[
{
"path": ".github/workflows/document_deploy.yml",
"content": "name: Deploy MkDocs site\n\non:\n push:\n branches:\n - main # 当推送到主分支时触发\n - vyokky/dev # 当推送到 vyokky_dev 分支时触发\n paths:\n - 'documents/**' # 当 docs 目录中的文件变化时触发\n\njobs:\n deploy:\n runs-on: ubuntu-latest\n permissions:\n contents: write\n\n steps:\n - name: Checkout repository\n uses: actions/checkout@v2\n\n - name: Set up Python\n uses: actions/setup-python@v2\n with:\n python-version: '3.9'\n\n - name: Install MkDocs and dependencies\n run: |\n pip install mkdocs mkdocs-material mkdocstrings mkdocstrings[python]\n\n - name: Deploy to GitHub Pages\n run: |\n cd documents\n mkdocs gh-deploy --config-file mkdocs.yml --force\n env:\n github_token: ${{ secrets.GITHUB_TOKEN }}"
},
{
"path": ".gitignore",
"content": "# Ignore login file\n*.bin\n\n# Ignore Jupyter Notebook checkpoints\n.ipynb_checkpoints\n/test/*\n/testing/*\n/deprecated/*\n/test/*.ipynb\n/logs/*\n/customization/*\n__pycache__/\n**/__pycache__/\n*.pyc\n*.ipynb\n/.VSCodeCounter\n/analysis/*\n/tla/*\n\n# Ignore the config file\nufo/config/config.yaml\nufo/config/config_llm.yaml\n\n\n# Ignore the helper files\nufo/rag/app_docs/*\nlearner/records.json\nvectordb/docs/*\nvectordb/experience/*\nvectordb/demonstration/*\n\n# Ignore the data files and scripts\ntasks/*\nscripts/*\nbackup/*\nnode_modules/*\n\n# Don't ignore the example files\n!vectordb/docs/example/\n!vectordb/demonstration/example.yaml\n\n.vscode\n\n# Ignore the record files\ntasks_status.json\ndatas\n_datas\ndatasUFO\n\n.venv/*\n\n# Ignore mkdocs build output\ndocuments/site/\n\n# Ignore frontend environment files with auto-generated content\ngalaxy/webui/frontend/.env.development.local\n\n# Ignore config files with sensitive data (API keys)\nconfig/*/agents.yaml\nconfig/*/agent.yaml\nufo/config/config.yaml\nufo/config/config_llm.yaml\n\n# But keep config templates and default configs (except agents.yaml and agent.yaml)\n!config/*/*.template\n!config/*/rag.yaml\n!config/*/system.yaml\n!config/*/mcp.yaml\n!config/*/prices.yaml\n!config/*/third_party.yaml\n!config/*/device.yaml\n!config/*/network.yaml\nnode_modules\n"
},
{
"path": "CODE_OF_CONDUCT.md",
"content": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n\nResources:\n\n- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)\n- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\n- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns\n"
},
{
"path": "CONTRIBUTING.md",
"content": "# Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to\nagree to a Contributor License Agreement (CLA) declaring that you have the right to,\nand actually do, grant us the rights to use your contribution. For details, visit\nhttps://cla.microsoft.com.\n\nWhen you submit a pull request, a CLA-bot will automatically determine whether you need\nto provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the\ninstructions provided by the bot. You will only need to do this once across all repositories using our CLA.\n\n## note\nYou should sunmit your pull request to the `pre-release` branch, not the `main` branch.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\nor contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments."
},
{
"path": "DISCLAIMER.md",
"content": "# Disclaimer: Code Execution and Data Handling Notice\n\nBy choosing to run the provided code, you acknowledge and agree to the following terms and conditions regarding the functionality and data handling practices:\n\n## 1. Code Functionality:\nThe code you are about to execute has the capability to capture screenshots of your working desktop environment and active applications. These screenshots will be processed and sent to the GPT model for inference.\n\n\n## 2. Data Privacy and Storage:\nIt is crucial to note that Microsoft, the provider of this code, explicitly states that it does not collect or save any of the transmitted data. The captured screenshots are processed in real-time for the purpose of inference, and no permanent storage or record of this data is retained by Microsoft.\n\n## 3. User Responsibility:\nBy running the code, you understand and accept the responsibility for the content and nature of the data present on your desktop during the execution period. It is your responsibility to ensure that no sensitive or confidential information is visible or captured during this process.\n\n## 4. Security Measures:\nMicrosoft has implemented security measures to safeguard the action execution. However, it is recommended that you run the code in a secure and controlled environment to minimize potential risks. Ensure that you are running the latest security updates on your system.\n\n## 5. Consent for Inference:\nYou explicitly provide consent for the GPT model to analyze the captured screenshots for the purpose of generating relevant outputs. This consent is inherent in the act of executing the code.\n\n## 6. No Guarantee of Accuracy:\nThe outputs generated by the GPT model are based on patterns learned during training and may not always be accurate or contextually relevant. Microsoft does not guarantee the accuracy or suitability of the inferences made by the model.\n\n## 7. Indemnification:\nUsers agree to defend, indemnify, and hold Microsoft harmless from and against all damages, costs, and attorneys' fees in connection with any claims arising from the use of this Repo.\n\n## 8. Reporting Infringements:\nIf anyone believes that this Repo infringes on their rights, please notify the project owner via the provided project owner email. Microsoft will investigate and take appropriate actions as necessary.\n\n## 9. Modifications to the Disclaimer:\nMicrosoft reserves the right to update or modify this disclaimer at any time without prior notice. It is your responsibility to review the disclaimer periodically for any changes.\n\nBy proceeding to execute the code, you acknowledge that you have read, understood, and agreed to the terms outlined in this disclaimer. If you do not agree with these terms, refrain from running the provided code."
},
{
"path": "LICENSE",
"content": "Copyright (c) Microsoft Corporation.\n\nMIT License\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
},
{
"path": "README.md",
"content": "\r\n\r\n
\r\n UFO³ : Weaving the Digital Agent Galaxy\r\n
\r\n
\r\n From Single Device Agent to Multi-Device Galaxy\r\n
\r\n\r\n#### 📊 Orchestration\r\n\r\n```\r\nTask1 → Running ✅\r\nTask2 → Pending ⏸️\r\nTask3 → Running 🔄\r\n ↓\r\n Completion\r\n ↓\r\n Final Report\r\n```\r\n\r\n**Orchestration:**\r\n- Real-time status updates\r\n- Automatic error recovery\r\n- Progress tracking with feedback\r\n\r\n
\r\n
\r\n
\r\n\r\n---\r\n\r\n### 🪟 UFO² Desktop AgentOS – Core Strengths\r\n\r\nUFO² serves dual roles: **standalone Windows automation** and **Galaxy device agent** for Windows platforms.\r\n\r\n
\r\n\r\n| Feature | Description | Documentation |\r\n|---------|-------------|---------------|\r\n| **Deep OS Integration** | Windows UIA, Win32, WinCOM native control | [Learn More](https://microsoft.github.io/UFO) |\r\n| **Hybrid Actions** | GUI clicks + API calls for optimal performance | [Learn More](https://microsoft.github.io/UFO/automator/overview) |\r\n| **Speculative Multi-Action** | Batch predictions → **51% fewer LLM calls** | [Learn More](https://microsoft.github.io/UFO/advanced_usage/multi_action) |\r\n| **Visual + UIA Detection** | Hybrid control detection for robustness | [Learn More](https://microsoft.github.io/UFO/advanced_usage/control_detection/hybrid_detection) |\r\n| **Knowledge Substrate** | RAG with docs, demos, execution traces | [Learn More](https://microsoft.github.io/UFO/advanced_usage/reinforce_appagent/overview/) |\r\n| **Device Agent Role** | Can serve as Windows executor in Galaxy orchestration | [Learn More](./galaxy/README.md) |\r\n\r\n
\r\n\r\n**As Galaxy Device Agent:**\r\n- Receives tasks from ConstellationAgent via Galaxy orchestration layer\r\n- Executes Windows-specific operations using proven UFO² capabilities\r\n- Reports status and results back to TaskOrchestrator\r\n- Participates in cross-device workflows seamlessly\r\n\r\n---\r\n\r\n## 🚀 Quick Start Guide\r\n\r\nChoose your path and follow the detailed setup guide:\r\n\r\n
\r\n
\r\n
\r\n\r\n### 🌌 Galaxy Quick Start\r\n\r\n**For cross-device orchestration**\r\n\r\n```powershell\r\n# 1. Install\r\npip install -r requirements.txt\r\n\r\n# 2. Configure ConstellationAgent\r\ncopy config\\galaxy\\agent.yaml.template config\\galaxy\\agent.yaml\r\n# Edit and add your API keys\r\n\r\n# 3. Configure devices\r\n# Edit config\\galaxy\\devices.yaml to register your devices\r\n\r\n# 4. Start device agents (with platform flags)\r\n# Windows: Start server + client\r\n# Linux: Start server + MCP servers + client \r\n# Mobile (Android): Start server + MCP servers + client\r\n# See platform-specific guides for detailed setup\r\n\r\n# 5. Launch Galaxy\r\npython -m galaxy --interactive\r\n```\r\n\r\n**📖 Complete Guide:**\r\n- [Galaxy README](./galaxy/README.md) – Architecture & concepts\r\n- [Online Quick Start](https://microsoft.github.io/UFO/getting_started/quick_start_galaxy/) – Step-by-step tutorial\r\n- [Windows Device Setup](https://microsoft.github.io/UFO/getting_started/quick_start_ufo2/)\r\n- [Linux Device Setup](https://microsoft.github.io/UFO/getting_started/quick_start_linux/)\r\n- [Mobile Device Setup](https://microsoft.github.io/UFO/getting_started/quick_start_mobile/) – Android agent setup\r\n- [Configuration](https://microsoft.github.io/UFO/configuration/system/galaxy_devices/) – Device pool configuration\r\n\r\n
\r\n
\r\n\r\n### 🪟 UFO² Quick Start\r\n\r\n**For Windows automation**\r\n\r\n```powershell\r\n# 1. Install\r\npip install -r requirements.txt\r\n\r\n# 2. Configure\r\ncopy config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\r\n# Edit and add your API keys\r\n\r\n# 3. Run\r\npython -m ufo --task \r\n```\r\n\r\n**📖 Complete Guide:**\r\n- [UFO² README](./ufo/README.md) – Full documentation\r\n- [Configuration Guide](./ufo/README.md#️-step-2-configure-the-llms) – LLM setup\r\n- [Advanced Features](https://microsoft.github.io/UFO/advanced_usage/overview/) – Multi-action, RAG\r\n\r\n
\r\n\r\n\r\n\r\n---\r\n\r\n## 📢 Latest Updates\r\n\r\n### 2025-11 – UFO³ Galaxy Framework Released 🌌\r\n**Major Research Breakthrough:** Multi-Device Orchestration System\r\n\r\n- 🌟 **Declarative DAG Decomposition**: TaskConstellation structure for workflow logic and dependencies\r\n- 🔄 **Dynamic Graph Evolution**: Living constellation that adapts through controlled rewrites\r\n- 🎯 **Heterogeneous Orchestration**: Safe, asynchronous execution with capability-based device matching\r\n- 🔌 **Unified AIP Protocol**: WebSocket-based secure agent coordination with fault tolerance\r\n- 🛠️ **MCP-Empowered Agent Framework**: Template-driven toolkit for rapid device agent development\r\n- 📄 **Research Paper**: [UFO³: Weaving the Digital Agent Galaxy](https://arxiv.org/abs/2511.11332)\r\n\r\n**Key Features:**\r\n- First multi-device orchestration framework for GUI agents\r\n- Result-driven adaptive execution instead of rigid workflows\r\n- Model Context Protocol (MCP) integration for tool augmentation\r\n- Formally verified correctness and concurrency safety guarantees\r\n\r\n### 2025-04 – UFO² v2.0.0\r\n- 📅 UFO² Desktop AgentOS released\r\n- 🏗️ Enhanced architecture with AgentOS concept\r\n- 📄 [Technical Report](https://arxiv.org/pdf/2504.14603) published\r\n- ✅ Entered Long-Term Support (LTS) status\r\n\r\n### 2024-02 – Original UFO\r\n- 🎈 First UFO release - UI-Focused agent for Windows\r\n- 📄 [Original Paper](https://arxiv.org/abs/2402.07939)\r\n- 🌍 Wide media coverage and adoption\r\n\r\n---\r\n\r\n## 📚 Citation\r\n\r\nIf you use UFO³ Galaxy or UFO² in your research, please cite the relevant papers:\r\n\r\n### UFO³ Galaxy Framework (2025)\r\n```bibtex\r\n@article{zhang2025ufo3,\r\n title={UFO$^3$: Weaving the Digital Agent Galaxy}, \r\n author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\r\n journal = {arXiv preprint arXiv:2511.11332},\r\n year = {2025},\r\n}\r\n```\r\n\r\n### UFO² Desktop AgentOS (2025)\r\n```bibtex\r\n@article{zhang2025ufo2,\r\n title = {{UFO2: The Desktop AgentOS}},\r\n author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\r\n journal = {arXiv preprint arXiv:2504.14603},\r\n year = {2025}\r\n}\r\n```\r\n\r\n### Original UFO (2024)\r\n```bibtex\r\n@article{zhang2024ufo,\r\n title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},\r\n author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},\r\n journal = {arXiv preprint arXiv:2402.07939},\r\n year = {2024}\r\n}\r\n```\r\n\r\n---\r\n\r\n## 🌐 Media & Community\r\n\r\n**Media Coverage:**\r\n- [微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」](https://www.jiqizhixin.com/articles/2025-05-06-13)\r\n- [Microsoft's UFO: Smarter Windows Experience](https://the-decoder.com/microsofts-ufo-abducts-traditional-user-interfaces-for-a-smarter-windows-experience/)\r\n- [下一代Windows系统曝光](https://baijiahao.baidu.com/s?id=1790938358152188625)\r\n- **[More coverage →](./ufo/README.md#-tracing-the-stars)**\r\n\r\n**Community:**\r\n- 💬 [GitHub Discussions](https://github.com/microsoft/UFO/discussions)\r\n- 🐛 [Issue Tracker](https://github.com/microsoft/UFO/issues)\r\n- 📧 Email: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)\r\n- 📺 [YouTube Channel](https://www.youtube.com/watch?v=QT_OhygMVXU)\r\n\r\n---\r\n\r\n## 🎨 Related Projects & Research\r\n\r\n**Microsoft Research:**\r\n- **[TaskWeaver](https://github.com/microsoft/TaskWeaver)** – Code-first LLM agent framework for data analytics and task automation\r\n\r\n**GUI Agent Research:**\r\n- **[LLM-Brained GUI Agents Survey](https://github.com/vyokky/LLM-Brained-GUI-Agents-Survey)** – Comprehensive survey of GUI automation agents\r\n- **[Interactive Survey Site](https://vyokky.github.io/LLM-Brained-GUI-Agents-Survey/)** – Explore latest GUI agent research and developments\r\n\r\n**Multi-Agent Systems:**\r\n- **UFO³ Galaxy** represents a novel approach to multi-device orchestration, introducing the Constellation framework for coordinating heterogeneous agents across platforms\r\n- Builds on multi-agent coordination research while addressing unique challenges of cross-device GUI automation\r\n\r\n**Benchmarks:**\r\n- **[Windows Agent Arena (WAA)](https://github.com/nice-mee/WindowsAgentArena)** – Evaluation benchmark for Windows automation agents\r\n- **[OSWorld](https://github.com/nice-mee/WindowsAgentArena/tree/2020-qqtcg/osworld)** – Cross-application task evaluation suite\r\n\r\n---\r\n\r\n## 💡 FAQ\r\n\r\n\r\n🤔 Should I use Galaxy or UFO²?\r\n\r\n**Start with UFO²** if:\r\n- You only need Windows automation\r\n- You want quick setup and learning\r\n- Tasks are relatively simple\r\n\r\n**Choose Galaxy** if:\r\n- You need cross-device coordination\r\n- Tasks are complex and multi-step\r\n- You want advanced orchestration\r\n- You're comfortable with active development\r\n\r\n**Hybrid approach** if:\r\n- You want best of both worlds\r\n- Some tasks are simple (UFO²), some complex (Galaxy)\r\n- You're gradually migrating\r\n\r\n\r\n\r\n\r\n⚠️ Will UFO² be deprecated?\r\n\r\n**No!** UFO² has entered **Long-Term Support (LTS)** status:\r\n- ✅ Actively maintained\r\n- ✅ Bug fixes and security updates\r\n- ✅ Performance improvements\r\n- ✅ Full community support\r\n- ✅ No plans for deprecation\r\n\r\nUFO² is the stable, proven solution for Windows automation.\r\n\r\n\r\n\r\n\r\n🔄 How do I migrate from UFO² to Galaxy?\r\n\r\nMigration is **gradual and optional**:\r\n\r\n1. **Phase 1: Learn** – Understand Galaxy concepts\r\n2. **Phase 2: Experiment** – Try Galaxy with non-critical tasks\r\n3. **Phase 3: Hybrid** – Use both frameworks\r\n4. **Phase 4: Migrate** – Gradually move complex tasks to Galaxy\r\n\r\n**No forced migration!** Continue using UFO² as long as it meets your needs.\r\n\r\nSee [Migration Guide](./documents/docs/getting_started/migration_ufo2_to_galaxy.md) for details.\r\n\r\n\r\n\r\n\r\n🎯 Can Galaxy do everything UFO² does?\r\n\r\n**Functionally: Yes.** Galaxy can use UFO² as a Windows device agent.\r\n\r\n**Practically: It depends.**\r\n- For **simple Windows tasks**: UFO² standalone is easier and more streamlined\r\n- For **complex workflows**: Galaxy orchestrates UFO² with other device agents\r\n\r\n**Recommendation:** Use the right tool for the job. UFO² can work standalone or as Galaxy's Windows device agent.\r\n\r\n\r\n\r\n\r\n📊 How mature is Galaxy?\r\n\r\n**Status: Active Development** 🚧\r\n\r\n**Stable:**\r\n- ✅ Core architecture\r\n- ✅ DAG orchestration\r\n- ✅ Basic multi-device support\r\n- ✅ Event system\r\n\r\n**In Development:**\r\n- 🔨 Advanced device types\r\n- 🔨 Enhanced monitoring\r\n- 🔨 Performance optimization\r\n- 🔨 Extended documentation\r\n\r\n**Recommendation:** Great for experimentation and non-critical workflows.\r\n\r\n\r\n\r\n\r\n🔧 Can I extend or customize?\r\n\r\n**Both frameworks are highly extensible:**\r\n\r\n**UFO²:**\r\n- Custom actions and automators\r\n- Custom knowledge sources (RAG)\r\n- Custom control detectors\r\n- Custom evaluation metrics\r\n\r\n**Galaxy:**\r\n- Custom agents\r\n- Custom device types\r\n- Custom orchestration strategies\r\n- Custom visualization components\r\n\r\nSee respective documentation for extension guides.\r\n\r\n\r\n\r\n\r\n🤝 How can I contribute?\r\n\r\nWe welcome contributions to both UFO² and Galaxy!\r\n\r\n**Ways to contribute:**\r\n- 🐛 Report bugs and issues\r\n- 💡 Suggest features and improvements\r\n- 📝 Improve documentation\r\n- 🧪 Add tests and examples\r\n- 🔧 Submit pull requests\r\n\r\nSee [CONTRIBUTING.md](./CONTRIBUTING.md) for guidelines.\r\n\r\n\r\n\r\n\r\n\r\n---\r\n\r\n## ⚠️ Disclaimer & License\r\n\r\n**Disclaimer:** By using this software, you acknowledge and agree to the terms in [DISCLAIMER.md](./DISCLAIMER.md).\r\n\r\n**License:** This project is licensed under the [MIT License](LICENSE).\r\n\r\n**Trademarks:** Use of Microsoft trademarks follows [Microsoft's Trademark Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).\r\n\r\n---\r\n\r\n
\n\n\n# Introduction\n\nThis repository contains the implementation of the **Data Collection** process for training the **Large Action Models** (LAMs) in the [**paper**](https://arxiv.org/abs/2412.10047). The **Data Collection** process is designed to streamline task processing, ensuring that all necessary steps are seamlessly integrated from initialization to execution. This module is part of the [**UFO**](https://arxiv.org/abs/2402.07939) project.\n\nIf you find this project useful, please give a star ⭐, and consider to cite our paper:\n\n```bibtex\n@misc{wang2024largeactionmodelsinception,\n title={Large Action Models: From Inception to Implementation}, \n author={Lu Wang and Fangkai Yang and Chaoyun Zhang and Junting Lu and Jiaxu Qian and Shilin He and Pu Zhao and Bo Qiao and Ray Huang and Si Qin and Qisheng Su and Jiayi Ye and Yudi Zhang and Jian-Guang Lou and Qingwei Lin and Saravan Rajmohan and Dongmei Zhang and Qi Zhang},\n year={2024},\n eprint={2412.10047},\n archivePrefix={arXiv},\n primaryClass={cs.AI},\n url={https://arxiv.org/abs/2412.10047}, \n}\n```\n\n\n\n# Dataflow\n\nDataflow uses UFO to implement `instantiation`, `execution`, and `dataflow` for a given task, with options for batch processing and single processing.\n\n1. **Instantiation**: Instantiation refers to the process of setting up and preparing a task for execution. This step typically involves `choosing template`, `prefill` and `filter`.\n2. **Execution**: Execution is the actual process of running the task. This step involves carrying out the actions or operations specified by the `Instantiation`. And after execution, an evaluate agent will evaluate the quality of the whole execution process.\n3. **Dataflow**: Dataflow is the overarching process that combines **instantiation** and **execution** into a single pipeline. It provides an end-to-end solution for processing tasks, ensuring that all necessary steps (from initialization to execution) are seamlessly integrated.\n\nYou can use `instantiation` and `execution` independently if you only need to perform one specific part of the process. When both steps are required for a task, the `dataflow` process streamlines them, allowing you to execute tasks from start to finish in a single pipeline.\n\nThe overall processing of dataflow is as below. Given a task-plan data, the LLMwill instantiatie the task-action data, including choosing template, prefill, filter.\n\n
\n \n
\n\n## How To Use\n\n### 1. Install Packages\n\nYou should install the necessary packages in the UFO root folder:\n\n```bash\npip install -r requirements.txt\n```\n\n### 2. Configure the LLMs\n\nBefore running dataflow, you need to provide your LLM configurations **individually for PrefillAgent and FilterAgent**. You can create your own config file `dataflow/config/config.yaml`, by copying the `dataflow/config/config.yaml.template` and editing config for **PREFILL_AGENT** and **FILTER_AGENT** as follows:\n\n#### OpenAI\n\n```bash\nVISUAL_MODE: True, # Whether to use the visual mode\nAPI_TYPE: \"openai\" , # The API type, \"openai\" for the OpenAI API. \nAPI_BASE: \"https://api.openai.com/v1/chat/completions\", # The the OpenAI API endpoint.\nAPI_KEY: \"sk-\", # The OpenAI API key, begin with sk-\nAPI_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\nAPI_MODEL: \"gpt-4-vision-preview\", # The only OpenAI model\n```\n\n#### Azure OpenAI (AOAI)\n\n```bash\nVISUAL_MODE: True, # Whether to use the visual mode\nAPI_TYPE: \"aoai\" , # The API type, \"aoai\" for the Azure OpenAI. \nAPI_BASE: \"YOUR_ENDPOINT\", # The AOAI API address. Format: https://{your-resource-name}.openai.azure.com\nAPI_KEY: \"YOUR_KEY\", # The aoai API key\nAPI_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\nAPI_MODEL: \"gpt-4-vision-preview\", # The only OpenAI model\nAPI_DEPLOYMENT_ID: \"YOUR_AOAI_DEPLOYMENT\", # The deployment id for the AOAI API\n```\n\nYou can also non-visial model (e.g., GPT-4) for each agent, by setting `VISUAL_MODE: False` and proper `API_MODEL` (openai) and `API_DEPLOYMENT_ID` (aoai).\n\n#### Non-Visual Model Configuration\n\nYou can utilize non-visual models (e.g., GPT-4) for each agent by configuring the following settings in the `config.yaml` file:\n\n- ``VISUAL_MODE: False # To enable non-visual mode.``\n- Specify the appropriate `API_MODEL` (OpenAI) and `API_DEPLOYMENT_ID` (AOAI) for each agent.\n\nEnsure you configure these settings accurately to leverage non-visual models effectively.\n\n#### Other Configurations\n\n`config_dev.yaml` specifies the paths of relevant files and contains default settings. The match strategy for the window match and control filter supports options: `'contains'`, `'fuzzy'`, and `'regex'`, allowing flexible matching strategy for users. The `MAX_STEPS` is the max step for the execute_flow, which can be set by users.\n\n#### NOTE 💡\n\n**BE CAREFUL!** If you are using GitHub or other open-source tools, do not expose your `config.yaml` online, as it contains your private keys.\n\n### 3. Prepare Files\n\nCertain files need to be prepared before running the task.\n\n#### 3.1. Tasks as JSON\n\nThe tasks that need to be instantiated should be organized in a folder of JSON files, with the default folder path set to dataflow `/tasks `. This path can be changed in the `dataflow/config/config.yaml` file, or you can specify it in the terminal, as mentioned in **4. Start Running**. For example, a task stored in `dataflow/tasks/prefill/` may look like this:\n\n```json\n{\n // The app you want to use\n \"app\": \"word\",\n // A unique ID to distinguish different tasks \n \"unique_id\": \"1\",\n // The task and steps to be instantiated\n \"task\": \"Type 'hello' and set the font type to Arial\",\n \"refined_steps\": [\n \"Type 'hello'\",\n \"Set the font to Arial\"\n ]\n}\n```\n\n#### 3.2. Templates and Descriptions\n\nYou should place an app file as a reference for instantiation in a folder named after the app.\n\nFor example, if you have `template1.docx` for Word, it should be located at `dataflow/templates/word/template1.docx`.\n\nAdditionally, for each app folder, there should be a `description.json` file located at `dataflow/templates/word/description.json`, which describes each template file in detail. It may look like this:\n\n```json\n{\n \"template1.docx\": \"A document with a rectangle shape\",\n \"template2.docx\": \"A document with a line of text\"\n}\n```\n\nIf a `description.json` file is not present, one template file will be selected at random.\n\n#### 3.3. Final Structure\n\nEnsure the following files are in place:\n\n- [X] JSON files to be instantiated\n- [X] Templates as references for instantiation\n- [X] Description file in JSON format\n\nThe structure of the files can be:\n\n```txt\ndataflow/\n|\n├── tasks\n│ └── prefill\n│ ├── bulleted.json\n│ ├── delete.json\n│ ├── draw.json\n│ ├── macro.json\n│ └── rotate.json\n├── templates\n│ └── word\n│ ├── description.json\n│ ├── template1.docx\n│ ├── template2.docx\n│ ├── template3.docx\n│ ├── template4.docx\n│ ├── template5.docx\n│ ├── template6.docx\n│ └── template7.docx\n└── ...\n```\n\n### 4. Start Running\n\nAfter finishing the previous steps, you can use the following commands in the command line. We provide single / batch process, for which you need to give the single file path / folder path. Determine the type of path provided by the user and automatically decide whether to process a single task or batch tasks.\n\nAlso, you can choose to use `instantiation` / `execution` sections individually, or use them as a whole section, which is named as `dataflow`.\n\nThe default task hub is set to be `\"TASKS_HUB\"` in `dataflow/config_dev.yaml`.\n\nYou can use `\"TEMPLATE_METHOD\"` in `dataflow/config_dev.yaml` to choose `LLM` or `SemanticSimilarity` as the backend for the template selection function. If you choose `LLM`, since the visual version is being used, you need to manually generate screenshots in the `templates/\"YOUR_APP\"/images` directory, and the filenames should match the template name and the screenshots should in `PNG` format.\n\n* Dataflow Task:\n\n ```bash\n python -m dataflow --dataflow --task_path path_to_task_file\n ```\n\n* Instantiation Task:\n\n ```bash\n python -m dataflow --instantiation --task_path path_to_task_file\n ```\n* Execution Task:\n\n ```bash\n python -m dataflow --execution --task_path path_to_task_file\n ```\n\n## Workflow\n\n### Instantiation\n\nThere are three key steps in the instantiation process:\n\n1. `Choose a template` file according to the specified app and instruction.\n2. `Prefill` the task using the current screenshot.\n3. `Filter` the established task.\n\nGiven the initial task, the dataflow first choose a template (`Phase 1`), the prefill the initial task based on word envrionment to obtain task-action data (`Phase 2`). Finnally, it will filter the established task to evaluate the quality of task-action data.\n\n
\n \n
\n\n#### 1. Choose Template File\n\nTemplates for your app must be defined and described in `dataflow/templates/app`. For instance, if you want to instantiate tasks for the Word application, place the relevant `.docx` files in dataflow `/templates/word `, along with a `description.json` file.\n\nThe appropriate template will be selected based on how well its description matches the instruction.\n\n#### 2. Prefill the Task\n\nAfter selecting the template file, it will be opened, and a screenshot will be taken. If the template file is currently in use, errors may occur.\n\nThe screenshot will be sent to the action prefill agent, which will return a modified task.\n\n#### 3. Filter Task\n\nThe completed task will be evaluated by a filter agent, which will assess it and provide feedback.\n\n### Execution\n\nThe instantiated plans will be executed by a execute task. After execution, evalution agent will evaluation the quality of the entire execution process.\n\nIn this phase, given the task-action data, the execution process will match the real controller based on word environment and execute the plan step by step.\n\n
\n \n
\n\n\n## Result\n\nThe structure of the results of the task is as below:\n\n```txt\nUFO/\n├── dataflow/ # Root folder for dataflow\n│ └── results/ # Directory for storing task processing results\n│ ├── saved_document/ # Directory for final document results\n│ ├── instantiation/ # Directory for instantiation results\n│ │ ├── instantiation_pass/ # Tasks successfully instantiated\n│ │ └── instantiation_fail/ # Tasks that failed instantiation\n│ ├── execution/ # Directory for execution results\n│ │ ├── execution_pass/ # Tasks successfully executed\n│ │ ├── execution_fail/ # Tasks that failed execution\n│ │ └── execution_unsure/ # Tasks with uncertain execution results\n│ ├── dataflow/ # Directory for dataflow results\n│ │ ├── execution_pass/ # Tasks successfully executed\n│ │ ├── execution_fail/ # Tasks that failed execution\n│ │ └── execution_unsure/ # Tasks with uncertain execution results\n│ └── ...\n└── ...\n```\n\n1. **General Description:**\n\n This directory structure organizes the results of task processing into specific categories, including instantiation, execution, and dataflow outcomes.\n2. **Instantiation:**\n\n The `instantiation` directory contains subfolders for tasks that were successfully instantiated (`instantiation_pass`) and those that failed during instantiation (`instantiation_fail`).\n3. **Execution:**\n\n Results of task execution are stored under the `execution` directory, categorized into successful tasks (`execution_pass`), failed tasks (`execution_fail`), and tasks with uncertain outcomes (`execution_unsure`).\n4. **Dataflow Results:**\n\n The `dataflow` directory similarly holds results of tasks based on execution success, failure, or uncertainty, providing a comprehensive view of the data processing pipeline.\n5. **Saved Documents:**\n\n Instantiated results are separately stored in the `saved_document` directory for easy access and reference.\n\n### Description\n\nhis section illustrates the structure of the result of the task, organized in a hierarchical format to describe the various fields and their purposes. The result data include `unique_id`，``app``, `original`, `execution_result`, `instantiation_result`, `time_cost`.\n\n#### 1. **Field Descriptions**\n\n- **Hierarchy**: The data is presented in a hierarchical manner to allow for a clearer understanding of field relationships.\n- **Type Description**: The type of each field (e.g., `string`, `array`, `object`) clearly specifies the format of the data.\n- **Field Purpose**: Each field has a brief description outlining its function.\n\n#### 2. **Execution Results and Errors**\n\n- **execution_result**: Contains the results of task execution, including subtask performance, completion status, and any encountered errors.\n- **instantiation_result**: Describes the process of task instantiation, including template selection, prefilled tasks, and instantiation evaluation.\n- **error**: If an error occurs during task execution, this field will contain the relevant error information.\n\n#### 3. **Time Consumption**\n\n- **time_cost**: The time spent on each phase of the task, from template selection to task execution, is recorded to analyze task efficiency.\n\n### Example Data\n\n```json\n{\n \"unique_id\": \"102\",\n \"app\": \"word\",\n \"original\": {\n \"original_task\": \"Find which Compatibility Mode you are in for Word\",\n \"original_steps\": [\n \"1.Click the **File** tab.\",\n \"2.Click **Info**.\",\n \"3.Check the **Compatibility Mode** indicator at the bottom of the document preview pane.\"\n ]\n },\n \"execution_result\": {\n \"result\": {\n \"reason\": \"The agent successfully identified the compatibility mode of the Word document.\",\n \"sub_scores\": {\n \"correct identification of compatibility mode\": \"yes\"\n },\n \"complete\": \"yes\"\n },\n \"error\": null\n },\n \"instantiation_result\": {\n \"choose_template\": {\n \"result\": \"dataflow\\\\results\\\\saved_document\\\\102.docx\",\n \"error\": null\n },\n \"prefill\": {\n \"result\": {\n \"instantiated_request\": \"Identify the Compatibility Mode of the Word document.\",\n \"instantiated_plan\": [\n {\n \"Step\": 1,\n \"Subtask\": \"Identify the Compatibility Mode\",\n \"Function\": \"summary\",\n \"Args\": {\n \"text\": \"The document is in '102 - Compatibility Mode'.\"\n },\n \"Success\": true\n }\n ]\n },\n \"error\": null\n },\n \"instantiation_evaluation\": {\n \"result\": {\n \"judge\": true,\n \"thought\": \"Identifying the Compatibility Mode of a Word document is a task that can be executed locally within Word.\"\n },\n \"error\": null\n }\n },\n \"time_cost\": {\n \"choose_template\": 0.017,\n \"prefill\": 11.304,\n \"instantiation_evaluation\": 2.38,\n \"total\": 34.584,\n \"execute\": 0.946,\n \"execute_eval\": 10.381\n }\n}\n```\n\n## Quick Start\n\nWe prepare two cases to show the dataflow, which can be found in `dataflow\\tasks\\prefill`. So after installing required packages, you can type the following command in the command line:\n\n```\npython -m dataflow -dataflow\n```\n\nAnd you can see the hints showing in the terminal, which means the dataflow is working.\n\n### Structure of related files\n\nAfter the two tasks are finished, the task and output files would appear as follows:\n\n```bash\nUFO/\n├── dataflow/\n│ └── results/\n│ ├── saved_document/ \t# Directory for saved documents\n│ │ ├── bulleted.docx \t# Result of the \"bulleted\" task\n│ │ └── rotate.docx \t# Result of the \"rotate\" task\n│ ├── dataflow/ \t\t # Dataflow results directory\n│ │ ├── execution_pass/ \t# Successfully executed tasks\n│ │ │ ├── bulleted.json \t# Execution result for the \"bulleted\" task\n│ │ │ ├── rotate.json \t # Execution result for the \"rotate\" task\n│ │ │ └── ...\n└── ...\n```\n\n### Result files\n\nThe result stucture of bulleted task is shown as below. This document provides a detailed breakdown of the task execution process for turning lines of text into a bulleted list in Word. It includes the original task description, execution results, and time analysis for each step.\n\n* **`unique_id`** : The identifier for the task, in this case, `\"5\"`.\n* **`app`** : The application being used, which is `\"word\"`.\n* **`original`** : Contains the original task description and the steps.\n\n * **`original_task`** : Describes the task in simple terms (turning text into a bulleted list).\n * **`original_steps`** : Lists the steps required to perform the task.\n* **`execution_result`** : Provides the result of executing the task.\n\n * **`result`** : Describes the outcome of the execution, including a success message and sub-scores for each part of the task. The `complete: \"yes\"` means the evaluation agent think the execution process is successful! The `sub_score` is the evaluation of each subtask, corresponding to the ` instantiated_plan` in the `prefill`.\n * **`error`** : If any error occurred during execution, it would be reported here, but it's `null` in this case.\n* **`instantiation_result`** : Details the instantiation of the task (setting up the task for execution).\n\n * **`choose_template`** : Path to the template or document created during the task (in this case, the bulleted list document).\n * **`prefill`** : Describes the `instantiated_request` and `instantiated_plan` and the steps involved, such as selecting text and clicking buttons, which is the result of prefill flow. The `Success` and `MatchedControlText` is added in the execution process. **`Success`** indicates whether the subtask was executed successfully. **`MatchedControlText`** refers to the control text that was matched during the execution process based on the plan.\n * **`instantiation_evaluation`** : Provides feedback on the task's feasibility and the evaluation of the request, which is result of the filter flow. **`\"judge\": true`** : This indicates that the evaluation of the task was positive, meaning the task is considered valid or successfully judged. And the `thought ` is the detailed reason.\n* **`time_cost`** : The time spent on different parts of the task, including template selection, prefill, instantiation evaluation, and execution. Total time is also given.\n\nThis structure follows your description and provides the necessary details in a consistent format.\n\n```json\n{\n \"unique_id\": \"5\",\n \"app\": \"word\",\n \"original\": {\n \"original_task\": \"Turning lines of text into a bulleted list in Word\",\n \"original_steps\": [\n \"1. Place the cursor at the beginning of the line of text you want to turn into a bulleted list\",\n \"2. Click the Bullets button in the Paragraph group on the Home tab and choose a bullet style\"\n ]\n },\n \"execution_result\": {\n \"result\": {\n \"reason\": \"The agent successfully selected the text 'text to edit' and then clicked on the 'Bullets' button in the Word application. The final screenshot shows that the text 'text to edit' has been converted into a bulleted list.\",\n \"sub_scores\": {\n \"text selection\": \"yes\",\n \"bulleted list conversion\": \"yes\"\n },\n \"complete\": \"yes\"\n },\n \"error\": null\n },\n \"instantiation_result\": {\n \"choose_template\": {\n \"result\": \"dataflow\\\\results\\\\saved_document\\\\bulleted.docx\",\n \"error\": null\n },\n \"prefill\": {\n \"result\": {\n \"instantiated_request\": \"Turn the line of text 'text to edit' into a bulleted list in Word.\",\n \"instantiated_plan\": [\n {\n \"Step\": 1,\n \"Subtask\": \"Place the cursor at the beginning of the text 'text to edit'\",\n \"ControlLabel\": null,\n \"ControlText\": \"\",\n \"Function\": \"select_text\",\n \"Args\": {\n \"text\": \"text to edit\"\n },\n \"Success\": true,\n \"MatchedControlText\": null\n },\n {\n \"Step\": 2,\n \"Subtask\": \"Click the Bullets button in the Paragraph group on the Home tab\",\n \"ControlLabel\": \"61\",\n \"ControlText\": \"Bullets\",\n \"Function\": \"click_input\",\n \"Args\": {\n \"button\": \"left\",\n \"double\": false\n },\n \"Success\": true,\n \"MatchedControlText\": \"Bullets\"\n }\n ]\n },\n \"error\": null\n },\n \"instantiation_evaluation\": {\n \"result\": {\n \"judge\": true,\n \"thought\": \"The task is specific and involves a basic function in Word that can be executed locally without any external dependencies.\",\n \"request_type\": \"None\"\n },\n \"error\": null\n }\n },\n \"time_cost\": {\n \"choose_template\": 0.012,\n \"prefill\": 15.649,\n \"instantiation_evaluation\": 2.469,\n \"execute\": 5.824,\n \"execute_eval\": 8.702,\n \"total\": 43.522\n }\n}\n```\n\n### Log files\n\nThe corresponding logs can be found in the directories `logs/bulleted` and `logs/rotate`, as shown below. Detailed logs for each workflow are recorded, capturing every step of the execution process.\n\n
\n \n
\n\n## Notes\n\n1. Users should be careful to save the original files while using this project; otherwise, the files will be closed when the app is shut down.\n2. After starting the project, users should not close the app window while the program is taking screenshots.\n"
},
{
"path": "dataflow/__main__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\nfrom dataflow import dataflow\n\nif __name__ == \"__main__\":\n # Execute the main script\n dataflow.main()\n"
},
{
"path": "dataflow/config/config.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom ufo.config import Config\n\n\nclass Config(Config):\n _instance = None\n\n def __init__(self, config_path=\"dataflow/config/\"):\n \"\"\"\n Initializes the Config class.\n :param config_path: The path to the config file.\n \"\"\"\n\n self.config_data = self.load_config(config_path)\n\n @staticmethod\n def get_instance():\n \"\"\"\n Get the instance of the Config class.\n :return: The instance of the Config class.\n \"\"\"\n\n if Config._instance is None:\n Config._instance = Config()\n\n return Config._instance\n\n def optimize_configs(self, configs):\n \"\"\"\n Optimize the configurations.\n :param configs: The configurations to optimize.\n :return: The optimized configurations.\n \"\"\"\n \n self.update_api_base(configs, \"PREFILL_AGENT\")\n self.update_api_base(configs, \"FILTER_AGENT\")\n\n return configs\n"
},
{
"path": "dataflow/config/config.yaml.template",
"content": "PREFILL_AGENT: {\n VISUAL_MODE: True, # Whether to use the visual mode\n\n API_TYPE: \"openai\" , # The API type, \"openai\" for the OpenAI API, \"aoai\" for the AOAI API, 'azure_ad' for the ad authority of the AOAI API. \n API_BASE: \"https://api.openai.com/v1/chat/completions\", # The the OpenAI API endpoint, \"https://api.openai.com/v1/chat/completions\" for the OpenAI API.\n API_KEY: \"sk-\", # The OpenAI API key, begin with sk-\n API_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\n API_MODEL: \"gpt-4-vision-preview\", # The only OpenAI model by now that accepts visual input\n\n\n ### Comment above and uncomment these if using \"aoai\".\n # API_TYPE: \"aoai\" , # The API type, \"openai\" for the OpenAI API, \"aoai\" for the Azure OpenAI. \n # API_BASE: \"YOUR_ENDPOINT\", # The the OpenAI API endpoint, \"https://api.openai.com/v1/chat/completions\" for the OpenAI API. As for the aoai, it should be https://{your-resource-name}.openai.azure.com\n # API_KEY: \"YOUR_KEY\", # The aoai API key\n # API_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\n # API_MODEL: \"YOUR_MODEL\", # The only OpenAI model by now that accepts visual input\n # API_DEPLOYMENT_ID: \"gpt-4-visual-preview\", # The deployment id for the AOAI API\n \n ### For Azure_AD\n # AAD_TENANT_ID: \"YOUR_TENANT_ID\", # Set the value to your tenant id for the llm model\n # AAD_API_SCOPE: \"YOUR_SCOPE\", # Set the value to your scope for the llm model\n # AAD_API_SCOPE_BASE: \"YOUR_SCOPE_BASE\" # Set the value to your scope base for the llm model, whose format is API://YOUR_SCOPE_BASE, and the only need is the YOUR_SCOPE_BASE\n}\n\nFILTER_AGENT: {\n VISUAL_MODE: True, # Whether to use the visual mode\n\n API_TYPE: \"openai\" , # The API type, \"openai\" for the OpenAI API, \"aoai\" for the AOAI API, 'azure_ad' for the ad authority of the AOAI API. \n API_BASE: \"https://api.openai.com/v1/chat/completions\", # The the OpenAI API endpoint, \"https://api.openai.com/v1/chat/completions\" for the OpenAI API.\n API_KEY: \"sk-\", # The OpenAI API key, begin with sk-\n API_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\n API_MODEL: \"gpt-4-vision-preview\", # The only OpenAI model by now that accepts visual input\n\n\n ### Comment above and uncomment these if using \"aoai\".\n # API_TYPE: \"aoai\" , # The API type, \"openai\" for the OpenAI API, \"aoai\" for the Azure OpenAI. \n # API_BASE: \"YOUR_ENDPOINT\", # The the OpenAI API endpoint, \"https://api.openai.com/v1/chat/completions\" for the OpenAI API. As for the aoai, it should be https://{your-resource-name}.openai.azure.com\n # API_KEY: \"YOUR_KEY\", # The aoai API key\n # API_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\n # API_MODEL: \"YOUR_MODEL\", # The only OpenAI model by now that accepts visual input\n # API_DEPLOYMENT_ID: \"gpt-4-visual-preview\", # The deployment id for the AOAI API\n \n ### For Azure_AD\n # AAD_TENANT_ID: \"YOUR_TENANT_ID\", # Set the value to your tenant id for the llm model\n # AAD_API_SCOPE: \"YOUR_SCOPE\", # Set the value to your scope for the llm model\n # AAD_API_SCOPE_BASE: \"YOUR_SCOPE_BASE\" # Set the value to your scope base for the llm model, whose format is API://YOUR_SCOPE_BASE, and the only need is the YOUR_SCOPE_BASE\n }\n\n\n### For parameters\nMAX_TOKENS: 2000 # The max token limit for the response completion\nMAX_RETRY: 3 # The max retry limit for the response completion\nTEMPERATURE: 0.0 # The temperature of the model: the lower the value, the more consistent the output of the model\nTOP_P: 0.0 # The top_p of the model: the lower the value, the more conservative the output of the model\nTIMEOUT: 60 # The call timeout(s), default is 10 minss"
},
{
"path": "dataflow/config/config_dev.yaml",
"content": "version: 0.1\n\nCONTROL_BACKEND: \"uia\" # The list of backend for control action, currently we support uia and win32, \nCONTROL_LIST: [\"Button\", \"Edit\", \"TabItem\", \"Document\", \"ListItem\", \"MenuItem\", \"ScrollBar\", \"TreeItem\", \"Hyperlink\", \"ComboBox\", \"RadioButton\", \"DataItem\", \"Spinner\"] \nPRINT_LOG: False # Whether to print the log \nLOG_LEVEL: \"INFO\" # The log level\nMATCH_STRATEGY: \"fuzzy\" # The match strategy for the control filter, support 'contains', 'fuzzy', 'regex'\n\nPREFILL_PROMPT: \"dataflow/prompts/instantiation/{mode}/prefill.yaml\" # The prompt for the action prefill\nFILTER_PROMPT: \"dataflow/prompts/instantiation/{mode}/filter.yaml\" # The prompt for the filter\nPREFILL_EXAMPLE_PROMPT: \"dataflow/prompts/instantiation/{mode}/prefill_example.yaml\" # The prompt for the action prefill example\nAPI_PROMPT: \"ufo/prompts/share/lite/api.yaml\" # The prompt for the API\n\n# Template Configuration\nTEMPLATE_METHOD: \"LLM\" # The method for the template, support 'SemanticSimilarity', 'LLM'.\nTEMPLATE_PROMPT: \"dataflow/prompts/instantiation/{mode}/template.yaml\" # The prompt for the template\n\n# Reformat Configuration\nREFORMAT_TO_BATCH: True # Whether to reformat the result of dataflow to the format of the UFO batch mode\nREFORMAT_TO_BATCH_HUB: \"datasUFO\" # The reformat result path\n\n# Default Task Configuration\nTASKS_HUB: \"dataflow/tasks/prefill\" # The default tasks hub for batch dataflow\nTEMPLATE_PATH: \"dataflow/templates\" # The template path for the exploration\n\n# Result Configuration\nRESULT_HUB: \"dataflow/results/{task_type}\" # The result hub, task_type is 'instantiation' or 'execution'\nINSTANTIATION_RESULT_SCHEMA: \"dataflow/schema/instantiation_schema.json\" # The JSON Schema for the result log\nEXECUTION_RESULT_SCHEMA: \"dataflow/schema/execution_schema.json\"\n\n# For control filtering\nCONTROL_FILTER_TYPE: [] # The list of control filter type, support 'TEXT', 'SEMANTIC', 'ICON'\nCONTROL_FILTER_MODEL_SEMANTIC_NAME: \"all-MiniLM-L6-v2\" # The control filter model name of semantic similarity\nCONTROL_EMBEDDING_CACHE_PATH: \"dataflow/cache/\" # The cache path for the control filter\nCONTROL_FILTER_TOP_K_PLAN: 2 # The control filter effect on top k plans from UFO, default is 2\n\n# log path\nLOG_PATH: \"dataflow/logs/{task}\"\nPREFILL_LOG_PATH: \"dataflow/logs/{task}/prefill/\"\nFILTER_LOG_PATH: \"dataflow/logs/{task}/filter/\"\nEXECUTE_LOG_PATH: \"dataflow/logs/{task}/execute/\"\n\nMAX_STEPS: 30 # The max step for the execute_flow\n"
},
{
"path": "dataflow/data_flow_controller.py",
"content": "import os\nimport time\nimport traceback\nfrom enum import Enum\nfrom typing import Any, Dict, Optional, List\nfrom jsonschema import validate, ValidationError\nimport shutil\n\nfrom dataflow.env.env_manager import WindowsAppEnv\nfrom dataflow.instantiation.workflow.choose_template_flow import ChooseTemplateFlow\nfrom dataflow.instantiation.workflow.prefill_flow import PrefillFlow\nfrom dataflow.instantiation.workflow.filter_flow import FilterFlow\nfrom dataflow.execution.workflow.execute_flow import ExecuteFlow\nfrom dataflow.config.config import Config\n\nfrom ufo.utils import print_with_color\nfrom learner.utils import load_json_file, save_json_file, reformat_json_file\n\nfrom ufo.agents.processors.app_agent_processor import AppAgentProcessor\nfrom ufo.module.context import Context\n\n# Set the environment variable for the run configuration.\nos.environ[\"RUN_CONFIGS\"] = \"True\"\n\n# Load configuration data.\n_configs = Config.get_instance().config_data\n\nINSTANTIATION_RESULT_MAP = {True: \"instantiation_pass\", False: \"instantiation_fail\"}\n\nEXECUTION_RESULT_MAP = {\n \"yes\": \"execution_pass\",\n \"no\": \"execution_fail\",\n \"unsure\": \"execution_unsure\",\n}\n\n\nclass AppEnum(Enum):\n \"\"\"\n Enum class for applications.\n \"\"\"\n\n WORD = 1, \"Word\", \".docx\", \"winword\"\n EXCEL = 2, \"Excel\", \".xlsx\", \"excel\"\n POWERPOINT = 3, \"PowerPoint\", \".pptx\", \"powerpnt\"\n\n def __init__(self, id: int, description: str, file_extension: str, win_app: str):\n \"\"\"\n Initialize the application enum.\n :param id: The ID of the application.\n :param description: The description of the application.\n :param file_extension: The file extension of the application.\n :param win_app: The Windows application name.\n \"\"\"\n\n self.id = id\n self.description = description\n self.file_extension = file_extension\n self.win_app = win_app\n self.app_root_name = win_app.upper() + \".EXE\"\n\n\nclass TaskObject:\n def __init__(self, task_file_path: str, task_type: str) -> None:\n \"\"\"\n Initialize the task object.\n :param task_file_path: The path to the task file.\n :param task_type: The task_type of the task object (dataflow, instantiation, or execution).\n \"\"\"\n\n self.task_file_path = task_file_path\n self.task_file_base_name = os.path.basename(task_file_path)\n self.task_file_name = self.task_file_base_name.split(\".\")[0]\n\n task_json_file = load_json_file(task_file_path)\n self.app_object = self._choose_app_from_json(task_json_file[\"app\"])\n # Initialize the task attributes based on the task_type\n self._init_attr(task_type, task_json_file)\n\n def _choose_app_from_json(self, task_app: str) -> AppEnum:\n \"\"\"\n Choose the app from the task json file.\n :param task_app: The app from the task json file.\n :return: The app enum.\n \"\"\"\n\n for app in AppEnum:\n if app.description.lower() == task_app.lower():\n return app\n raise ValueError(\"The APP in the task file is not supported.\")\n\n def _init_attr(self, task_type: str, task_json_file: Dict[str, Any]) -> None:\n \"\"\"\n Initialize the attributes of the task object.\n :param task_type: The task_type of the task object (dataflow, instantiation, or execution).\n :param task_json_file: The task JSON file.\n \"\"\"\n\n if task_type == \"dataflow\" or task_type == \"instantiation\":\n for key, value in task_json_file.items():\n setattr(self, key.lower().replace(\" \", \"_\"), value)\n elif task_type == \"execution\":\n self.app = task_json_file.get(\"app\")\n self.unique_id = task_json_file.get(\"unique_id\")\n original = task_json_file.get(\"original\", {})\n self.task = original.get(\"original_task\", None)\n self.refined_steps = original.get(\"original_steps\", None)\n else:\n raise ValueError(f\"Unsupported task_type: {task_type}\")\n\n\nclass DataFlowController:\n \"\"\"\n Flow controller class to manage the instantiation and execution process.\n \"\"\"\n\n def __init__(self, task_path: str, task_type: str) -> None:\n \"\"\"\n Initialize the flow controller.\n :param task_path: The path to the task file.\n :param task_type: The task_type of the flow controller (instantiation, execution, or dataflow).\n \"\"\"\n\n self.task_object = TaskObject(task_path, task_type)\n self.app_env = None\n self.app_name = self.task_object.app_object.description.lower()\n self.task_file_name = self.task_object.task_file_name\n\n self.schema = self._load_schema(task_type)\n\n self.task_type = task_type\n self.task_info = self.init_task_info()\n self.result_hub = _configs[\"RESULT_HUB\"].format(task_type=task_type)\n\n def init_task_info(self) -> Dict[str, Any]:\n \"\"\"\n Initialize the task information.\n :return: The initialized task information.\n \"\"\"\n init_task_info = None\n if self.task_type == \"execution\":\n # read from the instantiated task file\n init_task_info = load_json_file(self.task_object.task_file_path)\n else:\n init_task_info = {\n \"unique_id\": self.task_object.unique_id,\n \"app\": self.app_name,\n \"original\": {\n \"original_task\": self.task_object.task,\n \"original_steps\": self.task_object.refined_steps,\n },\n \"execution_result\": {\"result\": None, \"error\": None},\n \"instantiation_result\": {\n \"choose_template\": {\"result\": None, \"error\": None},\n \"prefill\": {\"result\": None, \"error\": None},\n \"instantiation_evaluation\": {\"result\": None, \"error\": None},\n },\n \"time_cost\": {},\n }\n return init_task_info\n\n def _load_schema(self, task_type: str) -> Dict[str, Any]:\n \"\"\"\n load the schema based on the task_type.\n :param task_type: The task_type of the task object (dataflow, instantiation, or execution).\n :return: The schema for the task_type.\n \"\"\"\n\n if task_type == \"instantiation\":\n return load_json_file(_configs[\"INSTANTIATION_RESULT_SCHEMA\"])\n elif task_type == \"execution\" or task_type == \"dataflow\":\n return load_json_file(_configs[\"EXECUTION_RESULT_SCHEMA\"])\n\n def execute_instantiation(self) -> Optional[List[Dict[str, Any]]]:\n \"\"\"\n Execute the instantiation process.\n :return: The instantiation plan if successful.\n \"\"\"\n\n print_with_color(\n f\"Instantiating task {self.task_object.task_file_name}...\", \"blue\"\n )\n\n template_copied_path = self.instantiation_single_flow(\n ChooseTemplateFlow,\n \"choose_template\",\n init_params=[self.task_object.app_object.file_extension],\n execute_params=[],\n )\n\n if template_copied_path:\n self.app_env.start(template_copied_path)\n\n prefill_result = self.instantiation_single_flow(\n PrefillFlow,\n \"prefill\",\n init_params=[self.app_env],\n execute_params=[\n template_copied_path,\n self.task_object.task,\n self.task_object.refined_steps,\n ],\n )\n self.app_env.close()\n\n if prefill_result:\n self.instantiation_single_flow(\n FilterFlow,\n \"instantiation_evaluation\",\n init_params=[],\n execute_params=[prefill_result[\"instantiated_request\"]],\n )\n return prefill_result[\"instantiated_plan\"]\n\n def execute_execution(self, request: str, plan: Dict[str, any]) -> None:\n \"\"\"\n Execute the execution process.\n :param request: The task request to be executed.\n :param plan: The execution plan containing detailed steps.\n \"\"\"\n\n print_with_color(\"Executing the execution process...\", \"blue\")\n execute_flow = None\n\n try:\n self.app_env.start(self.template_copied_path)\n # Initialize the execution context and flow\n context = Context()\n execute_flow = ExecuteFlow(self.task_file_name, context, self.app_env)\n\n # Execute the plan\n executed_plan, execute_result = execute_flow.execute(request, plan)\n\n # Update the instantiated plan\n self.instantiated_plan = executed_plan\n # Record execution results and time metrics\n self.task_info[\"execution_result\"][\"result\"] = execute_result\n self.task_info[\"time_cost\"][\"execute\"] = execute_flow.execution_time\n self.task_info[\"time_cost\"][\"execute_eval\"] = execute_flow.eval_time\n\n except Exception as e:\n # Handle and log any exceptions that occur during execution\n self.task_info[\"execution_result\"][\"error\"] = {\n \"type\": str(type(e).__name__),\n \"message\": str(e),\n \"traceback\": traceback.format_exc(),\n }\n print_with_color(f\"Error in Execution: {e}\", \"red\")\n raise e\n finally:\n # Record the total time cost of the execution process\n if execute_flow and hasattr(execute_flow, \"execution_time\"):\n self.task_info[\"time_cost\"][\"execute\"] = execute_flow.execution_time\n else:\n self.task_info[\"time_cost\"][\"execute\"] = None\n if execute_flow and hasattr(execute_flow, \"eval_time\"):\n self.task_info[\"time_cost\"][\"execute_eval\"] = execute_flow.eval_time\n else:\n self.task_info[\"time_cost\"][\"execute_eval\"] = None\n \n\n def instantiation_single_flow(\n self,\n flow_class: AppAgentProcessor,\n flow_type: str,\n init_params=None,\n execute_params=None,\n ) -> Optional[Dict[str, Any]]:\n \"\"\"\n Execute a single flow process in the instantiation phase.\n :param flow_class: The flow class to instantiate.\n :param flow_type: The type of the flow.\n :param init_params: The initialization parameters for the flow.\n :param execute_params: The execution parameters for the flow.\n :return: The result of the flow process.\n \"\"\"\n\n flow_instance = None\n try:\n flow_instance = flow_class(self.app_name, self.task_file_name, *init_params)\n result = flow_instance.execute(*execute_params)\n self.task_info[\"instantiation_result\"][flow_type][\"result\"] = result\n return result\n except Exception as e:\n self.task_info[\"instantiation_result\"][flow_type][\"error\"] = {\n \"type\": str(e.__class__),\n \"error_message\": str(e),\n \"traceback\": traceback.format_exc(),\n }\n print_with_color(f\"Error in {flow_type}: {e} {traceback.format_exc()}\")\n finally:\n if flow_instance and hasattr(flow_instance, \"execution_time\"):\n self.task_info[\"time_cost\"][flow_type] = flow_instance.execution_time\n else:\n self.task_info[\"time_cost\"][flow_type] = None\n\n def save_result(self) -> None:\n \"\"\"\n Validate and save the instantiated task result.\n \"\"\"\n\n validation_error = None\n\n # Validate the result against the schema\n try:\n validate(instance=self.task_info, schema=self.schema)\n except ValidationError as e:\n # Record the validation error but allow the process to continue\n validation_error = str(e.message)\n print_with_color(f\"Validation Error: {e.message}\", \"yellow\")\n\n # Determine the target directory based on task_type and quality/completeness\n target_file = None\n\n if self.task_type == \"instantiation\":\n # Determine the quality of the instantiation\n if not self.task_info[\"instantiation_result\"][\"instantiation_evaluation\"][\n \"result\"\n ]:\n target_file = INSTANTIATION_RESULT_MAP[False]\n else:\n is_quality_good = self.task_info[\"instantiation_result\"][\n \"instantiation_evaluation\"\n ][\"result\"][\"judge\"]\n target_file = INSTANTIATION_RESULT_MAP.get(\n is_quality_good, INSTANTIATION_RESULT_MAP[False]\n )\n\n else:\n # Determine the completion status of the execution\n if not self.task_info[\"execution_result\"][\"result\"]:\n target_file = EXECUTION_RESULT_MAP[\"no\"]\n else:\n is_completed = self.task_info[\"execution_result\"][\"result\"][\"complete\"]\n target_file = EXECUTION_RESULT_MAP.get(\n is_completed, EXECUTION_RESULT_MAP[\"no\"]\n )\n\n # Construct the full path to save the result\n new_task_path = os.path.join(\n self.result_hub, target_file, self.task_object.task_file_base_name\n )\n os.makedirs(os.path.dirname(new_task_path), exist_ok=True)\n save_json_file(new_task_path, self.task_info)\n\n print(f\"Task saved to {new_task_path}\")\n\n # If validation failed, indicate that the saved result may need further inspection\n if validation_error:\n print(\n \"The saved task result does not conform to the expected schema and may require review.\"\n )\n\n @property\n def template_copied_path(self) -> str:\n \"\"\"\n Get the copied template path from the task information.\n :return: The copied template path.\n \"\"\"\n\n return self.task_info[\"instantiation_result\"][\"choose_template\"][\"result\"]\n\n @property\n def instantiated_plan(self) -> List[Dict[str, Any]]:\n \"\"\"\n Get the instantiated plan from the task information.\n :return: The instantiated plan.\n \"\"\"\n\n return self.task_info[\"instantiation_result\"][\"prefill\"][\"result\"][\n \"instantiated_plan\"\n ]\n\n @instantiated_plan.setter\n def instantiated_plan(self, value: List[Dict[str, Any]]) -> None:\n \"\"\"\n Set the instantiated plan in the task information.\n :param value: New value for the instantiated plan.\n \"\"\"\n\n self.task_info.setdefault(\"instantiation_result\", {}).setdefault(\n \"prefill\", {}\n ).setdefault(\"result\", {})\n self.task_info[\"instantiation_result\"][\"prefill\"][\"result\"][\n \"instantiated_plan\"\n ] = value\n\n def reformat_to_batch(self, path) -> None:\n \"\"\"\n Transfer the result to the result hub.\n \"\"\"\n os.makedirs(path, exist_ok=True)\n source_files_path = os.path.join(\n self.result_hub,\n self.task_type + \"_pass\",\n )\n source_template_path = os.path.join(\n os.path.dirname(self.result_hub),\n \"saved_document\",\n )\n target_file_path = os.path.join(\n path,\n \"tasks\",\n )\n target_template_path = os.path.join(\n path,\n \"files\",\n )\n os.makedirs((target_file_path), exist_ok=True)\n os.makedirs((target_template_path), exist_ok=True)\n\n for file in os.listdir(source_files_path):\n if file.endswith(\".json\"):\n source_file = os.path.join(source_files_path, file)\n target_file = os.path.join(target_file_path, file)\n target_object = os.path.join(\n target_template_path, file.replace(\".json\", \".docx\")\n )\n is_successed = reformat_json_file(\n target_file,\n target_object,\n load_json_file(source_file),\n )\n if is_successed:\n shutil.copy(\n os.path.join(\n source_template_path, file.replace(\".json\", \".docx\")\n ),\n target_template_path,\n )\n\n def run(self) -> None:\n \"\"\"\n Run the instantiation and execution process.\n \"\"\"\n\n start_time = time.time()\n\n try:\n self.app_env = WindowsAppEnv(self.task_object.app_object)\n\n if self.task_type == \"dataflow\":\n plan = self.execute_instantiation()\n self.execute_execution(self.task_object.task, plan)\n elif self.task_type == \"instantiation\":\n self.execute_instantiation()\n elif self.task_type == \"execution\":\n plan = self.instantiated_plan\n self.execute_execution(self.task_object.task, plan)\n else:\n raise ValueError(f\"Unsupported task_type: {self.task_type}\")\n except Exception as e:\n raise e\n\n finally:\n # Update or record the total time cost of the process\n total_time = round(time.time() - start_time, 3)\n new_total_time = (\n self.task_info.get(\"time_cost\", {}).get(\"total\", 0) + total_time\n )\n self.task_info[\"time_cost\"][\"total\"] = round(new_total_time, 3)\n\n self.save_result()\n\n if _configs[\"REFORMAT_TO_BATCH\"]:\n self.reformat_to_batch(_configs[\"REFORMAT_TO_BATCH_HUB\"])\n"
},
{
"path": "dataflow/dataflow.py",
"content": "import argparse\nimport os\nimport traceback\nfrom ufo.utils import print_with_color\nfrom dataflow.config.config import Config\n\n_configs = Config.get_instance().config_data\n\n\ndef parse_args() -> argparse.Namespace:\n \"\"\"\n Parse command-line arguments. Automatically detect batch or single mode.\n \"\"\"\n parser = argparse.ArgumentParser(\n description=\"Run tasks automatically in single or batch mode.\"\n )\n\n # Add options for -dataflow, -instantiation, and -execution\n parser.add_argument(\n \"--dataflow\",\n action=\"store_const\",\n const=\"dataflow\",\n help=\"Indicates that the task type is dataflow.\",\n )\n parser.add_argument(\n \"--instantiation\",\n action=\"store_const\",\n const=\"instantiation\",\n help=\"Indicates that the task type is instantiation.\",\n )\n parser.add_argument(\n \"--execution\",\n action=\"store_const\",\n const=\"execution\",\n help=\"Indicates that the task type is execution.\",\n )\n\n # Task path argument\n parser.add_argument(\n \"--task_path\",\n type=str,\n default=_configs[\"TASKS_HUB\"],\n help=\"Path to the task file or directory.\",\n )\n\n return parser.parse_args()\n\n\ndef validate_path(path: str) -> str:\n \"\"\"\n Validate the given path and determine its type.\n :param path: The path to validate.\n :return: \"file\", \"directory\", or raises an error if invalid.\n \"\"\"\n if os.path.isfile(path):\n return \"file\"\n elif os.path.isdir(path):\n return \"directory\"\n else:\n print_with_color(f\"Invalid path: {path}\", \"red\")\n raise ValueError(f\"Path {path} is neither a file nor a directory.\")\n\n\ndef process_task(task_path: str, task_type: str) -> None:\n \"\"\"\n Process a single task file using the DataFlowController.\n \"\"\"\n from dataflow.data_flow_controller import DataFlowController\n\n try:\n print_with_color(f\"Processing task: {task_path}\", \"green\")\n flow_controller = DataFlowController(task_path, task_type)\n flow_controller.run()\n print_with_color(f\"Task {task_path} completed successfully.\", \"green\")\n except Exception as e:\n print_with_color(\n f\"Error processing {task_path}: {traceback.format_exc()}\", \"red\"\n )\n\n\ndef process_batch(task_dir: str, task_type: str) -> None:\n \"\"\"\n Process all task files in a directory.\n \"\"\"\n task_files = [\n os.path.join(task_dir, f)\n for f in os.listdir(task_dir)\n if os.path.isfile(os.path.join(task_dir, f))\n ]\n if not task_files:\n print_with_color(f\"No tasks found in directory: {task_dir}.\", \"yellow\")\n return\n\n print_with_color(f\"Found {len(task_files)} tasks in {task_dir}.\", \"blue\")\n for task_file in task_files:\n process_task(task_file, task_type)\n\n\ndef main():\n \"\"\"\n Main function to run tasks based on the provided arguments.\n You can use dataflow, instantiation, and execution modes to process the task.\n Also, you can run tasks in batch mode by providing the path to the task directory.\n See README to read the detailed usage.\n \"\"\"\n args = parse_args()\n\n # Ensure that a task type has been provided; if not, raise an error\n if not any([args.dataflow, args.instantiation, args.execution]):\n print_with_color(\n \"Error: You must specify one of the task types (--dataflow, --instantiation, or --execution).\",\n \"red\",\n )\n return\n\n task_type = args.dataflow or args.instantiation or args.execution\n\n path_type = validate_path(args.task_path)\n\n if path_type == \"file\":\n process_task(args.task_path, task_type)\n elif path_type == \"directory\":\n process_batch(args.task_path, task_type)\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "dataflow/env/env_manager.py",
"content": "import logging\nimport platform\nimport re\nfrom time import sleep\nfrom typing import Optional, Tuple, Dict, TYPE_CHECKING, Any\nimport psutil\n\nfrom fuzzywuzzy import fuzz\n\n# Conditional imports for Windows-specific packages\nif TYPE_CHECKING or platform.system() == \"Windows\":\n from pywinauto import Desktop\n from pywinauto.controls.uiawrapper import UIAWrapper\nelse:\n Desktop = None\n UIAWrapper = Any\n\nfrom dataflow.config.config import Config\nfrom ufo.config import Config as UFOConfig\nfrom aip.messages import ControlInfo\n\n# Load configuration settings\n_configs = Config.get_instance().config_data\n_ufo_configs = UFOConfig.get_instance().config_data\n\nif _ufo_configs is not None:\n _BACKEND = \"uia\"\nif _configs is not None:\n _MATCH_STRATEGY = _configs.get(\"MATCH_STRATEGY\", \"contains\")\n\n\nclass WindowsAppEnv:\n \"\"\"\n Represents the Windows Application Environment.\n \"\"\"\n\n def __init__(self, app_object: object) -> None:\n \"\"\"\n Initializes the Windows Application Environment.\n :param app_object: The app object containing information about the application.\n \"\"\"\n\n self.app_window = None\n self.app_root_name = app_object.app_root_name\n self.app_name = app_object.description.lower()\n self.win_app = app_object.win_app\n\n def start(self, copied_template_path: str) -> None:\n \"\"\"\n Starts the Windows environment.\n :param copied_template_path: The file path to the copied template to start the environment.\n \"\"\"\n\n from ufo.automator.ui_control import openfile\n\n file_controller = openfile.FileController(_BACKEND)\n try:\n file_controller.execute_code(\n {\"APP\": self.win_app, \"file_path\": copied_template_path}\n )\n except Exception as e:\n logging.exception(f\"Failed to start the application: {e}\")\n raise\n\n def close(self) -> None:\n \"\"\"\n Tries to gracefully close the application; if it fails or is not closed, forcefully terminates the process.\n \"\"\"\n\n try:\n # Gracefully close the application window\n if self.app_window and self.app_window.process_id():\n self.app_window.close()\n sleep(1)\n # Forcefully close the application window\n if self.app_window.element_info.name.lower() != \"\":\n self._check_and_kill_process()\n except Exception as e:\n logging.warning(\n f\"Graceful close failed: {e}. Attempting to forcefully terminate the process.\"\n )\n self._check_and_kill_process()\n raise e\n\n def _check_and_kill_process(self) -> None:\n \"\"\"\n Checks if the process is still running and kills it if it is.\n \"\"\"\n\n try:\n if self.app_window and self.app_window.process_id():\n process = psutil.Process(self.app_window.process_id())\n print(f\"Killing process: {self.app_window.process_id}\")\n process.terminate()\n except Exception as e:\n logging.error(f\"Error while checking window status: {e}\")\n raise e\n\n def find_matching_window(self, doc_name: str) -> Optional[UIAWrapper]:\n \"\"\"\n Finds a matching window based on the process name and the configured matching strategy.\n :param doc_name: The document name associated with the application.\n :return: The matched window or None if no match is found.\n \"\"\"\n\n desktop = Desktop(backend=_BACKEND)\n windows_list = desktop.windows()\n for window in windows_list:\n window_title = window.element_info.name.lower()\n if self._match_window_name(window_title, doc_name):\n self.app_window = window\n return window\n return None\n\n def _match_window_name(self, window_title: str, doc_name: str) -> bool:\n \"\"\"\n Matches the window name based on the strategy specified in the config file.\n :param window_title: The title of the window.\n :param doc_name: The document name associated with the application.\n :return: True if a match is found based on the strategy; False otherwise.\n \"\"\"\n\n app_name = self.app_name\n doc_name = doc_name.lower()\n\n if _MATCH_STRATEGY == \"contains\":\n return app_name in window_title and doc_name in window_title\n elif _MATCH_STRATEGY == \"fuzzy\":\n similarity_app = fuzz.partial_ratio(window_title, app_name)\n similarity_doc = fuzz.partial_ratio(window_title, doc_name)\n return similarity_app >= 70 and similarity_doc >= 70\n elif _MATCH_STRATEGY == \"regex\":\n combined_name_1 = f\"{app_name}.*{doc_name}\"\n combined_name_2 = f\"{doc_name}.*{app_name}\"\n pattern_1 = re.compile(combined_name_1, flags=re.IGNORECASE)\n pattern_2 = re.compile(combined_name_2, flags=re.IGNORECASE)\n return (\n re.search(pattern_1, window_title) is not None\n or re.search(pattern_2, window_title) is not None\n )\n else:\n logging.exception(f\"Unknown match strategy: {_MATCH_STRATEGY}\")\n raise ValueError(f\"Unknown match strategy: {_MATCH_STRATEGY}\")\n\n def _calculate_match_score(self, control: ControlInfo, control_text: str) -> int:\n \"\"\"\n Calculate the match score between a control and the given text.\n :param control: The control object to evaluate.\n :param control_text: The target text to match.\n :return: An integer score representing the match quality (higher is better).\n \"\"\"\n control_content = control.text_content or \"\"\n\n # Matching strategies\n if _MATCH_STRATEGY == \"contains\":\n return 100 if control_text in control_content else 0\n elif _MATCH_STRATEGY == \"fuzzy\":\n return fuzz.partial_ratio(control_content, control_text)\n elif _MATCH_STRATEGY == \"regex\":\n pattern = re.compile(f\"{re.escape(control_text)}\", flags=re.IGNORECASE)\n return 100 if re.search(pattern, control_content) else 0\n else:\n raise ValueError(f\"Unknown match strategy: {_MATCH_STRATEGY}\")\n\n def find_matching_controller(\n self, filtered_annotation_dict: Dict[int, ControlInfo], control_text: str\n ) -> Tuple[str, ControlInfo]:\n \"\"\" \"\n Select the best matched controller.\n :param filtered_annotation_dict: The filtered annotation dictionary.\n :param control_text: The text content of the control for additional context.\n :return: Tuple containing the key of the selected controller and the control object.s\n \"\"\"\n control_selected = None\n controller_key = None\n highest_score = 0\n\n # Iterate through the filtered annotation dictionary to find the best match\n for key, control in filtered_annotation_dict.items():\n # Calculate the matching score using the match function\n score = self._calculate_match_score(control, control_text)\n\n # Update the selected control if the score is higher\n if score > highest_score:\n highest_score = score\n controller_key = key\n control_selected = control\n\n return controller_key, control_selected\n"
},
{
"path": "dataflow/execution/agent/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/execution/agent/execute_agent.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom ufo.agents.agent.app_agent import AppAgent\n\n\nclass ExecuteAgent(AppAgent):\n \"\"\"\n The Agent for task execution.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n process_name: str,\n app_root_name: str,\n ):\n \"\"\"\n Initialize the ExecuteAgent.\n :param name: The name of the agent.\n :param process_name: The name of the process.\n :param app_root_name: The name of the app root.\n \"\"\"\n\n self._step = 0\n self._complete = False\n self._name = name\n self._status = None\n self._process_name = process_name\n self._app_root_name = app_root_name\n self.Puppeteer = self.create_puppeteer_interface()"
},
{
"path": "dataflow/execution/agent/execute_eval_agent.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom typing import Optional\n\nfrom dataflow.prompter.execution.execute_eval_prompter import ExecuteEvalAgentPrompter\nfrom ufo.agents.agent.evaluation_agent import EvaluationAgent\n\nclass ExecuteEvalAgent(EvaluationAgent):\n \"\"\"\n The Agent for task execution evaluation.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n app_root_name: str,\n is_visual: bool,\n main_prompt: str,\n example_prompt: str,\n api_prompt: str,\n ):\n \"\"\"\n Initialize the ExecuteEvalAgent.\n :param name: The name of the agent.\n :param app_root_name: The name of the app root.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param example_prompt: The example prompt.\n :param api_prompt: The API prompt.\n \"\"\"\n\n super().__init__(\n name=name,\n app_root_name=app_root_name,\n is_visual=is_visual,\n main_prompt=main_prompt,\n example_prompt=example_prompt,\n api_prompt=api_prompt,\n )\n\n def get_prompter(\n self,\n is_visual: bool,\n prompt_template: str,\n example_prompt_template: str,\n api_prompt_template: str,\n root_name: Optional[str] = None,\n ) -> ExecuteEvalAgentPrompter:\n \"\"\"\n Get the prompter for the agent.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param prompt_template: The prompt template.\n :param example_prompt_template: The example prompt template.\n :param api_prompt_template: The API prompt template.\n :param root_name: The name of the root.\n :return: The prompter.\n \"\"\"\n\n return ExecuteEvalAgentPrompter(\n is_visual=is_visual,\n prompt_template=prompt_template,\n example_prompt_template=example_prompt_template,\n api_prompt_template=api_prompt_template,\n root_name=root_name,\n )"
},
{
"path": "dataflow/execution/workflow/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/execution/workflow/execute_flow.py",
"content": "import os\nimport time\nfrom typing import Any, Dict, List, Tuple\n\nfrom dataflow.config.config import Config as InstantiationConfig\nfrom dataflow.env.env_manager import WindowsAppEnv\nfrom dataflow.execution.agent.execute_agent import ExecuteAgent\nfrom dataflow.execution.agent.execute_eval_agent import ExecuteEvalAgent\nfrom ufo import utils\nfrom ufo.agents.processors.app_agent_processor import AppAgentProcessor\nfrom ufo.automator.app_apis.basic import WinCOMReceiverBasic\nfrom ufo.config import Config as UFOConfig\nfrom ufo.module.basic import BaseSession, Context, ContextNames\n\n_configs = InstantiationConfig.get_instance().config_data\n_ufo_configs = UFOConfig.get_instance().config_data\n\n\nclass ExecuteFlow(AppAgentProcessor):\n \"\"\"\n ExecuteFlow class for executing the task and saving the result.\n \"\"\"\n\n _app_execute_agent_dict: Dict[str, ExecuteAgent] = {}\n _app_eval_agent_dict: Dict[str, ExecuteEvalAgent] = {}\n\n def __init__(\n self, task_file_name: str, context: Context, environment: WindowsAppEnv\n ) -> None:\n \"\"\"\n Initialize the execute flow for a task.\n :param task_file_name: Name of the task file being processed.\n :param context: Context object for the current session.\n :param environment: Environment object for the application being processed.\n \"\"\"\n\n super().__init__(agent=ExecuteAgent, context=context)\n\n self.execution_time = None\n self.eval_time = None\n self._app_env = environment\n self._task_file_name = task_file_name\n self._app_name = self._app_env.app_name\n\n log_path = _configs[\"EXECUTE_LOG_PATH\"].format(task=task_file_name)\n self._initialize_logs(log_path)\n\n self.application_window = self._app_env.find_matching_window(task_file_name)\n self.app_agent = self._get_or_create_execute_agent()\n self.eval_agent = self._get_or_create_evaluation_agent()\n\n self._matched_control = None # Matched control for the current step.\n\n def _get_or_create_execute_agent(self) -> ExecuteAgent:\n \"\"\"\n Retrieve or create a execute agent for the given application.\n :return: ExecuteAgent instance for the specified application.\n \"\"\"\n\n if self._app_name not in ExecuteFlow._app_execute_agent_dict:\n ExecuteFlow._app_execute_agent_dict[self._app_name] = ExecuteAgent(\n \"execute\",\n self._app_name,\n self._app_env.app_root_name,\n )\n return ExecuteFlow._app_execute_agent_dict[self._app_name]\n\n def _get_or_create_evaluation_agent(self) -> ExecuteEvalAgent:\n \"\"\"\n Retrieve or create an evaluation agent for the given application.\n :return: ExecuteEvalAgent instance for the specified application.\n \"\"\"\n\n if self._app_name not in ExecuteFlow._app_eval_agent_dict:\n ExecuteFlow._app_eval_agent_dict[self._app_name] = ExecuteEvalAgent(\n \"evaluation\",\n self._app_env.app_root_name,\n is_visual=True,\n main_prompt=_ufo_configs[\"EVALUATION_PROMPT\"],\n example_prompt=\"\",\n api_prompt=_ufo_configs[\"API_PROMPT\"],\n )\n return ExecuteFlow._app_eval_agent_dict[self._app_name]\n\n def _initialize_logs(self, log_path: str) -> None:\n \"\"\"\n Initialize logging for execute messages and responses.\n :param log_path: Path to save the logs.\n \"\"\"\n\n os.makedirs(log_path, exist_ok=True)\n self._execute_message_logger = BaseSession.initialize_logger(\n log_path, \"execute_log.json\", \"w\", _configs\n )\n self.context.set(ContextNames.LOG_PATH, log_path)\n self.context.set(ContextNames.LOGGER, self._execute_message_logger)\n\n def execute(\n self, request: str, instantiated_plan: List[Dict[str, Any]]\n ) -> Tuple[List[Dict[str, Any]], Dict[str, str]]:\n \"\"\"\n Execute the execute flow: Execute the task and save the result.\n :param request: Original request to be executed.\n :param instantiated_plan: Instantiated plan containing steps to execute.\n :return: Tuple containing task quality flag, comment, and task type.\n \"\"\"\n\n start_time = time.time()\n try:\n executed_plan = self.execute_plan(instantiated_plan)\n except Exception as error:\n raise RuntimeError(f\"Execution failed. {error}\")\n finally:\n self.execution_time = round(time.time() - start_time, 3)\n\n start_time = time.time()\n try:\n result, _ = self.eval_agent.evaluate(\n request=request, log_path=self.log_path\n )\n utils.print_with_color(f\"Result: {result}\", \"green\")\n except Exception as error:\n raise RuntimeError(f\"Evaluation failed. {error}\")\n finally:\n self.eval_time = round(time.time() - start_time, 3)\n\n return executed_plan, result\n\n def execute_plan(\n self, instantiated_plan: List[Dict[str, Any]]\n ) -> List[Dict[str, Any]]:\n \"\"\"\n Get the executed result from the execute agent.\n :param instantiated_plan: Plan containing steps to execute.\n :return: List of executed steps.\n \"\"\"\n\n # Initialize the step counter and capture the initial screenshot.\n self.session_step = 0\n try:\n time.sleep(1)\n # Initialize the API receiver\n self.app_agent.Puppeteer.receiver_manager.create_api_receiver(\n self.app_agent._app_root_name, self.app_agent._process_name\n )\n # Initialize the control receiver\n current_receiver = self.app_agent.Puppeteer.receiver_manager.receiver_list[\n -1\n ]\n\n if current_receiver is not None:\n self.application_window = self._app_env.find_matching_window(\n self._task_file_name\n )\n current_receiver.com_object = (\n current_receiver.get_object_from_process_name()\n )\n\n self.init_and_final_capture_screenshot()\n except Exception as error:\n raise RuntimeError(f\"Execution initialization failed. {error}\")\n\n # Initialize the success flag for each step.\n for index, step_plan in enumerate(instantiated_plan):\n instantiated_plan[index][\"Success\"] = None\n instantiated_plan[index][\"MatchedControlText\"] = None\n\n for index, step_plan in enumerate(instantiated_plan):\n try:\n self.session_step += 1\n\n # Check if the maximum steps have been exceeded.\n if self.session_step > _configs[\"MAX_STEPS\"]:\n raise RuntimeError(\"Maximum steps exceeded.\")\n\n self._parse_step_plan(step_plan)\n\n try:\n self.process()\n instantiated_plan[index][\"Success\"] = True\n instantiated_plan[index][\"ControlLabel\"] = self._control_label\n instantiated_plan[index][\n \"MatchedControlText\"\n ] = self._matched_control\n except Exception as ControllerNotFoundError:\n instantiated_plan[index][\"Success\"] = False\n raise ControllerNotFoundError\n\n except Exception as error:\n err_info = RuntimeError(\n f\"Step {self.session_step} execution failed. {error}\"\n )\n raise err_info\n # capture the final screenshot\n self.session_step += 1\n time.sleep(1)\n self.init_and_final_capture_screenshot()\n # save the final state of the app\n\n win_com_receiver = None\n for receiver in reversed(\n self.app_agent.Puppeteer.receiver_manager.receiver_list\n ):\n if isinstance(receiver, WinCOMReceiverBasic):\n if receiver.client is not None:\n win_com_receiver = receiver\n break\n\n if win_com_receiver is not None:\n win_com_receiver.save()\n time.sleep(1)\n win_com_receiver.client.Quit()\n\n print(\"Execution complete.\")\n\n return instantiated_plan\n\n def process(self) -> None:\n \"\"\"\n Process the current step.\n \"\"\"\n\n step_start_time = time.time()\n self.print_step_info()\n self.capture_screenshot()\n self.execute_action()\n self.time_cost = round(time.time() - step_start_time, 3)\n self.log_save()\n\n def print_step_info(self) -> None:\n \"\"\"\n Print the step information.\n \"\"\"\n\n utils.print_with_color(\n \"Step {step}: {subtask}\".format(\n step=self.session_step,\n subtask=self.subtask,\n ),\n \"magenta\",\n )\n\n def log_save(self) -> None:\n \"\"\"\n Log the constructed prompt message for the PrefillAgent.\n \"\"\"\n\n step_memory = {\n \"Step\": self.session_step,\n \"Subtask\": self.subtask,\n \"ControlLabel\": self._control_label,\n \"ControlText\": self.control_text,\n \"Action\": self.action,\n \"ActionType\": self.app_agent.Puppeteer.get_command_types(self._operation),\n \"Results\": self._results,\n \"Application\": self.app_agent._app_root_name,\n \"TimeCost\": self.time_cost,\n }\n self._memory_data.add_values_from_dict(step_memory)\n self.log(self._memory_data.to_dict())\n\n def _parse_step_plan(self, step_plan: Dict[str, Any]) -> None:\n \"\"\"\n Parse the response.\n :param step_plan: The step plan.\n \"\"\"\n\n self._matched_control = None\n self.subtask = step_plan.get(\"Subtask\", \"\")\n self.control_text = step_plan.get(\"ControlText\", \"\")\n self._operation = step_plan.get(\"Function\", \"\")\n self.question_list = step_plan.get(\"Questions\", [])\n self._args = utils.revise_line_breaks(step_plan.get(\"Args\", \"\"))\n\n # Compose the function call and the arguments string.\n self.action = self.app_agent.Puppeteer.get_command_string(\n self._operation, self._args\n )\n\n self.status = step_plan.get(\"Status\", \"\")\n\n def init_and_final_capture_screenshot(self) -> None:\n \"\"\"\n Capture the screenshot.\n \"\"\"\n\n # Define the paths for the screenshots saved.\n screenshot_save_path = self.log_path + f\"action_step{self.session_step}.png\"\n\n self._memory_data.add_values_from_dict(\n {\n \"CleanScreenshot\": screenshot_save_path,\n }\n )\n\n self.photographer.capture_app_window_screenshot(\n self.application_window, save_path=screenshot_save_path\n )\n # Capture the control screenshot.\n control_selected = self._app_env.app_window\n self.capture_control_screenshot(control_selected)\n\n def execute_action(self) -> None:\n \"\"\"\n Execute the action.\n \"\"\"\n\n control_selected = None\n # Find the matching window and control.\n self.application_window = self._app_env.find_matching_window(\n self._task_file_name\n )\n if self.control_text == \"\":\n control_selected = self.application_window\n else:\n self._control_label, control_selected = (\n self._app_env.find_matching_controller(\n self.filtered_annotation_dict, self.control_text\n )\n )\n if control_selected:\n self._matched_control = control_selected.window_text()\n\n if not control_selected:\n # If the control is not found, raise an error.\n raise RuntimeError(f\"Control with text '{self.control_text}' not found.\")\n\n try:\n # Get the selected control item from the annotation dictionary and LLM response.\n # The LLM response is a number index corresponding to the key in the annotation dictionary.\n if control_selected:\n\n if _ufo_configs.get(\"SHOW_VISUAL_OUTLINE_ON_SCREEN\", True):\n control_selected.draw_outline(colour=\"red\", thickness=3)\n time.sleep(_ufo_configs.get(\"RECTANGLE_TIME\", 0))\n\n control_coordinates = utils.coordinate_adjusted(\n self.application_window.rectangle(), control_selected.rectangle()\n )\n\n self._control_log = {\n \"control_class\": control_selected.element_info.class_name,\n \"control_type\": control_selected.element_info.control_type,\n \"control_automation_id\": control_selected.element_info.automation_id,\n \"control_friendly_class_name\": control_selected.friendly_class_name(),\n \"control_coordinates\": {\n \"left\": control_coordinates[0],\n \"top\": control_coordinates[1],\n \"right\": control_coordinates[2],\n \"bottom\": control_coordinates[3],\n },\n }\n\n self.app_agent.Puppeteer.receiver_manager.create_ui_control_receiver(\n control_selected, self.application_window\n )\n\n # Save the screenshot of the tagged selected control.\n self.capture_control_screenshot(control_selected)\n\n self._results = self.app_agent.Puppeteer.execute_command(\n self._operation, self._args\n )\n self.control_reannotate = None\n if not utils.is_json_serializable(self._results):\n self._results = \"\"\n\n return\n\n except Exception:\n self.general_error_handler()\n\n def general_error_handler(self) -> None:\n \"\"\"\n Handle general errors.\n \"\"\"\n\n pass\n"
},
{
"path": "dataflow/instantiation/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/instantiation/agent/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/instantiation/agent/filter_agent.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom typing import List\n\nfrom dataflow.prompter.instantiation.filter_prompter import FilterPrompter\nfrom ufo.agents.agent.basic import BasicAgent\n\n\nclass FilterAgent(BasicAgent):\n \"\"\"\n The Agent to evaluate the instantiated task is correct or not.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n process_name: str,\n is_visual: bool,\n main_prompt: str,\n example_prompt: str,\n api_prompt: str,\n ):\n \"\"\"\n Initialize the FilterAgent.\n :param name: The name of the agent.\n :param process_name: The name of the process.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param example_prompt: The example prompt.\n :param api_prompt: The API prompt.\n \"\"\"\n\n self._step = 0\n self._complete = False\n self._name = name\n self._status = None\n self.prompter: FilterPrompter = self.get_prompter(\n is_visual, main_prompt, example_prompt, api_prompt\n )\n self._process_name = process_name\n\n def get_prompter(\n self, is_visual: bool, main_prompt: str, example_prompt: str, api_prompt: str\n ) -> FilterPrompter:\n \"\"\"\n Get the prompt for the agent.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param example_prompt: The example prompt.\n :param api_prompt: The API prompt.\n :return: The prompt string.\n \"\"\"\n\n return FilterPrompter(is_visual, main_prompt, example_prompt, api_prompt)\n\n def message_constructor(self, request: str, app: str) -> List[str]:\n \"\"\"\n Construct the prompt message for the FilterAgent.\n :param request: The request sentence.\n :param app: The name of the operated app.\n :return: The prompt message.\n \"\"\"\n\n filter_agent_prompt_system_message = self.prompter.system_prompt_construction(\n app=app\n )\n filter_agent_prompt_user_message = self.prompter.user_content_construction(\n request\n )\n filter_agent_prompt_message = self.prompter.prompt_construction(\n filter_agent_prompt_system_message, filter_agent_prompt_user_message\n )\n\n return filter_agent_prompt_message\n\n def process_confirmation(self) -> None:\n \"\"\"\n Confirm the process.\n This is the abstract method from BasicAgent that needs to be implemented.\n \"\"\"\n\n pass\n"
},
{
"path": "dataflow/instantiation/agent/prefill_agent.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom typing import Dict, List\n\nfrom dataflow.prompter.instantiation.prefill_prompter import PrefillPrompter\n\nfrom ufo.agents.agent.basic import BasicAgent\n\n\nclass PrefillAgent(BasicAgent):\n \"\"\"\n The Agent for task instantialization and action sequence generation.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n process_name: str,\n is_visual: bool,\n main_prompt: str,\n example_prompt: str,\n api_prompt: str,\n ):\n \"\"\"\n Initialize the PrefillAgent.\n :param name: The name of the agent.\n :param process_name: The name of the process.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param example_prompt: The example prompt.\n :param api_prompt: The API prompt.\n \"\"\"\n\n self._step = 0\n self._complete = False\n self._name = name\n self._status = None\n self.prompter: PrefillPrompter = self.get_prompter(\n is_visual, main_prompt, example_prompt, api_prompt\n )\n self._process_name = process_name\n\n def get_prompter(\n self, is_visual: bool, main_prompt: str, example_prompt: str, api_prompt: str\n ) -> str:\n \"\"\"\n Get the prompt for the agent.\n This is the abstract method from BasicAgent that needs to be implemented.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param example_prompt: The example prompt.\n :param api_prompt: The API prompt.\n :return: The prompt string.\n \"\"\"\n\n return PrefillPrompter(is_visual, main_prompt, example_prompt, api_prompt)\n\n def message_constructor(\n self,\n dynamic_examples: str,\n given_task: str,\n reference_steps: List[str],\n log_path: str,\n ) -> List[str]:\n \"\"\"\n Construct the prompt message for the PrefillAgent.\n :param dynamic_examples: The dynamic examples retrieved from the self-demonstration and human demonstration.\n :param given_task: The given task.\n :param reference_steps: The reference steps.\n :param log_path: The path of the log.\n :return: The prompt message.\n \"\"\"\n\n prefill_agent_prompt_system_message = self.prompter.system_prompt_construction(\n dynamic_examples\n )\n prefill_agent_prompt_user_message = self.prompter.user_content_construction(\n given_task, reference_steps, log_path\n )\n appagent_prompt_message = self.prompter.prompt_construction(\n prefill_agent_prompt_system_message,\n prefill_agent_prompt_user_message,\n )\n\n return appagent_prompt_message\n\n def process_confirmation(self) -> None:\n \"\"\"\n Confirm the process.\n This is the abstract method from BasicAgent that needs to be implemented.\n \"\"\"\n\n pass\n"
},
{
"path": "dataflow/instantiation/agent/template_agent.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nfrom typing import Dict, List\n\nfrom dataflow.prompter.instantiation.template_prompter import TemplatePrompter\n\nfrom ufo.agents.agent.basic import BasicAgent\n\n\nclass TemplateAgent(BasicAgent):\n \"\"\"\n The Agent for choosing template.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n is_visual: bool,\n main_prompt: str,\n template_prompt: str = \"\",\n ):\n \"\"\"\n Initialize the TemplateAgent.\n :param name: The name of the agent.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param template_prompt: The description of the file.\n \"\"\"\n\n self._step = 0\n self._complete = False\n self._name = name\n self._status = None\n self.prompter: TemplatePrompter = self.get_prompter(\n is_visual, main_prompt, template_prompt\n )\n\n def get_prompter(\n self,\n is_visual: bool,\n main_prompt: str,\n template_prompt: str = \"\",\n ) -> str:\n \"\"\"\n Get the prompt for the agent.\n This is the abstract method from BasicAgent that needs to be implemented.\n :param is_visual: The flag indicating whether the agent is visual or not.\n :param main_prompt: The main prompt.\n :param template_prompt: The description of the file.\n :return: The prompt string.\n \"\"\"\n\n return TemplatePrompter(is_visual, main_prompt, template_prompt)\n\n def message_constructor(\n self,\n descriptions: Dict,\n request: str,\n path: str = r\"dataflow\\templates\\word\",\n ) -> List[str]:\n \"\"\"\n Construct the prompt message for the PrefillAgent.\n\n :return: The prompt message.\n \"\"\"\n\n template_agent_prompt_system_message = self.prompter.system_prompt_construction(\n descriptions\n )\n template_agent_prompt_user_message = self.prompter.user_content_construction(\n path=path, request=request\n )\n appagent_prompt_message = self.prompter.prompt_construction(\n template_agent_prompt_system_message,\n template_agent_prompt_user_message,\n )\n\n return appagent_prompt_message\n\n def process_confirmation(self) -> None:\n \"\"\"\n Confirm the process.\n This is the abstract method from BasicAgent that needs to be implemented.\n \"\"\"\n\n pass\n"
},
{
"path": "dataflow/instantiation/workflow/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/instantiation/workflow/choose_template_flow.py",
"content": "import json\nimport os\nimport random\nimport time\nimport warnings\nfrom datetime import datetime\nfrom pathlib import Path\nfrom typing import Dict\n\nfrom langchain.embeddings import CacheBackedEmbeddings\nfrom langchain.storage import LocalFileStore\nfrom langchain_community.embeddings import HuggingFaceEmbeddings\nfrom langchain_community.vectorstores import FAISS\nfrom dataflow.instantiation.agent.template_agent import TemplateAgent\n\nfrom dataflow.config.config import Config\n\n_configs = Config.get_instance().config_data\n\n\nclass ChooseTemplateFlow:\n \"\"\"\n Class to select and copy the most relevant template file based on the given task context.\n \"\"\"\n\n _SENTENCE_TRANSFORMERS_PREFIX = \"sentence-transformers/\"\n\n def __init__(self, app_name: str, task_file_name: str, file_extension: str):\n \"\"\"\n Initialize the flow with the given task context.\n :param app_name: The name of the application.\n :param file_extension: The file extension of the template.\n :param task_file_name: The name of the task file.\n \"\"\"\n\n self._app_name = app_name\n self._file_extension = file_extension\n self._task_file_name = task_file_name\n self.execution_time = None\n self._embedding_model = self._load_embedding_model(\n model_name=_configs[\"CONTROL_FILTER_MODEL_SEMANTIC_NAME\"]\n )\n\n def execute(self) -> str:\n \"\"\"\n Execute the flow and return the copied template path.\n :return: The path to the copied template file.\n \"\"\"\n\n start_time = time.time()\n try:\n template_copied_path = self._choose_template_and_copy()\n except Exception as e:\n raise e\n finally:\n self.execution_time = round(time.time() - start_time, 3)\n return template_copied_path\n\n def _create_copied_file(\n self, copy_from_path: Path, copy_to_folder_path: Path, file_name: str = None\n ) -> str:\n \"\"\"\n Create a cache file from the specified source.\n :param copy_from_path: The original path of the file.\n :param copy_to_folder_path: The path where the cache file will be created.\n :param file_name: Optional; the name of the task file.\n :return: The path to the newly created cache file.\n \"\"\"\n\n os.makedirs(copy_to_folder_path, exist_ok=True)\n copied_template_path = self._generate_copied_file_path(\n copy_to_folder_path, file_name\n )\n\n with open(copy_from_path, \"rb\") as f:\n ori_content = f.read()\n with open(copied_template_path, \"wb\") as f:\n f.write(ori_content)\n\n return copied_template_path\n\n def _generate_copied_file_path(self, folder_path: Path, file_name: str) -> str:\n \"\"\"\n Generate the file path for the copied template.\n :param folder_path: The folder where the file will be created.\n :param file_name: Optional; the name of the task file.\n :return: The path to the newly created file.\n \"\"\"\n\n template_extension = self._file_extension\n if file_name:\n return str(folder_path / f\"{file_name}{template_extension}\")\n timestamp = datetime.now().strftime(\"%Y-%m-%d-%H-%M-%S\")\n return str(folder_path / f\"{timestamp}{template_extension}\")\n\n def _get_chosen_file_path(self) -> str:\n \"\"\"\n Choose the most relevant template file based on the task.\n :return: The path to the most relevant template file.\n \"\"\"\n\n templates_description_path = (\n Path(_configs[\"TEMPLATE_PATH\"]) / self._app_name / \"description.json\"\n )\n\n try:\n with open(templates_description_path, \"r\") as f:\n return self._choose_target_template_file(\n self._task_file_name, json.load(f)\n )\n except FileNotFoundError:\n warnings.warn(\n f\"Warning: {templates_description_path} does not exist. Choosing a random template.\"\n )\n return self._choose_random_template()\n\n def _choose_random_template(self) -> str:\n \"\"\"\n Select a random template file from the template folder.\n :return: The path to the randomly selected template file.\n \"\"\"\n\n template_folder = Path(_configs[\"TEMPLATE_PATH\"]) / self._app_name\n template_files = [f for f in template_folder.iterdir() if f.is_file()]\n\n if not template_files:\n raise Exception(\"No template files found in the specified directory.\")\n\n chosen_template_file = random.choice(template_files)\n print(f\"Randomly selected template: {chosen_template_file.name}\")\n return str(chosen_template_file)\n\n def _choose_template_and_copy(self) -> str:\n \"\"\"\n Choose the template and copy it to the cache folder.\n :return: The path to the copied template file.\n \"\"\"\n\n chosen_template_file_path = self._get_chosen_file_path()\n chosen_template_full_path = (\n Path(_configs[\"TEMPLATE_PATH\"]) / self._app_name / chosen_template_file_path\n )\n\n target_template_folder_path = Path(\n _configs[\"RESULT_HUB\"].format(task_type=\"saved_document\")\n ) / (os.path.dirname(os.path.dirname(self._task_file_name)))\n\n return self._create_copied_file(\n chosen_template_full_path, target_template_folder_path, self._task_file_name\n )\n\n def _choose_target_template_file(\n self, given_task: str, doc_files_description: Dict[str, str]\n ) -> str:\n \"\"\"\n Get the target file based on the semantic similarity of the given task and the template file descriptions.\n :param given_task: The task to be matched.\n :param doc_files_description: A dictionary of template file descriptions.\n :return: The path to the chosen template file.\n \"\"\"\n\n if _configs[\"TEMPLATE_METHOD\"] == \"SemanticSimilarity\":\n return self._choose_target_template_file_semantic(\n given_task, doc_files_description\n )\n elif _configs[\"TEMPLATE_METHOD\"] == \"LLM\":\n self.template_agent = TemplateAgent(\n \"template\",\n is_visual=True,\n main_prompt=_configs[\"TEMPLATE_PROMPT\"],\n )\n return self._choose_target_template_file_llm(\n given_task, doc_files_description\n )\n else:\n raise ValueError(\"Invalid TEMPLATE_METHOD.\")\n\n def _choose_target_template_file_semantic(\n self, given_task: str, doc_files_description: Dict[str, str]\n ) -> str:\n \"\"\"\n Get the target file based on the semantic similarity of the given task and the template file descriptions.\n :param given_task: The task to be matched.\n :param doc_files_description: A dictionary of template file descriptions.\n :return: The path to the chosen template file.\n \"\"\"\n\n file_doc_map = {\n desc: file_name for file_name, desc in doc_files_description.items()\n }\n db = FAISS.from_texts(\n list(doc_files_description.values()), self._embedding_model\n )\n most_similar = db.similarity_search(given_task, k=1)\n\n if not most_similar:\n raise ValueError(\"No similar templates found.\")\n return file_doc_map[most_similar[0].page_content]\n\n def _choose_target_template_file_llm(\n self, given_task: str, doc_files_description: Dict[str, str]\n ) -> str:\n \"\"\"\n Get the target file based on the LLM of the given task and the template file descriptions.\n :param given_task: The task to be matched.\n :param doc_files_description: A dictionary of template file descriptions.\n :return: The path to the chosen template file.\n \"\"\"\n\n prompt_message = self.template_agent.message_constructor(\n doc_files_description, given_task\n )\n response_string, _ = self.template_agent.get_response(\n prompt_message, \"prefill\", use_backup_engine=True, configs=_configs\n )\n if response_string is None:\n raise ValueError(\"No similar templates found.\")\n elif \"```json\" in response_string:\n response_string = response_string[7:-3]\n response_json = json.loads(response_string)\n file_name = list(response_json.keys())[0]\n if file_name not in doc_files_description:\n print(f\"Template {file_name} not found in the description.\")\n raise ValueError(\"No similar templates found.\")\n return file_name\n\n @staticmethod\n def _load_embedding_model(model_name: str) -> CacheBackedEmbeddings:\n \"\"\"\n Load the embedding model.\n :param model_name: The name of the embedding model to load.\n :return: The loaded embedding model.\n \"\"\"\n\n store = LocalFileStore(_configs[\"CONTROL_EMBEDDING_CACHE_PATH\"])\n if not model_name.startswith(ChooseTemplateFlow._SENTENCE_TRANSFORMERS_PREFIX):\n model_name = ChooseTemplateFlow._SENTENCE_TRANSFORMERS_PREFIX + model_name\n embedding_model = HuggingFaceEmbeddings(model_name=model_name)\n return CacheBackedEmbeddings.from_bytes_store(\n embedding_model, store, namespace=model_name\n )\n"
},
{
"path": "dataflow/instantiation/workflow/filter_flow.py",
"content": "import json\nimport logging\nimport os\nimport time\nfrom typing import Dict, Tuple, Any\n\nfrom dataflow.config.config import Config\nfrom dataflow.instantiation.agent.filter_agent import FilterAgent\nfrom ufo.module.basic import BaseSession\n\n_configs = Config.get_instance().config_data\n\n\nclass FilterFlow:\n \"\"\"\n Class to refine the plan steps and prefill the file based on filtering criteria.\n \"\"\"\n\n _app_filter_agent_dict: Dict[str, FilterAgent] = {}\n\n def __init__(self, app_name: str, task_file_name: str) -> None:\n \"\"\"\n Initialize the filter flow for a task.\n :param app_name: Name of the application being processed.\n :param task_file_name: Name of the task file being processed.\n \"\"\"\n\n self.execution_time = None\n self._app_name = app_name\n self._log_path_configs = _configs[\"FILTER_LOG_PATH\"].format(task=task_file_name)\n self._filter_agent = self._get_or_create_filter_agent()\n self._initialize_logs()\n\n def _get_or_create_filter_agent(self) -> FilterAgent:\n \"\"\"\n Retrieve or create a filter agent for the given application.\n :return: FilterAgent instance for the specified application.\n \"\"\"\n\n if self._app_name not in FilterFlow._app_filter_agent_dict:\n FilterFlow._app_filter_agent_dict[self._app_name] = FilterAgent(\n \"filter\",\n self._app_name,\n is_visual=True,\n main_prompt=_configs[\"FILTER_PROMPT\"],\n example_prompt=\"\",\n api_prompt=_configs[\"API_PROMPT\"],\n ) \n return FilterFlow._app_filter_agent_dict[self._app_name]\n\n def execute(self, instantiated_request: str) -> Dict[str, Any]:\n \"\"\"\n Execute the filter flow: Filter the task and save the result.\n :param instantiated_request: Request object to be filtered.\n :return: Tuple containing task quality flag, comment, and task type.\n \"\"\"\n\n start_time = time.time()\n try:\n judge, thought, request_type = self._get_filtered_result(\n instantiated_request\n )\n except Exception as e:\n raise e\n finally:\n self.execution_time = round(time.time() - start_time, 3)\n return {\n \"judge\": judge,\n \"thought\": thought,\n \"request_type\": request_type,\n }\n \n def _initialize_logs(self) -> None:\n \"\"\"\n Initialize logging for filter messages and responses.\n \"\"\"\n\n os.makedirs(self._log_path_configs, exist_ok=True)\n self._filter_message_logger = BaseSession.initialize_logger(\n self._log_path_configs, \"filter_messages.json\", \"w\", _configs\n )\n self._filter_response_logger = BaseSession.initialize_logger(\n self._log_path_configs, \"filter_responses.json\", \"w\", _configs\n )\n\n def _get_filtered_result(self, instantiated_request: str) -> Tuple[bool, str, str]:\n \"\"\"\n Get the filtered result from the filter agent.\n :param instantiated_request: Request object containing task details.\n :return: Tuple containing task quality flag, request comment, and request type.\n \"\"\"\n\n # Construct the prompt message for the filter agent\n prompt_message = self._filter_agent.message_constructor(\n instantiated_request,\n self._app_name,\n )\n prompt_json = json.dumps(prompt_message, indent=4)\n self._filter_message_logger.info(prompt_json)\n\n # Get the response from the filter agent\n try:\n start_time = time.time()\n response_string, _ = self._filter_agent.get_response(\n prompt_message, \"filter\", use_backup_engine=True, configs=_configs\n )\n try:\n fixed_response_string = self._fix_json_commas(response_string)\n response_json = self._filter_agent.response_to_dict(\n fixed_response_string\n )\n except json.JSONDecodeError as e:\n logging.error(\n f\"JSONDecodeError: {e.msg} at position {e.pos}. Response: {response_string}\"\n )\n raise e\n\n execution_time = round(time.time() - start_time, 3)\n\n response_json[\"execution_time\"] = execution_time\n self._filter_response_logger.info(json.dumps(response_json, indent=4))\n\n return (\n response_json[\"judge\"],\n response_json[\"thought\"],\n response_json[\"type\"],\n )\n except Exception as e:\n logging.error(f\"Error occurred while filtering: {e}\")\n raise e\n\n def _fix_json_commas(self, json_string: str) -> str:\n \"\"\"\n Function to add missing commas between key-value pairs in a JSON string\n and remove newline characters for proper formatting.\n :param json_string: The JSON string to be fixed.\n :return: The fixed JSON string.\n \"\"\"\n\n # Remove newline characters\n json_string = json_string.replace(\"\\n\", \"\")\n\n return json_string"
},
{
"path": "dataflow/instantiation/workflow/prefill_flow.py",
"content": "import json\nimport logging\nimport os\nimport time\nfrom typing import Any, Dict, List, Tuple\n\nfrom dataflow.config.config import Config\nfrom dataflow.instantiation.agent.prefill_agent import PrefillAgent\nfrom dataflow.env.env_manager import WindowsAppEnv\nfrom ufo.agents.processors.app_agent_processor import AppAgentProcessor\nfrom ufo.automator.ui_control.inspector import ControlInspectorFacade\nfrom ufo.automator.ui_control.screenshot import PhotographerFacade\nfrom ufo.module.basic import BaseSession\nfrom ufo.config import Config as UFOConfig\n\n_configs = Config.get_instance().config_data\n_ufo_configs = UFOConfig.get_instance().config_data\n_BACKEND = \"uia\"\n\n\nclass PrefillFlow(AppAgentProcessor):\n \"\"\"\n Class to manage the prefill process by refining planning steps and automating UI interactions\n \"\"\"\n\n _app_prefill_agent_dict: Dict[str, PrefillAgent] = {}\n\n def __init__(\n self,\n app_name: str,\n task_file_name: str,\n environment: WindowsAppEnv,\n ) -> None:\n \"\"\"\n Initialize the prefill flow with the application context.\n :param app_name: The name of the application.\n :param task_file_name: The name of the task file for logging and tracking.\n :param environment: The environment of the app.\n \"\"\"\n\n self.execution_time = None\n self._app_name = app_name\n self._task_file_name = task_file_name\n self._app_env = environment\n # Create or reuse a PrefillAgent for the app\n if self._app_name not in PrefillFlow._app_prefill_agent_dict:\n PrefillFlow._app_prefill_agent_dict[self._app_name] = PrefillAgent(\n \"prefill\",\n self._app_name,\n is_visual=True,\n main_prompt=_configs[\"PREFILL_PROMPT\"],\n example_prompt=_configs[\"PREFILL_EXAMPLE_PROMPT\"],\n api_prompt=_configs[\"API_PROMPT\"],\n )\n self._prefill_agent = PrefillFlow._app_prefill_agent_dict[self._app_name]\n\n # Initialize execution step and UI control tools\n self._execute_step = 0\n self._control_inspector = ControlInspectorFacade(_BACKEND)\n self._photographer = PhotographerFacade()\n\n # Set default states\n self._status = \"\"\n\n # Initialize loggers for messages and responses\n self._log_path_configs = _configs[\"PREFILL_LOG_PATH\"].format(\n task=self._task_file_name\n )\n os.makedirs(self._log_path_configs, exist_ok=True)\n\n # Set up loggers\n self._message_logger = BaseSession.initialize_logger(\n self._log_path_configs, \"prefill_messages.json\", \"w\", _configs\n )\n self._response_logger = BaseSession.initialize_logger(\n self._log_path_configs, \"prefill_responses.json\", \"w\", _configs\n )\n\n def execute(\n self, template_copied_path: str, original_task: str, refined_steps: List[str]\n ) -> Dict[str, Any]:\n \"\"\"\n Start the execution by retrieving the instantiated result.\n :param template_copied_path: The path of the copied template to use.\n :param original_task: The original task to refine.\n :param refined_steps: The steps to guide the refinement process.\n :return: The refined task and corresponding action plans.\n \"\"\"\n\n start_time = time.time()\n try:\n instantiated_request, instantiated_plan = self._instantiate_task(\n template_copied_path, original_task, refined_steps\n )\n except Exception as e:\n raise e\n finally:\n self.execution_time = round(time.time() - start_time, 3)\n\n return {\n \"instantiated_request\": instantiated_request,\n \"instantiated_plan\": instantiated_plan,\n }\n\n def _instantiate_task(\n self, template_copied_path: str, original_task: str, refined_steps: List[str]\n ) -> Tuple[str, List[str]]:\n \"\"\"\n Retrieve and process the instantiated result for the task.\n Interacts with the PrefillAgent to refine the task and generate action plans.\n :param template_copied_path: The path of the copied template to use.\n :param original_task: The original task to refine.\n :param refined_steps: The steps to guide the refinement process.\n :return: The refined task and corresponding action plans.\n \"\"\"\n\n try:\n # Retrieve prefill actions and task plan\n instantiated_request, instantiated_plan = self._get_prefill_actions(\n original_task,\n refined_steps,\n template_copied_path,\n )\n\n print(f\"Original Task: {original_task}\")\n print(f\"Prefilled Task: {instantiated_request}\")\n\n except Exception as e:\n logging.exception(f\"Error in prefilling task: {e}\")\n raise e\n\n return instantiated_request, instantiated_plan\n\n def _update_state(self, file_path: str) -> None:\n \"\"\"\n Update the current state of the app by inspecting UI elements.\n :param file_path: Path of the app file to inspect.\n \"\"\"\n\n print(f\"Updating the app state using the file: {file_path}\")\n\n # Retrieve control elements in the app window\n control_list = self._control_inspector.find_control_elements_in_descendants(\n self._app_env.app_window,\n control_type_list=_ufo_configs[\"CONTROL_LIST\"],\n class_name_list=_ufo_configs[\"CONTROL_LIST\"],\n )\n\n # Capture UI control annotations\n self._annotation_dict = self._photographer.get_annotation_dict(\n self._app_env.app_window, control_list, annotation_type=\"number\"\n )\n\n # Filter out irrelevant control elements\n self._filtered_annotation_dict = self.get_filtered_annotation_dict(\n self._annotation_dict, configs=_configs\n )\n\n # Gather control info for both full and filtered lists\n self._control_info = self._control_inspector.get_control_info_list_of_dict(\n self._annotation_dict,\n [\"control_text\", \"control_type\" if _BACKEND == \"uia\" else \"control_class\"],\n )\n self._filtered_control_info = (\n self._control_inspector.get_control_info_list_of_dict(\n self._filtered_annotation_dict,\n [\n \"control_text\",\n \"control_type\" if _BACKEND == \"uia\" else \"control_class\",\n ],\n )\n )\n\n def _get_prefill_actions(\n self, given_task: str, reference_steps: List[str], file_path: str\n ) -> Tuple[str, List[str]]:\n \"\"\"\n Generate refined tasks and action plans using the PrefillAgent.\n :param given_task: The task to refine.\n :param reference_steps: Reference steps for the task.\n :param file_path: Path to the task template.\n :return: The refined task and corresponding action plans.\n \"\"\"\n\n self._update_state(file_path)\n execution_time = 0\n # Save a screenshot of the app state\n screenshot_path = os.path.join(self._log_path_configs, \"screenshot.png\")\n self._save_screenshot(self._task_file_name, screenshot_path)\n\n # Construct prompt message for the PrefillAgent\n prompt_message = self._prefill_agent.message_constructor(\n \"\",\n given_task,\n reference_steps,\n self._log_path_configs,\n )\n\n # Log the constructed message\n self._log_message(prompt_message)\n\n try:\n # Record start time and get PrefillAgent response\n start_time = time.time()\n response_string, _ = self._prefill_agent.get_response(\n prompt_message, \"prefill\", use_backup_engine=True, configs=_configs\n )\n execution_time = round(time.time() - start_time, 3)\n\n # Parse and log the response\n response_json = self._prefill_agent.response_to_dict(response_string)\n instantiated_request = response_json[\"New_task\"]\n instantiated_plan = response_json[\"Actions_plan\"]\n\n except Exception as e:\n self._status = \"ERROR\"\n logging.exception(f\"Error in prefilling task: {e}\")\n raise e\n finally:\n # Log the response and execution time\n self._log_response(response_json, execution_time)\n\n return instantiated_request, instantiated_plan\n\n def _log_message(self, prompt_message: str) -> None:\n \"\"\"\n Log the constructed prompt message for the PrefillAgent.\n :param prompt_message: The message constructed for PrefillAgent.\n \"\"\"\n\n messages_log_entry = {\n \"step\": self._execute_step,\n \"messages\": prompt_message,\n \"error\": \"\",\n }\n self._message_logger.info(json.dumps(messages_log_entry, indent=4))\n\n def _log_response(\n self, response_json: Dict[str, Any], execution_time: float\n ) -> None:\n \"\"\"\n Log the response received from PrefillAgent along with execution time.\n :param response_json: Response data from PrefillAgent.\n :param execution_time: Time taken for the PrefillAgent call.\n \"\"\"\n\n response_log_entry = {\n \"step\": self._execute_step,\n \"execution_time\": execution_time,\n \"agent_response\": response_json,\n \"error\": \"\",\n }\n self._response_logger.info(json.dumps(response_log_entry, indent=4))\n\n def _save_screenshot(self, doc_name: str, save_path: str) -> None:\n \"\"\"\n Captures a screenshot of the current window or the full screen if the window is not found.\n :param doc_name: The name or description of the document to match the window.\n :param save_path: The path where the screenshot will be saved.\n \"\"\"\n\n try:\n # Find the window matching the document name\n matched_window = self._app_env.find_matching_window(doc_name)\n if matched_window:\n screenshot = self._photographer.capture_app_window_screenshot(\n matched_window\n )\n else:\n logging.warning(\"Window not found, taking a full-screen screenshot.\")\n screenshot = self._photographer.capture_desktop_screen_screenshot()\n\n screenshot.save(save_path)\n print(f\"Screenshot saved to {save_path}\")\n except Exception as e:\n logging.exception(f\"Failed to save screenshot: {e}\")\n raise e\n"
},
{
"path": "dataflow/prompter/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/prompter/execution/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/prompter/execution/execute_eval_prompter.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport json\nimport os\nfrom typing import Dict, List, Optional\n\nfrom ufo.prompter.basic import BasicPrompter\nfrom ufo.prompter.eva_prompter import EvaluationAgentPrompter\n\nclass ExecuteEvalAgentPrompter(EvaluationAgentPrompter):\n \"\"\"\n Execute the prompt for the ExecuteEvalAgent.\n \"\"\"\n\n def __init__(\n self,\n is_visual: bool,\n prompt_template: str,\n example_prompt_template: str,\n api_prompt_template: str,\n root_name: Optional[str] = None,\n ):\n \"\"\"\n Initialize the CustomEvaluationAgentPrompter.\n :param is_visual: Whether the request is for visual model.\n :param prompt_template: The path of the prompt template.\n :param example_prompt_template: The path of the example prompt template.\n :param api_prompt_template: The path of the api prompt template.\n :param root_name: The name of the root application.\n \"\"\"\n\n super().__init__(\n is_visual,\n prompt_template,\n example_prompt_template,\n api_prompt_template,\n root_name,\n )\n\n @staticmethod\n def load_logs(log_path: str) -> List[Dict[str, str]]:\n \"\"\"\n Load logs from the log path.\n :param log_path: The path of the log.\n \"\"\"\n\n log_file_path = os.path.join(log_path, \"execute_log.json\")\n with open(log_file_path, \"r\") as f:\n logs = f.readlines()\n logs = [json.loads(log) for log in logs]\n return logs\n"
},
{
"path": "dataflow/prompter/instantiation/__init__.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License."
},
{
"path": "dataflow/prompter/instantiation/filter_prompter.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport json\nfrom typing import Dict, List\n\nfrom ufo.prompter.basic import BasicPrompter\nfrom ufo.prompter.prompt_sanitizer import sanitize_user_input\n\n\nclass FilterPrompter(BasicPrompter):\n \"\"\"\n Load the prompt for the FilterAgent.\n \"\"\"\n\n def __init__(\n self,\n is_visual: bool,\n prompt_template: str,\n example_prompt_template: str,\n api_prompt_template: str,\n ):\n \"\"\"\n Initialize the FilterPrompter.\n :param is_visual: The flag indicating whether the prompter is visual or not.\n :param prompt_template: The prompt template.\n :param example_prompt_template: The example prompt template.\n :param api_prompt_template: The API prompt template.\n \"\"\"\n\n super().__init__(is_visual, prompt_template, example_prompt_template)\n self.api_prompt_template = self.load_prompt_template(\n api_prompt_template, is_visual\n )\n\n def api_prompt_helper(self, apis: Dict = {}, verbose: int = 1) -> str:\n \"\"\"\n Construct the prompt for APIs.\n :param apis: The APIs.\n :param verbose: The verbosity level.\n :return: The prompt for APIs.\n \"\"\"\n\n # Construct the prompt for APIs\n if len(apis) == 0:\n api_list = [\n \"- The action type are limited to {actions}.\".format(\n actions=list(self.api_prompt_template.keys())\n )\n ]\n\n # Construct the prompt for each API\n for key in self.api_prompt_template.keys():\n api = self.api_prompt_template[key]\n if verbose > 0:\n api_text = \"{summary}\\n{usage}\".format(\n summary=api[\"summary\"], usage=api[\"usage\"]\n )\n else:\n api_text = api[\"summary\"]\n\n api_list.append(api_text)\n\n api_prompt = self.retrieved_documents_prompt_helper(\"\", \"\", api_list)\n else:\n api_list = [\n \"- The action type are limited to {actions}.\".format(\n actions=list(apis.keys())\n )\n ]\n\n # Construct the prompt for each API\n for key in apis.keys():\n api = apis[key]\n api_text = \"{description}\\n{example}\".format(\n description=api[\"description\"], example=api[\"example\"]\n )\n api_list.append(api_text)\n\n api_prompt = self.retrieved_documents_prompt_helper(\"\", \"\", api_list)\n\n return api_prompt\n\n def system_prompt_construction(self, app: str = \"\") -> str:\n \"\"\"\n Construct the prompt for the system.\n :param app: The app name.\n :return: The prompt for the system.\n \"\"\"\n\n try:\n ans = self.prompt_template[\"system\"]\n ans = ans.format(app=app)\n return ans\n except Exception as e:\n print(e)\n\n def user_prompt_construction(self, request: str) -> str:\n \"\"\"\n Construct the prompt for the user.\n :param request: The user request.\n :return: The prompt for the user.\n \"\"\"\n\n prompt = self.prompt_template[\"user\"].format(\n request=sanitize_user_input(request, \"request\"),\n )\n return prompt\n\n def user_content_construction(self, request: str) -> List[Dict]:\n \"\"\"\n Construct the prompt for LLMs.\n :param request: The user request.\n :return: The prompt for LLMs.\n \"\"\"\n\n user_content = []\n\n user_content.append(\n {\"type\": \"text\", \"text\": self.user_prompt_construction(request)}\n )\n\n return user_content\n\n def examples_prompt_helper(\n self,\n header: str = \"## Response Examples\",\n separator: str = \"Example\",\n additional_examples: List[str] = [],\n ) -> str:\n \"\"\"\n Construct the prompt for examples.\n :param header: The header of the prompt.\n :param separator: The separator of the prompt.\n :param additional_examples: The additional examples.\n :return: The prompt for examples.\n \"\"\"\n\n template = \"\"\"\n [User Request]:\n {request}\n [Response]:\n {response}\n [Tips]:\n {tip}\n \"\"\"\n\n example_list = []\n\n for key in self.example_prompt_template.keys():\n if key.startswith(\"example\"):\n example = template.format(\n request=self.example_prompt_template[key].get(\"Request\"),\n response=json.dumps(\n self.example_prompt_template[key].get(\"Response\")\n ),\n tip=self.example_prompt_template[key].get(\"Tips\", \"\"),\n )\n example_list.append(example)\n\n example_list += [json.dumps(example) for example in additional_examples]\n\n return self.retrieved_documents_prompt_helper(header, separator, example_list)\n"
},
{
"path": "dataflow/prompter/instantiation/prefill_prompter.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport json\nimport os\nfrom typing import Dict, List\n\nfrom ufo.prompter.basic import BasicPrompter\nfrom ufo.prompter.prompt_sanitizer import sanitize_user_input\n\n\nclass PrefillPrompter(BasicPrompter):\n \"\"\"\n Load the prompt for the PrefillAgent.\n \"\"\"\n\n def __init__(\n self,\n is_visual: bool,\n prompt_template: str,\n example_prompt_template: str,\n api_prompt_template: str,\n ):\n \"\"\"\n Initialize the PrefillPrompter.\n :param is_visual: The flag indicating whether the prompter is visual or not.\n :param prompt_template: The prompt template.\n :param example_prompt_template: The example prompt template.\n :param api_prompt_template: The API prompt template.\n \"\"\"\n\n super().__init__(is_visual, prompt_template, example_prompt_template)\n self.api_prompt_template = self.load_prompt_template(\n api_prompt_template, is_visual\n )\n\n def api_prompt_helper(self, verbose: int = 1) -> str:\n \"\"\"\n Construct the prompt for APIs.\n :param verbose: The verbosity level.\n :return: The prompt for APIs.\n \"\"\"\n\n # Construct the prompt for APIs\n api_list = [\n \"- The action type are limited to {actions}.\".format(\n actions=list(self.api_prompt_template.keys())\n )\n ]\n\n # Construct the prompt for each API\n for key in self.api_prompt_template.keys():\n api = self.api_prompt_template[key]\n if verbose > 0:\n api_text = \"{summary}\\n{usage}\".format(\n summary=api[\"summary\"], usage=api[\"usage\"]\n )\n else:\n api_text = api[\"summary\"]\n\n api_list.append(api_text)\n\n api_prompt = self.retrieved_documents_prompt_helper(\"\", \"\", api_list)\n\n return api_prompt\n\n def system_prompt_construction(self, additional_examples: List = []) -> str:\n \"\"\"\n Construct the prompt for the system.\n :param additional_examples: The additional examples.\n :return: The prompt for the system.\n \"\"\"\n\n examples = self.examples_prompt_helper(additional_examples=additional_examples)\n apis = self.api_prompt_helper(verbose=0)\n return self.prompt_template[\"system\"].format(apis=apis, examples=examples)\n\n def user_prompt_construction(\n self, given_task: str, reference_steps: List\n ) -> str:\n \"\"\"\n Construct the prompt for the user.\n :param given_task: The given task.\n :param reference_steps: The reference steps.\n :return: The prompt for the user.\n \"\"\"\n\n prompt = self.prompt_template[\"user\"].format(\n given_task=sanitize_user_input(given_task, \"given_task\"),\n reference_steps=sanitize_user_input(json.dumps(reference_steps), \"reference_steps\")\n )\n\n return prompt\n\n def load_screenshots(self, log_path: str) -> str:\n \"\"\"\n Load the first and last screenshots from the log path.\n :param log_path: The path of the log.\n :return: The screenshot URL.\n \"\"\"\n\n from ufo.prompter.eva_prompter import EvaluationAgentPrompter\n\n init_image = os.path.join(log_path, \"screenshot.png\")\n init_image_url = EvaluationAgentPrompter.load_single_screenshot(init_image)\n return init_image_url\n\n def user_content_construction(\n self,\n given_task: str,\n reference_steps: List,\n log_path: str,\n ) -> List[Dict]:\n \"\"\"\n Construct the prompt for LLMs.\n :param given_task: The given task.\n :param reference_steps: The reference steps.\n :param log_path: The path of the log.\n :return: The prompt for LLMs.\n \"\"\"\n\n user_content = []\n if self.is_visual:\n screenshot = self.load_screenshots(log_path)\n screenshot_text = \"\"\"You are a action prefill agent, responsible to prefill the given task.\n This is the screenshot of the current environment, please check it and give prefilled task accodingly.\"\"\"\n\n user_content.append({\"type\": \"text\", \"text\": screenshot_text})\n user_content.append({\"type\": \"image_url\", \"image_url\": {\"url\": screenshot}})\n\n user_content.append(\n {\n \"type\": \"text\",\n \"text\": self.user_prompt_construction(\n given_task, reference_steps\n ),\n }\n )\n\n return user_content\n\n def examples_prompt_helper(\n self,\n header: str = \"## Response Examples\",\n separator: str = \"Example\",\n additional_examples: List[str] = [],\n ) -> str:\n \"\"\"\n Construct the prompt for examples.\n :param header: The header of the prompt.\n :param separator: The separator of the prompt.\n :param additional_examples: The additional examples.\n :return: The prompt for examples.\n \"\"\"\n\n template = \"\"\"\n [User Request]:\n {request}\n [Response]:\n {response}\n [Tips]:\n {tip}\n \"\"\"\n\n example_list = []\n\n for key in self.example_prompt_template.keys():\n if key.startswith(\"example\"):\n example = template.format(\n request=self.example_prompt_template[key].get(\"Request\"),\n response=json.dumps(\n self.example_prompt_template[key].get(\"Response\")\n ),\n tip=self.example_prompt_template[key].get(\"Tips\", \"\"),\n )\n example_list.append(example)\n\n example_list += [json.dumps(example) for example in additional_examples]\n\n return self.retrieved_documents_prompt_helper(header, separator, example_list)\n"
},
{
"path": "dataflow/prompter/instantiation/template_prompter.py",
"content": "# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\nimport base64\nimport mimetypes\nimport os\nfrom typing import Dict, List, cast, Optional\n\nfrom ufo.prompter.basic import BasicPrompter\nfrom ufo.prompter.prompt_sanitizer import sanitize_user_input\n\n\nclass TemplatePrompter(BasicPrompter):\n \"\"\"\n Load the prompt for the TemplateAgent.\n \"\"\"\n\n def __init__(\n self,\n is_visual: bool,\n prompt_template: str,\n example_prompt_template: str,\n ):\n \"\"\"\n Initialize the FilterPrompter.\n :param is_visual: The flag indicating whether the prompter is visual or not.\n :param prompt_template: The prompt template.\n \"\"\"\n\n super().__init__(is_visual, prompt_template, example_prompt_template)\n\n def encode_image(self, image_path: str) -> str:\n \"\"\"\n Encode the image.\n :param image_path: The image path.\n :return: The encoded image.\n \"\"\"\n with open(image_path, \"rb\") as image_file:\n encoded_image = base64.b64encode(image_file.read()).decode(\"ascii\")\n\n mime_type = \"image/png\"\n\n image_url = f\"data:{mime_type};base64,\" + encoded_image\n return image_url\n\n def file_prompt_helper(self, path) -> str:\n \"\"\"\n Construct the prompt for files.\n :return: The prompt for files.\n \"\"\"\n image_path = os.path.join(path, \"images\")\n image_urls = []\n user_content = []\n for file in os.listdir(image_path):\n if file.endswith(\".png\"):\n image_urls.append(self.encode_image(os.path.join(image_path, file)))\n\n for i in range(len(image_urls)):\n user_content.append(\n {\n \"type\": \"text\",\n \"text\": \"This is the screenshot of \" + str(i + 1) + \".docx\",\n },\n )\n user_content.append(\n {\"type\": \"image_url\", \"image_url\": {\"url\": image_urls[i]}},\n )\n return user_content\n\n def system_prompt_construction(self, descriptions: str = \"\") -> str:\n \"\"\"\n Construct the prompt for the system.\n :param app: The app name.\n :return: The prompt for the system.\n \"\"\"\n\n try:\n ans = self.prompt_template[\"system\"]\n ans = ans.format(descriptions=descriptions)\n return ans\n except Exception as e:\n print(e)\n\n def user_prompt_construction(self, request: str) -> str:\n \"\"\"\n Construct the prompt for the user.\n :param request: The user request.\n :return: The prompt for the user.\n \"\"\"\n\n prompt = self.prompt_template[\"user\"].format(\n given_task=sanitize_user_input(request, \"given_task\"),\n )\n return prompt\n\n def user_content_construction(self, path: str, request: str) -> List[Dict]:\n \"\"\"\n Construct the prompt for LLMs.\n :param path: The path of the template.\n :param request: The user request.\n :return: The prompt for LLMs.\n \"\"\"\n\n user_content = self.file_prompt_helper(path)\n\n user_content.append(\n {\"type\": \"text\", \"text\": self.user_prompt_construction(request)}\n )\n\n return user_content\n"
},
{
"path": "dataflow/prompts/instantiation/visual/filter.yaml",
"content": "version: 1.0\n\nsystem: |-\n You are a task judge, will be provided with a task in the . You need to judge whether this task can be executed locally.\n\n ## Evaluation Dimension\n The task is only related to {app}.\n This task should be like a task, not subjective considerations. For example, if there are 'custom', 'you want' and other situations, they cannot be considered and should return false and be classified as Non_task. Any subjective will crash the system.\n This task should specify the element, for example, if there are only 'text' without the specific string, they cannot be considered and should return false and be classified as Non_task.\n This task should not involve interactions with other application plug-ins, etc., and only rely on Word. If 'Excel', 'Edge' and other interactions are involved, it should return false and be classified as App_involve.\n This task should not involve version updates and other interactions that depend on the environment, but only rely on the current version, and do not want to be upgraded or downgraded. It should return false and be classified as Env.\n There are other things that you think cannot be executed or are irrelevant, return False, and be classified as Others\n \n ## Response Format\n Your response should be strictly structured in a JSON format, consisting of three distinct parts with the following keys and corresponding content:\n {{\n \"judge\": true or false depends on you think this task whether can be performed,\n \"thought\": \"Outline the reason why you give the judgement.\",\n \"type\": \"None/Non_task/App_involve/Env/Others\"\n }}\n Make sure you answer must be strictly in JSON format only, without other redundant text such as json header. Otherwise it will crash the system.\n Below is only a example of the response. Do not fall in the example.\n\nuser: |-\n {request}\n "
},
{
"path": "dataflow/prompts/instantiation/visual/prefill.yaml",
"content": "version: 1.0\n\nsystem: |-\n You are a Agent Task Creator and planer.\n You will receive a that is abstract and your objective is to instantiate this task, and give the step-by-step actions to take.\n - You should review the doc screenshot to detail the to a .\n - You are provided with , you should review the acions carefully and choose the most suitable ones step-by-step .\n You are also provided with some steps to reference in \n - You should also review these steps carefully, to help you instantiate the original task and give the actions.\n \n\n ## Control item\n - The control item is the element on the page that you can interact with, we limit the actionable control item to the following:\n - \"Button\" is the control item that you can click.\n - \"Edit\" is the control item that you can click and input text.\n - \"TabItem\" is the control item that you can click and switch to another page.\n - \"ListItem\" is the control item that you can click and select.\n - \"MenuItem\" is the control item that you can click and select.\n - \"ScrollBar\" is the control item that you can scroll.\n - \"TreeItem\" is the control item that you can click and select.\n - \"Document\" is the control item that you can click and select text.\n - \"Hyperlink\" is the control item that you can click and open a link.\n - \"ComboBox\" is the control item that you can click and input text. The Google search box is an example of ComboBox.\n\n ## Available Actions on the control item\n - All the available actions are listed below:\n {apis}\n\n Besides, please prefill the task based on the screenshot. you will also be provided with a screenshot, one before the agent's execution and one after the agent's execution.\n\n ## The requirements for \n 1. The must based on the given task, but if more then one options exist in , you must choose one of them.\n 2. The must be able to be completed step-by-step by a Windows Operating System or an Application on Windows platform.\n 3. The should be specific and individual, you should not provide different options.\n 4. You should keep clear and objective, any vague vocabulary or any subjective terms are forbidden.\n 5. You should try your best not to make the become verbose, can only add up to 50 words into .\n 6. The detailed target in should be specific and clear based on the doc canvas content and control information.\n 7. The should be able to implemented by the available controls and actions.\n\n \n ## The requirements for \n 1. The should be step-by-step actions to take in the doc file environment.\n 2. Each action should be in the available actions from .\n 3. Each action should be generated with a \"step\" description which is the function description of the action.\n 4. No need to explain the purpose of the action, just give the actions to take.\n 5. Each plan should focus on a single action, if multiple actions need to be performed, you should separate them into different steps.\n \n ## Response Format\n - You are required to response in a JSON format, consisting of several distinct parts with the following keys and corresponding content:\n {{\n \"Observation\": ,\n \"Thought\": ,\n \"New_task\":,\n \"Actions_plan\":\n }}\n \n ### Action Call Format\n - The action call format is the same as the available actions in the API list.You are required to provide the action call format in a JSON format:\n {{\n \"Step\": \n \"Subtask\": \n \"ControlLabel\": . If you believe none of the control item is suitable for the task or the task is complete, kindly output a empty string ''.>\n \"ControlText\": .The control text must match exactly with the selected control label. \n If the function to call don't need specify controlText or the task is complete,you can kindly output an empty string ''.\n If the function to call need to specify controlText and none of the control item is suitable for the task,you should input a possible control name.> \n \"Function\": \n \"Args\": \n }}\n\n e.g.\n {{\n \"Step\": 1\n \"Subtask\": \"change the borders\",\n \"ControlLabel\": \"\",\n \"ControlText\": \"Borders\",\n \"Function\": \"click_input\",\n \"Args\": {{\n \"button\": \"left\",\n \"double\": false\n }}\n }}\n\n {{\n \"Step\": 2, \n \"Subtask\": \"change the borders\",\n \"ControlLabel\": \"101\",\n \"ControlText\": \"Borders\",\n \"Function\": \"click_input\",\n \"Args\": {{\n \"control_id\": \"101\",\n \"button\": \"left\",\n \"double\": false\n }}\n }}\n\n {{\n \"Step\": 3, \n \"Subtask\": \"select the target text\",\n \"ControlLabel\": \"\",\n \"ControlText\": \"\",\n \"Function\": \"select_text\",\n \"Args\": {{\n \"text\": \"Test For Fun\"\n }}\n }}\n\n - The field must be strictly in a format separated each action call by \"\\n\". The list format should be like this:\n \"action call 1\\naction call 2\\naction call 3\"\n - If you think the original task don't need to be detailed, you can directly copy the original task to the \"New_task\".\n - You should review the apis function carefully and if the function to call need to specify target control,the 'controlText' field\n cannot be set empty.\n - The \"Subtask\" description should be consistent with the action and also the thought.\n\n ## Here are some examples for you to complete the user request:\n {examples}\n\n ## Tips\n - Read the above instruction carefully. Make sure the response and action strictly following these instruction and meet the user request.\n - Make sure you answer must be strictly in JSON format only, without other redundant text such as json header. Your output must be able to be able to be parsed by json.loads(). Otherwise, it will crash the system and destroy the user's computer.\n - Your task is very important to improve the agent's performance. I will tip you 200$ if you do well. Thank you for your hard work!\n\nuser: |-\n {given_task}\n {reference_steps}\n "
},
{
"path": "dataflow/prompts/instantiation/visual/prefill_example.yaml",
"content": "\nversion: 1.0\n\nexample1: \n Request: |-\n Delete Text in document.\n Response:\n observation: |-\n I observe the canvas state is a Word document with a body containing a paragraph with a run element, which has a text element 'text to edit'.\n thought: |-\n My task is to detail the given task and give the step-by-step actions to take. \n The user needs to delete text in the Word document. \n Based on the canvas state, there is a text element 'text to edit'.\n And based on the available apis and controls,the user can use \"select_text\" to select the target to delete,and \"type_keys\" to type in delete.\n Therefore,the user can detail the task to delete 'text to edit' in the Word document.\n In this case, the user should select the text to edit in the Word document and press the 'Delete' key on the keyboard to delete the selected text.\n new_task: |-\n Delete the 'text to edit' in the Word document.\n action_plans: |-\n {{\"step 1\":\"choose the target text 'text to edit'\",\"controlLabel\": \"\", \"controlText\": \"\", \"function\": \"select_text\", \"args\": {{\"text\": \"text to edit\"}}}}\n {{\"step 2\":\"type in delete keys to finish delete\",\"controlLabel\": \"101\", \"controlText\": \"Edit\", \"function\": \"type_keys\", \"args\": {{\"text\": \"{DELETE}\"}}}}\n\nexample2: \n Request: |-\n Highlight Text in document.\n Response:\n observation: |-\n I observe the canvas state is a Word document with a body containing a paragraph with a run element, which has a text element 'text to edit'.\n thought: |-\n My task is to detail the given task and give the step-by-step actions to take. \n The user needs to highlight text in the Word document. \n Based on the canvas state, there is a text element 'text to edit'.\n And based on the available apis and controls,the user can use \"select_text\" to select the target to highlight and then to highlight the text.\n Since there is no \"Highlight\" button available,I should click to the 'Home' tab first and then click the 'Highlight' button.\n Therefore,the user can detail the task to highlight 'text to edit' in the Word document.\n In this case, the user should select the 'text to edit' in the Word document and press the 'Home' button and 'Highlight' button respectively.\n new_task: |-\n Highlight 'text to edit' in the Word document.\n action_plans: |-\n {{\"step 1\":\"choose the target text 'text to edit'\",\"controlLabel\": \"\", \"controlText\": \"\", \"function\": \"select_text\", \"args\": {{\"text\": \"text to edit\"}}}}\n {{\"step 2\":\"change ribbon to Home to show the highlight button\",\"controlLabel\": \"11\", \"controlText\": \"Home\", \"function\": \"click_input\", \"args\": {{\"button\": \"left\", \"double\": false}}}}\n {{\"step 3\":\"click the highlight button to finish highlight\",\"controlLabel\": \"\", \"controlText\": \"Highlight\", \"function\": \"click_input\", \"args\": {{\"button\": \"left\", \"double\": false}}}}\n"
},
{
"path": "dataflow/prompts/instantiation/visual/template.yaml",
"content": "version: 1.0\n\nsystem: |-\n You are a Word operator expert and you can easily perform any word-related operations.\n - What you need to do now is to judge and summarize the problems about the execution environment. \n - You should tell me what kind of document you think is needed as the execution environment.\n - Think step by step. \n\n ## Available File Descriptions\n - All the available description of the template files are listed below:\n {descriptions}\n\n Besides, please prefill the task based on the screenshot. you will also be provided with a screenshot, one before the agent's execution and one after the agent's execution.\n All I need is the document that you think is needed as the execution environment.\n Your reply only need reply in json model.\n\n ## Response Format\n - You are required to response in a JSON format, consisting of several distinct parts with the following keys and corresponding content:\n {{\"template_file_name\": \"short description of why you pick this\"}}\n \n For example:\n - Example 1:\n {{\"1.docx\": \"I think this is the most suitable one because it contains a rectangle the task needs.\"}}\n - Example 2:\n {{\"3.docx\": \"The task requires a chart, so I think this is the most suitable one.\"}}\n\nuser: |-\n {given_task}\n "
},
{
"path": "dataflow/schema/execution_schema.json",
"content": "{\n \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n \"type\": \"object\",\n \"properties\": {\n \"unique_id\": { \"type\": \"string\" },\n \"app\": { \"type\": \"string\" },\n \"original\": {\n \"type\": \"object\",\n \"properties\": {\n \"original_task\": { \"type\": \"string\" },\n \"original_steps\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" }\n }\n },\n \"required\": [\"original_task\", \"original_steps\"]\n },\n \"execution_result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"reason\": { \"type\": \"string\" },\n \"sub_scores\": {\n \"type\": \"object\",\n \"patternProperties\": {\n \".*\": { \"type\": \"string\" }\n }\n },\n \"complete\": { \"type\": \"string\" }\n },\n \"required\": [\"reason\", \"sub_scores\", \"complete\"]\n },\n \"error\": {\n \"type\": [\"null\", \"object\"],\n \"properties\": {\n \"type\": { \"type\": \"string\" },\n \"message\": { \"type\": \"string\" },\n \"traceback\": { \"type\": \"string\" }\n },\n \"required\": [\"type\", \"message\", \"traceback\"]\n }\n }\n },\n \"instantiation_result\": {\n \"type\": \"object\",\n \"properties\": {\n \"choose_template\": {\n \"type\": \"object\",\n \"properties\": {\n \"result\": { \"type\": [\"string\", \"null\"] },\n \"error\": { \"type\": [\"null\", \"string\"] }\n },\n \"required\": [\"result\", \"error\"]\n },\n \"prefill\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"instantiated_request\": { \"type\": \"string\" },\n \"instantiated_plan\": {\n \"type\":[\"array\", \"null\"],\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"Step\": { \"type\": \"integer\" },\n \"Subtask\": { \"type\": \"string\" },\n \"ControlLabel\": { \"type\": [\"string\", \"null\"] },\n \"ControlText\": { \"type\": \"string\" },\n \"Function\": { \"type\": \"string\" },\n \"Args\": { \"type\": \"object\", \"additionalProperties\": true },\n \"Success\": { \"type\": [\"boolean\", \"null\"] },\n \"MatchedControlText\": { \"type\": [\"string\", \"null\"] }\n },\n \"required\": [\"Step\", \"Subtask\", \"Function\", \"Args\", \"Success\", \"MatchedControlText\"]\n }\n }\n },\n \"required\": [\"instantiated_request\", \"instantiated_plan\"]\n },\n \"error\": { \"type\": [\"null\", \"string\"] }\n },\n \"required\": [\"result\", \"error\"]\n },\n \"instantiation_evaluation\": {\n \"type\": \"object\",\n \"properties\": {\n \"result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"judge\": { \"type\": \"boolean\" },\n \"thought\": { \"type\": \"string\" },\n \"request_type\": { \"type\": \"string\" }\n },\n \"required\": [\"judge\", \"thought\", \"request_type\"]\n },\n \"error\": { \"type\": [\"null\", \"string\"] }\n },\n \"required\": [\"result\", \"error\"]\n }\n }\n },\n \"time_cost\": {\n \"type\": \"object\",\n \"properties\": {\n \"choose_template\": { \"type\": [\"number\", \"null\"] },\n \"prefill\":{ \"type\": [\"number\", \"null\"] },\n \"instantiation_evaluation\": { \"type\": [\"number\", \"null\"] },\n \"execute\": { \"type\": [\"number\", \"null\"] },\n \"execute_eval\": { \"type\": [\"number\", \"null\"] },\n \"total\": { \"type\": [\"number\", \"null\"] }\n },\n \"required\": [\"choose_template\", \"prefill\", \"instantiation_evaluation\", \"execute\", \"execute_eval\", \"total\"]\n }\n },\n \"required\": [\"unique_id\", \"app\", \"original\", \"execution_result\", \"instantiation_result\", \"time_cost\"]\n}\n"
},
{
"path": "dataflow/schema/instantiation_schema.json",
"content": "{\n \"$schema\": \"http://json-schema.org/draft-07/schema#\",\n \"type\": \"object\",\n \"properties\": {\n \"unique_id\": { \"type\": \"string\" },\n \"app\": { \"type\": \"string\" },\n \"original\": {\n \"type\": \"object\",\n \"properties\": {\n \"original_task\": { \"type\": \"string\" },\n \"original_steps\": {\n \"type\": \"array\",\n \"items\": { \"type\": \"string\" }\n }\n },\n \"required\": [\"original_task\", \"original_steps\"]\n },\n \"execution_result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"result\": {\n \"type\":\"null\"\n },\n \"error\": {\n \"type\":\"null\"\n }\n }\n },\n \"instantiation_result\": {\n \"type\": \"object\",\n \"properties\": {\n \"choose_template\": {\n \"type\": \"object\",\n \"properties\": {\n \"result\": { \"type\": [\"string\", \"null\"] },\n \"error\": { \"type\": [\"null\", \"string\"] }\n },\n \"required\": [\"result\", \"error\"]\n },\n \"prefill\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"instantiated_request\": { \"type\": \"string\" },\n \"instantiated_plan\": {\n \"type\":[\"array\", \"null\"],\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"Step\": { \"type\": \"integer\" },\n \"Subtask\": { \"type\": \"string\" },\n \"ControlLabel\": { \"type\": [\"string\", \"null\"] },\n \"ControlText\": { \"type\": \"string\" },\n \"Function\": { \"type\": \"string\" },\n \"Args\": { \"type\": \"object\", \"additionalProperties\": true }\n },\n \"required\": [\"Step\", \"Subtask\", \"Function\", \"Args\"]\n }\n }\n },\n \"required\": [\"instantiated_request\", \"instantiated_plan\"]\n },\n \"error\": { \"type\": [\"null\", \"string\"] }\n },\n \"required\": [\"result\", \"error\"]\n },\n \"instantiation_evaluation\": {\n \"type\": \"object\",\n \"properties\": {\n \"result\": {\n \"type\": [\"object\", \"null\"],\n \"properties\": {\n \"judge\": { \"type\": \"boolean\" },\n \"thought\": { \"type\": \"string\" },\n \"request_type\": { \"type\": \"string\" }\n },\n \"required\": [\"judge\", \"thought\", \"request_type\"]\n },\n \"error\": { \"type\": [\"null\", \"string\"] }\n },\n \"required\": [\"result\", \"error\"]\n }\n }\n },\n \"time_cost\": {\n \"type\": \"object\",\n \"properties\": {\n \"choose_template\": { \"type\": [\"number\", \"null\"] },\n \"prefill\":{ \"type\": [\"number\", \"null\"] },\n \"instantiation_evaluation\": { \"type\": [\"number\", \"null\"] },\n \"total\": { \"type\": [\"number\", \"null\"] }\n },\n \"required\": [\"choose_template\", \"prefill\", \"instantiation_evaluation\", \"total\"]\n }\n },\n \"required\": [\"unique_id\", \"app\", \"original\", \"execution_result\", \"instantiation_result\", \"time_cost\"]\n}\n"
},
{
"path": "dataflow/templates/word/description.json",
"content": "{\n \"1.docx\":\"A doc with a rectangle shape\",\n \"2.docx\":\"A doc with a line of text\",\n \"3.docx\":\"A doc with a chart\",\n \"4.docx\":\"A doc with a text box\",\n \"5.docx\":\"A doc with comments and reviewer\",\n \"6.docx\":\"A doc with a list of items\",\n \"7.docx\":\"A doc with a table\"\n}\n"
},
{
"path": "documents/docs/about/CODE_OF_CONDUCT.md",
"content": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n\nResources:\n\n- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)\n- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\n- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns\n"
},
{
"path": "documents/docs/about/CONTRIBUTING.md",
"content": "# Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to\nagree to a Contributor License Agreement (CLA) declaring that you have the right to,\nand actually do, grant us the rights to use your contribution. For details, visit\nhttps://cla.microsoft.com.\n\nWhen you submit a pull request, a CLA-bot will automatically determine whether you need\nto provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the\ninstructions provided by the bot. You will only need to do this once across all repositories using our CLA.\n\n!!! note\n You should sunmit your pull request to the `pre-release` branch, not the `main` branch.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\nor contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments."
},
{
"path": "documents/docs/about/DISCLAIMER.md",
"content": "# Disclaimer: Code Execution and Data Handling Notice\n\nBy choosing to run the provided code, you acknowledge and agree to the following terms and conditions regarding the functionality and data handling practices:\n\n## 1. Code Functionality:\nThe code you are about to execute has the capability to capture screenshots of your working desktop environment and active applications. These screenshots will be processed and sent to the GPT model for inference.\n\n\n## 2. Data Privacy and Storage:\nIt is crucial to note that Microsoft, the provider of this code, explicitly states that it does not collect or save any of the transmitted data. The captured screenshots are processed in real-time for the purpose of inference, and no permanent storage or record of this data is retained by Microsoft.\n\n## 3. User Responsibility:\nBy running the code, you understand and accept the responsibility for the content and nature of the data present on your desktop during the execution period. It is your responsibility to ensure that no sensitive or confidential information is visible or captured during this process.\n\n## 4. Security Measures:\nMicrosoft has implemented security measures to safeguard the action execution. However, it is recommended that you run the code in a secure and controlled environment to minimize potential risks. Ensure that you are running the latest security updates on your system.\n\n## 5. Consent for Inference:\nYou explicitly provide consent for the GPT model to analyze the captured screenshots for the purpose of generating relevant outputs. This consent is inherent in the act of executing the code.\n\n## 6. No Guarantee of Accuracy:\nThe outputs generated by the GPT model are based on patterns learned during training and may not always be accurate or contextually relevant. Microsoft does not guarantee the accuracy or suitability of the inferences made by the model.\n\n## 7. Indemnification:\nUsers agree to defend, indemnify, and hold Microsoft harmless from and against all damages, costs, and attorneys' fees in connection with any claims arising from the use of this Repo.\n\n## 8. Reporting Infringements:\nIf anyone believes that this Repo infringes on their rights, please notify the project owner via the provided project owner email. Microsoft will investigate and take appropriate actions as necessary.\n\n## 9. Modifications to the Disclaimer:\nMicrosoft reserves the right to update or modify this disclaimer at any time without prior notice. It is your responsibility to review the disclaimer periodically for any changes.\n\nBy proceeding to execute the code, you acknowledge that you have read, understood, and agreed to the terms outlined in this disclaimer. If you do not agree with these terms, refrain from running the provided code."
},
{
"path": "documents/docs/about/LICENSE.md",
"content": "Copyright (c) Microsoft Corporation.\n\n## MIT License\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED **AS IS**, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE."
},
{
"path": "documents/docs/about/SUPPORT.md",
"content": "# Support\n\n## How to file issues and get help \n\nThis project uses GitHub Issues to track bugs and feature requests. Please search the existing \nissues before filing new issues to avoid duplicates. For new issues, file your bug or \nfeature request as a new Issue.\n\nYou may use [GitHub Issues](https://github.com/microsoft/UFO/issues) to raise questions, bug reports, and feature requests.\n\nFor help and questions about using this project, please please contact [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com).\n\n\n## Microsoft Support Policy \n\nSupport for this **PROJECT or PRODUCT** is limited to the resources listed above.\n"
},
{
"path": "documents/docs/aip/endpoints.md",
"content": "# AIP Endpoints\n\nEndpoints combine protocol, transport, and resilience components to provide production-ready AIP communication for servers, clients, and orchestrators.\n\n## Endpoint Types at a Glance\n\n| Endpoint Type | Role | Used By | Key Features |\n|---------------|------|---------|--------------|\n| **DeviceServerEndpoint** | Server | Device Agent Service | ✅ Multiplexed connections ✅ Session management ✅ Task dispatching ✅ Result aggregation |\n| **DeviceClientEndpoint** | Client | Device Agent Client | ✅ Auto-reconnection ✅ Heartbeat management ✅ Command execution ✅ Telemetry reporting |\n| **ConstellationEndpoint** | Orchestrator | ConstellationClient | ✅ Multi-device coordination ✅ Task distribution ✅ Device info querying ✅ Connection pooling |\n\n---\n\n## Endpoint Architecture\n\n**Endpoint Inheritance Hierarchy:**\n\nAIP provides three specialized endpoint implementations that all inherit common functionality from a shared base class:\n\n```mermaid\ngraph TB\n Base[AIPEndpoint Base]\n Base --> Server[DeviceServerEndpoint Server-Side]\n Base --> Client[DeviceClientEndpoint Client-Side]\n Base --> Constellation[ConstellationEndpoint Orchestrator]\n \n Base -.->|Protocol| P[Message Handling]\n Base -.->|Resilience| R[Reconnection + Heartbeat]\n Base -.->|Sessions| S[State Tracking]\n \n style Base fill:#e1f5ff\n style Server fill:#fff4e1\n style Client fill:#f0ffe1\n style Constellation fill:#ffe1f5\n```\n\nThe dashed arrows indicate capabilities that the base class provides to all subclasses. This inheritance design ensures consistent behavior across all endpoint types while allowing specialization for server, client, and orchestrator roles.\n\n**Base Endpoint Components:**\n\nAll endpoints inherit from `AIPEndpoint`, which provides:\n\n- **Protocol**: Message serialization and handling\n- **Reconnection Strategy**: Automatic reconnection with backoff\n- **Timeout Manager**: Operation timeout management\n- **Session Handlers**: Per-session state tracking\n\n## Base Endpoint: AIPEndpoint\n\n### Common Methods\n\n| Method | Purpose | Example Usage |\n|--------|---------|---------------|\n| `start()` | Start endpoint | `await endpoint.start()` |\n| `stop()` | Stop endpoint | `await endpoint.stop()` |\n| `is_connected()` | Check connection | `if endpoint.is_connected(): ...` |\n| `send_with_timeout()` | Send with timeout | `await endpoint.send_with_timeout(msg, 30.0)` |\n| `receive_with_timeout()` | Receive with timeout | `msg = await endpoint.receive_with_timeout(ServerMessage, 60.0)` |\n\n**Basic Usage Pattern:**\n\n```python\nfrom aip.endpoints.base import AIPEndpoint\n\n# Start endpoint\nawait endpoint.start()\n\n# Check connection\nif endpoint.is_connected():\n await endpoint.handle_message(msg)\n\n# Send with timeout\nawait endpoint.send_with_timeout(msg, timeout=30.0)\n\n# Clean shutdown\nawait endpoint.stop()\n```\n\n---\n\n## DeviceServerEndpoint\n\nWraps UFO's server-side WebSocket handler with AIP protocol support for managing multiple device connections simultaneously.\n\n### Configuration\n\n```python\nfrom aip.endpoints import DeviceServerEndpoint\n\nendpoint = DeviceServerEndpoint(\n ws_manager=ws_manager, # WebSocket connection manager\n session_manager=session_manager, # Session state manager\n local=False # Local vs remote deployment\n)\n```\n\n### Integration with FastAPI\n\n```python\nfrom fastapi import FastAPI, WebSocket\nfrom aip.endpoints import DeviceServerEndpoint\n\napp = FastAPI()\nendpoint = DeviceServerEndpoint(ws_manager, session_manager)\n\n@app.websocket(\"/ws\")\nasync def websocket_route(websocket: WebSocket):\n await endpoint.handle_websocket(websocket)\n```\n\n### Key Features\n\n| Feature | Description | Benefit |\n|---------|-------------|---------|\n| **Multiplexed Connections** | Handle multiple clients simultaneously | Scale to many devices |\n| **Session Management** | Track active sessions per device | Maintain conversation context |\n| **Task Dispatching** | Route tasks to appropriate clients | Targeted execution |\n| **Result Aggregation** | Collect and format execution results | Unified response handling |\n| **Auto Task Cancellation** | Cancel tasks on disconnect | Prevent orphaned tasks |\n\n**Backward Compatibility:**\n\nThe Device Server Endpoint maintains full compatibility with UFO's existing WebSocket handler.\n\n### Task Cancellation on Disconnection\n\n```python\n# Automatically called when device disconnects\nawait endpoint.cancel_device_tasks(\n device_id=\"device_001\",\n reason=\"device_disconnected\"\n)\n```\n\n---\n\n## DeviceClientEndpoint\n\nWraps UFO's client-side WebSocket client with AIP protocol support, automatic reconnection, and heartbeat management.\n\n### Configuration\n\n```python\nfrom aip.endpoints import DeviceClientEndpoint\n\nendpoint = DeviceClientEndpoint(\n ws_url=\"ws://localhost:8000/ws\",\n ufo_client=ufo_client,\n max_retries=3,\n timeout=120.0\n)\n```\n\n### Automatic Features\n\n| Feature | Default Behavior | Configuration |\n|---------|------------------|---------------|\n| **Heartbeat** | Starts on connection | 20s interval (fixed) |\n| **Reconnection** | Exponential backoff | `max_retries=3`, `initial_backoff=2.0` |\n| **Message Routing** | Auto-routes to UFO client | Handled internally |\n| **Connection Management** | Auto-connect on start | Transparent to user |\n\n**Lifecycle Management Example:**\n\n```python\n# Start and connect\nawait endpoint.start()\n\n# Handle messages automatically\n# (routed to underlying UFO client)\n\n# Stop heartbeat and close\nawait endpoint.stop()\n```\n\n### Reconnection Strategy\n\n```python\nfrom aip.resilience import ReconnectionStrategy\n\nreconnection_strategy = ReconnectionStrategy(\n max_retries=3,\n initial_backoff=2.0,\n max_backoff=60.0\n)\n\nendpoint = DeviceClientEndpoint(\n ws_url=url,\n ufo_client=client,\n reconnection_strategy=reconnection_strategy\n)\n```\n\n---\n\n## ConstellationEndpoint\n\nEnables the ConstellationClient to communicate with multiple devices simultaneously, managing connections, tasks, and queries.\n\n### Configuration\n\n```python\nfrom aip.endpoints import ConstellationEndpoint\n\nendpoint = ConstellationEndpoint(\n task_name=\"multi_device_task\",\n message_processor=processor # Optional custom processor\n)\n```\n\n### Multi-Device Operations\n\n| Operation | Method | Description |\n|-----------|--------|-------------|\n| **Connect** | `connect_to_device()` | Establish connection to device |\n| **Send Task** | `send_task_to_device()` | Dispatch task to specific device |\n| **Query Info** | `request_device_info()` | Get device telemetry |\n| **Check Status** | `is_device_connected()` | Verify connection health |\n| **Disconnect** | `disconnect_device()` | Close device connection |\n| **Disconnect All** | `stop()` | Shutdown all connections |\n\n### Connecting to Devices\n\n```python\n# Connect using AgentProfile\nconnection = await endpoint.connect_to_device(\n device_info=agent_profile, # AgentProfile object\n message_processor=processor\n)\n```\n\nLearn more about [AgentProfile configuration](../galaxy/client/device_manager.md) in the Galaxy documentation.\n\n### Sending Tasks\n\n```python\n# Dispatch task to specific device\nresult = await endpoint.send_task_to_device(\n device_id=\"device_001\",\n task_request={\n \"request\": \"Open Notepad\",\n \"task_name\": \"open_notepad\",\n \"session_id\": \"session_123\"\n }\n)\n```\n\n### Querying Device Info\n\n```python\n# Request telemetry update\ndevice_info = await endpoint.request_device_info(\"device_001\")\n\nif device_info:\n print(f\"OS: {device_info['os']}\")\n print(f\"CPU: {device_info['cpu']}\")\n print(f\"GPU: {device_info.get('gpu', 'N/A')}\")\n```\n\n### Connection Management\n\n**Managing Multiple Devices:**\n\n```python\n# Check connection before sending\nif endpoint.is_device_connected(\"device_001\"):\n await endpoint.send_task_to_device(...)\n\n# Disconnect specific device\nawait endpoint.disconnect_device(\"device_001\")\n\n# Disconnect all devices\nawait endpoint.stop()\n```\n\n### Disconnection Handling\n\n```python\n# Automatically triggered on device disconnect\nawait endpoint.on_device_disconnected(\"device_001\")\n\n# Cancels pending tasks\nawait endpoint.cancel_device_tasks(\n device_id=\"device_001\",\n reason=\"device_disconnected\"\n)\n\n# Attempts reconnection (if enabled)\nsuccess = await endpoint.reconnect_device(\"device_001\")\n```\n\n---\n\n## Endpoint Lifecycle Patterns\n\n### Server Lifecycle\n\n**Server Endpoint State Transitions:**\n\nThis state diagram shows the lifecycle of a server endpoint from initialization through connection handling to shutdown:\n\n```mermaid\nstateDiagram-v2\n [*] --> Initialize: Create endpoint\n Initialize --> Started: start()\n Started --> Listening: Accept connections\n Listening --> Handling: Handle WebSocket\n Handling --> Listening: Connection closed\n Listening --> Stopped: stop()\n Stopped --> [*]\n```\n\nThe `Listening → Handling` loop represents the server accepting multiple client connections. Each connection is handled independently while the server remains in the listening state.\n\n**Server Lifecycle Code:**\n\n```python\n# 1. Initialize\nendpoint = DeviceServerEndpoint(client_manager, session_manager)\n\n# 2. Start\nawait endpoint.start()\n\n# 3. Handle connections\n@app.websocket(\"/ws\")\nasync def handle_ws(websocket: WebSocket):\n await endpoint.handle_websocket(websocket)\n\n# 4. Stop (on shutdown)\nawait endpoint.stop()\n```\n\n### Client Lifecycle\n\n**Client Endpoint State Transitions with Auto-Reconnection:**\n\nThis diagram shows the client lifecycle including automatic reconnection attempts when the connection is lost:\n\n```mermaid\nstateDiagram-v2\n [*] --> Initialize: Create endpoint\n Initialize --> Connecting: start()\n Connecting --> Connected: Connection established\n Connected --> Heartbeat: Auto-start heartbeat\n Heartbeat --> Handling: Handle messages\n Handling --> Heartbeat: Continue\n Heartbeat --> Reconnecting: Connection lost\n Reconnecting --> Connected: Reconnect successful\n Reconnecting --> Stopped: Max retries\n Connected --> Stopped: stop()\n Stopped --> [*]\n```\n\nThe `Heartbeat → Handling` loop represents normal operation with periodic heartbeats. The `Reconnecting → Connected` transition shows automatic recovery from network failures.\n\n**Client Lifecycle Code:**\n\n```python\n# 1. Initialize\nendpoint = DeviceClientEndpoint(ws_url, ufo_client)\n\n# 2. Connect\nawait endpoint.start()\n\n# 3. Handle messages (automatic)\n# UFO client receives and processes messages\n\n# 4. Disconnect\nawait endpoint.stop()\n```\n\n### Constellation Lifecycle\n\n**Constellation Lifecycle Code:**\n\n```python\n# 1. Initialize\nendpoint = ConstellationEndpoint(task_name)\n\n# 2. Start\nawait endpoint.start()\n\n# 3. Connect to devices\nawait endpoint.connect_to_device(device_info1)\nawait endpoint.connect_to_device(device_info2)\n\n# 4. Send tasks\nawait endpoint.send_task_to_device(device_id, task_request)\n\n# 5. Cleanup (disconnects all devices)\nawait endpoint.stop()\n```\n\n---\n\n## Resilience Features\n\n!!!warning \"Built-In Resilience\"\n All endpoints include automatic reconnection, timeout management, and heartbeat monitoring for production reliability.\n\n### Resilience Configuration\n\n| Component | Configuration | Purpose |\n|-----------|---------------|---------|\n| **Reconnection** | `ReconnectionStrategy` | Auto-reconnect with backoff |\n| **Timeout** | `TimeoutManager` | Enforce operation timeouts |\n| **Heartbeat** | `HeartbeatManager` | Monitor connection health |\n\n**Configuring Resilience:**\n\n```python\nfrom aip.resilience import ReconnectionStrategy, ReconnectionPolicy\n\nstrategy = ReconnectionStrategy(\n max_retries=5,\n initial_backoff=1.0,\n max_backoff=60.0,\n backoff_multiplier=2.0,\n policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF\n)\n\nendpoint = DeviceClientEndpoint(\n ws_url=url,\n ufo_client=client,\n reconnection_strategy=strategy\n)\n```\n\n### Timeout Operations\n\n```python\n# Send with custom timeout\nawait endpoint.send_with_timeout(msg, timeout=30.0)\n\n# Receive with custom timeout\nmsg = await endpoint.receive_with_timeout(ServerMessage, timeout=60.0)\n```\n\n[→ See detailed resilience documentation](./resilience.md)\n\n---\n\n## Error Handling Patterns\n\n### Connection Errors\n\n```python\ntry:\n await endpoint.start()\nexcept ConnectionError as e:\n logger.error(f\"Failed to connect: {e}\")\n # Reconnection handled automatically if enabled\n```\n\n### Task Execution Errors\n\n```python\ntry:\n result = await endpoint.send_task_to_device(device_id, task)\nexcept TimeoutError:\n logger.error(\"Task execution timeout\")\nexcept Exception as e:\n logger.error(f\"Task failed: {e}\")\n```\n\n### Custom Disconnection Handling\n\n```python\nclass CustomEndpoint(DeviceClientEndpoint):\n async def on_device_disconnected(self, device_id: str) -> None:\n logger.warning(f\"Device {device_id} disconnected\")\n \n # Custom cleanup logic\n await self.custom_cleanup(device_id)\n \n # Call parent implementation\n await super().on_device_disconnected(device_id)\n```\n\n---\n\n## Best Practices\n\n**Endpoint Selection:**\n\n| Use Case | Endpoint Type |\n|----------|---------------|\n| Device agent server | `DeviceServerEndpoint` |\n| Device agent client | `DeviceClientEndpoint` |\n| Multi-device orchestrator | `ConstellationEndpoint` |\n\n!!!warning \"Configuration Guidelines\"\n - **Set appropriate timeouts** based on deployment environment\n - **Configure reconnection** based on network reliability\n - **Monitor connection health** with `is_connected()` checks\n - **Implement custom handlers** for application-specific cleanup\n\n!!!success \"Resource Management\"\n - **Always call `stop()`** during shutdown to prevent leaks\n - **Use message processors** for custom message handling\n - **Handle disconnections** with `on_device_disconnected` overrides\n\n**Custom Message Processor:**\n\n```python\nclass MyProcessor:\n async def process_message(self, msg):\n # Custom processing\n logger.info(f\"Processing: {msg.type}\")\n # ...\n\nendpoint = ConstellationEndpoint(\n task_name=\"task\",\n message_processor=MyProcessor()\n)\n```\n\n---\n\n## Quick Reference\n\n### Import Endpoints\n\n```python\nfrom aip.endpoints import (\n AIPEndpoint, # Base class\n DeviceServerEndpoint, # Server-side\n DeviceClientEndpoint, # Client-side\n ConstellationEndpoint, # Orchestrator-side\n)\n```\n\n### Related Documentation\n\n- [Protocol Reference](./protocols.md) - Protocol implementations used by endpoints\n- [Transport Layer](./transport.md) - Transport configuration and options\n- [Resilience](./resilience.md) - Reconnection and heartbeat management\n- [Messages](./messages.md) - Message types and validation\n- [Overview](./overview.md) - System architecture and design\n- [Galaxy Client](../galaxy/client/overview.md) - Multi-device orchestration with ConstellationClient\n- [UFO Server](../server/websocket_handler.md) - WebSocket server implementation\n- [UFO Client](../client/websocket_client.md) - WebSocket client implementation\n\n"
},
{
"path": "documents/docs/aip/messages.md",
"content": "# AIP Message Reference\n\nAIP uses **Pydantic-based messages** for automatic validation, serialization, and type safety. All messages transmit as JSON over WebSocket.\n\n## Message Overview\n\n### Bidirectional Communication\n\n**Message Flow Overview:**\n\nThis diagram illustrates all message types and their directions in the AIP protocol, showing how clients and servers communicate bidirectionally:\n\n```mermaid\ngraph LR\n Client[Device Client]\n Server[Device Service]\n \n Client -->|REGISTER| Server\n Client -->|COMMAND_RESULTS| Server\n Client -->|TASK_END| Server\n Client -->|HEARTBEAT| Server\n \n Server -->|TASK| Client\n Server -->|COMMAND| Client\n Server -->|HEARTBEAT| Client\n Server -->|TASK_END| Client\n \n Client <-->|DEVICE_INFO| Server\n Client <-->|ERROR| Server\n \n style Client fill:#f0ffe1\n style Server fill:#fff4e1\n```\n\nUnidirectional arrows indicate request-response patterns, while bidirectional arrows (`<-->`) indicate messages that can be initiated by either party. Note that both `HEARTBEAT` and `TASK_END` can flow in both directions depending on the scenario.\n\n### Message Types Quick Reference\n\n| Direction | Message Type | Purpose | Key Fields |\n|-----------|--------------|---------|------------|\n| **Client → Server** | | | |\n| | `REGISTER` | Initial capability advertisement | `client_id`, `metadata` |\n| | `COMMAND_RESULTS` | Return command execution results | `action_results`, `prev_response_id` |\n| | `TASK_END` | Notify task completion | `status`, `session_id` |\n| | `HEARTBEAT` | Keepalive signal | `client_id` |\n| **Server → Client** | | | |\n| | `TASK` | Task assignment | `user_request`, `task_name` |\n| | `COMMAND` | Command execution request | `actions`, `response_id` |\n| | `HEARTBEAT` | Keepalive acknowledgment | `response_id` |\n| | `TASK_END` | Task completion notification | `status`, `result` |\n| **Bidirectional** | | | |\n| | `DEVICE_INFO_REQUEST` | Request device telemetry | `request_id` |\n| | `DEVICE_INFO_RESPONSE` | Device information | Device specs |\n| | `ERROR` | Error condition | `error` message |\n\n--- \n---\n\n## Core Data Structures\n\nThese Pydantic models form the building blocks for all AIP messages.\n\n### Essential Types Summary\n\n| Type | Purpose | Key Fields | Usage |\n|------|---------|------------|-------|\n| **Rect** | UI element coordinates | `x`, `y`, `width`, `height` | UI automation |\n| **ControlInfo** | UI control metadata | `annotation_id`, `name`, `rectangle` | Control discovery |\n| **WindowInfo** | Window metadata | `process_id`, `is_active` (extends ControlInfo) | Window management |\n| **MCPToolInfo** | Tool definition | `tool_key`, `namespace`, `input_schema` | Capability advertisement |\n| **Command** | Execution request | `tool_name`, `parameters`, `call_id` | Action dispatch |\n| **Result** | Execution outcome | `status`, `result`, `error` | Result reporting |\n\n### Rect (Rectangle)\n\nRepresents UI element bounding box.\n\n```python\nrect = Rect(x=100, y=200, width=300, height=150)\n```\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `x` | int | X-coordinate (top-left) |\n| `y` | int | Y-coordinate (top-left) |\n| `width` | int | Width in pixels |\n| `height` | int | Height in pixels |\n\n### ControlInfo\n\nUI control element metadata.\n\n**ControlInfo Example:**\n\n```python\ncontrol = ControlInfo(\n annotation_id=\"ctrl_001\",\n name=\"Submit Button\",\n class_name=\"Button\",\n rectangle=Rect(x=100, y=200, width=80, height=30),\n is_enabled=True,\n is_visible=True\n)\n```\n\n**Complete Field List:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `annotation_id` | str? | Unique annotation identifier |\n| `name` | str? | Control name |\n| `title` | str? | Control title |\n| `handle` | int? | Windows handle (HWND) |\n| `class_name` | str? | UI class name |\n| `rectangle` | Rect? | Bounding rectangle |\n| `control_type` | str? | Type (Button, TextBox, etc.) |\n| `automation_id` | str? | UI Automation ID |\n| `is_enabled` | bool? | Enabled state |\n| `is_visible` | bool? | Visibility state |\n| `source` | str? | Data source identifier |\n| `text_content` | str? | Text content |\n\n### WindowInfo\n\nWindow metadata (extends ControlInfo).\n\n**Additional Fields:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `process_id` | int? | Process ID (PID) |\n| `process_name` | str? | Process name (e.g., \"notepad.exe\") |\n| `is_minimized` | bool? | Minimized state |\n| `is_maximized` | bool? | Maximized state |\n| `is_active` | bool? | Has focus |\n\n### MCPToolInfo\n\nMCP tool capability definition.\n\n**Tool Advertisement:**\n\nDevice agents use `MCPToolInfo` to advertise their capabilities during registration.\n\n```python\ntool_info = MCPToolInfo(\n tool_key=\"ui_automation.click_button\",\n tool_name=\"click_button\",\n namespace=\"ui_automation\",\n tool_type=\"action\",\n description=\"Click a button by its ID\",\n input_schema={\n \"type\": \"object\",\n \"properties\": {\n \"button_id\": {\"type\": \"string\"}\n }\n }\n)\n```\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `tool_key` | str | Unique key (`namespace.tool_name`) |\n| `tool_name` | str | Tool name |\n| `namespace` | str | MCP namespace |\n| `tool_type` | str | `\"action\"` or `\"data_collection\"` |\n| `description` | str? | Tool description |\n| `input_schema` | dict? | JSON schema for inputs |\n| `output_schema` | dict? | JSON schema for outputs |\n| `meta` | dict? | Metadata |\n| `annotations` | dict? | Additional annotations |\n\nLearn more about [MCP tools and capabilities](../mcp/overview.md).\n\n---\n\n## Command and Result Structures\n\n### Command\n\nExecution request sent to device agents.\n\n**Command Structure:**\n\n```python\ncmd = Command(\n tool_name=\"click_element\",\n parameters={\"control_id\": \"btn_submit\"},\n tool_type=\"action\",\n call_id=\"cmd_12345\"\n)\n```\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `tool_name` | str | ✅ | Name of tool to execute |\n| `parameters` | dict | | Tool parameters |\n| `tool_type` | str | ✅ | `\"data_collection\"` or `\"action\"` |\n| `call_id` | str | | Unique identifier for correlation |\n\n**Call ID Correlation:**\n\nUse `call_id` to match commands with their results in the `Result` object.\n\n### ResultStatus\n\nExecution outcome enumeration.\n\n| Status | Meaning | When to Use |\n|--------|---------|-------------|\n| `SUCCESS` | ✅ Completed successfully | Command executed without errors |\n| `FAILURE` | ❌ Failed with error | Execution encountered an error |\n| `SKIPPED` | ⏭️ Skipped execution | Conditional execution, not run |\n| `NONE` | ⚪ No status | Initial/unknown state |\n\n### Result\n\nCommand execution outcome.\n\n!!!warning \"Always Check Status\"\n Check `status` before accessing `result`. If `FAILURE`, use `error` field for diagnostics.\n\n```python\n# Success result\nresult = Result(\n status=ResultStatus.SUCCESS,\n result={\"element_found\": True, \"clicked\": True},\n namespace=\"ui_automation\",\n call_id=\"cmd_12345\"\n)\n\n# Failure result\nresult = Result(\n status=ResultStatus.FAILURE,\n error=\"Element not found: btn_submit\",\n namespace=\"ui_automation\",\n call_id=\"cmd_12345\"\n)\n```\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `status` | ResultStatus | Execution status |\n| `error` | str? | Error message (if FAILURE) |\n| `result` | Any | Result payload (type varies by tool) |\n| `namespace` | str? | Namespace of executed tool |\n| `call_id` | str? | Matches Command.call_id |\n\n---\n\n## Status Enumerations\n\n### TaskStatus\n\nTask lifecycle states.\n\n**State Transitions:**\n\n**Task Lifecycle State Machine:**\n\nThis diagram shows the possible state transitions during task execution, including the multi-turn loop and terminal states:\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE: Task starts\n CONTINUE --> CONTINUE: Multi-step\n CONTINUE --> COMPLETED: Success\n CONTINUE --> FAILED: Error\n OK --> OK: Heartbeat\n ERROR --> [*]: Terminal\n```\n\nThe `CONTINUE → CONTINUE` self-loop represents multi-turn execution where tasks request additional commands before completion. `COMPLETED` and `FAILED` are terminal success/failure states.\n\n| Status | Meaning | Usage |\n|--------|---------|-------|\n| `CONTINUE` | 🔄 Task ongoing | Multi-turn execution, more steps needed |\n| `COMPLETED` | ✅ Task done | Successful completion |\n| `FAILED` | ❌ Task failed | Error encountered |\n| `OK` | ✓ Acknowledgment | Heartbeat, health check passed |\n| `ERROR` | ⚠️ Protocol error | Protocol-level error |\n\n**Multi-Turn Execution:**\n\n`CONTINUE` enables agents to request additional commands before marking a task as complete, supporting complex multi-step workflows.\n\n---\n\n## Client Types\n\n### ClientType\n\nIdentifies the type of client connecting to the server.\n\n| Type | Role | Characteristics |\n|------|------|----------------|\n| `DEVICE` | Device agent executor | • Executes tasks locally • Reports telemetry • Single-device focus |\n| `CONSTELLATION` | Multi-device orchestrator | • Manages multiple devices • Coordinates tasks • Requires `target_id` |\n\n**Registration by Type:**\n\n```python\n# Device client\ndevice_msg = ClientMessage(\n type=ClientMessageType.REGISTER,\n client_type=ClientType.DEVICE,\n client_id=\"device_001\"\n)\n\n# Constellation client\nconstellation_msg = ClientMessage(\n type=ClientMessageType.REGISTER,\n client_type=ClientType.CONSTELLATION,\n client_id=\"orchestrator_001\",\n target_id=\"device_001\" # Target device\n)\n```\n\n---\n\n## ClientMessage (Client → Server)\n\nDevices and constellation clients use `ClientMessage` to communicate with the server.\n\n### Message Types\n\n| Type | Purpose | Required Fields |\n|------|---------|----------------|\n| **REGISTER** | Initial registration | `client_id`, `client_type` |\n| **HEARTBEAT** | Keepalive | `client_id`, `status=OK` |\n| **TASK** | Request task execution | `request`, `client_id` |\n| **TASK_END** | Notify completion | `session_id`, `status` |\n| **COMMAND_RESULTS** | Return results | `action_results`, `prev_response_id` |\n| **DEVICE_INFO_REQUEST** | Request telemetry | `request_id` |\n| **DEVICE_INFO_RESPONSE** | Provide telemetry | Device data |\n| **ERROR** | Report error | `error` |\n\n### Common Fields\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `type` | ClientMessageType | Message type |\n| `status` | TaskStatus | Current task status |\n| `client_type` | ClientType | DEVICE or CONSTELLATION |\n| `session_id` | str? | Session identifier |\n| `task_name` | str? | Human-readable task name |\n| `client_id` | str? | Unique client identifier |\n| `target_id` | str? | Target device (for constellation) |\n| `request` | str? | Request text (for TASK) |\n| `action_results` | List[Result]? | Command results |\n| `timestamp` | str? | ISO 8601 timestamp |\n| `request_id` | str? | Unique request identifier |\n| `prev_response_id` | str? | Previous response ID |\n| `error` | str? | Error message |\n| `metadata` | dict? | Additional metadata |\n\n### Example: REGISTER\n\n```python\nregister_msg = ClientMessage(\n type=ClientMessageType.REGISTER,\n client_type=ClientType.DEVICE,\n client_id=\"windows_agent_001\",\n status=TaskStatus.OK,\n timestamp=\"2024-11-04T10:30:00Z\",\n metadata={\n \"platform\": \"windows\",\n \"os_version\": \"Windows 11\",\n \"capabilities\": [\"ui_automation\", \"file_operations\"]\n }\n)\n```\n\n### Example: COMMAND_RESULTS\n\n```python\nresults_msg = ClientMessage(\n type=ClientMessageType.COMMAND_RESULTS,\n client_id=\"windows_agent_001\",\n session_id=\"session_123\",\n prev_response_id=\"resp_456\", # Links to server's COMMAND message\n status=TaskStatus.CONTINUE,\n action_results=[\n Result(status=ResultStatus.SUCCESS, result={\"clicked\": True}),\n Result(status=ResultStatus.SUCCESS, result={\"text_entered\": True})\n ],\n timestamp=\"2024-11-04T10:31:00Z\",\n request_id=\"req_789\"\n)\n```\n\n---\n\n## ServerMessage (Server → Client)\n\nDevice services use `ServerMessage` to assign tasks and send commands to clients.\n\n### Message Types\n\n| Type | Purpose | Required Fields |\n|------|---------|----------------|\n| **TASK** | Assign task | `user_request`, `task_name`, `session_id` |\n| **COMMAND** | Execute commands | `actions`, `response_id`, `session_id` |\n| **TASK_END** | Notify completion | `status`, `session_id` |\n| **HEARTBEAT** | Keepalive ack | `response_id` |\n| **DEVICE_INFO_REQUEST** | Request telemetry | `request_id` |\n| **DEVICE_INFO_RESPONSE** | Telemetry data | Device info |\n| **ERROR** | Error notification | `error` |\n\n### Common Fields\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `type` | ServerMessageType | Message type |\n| `status` | TaskStatus | Current task status |\n| `user_request` | str? | Original user request |\n| `agent_name` | str? | Agent handling task |\n| `process_name` | str? | Process for execution context |\n| `root_name` | str? | Root application name |\n| `actions` | List[Command]? | Commands to execute |\n| `messages` | List[str]? | Log messages |\n| `error` | str? | Error description |\n| `session_id` | str? | Session identifier |\n| `task_name` | str? | Task name |\n| `timestamp` | str? | ISO 8601 timestamp |\n| `response_id` | str? | Response identifier |\n| `result` | Any? | Result payload |\n\n### Example: TASK Assignment\n\n```python\ntask_msg = ServerMessage(\n type=ServerMessageType.TASK,\n status=TaskStatus.CONTINUE,\n user_request=\"Open Notepad and create a new file\",\n task_name=\"create_notepad_file\",\n session_id=\"session_123\",\n response_id=\"resp_001\",\n agent_name=\"AppAgent\",\n process_name=\"notepad.exe\",\n timestamp=\"2024-11-04T10:30:00Z\"\n)\n```\n\n### Example: COMMAND Execution\n\n```python\ncommand_msg = ServerMessage(\n type=ServerMessageType.COMMAND,\n status=TaskStatus.CONTINUE,\n session_id=\"session_123\",\n response_id=\"resp_456\",\n actions=[\n Command(\n tool_name=\"launch_application\",\n parameters={\"app_name\": \"notepad\"},\n tool_type=\"action\",\n call_id=\"cmd_001\"\n ),\n Command(\n tool_name=\"type_text\",\n parameters={\"text\": \"Hello World\"},\n tool_type=\"action\",\n call_id=\"cmd_002\"\n )\n ],\n timestamp=\"2024-11-04T10:30:30Z\"\n)\n```\n\n### Example: TASK_END\n\n```python\ntask_end_msg = ServerMessage(\n type=ServerMessageType.TASK_END,\n status=TaskStatus.COMPLETED,\n session_id=\"session_123\",\n response_id=\"resp_999\",\n result={\n \"file_created\": True,\n \"path\": \"C:\\\\Users\\\\user\\\\document.txt\"\n },\n timestamp=\"2024-11-04T10:35:00Z\"\n)\n```\n\n---\n\n## Message Validation\n\n!!!warning \"Built-In Validation\"\n AIP provides `MessageValidator` class for ensuring message integrity. Always validate messages before processing to prevent protocol errors.\n\n### Validation Methods\n\n| Method | Purpose | Requirements |\n|--------|---------|-------------|\n| `validate_registration()` | Check registration | `type=REGISTER`, `client_id` present |\n| `validate_task_request()` | Check task request | `type=TASK`, `request` and `client_id` present |\n| `validate_command_results()` | Check results | `type=COMMAND_RESULTS`, `prev_response_id` present |\n| `validate_server_message()` | Check server msg | `type` and `status` present |\n\n**Validation Usage:**\n\n```python\nfrom aip.messages import MessageValidator\n\n# Validate registration\nif MessageValidator.validate_registration(client_message):\n await process_registration(client_message)\n\n# Validate task request\nif MessageValidator.validate_task_request(client_message):\n await dispatch_task(client_message)\n\n# Validate command results\nif MessageValidator.validate_command_results(client_message):\n await process_results(client_message)\n```\n\n---\n\n## Message Correlation\n\nAIP uses identifier chains to maintain conversation context across multiple message exchanges.\n\n### Correlation Pattern\n\n**Message Identifier Chaining:**\n\nThis sequence diagram demonstrates how messages are linked together using correlation IDs to maintain conversation context:\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n \n C->>S: request_id: \"req_001\"\n S->>C: response_id: \"resp_001\"\n C->>S: request_id: \"req_002\" prev_response_id: \"resp_001\"\n S->>C: response_id: \"resp_002\"\n```\n\nEach new request includes `prev_response_id` pointing to the previous server response, forming a traceable conversation chain. This pattern enables audit trails, debugging, and request-response correlation in multi-turn conversations.\n\n### Correlation Fields\n\n| Field | Purpose | Example |\n|-------|---------|---------|\n| `request_id` | Unique request identifier | `\"req_abc123\"` |\n| `response_id` | Unique response identifier | `\"resp_def456\"` |\n| `prev_response_id` | Links to previous response | `\"resp_def456\"` |\n| `session_id` | Groups related messages | `\"session_xyz\"` |\n| `call_id` | Correlates commands/results | `\"cmd_001\"` |\n\n### Session Tracking\n\n**Session-Based Grouping:**\n\nAll messages within a task execution share the same `session_id` for traceability.\n\n```python\n# All messages use same session_id\nSESSION_ID = \"session_abc123\"\n\ntask_msg.session_id = SESSION_ID\ncommand_msg.session_id = SESSION_ID\nresults_msg.session_id = SESSION_ID\ntask_end_msg.session_id = SESSION_ID\n```\n\n---\n\n## Best Practices\n\n!!!success \"Message Construction\"\n **Timestamps**: Always use ISO 8601 format\n ```python\n from datetime import datetime, timezone\n timestamp = datetime.now(timezone.utc).isoformat()\n ```\n \n **Unique IDs**: Generate UUIDs for correlation\n ```python\n import uuid\n request_id = str(uuid.uuid4())\n ```\n\n!!!warning \"Error Handling\"\n - Check `Result.status` before accessing result data\n - Always provide meaningful error messages\n - Use `ResultStatus.FAILURE` with descriptive `error` field\n\n**Extensibility:**\n\n- Use `metadata` field for custom data without breaking protocol\n- Leverage Pydantic's validation for type safety\n- Always correlate messages with `prev_response_id`\n\n---\n\n## Quick Reference\n\n### Import Messages\n\n```python\nfrom aip.messages import (\n ClientMessage,\n ServerMessage,\n ClientMessageType,\n ServerMessageType,\n ClientType,\n TaskStatus,\n Command,\n Result,\n ResultStatus,\n MessageValidator,\n)\n```\n\n### Related Documentation\n\n- [Protocol Guide](./protocols.md) - How protocols construct and use messages\n- [Endpoints](./endpoints.md) - How endpoints handle messages\n- [Overview](./overview.md) - High-level message flow in system architecture\n- [Transport Layer](./transport.md) - WebSocket transport for message delivery\n- [Resilience](./resilience.md) - Message retry and timeout handling\n- [MCP Integration](../mcp/overview.md) - How MCP tools integrate with AIP messages\n"
},
{
"path": "documents/docs/aip/overview.md",
"content": "# Agent Interaction Protocol (AIP)\n\nThe orchestration model requires a communication substrate that remains **correct under continuous DAG evolution**, **dynamic agent participation**, and **fine-grained event propagation**. Legacy HTTP-based coordination approaches (e.g., A2A, ACP) assume short-lived, stateless interactions, incurring handshake overhead, stale capability views, and fragile recovery when partial failures occur mid-task. These assumptions make them unsuitable for the continuously evolving workflows and long-running reasoning loops characteristic of UFO².\n\n## Design Overview\n\nAIP serves as the **nervous system** of UFO², connecting the ConstellationClient, device agent services, and device clients under a unified, event-driven control plane. It is designed as a lightweight yet evolution-tolerant protocol to satisfy six goals:\n\n**Design Goals:**\n\n- **(G1)** Maintain persistent bidirectional sessions to eliminate per-request overhead\n- **(G2)** Unify heterogeneous capability discovery via multi-source profiling\n- **(G3)** Ensure fine-grained reliability through heartbeats and timeout managers for disconnection and failure detection\n- **(G4)** Preserve deterministic command ordering within sessions\n- **(G5)** Support composable extensibility for new message types and resilience strategies\n- **(G6)** Provide transparent reconnection and task continuity under transient failures\n\n| Legacy HTTP Coordination | AIP WebSocket-Based Design |\n|--------------------------|----------------------------|\n| ❌ Short-lived requests | ✅ Persistent sessions (G1) |\n| ❌ Stateless interactions | ✅ Session-aware task management |\n| ❌ High latency overhead | ✅ Low-latency event streaming |\n| ❌ Poor reconnection support | ✅ Seamless recovery from disconnections (G6) |\n| ❌ Manual state synchronization | ✅ Automatic DAG state propagation |\n| ❌ Fragile partial failures | ✅ Fine-grained reliability (G3) |\n\n## Five-Layer Architecture\n\nTo meet these requirements, AIP adopts a persistent, bidirectional WebSocket transport and decomposes the orchestration substrate into **five** logical strata, each responsible for a distinct aspect of reliability and adaptability. The architecture establishes a complete substrate where **L1** defines semantic contracts, **L2** provides transport flexibility, **L3** implements protocol logic, **L4** ensures operational resilience, and **L5** delivers deployment-ready orchestration primitives.\n\n**Architecture Diagram:**\n\nThe following diagram illustrates the five-layer architecture and the roles of each component:\n\n\n\n### Layer 1: Message Schema Layer\n\nDefines strongly-typed, Pydantic-validated contracts (`ClientMessage`, `ServerMessage`) for message direction, purpose, and task transitions. All messages are validated at schema level, preventing malformed messages from entering the protocol pipeline, enabling early error detection and simplifying debugging.\n\n| Responsibility | Implementation | Supports |\n|----------------|----------------|----------|\n| Message contracts | Pydantic models with validation | Human-readable + machine-verifiable |\n| Structured metadata | System info, capabilities | Unified capability discovery (G2) |\n| ID correlation | Explicit request/response linking | Deterministic ordering (G4) |\n\n### Layer 2: Transport Abstraction Layer\n\nProvides protocol-agnostic `Transport` interface with production-grade WebSocket implementation. The abstraction layer allows swapping transports without changing protocol logic, supporting future protocol evolution.\n\n| Feature | Benefit | Goals |\n|---------|---------|-------|\n| Configurable pings/timeouts | Connection health monitoring | G3 |\n| Large payload support | Handles complex task definitions | G1 |\n| Decoupled transport logic | Future extensibility (HTTP/3, gRPC) | G5 |\n| Low-latency persistent sessions | Eliminates per-request overhead | G1 |\n\n### Layer 3: Protocol Orchestration Layer\n\nImplements modular handlers for registration, task execution, heartbeat, and command dispatch. Each handler is independently testable and replaceable, supporting composable extensibility (G5) while maintaining ordered state transitions (G4).\n\n| Component | Purpose | Design |\n|-----------|---------|--------|\n| `AIPProtocol` base | Common handler infrastructure | Extensible base class |\n| Handler modules | Registration, tasks, heartbeat, commands | Pluggable handlers |\n| Middleware hooks | Logging, metrics, authentication | Composable extensions (G5) |\n| State transitions | Ordered message processing | Deterministic ordering (G4) |\n\n**Related Documentation:**\n- [Complete message reference](./messages.md)\n- [Protocol implementation details](./protocols.md)\n\n### Layer 4: Resilience and Health Management Layer\n\n!!!warning \"Fault Tolerance\"\n This layer guarantees fine-grained reliability (G3) and seamless task continuity under transient disconnections (G6), preventing cascade failures.\n\nEncapsulates reliability mechanisms ensuring operational continuity under failures:\n\n| Component | Mechanism | Goals |\n|-----------|-----------|-------|\n| `HeartbeatManager` | Periodic keepalive signals | G3 |\n| `TimeoutManager` | Configurable timeout policies | G3 |\n| `ReconnectionStrategy` | Exponential backoff with jitter | G6 |\n| Session recovery | Automatic state restoration | G6 |\n\n[→ Resilience implementation details](./resilience.md)\n\n### Layer 5: Endpoint Orchestration Layer\n\nProvides role-specific facades integrating lower layers into deployable components. These endpoints unify connection lifecycle, task routing, and health monitoring across roles, reinforcing G1–G6 through consistent implementation of lower-layer capabilities.\n\n| Endpoint | Role | Responsibilities |\n|----------|------|------------------|\n| `ConstellationEndpoint` | Orchestrator | Global agent registry, task assignment, DAG coordination |\n| `DeviceServerEndpoint` | Server | WebSocket connection management, task dispatch, result aggregation |\n| `DeviceClientEndpoint` | Executor | Local task execution, MCP tool invocation, telemetry reporting |\n\n**Endpoint Integration Benefits:**\n\n- ✅ Connection lifecycle management (G1, G6)\n- ✅ Role-specific protocol variants (G5)\n- ✅ Health monitoring integration (G3)\n- ✅ Task routing and session management (G4)\n\n[→ Endpoint setup guide](./endpoints.md)\n\n## Architecture Benefits\n\nTogether, these layers form a vertically integrated stack that enables UFO² to maintain **correctness and availability** under challenging conditions:\n\n| Challenge | How AIP Addresses It | Layers Involved |\n|-----------|----------------------|-----------------|\n| **DAG Evolution** | Deterministic ordering, extensible message types | L1, L3, L4, L5 (G4, G5) |\n| **Agent Churn** | Heartbeats, reconnection, session recovery | L4, L5 (G3, G6) |\n| **Heterogeneous Environments** | Persistent sessions, multi-source profiling | L1, L2, L5 (G1, G2) |\n| **Transient Failures** | Timeout management, automatic recovery | L4 (G3, G6) |\n| **Protocol Evolution** | Transport abstraction, middleware hooks | L2, L3 (G5) |\n\nAIP transforms distributed workflow execution into a **coherent, safe, and adaptive system** where reasoning and execution converge seamlessly across diverse agents and environments.\n\n## Core Capabilities\n\n### Agent Registration & Profiling\n\nEach agent is represented by an **AgentProfile** combining data from three sources for comprehensive capability discovery, supporting heterogeneous capability unification (G2):\n\n| Source | Provider | Information |\n|--------|----------|-------------|\n| **User Config** | ConstellationClient | Endpoint URLs, user preferences, device identity |\n| **Service Manifest** | Device Agent Service | Supported tools, capabilities, operational metadata |\n| **Client Telemetry** | Device Agent Client | OS, hardware specs, GPU status, runtime metrics |\n\n**Benefits of Multi-Level Profiling:**\n\n- ✅ Accurate task allocation based on real-time capabilities (G2)\n- ✅ Transparent adaptation to environmental changes (e.g., GPU availability) \n- ✅ No manual updates needed when device state changes \n- ✅ Informed scheduling decisions at scale\n\n!!!tip \"Dynamic Profile Updates\"\n Client telemetry continuously refreshes, so the orchestrator always sees current device state—critical for GPU-aware scheduling or cross-device load balancing (G2).\n\n[→ See detailed registration flow](./protocols.md)\n\n### Task Dispatch & Result Delivery\n\nAIP uses **long-lived WebSocket sessions** that span multiple task executions, eliminating per-request connection overhead and preserving context (G1).\n\n**Task Execution Sequence:**\n\nThe following sequence diagram shows the complete lifecycle of a task from assignment to completion, including intermediate execution steps and state updates:\n\n```mermaid\nsequenceDiagram\n participant CC as ConstellationClient\n participant DAS as Device Service\n participant DAC as Device Client\n \n CC->>DAS: TASK message (TaskStar)\n DAS->>DAC: Stream task payload\n DAC->>DAC: Execute using MCP tools\n DAC->>DAS: Stream execution logs\n DAS->>CC: TASK_END (status, logs, results)\n CC->>CC: Update TaskConstellation\n CC->>CC: Notify ConstellationAgent\n```\n\nEach arrow represents a message exchange, with vertical lifelines showing the temporal ordering of events. Note how logs stream back during execution, enabling real-time monitoring.\n\n| Stage | Message Type | Content |\n|-------|-------------|---------|\n| Assignment | `TASK` | TaskStar definition, target device, commands |\n| Execution | (internal) | MCP tool invocations, local computation |\n| Reporting | `TASK_END` | Status, logs, evaluator outputs, results |\n\n!!!warning \"Asynchronous Execution\"\n Tasks execute asynchronously. The orchestrator may assign multiple tasks to different devices simultaneously, with results arriving in non-deterministic order.\n\n**Related Documentation:**\n- [Message format details](./messages.md)\n- [TaskConstellation documentation](../galaxy/constellation/task_constellation.md)\n- [TaskStar (task nodes) documentation](../galaxy/constellation/task_star.md)\n\n### Command Execution\n\nWithin each task, AIP executes **individual commands** deterministically with preserved ordering, enabling precise control and error handling (G4).\n\n**Command Structure:**\n\n| Field | Purpose | Example |\n|-------|---------|---------|\n| `tool_name` | Tool/action name | `\"click_input\"` |\n| `parameters` | Typed arguments | `{\"target\": \"Save Button\", \"button\": \"left\"}` |\n| `tool_type` | Category | `\"action\"` or `\"data_collection\"` |\n| `call_id` | Unique identifier | `\"cmd_001\"` |\n\n**Execution Guarantees:**\n\n- ✅ **Sequential execution** within a session (deterministic order) (G4)\n- ✅ **Command batching** supported (reduces network overhead) \n- ✅ **Structured results** with status codes and error details \n- ✅ **Timeout propagation** for precise recovery strategies (G3)\n\n**Command Batching Example:**\n\n```json\n{\n \"actions\": [\n {\"tool_name\": \"click\", \"parameters\": {\"target\": \"File\"}, \"call_id\": \"1\"},\n {\"tool_name\": \"click\", \"parameters\": {\"target\": \"Save As\"}, \"call_id\": \"2\"},\n {\"tool_name\": \"type\", \"parameters\": {\"text\": \"document.pdf\"}, \"call_id\": \"3\"}\n ]\n}\n```\n\nAll three commands sent in one message, executed sequentially.\n\n[→ See command execution protocol](./protocols.md)\n\n## Message Protocol Overview\n\nAll AIP messages use **Pydantic models** for automatic validation, serialization, and type safety.\n\n### Bidirectional Message Types\n\n| Direction | Message Type | Purpose |\n|-----------|--------------|---------|\n| **Client → Server** | `REGISTER` | Initial capability advertisement |\n| | `COMMAND_RESULTS` | Return command execution results |\n| | `TASK_END` | Notify task completion |\n| | `HEARTBEAT` | Keepalive signal |\n| | `DEVICE_INFO_RESPONSE` | Device telemetry update |\n| **Server → Client** | `TASK` | Task assignment |\n| | `COMMAND` | Command execution request |\n| | `DEVICE_INFO_REQUEST` | Request telemetry refresh |\n| | `HEARTBEAT` | Keepalive acknowledgment |\n| **Bidirectional** | `ERROR` | Error condition reporting |\n\n**Message Correlation:**\n\nEvery message includes:\n\n- `timestamp`: ISO 8601 formatted \n- `request_id` / `response_id`: Unique identifier \n- `prev_response_id`: Links responses to requests \n- `session_id`: Session context\n\n[→ Complete message reference](./messages.md)\n\n## Resilient Connection Protocol\n\n!!!warning \"Network Instability Handling (G3, G6)\"\n AIP ensures **continuous orchestration** even under transient network failures or device disconnections through fine-grained reliability mechanisms and transparent reconnection.\n\n### Device Disconnection Flow\n\n**Connection State Transitions:**\n\nThis state diagram illustrates how devices transition between connection states and the actions triggered at each transition:\n\n```mermaid\nstateDiagram-v2\n [*] --> CONNECTED\n CONNECTED --> DISCONNECTED: Connection lost\n DISCONNECTED --> CONNECTED: Reconnection succeeds\n DISCONNECTED --> [*]: Timeout / Manual removal\n \n note right of DISCONNECTED\n • Excluded from scheduling\n • Tasks marked FAILED\n • Auto-reconnect triggered\n end note\n```\n\nThe `DISCONNECTED` state acts as a quarantine zone where the device is temporarily removed from the scheduling pool while auto-reconnection attempts are made. If reconnection fails after timeout, the device is permanently removed.\n\n| Event | Orchestrator Action | Device Action |\n|-------|---------------------|---------------|\n| **Device disconnects** | Mark as `DISCONNECTED` Exclude from scheduling Trigger auto-reconnect (G6) | N/A |\n| **Reconnection succeeds** | Mark as `CONNECTED` Resume scheduling | Session restored (G6) |\n| **Disconnect during task** | Mark tasks as `FAILED` Propagate to ConstellationAgent Trigger DAG edit | N/A |\n\n### ConstellationClient Disconnection\n\n!!!danger \"Bidirectional Fault Handling\"\n When the **ConstellationClient** disconnects, all Device Agent Services:\n \n 1. Receive termination signal \n 2. **Abort all ongoing tasks** tied to that client \n 3. Prevent resource leakage and zombie processes \n 4. Maintain end-to-end consistency\n\n**Guarantees:**\n\n- ✅ No orphaned tasks \n- ✅ Synchronized state across client-server boundary \n- ✅ Rapid recovery when connection restored (G6)\n- ✅ Consistent TaskConstellation state (G4)\n\n[→ See resilience implementation](./resilience.md)\n\n## Extensibility Mechanisms\n\nAIP provides multiple extension points for domain-specific needs without modifying the core protocol, supporting composable extensibility (G5).\n\n### 1. Protocol Middleware\n\nAdd custom processing to message pipeline:\n\n```python\nfrom aip.protocol.base import ProtocolMiddleware\n\nclass AuditMiddleware(ProtocolMiddleware):\n async def process_outgoing(self, msg):\n log_to_audit_trail(msg)\n return msg\n \n async def process_incoming(self, msg):\n log_to_audit_trail(msg)\n return msg\n```\n\n### 2. Custom Message Handlers\n\nRegister handlers for new message types:\n\n```python\nprotocol.register_handler(\"custom_type\", handle_custom_message)\n```\n\n### 3. Transport Layer\n\nPluggable transport (default: WebSocket) (G5):\n\n```python\nfrom aip.transport import CustomTransport\nprotocol.transport = CustomTransport(config)\n```\n\n[→ See extensibility guide](./protocols.md)\n\n## Integration with UFO² Ecosystem\n\n| Component | Integration Point | Benefit |\n|-----------|-------------------|---------|\n| **MCP Servers** | Command execution model aligns with MCP message formats | Unified interface for system actions and LLM tool calls |\n| **TaskConstellation** | Real-time state synchronization via AIP messages | Planning DAG always reflects distributed execution state |\n| **Configuration System** | Agent endpoints, capabilities managed via UFO² config | Centralized management, type-safe validation |\n| **Logging & Monitoring** | Comprehensive logging at all protocol layers | Debugging, performance monitoring, audit trails |\n\nAIP abstracts network/device heterogeneity, allowing the orchestrator to treat all agents as **first-class citizens** in a single event-driven control plane.\n\n**Related Documentation:**\n\n- [TaskConstellation (DAG orchestrator)](../galaxy/constellation/task_constellation.md)\n- [ConstellationAgent (orchestration agent)](../galaxy/constellation_agent/overview.md)\n- [MCP Integration Guide](../mcp/overview.md)\n- [Configuration System](../configuration/system/system_config.md)\n**Next Steps:**\n\n- 📖 [Message Reference](./messages.md) - Complete message type documentation \n- 🔧 [Protocol Guide](./protocols.md) - Implementation details and best practices \n- 🌐 [Transport Layer](./transport.md) - WebSocket configuration and optimization \n- 🔌 [Endpoints](./endpoints.md) - Endpoint setup and usage patterns \n- 🛡️ [Resilience](./resilience.md) - Connection management and fault tolerance\n\n## Summary\n\nAIP transforms distributed workflow execution into a **coherent, safe, and adaptive system** where reasoning and execution converge seamlessly across diverse agents and environments.\n\n**Key Takeaways:**\n\n| Aspect | Impact | Goals |\n|--------|--------|-------|\n| **Persistence** | Long-lived connections reduce overhead, maintain context | G1 |\n| **Low Latency** | WebSocket enables real-time event propagation | G1 |\n| **Capability Discovery** | Multi-source profiling unifies heterogeneous agents | G2 |\n| **Reliability** | Heartbeats, timeouts, auto-reconnection ensure graceful degradation | G3, G6 |\n| **Determinism** | Sequential command execution, explicit ID correlation | G4 |\n| **Extensibility** | Middleware hooks, pluggable transports, custom handlers | G5 |\n| **Developer UX** | Strongly-typed messages, clear errors reduce integration effort | G5 |\n\nBy decomposing orchestration into five logical layers—each addressing specific reliability and adaptability concerns—AIP enables UFO² to maintain **correctness and availability** under DAG evolution (G4, G5), agent churn (G3, G6), and heterogeneous execution environments (G1, G2).\n"
},
{
"path": "documents/docs/aip/protocols.md",
"content": "# AIP Protocol Reference\n\n## Protocol Stack Overview\n\nAIP uses a three-layer architecture where specialized protocols handle domain-specific concerns, the core protocol manages message processing, and the transport layer provides network communication.\n\n```mermaid\ngraph TB\n subgraph \"Specialized Protocols\"\n RP[RegistrationProtocol]\n TEP[TaskExecutionProtocol]\n CP[CommandProtocol]\n HP[HeartbeatProtocol]\n DIP[DeviceInfoProtocol]\n end\n \n subgraph \"Core Protocol\"\n AIP[\"AIPProtocol Message serialization Middleware pipeline Message routing\"]\n end\n \n subgraph \"Transport Layer\"\n WS[WebSocket]\n HTTP3[HTTP/3 Future]\n GRPC[gRPC Future]\n end\n \n RP --> AIP\n TEP --> AIP\n CP --> AIP\n HP --> AIP\n DIP --> AIP\n \n AIP --> WS\n AIP -.-> HTTP3\n AIP -.-> GRPC\n \n style AIP fill:#e1f5ff\n style WS fill:#f0ffe1\n```\n\nThis layered design enables clean separation of concerns: specialized protocols implement domain logic, the core protocol handles serialization and routing, and the transport layer abstracts network details. Dashed arrows indicate future transport options.\n\n### Protocol Comparison\n\n| Protocol | Purpose | Key Messages | Use When |\n|----------|---------|--------------|----------|\n| **RegistrationProtocol** | Agent capability advertisement | `REGISTER`, `HEARTBEAT(OK)` | Device joins constellation |\n| **TaskExecutionProtocol** | Task lifecycle management | `TASK`, `COMMAND`, `TASK_END` | Executing multi-step tasks |\n| **CommandProtocol** | Command validation | Validation utilities | Before sending/receiving commands |\n| **HeartbeatProtocol** | Connection health monitoring | `HEARTBEAT` | Periodic keepalive |\n| **DeviceInfoProtocol** | Telemetry exchange | `DEVICE_INFO_REQUEST/RESPONSE` | Querying device state |\n\n---\n\n## Core Protocol: AIPProtocol\n\n`AIPProtocol` provides transport-agnostic message handling with middleware support and automatic serialization.\n\n### Quick Start\n\n```python\nfrom aip.protocol import AIPProtocol\nfrom aip.transport import WebSocketTransport\n\ntransport = WebSocketTransport()\nprotocol = AIPProtocol(transport)\n```\n\n### Core Operations\n\n| Operation | Method | Description |\n|-----------|--------|-------------|\n| **Send** | `send_message(msg)` | Serialize and send Pydantic message |\n| **Receive** | `receive_message(MsgType)` | Receive and deserialize to type |\n| **Dispatch** | `dispatch_message(msg)` | Route to registered handler |\n| **Error** | `send_error(error, id)` | Send error notification |\n| **Status** | `is_connected()` | Check connection state |\n\n### Middleware Pipeline\n\nAdd middleware for logging, authentication, metrics, or custom transformations.\n\n```python\nfrom aip.protocol.base import ProtocolMiddleware\n\nclass LoggingMiddleware(ProtocolMiddleware):\n async def process_outgoing(self, msg):\n logger.info(f\"→ {msg.type}\")\n return msg\n \n async def process_incoming(self, msg):\n logger.info(f\"← {msg.type}\")\n return msg\n\nprotocol.add_middleware(LoggingMiddleware())\n```\n\n**Execution Order:**\n\n- **Outgoing**: First added → First executed\n- **Incoming**: Last added → First executed (reverse)\n\n### Message Handler Registration\n\n```python\nasync def handle_task(msg):\n logger.info(f\"Handling task: {msg.task_name}\")\n # Process task...\n\nprotocol.register_handler(\"task\", handle_task)\n\n# Auto-dispatch to handler\nawait protocol.dispatch_message(server_msg)\n```\n\n[→ See transport configuration](./transport.md)\n\n---\n\n## RegistrationProtocol {#registration-protocol}\n\nHandles initial registration and capability advertisement when agents join the constellation.\n\n### Registration Flow\n\nThe following diagram shows the two-way handshake for device registration, including validation and acknowledgment:\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n \n C->>S: REGISTER (device_id, metadata, capabilities)\n S->>S: Validate registration and Store AgentProfile\n alt Success\n S->>C: HEARTBEAT (OK)\n else Failure\n S->>C: ERROR (reason)\n end\n```\n\nUpon successful registration, the server stores the `AgentProfile` and responds with a `HEARTBEAT` acknowledgment. Failed registrations (e.g., duplicate device_id) return an `ERROR` message with diagnostic details.\n\n### Device Registration\n\n**Client-Side Registration:**\n\n```python\nfrom aip.protocol import RegistrationProtocol\n\nreg_protocol = RegistrationProtocol(transport)\n\nsuccess = await reg_protocol.register_as_device(\n device_id=\"windows_agent_001\",\n metadata={\n \"platform\": \"windows\",\n \"os_version\": \"Windows 11\",\n \"cpu\": \"Intel i7\",\n \"ram_gb\": 16,\n \"capabilities\": [\"ui_automation\", \"file_operations\"]\n },\n platform=\"windows\"\n)\n```\n\n**Auto-Added Fields:**\n\n- `timestamp`: Registration time (ISO 8601)\n- `client_type`: Set to `ClientType.DEVICE`\n\n[→ See ClientType and ClientMessage in Message Reference](./messages.md)\n\n### Constellation Registration\n\n**Orchestrator Registration:**\n\n```python\nsuccess = await reg_protocol.register_as_constellation(\n constellation_id=\"orchestrator_001\",\n target_device=\"windows_agent_001\", # Required\n metadata={\n \"orchestrator_version\": \"2.0.0\",\n \"max_concurrent_tasks\": 10\n }\n)\n```\n\n!!!warning \"Target Device Required\"\n Constellation clients **must** specify `target_device` to indicate which device they coordinate.\n\n### Server-Side Handlers\n\n| Method | Purpose | When to Use |\n|--------|---------|-------------|\n| `send_registration_confirmation()` | Acknowledge successful registration | After validating and storing profile |\n| `send_registration_error()` | Report registration failure | Invalid ID, duplicate, or validation error |\n\n---\n\n## TaskExecutionProtocol {#task-execution-protocol}\n\nManages the complete task lifecycle: assignment → command execution → result reporting → completion.\n\n### Task Lifecycle\n\nThis state diagram shows the complete task execution lifecycle, including the multi-turn command loop where agents can request additional commands before completion:\n\n```mermaid\nstateDiagram-v2\n [*] --> TaskAssigned: TASK\n TaskAssigned --> CommandSent: COMMAND\n CommandSent --> ResultsReceived: COMMAND_RESULTS\n ResultsReceived --> CommandSent: CONTINUE\n ResultsReceived --> TaskCompleted: COMPLETED/FAILED\n TaskCompleted --> [*]: TASK_END\n \n note right of ResultsReceived\n Multi-turn: Agent can request\n more commands before completion\n end note\n```\n\nThe `CONTINUE` loop (ResultsReceived → CommandSent) enables iterative task refinement where the agent can execute commands, evaluate results, and request follow-up commands before declaring completion.\n\n### Client → Server: Task Request\n\n```python\nfrom aip.protocol import TaskExecutionProtocol\n\ntask_protocol = TaskExecutionProtocol(transport)\n\nawait task_protocol.send_task_request(\n request=\"Open Notepad and create test.txt\",\n task_name=\"create_notepad_file\",\n session_id=\"session_123\",\n client_id=\"windows_agent_001\",\n client_type=ClientType.DEVICE,\n metadata={\"priority\": \"high\"}\n)\n```\n\n### Server → Client: Task Assignment\n\n```python\nawait task_protocol.send_task_assignment(\n user_request=\"Open Notepad and create a file\",\n task_name=\"create_notepad_file\",\n session_id=\"session_123\",\n response_id=\"resp_001\",\n agent_name=\"AppAgent\",\n process_name=\"notepad.exe\"\n)\n```\n\n### Server → Client: Command Dispatch\n\nSend multiple commands in one message to reduce network overhead.\n\n**Method 1: Using ServerMessage**\n\n```python\nfrom aip.messages import ServerMessage, Command, TaskStatus\n\nserver_msg = ServerMessage(\n type=ServerMessageType.COMMAND,\n status=TaskStatus.CONTINUE,\n session_id=\"session_123\",\n response_id=\"resp_002\",\n actions=[\n Command(tool_name=\"launch_application\", \n parameters={\"app_name\": \"notepad\"}, \n tool_type=\"action\", call_id=\"cmd_001\"),\n Command(tool_name=\"type_text\", \n parameters={\"text\": \"Hello\"}, \n tool_type=\"action\", call_id=\"cmd_002\")\n ]\n)\n\nawait task_protocol.send_command(server_msg)\n```\n\n**Method 2: Using send_commands**\n\n```python\nawait task_protocol.send_commands(\n actions=[Command(...)],\n session_id=\"session_123\",\n response_id=\"resp_003\",\n status=TaskStatus.CONTINUE,\n agent_name=\"AppAgent\"\n)\n```\n\n### Client → Server: Command Results\n\n```python\nfrom aip.messages import Result, ResultStatus\n\nawait task_protocol.send_command_results(\n action_results=[\n Result(status=ResultStatus.SUCCESS, \n result={\"app_launched\": True}, \n call_id=\"cmd_001\"),\n Result(status=ResultStatus.SUCCESS, \n result={\"text_entered\": True}, \n call_id=\"cmd_002\")\n ],\n session_id=\"session_123\",\n client_id=\"windows_agent_001\",\n prev_response_id=\"resp_002\", # Links to COMMAND message\n status=TaskStatus.CONTINUE\n)\n```\n\n[→ See Result and ResultStatus definitions in Message Reference](./messages.md)\n\n### Task Completion\n\n**Server → Client: Success**\n\n```python\nawait task_protocol.send_task_end(\n session_id=\"session_123\",\n status=TaskStatus.COMPLETED,\n result={\n \"file_created\": True,\n \"path\": \"C:\\\\Users\\\\user\\\\test.txt\"\n },\n response_id=\"resp_999\"\n)\n```\n\n**Server → Client: Failure**\n\n```python\nawait task_protocol.send_task_end(\n session_id=\"session_123\",\n status=TaskStatus.FAILED,\n error=\"Notepad failed to launch: Access denied\",\n response_id=\"resp_999\"\n)\n```\n\n### Complete Task Flow\n\nThis comprehensive sequence diagram shows the complete flow from task request to completion, including the multi-turn command loop where the agent iteratively executes commands and requests follow-up actions:\n\n```mermaid\nsequenceDiagram\n participant CC as ConstellationClient\n participant CA as ConstellationAgent\n participant DS as DeviceService\n participant DC as DeviceClient\n \n CC->>CA: TASK request\n CA->>DS: TASK assignment\n DS->>DC: TASK (forward)\n \n loop Multi-turn execution\n DC->>DS: Request COMMAND\n DS->>CA: Forward request\n CA->>CA: Plan next action\n CA->>DS: COMMAND\n DS->>DC: COMMAND (forward)\n DC->>DC: Execute\n DC->>DS: COMMAND_RESULTS\n DS->>CA: COMMAND_RESULTS\n end\n \n CA->>DS: TASK_END\n DS->>DC: TASK_END (forward)\n CC->>CC: Update TaskConstellation\n```\n\nThe loop in the middle represents iterative task execution where the agent can perform multiple command cycles before determining the task is complete. Each cycle involves planning, execution, and result evaluation.\n\n---\n\n## CommandProtocol\n\nProvides validation utilities for commands and results before transmission.\n\n### Validation Methods\n\n| Method | Validates | Returns |\n|--------|-----------|---------|\n| `validate_command(cmd)` | Single command structure | `bool` |\n| `validate_commands(cmds)` | List of commands | `bool` |\n| `validate_result(result)` | Single result structure | `bool` |\n| `validate_results(results)` | List of results | `bool` |\n\n### Usage Pattern\n\n```python\nfrom aip.protocol import CommandProtocol\n\ncmd_protocol = CommandProtocol(transport)\n\n# Validate before sending\ncmd = Command(tool_name=\"click\", parameters={\"id\": \"btn\"}, tool_type=\"action\")\n\nif cmd_protocol.validate_command(cmd):\n await task_protocol.send_commands([cmd], ...)\nelse:\n logger.error(\"Invalid command structure\")\n\n# Validate results before transmission\nresults = [Result(...), Result(...)]\n\nif cmd_protocol.validate_results(results):\n await task_protocol.send_command_results(results, ...)\n```\n\n!!!warning \"Validation Best Practice\"\n Always validate commands and results before transmission to catch protocol errors early and prevent runtime failures.\n\n---\n\n## HeartbeatProtocol {#heartbeat-protocol}\n\nPeriodic keepalive messages detect broken connections and network issues.\n\n### Heartbeat Flow\n\nThe heartbeat protocol uses a simple ping-pong pattern to verify connection health at regular intervals:\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n \n loop Every 20-30s\n C->>S: HEARTBEAT (client_id)\n S->>S: Update last_seen timestamp\n S->>C: HEARTBEAT (OK)\n end\n \n Note over C,S: If no response → Connection dead\n```\n\nIf the server fails to receive a heartbeat within the timeout window, it marks the connection as dead and triggers disconnection handling. This prevents silent connection failures from going undetected.\n\n### Client-Side Heartbeat\n\n```python\nfrom aip.protocol import HeartbeatProtocol\n\nheartbeat_protocol = HeartbeatProtocol(transport)\n\nawait heartbeat_protocol.send_heartbeat(\n client_id=\"windows_agent_001\",\n metadata={\"custom_info\": \"value\"} # Optional\n)\n```\n\n### Server-Side Response\n\n```python\nawait heartbeat_protocol.send_heartbeat_ack(\n response_id=\"resp_hb_001\"\n)\n```\n\n!!!tip \"Automatic Management\"\n The `HeartbeatManager` automates heartbeat sending—you rarely need to call these methods directly.\n\n[→ See HeartbeatManager](./resilience.md#heartbeat-manager)\n\n---\n\n## DeviceInfoProtocol\n\nRequest and report device hardware/software information for informed scheduling.\n\n### Info Request Flow\n\nThe server can request fresh device information at any time to make informed scheduling decisions:\n\n```mermaid\nsequenceDiagram\n participant S as Server\n participant C as Client\n \n S->>C: DEVICE_INFO_REQUEST\n C->>C: Collect telemetry (OS, CPU, GPU, RAM, etc.)\n C->>S: DEVICE_INFO_RESPONSE (device specs)\n```\n\nThis pull-based telemetry model allows the orchestrator to query device capabilities on-demand (e.g., before assigning a GPU-intensive task) rather than relying on stale registration data.\n\n### Constellation → Server: Request Info\n\n```python\nfrom aip.protocol import DeviceInfoProtocol\n\ninfo_protocol = DeviceInfoProtocol(transport)\n\nawait info_protocol.request_device_info(\n constellation_id=\"orchestrator_001\",\n target_device=\"windows_agent_001\",\n request_id=\"req_info_001\"\n)\n```\n\n### Server → Client: Provide Info\n\nThe server responds with device information (or an error if collection failed):\n\n```python\ndevice_info = {\n \"os\": \"Windows 11\",\n \"cpu\": \"Intel i7-12700K\",\n \"ram_gb\": 32,\n \"gpu\": \"NVIDIA RTX 3080\",\n \"disk_free_gb\": 500,\n \"active_processes\": 145,\n \"network_status\": \"connected\"\n}\n\nawait info_protocol.send_device_info_response(\n device_info=device_info,\n request_id=\"req_info_001\",\n error=None # Set to error message string if info collection failed\n)\n```\n\n### Use Cases\n\n!!!success \"Device-Aware Task Scheduling\"\n - **GPU-aware scheduling**: Check GPU availability before assigning vision tasks\n - **Load balancing**: Distribute tasks based on CPU/RAM usage\n - **Health monitoring**: Track device status over time\n\n---\n\n## Protocol Patterns\n\n### Multi-Turn Conversations\n\nUse `prev_response_id` to maintain conversation context across multiple exchanges.\n\nThis diagram shows how messages are chained together using `prev_response_id` to maintain conversation context:\n\n```mermaid\ngraph LR\n A[\"Server: COMMAND response_id=001\"] --> B[\"Client: RESULTS prev_response_id=001 request_id=002\"]\n B --> C[\"Server: COMMAND response_id=003\"]\n C --> D[\"Client: RESULTS prev_response_id=003 request_id=004\"]\n```\n\nEach response references the previous message's `response_id` in its `prev_response_id` field, forming a traceable conversation chain. This enables debugging, audit trails, and request-response correlation.\n\n```python\n# Turn 1: Server sends command\nawait protocol.send_message(ServerMessage(\n type=ServerMessageType.COMMAND,\n response_id=\"resp_001\",\n ...\n))\n\n# Turn 2: Client sends results\nawait protocol.send_message(ClientMessage(\n type=ClientMessageType.COMMAND_RESULTS,\n request_id=\"req_001\",\n prev_response_id=\"resp_001\", # Links to previous\n ...\n))\n```\n\n### Session-Based Communication\n\nAll messages in a task share the same `session_id` for traceability.\n\n```python\nSESSION_ID = \"session_abc123\"\n\n# All use same session_id\ntask_msg.session_id = SESSION_ID\ncommand_msg.session_id = SESSION_ID\nresults_msg.session_id = SESSION_ID\ntask_end_msg.session_id = SESSION_ID\n```\n\n### Error Recovery\n\n**Protocol-Level Errors (Connection Issues):**\n\n```python\ntry:\n await protocol.send_message(msg)\nexcept ConnectionError:\n await reconnect()\nexcept IOError as e:\n logger.error(f\"I/O error: {e}\")\n```\n\n**Application-Level Errors (Task Failures):**\n\n```python\n# Send error through protocol\nawait protocol.send_error(\n error_msg=\"Invalid command: tool_name missing\",\n response_id=msg.response_id\n)\n```\n\n---\n\n## Best Practices\n\n### Protocol Selection\n\nUse specialized protocols instead of manually constructing messages with `AIPProtocol`.\n\n| Task | Protocol |\n|------|----------|\n| Agent registration | `RegistrationProtocol` |\n| Task execution | `TaskExecutionProtocol` |\n| Command validation | `CommandProtocol` |\n| Keepalive | `HeartbeatProtocol` |\n| Device telemetry | `DeviceInfoProtocol` |\n\n### Validation\n\n- Always validate commands/results before transmission\n- Use `MessageValidator` for message integrity checks\n- Catch validation errors early\n\n### Session Management\n\n- **Always set `session_id`** for task-related messages\n- Use **correlation IDs** (`prev_response_id`) for multi-turn conversations\n- **Generate unique IDs** with `uuid.uuid4()`\n\n### Error Handling\n\n- **Distinguish** protocol errors (connection) from application errors (task failure)\n- **Propagate errors** explicitly through error messages\n- **Leverage middleware** for cross-cutting concerns (logging, metrics, auth)\n\n!!!danger \"Resource Cleanup\"\n Always close protocols when done to release transport resources.\n\n---\n\n## Quick Reference\n\n### Import Protocols\n\n```python\nfrom aip.protocol import (\n AIPProtocol,\n RegistrationProtocol,\n TaskExecutionProtocol,\n CommandProtocol,\n HeartbeatProtocol,\n DeviceInfoProtocol,\n)\n```\n\n### Related Documentation\n\n- [Message Reference](./messages.md) - Message types and structures\n- [Transport Layer](./transport.md) - WebSocket implementation \n- [Endpoints](./endpoints.md) - Protocol usage in endpoints\n- [Resilience](./resilience.md) - Connection management and recovery\n- [Overview](./overview.md) - System architecture\n"
},
{
"path": "documents/docs/aip/resilience.md",
"content": "# AIP Resilience\n\nAIP's resilience layer ensures stable communication and consistent orchestration across distributed agent constellations through automatic reconnection, heartbeat monitoring, and timeout management.\n\n## Resilience Components\n\n| Component | Purpose | Key Features |\n|-----------|---------|--------------|\n| **ReconnectionStrategy** | Auto-reconnect on disconnect | Exponential backoff, max retries, policies |\n| **HeartbeatManager** | Connection health monitoring | Periodic keepalive, failure detection |\n| **TimeoutManager** | Operation timeout enforcement | Configurable timeouts, async cancellation |\n| **ConnectionProtocol** | State management | Bidirectional fault handling, task cleanup |\n\n---\n\n## Resilient Connection Protocol\n\nThe Resilient Connection Protocol governs how connection disruptions are detected, handled, and recovered between ConstellationClient and Device Agents.\n\n### Connection State Diagram\n\nThis state diagram shows how devices transition between connection states and the internal sub-states during disconnection recovery:\n\n```mermaid\nstateDiagram-v2\n [*] --> CONNECTED: Initial connection\n CONNECTED --> DISCONNECTED: Connection lost\n DISCONNECTED --> CONNECTED: Reconnect succeeds\n DISCONNECTED --> [*]: Max retries / Manual removal\n \n state DISCONNECTED {\n [*] --> DetectFailure\n DetectFailure --> CancelTasks\n CancelTasks --> NotifyOrchestrator\n NotifyOrchestrator --> AttemptReconnect\n AttemptReconnect --> [*]: Success\n }\n \n note right of DISCONNECTED\n • Invisible to scheduler\n • Tasks marked FAILED\n • Auto-reconnect triggered\n end note\n```\n\nThe nested states within `DISCONNECTED` show the cleanup and recovery sequence: detect the failure, cancel running tasks, notify the orchestrator, then attempt reconnection with exponential backoff.\n\n### Device Disconnection Workflow\n\n!!!danger \"Impact on Running Tasks\"\n All tasks running on a disconnected device are **immediately marked as FAILED** to maintain TaskConstellation consistency.\n\n| Phase | Action | Trigger |\n|-------|--------|---------|\n| **1. Detection** | Connection failure detected | WebSocket close, heartbeat timeout, network error |\n| **2. State Transition** | `CONNECTED` → `DISCONNECTED` | Agent excluded from scheduler |\n| **3. Task Failure** | Mark tasks as `TASK_FAILED` | Propagate to ConstellationAgent |\n| **4. Auto-Reconnect** | Background routine triggered | Exponential backoff |\n| **5. Recovery** | `DISCONNECTED` → `CONNECTED` | Resume scheduling |\n\n**Task Cancellation:**\n\n```python\n# Automatically called on disconnection\nawait device_server.cancel_device_tasks(client_id, reason=\"device_disconnected\")\n```\n\n### ConstellationClient Disconnection\n\nWhen ConstellationClient disconnects, Device Agent Servers proactively clean up to prevent orphaned tasks.\n\nThis sequence diagram shows the proactive cleanup sequence when the orchestrator disconnects, ensuring all running tasks are properly aborted:\n\n```mermaid\nsequenceDiagram\n participant CC as ConstellationClient\n participant DAS as Device Agent Server\n participant Tasks as Running Tasks\n \n CC-xDAS: Connection lost\n DAS->>DAS: Detect termination signal\n DAS->>Tasks: Abort all tasks for client\n Tasks->>Tasks: Cleanup resources\n DAS->>DAS: Maintain consistency\n \n Note over DAS: Prevents: • Resource leaks • Orphaned tasks • Inconsistent states\n```\n\nThe `x` marker on the connection arrow indicates an abnormal termination. The server immediately detects this and cascades the cleanup signal to all associated tasks, preventing resource leaks.\n\n**Guarantees:**\n\n- ✅ No orphaned tasks or zombie processes\n- ✅ End-to-end consistency across client-server boundary \n- ✅ Automatic resource cleanup\n- ✅ Synchronized task state reflection\n\n---\n\n## ReconnectionStrategy\n\nManages reconnection attempts with configurable backoff policies to handle transient network failures.\n\n### Configuration\n\n```python\nfrom aip.resilience import ReconnectionStrategy, ReconnectionPolicy\n\nstrategy = ReconnectionStrategy(\n max_retries=5, # Maximum attempts\n initial_backoff=1.0, # Initial delay (seconds)\n max_backoff=60.0, # Maximum delay (seconds)\n backoff_multiplier=2.0, # Exponential multiplier\n policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF\n)\n```\n\n[→ See how ReconnectionStrategy is used in endpoints](./endpoints.md)\n\n### Backoff Policies\n\nSelect the policy that matches your deployment environment's network characteristics.\n\n| Policy | Backoff Pattern | Best For | Example Sequence |\n|--------|----------------|----------|------------------|\n| **EXPONENTIAL_BACKOFF** | Doubles each attempt | Internet, unreliable networks | 1s → 2s → 4s → 8s → 16s |\n| **LINEAR_BACKOFF** | Linear increase | Local networks, testing | 1s → 2s → 3s → 4s → 5s |\n| **IMMEDIATE** | No delay | ⚠️ Testing only | 0s → 0s → 0s → 0s → 0s |\n| **NONE** | No reconnection | Manual control | Disabled |\n\n!!!danger \"IMMEDIATE Policy Warning\"\n `IMMEDIATE` policy can overwhelm servers with rapid retry attempts. **Use only for testing.**\n\n### Reconnection Workflow\n\nThis flowchart shows the complete reconnection logic from failure detection through recovery or permanent failure:\n\n```mermaid\ngraph TD\n A[Connection Lost] --> B[Cancel Pending Tasks]\n B --> C[Notify Upper Layers]\n C --> D{Retry Count < Max?}\n D -->|Yes| E[Calculate Backoff]\n E --> F[Wait Backoff Duration]\n F --> G[Attempt Reconnect]\n G --> H{Success?}\n H -->|Yes| I[Restore Session]\n H -->|No| J[Increment Retry Count]\n J --> D\n D -->|No| K[Max Retries Reached]\n K --> L[Permanent Failure]\n I --> M[Resume Operations]\n \n style I fill:#d4edda\n style L fill:#f8d7da\n```\n\nThe loop between \"Attempt Reconnect\" and \"Increment Retry Count\" continues until either reconnection succeeds (green path) or max retries are exhausted (red path). Backoff duration increases with each failed attempt.\n\n### Reconnection Example\n\n```python\nasync def handle_disconnection(\n endpoint: AIPEndpoint,\n device_id: str,\n on_reconnect: Optional[Callable] = None\n):\n # Step 1: Cancel pending tasks\n await strategy._cancel_pending_tasks(endpoint, device_id)\n \n # Step 2: Notify upper layers\n await strategy._notify_disconnection(endpoint, device_id)\n \n # Step 3: Attempt reconnection\n reconnected = await strategy.attempt_reconnection(endpoint, device_id)\n \n # Step 4: Call reconnection callback\n if reconnected and on_reconnect:\n await on_reconnect()\n```\n\n### Custom Reconnection Callback\n\n```python\nasync def on_reconnected():\n logger.info(\"Device reconnected, resuming tasks\")\n await restore_task_queue()\n await sync_device_state()\n\nawait strategy.handle_disconnection(\n endpoint=endpoint,\n device_id=\"device_001\",\n on_reconnect=on_reconnected\n)\n```\n\n---\n\n## HeartbeatManager {#heartbeat-manager}\n\nSends periodic keepalive messages to detect broken connections before they cause failures.\n\n### Configuration\n\n```python\nfrom aip.resilience import HeartbeatManager\nfrom aip.protocol import HeartbeatProtocol\n\nheartbeat_protocol = HeartbeatProtocol(transport)\nheartbeat_manager = HeartbeatManager(\n protocol=heartbeat_protocol,\n default_interval=30.0 # 30 seconds\n)\n```\n\n[→ See HeartbeatProtocol reference](./protocols.md#heartbeat-protocol)\n\n### Lifecycle Management\n\n| Operation | Method | Description |\n|-----------|--------|-------------|\n| **Start** | `start_heartbeat(client_id, interval)` | Begin periodic heartbeat for client |\n| **Stop** | `stop_heartbeat(client_id)` | Stop heartbeat for specific client |\n| **Stop All** | `stop_all()` | Stop all active heartbeats |\n| **Check Status** | `is_running(client_id)` | Verify if heartbeat is active |\n| **Get Interval** | `get_interval(client_id)` | Retrieve current interval |\n\n### Usage Example\n\n```python\n# Start heartbeat for a client\nawait heartbeat_manager.start_heartbeat(\n client_id=\"device_001\",\n interval=20.0 # Override default\n)\n\n# Check if running\nif heartbeat_manager.is_running(\"device_001\"):\n logger.info(\"Heartbeat active\")\n\n# Stop for specific client\nawait heartbeat_manager.stop_heartbeat(\"device_001\")\n\n# Stop all heartbeats (cleanup)\nawait heartbeat_manager.stop_all()\n```\n\n### Heartbeat Loop Internals\n\nThe heartbeat manager automatically sends periodic heartbeats. If the protocol is not connected, it logs a warning and continues the loop:\n\n```python\nasync def _heartbeat_loop(client_id: str, interval: float):\n \"\"\"Internal heartbeat loop (automatic)\"\"\"\n try:\n while True:\n await asyncio.sleep(interval)\n \n if protocol.is_connected():\n try:\n await protocol.send_heartbeat(client_id)\n except Exception as e:\n logger.error(f\"Error sending heartbeat: {e}\")\n # Continue loop, connection manager handles disconnection\n else:\n logger.warning(\"Protocol not connected, skipping heartbeat\")\n \n except asyncio.CancelledError:\n logger.debug(\"Heartbeat loop cancelled\")\n```\n\n### Failure Detection\n\nWhen the transport layer fails to send a heartbeat (connection closed), errors are logged but the loop continues running. The connection manager is responsible for detecting the disconnection through transport-level errors and triggering the reconnection strategy.\n\nThis sequence diagram shows how heartbeat errors are handled:\n\n```mermaid\nsequenceDiagram\n participant HM as HeartbeatManager\n participant P as Protocol\n participant T as Transport\n \n loop Every interval\n HM->>P: send_heartbeat()\n P->>T: Send via WebSocket\n alt Connection alive\n T-->>P: Success\n P-->>HM: Continue\n else Connection dead\n T-xP: ConnectionError\n P-xHM: Error (caught)\n HM->>HM: Log error, continue loop\n Note over HM: Connection manager handles disconnection at transport level\n end\n end\n```\n\nThe `x` markers indicate error paths. When the transport layer fails to send a heartbeat, the error is caught and logged. The heartbeat loop continues, while the connection manager detects the disconnection at the transport level and initiates recovery.\n\n### Interval Guidelines\n\n| Environment | Recommended Interval | Rationale |\n|-------------|---------------------|-----------|\n| **Local network** | 10-20s | Quick failure detection, low latency |\n| **Internet** | 30-60s | Balance overhead vs detection speed |\n| **Mobile/Unreliable** | 60-120s | Reduce battery/bandwidth usage |\n| **Critical systems** | 5-10s | Fastest failure detection |\n\n---\n\n## TimeoutManager\n\nPrevents operations from hanging indefinitely by enforcing configurable timeouts with automatic cancellation.\n\n### Configuration\n\n```python\nfrom aip.resilience import TimeoutManager\n\ntimeout_manager = TimeoutManager(\n default_timeout=120.0 # 120 seconds\n)\n```\n\n[→ See how timeouts are used in protocol operations](./protocols.md)\n\n### Usage Patterns\n\n**Default Timeout:**\n\n```python\nresult = await timeout_manager.with_timeout(\n protocol.send_message(msg),\n operation_name=\"send_message\"\n)\n```\n\n**Custom Timeout:**\n\n```python\nresult = await timeout_manager.with_timeout(\n protocol.receive_message(ServerMessage),\n timeout=60.0,\n operation_name=\"receive_message\"\n)\n```\n\n### Error Handling\n\n```python\nfrom asyncio import TimeoutError\n\ntry:\n result = await timeout_manager.with_timeout(\n long_running_operation(),\n timeout=30.0\n )\nexcept TimeoutError:\n logger.error(\"Operation timed out after 30 seconds\")\n # Handle timeout: retry, fail task, notify user\n```\n\n### Recommended Timeouts\n\n| Operation | Timeout | Rationale |\n|-----------|---------|-----------|\n| **Registration** | 10-30s | Simple message exchange |\n| **Task Dispatch** | 30-60s | May involve scheduling logic |\n| **Command Execution** | 60-300s | Depends on command complexity |\n| **Heartbeat** | 5-10s | Fast failure detection needed |\n| **Disconnection** | 5-15s | Clean shutdown |\n| **Device Info Query** | 15-30s | Telemetry collection |\n\n---\n\n## Integration with Endpoints\n\nEndpoints automatically integrate all resilience components—no manual wiring needed.\n\n### Example: DeviceClientEndpoint\n\n```python\nfrom aip.endpoints import DeviceClientEndpoint\n\nendpoint = DeviceClientEndpoint(\n ws_url=\"ws://localhost:8000/ws\",\n ufo_client=client,\n max_retries=3, # Reconnection retries\n timeout=120.0 # Connection timeout\n)\n\n# Resilience handled automatically on start\nawait endpoint.start()\n```\n\n**Note**: The endpoint creates its own `ReconnectionStrategy` internally with the specified `max_retries`.\n\n### Built-In Features\n\n| Feature | Behavior | Configuration |\n|---------|----------|---------------|\n| **Auto-Reconnection** | Triggered on disconnect | Via `ReconnectionStrategy` |\n| **Heartbeat** | Starts on connection | Managed by `HeartbeatManager` |\n| **Timeout Enforcement** | Applied to all operations | Via `TimeoutManager` |\n| **Task Cancellation** | Auto-cancel on disconnect | Built-in to endpoint |\n\n[→ See endpoint documentation](./endpoints.md)\n[→ See WebSocket transport details](./transport.md)\n\n---\n\n## Best Practices by Environment\n\n### Local Network (Low Latency, High Reliability)\n\n```python\nstrategy = ReconnectionStrategy(\n max_retries=3,\n initial_backoff=1.0,\n max_backoff=10.0,\n policy=ReconnectionPolicy.LINEAR_BACKOFF\n)\nheartbeat_interval = 20.0 # Quick detection\ntimeout_default = 60.0\n```\n\n### Internet (Variable Latency, Moderate Reliability)\n\n```python\nstrategy = ReconnectionStrategy(\n max_retries=5,\n initial_backoff=2.0,\n max_backoff=60.0,\n policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF\n)\nheartbeat_interval = 30.0 # Balance overhead and detection\ntimeout_default = 120.0\n```\n\n### Unreliable Network (High Latency, Low Reliability)\n\n```python\nstrategy = ReconnectionStrategy(\n max_retries=10,\n initial_backoff=5.0,\n max_backoff=300.0, # Up to 5 minutes\n policy=ReconnectionPolicy.EXPONENTIAL_BACKOFF\n)\nheartbeat_interval = 60.0 # Reduce overhead\ntimeout_default = 180.0\n```\n\n---\n\n## Error Scenarios\n\n### Scenario 1: Transient Network Failure\n\n**Problem**: Network glitch disconnects client for 3 seconds.\n\n**Resolution**:\n1. ✅ Disconnection detected via heartbeat timeout\n2. ✅ Automatic reconnection triggered (1st attempt after 2s)\n3. ✅ Connection restored successfully\n4. ✅ Heartbeat resumes\n5. ✅ Tasks continue\n\n### Scenario 2: Prolonged Outage\n\n**Problem**: Device offline for 10 minutes.\n\n**Resolution**:\n1. ❌ Initial disconnection detected\n2. ⏳ Multiple reconnection attempts (exponential backoff: 2s, 4s, 8s, 16s, 32s)\n3. ❌ All attempts fail (max retries reached)\n4. ⚠️ Tasks marked as FAILED\n5. 📢 ConstellationAgent notified\n6. ♻️ Tasks reassigned to other devices\n\n### Scenario 3: Server Restart\n\n**Problem**: Server restarts, causing all clients to disconnect at once.\n\n**Resolution**:\n1. ⚠️ All clients detect disconnection\n2. ⏳ Each client begins reconnection (with jitter to avoid thundering herd)\n3. ✅ Server restarts and accepts connections\n4. ✅ Clients reconnect and re-register\n5. ✅ Task execution resumes\n\n### Scenario 4: Heartbeat Timeout\n\n**Problem**: Heartbeat not received within timeout period.\n\n**Resolution**:\n 1. ⏰ HeartbeatManager detects missing pong\n 2. ⚠️ Connection marked as potentially dead\n 3. 🔄 Disconnection handling triggered\n 4. ⏳ Reconnection attempted\n 5. ✅ If successful, heartbeat resumes\n\n---\n\n## Monitoring and Observability\n\n### Enable Resilience Logging\n\n```python\nimport logging\n\n# Enable detailed resilience logs\nlogging.getLogger(\"aip.resilience\").setLevel(logging.INFO)\n```\n\n### Custom Event Handlers\n\n```python\nclass CustomEndpoint(DeviceClientEndpoint):\n async def on_device_disconnected(self, device_id: str) -> None:\n # Custom cleanup\n await self.cleanup_resources(device_id)\n logger.warning(f\"Device {device_id} disconnected\")\n \n # Call parent implementation\n await super().on_device_disconnected(device_id)\n \n async def reconnect_device(self, device_id: str) -> bool:\n # Custom reconnection logic\n success = await self.custom_reconnect(device_id)\n \n if success:\n await self.restore_state(device_id)\n logger.info(f\"Device {device_id} reconnected\")\n \n return success\n```\n\n### Graceful Degradation\n\n```python\nif not await strategy.attempt_reconnection(endpoint, device_id):\n logger.error(f\"Failed to reconnect {device_id} after max retries\")\n \n # Graceful degradation\n await notify_operator(f\"Device {device_id} offline\")\n await reassign_tasks_to_other_devices(device_id)\n await update_monitoring_dashboard(device_id, \"offline\")\n```\n\n---\n\n## Testing Resilience\n\nTest resilience by simulating network failures and verifying recovery.\n\n```python\n# Simulate disconnection\nawait transport.close()\n\n# Verify reconnection\nassert await endpoint.reconnect_device(device_id)\n\n# Verify heartbeat resumes\nawait asyncio.sleep(1)\nassert heartbeat_manager.is_running(device_id)\n\n# Verify task state\nassert all(task.status == TaskStatus.FAILED for task in orphaned_tasks)\n```\n\n---\n\n## Quick Reference\n\n### Import Resilience Components\n\n```python\nfrom aip.resilience import (\n ReconnectionStrategy,\n ReconnectionPolicy,\n HeartbeatManager,\n TimeoutManager,\n)\n```\n\n### Related Documentation\n\n- [Endpoints](./endpoints.md) - How endpoints use resilience\n- [Transport Layer](./transport.md) - Transport-level connection management \n- [Protocol Reference](./protocols.md) - Protocol-level error handling\n- [Overview](./overview.md) - System architecture and design\n"
},
{
"path": "documents/docs/aip/transport.md",
"content": "# AIP Transport Layer\n\nThe transport layer provides a pluggable abstraction for AIP's network communication, decoupling protocol logic from underlying network implementations through a unified Transport interface.\n\n## Transport Architecture\n\nAIP uses a transport abstraction pattern that allows different network protocols to be swapped without changing higher-level protocol logic. The current implementation focuses on WebSocket, with future support planned for HTTP/3 and gRPC:\n\n```mermaid\ngraph TD\n subgraph \"Transport Abstraction\"\n TI[Transport Interface]\n TI --> |implements| WST[WebSocketTransport]\n TI --> |future| H3T[HTTP/3 Transport]\n TI --> |future| GRPC[gRPC Transport]\n end\n \n subgraph \"WebSocket Transport\"\n WST --> |client-side| WSC[websockets library]\n WST --> |server-side| FAPI[FastAPI WebSocket]\n WST --> |adapter| ADP[Unified Adapter]\n end\n \n subgraph \"Protocol Layer\"\n PROTO[AIP Protocols]\n PROTO --> |uses| TI\n end\n \n style WST fill:#d4edda\n style TI fill:#d1ecf1\n```\n\nThe unified adapter bridges client and server WebSocket libraries, providing a consistent interface regardless of which side of the connection you're on. This design pattern enables protocol code to be transport-agnostic.\n\n---\n\n## Transport Interface\n\nAll transport implementations must implement the `Transport` interface for interoperability.\n\n### Core Operations\n\n| Method | Purpose | Return Type |\n|--------|---------|-------------|\n| `connect(url, **kwargs)` | Establish connection to remote endpoint | `None` |\n| `send(data)` | Send raw bytes | `None` |\n| `receive()` | Receive raw bytes | `bytes` |\n| `close()` | Close connection gracefully | `None` |\n| `wait_closed()` | Wait for connection to fully close | `None` |\n| `is_connected` (property) | Check connection status | `bool` |\n\n### Interface Definition\n\n```python\nfrom aip.transport import Transport\n\nclass Transport(ABC):\n @abstractmethod\n async def connect(self, url: str, **kwargs) -> None:\n \"\"\"Connect to remote endpoint\"\"\"\n \n @abstractmethod\n async def send(self, data: bytes) -> None:\n \"\"\"Send data\"\"\"\n \n @abstractmethod\n async def receive(self) -> bytes:\n \"\"\"Receive data\"\"\"\n \n @abstractmethod\n async def close(self) -> None:\n \"\"\"Close connection\"\"\"\n \n @abstractmethod\n async def wait_closed(self) -> None:\n \"\"\"Wait for connection to fully close\"\"\"\n \n @property\n @abstractmethod\n def is_connected(self) -> bool:\n \"\"\"Check connection status\"\"\"\n```\n\n---\n\n## WebSocket Transport\n\n`WebSocketTransport` provides persistent, full-duplex, bidirectional communication over WebSocket protocol (RFC 6455).\n\n### Quick Start\n\n**Client-Side:**\n\n```python\nfrom aip.transport import WebSocketTransport\n\n# Create and configure\ntransport = WebSocketTransport(\n ping_interval=30.0,\n ping_timeout=180.0,\n close_timeout=10.0,\n max_size=100 * 1024 * 1024 # 100MB\n)\n\n# Connect\nawait transport.connect(\"ws://localhost:8000/ws\")\n\n# Communicate\nawait transport.send(b\"Hello Server\")\ndata = await transport.receive()\n\n# Cleanup\nawait transport.close()\n```\n\n**Server-Side (FastAPI):**\n\n```python\nfrom fastapi import WebSocket\nfrom aip.transport import WebSocketTransport\n\nasync def websocket_endpoint(websocket: WebSocket):\n await websocket.accept()\n \n # Wrap existing WebSocket\n transport = WebSocketTransport(websocket=websocket)\n \n # Use unified interface\n data = await transport.receive()\n await transport.send(b\"Response\")\n```\n\n**Note**: WebSocketTransport automatically detects whether it's wrapping a FastAPI WebSocket or a client connection and selects the appropriate adapter.\n\n[→ See how endpoints use WebSocketTransport](./endpoints.md)\n\n### Configuration Parameters\n\n\n🔧 Configuration Options (Click to expand)\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| **ping_interval** | `float` | `30.0` | Time between ping messages (seconds). Keepalive mechanism. |\n| **ping_timeout** | `float` | `180.0` | Max wait for pong response (seconds). Connection marked dead if exceeded. |\n| **close_timeout** | `float` | `10.0` | Timeout for graceful close handshake (seconds). |\n| **max_size** | `int` | `104857600` | Max message size in bytes (100MB). Messages exceeding this are rejected. |\n\n\n\n**Usage Guidelines:**\n\n!!!warning \"max_size for Large Payloads\"\n Set `max_size` based on application needs. Large screenshots, models, or binary data may require higher limits. Consider compression for payloads approaching this limit.\n\n### Connection States\n\nWebSocket connections transition through multiple states during their lifecycle. This diagram shows all possible states and transitions:\n\n```mermaid\nstateDiagram-v2\n [*] --> DISCONNECTED\n DISCONNECTED --> CONNECTING: connect()\n CONNECTING --> CONNECTED: Success\n CONNECTING --> ERROR: Failure\n CONNECTED --> DISCONNECTING: close()\n DISCONNECTING --> DISCONNECTED: Complete\n CONNECTED --> ERROR: Network failure\n ERROR --> DISCONNECTED: Reset\n \n note right of CONNECTED\n • is_connected = True\n • send/receive active\n • Ping/pong running\n end note\n```\n\nOnly the `CONNECTED` state allows data transmission. The `ERROR` state is a terminal state that requires reset before attempting reconnection.\n\n**State Definitions:**\n\n| State | Meaning | Actions Allowed |\n|-------|---------|-----------------|\n| `DISCONNECTED` | No active connection | `connect()` |\n| `CONNECTING` | Connection in progress | Wait for result |\n| `CONNECTED` | Active connection | `send()`, `receive()`, `close()` |\n| `DISCONNECTING` | Closing in progress | Wait for completion |\n| `ERROR` | Error occurred | Investigate, reset |\n\n**Check State:**\n\n```python\nfrom aip.transport import TransportState\n\nif transport.state == TransportState.CONNECTED:\n await transport.send(data)\nelse:\n logger.warning(\"Transport not connected\")\n```\n\n### Ping/Pong Keepalive\n\nWebSocket automatically sends ping frames at `ping_interval` to detect broken connections.\n\nThis sequence diagram shows the automatic ping/pong mechanism for detecting broken connections:\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n \n loop Every ping_interval\n C->>S: ping frame\n S->>C: pong frame\n Note over C: Connection healthy\n end\n \n C->>S: ping frame\n S-xC: No response\n Note over C: Timeout after ping_timeout\n C->>C: Mark connection dead\n C->>C: Close connection\n```\n\nThe `x` marker indicates a failed pong response. After `ping_timeout` expires without receiving a pong, the connection is automatically marked dead and closed, triggering reconnection logic.\n\n**Timeout Behavior:**\n\n- ✅ **Pong received within `ping_timeout`**: Connection healthy, continue\n- ❌ **No pong within `ping_timeout`**: Connection marked dead, automatic close triggered\n\n### Error Handling\n\n!!!danger \"Always Handle ConnectionError\"\n Connection failures can occur at any time due to network issues. Wrap send/receive in try-except blocks.\n\n**Connection Errors:**\n\n```python\ntry:\n await transport.connect(\"ws://localhost:8000/ws\")\nexcept ConnectionError as e:\n logger.error(f\"Failed to connect: {e}\")\n await handle_connection_failure()\n```\n\n**Send/Receive Errors:**\n\n```python\ntry:\n await transport.send(data)\n response = await transport.receive()\nexcept ConnectionError:\n logger.warning(\"Connection closed during operation\")\n await reconnect()\nexcept IOError as e:\n logger.error(f\"I/O error: {e}\")\n await handle_io_error(e)\n```\n\n**Graceful Shutdown:**\n\n```python\ntry:\n # Close with timeout\n await transport.close()\n \n # Wait for complete shutdown\n await transport.wait_closed()\nexcept Exception as e:\n logger.error(f\"Error during shutdown: {e}\")\n```\n\n**Note**: The transport sends a WebSocket close frame and waits for the peer's close frame within `close_timeout` before terminating the connection.\n\n### Adapter Pattern\n\nAIP uses adapters to provide a unified interface across different WebSocket libraries without exposing implementation details.\n\n**Supported WebSocket Implementations:**\n\n| Implementation | Use Case | Adapter |\n|----------------|----------|---------|\n| **websockets library** | Client-side connections | `WebSocketsLibAdapter` |\n| **FastAPI WebSocket** | Server-side endpoints | `FastAPIWebSocketAdapter` |\n\n**Automatic Detection:**\n\n```python\n# Server-side: Automatically uses FastAPIWebSocketAdapter\ntransport = WebSocketTransport(websocket=fastapi_websocket)\n\n# Client-side: Automatically uses WebSocketsLibAdapter\ntransport = WebSocketTransport()\nawait transport.connect(\"ws://server:8000/ws\")\n```\n\n**Benefits:**\n\n- ✅ Protocol-level code remains unchanged across client/server\n- ✅ API differences abstracted by adapters\n- ✅ Easy to add new WebSocket implementations\n- ✅ Testability through adapter mocking\n\n---\n\n## Message Encoding\n\nAIP uses UTF-8 encoded JSON for all messages, leveraging Pydantic for serialization/deserialization.\n\n### Encoding Flow\n\nThis diagram shows the transformation steps from Pydantic model to network bytes:\n\n```mermaid\ngraph LR\n A[Pydantic Model] -->|model_dump_json| B[JSON String]\n B -->|encode utf-8| C[bytes]\n C -->|transport.send| D[Network]\n \n style A fill:#d4edda\n style D fill:#d1ecf1\n```\n\nPydantic handles type validation and JSON serialization, UTF-8 encoding converts to bytes, then the transport layer sends over the network. Decoding follows the reverse path.\n\n**Send Example:**\n\n```python\nfrom aip.messages import ClientMessage\n\n# 1. Create Pydantic model\nmsg = ClientMessage(\n message_type=\"TASK_RESULT\",\n task_id=\"task_123\",\n result={\"status\": \"success\"}\n)\n\n# 2. Serialize to JSON string\njson_str = msg.model_dump_json()\n\n# 3. Encode to bytes\nbytes_data = json_str.encode('utf-8')\n\n# 4. Send via transport\nawait transport.send(bytes_data)\n```\n\n### Decoding Flow\n\n```mermaid\ngraph LR\n A[Network] -->|transport.receive| B[bytes]\n B -->|decode utf-8| C[JSON String]\n C -->|model_validate_json| D[Pydantic Model]\n \n style A fill:#d1ecf1\n style D fill:#d4edda\n```\n\n**Receive Example:**\n\n```python\nfrom aip.messages import ServerMessage\n\n# 1. Receive bytes\nbytes_data = await transport.receive()\n\n# 2. Decode to JSON string\njson_str = bytes_data.decode('utf-8')\n\n# 3. Deserialize to Pydantic model\nmsg = ServerMessage.model_validate_json(json_str)\n\n# 4. Use typed data\nprint(f\"Task ID: {msg.task_id}\")\n```\n\n---\n\n## Performance Optimization\n\n### Performance Comparison\n\n| Scenario | Recommended Configuration | Rationale |\n|----------|---------------------------|-----------|\n| **Large Messages** | `max_size=500MB`, compression | Screenshots, binary data |\n| **High Throughput** | Batch messages, `ping_interval=60s` | Reduce overhead per message |\n| **Low Latency** | Dedicated connections, `ping_interval=10s` | Fast failure detection |\n| **Mobile Networks** | `ping_interval=60s`, compression | Reduce battery/bandwidth usage |\n\n### Optimization Strategies\n\n**Large Messages Strategy:**\n\nFor messages approaching `max_size`:\n\n**Option 1: Compression**\n```python\n import gzip\n \n compressed = gzip.compress(large_data)\n await transport.send(compressed)\n ```\n\n**Option 2: Chunking**\n```python\n chunk_size = 1024 * 1024 # 1MB chunks\n for i in range(0, len(large_data), chunk_size):\n chunk = large_data[i:i+chunk_size]\n await transport.send(chunk)\n ```\n\n**Option 3: Streaming Protocol**\n\nConsider implementing a custom streaming protocol for very large payloads.\n\n[→ See message encoding details in Protocol Reference](./protocols.md)\n\n**High Throughput Strategy:**\n\nFor high message rates:\n\n**Batch Messages:**\n```python\n batch = [msg1, msg2, msg3, msg4]\n batch_json = json.dumps([msg.model_dump() for msg in batch])\n await transport.send(batch_json.encode('utf-8'))\n ```\n\n**Reduce Ping Frequency:**\n```python\ntransport = WebSocketTransport(\n ping_interval=60.0 # Less overhead\n)\n```\n\n**Low Latency Strategy:**\n\nFor real-time applications:\n\n**Fast Failure Detection:**\n```python\n transport = WebSocketTransport(\n ping_interval=10.0, # Quick detection\n ping_timeout=30.0\n )\n ```\n\n**Dedicated Connections:**\n```python\n# One transport per device (no sharing)\ndevice_transports = {\n device_id: WebSocketTransport()\n for device_id in devices\n}\n```\n\n---\n\n## Transport Extensions\n\n!!!warning \"Future Implementations\"\n AIP's architecture supports multiple transport implementations. The following are planned but not yet implemented.\n\n### HTTP/3 Transport (Planned)\n\n**Benefits:**\n\n- ✅ Multiplexing without head-of-line blocking (QUIC protocol)\n- ✅ 0-RTT connection resumption (faster reconnection)\n- ✅ Better mobile network performance (connection migration)\n- ✅ Built-in encryption (TLS 1.3)\n\n**Use Cases:**\n\n- High-latency networks (satellite, mobile)\n- Frequent reconnections (mobile roaming)\n- Multiple concurrent streams per connection\n\n### gRPC Transport (Planned)\n\n**Benefits:**\n\n- ✅ Strong typing with Protocol Buffers\n- ✅ Built-in load balancing\n- ✅ Bidirectional streaming RPCs\n- ✅ Code generation for multiple languages\n\n**Use Cases:**\n\n- Cross-language interoperability\n- Microservices communication\n- Performance-critical paths\n\n### Custom Transport Implementation\n\nImplement custom transports for specialized protocols:\n\n```python\nfrom aip.transport.base import Transport\n\nclass CustomTransport(Transport):\n async def connect(self, url: str, **kwargs) -> None:\n # Custom connection logic\n self._connection = await custom_protocol.connect(url)\n \n async def send(self, data: bytes) -> None:\n await self._connection.write(data)\n \n async def receive(self) -> bytes:\n return await self._connection.read()\n \n async def close(self) -> None:\n await self._connection.shutdown()\n \n @property\n def is_connected(self) -> bool:\n return self._connection is not None and self._connection.is_open\n```\n\n**Integration:**\n\nCustom transports can be used directly with protocols:\n\n```python\nfrom aip.protocol import AIPProtocol\n\n# Use custom transport with protocol\ntransport = CustomTransport()\nawait transport.connect(\"custom://server:port\")\n\nprotocol = AIPProtocol(transport)\nawait protocol.send_message(message)\n```\n\n[→ See Transport interface specification above](#transport-interface)\n[→ See Protocol usage examples](./protocols.md)\n\n---\n\n## Best Practices\n\n### Environment-Specific Configuration\n\nAdapt transport settings to your deployment environment's characteristics.\n\n| Environment | ping_interval | ping_timeout | max_size | close_timeout |\n|-------------|--------------|--------------|----------|---------------|\n| **Local Network** | 10-20s | 30-60s | 100MB | 5s |\n| **Internet** | 30-60s | 120-180s | 100MB | 10s |\n| **Unreliable Network** | 60-120s | 180-300s | 50MB | 15s |\n| **Mobile** | 60s | 180s | 10MB | 10s |\n\n**Local Network Example:**\n\n```python\ntransport = WebSocketTransport(\n ping_interval=15.0, # Quick failure detection\n ping_timeout=45.0,\n close_timeout=5.0\n)\n```\n\n**Internet Example:**\n\n```python\ntransport = WebSocketTransport(\n ping_interval=30.0, # Balance overhead and detection\n ping_timeout=180.0,\n close_timeout=10.0\n)\n```\n\n**Mobile Network Example:**\n\n```python\ntransport = WebSocketTransport(\n ping_interval=60.0, # Reduce battery usage\n ping_timeout=180.0,\n max_size=10 * 1024 * 1024 # 10MB for mobile\n)\n```\n\n### Connection Health Monitoring\n\nAlways verify connection status before critical operations:\n\n```python\n# Check before sending\nif not transport.is_connected:\n logger.warning(\"Transport not connected, attempting reconnection\")\n await reconnect_transport()\n\n# Proceed with send\nawait transport.send(data)\n```\n\n### Resilience Integration\n\nTransport alone provides low-level communication. Combine with resilience components for production readiness:\n\n```python\nfrom aip.resilience import ReconnectionStrategy\n\nstrategy = ReconnectionStrategy(max_retries=5)\n\ntry:\n await transport.send(data)\nexcept ConnectionError:\n # Trigger reconnection\n await strategy.handle_disconnection(endpoint, device_id)\n```\n\n[→ See Resilience documentation](./resilience.md)\n[→ See HeartbeatManager for connection health monitoring](./resilience.md#heartbeat-manager)\n\n### Logging and Observability\n\n```python\nimport logging\n\n# Enable transport debug logs\nlogging.getLogger(\"aip.transport\").setLevel(logging.DEBUG)\n\n# Custom transport event logging\nclass LoggedTransport(WebSocketTransport):\n async def send(self, data: bytes) -> None:\n logger.debug(f\"Sending {len(data)} bytes\")\n await super().send(data)\n \n async def receive(self) -> bytes:\n data = await super().receive()\n logger.debug(f\"Received {len(data)} bytes\")\n return data\n```\n\n### Resource Cleanup\n\n!!!danger \"Prevent Resource Leaks\"\n Always close transports to prevent socket/memory leaks:\n\n**Context Manager Pattern (Recommended):**\n\n```python\nasync with WebSocketTransport() as transport:\n await transport.connect(\"ws://localhost:8000/ws\")\n await transport.send(data)\n # Automatic cleanup on exit\n```\n\n**Try-Finally Pattern:**\n\n```python\ntransport = WebSocketTransport()\ntry:\n await transport.connect(\"ws://localhost:8000/ws\")\n await transport.send(data)\nfinally:\n await transport.close()\n```\n\n---\n\n## Quick Reference\n\n### Import Transport Components\n\n```python\nfrom aip.transport import (\n Transport, # Abstract base class\n WebSocketTransport, # WebSocket implementation\n TransportState, # Connection states enum\n)\n```\n\n### Common Patterns\n\n| Pattern | Code |\n|---------|------|\n| **Create transport** | `transport = WebSocketTransport()` |\n| **Connect** | `await transport.connect(\"ws://host:port/path\")` |\n| **Send** | `await transport.send(data.encode('utf-8'))` |\n| **Receive** | `data = await transport.receive()` |\n| **Check status** | `if transport.is_connected: ...` |\n| **Close** | `await transport.close()` |\n\n### Related Documentation\n\n- [Protocol Reference](./protocols.md) - How protocols use transports\n- [Resilience](./resilience.md) - Connection management and reconnection\n- [Endpoints](./endpoints.md) - Transport usage in endpoints\n- [Messages](./messages.md) - Message encoding/decoding\n"
},
{
"path": "documents/docs/choose_path.md",
"content": "# Choosing Your Path: UFO² or UFO³ Galaxy?\n\nNot sure which UFO framework to use? This guide will help you make the right choice based on your specific needs.\n\n---\n\n## 🗺️ Quick Decision Tree\n\nUse this interactive flowchart to find the best solution for your use case:\n\n\n```mermaid\ngraph TD\n Start[What are you trying to automate?] --> Q1{Involves multiple devices/platforms?}\n \n Q1 -->|Yes| Q2{Need parallel execution across devices?}\n Q1 -->|No| Q3{Complex multi-app workflow on Windows?}\n \n Q2 -->|Yes| Galaxy[✨ Use UFO³ Galaxy]\n Q2 -->|No, sequential| Q4{Can tasks run independently?}\n \n Q4 -->|Yes, independent| UFO2_Multi[Use UFO² on each device separately]\n Q4 -->|No, dependencies| Galaxy\n \n Q3 -->|Yes| UFO2[🪟 Use UFO²]\n Q3 -->|No, simple task| UFO2\n \n Q3 -->|Might scale later| Hybrid[Use UFO² now, Galaxy-ready setup]\n \n Galaxy --> GalaxyDoc[📖 See Galaxy Quick Start]\n UFO2 --> UFO2Doc[📖 See UFO² Quick Start]\n UFO2_Multi --> UFO2Doc\n Hybrid --> MigrationDoc[📖 See Migration Guide]\n \n style Galaxy fill:#fff9c4\n style UFO2 fill:#c8e6c9\n style UFO2_Multi fill:#c8e6c9\n style Hybrid fill:#e1bee7\n \n click GalaxyDoc \"./getting_started/quick_start_galaxy.md\"\n click UFO2Doc \"./getting_started/quick_start_ufo2.md\"\n click MigrationDoc \"./getting_started/migration_ufo2_to_galaxy.md\"\n```\n\n---\n\n## 📊 Quick Comparison Matrix\n\n| Dimension | UFO² Desktop AgentOS | UFO³ Galaxy |\n|-----------|---------------------|-------------|\n| **Target Scope** | Single Windows desktop | Multiple devices (Windows/Linux/macOS) |\n| **Best For** | Simple local automation | Complex cross-device workflows |\n| **Setup Complexity** | ⭐ Simple | ⭐⭐⭐ Moderate (requires device pool) |\n| **Learning Curve** | ⭐⭐ Easy | ⭐⭐⭐⭐ Advanced |\n| **Execution Model** | Sequential multi-app | Parallel DAG orchestration |\n| **Network Required** | ❌ No | ✅ Yes (WebSocket between devices) |\n| **Parallelism** | Within single device | Across multiple devices |\n| **Fault Tolerance** | Retry on same device | Retry + task migration |\n| **Typical Latency** | 10-30s (local) | 20-60s (includes orchestration) |\n| **Ideal Task Count** | 1-5 steps | 5-20+ steps with dependencies |\n\n**Quick Rule of Thumb:**\n- **1 device + simple workflow** → UFO²\n- **2+ devices OR complex dependencies** → Galaxy\n- **Not sure?** → Start with UFO², migrate later ([Migration Guide](./getting_started/migration_ufo2_to_galaxy.md))\n\n---\n\n## 🎯 Scenario-Based Recommendations\n\n### Scenario 1: Desktop Productivity Automation\n\n**Task:** \"Create a weekly report: extract data from Excel, generate charts in PowerPoint, send via Outlook\"\n\n**Recommendation:** ✅ **UFO²**\n\n**Why:**\n- All applications on one Windows desktop\n- Sequential workflow (Excel → PowerPoint → Outlook)\n- No cross-device dependencies\n\n**Learn More:** [UFO² Overview](./ufo2/overview.md)\n\n---\n\n### Scenario 2: Development Workflow Automation\n\n**Task:** \"Clone repo on my laptop, build Docker image on GPU server, run tests on CI cluster, open results on my desktop\"\n\n**Recommendation:** ✅ **UFO³ Galaxy**\n\n**Why:**\n- Spans 3+ devices (laptop, GPU server, CI cluster, desktop)\n- Sequential dependencies (clone → build → test → display)\n- Requires device coordination and data transfer\n\n**Learn More:** [Galaxy Overview](./galaxy/overview.md)\n\n---\n\n### Scenario 3: Batch Data Processing\n\n**Task:** \"Process 100 files: fetch from cloud, clean data, run ML model, save results\"\n\n**Recommendation:** **Depends on setup**\n\n| Setup | Recommendation | Why |\n|-------|---------------|-----|\n| **Single powerful workstation** | ✅ UFO² | All processing on one machine, simpler |\n| **Distributed cluster** | ✅ Galaxy | Parallel processing across nodes, faster |\n| **Mix (local + cloud GPU)** | ✅ Galaxy | Heterogeneous resources |\n\n**Learn More:** \n- [UFO² for Single Device](./getting_started/quick_start_ufo2.md)\n- [Galaxy for Distributed](./getting_started/quick_start_galaxy.md)\n\n---\n\n### Scenario 4: Cross-Platform Testing\n\n**Task:** \"Test web app on Windows Chrome, Linux Firefox, and macOS Safari\"\n\n**Recommendation:** ✅ **UFO³ Galaxy**\n\n**Why:**\n- Requires 3 different OS platforms\n- Parallel execution saves time\n- Centralized result aggregation\n\n**Learn More:** [Galaxy Multi-Platform Support](./galaxy/overview.md#cross-device-collaboration)\n\n---\n\n### Scenario 5: File Management & Organization\n\n**Task:** \"Organize Downloads folder by file type, compress old files, upload to cloud\"\n\n**Recommendation:** ✅ **UFO²**\n\n**Why:**\n- Single-device local file operations\n- No network dependencies\n- Simple sequential workflow\n\n**Learn More:** [UFO² Quick Start](./getting_started/quick_start_ufo2.md)\n\n---\n\n### Scenario 6: Multi-Stage Data Pipeline\n\n**Task:** \"Collect logs from 5 Linux servers, aggregate on central server, analyze, generate dashboard on Windows\"\n\n**Recommendation:** ✅ **UFO³ Galaxy**\n\n**Why:**\n- Multiple source devices (5 Linux servers)\n- Parallel log collection (5x faster than sequential)\n- Cross-platform (Linux → Windows)\n- Complex dependency graph\n\n**Learn More:** [Galaxy Task Constellation](./galaxy/constellation/overview.md)\n\n---\n\n### Scenario 7: Learning Agent Development\n\n**Task:** \"I'm new to agent development and want to learn by building simple automation\"\n\n**Recommendation:** ✅ **UFO²**\n\n**Why:**\n- Simpler architecture (easier to understand)\n- Faster feedback loop (local execution)\n- Comprehensive documentation and examples\n- Can upgrade to Galaxy later\n\n**Learn More:** [UFO² Quick Start](./getting_started/quick_start_ufo2.md)\n\n---\n\n### Scenario 8: Enterprise Workflow Integration\n\n**Task:** \"Integrate with existing CI/CD pipeline across dev laptops, build servers, and test farms\"\n\n**Recommendation:** ✅ **UFO³ Galaxy**\n\n**Why:**\n- Enterprise-scale device coordination\n- Fault tolerance with automatic recovery\n- Formal safety guarantees for correctness\n- Supports heterogeneous infrastructure\n\n**Learn More:** [Galaxy Architecture](./galaxy/overview.md#architecture)\n\n---\n\n## 🔀 Hybrid Approaches\n\nYou don't have to choose just one! Here are common hybrid patterns:\n\n### Pattern 1: UFO² as Galaxy Device\n\n**Setup:** Run UFO² as a Galaxy device (requires both server and client)\n\n```bash\n# Terminal 1: Start UFO² Server on Windows desktop\npython -m ufo.server.app --port 5000\n\n# Terminal 2: Start UFO² Client (connect to server)\npython -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id my_windows_device --platform windows\n```\n\n**Benefits:**\n- Keep UFO² for local Windows expertise\n- Gain Galaxy's cross-device orchestration\n- Best of both worlds\n\n**Learn More:** [UFO² as Galaxy Device](./ufo2/as_galaxy_device.md)\n\n---\n\n### Pattern 2: Gradual Migration\n\n**Strategy:** Start with UFO² for immediate needs, prepare for Galaxy expansion\n\n**Phase 1:** Use UFO² standalone\n```bash\npython -m ufo --task \"Your current task\"\n```\n\n**Phase 2:** Make UFO² Galaxy-compatible\n```yaml\n# config/galaxy/devices.yaml (prepare in advance)\ndevices:\n - device_id: \"my_windows\"\n server_url: \"ws://localhost:5000/ws\" # Where UFO client connects to UFO server\n os: \"windows\"\n capabilities: [\"office\", \"web\"]\n```\n\n**Phase 3:** Start UFO device agent and connect to Galaxy\n```bash\n# Terminal 1: Start UFO Server on your Windows machine\npython -m ufo.server.app --port 5000\n\n# Terminal 2: Start UFO Client (connects to UFO server above)\npython -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id my_windows --platform windows\n\n# Terminal 3: Start Galaxy (on control machine, can be same or different)\npython -m galaxy --request \"Cross-device workflow\"\n```\n\n**Learn More:** [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md)\n\n---\n\n### Pattern 3: Domain-Specific Split\n\n**Strategy:** Use different frameworks for different workflow types\n\n| Workflow Type | Framework | Example |\n|--------------|-----------|---------|\n| **Daily desktop tasks** | UFO² | Email processing, document creation |\n| **Development workflows** | Galaxy | Code build → test → deploy |\n| **Data processing** | Galaxy (if distributed) | Multi-node ML training |\n| **Quick automation** | UFO² | One-off tasks |\n\n**Learn More:** [When to Use Which](./getting_started/migration_ufo2_to_galaxy.md#when-to-use-which)\n\n---\n\n## 🚫 Common Misconceptions\n\n### Misconception 1: \"Galaxy is always better because it's newer\"\n\n**Reality:** UFO² is better for simple single-device tasks due to:\n- Lower latency (no network overhead)\n- Simpler setup and debugging\n- Battle-tested stability\n\n**Use Galaxy only when you actually need multi-device orchestration.**\n\n---\n\n### Misconception 2: \"I need to rewrite everything to migrate to Galaxy\"\n\n**Reality:** UFO² can run as a Galaxy device with minimal changes:\n```bash\n# Terminal 1: Start UFO Server\npython -m ufo.server.app --port 5000\n\n# Terminal 2: Start UFO Client in WebSocket mode\npython -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id my_device --platform windows\n```\n\n**Learn More:** [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md#option-2-convert-ufo2-instance-to-galaxy-device)\n\n---\n\n### Misconception 3: \"Galaxy can't run on a single device\"\n\n**Reality:** Galaxy works perfectly on one device if you need:\n- DAG-based workflow planning\n- Advanced monitoring and trajectory reports\n- Preparation for future multi-device expansion\n\n```yaml\n# Single-device Galaxy setup\ndevices:\n - device_id: \"localhost\"\n server_url: \"ws://localhost:5005/ws\"\n```\n\n---\n\n### Misconception 4: \"UFO² is deprecated in favor of Galaxy\"\n\n**Reality:** UFO² is actively maintained and recommended for single-device use:\n- More efficient for local tasks\n- Simpler for beginners\n- Core component when used as Galaxy device\n\n**Both frameworks are complementary, not competing.**\n\n---\n\n## 🎓 Learning Paths\n\n### For Beginners\n\n**Week 1-2: Start with UFO²**\n1. [UFO² Quick Start](./getting_started/quick_start_ufo2.md)\n2. Build simple automation (file management, email, etc.)\n3. Understand HostAgent/AppAgent architecture\n\n**Week 3-4: Explore Advanced UFO²**\n4. [Hybrid GUI-API Actions](./ufo2/core_features/hybrid_actions.md)\n5. [MCP Server Integration](./mcp/overview.md)\n6. [Customization & Learning](./ufo2/advanced_usage/customization.md)\n\n**Week 5+: Graduate to Galaxy (if needed)**\n7. [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md)\n8. [Galaxy Quick Start](./getting_started/quick_start_galaxy.md)\n9. Build cross-device workflows\n\n---\n\n### For Experienced Developers\n\n**Direct to Galaxy** if you already know you need multi-device:\n1. [Galaxy Quick Start](./getting_started/quick_start_galaxy.md)\n2. [Task Constellation Concepts](./galaxy/constellation/overview.md)\n3. [ConstellationAgent Deep Dive](./galaxy/constellation_agent/overview.md)\n4. [Performance Monitoring](./galaxy/evaluation/performance_metrics.md)\n\n---\n\n## 📋 Decision Checklist\n\nStill unsure? Answer these questions:\n\n**Q1: Does your workflow involve 2+ physical devices?**\n\n- ✅ Yes → **Galaxy**\n- ❌ No → Continue to Q2\n\n**Q2: Do you need parallel execution across different machines?**\n\n- ✅ Yes → **Galaxy**\n- ❌ No → Continue to Q3\n\n**Q3: Does your workflow have complex dependencies (DAG structure)?**\n\n- ✅ Yes, complex DAG → **Galaxy**\n- ❌ No, simple sequence → Continue to Q4\n\n**Q4: Are you comfortable with distributed systems concepts?**\n\n- ✅ Yes → **Galaxy** (if any of Q1-Q3 is yes)\n- ❌ No → **UFO²** (learn basics first)\n\n**Q5: Do you need cross-platform support (Windows + Linux)?**\n\n- ✅ Yes → **Galaxy**\n- ❌ No, Windows only → **UFO²**\n\n---\n\n**Result:**\n\n- **3+ \"Galaxy\" answers** → Use Galaxy ([Quick Start](./getting_started/quick_start_galaxy.md))\n- **Mostly \"UFO²\" answers** → Use UFO² ([Quick Start](./getting_started/quick_start_ufo2.md))\n- **Mixed answers** → Start with UFO², keep Galaxy option open ([Migration Guide](./getting_started/migration_ufo2_to_galaxy.md))\n\n---\n\n## 🔗 Next Steps\n\n### If you chose UFO²:\n1. 📖 [UFO² Quick Start Guide](./getting_started/quick_start_ufo2.md)\n2. 🎯 [UFO² Overview & Architecture](./ufo2/overview.md)\n3. 🛠️ [Configuration Guide](./configuration/system/overview.md)\n\n### If you chose Galaxy:\n1. 📖 [Galaxy Quick Start Guide](./getting_started/quick_start_galaxy.md)\n2. 🎯 [Galaxy Overview & Architecture](./galaxy/overview.md)\n3. 🌟 [Task Constellation Concepts](./galaxy/constellation/overview.md)\n\n### If you're still exploring:\n1. 📊 [Detailed Comparison](./getting_started/migration_ufo2_to_galaxy.md#when-to-use-which)\n2. 🎬 [Demo Video](https://www.youtube.com/watch?v=QT_OhygMVXU)\n3. 📄 [Research Paper](https://arxiv.org/abs/2504.14603)\n\n---\n\n## 💡 Pro Tips\n\n!!! tip \"Start Simple\"\n When in doubt, start with **UFO²**. It's easier to scale up to Galaxy later than to debug a complex Galaxy setup when you don't need it.\n\n!!! tip \"Hybrid is Valid\"\n Don't feel locked into one choice. You can use **UFO² for local tasks** and **Galaxy for cross-device workflows** simultaneously.\n\n!!! tip \"Test Before Committing\"\n Try both for a simple workflow to see which feels more natural for your use case:\n ```bash\n # UFO² test\n python -m ufo --task \"Create test report\"\n \n # Galaxy test \n python -m galaxy --request \"Create test report\"\n ```\n\n!!! warning \"Network Requirements\"\n Galaxy requires **stable network connectivity** between devices. If your environment has network restrictions, UFO² might be more reliable.\n\n---\n\n## 🤝 Getting Help\n\n- **Documentation:** [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)\n- **GitHub Issues:** [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues)\n- **Discussions:** [https://github.com/microsoft/UFO/discussions](https://github.com/microsoft/UFO/discussions)\n\nStill have questions? Check the [Migration FAQ](./getting_started/migration_ufo2_to_galaxy.md#getting-help) or open a discussion on GitHub!\n"
},
{
"path": "documents/docs/client/computer.md",
"content": "# Computer\n\nThe **Computer** class is the core execution layer of the UFO client. It manages MCP (Model Context Protocol) tool execution, maintains tool registries, and provides thread-isolated execution for reliability. Each Computer instance represents a distinct execution context with its own namespace and resource management.\n\n## Architecture Overview\n\nThe Computer layer provides the execution engine for MCP tools with three main components:\n\n```mermaid\ngraph TB\n CommandRouter[\"CommandRouter Command Routing\"]\n ComputerManager[\"ComputerManager Instance Management\"]\n Computer[\"Computer Core Execution Layer\"]\n MCPServerManager[\"MCP Server Manager Process Isolation\"]\n \n CommandRouter -->|Routes To| ComputerManager\n ComputerManager -->|Creates & Manages| Computer\n Computer -->|Data Collection| DataServers[\"Data Collection Servers screenshot, ui_detection, etc.\"]\n Computer -->|Actions| ActionServers[\"Action Servers gui_automation, file_operations, etc.\"]\n Computer -->|Uses| ToolsRegistry[\"Tools Registry tool_type::tool_name → MCPToolCall\"]\n Computer -->|Provides| MetaTools[\"Meta Tools list_tools built-in introspection\"]\n Computer -->|Delegates To| MCPServerManager\n```\n\n**Computer** manages MCP tool execution with thread isolation and timeout control (6000-second timeout, 10-worker thread pool). \n**ComputerManager** handles multiple Computer instances with namespace-based routing. \n**CommandRouter** routes and executes commands across Computer instances with early-exit support.\n\n### Key Responsibilities\n\n- **Tool Registration**: Register tools from multiple MCP servers with namespace isolation\n- **Command Routing**: Convert high-level commands to MCP tool calls\n- **Execution Management**: Execute tools in isolated thread pools with timeout protection\n- **Meta Tools**: Provide introspection capabilities (e.g., `list_tools`)\n\n## Table of Contents\n\n## Core Components\n\n### 1. Computer Class\n\nThe `Computer` class manages a single logical computer with its own set of MCP servers and tools.\n\n#### Key Attributes\n\n| Attribute | Type | Description |\n|-----------|------|-------------|\n| `_name` | `str` | Unique identifier for the computer instance |\n| `_process_name` | `str` | Associated process name for MCP server isolation |\n| `_data_collection_servers` | `Dict[str, BaseMCPServer]` | Servers for data collection (screenshot, UI detection, etc.) |\n| `_action_servers` | `Dict[str, BaseMCPServer]` | Servers for actions (GUI automation, file operations, etc.) |\n| `_tools_registry` | `Dict[str, MCPToolCall]` | Registry of all available tools (key: `tool_type::tool_name`) |\n| `_meta_tools` | `Dict[str, Callable]` | Built-in introspection tools |\n| `_executor` | `ThreadPoolExecutor` | Thread pool for isolated tool execution (10 workers) |\n| `_tool_timeout` | `int` | Tool execution timeout (6000 seconds = 100 minutes) |\n\n#### Tool Namespaces\n\nComputer supports two types of tool namespaces:\n\n- **`data_collection`**: Tools for gathering information (non-destructive operations)\n- **`action`**: Tools for performing actions (state-changing operations)\n\n```python\n# Tool key format: \"tool_type::tool_name\"\n\"data_collection::screenshot\" # Take screenshot\n\"data_collection::ui_detection\" # Detect UI elements\n\"action::click\" # Click UI element\n\"action::type_text\" # Type text\n```\n\n> **Note:** Different namespaces allow the same tool name to exist in both data collection and action contexts. For example, both `data_collection::get_file_info` and `action::get_file_info` can coexist.\n\n### 2. ComputerManager Class\n\nThe `ComputerManager` creates and manages multiple `Computer` instances based on agent configurations.\n\n#### Computer Instance Key\n\nEach computer instance is identified by a unique key:\n\n```python\nkey = f\"{agent_name}::{process_name}::{root_name}\"\n```\n\n**Example:**\n```python\n\"host_agent::chrome::default\" # Default chrome computer for host_agent\n\"host_agent::vscode::custom_config\" # Custom VSCode computer for host_agent\n```\n\n#### Configuration Structure\n\n```yaml\nmcp:\n host_agent:\n default:\n data_collection:\n - namespace: \"screenshot\"\n server_type: \"local\"\n module: \"ufo.client.mcp.local_servers.screenshot\"\n reset: false\n - namespace: \"ui_detection\"\n server_type: \"local\"\n module: \"ufo.client.mcp.local_servers.ui_detection\"\n reset: false\n action:\n - namespace: \"gui_automation\"\n server_type: \"local\"\n module: \"ufo.client.mcp.local_servers.gui_automation\"\n reset: false\n```\n\n**Configuration Requirements**\n\n- Each agent must have at least a `default` root configuration\n- If `root_name` is not found, the manager falls back to `default`\n- Missing configurations will raise a `ValueError`\n\n### 3. CommandRouter Class\n\nThe `CommandRouter` executes commands on the appropriate `Computer` instance by routing through the `ComputerManager`.\n\n#### Execution Flow\n\n```mermaid\ngraph LR\n Command --> CommandRouter\n CommandRouter --> ComputerManager\n ComputerManager -->|get_or_create| Computer\n Computer -->|command2tool| ToolCall[MCPToolCall]\n ToolCall -->|run_actions| Result[MCP Tool Result]\n```\n\n## Initialization\n\n### Computer Initialization\n\n```python\nfrom ufo.client.computer import Computer\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\n# Create MCP server manager\nmcp_manager = MCPServerManager()\n\n# Initialize computer\ncomputer = Computer(\n name=\"my_computer\",\n process_name=\"my_process\",\n mcp_server_manager=mcp_manager,\n data_collection_servers_config=[\n {\n \"namespace\": \"screenshot\",\n \"server_type\": \"local\",\n \"module\": \"ufo.client.mcp.local_servers.screenshot\"\n }\n ],\n action_servers_config=[\n {\n \"namespace\": \"gui_automation\",\n \"server_type\": \"local\",\n \"module\": \"ufo.client.mcp.local_servers.gui_automation\"\n }\n ]\n)\n\n# Async initialization (required)\nawait computer.async_init()\n```\n\n> **⚠️ Important:** You **must** call `await computer.async_init()` after creating a `Computer` instance. This registers all MCP servers and their tools asynchronously.\n\n### ComputerManager Initialization\n\n```python\nfrom ufo.client.computer import ComputerManager\n\n# Load configuration\nwith open(\"config.yaml\") as f:\n configs = yaml.safe_load(f)\n\n# Create manager\nmanager = ComputerManager(\n configs=configs,\n mcp_server_manager=mcp_manager\n)\n\n# Get or create computer instance\ncomputer = await manager.get_or_create(\n agent_name=\"host_agent\",\n process_name=\"chrome\",\n root_name=\"default\"\n)\n```\n\n## Tool Execution\n\n### Basic Tool Execution\n\n```python\nfrom aip.messages import MCPToolCall\n\n# Create tool call\ntool_call = MCPToolCall(\n tool_key=\"data_collection::screenshot\",\n tool_name=\"screenshot\",\n parameters={\"region\": \"full_screen\"}\n)\n\n# Execute tool\nresults = await computer.run_actions([tool_call])\n\n# Check result\nif results[0].is_error:\n print(f\"Error: {results[0].content}\")\nelse:\n print(f\"Success: {results[0].data}\")\n```\n\n### Command to Tool Conversion\n\nThe `command2tool()` method converts high-level `Command` objects to `MCPToolCall` objects:\n\n```python\nfrom aip.messages import Command\n\n# Create command\ncommand = Command(\n tool_name=\"screenshot\",\n tool_type=\"data_collection\",\n parameters={\"region\": \"active_window\"}\n)\n\n# Convert to tool call\ntool_call = computer.command2tool(command)\n\n# Execute\nresults = await computer.run_actions([tool_call])\n```\n\nIf `tool_type` is not specified in the command, the `command2tool()` method will automatically detect whether the tool is registered as `data_collection` or `action`.\n\n### Batch Tool Execution\n\n```python\n# Execute multiple tools sequentially\ntool_calls = [\n MCPToolCall(tool_key=\"data_collection::screenshot\", tool_name=\"screenshot\"),\n MCPToolCall(tool_key=\"data_collection::ui_detection\", tool_name=\"detect_ui\"),\n MCPToolCall(tool_key=\"action::click\", tool_name=\"click\", parameters={\"x\": 100, \"y\": 200})\n]\n\nresults = await computer.run_actions(tool_calls)\n\nfor i, result in enumerate(results):\n print(f\"Tool {i}: {'Success' if not result.is_error else 'Failed'}\")\n```\n\n## Thread Isolation & Timeout\n\n### Why Thread Isolation?\n\nMCP tools may contain **blocking operations** (e.g., `time.sleep()`, synchronous I/O) that can block the event loop and cause WebSocket disconnections. To prevent this:\n\n1. Each tool call runs in a **separate thread** with its own event loop\n2. The thread pool has **10 concurrent workers**\n3. Each tool call has a **timeout of 6000 seconds** (100 minutes)\n\n### Implementation Details\n\n```python\ndef _call_tool_in_thread():\n \"\"\"Execute MCP tool call in isolated thread with its own event loop.\"\"\"\n loop = asyncio.new_event_loop()\n asyncio.set_event_loop(loop)\n try:\n async def _do_call():\n async with Client(server) as client:\n return await client.call_tool(\n name=tool_name,\n arguments=params,\n raise_on_error=False\n )\n return loop.run_until_complete(_do_call())\n finally:\n loop.close()\n\n# Execute in thread pool with timeout\nresult = await asyncio.wait_for(\n loop.run_in_executor(self._executor, _call_tool_in_thread),\n timeout=self._tool_timeout\n)\n```\n\nIf a tool execution exceeds 6000 seconds, it will be cancelled and return a timeout error:\n\n```python\nCallToolResult(\n is_error=True,\n content=[TextContent(text=\"Tool execution timed out after 6000s\")]\n)\n```\n\n## Meta Tools\n\nMeta tools are **built-in introspection tools** that provide information about the computer's capabilities.\n\n### Registering Meta Tools\n\nUse the `@Computer.meta_tool()` decorator to register a method as a meta tool:\n\n```python\nclass Computer:\n @meta_tool(\"list_tools\")\n async def list_tools(\n self,\n tool_type: Optional[str] = None,\n namespace: Optional[str] = None,\n remove_meta: bool = True\n ) -> CallToolResult:\n \"\"\"List all available tools.\"\"\"\n # Implementation...\n```\n\n### Using Meta Tools\n\n```python\n# List all action tools\ntool_call = MCPToolCall(\n tool_key=\"action::list_tools\",\n tool_name=\"list_tools\",\n parameters={\"tool_type\": \"action\"}\n)\n\nresult = await computer.run_actions([tool_call])\ntools = result[0].data # List of available action tools\n```\n\n**Example:**\n\n```python\n# List all tools in \"screenshot\" namespace\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::list_tools\",\n tool_name=\"list_tools\",\n parameters={\"namespace\": \"screenshot\", \"remove_meta\": True}\n )\n])\n\n# Returns: [{\"tool_name\": \"take_screenshot\", \"description\": \"...\", ...}]\n```\n\n## Dynamic Server Management\n\n### Adding a Server\n\n```python\nfrom ufo.client.mcp.mcp_server_manager import BaseMCPServer\n\n# Create new MCP server\nnew_server = mcp_manager.create_or_get_server(\n mcp_config={\n \"namespace\": \"custom_tools\",\n \"server_type\": \"local\",\n \"module\": \"my_custom_mcp_server\"\n },\n reset=False,\n process_name=\"my_process\"\n)\n\n# Add to computer\nawait computer.add_server(\n namespace=\"custom_tools\",\n mcp_server=new_server,\n tool_type=\"action\"\n)\n```\n\n### Removing a Server\n\n```python\n# Remove server and all its tools\nawait computer.delete_server(\n namespace=\"custom_tools\",\n tool_type=\"action\"\n)\n```\n\n**Use cases for dynamic server management:**\n\n- Add specialized tools for specific tasks\n- Remove servers to reduce memory footprint\n- Hot-reload MCP servers during development\n\n## Command Routing\n\nThe `CommandRouter` orchestrates command execution across multiple computers.\n\n### Basic Usage\n\n```python\nfrom ufo.client.computer import CommandRouter\nfrom aip.messages import Command, Result\n\n# Create router\nrouter = CommandRouter(computer_manager=manager)\n\n# Execute commands\ncommands = [\n Command(tool_name=\"screenshot\", tool_type=\"data_collection\"),\n Command(tool_name=\"click\", tool_type=\"action\", parameters={\"x\": 100, \"y\": 200})\n]\n\nresults = await router.execute(\n agent_name=\"host_agent\",\n process_name=\"chrome\",\n root_name=\"default\",\n commands=commands,\n early_exit=True # Stop on first error\n)\n\nfor result in results:\n print(f\"Status: {result.status}\")\n print(f\"Data: {result.data}\")\n```\n\n### Error Handling\n\n```python\n# early_exit=True: Stop on first error\nresults = await router.execute(\n agent_name=\"host_agent\",\n process_name=\"chrome\",\n root_name=\"default\",\n commands=commands,\n early_exit=True\n)\n\n# early_exit=False: Execute all commands even if some fail\nresults = await router.execute(\n agent_name=\"host_agent\",\n process_name=\"chrome\",\n root_name=\"default\",\n commands=commands,\n early_exit=False\n)\n```\n\n> **⚠️ Warning:** When `early_exit=True`, if a command fails, subsequent commands will **not** be executed, and their results will be set to `ResultStatus.SKIPPED`.\n\n## Tool Registry\n\nThe tools registry maintains a mapping of all available tools.\n\n### Tool Key Format\n\n```python\ntool_key = f\"{tool_type}::{tool_name}\"\n\n# Examples:\n\"data_collection::screenshot\"\n\"action::click\"\n\"data_collection::list_tools\" # Meta tool\n```\n\n### Accessing Tools\n\n```python\n# Get tool info\ntool_info = computer._tools_registry.get(\"action::click\")\n\n# Tool info contains:\nprint(tool_info.tool_name) # \"click\"\nprint(tool_info.tool_type) # \"action\"\nprint(tool_info.namespace) # e.g., \"gui_automation\"\nprint(tool_info.description) # Tool description\nprint(tool_info.input_schema) # JSON schema for input parameters\nprint(tool_info.mcp_server) # Reference to MCP server\n```\n\n## Best Practices\n\n### Configuration\n\n1. **Use namespaces wisely**: Group related tools under meaningful namespaces\n2. **Separate concerns**: Use `data_collection` for read-only operations, `action` for state changes\n3. **Configure timeouts**: Adjust `_tool_timeout` for long-running operations\n4. **Use default root**: Always provide a `default` root configuration as fallback\n\n### Performance Optimization\n\n1. **Register servers in parallel**: The `async_init()` method already does this via `asyncio.gather()`\n2. **Reuse Computer instances**: Let `ComputerManager` cache instances rather than creating new ones\n3. **Limit concurrent tools**: The thread pool has 10 workers; excessive parallel tools may queue\n4. **Reset servers carefully**: Setting `reset=True` in server config will restart the MCP server process\n\n### Common Pitfalls\n\n> **⚠️ Important:** Avoid these common mistakes:\n> - **Forgetting `async_init()`**: Always call after creating a `Computer` instance\n> - **Tool key collisions**: Ensure tool names are unique within each `tool_type`\n> - **Timeout too short**: Some operations (e.g., file downloads) may need longer timeouts\n> - **Blocking in meta tools**: Meta tools should be fast; avoid I/O operations\n\n## Error Handling\n\n### Tool Execution Errors\n\n```python\ntry:\n results = await computer.run_actions([tool_call])\n if results[0].is_error:\n error_message = results[0].content[0].text\n print(f\"Tool error: {error_message}\")\nexcept ValueError as e:\n print(f\"Tool not registered: {e}\")\nexcept asyncio.TimeoutError:\n print(\"Tool execution timed out\")\nexcept Exception as e:\n print(f\"Unexpected error: {e}\")\n```\n\n### Configuration Errors\n\n```python\ntry:\n computer = await manager.get_or_create(\n agent_name=\"host_agent\",\n process_name=\"chrome\",\n root_name=\"invalid_root\"\n )\nexcept ValueError as e:\n print(f\"Configuration error: {e}\")\n # Fallback to default\n computer = await manager.get_or_create(\n agent_name=\"host_agent\",\n process_name=\"chrome\",\n root_name=\"default\"\n )\n```\n\n## Integration Points\n\n### With UFO Client\n\nThe `Computer` is created and managed by the `UFOClient`:\n\n```python\n# In UFOClient\nself.command_router = CommandRouter(computer_manager)\n\n# Execute commands from server\nresults = await self.command_router.execute(\n agent_name=self.agent_name,\n process_name=self.process_name,\n root_name=self.root_name,\n commands=command_list\n)\n```\n\n### With MCP Server Manager\n\nThe `Computer` relies on `MCPServerManager` for server lifecycle management:\n\n```python\n# Create or get existing MCP server\nmcp_server = self.mcp_server_manager.create_or_get_server(\n mcp_config=server_config,\n reset=False,\n process_name=self._process_name\n)\n```\n\nSee [MCP Integration](mcp_integration.md) for more details on MCP server management.\n\n## Related Documentation\n\n- [UFO Client Overview](overview.md) - High-level client architecture\n- [UFO Client](ufo_client.md) - Command execution orchestration\n- [Computer Manager](computer_manager.md) - Multi-computer instance management\n- [MCP Integration](mcp_integration.md) - MCP server details\n- [AIP Messages](../aip/messages.md) - Command and Result message formats\n"
},
{
"path": "documents/docs/client/computer_manager.md",
"content": "# Computer Manager & Computer\n\nThe **Computer Manager** orchestrates multiple **Computer** instances, each representing an isolated execution namespace with dedicated MCP servers and tools. This enables context-specific tool routing and fine-grained control over data collection vs. action execution.\n\n---\n\n## Overview\n\nThe Computer layer consists of two components working together:\n\n- **ComputerManager**: High-level orchestrator managing multiple Computer instances\n- **Computer**: Individual execution namespace with its own MCP servers and tool registry\n\n### Computer Manager Responsibilities\n\n| Capability | Description | Implementation |\n|------------|-------------|----------------|\n| **Multi-Computer Management** | Create and manage multiple Computer instances | Per-process, per-agent namespaces |\n| **Namespace Isolation** | Separate tool namespaces for different contexts | Independent MCP servers per Computer |\n| **Command Routing** | Route commands to appropriate Computer instances | CommandRouter resolves by agent/process/root |\n| **MCP Server Configuration** | Configure data collection and action servers | Config-driven server initialization |\n| **Lifecycle Management** | Initialize, reset, and tear down Computers | Async initialization, cascading reset |\n\n### Computer (Instance) Responsibilities\n\n| Capability | Description | Implementation |\n|------------|-------------|----------------|\n| **Tool Registry** | Maintain registry of available MCP tools | `_tools_registry` dict |\n| **Tool Execution** | Execute MCP tool calls with timeout protection | Thread pool isolation (max 10 workers) |\n| **Server Management** | Manage data collection and action MCP servers | Separate namespaces |\n| **Meta Tools** | Provide built-in tools (list_tools, etc.) | Decorated meta tool methods |\n| **Async Initialization** | Initialize MCP servers asynchronously | `async_init()` |\n\n**Architectural Relationship:**\n\n```mermaid\ngraph TB\n subgraph \"Computer Manager Layer\"\n CM[Computer Manager]\n CR[Command Router]\n end\n \n subgraph \"Computer Instances\"\n C1[Computer: default]\n C2[Computer: notepad.exe]\n C3[Computer: explorer.exe]\n end\n \n subgraph \"Computer 1 Components\"\n C1 --> DC1[Data Collection Servers]\n C1 --> AS1[Action Servers]\n C1 --> TR1[Tool Registry]\n C1 --> MT1[Meta Tools]\n end\n \n CM -->|manages| C1\n CM -->|manages| C2\n CM -->|manages| C3\n CR -->|routes to| C1\n CR -->|routes to| C2\n CR -->|routes to| C3\n \n style CM fill:#ffe0b2\n style C1 fill:#bbdefb\n style C2 fill:#bbdefb\n style C3 fill:#bbdefb\n```\n\n---\n\n## 🏗️ Computer Manager Architecture\n\n### Computer Instance Management\n\n```mermaid\ngraph LR\n subgraph \"ComputerManager\"\n Config[UFO Config]\n Registry[Computer Registry]\n end\n \n subgraph \"Computers\"\n Default[default_agent]\n Proc1[notepad.exe]\n Proc2[explorer.exe]\n end\n \n Config -->|creates| Default\n Config -->|creates| Proc1\n Config -->|creates| Proc2\n \n Registry -->|tracks| Default\n Registry -->|tracks| Proc1\n Registry -->|tracks| Proc2\n \n style Config fill:#fff3e0\n style Registry fill:#e1f5fe\n```\n\n**Computer Namespaces:**\n\n| Namespace Type | Purpose | Example |\n|----------------|---------|---------|\n| **Data Collection** | Gathering information, non-invasive queries | Screenshots, UI element detection, app state |\n| **Action** | Performing actions, invasive operations | GUI automation, file operations, app control |\n\nData collection tools are designed for non-invasive information gathering, while action tools have full control for state-changing operations.\n\n---\n\n## Computer Manager Architecture\n\n## 🖥️ Computer (Instance) Architecture\n\n### Internal Structure\n\n```mermaid\ngraph TB\n subgraph \"Computer Instance\"\n Init[Initialization]\n Servers[MCP Servers]\n Registry[Tool Registry]\n Execution[Tool Execution]\n end\n \n subgraph \"MCP Servers\"\n Servers --> DC[Data Collection Servers]\n Servers --> AS[Action Servers]\n end\n \n subgraph \"Tool Registry\"\n Registry --> TR[_tools_registry Dict]\n TR -->|key: action::click| T1[MCPToolCall]\n TR -->|key: data_collection::screenshot| T2[MCPToolCall]\n TR -->|key: action::list_tools| T3[Meta Tool]\n end\n \n subgraph \"Execution Engine\"\n Execution --> TP[Thread Pool Executor]\n Execution --> TO[Timeout Protection]\n TP -->|max 10 workers| Threads[Isolated Threads]\n end\n \n Init --> Servers\n Servers --> Registry\n Registry --> Execution\n \n style Init fill:#c8e6c9\n style Servers fill:#bbdefb\n style Registry fill:#fff9c4\n style Execution fill:#ffccbc\n```\n\n**Key Attributes:**\n\n| Attribute | Type | Purpose |\n|-----------|------|---------|\n| `_name` | `str` | Computer name (identifier) |\n| `_process_name` | `str` | Associated process (e.g., \"notepad.exe\") |\n| `_data_collection_servers` | `Dict[str, BaseMCPServer]` | Namespace → MCP server mapping (data collection) |\n| `_action_servers` | `Dict[str, BaseMCPServer]` | Namespace → MCP server mapping (actions) |\n| `_tools_registry` | `Dict[str, MCPToolCall]` | Tool key → tool info mapping |\n| `_meta_tools` | `Dict[str, Callable]` | Built-in meta tools |\n| `_executor` | `ThreadPoolExecutor` | Thread pool for tool execution (10 workers) |\n| `_tool_timeout` | `int` | Tool execution timeout: **6000 seconds (100 minutes)** |\n\n> **Note:** The tool execution timeout is 6000 seconds (100 minutes), allowing for very long-running operations while preventing indefinite hangs.\n\n---\n\n## Initialization\n\n### Computer Manager Initialization\n\n**Creating Computer Manager:**\n\n```python\nfrom ufo.client.computer import ComputerManager\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\nfrom config.config_loader import get_ufo_config\n\n# 1. Get UFO configuration\nufo_config = get_ufo_config()\n\n# 2. Initialize MCP server manager\nmcp_server_manager = MCPServerManager()\n\n# 3. Create computer manager\ncomputer_manager = ComputerManager(\n ufo_config.to_dict(),\n mcp_server_manager\n)\n```\n\n### Computer Instance Initialization\n\n**Computer Async Initialization:**\n\n```python\ncomputer = Computer(\n name=\"default_agent\",\n process_name=\"explorer.exe\",\n mcp_server_manager=mcp_server_manager,\n data_collection_servers_config=[...],\n action_servers_config=[...]\n)\n\n# Async initialization (required)\nawait computer.async_init()\n```\n\n**Initialization Flow:**\n\n```mermaid\nsequenceDiagram\n participant Code\n participant Computer\n participant MCP as MCP Server Manager\n participant Servers\n \n Code->>Computer: __init__(name, process, configs)\n Computer->>Computer: Create thread pool executor\n Computer->>Computer: Register meta tools\n \n Code->>Computer: async_init()\n Computer->>Computer: _init_data_collection_servers()\n Computer->>MCP: create_or_get_server(config)\n MCP-->>Computer: BaseMCPServer\n \n Computer->>Computer: _init_action_servers()\n Computer->>MCP: create_or_get_server(config)\n MCP-->>Computer: BaseMCPServer\n \n par Register Data Collection Servers\n Computer->>Servers: register_mcp_servers(data_collection)\n and Register Action Servers\n Computer->>Servers: register_mcp_servers(action)\n end\n \n Servers-->>Computer: Tools registered\n```\n\n**Configuration Example:**\n\n```yaml\ndata_collection_servers:\n - namespace: screenshot_collector\n type: local\n module: ufo.client.mcp.local_servers.screenshot_server\n reset: false\n - namespace: ui_collector\n type: local\n module: ufo.client.mcp.local_servers.ui_server\n reset: false\n\naction_servers:\n - namespace: gui_automator\n type: local\n module: ufo.client.mcp.local_servers.automation_server\n reset: false\n```\n\n---\n\n## 🔀 Command Routing\n\n### CommandRouter\n\nThe CommandRouter resolves which Computer instance should handle each command based on agent/process/root context.\n\n**Routing Signature:**\n\n```python\nasync def execute(\n self,\n agent_name: str,\n process_name: str,\n root_name: str,\n commands: List[Command]\n) -> List[Result]\n```\n\n**Routing Logic:**\n\n```mermaid\ngraph TD\n Start[Command List]\n Start --> Resolve[Resolve Computer Instance]\n Resolve -->|agent_name, process_name, root_name| Computer[Get/Create Computer]\n \n Computer --> Loop[For Each Command]\n Loop --> Parse[Parse Command to MCPToolCall]\n Parse --> Lookup[Lookup Tool in Registry]\n \n Lookup -->|Found| Execute[Execute Tool]\n Lookup -->|Not Found| Error[Return Error Result]\n \n Execute --> Timeout[Tool Execution with Timeout]\n Timeout -->|Success| Result[Return Result]\n Timeout -->|Timeout| TimeoutError[Timeout Error Result]\n Timeout -->|Exception| ExecError[Execution Error Result]\n \n Result --> Collect[Collect Results]\n Error --> Collect\n TimeoutError --> Collect\n ExecError --> Collect\n \n Collect --> Return[Return List[Result]]\n \n style Start fill:#e1f5fe\n style Computer fill:#bbdefb\n style Execute fill:#c8e6c9\n style Collect fill:#fff9c4\n```\n\n---\n\n## 🔧 Tool Execution\n\n### Tool Execution Pipeline\n\nMCP tools are executed in isolated threads to prevent blocking operations (like `time.sleep`) from blocking the main event loop and causing WebSocket disconnections.\n\n**Execution Flow:**\n\n```mermaid\nsequenceDiagram\n participant Computer\n participant TP as Thread Pool\n participant Thread\n participant Loop as New Event Loop\n participant MCP as MCP Server\n \n Computer->>Computer: _run_action(tool_call)\n Computer->>Computer: Lookup tool in registry\n \n alt Meta Tool\n Computer->>Computer: Execute meta tool directly\n Computer-->>Computer: Result\n else MCP Tool\n Computer->>TP: Submit _call_tool_in_thread()\n TP->>Thread: Execute in thread\n Thread->>Loop: Create new event loop\n Loop->>MCP: client.call_tool(name, params)\n \n alt Success (within timeout)\n MCP-->>Loop: Result\n Loop-->>Thread: Result\n Thread-->>TP: Result\n TP-->>Computer: CallToolResult\n else Timeout (> 6000s)\n Note over Computer,MCP: Tool execution timeout\n Computer-->>Computer: TimeoutError Result\n else Exception\n Note over Computer,MCP: Tool execution failed\n Computer-->>Computer: Error Result\n end\n end\n```\n\n**Thread Pool Configuration:**\n\n| Parameter | Value | Purpose |\n|-----------|-------|---------|\n| `max_workers` | **10** | Maximum concurrent tool executions |\n| `thread_name_prefix` | `\"mcp_tool_\"` | Thread naming for debugging |\n| Timeout | **6000 seconds (100 minutes)** | Per-tool execution timeout |\n\n**Code Implementation:**\n\n```python\ndef _call_tool_in_thread():\n \"\"\"\n Execute MCP tool call in an isolated thread with its own event loop.\n This prevents blocking operations in MCP tools from blocking the main event loop.\n \"\"\"\n # Create a new event loop for this thread\n loop = asyncio.new_event_loop()\n asyncio.set_event_loop(loop)\n try:\n async def _do_call():\n async with Client(server) as client:\n return await client.call_tool(\n name=tool_name, arguments=params, raise_on_error=False\n )\n return loop.run_until_complete(_do_call())\n finally:\n loop.close()\n\n# Execute in thread pool with timeout protection\nresult = await asyncio.wait_for(\n loop.run_in_executor(self._executor, _call_tool_in_thread),\n timeout=self._tool_timeout\n)\n```\n\n---\n\n## 🛠️ Tool Registry\n\n### Tool Registration\n\nTools are discovered from MCP servers during initialization and registered with unique keys.\n\n**Tool Key Format:**\n\n```\n::\n\nExamples:\n- action::click\n- action::type_text\n- data_collection::screenshot\n- data_collection::get_ui_elements\n```\n\n**Registration Process:**\n\n```python\nasync def register_one_mcp_server(\n self, namespace: str, tool_type: str, mcp_server: BaseMCPServer\n) -> None:\n async with Client(mcp_server.server) as client:\n tools = await client.list_tools()\n \n for tool in tools:\n tool_key = self.make_tool_key(tool_type, tool.name)\n \n self._register_tool(\n tool_key=tool_key,\n tool_name=tool.name,\n title=tool.title,\n namespace=namespace,\n tool_type=tool_type,\n description=tool.description,\n input_schema=tool.inputSchema,\n output_schema=tool.outputSchema,\n mcp_server=mcp_server\n )\n```\n\n**MCPToolCall Structure:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `tool_key` | `str` | Unique key (e.g., \"action::click\") |\n| `tool_name` | `str` | Tool name (e.g., \"click\") |\n| `title` | `str` | Display title |\n| `namespace` | `str` | Server namespace |\n| `tool_type` | `str` | \"action\" or \"data_collection\" |\n| `description` | `str` | Tool description |\n| `input_schema` | `Dict` | Input parameters schema |\n| `output_schema` | `Dict` | Output schema |\n| `mcp_server` | `BaseMCPServer` | Reference to server |\n\n---\n\n## Meta Tools\n\nMeta tools are built-in methods decorated with `@meta_tool` that provide computer-level operations.\n\n**Example: list_tools Meta Tool**\n\n```python\n@Computer.meta_tool(\"list_tools\")\nasync def list_tools(\n self,\n tool_type: Optional[str] = None,\n namespace: Optional[str] = None,\n remove_meta: bool = True\n) -> CallToolResult:\n \"\"\"\n Get available tools of a specific type.\n \"\"\"\n tools = []\n \n for tool in self._tools_registry.values():\n if ((tool_type is None or tool.tool_type == tool_type)\n and (namespace is None or tool.namespace == namespace)\n and (not remove_meta or tool.tool_name not in self._meta_tools)):\n tools.append(tool.tool_info.model_dump())\n \n return CallToolResult(\n content=[TextContent(type=\"text\", text=json.dumps(tools))]\n )\n```\n\n**Meta Tool Registration:**\n\n```python\n# In __init__:\nfor attr in dir(self):\n method = getattr(self, attr)\n if callable(method) and hasattr(method, \"_meta_tool_name\"):\n name = getattr(method, \"_meta_tool_name\")\n self._meta_tools[name] = method\n```\n\n---\n\n## 🔄 Lifecycle Management\n\n### Reset\n\n```python\n# Computer Manager reset (cascades to all computers)\ncomputer_manager.reset()\n\n# Computer instance reset\ncomputer.reset()\n```\n\n**Reset Operations:**\n\n| Component | Reset Action |\n|-----------|--------------|\n| Computer Manager | Reset all Computer instances |\n| Computer | Clear tool registry, reset MCP servers |\n| MCP Servers | Reset server state |\n\n---\n\n## Best Practices\n\n### Monitor Tool Execution Times\n\n```python\nimport time\nstart = time.time()\nresult = await computer._run_action(tool_call)\nduration = time.time() - start\nif duration > 300: # 5 minutes\n logger.warning(f\"Slow tool: {tool_call.tool_name} took {duration}s\")\n```\n\n### Handle Timeouts Gracefully\n\n```python\n# 100-minute timeout is generous but not infinite\n# Design tools to complete within reasonable time\n```\n\n### Use Namespace Isolation\n\n```python\n# Separate data collection from actions\ndata_tools = await computer.list_tools(tool_type=\"data_collection\")\naction_tools = await computer.list_tools(tool_type=\"action\")\n```\n\n---\n\n## 🚀 Next Steps\n\n👉 [Device Info Provider](./device_info.md) - System profiling \n👉 [MCP Integration](./mcp_integration.md) - MCP server details \n👉 [UFO Client](./ufo_client.md) - Execution orchestration \n👉 [Quick Start](./quick_start.md) - Get started with client \n👉 [Configuration](../configuration/system/overview.md) - UFO configuration\n"
},
{
"path": "documents/docs/client/device_info.md",
"content": "# 📱 Device Info Provider\n\nThe **Device Info Provider** collects comprehensive system information from client devices during registration, enabling intelligent task assignment and device selection in constellation (multi-device) scenarios.\n\nDevice information is proactively collected during client registration and pushed to the server, reducing latency and enabling immediate task routing decisions.\n\n---\n\n## 📋 Overview\n\n**Core Capabilities:**\n\n| Capability | Description | Use Case |\n|------------|-------------|----------|\n| **System Detection** | Auto-detect OS, version, architecture | Platform-specific task routing |\n| **Hardware Profiling** | CPU count, memory capacity | Resource-aware task assignment |\n| **Network Discovery** | Hostname, IP address | Network topology mapping |\n| **Feature Detection** | GUI, CLI, browser, office apps | Capability-based device selection |\n| **Extensibility** | Custom metadata support | Environment-specific configuration |\n\n**Supported Platforms:**\n\n| Platform | Status | Features Detected |\n|----------|--------|-------------------|\n| **Windows** | ✅ Full Support | GUI, CLI, browser, file system, office, Windows apps |\n| **Linux** | ✅ Full Support | GUI, CLI, browser, file system, office, Linux apps |\n| **macOS** | ✅ Full Support | GUI, CLI, browser, file system, office |\n| **Mobile** | 🔮 Planned | Touch, mobile apps, sensors |\n| **IoT** | 🔮 Planned | Sensors, actuators, limited resources |\n\n---\n\n## 🏗️ Architecture\n\n### DeviceSystemInfo Dataclass\n\nThe device info structure captures essential information to minimize registration overhead:\n\n```mermaid\nclassDiagram\n class DeviceSystemInfo {\n +string device_id\n +string platform\n +string os_version\n +int cpu_count\n +float memory_total_gb\n +string hostname\n +string ip_address\n +List~string~ supported_features\n +string platform_type\n +string schema_version\n +Dict custom_metadata\n +to_dict() Dict\n }\n \n class DeviceInfoProvider {\n +collect_system_info() DeviceSystemInfo\n -_get_platform() string\n -_get_os_version() string\n -_get_cpu_count() int\n -_get_memory_total_gb() float\n -_get_hostname() string\n -_get_ip_address() string\n -_detect_features() List~string~\n -_get_platform_type() string\n }\n \n DeviceInfoProvider ..> DeviceSystemInfo : creates\n```\n\n**Field Reference:**\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `device_id` | `str` | Unique client identifier | `\"device_windows_001\"` |\n| `platform` | `str` | OS platform (lowercase) | `\"windows\"`, `\"linux\"`, `\"darwin\"` |\n| `os_version` | `str` | OS version string | `\"10.0.19045\"` (Windows 10) |\n| `cpu_count` | `int` | Number of CPU cores | `8` |\n| `memory_total_gb` | `float` | Total RAM in GB (rounded to 2 decimals) | `16.0` |\n| `hostname` | `str` | Network hostname | `\"DESKTOP-ABC123\"` |\n| `ip_address` | `str` | Local IP address | `\"192.168.1.100\"` |\n| `supported_features` | `List[str]` | Detected capabilities | `[\"gui\", \"cli\", \"browser\", \"office\"]` |\n| `platform_type` | `str` | Device category | `\"computer\"`, `\"mobile\"`, `\"web\"`, `\"iot\"` |\n| `schema_version` | `str` | Schema version for compatibility | `\"1.0\"` |\n| `custom_metadata` | `Dict` | User-defined metadata | `{\"environment\": \"production\"}` |\n\n---\n\n## 🔍 Collection Process\n\n### Automatic Collection\n\n```python\nfrom ufo.client.device_info_provider import DeviceInfoProvider\n\n# Collect system information\nsystem_info = DeviceInfoProvider.collect_system_info(\n client_id=\"device_windows_001\",\n custom_metadata=None # Or load from config\n)\n\n# Result: DeviceSystemInfo object\nprint(system_info.platform) # \"windows\"\nprint(system_info.cpu_count) # 8\nprint(system_info.memory_total_gb) # 16.0\nprint(system_info.supported_features) # [\"gui\", \"cli\", \"browser\", ...]\n\n# Convert to dict for transmission\ndevice_dict = system_info.to_dict()\n```\n\n**Collection Flow:**\n\n```mermaid\nsequenceDiagram\n participant Client\n participant DIP as Device Info Provider\n participant OS as Operating System\n \n Client->>DIP: collect_system_info(client_id, custom_metadata)\n \n par Collect Basic Info\n DIP->>OS: platform.system()\n OS-->>DIP: \"Windows\"\n \n DIP->>OS: platform.version()\n OS-->>DIP: \"10.0.19045\"\n and Collect Hardware Info\n DIP->>OS: os.cpu_count()\n OS-->>DIP: 8\n \n DIP->>OS: psutil.virtual_memory()\n OS-->>DIP: 16GB\n and Collect Network Info\n DIP->>OS: socket.gethostname()\n OS-->>DIP: \"DESKTOP-ABC123\"\n \n DIP->>OS: socket.getsockname()\n OS-->>DIP: \"192.168.1.100\"\n end\n \n DIP->>DIP: _detect_features()\n DIP->>DIP: _get_platform_type()\n \n DIP-->>Client: DeviceSystemInfo\n```\n\n---\n\n## 🎯 Feature Detection\n\n### Platform-Specific Features\n\nFeatures are automatically detected based on the platform to enable capability-based device selection.\n\n**Windows Features:**\n\n```python\nfeatures = [\n \"gui\", # Graphical user interface\n \"cli\", # Command line interface\n \"browser\", # Web browser support\n \"file_system\", # File system operations\n \"office\", # Office applications (Word, Excel, etc.)\n \"windows_apps\" # Windows-specific applications\n]\n```\n\n**Linux Features:**\n\n```python\nfeatures = [\n \"gui\", # Graphical user interface (X11/Wayland)\n \"cli\", # Bash/shell\n \"browser\", # Firefox, Chrome, etc.\n \"file_system\", # Linux file system\n \"office\", # LibreOffice, etc.\n \"linux_apps\" # Linux-specific applications\n]\n```\n\n**macOS Features:**\n\n```python\nfeatures = [\n \"gui\", # macOS GUI\n \"cli\", # Terminal\n \"browser\", # Safari, Chrome, etc.\n \"file_system\", # macOS file system\n \"office\" # Office for Mac\n]\n```\n\n**Feature Detection Logic:**\n\n| Platform | Detected Features | Rationale |\n|----------|-------------------|-----------|\n| `windows`, `linux`, `darwin` | GUI, CLI, browser, file_system, office | Desktop/laptop computers have full capabilities |\n| `android`, `ios` (future) | Touch, mobile apps, camera | Mobile-specific features |\n| Custom | User-defined | Extensible via custom_metadata |\n\n---\n\n## 💡 Usage Examples\n\n### Basic Collection\n\n```python\nfrom ufo.client.device_info_provider import DeviceInfoProvider\n\n# Collect with auto-detection\ninfo = DeviceInfoProvider.collect_system_info(\n client_id=\"device_001\",\n custom_metadata=None\n)\n\nprint(f\"Platform: {info.platform}\")\nprint(f\"CPU Cores: {info.cpu_count}\")\nprint(f\"Memory: {info.memory_total_gb} GB\")\nprint(f\"Features: {', '.join(info.supported_features)}\")\n```\n\n### With Custom Metadata\n\n```python\n# Add environment-specific metadata\ncustom_meta = {\n \"environment\": \"production\",\n \"datacenter\": \"us-east-1\",\n \"role\": \"automation_worker\",\n \"team\": \"qa\"\n}\n\ninfo = DeviceInfoProvider.collect_system_info(\n client_id=\"device_prod_001\",\n custom_metadata=custom_meta\n)\n\n# Custom metadata is preserved\nprint(info.custom_metadata[\"environment\"]) # \"production\"\n```\n\n### JSON Serialization\n\n```python\n# Convert to dictionary for transmission\ndevice_dict = info.to_dict()\n\n# Serialize to JSON\nimport json\njson_str = json.dumps(device_dict, indent=2)\n\n# Example output:\n# {\n# \"device_id\": \"device_001\",\n# \"platform\": \"windows\",\n# \"os_version\": \"10.0.19045\",\n# \"cpu_count\": 8,\n# \"memory_total_gb\": 16.0,\n# \"hostname\": \"DESKTOP-ABC123\",\n# \"ip_address\": \"192.168.1.100\",\n# \"supported_features\": [\"gui\", \"cli\", \"browser\", \"file_system\", \"office\", \"windows_apps\"],\n# \"platform_type\": \"computer\",\n# \"schema_version\": \"1.0\",\n# \"custom_metadata\": {}\n# }\n```\n\n---\n\n## ⚠️ Error Handling\n\n### Graceful Degradation\n\nIf any detection method fails, the provider returns minimal info instead of crashing.\n\n**Error Handling Strategy:**\n\n```python\ntry:\n # Attempt full collection\n return DeviceSystemInfo(...)\nexcept Exception as e:\n logger.error(f\"Error collecting system info: {e}\", exc_info=True)\n # Return minimal info on error\n return DeviceSystemInfo(\n device_id=client_id,\n platform=\"unknown\",\n os_version=\"unknown\",\n cpu_count=0,\n memory_total_gb=0.0,\n hostname=\"unknown\",\n ip_address=\"unknown\",\n supported_features=[],\n platform_type=\"unknown\",\n custom_metadata=custom_metadata or {}\n )\n```\n\n**Individual Method Failures:**\n\n| Method | Failure Behavior | Fallback Value |\n|--------|------------------|----------------|\n| `_get_platform()` | Catch exception | `\"unknown\"` |\n| `_get_os_version()` | Catch exception | `\"unknown\"` |\n| `_get_cpu_count()` | Catch exception | `0` |\n| `_get_memory_total_gb()` | psutil not installed or exception | `0.0` |\n| `_get_hostname()` | Catch exception | `\"unknown\"` |\n| `_get_ip_address()` | Primary method fails | Try hostname resolution, then `\"unknown\"` |\n\n---\n\n## 🔧 Memory Detection Details\n\n### psutil Dependency\n\n!!!warning \"Optional Dependency\"\n Memory detection requires `psutil`. If not installed, memory will be reported as `0.0`.\n\n**Installation:**\n\n```bash\npip install psutil\n```\n\n**Detection Code:**\n\n```python\n@staticmethod\ndef _get_memory_total_gb() -> float:\n \"\"\"Get total memory in GB\"\"\"\n try:\n import psutil\n total_memory = psutil.virtual_memory().total\n return round(total_memory / (1024**3), 2) # Convert to GB, round to 2 decimals\n except ImportError:\n logger.warning(\"psutil not installed, memory info unavailable\")\n return 0.0\n except Exception:\n return 0.0\n```\n\n---\n\n## 🌐 IP Address Detection\n\n### Multi-Method Approach\n\n!!!tip \"Robust IP Detection\"\n IP detection uses a two-stage approach for reliability.\n\n**Primary Method (Socket Connection):**\n\n```python\n# Connect to external address (doesn't actually send data)\ns = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\ns.connect((\"8.8.8.8\", 80)) # Google DNS\nip = s.getsockname()[0]\ns.close()\n```\n\n**Fallback Method (Hostname Resolution):**\n\n```python\n# If primary fails, resolve via hostname\nip = socket.gethostbyname(socket.gethostname())\n```\n\n**Final Fallback:**\n\n```python\n# If all methods fail\nreturn \"unknown\"\n```\n\n---\n\n## 🚀 Integration Points\n\n### WebSocket Client Registration\n\nThe WebSocket client uses the Device Info Provider during registration:\n\n```python\n# In websocket client's register_client()\nfrom ufo.client.device_info_provider import DeviceInfoProvider\n\nsystem_info = DeviceInfoProvider.collect_system_info(\n self.ufo_client.client_id,\n custom_metadata=None\n)\n\nmetadata = {\n \"system_info\": system_info.to_dict(),\n \"registration_time\": datetime.now(timezone.utc).isoformat()\n}\n\nawait self.registration_protocol.register_as_device(\n device_id=self.ufo_client.client_id,\n metadata=metadata,\n platform=self.ufo_client.platform\n)\n```\n\nSee [WebSocket Client](./websocket_client.md) for complete registration flow details.\n\n### Agent Server\n\nThe server receives device info during registration and stores it in the agent profile:\n\n```python\n# Server-side AgentProfile integration\ndevice_info = registration_data[\"metadata\"][\"system_info\"]\nagent_profile.add_device(device_id, device_info)\n```\n\nSee [Server Quick Start](../server/quick_start.md) for server-side processing details.\n\n---\n\n## ✅ Best Practices\n\n**1. Add Custom Metadata for Environment Tracking**\n\n```python\ncustom_meta = {\n \"environment\": os.getenv(\"ENVIRONMENT\", \"development\"),\n \"version\": \"1.0.0\",\n \"deployment_region\": \"us-west-2\",\n \"cost_center\": \"engineering\"\n}\n\nsystem_info = DeviceInfoProvider.collect_system_info(\n client_id=\"device_001\",\n custom_metadata=custom_meta\n)\n```\n\n**2. Install psutil for Accurate Memory Detection**\n\n```bash\npip install psutil\n```\n\n**3. Use Descriptive Client IDs**\n\n```python\n# Include environment and location in client_id\nclient_id = f\"device_{platform}_{env}_{location}_{instance_id}\"\n# Example: \"device_windows_prod_us-west_001\"\n```\n\n**4. Log Collection Results**\n\n```python\nsystem_info = DeviceInfoProvider.collect_system_info(...)\n\nlogger.info(\n f\"Collected device info: \"\n f\"platform={system_info.platform}, \"\n f\"cpu={system_info.cpu_count}, \"\n f\"memory={system_info.memory_total_gb}GB, \"\n f\"features={system_info.supported_features}\"\n)\n```\n\n**5. Validate Before Sending**\n\n```python\nsystem_info = DeviceInfoProvider.collect_system_info(...)\n\n# Validate essential fields\nassert system_info.device_id, \"Device ID required\"\nassert system_info.platform != \"unknown\", \"Platform detection failed\"\nassert system_info.cpu_count > 0, \"CPU detection failed\"\n```\n\n---\n\n## 🚀 Next Steps\n\n- [WebSocket Client](./websocket_client.md) - See how device info is used in registration\n- [Quick Start](./quick_start.md) - Connect your device to the server\n- [MCP Integration](./mcp_integration.md) - Understand client tool capabilities\n- [Server Quick Start](../server/quick_start.md) - Learn server-side registration processing\n"
},
{
"path": "documents/docs/client/mcp_integration.md",
"content": "# 🔌 MCP Integration\n\n**MCP (Model Context Protocol)** provides the tool execution layer in UFO clients, enabling agents to collect system state and execute actions through a standardized interface. This page provides a **client-focused overview** of how MCP integrates into the client architecture.\n\n**Related Documentation:**\n\n- [MCP Overview](../mcp/overview.md) - Core MCP concepts and architecture\n- [Configuration Guide](../mcp/configuration.md) - Server configuration details\n- [Data Collection Servers](../mcp/data_collection.md) - Observation tools\n- [Action Servers](../mcp/action.md) - Execution tools\n- [Creating MCP Servers](../tutorials/creating_mcp_servers.md) - Build custom tools\n\n---\n\n## 🏗️ MCP in Client Architecture\n\n### Role in the Client Stack\n\n```mermaid\ngraph TB\n Server[Agent Server via WebSocket]\n Client[UFO Client Session Orchestration]\n Router[Command Router Command Execution]\n Computer[Computer MCP Tool Manager]\n MCPMgr[MCP Server Manager Server Lifecycle]\n \n DataServers[Data Collection Servers UICollector, etc.]\n ActionServers[Action Servers UIExecutor, CommandLineExecutor]\n \n Server -->|AIP Commands| Client\n Client -->|Execute Actions| Router\n Router -->|Route to Computer| Computer\n Computer -->|Manage Servers| MCPMgr\n Computer -->|Register & Execute| DataServers\n Computer -->|Register & Execute| ActionServers\n \n style Computer fill:#e1f5ff\n style MCPMgr fill:#fff4e6\n style DataServers fill:#e8f5e9\n style ActionServers fill:#fff3e0\n```\n\n**Key Components:**\n\n| Component | Location | Responsibility |\n|-----------|----------|----------------|\n| **Computer** | `ufo.client.computer.Computer` | Manages MCP servers, routes tool calls, executes in thread pool |\n| **MCP Server Manager** | `ufo.client.mcp.mcp_server_manager.MCPServerManager` | Creates/manages server instances (local/http/stdio) |\n| **Command Router** | `ufo.client.computer.CommandRouter` | Routes commands to appropriate Computer instances |\n| **Data Collection Servers** | Various MCP servers | Tools for gathering system state (read-only) |\n| **Action Servers** | Various MCP servers | Tools for performing state changes |\n\n---\n\n## 🔄 Client-MCP Integration Flow\n\n### End-to-End Execution\n\n```mermaid\nsequenceDiagram\n participant Server as Agent Server\n participant Client as UFO Client\n participant Router as Command Router\n participant Computer as Computer\n participant MCP as MCP Server\n \n Server->>Client: AIP Command (tool_name, parameters)\n Client->>Router: execute_actions(commands)\n Router->>Computer: command2tool()\n Computer->>Computer: Convert to MCPToolCall\n Router->>Computer: run_actions([tool_call])\n Computer->>MCP: call_tool(tool_name, parameters)\n MCP-->>Computer: CallToolResult\n Computer-->>Router: Results\n Router-->>Client: List[Result]\n Client-->>Server: AIP Result message\n```\n\n**Execution Stages:**\n\n| Stage | Component | Description |\n|-------|-----------|-------------|\n| **1. Command Reception** | UFO Client | Receives AIP Command from server |\n| **2. Command Routing** | Command Router | Routes to appropriate Computer instance |\n| **3. Command Conversion** | Computer | AIP Command → MCPToolCall |\n| **4. Tool Execution** | Computer | Executes tool via MCP Server |\n| **5. Result Return** | UFO Client | Packages result for server |\n\n---\n\n## 💻 Computer: The MCP Manager\n\n### Computer Class Overview\n\nThe `Computer` class is the **client-side MCP manager**, handling server registration, tool discovery, and execution.\n\n**Core Responsibilities:**\n\n```python\nfrom ufo.client.computer import Computer\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\n# Initialize Computer with MCP servers\ncomputer = Computer(\n name=\"notepad_computer\",\n process_name=\"notepad.exe\",\n mcp_server_manager=mcp_manager,\n data_collection_servers_config=[\n {\"namespace\": \"UICollector\", \"type\": \"local\", \"reset\": False}\n ],\n action_servers_config=[\n {\"namespace\": \"HostUIExecutor\", \"type\": \"local\", \"reset\": False}\n ]\n)\n\n# Async initialization registers all tools\nawait computer.async_init()\n```\n\n**Initialization Sequence:**\n\n| Step | Action | Result |\n|------|--------|--------|\n| 1. Create MCP Server Manager | Initialize server lifecycle manager | Ready to create servers |\n| 2. Initialize data_collection servers | Register observation tools | UICollector ready |\n| 3. Initialize action servers | Register execution tools | HostUIExecutor, CommandLineExecutor ready |\n| 4. Register MCP servers | Query each server for tools | Tool registry populated |\n\nSee [Computer](./computer.md) for detailed class documentation.\n\n---\n\n## 🛠️ Two Server Types\n\n### Data Collection vs Action\n\nUnderstanding the difference between server types is essential for proper MCP usage:\n\n**Comparison:**\n\n| Aspect | Data Collection Servers | Action Servers |\n|--------|------------------------|----------------|\n| **Purpose** | Observe system state | Modify system state |\n| **Examples** | `take_screenshot`, `detect_ui_elements` | `click`, `type_text`, `run_command` |\n| **Invocation** | LLM-selected tools | LLM-selected tools |\n| **Side Effects** | ❌ None (read-only) | ✅ Yes (state changes) |\n| **Namespace** | `\"data_collection\"` | `\"action\"` |\n| **Tool Key Format** | `data_collection::tool_name` | `action::tool_name` |\n\n**Data Collection Example:**\n\n```python\n# Example: Take screenshot for UI analysis\nresult = await computer.run_actions([\n computer.command2tool(Command(\n tool_name=\"take_screenshot\",\n tool_type=\"data_collection\",\n parameters={\"region\": \"active_window\"}\n ))\n])\n```\n\n**Action Example:**\n\n```python\n# Example: Click a button\nresult = await computer.run_actions([\n computer.command2tool(Command(\n tool_name=\"click\",\n tool_type=\"action\",\n parameters={\n \"control_text\": \"Save\",\n \"control_type\": \"Button\"\n }\n ))\n])\n```\n\nSee [MCP Overview - Server Types](../mcp/overview.md#1-two-server-types) for detailed comparison.\n\n---\n\n## 📋 Server Configuration\n\n### Configuration File\n\nMCP servers are configured in `config/ufo/mcp.yaml`:\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector # Server namespace\n type: local # local, http, or stdio\n reset: false # Reset on each step?\n \n action:\n - namespace: HostUIExecutor # Server namespace\n type: local\n reset: false\n \n - namespace: CommandLineExecutor # Multiple servers allowed\n type: local\n reset: false\n```\n\n**Configuration Parameters:**\n\n| Parameter | Type | Description | Example |\n|-----------|------|-------------|---------|\n| `namespace` | `str` | Server identifier (must match registered name) | `\"UICollector\"` |\n| `type` | `str` | Deployment type: `local`, `http`, `stdio` | `\"local\"` |\n| `reset` | `bool` | Reset server state on each step | `false` |\n\n!!!tip \"📖 Full Configuration Guide\"\n See [MCP Configuration](../mcp/configuration.md) for advanced configuration including:\n - HTTP server endpoints\n - Stdio server commands\n - Custom server parameters\n - Environment-specific configs\n\n---\n\n## 🔧 Tool Registry & Execution\n\n### Tool Discovery\n\nThe Computer automatically discovers and registers tools from all configured MCP servers during initialization:\n\n**Automatic Registration:**\n\n```python\n# During computer.async_init()\nasync def register_mcp_servers(self, servers, tool_type):\n \"\"\"Register tools from all MCP servers\"\"\"\n for namespace, server in servers.items():\n # Connect to MCP server\n async with Client(server.server) as client:\n # List available tools\n tools = await client.list_tools()\n \n # Register each tool with unique key\n for tool in tools:\n tool_key = self.make_tool_key(tool_type, tool.name)\n self._tools_registry[tool_key] = MCPToolCall(\n tool_key=tool_key,\n tool_name=tool.name,\n title=tool.title,\n namespace=namespace,\n tool_type=tool_type,\n description=tool.description,\n input_schema=tool.inputSchema,\n output_schema=tool.outputSchema,\n mcp_server=server\n )\n```\n\n**Tool Registry Structure:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `tool_key` | `str` | Unique key: `\"tool_type::tool_name\"` |\n| `tool_name` | `str` | Tool name (e.g., `\"take_screenshot\"`) |\n| `title` | `str` | Display title |\n| `namespace` | `str` | Server namespace (e.g., `\"UICollector\"`) |\n| `tool_type` | `str` | `\"data_collection\"` or `\"action\"` |\n| `description` | `str` | Tool description |\n| `input_schema` | `dict` | JSON schema for parameters |\n| `output_schema` | `dict` | JSON schema for results |\n| `mcp_server` | `BaseMCPServer` | Server instance |\n\n### Tool Execution\n\nTools execute in isolated threads with timeout protection (default: 6000 seconds = 100 minutes per tool):\n\n```python\n# Thread pool configuration\nself._executor = concurrent.futures.ThreadPoolExecutor(\n max_workers=10,\n thread_name_prefix=\"mcp_tool_\"\n)\nself._tool_timeout = 6000 # 100 minutes\n```\n\nSee [Computer](./computer.md) for execution details.\n\n---\n\n## 🚀 Integration Examples\n\n### Basic Usage\n\n```python\nfrom ufo.client.computer import ComputerManager, CommandRouter\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\nfrom aip.messages import Command\n\n# Create MCP server manager\nmcp_server_manager = MCPServerManager()\n\n# Create computer manager (manages Computer instances)\ncomputer_manager = ComputerManager(config, mcp_server_manager)\n\n# Create command router\ncommand_router = CommandRouter(computer_manager)\n\n# Execute action through MCP\ncommand = Command(\n tool_name=\"click\",\n tool_type=\"action\",\n parameters={\n \"control_text\": \"Save\",\n \"control_type\": \"Button\"\n }\n)\n\n# Router creates Computer instance and executes\nresults = await command_router.execute(\n agent_name=\"HostAgent\",\n process_name=\"notepad.exe\",\n root_name=\"default\",\n commands=[command]\n)\n```\n\n### Custom MCP Server\n\n```python\nfrom fastmcp import FastMCP\n\n# Define custom MCP server\nmcp = FastMCP(\"CustomTools\")\n\n@mcp.tool()\nasync def custom_action(param: str) -> str:\n \"\"\"Execute custom action\"\"\"\n return f\"Executed: {param}\"\n\n# Register in config/ufo/mcp.yaml:\n# action:\n# - namespace: CustomTools\n# type: local\n# reset: false\n```\n\n**For step-by-step instructions:**\n\n- [Creating MCP Servers](../tutorials/creating_mcp_servers.md) - Build your own MCP tools\n\n---\n\n## 🔗 Integration Points\n\n### With Other Client Components\n\n**UFO Client:**\n- Receives AIP Commands from server\n- Delegates to Command Router\n- Returns AIP Results\n\n**Command Router:**\n- Routes commands to appropriate Computer instance (by agent/process/root name)\n- Manages command execution with early-exit support\n\n**Computer:**\n- **MCP entry point**: Manages all MCP servers\n- Executes tools via MCP Server Manager\n- Maintains tool registry\n\n**MCP Server Manager:**\n- Creates and manages MCP server instances\n- Supports local, HTTP, and stdio deployment types\n\nSee [UFO Client](./ufo_client.md) and [Computer](./computer.md) for integration details.\n\n---\n\n## 📚 Related Documentation\n\n### Client Components\n\n| Component | Description | Link |\n|-----------|-------------|------|\n| **Computer** | Core MCP execution layer | [Computer](./computer.md) |\n| **UFO Client** | Session orchestration | [UFO Client](./ufo_client.md) |\n| **WebSocket Client** | Server communication | [WebSocket Client](./websocket_client.md) |\n\n### MCP Deep Dive\n\n| Topic | Description | Link |\n|-------|-------------|------|\n| **MCP Overview** | Architecture, concepts, deployment models | [Overview](../mcp/overview.md) |\n| **Data Collection** | Observation tools (UI, screenshots, system) | [Data Collection](../mcp/data_collection.md) |\n| **Action Servers** | Execution tools (click, type, run) | [Action](../mcp/action.md) |\n| **Configuration** | YAML configuration guide | [Configuration](../mcp/configuration.md) |\n| **Local Servers** | Built-in in-process servers | [Local Servers](../mcp/local_servers.md) |\n| **Remote Servers** | HTTP/Stdio deployment | [Remote Servers](../mcp/remote_servers.md) |\n| **Creating MCP Servers** | Build your own tools | [Creating MCP Servers](../tutorials/creating_mcp_servers.md) |\n\n---\n\n## 🎯 Key Takeaways\n\n**MCP in Client - Summary**\n\n**1. Computer is the MCP Manager**\n- Manages all MCP server instances\n- Routes tool calls to appropriate servers\n- Executes in thread pool for isolation\n\n**2. Two Server Types**\n- **Data Collection**: Read-only, observation tools\n- **Action**: State-changing, execution tools\n\n**3. Configuration-Driven**\n- Servers configured in `config/ufo/mcp.yaml`\n- Supports local, HTTP, and stdio deployment\n\n**4. Automatic Registration**\n- Tools auto-discovered during initialization\n- Tool registry built from server metadata\n\n**5. Detailed Docs Available**\n- Full MCP section at [MCP Overview](../mcp/overview.md)\n- Custom server guides, examples, troubleshooting\n\n---\n\n## 🚀 Next Steps\n\n- [MCP Overview](../mcp/overview.md) - Understand MCP architecture in depth\n- [Computer](./computer.md) - See how MCP servers are managed\n- [Creating MCP Servers](../tutorials/creating_mcp_servers.md) - Build your own MCP tools\n"
},
{
"path": "documents/docs/client/overview.md",
"content": "# UFO Client Overview\n\nThe **UFO Client** runs on target devices and serves as the **execution layer** of UFO's distributed agent system. It manages MCP (Model Context Protocol) servers, executes commands deterministically, and communicates with the Agent Server through the Agent Interaction Protocol (AIP).\n\n**Quick Start:** Jump to the [Quick Start Guide](./quick_start.md) to connect your device. Make sure the [Agent Server](../server/quick_start.md) is running first.\n\n---\n\n## 🎯 What is the UFO Client?\n\n```mermaid\ngraph LR\n subgraph \"Agent Server (Brain)\"\n Reasoning[High-Level Reasoning]\n Planning[Task Planning]\n Strategy[Strategy Selection]\n end\n \n subgraph \"Agent Client (Hands)\"\n Execution[Command Execution]\n Tools[Tool Management]\n Reporting[Status Reporting]\n end\n \n subgraph \"Device Environment\"\n Apps[Applications]\n Files[File System]\n UI[User Interface]\n end\n \n Reasoning -->|Directives| Execution\n Planning -->|Commands| Execution\n Strategy -->|Tasks| Execution\n \n Execution --> Tools\n Tools --> Apps\n Tools --> Files\n Tools --> UI\n \n Reporting -->|Results| Reasoning\n \n style Reasoning fill:#bbdefb\n style Execution fill:#c8e6c9\n style Tools fill:#fff9c4\n```\n\n**The UFO Client is a stateless execution agent that:**\n\n| Capability | Description | Benefit |\n|------------|-------------|---------|\n| **🔧 Executes Commands** | Translates server directives into concrete actions | Deterministic, reliable execution |\n| **🛠️ Manages MCP Servers** | Orchestrates local and remote tool interfaces | Extensible tool ecosystem |\n| **📊 Reports Device Info** | Provides hardware and software profile to server | Intelligent task assignment |\n| **📡 Communicates via AIP** | Maintains persistent WebSocket connection | Real-time bidirectional communication |\n| **🚫 Remains Stateless** | Executes directives without high-level reasoning | Independent updates, simple architecture |\n\n**Stateless Design Philosophy:** The client focuses purely on execution. All reasoning and decision-making happens on the server, allowing independent updates to server logic and client tools, simple client architecture, intelligent orchestration of multiple clients, and resource-efficient operation.\n\n**Architecture:** The UFO Client is part of UFO's distributed **server-client architecture**, where it handles command execution and resource access while the [Agent Server](../server/overview.md) handles orchestration and decision-making. See [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md) for the complete design rationale, communication protocols, and deployment patterns.\n\n---\n\n## 🏗️ Architecture\n\nThe client implements a **layered architecture** separating communication, execution, and tool management for maximum flexibility and maintainability.\n\n```mermaid\ngraph TB\n subgraph \"Communication\"\n WSC[WebSocket Client AIP Protocol]\n end\n \n subgraph \"Orchestration\"\n UFC[UFO Client]\n CM[Computer Manager]\n end\n \n subgraph \"Execution\"\n COMP[Computer]\n MCPM[MCP Manager]\n end\n \n subgraph \"Tools\"\n LOCAL[Local MCP Servers]\n REMOTE[Remote MCP Servers]\n end\n \n WSC --> UFC\n UFC --> CM\n CM --> COMP\n COMP --> MCPM\n MCPM --> LOCAL\n MCPM --> REMOTE\n \n style WSC fill:#bbdefb\n style UFC fill:#c8e6c9\n style COMP fill:#fff9c4\n style MCPM fill:#ffcdd2\n```\n\n### Core Components\n\n| Component | Responsibility | Key Features | Documentation |\n|-----------|---------------|--------------|---------------|\n| **WebSocket Client** | AIP communication | • Connection management • Registration • Heartbeat monitoring • Message routing | [Details →](./websocket_client.md) |\n| **UFO Client** | Execution orchestration | • Command execution • Result aggregation • Error handling • Session management | [Details →](./ufo_client.md) |\n| **Computer Manager** | Multi-computer abstraction | • Computer instance management • Namespace routing • Resource isolation | [Details →](./computer_manager.md) |\n| **Computer** | Tool management | • MCP server registration • Tool registry • Execution isolation • Thread pool management | [Details →](./computer.md) |\n| **MCP Server Manager** | MCP lifecycle | • Server creation • Configuration loading • Connection pooling • Health monitoring | [MCP Documentation →](../mcp/overview.md) |\n| **Device Info Provider** | System profiling | • Hardware detection • Capability reporting • Platform identification • Feature enumeration | [Details →](./device_info.md) |\n\nFor detailed component documentation:\n\n- [WebSocket Client](./websocket_client.md) - AIP protocol implementation\n- [UFO Client](./ufo_client.md) - Execution orchestration\n- [Computer Manager](./computer_manager.md) - Multi-computer management\n- [Device Info Provider](./device_info.md) - System profiling\n- [MCP Integration](../mcp/overview.md) - MCP server management (comprehensive documentation)\n\n---\n\n## 🚀 Key Capabilities\n\n### 1. Deterministic Command Execution\n\nThe client executes commands **exactly as specified** without interpretation or reasoning, ensuring predictable behavior.\n\n```mermaid\nsequenceDiagram\n participant Server\n participant Client as UFO Client\n participant Computer\n participant Tool as MCP Tool\n \n Server->>Client: COMMAND (AIP)\n Client->>Computer: Execute Command\n Computer->>Computer: Lookup Tool\n Computer->>Tool: Execute with Timeout\n Tool-->>Computer: Result\n Computer-->>Client: Aggregated Result\n Client-->>Server: COMMAND_RESULTS (AIP)\n```\n\n**Execution Flow:**\n\n| Step | Action | Purpose |\n|------|--------|---------|\n| 1️⃣ **Receive** | Get structured command from server via AIP | Ensure well-formed input |\n| 2️⃣ **Route** | Dispatch to appropriate computer instance | Support multi-namespace execution |\n| 3️⃣ **Lookup** | Find tool in MCP registry | Dynamic tool resolution |\n| 4️⃣ **Execute** | Run tool in isolated thread pool | Fault isolation and timeout protection |\n| 5️⃣ **Aggregate** | Combine results from multiple tools | Structured response format |\n| 6️⃣ **Return** | Send results back to server via AIP | Complete the execution loop |\n\n**Execution Guarantees:**\n- **Isolation**: Each tool runs in separate thread pool\n- **Timeouts**: Configurable timeout (default: 6000 seconds/100 minutes)\n- **Fault Tolerance**: One failed tool doesn't crash entire client\n- **Thread Safety**: Concurrent tool execution supported\n- **Error Reporting**: Structured errors returned to server\n\n### 2. MCP Server Management\n\nThe client manages a collection of **MCP (Model Context Protocol) servers** to provide diverse tool access for automation tasks. The client is responsible for registering, managing, and executing these tools, while the [Agent Server](../server/overview.md) handles command orchestration. See [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md#client-command-execution-and-resource-access) for how MCP integration fits into the overall architecture.\n\n**MCP Server Categories:**\n\n**Data Collection Servers** gather information from the device:\n \n| Server Type | Tools Provided | Use Cases |\n|-------------|---------------|-----------|\n| **System Info** | CPU, memory, disk stats | Resource monitoring |\n| **Application State** | Running apps, windows | Context awareness |\n| **Screenshot** | Screen capture | Visual verification |\n| **UI Element Detection** | Control trees, accessibility | UI automation |\n \nExample Tools: `get_system_info()`, `list_running_apps()`, `capture_screenshot()`, `get_ui_tree()`\n\n**Action Servers** perform actions on the device:\n \n| Server Type | Tools Provided | Use Cases |\n|-------------|---------------|-----------|\n| **GUI Automation** | Keyboard, mouse, clicks | UI interaction |\n| **Application Control** | Launch, close, focus | App management |\n| **File System** | Read, write, delete | File operations |\n| **Command Execution** | Shell commands | System automation |\n \nExample Tools: `click_button(label)`, `type_text(text)`, `open_application(name)`, `execute_command(cmd)`\n\n**Server Types:**\n\n| Type | Deployment | Pros | Cons |\n|------|------------|------|------|\n| **Local MCP Servers** | Run in same process via FastMCP | Fast, no network overhead | Limited to local capabilities |\n| **Remote MCP Servers** | Connect via HTTP/SSE | Scalable, shared services | Network latency, external dependency |\n\n**Example MCP Server Configuration:**\n\n```yaml\nmcp_servers:\n data_collection:\n - name: \"system_info\"\n type: \"local\"\n class: \"SystemInfoServer\"\n - name: \"ui_detector\"\n type: \"local\"\n class: \"UIDetectionServer\"\n \n action:\n - name: \"gui_automation\"\n type: \"local\"\n class: \"GUIAutomationServer\"\n - name: \"file_ops\"\n type: \"remote\"\n url: \"http://localhost:8080/mcp\"\n```\n\nSee [MCP Integration](../mcp/overview.md) for comprehensive MCP server documentation.\n\n### 3. Device Profiling\n\nThe client automatically collects and reports **device information** to enable the server to make intelligent task routing decisions.\n\n**Device Profile Structure:**\n\n```json\n{\n \"device_id\": \"device_windows_001\",\n \"platform\": \"windows\",\n \"platform_type\": \"computer\",\n \"os_version\": \"10.0.22631\",\n \"system_info\": {\n \"cpu_count\": 8,\n \"memory_total_gb\": 16.0,\n \"disk_total_gb\": 512.0,\n \"hostname\": \"DESKTOP-ABC123\",\n \"ip_address\": \"192.168.1.100\"\n },\n \"supported_features\": [\n \"gui_automation\",\n \"cli_execution\",\n \"browser_control\",\n \"office_integration\",\n \"windows_apps\"\n ],\n \"installed_applications\": [\n \"Chrome\",\n \"Excel\",\n \"PowerPoint\",\n \"VSCode\"\n ],\n \"screen_resolution\": \"1920x1080\",\n \"connected_at\": \"2025-11-05T10:30:00Z\"\n}\n```\n\n**Profile Usage on Server:**\n\n```mermaid\ngraph LR\n Client[Client Detects Device Info]\n Server[Server Stores Profile]\n Route[Server Routes Tasks]\n \n Client -->|Report Profile| Server\n Server -->|Match Requirements| Route\n Route -->|Dispatch Task| Client\n \n style Client fill:#bbdefb\n style Server fill:#c8e6c9\n style Route fill:#fff9c4\n```\n\n**Server Uses Profile For:**\n\n| Use Case | Example Logic |\n|----------|--------------|\n| **Platform Matching** | Route Excel task to Windows device |\n| **Capability Filtering** | Only send browser tasks to devices with Chrome |\n| **Load Balancing** | Distribute tasks based on CPU/memory |\n| **Failure Recovery** | Reassign task if device disconnects |\n\nSee [Device Info Provider](./device_info.md) for detailed profiling documentation.\n\n### 4. Resilient Communication\n\nRobust, fault-tolerant communication with the server using strongly-typed AIP messages.\n\n**Connection Lifecycle:**\n\n```mermaid\nstateDiagram-v2\n [*] --> Disconnected\n Disconnected --> Connecting: Initiate Connection\n Connecting --> Registering: WebSocket Established\n Registering --> Connected: Registration Success\n Connecting --> Disconnected: Connection Failed\n Registering --> Disconnected: Registration Failed\n \n Connected --> Heartbeating: Start Heartbeat Loop\n Heartbeating --> Heartbeating: Send/Receive Heartbeat\n Heartbeating --> Disconnected: Heartbeat Timeout\n Heartbeating --> Disconnected: WebSocket Closed\n \n Disconnected --> Connecting: Retry (Exponential Backoff)\n \n note right of Connected\n • Receive commands\n • Execute tasks\n • Report results\n end note\n \n note right of Heartbeating\n Default interval: 30s\n Timeout: 60s\n end note\n```\n\n**Connection Features:**\n\n| Feature | Description | Configuration |\n|---------|-------------|---------------|\n| **Auto Registration** | Registers with server on connect | Device ID, platform, capabilities |\n| **Exponential Backoff** | Smart retry on connection failure | Max retries: 5 (default) |\n| **Heartbeat Monitoring** | Keep-alive mechanism | Interval: 30s (configurable) |\n| **Graceful Reconnection** | Resume operation after disconnect | Auto-reconnect on network recovery |\n\n**Message Types:**\n\n| Message | Direction | Purpose |\n|---------|-----------|---------|\n| `REGISTRATION` | Client → Server | Register device with capabilities |\n| `REGISTRATION_ACK` | Server → Client | Confirm registration |\n| `HEARTBEAT` | Client ↔ Server | Keep connection alive |\n| `COMMAND` | Server → Client | Execute task command |\n| `COMMAND_RESULTS` | Client → Server | Return execution results |\n| `ERROR` | Client → Server | Report execution errors |\n\nSee [WebSocket Client](./websocket_client.md) and [AIP Protocol](../aip/overview.md) for protocol details.\n\n---\n\n## 📋 Workflow Examples\n\n### Client Initialization & Registration\n\n```mermaid\nsequenceDiagram\n participant Main as Client Main\n participant MCP as MCP Manager\n participant WSC as WebSocket Client\n participant Server\n \n Main->>MCP: Initialize MCP Servers\n MCP-->>Main: Server Registry Ready\n \n Main->>WSC: Create Client & Connect\n WSC->>Server: WebSocket Connect\n Server-->>WSC: Connection Established\n \n WSC->>WSC: Collect Device Info\n WSC->>Server: REGISTRATION\n Server-->>WSC: REGISTRATION_ACK\n \n WSC->>WSC: Start Heartbeat Loop\n \n loop Every 30 seconds\n WSC->>Server: HEARTBEAT\n Server-->>WSC: HEARTBEAT_ACK\n end\n \n Note over WSC,Server: Ready to Execute Commands\n```\n\n**Initialization Steps:**\n\n| Step | Action | Details |\n|------|--------|---------|\n| 1️⃣ **Parse Args** | Process command-line arguments | `--client-id`, `--ws-server`, `--platform` |\n| 2️⃣ **Load Config** | Load UFO configuration | MCP servers, tools, settings |\n| 3️⃣ **Init MCP** | Initialize MCP server manager | Create local/remote servers |\n| 4️⃣ **Create Managers** | Create computer manager | Register MCP servers with computers |\n| 5️⃣ **Connect** | Establish WebSocket connection | Connect to server |\n| 6️⃣ **Register** | Send device profile | Platform, capabilities, system info |\n| 7️⃣ **Heartbeat** | Start keep-alive loop | Default: 30s interval |\n| 8️⃣ **Listen** | Wait for commands | Ready for task execution |\n\n### Command Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Server\n participant Client as UFO Client\n participant Comp as Computer\n participant Tool as MCP Tool\n \n Server->>Client: COMMAND {type: \"click_button\", args: {...}}\n Client->>Comp: execute_command()\n Comp->>Comp: find_tool(\"click_button\")\n \n alt Tool Found\n Comp->>Tool: execute(args)\n Note over Tool: Thread Pool Execution 6000s timeout\n Tool-->>Comp: Success\n Comp-->>Client: Result\n Client-->>Server: COMMAND_RESULTS {status: \"completed\"}\n else Tool Not Found\n Comp-->>Client: Error\n Client-->>Server: ERROR {error: \"Tool not found\"}\n end\n```\n\n---\n\n## 🖥️ Platform Support\n\nThe client supports multiple platforms with platform-specific tool implementations.\n\n| Platform | Status | Features | Native Tools |\n|----------|--------|----------|--------------|\n| **Windows** | ✅ **Full Support** | • UI Automation (UIAutomation API) • COM API integration • Office automation • Windows-specific apps | PowerShell, Registry, WMI, Win32 API |\n| **Linux** | ✅ **Full Support** | • Bash automation • X11/Wayland GUI tools • Package managers • Linux applications | bash, apt/yum, systemd, xdotool |\n| **macOS** | 🚧 **In Development** | • macOS applications • Automator integration • AppleScript support | osascript, Automator, launchctl |\n| **Mobile** | 🔮 **Planned** | • Touch interface • Mobile apps • Gesture control | ADB (Android), XCTest (iOS) |\n\n**Platform Detection:**\n\n- **Automatic**: Detected via `platform.system()` on startup\n- **Override**: Use `--platform` flag to specify manually\n- **Validation**: Server validates platform matches task requirements\n\n**Platform-Specific Example:**\n\n**Windows:**\n```python\n# Windows-specific tools\ntools = [\n \"open_windows_app(name='Excel')\",\n \"execute_powershell(script='Get-Process')\",\n \"read_registry(key='HKLM\\\\Software')\"\n]\n```\n\n**Linux:**\n```python\n# Linux-specific tools\ntools = [\n \"execute_bash(command='ls -la')\",\n \"install_package(name='vim')\",\n \"control_systemd(service='nginx', action='restart')\"\n]\n```\n\n---\n\n## ⚙️ Configuration\n\n### Command-Line Arguments\n\nStart the UFO client with:\n\n```bash\npython -m ufo.client.client [OPTIONS]\n```\n\n**Available Options:**\n\n| Option | Type | Default | Description | Example |\n|--------|------|---------|-------------|---------|\n| `--client-id` | `str` | `client_001` | Unique client identifier | `--client-id device_win_001` |\n| `--ws-server` | `str` | `ws://localhost:5000/ws` | WebSocket server URL | `--ws-server ws://192.168.1.10:5000/ws` |\n| `--ws` | `flag` | `False` | **Enable WebSocket mode** (required) | `--ws` |\n| `--max-retries` | `int` | `5` | Connection retry limit | `--max-retries 10` |\n| `--platform` | `str` | Auto-detect | Platform override | `--platform windows` |\n| `--log-level` | `str` | `WARNING` | Logging verbosity | `--log-level DEBUG` |\n\n**Quick Start Command:**\n\n```bash\n# Minimal command (default server)\npython -m ufo.client.client --ws --client-id my_device\n\n# Production command (custom server)\npython -m ufo.client.client \\\n --ws \\\n --client-id device_production_01 \\\n --ws-server ws://ufo-server.company.com:5000/ws \\\n --max-retries 10 \\\n --log-level INFO\n```\n\n### UFO Configuration\n\nThe client inherits settings from `config_dev.yaml`:\n\n**Key Configuration Sections:**\n\n| Section | Purpose | Example |\n|---------|---------|---------|\n| **MCP Servers** | Define data collection and action servers | `mcp_servers.data_collection`, `mcp_servers.action` |\n| **Tool Settings** | Tool-specific parameters | Timeouts, retries, API keys |\n| **Logging** | Log levels, formats, destinations | File logging, console output |\n| **Platform Settings** | OS-specific configurations | Windows UI automation settings |\n\n**Sample Configuration:**\n\n```yaml\nclient:\n heartbeat_interval: 30 # seconds\n command_timeout: 6000 # seconds (100 minutes)\n max_concurrent_tools: 10\n\nmcp_servers:\n data_collection:\n - name: system_info\n type: local\n enabled: true\n action:\n - name: gui_automation\n type: local\n enabled: true\n settings:\n click_delay: 0.5\n typing_speed: 100 # chars per minute\n\nlogging:\n level: INFO\n format: \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n file: \"logs/client.log\"\n```\n\nSee [Configuration Guide](../configuration/system/overview.md) for comprehensive documentation.\n\n---\n\n## ⚠️ Error Handling\n\nThe client is designed to handle various failure scenarios gracefully without crashing.\n\n### Connection Failures\n\n```mermaid\nstateDiagram-v2\n [*] --> Attempting\n Attempting --> Connected: Success\n Attempting --> Failed: Error\n \n Failed --> Waiting: Exponential Backoff\n Waiting --> Attempting: Retry (2^n seconds)\n \n Failed --> [*]: Max Retries Exceeded\n \n note right of Waiting\n Retry Delays:\n 1st: 2s\n 2nd: 4s\n 3rd: 8s\n 4th: 16s\n 5th: 32s\n end note\n```\n\n**Connection Error Handling:**\n\n| Scenario | Client Behavior | Configuration |\n|----------|----------------|---------------|\n| **Initial Connection Failed** | Exponential backoff retry | `--max-retries` (default: 5) |\n| **Connection Lost** | Attempt reconnection | Automatic |\n| **Max Retries Exceeded** | Exit with error code | Log error, exit |\n| **Server Unreachable** | Log error, retry | Backoff between retries |\n\n### Tool Execution Failures\n\n**Protection Mechanisms:**\n\n| Mechanism | Purpose | Default Value |\n|-----------|---------|---------------|\n| **Thread Pool Isolation** | Prevent one tool from blocking others | Enabled |\n| **Execution Timeout** | Kill hung tools | 6000 seconds (100 minutes) |\n| **Exception Catching** | Graceful error handling | All tools wrapped |\n| **Error Reporting** | Notify server of failures | Structured error messages |\n\n**Error Handling Example:**\n\n```python\n# Client automatically handles tool errors\ntry:\n result = tool.execute(args)\n return {\"status\": \"success\", \"result\": result}\nexcept TimeoutError:\n return {\"status\": \"error\", \"error\": \"Tool execution timeout\"}\nexcept Exception as e:\n return {\"status\": \"error\", \"error\": str(e)}\n```\n\n### Server Disconnection\n\n**Graceful Shutdown Process:**\n\n1. **Detect Disconnection** - WebSocket connection lost\n2. **Stop Heartbeat** - Terminate keep-alive loop\n3. **Cancel Pending Tasks** - Abort in-progress commands\n4. **Attempt Reconnection** - Use exponential backoff\n5. **Clean Shutdown** - If max retries exceeded\n\n---\n\n## ✅ Best Practices\n\n### Development Best Practices\n\n**1. Use Unique Client IDs**\n\n```bash\n# Bad: Generic ID\n--client-id client_001\n\n# Good: Descriptive ID\n--client-id device_win_dev_john_laptop\n```\n\n**2. Start with INFO Logging**\n\n```bash\n# Development: WARNING for normal operation (default)\n--log-level WARNING\n\n# Debugging: DEBUG for troubleshooting\n--log-level DEBUG\n```\n\n**3. Test MCP Connectivity First**\n\n```python\n# Verify MCP servers are accessible before running client\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\nmanager = MCPServerManager()\n# Test server creation from configuration\n```\n\n### Production Best Practices\n\n**1. Use Descriptive Client IDs**\n\n```bash\n# Include environment, location, purpose\n--client-id device_windows_production_office_01\n--client-id device_linux_staging_lab_02\n```\n\n**2. Configure Automatic Restart**\n\n**systemd (Linux):**\n\n```ini\n[Unit]\nDescription=UFO Agent Client\nAfter=network.target\n \n[Service]\nType=simple\nUser=ufo\nWorkingDirectory=/opt/ufo\nExecStart=/usr/bin/python3 -m ufo.client.client \\\n --ws \\\n --client-id device_linux_prod_01 \\\n --ws-server ws://ufo-server.internal:5000/ws \\\n --log-level INFO\nRestart=always\nRestartSec=10\n \n[Install]\nWantedBy=multi-user.target\n```\n\n**PM2 (Cross-platform):**\n\n```json\n{\n \"apps\": [{\n \"name\": \"ufo-client\",\n \"script\": \"python\",\n \"args\": [\n \"-m\", \"ufo.client.client\",\n \"--ws\",\n \"--client-id\", \"device_win_prod_01\",\n \"--ws-server\", \"ws://ufo-server.internal:5000/ws\",\n \"--log-level\", \"INFO\"\n ],\n \"cwd\": \"C:\\\\ufo\",\n \"restart_delay\": 5000,\n \"max_restarts\": 10\n }]\n}\n```\n\n**3. Monitor Connection Health**\n\n```bash\n# Check logs for connection status\ntail -f logs/client.log | grep -E \"Connected|Disconnected|ERROR\"\n```\n\n### Security Best Practices\n\n!!! warning \"Security Considerations\"\n \n | Practice | Description | Implementation |\n |----------|-------------|----------------|\n | **Use WSS** | Encrypt WebSocket communication | `wss://server:5000/ws` instead of `ws://` |\n | **Validate Server** | Verify server certificate | Configure SSL/TLS verification |\n | **Restrict Tools** | Limit MCP server access | Only enable necessary tools |\n | **Least Privilege** | Run with minimum permissions | Create dedicated user account |\n | **Network Isolation** | Use firewalls and VPNs | Restrict server access to internal network |\n\n---\n\n## 🎓 Documentation Map\n\n### Getting Started\n\n| Document | Purpose | When to Read |\n|----------|---------|--------------|\n| [Quick Start](./quick_start.md) | Connect your device quickly | First time setup |\n| [Server Quick Start](../server/quick_start.md) | Understand server-side setup | Before running client |\n\n### Component Details\n\n| Document | Component | Topics Covered |\n|----------|-----------|----------------|\n| [WebSocket Client](./websocket_client.md) | Communication layer | AIP protocol, connection management |\n| [UFO Client](./ufo_client.md) | Orchestration | Session tracking, command execution |\n| [Computer Manager](./computer_manager.md) | Multi-computer abstraction | Namespace management, routing |\n| [Computer](./computer.md) | Tool management | MCP registry, execution |\n| [Device Info](./device_info.md) | System profiling | Hardware detection, capabilities |\n| [MCP Integration](./mcp_integration.md) | MCP servers | Server types, configuration |\n\n### Related Documentation\n\n| Document | Topic | Relevance |\n|----------|-------|-----------|\n| [Server Overview](../server/overview.md) | Server architecture | Understand the other half |\n| [AIP Protocol](../aip/overview.md) | Communication protocol | Deep dive into messaging |\n| [Configuration](../configuration/system/overview.md) | UFO configuration | Customize behavior |\n\n---\n\n## 🔄 Client vs. Server\n\nUnderstanding the **clear division** between client and server responsibilities is crucial for effective system design.\n\n**Responsibility Matrix:**\n\n| Aspect | Client (Execution) | Server (Orchestration) |\n|--------|-------------------|------------------------|\n| **Primary Role** | Execute directives deterministically | Reason about tasks, plan actions |\n| **State Management** | Stateless (no session memory) | Stateful (maintains sessions) |\n| **Reasoning** | None (pure execution) | Full (high-level decision-making) |\n| **Tools** | MCP servers (local/remote) | Agent strategies, prompts, LLMs |\n| **Communication** | Device ↔ Server (AIP) | Multi-client coordination |\n| **Updates** | Tool implementation changes | Strategy and logic updates |\n| **Complexity** | Low (simple execution loop) | High (complex orchestration) |\n| **Dependencies** | MCP servers, system APIs | LLMs, databases, client registry |\n\n**Workflow Comparison:**\n\n```mermaid\ngraph TB\n subgraph \"Server Workflow\"\n S1[Receive User Request]\n S2[Reason About Task]\n S3[Plan Execution Steps]\n S4[Select Target Device]\n S5[Send Commands]\n end\n \n subgraph \"Client Workflow\"\n C1[Receive Command]\n C2[Lookup Tool]\n C3[Execute Tool]\n C4[Return Result]\n end\n \n S1 --> S2\n S2 --> S3\n S3 --> S4\n S4 --> S5\n S5 -.->|AIP| C1\n C1 --> C2\n C2 --> C3\n C3 --> C4\n C4 -.->|AIP| S5\n \n style S1 fill:#bbdefb\n style S2 fill:#bbdefb\n style S3 fill:#bbdefb\n style C1 fill:#c8e6c9\n style C2 fill:#c8e6c9\n style C3 fill:#c8e6c9\n```\n\n**Decoupled Architecture Benefits:**\n- Independent Updates: Modify server logic without touching clients\n- Flexible Deployment: Run clients on any platform\n- Scalability: Add more clients without server changes\n- Maintainability: Simpler client code, easier debugging\n- Testability: Test client and server independently\n\n---\n\n## 🚀 Next Steps\n\n**1. Run Your First Client**\n\n```bash\n# Follow the quick start guide\npython -m ufo.client.client \\\n --ws \\\n --client-id my_first_device \\\n --ws-server ws://localhost:5000/ws\n```\n👉 [Quick Start Guide](./quick_start.md)\n\n**2. Understand Registration Process**\n\nLearn how clients register with the server, device profile structure, and registration acknowledgment.\n\n👉 [Server Quick Start](../server/quick_start.md) - Start server and connect clients\n\n**3. Explore MCP Integration**\n\nLearn about MCP servers, configure custom tools, and create your own MCP servers.\n\n👉 [MCP Integration](../mcp/overview.md)\n\n**4. Configure for Your Environment**\n\nCustomize MCP servers, adjust timeouts and retries, and configure platform-specific settings.\n\n👉 [Configuration Guide](../configuration/system/overview.md)\n\n**5. Master the Protocol**\n\nDeep dive into AIP messages, understand message flow, and error handling patterns.\n\n👉 [AIP Protocol](../aip/overview.md)\n"
},
{
"path": "documents/docs/client/quick_start.md",
"content": "# ⚡ Quick Start\n\nGet your device connected to the UFO Agent Server and start executing tasks in minutes. No complex setup—just run a single command.\n\n---\n\n## 📋 Prerequisites\n\nBefore connecting a client device, ensure these requirements are met:\n\n| Requirement | Version/Details | Verification Command |\n|-------------|-----------------|----------------------|\n| **Python** | 3.10 or higher | `python --version` |\n| **UFO Installation** | Latest version with dependencies | `python -c \"import ufo; print('✅ Installed')\"` |\n| **Running Server** | Agent server accessible on network | `curl http://server:5000/api/health` |\n| **Network Access** | Client can reach server WebSocket endpoint | Test connectivity to server |\n\n!!! tip \"Server First!\"\n **Always start the Agent Server before connecting clients.** The server must be running and accessible for clients to register successfully.\n \n 👉 [Server Quick Start Guide](../server/quick_start.md)\n\n**Verify Server Status:**\n\n**Windows:**\n```powershell\n# Test HTTP API\nInvoke-WebRequest -Uri http://localhost:5000/api/health\n \n# Test WebSocket (requires wscat)\nwscat -c ws://localhost:5000/ws\n```\n\n**Linux/macOS:**\n```bash\n# Test HTTP API\ncurl http://localhost:5000/api/health\n \n# Test WebSocket (requires wscat)\nwscat -c ws://localhost:5000/ws\n```\n\n---\n\n## 🚀 Starting a Device Client\n\n### Minimal Command (Local Server)\n\nConnect to a server running on the same machine with default settings:\n\n```bash\npython -m ufo.client.client --ws --client-id my_device\n```\n\n**What This Does:**\n\n| Parameter | Default Value | Purpose |\n|-----------|---------------|---------|\n| `--ws` | N/A (flag) | **Enable WebSocket mode** (required) |\n| `--client-id` | `my_device` | Unique identifier for this device |\n| `--ws-server` | `ws://localhost:5000/ws` | Connect to local server |\n| `--platform` | Auto-detected | Detected from `platform.system()` |\n| `--max-retries` | `5` | Connection retry attempts |\n\n### Connect to Remote Server\n\nConnect to a server running on a different machine in your network:\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5000/ws \\\n --client-id device_windows_001\n```\n\n**Network Requirements:**\n\n- ✅ Client can ping the server: `ping 192.168.1.100`\n- ✅ Port **5000** is accessible (firewall allows)\n- ✅ Server is running and listening on correct port\n\n### Override Platform Detection\n\n!!! tip \"When to Override\"\n Normally, the client auto-detects the platform (`windows` or `linux`). Override when:\n \n - Running in container/VM with mismatched OS\n - Testing cross-platform behavior\n - Platform detection fails\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://127.0.0.1:5000/ws \\\n --client-id my_linux_device \\\n --platform linux\n```\n\n### Complete Command (All Options)\n\nProduction-ready configuration with all available options:\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5000/ws \\\n --client-id device_windows_prod_01 \\\n --platform windows \\\n --max-retries 10 \\\n --log-level WARNING\n```\n\n**Enhancements:**\n\n- 🔁 **10 retries**: Resilient to temporary network issues\n- 📋 **WARNING logging**: Default level (less verbose than INFO)\n- 🏷️ **Descriptive ID**: `device_windows_prod_01` clearly identifies environment\n\n---\n\n## 📝 Connection Parameters Reference\n\nAll available command-line options for the UFO client.\n\n### Required Parameters\n\n| Parameter | Description | Example |\n|-----------|-------------|---------|\n| `--ws` | **Enable WebSocket mode** (flag, no value) | `--ws` |\n\n### Connection Parameters\n\n| Parameter | Type | Default | Description | Example |\n|-----------|------|---------|-------------|---------|\n| `--ws-server` | `str` | `ws://localhost:5000/ws` | WebSocket server URL | `--ws-server ws://192.168.1.10:5000/ws` |\n| `--max-retries` | `int` | `5` | Maximum connection retry attempts | `--max-retries 10` |\n\n### Device Parameters\n\n| Parameter | Type | Default | Description | Example |\n|-----------|------|---------|-------------|---------|\n| `--client-id` | `str` | `client_001` | **Unique device identifier** | `--client-id device_win_prod_01` |\n| `--platform` | `str` | Auto-detect | Platform override: `windows` or `linux` | `--platform linux` |\n\n### Logging Parameters\n\n| Parameter | Type | Default | Description | Example |\n|-----------|------|---------|-------------|---------|\n| `--log-level` | `str` | `WARNING` | Logging verbosity: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, `OFF` | `--log-level DEBUG` |\n\n!!! warning \"Unique Client IDs - Critical!\"\n **Each device MUST have a unique `--client-id`.** Duplicate IDs will cause:\n \n - ❌ Connection conflicts (devices disconnecting each other)\n - ❌ Task routing failures (tasks sent to wrong device)\n - ❌ Session corruption (server state confusion)\n \n **Best Practice:** Use descriptive IDs:\n ```\n ✅ device_windows_prod_datacenter1_rack3\n ✅ device_linux_staging_jenkins_worker2\n ❌ client_001\n ❌ device1\n ```\n\n---\n\n## ✅ Successful Connection\n\n### Client Logs\n\nWhen the client connects successfully, you'll see this sequence:\n\n```log\nINFO - Platform detected/specified: windows\nINFO - UFO Client initialized for platform: windows\nINFO - [WS] Connecting to ws://127.0.0.1:5000/ws (attempt 1/5)\nINFO - [WS] [AIP] Collected device info: platform=windows, cpu=8, memory=16.0GB\nINFO - [WS] [AIP] Attempting to register as device_windows_001\nINFO - [WS] [AIP] ✅ Successfully registered as device_windows_001\nINFO - [WS] Heartbeat loop started (interval: 30s)\n```\n\n**Registration Flow:**\n\n```mermaid\nsequenceDiagram\n participant C as Client\n participant S as Server\n \n C->>C: Load Config & Initialize MCP\n C->>S: WebSocket Connect\n S-->>C: Connection Ack\n \n C->>C: Collect Device Info\n C->>S: REGISTRATION (id, platform, capabilities)\n S->>S: Validate & Store\n S-->>C: REGISTRATION_ACK\n \n loop Every 30s\n C->>S: HEARTBEAT\n S-->>C: HEARTBEAT_ACK\n end\n \n Note over C,S: Ready for Commands\n```\n\n### Server Logs\n\nOn the server side, you'll see:\n\n```log\nINFO - [WS] ✅ Registered device client: device_windows_001\nINFO - [WS] Device device_windows_001 capabilities: {\n \"platform\": \"windows\",\n \"cpu_count\": 8,\n \"memory_gb\": 16.0,\n \"mcp_servers\": [\"system_info\", \"gui_automation\"]\n}\n```\n\n---\n\n## 🔍 Verify Connection\n\n### Check Connected Clients (HTTP API)\n\nFrom the server machine or any network-accessible machine:\n\n**cURL:**\n```bash\ncurl http://localhost:5000/api/clients\n```\n\n**PowerShell:**\n```powershell\nInvoke-RestMethod -Uri http://localhost:5000/api/clients | ConvertTo-Json\n```\n\n**Python:**\n```python\nimport requests\nresponse = requests.get(\"http://localhost:5000/api/clients\")\nprint(response.json())\n```\n\n**Expected Response:**\n\n```json\n{\n \"clients\": [\n {\n \"client_id\": \"device_windows_001\",\n \"type\": \"device\",\n \"platform\": \"windows\",\n \"connected_at\": 1730736000.0,\n \"uptime_seconds\": 45,\n \"capabilities\": {\n \"cpu_count\": 8,\n \"memory_gb\": 16.0,\n \"mcp_servers\": [\"system_info\", \"gui_automation\"]\n }\n }\n ],\n \"total\": 1\n}\n```\n\n**Client Status Indicators:**\n\n| Field | Description | Example |\n|-------|-------------|---------|\n| `client_id` | Unique device identifier | `device_windows_001` |\n| `type` | Client type (always `\"device\"`) | `device` |\n| `platform` | Operating system | `windows`, `linux` |\n| `connected_at` | Unix timestamp of connection | `1730736000.0` |\n| `uptime_seconds` | Seconds since connection | `45` |\n| `capabilities` | Device hardware/software profile | CPU, memory, MCP servers |\n\n### Monitor Heartbeats\n\nThe client sends **heartbeat messages every 30 seconds** to prove it's still alive.\n\n**Client Logs (DEBUG level):**\n\n```log\nDEBUG - [WS] [AIP] Heartbeat sent\nDEBUG - [WS] [AIP] Heartbeat acknowledged\n```\n\n**Server Logs (DEBUG level):**\n\n```log\nDEBUG - [WS] Heartbeat received from device_windows_001\nDEBUG - [WS] Heartbeat acknowledged for device_windows_001\n```\n\n---\n\n## 🎯 Running Your First Task\n\nOnce the client is connected, dispatch a simple task from the server to verify end-to-end functionality.\n\n### Dispatch Task via HTTP API\n\n**cURL:**\n```bash\ncurl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"device_windows_001\",\n \"request\": \"Open Notepad and type Hello from UFO\"\n }'\n```\n\n**PowerShell:**\n```powershell\n$body = @{\n client_id = \"device_windows_001\"\n request = \"Open Notepad and type Hello from UFO\"\n} | ConvertTo-Json\n \nInvoke-RestMethod -Uri http://localhost:5000/api/dispatch `\n -Method POST `\n -ContentType \"application/json\" `\n -Body $body\n```\n\n**Python:**\n```python\nimport requests\n \nresponse = requests.post(\n \"http://localhost:5000/api/dispatch\",\n json={\n \"client_id\": \"device_windows_001\",\n \"request\": \"Open Notepad and type Hello from UFO\"\n }\n)\nprint(response.json())\n```\n\n### Server Response\n\n```json\n{\n \"status\": \"success\",\n \"session_id\": \"session_20251104_143022_abc123\",\n \"message\": \"Task dispatched to device_windows_001\",\n \"client_id\": \"device_windows_001\"\n}\n```\n\n**Response Fields:**\n\n| Field | Description |\n|-------|-------------|\n| `status` | `\"success\"` or `\"error\"` |\n| `session_id` | Unique session identifier for tracking |\n| `message` | Human-readable status message |\n| `client_id` | Target device that received the task |\n\n### Client Execution Logs\n\n```log\nINFO - [WS] Starting task: Open Notepad and type Hello from UFO\nINFO - [WS] [AIP] Sent task request with platform: windows\nINFO - Executing 3 actions in total\nINFO - [WS] [AIP] Sent client result for prev_response_id: resp_abc123\nINFO - [WS] Task session_20251104_143022_abc123 completed\n```\n\n**Execution Flow:**\n\n```mermaid\nsequenceDiagram\n participant API as HTTP API\n participant Server\n participant Client\n participant App as Notepad\n \n API->>Server: POST /dispatch\n Server->>Server: Create Session\n Server-->>API: {session_id, status}\n \n Server->>Client: COMMAND\n Client->>App: Launch & Type\n App-->>Client: Done\n Client->>Server: COMMAND_RESULTS\n```\n\n---\n\n## ⚠️ Common Issues\n\n### 1. Connection Refused\n\n**Symptom:**\n```log\nERROR - [WS] Unexpected error: [Errno 10061] Connect call failed\nERROR - [WS] Max retries reached. Exiting.\n```\n\n**Root Causes:**\n\n| Cause | Verification | Solution |\n|-------|--------------|----------|\n| Server not running | `curl http://localhost:5000/api/health` | Start server first |\n| Wrong port | Check server startup logs | Use correct port (`--ws-server ws://...`) |\n| Firewall blocking | `telnet server 5000` | Allow port 5000 in firewall |\n| Server using `--local` flag | Check server CLI args | Connect from localhost only |\n\n**Solutions:**\n\n**Verify Server:**\n```bash\n# Check if server is running\ncurl http://localhost:5000/api/health\n \n# Expected response:\n# {\"status\": \"healthy\", \"uptime_seconds\": 123}\n```\n\n**Check Firewall:**\n```bash\n# Windows: Check if port is listening\nnetstat -an | findstr \":5000\"\n \n# Linux: Check if port is listening\nnetstat -tuln | grep :5000\n```\n\n**Fix Connection:**\n```bash\n# Ensure server and client match:\n# Server: --port 5000\n# Client: --ws-server ws://localhost:5000/ws\n```\n\n### 2. Registration Failed\n\n**Symptom:**\n```log\nERROR - [WS] [AIP] ❌ Failed to register as device_windows_001\nRuntimeError: Registration failed for device_windows_001\n```\n\n**Root Causes:**\n\n| Cause | Explanation | Solution |\n|-------|-------------|----------|\n| Duplicate client ID | Another device using same ID | Use unique `--client-id` |\n| Server rejecting connection | Server validation error | Check server logs for details |\n| Network interruption | Connection dropped during registration | Retry connection |\n| Device info collection error | Failed to gather system info | Check MCP server initialization |\n\n**Solutions:**\n\n**Check Duplicate IDs:**\n```bash\n# List all connected clients\ncurl http://localhost:5000/api/clients | grep client_id\n \n# If your ID appears, choose a different one\npython -m ufo.client.client --ws --client-id NEW_UNIQUE_ID\n```\n\n**Check Server Logs:**\n```bash\n# Server logs show detailed rejection reasons\n# Example: \"Client ID already exists\"\n# Example: \"Platform mismatch\"\n```\n\n### 3. Platform Detection Issues\n\n**Symptom:**\n```log\nWARNING - Platform not detected correctly\nWARNING - Defaulting to platform: unknown\n```\n\n**Solution:**\n\nExplicitly set the platform:\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://127.0.0.1:5000/ws \\\n --client-id my_device \\\n --platform windows # or 'linux'\n```\n\n**Platform Values:**\n\n| Value | OS | Auto-Detection |\n|-------|----|-|\n| `windows` | Windows 10/11, Server 2016+ | `platform.system() == \"Windows\"` |\n| `linux` | Ubuntu, Debian, RHEL, etc. | `platform.system() == \"Linux\"` |\n\n### 4. Heartbeat Timeout\n\n**Symptom:**\n```log\nERROR - [WS] Connection closed: ConnectionClosedError\nINFO - [WS] Reconnecting... (attempt 2/5)\n```\n\n**Root Causes:**\n\n| Cause | Description | Solution |\n|-------|-------------|----------|\n| Network instability | Wi-Fi dropouts, packet loss | Use wired connection |\n| Server crashed | Server process terminated | Restart server |\n| Proxy interference | Corporate proxy blocking WebSocket | Configure proxy bypass |\n| Firewall timeout | Idle connection timeout | Reduce heartbeat interval |\n\n**Solutions:**\n\n**Increase Retries:**\n```bash\n# For unreliable networks\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://server:5000/ws \\\n --client-id my_device \\\n --max-retries 20\n```\n\n**Check Network:**\n```bash\n# Test sustained connection\nping -t server # Windows\nping server # Linux (Ctrl+C to stop)\n```\n\n**Verify Server:**\n```bash\n# Check if server is still running\ncurl http://server:5000/api/health\n```\n\n---\n\n## 🌐 Multiple Devices\n\nConnect multiple devices to the same server for **fleet management** and **task distribution**.\n\n### Example Configuration\n\n**Device 1 (Windows Desktop):**\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5000/ws \\\n --client-id device_windows_desktop_001\n```\n\n**Device 2 (Linux Server):**\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5000/ws \\\n --client-id device_linux_server_001 \\\n --platform linux\n```\n\n**Device 3 (Windows Laptop):**\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5000/ws \\\n --client-id device_windows_laptop_002\n```\n\n### Verify All Connected\n\n```bash\ncurl http://192.168.1.100:5000/api/clients\n```\n\n**Expected Response:**\n\n```json\n{\n \"clients\": [\n {\n \"client_id\": \"device_windows_desktop_001\",\n \"type\": \"device\",\n \"platform\": \"windows\",\n \"uptime_seconds\": 120\n },\n {\n \"client_id\": \"device_linux_server_001\",\n \"type\": \"device\",\n \"platform\": \"linux\",\n \"uptime_seconds\": 95\n },\n {\n \"client_id\": \"device_windows_laptop_002\",\n \"type\": \"device\",\n \"platform\": \"windows\",\n \"uptime_seconds\": 45\n }\n ],\n \"total\": 3\n}\n```\n\n**Client ID Naming Convention:**\n\n```\ndevice____\n\nExamples:\n- device_windows_prod_datacenter1_001\n- device_linux_staging_cloud_aws_002\n- device_windows_dev_office_laptop_john\n```\n\n---\n\n## 🔧 Running as Background Service\n\n!!! tip \"Production Deployment\"\n For production use, run the client as a **system service** that starts automatically and restarts on failure.\n\n### Linux (systemd)\n\nCreate `/etc/systemd/system/ufo-client.service`:\n\n```ini\n[Unit]\nDescription=UFO Device Client - Execution Agent\nDocumentation=https://github.com/microsoft/UFO\nAfter=network-online.target\nWants=network-online.target\n\n[Service]\nType=simple\nUser=ufouser\nGroup=ufouser\nWorkingDirectory=/home/ufouser/UFO2\n\n# Environment variables (if needed)\nEnvironment=\"PYTHONUNBUFFERED=1\"\n\n# Main command\nExecStart=/usr/bin/python3 -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5000/ws \\\n --client-id device_linux_prod_01 \\\n --platform linux \\\n --log-level INFO\n\n# Restart policy\nRestart=always\nRestartSec=10\nStartLimitBurst=5\nStartLimitIntervalSec=300\n\n# Resource limits (optional)\nLimitNOFILE=65536\nMemoryLimit=2G\n\n# Logging\nStandardOutput=journal\nStandardError=journal\nSyslogIdentifier=ufo-client\n\n[Install]\nWantedBy=multi-user.target\n```\n\n**Enable and Start:**\n\n```bash\n# Reload systemd configuration\nsudo systemctl daemon-reload\n\n# Enable service (start on boot)\nsudo systemctl enable ufo-client\n\n# Start service now\nsudo systemctl start ufo-client\n\n# Check status\nsudo systemctl status ufo-client\n\n# View logs\nsudo journalctl -u ufo-client -f\n```\n\n**Service Management:**\n\n| Command | Purpose |\n|---------|---------|\n| `systemctl start ufo-client` | Start the service |\n| `systemctl stop ufo-client` | Stop the service |\n| `systemctl restart ufo-client` | Restart the service |\n| `systemctl status ufo-client` | Check service status |\n| `journalctl -u ufo-client -f` | Follow logs in real-time |\n| `systemctl disable ufo-client` | Disable auto-start |\n\n### Windows (NSSM)\n\n**NSSM** (Non-Sucking Service Manager) wraps any application as a Windows service.\n\n**1. Download NSSM:**\n\nDownload from [nssm.cc](https://nssm.cc/download)\n\n**2. Install Service:**\n\n```powershell\n# Install as service\nnssm install UFOClient \"C:\\Python310\\python.exe\" `\n \"-m\" \"ufo.client.client\" `\n \"--ws\" `\n \"--ws-server\" \"ws://192.168.1.100:5000/ws\" `\n \"--client-id\" \"device_windows_prod_01\" `\n \"--log-level\" \"INFO\"\n\n# Set working directory\nnssm set UFOClient AppDirectory \"C:\\UFO2\"\n\n# Set restart policy\nnssm set UFOClient AppExit Default Restart\nnssm set UFOClient AppRestartDelay 10000\n\n# Set logging\nnssm set UFOClient AppStdout \"C:\\UFO2\\logs\\client-stdout.log\"\nnssm set UFOClient AppStderr \"C:\\UFO2\\logs\\client-stderr.log\"\n```\n\n**3. Manage Service:**\n\n```powershell\n# Start service\nnssm start UFOClient\n\n# Check status\nnssm status UFOClient\n\n# Stop service\nnssm stop UFOClient\n\n# Remove service\nnssm remove UFOClient confirm\n```\n\n**Alternative: Windows Task Scheduler**\n\n```powershell\n# Create scheduled task to run on startup\n$action = New-ScheduledTaskAction -Execute \"python.exe\" `\n -Argument \"-m ufo.client.client --ws --ws-server ws://server:5000/ws --client-id device_win_01\"\n$trigger = New-ScheduledTaskTrigger -AtStartup\n$settings = New-ScheduledTaskSettingsSet -RestartCount 3 -RestartInterval (New-TimeSpan -Minutes 1)\n\nRegister-ScheduledTask -TaskName \"UFOClient\" `\n -Action $action `\n -Trigger $trigger `\n -Settings $settings `\n -User \"System\" `\n -RunLevel Highest\n```\n\n### PM2 (Cross-Platform)\n\n**PM2** is a cross-platform process manager with built-in load balancing, monitoring, and auto-restart.\n\n**1. Install PM2:**\n\n```bash\nnpm install -g pm2\n```\n\n**2. Create Ecosystem File (`ecosystem.config.js`):**\n\n```javascript\nmodule.exports = {\n apps: [{\n name: \"ufo-client\",\n script: \"python\",\n args: [\n \"-m\", \"ufo.client.client\",\n \"--ws\",\n \"--ws-server\", \"ws://192.168.1.100:5000/ws\",\n \"--client-id\", \"device_prod_01\",\n \"--log-level\", \"INFO\"\n ],\n cwd: \"/home/user/UFO2\",\n interpreter: \"none\",\n autorestart: true,\n watch: false,\n max_restarts: 10,\n min_uptime: \"10s\",\n restart_delay: 5000,\n env: {\n PYTHONUNBUFFERED: \"1\"\n }\n }]\n};\n```\n\n**3. Start with PM2:**\n\n```bash\n# Start from ecosystem file\npm2 start ecosystem.config.js\n\n# Or start directly\npm2 start \"python -m ufo.client.client --ws --ws-server ws://192.168.1.100:5000/ws --client-id device_001\" \\\n --name ufo-client\n\n# Save PM2 configuration\npm2 save\n\n# Enable startup script (auto-start on boot)\npm2 startup\n# Follow the instructions printed by the command\n\n# Monitor\npm2 monit\n\n# View logs\npm2 logs ufo-client\n```\n\n**PM2 Management:**\n\n| Command | Purpose |\n|---------|---------|\n| `pm2 list` | List all processes |\n| `pm2 start ufo-client` | Start process |\n| `pm2 stop ufo-client` | Stop process |\n| `pm2 restart ufo-client` | Restart process |\n| `pm2 delete ufo-client` | Remove process |\n| `pm2 logs ufo-client` | View logs |\n| `pm2 monit` | Real-time monitoring dashboard |\n\n---\n\n## 🏭 Production Deployment Best Practices\n\nFollow these best practices for reliable production deployments.\n\n### 1. Descriptive Client IDs\n\n```bash\n# ❌ Bad: Generic, non-unique\n--client-id client_001\n--client-id device1\n\n# ✅ Good: Descriptive, environment, location\n--client-id production_windows_datacenter1_rack3_slot1\n--client-id staging_linux_cloud_aws_us-east-1_worker2\n--client-id dev_windows_office_john_laptop\n```\n\n**ID Structure:**\n\n```\n___\n\n- environment: production, staging, development, test\n- platform: windows, linux\n- location: datacenter1, cloud_aws, office\n- identifier: unique number or name\n```\n\n### 2. Structured Logging\n\n**File Logging:**\n```bash\n# Redirect to log file with rotation\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://server:5000/ws \\\n --client-id device_prod_01 \\\n --log-level INFO \\\n > /var/log/ufo-client.log 2>&1\n```\n\n**Systemd Journal:**\n```bash\n# Already configured in systemd service\n# View logs:\njournalctl -u ufo-client -f --since \"1 hour ago\"\n```\n\n**Syslog:**\n```bash\n# Configure Python logging to send to syslog\n# Add to config_dev.yaml:\n# logging:\n# handlers:\n# syslog:\n# class: logging.handlers.SysLogHandler\n# address: /dev/log\n```\n\n### 3. Automatic Restart on Failure\n\n**Service Configuration:**\n\n| Platform | Mechanism | Restart Delay | Max Restarts |\n|----------|-----------|---------------|--------------|\n| Linux | systemd | 10 seconds | Unlimited (with rate limiting) |\n| Windows | NSSM | 10 seconds | Unlimited |\n| Cross-platform | PM2 | 5 seconds | 10 attempts, then manual |\n\n### 4. Health Monitoring\n\n**Monitoring Script:**\n\n```bash\n#!/bin/bash\n# check-ufo-client.sh\n\nCLIENT_ID=\"device_prod_01\"\nSERVER_URL=\"http://192.168.1.100:5000\"\n\n# Check if client is connected\nresponse=$(curl -s \"${SERVER_URL}/api/clients\" | grep -c \"${CLIENT_ID}\")\n\nif [ \"$response\" -eq \"0\" ]; then\n echo \"ALERT: Client ${CLIENT_ID} is not connected!\"\n # Send alert (email, Slack, PagerDuty, etc.)\n exit 1\nelse\n echo \"OK: Client ${CLIENT_ID} is connected\"\n exit 0\nfi\n```\n\n**Run via cron:**\n```cron\n# Check every 5 minutes\n*/5 * * * * /usr/local/bin/check-ufo-client.sh\n```\n\n### 5. Secure Communication\n\n!!! danger \"Production Security\"\n **Never expose clients to the internet without these security measures:**\n\n**Use WSS (WebSocket Secure):**\n\n```bash\n# Production: Encrypted WebSocket\n--ws-server wss://ufo-server.company.com/ws\n\n# Development only: Unencrypted\n--ws-server ws://localhost:5000/ws\n```\n\n**Server-Side TLS Configuration:**\n\n```bash\n# Server with TLS\npython -m ufo.server.app \\\n --port 5000 \\\n --ssl-cert /path/to/cert.pem \\\n --ssl-key /path/to/key.pem\n```\n\n**Network Security:**\n\n| Measure | Implementation |\n|---------|----------------|\n| **Firewall Rules** | Allow only server IP on port 5000 |\n| **VPN/Private Network** | Run server on internal network only |\n| **Authentication** | Implement client authentication (future feature) |\n| **Certificate Validation** | Verify server TLS certificates |\n\n---\n\n## 🔧 Troubleshooting Commands\n\nUse these commands to diagnose connection and execution issues.\n\n### Test Server Connectivity\n\n**HTTP Health Check:**\n```bash\ncurl http://localhost:5000/api/health\n \n# Expected response:\n# {\"status\": \"healthy\", \"uptime_seconds\": 3456}\n```\n\n**WebSocket Test:**\n```bash\n# Install wscat\nnpm install -g wscat\n \n# Test WebSocket connection\nwscat -c ws://localhost:5000/ws\n \n# You should see connection established\n# Send a test message (will likely be rejected, but connection works)\n```\n\n**Network Connectivity:**\n```bash\n# Test if server is reachable\nping 192.168.1.100\n \n# Test if port is open\ntelnet 192.168.1.100 5000 # Windows/Linux\nnc -zv 192.168.1.100 5000 # Linux/macOS\n```\n\n### Check Connected Clients\n\n```bash\n# List all connected clients\ncurl http://localhost:5000/api/clients | python -m json.tool\n\n# Check specific client\ncurl http://localhost:5000/api/clients | grep \"device_windows_001\"\n```\n\n### Monitor Client Logs\n\n**Increase Verbosity:**\n```bash\n# Enable DEBUG logging\npython -m ufo.client.client \\\n --ws \\\n --client-id my_device \\\n --log-level DEBUG\n```\n\n**Filter Logs:**\n```bash\n# Only show errors\npython -m ufo.client.client --ws --client-id my_device 2>&1 | grep ERROR\n \n# Only show connection events\npython -m ufo.client.client --ws --client-id my_device 2>&1 | grep -E \"Connect|Register\"\n```\n\n### Test Task Dispatch\n\n```bash\n# Dispatch simple test task\ncurl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"device_windows_001\",\n \"request\": \"List all files in the current directory\"\n }'\n```\n\n---\n\n## 🚀 Next Steps\n\n!!! tip \"Continue Learning\"\n Now that your client is connected and running tasks:\n\n**1. Understand Registration Flow**\n\nLearn how clients register with the server and exchange device profiles:\n\n👉 [UFO Client Overview](./overview.md)\n\n**2. Explore Device Information**\n\nDeep dive into what device information is collected and how it's used for task assignment:\n\n👉 [Device Info Provider](./device_info.md)\n\n**3. Master WebSocket Communication**\n\nUnderstand the AIP protocol and WebSocket message flow:\n\n👉 [WebSocket Client](./websocket_client.md)\n\n**4. Configure MCP Servers**\n\nLearn how to add custom tools and configure MCP servers:\n\n👉 [MCP Integration](../mcp/overview.md)\n\n**5. Study the AIP Protocol**\n\nDeep dive into message types, flow control, and error handling:\n\n👉 [AIP Protocol](../aip/overview.md)\n\n**6. Production Deployment**\n\nBest practices for running clients in production environments:\n\n👉 [Configuration Guide](../configuration/system/overview.md)\n"
},
{
"path": "documents/docs/client/ufo_client.md",
"content": "# 🎯 UFO Client\n\nThe **UFO Client** is the execution engine that receives commands from the server, routes them to appropriate tools via the CommandRouter, and aggregates results. It focuses on stateless command execution, delegating all decision-making to the server.\n\n## 📋 Overview\n\nThe UFO Client bridges network communication and local tool execution.\n\n**Key Capabilities:**\n\n| Capability | Description | Implementation |\n|------------|-------------|----------------|\n| **Command Execution** | Processes server commands deterministically | `execute_step()`, `execute_actions()` |\n| **Session Management** | Tracks session state and metadata | Session ID, agent/process/root names |\n| **Result Aggregation** | Collects and structures tool execution results | Returns `List[Result]` |\n| **Thread Safety** | Ensures safe concurrent execution | `asyncio.Lock` (`task_lock`) |\n| **State Management** | Maintains agent, process, and root names | Property setters with validation |\n| **Manager Coordination** | Orchestrates ComputerManager and MCPServerManager | `reset()` cascades to all managers |\n\nThe UFO Client follows a stateless execution philosophy:\n\n- Executes commands sent by the server\n- Routes commands to the appropriate tools\n- Returns execution results\n- Does **not** decide which commands to run\n- Does **not** interpret user requests\n- Does **not** store long-term state\n\n**Architectural Position:**\n\n```mermaid\ngraph LR\n subgraph Server[\"Server Side (Orchestration)\"]\n SRV[Agent Server]\n LLM[LLM Reasoning]\n end\n \n subgraph Network[\"Network Layer\"]\n WSC[WebSocket Client]\n end\n \n subgraph Client[\"Client Side (Execution)\"]\n UFC[UFO Client]\n CR[Command Router]\n Tools[MCP Tools]\n end\n \n SRV -->|Commands| WSC\n WSC -->|execute_step| UFC\n UFC -->|execute| CR\n CR -->|tool calls| Tools\n Tools -->|results| CR\n CR -->|results| UFC\n UFC -->|results| WSC\n WSC -->|results| SRV\n \n LLM -->|planning| SRV\n \n style SRV fill:#ffe0b2\n style UFC fill:#bbdefb\n style Tools fill:#c8e6c9\n```\n\n## 🏗️ Architecture\n\nThe UFO Client has a minimal API surface—just initialization, execution, and reset.\n\n### Component Structure\n\n```mermaid\ngraph TB\n subgraph \"UFOClient\"\n State[Session State]\n Execution[Execution Methods]\n Dependencies[Manager Dependencies]\n end\n \n subgraph \"Session State\"\n State1[session_id]\n State2[agent_name]\n State3[process_name]\n State4[root_name]\n State5[task_lock]\n end\n \n subgraph \"Execution Methods\"\n Exec1[execute_step]\n Exec2[execute_actions]\n Exec3[reset]\n end\n \n subgraph \"Dependencies\"\n Dep1[CommandRouter]\n Dep2[ComputerManager]\n Dep3[MCPServerManager]\n end\n \n State --> State1\n State --> State2\n State --> State3\n State --> State4\n State --> State5\n \n Execution --> Exec1\n Execution --> Exec2\n Execution --> Exec3\n \n Dependencies --> Dep1\n Dependencies --> Dep2\n Dependencies --> Dep3\n \n Exec1 --> Exec2\n Exec2 --> Dep1\n Exec3 --> Dep2\n Exec3 --> Dep3\n \n style State fill:#e3f2fd\n style Execution fill:#f1f8e9\n style Dependencies fill:#fff3e0\n```\n\n**Class Attributes:**\n\n| Attribute | Type | Purpose |\n|-----------|------|---------|\n| `mcp_server_manager` | `MCPServerManager` | Manages MCP server lifecycle |\n| `computer_manager` | `ComputerManager` | Manages computer instances (tool namespaces) |\n| `command_router` | `CommandRouter` | Routes commands to appropriate computers |\n| `task_lock` | `asyncio.Lock` | Ensures thread-safe execution |\n| `client_id` | `str` | Unique identifier for this client (default: `\"client_001\"`) |\n| `platform` | `str` | Platform type (`\"windows\"` or `\"linux\"`) - auto-detected if not provided |\n| `session_id` | `Optional[str]` | Current session identifier |\n| `agent_name` | `Optional[str]` | Active agent (e.g., `\"HostAgent\"`, `\"AppAgent\"`) |\n| `process_name` | `Optional[str]` | Process context (e.g., `\"notepad.exe\"`) |\n| `root_name` | `Optional[str]` | Root operation name |\n\n## 🚀 Initialization\n\nCreating a UFO Client requires two manager instances: MCPServerManager and ComputerManager.\n\n```python\nfrom ufo.client.ufo_client import UFOClient\nfrom ufo.client.computer import ComputerManager\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\n# 1. Initialize MCP Server Manager\nmcp_server_manager = MCPServerManager()\nmcp_server_manager.create_servers_from_config() # Load from config_dev.yaml\n\n# 2. Initialize Computer Manager\ncomputer_manager = ComputerManager(\n ufo_config.to_dict(),\n mcp_server_manager\n)\n\n# 3. Create UFO Client\nclient = UFOClient(\n mcp_server_manager=mcp_server_manager,\n computer_manager=computer_manager,\n client_id=\"device_windows_001\",\n platform=\"windows\"\n)\n```\n\n**Constructor Parameters:**\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `mcp_server_manager` | `MCPServerManager` | ✅ Yes | - | MCP server lifecycle manager |\n| `computer_manager` | `ComputerManager` | ✅ Yes | - | Computer instance manager |\n| `client_id` | `str` | No | `\"client_001\"` | Unique client identifier |\n| `platform` | `str` | No | Auto-detected | Platform type: `\"windows\"` or `\"linux\"` |\n\n**Initialization Side Effects:**\n\n1. Creates `CommandRouter` instance (delegates to ComputerManager)\n2. Initializes `task_lock` (`asyncio.Lock()`)\n3. Sets session state to `None` (session_id, agent_name, process_name, root_name)\n\n## 📊 Session State Management\n\nThe UFO Client maintains contextual metadata for the current execution session.\n\n### Session ID\n\n**Purpose:** Unique identifier for the current task session\n\n```python\n# Set session ID (typically set by server)\nclient.session_id = \"session_20251104_143022_abc123\"\n\n# Get session ID\ncurrent_session = client.session_id # \"session_20251104_143022_abc123\"\n\n# Clear session ID\nclient.reset() # Sets session_id to None\n```\n\n**Validation:**\n\n```python\n# ✅ Valid\nclient.session_id = \"session_123\"\nclient.session_id = None\n\n# ❌ Invalid - raises ValueError\nclient.session_id = 12345 # Not a string\n```\n\n### Agent Name\n\n**Purpose:** Identifies the active agent (HostAgent, AppAgent, etc.)\n\n```python\n# Set agent name (from server message)\nclient.agent_name = \"HostAgent\"\n\n# Get agent name\nagent = client.agent_name # \"HostAgent\"\n```\n\n**Common Agent Names:**\n\n| Agent Name | Purpose |\n|------------|---------|\n| `HostAgent` | OS-level operations (start apps, manage files) |\n| `AppAgent` | Application-specific operations (UI automation) |\n| `FollowerAgent` | Follow predefined workflows |\n\n### Process Name\n\n**Purpose:** Identifies the process context\n\n```python\n# Set process name (from server message)\nclient.process_name = \"notepad.exe\"\n\n# Get process name\nprocess = client.process_name # \"notepad.exe\"\n```\n\n**Usage:** Helps route commands to the correct application context\n\n### Root Name\n\n**Purpose:** Identifies the root operation name\n\n```python\n# Set root name (from server message)\nclient.root_name = \"open_application\"\n\n# Get root name\nroot = client.root_name # \"open_application\"\n```\n\n**Property Validation:**\n\nAll properties validate their inputs:\n\n```python\ntry:\n client.agent_name = 123 # Not a string\nexcept ValueError as e:\n print(e) # \"Agent name must be a string or None.\"\n```\n\n**Validation Table:**\n\n| Property | Valid Types | Raises on Invalid |\n|----------|-------------|-------------------|\n| `session_id` | `str`, `None` | `ValueError` |\n| `agent_name` | `str`, `None` | `ValueError` |\n| `process_name` | `str`, `None` | `ValueError` |\n| `root_name` | `str`, `None` | `ValueError` |\n\n## ⚙️ Command Execution\n\n### Execute Step (Main Entry Point)\n\n`execute_step()` processes one complete server message, extracting metadata and executing all commands.\n\n**Signature:**\n\n```python\nasync def execute_step(self, response: ServerMessage) -> List[Result]:\n \"\"\"\n Perform a single step execution.\n :param response: The ServerMessage instance to process.\n :return: A list of Result instances.\n \"\"\"\n```\n\n**Execution Flow:**\n\n```mermaid\nsequenceDiagram\n participant WSC as WebSocket Client\n participant UFC as UFO Client\n participant CR as Command Router\n participant Tools\n \n WSC->>UFC: execute_step(ServerMessage)\n \n Note over UFC: 1. Extract Metadata\n UFC->>UFC: self.agent_name = response.agent_name\n UFC->>UFC: self.process_name = response.process_name\n UFC->>UFC: self.root_name = response.root_name\n \n Note over UFC: 2. Execute Actions\n UFC->>UFC: execute_actions(response.actions)\n \n UFC->>CR: command_router.execute( agent_name, process_name, root_name, commands)\n \n CR->>Tools: Route commands to tools\n Tools-->>CR: Results\n CR-->>UFC: List[Result]\n \n UFC-->>WSC: List[Result]\n```\n\n**Implementation:**\n\n```python\nasync def execute_step(self, response: ServerMessage) -> List[Result]:\n \"\"\"Perform a single step execution.\"\"\"\n \n # Extract metadata from server response\n self.agent_name = response.agent_name\n self.process_name = response.process_name\n self.root_name = response.root_name\n \n # Execute actions\n action_results = await self.execute_actions(response.actions)\n \n return action_results\n```\n\n**Example Usage:**\n\n```python\nfrom aip.messages import ServerMessage\n\n# Receive server message\nserver_response = ServerMessage.model_validate_json(msg)\n\n# Execute step\naction_results = await client.execute_step(server_response)\n\n# action_results is List[Result]\nfor result in action_results:\n print(f\"Action: {result.action}, Status: {result.status}\")\n```\n\n### Execute Actions\n\n`execute_actions()` executes a list of commands via the CommandRouter.\n\n**Signature:**\n\n```python\nasync def execute_actions(self, commands: Optional[List[Command]]) -> List[Result]:\n \"\"\"\n Execute the actions provided by the server.\n :param commands: List of actions to execute.\n :returns: Results of the executed actions.\n \"\"\"\n```\n\n**Implementation:**\n\n```python\nasync def execute_actions(self, commands: Optional[List[Command]]) -> List[Result]:\n \"\"\"Execute the actions provided by the server.\"\"\"\n \n action_results = []\n \n if commands:\n self.logger.info(f\"Executing {len(commands)} actions in total\")\n \n # Delegate to CommandRouter\n action_results = await self.command_router.execute(\n agent_name=self.agent_name,\n process_name=self.process_name,\n root_name=self.root_name,\n commands=commands\n )\n \n return action_results\n```\n\n**Example:**\n\n```python\nfrom aip.messages import Command\n\ncommands = [\n Command(\n action=\"click\",\n parameters={\"control_label\": \"Start\", \"x\": 10, \"y\": 10}\n ),\n Command(\n action=\"type_text\",\n parameters={\"text\": \"notepad\"}\n ),\n Command(\n action=\"press_key\",\n parameters={\"key\": \"enter\"}\n )\n]\n\n# Execute all commands\nresults = await client.execute_actions(commands)\n\n# results contains Result object for each command\n```\n\n**Command Execution Table:**\n\n| Step | Action | Component |\n|------|--------|-----------|\n| 1 | Receive commands | UFO Client |\n| 2 | Log command count | UFO Client |\n| 3 | Call CommandRouter | UFO Client |\n| 4 | Route to Computer | CommandRouter |\n| 5 | Execute via MCP | Computer |\n| 6 | Collect results | CommandRouter |\n| 7 | Return results | UFO Client |\n\nSee [Computer Manager](./computer_manager.md) for command routing details.\n\n## 🔄 State Reset\n\n!!!warning \"Critical for Multi-Task Execution\"\n Always reset state between tasks to prevent data leakage between sessions.\n\n**Signature:**\n\n```python\ndef reset(self):\n \"\"\"Reset session state and dependent managers.\"\"\"\n```\n\n**Implementation:**\n\n```python\ndef reset(self):\n \"\"\"Reset session state and dependent managers.\"\"\"\n \n # Clear session state\n self._session_id = None\n self._agent_name = None\n self._process_name = None\n self._root_name = None\n \n # Reset managers\n self.computer_manager.reset()\n self.mcp_server_manager.reset()\n \n self.logger.info(\"Client state has been reset.\")\n```\n\n**Reset Cascade:**\n\n```mermaid\ngraph TD\n Reset[client.reset]\n \n Reset --> S1[session_id = None]\n Reset --> S2[agent_name = None]\n Reset --> S3[process_name = None]\n Reset --> S4[root_name = None]\n \n Reset --> M1[computer_manager.reset]\n Reset --> M2[mcp_server_manager.reset]\n \n M1 --> C1[Clear computer instances]\n M2 --> M3[Reset MCP servers]\n \n style Reset fill:#ffcdd2\n style M1 fill:#fff9c4\n style M2 fill:#fff9c4\n```\n\n**When to Reset:**\n\n| Scenario | Why Reset |\n|----------|-----------|\n| **Before starting new task** | Clear previous task state |\n| **On task completion** | Prepare for next task |\n| **On task failure** | Clean up failed state |\n| **On server disconnection** | Reset to known good state |\n\n**Note:** The WebSocket client automatically calls `reset()` before starting new tasks:\n\n```python\nasync with self.ufo_client.task_lock:\n self.ufo_client.reset() # Automatic\n await self.task_protocol.send_task_request(...)\n```\n\n## 🔒 Thread Safety\n\nThe UFO Client uses `asyncio.Lock` to prevent concurrent state modifications.\n\n**Lock Implementation:**\n\n```python\n# In UFOClient.__init__\nself.task_lock = asyncio.Lock()\n```\n\n**Usage in WebSocket Client:**\n\n```python\n# In WebSocket client\nasync with client.task_lock:\n client.reset()\n await client.execute_step(server_response)\n```\n\n**Protected Operations:**\n\n| Operation | Protected By | Reason |\n|-----------|--------------|--------|\n| Session state modifications | `task_lock` | Prevent race conditions |\n| Command execution | `task_lock` | Ensure one task at a time |\n| State reset | `task_lock` | Atomic reset operation |\n\n!!!warning \"Single Task Execution\"\n The lock ensures only **one task executes at a time**. Attempting concurrent execution will block until the lock is released.\n\n## 📋 Complete Execution Pipeline\n\n```mermaid\nsequenceDiagram\n participant Server\n participant WSC as WebSocket Client\n participant UFC as UFO Client\n participant CR as Command Router\n participant CM as Computer Manager\n participant Comp as Computer\n participant Tool as MCP Tool\n \n Note over Server,Tool: Full Execution Pipeline\n \n Server->>WSC: COMMAND message\n WSC->>UFC: execute_step(ServerMessage)\n \n Note over UFC: Extract Metadata\n UFC->>UFC: agent_name = \"HostAgent\"\n UFC->>UFC: process_name = \"explorer.exe\"\n UFC->>UFC: root_name = \"navigate\"\n \n Note over UFC: Execute Actions\n UFC->>CR: execute(agent, process, root, commands)\n \n CR->>CM: Route commands\n CM->>Comp: Get computer instance\n Comp->>Tool: Execute tool\n \n Tool-->>Comp: Result\n Comp-->>CM: Result\n CM-->>CR: List[Result]\n CR-->>UFC: List[Result]\n \n UFC-->>WSC: List[Result]\n WSC->>Server: COMMAND_RESULTS (via AIP)\n```\n\n## ⚠️ Error Handling\n\n### Command Execution Errors\n\nIndividual command failures are captured in `Result` objects, not thrown as exceptions.\n\n**Error Result Structure:**\n\n```python\nfrom aip.messages import Result, ResultStatus\n\nerror_result = Result(\n action=\"click\",\n status=ResultStatus.ERROR,\n error_message=\"Control not found\",\n observation=\"Failed to locate control with label 'Start'\"\n)\n```\n\n**Handling Execution Errors:**\n\n```python\ntry:\n results = await client.execute_actions(commands)\n \n # Check each result\n for result in results:\n if result.status == ResultStatus.ERROR:\n logger.error(f\"Action {result.action} failed: {result.error_message}\")\n else:\n logger.info(f\"Action {result.action} succeeded\")\n \nexcept Exception as e:\n # Unexpected error (not tool failure)\n logger.error(f\"Command execution failed: {e}\", exc_info=True)\n```\n\n### Property Validation Errors\n\n```python\ntry:\n client.session_id = 12345 # Invalid type\nexcept ValueError as e:\n logger.error(f\"Invalid session ID: {e}\")\n # ValueError: Session ID must be a string or None.\n```\n\n**Error Handling Table:**\n\n| Error Type | Raised By | Handling |\n|------------|-----------|----------|\n| Tool execution error | MCP tools | Captured in `Result.error_message` |\n| Property validation error | Property setters | `ValueError` exception |\n| Unexpected errors | Any component | Logged, may propagate |\n\n## 📝 Logging\n\nThe UFO Client logs all major events for debugging and monitoring.\n\n**Log Examples:**\n\n**Initialization:**\n\n```log\nINFO - UFO Client initialized for platform: windows\n```\n\n**Session State Changes:**\n\n```log\nINFO - Session ID set to: session_20251104_143022_abc123\nINFO - Agent name set to: HostAgent\nINFO - Process name set to: notepad.exe\nINFO - Root name set to: open_application\n```\n\n**Execution:**\n\n```log\nINFO - Executing 5 actions in total\n```\n\n**Reset:**\n\n```log\nINFO - Client state has been reset.\n```\n\n**Log Level Recommendations:**\n\n| Environment | Level | Rationale |\n|-------------|-------|-----------|\n| Development | `DEBUG` | See all operations |\n| Staging | `INFO` | Track execution flow |\n| Production | `INFO` | Monitor without spam |\n| Troubleshooting | `DEBUG` | Diagnose issues |\n\n## 💡 Usage Example\n\n### Complete Workflow\n\nThis example shows how to use the UFO Client in a typical workflow.\n\n```python\nimport asyncio\nfrom ufo.client.ufo_client import UFOClient\nfrom aip.messages import ServerMessage, Command, ServerMessageType, TaskStatus\n\nasync def main():\n # 1. Initialize client\n client = UFOClient(\n mcp_server_manager=mcp_manager,\n computer_manager=computer_manager,\n client_id=\"device_windows_001\",\n platform=\"windows\"\n )\n \n # 2. Simulate server message\n server_msg = ServerMessage(\n type=ServerMessageType.COMMAND,\n session_id=\"session_123\",\n response_id=\"resp_456\",\n agent_name=\"HostAgent\",\n process_name=\"explorer.exe\",\n root_name=\"navigate_folder\",\n actions=[\n Command(action=\"click\", parameters={\"label\": \"File\"}),\n Command(action=\"click\", parameters={\"label\": \"New Folder\"})\n ],\n status=TaskStatus.PROCESSING\n )\n \n # 3. Execute step\n async with client.task_lock: # Thread-safe execution\n results = await client.execute_step(server_msg)\n \n # 4. Process results\n for result in results:\n print(f\"Action: {result.action}\")\n print(f\"Status: {result.status}\")\n print(f\"Observation: {result.observation}\")\n if result.status == ResultStatus.ERROR:\n print(f\"Error: {result.error_message}\")\n \n # 5. Reset for next task\n client.reset()\n\nasyncio.run(main())\n```\n\n## ✅ Best Practices\n\n### Development Best Practices\n\n**1. Always Reset Between Tasks**\n\n```python\nasync with client.task_lock:\n client.reset() # Clear previous state\n await client.execute_step(new_server_response)\n```\n\n**2. Use Property Setters (Not Direct Assignment)**\n\n```python\n# ✅ Good - validates input\nclient.session_id = \"session_123\"\n\n# ❌ Bad - bypasses validation\nclient._session_id = \"session_123\"\n```\n\n**3. Log Execution Progress**\n\n```python\nself.logger.info(f\"Executing {len(commands)} actions for {self.agent_name}\")\n```\n\n**4. Handle Errors Gracefully**\n\n```python\ntry:\n results = await client.execute_actions(commands)\nexcept Exception as e:\n self.logger.error(f\"Execution failed: {e}\", exc_info=True)\n # Error is also captured in results\n```\n\n### Production Best Practices\n\n**1. Use Thread Locks Consistently**\n\n```python\n# Always use task_lock for state operations\nasync with client.task_lock:\n client.reset()\n results = await client.execute_step(msg)\n```\n\n**2. Monitor Execution Times**\n\n```python\nimport time\n\nstart = time.time()\nresults = await client.execute_actions(commands)\nduration = time.time() - start\n\nif duration > 60: # Alert if > 1 minute\n logger.warning(f\"Slow execution: {duration}s for {len(commands)} commands\")\n```\n\n**3. Validate Results**\n\n```python\n# Check for failures\nfailed_actions = [r for r in results if r.status == ResultStatus.ERROR]\nif failed_actions:\n logger.error(f\"{len(failed_actions)} actions failed\")\n # Report to monitoring system\n```\n\n## 🔗 Integration Points\n\n### WebSocket Client Integration\n\nThe WebSocket client uses UFO Client for all command execution.\n\n**Integration:**\n\n```python\n# In WebSocket client\naction_results = await self.ufo_client.execute_step(server_response)\n```\n\nSee [WebSocket Client](./websocket_client.md) for communication details.\n\n### Command Router Integration\n\nThe UFO Client delegates all execution to the CommandRouter.\n\n**Integration:**\n\n```python\naction_results = await self.command_router.execute(\n agent_name=self.agent_name,\n process_name=self.process_name,\n root_name=self.root_name,\n commands=commands\n)\n```\n\nSee [Computer Manager](./computer_manager.md) for routing details.\n\n### Computer Manager Integration\n\nThe Computer Manager maintains computer instances for tool execution.\n\n**Integration:**\n\n```python\n# Reset cascades to computer manager\nself.computer_manager.reset()\n```\n\nSee [Computer Manager](./computer_manager.md) for management details.\n\n### MCP Server Manager Integration\n\nThe MCP Server Manager handles MCP server creation and cleanup.\n\n**Integration:**\n\n```python\n# Reset cascades to MCP server manager\nself.mcp_server_manager.reset()\n```\n\nSee [MCP Integration](./mcp_integration.md) for MCP details.\n\n## 🚀 Next Steps\n\n**Continue Learning**\n\n1. **Understand Network Communication** - Learn how the WebSocket client uses UFO Client: [WebSocket Client](./websocket_client.md)\n\n2. **Explore Command Routing** - See how commands are routed to the right tools: [Computer Manager](./computer_manager.md)\n\n3. **Study Device Profiling** - Understand device information collection: [Device Info Provider](./device_info.md)\n\n4. **Learn About MCP Integration** - Deep dive into MCP server management: [MCP Integration](./mcp_integration.md)\n\n5. **Master AIP Messages** - Understand message structures: [AIP Messages](../aip/messages.md)"
},
{
"path": "documents/docs/client/websocket_client.md",
"content": "# 🔌 WebSocket Client\n\nThe **WebSocket Client** implements the **AIP (Agent Interaction Protocol)** for reliable, bidirectional communication between device clients and the Agent Server. It provides the low-level communication infrastructure for UFO device clients.\n\n## 📋 Overview\n\nThe WebSocket client handles all network communication aspects, allowing the UFO Client to focus on task execution.\n\n**Key Responsibilities:**\n\n| Capability | Description | Implementation |\n|------------|-------------|----------------|\n| **Connection Management** | Persistent WebSocket connection with automatic retry | Exponential backoff, configurable max retries |\n| **AIP Protocol Implementation** | Structured message handling via Registration, Heartbeat, Task Execution | Three protocol handlers |\n| **Device Registration** | Automatic registration with device profile on connect | Push model (proactive info collection) |\n| **Heartbeat Monitoring** | Regular keepalive messages for connection health | Configurable interval (default: 30s) |\n| **Message Routing** | Dispatch incoming messages to appropriate handlers | Type-based routing |\n| **Error Handling** | Graceful error recovery and reporting | Retry logic, error propagation via AIP |\n\n**Message Flow Overview:**\n\n```mermaid\ngraph LR\n subgraph \"Client Side\"\n WSC[WebSocket Client]\n AIP[AIP Protocols]\n UFC[UFO Client]\n end\n \n subgraph \"Network\"\n WS[WebSocket Connection]\n end\n \n subgraph \"Server Side\"\n Server[Agent Server]\n end\n \n WSC <-->|AIP Messages| AIP\n AIP <-->|WebSocket| WS\n WS <-->|TCP/IP| Server\n WSC -->|Delegate Execution| UFC\n \n style WSC fill:#bbdefb\n style AIP fill:#c8e6c9\n style Server fill:#ffe0b2\n```\n\n## 🏗️ Architecture\n\nThe WebSocket client is organized into distinct layers for connection management, protocol handling, and message routing.\n\n### Component Structure\n\n```mermaid\ngraph TB\n subgraph \"UFOWebSocketClient\"\n CM[Connection Management Layer]\n PH[Protocol Handler Layer]\n MR[Message Routing Layer]\n end\n \n subgraph \"Connection Management\"\n CM1[connect_and_listen]\n CM2[Retry Logic]\n CM3[State Tracking]\n end\n \n subgraph \"AIP Protocols\"\n PH1[RegistrationProtocol]\n PH2[HeartbeatProtocol]\n PH3[TaskExecutionProtocol]\n end\n \n subgraph \"Message Handlers\"\n MR1[recv_loop]\n MR2[handle_message]\n MR3[handle_commands]\n MR4[handle_task_end]\n end\n \n CM --> CM1\n CM --> CM2\n CM --> CM3\n \n PH --> PH1\n PH --> PH2\n PH --> PH3\n \n MR --> MR1\n MR --> MR2\n MR --> MR3\n MR --> MR4\n \n CM1 --> PH\n PH --> MR\n \n style CM fill:#e3f2fd\n style PH fill:#f1f8e9\n style MR fill:#fff3e0\n```\n\n### Class Structure\n\n| Component | Type | Purpose |\n|-----------|------|---------|\n| **UFOWebSocketClient** | Main Class | Orchestrates all WebSocket communication |\n| **WebSocketTransport** | AIP Component | Low-level WebSocket send/receive |\n| **RegistrationProtocol** | AIP Protocol | Client registration messages |\n| **HeartbeatProtocol** | AIP Protocol | Connection keepalive messages |\n| **TaskExecutionProtocol** | AIP Protocol | Task request/result messages |\n\n---\n\n## 🔄 Connection Lifecycle\n\n### Initialization & Connection Flow\n\n```mermaid\nsequenceDiagram\n participant Main as Client Main\n participant WSC as WebSocket Client\n participant WS as WebSocket\n participant Server\n \n Note over Main: 1. Initialization\n Main->>WSC: Create UFOWebSocketClient(ws_url, ufo_client)\n WSC->>WSC: Initialize attributes (max_retries=3, timeout=120)\n \n Note over WSC,Server: 2. Connection Attempt\n WSC->>WS: websockets.connect(ws_url)\n WS->>Server: TCP Handshake\n Server-->>WS: WebSocket Upgrade\n WS-->>WSC: Connection Established\n \n Note over WSC,Server: 3. AIP Protocol Initialization\n WSC->>WSC: Create WebSocketTransport(ws)\n WSC->>WSC: Create RegistrationProtocol(transport)\n WSC->>WSC: Create HeartbeatProtocol(transport)\n WSC->>WSC: Create TaskExecutionProtocol(transport)\n \n Note over WSC,Server: 4. Device Registration\n WSC->>WSC: Collect Device Info\n WSC->>Server: REGISTRATION (via AIP)\n Server-->>WSC: REGISTRATION_ACK\n WSC->>WSC: Set connected_event\n \n Note over WSC,Server: 5. Message Handling\n par Receive Loop\n loop Continuous\n Server->>WSC: Server Messages\n WSC->>WSC: Route to Handlers\n end\n and Heartbeat Loop\n loop Every 30s\n WSC->>Server: HEARTBEAT\n Server-->>WSC: HEARTBEAT_ACK\n end\n end\n```\n\n### Initialization Code\n\nCreating a WebSocket client:\n\n```python\nfrom ufo.client.websocket import UFOWebSocketClient\nfrom ufo.client.ufo_client import UFOClient\n\n# Create UFO client (execution engine)\nufo_client = UFOClient(\n mcp_server_manager=mcp_manager,\ncomputer_manager=computer_manager,\n client_id=\"device_windows_001\",\n platform=\"windows\"\n)\n\n# Create WebSocket client (communication layer)\nws_client = UFOWebSocketClient(\n ws_url=\"ws://localhost:5000/ws\",\n ufo_client=ufo_client,\n max_retries=3, # Default: 3 attempts\n timeout=120 # Heartbeat interval in seconds (default: 120)\n)\n\n# Connect and start listening (blocking call)\nawait ws_client.connect_and_listen()\n```\n\n**Constructor Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `ws_url` | `str` | Required | WebSocket server URL (e.g., `ws://localhost:5000/ws`) |\n| `ufo_client` | `UFOClient` | Required | UFO client instance for command execution |\n| `max_retries` | `int` | `3` | Maximum connection retry attempts |\n| `timeout` | `float` | `120` | Heartbeat interval in seconds (passed to `heartbeat_loop()`) |\n\n**Note:** The `timeout` parameter is passed to `heartbeat_loop(interval)` to control heartbeat frequency. While `heartbeat_loop()` has a default of 30s in its signature, the client constructor uses 120s which is passed when calling the method.\n\n### Connection Establishment Details\n\nThe client uses specific WebSocket parameters optimized for long-running task execution:\n\n**WebSocket Connection Parameters:**\n\n```python\nasync with websockets.connect(\n self.ws_url,\n ping_interval=20, # Send WebSocket ping every 20 seconds\n ping_timeout=180, # Wait up to 3 minutes for pong response\n close_timeout=10, # 10 second close handshake timeout\n max_size=100 * 1024 * 1024 # 100MB max message size\n) as ws:\n # Connection established\n```\n\n**Parameter Rationale:**\n\n| Parameter | Value | Reason |\n|-----------|-------|--------|\n| `ping_interval` | **20 seconds** | Frequent keepalive to detect connection loss quickly |\n| `ping_timeout` | **180 seconds** | Tolerates long-running operations (e.g., complex tasks) |\n| `close_timeout` | **10 seconds** | Quick cleanup on intentional disconnect |\n| `max_size` | **100 MB** | Supports large screenshots, logs, file transfers |\n\n**Note:** The 180-second `ping_timeout` ensures the connection stays alive during lengthy tool executions (up to 100 minutes per tool).\n\n## 📝 Registration Flow\n\n### Device Information Collection\n\nUFO uses a **push model** for device information: clients proactively send their profile during registration, rather than waiting for the server to request it. This reduces latency for constellation (multi-client) scenarios.\n\n**Device Info Collection:**\n\n```python\nfrom ufo.client.device_info_provider import DeviceInfoProvider\n\n# Collect comprehensive system information\nsystem_info = DeviceInfoProvider.collect_system_info(\n client_id=self.ufo_client.client_id,\n custom_metadata=None # Server adds custom metadata if configured\n)\n\n# System info includes:\n# - platform (windows/linux/darwin)\n# - os_version\n# - cpu_count\n# - memory_total_gb\n# - hostname\n# - ip_address\n# - supported_features\n# - platform_type\n```\n\n**Metadata Structure:**\n\n```python\nmetadata = {\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"Windows-10-10.0.19045\",\n \"cpu_count\": 8,\n \"memory_total_gb\": 16.0,\n \"hostname\": \"DESKTOP-ABC123\",\n \"ip_address\": \"192.168.1.100\",\n # ... additional fields\n },\n \"registration_time\": \"2025-11-05T14:30:00.123Z\"\n}\n```\n\nSee [Device Info Provider](./device_info.md) for complete field descriptions.\n\n### Registration Message Exchange\n\n```mermaid\nsequenceDiagram\n participant Client\n participant AIP as AIP Registration Protocol\n participant Server\n \n Note over Client: Collect Device Info\n Client->>Client: DeviceInfoProvider.collect_system_info()\n \n Note over Client,Server: Registration Request\n Client->>AIP: register_as_device( device_id, metadata, platform)\n AIP->>Server: REGISTRATION {device_id, metadata, platform}\n \n Note over Server: Validate & Store\n Server->>Server: Check for duplicate ID\n Server->>Server: Store device info\n Server->>Server: Add to client registry\n \n Note over Client,Server: Registration Response\n Server-->>AIP: REGISTRATION_ACK {success: true}\n AIP-->>Client: success = True\n \n Client->>Client: Set connected_event\n Client->>Client: Log success\n```\n\n**Registration Code:**\n\n```python\nasync def register_client(self):\n \"\"\"Send client_id and device system information to server.\"\"\"\n \n # Collect device info\n try:\n system_info = DeviceInfoProvider.collect_system_info(\n self.ufo_client.client_id,\n custom_metadata=None\n )\n metadata = {\n \"system_info\": system_info.to_dict(),\n \"registration_time\": datetime.datetime.now(\n datetime.timezone.utc\n ).isoformat(),\n }\n self.logger.info(\n f\"[WS] \\[AIP] Collected device info: platform={system_info.platform}, \"\n f\"cpu={system_info.cpu_count}, memory={system_info.memory_total_gb}GB\"\n )\n except Exception as e:\n self.logger.error(f\"[WS] \\[AIP] Error collecting device info: {e}\")\n # Continue with minimal metadata\n metadata = {\n \"registration_time\": datetime.datetime.now(\n datetime.timezone.utc\n ).isoformat(),\n }\n \n # Use AIP RegistrationProtocol\n success = await self.registration_protocol.register_as_device(\n device_id=self.ufo_client.client_id,\n metadata=metadata,\n platform=self.ufo_client.platform\n )\n \n if success:\n self.connected_event.set() # Signal successful registration\n self.logger.info(f\"[WS] \\[AIP] ✅ Successfully registered as {self.ufo_client.client_id}\")\n else:\n self.logger.error(f\"[WS] \\[AIP] ❌ Failed to register as {self.ufo_client.client_id}\")\n raise RuntimeError(f\"Registration failed for {self.ufo_client.client_id}\")\n```\n\n### Registration Outcomes\n\n**Success Scenario:**\n\n```log\nINFO - [WS] \\[AIP] Collected device info: platform=windows, cpu=8, memory=16.0GB\nINFO - [WS] \\[AIP] Attempting to register as device_windows_001\nINFO - [WS] \\[AIP] ✅ Successfully registered as device_windows_001\n```\n\n- `connected_event` is set (allows task requests)\n- Client enters message handling loops\n\n**Failure Scenario:**\n\n```log\nERROR - [WS] \\[AIP] ❌ Failed to register as device_windows_001\nRuntimeError: Registration failed for device_windows_001\n```\n\n- Connection is closed\n- Retry logic engages (exponential backoff)\n\n**Common Failure Causes:**\n\n| Cause | Server Behavior | Client Action |\n|-------|----------------|---------------|\n| Duplicate client ID | Reject registration | Change client ID, retry |\n| Server capacity limit | Reject registration | Wait and retry later |\n| Network interruption | Timeout | Automatic retry with backoff |\n| Invalid platform | Reject registration | Fix platform parameter |\n\n---\n\n## 💓 Heartbeat Mechanism\n\nHeartbeats prove the client is still alive and responsive, allowing the server to detect disconnected clients quickly.\n\n### Heartbeat Loop Implementation\n\n**Default Configuration:**\n\n| Parameter | Value | Configurable |\n|-----------|-------|--------------|\n| **Interval** | 30 seconds | ✅ Yes (function parameter) |\n| **Protocol** | AIP HeartbeatProtocol | No |\n| **Error Handling** | Break loop on failure | No |\n\n**Heartbeat Code:**\n\n```python\nasync def heartbeat_loop(self, interval: float = 30) -> None:\n \"\"\"\n Send periodic heartbeat messages using AIP HeartbeatProtocol.\n :param interval: Interval between heartbeats in seconds (default: 30)\n \"\"\"\n while True:\n await asyncio.sleep(interval)\n try:\n await self.heartbeat_protocol.send_heartbeat(\n self.ufo_client.client_id\n )\n self.logger.debug(\"[WS] \\[AIP] Heartbeat sent\")\n except (ConnectionError, IOError) as e:\n self.logger.debug(\n f\"[WS] \\[AIP] Heartbeat failed (connection closed): {e}\"\n )\n break # Exit loop if connection is closed\n```\n\n**Customizing Heartbeat Interval:**\n\nAdjust the interval when calling the heartbeat loop:\n\n```python\n# In handle_messages():\nawait asyncio.gather(\n self.recv_loop(),\n self.heartbeat_loop(interval=60) # Custom 60-second interval\n)\n```\n\n### Heartbeat Message Structure\n\n**Client → Server (Heartbeat):**\n\n```json\n{\n \"type\": \"HEARTBEAT\",\n \"client_id\": \"device_windows_001\",\n \"timestamp\": \"2025-11-05T14:30:22.123Z\"\n}\n```\n\n**Server → Client (Heartbeat Ack - Optional):**\n\n```json\n{\n \"type\": \"HEARTBEAT\",\n \"timestamp\": \"2025-11-05T14:30:22.456Z\"\n}\n```\n\n### Heartbeat State Diagram\n\n```mermaid\nstateDiagram-v2\n [*] --> Sleeping\n Sleeping --> SendingHeartbeat: After interval (30s)\n SendingHeartbeat --> Success: Sent successfully\n SendingHeartbeat --> Failed: Connection error\n \n Success --> Sleeping: Continue loop\n Failed --> [*]: Exit loop\n \n note right of Sleeping\n Wait for interval duration\n (default: 30 seconds)\n end note\n \n note right of Failed\n Connection closed\n recv_loop will also exit\n Outer retry logic activates\n end note\n```\n\n---\n\n## 📨 Message Handling\n\n### Message Router\n\nAll incoming messages are validated against the AIP schema and routed based on their `type` field.\n\n**Message Dispatcher Code:**\n\n```python\nasync def handle_message(self, msg: str):\n \"\"\"Dispatch messages based on their type.\"\"\"\n try:\n # Parse and validate message\n data = ServerMessage.model_validate_json(msg)\n msg_type = data.type\n \n self.logger.info(f\"[WS] Received message: {data}\")\n \n # Route by type\n if msg_type == ServerMessageType.TASK:\n await self.start_task(data.user_request, data.task_name)\n elif msg_type == ServerMessageType.HEARTBEAT:\n self.logger.info(\"[WS] Heartbeat received\")\n elif msg_type == ServerMessageType.TASK_END:\n await self.handle_task_end(data)\n elif msg_type == ServerMessageType.ERROR:\n self.logger.error(f\"[WS] Server error: {data.error}\")\n elif msg_type == ServerMessageType.COMMAND:\n await self.handle_commands(data)\n else:\n self.logger.warning(f\"[WS] Unknown message type: {msg_type}\")\n \n except Exception as e:\n self.logger.error(f\"[WS] Error handling message: {e}\", exc_info=True)\n```\n\n**Message Type Routing:**\n\n| Server Message Type | Handler Method | Purpose |\n|---------------------|----------------|---------|\n| `TASK` | `start_task()` | Begin new task execution |\n| `COMMAND` | `handle_commands()` | Execute specific commands |\n| `TASK_END` | `handle_task_end()` | Process task completion |\n| `HEARTBEAT` | Log only | Acknowledge keepalive |\n| `ERROR` | Log error | Handle server-side errors |\n| Unknown | Log warning | Ignore unrecognized types |\n\n### Task Start Handler\n\n!!!warning \"Single Task Execution\"\n The client executes **only one task at a time**. New task requests are ignored if a task is currently running.\n\n**Task Start Flow:**\n\n```mermaid\nsequenceDiagram\n participant Server\n participant WSC as WebSocket Client\n participant UFC as UFO Client\n participant Task as Task Coroutine\n \n Server->>WSC: TASK message {user_request, task_name}\n \n alt Current Task Running\n WSC->>WSC: Check current_task.done()\n WSC->>Server: ⚠️ Ignore (log warning)\n else No Task Running\n WSC->>Task: Create task_loop() coroutine\n Task->>UFC: Reset session state\n Task->>Task: Build metadata (platform)\n Task->>Server: TASK_REQUEST (via AIP)\n Server-->>Task: Acknowledgment\n Task->>WSC: Task coroutine running\n end\n```\n\n**Task Start Code:**\n\n```python\nasync def start_task(self, request_text: str, task_name: str | None):\n \"\"\"Start a new task based on server request.\"\"\"\n \n # Check if task is already running\n if self.current_task is not None and not self.current_task.done():\n self.logger.warning(\n f\"[WS] Task {self.session_id} is still running, ignoring new task\"\n )\n return\n \n self.logger.info(f\"[WS] Starting task: {request_text}\")\n \n async def task_loop():\n try:\n async with self.ufo_client.task_lock:\n self.ufo_client.reset() # Clear previous session state\n \n # Build metadata with platform info\n metadata = {}\n if self.ufo_client.platform:\n metadata[\"platform\"] = self.ufo_client.platform\n \n # Send task request via AIP\n await self.task_protocol.send_task_request(\n request=request_text,\n task_name=task_name if task_name else str(uuid4()),\n session_id=self.ufo_client.session_id,\n client_id=self.ufo_client.client_id,\n metadata=metadata if metadata else None\n )\n \n self.logger.info(\n f\"[WS] \\[AIP] Sent task request with platform: {self.ufo_client.platform}\"\n )\n except Exception as e:\n self.logger.error(f\"[WS] \\[AIP] Error sending task request: {e}\")\n # Send error via AIP\n error_msg = ClientMessage(\n type=ClientMessageType.ERROR,\n error=str(e),\n client_id=self.ufo_client.client_id,\n timestamp=datetime.datetime.now(datetime.timezone.utc).isoformat()\n )\n await self.transport.send(error_msg.model_dump_json().encode())\n \n # Create task coroutine\n self.current_task = asyncio.create_task(task_loop())\n```\n\n### Command Execution Handler\n\nThe server sends specific commands (tool calls) to execute, and the client returns results.\n\n**Command Execution Flow:**\n\n```python\nasync def handle_commands(self, server_response: ServerMessage):\n \"\"\"\n Handle commands received from server.\n Uses AIP TaskExecutionProtocol to send results back.\n \"\"\"\n response_id = server_response.response_id\n task_status = server_response.status\n self.session_id = server_response.session_id\n \n # Execute commands via UFO Client\n action_results = await self.ufo_client.execute_step(server_response)\n \n # Send results via AIP\n await self.task_protocol.send_task_result(\n session_id=self.session_id,\n prev_response_id=response_id,\n action_results=action_results,\n status=task_status,\n client_id=self.ufo_client.client_id\n )\n \n self.logger.info(\n f\"[WS] \\[AIP] Sent client result for prev_response_id: {response_id}\"\n )\n \n # Check for task completion\n if task_status in [TaskStatus.COMPLETED, TaskStatus.FAILED]:\n await self.handle_task_end(server_response)\n```\n\n**Execution Steps:**\n\n1. **Extract Metadata**: Get `response_id`, `task_status`, `session_id`\n2. **Execute Commands**: Delegate to `ufo_client.execute_step()`\n3. **Send Results**: Use `TaskExecutionProtocol.send_task_result()`\n4. **Check Completion**: Handle task end if status is terminal\n\n### Task Completion Handler\n\n```python\nasync def handle_task_end(self, server_response: ServerMessage):\n \"\"\"Handle task end messages from server.\"\"\"\n \n if server_response.status == TaskStatus.COMPLETED:\n self.logger.info(\n f\"[WS] Task {self.session_id} completed, result: {server_response.result}\"\n )\n elif server_response.status == TaskStatus.FAILED:\n self.logger.info(\n f\"[WS] Task {self.session_id} failed, with error: {server_response.error}\"\n )\n else:\n self.logger.warning(\n f\"[WS] Unknown task status for {self.session_id}: {server_response.status}\"\n )\n```\n\n---\n\n## ⚠️ Error Handling\n\n### Connection Error Recovery\n\nThe client automatically retries failed connections using exponential backoff to avoid overwhelming the server.\n\n**Retry Logic:**\n\n```python\nasync def connect_and_listen(self):\n \"\"\"Connect with automatic retry.\"\"\"\n while self.retry_count < self.max_retries:\n try:\n async with websockets.connect(...) as ws:\n # Initialize protocols\n self.transport = WebSocketTransport(ws)\n self.registration_protocol = RegistrationProtocol(self.transport)\n self.heartbeat_protocol = HeartbeatProtocol(self.transport)\n self.task_protocol = TaskExecutionProtocol(self.transport)\n \n await self.register_client()\n self.retry_count = 0 # Reset on successful connection\n await self.handle_messages()\n \n except (websockets.ConnectionClosedError, websockets.ConnectionClosedOK) as e:\n self.logger.error(f\"[WS] Connection closed: {e}\")\n self.retry_count += 1\n await self._maybe_retry()\n \n except Exception as e:\n self.logger.error(f\"[WS] Unexpected error: {e}\", exc_info=True)\n self.retry_count += 1\n await self._maybe_retry()\n \n self.logger.error(\"[WS] Max retries reached. Exiting.\")\n```\n\n**Exponential Backoff:**\n\n```python\nasync def _maybe_retry(self):\n \"\"\"Exponential backoff before retry.\"\"\"\n if self.retry_count < self.max_retries:\n wait_time = 2 ** self.retry_count # 2s, 4s, 8s, 16s...\n self.logger.info(f\"[WS] Retrying in {wait_time}s...\")\n await asyncio.sleep(wait_time)\n```\n\n**Retry Schedule:**\n\n| Attempt | Wait Time | Cumulative Wait |\n|---------|-----------|-----------------|\n| 1st retry | 2 seconds | 2s |\n| 2nd retry | 4 seconds | 6s |\n| 3rd retry | 8 seconds | 14s |\n| **Max retries reached** | Exit | - |\n\n**Default Max Retries = 3**\n\nBased on source code: `max_retries: int = 3` in constructor. Increase for unreliable networks:\n\n```python\nws_client = UFOWebSocketClient(\n ws_url=\"ws://...\",\n ufo_client=ufo_client,\n max_retries=10 # More resilient\n)\n```\n\n### Message Parsing Errors\n\n**Graceful Error Handling:**\n\n```python\ntry:\n data = ServerMessage.model_validate_json(msg)\n # Process message...\nexcept Exception as e:\n self.logger.error(f\"[WS] Error handling message: {e}\", exc_info=True)\n # Message is dropped, client continues listening\n```\n\nMessage parsing errors don't crash the client—the error is logged and the receive loop continues.\n\n### Registration Error Handling\n\n**Fallback to Minimal Metadata:**\n\n```python\ntry:\n system_info = DeviceInfoProvider.collect_system_info(...)\n metadata = {\"system_info\": system_info.to_dict()}\nexcept Exception as e:\n self.logger.error(f\"[WS] \\[AIP] Error collecting device info: {e}\")\n # Continue with minimal metadata\n metadata = {\n \"registration_time\": datetime.datetime.now(datetime.timezone.utc).isoformat()\n }\n```\n\nIf device info collection fails, registration still proceeds with minimal metadata (timestamp only).\n\n---\n\n## 🔌 AIP Protocol Integration\n\nThe WebSocket client uses three specialized AIP protocols for different communication patterns.\n\n### 1. Registration Protocol\n\n**Purpose:** Client registration and device profile exchange\n\n```python\nfrom aip.protocol.registration import RegistrationProtocol\n\nself.registration_protocol = RegistrationProtocol(self.transport)\n\n# Register as device\nsuccess = await self.registration_protocol.register_as_device(\n device_id=\"device_windows_001\",\n metadata={\"system_info\": {...}},\n platform=\"windows\"\n)\n```\n\n**Key Methods:**\n\n| Method | Parameters | Returns | Purpose |\n|--------|------------|---------|---------|\n| `register_as_device()` | `device_id`, `metadata`, `platform` | `bool` | Register client as device |\n\nSee [AIP Registration Protocol](../aip/protocols.md#registration-protocol) for message format details.\n\n### 2. Heartbeat Protocol\n\n**Purpose:** Connection keepalive and health monitoring\n\n```python\nfrom aip.protocol.heartbeat import HeartbeatProtocol\n\nself.heartbeat_protocol = HeartbeatProtocol(self.transport)\n\n# Send heartbeat\nawait self.heartbeat_protocol.send_heartbeat(\"device_windows_001\")\n```\n\n**Key Methods:**\n\n| Method | Parameters | Returns | Purpose |\n|--------|------------|---------|---------|\n| `send_heartbeat()` | `client_id` | `None` | Send keepalive message |\n\nSee [AIP Heartbeat Protocol](../aip/protocols.md#heartbeat-protocol) for message format details.\n\n### 3. Task Execution Protocol\n\n**Purpose:** Task request and result exchange\n\n```python\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\nself.task_protocol = TaskExecutionProtocol(self.transport)\n\n# Send task request\nawait self.task_protocol.send_task_request(\n request=\"Open Notepad\",\n task_name=\"task_001\",\n session_id=None,\n client_id=\"device_windows_001\",\n metadata={\"platform\": \"windows\"}\n)\n\n# Send task result\nawait self.task_protocol.send_task_result(\n session_id=\"session_123\",\n prev_response_id=\"resp_456\",\n action_results=[...],\n status=TaskStatus.COMPLETED,\n client_id=\"device_windows_001\"\n)\n```\n\n**Key Methods:**\n\n| Method | Parameters | Returns | Purpose |\n|--------|------------|---------|---------|\n| `send_task_request()` | `request`, `task_name`, `session_id`, `client_id`, `metadata` | `None` | Request task execution |\n| `send_task_result()` | `session_id`, `prev_response_id`, `action_results`, `status`, `client_id` | `None` | Return execution results |\n\nSee [AIP Task Execution Protocol](../aip/protocols.md#task-execution-protocol) for message format details.\n\n---\n\n## 🔍 Connection State Management\n\n### State Checking\n\nUse `is_connected()` to check if the client is ready to send messages.\n\n**Implementation:**\n\n```python\ndef is_connected(self) -> bool:\n \"\"\"Check if WebSocket is connected and registered.\"\"\"\n return (\n self.connected_event.is_set() # Registration succeeded\n and self._ws is not None # WebSocket exists\n and not self._ws.closed # WebSocket is open\n )\n```\n\n**Usage Example:**\n\n```python\nif ws_client.is_connected():\n await ws_client.start_task(\"Open Calculator\", \"task_calc\")\nelse:\n logger.error(\"Not connected to server - cannot send task\")\n```\n\n### Connected Event\n\nThe `connected_event` is an `asyncio.Event` that signals successful registration.\n\n**Usage Pattern:**\n\n```python\n# Wait for connection before sending requests\nawait ws_client.connected_event.wait()\n\n# Now safe to send task requests\nawait ws_client.start_task(\"Open Notepad\", \"task_notepad\")\n```\n\n**Event Lifecycle:**\n\n| State | Event Status | Meaning |\n|-------|--------------|---------|\n| Initial | Not set | Client not connected |\n| Connecting | Not set | WebSocket connecting, registering |\n| Registered | **Set** | ✅ Ready to send/receive messages |\n| Disconnected | Cleared | Connection lost, will retry |\n\n## ✅ Best Practices\n\n### Development Best Practices\n\n**1. Enable DEBUG Logging**\n\n```python\nimport logging\nlogging.basicConfig(level=logging.DEBUG)\n```\n\n**Output:**\n```log\nDEBUG - [WS] [AIP] Heartbeat sent\nDEBUG - [WS] [AIP] Heartbeat failed (connection closed): ...\nINFO - [WS] Received message: ServerMessage(type='COMMAND', ...)\n```\n\n**2. Test Connection Before Full Integration**\n\n```python\n# Test just connection and registration\nws_client = UFOWebSocketClient(ws_url, ufo_client)\nawait ws_client.connect_and_listen() # Should register successfully\n```\n\n**3. Handle Connection Loss Gracefully**\n\n```python\ntry:\n await ws_client.connect_and_listen()\nexcept Exception as e:\n logger.error(f\"WebSocket client error: {e}\")\n # Implement recovery (e.g., alert, restart)\n```\n\n### Production Best Practices\n\n**1. Use Appropriate Retry Limits**\n\nFor production networks with occasional instability:\n\n```python\nws_client = UFOWebSocketClient(\n ws_url=\"wss://production-server.com/ws\",\n ufo_client=ufo_client,\n max_retries=10 # More retries for resilience\n)\n```\n\n**2. Monitor Connection Health**\n\nLog heartbeat success/failure for alerting:\n\n```python\n# In heartbeat_loop (add custom monitoring):\ntry:\n await self.heartbeat_protocol.send_heartbeat(...)\n self.logger.debug(\"[WS] ✅ Heartbeat sent successfully\")\n # Update metrics: heartbeat_success_count++\nexcept Exception as e:\n self.logger.error(f\"[WS] ❌ Heartbeat failed: {e}\")\n # Trigger alert: connection_health_alert()\n```\n\n**3. Use Secure WebSocket (WSS)**\n\n```python\n# Production: Encrypted WebSocket\nws_client = UFOWebSocketClient(\n ws_url=\"wss://ufo-server.company.com/ws\", # WSS, not WS\n ufo_client=ufo_client\n)\n```\n\n**4. Clean State on Reconnection**\n\nThe client automatically resets state:\n\n```python\nasync with self.ufo_client.task_lock:\n self.ufo_client.reset() # Clears session state\n # Send new task request\n```\n\n### Error Handling Best Practices\n\n!!!warning \"Defensive Programming\"\n \n **1. Expect Transient Failures**\n ```python\n # Increase retries for unreliable networks\n max_retries=10\n \n # Monitor retry count in logs\n self.logger.info(f\"[WS] Retry {self.retry_count}/{self.max_retries}\")\n ```\n \n **2. Validate Messages Before Processing**\n ```python\n # Already handled by Pydantic in source code:\n data = ServerMessage.model_validate_json(msg) # Raises on invalid\n ```\n \n **3. Report Errors via AIP**\n ```python\n # Send structured error messages back to server\n error_msg = ClientMessage(\n type=ClientMessageType.ERROR,\n error=str(e),\n client_id=self.ufo_client.client_id,\n timestamp=datetime.datetime.now(datetime.timezone.utc).isoformat()\n )\n await self.transport.send(error_msg.model_dump_json().encode())\n ```\n\n---\n\n## 🔗 Integration Points\n\n### UFO Client Integration\n\nThe WebSocket client delegates all command execution to the UFO Client.\n\n**Execution Flow:**\n\n```python\n# WebSocket client receives command\naction_results = await self.ufo_client.execute_step(server_response)\n```\n\n**Integration:**\n\n| WebSocket Client Role | UFO Client Role |\n|----------------------|-----------------|\n| Receive commands from server | Execute commands via MCP tools |\n| Parse server messages | Manage computer/tool registry |\n| Send results back | Collect execution results |\n| Handle connection errors | Handle execution errors |\n\nSee [UFO Client](./ufo_client.md) for execution details.\n\n### Device Info Provider Integration\n\nDevice information is collected once during registration.\n\n**Integration:**\n\n```python\nfrom ufo.client.device_info_provider import DeviceInfoProvider\n\nsystem_info = DeviceInfoProvider.collect_system_info(\n client_id=self.ufo_client.client_id,\n custom_metadata=None\n)\n```\n\nSee [Device Info Provider](./device_info.md) for profiling details.\n\n### AIP Transport Integration\n\nAll messages go through the WebSocket transport layer.\n\n**Transport Creation:**\n\n```python\nfrom aip.transport.websocket import WebSocketTransport\n\nself.transport = WebSocketTransport(ws)\n```\n\n**Transport Usage:**\n\n- **Protocols use transport** for sending messages\n- **Direct transport access** for error messages\n\nSee [AIP Transport Layer](../aip/transport.md) for transport details.\n\n## 🚀 Next Steps\n\n**Continue Learning**\n\n1. **Connect Your Client** - Follow the step-by-step guide: [Quick Start Guide](./quick_start.md)\n\n2. **Understand Command Execution** - Learn how the UFO Client executes commands: [UFO Client Documentation](./ufo_client.md)\n\n3. **Explore Device Profiling** - See what device information is collected: [Device Info Provider](./device_info.md)\n\n4. **Master the AIP Protocol** - Deep dive into message formats: [AIP Protocol Guide](../aip/protocols.md)\n\n5. **Study Server-Side Registration** - Understand how the server handles registration: [Server Overview](../server/overview.md)\n"
},
{
"path": "documents/docs/configuration/models/azure_openai.md",
"content": "# Azure OpenAI (AOAI)\n\n## Step 1: Create Azure OpenAI Resource\n\nTo use the Azure OpenAI API, create an account on the [Azure OpenAI website](https://azure.microsoft.com/en-us/products/ai-services/openai-service). After creating an account, deploy a model and obtain your API key and endpoint.\n\n## Step 2: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Azure OpenAI API.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your Azure OpenAI configuration:\n\n### Option 1: API Key Authentication (Recommended for Development)\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Enable visual mode to understand screenshots\n REASONING_MODEL: False # Set to True for o-series models\n API_TYPE: \"aoai\" # Use Azure OpenAI API\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\" # Your Azure endpoint\n API_KEY: \"YOUR_AOAI_KEY\" # Your Azure OpenAI API key\n API_VERSION: \"2024-02-15-preview\" # API version\n API_MODEL: \"gpt-4o\" # Model name\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\" # Your deployment name\n\nAPP_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o-mini\" # Use gpt-4o-mini for cost efficiency\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n```\n\n### Option 2: Azure AD Authentication (Recommended for Production)\n\nFor Azure Active Directory authentication, use `API_TYPE: \"azure_ad\"`:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"azure_ad\" # Use Azure AD authentication\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\" # Your Azure endpoint\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o\"\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n \n # Azure AD Configuration\n AAD_TENANT_ID: \"YOUR_TENANT_ID\" # Your Azure tenant ID\n AAD_API_SCOPE: \"YOUR_SCOPE\" # Your API scope\n AAD_API_SCOPE_BASE: \"YOUR_SCOPE_BASE\" # Scope base (without api:// prefix)\n\nAPP_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"azure_ad\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o-mini\"\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n AAD_TENANT_ID: \"YOUR_TENANT_ID\"\n AAD_API_SCOPE: \"YOUR_SCOPE\"\n AAD_API_SCOPE_BASE: \"YOUR_SCOPE_BASE\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Ensure your deployment supports visual inputs\n- **`API_TYPE`**: Use `\"aoai\"` for API key auth or `\"azure_ad\"` for Azure AD auth\n- **`API_BASE`**: Your Azure OpenAI endpoint URL (format: `https://{resource-name}.openai.azure.com`)\n- **`API_KEY`**: Your Azure OpenAI API key (not needed for Azure AD auth)\n- **`API_VERSION`**: Azure API version (e.g., `\"2024-02-15-preview\"`)\n- **`API_MODEL`**: Model identifier (e.g., `gpt-4o`, `gpt-4o-mini`)\n- **`API_DEPLOYMENT_ID`**: Your Azure deployment name (required for AOAI)\n- **`AAD_TENANT_ID`**: Azure tenant ID (required for Azure AD auth)\n- **`AAD_API_SCOPE`**: Azure AD API scope (required for Azure AD auth)\n- **`AAD_API_SCOPE_BASE`**: Scope base without `api://` prefix (required for Azure AD auth)\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n- [OpenAI](openai.md) - Standard OpenAI API setup\n\n## Step 3: Start Using UFO\n\nAfter configuration, you can start using UFO with the Azure OpenAI API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks."
},
{
"path": "documents/docs/configuration/models/claude.md",
"content": "# Anthropic Claude\n\n## Step 1: Obtain API Key\n\nTo use the Claude API, create an account on the [Anthropic Console](https://console.anthropic.com/) and access your API key from the API keys section.\n\n## Step 2: Install Dependencies\n\nInstall the required Anthropic Python package:\n\n```bash\npip install -U anthropic==0.37.1\n```\n\n## Step 3: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Claude API.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your Claude configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Enable visual mode to understand screenshots\n API_TYPE: \"claude\" # Use Claude API\n API_BASE: \"https://api.anthropic.com\" # Claude API endpoint\n API_KEY: \"YOUR_CLAUDE_API_KEY\" # Your Claude API key\n API_MODEL: \"claude-3-5-sonnet-20241022\" # Model name\n API_VERSION: \"2023-06-01\" # API version\n\nAPP_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"claude\"\n API_BASE: \"https://api.anthropic.com\"\n API_KEY: \"YOUR_CLAUDE_API_KEY\"\n API_MODEL: \"claude-3-5-sonnet-20241022\"\n API_VERSION: \"2023-06-01\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Most Claude 3+ models support visual inputs (see [Claude models](https://www.anthropic.com/pricing#anthropic-api))\n- **`API_TYPE`**: Use `\"claude\"` for Claude API (case-sensitive in code: lowercase)\n- **`API_BASE`**: Claude API endpoint - `https://api.anthropic.com`\n- **`API_KEY`**: Your Anthropic API key from the console\n- **`API_MODEL`**: Model identifier (e.g., `claude-3-5-sonnet-20241022`, `claude-3-opus-20240229`)\n- **`API_VERSION`**: API version identifier\n\n**Available Models:**\n\n- **Claude 3.5 Sonnet**: `claude-3-5-sonnet-20241022` - Best balance of intelligence and speed\n- **Claude 3 Opus**: `claude-3-opus-20240229` - Most capable model\n- **Claude 3 Sonnet**: `claude-3-sonnet-20240229` - Balanced performance\n- **Claude 3 Haiku**: `claude-3-haiku-20240307` - Fast and cost-effective\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n- [Anthropic Documentation](https://docs.anthropic.com/) - Official Claude API docs\n\n## Step 4: Start Using UFO\n\nAfter configuration, you can start using UFO with the Claude API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks."
},
{
"path": "documents/docs/configuration/models/custom_model.md",
"content": "# Customized LLM Models\n\nUFO supports and welcomes the integration of custom LLM models. If you have a custom LLM model that you would like to use with UFO, follow the steps below to configure it.\n\n## Step 1: Create and Serve Your Model\n\nCreate a custom LLM model and serve it on your local or remote environment. Ensure your model has an accessible API endpoint.\n\n## Step 2: Implement Model Service Class\n\nCreate a Python script under the `ufo/llm` directory and implement your own LLM model class by inheriting the `BaseService` class from `ufo/llm/base.py`.\n\n**Reference Example:** See `PlaceHolderService` in `ufo/llm/placeholder.py` as a template.\n\nYou must implement the `chat_completion` method:\n\n```python\ndef chat_completion(\n self,\n messages: List[Dict[str, str]],\n n: int = 1,\n temperature: Optional[float] = None,\n max_tokens: Optional[int] = None,\n top_p: Optional[float] = None,\n **kwargs: Any,\n) -> Tuple[List[str], Optional[float]]:\n \"\"\"\n Generates completions for a given list of messages.\n \n Args:\n messages: The list of messages to generate completions for.\n n: The number of completions to generate for each message.\n temperature: Controls the randomness (higher = more random).\n max_tokens: The maximum number of tokens in completions.\n top_p: Controls diversity (higher = more diverse).\n **kwargs: Additional keyword arguments.\n \n Returns:\n Tuple[List[str], Optional[float]]: \n - List of generated completions for each message\n - Cost of the API call (None if not applicable)\n \n Raises:\n Exception: If an error occurs while making the API request.\n \"\"\"\n # Your implementation here\n pass\n```\n\n**Key Implementation Points:**\n\n- Handle message formatting according to your model's API\n- Process visual inputs if `VISUAL_MODE` is enabled\n- Implement retry logic for failed requests\n- Calculate and return cost if applicable\n\n## Step 3: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use your custom model.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your custom model configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Set based on your model's capabilities\n API_TYPE: \"custom_model\" # Use custom model type\n API_BASE: \"http://your-endpoint:port\" # Your model's API endpoint\n API_KEY: \"YOUR_API_KEY\" # Your API key (if required)\n API_MODEL: \"your-model-name\" # Your model identifier\n\nAPP_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"custom_model\"\n API_BASE: \"http://your-endpoint:port\"\n API_KEY: \"YOUR_API_KEY\"\n API_MODEL: \"your-model-name\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` if your model supports visual inputs\n- **`API_TYPE`**: Use `\"custom_model\"` for custom implementations\n- **`API_BASE`**: Your custom model's API endpoint URL\n- **`API_KEY`**: Authentication key (if your model requires it)\n- **`API_MODEL`**: Model identifier or name\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n\n## Step 4: Register Your Model\n\nUpdate the model factory in `ufo/llm/__init__.py` to include your custom model class:\n\n```python\nfrom ufo.llm.your_model import YourModelService\n\n# Add to the model factory mapping\nMODEL_FACTORY = {\n # ... existing models ...\n \"custom_model\": YourModelService,\n}\n```\n\n## Step 5: Start Using UFO\n\nAfter configuration, you can start using UFO with your custom model. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.\n\n**Testing Your Integration:**\n\n1. Test with simple requests first\n2. Verify visual mode works (if applicable)\n3. Check error handling and retry logic\n4. Monitor response quality and latency"
},
{
"path": "documents/docs/configuration/models/deepseek.md",
"content": "# DeepSeek Model\n\n## Step 1: Obtain API Key\n\nDeepSeek is developed by DeepSeek AI. To use DeepSeek models, go to [DeepSeek Platform](https://www.deepseek.com/), register an account, and obtain your API key from the API management console.\n\n## Step 2: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the DeepSeek model.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your DeepSeek configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: False # DeepSeek models typically don't support visual inputs\n API_TYPE: \"deepseek\" # Use DeepSeek API\n API_KEY: \"YOUR_DEEPSEEK_API_KEY\" # Your DeepSeek API key\n API_MODEL: \"deepseek-chat\" # Model name\n\nAPP_AGENT:\n VISUAL_MODE: False\n API_TYPE: \"deepseek\"\n API_KEY: \"YOUR_DEEPSEEK_API_KEY\"\n API_MODEL: \"deepseek-chat\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `False` - Most DeepSeek models don't support visual inputs\n- **`API_TYPE`**: Use `\"deepseek\"` for DeepSeek API (case-sensitive in code: lowercase)\n- **`API_KEY`**: Your DeepSeek API key\n- **`API_MODEL`**: Model identifier (e.g., `deepseek-chat`, `deepseek-coder`)\n\n**Available Models:**\n\n- **DeepSeek-Chat**: `deepseek-chat` - General conversation model\n- **DeepSeek-Coder**: `deepseek-coder` - Code-specialized model\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n\n## Step 3: Start Using UFO\n\nAfter configuration, you can start using UFO with the DeepSeek model. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.\n\n**Note:** Since DeepSeek models don't support visual mode, UFO will operate in text-only mode, which may limit some UI automation capabilities that rely on screenshot understanding.\n"
},
{
"path": "documents/docs/configuration/models/gemini.md",
"content": "# Google Gemini\n\n## Step 1: Obtain API Key\n\nTo use the Google Gemini API, create an account on [Google AI Studio](https://ai.google.dev/) and generate your API key from the API keys section.\n\n## Step 2: Install Dependencies\n\nInstall the required Google GenAI Python package:\n\n```bash\npip install -U google-genai==1.12.1\n```\n\n## Step 3: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Google Gemini API.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your Gemini configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Enable visual mode to understand screenshots\n JSON_SCHEMA: True # Enable JSON schema for structured responses\n API_TYPE: \"gemini\" # Use Gemini API\n API_BASE: \"https://generativelanguage.googleapis.com\" # Gemini API endpoint\n API_KEY: \"YOUR_GEMINI_API_KEY\" # Your Gemini API key\n API_MODEL: \"gemini-2.0-flash-exp\" # Model name\n API_VERSION: \"v1beta\" # API version\n\nAPP_AGENT:\n VISUAL_MODE: True\n JSON_SCHEMA: True\n API_TYPE: \"gemini\"\n API_BASE: \"https://generativelanguage.googleapis.com\"\n API_KEY: \"YOUR_GEMINI_API_KEY\"\n API_MODEL: \"gemini-2.0-flash-exp\"\n API_VERSION: \"v1beta\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Most Gemini models support visual inputs (see [Gemini models](https://ai.google.dev/gemini-api/docs/models/gemini))\n- **`JSON_SCHEMA`**: Set to `True` to enable structured JSON output formatting\n- **`API_TYPE`**: Use `\"gemini\"` for Google Gemini API (case-sensitive in code: lowercase)\n- **`API_BASE`**: Gemini API endpoint - `https://generativelanguage.googleapis.com`\n- **`API_KEY`**: Your Google AI API key\n- **`API_MODEL`**: Model identifier (e.g., `gemini-2.0-flash-exp`, `gemini-1.5-pro`)\n- **`API_VERSION`**: API version (typically `v1beta`)\n\n**Available Models:**\n\n- **Gemini 2.0 Flash**: `gemini-2.0-flash-exp` - Latest experimental model with multimodal capabilities\n- **Gemini 1.5 Pro**: `gemini-1.5-pro` - Advanced reasoning and long context\n- **Gemini 1.5 Flash**: `gemini-1.5-flash` - Fast and efficient\n\n**Rate Limits:**\n\nIf you encounter `429 Resource has been exhausted` errors, you've hit the rate limit of your Gemini API quota. Consider:\n- Reducing request frequency\n- Upgrading your API tier\n- Using exponential backoff for retries\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n- [Gemini API Documentation](https://ai.google.dev/gemini-api) - Official Gemini API docs\n\n## Step 4: Start Using UFO\n\nAfter configuration, you can start using UFO with the Gemini API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks."
},
{
"path": "documents/docs/configuration/models/ollama.md",
"content": "# Ollama\n\n## Step 1: Install and Start Ollama\n\nGo to [Ollama](https://github.com/jmorganca/ollama) and follow the installation instructions for your platform.\n\n**For Linux & WSL2:**\n\n```bash\n# Install Ollama\ncurl https://ollama.ai/install.sh | sh\n\n# Start the Ollama server\nollama serve\n```\n\n**For Windows/Mac:** Download and install from the [Ollama website](https://ollama.ai/).\n\n## Step 2: Pull and Test a Model\n\nOpen a new terminal and pull a model:\n\n```bash\n# Pull a model (e.g., llama2)\nollama pull llama2\n\n# Test the model\nollama run llama2\n```\n\nBy default, Ollama starts a server at `http://localhost:11434`, which will be used as the API base in your configuration.\n\n## Step 3: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use Ollama.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your Ollama configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Enable if model supports vision (e.g., llava)\n API_TYPE: \"ollama\" # Use Ollama API\n API_BASE: \"http://localhost:11434\" # Ollama server endpoint\n API_KEY: \"ollama\" # Placeholder (not used but required)\n API_MODEL: \"llama2\" # Model name (must match pulled model)\n\nAPP_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"ollama\"\n API_BASE: \"http://localhost:11434\"\n API_KEY: \"ollama\"\n API_MODEL: \"llama2\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` only for vision-capable models like `llava`\n- **`API_TYPE`**: Use `\"ollama\"` for Ollama API (case-sensitive in code: lowercase)\n- **`API_BASE`**: Ollama server URL (default: `http://localhost:11434`)\n- **`API_KEY`**: Placeholder value (not used but required in config)\n- **`API_MODEL`**: Model name matching your pulled model\n\n**Important: Increase Context Length**\n\nUFO requires at least 20,000 tokens to function properly. Ollama's default context length is 2048 tokens, which is insufficient. You must create a custom model with increased context:\n\n1. Create a `Modelfile`:\n\n```text\nFROM llama2\nPARAMETER num_ctx 32768\n```\n\n2. Build the custom model:\n\n```bash\nollama create llama2-max-ctx -f Modelfile\n```\n\n3. Use the custom model in your config:\n\n```yaml\nAPI_MODEL: \"llama2-max-ctx\"\n```\n\nFor more details, see [Ollama's Modelfile documentation](https://github.com/ollama/ollama/blob/main/docs/modelfile.md).\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n\n## Step 4: Start Using UFO\n\nAfter configuration, you can start using UFO with Ollama. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.\n\n\n\n"
},
{
"path": "documents/docs/configuration/models/openai.md",
"content": "# OpenAI\n\n## Step 1: Obtain API Key\n\nTo use the OpenAI API, create an account on the [OpenAI website](https://platform.openai.com/signup). After creating an account, you can access your API key from the [API keys page](https://platform.openai.com/account/api-keys).\n\n## Step 2: Configure Agent Settings\n\nAfter obtaining the API key, configure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the OpenAI API.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your OpenAI configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Enable visual mode to understand screenshots\n REASONING_MODEL: False # Set to True for o-series models (o1, o3, o3-mini)\n API_TYPE: \"openai\" # Use OpenAI API\n API_BASE: \"https://api.openai.com/v1\" # OpenAI API endpoint\n API_KEY: \"sk-YOUR_KEY_HERE\" # Your OpenAI API key (starts with sk-)\n API_VERSION: \"2025-02-01-preview\" # API version\n API_MODEL: \"gpt-4o\" # Model name (gpt-4o, gpt-4o-mini, etc.)\n\nAPP_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-4o-mini\" # Use gpt-4o-mini for cost efficiency\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` to enable vision capabilities. Ensure your selected model supports visual inputs (see [OpenAI models](https://platform.openai.com/docs/models))\n- **`REASONING_MODEL`**: Set to `True` when using o-series models (o1, o3, o3-mini) which have different behavior\n- **`API_TYPE`**: Use `\"openai\"` for OpenAI API\n- **`API_BASE`**: OpenAI API base URL - `https://api.openai.com/v1`\n- **`API_KEY`**: Your OpenAI API key from the API keys page\n- **`API_VERSION`**: API version identifier\n- **`API_MODEL`**: Model identifier (e.g., `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`)\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n- [Azure OpenAI](azure_openai.md) - Alternative Azure-hosted OpenAI setup\n\n## Step 3: Start Using UFO\n\nAfter configuration, you can start using UFO with the OpenAI API. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks."
},
{
"path": "documents/docs/configuration/models/operator.md",
"content": "# OpenAI CUA (Operator)\n\nThe [Operator](https://openai.com/index/computer-using-agent/) is a specialized agentic model tailored for Computer-Using Agents (CUA). It's currently available via the Azure OpenAI API (AOAI) using the [Response API](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/responses?tabs=python-secure).\n\n## Step 1: Create Azure OpenAI Resource\n\nTo use the Operator model, create an account on the [Azure OpenAI website](https://azure.microsoft.com/en-us/products/ai-services/openai-service). After creating an account, deploy the Operator model and access your API key.\n\n## Step 2: Configure Operator Agent\n\nConfigure the `OPERATOR` in the `config/ufo/agents.yaml` file to use the Azure OpenAI Operator model.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your Operator configuration:\n\n```yaml\nOPERATOR:\n SCALER: [1024, 768] # Visual input resolution [width, height]\n API_TYPE: \"azure_ad\" # Use Azure AD authentication\n API_MODEL: \"computer-use-preview-20250311\" # Operator model name\n API_VERSION: \"2025-03-01-preview\" # API version for Operator\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\" # Your Azure endpoint\n \n # Azure AD Authentication (required)\n AAD_TENANT_ID: \"YOUR_TENANT_ID\" # Your Azure tenant ID\n AAD_API_SCOPE: \"YOUR_SCOPE\" # Your API scope\n AAD_API_SCOPE_BASE: \"YOUR_SCOPE_BASE\" # Scope base (without api:// prefix)\n```\n\n**Configuration Fields:**\n\n- **`SCALER`**: Resolution for visual input `[width, height]` (recommended: `[1024, 768]`)\n- **`API_TYPE`**: Use `\"azure_ad\"` for Azure AD authentication (or `\"aoai\"` for API key auth)\n- **`API_MODEL`**: Operator model identifier (e.g., `computer-use-preview-20250311`)\n- **`API_VERSION`**: API version for Operator (e.g., `2025-03-01-preview`)\n- **`API_BASE`**: Your Azure OpenAI endpoint URL\n- **`AAD_TENANT_ID`**: Azure tenant ID (required for Azure AD auth)\n- **`AAD_API_SCOPE`**: Azure AD API scope (required for Azure AD auth)\n- **`AAD_API_SCOPE_BASE`**: Scope base without `api://` prefix (required for Azure AD auth)\n\n**For API Key Authentication (Development):**\n\nIf you prefer API key authentication instead of Azure AD:\n\n```yaml\nOPERATOR:\n SCALER: [1024, 768]\n API_TYPE: \"aoai\" # Use API key authentication\n API_MODEL: \"computer-use-preview-20250311\"\n API_VERSION: \"2025-03-01-preview\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\" # Your Azure OpenAI API key\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\" # Your deployment name\n```\n\n## Step 3: Run Operator in UFO\n\nUFO supports running Operator in two modes:\n\n1. **Standalone Agent**: Run Operator as a single agent\n2. **As AppAgent**: Call Operator as a separate `AppAgent` from the `HostAgent`\n\nOperator uses a specialized visual-only workflow different from other models and currently does not support the standard `AppAgent` workflow.\n\n**For detailed usage instructions, see:**\n\n- [Operator as AppAgent](../../ufo2/advanced_usage/operator_as_app_agent.md) - How to integrate Operator into UFO workflows\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Azure OpenAI](azure_openai.md) - General Azure OpenAI setup\n\n**Important Notes:**\n\n- Operator is a visual-only model optimized for computer control tasks\n- It uses a different workflow from standard text-based models\n- Best suited for direct UI manipulation and visual understanding tasks\n- Requires Azure OpenAI deployment (not available via standard OpenAI API)\n\n"
},
{
"path": "documents/docs/configuration/models/overview.md",
"content": "# Supported Models\n\nUFO supports a wide variety of LLM models and APIs. You can configure different models for `HOST_AGENT`, `APP_AGENT`, `BACKUP_AGENT`, and `EVALUATION_AGENT` in the `config/ufo/agents.yaml` file to optimize for performance, cost, or specific capabilities.\n\n## Available Model Integrations\n\n| Provider | Documentation | Visual Support | Authentication |\n| --- | --- | --- | --- |\n| **OpenAI** | [OpenAI API](./openai.md) | ✅ | API Key |\n| **Azure OpenAI (AOAI)** | [Azure OpenAI API](./azure_openai.md) | ✅ | API Key / Azure AD |\n| **Google Gemini** | [Gemini API](./gemini.md) | ✅ | API Key |\n| **Anthropic Claude** | [Claude API](./claude.md) | ✅ | API Key |\n| **Qwen (Alibaba)** | [Qwen API](./qwen.md) | ✅ | API Key |\n| **DeepSeek** | [DeepSeek API](./deepseek.md) | ❌ | API Key |\n| **Ollama** | [Ollama API](./ollama.md) | ⚠️ Limited | Local |\n| **OpenAI Operator** | [Operator (CUA)](./operator.md) | ✅ | Azure AD |\n| **Custom Models** | [Custom API](./custom_model.md) | Depends | Varies |\n\n## Model Selection Guide\n\n### By Use Case\n\n**For Production Deployments:**\n- **Primary**: OpenAI GPT-4o or Azure OpenAI (enterprise features)\n- **Cost-optimized**: GPT-4o-mini for APP_AGENT, GPT-4o for HOST_AGENT\n- **Privacy-sensitive**: Ollama (local models)\n\n**For Development & Testing:**\n- **Fast iteration**: Gemini 2.0 Flash (high speed, low cost)\n- **Local testing**: Ollama with llama2 or similar\n- **Budget-friendly**: DeepSeek or Qwen models\n\n**For Specialized Tasks:**\n- **Computer control**: OpenAI Operator (CUA model)\n- **Code generation**: DeepSeek-Coder or Claude\n- **Long context**: Gemini 1.5 Pro (large context window)\n\n### By Capability\n\n**Vision Support (Screenshot Understanding):**\n- ✅ OpenAI GPT-4o, GPT-4-turbo\n- ✅ Azure OpenAI (vision-enabled deployments)\n- ✅ Google Gemini (all 1.5+ models)\n- ✅ Claude 3+ (all variants)\n- ✅ Qwen-VL models\n- ⚠️ Ollama (llava models only)\n- ❌ DeepSeek (text-only)\n\n**JSON Schema Support:**\n- ✅ OpenAI / Azure OpenAI\n- ✅ Google Gemini\n- ⚠️ Limited: Claude, Qwen, Ollama\n\n## Configuration Architecture\n\nEach model is implemented as a separate class in the `ufo/llm` directory, inheriting from the `BaseService` class in `ufo/llm/base.py`. All models implement the `chat_completion` method to maintain a consistent interface.\n\n**Key Configuration Files:**\n\n- **`config/ufo/agents.yaml`**: Primary agent configuration (HOST, APP, BACKUP, EVALUATION, OPERATOR)\n- **`config/ufo/system.yaml`**: System-wide LLM parameters (MAX_TOKENS, TEMPERATURE, etc.)\n- **`config/ufo/prices.yaml`**: Cost tracking for different models\n\n## Multi-Provider Setup\n\nYou can mix and match providers for different agents to optimize cost and performance:\n\n```yaml\n# Use OpenAI for planning\nHOST_AGENT:\n API_TYPE: \"openai\"\n API_MODEL: \"gpt-4o\"\n\n# Use Azure OpenAI for execution (cost control)\nAPP_AGENT:\n API_TYPE: \"aoai\"\n API_MODEL: \"gpt-4o-mini\"\n\n# Use Claude for evaluation\nEVALUATION_AGENT:\n API_TYPE: \"claude\"\n API_MODEL: \"claude-3-5-sonnet-20241022\"\n```\n\n## Getting Started\n\n1. Choose your LLM provider from the table above\n2. Follow the provider-specific documentation to obtain API keys\n3. Configure `config/ufo/agents.yaml` with your credentials\n4. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) to begin\n\n**For detailed configuration options:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete configuration reference\n- [System Configuration](../system/system_config.md) - LLM parameters and behavior\n- [Quick Start Guide](../../getting_started/quick_start_ufo2.md) - Step-by-step setup"
},
{
"path": "documents/docs/configuration/models/qwen.md",
"content": "# Qwen Model\n\n## Step 1: Obtain API Key\n\nQwen (Tongyi Qianwen) is developed by Alibaba DAMO Academy. To use Qwen models, go to [DashScope](https://dashscope.aliyun.com/), register an account, and obtain your API key. Detailed instructions are available in the [DashScope documentation](https://help.aliyun.com/zh/dashscope/developer-reference/activate-dashscope-and-create-an-api-key) (Chinese).\n\n## Step 2: Configure Agent Settings\n\nConfigure the `HOST_AGENT` and `APP_AGENT` in the `config/ufo/agents.yaml` file to use the Qwen model.\n\nIf the file doesn't exist, copy it from the template:\n\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\nEdit `config/ufo/agents.yaml` with your Qwen configuration:\n\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True # Enable visual mode for vision-capable models\n API_TYPE: \"qwen\" # Use Qwen API\n API_KEY: \"YOUR_QWEN_API_KEY\" # Your DashScope API key\n API_MODEL: \"qwen-vl-max\" # Model name (e.g., qwen-vl-max, qwen-max)\n\nAPP_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"qwen\"\n API_KEY: \"YOUR_QWEN_API_KEY\"\n API_MODEL: \"qwen-vl-max\"\n```\n\n**Configuration Fields:**\n\n- **`VISUAL_MODE`**: Set to `True` for vision-capable models (qwen-vl-*). Set to `False` for text-only models\n- **`API_TYPE`**: Use `\"qwen\"` for Qwen API (case-sensitive in code: lowercase)\n- **`API_KEY`**: Your DashScope API key\n- **`API_MODEL`**: Model identifier (see [Qwen model list](https://help.aliyun.com/zh/dashscope/developer-reference/model-square/))\n\n**Available Models:**\n\n- **Qwen-VL-Max**: `qwen-vl-max` - Vision and language model\n- **Qwen-Max**: `qwen-max` - Text-only advanced model\n- **Qwen-Plus**: `qwen-plus` - Balanced performance model\n\n**For detailed configuration options, see:**\n\n- [Agent Configuration Guide](../system/agents_config.md) - Complete agent settings reference\n- [Model Configuration Overview](overview.md) - Compare different LLM providers\n\n## Step 3: Start Using UFO\n\nAfter configuration, you can start using UFO with the Qwen model. Refer to the [Quick Start Guide](../../getting_started/quick_start_ufo2.md) for detailed instructions on running your first tasks.\n"
},
{
"path": "documents/docs/configuration/system/agents_config.md",
"content": "# Agent Configuration (agents.yaml)\n\nConfigure all LLM models and agent-specific settings for UFO². Each agent type can use different models and API configurations for optimal performance.\n\n## Overview\n\nThe `agents.yaml` file defines LLM settings for all agents in UFO². This is the **most important configuration file** as it contains your API keys and model selections.\n\n**File Location**: `config/ufo/agents.yaml`\n\n**Initial Setup Required:**\n\n1. **Copy the template file**:\n ```powershell\n Copy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n ```\n\n2. **Edit `config/ufo/agents.yaml`** with your API keys and settings\n\n3. **Never commit `agents.yaml`** to version control (it contains secrets)\n\n## Quick Start\n\n### Step 1: Create Configuration File\n\n```powershell\n# Copy template to create your configuration\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\n### Step 2: Configure Your LLM Provider\n\nChoose your LLM provider and edit `config/ufo/agents.yaml`:\n\n**OpenAI:**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_OPENAI_KEY_HERE\"\n API_MODEL: \"gpt-4o\"\n API_VERSION: \"2025-02-01-preview\"\n \nAPP_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_OPENAI_KEY_HERE\"\n API_MODEL: \"gpt-4o-mini\"\n API_VERSION: \"2025-02-01-preview\"\n```\n\n**Azure OpenAI:**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_MODEL: \"gpt-4o\"\n API_VERSION: \"2024-02-15-preview\"\n API_DEPLOYMENT_ID: \"gpt-4o-deployment\"\n \nAPP_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_MODEL: \"gpt-4o-mini\"\n API_VERSION: \"2024-02-15-preview\"\n API_DEPLOYMENT_ID: \"gpt-4o-mini-deployment\"\n```\n\n**Google Gemini:**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"gemini\"\n API_BASE: \"https://generativelanguage.googleapis.com\"\n API_KEY: \"YOUR_GEMINI_API_KEY\"\n API_MODEL: \"gemini-2.0-flash-exp\"\n API_VERSION: \"v1beta\"\n```\n\n**Anthropic Claude:**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"claude\"\n API_BASE: \"https://api.anthropic.com\"\n API_KEY: \"YOUR_CLAUDE_API_KEY\"\n API_MODEL: \"claude-3-5-sonnet-20241022\"\n API_VERSION: \"2023-06-01\"\n```\n\n### Step 3: Verify Configuration\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\nprint(f\"HOST_AGENT model: {config.host_agent.api_model}\")\nprint(f\"APP_AGENT model: {config.app_agent.api_model}\")\n```\n\n---\n\n## Agent Types\n\nUFO² uses different agents for different purposes. Each can be configured with different models.\n\n| Agent | Purpose | Recommended Model | Frequency |\n|-------|---------|-------------------|-----------|\n| **HOST_AGENT** | Task planning, app coordination | GPT-4o, GPT-4 | Low (planning) |\n| **APP_AGENT** | Action execution, UI interaction | GPT-4o-mini, GPT-4o | High (every action) |\n| **BACKUP_AGENT** | Fallback when others fail | GPT-4-vision-preview | Rare (errors) |\n| **EVALUATION_AGENT** | Task completion evaluation | GPT-4o | Low (end of task) |\n| **OPERATOR** | CUA-based automation | computer-use-preview | Optional |\n\n**Cost Optimization Tips:**\n\n- Use **GPT-4o** for HOST_AGENT (complex planning)\n- Use **GPT-4o-mini** for APP_AGENT (frequent actions, 60% cheaper)\n- Same model can be used for BACKUP_AGENT and EVALUATION_AGENT\n\n## Configuration Fields\n\n### Common Fields (All Agents)\n\nThese fields are available for `HOST_AGENT`, `APP_AGENT`, `BACKUP_AGENT`, `EVALUATION_AGENT`, and `OPERATOR`.\n\n#### Core Settings\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `VISUAL_MODE` | Boolean | ❌ | `True` | Enable vision capabilities (screenshot understanding) |\n| `REASONING_MODEL` | Boolean | ❌ | `False` | Whether model is a reasoning model (o1, o3, o3-mini) |\n| `API_TYPE` | String | ✅ | `\"openai\"` | LLM provider type |\n| `API_BASE` | String | ✅ | varies | API endpoint URL |\n| `API_KEY` | String | ✅ | `\"\"` | API authentication key |\n| `API_MODEL` | String | ✅ | varies | Model identifier |\n| `API_VERSION` | String | ❌ | `\"2025-02-01-preview\"` | API version |\n\n**Legend:** ✅ = Required (must be set), ❌ = Optional (has default value)\n\n#### API_TYPE Options\n\n| API_TYPE | Provider | Example API_BASE |\n|----------|----------|------------------|\n| `\"openai\"` | OpenAI | `https://api.openai.com/v1/chat/completions` |\n| `\"aoai\"` | Azure OpenAI | `https://YOUR_RESOURCE.openai.azure.com` |\n| `\"azure_ad\"` | Azure OpenAI (AD auth) | `https://YOUR_RESOURCE.openai.azure.com` |\n| `\"gemini\"` | Google Gemini | `https://generativelanguage.googleapis.com` |\n| `\"claude\"` | Anthropic Claude | `https://api.anthropic.com` |\n| `\"qwen\"` | Alibaba Qwen | varies |\n| `\"ollama\"` | Ollama (local) | `http://localhost:11434` |\n\n#### Azure OpenAI Additional Fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `API_DEPLOYMENT_ID` | String | ✅ (for AOAI) | Azure deployment name |\n\n**Example**:\n```yaml\nHOST_AGENT:\n API_TYPE: \"aoai\"\n API_BASE: \"https://myresource.openai.azure.com\"\n API_KEY: \"abc123...\"\n API_MODEL: \"gpt-4o\"\n API_DEPLOYMENT_ID: \"gpt-4o-deployment-name\"\n```\n\n#### Azure AD Authentication Fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `AAD_TENANT_ID` | String | ✅ (for azure_ad) | Azure AD tenant ID |\n| `AAD_API_SCOPE` | String | ✅ (for azure_ad) | Azure AD API scope |\n| `AAD_API_SCOPE_BASE` | String | ✅ (for azure_ad) | Scope base URL |\n\n**Example**:\n```yaml\nHOST_AGENT:\n API_TYPE: \"azure_ad\"\n API_BASE: \"https://myresource.openai.azure.com\"\n AAD_TENANT_ID: \"your-tenant-id\"\n AAD_API_SCOPE: \"your-scope\"\n AAD_API_SCOPE_BASE: \"API://your-scope-base\"\n API_MODEL: \"gpt-4o\"\n API_DEPLOYMENT_ID: \"gpt-4o-deployment\"\n```\n\n#### Prompt Configuration\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `PROMPT` | String | ❌ | Path to main prompt template |\n| `EXAMPLE_PROMPT` | String | ❌ | Path to example prompt template |\n| `API_PROMPT` | String | ❌ | Path to API usage prompt (APP_AGENT only) |\n\n**Default Prompt Paths:**\n```yaml\nHOST_AGENT:\n PROMPT: \"ufo/prompts/share/base/host_agent.yaml\"\n EXAMPLE_PROMPT: \"ufo/prompts/examples/{mode}/host_agent_example.yaml\"\n\nAPP_AGENT:\n PROMPT: \"ufo/prompts/share/base/app_agent.yaml\"\n EXAMPLE_PROMPT: \"ufo/prompts/examples/{mode}/app_agent_example.yaml\"\n API_PROMPT: \"ufo/prompts/share/base/api.yaml\"\n```\n\nYou can customize prompts by creating your own YAML files and updating these paths. See the [Customization Guide](../../ufo2/advanced_usage/customization.md) for details.\n\n#### OPERATOR-Specific Fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `SCALER` | List[int] | ❌ | Screen dimensions for visual input `[width, height]`, default: `[1024, 768]` |\n\n**Example:**\n```yaml\nOPERATOR:\n SCALER: [1920, 1080] # Full HD resolution\n API_MODEL: \"computer-use-preview-20250311\"\n # ... other settings\n```\n\n## Complete Configuration Example\n\nHere's a complete `agents.yaml` with all agent types configured:\n\n```yaml\n# HOST_AGENT - Task planning and coordination\nHOST_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"gpt-4o\"\n API_VERSION: \"2025-02-01-preview\"\n PROMPT: \"ufo/prompts/share/base/host_agent.yaml\"\n EXAMPLE_PROMPT: \"ufo/prompts/examples/{mode}/host_agent_example.yaml\"\n\n# APP_AGENT - Action execution\nAPP_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"gpt-4o-mini\" # Cheaper for frequent actions\n API_VERSION: \"2025-02-01-preview\"\n PROMPT: \"ufo/prompts/share/base/app_agent.yaml\"\n EXAMPLE_PROMPT: \"ufo/prompts/examples/{mode}/app_agent_example.yaml\"\n API_PROMPT: \"ufo/prompts/share/base/api.yaml\"\n\n# BACKUP_AGENT - Fallback agent\nBACKUP_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"gpt-4-vision-preview\"\n API_VERSION: \"2025-02-01-preview\"\n\n# EVALUATION_AGENT - Task evaluation\nEVALUATION_AGENT:\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"gpt-4o\"\n API_VERSION: \"2025-02-01-preview\"\n\n# OPERATOR - OpenAI Operator (optional)\nOPERATOR:\n SCALER: [1024, 768] # Screen resolution for visual input\n VISUAL_MODE: True\n REASONING_MODEL: False\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"computer-use-preview-20250311\"\n API_VERSION: \"2025-03-01-preview\"\n```\n\n## Multi-Provider Configuration\n\nYou can use different providers for different agents:\n\n```yaml\n# Use OpenAI for planning\nHOST_AGENT:\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_OPENAI_KEY\"\n API_MODEL: \"gpt-4o\"\n\n# Use Azure OpenAI for actions (cost control)\nAPP_AGENT:\n API_TYPE: \"aoai\"\n API_BASE: \"https://mycompany.openai.azure.com\"\n API_KEY: \"YOUR_AZURE_KEY\"\n API_MODEL: \"gpt-4o-mini\"\n API_DEPLOYMENT_ID: \"gpt-4o-mini-deploy\"\n\n# Use Claude for evaluation\nEVALUATION_AGENT:\n API_TYPE: \"claude\"\n API_BASE: \"https://api.anthropic.com\"\n API_KEY: \"YOUR_CLAUDE_KEY\"\n API_MODEL: \"claude-3-5-sonnet-20241022\"\n```\n\n## Model Recommendations\n\n### For HOST_AGENT (Planning)\n\n| Model | Provider | Pros | Cons |\n|-------|----------|------|------|\n| **gpt-4o** | OpenAI | Best overall, fast, multimodal | $$ |\n| **gpt-4-turbo** | OpenAI | Good quality, cheaper than GPT-4 | Slower |\n| **claude-3-5-sonnet** | Anthropic | Excellent reasoning | No vision API yet |\n| **gemini-2.0-flash** | Google | Fast, cheap, multimodal | New, less tested |\n\n### For APP_AGENT (Execution)\n\n| Model | Provider | Pros | Cons |\n|-------|----------|------|------|\n| **gpt-4o-mini** | OpenAI | 60% cheaper, fast, good quality | Slightly less capable |\n| **gpt-4o** | OpenAI | Best quality | More expensive |\n| **gemini-1.5-flash** | Google | Very cheap, fast | Less accurate |\n\n### For OPERATOR (CUA Mode)\n\n| Model | Provider | Notes |\n|-------|----------|-------|\n| **computer-use-preview-20250311** | OpenAI | Supported model for Operator mode (Computer Use Agent) |\n\n## Reasoning Models\n\nFor models like OpenAI o1, o3, o3-mini, set `REASONING_MODEL: True`:\n\n```yaml\nHOST_AGENT:\n REASONING_MODEL: True # Enable for o1/o3/o3-mini\n API_TYPE: \"openai\"\n API_MODEL: \"o3-mini\"\n # ... other settings\n```\n\n**Note:** Reasoning models have different behavior including no streaming responses, different token limits, and may have different pricing.\n\n## Environment Variables\n\nInstead of hardcoding API keys, you can use environment variables:\n\n```yaml\nHOST_AGENT:\n API_KEY: \"${OPENAI_API_KEY}\" # Reads from environment variable\n\nAPP_AGENT:\n API_KEY: \"${AZURE_OPENAI_KEY}\"\n```\n\n**Setting environment variables**:\n\n**Windows (PowerShell):**\n```powershell\n$env:OPENAI_API_KEY = \"sk-your-key\"\n$env:AZURE_OPENAI_KEY = \"your-azure-key\"\n```\n\n**Windows (Persistent):**\n```powershell\n[System.Environment]::SetEnvironmentVariable('OPENAI_API_KEY', 'sk-your-key', 'User')\n```\n\n**Linux/macOS:**\n```bash\nexport OPENAI_API_KEY=\"sk-your-key\"\nexport AZURE_OPENAI_KEY=\"your-azure-key\"\n```\n\n## Programmatic Access\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Access HOST_AGENT settings\nhost_model = config.host_agent.api_model\nhost_type = config.host_agent.api_type\nhost_visual = config.host_agent.visual_mode\n\n# Access APP_AGENT settings\napp_model = config.app_agent.api_model\napp_key = config.app_agent.api_key\n\n# Check if agent is configured\nif config.host_agent.api_key:\n print(\"HOST_AGENT is configured\")\nelse:\n print(\"Warning: HOST_AGENT API key not set\")\n```\n\n## Troubleshooting\n\n### Issue 1: \"agents.yaml not found\"\n\n**Error Message:**\n```\nFileNotFoundError: config/ufo/agents.yaml not found\n```\n\n**Solution:** Copy the template file\n```powershell\nCopy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n```\n\n### Issue 2: API Authentication Errors\n\n**Error Message:**\n```\nopenai.AuthenticationError: Invalid API key\n```\n\n**Solutions:**\n1. Verify API key is correct\n2. Check for extra spaces or quotes\n3. Ensure API_TYPE matches your provider\n4. For Azure, verify API_DEPLOYMENT_ID is set\n\n### Issue 3: Model Not Found\n\n**Error Message:**\n```\nopenai.NotFoundError: The model 'gpt-4o' does not exist\n```\n\n**Solutions:**\n1. Verify model name is correct (check provider's documentation)\n2. For Azure, ensure deployment exists and API_DEPLOYMENT_ID matches\n3. Check if you have access to the model\n\n### Issue 4: Rate Limits\n\n**Error Message:**\n```\nopenai.RateLimitError: Rate limit exceeded\n```\n\n**Solutions:**\n1. Add delays between requests (configure in `system.yaml`)\n2. Upgrade your API plan\n3. Use different API keys for different agents\n\n## Security Best Practices\n\n**API Key Security Guidelines:**\n\n1. ✅ **Never commit `agents.yaml` to Git**\n - Add to `.gitignore`\n - Only commit `agents.yaml.template`\n\n2. ✅ **Use environment variables** for production\n ```yaml\n API_KEY: \"${OPENAI_API_KEY}\"\n ```\n\n3. ✅ **Rotate keys regularly**\n\n4. ✅ **Use separate keys** for dev/prod environments\n\n5. ✅ **Restrict key permissions** (e.g., read-only for evaluation agents)\n\n## Related Documentation\n\n- **[Third-Party Agent Configuration](third_party_config.md)** - Configure external agents like LinuxAgent and HardwareAgent\n- **[Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md)** - Build your own specialized agents\n- **[System Configuration](system_config.md)** - Runtime and execution settings\n- **[MCP Configuration](mcp_reference.md)** - Tool server configuration\n- **[RAG Configuration](rag_config.md)** - Knowledge retrieval settings\n- **[Model Setup Guide](../models/overview.md)** - Provider-specific setup\n- **[Migration Guide](migration.md)** - Migrating from legacy config\n\n## Summary\n\n**Key Takeaways:**\n\n✅ **Copy template first**: `Copy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml` \n✅ **Add your API keys**: Edit `agents.yaml` with your credentials \n✅ **Choose models wisely**: GPT-4o for planning, GPT-4o-mini for actions \n✅ **Never commit secrets**: Keep `agents.yaml` out of version control \n✅ **Use environment variables**: For production deployments\n\n**Your agents are now ready to work!** 🚀\n"
},
{
"path": "documents/docs/configuration/system/extending.md",
"content": "# Extending Configuration\n\nThis guide shows you how to add custom configuration options to UFO2.\n\n**Three Ways to Extend:**\n\n1. **Simple YAML files** - Quick custom settings in existing files\n2. **New configuration files** - Organize new features separately \n3. **Typed configuration schemas** - Full type safety with Python dataclasses\n\n## Method 1: Adding Fields to Existing Files\n\nFor simple customizations, add fields directly to existing configuration files.\n\n```yaml\n # config/ufo/system.yaml\n MAX_STEP: 20\n SLEEP_TIME: 1.0\n\n # Your custom fields\n CUSTOM_TIMEOUT: 300\n DEBUG_MODE: true\n FEATURE_FLAGS:\n enable_telemetry: false\n use_experimental_api: true\n```\n\n### Accessing Custom Fields\n\n```python\n from config.config_loader import get_ufo_config\n\n config = get_ufo_config()\n\n # Access custom fields dynamically\n timeout = config.system.CUSTOM_TIMEOUT # 300\n debug = config.system.DEBUG_MODE # True\n use_experimental = config.system.FEATURE_FLAGS['use_experimental_api'] # True\n```\n\nCustom fields are automatically discovered and loaded - no code modifications needed!\n\n---\n\n## Method 2: Creating New Configuration Files\n\nFor larger features, create dedicated configuration files.\n\n```yaml\n # config/ufo/analytics.yaml\n ANALYTICS:\n enabled: true\n backend: \"influxdb\"\n endpoint: \"http://localhost:8086\"\n database: \"ufo_metrics\"\n retention: \"30d\"\n \n metrics:\n - name: \"task_duration\"\n type: \"histogram\"\n - name: \"success_rate\"\n type: \"counter\"\n ```\n\n### Automatic Discovery\n\nThe config loader automatically discovers and loads all YAML files in `config/ufo/`:\n\n```python\n# No registration needed!\nconfig = get_ufo_config()\n\n# Your new file is automatically loaded\nanalytics_enabled = config.ANALYTICS['enabled']\nmetrics = config.ANALYTICS['metrics']\n```\n\n---\n\n## Method 3: Typed Configuration Schemas\n\nFor production features requiring type safety and validation, define typed schemas.\n\n```python\n # config/config_schemas.py\n from dataclasses import dataclass, field\n from typing import List, Literal\n\n @dataclass\n class MetricConfig:\n \"\"\"Configuration for a single metric.\"\"\"\n name: str\n type: Literal[\"counter\", \"histogram\", \"gauge\"]\n tags: List[str] = field(default_factory=list)\n\n @dataclass\n class AnalyticsConfig:\n \"\"\"Analytics system configuration.\"\"\"\n \n # Required fields\n enabled: bool\n backend: Literal[\"influxdb\", \"prometheus\", \"datadog\"]\n endpoint: str\n \n # Optional fields with defaults\n database: str = \"ufo_metrics\"\n retention: str = \"30d\"\n batch_size: int = 100\n flush_interval: float = 10.0\n \n # Nested configuration\n metrics: List[MetricConfig] = field(default_factory=list)\n \n def __post_init__(self):\n \"\"\"Validate configuration after initialization.\"\"\"\n if self.enabled and not self.endpoint:\n raise ValueError(\"endpoint required when analytics enabled\")\n \n if self.batch_size <= 0:\n raise ValueError(\"batch_size must be positive\")\n```\n\n### Step 2: Integrate into UFOConfig\n\n```python\n # config/config_schemas.py\n from dataclasses import dataclass\n\n @dataclass\n class UFOConfig:\n \"\"\"Main UFO configuration.\"\"\"\n host_agent: AgentConfig\n app_agent: AgentConfig\n system: SystemConfig\n rag: RAGConfig\n analytics: AnalyticsConfig # Add your new config\n \n # ... rest of implementation\n```\n\n### Step 3: Use Typed Configuration\n\n```python\n from config.config_loader import get_ufo_config\n\n config = get_ufo_config()\n\n # Type-safe access with IDE autocomplete\n if config.analytics.enabled:\n for metric in config.analytics.metrics:\n print(f\"Metric: {metric.name}, Type: {metric.type}\")\n \n # Validation happens automatically\n batch_size = config.analytics.batch_size # Guaranteed > 0\n```\n\n---\n\n## Common Patterns\n\n### Environment-Specific Overrides\n\n```yaml\n # config/ufo/system.yaml (base)\n LOG_LEVEL: \"INFO\"\n DEBUG_MODE: false\n CACHE_SIZE: 1000\n\n # config/ufo/system.dev.yaml (development override)\n LOG_LEVEL: \"DEBUG\"\n DEBUG_MODE: true\n PROFILING_ENABLED: true\n\n # config/ufo/system.prod.yaml (production override)\n LOG_LEVEL: \"WARNING\"\n CACHE_SIZE: 10000\n MONITORING_ENABLED: true\n```\n\n### Feature Flags\n\n```yaml\n # config/ufo/features.yaml\n FEATURES:\n experimental_actions: false\n multi_device_mode: true\n advanced_logging: false\n \n # Per-agent feature flags\n agent_features:\n host_agent:\n use_vision_model: true\n parallel_processing: false\n app_agent:\n speculative_execution: true\n action_batching: true\n```\n\n### Plugin Configuration\n\n```yaml\n # config/ufo/plugins.yaml\n PLUGINS:\n enabled: true\n auto_discover: true\n load_order:\n - \"core\"\n - \"analytics\"\n - \"custom\"\n \n plugins:\n analytics:\n enabled: true\n config_file: \"config/plugins/analytics.yaml\"\n \n custom_processor:\n enabled: false\n class: \"plugins.custom.MyProcessor\"\n priority: 100\n```\n\n---\n\n## Best Practices\n\n**DO - Recommended Practices**\n\n- ✅ **Group related settings** in dedicated files\n- ✅ **Use typed schemas** for production features\n- ✅ **Provide sensible defaults** for all optional fields\n- ✅ **Add validation** in `__post_init__` methods\n- ✅ **Document all fields** with docstrings\n- ✅ **Use environment overrides** for deployment-specific settings\n- ✅ **Version your config schemas** when making breaking changes\n- ✅ **Test configuration loading** in CI/CD pipelines\n\n**DON'T - Anti-Patterns**\n\n- ❌ **Don't hardcode secrets** - use environment variables\n- ❌ **Don't duplicate settings** across multiple files\n- ❌ **Don't use dynamic field names** - breaks type safety\n- ❌ **Don't skip validation** - catch errors early\n- ❌ **Don't mix concerns** - keep configs focused\n- ❌ **Don't ignore warnings** from config loader\n- ❌ **Don't commit sensitive data** - use .env files\n\n---\n\n## Security Considerations\n\n!!!warning \"Secrets Management\"\n Never commit sensitive data to configuration files:\n \n ```yaml\n # ? BAD - Hardcoded secrets\n DATABASE:\n password: \"my-secret-password\"\n api_key: \"sk-1234567890\"\n \n # ? GOOD - Environment variable references\n DATABASE:\n password: \"${DB_PASSWORD}\"\n api_key: \"${API_KEY}\"\n ```\n\n### \"Environment Variables\"\nUse environment variables for secrets:\n\n```python\nimport os\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Resolve environment variables\ndb_password = os.getenv('DB_PASSWORD')\napi_key = os.getenv('API_KEY')\n```\n\n---\n\n## Testing Your Configuration\n\n```python\n import pytest\n from config.config_loader import ConfigLoader\n from config.ufo.schemas.analytics_config import AnalyticsConfig\n\n def test_analytics_config_defaults():\n \"\"\"Test analytics configuration defaults.\"\"\"\n config_data = {\n 'enabled': True,\n 'backend': 'influxdb',\n 'endpoint': 'http://localhost:8086'\n }\n \n analytics = AnalyticsConfig(**config_data)\n \n assert analytics.enabled is True\n assert analytics.database == 'ufo_metrics' # Default\n assert analytics.batch_size == 100 # Default\n\n def test_analytics_config_validation():\n \"\"\"Test analytics configuration validation.\"\"\"\n with pytest.raises(ValueError, match=\"endpoint required\"):\n AnalyticsConfig(enabled=True, backend='influxdb', endpoint='')\n \n with pytest.raises(ValueError, match=\"batch_size must be positive\"):\n AnalyticsConfig(\n enabled=True,\n backend='influxdb',\n endpoint='http://localhost',\n batch_size=-1\n )\n\n def test_config_loading():\n \"\"\"Test full configuration loading.\"\"\"\n loader = ConfigLoader()\n config = loader.load_ufo_config('config/ufo')\n \n # Verify custom configuration loaded\n assert hasattr(config, 'analytics')\n assert config.analytics.enabled in [True, False]\n```\n\n---\n\n## Next Steps\n\n- **[Agents Configuration](./agents_config.md)** - LLM and agent settings\n- **[System Configuration](./system_config.md)** - Runtime and execution settings\n- **[RAG Configuration](./rag_config.md)** - Knowledge retrieval settings\n- **[Migration Guide](./migration.md)** - Migrate from legacy configuration\n- **[Configuration Overview](./overview.md)** - Understand configuration system design\n"
},
{
"path": "documents/docs/configuration/system/galaxy_agent.md",
"content": "# Galaxy Constellation Agent Configuration\n\n**agent.yaml** configures the **Constellation Agent** - the AI agent responsible for creating constellations (task decomposition) and editing them based on execution results.\n\n---\n\n## Overview\n\nThe **agent.yaml** configuration file provides **LLM and API settings** for the Constellation Agent. This agent is responsible for:\n\n- **Constellation Creation**: Breaking down user requests into device-specific tasks\n- **Constellation Editing**: Adjusting task plans based on execution results\n- **Device Selection**: Choosing appropriate devices for each sub-task\n- **Task Orchestration**: Coordinating multi-device workflows\n\n**Configuration Separation:**\n\n- **agent.yaml** - LLM configuration for constellation agent (this document)\n- **constellation.yaml** - Runtime settings for orchestrator ([Galaxy Constellation Configuration](./galaxy_constellation.md))\n- **devices.yaml** - Device definitions ([Galaxy Devices Configuration](./galaxy_devices.md))\n\n**Agent Role in System:**\n\n```mermaid\ngraph TB\n A[User Request] -->|Natural Language| B[Constellation Agent]\n B -->|Uses LLM Config| C[agent.yaml]\n B -->|Creates/Edits| D[Constellation Plan]\n D -->|Tasks| E[Device Agent 1]\n D -->|Tasks| F[Device Agent 2]\n D -->|Tasks| G[Device Agent N]\n \n style B fill:#e1f5ff\n style C fill:#ffe1e1\n style D fill:#fff4e1\n```\n\n---\n\n## File Location\n\n**Standard Location:**\n\n```\nUFO2/\n├── config/\n│ └── galaxy/\n│ ├── agent.yaml # ← Constellation agent config (copy from template)\n│ ├── agent.yaml.template # ← Template for initial setup\n│ ├── constellation.yaml # ← Runtime settings\n│ └── devices.yaml # ← Device definitions\n```\n\n!!!warning \"Setup Required\"\n 1. Copy `agent.yaml.template` to `agent.yaml`\n 2. Fill in your API credentials (API_KEY, AAD_TENANT_ID, etc.)\n 3. Never commit `agent.yaml` with real credentials to version control\n\n**Loading in Code:**\n\n```python\nfrom config.config_loader import get_galaxy_config\n\n# Load Galaxy configuration (includes agent settings)\nconfig = get_galaxy_config()\n\n# Access constellation agent settings\nagent_config = config.constellation_agent\nreasoning_model = agent_config.reasoning_model\napi_type = agent_config.api_type\napi_model = agent_config.api_model\n```\n\n---\n\n## Configuration Schema\n\n### Complete Schema\n\n```yaml\n# Galaxy Constellation Agent Configuration\n\nCONSTELLATION_AGENT:\n # Reasoning\n REASONING_MODEL: bool # Enable reasoning/chain-of-thought\n \n # API Connection\n API_TYPE: string # API provider type\n API_BASE: string # API base URL\n API_KEY: string # API authentication key\n API_VERSION: string # API version\n API_MODEL: string # Model name/deployment\n \n # Azure AD Authentication (for azure_ad API_TYPE)\n AAD_TENANT_ID: string # Azure AD tenant ID\n AAD_API_SCOPE: string # API scope name\n AAD_API_SCOPE_BASE: string # API scope base GUID\n \n # Prompt Configuration\n CONSTELLATION_CREATION_PROMPT: string # Path to creation prompt template\n CONSTELLATION_EDITING_PROMPT: string # Path to editing prompt template\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: string # Path to creation examples\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: string # Path to editing examples\n```\n\n---\n\n## Configuration Fields\n\n### Reasoning Capabilities\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `REASONING_MODEL` | `bool` | No | `False` | Enable chain-of-thought reasoning for complex planning |\n\n**Example:**\n\n```yaml\nCONSTELLATION_AGENT:\n REASONING_MODEL: False # Standard LLM response (faster)\n```\n\n!!!tip \"Reasoning Model\"\n Set `REASONING_MODEL: True` for:\n - Complex multi-device workflows\n - Tasks requiring step-by-step planning\n - Debugging constellation failures\n \n **Trade-off:** Slower response time, higher token cost\n\n---\n\n### API Connection Settings\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `API_TYPE` | `string` | Yes | - | API provider: `\"openai\"`, `\"azure\"`, `\"azure_ad\"`, `\"aoai\"` |\n| `API_BASE` | `string` | Yes* | - | API base URL (required for Azure) |\n| `API_KEY` | `string` | Yes* | - | API authentication key (required for non-AAD auth) |\n| `API_VERSION` | `string` | Yes* | - | API version (required for Azure) |\n| `API_MODEL` | `string` | Yes | - | Model name or deployment name |\n\n**Supported API Types:**\n\n| API_TYPE | Provider | Authentication | Example API_BASE |\n|----------|----------|----------------|------------------|\n| `openai` | OpenAI | API Key | Not required (uses default) |\n| `azure` | Azure OpenAI | API Key | `https://your-resource.openai.azure.com/` |\n| `azure_ad` | Azure OpenAI | Azure AD (AAD) | `https://your-resource.azure-api.net/` |\n| `aoai` | Azure OpenAI (alias) | API Key | `https://your-resource.openai.azure.com/` |\n\n---\n\n#### Example 1: OpenAI Configuration\n\n```yaml\nCONSTELLATION_AGENT:\n API_TYPE: \"openai\"\n API_KEY: \"sk-proj-...\" # Your OpenAI API key\n API_MODEL: \"gpt-4o\" # OpenAI model name\n API_VERSION: \"2024-02-01\" # Optional for OpenAI\n```\n\n---\n\n#### Example 2: Azure OpenAI (API Key Auth)\n\n```yaml\nCONSTELLATION_AGENT:\n API_TYPE: \"azure\"\n API_BASE: \"https://my-resource.openai.azure.com/\"\n API_KEY: \"abc123...\" # Azure OpenAI API key\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-4o-deployment\" # Your deployment name\n```\n\n---\n\n#### Example 3: Azure OpenAI (Azure AD Auth)\n\n```yaml\nCONSTELLATION_AGENT:\n API_TYPE: \"azure_ad\"\n API_BASE: \"https://cloudgpt-openai.azure-api.net/\"\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-5-chat-20251003\"\n \n # Azure AD Configuration\n AAD_TENANT_ID: \"72f988bf-86f1-41af-91ab-2d7cd011db47\"\n AAD_API_SCOPE: \"openai\"\n AAD_API_SCOPE_BASE: \"feb7b661-cac7-44a8-8dc1-163b63c23df2\"\n```\n\n!!!warning \"Azure AD Authentication\"\n When using `API_TYPE: \"azure_ad\"`:\n - No `API_KEY` needed (uses Azure AD token)\n - Requires `AAD_TENANT_ID`, `AAD_API_SCOPE`, `AAD_API_SCOPE_BASE`\n - User must be authenticated with `az login` or have proper AAD credentials\n\n---\n\n### Azure AD Fields (azure_ad API_TYPE only)\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `AAD_TENANT_ID` | `string` | Yes* | Azure AD tenant GUID |\n| `AAD_API_SCOPE` | `string` | Yes* | API scope identifier (e.g., \"openai\") |\n| `AAD_API_SCOPE_BASE` | `string` | Yes* | API scope base GUID |\n\n*Required only when `API_TYPE: \"azure_ad\"`\n\n---\n\n### Prompt Configuration Paths\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `CONSTELLATION_CREATION_PROMPT` | `string` | Yes | - | Path to constellation creation prompt template |\n| `CONSTELLATION_EDITING_PROMPT` | `string` | Yes | - | Path to constellation editing prompt template |\n| `CONSTELLATION_CREATION_EXAMPLE_PROMPT` | `string` | Yes | - | Path to creation examples (few-shot learning) |\n| `CONSTELLATION_EDITING_EXAMPLE_PROMPT` | `string` | Yes | - | Path to editing examples (few-shot learning) |\n\n**Default Prompt Paths:**\n\n```yaml\nCONSTELLATION_AGENT:\n CONSTELLATION_CREATION_PROMPT: \"galaxy/prompts/constellation/share/constellation_creation.yaml\"\n CONSTELLATION_EDITING_PROMPT: \"galaxy/prompts/constellation/share/constellation_editing.yaml\"\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_creation_example.yaml\"\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_editing_example.yaml\"\n```\n\n!!!tip \"Custom Prompts\"\n You can customize prompts for your use case:\n ```yaml\n CONSTELLATION_CREATION_PROMPT: \"custom_prompts/my_constellation_creation.yaml\"\n ```\n\n---\n\n## Complete Examples\n\n### Example 1: Production (Azure AD)\n\n```yaml\n# Galaxy Constellation Agent Configuration - Production\n# Uses Azure OpenAI with Azure AD authentication\n\nCONSTELLATION_AGENT:\n # Capabilities\n REASONING_MODEL: False\n \n # Azure OpenAI (Azure AD Auth)\n API_TYPE: \"azure_ad\"\n API_BASE: \"https://cloudgpt-openai.azure-api.net/\"\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-5-chat-20251003\"\n \n # Azure AD Configuration\n AAD_TENANT_ID: \"72f988bf-86f1-41af-91ab-2d7cd011db47\"\n AAD_API_SCOPE: \"openai\"\n AAD_API_SCOPE_BASE: \"feb7b661-cac7-44a8-8dc1-163b63c23df2\"\n \n # Prompt Configurations\n CONSTELLATION_CREATION_PROMPT: \"galaxy/prompts/constellation/share/constellation_creation.yaml\"\n CONSTELLATION_EDITING_PROMPT: \"galaxy/prompts/constellation/share/constellation_editing.yaml\"\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_creation_example.yaml\"\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_editing_example.yaml\"\n```\n\n---\n\n### Example 2: Development (OpenAI)\n\n```yaml\n# Galaxy Constellation Agent Configuration - Development\n# Uses OpenAI API for quick testing\n\nCONSTELLATION_AGENT:\n # Capabilities\n REASONING_MODEL: True # Enable for debugging\n \n # OpenAI API\n API_TYPE: \"openai\"\n API_KEY: \"sk-proj-...\" # Your OpenAI API key (DO NOT COMMIT!)\n API_MODEL: \"gpt-4o\"\n API_VERSION: \"2024-02-01\"\n \n # Prompt Configurations (default paths)\n CONSTELLATION_CREATION_PROMPT: \"galaxy/prompts/constellation/share/constellation_creation.yaml\"\n CONSTELLATION_EDITING_PROMPT: \"galaxy/prompts/constellation/share/constellation_editing.yaml\"\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_creation_example.yaml\"\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_editing_example.yaml\"\n```\n\n---\n\n### Example 3: Azure OpenAI (API Key)\n\n```yaml\n# Galaxy Constellation Agent Configuration - Azure (API Key Auth)\n# Uses Azure OpenAI with API key authentication\n\nCONSTELLATION_AGENT:\n # Capabilities\n REASONING_MODEL: False\n \n # Azure OpenAI (API Key Auth)\n API_TYPE: \"azure\"\n API_BASE: \"https://my-openai-resource.openai.azure.com/\"\n API_KEY: \"abc123...\" # Azure OpenAI API key (DO NOT COMMIT!)\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-4o-deployment-name\"\n \n # Prompt Configurations\n CONSTELLATION_CREATION_PROMPT: \"galaxy/prompts/constellation/share/constellation_creation.yaml\"\n CONSTELLATION_EDITING_PROMPT: \"galaxy/prompts/constellation/share/constellation_editing.yaml\"\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_creation_example.yaml\"\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_editing_example.yaml\"\n```\n\n---\n\n## Security Best Practices\n\n!!!danger \"Never Commit Credentials\"\n **DO NOT commit `agent.yaml` with real credentials to version control!**\n \n ✅ **Recommended Workflow:**\n ```bash\n # 1. Copy template\n cp config/galaxy/agent.yaml.template config/galaxy/agent.yaml\n \n # 2. Edit agent.yaml with your credentials\n # (This file is .gitignored)\n \n # 3. Commit only the template\n git add config/galaxy/agent.yaml.template\n git commit -m \"Update agent template\"\n ```\n\n**Use Environment Variables for Sensitive Data:**\n\n```yaml\n# In agent.yaml\nCONSTELLATION_AGENT:\n API_KEY: ${GALAXY_API_KEY} # Read from environment variable\n```\n\n```bash\n# In your shell\nexport GALAXY_API_KEY=\"sk-proj-...\"\n```\n\n---\n\n## Integration with Other Configurations\n\nThe agent configuration works together with other Galaxy configs:\n\n**agent.yaml** (LLM config) + **constellation.yaml** (runtime) + **devices.yaml** (devices) → **Complete Galaxy System**\n\n### Complete Initialization Example\n\n```python\nfrom config.config_loader import get_galaxy_config\nfrom galaxy.agents.constellation_agent import ConstellationAgent\nfrom galaxy.client.device_manager import ConstellationDeviceManager\nimport yaml\n\n# 1. Load all Galaxy configurations\ngalaxy_config = get_galaxy_config()\n\n# 2. Initialize Constellation Agent with LLM config\nagent = ConstellationAgent(\n reasoning_model=galaxy_config.constellation_agent.reasoning_model,\n api_type=galaxy_config.constellation_agent.api_type,\n api_base=galaxy_config.constellation_agent.api_base,\n api_key=galaxy_config.constellation_agent.api_key,\n api_version=galaxy_config.constellation_agent.api_version,\n api_model=galaxy_config.constellation_agent.api_model\n)\n\n# 3. Load constellation runtime settings\nwith open(\"config/galaxy/constellation.yaml\", \"r\") as f:\n constellation_config = yaml.safe_load(f)\n\n# 4. Initialize Device Manager with runtime settings\ndevice_manager = ConstellationDeviceManager(\n task_name=constellation_config[\"CONSTELLATION_ID\"],\n heartbeat_interval=constellation_config[\"HEARTBEAT_INTERVAL\"],\n reconnect_delay=constellation_config[\"RECONNECT_DELAY\"]\n)\n\n# 5. Load and register devices\ndevice_config_path = constellation_config[\"DEVICE_INFO\"]\nwith open(device_config_path, \"r\") as f:\n devices_config = yaml.safe_load(f)\n\nfor device in devices_config[\"devices\"]:\n await device_manager.register_device(**device)\n\nprint(\"✅ Galaxy Constellation System Initialized\")\nprint(f\" Agent Model: {galaxy_config.constellation_agent.api_model}\")\nprint(f\" Constellation ID: {constellation_config['CONSTELLATION_ID']}\")\nprint(f\" Devices: {len(devices_config['devices'])}\")\n```\n\n---\n\n## Best Practices\n\n**Configuration Best Practices:**\n\n1. **Use Templates for Team Collaboration**\n ```bash\n # Share template, not credentials\n config/galaxy/agent.yaml.template # ✅ Commit this\n config/galaxy/agent.yaml # ❌ Never commit this\n ```\n\n2. **Test with OpenAI, Deploy with Azure**\n ```yaml\n # Development: OpenAI (fast iteration)\n API_TYPE: \"openai\"\n \n # Production: Azure (enterprise features)\n API_TYPE: \"azure_ad\"\n ```\n\n3. **Use Reasoning Mode Selectively**\n ```yaml\n # For complex workflows\n REASONING_MODEL: True\n \n # For simple tasks\n REASONING_MODEL: False # Faster\n ```\n\n---\n\n## Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Constellation Runtime** | [Galaxy Constellation Configuration](./galaxy_constellation.md) | Runtime settings for orchestrator |\n| **Device Configuration** | [Galaxy Devices Configuration](./galaxy_devices.md) | Device definitions |\n| **System Configuration** | [Configuration Overview](./overview.md) | Overall configuration architecture |\n\n---\n\n## Next Steps\n\n1. **Copy Template**: `cp agent.yaml.template agent.yaml`\n2. **Configure Credentials**: Fill in API_KEY or AAD settings\n3. **Configure Runtime**: See [Galaxy Constellation Configuration](./galaxy_constellation.md)\n4. **Configure Devices**: See [Galaxy Devices Configuration](./galaxy_devices.md)\n5. **Test Constellation**: Run Galaxy orchestrator\n\n---\n\n## Source Code References\n\n- **ConstellationAgent**: `galaxy/agents/constellation_agent.py`\n- **Configuration Loading**: `config/config_loader.py`\n- **Configuration Schemas**: `config/config_schemas.py`\n- **Prompt Templates**: `galaxy/prompts/constellation/`\n"
},
{
"path": "documents/docs/configuration/system/galaxy_constellation.md",
"content": "# Galaxy Constellation Runtime Configuration\n\n**constellation.yaml** defines constellation-wide runtime settings that control how the Galaxy orchestrator manages devices, tasks, and logging across the entire constellation system.\n\n---\n\n## Overview\n\nThe **constellation.yaml** configuration file provides **constellation-level runtime settings** that apply to the entire Galaxy system. These settings control:\n\n- Constellation identification and logging\n- Heartbeat and connection management\n- Task concurrency and step limits\n- Device configuration file path\n\n**Configuration Separation:**\n\n- **constellation.yaml** - Runtime settings for the constellation orchestrator (this document)\n- **devices.yaml** - Individual device definitions ([Galaxy Devices Configuration](./galaxy_devices.md))\n- **agent.yaml** - LLM configuration for constellation agent ([Galaxy Agent Configuration](./galaxy_agent.md))\n\n**Configuration Relationship:**\n\n```mermaid\ngraph TB\n A[constellation.yaml] -->|Runtime Settings| B[ConstellationDeviceManager]\n C[devices.yaml] -->|Device Definitions| B\n D[agent.yaml] -->|LLM Config| E[ConstellationAgent]\n B -->|Orchestrates| F[Device Agents]\n E -->|Plans Tasks| B\n \n style A fill:#e1f5ff\n style C fill:#fff4e1\n style D fill:#ffe1e1\n```\n\n---\n\n## File Location\n\n**Standard Location:**\n\n```\nUFO2/\n├── config/\n│ └── galaxy/\n│ ├── constellation.yaml # ← Runtime settings (this file)\n│ ├── devices.yaml # ← Device definitions\n│ └── agent.yaml.template # ← Agent LLM configuration template\n```\n\n**Loading in Code:**\n\n```python\nimport yaml\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Load constellation configuration\nwith open(\"config/galaxy/constellation.yaml\", \"r\", encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n\n# Initialize ConstellationDeviceManager with runtime settings\nmanager = ConstellationDeviceManager(\n task_name=config[\"CONSTELLATION_ID\"],\n heartbeat_interval=config[\"HEARTBEAT_INTERVAL\"],\n reconnect_delay=config[\"RECONNECT_DELAY\"]\n)\n\n# Load device configuration from specified path\ndevice_config_path = config[\"DEVICE_INFO\"]\nwith open(device_config_path, \"r\", encoding=\"utf-8\") as f:\n devices_config = yaml.safe_load(f)\n\n# Register devices\nfor device in devices_config[\"devices\"]:\n await manager.register_device(**device)\n```\n\n---\n\n## Configuration Schema\n\n### Complete Schema\n\n```yaml\n# Galaxy Constellation Configuration\n# Runtime settings for constellation system\n\n# Constellation Identity & Logging\nCONSTELLATION_ID: string # Unique constellation identifier\nLOG_TO_MARKDOWN: bool # Save trajectory logs to markdown\n\n# Connection & Health Management\nHEARTBEAT_INTERVAL: float # Heartbeat check interval (seconds)\nRECONNECT_DELAY: float # Reconnection delay (seconds)\n\n# Task & Execution Limits\nMAX_CONCURRENT_TASKS: int # Maximum concurrent tasks\nMAX_STEP: int # Maximum steps per session\n\n# Device Configuration Reference\nDEVICE_INFO: string # Path to devices.yaml file\n```\n\n---\n\n## Configuration Fields\n\n### Constellation Identity & Logging\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `CONSTELLATION_ID` | `string` | Yes | - | Unique identifier for this constellation instance |\n| `LOG_TO_MARKDOWN` | `bool` | No | `true` | Whether to save trajectory logs in markdown format |\n\n**Example:**\n\n```yaml\nCONSTELLATION_ID: \"production_constellation\"\nLOG_TO_MARKDOWN: true\n```\n\n**Constellation ID Best Practices:**\n\nUse descriptive names that indicate environment and purpose:\n- `production_main` - Main production constellation\n- `dev_testing` - Development testing constellation\n- `qa_regression` - QA regression testing constellation\n\n---\n\n### Connection & Health Management\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `HEARTBEAT_INTERVAL` | `float` | No | `30.0` | Interval (in seconds) between heartbeat checks for connected devices |\n| `RECONNECT_DELAY` | `float` | No | `5.0` | Delay (in seconds) before attempting to reconnect a failed device |\n\n**Example:**\n\n```yaml\nHEARTBEAT_INTERVAL: 30.0 # Check device health every 30 seconds\nRECONNECT_DELAY: 5.0 # Wait 5 seconds before reconnecting\n```\n\n!!!info \"Heartbeat Mechanism\"\n The heartbeat system monitors device agent connections:\n - Every `HEARTBEAT_INTERVAL` seconds, the constellation checks if devices are responsive\n - If a device fails to respond, it is marked as `FAILED`\n - After `RECONNECT_DELAY` seconds, automatic reconnection is attempted\n - Reconnection continues until `max_retries` is reached (configured per-device in devices.yaml)\n\n**Tuning Guidelines:**\n\n| Environment | HEARTBEAT_INTERVAL | RECONNECT_DELAY | Rationale |\n|-------------|-------------------|-----------------|-----------|\n| **Production** | 10.0 - 30.0 | 5.0 - 10.0 | Balance responsiveness with network overhead |\n| **Development** | 30.0 - 60.0 | 3.0 - 5.0 | Reduce noise during debugging |\n| **Testing** | 5.0 - 10.0 | 2.0 - 3.0 | Faster failure detection for tests |\n\n---\n\n### Task & Execution Limits\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `MAX_CONCURRENT_TASKS` | `int` | No | `6` | Maximum number of tasks that can run concurrently across all devices |\n| `MAX_STEP` | `int` | No | `15` | Maximum number of steps allowed per session before termination |\n\n**Example:**\n\n```yaml\nMAX_CONCURRENT_TASKS: 6 # Allow 6 tasks to run simultaneously\nMAX_STEP: 15 # Limit sessions to 15 steps\n```\n\n!!!warning \"Concurrency Considerations\"\n - **MAX_CONCURRENT_TASKS** controls task queue parallelism across the entire constellation\n - Each device can handle 1 task at a time (per device, not global)\n - Example: 6 devices + MAX_CONCURRENT_TASKS=6 → All devices can be busy simultaneously\n - Example: 10 devices + MAX_CONCURRENT_TASKS=4 → Only 4 devices busy at once, 6 idle\n\n**Task Concurrency Calculation:**\n\n```python\n# Effective concurrency\neffective_concurrency = min(\n num_registered_devices,\n MAX_CONCURRENT_TASKS\n)\n\n# Example 1: 3 devices, MAX_CONCURRENT_TASKS=6\n# → effective_concurrency = 3 (device-limited)\n\n# Example 2: 10 devices, MAX_CONCURRENT_TASKS=4\n# → effective_concurrency = 4 (config-limited)\n```\n\n**MAX_STEP Guidelines:**\n\n| Use Case | MAX_STEP | Rationale |\n|----------|----------|-----------|\n| **Simple Automation** | 5 - 10 | Quick tasks (open app, click button) |\n| **Complex Workflows** | 15 - 30 | Multi-step processes (data entry, reporting) |\n| **Unrestricted** | 100+ | Research, exploratory tasks |\n\n---\n\n### Device Configuration Reference\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `DEVICE_INFO` | `string` | Yes | - | Relative or absolute path to `devices.yaml` configuration file |\n\n**Example:**\n\n```yaml\nDEVICE_INFO: \"config/galaxy/devices.yaml\"\n```\n\n**Path Resolution:**\n\n- **Relative paths** are resolved from the UFO2 project root\n- **Absolute paths** are supported for external configuration files\n- The loader validates that the file exists and is readable\n\n**Example Paths:**\n\n```yaml\n# Relative path (recommended)\nDEVICE_INFO: \"config/galaxy/devices.yaml\"\n\n# Absolute path\nDEVICE_INFO: \"/etc/ufo/galaxy/devices.yaml\"\n\n# Different config for testing\nDEVICE_INFO: \"config/galaxy/devices_test.yaml\"\n```\n\n---\n\n## Complete Examples\n\n### Example 1: Production Configuration\n\n```yaml\n# Galaxy Constellation Configuration - Production\n# High reliability, moderate concurrency\n\n# Identity & Logging\nCONSTELLATION_ID: \"production_main\"\nLOG_TO_MARKDOWN: true\n\n# Connection & Health\nHEARTBEAT_INTERVAL: 15.0 # Fast failure detection\nRECONNECT_DELAY: 10.0 # Give devices time to recover\n\n# Task Limits\nMAX_CONCURRENT_TASKS: 10 # High concurrency for production load\nMAX_STEP: 20 # Allow complex workflows\n\n# Device Configuration\nDEVICE_INFO: \"config/galaxy/devices.yaml\"\n```\n\n**Use Case:** Production constellation managing office automation across 10+ devices.\n\n---\n\n### Example 2: Development Configuration\n\n```yaml\n# Galaxy Constellation Configuration - Development\n# Relaxed settings for testing and debugging\n\n# Identity & Logging\nCONSTELLATION_ID: \"dev_testing\"\nLOG_TO_MARKDOWN: true\n\n# Connection & Health\nHEARTBEAT_INTERVAL: 60.0 # Reduce noise during debugging\nRECONNECT_DELAY: 5.0 # Fast reconnects for quick iteration\n\n# Task Limits\nMAX_CONCURRENT_TASKS: 3 # Limit concurrency for easier debugging\nMAX_STEP: 50 # Allow exploration and experimentation\n\n# Device Configuration\nDEVICE_INFO: \"config/galaxy/devices_dev.yaml\"\n```\n\n**Use Case:** Development environment with 2-3 test devices for feature development.\n\n---\n\n### Example 3: Testing/CI Configuration\n\n```yaml\n# Galaxy Constellation Configuration - CI/CD\n# Fast failure detection, limited concurrency\n\n# Identity & Logging\nCONSTELLATION_ID: \"ci_regression\"\nLOG_TO_MARKDOWN: true\n\n# Connection & Health\nHEARTBEAT_INTERVAL: 5.0 # Very fast detection for CI\nRECONNECT_DELAY: 2.0 # Quick retries in CI environment\n\n# Task Limits\nMAX_CONCURRENT_TASKS: 4 # Parallel test execution\nMAX_STEP: 15 # Strict limits for regression tests\n\n# Device Configuration\nDEVICE_INFO: \"config/galaxy/devices_ci.yaml\"\n```\n\n**Use Case:** Automated testing in CI/CD pipeline with controlled test devices.\n\n---\n\n## Integration with Device Configuration\n\nThe constellation configuration works together with device configuration:\n\n**constellation.yaml (runtime)** + **devices.yaml (device definitions)** → **Complete Constellation System**\n\n### Loading Workflow\n\n```mermaid\nsequenceDiagram\n participant App as Application\n participant Config as constellation.yaml\n participant DevConfig as devices.yaml\n participant Manager as ConstellationDeviceManager\n \n App->>Config: Load constellation.yaml\n Config-->>App: Runtime settings\n \n App->>Manager: Initialize with runtime settings\n Note over Manager: CONSTELLATION_ID, HEARTBEAT_INTERVAL, etc.\n \n App->>Config: Read DEVICE_INFO path\n Config-->>App: \"config/galaxy/devices.yaml\"\n \n App->>DevConfig: Load devices.yaml from path\n DevConfig-->>App: Device definitions\n \n App->>Manager: Register devices\n Manager->>Manager: Apply runtime settings to all devices\n```\n\n### Example: Complete Initialization\n\n```python\nimport yaml\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# 1. Load constellation runtime settings\nwith open(\"config/galaxy/constellation.yaml\", \"r\", encoding=\"utf-8\") as f:\n constellation_config = yaml.safe_load(f)\n\n# 2. Initialize manager with runtime settings\nmanager = ConstellationDeviceManager(\n task_name=constellation_config[\"CONSTELLATION_ID\"],\n heartbeat_interval=constellation_config[\"HEARTBEAT_INTERVAL\"],\n reconnect_delay=constellation_config[\"RECONNECT_DELAY\"]\n)\n\n# 3. Load device configuration from path specified in constellation.yaml\ndevice_config_path = constellation_config[\"DEVICE_INFO\"]\nwith open(device_config_path, \"r\", encoding=\"utf-8\") as f:\n devices_config = yaml.safe_load(f)\n\n# 4. Register all devices\nfor device in devices_config[\"devices\"]:\n await manager.register_device(\n device_id=device[\"device_id\"],\n server_url=device[\"server_url\"],\n os=device.get(\"os\"),\n capabilities=device.get(\"capabilities\", []),\n metadata=device.get(\"metadata\", {}),\n max_retries=device.get(\"max_retries\", 5),\n auto_connect=device.get(\"auto_connect\", True)\n )\n\nprint(f\"✅ Constellation '{constellation_config['CONSTELLATION_ID']}' initialized\")\nprint(f\" Devices registered: {len(devices_config['devices'])}\")\nprint(f\" Max concurrent tasks: {constellation_config['MAX_CONCURRENT_TASKS']}\")\n```\n\n---\n\n## Best Practices\n\n**Configuration Best Practices:**\n\n1. **Use Environment-Specific Configurations**\n ```bash\n config/galaxy/\n ├── constellation.yaml # Base production config\n ├── constellation_dev.yaml # Development overrides\n ├── constellation_test.yaml # Testing overrides\n ```\n\n2. **Tune Heartbeat for Your Network**\n ```yaml\n # Local network - fast heartbeats\n HEARTBEAT_INTERVAL: 10.0\n \n # WAN/Internet - slower heartbeats\n HEARTBEAT_INTERVAL: 60.0\n ```\n\n3. **Match Concurrency to Use Case**\n ```yaml\n # High-throughput automation\n MAX_CONCURRENT_TASKS: 20\n \n # Resource-constrained environment\n MAX_CONCURRENT_TASKS: 3\n ```\n\n4. **Set Reasonable Step Limits**\n ```yaml\n # Prevent runaway sessions\n MAX_STEP: 30\n \n # For debugging (see all steps)\n MAX_STEP: 100\n ```\n\n---\n\n## Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Device Configuration** | [Galaxy Devices Configuration](./galaxy_devices.md) | Device definitions and capabilities |\n| **Agent Configuration** | [Galaxy Agent Configuration](./galaxy_agent.md) | LLM settings for constellation agent |\n| **Agent Registration** | [Agent Registration Overview](../../galaxy/agent_registration/overview.md) | Registration process and architecture |\n| **System Configuration** | [Configuration Overview](./overview.md) | Overall configuration architecture |\n\n---\n\n## Next Steps\n\n1. **Configure Devices**: See [Galaxy Devices Configuration](./galaxy_devices.md)\n2. **Configure Agent**: See [Galaxy Agent Configuration](./galaxy_agent.md)\n3. **Understand Registration**: Read [Agent Registration Overview](../../galaxy/agent_registration/overview.md)\n4. **Run Constellation**: Check Galaxy orchestrator documentation\n\n---\n\n## Source Code References\n\n- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`\n- **Configuration Loading**: `config/config_loader.py`\n- **Configuration Schemas**: `config/config_schemas.py`\n"
},
{
"path": "documents/docs/configuration/system/galaxy_devices.md",
"content": "# Galaxy Devices Configuration\n\nDevice configuration in **devices.yaml** defines the constellation's device agents, providing device identity, capabilities, metadata, and connection parameters for each agent in the constellation.\n\n---\n\n## Overview\n\nThe **devices.yaml** configuration file defines the **devices array** for the Galaxy constellation system. It provides:\n\n- Device identity and endpoint information\n- User-specified capabilities\n- Custom metadata and preferences\n- Connection and retry parameters\n\n**Constellation vs Device Configuration:**\n\n- **devices.yaml** - Defines individual device agents (this document)\n- **constellation.yaml** - Defines constellation-wide runtime settings\n- See [Galaxy Constellation Configuration](./galaxy_constellation.md) for runtime settings\n\n**Configuration Flow:**\n\n```mermaid\ngraph LR\n A[devices.yaml] -->|Load| B[ConstellationDeviceManager]\n B -->|Parse| C[Device Entries]\n C -->|For Each Device| D[DeviceRegistry.register_device]\n D -->|Create| E[AgentProfile v1]\n E -->|If auto_connect| F[Connection Process]\n F -->|Merge| G[Complete AgentProfile]\n \n style A fill:#e1f5ff\n style E fill:#fff4e1\n style G fill:#c8e6c9\n```\n\n---\n\n## 📁 File Location\n\n**Standard Location:**\n\n```\nUFO2/\n├── config/\n └── galaxy/\n ├── devices.yaml # 📄 Device definitions (this file)\n ├── constellation.yaml # ⚙️ Runtime settings\n └── agent.yaml.template # 🤖 Agent LLM configuration template\n```\n\n**Loading in Code:**\n\n```python\nfrom galaxy.client.device_manager import ConstellationDeviceManager\nimport yaml\n\n# Load device configuration\nwith open(\"config/galaxy/devices.yaml\", \"r\", encoding=\"utf-8\") as f:\n devices_config = yaml.safe_load(f)\n\n# Load constellation configuration\nwith open(\"config/galaxy/constellation.yaml\", \"r\", encoding=\"utf-8\") as f:\n constellation_config = yaml.safe_load(f)\n\n# Initialize manager with constellation settings\nmanager = ConstellationDeviceManager(\n task_name=constellation_config.get(\"CONSTELLATION_ID\", \"default\"),\n heartbeat_interval=constellation_config.get(\"HEARTBEAT_INTERVAL\", 30.0),\n reconnect_delay=constellation_config.get(\"RECONNECT_DELAY\", 5.0)\n)\n\n# Register devices from devices.yaml\nfor device_config in devices_config[\"devices\"]:\n await manager.register_device(\n device_id=device_config[\"device_id\"],\n server_url=device_config[\"server_url\"],\n os=device_config.get(\"os\"),\n capabilities=device_config.get(\"capabilities\", []),\n metadata=device_config.get(\"metadata\", {}),\n max_retries=device_config.get(\"max_retries\", 5),\n auto_connect=device_config.get(\"auto_connect\", True)\n )\n```\n\n---\n\n## 📝 Configuration Schema\n\n### File Structure\n\n```yaml\n# Device Configuration - YAML Format\n# Defines devices for the constellation\n# Runtime settings are configured in constellation.yaml\n\ndevices: # List of device configurations\n - device_id: string # Unique device identifier\n server_url: string # WebSocket URL of device agent\n os: string # Operating system\n capabilities: list[string] # Device capabilities\n metadata: dict # Custom metadata\n max_retries: int # Connection retry limit\n auto_connect: bool # Auto-connect on registration\n```\n\n---\n\n### Device Configuration Fields\n\n#### Required Fields\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `device_id` | `string` | **Unique device identifier** | `\"windowsagent\"`, `\"linux_server_01\"` |\n| `server_url` | `string` | **WebSocket endpoint URL** | `\"ws://localhost:5005/ws\"` |\n\n!!!danger \"Required Fields\"\n `device_id` and `server_url` are **required** for every device. Registration will fail without them.\n\n#### Optional Fields\n\n| Field | Type | Default | Description | Example |\n|-------|------|---------|-------------|---------|\n| `os` | `string` | `None` | Operating system type | `\"windows\"`, `\"linux\"`, `\"darwin\"` |\n| `capabilities` | `list[string]` | `[]` | Device capabilities | `[\"web_browsing\", \"office\"]` |\n| `metadata` | `dict` | `{}` | Custom metadata | See [Metadata Fields](#metadata-fields) |\n| `max_retries` | `int` | `5` | Maximum connection retries | `3`, `10` |\n| `auto_connect` | `bool` | `true` | Auto-connect after registration | `true`, `false` |\n\n!!!danger \"Required Fields\"\n `device_id` and `server_url` are **required** for every device. Registration will fail without them.\n\n---\n\n---\n\n### Metadata Fields\n\nThe `metadata` dictionary is **completely flexible** and can contain any custom fields. However, some common patterns are recommended:\n\n**Recommended Metadata Fields:**\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `location` | `string` | Physical location | `\"office_desktop\"`, `\"datacenter_rack_a42\"` |\n| `performance` | `string` | Performance tier | `\"low\"`, `\"medium\"`, `\"high\"`, `\"very_high\"` |\n| `description` | `string` | Human-readable description | `\"Primary Windows workstation\"` |\n| `tags` | `list[string]` | Custom tags | `[\"production\", \"gpu\", \"critical\"]` |\n| `operation_engineer_email` | `string` | Contact email | `\"admin@example.com\"` |\n| `operation_engineer_name` | `string` | Contact name | `\"John Doe\"` |\n\n**Custom Fields (Application-Specific):**\n\n```yaml\nmetadata:\n # File paths\n logs_file_path: \"/var/log/application.log\"\n dev_path: \"/home/deploy/projects/\"\n app_log_file: \"log_detailed.xlsx\"\n \n # Excel logging\n sheet_name_for_writing_log_in_excel: \"report\"\n \n # Email configuration\n sender_name: \"Automation Bot\"\n \n # Log patterns\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR or FATAL\"\n \n # GPU information\n gpu_type: \"NVIDIA RTX 4090\"\n gpu_count: 2\n gpu_memory_gb: 48\n```\n\n---\n\n## 📚 Complete Example\n\n### Example 1: Multi-Device Constellation\n\n```yaml\n# Device Configuration - YAML Format\n# Defines devices for the constellation\n# Runtime settings (constellation_id, heartbeat_interval, etc.) are configured in constellation.yaml\n\ndevices:\n # ===== Windows Desktop Agent =====\n - device_id: \"windowsagent\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"web_browsing\"\n - \"office_applications\"\n - \"file_management\"\n - \"email_sending\"\n metadata:\n location: \"office_desktop\"\n performance: \"high\"\n description: \"Primary Windows workstation for office automation\"\n operation_engineer_email: \"admin@example.com\"\n operation_engineer_name: \"John Doe\"\n sender_name: \"Office Bot\"\n app_log_file: \"automation_log.xlsx\"\n sheet_name_for_writing_log_in_excel: \"report\"\n tags:\n - \"production\"\n - \"office\"\n - \"critical\"\n max_retries: 5\n auto_connect: true\n\n # ===== Linux Server 1 =====\n - device_id: \"linux_server_01\"\n server_url: \"ws://10.0.1.50:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server_management\"\n - \"log_monitoring\"\n - \"database_operations\"\n metadata:\n location: \"datacenter_rack_a42\"\n performance: \"medium\"\n description: \"Production Linux server for backend services\"\n logs_file_path: \"/var/log/application.log\"\n dev_path: \"/home/deploy/projects/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR or FATAL\"\n tags:\n - \"production\"\n - \"backend\"\n - \"monitoring\"\n max_retries: 3\n auto_connect: true\n\n # ===== Linux Server 2 =====\n - device_id: \"linux_server_02\"\n server_url: \"ws://10.0.1.51:5002/ws\"\n os: \"linux\"\n capabilities:\n - \"server_management\"\n - \"log_monitoring\"\n - \"database_operations\"\n metadata:\n location: \"datacenter_rack_a43\"\n performance: \"medium\"\n description: \"Secondary Linux server for load balancing\"\n logs_file_path: \"/var/log/application.log\"\n dev_path: \"/home/deploy/projects/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR or FATAL\"\n tags:\n - \"production\"\n - \"backend\"\n - \"load_balancer\"\n max_retries: 3\n auto_connect: true\n\n # ===== GPU Workstation =====\n - device_id: \"gpu_workstation\"\n server_url: \"ws://192.168.1.100:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"gpu_computation\"\n - \"model_training\"\n - \"data_processing\"\n - \"deep_learning\"\n metadata:\n location: \"ml_lab\"\n performance: \"very_high\"\n description: \"High-performance GPU workstation for ML training\"\n operation_engineer_email: \"ml-team@example.com\"\n gpu_type: \"NVIDIA RTX 4090\"\n gpu_count: 2\n gpu_memory_gb: 48\n cpu_count: 32\n memory_total_gb: 128\n tags:\n - \"production\"\n - \"ml\"\n - \"gpu\"\n - \"high_priority\"\n max_retries: 10\n auto_connect: true\n```\n\n### Example 2: Development Environment\n\n```yaml\n# Device Configuration - YAML Format\n# Runtime settings are configured in constellation.yaml\n\ndevices:\n - device_id: \"dev_windows\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"web_browsing\"\n - \"office_applications\"\n metadata:\n location: \"developer_laptop\"\n performance: \"medium\"\n description: \"Development Windows machine\"\n environment: \"development\"\n max_retries: 3\n auto_connect: true\n\n - device_id: \"dev_linux\"\n server_url: \"ws://localhost:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"cli\"\n - \"file_system\"\n metadata:\n location: \"developer_laptop\"\n performance: \"medium\"\n description: \"Development Linux VM\"\n environment: \"development\"\n max_retries: 3\n auto_connect: false # Manual connection for debugging\n```\n\n---\n\n## 🔄 Multi-Source Metadata Merging\n\nThe `metadata` field in configuration is **Source 1** in the multi-source profiling architecture. It will be merged with:\n\n- **Source 2**: Service-level manifest (registration data)\n- **Source 3**: Client telemetry (DeviceInfoProvider)\n\n### Merging Process\n\n```mermaid\ngraph TB\n subgraph \"Source 1: User Config\"\n UC[metadata in devices.yaml]\n UC --> |location, performance, tags| Final\n end\n \n subgraph \"Source 2: Service Manifest\"\n SM[AIP Registration]\n SM --> |platform, registration_time| Final\n end\n \n subgraph \"Source 3: Client Telemetry\"\n CT[DeviceInfoProvider]\n CT --> |system_info object| Final\n end\n \n Final[Complete metadata in AgentProfile]\n \n style UC fill:#e1f5ff\n style SM fill:#fff4e1\n style CT fill:#e8f5e9\n style Final fill:#f3e5f5\n```\n\n**Before Merging (User Config Only):**\n\n```yaml\nmetadata:\n location: \"office_desktop\"\n performance: \"high\"\n description: \"Primary Windows workstation\"\n```\n\n**After Merging (All Sources):**\n\n```python\nmetadata = {\n # Source 1: User Config\n \"location\": \"office_desktop\",\n \"performance\": \"high\",\n \"description\": \"Primary Windows workstation\",\n \n # Source 2: Service Manifest\n \"platform\": \"windows\",\n \"registration_time\": \"2025-11-06T10:30:00Z\",\n \n # Source 3: Client Telemetry\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n }\n}\n```\n\nSee [AgentProfile Documentation](../../galaxy/agent_registration/agent_profile.md#multi-source-construction) for merging details.\n\n---\n\n## 🎯 Use Cases and Patterns\n\n### Pattern 1: Office Automation\n\n```yaml\ndevices:\n - device_id: \"office_pc\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"web_browsing\"\n - \"office_applications\"\n - \"email_sending\"\n - \"file_management\"\n metadata:\n location: \"office_desktop\"\n performance: \"medium\"\n description: \"Office PC for daily automation tasks\"\n operation_engineer_email: \"it@company.com\"\n sender_name: \"Office Automation\"\n app_log_file: \"office_automation.xlsx\"\n```\n\n**Task Assignment:**\n\n```python\n# Find device with office capabilities\ndevices = manager.get_all_devices(connected=True)\nfor device_id, profile in devices.items():\n if \"office_applications\" in profile.capabilities:\n await manager.assign_task_to_device(\n task_id=\"create_report\",\n device_id=device_id,\n task_description=\"Create monthly report in Excel\",\n task_data={\"template\": \"monthly_template.xlsx\"}\n )\n```\n\n### Pattern 2: Server Monitoring\n\n```yaml\ndevices:\n - device_id: \"prod_server_01\"\n server_url: \"ws://10.0.1.50:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server_management\"\n - \"log_monitoring\"\n metadata:\n location: \"datacenter_us_west\"\n performance: \"high\"\n logs_file_path: \"/var/log/app.log\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n```\n\n**Task Assignment:**\n\n```python\n# Monitor server logs\nawait manager.assign_task_to_device(\n task_id=\"monitor_logs\",\n device_id=\"prod_server_01\",\n task_description=\"Check logs for errors\",\n task_data={\n \"log_file\": profile.metadata[\"logs_file_path\"],\n \"error_pattern\": profile.metadata[\"error_log_pattern\"]\n }\n)\n```\n\n### Pattern 3: GPU Computation\n\n```yaml\ndevices:\n - device_id: \"gpu_node_01\"\n server_url: \"ws://192.168.1.100:5005/ws\"\n os: \"linux\"\n capabilities:\n - \"gpu_computation\"\n - \"model_training\"\n - \"data_processing\"\n metadata:\n location: \"ml_lab_rack_01\"\n performance: \"very_high\"\n gpu_type: \"NVIDIA A100\"\n gpu_count: 4\n gpu_memory_gb: 320 # 4 × 80GB\n cpu_count: 96\n memory_total_gb: 1024\n```\n\n**Task Assignment:**\n\n```python\n# Select GPU device based on metadata\ndevices = manager.get_all_devices(connected=True)\nfor device_id, profile in devices.items():\n metadata = profile.metadata\n if (\n \"gpu_computation\" in profile.capabilities\n and metadata.get(\"gpu_count\", 0) >= 4\n and metadata.get(\"gpu_memory_gb\", 0) >= 300\n ):\n await manager.assign_task_to_device(\n task_id=\"train_model\",\n device_id=device_id,\n task_description=\"Train large language model\",\n task_data={\"model\": \"llama-70b\", \"dataset\": \"training_data.json\"}\n )\n```\n\n---\n\n## ⚠️ Validation and Best Practices\n\n### Required Field Validation\n\n```python\ndef validate_device_config(device: dict) -> bool:\n \"\"\"Validate device configuration.\"\"\"\n \n # Required fields\n if \"device_id\" not in device:\n logger.error(\"Missing required field: device_id\")\n return False\n \n if \"server_url\" not in device:\n logger.error(\"Missing required field: server_url\")\n return False\n \n # Validate server_url format\n if not device[\"server_url\"].startswith(\"ws://\") and \\\n not device[\"server_url\"].startswith(\"wss://\"):\n logger.error(f\"Invalid server_url: {device['server_url']}\")\n return False\n \n return True\n```\n\n### Best Practices\n\n!!!tip \"Configuration Best Practices\"\n \n **1. Use Meaningful device_id**\n ```yaml\n # ✅ Good: Descriptive and unique\n device_id: \"windows_office_pc_01\"\n device_id: \"linux_prod_server_us_west_01\"\n device_id: \"gpu_ml_workstation_lab_a\"\n \n # ❌ Bad: Generic or ambiguous\n device_id: \"device1\"\n device_id: \"test\"\n device_id: \"agent\"\n ```\n \n **2. Specify Granular Capabilities**\n ```yaml\n # ✅ Good: Specific capabilities\n capabilities:\n - \"web_browsing_chrome\"\n - \"office_excel_automation\"\n - \"email_outlook\"\n \n # ❌ Bad: Vague capabilities\n capabilities:\n - \"office\"\n - \"internet\"\n ```\n \n **3. Include Rich Metadata**\n ```yaml\n # ✅ Good: Comprehensive metadata\n metadata:\n location: \"datacenter_us_west_rack_a42\"\n performance: \"very_high\"\n description: \"Production GPU server for ML training\"\n tags: [\"production\", \"ml\", \"gpu\", \"critical\"]\n operation_engineer_email: \"ml-ops@company.com\"\n gpu_type: \"NVIDIA A100\"\n gpu_count: 4\n \n # ❌ Bad: Minimal metadata\n metadata:\n location: \"server room\"\n ```\n \n **4. Set Appropriate max_retries**\n ```yaml\n # Critical production devices\n max_retries: 10\n \n # Development/test devices\n max_retries: 3\n ```\n \n **5. Use auto_connect Wisely**\n ```yaml\n # Production: auto-connect\n auto_connect: true\n \n # Development/debugging: manual connect\n auto_connect: false\n ```\n\n---\n\n## 🔧 Loading and Parsing\n\n### Basic Loading\n\n```python\nimport yaml\n\nwith open(\"config/galaxy/devices.yaml\", \"r\", encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n\n# Access constellation-level settings\nconstellation_id = config.get(\"constellation_id\", \"default\")\nheartbeat_interval = config.get(\"heartbeat_interval\", 30.0)\n\n# Access devices\ndevices = config.get(\"devices\", [])\n```\n\n### Loading with Validation\n\n```python\nimport yaml\nfrom typing import Dict, List, Any\n\ndef load_and_validate_config(config_path: str) -> Dict[str, Any]:\n \"\"\"Load and validate devices configuration.\"\"\"\n \n with open(config_path, \"r\", encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n \n # Validate top-level structure\n if \"devices\" not in config:\n raise ValueError(\"Configuration must contain 'devices' list\")\n \n if not isinstance(config[\"devices\"], list):\n raise ValueError(\"'devices' must be a list\")\n \n # Validate each device\n for i, device in enumerate(config[\"devices\"]):\n if \"device_id\" not in device:\n raise ValueError(f\"Device {i}: Missing 'device_id'\")\n \n if \"server_url\" not in device:\n raise ValueError(f\"Device {i}: Missing 'server_url'\")\n \n # Validate URL format\n if not device[\"server_url\"].startswith((\"ws://\", \"wss://\")):\n raise ValueError(\n f\"Device {device['device_id']}: Invalid server_url format\"\n )\n \n return config\n```\n\n### Registration from Config\n\n```python\nasync def register_devices_from_config(\n manager: ConstellationDeviceManager,\n config_path: str\n) -> List[str]:\n \"\"\"Register all devices from configuration file.\"\"\"\n \n config = load_and_validate_config(config_path)\n \n registered = []\n failed = []\n \n for device_config in config[\"devices\"]:\n try:\n success = await manager.register_device(\n device_id=device_config[\"device_id\"],\n server_url=device_config[\"server_url\"],\n os=device_config.get(\"os\"),\n capabilities=device_config.get(\"capabilities\", []),\n metadata=device_config.get(\"metadata\", {}),\n max_retries=device_config.get(\"max_retries\", 5),\n auto_connect=device_config.get(\"auto_connect\", True)\n )\n \n if success:\n registered.append(device_config[\"device_id\"])\n else:\n failed.append(device_config[\"device_id\"])\n \n except Exception as e:\n logger.error(\n f\"Failed to register {device_config['device_id']}: {e}\"\n )\n failed.append(device_config[\"device_id\"])\n \n logger.info(f\"Registered: {len(registered)} devices\")\n if failed:\n logger.warning(f\"Failed: {len(failed)} devices - {failed}\")\n \n return registered\n```\n\n---\n\n## 🔗 Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture |\n| **AgentProfile** | [AgentProfile](../../galaxy/agent_registration/agent_profile.md) | Profile structure and merging |\n| **Registration Flow** | [Registration Flow](../../galaxy/agent_registration/registration_flow.md) | Registration process |\n| **Device Registry** | [Device Registry](../../galaxy/agent_registration/device_registry.md) | Registry component |\n| **Device Info** | [Device Info Provider](../../client/device_info.md) | Telemetry (Source 3) |\n\n---\n\n## 💡 Tips and Tricks\n\n!!!tip \"Advanced Configuration Tips\"\n \n **Use YAML Anchors for Reusable Metadata**\n ```yaml\n # Define reusable metadata templates\n _metadata_templates:\n production_server: &prod_server\n environment: \"production\"\n tags: [\"production\", \"critical\"]\n max_retries: 10\n \n dev_server: &dev_server\n environment: \"development\"\n tags: [\"development\", \"testing\"]\n max_retries: 3\n \n devices:\n - device_id: \"prod_server_01\"\n server_url: \"ws://10.0.1.50:5001/ws\"\n metadata:\n <<: *prod_server # Merge production template\n location: \"datacenter_us_west\"\n \n - device_id: \"dev_server_01\"\n server_url: \"ws://localhost:5001/ws\"\n metadata:\n <<: *dev_server # Merge dev template\n location: \"developer_laptop\"\n ```\n \n **Environment Variable Substitution**\n ```yaml\n # Use environment variables for sensitive data\n devices:\n - device_id: \"prod_server\"\n server_url: \"${SERVER_URL}\" # From environment\n metadata:\n api_key: \"${API_KEY}\"\n ```\n\n---\n\n## 🚀 Next Steps\n\n1. **Create Your Configuration**: Copy example and customize\n2. **Validate Configuration**: Use validation function\n3. **Register Devices**: Load config and register\n4. **Monitor Status**: Check device status after registration\n\n---\n\n## 📚 Source Code References\n\n- **Example Config**: `config/galaxy/devices.yaml`\n- **Loading Logic**: `galaxy/client/device_manager.py`\n- **DeviceRegistry**: `galaxy/client/components/device_registry.py`\n- **AgentProfile**: `galaxy/client/components/types.py`\n"
},
{
"path": "documents/docs/configuration/system/mcp_reference.md",
"content": "# MCP Configuration Reference\n\nThis document provides a quick reference for MCP (Model Context Protocol) server configuration in UFO².\n\nFor comprehensive MCP configuration guide with examples, best practices, and detailed explanations, see:\n\n- **[MCP Configuration Guide](../../mcp/configuration.md)** - Complete configuration documentation\n- [MCP Overview](../../mcp/overview.md) - Architecture and concepts\n- [Data Collection Servers](../../mcp/data_collection.md) - Observation tools\n- [Action Servers](../../mcp/action.md) - Execution tools\n\n## Quick Reference\n\n**Configuration File**: `config/ufo/mcp.yaml`\n\n### Structure\n\n```yaml\nAgentName: # e.g., \"HostAgent\", \"AppAgent\"\n SubType: # \"default\" or app name (e.g., \"WINWORD.EXE\")\n data_collection: # Data collection servers (read-only)\n - namespace: ...\n type: ... # \"local\", \"http\", or \"stdio\"\n action: # Action servers (state-changing)\n - namespace: ...\n type: ...\n```\n\n### Server Types\n\n| Type | Description | Use Case |\n|------|-------------|----------|\n| `local` | In-process server | Fast, built-in tools |\n| `http` | Remote HTTP server | Cross-machine, language-agnostic |\n| `stdio` | Child process via stdin/stdout | Process isolation |\n\n### Common Fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `namespace` | String | ✅ Yes | Unique server identifier |\n| `type` | String | ✅ Yes | Server type: `local`, `http`, or `stdio` |\n| `reset` | Boolean | ❌ No | Reset on context switch (default: `false`) |\n\n### Local Server Example\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: HostUIExecutor\n type: local\n reset: false\n```\n\n### HTTP Server Example\n\n```yaml\nHardwareAgent:\n default:\n data_collection:\n - namespace: HardwareCollector\n type: http\n host: \"localhost\"\n port: 8006\n path: \"/mcp\"\n reset: false\n```\n\n### Stdio Server Example\n\n```yaml\nCustomAgent:\n default:\n action:\n - namespace: CustomProcessor\n type: stdio\n command: \"python\"\n start_args: [\"-m\", \"custom_mcp_server\"]\n env: {\"API_KEY\": \"secret\"}\n cwd: \"/path/to/server\"\n```\n\n## Built-in Agent Configurations\n\n### HostAgent (System-Level)\n\n- **Data Collection**: UICollector\n- **Actions**: HostUIExecutor, CommandLineExecutor\n\n### AppAgent (Application-Level)\n\n**Default**: UICollector, AppUIExecutor, CommandLineExecutor\n\n**App-Specific**:\n- **WINWORD.EXE**: + WordCOMExecutor\n- **EXCEL.EXE**: + ExcelCOMExecutor\n- **POWERPNT.EXE**: + PowerPointCOMExecutor\n- **explorer.exe**: + PDFReaderExecutor\n\n### ConstellationAgent\n\n- **Actions**: ConstellationEditor\n\n### HardwareAgent\n\n- **Data Collection**: HardwareCollector (HTTP)\n- **Actions**: HardwareExecutor (HTTP)\n\n### LinuxAgent\n\n- **Actions**: BashExecutor (HTTP)\n\n## Reset Behavior\n\n!!!tip \"When to Use `reset: true`\"\n - **COM executors** (Word, Excel, PowerPoint) - Prevents state leakage between documents\n - **Stateful tools** - Requires clean state per task\n \n **Default: `false`** - Server persists across context switches\n\n## Access in Code\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\nmcp_config = config.MCP\n\n# Get agent-specific config\nhost_agent = mcp_config.get(\"HostAgent\", {})\napp_agent = mcp_config.get(\"AppAgent\", {})\n\n# Get sub-type config\nword_config = app_agent.get(\"WINWORD.EXE\", app_agent.get(\"default\", {}))\n```\n\n## Complete Documentation\n\nFor detailed configuration guide including:\n- Complete field reference for all server types\n- Agent-specific configuration examples\n- Best practices and anti-patterns\n- Configuration validation\n- Debugging and troubleshooting\n- Migration guide\n\nSee **[MCP Configuration Guide](../../mcp/configuration.md)**\n\n!!!tip \"Creating Custom MCP Servers\"\n Want to create your own MCP servers? See the **[Creating Custom MCP Servers Tutorial](../../tutorials/creating_mcp_servers.md)** for step-by-step instructions on building local, HTTP, and stdio servers.\n\n## Related Documentation\n\n- [MCP Overview](../../mcp/overview.md) - MCP architecture\n- [Data Collection Servers](../../mcp/data_collection.md) - Read-only tools\n- [Action Servers](../../mcp/action.md) - State-changing tools\n- [Local Servers](../../mcp/local_servers.md) - Built-in servers\n- [Remote Servers](../../mcp/remote_servers.md) - HTTP/Stdio deployment\n- **[Creating Custom MCP Servers Tutorial](../../tutorials/creating_mcp_servers.md)** - Build your own servers\n- [Configuration Overview](./overview.md) - General configuration system\n- [System Configuration](./system_config.md) - MCP-related system settings\n\n"
},
{
"path": "documents/docs/configuration/system/migration.md",
"content": "# Configuration Migration Guide\n\nThis guide helps you migrate from the legacy configuration system (`ufo/config/config.yaml`) to the new modular configuration system (`config/ufo/`).\n\n**Migration Overview:** Migrating to the new configuration system is **optional but recommended**. Your existing configuration will continue to work, but the new system offers better organization, type safety, and IDE support.\n\n## Why Migrate?\n\nThe new configuration system offers several advantages:\n\n| Feature | Legacy (`ufo/config/`) | New (`config/ufo/`) |\n|---------|----------------------|-------------------|\n| **Structure** | Single monolithic YAML | Modular domain-specific files |\n| **Type Safety** | Dict access only | Typed + dynamic access |\n| **IDE Support** | No autocomplete | Full IntelliSense |\n| **Scalability** | Hard to maintain | Easy to extend |\n| **Documentation** | External docs | Self-documenting structure |\n| **Environment Support** | Manual | Built-in dev/test/prod |\n\n## Migration Methods\n\n### Option 1: Automatic Migration (Recommended)\n\nUse the built-in migration tool:\n\n**Automatic Migration Tool**:\n\n```bash\n# From UFO2 root directory\npython -m ufo.tools.migrate_config\n\n# Or with options\npython -m ufo.tools.migrate_config --backup --validate\n```\n\n**What it does**:\n1. ✅ Reads your legacy `ufo/config/config.yaml`\n2. ✅ Splits into modular files by domain\n3. ✅ Creates backup of original file\n4. ✅ Validates the new configuration\n5. ✅ Provides migration report\n\n!!!warning \"Backup Reminder\"\n Always backup your configuration before migration! The tool creates a backup automatically, but it's good practice to keep your own copy.\n\n### Option 2: Manual Migration\n\nStep-by-step manual migration process.\n\n#### Step 1: Create Directory Structure\n\n```bash\n# Create new config directories\nmkdir -p config/ufo\nmkdir -p config/galaxy # If using Galaxy\n```\n\n#### Step 2: Copy Templates\n\n```bash\n# Copy template files\ncp config/ufo/agents.yaml.template config/ufo/agents.yaml\ncp config/galaxy/agent.yaml.template config/galaxy/agent.yaml # If using Galaxy\n```\n\n#### Step 3: Split Configuration\n\nSplit your `ufo/config/config.yaml` into modular files:\n\n**Legacy config.yaml**:\n```yaml\n# ufo/config/config.yaml (OLD - Monolithic)\nHOST_AGENT:\n API_TYPE: \"openai\"\n API_KEY: \"sk-...\"\n API_MODEL: \"gpt-4o\"\n\nAPP_AGENT:\n API_TYPE: \"openai\"\n API_KEY: \"sk-...\"\n API_MODEL: \"gpt-4o\"\n\nMAX_STEP: 50\nMAX_RETRY: 20\nTEMPERATURE: 0.0\n\nRAG_OFFLINE_DOCS: False\nRAG_EXPERIENCE: True\n```\n\n**New modular structure**:\n\n`config/ufo/agents.yaml`:\n```yaml\n# Agent LLM configurations\nHOST_AGENT:\n API_TYPE: \"openai\"\n API_KEY: \"sk-...\"\n API_MODEL: \"gpt-4o\"\n\nAPP_AGENT:\n API_TYPE: \"openai\"\n API_KEY: \"sk-...\"\n API_MODEL: \"gpt-4o\"\n```\n\n`config/ufo/system.yaml`:\n```yaml\n# System and runtime configurations\nMAX_STEP: 50\nMAX_RETRY: 20\nTEMPERATURE: 0.0\n```\n\n`config/ufo/rag.yaml`:\n```yaml\n# RAG knowledge configurations\nRAG_OFFLINE_DOCS: False\nRAG_EXPERIENCE: True\n```\n\n#### Step 4: Verify Configuration\n\n**Verification Script**:\n\n```python\n# Test your new configuration\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Verify values loaded correctly\nprint(f\"Max step: {config.system.max_step}\")\nprint(f\"Host agent model: {config.host_agent.api_model}\")\nprint(f\"RAG experience: {config.rag.experience}\")\n```\n\n#### Step 5: Update Code (Optional)\n\nModernize configuration access patterns:\n\n```python\n# OLD (still works but deprecated)\nconfig = Config()\nmax_step = config[\"MAX_STEP\"]\napi_model = config[\"HOST_AGENT\"][\"API_MODEL\"]\n\n# NEW (recommended)\nconfig = get_ufo_config()\nmax_step = config.system.max_step # Type-safe!\napi_model = config.host_agent.api_model # IDE autocomplete!\n```\n\n#### Step 6: Clean Up Legacy Config\n\n!!!danger \"Remove Legacy Config Only After Verification\"\n Only remove the legacy config after thoroughly testing that the new configuration works correctly!\n\n```bash\n# Backup legacy config\ncp ufo/config/config.yaml ufo/config/config.yaml.backup\n\n# Remove legacy config (after verifying new config works)\nrm ufo/config/config.yaml\n```\n\n## Field Mapping Reference\n\n### Agent Configurations\n\n| Legacy Location | New Location | Notes |\n|----------------|--------------|-------|\n| `HOST_AGENT.*` | `config/ufo/agents.yaml` → `HOST_AGENT.*` | Same structure |\n| `APP_AGENT.*` | `config/ufo/agents.yaml` → `APP_AGENT.*` | Same structure |\n| `BACKUP_AGENT.*` | `config/ufo/agents.yaml` → `BACKUP_AGENT.*` | Same structure |\n| `EVALUATION_AGENT.*` | `config/ufo/agents.yaml` → `EVALUATION_AGENT.*` | Same structure |\n| `OPERATOR.*` | `config/ufo/agents.yaml` → `OPERATOR.*` | New in UFO² |\n\n### System Configurations\n\n| Legacy Field | New Location | New Access Pattern |\n|-------------|--------------|-------------------|\n| `MAX_STEP` | `config/ufo/system.yaml` | `config.system.max_step` |\n| `MAX_RETRY` | `config/ufo/system.yaml` | `config.system.max_retry` |\n| `TEMPERATURE` | `config/ufo/system.yaml` | `config.system.temperature` |\n| `CONTROL_BACKEND` | `config/ufo/system.yaml` | `config.system.control_backend` |\n| `ACTION_SEQUENCE` | `config/ufo/system.yaml` | `config.system.action_sequence` |\n\n### RAG Configurations\n\n| Legacy Field | New Location | New Access Pattern |\n|-------------|--------------|-------------------|\n| `RAG_OFFLINE_DOCS` | `config/ufo/rag.yaml` | `config.rag.offline_docs` |\n| `RAG_EXPERIENCE` | `config/ufo/rag.yaml` | `config.rag.experience` |\n| `RAG_DEMONSTRATION` | `config/ufo/rag.yaml` | `config.rag.demonstration` |\n| `BING_API_KEY` | `config/ufo/rag.yaml` | `config.rag.BING_API_KEY` |\n\n### MCP Configurations\n\n| Legacy Field | New Location | Notes |\n|-------------|--------------|-------|\n| `USE_MCP` | `config/ufo/system.yaml` | Keep in system config |\n| `MCP_SERVERS_CONFIG` | `config/ufo/system.yaml` | Points to `config/ufo/mcp.yaml` |\n| MCP server definitions | `config/ufo/mcp.yaml` | New dedicated file |\n\n## Common Migration Scenarios\n\n### Scenario 1: Different Models for Different Agents\n\n**Legacy approach** (duplicated config):\n```yaml\n# ufo/config/config.yaml\nHOST_AGENT:\n API_MODEL: \"gpt-4o\"\n # ... other settings\n\nAPP_AGENT:\n API_MODEL: \"gpt-4o-mini\" # Different model\n # ... other settings\n```\n\n**New approach** (clear separation):\n```yaml\n# config/ufo/agents.yaml\nHOST_AGENT:\n API_MODEL: \"gpt-4o\"\n\nAPP_AGENT:\n API_MODEL: \"gpt-4o-mini\"\n```\n\n### Scenario 2: Environment-Specific Settings\n\n**Legacy approach** (manual switching):\n```yaml\n# ufo/config/config.yaml\n# Manually comment/uncomment for different environments\n# MAX_STEP: 10 # Development\nMAX_STEP: 50 # Production\n```\n\n**New approach** (automatic environment support):\n```yaml\n# config/ufo/system.yaml (base)\nMAX_STEP: 50\n\n# config/ufo/system_dev.yaml (development override)\nMAX_STEP: 10\nLOG_LEVEL: \"DEBUG\"\n```\n\n```bash\n# Set environment\nexport UFO_ENV=dev # Automatically uses system_dev.yaml overrides\n```\n\n### Scenario 3: Custom Experimental Features\n\n**Legacy approach** (modify code):\n```python\n# Had to modify Config class\nclass Config:\n def __init__(self):\n self.MY_CUSTOM_FEATURE = True # Added to code\n```\n\n**New approach** (just add to YAML):\n```yaml\n# config/ufo/custom.yaml (new file)\nMY_CUSTOM_FEATURE: True\nEXPERIMENTAL_SETTING: \"value\"\n```\n\n```python\n# Automatically available\nconfig = get_ufo_config()\nif config.MY_CUSTOM_FEATURE:\n value = config.EXPERIMENTAL_SETTING\n```\n\n## Validation After Migration\n\n### 1. Test Configuration Loading\n\n```python\nfrom config.config_loader import get_ufo_config\n\n# Load configuration\nconfig = get_ufo_config()\n\n# Verify critical settings\nassert config.system.max_step > 0\nassert config.host_agent.api_key != \"\"\nassert config.app_agent.api_model != \"\"\n\nprint(\"✅ Configuration loaded successfully!\")\n```\n\n### 2. Test Backward Compatibility\n\n```python\n# Old access patterns should still work\nconfig = get_ufo_config()\n\n# Dict-style access (legacy)\nmax_step_old = config[\"MAX_STEP\"]\nhost_agent_old = config[\"HOST_AGENT\"]\n\n# Verify they match new access\nassert max_step_old == config.system.max_step\nassert host_agent_old[\"API_MODEL\"] == config.host_agent.api_model\n\nprint(\"✅ Backward compatibility verified!\")\n```\n\n### 3. Run Application Tests\n\n```bash\n# Test with simple task\npython -m ufo --task \"Open Notepad\"\n\n# Check logs for configuration warnings\n# Should not see \"LEGACY CONFIG PATH DETECTED\" after migration\n```\n\n## Troubleshooting\n\n### Issue: \"No configuration found\"\n\n**Cause**: Configuration files not in expected locations\n\n!!!bug \"Solution\"\n Verify file locations and permissions\n\n```bash\n# Verify file locations\nls config/ufo/agents.yaml\nls config/ufo/system.yaml\n\n# Check file permissions\nchmod 644 config/ufo/*.yaml\n```\n\n### Issue: \"Configuration conflicts detected\"\n\n**Cause**: Both legacy and new configs exist\n\n!!!warning \"Conflict Resolution\"\n Choose one of these options to resolve conflicts\n\n```bash\n# Option 1: Remove legacy config (after backup)\nmv ufo/config/config.yaml ufo/config/config.yaml.backup\n\n# Option 2: Disable automatic fallback (in code)\nconfig = get_ufo_config() # Will warn but use new path\n```\n\n### Issue: \"Missing required fields\"\n\n**Cause**: Required fields not present in new configuration\n\n!!!failure \"Required Fields Missing\"\n Ensure all required agent fields are present\n\n```yaml\n# config/ufo/agents.yaml\n# Ensure all required agent fields present:\nHOST_AGENT:\n API_TYPE: \"openai\" # Required\n API_BASE: \"...\" # Required\n API_KEY: \"...\" # Required\n API_MODEL: \"...\" # Required\n```\n ```\n\n### Issue: \"Type errors in code\"\n\n**Cause**: Using old dict-style access with new typed config\n\n**Solution**:\n```python\n# OLD (can cause type issues)\nconfig[\"HOST_AGENT\"][\"API_MODEL\"]\n\n# NEW (type-safe)\nconfig.host_agent.api_model\n\n# Or keep old style for now\nconfig[\"HOST_AGENT\"][\"API_MODEL\"] # Still works!\n```\n\n## Migration Checklist\n\n- [ ] Backup legacy configuration\n- [ ] Create `config/ufo/` directory\n- [ ] Copy and customize template files\n- [ ] Split monolithic config into modular files\n- [ ] Test configuration loading\n- [ ] Verify backward compatibility\n- [ ] Update code to use new access patterns (optional)\n- [ ] Run application tests\n- [ ] Remove legacy configuration (after verification)\n- [ ] Update documentation/README\n- [ ] Commit changes to version control\n\n## Rollback Procedure\n\nIf migration causes issues:\n\n!!!danger \"Emergency Rollback\"\n Your application will immediately fall back to the legacy configuration without any code changes.\n\n```bash\n# 1. Restore legacy config from backup\ncp ufo/config/config.yaml.backup ufo/config/config.yaml\n\n# 2. Remove new config files\nrm -rf config/ufo/*.yaml\n\n# 3. Restart application\n# Old configuration will be used automatically\n```\n\n## Getting Help\n\nIf you encounter issues during migration:\n\n1. **Check the logs** for detailed error messages\n2. **Review configuration guides** ([Agents Config](./agents_config.md), [System Config](./system_config.md), [RAG Config](./rag_config.md)) for correct field names\n3. **Consult [Configuration Overview](./overview.md)** for system design\n4. **Open an issue** on GitHub with:\n - Your legacy config (redacted sensitive data)\n - Error messages\n - Steps you've tried\n\n## Next Steps\n\nAfter successful migration:\n\n- **[Agents Configuration](./agents_config.md)** - Configure LLM and agent settings\n- **[System Configuration](./system_config.md)** - Configure runtime and execution settings\n- **[RAG Configuration](./rag_config.md)** - Configure knowledge retrieval\n- **[Extending Configuration](./extending.md)** - Learn how to add custom settings\n"
},
{
"path": "documents/docs/configuration/system/overview.md",
"content": "# Configuration Architecture\n\nUFO² features a modern, modular configuration system designed for flexibility, maintainability, and backward compatibility. This guide explains the overall architecture and design principles.\n\n## Design Philosophy\n\nThe configuration system follows professional software engineering best practices:\n\n### Separation of Concerns\n\nConfiguration files are organized by domain rather than monolithic structure:\n\n- **Agent configurations** (`agents.yaml`) - LLM settings for different agents → [Agent Config Guide](./agents_config.md)\n- **System configurations** (`system.yaml`) - Execution and runtime settings → [System Config Guide](./system_config.md)\n- **RAG configurations** (`rag.yaml`) - Knowledge retrieval settings → [RAG Config Guide](./rag_config.md)\n- **MCP configurations** (`mcp.yaml`) - Model Context Protocol servers → [MCP Config Guide](./mcp_reference.md)\n- **Pricing configurations** (`prices.yaml`) - Cost tracking for different models → [Pricing Config Guide](./prices_config.md)\n- **Third-party configurations** (`third_party.yaml`) - External agent integration (LinuxAgent, HardwareAgent) → [Third-Party Config Guide](./third_party_config.md)\n\n### Type Safety + Flexibility\n\nHybrid approach combining:\n\n- **Fixed typed fields** - IDE autocomplete, type checking, and IntelliSense\n- **Dynamic YAML fields** - Add new settings without code changes\n\n**Example:**\n\n```python\n# Type-safe access (recommended)\nconfig = get_ufo_config()\nmax_step = config.system.max_step # IDE autocomplete!\napi_model = config.app_agent.api_model\n\n# Dynamic access (for custom fields)\ncustom_value = config.CUSTOM_FEATURE_FLAG\nnew_setting = config[\"NEW_YAML_KEY\"]\n\n# Backward compatible (legacy code still works)\nmax_step_old = config[\"MAX_STEP\"]\n```\n\n### Backward Compatibility\n\nZero breaking changes - existing code continues to work:\n\n- Old configuration paths still supported (`ufo/config/`)\n- Old access patterns still work (`config[\"MAX_STEP\"]`)\n- Automatic migration warnings guide users to new structure\n\nYour existing code will continue to work without any modifications. The system automatically falls back to legacy paths and access patterns. See the [Migration Guide](./migration.md) for details on upgrading to the new structure.\n\n### Auto-Discovery\n\nNo manual file registration needed:\n\n- All `*.yaml` files in `config/ufo/` are automatically loaded\n- Files are merged intelligently with deep merging\n- Environment-specific overrides (`*_dev.yaml`, `*_test.yaml`) supported\n\n## Directory Structure\n\n```\nUFO/\n├── config/ ← New Configuration Root (Recommended)\n│ ├── ufo/ ← UFO² Configurations\n│ │ ├── agents.yaml # LLM agent settings\n│ │ ├── agents.yaml.template # Template for setup\n│ │ ├── system.yaml # System and runtime settings\n│ │ ├── rag.yaml # RAG knowledge settings\n│ │ ├── mcp.yaml # MCP server configurations\n│ │ ├── prices.yaml # Model pricing\n│ │ └── third_party.yaml # Third-party agents (optional)\n│ │\n│ ├── galaxy/ ← Galaxy Configurations\n│ │ ├── agent.yaml # Constellation agent settings\n│ │ ├── agent.yaml.template # Template for setup\n│ │ ├── constellation.yaml # Constellation runtime settings\n│ │ └── devices.yaml # Device/client configurations\n│ │\n│ ├── config_loader.py # Modern config loader\n│ └── config_schemas.py # Type definitions\n│\n└── ufo/config/ ← Legacy Path (Still Supported)\n └── config.yaml # Old monolithic config\n```\n\n---\n\n## Galaxy Configuration Files\n\nThe Galaxy constellation system has its own set of configuration files in `config/galaxy/`:\n\n| File | Purpose | Template | Documentation |\n|------|---------|----------|---------------|\n| **constellation.yaml** | Constellation runtime settings (heartbeat, concurrency, step limits) | No | [Galaxy Constellation Config](./galaxy_constellation.md) |\n| **devices.yaml** | Device agent definitions (device_id, server_url, capabilities, metadata) | No | [Galaxy Devices Config](./galaxy_devices.md) |\n| **agent.yaml** | Constellation agent LLM configuration (API settings, prompts) | **Yes** (.template) | [Galaxy Agent Config](./galaxy_agent.md) |\n\n### Galaxy Configuration Structure\n\n```\nconfig/galaxy/\n├── constellation.yaml # Runtime settings for orchestrator\n│ ├── CONSTELLATION_ID # Constellation identifier\n│ ├── HEARTBEAT_INTERVAL # Health check frequency\n│ ├── RECONNECT_DELAY # Reconnection delay\n│ ├── MAX_CONCURRENT_TASKS # Task concurrency limit\n│ ├── MAX_STEP # Step limit per session\n│ ├── DEVICE_INFO # Path to devices.yaml\n│ └── LOG_TO_MARKDOWN # Markdown logging flag\n│\n├── devices.yaml # Device definitions\n│ └── devices: [] # Array of device configurations\n│ ├── device_id # Unique device identifier\n│ ├── server_url # WebSocket endpoint\n│ ├── os # Operating system\n│ ├── capabilities # Device capabilities\n│ ├── metadata # Custom metadata\n│ ├── max_retries # Connection retry limit\n│ └── auto_connect # Auto-connect flag\n│\n└── agent.yaml # Constellation agent LLM config\n └── CONSTELLATION_AGENT:\n ├── REASONING_MODEL # Enable reasoning mode\n ├── API_TYPE # API provider (openai, azure, azure_ad)\n ├── API_BASE # API base URL\n ├── API_KEY # API authentication key\n ├── API_VERSION # API version\n ├── API_MODEL # Model name/deployment\n ├── AAD_* # Azure AD auth settings\n └── *_PROMPT # Prompt template paths\n```\n\n### Galaxy Configuration Loading\n\n```python\n# Load Galaxy configurations\nfrom config.config_loader import get_galaxy_config\n\n# Load Galaxy configuration (includes agent and constellation settings)\ngalaxy_config = get_galaxy_config()\n\n# Access agent configuration (LLM settings)\nagent_config = galaxy_config.agent.constellation_agent\n\n# Access constellation runtime settings\nconstellation_settings = galaxy_config.constellation\n\n# Or use raw dict access for backward compatibility\nconstellation_id = galaxy_config[\"CONSTELLATION_ID\"]\n```\n\n**Galaxy vs UFO Configuration:**\n\n- **UFO Configurations** (`config/ufo/`) - Single-agent automation settings\n- **Galaxy Configurations** (`config/galaxy/`) - Multi-device constellation settings\n- Both systems can coexist in the same project\n\n---\n\n## Configuration Loading Process\n\n### Priority Chain\n\nThe configuration system uses a clear priority chain (highest to lowest):\n\n1. **New modular configs** - `config/{module}/*.yaml`\n2. **Legacy monolithic config** - `{module}/config/config.yaml` \n3. **Environment variables** - Runtime overrides\n\nWhen the same setting exists in multiple locations, the **new modular config** takes precedence over legacy configs. Values are merged with later sources overriding earlier ones.\n\n### Loading Algorithm\n\n```python\ndef load_config():\n # Step 1: Start with environment variables (lowest priority)\n config_data = dict(os.environ)\n \n # Step 2: Load legacy config if exists (middle priority)\n if exists(\"ufo/config/config.yaml\"):\n legacy_data = load_yaml(\"ufo/config/config.yaml\")\n merge(config_data, legacy_data)\n \n # Step 3: Load new modular configs (highest priority)\n for yaml_file in discover(\"config/ufo/*.yaml\"):\n new_data = load_yaml(yaml_file)\n merge(config_data, new_data)\n \n # Step 4: Create typed config object\n return UFOConfig.from_dict(config_data)\n```\n\n### Deep Merging\n\nConfiguration files are merged recursively, allowing you to split configurations across multiple files without duplication:\n\n```yaml\n# config/ufo/agents.yaml\nHOST_AGENT:\n API_TYPE: \"openai\"\n API_MODEL: \"gpt-4o\"\n\n# config/ufo/custom.yaml (added later)\nHOST_AGENT:\n TEMPERATURE: 0.5 # Added to HOST_AGENT\n\n# Result: HOST_AGENT has all three fields\n```\n\nFields from later files are added to (not replacing) earlier configurations.\n\n## File Organization Patterns\n\n### Split by Domain (Current Approach)\n\n```\nconfig/ufo/\n├── agents.yaml # All agent LLM configs\n├── system.yaml # All system settings\n├── rag.yaml # All RAG settings\n├── mcp.yaml # All MCP servers\n└── prices.yaml # Model pricing\n```\n\n**Advantages:** Easy to find related settings, clear separation of concerns, good for documentation.\n\n### Alternative: Split by Agent\n\n```\nconfig/ufo/\n├── host_agent.yaml # HOST_AGENT config\n├── app_agent.yaml # APP_AGENT config\n├── system.yaml # Shared system config\n└── rag.yaml # Shared RAG config\n```\n\n**Advantages:** Agent-specific settings isolated, easy to customize per agent, good for multi-agent scenarios.\n\nBoth patterns work! The loader auto-discovers and merges all YAML files.\n\n## Environment-Specific Overrides\n\nSupport for development, testing, and production environments:\n\n```bash\n# Base configuration\nconfig/ufo/agents.yaml # All environments\n\n# Environment-specific overrides\nconfig/ufo/agents_dev.yaml # Development only\nconfig/ufo/agents_test.yaml # Testing only\nconfig/ufo/agents_prod.yaml # Production only\n```\n\n**Activation**:\n```bash\n# Set environment\nexport UFO_ENV=dev # Linux/Mac\n$env:UFO_ENV = \"dev\" # Windows PowerShell\n\n# Configuration loads:\n# 1. agents.yaml (base)\n# 2. agents_dev.yaml (overrides)\n```\n\n## Type System\n\n### Fixed Types (Recommended)\n\nProvides IDE autocomplete and type safety:\n\n```python\n@dataclass\nclass SystemConfig:\n max_step: int = 50\n max_retry: int = 20\n temperature: float = 0.0\n # ...\n\n# Usage - IDE knows the types!\nconfig.system.max_step # int\nconfig.system.temperature # float\n```\n\n### Dynamic Types (Flexible)\n\nFor custom or experimental settings. Learn more about adding custom fields in the [Extending Configuration guide](./extending.md).\n\n**Example:**\n\n```python\n# In YAML\nMY_CUSTOM_FEATURE: True\nNEW_EXPERIMENTAL_SETTING: \"value\"\n\n# In code - dynamic access\nif config.MY_CUSTOM_FEATURE:\n setting = config.NEW_EXPERIMENTAL_SETTING\n```\n\n### Hybrid Approach\n\nBest of both worlds:\n\n```python\nclass SystemConfig:\n # Fixed fields\n max_step: int = 50\n \n # Dynamic extras\n _extras: Dict[str, Any]\n \n def __getattr__(self, name):\n # Try extras for unknown fields\n return self._extras.get(name)\n```\n\n## Migration Warnings\n\nThe system provides clear warnings when using legacy paths:\n\n```\n⚠️ LEGACY CONFIG PATH DETECTED: UFO\n\nUsing legacy config: ufo/config/\nPlease migrate to: config/ufo/\n\nQuick migration:\n mkdir -p config/ufo\n cp ufo/config/*.yaml config/ufo/\n\nOr use migration tool:\n python -m ufo.tools.migrate_config\n```\n\nThese warnings appear once per session and guide you to migrate to the new structure.\n\n## Best Practices\n\n**Recommended Practices:**\n\n- **Use modular files** - Split by domain or agent\n- **Use typed access** - `config.system.max_step` over `config[\"MAX_STEP\"]`\n- **Add templates** - Provide `.template` files for sensitive data\n- **Document custom fields** - Add comments in YAML\n- **Use environment overrides** - For dev/test/prod differences\n\n**Anti-Patterns to Avoid:**\n\n- **Mix old and new** - Migrate fully to new structure\n- **Put secrets in YAML** - Use environment variables instead\n- **Duplicate settings** - Leverage deep merging\n- **Break backward compat** - Keep `config[\"OLD_KEY\"]` working\n- **Hardcode paths** - Use config system\n\n## Configuration Lifecycle\n\n```mermaid\ngraph LR\n A[Application Start] --> B[Load Environment Vars]\n B --> C[Check for Legacy Config]\n C --> D[Load New Modular Configs]\n D --> E[Deep Merge All Sources]\n E --> F[Apply Transformations]\n F --> G[Create Typed Config Object]\n G --> H[Cache for Reuse]\n H --> I[Application Running]\n```\n\n## Next Steps\n\n### UFO Configuration Guides\n- **[Agent Configuration](./agents_config.md)** - LLM and API settings for all agents\n- **[System Configuration](./system_config.md)** - Runtime and execution settings\n- **[RAG Configuration](./rag_config.md)** - Knowledge retrieval and learning settings\n- **[MCP Configuration](./mcp_reference.md)** - Model Context Protocol servers\n- **[Pricing Configuration](./prices_config.md)** - LLM cost tracking\n- **[Third-Party Configuration](./third_party_config.md)** - External agent integration (LinuxAgent, HardwareAgent)\n- **[Migration Guide](./migration.md)** - How to migrate from old to new config\n- **[Extending Configuration](./extending.md)** - How to add new configuration options\n\n### Galaxy Configuration Guides\n- **[Galaxy Constellation Configuration](./galaxy_constellation.md)** - Runtime settings for constellation orchestrator\n- **[Galaxy Devices Configuration](./galaxy_devices.md)** - Device definitions and capabilities\n- **[Galaxy Agent Configuration](./galaxy_agent.md)** - LLM configuration for constellation agent\n\n## API Reference\n\nFor detailed API documentation of configuration classes and methods, see:\n\n- `config.config_loader.ConfigLoader` - Configuration loading and caching\n- `config.config_schemas.UFOConfig` - UFO configuration schema\n- `config.config_schemas.GalaxyConfig` - Galaxy configuration schema\n- `config.config_loader.get_ufo_config()` - Get UFO configuration instance\n- `config.config_loader.get_galaxy_config()` - Get Galaxy configuration instance\n"
},
{
"path": "documents/docs/configuration/system/prices_config.md",
"content": "# Pricing Configuration (prices.yaml)\n\nConfigure token pricing for different LLM models to track and estimate API costs during UFO² execution.\n\n---\n\n## Overview\n\nThe `prices.yaml` file defines the cost per 1,000 tokens for different LLM models. UFO² uses this information to calculate and report the estimated cost of task executions.\n\n**File Location**: `config/ufo/prices.yaml`\n\n!!!warning \"Pricing May Be Outdated\"\n The pricing information in this file **may not be current**. LLM providers frequently update their pricing.\n \n - Always verify current pricing on provider websites\n - This file will be updated periodically\n - Use these values as estimates only\n\n---\n\n## Quick Start\n\n### View Current Pricing\n\n```yaml\n# config/ufo/prices.yaml\ngpt-4o:\n prompt: 0.0025\n completion: 0.01\n\ngpt-4o-mini:\n prompt: 0.00015\n completion: 0.0006\n\ngpt-4-turbo:\n prompt: 0.01\n completion: 0.03\n```\n\n### Add Your Model\n\n```yaml\n# Add pricing for your custom model\nmy-custom-model:\n prompt: 0.001 # USD per 1K prompt tokens\n completion: 0.003 # USD per 1K completion tokens\n```\n\n---\n\n## Configuration Format\n\n### Structure\n\nEach model has two pricing fields:\n\n```yaml\nmodel-name:\n prompt: \n completion: \n```\n\n| Field | Type | Unit | Description |\n|-------|------|------|-------------|\n| `prompt` | Float | USD/1K tokens | Cost per 1,000 input (prompt) tokens |\n| `completion` | Float | USD/1K tokens | Cost per 1,000 output (completion) tokens |\n\n---\n\n## Common Models (As of Template)\n\n!!!info \"Verify Current Pricing\"\n These prices are from the template and **may be outdated**. Always check provider websites for current pricing:\n \n - [OpenAI Pricing](https://openai.com/pricing)\n - [Azure OpenAI Pricing](https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/)\n - [Anthropic Pricing](https://www.anthropic.com/pricing)\n - [Google AI Pricing](https://ai.google.dev/pricing)\n\n### OpenAI Models\n\n| Model | Prompt ($/1K) | Completion ($/1K) | Notes |\n|-------|---------------|-------------------|-------|\n| `gpt-4o` | $0.0025 | $0.01 | Latest GPT-4 optimized |\n| `gpt-4o-mini` | $0.00015 | $0.0006 | Cheaper alternative |\n| `gpt-4-turbo` | $0.01 | $0.03 | GPT-4 Turbo |\n| `gpt-4-vision-preview` | $0.01 | $0.03 | GPT-4 with vision |\n| `gpt-3.5-turbo` | $0.0005 | $0.0015 | GPT-3.5 |\n\n### Example Configuration\n\n```yaml\n# OpenAI Models\ngpt-4o:\n prompt: 0.0025\n completion: 0.01\n\ngpt-4o-mini:\n prompt: 0.00015\n completion: 0.0006\n\ngpt-4-turbo:\n prompt: 0.01\n completion: 0.03\n\ngpt-4-vision-preview:\n prompt: 0.01\n completion: 0.03\n\ngpt-3.5-turbo:\n prompt: 0.0005\n completion: 0.0015\n\n# Claude Models (example)\nclaude-3-5-sonnet-20241022:\n prompt: 0.003\n completion: 0.015\n\n# Gemini Models (example)\ngemini-2.0-flash-exp:\n prompt: 0.0\n completion: 0.0\n```\n\n---\n\n## Cost Tracking\n\nUFO² automatically tracks costs when pricing information is available.\n\n### During Execution\n\n```python\n# UFO² automatically calculates costs\nSession logs show:\n- Total prompt tokens used\n- Total completion tokens used\n- Estimated cost (based on prices.yaml)\n```\n\n### View Cost Summary\n\nAfter task execution, check logs:\n\n```\nlogs//cost_summary.json\n```\n\n**Example output**:\n```json\n{\n \"total_cost_usd\": 0.15,\n \"prompt_tokens\": 5000,\n \"completion_tokens\": 2000,\n \"model\": \"gpt-4o\"\n}\n```\n\n---\n\n## Updating Pricing\n\n### Step 1: Check Current Pricing\n\nVisit your LLM provider's pricing page:\n\n- **OpenAI**: https://openai.com/pricing\n- **Azure OpenAI**: https://azure.microsoft.com/pricing/details/cognitive-services/openai-service/\n- **Anthropic**: https://www.anthropic.com/pricing\n- **Google**: https://ai.google.dev/pricing\n\n### Step 2: Update prices.yaml\n\n```yaml\n# Update with current pricing\ngpt-4o:\n prompt: 0.0025 # Update if changed\n completion: 0.01\n```\n\n### Step 3: Add New Models\n\n```yaml\n# Add newly released models\ngpt-5:\n prompt: 0.005\n completion: 0.02\n```\n\n---\n\n## Programmatic Access\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Get pricing for a specific model\nmodel_name = \"gpt-4o\"\nif model_name in config.prices:\n prompt_cost = config.prices[model_name][\"prompt\"]\n completion_cost = config.prices[model_name][\"completion\"]\n print(f\"{model_name}:\")\n print(f\" Prompt: ${prompt_cost}/1K tokens\")\n print(f\" Completion: ${completion_cost}/1K tokens\")\nelse:\n print(f\"No pricing info for {model_name}\")\n```\n\n---\n\n## Cost Estimation Example\n\n```python\n# Example: Estimate cost for a task\nprompt_tokens = 10000 # 10K prompt tokens\ncompletion_tokens = 5000 # 5K completion tokens\nmodel = \"gpt-4o\"\n\n# Get pricing\nprompt_cost_per_1k = 0.0025\ncompletion_cost_per_1k = 0.01\n\n# Calculate\ntotal_cost = (\n (prompt_tokens / 1000) * prompt_cost_per_1k +\n (completion_tokens / 1000) * completion_cost_per_1k\n)\n\nprint(f\"Estimated cost: ${total_cost:.4f}\")\n# Output: Estimated cost: $0.0750\n```\n\n---\n\n## Notes\n\n!!!info \"Important Notes\"\n - ✅ Pricing is for **cost estimation only**, not billing\n - ✅ Actual costs may vary based on your provider contract\n - ✅ Different Azure regions may have different pricing\n - ✅ Some models have tiered pricing based on volume\n - ✅ Prices change frequently - update regularly\n\n---\n\n## Related Documentation\n\n- **[Agent Configuration](agents_config.md)** - LLM model selection\n- **[System Configuration](system_config.md)** - Token limits and usage\n\n---\n\n## Summary\n\n!!!success \"Key Takeaways\"\n ✅ **prices.yaml tracks LLM costs** - Estimates API spending \n ✅ **Pricing may be outdated** - Always verify current rates \n ✅ **Update regularly** - Providers change pricing frequently \n ✅ **Add new models** - Include pricing for any custom models \n ✅ **Cost tracking is automatic** - UFO² calculates costs during execution \n \n **Keep pricing updated for accurate cost tracking!** 💰\n"
},
{
"path": "documents/docs/configuration/system/rag_config.md",
"content": "# RAG Configuration (rag.yaml)\n\nConfigure Retrieval-Augmented Generation (RAG) to enhance UFO² with external knowledge sources, online search, experience learning, and demonstration-based learning.\n\n---\n\n## Overview\n\nThe `rag.yaml` file configures knowledge retrieval systems that augment UFO²'s capabilities beyond its base LLM knowledge. RAG helps UFO² make better decisions by providing:\n\n- **Offline Documentation**: Application manuals and documentation\n- **Online Search**: Real-time web search via Bing\n- **Experience Learning**: Learn from past successful executions\n- **Demonstration Learning**: Learn from user demonstrations\n\n**File Location**: `config/ufo/rag.yaml`\n\n**Optional Configuration:** RAG features are **optional**. UFO² works without them, but they can significantly improve performance on complex or domain-specific tasks.\n\n---\n\n## Quick Start\n\n### Disable All RAG (Default)\n\n```yaml\n# Minimal configuration - no external knowledge\nRAG_OFFLINE_DOCS: False\nRAG_ONLINE_SEARCH: False\nRAG_EXPERIENCE: False\nRAG_DEMONSTRATION: False\n```\n\n### Enable Online Search Only\n\n```yaml\n# Most useful for general tasks\nRAG_OFFLINE_DOCS: False\n\nRAG_ONLINE_SEARCH: True\nBING_API_KEY: \"YOUR_BING_API_KEY_HERE\"\nRAG_ONLINE_SEARCH_TOPK: 5\nRAG_ONLINE_RETRIEVED_TOPK: 5\n\nRAG_EXPERIENCE: False\nRAG_DEMONSTRATION: False\n```\n\n### Enable Experience Learning\n\n```yaml\n# Learn from past executions\nRAG_OFFLINE_DOCS: False\nRAG_ONLINE_SEARCH: False\n\nRAG_EXPERIENCE: True\nRAG_EXPERIENCE_RETRIEVED_TOPK: 5\n\nRAG_DEMONSTRATION: False\n```\n\n### Enable All Features\n\n```yaml\n# Maximum knowledge augmentation\nRAG_OFFLINE_DOCS: True\nRAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1\n\nRAG_ONLINE_SEARCH: True\nBING_API_KEY: \"YOUR_BING_API_KEY_HERE\"\nRAG_ONLINE_SEARCH_TOPK: 5\nRAG_ONLINE_RETRIEVED_TOPK: 5\n\nRAG_EXPERIENCE: True\nRAG_EXPERIENCE_RETRIEVED_TOPK: 5\n\nRAG_DEMONSTRATION: True\nRAG_DEMONSTRATION_RETRIEVED_TOPK: 5\n```\n\n---\n\n## RAG Components\n\n### 1. Offline Documentation\n\nRetrieve relevant documentation from local knowledge bases (app manuals, guides, API docs).\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `RAG_OFFLINE_DOCS` | Boolean | `False` | Enable offline documentation retrieval |\n| `RAG_OFFLINE_DOCS_RETRIEVED_TOPK` | Integer | `1` | Number of documents to retrieve |\n\n**Example**:\n```yaml\nRAG_OFFLINE_DOCS: True\nRAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1\n```\n\n!!!info \"Use Case\"\n - Application-specific tasks (Excel formulas, Word formatting)\n - Domain-specific workflows (accounting, design)\n - Requires pre-indexed documentation\n\n**Setup**:\n1. Place documentation in `vectordb/docs/`\n2. Index documents: `python -m learner`\n3. Enable in `rag.yaml`\n\n---\n\n### 2. Online Search\n\nSearch the web in real-time using Bing Search API.\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `RAG_ONLINE_SEARCH` | Boolean | `False` | Enable online Bing search |\n| `BING_API_KEY` | String | `\"\"` | Bing Search API key |\n| `RAG_ONLINE_SEARCH_TOPK` | Integer | `5` | Number of search results to fetch |\n| `RAG_ONLINE_RETRIEVED_TOPK` | Integer | `5` | Number of results to include in prompt |\n\n**Example**:\n```yaml\nRAG_ONLINE_SEARCH: True\nBING_API_KEY: \"abc123xyz...\"\nRAG_ONLINE_SEARCH_TOPK: 5\nRAG_ONLINE_RETRIEVED_TOPK: 5\n```\n\n!!!tip \"Getting Bing API Key\"\n 1. Go to [Azure Portal](https://portal.azure.com)\n 2. Create a \"Bing Search v7\" resource\n 3. Copy the API key from \"Keys and Endpoint\"\n 4. Add to `rag.yaml`: `BING_API_KEY: \"your-key\"`\n\n**Use Cases**:\n- Tasks requiring current information\n- Unfamiliar applications or features\n- Troubleshooting specific error messages\n- Finding how-to guides dynamically\n\n**Example Query Flow**:\n```\nUser Request: \"Create a pivot table in Excel\"\n↓\nBing Search: \"how to create pivot table in Excel\"\n↓\nRetrieved: Top 5 results about pivot tables\n↓\nLLM receives context from search results\n↓\nBetter action decisions\n```\n\n---\n\n### 3. Experience Learning\n\nLearn from UFO²'s own past successful task executions.\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `RAG_EXPERIENCE` | Boolean | `False` | Enable experience learning |\n| `RAG_EXPERIENCE_RETRIEVED_TOPK` | Integer | `5` | Number of past experiences to retrieve |\n| `EXPERIENCE_SAVED_PATH` | String | Auto-generated | Path to experience database |\n| `EXPERIENCE_PROMPT` | String | Auto-generated | Experience prompt template |\n\n**Example**:\n```yaml\nRAG_EXPERIENCE: True\nRAG_EXPERIENCE_RETRIEVED_TOPK: 5\n```\n\n!!!info \"How It Works\"\n 1. UFO² completes a task successfully\n 2. Task steps are saved to experience database\n 3. For future similar tasks, relevant past experiences are retrieved\n 4. LLM learns from successful patterns\n\n**Use Cases**:\n- Repetitive tasks with slight variations\n- Learning organizational-specific workflows\n- Improving over time on common tasks\n\n**Example**:\n```\nFirst Time: \"Create a monthly sales report\"\n→ Task succeeds, 15 steps recorded\n\nSecond Time: \"Create a quarterly sales report\"\n→ Retrieves \"monthly report\" experience\n→ Adapts the pattern, faster execution\n```\n\n**Default Paths**:\n```yaml\n# Auto-generated if not specified\nEXPERIENCE_SAVED_PATH: \"vectordb/experience\"\nEXPERIENCE_PROMPT: \"ufo/prompts/share/experience/experience.yaml\"\n```\n\n---\n\n### 4. Demonstration Learning\n\nLearn from user demonstrations (you show UFO² how to do a task).\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `RAG_DEMONSTRATION` | Boolean | `False` | Enable demonstration learning |\n| `RAG_DEMONSTRATION_RETRIEVED_TOPK` | Integer | `5` | Number of demonstrations to retrieve |\n| `DEMONSTRATION_SAVED_PATH` | String | Auto-generated | Path to demonstration database |\n| `DEMONSTRATION_PROMPT` | String | Auto-generated | Demonstration prompt template |\n\n**Example**:\n```yaml\nRAG_DEMONSTRATION: True\nRAG_DEMONSTRATION_RETRIEVED_TOPK: 5\n```\n\n!!!info \"How It Works\"\n 1. User demonstrates a task (UFO² records it)\n 2. Demonstration is saved with annotations\n 3. For similar future tasks, demonstrations are retrieved\n 4. LLM mimics the demonstrated behavior\n\n**Use Cases**:\n- Complex, domain-specific workflows\n- Organizational-specific procedures\n- Tasks with many edge cases\n\n**Workflow**:\n```\n1. Record Demonstration:\n python -m ufo --mode demonstration\n → Perform task manually\n → UFO² records your actions\n\n2. Save Demonstration:\n → Stored in vectordb/demonstration/\n\n3. Future Task:\n \"Do the same report formatting\"\n → Retrieves your demonstration\n → Replicates your steps\n```\n\n**Default Paths**:\n```yaml\n# Auto-generated if not specified\nDEMONSTRATION_SAVED_PATH: \"vectordb/demonstration\"\nDEMONSTRATION_PROMPT: \"ufo/prompts/share/demonstration/demonstration.yaml\"\n```\n\n---\n\n## Complete Configuration Examples\n\n### Minimal (No RAG)\n\n```yaml\n# config/ufo/rag.yaml\nRAG_OFFLINE_DOCS: False\nRAG_ONLINE_SEARCH: False\nRAG_EXPERIENCE: False\nRAG_DEMONSTRATION: False\n```\n\n### Online Search Only\n\n```yaml\nRAG_OFFLINE_DOCS: False\n\nRAG_ONLINE_SEARCH: True\nBING_API_KEY: \"your-bing-api-key-here\"\nRAG_ONLINE_SEARCH_TOPK: 5\nRAG_ONLINE_RETRIEVED_TOPK: 5\n\nRAG_EXPERIENCE: False\nRAG_DEMONSTRATION: False\n```\n\n### Experience Learning Only\n\n```yaml\nRAG_OFFLINE_DOCS: False\nRAG_ONLINE_SEARCH: False\n\nRAG_EXPERIENCE: True\nRAG_EXPERIENCE_RETRIEVED_TOPK: 5\nEXPERIENCE_SAVED_PATH: \"vectordb/experience\"\nEXPERIENCE_PROMPT: \"ufo/prompts/share/experience/experience.yaml\"\n\nRAG_DEMONSTRATION: False\n```\n\n### Full RAG Setup\n\n```yaml\n# Offline docs\nRAG_OFFLINE_DOCS: True\nRAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1\n\n# Online search\nRAG_ONLINE_SEARCH: True\nBING_API_KEY: \"your-bing-api-key\"\nRAG_ONLINE_SEARCH_TOPK: 5\nRAG_ONLINE_RETRIEVED_TOPK: 5\n\n# Experience\nRAG_EXPERIENCE: True\nRAG_EXPERIENCE_RETRIEVED_TOPK: 5\nEXPERIENCE_SAVED_PATH: \"vectordb/experience\"\nEXPERIENCE_PROMPT: \"ufo/prompts/share/experience/experience.yaml\"\n\n# Demonstration\nRAG_DEMONSTRATION: True\nRAG_DEMONSTRATION_RETRIEVED_TOPK: 5\nDEMONSTRATION_SAVED_PATH: \"vectordb/demonstration\"\nDEMONSTRATION_PROMPT: \"ufo/prompts/share/demonstration/demonstration.yaml\"\n```\n\n---\n\n## Setting Up Each RAG Component\n\n### Setup: Offline Documentation\n\n**Step 1**: Prepare documentation\n```powershell\n# Place docs in vectordb/docs/\nNew-Item -ItemType Directory -Path \"vectordb\\docs\\excel\" -Force\nCopy-Item \"C:\\path\\to\\excel_guide.pdf\" \"vectordb\\docs\\excel\\\"\n```\n\n**Step 2**: Index documents\n```powershell\npython -m learner --index-docs\n```\n\n**Step 3**: Enable in config\n```yaml\nRAG_OFFLINE_DOCS: True\nRAG_OFFLINE_DOCS_RETRIEVED_TOPK: 1\n```\n\n---\n\n### Setup: Online Search\n\n**Step 1**: Get Bing API key\n\n1. Go to [Azure Portal](https://portal.azure.com)\n2. Create resource → Search for \"Bing Search v7\"\n3. Create the resource\n4. Go to \"Keys and Endpoint\"\n5. Copy Key 1\n\n**Step 2**: Add to config\n```yaml\nRAG_ONLINE_SEARCH: True\nBING_API_KEY: \"your-copied-key-here\"\nRAG_ONLINE_SEARCH_TOPK: 5\nRAG_ONLINE_RETRIEVED_TOPK: 5\n```\n\n**Step 3**: Test\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\nprint(f\"Bing search enabled: {config.rag.online_search}\")\nprint(f\"API key set: {bool(config.rag.BING_API_KEY)}\")\n```\n\n---\n\n### Setup: Experience Learning\n\n**Step 1**: Enable in config\n```yaml\nRAG_EXPERIENCE: True\nRAG_EXPERIENCE_RETRIEVED_TOPK: 5\n```\n\n**Step 2**: Run tasks normally\n```powershell\npython -m ufo --request \"Create a sales report\"\n```\n\n**Step 3**: Successful tasks are auto-saved\n```\nExperience saved to: vectordb/experience/\n```\n\n**Step 4**: Future tasks retrieve experiences\n```powershell\n# Similar task will use past experience\npython -m ufo --request \"Create a quarterly report\"\n```\n\n---\n\n### Setup: Demonstration Learning\n\n**Step 1**: Record demonstration\n```powershell\npython -m ufo --mode demonstration --task \"format_monthly_report\"\n```\n\n**Step 2**: Perform task manually\n- UFO² records your every action\n- Add annotations/comments\n\n**Step 3**: Save demonstration\n```\nDemonstration saved to: vectordb/demonstration/\n```\n\n**Step 4**: Enable in config\n```yaml\nRAG_DEMONSTRATION: True\nRAG_DEMONSTRATION_RETRIEVED_TOPK: 5\n```\n\n**Step 5**: Use demonstrations\n```powershell\npython -m ufo --request \"Format the report like I showed you\"\n```\n\n---\n\n## Programmatic Access\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Check RAG settings\nif config.rag.online_search:\n print(f\"Online search enabled\")\n print(f\"Top K: {config.rag.online_search_topk}\")\n \nif config.rag.experience:\n print(f\"Experience learning enabled\")\n print(f\"Experience path: {config.rag.EXPERIENCE_SAVED_PATH}\")\n\nif config.rag.offline_docs:\n print(f\"Offline docs enabled\")\n\n# Access specific fields\nbing_key = config.rag.BING_API_KEY\nexp_topk = config.rag.experience_retrieved_topk\n```\n\n---\n\n## Performance Considerations\n\n### Impact on Speed\n\n| RAG Type | Speed Impact | When to Use |\n|----------|--------------|-------------|\n| **Offline Docs** | Low | Always (if indexed) |\n| **Online Search** | Medium | For unfamiliar tasks |\n| **Experience** | Low | Always (improves over time) |\n| **Demonstration** | Low | For specific workflows |\n\n### Impact on Cost\n\n| RAG Type | Cost Impact | Notes |\n|----------|-------------|-------|\n| **Offline Docs** | None | One-time indexing cost |\n| **Online Search** | Low | Bing API: ~$3/1000 queries |\n| **Experience** | None | Free storage |\n| **Demonstration** | None | Free storage |\n\n!!!tip \"Recommended Configuration\"\n For most users:\n ```yaml\n RAG_ONLINE_SEARCH: True # Useful for general tasks\n RAG_EXPERIENCE: True # Improves over time\n RAG_OFFLINE_DOCS: False # Unless you have specific docs\n RAG_DEMONSTRATION: False # Unless training specific workflows\n ```\n\n---\n\n## Troubleshooting\n\n### Issue 1: Bing Search Not Working\n\n!!!bug \"Error Message\"\n ```\n BingSearchError: Invalid API key\n ```\n \n **Solutions**:\n 1. Verify API key is correct\n 2. Check key has not expired\n 3. Ensure Bing Search v7 resource is active\n 4. Check Azure subscription is active\n\n---\n\n### Issue 2: Experience Not Retrieved\n\n!!!bug \"Symptom\"\n UFO² doesn't seem to learn from past tasks\n \n **Solutions**:\n 1. Check experience database exists:\n ```powershell\n Test-Path \"vectordb\\experience\"\n ```\n 2. Verify tasks completed successfully\n 3. Check similarity threshold (may be too strict)\n 4. Increase `RAG_EXPERIENCE_RETRIEVED_TOPK`\n\n---\n\n### Issue 3: Offline Docs Not Indexed\n\n!!!bug \"Error Message\"\n ```\n No offline documents found\n ```\n \n **Solutions**:\n 1. Run indexing:\n ```powershell\n python -m learner --index-docs\n ```\n 2. Check documents are in `vectordb/docs/`\n 3. Verify supported formats (PDF, TXT, MD)\n\n---\n\n### Issue 4: Too Much Context\n\n!!!bug \"Symptom\"\n Token limits exceeded, slow responses\n \n **Solution**: Reduce Top-K values\n ```yaml\n RAG_ONLINE_RETRIEVED_TOPK: 3 # Instead of 5\n RAG_EXPERIENCE_RETRIEVED_TOPK: 3\n RAG_DEMONSTRATION_RETRIEVED_TOPK: 3\n ```\n\n---\n\n## Best Practices\n\n### When to Enable Each Component\n\n| Scenario | Recommended RAG |\n|----------|----------------|\n| **General automation** | Online Search |\n| **Repetitive tasks** | Experience Learning |\n| **Domain-specific workflows** | Offline Docs + Demonstration |\n| **Learning over time** | Experience |\n| **New to UFO²** | Online Search only |\n| **Production deployment** | Experience + Offline Docs |\n\n### Top-K Selection\n\n| Field | Recommended Range | Notes |\n|-------|-------------------|-------|\n| `RAG_ONLINE_SEARCH_TOPK` | 3-10 | More = better context, slower |\n| `RAG_ONLINE_RETRIEVED_TOPK` | 3-5 | Balance quality vs tokens |\n| `RAG_EXPERIENCE_RETRIEVED_TOPK` | 3-5 | Most relevant experiences |\n| `RAG_DEMONSTRATION_RETRIEVED_TOPK` | 1-3 | Usually need few examples |\n| `RAG_OFFLINE_DOCS_RETRIEVED_TOPK` | 1-2 | Docs are usually long |\n\n---\n\n## Environment Variables\n\nStore API keys securely:\n\n```yaml\n# Use environment variable instead of hardcoded key\nBING_API_KEY: \"${BING_API_KEY}\"\n```\n\n**Set environment variable**:\n\n**Windows PowerShell:**\n```powershell\n$env:BING_API_KEY = \"your-key-here\"\n```\n\n**Windows (Persistent):**\n```powershell\n[System.Environment]::SetEnvironmentVariable('BING_API_KEY', 'your-key', 'User')\n```\n\n---\n\n## Related Documentation\n\n- **[Agent Configuration](agents_config.md)** - LLM settings\n- **[System Configuration](system_config.md)** - Runtime settings\n\n---\n\n## Summary\n\n!!!success \"Key Takeaways\"\n ✅ **RAG is optional** - UFO² works without it \n ✅ **Online Search** - Most useful for general tasks (needs Bing API key) \n ✅ **Experience** - Free, improves over time automatically \n ✅ **Offline Docs** - Great for domain-specific knowledge \n ✅ **Demonstration** - Best for complex, specific workflows \n ✅ **Start simple** - Enable Online Search first, add others as needed \n \n **Enhance UFO² with knowledge retrieval!** 🧠\n"
},
{
"path": "documents/docs/configuration/system/system_config.md",
"content": "# System Configuration (system.yaml)\n\nConfigure UFO²'s runtime behavior, execution limits, control backends, logging, and operational parameters. This file controls how UFO² interacts with the Windows environment.\n\n## Overview\n\nThe `system.yaml` file defines runtime settings that control UFO²'s behavior during task execution. Unlike `agents.yaml` (which configures LLMs), this file configures **how** UFO² operates on Windows.\n\n**File Location**: `config/ufo/system.yaml`\n\n**Note:** Unlike `agents.yaml`, the `system.yaml` file is **already present** in the repository with sensible defaults. You can use it as-is or customize it for your needs.\n\n## Quick Configuration\n\n### Default Configuration (Works Out of Box)\n\n```yaml\n# Most users can use default settings\nMAX_STEP: 50\nMAX_ROUND: 1\nCONTROL_BACKEND: [\"uia\"]\nUSE_MCP: True\nPRINT_LOG: False\n```\n\n### Recommended for Development\n\n```yaml\n# More verbose logging for debugging\nMAX_STEP: 50\nMAX_ROUND: 1\nPRINT_LOG: True\nLOG_LEVEL: \"DEBUG\"\nCONTROL_BACKEND: [\"uia\"]\n```\n\n### Recommended for Production\n\n```yaml\n# Optimized for reliability\nMAX_STEP: 100\nMAX_ROUND: 3\nCONTROL_BACKEND: [\"uia\"]\nUSE_MCP: True\nSAFE_GUARD: True\nLOG_TO_MARKDOWN: True\n```\n\n## Configuration Categories\n\nThe `system.yaml` file is organized into logical sections:\n\n| Category | Purpose | Key Fields |\n|----------|---------|------------|\n| **[LLM Parameters](#llm-parameters)** | API call settings | `MAX_TOKENS`, `TEMPERATURE`, `TIMEOUT` |\n| **[Execution Limits](#execution-limits)** | Task boundaries | `MAX_STEP`, `MAX_ROUND`, `SLEEP_TIME` |\n| **[Control Backend](#control-backend)** | UI detection methods | `CONTROL_BACKEND`, `IOU_THRESHOLD` |\n| **[Action Configuration](#action-configuration)** | Interaction behavior | `CLICK_API`, `INPUT_TEXT_API`, `MAXIMIZE_WINDOW` |\n| **[Logging](#logging)** | Output and debugging | `PRINT_LOG`, `LOG_LEVEL`, `LOG_XML` |\n| **[MCP Settings](#mcp-settings)** | Tool server integration | `USE_MCP`, `MCP_SERVERS_CONFIG` |\n| **[Safety](#safety)** | Security controls | `SAFE_GUARD`, `CONTROL_LIST` |\n| **[Control Filtering](#control-filtering)** | UI element filtering | `CONTROL_FILTER_TYPE`, `CONTROL_FILTER_TOP_K` |\n\n## LLM Parameters\n\nThese settings control how UFO² communicates with LLM APIs.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `MAX_TOKENS` | Integer | `2000` | Maximum tokens for LLM response |\n| `MAX_RETRY` | Integer | `20` | Maximum retries for failed API calls |\n| `TEMPERATURE` | Float | `0.0` | Sampling temperature (0.0 = deterministic, 1.0 = creative) |\n| `TOP_P` | Float | `0.0` | Nucleus sampling threshold |\n| `TIMEOUT` | Integer | `60` | API call timeout (seconds) |\n\n### Example\n\n```yaml\n# Conservative settings (recommended)\nMAX_TOKENS: 2000\nMAX_RETRY: 20\nTEMPERATURE: 0.0 # Deterministic\nTOP_P: 0.0\nTIMEOUT: 60\n\n# Creative settings (experimental)\n# MAX_TOKENS: 4000\n# TEMPERATURE: 0.7 # More creative\n# TOP_P: 0.9\n```\n\n**When to Adjust:**\n\n- **Increase MAX_TOKENS** if responses are getting cut off\n- **Increase TEMPERATURE** if you want more varied responses (not recommended)\n- **Keep at 0.0** for consistent, repeatable automation\n- **Increase TIMEOUT** for slow API connections\n\n## Execution Limits\n\nControl how long and how many attempts UFO² makes for tasks.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `MAX_STEP` | Integer | `50` | Maximum steps per task |\n| `MAX_ROUND` | Integer | `1` | Maximum rounds per task (retries from start) |\n| `SLEEP_TIME` | Integer | `1` | Wait time between steps (seconds) |\n| `RECTANGLE_TIME` | Integer | `1` | Duration to show visual highlights (seconds) |\n\n### Example\n\n```yaml\n# Default settings\nMAX_STEP: 50\nMAX_ROUND: 1\nSLEEP_TIME: 1\nRECTANGLE_TIME: 1\n\n# For complex tasks\n# MAX_STEP: 100\n# MAX_ROUND: 3\n\n# For faster execution (risky)\n# SLEEP_TIME: 0\n```\n\n**Note on Step vs Round:**\n\n- **STEP**: Individual action (click, type, etc.)\n- **ROUND**: Complete task attempt from start\n\nExample: If `MAX_ROUND: 3`, UFO² will retry the entire task up to 3 times if it fails.\n\n## Control Backend\n\nConfigure how UFO² detects and interacts with UI elements.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `CONTROL_BACKEND` | List[String] | `[\"uia\"]` | UI detection backends to use |\n| `IOU_THRESHOLD_FOR_MERGE` | Float | `0.1` | IoU threshold for merging overlapping controls |\n\n### Available Backends\n\n| Backend | Description | Pros | Cons |\n|---------|-------------|------|------|\n| `\"uia\"` | UI Automation | Fast, reliable, Windows native | May miss some controls |\n| `\"omniparser\"` | Vision-based | Finds visual-only elements | Requires GPU, slow |\n\n**Note:** `win32` backend is no longer supported.\n\n### Example\n\n```yaml\n# Recommended: Use UIA (default)\nCONTROL_BACKEND: [\"uia\"]\nIOU_THRESHOLD_FOR_MERGE: 0.1\n\n# With vision-based parsing (slow)\n# CONTROL_BACKEND: [\"uia\", \"omniparser\"]\n```\n\n**Best Practice:** Use `[\"uia\"]` as the default backend. Add `\"omniparser\"` only if you need vision-based control detection.\n\n## Action Configuration\n\nConfigure how UFO² performs actions on UI elements.\n\n### Core Action Settings\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `ACTION_SEQUENCE` | Boolean | `False` | Enable multi-action sequences in one step |\n| `SHOW_VISUAL_OUTLINE_ON_SCREEN` | Boolean | `False` | Show visual highlights during execution |\n| `MAXIMIZE_WINDOW` | Boolean | `False` | Maximize application windows before actions |\n| `JSON_PARSING_RETRY` | Integer | `3` | Retries for parsing LLM JSON responses |\n\n### Click Settings\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `CLICK_API` | String | `\"click_input\"` | Click method to use |\n| `AFTER_CLICK_WAIT` | Integer | `0` | Wait time after clicking (seconds) |\n\n### Input Settings\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `INPUT_TEXT_API` | String | `\"type_keys\"` | Text input method |\n| `INPUT_TEXT_ENTER` | Boolean | `False` | Press Enter after typing |\n| `INPUT_TEXT_INTER_KEY_PAUSE` | Float | `0.05` | Pause between keystrokes (seconds) |\n\n### Example\n\n```yaml\n# Recommended settings\nACTION_SEQUENCE: True # Enable multi-action for speed\nSHOW_VISUAL_OUTLINE_ON_SCREEN: False\nMAXIMIZE_WINDOW: False\nJSON_PARSING_RETRY: 3\n\nCLICK_API: \"click_input\"\nAFTER_CLICK_WAIT: 0\n\nINPUT_TEXT_API: \"type_keys\"\nINPUT_TEXT_ENTER: False\nINPUT_TEXT_INTER_KEY_PAUSE: 0.05\n\n# For visual debugging\n# SHOW_VISUAL_OUTLINE_ON_SCREEN: True\n\n# If clicks are too fast\n# AFTER_CLICK_WAIT: 1\n\n# For automation that needs Enter key\n# INPUT_TEXT_ENTER: True\n```\n\n!!!info \"Input Methods\"\n - **`type_keys`**: Simulates keyboard (slower, more realistic)\n - **`set_text`**: Direct text insertion (faster, may not trigger events)\n\n---\n\n## Logging\n\nControl UFO²'s logging output and debugging information.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `PRINT_LOG` | Boolean | `False` | Print logs to console |\n| `LOG_LEVEL` | String | `\"DEBUG\"` | Logging verbosity level |\n| `LOG_TO_MARKDOWN` | Boolean | `True` | Save logs as Markdown files |\n| `LOG_XML` | Boolean | `False` | Log UI tree XML at each step |\n| `CONCAT_SCREENSHOT` | Boolean | `False` | Concatenate control screenshots |\n| `INCLUDE_LAST_SCREENSHOT` | Boolean | `True` | Include previous screenshot in context |\n| `SCREENSHOT_TO_MEMORY` | Boolean | `True` | Load screenshots into memory |\n| `REQUEST_TIMEOUT` | Integer | `250` | Request timeout for vision models |\n\n### Log Levels\n\n| Level | Usage | When to Use |\n|-------|-------|-------------|\n| `\"DEBUG\"` | Detailed debugging info | Development, troubleshooting |\n| `\"INFO\"` | General information | Normal operation |\n| `\"WARNING\"` | Warning messages | Production |\n| `\"ERROR\"` | Errors only | Production (minimal logs) |\n\n### Example\n\n```yaml\n# Development settings\nPRINT_LOG: True\nLOG_LEVEL: \"DEBUG\"\nLOG_TO_MARKDOWN: True\nLOG_XML: True # Useful for debugging UI detection\n\n# Production settings\n# PRINT_LOG: False\n# LOG_LEVEL: \"WARNING\"\n# LOG_TO_MARKDOWN: True\n# LOG_XML: False\n\n# Memory optimization\n# SCREENSHOT_TO_MEMORY: False\n```\n\n!!!tip \"Log Files Location\"\n Logs are saved to `logs//` directory.\n\n---\n\n## MCP Settings\n\nConfigure Model Context Protocol (MCP) tool servers.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `USE_MCP` | Boolean | `True` | Enable MCP tool integration |\n| `MCP_SERVERS_CONFIG` | String | `\"config/ufo/mcp.yaml\"` | Path to MCP servers config |\n| `MCP_PREFERRED_APPS` | List[String] | `[]` | Apps that prefer MCP over UI automation |\n| `MCP_FALLBACK_TO_UI` | Boolean | `True` | Fall back to UI if MCP fails |\n| `MCP_INSTRUCTIONS_PATH` | String | `\"ufo/config/mcp_instructions\"` | MCP instruction templates path |\n| `MCP_TOOL_TIMEOUT` | Integer | `30` | MCP tool execution timeout (seconds) |\n| `MCP_LOG_EXECUTION` | Boolean | `False` | Log detailed MCP execution |\n\n### Example\n\n```yaml\n# Recommended settings\nUSE_MCP: True\nMCP_SERVERS_CONFIG: \"config/ufo/mcp.yaml\"\nMCP_FALLBACK_TO_UI: True\nMCP_TOOL_TIMEOUT: 30\nMCP_LOG_EXECUTION: False\n\n# Prefer MCP for VS Code and Terminal\nMCP_PREFERRED_APPS:\n - \"Code.exe\"\n - \"WindowsTerminal.exe\"\n\n# Debugging MCP issues\n# MCP_LOG_EXECUTION: True\n# MCP_TOOL_TIMEOUT: 60\n```\n\n!!!info \"What is MCP?\"\n MCP (Model Context Protocol) provides programmatic APIs for applications, offering more reliable automation than UI-based control.\n \n See [MCP Configuration](mcp_reference.md) for details.\n\n---\n\n## Safety\n\nSecurity and safety controls to prevent dangerous operations.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `SAFE_GUARD` | Boolean | `False` | Enable safety checks |\n| `CONTROL_LIST` | List[String] | See below | Allowed UI control types |\n\n### Default CONTROL_LIST\n\n```yaml\nCONTROL_LIST:\n - \"Button\"\n - \"Edit\"\n - \"TabItem\"\n - \"Document\"\n - \"ListItem\"\n - \"MenuItem\"\n - \"ScrollBar\"\n - \"TreeItem\"\n - \"Hyperlink\"\n - \"ComboBox\"\n - \"RadioButton\"\n - \"Spinner\"\n - \"CheckBox\"\n - \"Group\"\n - \"Text\"\n```\n\n### Example\n\n```yaml\n# Enable safety for production\nSAFE_GUARD: True\nCONTROL_LIST:\n - \"Button\"\n - \"Edit\"\n - \"TabItem\"\n # Add only safe control types\n\n# Disable for full automation (risky)\n# SAFE_GUARD: False\n```\n\n!!!danger \"Safety Warning\"\n When `SAFE_GUARD: True`, UFO² will only interact with control types in `CONTROL_LIST`. This prevents accidental dangerous operations but may limit functionality.\n\n---\n\n## Control Filtering\n\nAdvanced UI element filtering using semantic and icon similarity.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `CONTROL_FILTER_TYPE` | List[String] | `[]` | Filter types to enable |\n| `CONTROL_FILTER_TOP_K_PLAN` | Integer | `2` | Top K plans to consider |\n| `CONTROL_FILTER_TOP_K_SEMANTIC` | Integer | `15` | Top K controls by text similarity |\n| `CONTROL_FILTER_TOP_K_ICON` | Integer | `15` | Top K controls by icon similarity |\n| `CONTROL_FILTER_MODEL_SEMANTIC_NAME` | String | `\"all-MiniLM-L6-v2\"` | Semantic embedding model |\n| `CONTROL_FILTER_MODEL_ICON_NAME` | String | `\"clip-ViT-B-32\"` | Icon embedding model |\n\n### Filter Types\n\n| Type | Description | Use Case |\n|------|-------------|----------|\n| `\"TEXT\"` | Text-based filtering | Filter by control labels |\n| `\"SEMANTIC\"` | Semantic similarity | Find similar controls by meaning |\n| `\"ICON\"` | Icon similarity | Find controls by icon appearance |\n\n### Example\n\n```yaml\n# Disable filtering (use all controls)\nCONTROL_FILTER_TYPE: []\n\n# Enable semantic filtering (recommended)\nCONTROL_FILTER_TYPE: [\"SEMANTIC\"]\nCONTROL_FILTER_TOP_K_SEMANTIC: 15\nCONTROL_FILTER_MODEL_SEMANTIC_NAME: \"all-MiniLM-L6-v2\"\n\n# Enable all filtering (most selective)\n# CONTROL_FILTER_TYPE: [\"TEXT\", \"SEMANTIC\", \"ICON\"]\n# CONTROL_FILTER_TOP_K_SEMANTIC: 20\n# CONTROL_FILTER_TOP_K_ICON: 20\n```\n\n!!!warning \"Performance Impact\"\n - Filtering reduces the number of controls sent to LLM (faster, cheaper)\n - But may filter out the target control (less reliable)\n - Start without filtering, add if you have too many controls\n\n---\n\n## API Usage Configuration\n\nConfigure native API usage for Office applications.\n\n### Fields\n\n| Field | Type | Default | Description |\n|-------|------|---------|-------------|\n| `USE_APIS` | Boolean | `True` | Enable COM API usage for Office applications |\n| `API_PROMPT` | String | `\"ufo/prompts/share/base/api.yaml\"` | API prompt template |\n| `APP_API_PROMPT_ADDRESS` | Dict | See below | App-specific API prompts |\n\n### Default APP_API_PROMPT_ADDRESS\n\n```yaml\nAPP_API_PROMPT_ADDRESS:\n \"WINWORD.EXE\": \"ufo/prompts/apps/word/api.yaml\"\n \"EXCEL.EXE\": \"ufo/prompts/apps/excel/api.yaml\"\n \"msedge.exe\": \"ufo/prompts/apps/web/api.yaml\"\n \"chrome.exe\": \"ufo/prompts/apps/web/api.yaml\"\n \"POWERPNT.EXE\": \"ufo/prompts/apps/powerpoint/api.yaml\"\n```\n\n### Example\n\n```yaml\n# Enable API usage (recommended for Office)\nUSE_APIS: True\nAPI_PROMPT: \"ufo/prompts/share/base/api.yaml\"\nAPP_API_PROMPT_ADDRESS:\n \"WINWORD.EXE\": \"ufo/prompts/apps/word/api.yaml\"\n \"EXCEL.EXE\": \"ufo/prompts/apps/excel/api.yaml\"\n\n# Disable for pure UI automation\n# USE_APIS: False\n```\n\n!!!tip \"When to Use APIs\"\n COM APIs are faster and more reliable for Office applications. Keep `USE_APIS: True` for best results with Word, Excel, PowerPoint.\n\n---\n\n## Complete Example Configuration\n\nHere's a complete, production-ready `system.yaml`:\n\n```yaml\n# LLM Parameters\nMAX_TOKENS: 2000\nMAX_RETRY: 20\nTEMPERATURE: 0.0\nTOP_P: 0.0\nTIMEOUT: 60\n\n# Execution Limits\nMAX_STEP: 100\nMAX_ROUND: 3\nSLEEP_TIME: 1\nRECTANGLE_TIME: 1\n\n# Control Backend\nCONTROL_BACKEND: [\"uia\"]\nIOU_THRESHOLD_FOR_MERGE: 0.1\n\n# Action Configuration\nACTION_SEQUENCE: True\nSHOW_VISUAL_OUTLINE_ON_SCREEN: False\nMAXIMIZE_WINDOW: False\nJSON_PARSING_RETRY: 3\n\nCLICK_API: \"click_input\"\nAFTER_CLICK_WAIT: 0\n\nINPUT_TEXT_API: \"type_keys\"\nINPUT_TEXT_ENTER: False\nINPUT_TEXT_INTER_KEY_PAUSE: 0.05\n\n# Logging\nPRINT_LOG: False\nLOG_LEVEL: \"INFO\"\nLOG_TO_MARKDOWN: True\nLOG_XML: False\nCONCAT_SCREENSHOT: False\nINCLUDE_LAST_SCREENSHOT: True\nSCREENSHOT_TO_MEMORY: True\nREQUEST_TIMEOUT: 250\n\n# MCP Settings\nUSE_MCP: True\nMCP_SERVERS_CONFIG: \"config/ufo/mcp.yaml\"\nMCP_PREFERRED_APPS:\n - \"Code.exe\"\n - \"WindowsTerminal.exe\"\nMCP_FALLBACK_TO_UI: True\nMCP_TOOL_TIMEOUT: 30\nMCP_LOG_EXECUTION: False\n\n# Safety\nSAFE_GUARD: True\nCONTROL_LIST:\n - \"Button\"\n - \"Edit\"\n - \"TabItem\"\n - \"Document\"\n - \"ListItem\"\n - \"MenuItem\"\n - \"ScrollBar\"\n - \"TreeItem\"\n - \"Hyperlink\"\n - \"ComboBox\"\n - \"RadioButton\"\n\n# API Usage\nUSE_APIS: True\nAPI_PROMPT: \"ufo/prompts/share/base/api.yaml\"\nAPP_API_PROMPT_ADDRESS:\n \"WINWORD.EXE\": \"ufo/prompts/apps/word/api.yaml\"\n \"EXCEL.EXE\": \"ufo/prompts/apps/excel/api.yaml\"\n \"msedge.exe\": \"ufo/prompts/apps/web/api.yaml\"\n\n# Control Filtering (disabled by default)\nCONTROL_FILTER_TYPE: []\nCONTROL_FILTER_TOP_K_PLAN: 2\nCONTROL_FILTER_TOP_K_SEMANTIC: 15\nCONTROL_FILTER_TOP_K_ICON: 15\nCONTROL_FILTER_MODEL_SEMANTIC_NAME: \"all-MiniLM-L6-v2\"\nCONTROL_FILTER_MODEL_ICON_NAME: \"clip-ViT-B-32\"\n```\n\n---\n\n## Programmatic Access\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Access system settings\nmax_step = config.system.max_step\nlog_level = config.system.log_level\ncontrol_backends = config.system.control_backend\n\n# Check MCP settings\nif config.system.use_mcp:\n mcp_config_path = config.system.mcp_servers_config\n print(f\"MCP enabled, config: {mcp_config_path}\")\n\n# Modify at runtime (not recommended)\n# config.system.max_step = 200\n```\n\n---\n\n## Troubleshooting\n\n### Issue 1: Tasks Failing After X Steps\n\n!!!bug \"Error Message\"\n ```\n Task stopped: Maximum steps (50) reached\n ```\n \n **Solution**: Increase `MAX_STEP`\n ```yaml\n MAX_STEP: 100 # or higher\n ```\n\n### Issue 2: Controls Not Detected\n\n**Symptom:** UFO² can't find UI elements\n\n**Solutions:**\n1. Try enabling omniparser for vision-based detection:\n ```yaml\n CONTROL_BACKEND: [\"uia\", \"omniparser\"]\n ```\n2. Disable filtering:\n ```yaml\n CONTROL_FILTER_TYPE: []\n ```\n\n### Issue 3: Actions Too Fast\n\n**Symptom:** Actions execute before UI is ready\n\n**Solution:** Add delays\n```yaml\nSLEEP_TIME: 2\nAFTER_CLICK_WAIT: 1\n```\n\n### Issue 4: Logs Too Verbose\n\n**Symptom:** Too much console output\n\n**Solution:** Reduce logging\n```yaml\n PRINT_LOG: False\n LOG_LEVEL: \"WARNING\"\n ```\n\n---\n\n## Performance Tuning\n\n### For Speed\n\n```yaml\nMAX_STEP: 50\nSLEEP_TIME: 0\nCONTROL_BACKEND: [\"uia\"]\nCONTROL_FILTER_TYPE: [\"SEMANTIC\"] # Reduce LLM input\nACTION_SEQUENCE: True # Multi-action in one step\n```\n\n### For Reliability\n\n```yaml\nMAX_STEP: 100\nMAX_ROUND: 3\nSLEEP_TIME: 2\nAFTER_CLICK_WAIT: 1\nCONTROL_BACKEND: [\"uia\"]\nCONTROL_FILTER_TYPE: [] # Don't filter out controls\n```\n\n### For Debugging\n\n```yaml\nPRINT_LOG: True\nLOG_LEVEL: \"DEBUG\"\nLOG_XML: True\nSHOW_VISUAL_OUTLINE_ON_SCREEN: True\nMCP_LOG_EXECUTION: True\n```\n\n---\n\n## Related Documentation\n\n- **[Agent Configuration](agents_config.md)** - LLM and API settings\n- **[MCP Configuration](mcp_reference.md)** - Tool server configuration\n- **[RAG Configuration](rag_config.md)** - Knowledge retrieval\n\n## Summary\n\n**Key Takeaways:**\n\n✅ **Default settings work** - Start with defaults, adjust as needed \n✅ **Increase MAX_STEP** for complex tasks \n✅ **Use [\"uia\"]** for control detection \n✅ **Enable ACTION_SEQUENCE** for faster execution \n✅ **Adjust logging** based on dev vs production \n✅ **Enable MCP** for better Office automation\n\n**Fine-tune system settings for optimal performance!** ⚙️\n"
},
{
"path": "documents/docs/configuration/system/third_party_config.md",
"content": "# Third-Party Agent Configuration (third_party.yaml)\n\nConfigure third-party agents that extend UFO²'s capabilities beyond Windows GUI automation, such as LinuxAgent for CLI operations and HardwareAgent for physical device control.\n\n---\n\n## Overview\n\nThe `third_party.yaml` file configures external agents that integrate with UFO² to provide specialized capabilities. These agents work alongside the standard HostAgent and AppAgent to handle tasks that require non-GUI interactions.\n\n**File Location**: `config/ufo/third_party.yaml`\n\n**Advanced Feature:** Third-party agent configuration is an **advanced optional feature**. Most users only need the core agents (HostAgent, AppAgent). Configure third-party agents only when you need specialized capabilities.\n\n!!!tip \"Creating Custom Third-Party Agents\"\n Want to build your own third-party agent? See the **[Creating Custom Third-Party Agents Tutorial](../../tutorials/creating_third_party_agents.md)** for a complete step-by-step guide using HardwareAgent as an example.\n\n---\n\n## Quick Start\n\n### Default Configuration\n\n```yaml\n# Enable third-party agents\nENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n LinuxAgent:\n AGENT_NAME: \"LinuxAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/linux_agent_example.yaml\"\n INTRODUCTION: \"For Linux Use Only.\"\n```\n\n### Disable All Third-Party Agents\n\n```yaml\n# Disable all third-party agents\nENABLED_THIRD_PARTY_AGENTS: []\n```\n\n---\n\n## Available Third-Party Agents\n\n### LinuxAgent\n\n**Purpose**: Execute Linux CLI commands and server operations.\n\n!!!info \"UFO³ Integration\"\n LinuxAgent is used by **UFO³ Galaxy** to orchestrate Linux devices as sub-agents in multi-device workflows. When Galaxy routes a task to a Linux device, it uses LinuxAgent to execute commands via CLI.\n\n**Configuration**:\n```yaml\nTHIRD_PARTY_AGENT_CONFIG:\n LinuxAgent:\n AGENT_NAME: \"LinuxAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/linux_agent_example.yaml\"\n INTRODUCTION: \"For Linux Use Only.\"\n```\n\n**Fields**:\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `AGENT_NAME` | String | Agent identifier (must be \"LinuxAgent\") |\n| `APPAGENT_PROMPT` | String | Path to main prompt template |\n| `APPAGENT_EXAMPLE_PROMPT` | String | Path to example prompt template |\n| `INTRODUCTION` | String | Agent description for LLM context |\n\n**When to Enable**:\n- ✅ Using UFO³ Galaxy with Linux devices\n- ✅ Need to execute Linux CLI commands\n- ✅ Managing Linux servers from Windows\n- ✅ Cross-platform automation workflows\n\n**Related Documentation**:\n- [Linux Agent as Galaxy Device](../../linux/as_galaxy_device.md)\n- [Linux Agent Quick Start](../../getting_started/quick_start_linux.md)\n\n---\n\n### HardwareAgent\n\n**Purpose**: Control physical hardware components (robotic arms, USB devices, etc.).\n\n!!!warning \"Experimental Feature\"\n HardwareAgent is an experimental feature for controlling physical hardware. Requires specialized hardware setup and is not commonly used.\n\n**Configuration**:\n```yaml\nTHIRD_PARTY_AGENT_CONFIG:\n HardwareAgent:\n VISUAL_MODE: True\n AGENT_NAME: \"HardwareAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/share/base/app_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/examples/visual/app_agent_example.yaml\"\n API_PROMPT: \"ufo/prompts/third_party/hardware_agent_api.yaml\"\n INTRODUCTION: \"The HardwareAgent is used to manipulate hardware components of the computer without using GUI, such as robotic arms for keyboard input and mouse control, plug and unplug devices such as USB drives, and other hardware-related tasks.\"\n```\n\n**Fields**:\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `VISUAL_MODE` | Boolean | Enable visual mode (screenshot understanding) |\n| `AGENT_NAME` | String | Agent identifier (must be \"HardwareAgent\") |\n| `APPAGENT_PROMPT` | String | Path to main prompt template |\n| `APPAGENT_EXAMPLE_PROMPT` | String | Path to example prompt template |\n| `API_PROMPT` | String | Path to hardware API prompt template |\n| `INTRODUCTION` | String | Agent description for LLM context |\n\n**When to Enable**:\n- ✅ Using robotic arms for physical input\n- ✅ Automated USB device management\n- ✅ Physical hardware testing/automation\n- ✅ Research projects with hardware control\n\n**Related Documentation**:\n- [Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md) - Tutorial using HardwareAgent as example\n\n---\n\n## Configuration Fields\n\n### ENABLED_THIRD_PARTY_AGENTS\n\n**Type**: `List[String]` \n**Default**: `[]` (no third-party agents enabled)\n\nList of third-party agent names to enable. Only agents listed here will be loaded and available.\n\n**Options**:\n- `\"LinuxAgent\"` - Linux CLI execution\n- `\"HardwareAgent\"` - Physical hardware control\n\n**Examples**:\n```yaml\n# Enable LinuxAgent only (recommended for UFO³)\nENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\"]\n\n# Enable both agents\nENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\", \"HardwareAgent\"]\n\n# Disable all third-party agents\nENABLED_THIRD_PARTY_AGENTS: []\n```\n\n### THIRD_PARTY_AGENT_CONFIG\n\n**Type**: `Dict[String, Dict]`\n\nConfiguration dictionary for each third-party agent. Each agent has its own configuration block.\n\n**Structure**:\n```yaml\nTHIRD_PARTY_AGENT_CONFIG:\n AgentName:\n AGENT_NAME: \"AgentName\"\n # Agent-specific fields...\n```\n\n---\n\n## Complete Configuration Example\n\n### For UFO³ Galaxy (Recommended)\n\n```yaml\n# Enable LinuxAgent for UFO³ Galaxy multi-device orchestration\nENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n LinuxAgent:\n AGENT_NAME: \"LinuxAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/linux_agent_example.yaml\"\n INTRODUCTION: \"For Linux Use Only.\"\n```\n\n### With Hardware Support\n\n```yaml\n# Enable both Linux and Hardware agents\nENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\", \"HardwareAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n LinuxAgent:\n AGENT_NAME: \"LinuxAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/linux_agent_example.yaml\"\n INTRODUCTION: \"For Linux Use Only.\"\n \n HardwareAgent:\n VISUAL_MODE: True\n AGENT_NAME: \"HardwareAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/share/base/app_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/examples/visual/app_agent_example.yaml\"\n API_PROMPT: \"ufo/prompts/third_party/hardware_agent_api.yaml\"\n INTRODUCTION: \"The HardwareAgent is used to manipulate hardware components of the computer without using GUI, such as robotic arms for keyboard input and mouse control, plug and unplug devices such as USB drives, and other hardware-related tasks.\"\n```\n\n### Minimal (No Third-Party Agents)\n\n```yaml\n# Disable all third-party agents (default for standalone UFO²)\nENABLED_THIRD_PARTY_AGENTS: []\n```\n\n---\n\n## UFO³ Galaxy Integration\n\nWhen using UFO³ Galaxy for multi-device orchestration, LinuxAgent must be enabled to support Linux devices.\n\n### Setup for Galaxy\n\n**Step 1**: Enable LinuxAgent in `config/ufo/third_party.yaml`\n\n```yaml\nENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n LinuxAgent:\n AGENT_NAME: \"LinuxAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/linux_agent_example.yaml\"\n INTRODUCTION: \"For Linux Use Only.\"\n```\n\n**Step 2**: Configure Linux devices in `config/galaxy/devices.yaml`\n\n```yaml\ndevices:\n - device_id: \"linux_server_1\"\n server_url: \"ws://192.168.1.100:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server\"\n - \"cli\"\n - \"database\"\n```\n\n**Step 3**: Start Linux Agent components\n\nSee [Linux Agent as Galaxy Device](../../linux/as_galaxy_device.md) for complete setup.\n\n---\n\n## Programmatic Access\n\n```python\nfrom config.config_loader import get_ufo_config\n\nconfig = get_ufo_config()\n\n# Check which third-party agents are enabled\nenabled_agents = config.ENABLED_THIRD_PARTY_AGENTS\nprint(f\"Enabled third-party agents: {enabled_agents}\")\n\n# Access agent configuration\nif \"LinuxAgent\" in enabled_agents:\n linux_config = config.THIRD_PARTY_AGENT_CONFIG[\"LinuxAgent\"]\n print(f\"LinuxAgent prompt: {linux_config['APPAGENT_PROMPT']}\")\n\n# Check if specific agent is enabled\nlinux_enabled = \"LinuxAgent\" in config.ENABLED_THIRD_PARTY_AGENTS\nprint(f\"LinuxAgent enabled: {linux_enabled}\")\n```\n\n---\n\n## Adding Custom Third-Party Agents\n\nYou can add your own third-party agents by following the patterns described below. For a complete tutorial, see **[Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md)**.\n\n### Quick Overview\n\n### Step 1: Create Agent Implementation\n\n```python\n# ufo/agents/third_party/my_agent.py\nclass MyCustomAgent:\n def __init__(self, config):\n self.config = config\n # Initialize your agent\n```\n\n### Step 2: Add Configuration\n\n```yaml\nENABLED_THIRD_PARTY_AGENTS: [\"MyCustomAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n MyCustomAgent:\n AGENT_NAME: \"MyCustomAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/my_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/my_agent_example.yaml\"\n INTRODUCTION: \"Custom agent description.\"\n # Your custom fields\n CUSTOM_FIELD: \"value\"\n```\n\n### Step 3: Register Agent\n\nAdd your agent to the third-party agent registry in UFO²'s agent loader.\n\n---\n\n## Troubleshooting\n\n### Issue 1: LinuxAgent Not Working\n\n!!!bug \"Error Message\"\n ```\n LinuxAgent not found or not enabled\n ```\n \n **Solution**: Check configuration\n ```yaml\n # Verify LinuxAgent is in enabled list\n ENABLED_THIRD_PARTY_AGENTS: [\"LinuxAgent\"]\n ```\n\n### Issue 2: Prompt Files Not Found\n\n!!!bug \"Error Message\"\n ```\n FileNotFoundError: ufo/prompts/third_party/linux_agent.yaml\n ```\n \n **Solution**: Verify prompt files exist\n ```powershell\n # Check if prompt files exist\n Test-Path \"ufo\\prompts\\third_party\\linux_agent.yaml\"\n Test-Path \"ufo\\prompts\\third_party\\linux_agent_example.yaml\"\n ```\n\n### Issue 3: Agent Configuration Not Loaded\n\n!!!bug \"Symptom\"\n Third-party agent configuration changes not taking effect\n \n **Solution**: Restart UFO² application\n ```powershell\n # Configuration is loaded at startup\n # Restart UFO² to apply changes\n ```\n\n---\n\n## Best Practices\n\n!!!tip \"Recommendations\"\n - ✅ **Enable only what you need** - Don't enable agents you're not using\n - ✅ **For UFO³ Galaxy** - Always enable LinuxAgent when using Linux devices\n - ✅ **Keep prompts up to date** - Ensure prompt files exist and are current\n - ✅ **Document custom agents** - Add clear introduction text for LLM context\n - ✅ **Test configurations** - Verify agents load correctly after configuration changes\n\n!!!danger \"Warnings\"\n - ❌ **Don't enable HardwareAgent** without proper hardware setup\n - ❌ **Don't modify AGENT_NAME** - Must match the agent class name\n - ❌ **Don't delete prompt files** - Agents will fail to initialize\n\n---\n\n## Related Documentation\n\n- **[Creating Custom Third-Party Agents](../../tutorials/creating_third_party_agents.md)** - Complete tutorial for building third-party agents\n- **[Linux Agent as Galaxy Device](../../linux/as_galaxy_device.md)** - Using LinuxAgent in UFO³\n- **[Linux Agent Quick Start](../../getting_started/quick_start_linux.md)** - Setting up Linux Agent\n- **[Agent Configuration](./agents_config.md)** - Core agent LLM settings\n- **[Galaxy Devices Configuration](./galaxy_devices.md)** - Multi-device setup\n\n---\n\n## Summary\n\n!!!success \"Key Takeaways\"\n ✅ **third_party.yaml is optional** - Only needed for specialized agents \n ✅ **LinuxAgent for UFO³** - Required when using Linux devices in Galaxy \n ✅ **HardwareAgent is experimental** - For physical hardware control \n ✅ **Enable selectively** - Only enable agents you actually use \n ✅ **Configuration is simple** - Just add agent names to enabled list \n \n **Extend UFO² with specialized capabilities!** 🔧\n"
},
{
"path": "documents/docs/faq.md",
"content": "# Frequently Asked Questions (FAQ)\n\nQuick answers to common questions about UFO³ Galaxy, UFO², Linux Agents, and general troubleshooting.\n\n---\n\n## 🎯 General Questions\n\n### Q: What is UFO³?\n\n**A:** UFO³ is the third iteration of the UFO project, encompassing three major frameworks:\n\n- **UFO²** - Desktop AgentOS for Windows automation\n- **UFO³ Galaxy** - Multi-device orchestration framework \n- **Linux Agent** - Server and CLI automation for Linux\n\n### Q: Why is it called UFO?\n\n**A:** UFO stands for **U**I **Fo**cused agent. The name was given to the first version of the project and has been retained through all iterations (UFO v1, UFO², UFO³) as the project evolved from a simple UI-focused agent to a comprehensive multi-device orchestration framework.\n\n### Q: Which version should I use?\n\n**A:** Choose based on your needs:\n\n| Use Case | Recommended Version |\n|----------|-------------------|\n| Windows desktop automation only | [UFO²](getting_started/quick_start_ufo2.md) |\n| Cross-device workflows (Windows + Linux) | [UFO³ Galaxy](getting_started/quick_start_galaxy.md) |\n| Linux server management only | [Linux Agent](getting_started/quick_start_linux.md) |\n| Multi-device orchestration | [UFO³ Galaxy](getting_started/quick_start_galaxy.md) |\n\n### Q: What's the difference between UFO² and UFO³ Galaxy?\n\n**A: UFO²** is for single Windows desktop automation with:\n- Deep Windows OS integration (UIA, Win32, COM)\n- Office application automation\n- GUI + API hybrid execution\n\n**UFO³ Galaxy** orchestrates multiple devices with:\n- Cross-platform support (Windows + Linux)\n- Distributed task execution\n- Device capability-based routing\n- Constellation-based DAG orchestration\n\nSee [Migration Guide](getting_started/migration_ufo2_to_galaxy.md) for details.\n\n### Q: Can I use UFO on Linux or macOS?\n\n**A:** Yes and No:\n\n- **✅ Linux:** Supported via Linux Agent for server/CLI automation\n- **❌ macOS:** Not currently supported (Windows and Linux only)\n- **Windows:** Full UFO² desktop automation support\n\n---\n\n## 🔧 Installation & Setup\n\n### Q: Which Python version do I need?\n\n**A:** Python **3.10 or higher** is required for all UFO³ components.\n\n```bash\n# Check your Python version\npython --version\n```\n\n### Q: What models does UFO support?\n\n**A:** UFO³ supports multiple LLM providers:\n\n- **OpenAI** - GPT-4o, GPT-4, GPT-3.5\n- **Azure OpenAI** - All Azure-hosted models\n- **Google Gemini** - Gemini Pro, Gemini Flash\n- **Anthropic Claude** - Claude 3.5, Claude 3\n- **Qwen** - Local or API deployment\n- **DeepSeek** - DeepSeek models\n- **Ollama** - Local model hosting\n- And more...\n\nSee [Model Configuration Guide](configuration/models/overview.md) for the complete list and setup instructions.\n\n### Q: Can I use non-vision models in UFO?\n\n**A:** Yes! You can disable visual mode:\n\n```yaml\n# config/ufo/system.yaml\nVISUAL_MODE: false\n```\n\nHowever, UFO² is designed for vision models. Non-vision models may have reduced performance for GUI automation tasks.\n\n### Q: Can I host my own LLM endpoint?\n\n**A:** Yes! UFO³ supports custom endpoints:\n\n```yaml\n# config/ufo/agents.yaml\nHOST_AGENT:\n API_TYPE: \"openai\" # Or compatible API\n API_BASE: \"http://your-endpoint.com/v1/chat/completions\"\n API_KEY: \"your-key\"\n API_MODEL: \"your-model-name\"\n```\n\nSee [Model Configuration](configuration/models/overview.md) for details.\n\n### Q: Do I need API keys for all agents?\n\n**A:** No, only for LLM-powered agents:\n\n| Component | Requires API Key | Purpose |\n|-----------|-----------------|---------|\n| **ConstellationAgent** (Galaxy) | ✅ Yes | Orchestration reasoning |\n| **HostAgent** (UFO²) | ✅ Yes | Task planning |\n| **AppAgent** (UFO²) | ✅ Yes | Action execution |\n| **LinuxAgent** | ✅ Yes | Command planning |\n| **Device Server** | ❌ No | Message routing only |\n| **MCP Servers** | ❌ No | Tool provider only |\n\n---\n\n## ⚙️ Configuration\n\n### Q: Where are configuration files located?\n\n**A:** UFO³ uses a modular configuration system in `config/`:\n\n```\nconfig/\n├── ufo/ # UFO² configuration\n│ ├── agents.yaml # LLM and agent settings\n│ ├── system.yaml # Runtime settings\n│ ├── rag.yaml # Knowledge retrieval\n│ └── mcp.yaml # MCP server configuration\n└── galaxy/ # Galaxy configuration\n ├── agent.yaml # ConstellationAgent LLM\n ├── devices.yaml # Device pool\n └── constellation.yaml # Runtime settings\n```\n\n### Q: Can I still use the old `ufo/config/config.yaml`?\n\n**A:** Yes, for backward compatibility, but we recommend migrating to the new modular system:\n\n```bash\n# Check current configuration\npython -m ufo.tools.validate_config ufo --show-config\n\n# Migrate from legacy to new\npython -m ufo.tools.migrate_config\n```\n\nSee [Configuration Migration Guide](configuration/system/migration.md) for details.\n\n### Q: How do I protect my API keys?\n\n**A:** Best practices for API key security:\n\n1. **Never commit `.yaml` files with keys** - Use `.template` files\n ```bash\n # Good pattern\n config/ufo/agents.yaml.template # Commit this (with placeholders)\n config/ufo/agents.yaml # DON'T commit (has real keys)\n ```\n\n2. **Use environment variables** for sensitive data:\n ```yaml\n # In agents.yaml\n HOST_AGENT:\n API_KEY: ${OPENAI_API_KEY} # Reads from environment\n ```\n\n3. **Add to `.gitignore`**:\n ```\n config/**/agents.yaml\n config/**/agent.yaml\n !**/*.template\n ```\n\n---\n\n## 🌌 UFO³ Galaxy Questions\n\n### Q: What's the minimum number of devices for Galaxy?\n\n**A:** Galaxy requires **at least 1 device agent** (Windows or Linux) to be useful, but you can start with just one device and add more later.\n\n```yaml\n# Minimal Galaxy setup (1 device)\ndevices:\n - device_id: \"my_windows_pc\"\n server_url: \"ws://localhost:5000/ws\"\n os: \"windows\"\n```\n\n### Q: Can Galaxy mix Windows and Linux devices?\n\n**A:** Yes! Galaxy can orchestrate heterogeneous devices:\n\n```yaml\ndevices:\n - device_id: \"windows_desktop\"\n os: \"windows\"\n capabilities: [\"office\", \"excel\", \"outlook\"]\n \n - device_id: \"linux_server\"\n os: \"linux\"\n capabilities: [\"server\", \"database\", \"log_analysis\"]\n```\n\nGalaxy automatically routes tasks based on device capabilities.\n\n### Q: Do all devices need to be on the same network?\n\n**A:** No, devices can be distributed across networks using SSH tunneling:\n\n- **Same network:** Direct WebSocket connections\n- **Different networks:** Use SSH tunnels (reverse/forward)\n- **Cloud + local:** SSH tunnels with public gateways\n\nSee [Linux Quick Start - SSH Tunneling](getting_started/quick_start_linux.md#network-connectivity-ssh-tunneling) for examples.\n\n### Q: How does Galaxy decide which device to use?\n\n**A:** Galaxy uses **capability-based routing**:\n\n1. Analyzes the task requirements\n2. Matches against device `capabilities` in `devices.yaml`\n3. Considers device `metadata` (OS, performance, etc.)\n4. Selects the best-fit device(s)\n\nExample:\n```yaml\n# Task: \"Analyze error logs on the production server\"\n# → Galaxy routes to device with:\ncapabilities:\n - \"log_analysis\"\n - \"server_management\"\nos: \"linux\"\n```\n\n---\n\n## 🐧 Linux Agent Questions\n\n### Q: Does the Linux Agent require a GUI?\n\n**A:** No! The Linux Agent is designed for headless servers:\n\n- Executes CLI commands via MCP\n- No X11/desktop environment needed\n- Works over SSH\n- Perfect for remote servers\n\n### Q: Can I run multiple Linux Agents on one machine?\n\n**A:** Yes, using different ports and client IDs:\n\n```bash\n# Agent 1\npython -m ufo.server.app --port 5001\npython -m ufo.client.client --ws --client-id linux_1 --platform linux\n\n# Agent 2 (same machine)\npython -m ufo.server.app --port 5002\npython -m ufo.client.client --ws --client-id linux_2 --platform linux\n```\n\n### Q: What's the MCP service for?\n\n**A:** The MCP (Model Context Protocol) service provides the **actual command execution tools** for the Linux Agent:\n\n```\nLinux Agent (LLM reasoning)\n ↓\nMCP Service (tool provider)\n ↓\nBash commands (actual execution)\n```\n\nWithout MCP, the Linux Agent can't execute commands - it can only plan them.\n\n---\n\n## 🪟 UFO² Questions\n\n### Q: Does UFO² work on Windows 10?\n\n**A:** Yes! UFO² supports:\n- ✅ Windows 11 (recommended)\n- ✅ Windows 10 (fully supported)\n- ❌ Windows 8.1 or earlier (not tested)\n\n### Q: Can UFO² automate Office apps?\n\n**A:** Yes! UFO² has enhanced Office support through:\n- **MCP Office servers** - Direct API access to Excel, Word, Outlook, PowerPoint\n- **GUI automation** - Fallback for unsupported operations\n- **Hybrid execution** - Automatically chooses API or GUI\n\nEnable MCP in `config/ufo/mcp.yaml` for better Office automation.\n\n### Q: Does UFO² interrupt my work?\n\n**A:** UFO² can run automation tasks on your current desktop. For non-disruptive operation, you can run it on a separate machine or virtual desktop environment.\n\n> **Note:** Picture-in-Picture mode is planned for future releases.\n\n### Q: Can I use UFO² without MCP?\n\n**A:** UFO² requires MCP (Model Context Protocol) servers for tool execution. MCP provides the interface between the LLM agents and system operations (Windows APIs, Office automation, etc.). Without MCP, UFO² cannot perform actions.\n\n---\n\n## 🐛 Common Issues & Troubleshooting\n\n### Issue: \"Configuration file not found\"\n\n**Error:**\n```\nFileNotFoundError: config/ufo/agents.yaml not found\n```\n\n**Solution:**\n```bash\n# Copy template files\ncp config/ufo/agents.yaml.template config/ufo/agents.yaml\n\n# Edit with your API keys\nnotepad config/ufo/agents.yaml # Windows\nnano config/ufo/agents.yaml # Linux\n```\n\n### Issue: \"API Authentication Error\"\n\n**Error:**\n```\nopenai.AuthenticationError: Invalid API key\n```\n\n**Solutions:**\n\n1. **Check API key format:**\n ```yaml\n API_KEY: \"sk-...\" # OpenAI starts with sk-\n API_KEY: \"...\" # Azure uses deployment key\n ```\n\n2. **Verify API_TYPE matches your provider:**\n ```yaml\n API_TYPE: \"openai\" # For OpenAI\n API_TYPE: \"aoai\" # For Azure OpenAI\n ```\n\n3. **Check for extra spaces/quotes** in YAML\n\n4. **For Azure:** Verify `API_DEPLOYMENT_ID` is set\n\n### Issue: \"Connection aborted / Remote end closed connection\"\n\n**Error:**\n```\nError making API request: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))\n```\n\n**Solutions:**\n\n- Check network connection (VPN, proxy, firewall)\n- Verify LLM endpoint is accessible: `curl https://api.openai.com/v1/models`\n- Check endpoint status (Azure, OpenAI, etc.)\n- Try increasing timeout in config\n- Verify API base URL is correct\n\n### Issue: \"Device not connecting to Galaxy\"\n\n**Error:**\n```\nERROR - [WS] Failed to connect to ws://localhost:5000/ws\nConnection refused\n```\n\n**Checklist:**\n\n- [ ] Is the server running? (`curl http://localhost:5000/api/health`)\n- [ ] Port number correct? (Server: `--port 5000`, Client: `ws://...:5000/ws`)\n- [ ] Platform flag set? (`--platform windows` or `--platform linux`)\n- [ ] Firewall blocking? (Allow port 5000)\n- [ ] SSH tunnel established? (If using remote devices)\n\n### Issue: \"device_id mismatch in Galaxy\"\n\n**Error:**\n```\nERROR - Device 'linux_agent_1' not found in configuration\n```\n\n**Cause:** Mismatch between `devices.yaml` and client command\n\n**Solution:** Ensure exact match:\n\n| Location | Field | Example |\n|----------|-------|---------|\n| `devices.yaml` | `device_id:` | `\"linux_agent_1\"` |\n| Client command | `--client-id` | `linux_agent_1` |\n\n**Critical:** IDs must match **exactly** (case-sensitive, no typos).\n\n### Issue: \"MCP service not responding (Linux)\"\n\n**Error:**\n```\nERROR - Cannot connect to MCP server at http://127.0.0.1:8010\n```\n\n**Solutions:**\n\n1. **Check if MCP service is running:**\n ```bash\n curl http://localhost:8010/health\n ps aux | grep linux_mcp_server\n ```\n\n2. **Restart MCP service:**\n ```bash\n pkill -f linux_mcp_server\n python -m ufo.client.mcp.http_servers.linux_mcp_server\n ```\n\n3. **Check port conflict:**\n ```bash\n lsof -i :8010\n # If port taken, use different port:\n python -m ufo.client.mcp.http_servers.linux_mcp_server --port 8011\n ```\n\n### Issue: \"Tasks failing after X steps\"\n\n**Cause:** `MAX_STEP` limit reached\n\n**Solution:** Increase step limit in `config/ufo/system.yaml`:\n\n```yaml\n# Default is 50\nMAX_STEP: 100 # For complex tasks\n\n# Or disable limit (not recommended)\nMAX_STEP: -1\n```\n\n### Issue: \"Too many LLM calls / high cost\"\n\n**Solutions:**\n\n1. **Enable action sequences** (bundles actions):\n ```yaml\n # config/ufo/system.yaml\n ACTION_SEQUENCE: true\n ```\n\n2. **Use vision-capable models for GUI tasks:**\n ```yaml\n # config/ufo/agents.yaml\n APP_AGENT:\n API_MODEL: \"gpt-4o\" # Use vision models for GUI automation\n ```\n \n > **Note:** Non-vision models like gpt-3.5-turbo cannot process screenshots and should not be used for GUI automation tasks.\n\n3. **Enable experience learning** (reuse patterns):\n ```yaml\n # config/ufo/rag.yaml\n RAG_EXPERIENCE: true\n ```\n\n### Issue: \"Why is the latency high?\"\n\n**A:** Latency depends on several factors:\n\n- **LLM response time** - GPT-4o typically takes 10-30 seconds per step\n- **Network speed** - API calls to OpenAI/Azure endpoints\n- **Endpoint workload** - Provider server load\n- **Visual mode** - Image processing adds overhead\n\n**To reduce latency:**\n- Use faster models (gpt-3.5-turbo vs gpt-4o)\n- Enable action sequences to batch operations\n- Use local models (Ollama) if acceptable\n- Disable visual mode if not needed\n\n### Issue: \"Can I use non-English requests?\"\n\n**A:** Yes! Most modern LLMs support multiple languages:\n\n- GPT-4o, GPT-4: Excellent multilingual support\n- Gemini: Good multilingual support\n- Qwen: Excellent for Chinese\n- Claude: Good multilingual support\n\nPerformance may vary by language and model. Test with your specific language and model combination.\n\n---\n\n## 📚 Where to Find More Help\n\n### Documentation\n\n| Topic | Link |\n|-------|------|\n| **Getting Started** | [UFO² Quick Start](getting_started/quick_start_ufo2.md), [Galaxy Quick Start](getting_started/quick_start_galaxy.md), [Linux Quick Start](getting_started/quick_start_linux.md) |\n| **Configuration** | [Configuration Overview](configuration/system/overview.md) |\n| **Troubleshooting** | Quick start guides have detailed troubleshooting sections |\n| **Architecture** | [Project Structure](project_directory_structure.md) |\n| **More Guidance** | [User & Developer Guide](getting_started/more_guidance.md) |\n\n### Community & Support\n\n- **GitHub Discussions:** [https://github.com/microsoft/UFO/discussions](https://github.com/microsoft/UFO/discussions)\n- **GitHub Issues:** [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues)\n- **Email:** ufo-agent@microsoft.com\n\n### Debugging Tips\n\n1. **Enable debug logging:**\n ```yaml\n # config/ufo/system.yaml\n LOG_LEVEL: \"DEBUG\"\n ```\n\n2. **Check log files:**\n ```\n logs//\n ├── request.log # Request logs\n ├── response.log # Response logs\n ├── action_step*.png # Screenshots at each step\n └── action_step*_annotated.png # Annotated screenshots\n ```\n\n3. **Validate configuration:**\n ```bash\n python -m ufo.tools.validate_config ufo --show-config\n python -m ufo.tools.validate_config galaxy --show-config\n ```\n\n4. **Test LLM connectivity:**\n ```python\n # Test your API key\n from openai import OpenAI\n client = OpenAI(api_key=\"your-key\")\n response = client.chat.completions.create(\n model=\"gpt-4o\",\n messages=[{\"role\": \"user\", \"content\": \"Hello\"}]\n )\n print(response.choices[0].message.content)\n ```\n\n---\n\n> **💡 Still have questions?** Check the [More Guidance](getting_started/more_guidance.md) page for additional resources, or reach out to the community!\n"
},
{
"path": "documents/docs/galaxy/agent_registration/agent_profile.md",
"content": "# 📊 AgentProfile - Comprehensive Agent Representation\n\nThe **AgentProfile** is a multi-source data structure that consolidates administrator configuration, service-level capabilities, and real-time client telemetry into a unified, dynamically updated representation of each constellation agent.\n\n---\n\n## 📋 Overview\n\nThe **AgentProfile** is the primary data structure representing a registered constellation agent. It aggregates information from **three distinct sources** to provide a comprehensive view of each agent's identity, capabilities, operational status, and hardware characteristics.\n\nFor a complete understanding of how agents work in the constellation system, see:\n\n- [Constellation Overview](../constellation/overview.md) - Architecture and multi-device coordination\n- [Constellation Agent](../constellation_agent/overview.md) - Agent behavior and lifecycle\n\n| Function | Description |\n|----------|-------------|\n| **Identity Management** | Unique identification and endpoint tracking |\n| **Capability Advertisement** | Declare supported features and tools |\n| **Status Monitoring** | Real-time operational state tracking |\n| **Resource Profiling** | Hardware and system information |\n| **Task Assignment** | Enable intelligent task routing decisions |\n\n---\n\n## 🏗️ Structure Definition\n\n### Core Dataclass\n\n```python\nfrom dataclasses import dataclass, field\nfrom typing import Dict, List, Optional, Any\nfrom datetime import datetime\nfrom enum import Enum\n\nclass DeviceStatus(Enum):\n \"\"\"Device connection status\"\"\"\n DISCONNECTED = \"disconnected\"\n CONNECTING = \"connecting\"\n CONNECTED = \"connected\"\n FAILED = \"failed\"\n REGISTERING = \"registering\"\n BUSY = \"busy\"\n IDLE = \"idle\"\n\n@dataclass\nclass AgentProfile:\n \"\"\"\n Device information and capabilities.\n \n Consolidates information from three sources:\n 1. User-specified registration (devices.yaml)\n 2. Service-level manifest (AIP registration)\n 3. Client-side telemetry (DeviceInfoProvider)\n \"\"\"\n \n # === Identity ===\n device_id: str # Unique device identifier\n server_url: str # WebSocket endpoint URL\n \n # === Platform & Capabilities ===\n os: Optional[str] = None # Operating system (windows, linux, darwin)\n capabilities: List[str] = field(default_factory=list) # Advertised capabilities\n metadata: Dict[str, Any] = field(default_factory=dict) # Multi-source metadata\n \n # === Operational Status ===\n status: DeviceStatus = DeviceStatus.DISCONNECTED # Current state\n last_heartbeat: Optional[datetime] = None # Last heartbeat timestamp\n \n # === Connection Management ===\n connection_attempts: int = 0 # Connection retry counter\n max_retries: int = 5 # Maximum retry attempts\n \n # === Task Execution ===\n current_task_id: Optional[str] = None # Currently executing task ID\n```\n\n---\n\n## 🔍 Field Reference\n\n### Identity Fields\n\n| Field | Type | Source | Description | Example |\n|-------|------|--------|-------------|---------|\n| `device_id` | `str` | User Config | Unique identifier for the device | `\"windowsagent\"`, `\"linux_gpu_01\"` |\n| `server_url` | `str` | User Config | WebSocket endpoint of device agent server | `\"ws://localhost:5005/ws\"` |\n\nThe `device_id` must be unique across the entire constellation. Attempting to register a duplicate `device_id` will fail.\n\n### Platform & Capabilities\n\n| Field | Type | Source | Description | Example |\n|-------|------|--------|-------------|---------|\n| `os` | `Optional[str]` | User Config + Telemetry | Operating system type | `\"windows\"`, `\"linux\"`, `\"darwin\"` |\n| `capabilities` | `List[str]` | User Config + Telemetry | Advertised capabilities/features | `[\"gui\", \"browser\", \"office\"]` |\n| `metadata` | `Dict[str, Any]` | All Sources | Multi-source metadata aggregation | See [Metadata Structure](#metadata-structure) |\n\n**Capabilities Merging:**\n\n```python\n# Initial capabilities from user config\ncapabilities = [\"web_browsing\", \"office_applications\"]\n\n# After telemetry collection, auto-detected features are merged\n# Result: [\"web_browsing\", \"office_applications\", \"gui\", \"cli\", \"browser\", \"file_system\"]\n```\n\n### Operational Status\n\n| Field | Type | Source | Description | Example |\n|-------|------|--------|-------------|---------|\n| `status` | `DeviceStatus` | Runtime | Current connection/operational state | `DeviceStatus.IDLE` |\n| `last_heartbeat` | `Optional[datetime]` | Runtime | Timestamp of last heartbeat | `2025-11-06T10:30:45Z` |\n\n**Status Values:**\n\n```python\nDeviceStatus.DISCONNECTED # Not connected\nDeviceStatus.CONNECTING # Connection in progress\nDeviceStatus.CONNECTED # WebSocket established\nDeviceStatus.REGISTERING # Performing AIP registration\nDeviceStatus.IDLE # Ready for tasks\nDeviceStatus.BUSY # Executing a task\nDeviceStatus.FAILED # Connection or execution failed\n```\n\n### Connection Management\n\n| Field | Type | Source | Description | Example |\n|-------|------|--------|-------------|---------|\n| `connection_attempts` | `int` | Runtime | Number of connection attempts made | `0`, `3` |\n| `max_retries` | `int` | User Config | Maximum reconnection attempts before giving up | `5`, `10` |\n\nWhen a device disconnects, the system automatically retries connection up to `max_retries` times with exponential backoff.\n\n### Task Execution\n\n| Field | Type | Source | Description | Example |\n|-------|------|--------|-------------|---------|\n| `current_task_id` | `Optional[str]` | Runtime | ID of task currently being executed | `\"task_12345\"`, `None` |\n\n**Usage in Task Queue:**\n\n```python\n# When task is assigned\nprofile.status = DeviceStatus.BUSY\nprofile.current_task_id = \"task_12345\"\n\n# When task completes\nprofile.status = DeviceStatus.IDLE\nprofile.current_task_id = None\n```\n\n---\n\n## 🗂️ Metadata Structure\n\nThe `metadata` dictionary is a flexible container that aggregates information from all three profiling sources:\n\n### Metadata Schema\n\n```python\nmetadata = {\n # ===== Source 1: User Configuration =====\n \"location\": str, # Physical location\n \"performance\": str, # Performance tier\n \"description\": str, # Human-readable description\n \"operation_engineer_email\": str, # Contact information\n \"tags\": List[str], # Custom tags\n # ... any custom user-defined fields\n \n # ===== Source 2: Service Manifest =====\n \"platform\": str, # Platform type (from registration)\n \"registration_time\": str, # ISO timestamp of registration\n \n # ===== Source 3: Client Telemetry =====\n \"system_info\": {\n \"platform\": str, # OS platform (windows, linux, darwin)\n \"os_version\": str, # OS version string\n \"cpu_count\": int, # Number of CPU cores\n \"memory_total_gb\": float, # Total RAM in GB\n \"hostname\": str, # Device hostname\n \"ip_address\": str, # Device IP address\n \"platform_type\": str, # Device category (computer, mobile, etc.)\n \"schema_version\": str # Telemetry schema version\n },\n \"custom_metadata\": { # Optional custom metadata from config\n \"datacenter\": str,\n \"tier\": str,\n # ... server-configured metadata\n }\n}\n```\n\n### Example Metadata\n\n```python\n# Complete metadata example from a Windows GPU workstation\nmetadata = {\n # User Configuration\n \"location\": \"office_desktop\",\n \"performance\": \"very_high\",\n \"description\": \"Primary Windows workstation with GPU\",\n \"operation_engineer_email\": \"admin@example.com\",\n \"tags\": [\"production\", \"gpu-enabled\", \"high-priority\"],\n \n # Service Manifest\n \"platform\": \"windows\",\n \"registration_time\": \"2025-11-06T10:30:00.000Z\",\n \n # Client Telemetry\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-GPU01\",\n \"ip_address\": \"192.168.1.100\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n },\n \"custom_metadata\": {\n \"datacenter\": \"us-west-2\",\n \"tier\": \"premium\",\n \"gpu_type\": \"NVIDIA RTX 4090\",\n \"gpu_count\": 1\n }\n}\n```\n\n---\n\n## 🔄 Multi-Source Construction\n\n### Three-Source Architecture\n\n```mermaid\ngraph LR\n A[User Config devices.yaml]\n B[AIP Registration Service Manifest]\n C[Device Telemetry DeviceInfoProvider]\n \n A -->|device_id, server_url capabilities, metadata| D[AgentProfile]\n B -->|platform, registration_time| D\n C -->|system_info, features| D\n \n style A fill:#e1f5ff\n style B fill:#fff4e1\n style C fill:#e8f5e9\n style D fill:#f3e5f5\n```\n\n### Construction Timeline\n\n```mermaid\nsequenceDiagram\n participant Config as devices.yaml\n participant Manager as DeviceManager\n participant Server as UFO Server\n participant Telemetry as DeviceInfoProvider\n \n Note over Config,Telemetry: Phase 1: Initial Registration\n Config->>Manager: Load device config\n Manager->>Manager: Create AgentProfile (device_id, server_url, capabilities)\n \n Note over Config,Telemetry: Phase 2: Service Registration\n Manager->>Server: WebSocket REGISTER\n Server-->>Manager: Add platform, registration_time\n \n Note over Config,Telemetry: Phase 3: Telemetry Collection\n Manager->>Server: request_device_info()\n Server->>Telemetry: collect_system_info()\n Telemetry-->>Server: system_info\n Server-->>Manager: system_info\n Manager->>Manager: Update AgentProfile (merge system_info & features)\n```\n\n### Merging Strategy\n\n**1. User Configuration (Priority: Baseline)**\n\n```python\n# Initial AgentProfile creation\nprofile = AgentProfile(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\", \"office_applications\"],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\"\n }\n)\n```\n\n**2. Service Manifest (Priority: Override `os`, Add registration data)**\n\n```python\n# During AIP registration\nprofile.metadata.update({\n \"platform\": \"windows\", # From registration message\n \"registration_time\": \"2025-11-06T10:30:00Z\"\n})\n```\n\n**3. Client Telemetry (Priority: Merge capabilities, Add system_info)**\n\n```python\n# After DeviceInfoProvider collects data\nsystem_info = {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-GPU01\",\n \"ip_address\": \"192.168.1.100\",\n \"supported_features\": [\"gui\", \"cli\", \"browser\", \"file_system\", \"office\", \"windows_apps\"],\n \"platform_type\": \"computer\"\n}\n\n# Update OS if not already set\nif not profile.os:\n profile.os = system_info[\"platform\"]\n\n# Merge capabilities (avoid duplicates)\nexisting_caps = set(profile.capabilities)\nnew_caps = set(system_info[\"supported_features\"])\nprofile.capabilities = list(existing_caps.union(new_caps))\n# Result: [\"web_browsing\", \"office_applications\", \"gui\", \"cli\", \"browser\", \"file_system\", \"windows_apps\"]\n\n# Add system_info to metadata\nprofile.metadata[\"system_info\"] = system_info\n```\n\n---\n\n## 📊 Example Profiles\n\n### Example 1: Windows GPU Workstation\n\n```python\nAgentProfile(\n # Identity\n device_id=\"gpu_workstation_01\",\n server_url=\"ws://192.168.1.100:5005/ws\",\n \n # Platform & Capabilities\n os=\"windows\",\n capabilities=[\n # User-configured\n \"web_browsing\",\n \"office_applications\",\n \"gpu_computation\",\n \"model_training\",\n # Auto-detected\n \"gui\",\n \"cli\",\n \"browser\",\n \"file_system\",\n \"windows_apps\"\n ],\n \n # Metadata\n metadata={\n # User Configuration\n \"location\": \"office_desktop\",\n \"performance\": \"very_high\",\n \"description\": \"Primary GPU workstation for ML training\",\n \"operation_engineer_email\": \"ml-team@example.com\",\n \"tags\": [\"production\", \"gpu\", \"ml\"],\n \n # Service Manifest\n \"platform\": \"windows\",\n \"registration_time\": \"2025-11-06T10:30:00Z\",\n \n # Client Telemetry\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 64.0,\n \"hostname\": \"DESKTOP-GPU01\",\n \"ip_address\": \"192.168.1.100\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n },\n \"custom_metadata\": {\n \"gpu_type\": \"NVIDIA RTX 4090\",\n \"gpu_count\": 2,\n \"gpu_memory_gb\": 48\n }\n },\n \n # Status\n status=DeviceStatus.IDLE,\n last_heartbeat=datetime(2025, 11, 6, 10, 45, 30),\n \n # Connection\n connection_attempts=0,\n max_retries=5,\n \n # Task\n current_task_id=None\n)\n```\n\n### Profile Summary\n\n```mermaid\ngraph TB\n subgraph \"AgentProfile: gpu_workstation_01\"\n A[\"Status: IDLE Last Heartbeat: 10:45:30\"]\n \n B[\"System ━━━━━ OS: Windows 10.0.22631 CPU: 16 cores Memory: 64.0 GB Host: DESKTOP-GPU01 IP: 192.168.1.100\"]\n \n C[\"Capabilities ━━━━━ • web_browsing • office_applications • gpu_computation • model_training • gui, cli, browser • file_system\"]\n \n D[\"Metadata ━━━━━ Location: office_desktop Performance: very_high Tags: production, gpu, ml GPU: 2× NVIDIA RTX 4090\"]\n end\n \n style A fill:#e3f2fd\n style B fill:#f3e5f5\n style C fill:#e8f5e9\n style D fill:#fff3e0\n```\n\n### Example 2: Linux Server\n\n```python\nAgentProfile(\n # Identity\n device_id=\"linux_server_01\",\n server_url=\"ws://10.0.0.50:5001/ws\",\n \n # Platform & Capabilities\n os=\"linux\",\n capabilities=[\n # User-configured\n \"server_management\",\n \"log_monitoring\",\n \"database_operations\",\n # Auto-detected\n \"cli\",\n \"file_system\",\n \"linux_apps\"\n ],\n \n # Metadata\n metadata={\n # User Configuration\n \"location\": \"datacenter_rack_a42\",\n \"performance\": \"medium\",\n \"description\": \"Production Linux server for backend services\",\n \"logs_file_path\": \"/var/log/application.log\",\n \"dev_path\": \"/home/deploy/\",\n \n # Service Manifest\n \"platform\": \"linux\",\n \"registration_time\": \"2025-11-06T09:15:00Z\",\n \n # Client Telemetry\n \"system_info\": {\n \"platform\": \"linux\",\n \"os_version\": \"#1 SMP PREEMPT_DYNAMIC Wed Nov 1 15:36:23 UTC 2023\",\n \"cpu_count\": 8,\n \"memory_total_gb\": 16.0,\n \"hostname\": \"prod-server-01\",\n \"ip_address\": \"10.0.0.50\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n }\n },\n \n # Status\n status=DeviceStatus.BUSY,\n last_heartbeat=datetime(2025, 11, 6, 10, 44, 15),\n \n # Connection\n connection_attempts=0,\n max_retries=3,\n \n # Task\n current_task_id=\"task_monitoring_567\"\n)\n```\n\n---\n\n## 🔄 Lifecycle Operations\n\n### Creation\n\n```python\nfrom galaxy.client.components import DeviceRegistry, AgentProfile, DeviceStatus\n\nregistry = DeviceRegistry()\n\n# Create AgentProfile during registration\nprofile = registry.register_device(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\", \"office\"],\n metadata={\"location\": \"office\"},\n max_retries=5\n)\n\nprint(f\"Created: {profile.device_id}\")\nprint(f\"Status: {profile.status.value}\") # \"disconnected\"\n```\n\n### Status Updates\n\n```python\n# Update connection status\nregistry.update_device_status(\"windowsagent\", DeviceStatus.CONNECTING)\nregistry.update_device_status(\"windowsagent\", DeviceStatus.CONNECTED)\nregistry.update_device_status(\"windowsagent\", DeviceStatus.IDLE)\n\n# Set device busy with task\nregistry.set_device_busy(\"windowsagent\", task_id=\"task_123\")\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Status: {profile.status.value}\") # \"busy\"\nprint(f\"Current Task: {profile.current_task_id}\") # \"task_123\"\n\n# Set device idle (task complete)\nregistry.set_device_idle(\"windowsagent\")\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Status: {profile.status.value}\") # \"idle\"\nprint(f\"Current Task: {profile.current_task_id}\") # None\n```\n\n### System Info Updates\n\n```python\n# Update with telemetry data\nsystem_info = {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"supported_features\": [\"gui\", \"cli\", \"browser\", \"file_system\", \"office\"],\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n}\n\nregistry.update_device_system_info(\"windowsagent\", system_info)\n\n# Verify update\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"OS: {profile.os}\") # \"windows\"\nprint(f\"CPU Cores: {profile.metadata['system_info']['cpu_count']}\") # 16\nprint(f\"Memory: {profile.metadata['system_info']['memory_total_gb']} GB\") # 32.0\nprint(f\"Capabilities: {profile.capabilities}\")\n# [\"web_browsing\", \"office\", \"gui\", \"cli\", \"browser\", \"file_system\"]\n```\n\n### Heartbeat Tracking\n\n```python\nfrom datetime import datetime, timezone\n\n# Update heartbeat\nregistry.update_heartbeat(\"windowsagent\")\n\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Last Heartbeat: {profile.last_heartbeat}\")\n# 2025-11-06 10:45:30.123456+00:00\n```\n\n### Connection Retry Management\n\n```python\n# Increment connection attempts\nattempts = registry.increment_connection_attempts(\"windowsagent\")\nprint(f\"Attempts: {attempts}/{profile.max_retries}\")\n\n# Reset after successful connection\nregistry.reset_connection_attempts(\"windowsagent\")\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Attempts: {profile.connection_attempts}\") # 0\n```\n\n---\n\n## 🎯 Usage Patterns\n\nThe following patterns demonstrate how AgentProfile is used for intelligent task routing and device management. For more details on task constellation concepts, see [Constellation Overview](../constellation/overview.md).\n\n### Task Assignment Decision\n\n```python\ndef can_assign_task(profile: AgentProfile, required_capabilities: List[str]) -> bool:\n \"\"\"\n Check if device can handle a task based on its profile.\n \"\"\"\n # Check if device is available\n if profile.status != DeviceStatus.IDLE:\n return False\n \n # Check if all required capabilities are supported\n device_caps = set(profile.capabilities)\n required_caps = set(required_capabilities)\n \n if not required_caps.issubset(device_caps):\n return False\n \n # Optional: Check system resources\n system_info = profile.metadata.get(\"system_info\", {})\n if system_info.get(\"memory_total_gb\", 0) < 8: # Require at least 8GB\n return False\n \n return True\n\n# Usage\nprofile = registry.get_device(\"windowsagent\")\nif can_assign_task(profile, [\"browser\", \"gui\"]):\n await manager.assign_task_to_device(\n task_id=\"task_web_001\",\n device_id=\"windowsagent\",\n task_description=\"Navigate to website and extract data\",\n task_data={\"url\": \"https://example.com\"}\n )\n```\n\n### Device Selection\n\n```python\ndef select_best_device(\n all_devices: Dict[str, AgentProfile],\n required_capabilities: List[str],\n prefer_high_performance: bool = True\n) -> Optional[str]:\n \"\"\"\n Select the best available device for a task.\n \"\"\"\n candidates = []\n \n for device_id, profile in all_devices.items():\n # Must be idle\n if profile.status != DeviceStatus.IDLE:\n continue\n \n # Must have required capabilities\n device_caps = set(profile.capabilities)\n if not set(required_capabilities).issubset(device_caps):\n continue\n \n # Calculate score\n score = 0\n if profile.metadata.get(\"performance\") == \"very_high\":\n score += 10\n elif profile.metadata.get(\"performance\") == \"high\":\n score += 5\n \n # Prefer devices with more memory\n system_info = profile.metadata.get(\"system_info\", {})\n score += system_info.get(\"memory_total_gb\", 0) / 10\n \n candidates.append((device_id, score))\n \n if not candidates:\n return None\n \n # Sort by score (descending)\n candidates.sort(key=lambda x: x[1], reverse=True)\n return candidates[0][0]\n\n# Usage\nall_devices = registry.get_all_devices(connected=True)\nbest_device = select_best_device(\n all_devices,\n required_capabilities=[\"gpu_computation\", \"model_training\"],\n prefer_high_performance=True\n)\nprint(f\"Selected device: {best_device}\")\n```\n\n### Health Monitoring\n\n```python\nfrom datetime import datetime, timezone, timedelta\n\ndef check_device_health(profile: AgentProfile) -> Dict[str, Any]:\n \"\"\"\n Check device health based on profile data.\n \"\"\"\n health = {\n \"device_id\": profile.device_id,\n \"healthy\": True,\n \"warnings\": [],\n \"errors\": []\n }\n \n # Check heartbeat freshness\n if profile.last_heartbeat:\n age = datetime.now(timezone.utc) - profile.last_heartbeat\n if age > timedelta(minutes=5):\n health[\"warnings\"].append(\n f\"No heartbeat for {age.total_seconds():.0f} seconds\"\n )\n if age > timedelta(minutes=10):\n health[\"errors\"].append(\"Heartbeat timeout\")\n health[\"healthy\"] = False\n \n # Check connection attempts\n if profile.connection_attempts > profile.max_retries / 2:\n health[\"warnings\"].append(\n f\"High connection attempts: {profile.connection_attempts}/{profile.max_retries}\"\n )\n \n # Check if device is stuck in BUSY state\n if profile.status == DeviceStatus.BUSY and profile.current_task_id:\n # Would need to check task age here\n health[\"warnings\"].append(f\"Device busy with task {profile.current_task_id}\")\n \n return health\n\n# Usage\nprofile = registry.get_device(\"windowsagent\")\nhealth = check_device_health(profile)\nprint(f\"Health: {health['healthy']}\")\nprint(f\"Warnings: {health['warnings']}\")\nprint(f\"Errors: {health['errors']}\")\n```\n\n---\n\n## 🔗 Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture and process |\n| **Registration Flow** | [Registration Flow](./registration_flow.md) | Step-by-step registration process |\n| **Device Registry** | [Device Registry](./device_registry.md) | Registry component implementation |\n| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML configuration reference |\n| **Device Info** | [Device Info Provider](../../client/device_info.md) | Telemetry collection details |\n| **AIP Protocol** | [AIP Overview](../../aip/overview.md) | Agent Interaction Protocol |\n| **Constellation System** | [Constellation Overview](../constellation/overview.md) | Multi-device coordination |\n| **WebSocket Client** | [Client AIP Integration](../client/aip_integration.md) | Client-side implementation |\n\n---\n\n## 💡 Best Practices\n\n### 1. Meaningful Capabilities\n\n```python\n# ✅ Good: Specific, actionable capabilities\ncapabilities = [\"web_browsing\", \"office_excel\", \"file_management\", \"email_sending\"]\n\n# ❌ Bad: Vague capabilities\ncapabilities = [\"desktop\", \"general\"]\n```\n\n### 2. Rich Metadata\n\n```python\n# ✅ Good: Comprehensive metadata for smart routing\nmetadata = {\n \"location\": \"datacenter_us_west\",\n \"performance\": \"high\",\n \"description\": \"GPU workstation for ML training\",\n \"tags\": [\"production\", \"gpu\", \"ml\"],\n \"operation_engineer_email\": \"ml-team@example.com\"\n}\n```\n\n### 3. Monitor Heartbeats\n\n```python\n# Regularly check heartbeat freshness\nif profile.last_heartbeat:\n age = datetime.now(timezone.utc) - profile.last_heartbeat\n if age > timedelta(minutes=5):\n logger.warning(f\"Device {profile.device_id} heartbeat stale\")\n```\n\n### 4. Use System Info for Resource-Aware Routing\n\n```python\n# Check if device has enough resources\nsystem_info = profile.metadata.get(\"system_info\", {})\nif system_info.get(\"memory_total_gb\", 0) >= 16:\n # Assign memory-intensive task\n pass\n```\n\n---\n\n## 🚀 Next Steps\n\n1. **Learn Registration Process**: Read [Registration Flow](./registration_flow.md)\n2. **Configure Devices**: See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)\n3. **Understand DeviceRegistry**: Check [Device Registry](./device_registry.md)\n4. **Study Telemetry**: Read [Device Info Provider](../../client/device_info.md)\n\n---\n\n## 📚 Source Code References\n\n- **AgentProfile Definition**: `galaxy/client/components/types.py`\n- **DeviceRegistry**: `galaxy/client/components/device_registry.py`\n- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`\n- **DeviceInfoProvider**: `ufo/client/device_info_provider.py`\n"
},
{
"path": "documents/docs/galaxy/agent_registration/device_registry.md",
"content": "# 🗄️ DeviceRegistry - Device Data Management\n\n## 📋 Overview\n\nThe **DeviceRegistry** is a focused component that manages device registration and information storage, providing a clean separation of concerns in the constellation architecture. It is responsible for **device data management only** - storing, retrieving, and updating AgentProfile instances without handling networking, task execution, or protocol logic.\n\n> For details on how devices connect and register using the AIP protocol, see [Registration Flow](./registration_flow.md).\n\n**Core Responsibilities:**\n\n| Responsibility | Description |\n|----------------|-------------|\n| **Registration** | Create and store AgentProfile instances |\n| **Status Tracking** | Update device connection and operational states |\n| **Metadata Management** | Store and update device metadata from all sources |\n| **Information Retrieval** | Provide device information to other components |\n| **Task State Tracking** | Track which device is executing which task |\n\n**Delegation to Other Components:**\n\n- Network communication → [`WebSocketConnectionManager`](../client/components.md#websocketconnectionmanager-network-communication-handler)\n- Message processing → [`MessageProcessor`](../client/components.md#messageprocessor-message-router-and-handler)\n- Task execution → [`TaskQueueManager`](../client/components.md#taskqueuemanager-task-scheduling-and-queuing)\n- Heartbeat monitoring → [`HeartbeatManager`](../client/components.md#heartbeatmanager-connection-health-monitor)\n\n## 🏗️ Architecture\n\n### Class Structure\n\n```mermaid\nclassDiagram\n class DeviceRegistry {\n -Dict~str, AgentProfile~ _devices\n -Dict~str, Dict~ _device_capabilities\n -Logger logger\n \n +register_device(device_id, server_url, ...) AgentProfile\n +get_device(device_id) Optional~AgentProfile~\n +get_all_devices(connected) Dict~str, AgentProfile~\n +update_device_status(device_id, status)\n +set_device_busy(device_id, task_id)\n +set_device_idle(device_id)\n +is_device_busy(device_id) bool\n +get_current_task(device_id) Optional~str~\n +increment_connection_attempts(device_id) int\n +reset_connection_attempts(device_id)\n +update_heartbeat(device_id)\n +update_device_system_info(device_id, system_info) bool\n +get_device_system_info(device_id) Optional~Dict~\n +get_device_capabilities(device_id) Dict\n +get_connected_devices() List~str~\n +is_device_registered(device_id) bool\n +remove_device(device_id) bool\n }\n \n class AgentProfile {\n +str device_id\n +str server_url\n +Optional~str~ os\n +List~str~ capabilities\n +Dict metadata\n +DeviceStatus status\n +Optional~datetime~ last_heartbeat\n +int connection_attempts\n +int max_retries\n +Optional~str~ current_task_id\n }\n \n class DeviceStatus {\n <>\n DISCONNECTED\n CONNECTING\n CONNECTED\n REGISTERING\n IDLE\n BUSY\n FAILED\n }\n \n DeviceRegistry \"1\" --> \"*\" AgentProfile : manages\n AgentProfile --> DeviceStatus : has\n```\n\n### Internal Storage\n\n```python\nclass DeviceRegistry:\n def __init__(self):\n # Primary storage: device_id -> AgentProfile\n self._devices: Dict[str, AgentProfile] = {}\n \n # Secondary storage: device_id -> capabilities dict\n # (Legacy, mostly superseded by AgentProfile.capabilities)\n self._device_capabilities: Dict[str, Dict[str, Any]] = {}\n \n self.logger = logging.getLogger(f\"{__name__}.DeviceRegistry\")\n```\n\n**Storage Structure:**\n\n```python\n# Internal state example\n_devices = {\n \"windowsagent\": AgentProfile(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"gui\", \"browser\", \"office\"],\n metadata={...},\n status=DeviceStatus.IDLE,\n ...\n ),\n \"linux_server_01\": AgentProfile(\n device_id=\"linux_server_01\",\n server_url=\"ws://10.0.0.50:5001/ws\",\n os=\"linux\",\n capabilities=[\"cli\", \"server\"],\n metadata={...},\n status=DeviceStatus.BUSY,\n current_task_id=\"task_123\"\n )\n}\n```\n\n---\n\n## 🔧 Core Operations\n\n### 1. Device Registration\n\n#### Method: `register_device()`\n\n```python\ndef register_device(\n self,\n device_id: str,\n server_url: str,\n os: Optional[str] = None,\n capabilities: Optional[List[str]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n max_retries: int = 5,\n) -> AgentProfile:\n \"\"\"\n Register a new device.\n\n :param device_id: Unique device identifier\n :param server_url: UFO WebSocket server URL\n :param os: Operating system type\n :param capabilities: Device capabilities\n :param metadata: Additional metadata\n :param max_retries: Maximum connection retry attempts\n :return: Created AgentProfile object\n \"\"\"\n```\n\n**Process:**\n\n```mermaid\nsequenceDiagram\n participant Caller\n participant Registry as DeviceRegistry\n participant Profile as AgentProfile\n \n Caller->>Registry: register_device(device_id, server_url, ...)\n \n Registry->>Profile: Create AgentProfile\n Note over Profile: device_id, server_url os, capabilities metadata, max_retries status=DISCONNECTED\n \n Registry->>Registry: Store in _devices[device_id]\n Registry->>Registry: Log registration\n \n Registry-->>Caller: Return AgentProfile\n```\n\n**Example:**\n\n```python\nregistry = DeviceRegistry()\n\n# Register device\nprofile = registry.register_device(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\", \"office_applications\"],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\"\n },\n max_retries=5\n)\n\nprint(f\"Registered: {profile.device_id}\")\nprint(f\"Status: {profile.status.value}\") # \"disconnected\"\n```\n\n> **Note:** The `register_device()` method will overwrite an existing device if the same `device_id` is used. Consider adding validation if duplicate prevention is needed.\n\n### 2. Device Retrieval\n\n#### Method: `get_device()`\n\n```python\ndef get_device(self, device_id: str) -> Optional[AgentProfile]:\n \"\"\"Get device information by ID\"\"\"\n return self._devices.get(device_id)\n```\n\n**Example:**\n\n```python\nprofile = registry.get_device(\"windowsagent\")\n\nif profile:\n print(f\"Device: {profile.device_id}\")\n print(f\"Status: {profile.status.value}\")\n print(f\"Capabilities: {profile.capabilities}\")\nelse:\n print(\"Device not found\")\n```\n\n#### Method: `get_all_devices()`\n\n```python\ndef get_all_devices(self, connected: bool = False) -> Dict[str, AgentProfile]:\n \"\"\"\n Get all registered devices\n :param connected: If True, return only connected devices\n :return: Dictionary of device_id to AgentProfile\n \"\"\"\n```\n\n**Example:**\n\n```python\n# Get all devices\nall_devices = registry.get_all_devices(connected=False)\nprint(f\"Total devices: {len(all_devices)}\")\n\n# Get only connected devices\nconnected_devices = registry.get_all_devices(connected=True)\nprint(f\"Connected devices: {len(connected_devices)}\")\n\nfor device_id, profile in connected_devices.items():\n print(f\" - {device_id}: {profile.status.value}\")\n```\n\n**Connected Device Filter:**\n\n```python\n# Implementation detail\nif connected:\n return {\n device_id: device_info\n for device_id, device_info in self._devices.items()\n if device_info.status in [\n DeviceStatus.CONNECTED,\n DeviceStatus.IDLE,\n DeviceStatus.BUSY\n ]\n }\n```\n\n#### Method: `get_connected_devices()`\n\n```python\ndef get_connected_devices(self) -> List[str]:\n \"\"\"Get list of connected device IDs\"\"\"\n return [\n device_id\n for device_id, device_info in self._devices.items()\n if device_info.status == DeviceStatus.CONNECTED\n ]\n```\n\n**Example:**\n\n```python\nconnected = registry.get_connected_devices()\nprint(f\"Connected: {connected}\")\n# ['windowsagent', 'linux_server_01']\n```\n\n---\n\n### 3. Status Management\n\n#### Method: `update_device_status()`\n\n```python\ndef update_device_status(self, device_id: str, status: DeviceStatus) -> None:\n \"\"\"Update device connection status\"\"\"\n if device_id in self._devices:\n self._devices[device_id].status = status\n```\n\n**Example:**\n\n```python\n# Update status progression\nregistry.update_device_status(\"windowsagent\", DeviceStatus.CONNECTING)\nregistry.update_device_status(\"windowsagent\", DeviceStatus.CONNECTED)\nregistry.update_device_status(\"windowsagent\", DeviceStatus.IDLE)\n```\n\n**Status Lifecycle:**\n\n```mermaid\nstateDiagram-v2\n [*] --> DISCONNECTED: register_device()\n \n DISCONNECTED --> CONNECTING: update_device_status()\n CONNECTING --> CONNECTED: update_device_status()\n CONNECTING --> FAILED: update_device_status()\n \n CONNECTED --> REGISTERING: update_device_status()\n REGISTERING --> IDLE: update_device_status()\n REGISTERING --> FAILED: update_device_status()\n \n IDLE --> BUSY: set_device_busy()\n BUSY --> IDLE: set_device_idle()\n \n IDLE --> DISCONNECTED: update_device_status()\n BUSY --> DISCONNECTED: update_device_status()\n \n FAILED --> CONNECTING: update_device_status()\n \n DISCONNECTED --> [*]: remove_device()\n```\n\n---\n\n### 4. Task State Management\n\n#### Method: `set_device_busy()`\n\n```python\ndef set_device_busy(self, device_id: str, task_id: str) -> None:\n \"\"\"\n Set device to BUSY status and track current task.\n\n :param device_id: Device ID\n :param task_id: Task ID being executed\n \"\"\"\n if device_id in self._devices:\n self._devices[device_id].status = DeviceStatus.BUSY\n self._devices[device_id].current_task_id = task_id\n self.logger.info(f\"🔄 Device {device_id} set to BUSY (task: {task_id})\")\n```\n\n**Example:**\n\n```python\n# Assign task to device\nregistry.set_device_busy(\"windowsagent\", task_id=\"task_12345\")\n\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Status: {profile.status.value}\") # \"busy\"\nprint(f\"Current Task: {profile.current_task_id}\") # \"task_12345\"\n```\n\n#### Method: `set_device_idle()`\n\n```python\ndef set_device_idle(self, device_id: str) -> None:\n \"\"\"\n Set device to IDLE status and clear current task.\n\n :param device_id: Device ID\n \"\"\"\n if device_id in self._devices:\n self._devices[device_id].status = DeviceStatus.IDLE\n self._devices[device_id].current_task_id = None\n self.logger.info(f\"✅ Device {device_id} set to IDLE\")\n```\n\n**Example:**\n\n```python\n# Task completes\nregistry.set_device_idle(\"windowsagent\")\n\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Status: {profile.status.value}\") # \"idle\"\nprint(f\"Current Task: {profile.current_task_id}\") # None\n```\n\n#### Method: `is_device_busy()`\n\n```python\ndef is_device_busy(self, device_id: str) -> bool:\n \"\"\"\n Check if device is currently busy.\n\n :param device_id: Device ID\n :return: True if device is busy\n \"\"\"\n if device_id in self._devices:\n return self._devices[device_id].status == DeviceStatus.BUSY\n return False\n```\n\n**Example:**\n\n```python\nif registry.is_device_busy(\"windowsagent\"):\n print(\"Device is busy, task will be queued\")\nelse:\n print(\"Device is available\")\n```\n\n#### Method: `get_current_task()`\n\n```python\ndef get_current_task(self, device_id: str) -> Optional[str]:\n \"\"\"\n Get the current task ID being executed on device.\n\n :param device_id: Device ID\n :return: Current task ID or None\n \"\"\"\n if device_id in self._devices:\n return self._devices[device_id].current_task_id\n return None\n```\n\n**Example:**\n\n```python\ntask_id = registry.get_current_task(\"windowsagent\")\nif task_id:\n print(f\"Device executing: {task_id}\")\nelse:\n print(\"Device idle\")\n```\n\n---\n\n### 5. Connection Management\n\n#### Method: `increment_connection_attempts()`\n\n```python\ndef increment_connection_attempts(self, device_id: str) -> int:\n \"\"\"Increment connection attempts counter\"\"\"\n if device_id in self._devices:\n self._devices[device_id].connection_attempts += 1\n return self._devices[device_id].connection_attempts\n return 0\n```\n\n**Example:**\n\n```python\nattempts = registry.increment_connection_attempts(\"windowsagent\")\nprint(f\"Attempts: {attempts}\")\n\nprofile = registry.get_device(\"windowsagent\")\nif profile.connection_attempts >= profile.max_retries:\n print(\"Max retries reached, giving up\")\n```\n\n#### Method: `reset_connection_attempts()`\n\n```python\ndef reset_connection_attempts(self, device_id: str) -> None:\n \"\"\"Reset connection attempts counter to 0\"\"\"\n if device_id in self._devices:\n self._devices[device_id].connection_attempts = 0\n self.logger.info(f\"🔄 Reset connection attempts for device {device_id}\")\n```\n\n**Example:**\n\n```python\n# After successful connection\nregistry.reset_connection_attempts(\"windowsagent\")\n\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Attempts: {profile.connection_attempts}\") # 0\n```\n\n---\n\n### 6. Heartbeat Tracking\n\n#### Method: `update_heartbeat()`\n\n```python\ndef update_heartbeat(self, device_id: str) -> None:\n \"\"\"Update last heartbeat timestamp\"\"\"\n if device_id in self._devices:\n self._devices[device_id].last_heartbeat = datetime.now(timezone.utc)\n```\n\n**Example:**\n\n```python\nfrom datetime import datetime, timezone, timedelta\n\n# Update heartbeat\nregistry.update_heartbeat(\"windowsagent\")\n\nprofile = registry.get_device(\"windowsagent\")\nprint(f\"Last heartbeat: {profile.last_heartbeat}\")\n\n# Check heartbeat freshness\nage = datetime.now(timezone.utc) - profile.last_heartbeat\nif age > timedelta(minutes=5):\n print(\"⚠️ Heartbeat stale!\")\n```\n\n---\n\n### 7. System Information Management\n\n#### Method: `update_device_system_info()`\n\n```python\ndef update_device_system_info(\n self, device_id: str, system_info: Dict[str, Any]\n) -> bool:\n \"\"\"\n Update AgentProfile with system information retrieved from server.\n\n This method updates the device's OS, capabilities, and metadata with\n the system information that was automatically collected by the device\n and stored on the server.\n\n :param device_id: Device ID\n :param system_info: System information dictionary from server\n :return: True if update successful, False if device not found\n \"\"\"\n```\n\n> **Note:** System information is collected from the device agent and retrieved via the server. See [Client Connection Manager](../../server/client_connection_manager.md) for server-side information management.\n\n**Process:**\n\n```mermaid\nsequenceDiagram\n participant Caller\n participant Registry as DeviceRegistry\n participant Profile as AgentProfile\n \n Caller->>Registry: update_device_system_info(device_id, system_info)\n \n Registry->>Profile: Get device\n \n alt Device exists\n Registry->>Profile: Update os = system_info[\"platform\"]\n Registry->>Profile: Merge supported_features into capabilities\n Registry->>Profile: Add system_info to metadata\n Registry->>Profile: Add custom_metadata if present\n Registry->>Registry: Log update\n Registry-->>Caller: True\n else Device not found\n Registry->>Registry: Log warning\n Registry-->>Caller: False\n end\n```\n\n**Implementation:**\n\n```python\ndevice_info = self.get_device(device_id)\nif not device_info:\n self.logger.warning(f\"Cannot update system info: device {device_id} not found\")\n return False\n\n# 1. Update OS information\nif \"platform\" in system_info:\n device_info.os = system_info[\"platform\"]\n\n# 2. Merge capabilities with supported features (avoid duplicates)\nif \"supported_features\" in system_info:\n features = system_info[\"supported_features\"]\n existing_caps = set(device_info.capabilities)\n new_caps = existing_caps.union(set(features))\n device_info.capabilities = list(new_caps)\n\n# 3. Update metadata with system information\ndevice_info.metadata.update({\n \"system_info\": {\n \"platform\": system_info.get(\"platform\"),\n \"os_version\": system_info.get(\"os_version\"),\n \"cpu_count\": system_info.get(\"cpu_count\"),\n \"memory_total_gb\": system_info.get(\"memory_total_gb\"),\n \"hostname\": system_info.get(\"hostname\"),\n \"ip_address\": system_info.get(\"ip_address\"),\n \"platform_type\": system_info.get(\"platform_type\"),\n \"schema_version\": system_info.get(\"schema_version\"),\n }\n})\n\n# 4. Add custom metadata if present\nif \"custom_metadata\" in system_info:\n device_info.metadata[\"custom_metadata\"] = system_info[\"custom_metadata\"]\n\n# 5. Add tags if present\nif \"tags\" in system_info:\n device_info.metadata[\"tags\"] = system_info[\"tags\"]\n\nreturn True\n```\n\n**Example:**\n\n```python\nsystem_info = {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"supported_features\": [\"gui\", \"cli\", \"browser\", \"file_system\", \"office\"],\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n}\n\nsuccess = registry.update_device_system_info(\"windowsagent\", system_info)\n\nif success:\n profile = registry.get_device(\"windowsagent\")\n print(f\"OS: {profile.os}\") # \"windows\"\n print(f\"CPU: {profile.metadata['system_info']['cpu_count']}\") # 16\n print(f\"Memory: {profile.metadata['system_info']['memory_total_gb']} GB\") # 32.0\n```\n\n#### Method: `get_device_system_info()`\n\n```python\ndef get_device_system_info(self, device_id: str) -> Optional[Dict[str, Any]]:\n \"\"\"\n Get device system information (hardware, OS, features).\n\n :param device_id: Device ID\n :return: System information dictionary or None if not available\n \"\"\"\n device_info = self.get_device(device_id)\n if not device_info:\n return None\n \n return device_info.metadata.get(\"system_info\")\n```\n\n**Example:**\n\n```python\nsystem_info = registry.get_device_system_info(\"windowsagent\")\n\nif system_info:\n print(f\"Platform: {system_info['platform']}\")\n print(f\"CPU Cores: {system_info['cpu_count']}\")\n print(f\"Memory: {system_info['memory_total_gb']} GB\")\n print(f\"Hostname: {system_info['hostname']}\")\nelse:\n print(\"System info not available\")\n```\n\n---\n\n### 8. Capabilities Management\n\n#### Method: `set_device_capabilities()`\n\n```python\ndef set_device_capabilities(\n self, device_id: str, capabilities: Dict[str, Any]\n) -> None:\n \"\"\"Store device capabilities information\"\"\"\n self._device_capabilities[device_id] = capabilities\n\n # Also update device info with capabilities\n if device_id in self._devices:\n device_info = self._devices[device_id]\n if \"capabilities\" in capabilities:\n device_info.capabilities.extend(capabilities[\"capabilities\"])\n if \"metadata\" in capabilities:\n device_info.metadata.update(capabilities[\"metadata\"])\n```\n\n> **Note:** This method is primarily for backwards compatibility. Modern code should use `update_device_system_info()` instead.\n\n#### Method: `get_device_capabilities()`\n\n```python\ndef get_device_capabilities(self, device_id: str) -> Dict[str, Any]:\n \"\"\"Get device capabilities\"\"\"\n return self._device_capabilities.get(device_id, {})\n```\n\n---\n\n### 9. Utility Methods\n\n#### Method: `is_device_registered()`\n\n```python\ndef is_device_registered(self, device_id: str) -> bool:\n \"\"\"Check if device is registered\"\"\"\n return device_id in self._devices\n```\n\n**Example:**\n\n```python\nif registry.is_device_registered(\"windowsagent\"):\n print(\"Device exists\")\nelse:\n print(\"Device not registered\")\n```\n\n#### Method: `remove_device()`\n\n```python\ndef remove_device(self, device_id: str) -> bool:\n \"\"\"Remove a device from registry\"\"\"\n if device_id in self._devices:\n del self._devices[device_id]\n self._device_capabilities.pop(device_id, None)\n return True\n return False\n```\n\n**Example:**\n\n```python\nsuccess = registry.remove_device(\"windowsagent\")\nif success:\n print(\"Device removed\")\nelse:\n print(\"Device not found\")\n```\n\n---\n\n## 💡 Usage Patterns\n\n### Pattern 1: Complete Registration Flow\n\n```python\nfrom galaxy.client.components import DeviceRegistry, DeviceStatus\n\nregistry = DeviceRegistry()\n\n# 1. Register device\nprofile = registry.register_device(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\"],\n metadata={\"location\": \"office\"},\n max_retries=5\n)\n\n# 2. Update status through connection process\nregistry.update_device_status(\"windowsagent\", DeviceStatus.CONNECTING)\nregistry.increment_connection_attempts(\"windowsagent\")\nregistry.update_device_status(\"windowsagent\", DeviceStatus.CONNECTED)\nregistry.reset_connection_attempts(\"windowsagent\")\n\n# 3. Update with system info\nsystem_info = {\n \"platform\": \"windows\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"supported_features\": [\"gui\", \"cli\", \"browser\"]\n}\nregistry.update_device_system_info(\"windowsagent\", system_info)\n\n# 4. Set to IDLE (ready for tasks)\nregistry.set_device_idle(\"windowsagent\")\n\n# 5. Update heartbeat\nregistry.update_heartbeat(\"windowsagent\")\n```\n\n### Pattern 2: Task Assignment\n\n```python\n# Check if device can accept task\nif not registry.is_device_busy(\"windowsagent\"):\n # Assign task\n registry.set_device_busy(\"windowsagent\", task_id=\"task_123\")\n \n # ... execute task ...\n \n # Task complete\n registry.set_device_idle(\"windowsagent\")\nelse:\n print(\"Device busy, task queued\")\n```\n\n### Pattern 3: Device Selection\n\n```python\ndef find_available_device_with_capability(\n registry: DeviceRegistry,\n required_capability: str\n) -> Optional[str]:\n \"\"\"Find an idle device with specific capability.\"\"\"\n \n all_devices = registry.get_all_devices(connected=True)\n \n for device_id, profile in all_devices.items():\n # Check if idle\n if profile.status != DeviceStatus.IDLE:\n continue\n \n # Check capability\n if required_capability in profile.capabilities:\n return device_id\n \n return None\n\n# Usage\ndevice_id = find_available_device_with_capability(registry, \"browser\")\nif device_id:\n print(f\"Selected: {device_id}\")\n```\n\n### Pattern 4: Health Monitoring\n\n```python\nfrom datetime import datetime, timezone, timedelta\n\ndef check_all_devices_health(registry: DeviceRegistry):\n \"\"\"Check health of all registered devices.\"\"\"\n \n all_devices = registry.get_all_devices()\n \n for device_id, profile in all_devices.items():\n print(f\"\\n{device_id}:\")\n print(f\" Status: {profile.status.value}\")\n \n # Check heartbeat\n if profile.last_heartbeat:\n age = datetime.now(timezone.utc) - profile.last_heartbeat\n print(f\" Heartbeat age: {age.total_seconds():.0f}s\")\n \n if age > timedelta(minutes=5):\n print(f\" ⚠️ WARNING: Stale heartbeat!\")\n else:\n print(f\" ⚠️ WARNING: No heartbeat recorded\")\n \n # Check connection attempts\n if profile.connection_attempts > 0:\n print(f\" Connection attempts: {profile.connection_attempts}/{profile.max_retries}\")\n \n # Check task status\n if profile.current_task_id:\n print(f\" Current task: {profile.current_task_id}\")\n```\n\n---\n\n## 🔗 Integration with Other Components\n\nDeviceRegistry is used internally by other components in the constellation system. See [Components Overview](../client/components.md) for details on the component architecture.\n\n### With ConstellationDeviceManager\n\n```python\n# ConstellationDeviceManager uses DeviceRegistry internally\n\nclass ConstellationDeviceManager:\n def __init__(self, ...):\n self.device_registry = DeviceRegistry() # Internal registry\n \n async def register_device(self, ...):\n # Delegate to registry\n self.device_registry.register_device(...)\n \n def get_device_info(self, device_id: str):\n # Delegate to registry\n return self.device_registry.get_device(device_id)\n```\n\n### With MessageProcessor\n\n```python\n# MessageProcessor updates registry when messages arrive\n\nclass MessageProcessor:\n def __init__(self, device_registry: DeviceRegistry, ...):\n self.device_registry = device_registry\n \n async def handle_heartbeat(self, device_id: str):\n # Update heartbeat in registry\n self.device_registry.update_heartbeat(device_id)\n```\n\n### With TaskQueueManager\n\n```python\n# TaskQueueManager checks device status via registry\n\nclass TaskQueueManager:\n def can_assign_task(self, device_id: str) -> bool:\n # Check if device is busy\n return not self.device_registry.is_device_busy(device_id)\n```\n\n---\n\n## 🔗 Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture |\n| **AgentProfile** | [AgentProfile](./agent_profile.md) | Profile structure details |\n| **Registration Flow** | [Registration Flow](./registration_flow.md) | Step-by-step registration |\n| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML config reference |\n| **Components** | [Client Components](../client/components.md) | Component architecture |\n\n---\n\n## 💡 Best Practices\n\n**1. Always Check Device Exists**\n\n```python\nprofile = registry.get_device(device_id)\nif not profile:\n logger.error(f\"Device {device_id} not found\")\n return\n```\n\n**2. Use Defensive Copies for Lists/Dicts**\n\n```python\n# Registry already creates copies, but be aware\ncapabilities = [\"web\", \"office\"]\nregistry.register_device(..., capabilities=capabilities)\n# Modifying original list won't affect registry\ncapabilities.append(\"new\") # Safe\n```\n\n**3. Monitor Heartbeats Regularly**\n\n```python\n# Periodic check\nfor device_id in registry.get_all_devices():\n profile = registry.get_device(device_id)\n if profile.last_heartbeat:\n age = datetime.now(timezone.utc) - profile.last_heartbeat\n if age > timedelta(minutes=5):\n logger.warning(f\"Stale heartbeat: {device_id}\")\n```\n\n**4. Clear Task State After Completion**\n\n```python\n# Always set to IDLE after task completes\nregistry.set_device_idle(device_id)\n# This automatically clears current_task_id\n```\n\n---\n\n## 🚀 Next Steps\n\n1. **Understand AgentProfile**: Read [AgentProfile Documentation](./agent_profile.md)\n2. **Learn Configuration**: See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)\n3. **Study Registration**: Check [Registration Flow](./registration_flow.md)\n4. **Explore Components**: See ConstellationDeviceManager implementation\n\n---\n\n## 📚 Source Code Reference\n\n- **DeviceRegistry**: `galaxy/client/components/device_registry.py`\n- **AgentProfile**: `galaxy/client/components/types.py`\n- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`\n"
},
{
"path": "documents/docs/galaxy/agent_registration/overview.md",
"content": "# 🌟 Agent Registration & Profiling - Overview\n\n**Agent Registration** is the cornerstone of the AIP (Agent Interaction Protocol) initialization process. It enables dynamic discovery, capability advertisement, and intelligent task allocation across distributed constellation agents.\n\n---\n\n## 📋 Introduction\n\n\n*An overview of the Constellation Agent architecture showing the registration and profiling system.*\n\nAt the core of AIP's initialization process is the **ConstellationClient** (implemented as `ConstellationDeviceManager`), which maintains a global registry of active agents. Any device agent service that exposes a WebSocket endpoint and implements the AIP task dispatch and result-return protocol can be seamlessly integrated into UFO, providing remarkable **extensibility**.\n\nThe multi-source profiling pipeline enables **transparent capability discovery** and **safe adaptation** to environmental drift without direct administrator intervention.\n\nFor a complete understanding of the constellation system, see:\n\n- [Constellation Overview](../constellation/overview.md) - Multi-device coordination architecture\n- [Constellation Agent Overview](../constellation_agent/overview.md) - Agent behavior and patterns\n- [AIP Protocol Overview](../../aip/overview.md) - Message protocol details\n\n---\n\n## 🎯 Core Concepts\n\n### Agent Registry\n\nThe agent registry is a centralized store that tracks all active constellation agents. Each registered agent is represented by an **AgentProfile** that consolidates comprehensive information about the agent's capabilities, system resources, and operational status.\n\n| Component | Responsibility | Location |\n|-----------|---------------|----------|\n| **ConstellationDeviceManager** | Central coordinator for device management | `galaxy/client/device_manager.py` |\n| **DeviceRegistry** | Device registration and information storage | `galaxy/client/components/device_registry.py` |\n| **AgentProfile** | Multi-source agent metadata representation | `galaxy/client/components/types.py` |\n| **ClientConnectionManager** | Server-side client connection tracking | `ufo/server/services/client_connection_manager.py` |\n\n### Multi-Source Profiling\n\nEach **AgentProfile** consolidates information from **three distinct sources**, creating a comprehensive and dynamically updated view of each agent.\n\n```mermaid\ngraph TB\n subgraph Sources\n UC[User Config devices.yaml]\n SM[AIP Registration Service Manifest]\n CT[Device Telemetry DeviceInfoProvider]\n end\n \n UC -->|device_id, capabilities metadata| AP[AgentProfile]\n SM -->|platform, client_type registration_time| AP\n CT -->|system_info supported_features| AP\n \n AP --> CR[ConstellationDeviceManager]\n CR --> TA[Intelligent Task Routing]\n \n style UC fill:#e1f5ff\n style SM fill:#fff4e1\n style CT fill:#e8f5e9\n style AP fill:#f3e5f5\n```\n\n**Source Details:**\n\n| Source | Provider | Information Type | Update Frequency |\n|--------|----------|------------------|------------------|\n| **1. User Configuration** | Administrator (devices.yaml + constellation.yaml) | Endpoint identity, user preferences, capabilities | Static (config load) |\n| **2. Service Manifest** | Device Agent Service (AIP) | Client type, platform, registration metadata | On registration |\n| **3. Client Telemetry** | Device Client (DeviceInfoProvider) | Hardware specs, OS info, network status | On connection + periodic updates |\n\n**Note:** While constellation.yaml contains runtime settings like heartbeat intervals, the device-specific configuration is in devices.yaml.\n\n---\n\n## 🔄 Registration Flow\n\n\n*Agent registration flow: multi-source AgentProfile construction and registration.*\n\n### Registration Process Overview\n\nThe registration process follows a well-defined sequence that ensures comprehensive profiling and validation:\n\n```mermaid\nsequenceDiagram\n participant Admin as Administrator\n participant CDM as ConstellationDeviceManager\n participant Server as UFO Server\n participant DIP as DeviceInfoProvider\n \n Note over Admin,DIP: Phase 1: User Configuration\n Admin->>CDM: register_device(device_id, capabilities)\n CDM->>CDM: Create AgentProfile\n \n Note over Admin,DIP: Phase 2: WebSocket Connection\n CDM->>Server: connect_device()\n Server-->>CDM: Connection established\n \n Note over Admin,DIP: Phase 3: Service Registration\n CDM->>Server: REGISTER message\n Server-->>CDM: Registration confirmed\n \n Note over Admin,DIP: Phase 4: Telemetry Collection\n CDM->>Server: request_device_info()\n Server->>DIP: collect_system_info()\n DIP-->>Server: system_info\n Server-->>CDM: system_info\n CDM->>CDM: Merge into AgentProfile\n \n Note over Admin,DIP: Phase 5: Ready for Tasks\n CDM->>CDM: Set device to IDLE\n```\n\n**Registration Phases:**\n\n| Phase | Description | Components Involved | Result |\n|-------|-------------|---------------------|--------|\n| **1. User Configuration** | Administrator registers device with endpoint and capabilities | ConstellationDeviceManager, DeviceRegistry | AgentProfile created with user-specified data |\n| **2. WebSocket Connection** | Establish persistent connection to device agent server | WebSocketConnectionManager | Active WebSocket channel |\n| **3. Service Registration** | AIP registration protocol exchange with capability advertisement | RegistrationProtocol, UFOWebSocketHandler | Client type and platform recorded |\n| **4. Telemetry Collection** | Retrieve runtime system information from device | DeviceInfoProvider, DeviceInfoProtocol | Hardware, OS, and feature data merged |\n| **5. Activation** | Set device to IDLE state, ready for task assignment | DeviceRegistry | Agent ready for constellation tasks |\n\nDevices can be registered with `auto_connect=True` to automatically establish connection, or `auto_connect=False` to require manual connection via `connect_device()`.\n\n---\n\n## 📊 AgentProfile Structure\n\nThe **AgentProfile** is the primary data structure representing a registered constellation agent. For detailed information about the AgentProfile and its lifecycle operations, see [Agent Profile Documentation](./agent_profile.md).\n\n### Core Fields\n\nThe **AgentProfile** is the primary data structure representing a registered constellation agent:\n\n```python\n@dataclass\nclass AgentProfile:\n \"\"\"Device information and capabilities\"\"\"\n \n # Identity\n device_id: str # Unique device identifier\n server_url: str # WebSocket endpoint URL\n \n # Platform & Capabilities\n os: Optional[str] = None # Operating system (windows, linux, darwin)\n capabilities: List[str] # Advertised capabilities/features\n metadata: Dict[str, Any] # Additional metadata\n \n # Operational Status\n status: DeviceStatus # Current connection/operational status\n last_heartbeat: Optional[datetime] # Last heartbeat timestamp\n \n # Connection Management\n connection_attempts: int = 0 # Connection retry counter\n max_retries: int = 5 # Maximum retry attempts\n \n # Task Execution\n current_task_id: Optional[str] = None # Currently executing task ID\n```\n\n**Field Categories:**\n\n| Category | Fields | Purpose |\n|----------|--------|---------|\n| **Identity** | `device_id`, `server_url` | Unique identification and endpoint location |\n| **Platform** | `os`, `capabilities`, `metadata` | System type and advertised features |\n| **Status** | `status`, `last_heartbeat` | Real-time operational state tracking |\n| **Resilience** | `connection_attempts`, `max_retries` | Connection retry management |\n| **Execution** | `current_task_id` | Task assignment tracking |\n\n### Metadata Structure\n\nThe `metadata` field is a flexible dictionary that aggregates information from all three sources:\n\n```python\nmetadata = {\n # From User Configuration (Source 1)\n \"location\": \"home_office\",\n \"performance\": \"high\",\n \"description\": \"Primary development laptop\",\n \"operation_engineer_email\": \"admin@example.com\",\n \n # From Service Manifest (Source 2)\n \"platform\": \"windows\",\n \"registration_time\": \"2025-11-06T10:30:00Z\",\n \n # From Client Telemetry (Source 3)\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n },\n \"custom_metadata\": {\n \"datacenter\": \"us-west-2\",\n \"tier\": \"production\"\n }\n}\n```\n\nFor a complete example, see the [Agent Profile Documentation](./agent_profile.md#example-profiles).\n\n---\n\n## 🔄 Agent Lifecycle States\n\n\n*Lifecycle state transitions of the Constellation Agent.*\n\nThe agent lifecycle is managed through a state machine that tracks connection, registration, and task execution states. For more details on agent behavior and state management, see [Constellation Agent State Management](../constellation_agent/state.md).\n\n### State Definitions\n\n```python\nclass DeviceStatus(Enum):\n \"\"\"Device connection status\"\"\"\n DISCONNECTED = \"disconnected\" # Not connected to server\n CONNECTING = \"connecting\" # Attempting to establish connection\n CONNECTED = \"connected\" # Connected, initializing\n REGISTERING = \"registering\" # Performing registration handshake\n IDLE = \"idle\" # Connected and ready for tasks\n BUSY = \"busy\" # Executing a task\n FAILED = \"failed\" # Connection/execution failed\n```\n\n### State Transition Diagram\n\n```mermaid\nstateDiagram-v2\n [*] --> DISCONNECTED: Initial State\n \n DISCONNECTED --> CONNECTING: register_device() / connect_device()\n CONNECTING --> CONNECTED: WebSocket established\n CONNECTING --> FAILED: Connection error\n \n CONNECTED --> REGISTERING: Send REGISTER message\n REGISTERING --> IDLE: Registration confirmed + system info collected\n REGISTERING --> FAILED: Registration rejected\n \n IDLE --> BUSY: assign_task_to_device()\n BUSY --> IDLE: Task completed\n BUSY --> FAILED: Task failed / device disconnected\n \n FAILED --> CONNECTING: Automatic reconnection\n \n IDLE --> DISCONNECTED: disconnect_device() / connection lost\n BUSY --> DISCONNECTED: disconnect_device() / connection lost\n \n DISCONNECTED --> [*]: shutdown()\n```\n\n**Transition Events:**\n\n| From State | To State | Trigger | Action |\n|------------|----------|---------|--------|\n| DISCONNECTED | CONNECTING | `connect_device()` | Initiate WebSocket connection |\n| CONNECTING | CONNECTED | WebSocket handshake complete | Update status |\n| CONNECTED | REGISTERING | Send REGISTER message | AIP registration protocol |\n| REGISTERING | IDLE | Registration confirmed | Collect system info, ready for tasks |\n| IDLE | BUSY | `assign_task_to_device()` | Execute task |\n| BUSY | IDLE | Task completes | Clear current_task_id |\n| Any | DISCONNECTED | Connection lost | Cleanup, schedule reconnection |\n| FAILED | CONNECTING | Retry timer | Attempt reconnection (if under max_retries) |\n\n**Important:** When a device disconnects or enters FAILED state, the system automatically schedules reconnection attempts up to `max_retries` times with `reconnect_delay` interval.\n\n---\n\n## 🛠️ Key Components\n\n### 1. ConstellationDeviceManager\n\n**File:** `galaxy/client/device_manager.py`\n\nThe central coordinator for all device management operations in the constellation system.\n\n**Responsibilities:**\n\n- Device registration and lifecycle management\n- Connection establishment and monitoring\n- Task assignment and execution coordination\n- Automatic reconnection handling\n\n**Key Methods:**\n\n```python\nclass ConstellationDeviceManager:\n async def register_device(\n device_id: str,\n server_url: str,\n os: str,\n capabilities: List[str],\n metadata: Dict[str, Any],\n auto_connect: bool = True\n ) -> bool\n \n async def connect_device(device_id: str) -> bool\n \n async def assign_task_to_device(\n task_id: str,\n device_id: str,\n task_description: str,\n task_data: Dict[str, Any]\n ) -> ExecutionResult\n \n def get_device_info(device_id: str) -> Optional[AgentProfile]\n```\n\nSee [Device Registry Documentation](./device_registry.md) for DeviceRegistry details.\n\n### 2. DeviceRegistry\n\n**File:** `galaxy/client/components/device_registry.py`\n\nManages device registration and information storage with a focus on data management.\n\n**Responsibilities:**\n\n- Store and retrieve AgentProfile instances\n- Update device status and metadata\n- Track connection attempts and heartbeats\n- Merge multi-source information\n\n**Key Methods:**\n\n```python\nclass DeviceRegistry:\n def register_device(...) -> AgentProfile\n def update_device_status(device_id: str, status: DeviceStatus)\n def update_device_system_info(device_id: str, system_info: Dict)\n def set_device_busy(device_id: str, task_id: str)\n def set_device_idle(device_id: str)\n```\n\n### 3. RegistrationProtocol (AIP)\n\n**File:** `aip/protocol/registration.py`\n\nHandles AIP registration message exchange for both device and constellation clients.\n\n**Responsibilities:**\n\n- Device agent registration\n- Constellation client registration\n- Capability advertisement\n- Registration validation and confirmation\n\n**Key Methods:**\n\n```python\nclass RegistrationProtocol(AIPProtocol):\n async def register_as_device(\n device_id: str,\n metadata: Dict[str, Any],\n platform: str\n ) -> bool\n \n async def register_as_constellation(\n constellation_id: str,\n target_device: str,\n metadata: Dict[str, Any]\n ) -> bool\n \n async def send_registration_confirmation()\n async def send_registration_error(error: str)\n```\n\nSee [AIP Protocol Documentation](../../aip/overview.md) for protocol details.\n\n### 4. DeviceInfoProvider\n\n**File:** `ufo/client/device_info_provider.py`\n\nCollects device system information (telemetry source).\n\n**Responsibilities:**\n\n- Auto-detect platform, OS, and hardware\n- Collect CPU, memory, network information\n- Detect supported features based on platform\n- Provide DeviceSystemInfo structure\n\n**Key Methods:**\n\n```python\nclass DeviceInfoProvider:\n @staticmethod\n def collect_system_info(\n client_id: str,\n custom_metadata: Optional[Dict]\n ) -> DeviceSystemInfo\n```\n\nSee [Device Info Provider Documentation](../../client/device_info.md) for telemetry details.\n\n### 5. ClientConnectionManager (Server)\n\n**File:** `ufo/server/services/client_connection_manager.py`\n\nServer-side client connection tracking and management. For detailed information about the server-side implementation, see [Client Connection Manager Documentation](../../server/client_connection_manager.md).\n\n**Responsibilities:**\n\n- Track connected clients (devices and constellations)\n- Store device system information received during registration\n- Manage session-to-client mappings\n- Merge server configuration with client telemetry\n\n**Key Methods:**\n\n```python\nclass ClientConnectionManager:\n def add_client(\n client_id: str,\n platform: str,\n ws: WebSocket,\n client_type: ClientType,\n metadata: Dict\n )\n def get_device_system_info(device_id: str) -> Optional[Dict]\n```\n\n---\n\n## 📝 Configuration\n\nAgent registration uses two configuration files:\n\n**1. `config/galaxy/devices.yaml`** - Device definitions:\n- Device endpoints and identities\n- User-specified capabilities and metadata\n- Connection parameters (max retries, auto-connect)\n\n**2. `config/galaxy/constellation.yaml`** - Runtime settings:\n- Constellation identification and logging\n- Heartbeat interval and reconnection delay\n- Task concurrency and step limits\n\nSee [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) and [Galaxy Constellation Configuration](../../configuration/system/galaxy_constellation.md) for details.\n\n**Example Device Configuration (devices.yaml):**\n\n```yaml\n# Device Configuration - YAML Format\n# Runtime settings are configured in constellation.yaml\n\ndevices:\n - device_id: \"windowsagent\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities: [\"web_browsing\", \"office_applications\"]\n metadata:\n location: \"office_desktop\"\n performance: \"high\"\n max_retries: 5\n auto_connect: true\n```\n\nFor complete configuration schema, examples, and best practices, see:\n\n👉 **[Galaxy Devices Configuration Guide](../../configuration/system/galaxy_devices.md)**\n\n---\n\n## 🚀 Usage Example\n\n### Basic Registration\n\n```python\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Initialize manager\nmanager = ConstellationDeviceManager(\n task_name=\"test_constellation\",\n heartbeat_interval=30.0,\n reconnect_delay=5.0\n)\n\n# Register and connect device\nsuccess = await manager.register_device(\n device_id=\"windows_workstation\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"gui\", \"browser\", \"office\"],\n metadata={\n \"location\": \"home_office\",\n \"performance\": \"medium\"\n },\n auto_connect=True # Automatically connect after registration\n)\n\nif success:\n print(\"✅ Device registered and connected\")\n \n # Get device profile\n profile = manager.get_device_info(\"windows_workstation\")\n print(f\"Device: {profile.device_id}\")\n print(f\"Status: {profile.status.value}\")\n print(f\"Capabilities: {profile.capabilities}\")\n print(f\"System Info: {profile.metadata.get('system_info')}\")\n```\n\n### Task Assignment\n\n```python\n# Assign task to registered device\nresult = await manager.assign_task_to_device(\n task_id=\"task_001\",\n device_id=\"windows_workstation\",\n task_description=\"Open Excel and create a report\",\n task_data={\"file_path\": \"C:\\\\Reports\\\\monthly.xlsx\"},\n timeout=300.0\n)\n\nprint(f\"Task Status: {result.status}\")\nprint(f\"Result: {result.result}\")\n```\n\nFor more details on task assignment and execution, see:\n- [Registration Flow Documentation](./registration_flow.md) - Detailed examples\n- [Constellation Task Distribution](../constellation/overview.md) - Task routing strategies\n\n---\n\n## 🔗 Cross-References\n\n### Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Device Info Collection** | [Device Info Provider](../../client/device_info.md) | Client-side telemetry collection |\n| **AIP Protocol** | [AIP Overview](../../aip/overview.md) | Agent Interaction Protocol fundamentals |\n| **AIP Messages** | [AIP Messages](../../aip/messages.md) | Message structure and types |\n| **Agent Profile** | [Agent Profile](./agent_profile.md) | Detailed AgentProfile structure |\n| **Registration Flow** | [Registration Flow](./registration_flow.md) | Step-by-step registration process |\n| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML configuration reference |\n| **Device Registry** | [Device Registry](./device_registry.md) | Registry component details |\n| **Constellation System** | [Constellation Overview](../constellation/overview.md) | Multi-device coordination |\n| **Client Connection Manager** | [Server Connection Manager](../../server/client_connection_manager.md) | Server-side connection tracking |\n\n### Architecture Diagrams\n\n- **Constellation Agent Overview**: `documents/docs/img/constellation_agent.png`\n- **Agent Registration Flow**: `documents/docs/img/agent_registry.png`\n- **Agent Lifecycle States**: `documents/docs/img/agent_state.png`\n\n---\n\n## 💡 Key Benefits\n\nThe multi-source profiling approach provides several advantages:\n\n**1. Improved Task Allocation Accuracy**\n\n- Administrators specify high-level capabilities\n- Service manifests advertise supported tools\n- Telemetry provides real-time hardware status\n\n**2. Transparent Capability Discovery**\n\n- No manual system info entry required\n- Automatic feature detection based on platform\n- Dynamic updates without configuration changes\n\n**3. Safe Adaptation to Environmental Drift**\n\n- System changes (upgrades, hardware additions) automatically reflected\n- No administrator intervention needed for routine updates\n- Consistent metadata across distributed agents\n\n**4. Reliable Scheduling Decisions**\n\n- Fresh and accurate information for task routing\n- Hardware-aware task assignment (CPU/memory requirements)\n- Platform-specific capability matching\n\n---\n\n## 🎯 Next Steps\n\n1. **Understand AgentProfile in Detail**: Read [Agent Profile Documentation](./agent_profile.md)\n2. **Learn Registration Process**: Follow [Registration Flow](./registration_flow.md)\n3. **Configure Your Devices**: See [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)\n4. **Explore Device Registry**: Check [Device Registry](./device_registry.md)\n5. **Study AIP Protocol**: Read [AIP Documentation](../../aip/overview.md)\n\n---\n\n## 📚 Additional Resources\n\n- **Source Code**: `galaxy/client/device_manager.py`\n- **AIP Protocol**: `aip/protocol/registration.py`\n- **Device Info**: `ufo/client/device_info_provider.py`\n- **Configuration**: `config/galaxy/devices.yaml`\n\n**Best Practice:** Always configure devices with meaningful metadata and capabilities to enable intelligent task routing. The system will automatically enhance this information with telemetry data.\n"
},
{
"path": "documents/docs/galaxy/agent_registration/registration_flow.md",
"content": "# 🔄 Registration Flow - Complete Process Guide\n\n## 📋 Overview\n\nThe registration flow transforms a device configuration entry into a fully profiled, connected, and task-ready constellation agent through a coordinated **5-phase process**:\n\n1. **Loads user configuration** from YAML\n2. **Establishes WebSocket connection** to device agent server \n3. **Performs AIP registration protocol** exchange\n4. **Collects client telemetry** data\n5. **Activates the agent** as task-ready\n\nSee [Agent Registration Overview](./overview.md) for architecture context and [DeviceRegistry](./device_registry.md) for data management details.\n\n\n*Multi-source AgentProfile construction and registration flow.*\n\n## 🎯 Registration Phases\n\n### Phase Overview\n\n```mermaid\ngraph TB\n Start([Start]) --> P1[Phase 1: User Configuration]\n P1 --> P2[Phase 2: WebSocket Connection]\n P2 --> P3[Phase 3: Service Registration]\n P3 --> P4[Phase 4: Telemetry Collection]\n P4 --> P5[Phase 5: Agent Activation]\n P5 --> End([Agent Ready])\n \n P2 -->|Connection Failed| Retry{Retry < Max?}\n Retry -->|Yes| P2\n Retry -->|No| Failed([Failed])\n \n P3 -->|Registration Rejected| Failed\n \n style P1 fill:#e1f5ff\n style P2 fill:#fff4e1\n style P3 fill:#ffe1e1\n style P4 fill:#e8f5e9\n style P5 fill:#f3e5f5\n style End fill:#c8e6c9\n style Failed fill:#ffcdd2\n```\n\n| Phase | Duration | Can Fail? | Retry? | Result |\n|-------|----------|-----------|--------|--------|\n| **1. User Configuration** | < 1s | Yes | No | AgentProfile created |\n| **2. WebSocket Connection** | 1-5s | Yes | Yes (up to max_retries) | Active WebSocket |\n| **3. Service Registration** | 1-2s | Yes | No | Client type recorded |\n| **4. Telemetry Collection** | 1-3s | No (graceful degradation) | No | System info merged |\n| **5. Agent Activation** | < 1s | No | No | Status = IDLE |\n\n## 📝 Phase 1: User Configuration\n\n### Purpose\n\nLoad device configuration from YAML file and create initial AgentProfile with user-specified data.\n\n### Input\n\n`config/galaxy/devices.yaml`:\n\n```yaml\ndevices:\n - device_id: \"windowsagent\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"web_browsing\"\n - \"office_applications\"\n - \"file_management\"\n metadata:\n location: \"office_desktop\"\n performance: \"high\"\n description: \"Primary Windows workstation\"\n operation_engineer_email: \"admin@example.com\"\n max_retries: 5\n auto_connect: true\n```\n\n### Process\n\n```mermaid\nsequenceDiagram\n participant YAML as devices.yaml\n participant Manager as DeviceManager\n participant Registry as DeviceRegistry\n \n YAML->>Manager: Load configuration\n Manager->>Manager: Parse YAML\n \n loop For each device in config\n Manager->>Registry: register_device(device_id, server_url, ...)\n Registry->>Registry: Create AgentProfile\n Registry->>Registry: Set status = DISCONNECTED\n Registry-->>Manager: AgentProfile created\n end\n \n Manager->>Manager: Check auto_connect flag\n \n alt auto_connect == true\n Manager->>Manager: Schedule connect_device()\n end\n```\n\n### Code Example\n\n```python\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Initialize manager\nmanager = ConstellationDeviceManager(\n task_name=\"production_constellation\",\n heartbeat_interval=30.0,\n reconnect_delay=5.0\n)\n\n# Phase 1: Register device from configuration\nsuccess = await manager.register_device(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\", \"office_applications\", \"file_management\"],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\",\n \"description\": \"Primary Windows workstation\"\n },\n auto_connect=True # Proceed to Phase 2 automatically\n)\n```\n\n### Output\n\n**AgentProfile (Version 1):**\n\n```python\nAgentProfile(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\", \"office_applications\", \"file_management\"],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\",\n \"description\": \"Primary Windows workstation\"\n },\n status=DeviceStatus.DISCONNECTED,\n last_heartbeat=None,\n connection_attempts=0,\n max_retries=5,\n current_task_id=None\n)\n```\n\n> **Phase 1 Complete:** Device registered in local registry with user-specified configuration. Status: `DISCONNECTED`\n\n## 🌐 Phase 2: WebSocket Connection\n\n### Purpose\n\nEstablish a persistent WebSocket connection to the device agent's UFO server. This connection is managed by the `WebSocketConnectionManager` component.\n\nSee [Client Components](../client/components.md) for component architecture details.\n\n### Process\n\n```mermaid\nsequenceDiagram\n participant Manager as DeviceManager\n participant Registry as DeviceRegistry\n participant WSManager as WebSocketConnectionManager\n participant Server as UFO Server\n participant MsgProc as MessageProcessor\n participant HB as HeartbeatManager\n \n Manager->>Registry: update_device_status(CONNECTING)\n Manager->>Registry: increment_connection_attempts()\n \n Manager->>WSManager: connect_to_device(device_info, message_processor)\n WSManager->>Server: WebSocket handshake (ws://...)\n \n alt Connection Successful\n Server-->>WSManager: Connection accepted\n WSManager->>MsgProc: start_message_handler(device_id, websocket)\n MsgProc->>MsgProc: Start listening for messages\n WSManager-->>Manager: Connection established\n \n Manager->>Registry: update_device_status(CONNECTED)\n Manager->>Registry: update_heartbeat()\n Manager->>HB: start_heartbeat(device_id)\n HB->>HB: Start periodic heartbeat checks\n else Connection Failed\n Server-->>WSManager: Connection refused / timeout\n WSManager-->>Manager: ConnectionError\n Manager->>Registry: update_device_status(FAILED)\n Manager->>Manager: schedule_reconnection()\n end\n```\n\n### Connection Parameters\n\n| Parameter | Value | Description |\n|-----------|-------|-------------|\n| **URL** | `ws://host:port/ws` | WebSocket endpoint from configuration |\n| **Timeout** | 30 seconds | Connection timeout |\n| **Protocols** | WebSocket standard | No special sub-protocols |\n| **Headers** | None | Standard WebSocket headers |\n\n### Retry Strategy\n\n```python\nasync def connect_device(self, device_id: str, is_reconnection: bool = False) -> bool:\n \"\"\"Connect to a registered device with retry logic.\"\"\"\n \n device_info = self.device_registry.get_device(device_id)\n \n # Update status\n self.device_registry.update_device_status(device_id, DeviceStatus.CONNECTING)\n \n # Increment attempts (only for initial connection, not reconnections)\n if not is_reconnection:\n self.device_registry.increment_connection_attempts(device_id)\n \n try:\n # Establish WebSocket connection\n await self.connection_manager.connect_to_device(\n device_info,\n message_processor=self.message_processor\n )\n \n # Success: Update status\n self.device_registry.update_device_status(device_id, DeviceStatus.CONNECTED)\n self.device_registry.update_heartbeat(device_id)\n \n # Start heartbeat monitoring\n self.heartbeat_manager.start_heartbeat(device_id)\n \n return True\n \n except (websockets.WebSocketException, OSError, asyncio.TimeoutError) as e:\n self.logger.error(f\"Connection failed: {e}\")\n self.device_registry.update_device_status(device_id, DeviceStatus.FAILED)\n \n # Schedule reconnection if under retry limit\n if device_info.connection_attempts < device_info.max_retries:\n self._schedule_reconnection(device_id)\n \n return False\n```\n\n### Reconnection Logic\n\n```mermaid\ngraph TB\n Disconnect[Connection Lost] --> CheckRetries{Attempts < Max?}\n \n CheckRetries -->|Yes| Wait[Wait reconnect_delay seconds]\n Wait --> Attempt[Attempt Reconnection]\n Attempt --> Success{Success?}\n \n Success -->|Yes| Connected[CONNECTED]\n Success -->|No| Increment[Increment Retry Counter]\n Increment --> CheckRetries\n \n CheckRetries -->|No| Failed[FAILED - Give Up]\n \n Connected --> End([Ready for Phase 3])\n Failed --> End2([Registration Failed])\n \n style Connected fill:#c8e6c9\n style Failed fill:#ffcdd2\n```\n\n**Reconnection Parameters:**\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `max_retries` | 5 | Maximum reconnection attempts |\n| `reconnect_delay` | 5.0 seconds | Delay between attempts |\n| `retry_counter` | Per-device | Tracked in AgentProfile.connection_attempts |\n\n> **Warning:** If a device fails to connect after `max_retries` attempts, it enters `FAILED` status and requires manual intervention (e.g., restarting the device agent server).\n\n### Output\n\n- **WebSocket connection** established and active\n- **Message handler** listening for incoming messages\n- **Heartbeat monitoring** started\n- **Status**: `CONNECTED`\n\n> **Phase 2 Complete:** WebSocket connection established. Message handler and heartbeat monitoring active.\n\n## 📡 Phase 3: Service Registration (AIP)\n\n### Purpose\n\nPerform AIP registration protocol exchange to:\n\n- Identify client type (DEVICE vs CONSTELLATION)\n- Advertise platform information\n- Validate registration with server\n\nSee [AIP Protocol Documentation](../../aip/protocols.md#registration-protocol) for detailed protocol specifications.\n\n### Process\n\n```mermaid\nsequenceDiagram\n participant Manager as DeviceManager\n participant WSManager as WebSocketConnectionManager\n participant Transport as WebSocketTransport\n participant RegProtocol as RegistrationProtocol\n participant Server as UFO Server Handler\n \n Note over Manager,Server: Device Agent Client Registration\n \n Manager->>RegProtocol: register_as_device(device_id, metadata, platform)\n \n RegProtocol->>RegProtocol: Prepare ClientMessage\n Note over RegProtocol: type: REGISTER client_type: DEVICE metadata: system_info, etc.\n \n RegProtocol->>Transport: send_message(ClientMessage)\n Transport->>Server: WebSocket: REGISTER message\n \n Server->>Server: Parse ClientMessage\n Server->>Server: Validate registration\n Server->>Server: Extract metadata, system_info\n Server->>Server: Store in ClientConnectionManager\n \n Server->>Transport: ServerMessage (status: OK)\n Transport->>RegProtocol: receive_message(ServerMessage)\n \n alt Registration Successful\n RegProtocol-->>Manager: True (registration successful)\n Manager->>Registry: update_device_status(CONNECTED)\n else Registration Failed\n RegProtocol-->>Manager: False (registration failed)\n Manager->>Registry: update_device_status(FAILED)\n end\n```\n\n### Registration Message Structure\n\n**Client → Server (REGISTER message):**\n\n```python\nClientMessage(\n type=ClientMessageType.REGISTER,\n client_id=\"windowsagent\",\n client_type=ClientType.DEVICE,\n status=TaskStatus.OK,\n timestamp=\"2025-11-06T10:30:00.000Z\",\n metadata={\n \"platform\": \"windows\",\n \"registration_time\": \"2025-11-06T10:30:00.000Z\",\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"supported_features\": [\"gui\", \"cli\", \"browser\", \"file_system\", \"office\"],\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n }\n }\n)\n```\n\n**Server → Client (Confirmation):**\n\n```python\nServerMessage(\n type=ServerMessageType.HEARTBEAT,\n status=TaskStatus.OK,\n timestamp=\"2025-11-06T10:30:01.000Z\",\n response_id=\"reg_confirmation_12345\"\n)\n```\n\n### Server-Side Processing\n\n```python\n# In UFOWebSocketHandler.connect()\n\nasync def connect(self, websocket: WebSocket) -> str:\n \"\"\"Server-side registration handling.\"\"\"\n \n await websocket.accept()\n \n # Initialize AIP protocols\n self.transport = WebSocketTransport(websocket)\n self.registration_protocol = RegistrationProtocol(self.transport)\n \n # Parse registration message\n reg_info = await self._parse_registration_message()\n \n # Validate client type\n client_type = reg_info.client_type # DEVICE or CONSTELLATION\n platform = reg_info.metadata.get(\"platform\", \"windows\")\n \n # Register client\n client_id = reg_info.client_id\n self.client_manager.add_client(\n client_id,\n platform,\n websocket,\n client_type,\n reg_info.metadata # Contains system_info\n )\n \n # Send confirmation\n await self._send_registration_confirmation()\n \n return client_id\n```\n\n### Constellation Client Registration\n\nFor constellation clients (not device agents), the registration differs:\n\n```python\n# Constellation client registration\nClientMessage(\n type=ClientMessageType.REGISTER,\n client_id=\"constellation_orchestrator\",\n client_type=ClientType.CONSTELLATION,\n target_id=\"windowsagent\", # Target device for this constellation\n status=TaskStatus.OK,\n timestamp=\"2025-11-06T10:30:00.000Z\",\n metadata={\n \"type\": \"constellation_client\",\n \"targeted_device_id\": \"windowsagent\"\n }\n)\n```\n\n> **Note:** Device clients register as `ClientType.DEVICE`, while constellation orchestrators register as `ClientType.CONSTELLATION` with a `target_id` pointing to the device they want to control.\n\n### Output\n\n- Client registered in server's `ClientConnectionManager`\n- Client type (DEVICE/CONSTELLATION) recorded\n- Platform information stored\n- Registration confirmation received\n\n> **Phase 3 Complete:** AIP registration protocol completed. Client type and platform recorded on server.\n\n## 📊 Phase 4: Telemetry Collection\n\n### Purpose\n\nCollect real-time system information from the device client and merge it into the AgentProfile. The system information is collected by the device's `DeviceInfoProvider` during registration and sent to the server as part of the registration metadata.\n\nSee [Device Info Provider](../../client/device_info.md) for details on telemetry collection.\n\n### Process\n\n```mermaid\nsequenceDiagram\n participant Manager as DeviceManager\n participant WSManager as WebSocketConnectionManager\n participant Server as UFO Server\n participant DIP as DeviceInfoProvider\n participant Registry as DeviceRegistry\n \n Note over Manager,Registry: Request Device Info\n \n Manager->>WSManager: request_device_info(device_id)\n WSManager->>Server: Send DEVICE_INFO_REQUEST\n \n Note over Server,DIP: Server has already received system_info during registration\n \n Server->>Server: Retrieve stored system_info from ClientConnectionManager\n Server-->>WSManager: Return system_info\n WSManager-->>Manager: system_info dict\n \n Note over Manager,Registry: Merge System Info into AgentProfile\n \n Manager->>Registry: update_device_system_info(device_id, system_info)\n \n Registry->>Registry: Update OS if not set\n Registry->>Registry: Merge supported_features into capabilities\n Registry->>Registry: Add system_info to metadata\n Registry->>Registry: Add custom_metadata if present\n \n Registry-->>Manager: Update complete\n```\n\n### DeviceInfoProvider (Client-Side)\n\nThe device client collects system info **during registration** (before Phase 4):\n\n```python\n# In WebSocket client's register_client() method\n\nfrom ufo.client.device_info_provider import DeviceInfoProvider\n\n# Collect device info\nsystem_info = DeviceInfoProvider.collect_system_info(\n client_id=self.ufo_client.client_id,\n custom_metadata=None\n)\n\n# Prepare metadata for registration\nmetadata = {\n \"system_info\": system_info.to_dict(),\n \"registration_time\": datetime.now(timezone.utc).isoformat()\n}\n\n# Register with AIP (includes system_info in metadata)\nawait self.registration_protocol.register_as_device(\n device_id=self.ufo_client.client_id,\n metadata=metadata,\n platform=self.ufo_client.platform\n)\n```\n\n### System Info Structure\n\n```python\n{\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"supported_features\": [\n \"gui\",\n \"cli\",\n \"browser\",\n \"file_system\",\n \"office\",\n \"windows_apps\"\n ],\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n}\n```\n\nSee [Device Info Provider](../../client/device_info.md) for telemetry collection details.\n\n### Merging Logic\n\n```python\ndef update_device_system_info(\n self, device_id: str, system_info: Dict[str, Any]\n) -> bool:\n \"\"\"Update AgentProfile with system information.\"\"\"\n \n device_info = self.get_device(device_id)\n if not device_info:\n return False\n \n # 1. Update OS information\n if \"platform\" in system_info:\n device_info.os = system_info[\"platform\"]\n \n # 2. Merge capabilities with supported features (avoid duplicates)\n if \"supported_features\" in system_info:\n features = system_info[\"supported_features\"]\n existing_caps = set(device_info.capabilities)\n new_caps = existing_caps.union(set(features))\n device_info.capabilities = list(new_caps)\n \n # 3. Update metadata with system information\n device_info.metadata.update({\n \"system_info\": {\n \"platform\": system_info.get(\"platform\"),\n \"os_version\": system_info.get(\"os_version\"),\n \"cpu_count\": system_info.get(\"cpu_count\"),\n \"memory_total_gb\": system_info.get(\"memory_total_gb\"),\n \"hostname\": system_info.get(\"hostname\"),\n \"ip_address\": system_info.get(\"ip_address\"),\n \"platform_type\": system_info.get(\"platform_type\"),\n \"schema_version\": system_info.get(\"schema_version\")\n }\n })\n \n # 4. Add custom metadata if present\n if \"custom_metadata\" in system_info:\n device_info.metadata[\"custom_metadata\"] = system_info[\"custom_metadata\"]\n \n # 5. Add tags if present\n if \"tags\" in system_info:\n device_info.metadata[\"tags\"] = system_info[\"tags\"]\n \n return True\n```\n\n### Before & After\n\n**Before Telemetry (AgentProfile v2):**\n\n```python\nAgentProfile(\n device_id=\"windowsagent\",\n os=\"windows\", # From user config\n capabilities=[\"web_browsing\", \"office_applications\", \"file_management\"],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\"\n }\n)\n```\n\n**After Telemetry (AgentProfile v3 - Complete):**\n\n```python\nAgentProfile(\n device_id=\"windowsagent\",\n os=\"windows\", # Confirmed by telemetry\n capabilities=[\n \"web_browsing\", \"office_applications\", \"file_management\", # User config\n \"gui\", \"cli\", \"browser\", \"file_system\", \"office\", \"windows_apps\" # Auto-detected\n ],\n metadata={\n # User config\n \"location\": \"office_desktop\",\n \"performance\": \"high\",\n \n # Telemetry\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n }\n }\n)\n```\n\n> **Phase 4 Complete:** System information collected and merged into AgentProfile. Capabilities expanded with auto-detected features.\n\n## ✅ Phase 5: Agent Activation\n\n### Purpose\n\nFinalize agent registration and set it to IDLE status, ready to accept task assignments.\n\n### Process\n\n```mermaid\nsequenceDiagram\n participant Manager as DeviceManager\n participant Registry as DeviceRegistry\n participant HB as HeartbeatManager\n \n Manager->>Registry: set_device_idle(device_id)\n \n Registry->>Registry: Update status = IDLE\n Registry->>Registry: Clear current_task_id = None\n Registry-->>Manager: Status updated\n \n Manager->>HB: Verify heartbeat active\n HB-->>Manager: Heartbeat OK\n \n Manager->>Manager: Log successful registration\n Note over Manager: ✅ Device ready for tasks\n```\n\n### Code\n\n```python\n# Set device to IDLE (ready to accept tasks)\nself.device_registry.set_device_idle(device_id)\n\nself.logger.info(f\"✅ Successfully connected to device {device_id}\")\n```\n\n### Final AgentProfile State\n\n```python\nAgentProfile(\n # Identity\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n \n # Platform & Capabilities\n os=\"windows\",\n capabilities=[\n \"web_browsing\", \"office_applications\", \"file_management\",\n \"gui\", \"cli\", \"browser\", \"file_system\", \"office\", \"windows_apps\"\n ],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\",\n \"platform\": \"windows\",\n \"registration_time\": \"2025-11-06T10:30:00Z\",\n \"system_info\": {\n \"platform\": \"windows\",\n \"os_version\": \"10.0.22631\",\n \"cpu_count\": 16,\n \"memory_total_gb\": 32.0,\n \"hostname\": \"DESKTOP-DEV01\",\n \"ip_address\": \"192.168.1.100\",\n \"platform_type\": \"computer\",\n \"schema_version\": \"1.0\"\n }\n },\n \n # Status\n status=DeviceStatus.IDLE, # ✅ Ready for tasks!\n last_heartbeat=datetime(2025, 11, 6, 10, 30, 45),\n \n # Connection\n connection_attempts=0, # Reset after successful connection\n max_retries=5,\n \n # Task\n current_task_id=None\n)\n```\n\n> **Phase 5 Complete:** Agent fully registered, profiled, and activated. Status: `IDLE` - Ready to accept task assignments.\n\n## 🎯 Complete End-to-End Example\n\n### Scenario: Register Windows Workstation\n\n```python\nimport asyncio\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\nasync def register_windows_workstation():\n \"\"\"Complete registration flow example.\"\"\"\n \n # Initialize manager\n manager = ConstellationDeviceManager(\n task_name=\"office_constellation\",\n heartbeat_interval=30.0,\n reconnect_delay=5.0\n )\n \n print(\"📝 Phase 1: User Configuration\")\n # Register device from user config\n success = await manager.register_device(\n device_id=\"windowsagent\",\n server_url=\"ws://localhost:5005/ws\",\n os=\"windows\",\n capabilities=[\"web_browsing\", \"office_applications\", \"file_management\"],\n metadata={\n \"location\": \"office_desktop\",\n \"performance\": \"high\",\n \"description\": \"Primary Windows workstation\",\n \"operation_engineer_email\": \"admin@example.com\"\n },\n max_retries=5,\n auto_connect=True # Will proceed to Phase 2-5 automatically\n )\n \n if success:\n print(\"✅ Registration successful!\")\n \n # Get complete profile\n profile = manager.get_device_info(\"windowsagent\")\n \n print(f\"\\n📊 AgentProfile:\")\n print(f\" Device ID: {profile.device_id}\")\n print(f\" Status: {profile.status.value}\")\n print(f\" OS: {profile.os}\")\n print(f\" Capabilities: {profile.capabilities}\")\n print(f\" System Info:\")\n system_info = profile.metadata.get(\"system_info\", {})\n print(f\" - CPU Cores: {system_info.get('cpu_count')}\")\n print(f\" - Memory: {system_info.get('memory_total_gb')} GB\")\n print(f\" - Hostname: {system_info.get('hostname')}\")\n print(f\" - IP: {system_info.get('ip_address')}\")\n \n # Device is now ready for tasks\n print(f\"\\n🚀 Device is ready to receive tasks!\")\n \n else:\n print(\"❌ Registration failed\")\n\n# Run the example\nasyncio.run(register_windows_workstation())\n```\n\n**Output:**\n\n```\n📝 Phase 1: User Configuration\n🌐 Phase 2: WebSocket Connection\n Connecting to ws://localhost:5005/ws...\n Connection established\n📡 Phase 3: Service Registration\n Sending REGISTER message...\n Registration confirmed\n📊 Phase 4: Telemetry Collection\n Collecting system information...\n System info merged\n✅ Phase 5: Agent Activation\n Device set to IDLE\n\n✅ Registration successful!\n\n📊 AgentProfile:\n Device ID: windowsagent\n Status: idle\n OS: windows\n Capabilities: ['web_browsing', 'office_applications', 'file_management', 'gui', 'cli', 'browser', 'file_system', 'office', 'windows_apps']\n System Info:\n - CPU Cores: 16\n - Memory: 32.0 GB\n - Hostname: DESKTOP-DEV01\n - IP: 192.168.1.100\n\n🚀 Device is ready to receive tasks!\n```\n\n---\n\n## 🔧 Error Handling\n\n### Connection Failures\n\n```python\ntry:\n success = await manager.register_device(...)\nexcept websockets.WebSocketException as e:\n logger.error(f\"WebSocket error: {e}\")\n # Will automatically retry if under max_retries\nexcept OSError as e:\n logger.error(f\"Network error: {e}\")\n # Check network connectivity\nexcept asyncio.TimeoutError:\n logger.error(\"Connection timeout\")\n # Server may be down or unreachable\n```\n\n### Registration Rejection\n\n```python\n# Server-side validation\nif not self.client_manager.is_device_connected(claimed_device_id):\n error_msg = f\"Target device '{claimed_device_id}' is not connected\"\n await self._send_error_response(error_msg)\n await self.transport.close()\n raise ValueError(error_msg)\n```\n\n### Telemetry Collection Failure\n\n```python\n# Graceful degradation - always returns valid DeviceSystemInfo\ntry:\n return DeviceSystemInfo(...)\nexcept Exception as e:\n logger.error(f\"Error collecting system info: {e}\")\n # Return minimal info instead of failing\n return DeviceSystemInfo(\n device_id=client_id,\n platform=\"unknown\",\n os_version=\"unknown\",\n cpu_count=0,\n memory_total_gb=0.0,\n hostname=\"unknown\",\n ip_address=\"unknown\",\n supported_features=[],\n platform_type=\"unknown\"\n )\n```\n\n---\n\n## 🔗 Related Documentation\n\n| Topic | Document | Description |\n|-------|----------|-------------|\n| **Overview** | [Agent Registration Overview](./overview.md) | Registration architecture |\n| **AgentProfile** | [AgentProfile](./agent_profile.md) | Profile structure details |\n| **Device Registry** | [Device Registry](./device_registry.md) | Registry component |\n| **Galaxy Devices Config** | [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md) | YAML config reference |\n| **Device Info** | [Device Info Provider](../../client/device_info.md) | Telemetry collection |\n| **AIP Protocol** | [AIP Overview](../../aip/overview.md) | Protocol fundamentals |\n\n## 💡 Best Practices\n\n**1. Use auto_connect for Production**\n\n```python\nawait manager.register_device(..., auto_connect=True)\n# Automatically completes all 5 phases\n```\n\n**2. Configure Appropriate max_retries**\n\n```python\n# Critical devices: higher retries\nmax_retries=10 # For production servers\n\n# Test devices: lower retries\nmax_retries=3 # For development environments\n```\n\n**3. Monitor Registration Status**\n\n```python\nprofile = manager.get_device_info(device_id)\nif profile.status == DeviceStatus.FAILED:\n logger.error(f\"Device {device_id} failed to register\")\n # Take corrective action\n```\n\n**4. Provide Rich Metadata**\n\n```python\nmetadata={\n \"location\": \"datacenter_us_west\",\n \"performance\": \"high\",\n \"tags\": [\"production\", \"critical\"],\n \"operation_engineer_email\": \"ops@example.com\"\n}\n```\n\n## 🚀 Next Steps\n\n1. **Configure Devices**: Read [Galaxy Devices Configuration](../../configuration/system/galaxy_devices.md)\n2. **Understand DeviceRegistry**: Check [Device Registry](./device_registry.md)\n3. **Learn Task Assignment**: See [Task Execution Documentation](../constellation_orchestrator/overview.md)\n4. **Study AIP Messages**: Read [AIP Messages](../../aip/messages.md)\n\n## 📚 Source Code References\n\n- **ConstellationDeviceManager**: `galaxy/client/device_manager.py`\n- **DeviceRegistry**: `galaxy/client/components/device_registry.py`\n- **RegistrationProtocol**: `aip/protocol/registration.py`\n- **UFOWebSocketHandler**: `ufo/server/ws/handler.py`\n- **DeviceInfoProvider**: `ufo/client/device_info_provider.py`\n"
},
{
"path": "documents/docs/galaxy/client/aip_integration.md",
"content": "# AIP Protocol Integration\n\nThe Agent Interaction Protocol (AIP) is the communication protocol used throughout Galaxy Client for device coordination. This document explains how Galaxy Client integrates with AIP, the message flow patterns, and how different components use the protocol.\n\n## Related Documentation\n\n- [Overview](./overview.md) - Overall Galaxy Client architecture\n- [DeviceManager](./device_manager.md) - Connection management using AIP\n- [Components](./components.md) - Component-level AIP usage\n- [AIP Protocol Specification](../../aip/overview.md) - Complete protocol reference\n- [AIP Message Reference](../../aip/messages.md) - Detailed message structures\n\n---\n\n## What is AIP?\n\nAIP (Agent Interaction Protocol) is a WebSocket-based message protocol for agent communication. It defines structured message types, status codes, and communication patterns for device registration, task execution, health monitoring, and information exchange.\n\n**Core Principles:**\n\n**Transport Agnostic**: AIP runs over WebSocket in Galaxy Client, but the protocol itself is transport-independent. You could implement AIP over HTTP, gRPC, or any other transport.\n\n**Strongly Typed**: All messages are Pydantic models with strict validation. Invalid messages are rejected immediately, preventing protocol errors from propagating.\n\n**Bidirectional**: Both client and server can initiate messages. Clients send REGISTER, TASK_END, HEARTBEAT responses. Server sends TASK, DEVICE_INFO_REQUEST, HEARTBEAT requests.\n\n**Status-Based**: Every message includes a status field (OK, ERROR, CONTINUE, COMPLETED, FAILED) indicating the message's semantic meaning and guiding response handling.\n\n**Key Message Types:**\n\n```\nRegistration & Connection:\n- REGISTER: Device announces itself to server\n- REGISTER_CONFIRMATION: Server acknowledges registration\n\nHealth Monitoring:\n- HEARTBEAT (client→server): \"I'm alive\"\n- HEARTBEAT (server→client): \"Are you alive?\"\n\nTask Execution:\n- TASK (server→client): \"Execute this task\"\n- COMMAND (server→client): \"Execute these commands\"\n- COMMAND_RESULTS (client→server): \"Command execution results\"\n- TASK_END (client→server): \"Task completed\"\n\nDevice Information:\n- DEVICE_INFO_REQUEST (client→server): \"What are your system specs?\"\n- DEVICE_INFO_RESPONSE (server→client): \"Here's my system info\"\n\nError Handling:\n- ERROR: \"Something went wrong\"\n```\n\n---\n\n## Protocol Architecture in Galaxy Client\n\nGalaxy Client uses AIP at multiple levels:\n\n### Layer 1: Transport (WebSocket)\n\nWebSocketConnectionManager handles raw WebSocket communication:\n\n```python\n# Establish WebSocket connection\nws = await websockets.connect(server_url)\n\n# Send raw bytes\nawait ws.send(message_bytes)\n\n# Receive raw bytes\nmessage_bytes = await ws.recv()\n```\n\nWebSocketConnectionManager knows nothing about AIP message structure. It's purely a transport layer.\n\n### Layer 2: Protocol (AIP)\n\nAIPProtocol class (from `aip/protocol/base.py`) handles message serialization and deserialization:\n\n```python\nfrom aip.protocol import AIPProtocol\nfrom aip.transport import WebSocketTransport\n\n# Wrap WebSocket in Transport abstraction\ntransport = WebSocketTransport(ws)\n\n# Create protocol handler\nprotocol = AIPProtocol(transport)\n\n# Send structured message\nawait protocol.send_message(ClientMessage(\n type=ClientMessageType.REGISTER,\n payload={\"device_id\": \"windows_pc\"}\n))\n\n# Receive structured message\nmessage = await protocol.receive_message(ServerMessage)\n```\n\nAIPProtocol converts between Pydantic models and bytes, applies middleware, and handles serialization errors.\n\n### Layer 3: Message Processing (MessageProcessor)\n\nMessageProcessor (from DeviceManager components) routes messages to handlers:\n\n```python\n# Register handler for TASK messages\nmessage_processor.register_handler(\n message_type=\"task\",\n handler=handle_task_message\n)\n\n# Start listening for messages\nawait message_processor.start_message_handler(device_id)\n\n# Messages automatically routed to registered handlers\n```\n\nMessageProcessor implements the observer pattern, dispatching incoming messages to registered callbacks.\n\n### Layer 4: Application Logic (DeviceManager, ConstellationClient)\n\nApplication components use MessageProcessor to send/receive messages without dealing with protocol details:\n\n```python\n# Send REGISTER message\nawait message_processor.send_message(\n device_id=device_id,\n message_type=\"REGISTER\",\n payload={\"device_id\": device_id, \"capabilities\": [\"office\"]}\n)\n\n# Wait for REGISTER_CONFIRMATION\nconfirmation = await message_processor.wait_for_response(\n device_id=device_id,\n message_type=\"REGISTER_CONFIRMATION\",\n timeout=10.0\n)\n```\n\nThis layered architecture separates concerns and makes each layer testable.\n\n---\n\n## Message Flow Patterns\n\n### Device Registration Flow\n\nWhen DeviceManager connects to a device, it performs AIP registration:\n\n```mermaid\nsequenceDiagram\n participant DM as DeviceManager\n participant MP as MessageProcessor\n participant P as AIPProtocol\n participant T as WebSocketTransport\n participant Server\n \n Note over DM,Server: 1. WebSocket Connection\n DM->>T: connect(server_url)\n T->>Server: WebSocket handshake\n Server-->>T: Connection established\n \n Note over DM,Server: 2. Device Registration\n DM->>MP: send_message(REGISTER)\n MP->>P: send_message(ClientMessage)\n P->>P: Serialize to JSON\n P->>T: send(bytes)\n T->>Server: REGISTER message\n \n Note over DM,Server: 3. Server Confirmation\n Server->>T: REGISTER_CONFIRMATION\n T-->>P: recv(bytes)\n P->>P: Deserialize from JSON\n P-->>MP: ServerMessage(REGISTER_CONFIRMATION)\n MP-->>DM: Registration confirmed\n \n Note over DM,Server: 4. Device Info Exchange\n DM->>MP: send_message(DEVICE_INFO_REQUEST)\n MP->>Server: DEVICE_INFO_REQUEST\n Server->>MP: DEVICE_INFO_RESPONSE\n MP-->>DM: Device telemetry\n```\n\n**Message Details:**\n\n**Step 2 - REGISTER Message:**\n```json\n{\n \"type\": \"register\",\n \"client_type\": \"constellation\",\n \"payload\": {\n \"device_id\": \"windows_pc\",\n \"capabilities\": [\"office\", \"web\", \"email\"],\n \"metadata\": {\n \"location\": \"office\",\n \"user\": \"john\"\n }\n },\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:30:00Z\"\n}\n```\n\n**Step 3 - REGISTER_CONFIRMATION:**\n```json\n{\n \"type\": \"heartbeat\",\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:30:01Z\",\n \"response_id\": \"reg_conf_abc123\"\n}\n```\n\nNote: The server confirms registration by sending a HEARTBEAT message with OK status, which serves as the registration confirmation in the AIP protocol.\n\n**Step 4 - DEVICE_INFO_REQUEST:**\n```json\n{\n \"type\": \"device_info_request\",\n \"client_type\": \"constellation\",\n \"payload\": {\n \"request_id\": \"req_xyz789\"\n },\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:30:02Z\"\n}\n```\n\n**Step 4 - DEVICE_INFO_RESPONSE:**\n```json\n{\n \"type\": \"device_info_response\",\n \"status\": \"ok\",\n \"result\": {\n \"device_id\": \"windows_pc\",\n \"device_info\": {\n \"os\": \"Windows 11\",\n \"cpu_count\": 8,\n \"memory_gb\": 32,\n \"screen_resolution\": \"1920x1080\",\n \"python_version\": \"3.11.5\",\n \"installed_apps\": [\"Microsoft Office\", \"Chrome\", \"VSCode\"]\n }\n },\n \"timestamp\": \"2025-11-06T10:30:03Z\",\n \"response_id\": \"info_resp_xyz789\"\n}\n```\n\n### Heartbeat Flow\n\nHeartbeatManager sends periodic HEARTBEAT messages to monitor device health:\n\n```mermaid\nsequenceDiagram\n participant HM as HeartbeatManager\n participant MP as MessageProcessor\n participant Server\n \n Note over HM: Every 30 seconds\n \n HM->>MP: send_message(HEARTBEAT)\n MP->>Server: HEARTBEAT\n \n alt Server responds\n Server-->>MP: HEARTBEAT (response)\n MP-->>HM: Response received\n HM->>HM: Update last_heartbeat timestamp\n else Timeout (no response in 10s)\n MP-->>HM: TimeoutError\n HM->>HM: Mark device as unhealthy\n HM->>DM: _handle_device_disconnection(\"heartbeat_timeout\")\n end\n```\n\n**HEARTBEAT Message (client→server):**\n```json\n{\n \"type\": \"heartbeat\",\n \"client_type\": \"constellation\",\n \"client_id\": \"constellation_client_id\",\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:35:00Z\"\n}\n```\n\n**HEARTBEAT Response (server→client):**\n```json\n{\n \"type\": \"heartbeat\",\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:35:00Z\",\n \"response_id\": \"hb_resp_123\"\n}\n```\n\nHeartbeat is a simple request-response pattern. If the server doesn't respond within timeout, HeartbeatManager assumes connection failure and triggers reconnection.\n\n### Task Execution Flow\n\nTask execution involves multiple message exchanges:\n\n```mermaid\nsequenceDiagram\n participant Orch as TaskOrchestrator\n participant DM as DeviceManager\n participant MP as MessageProcessor\n participant Server\n participant Device\n \n Note over Orch,Device: 1. Task Assignment\n Orch->>DM: assign_task_to_device(task_id, device_id, ...)\n DM->>MP: send_message(TASK)\n MP->>Server: TASK\n Server->>Device: Forward TASK\n \n Note over Orch,Device: 2. Task Execution (on device)\n Device->>Device: Plan task steps\n \n loop For each step\n Device->>Server: Request command execution\n Server->>Device: COMMAND (action to take)\n Device->>Device: Execute command\n Device->>Server: COMMAND_RESULTS\n Server->>Server: Store results\n end\n \n Note over Orch,Device: 3. Task Completion\n Device->>Server: TASK_END (status=completed)\n Server->>MP: TASK_END\n MP-->>DM: Task result\n DM-->>Orch: Task completed\n```\n\n**TASK Message (server→client):**\n```json\n{\n \"type\": \"task\",\n \"status\": \"continue\",\n \"user_request\": \"Open Excel and create a chart\",\n \"task_name\": \"galaxy/production/excel_task\",\n \"session_id\": \"sess_task_abc123\",\n \"timestamp\": \"2025-11-06T10:40:00Z\",\n \"response_id\": \"task_req_001\"\n}\n```\n\n**COMMAND Message (server→client):**\n```json\n{\n \"type\": \"command\",\n \"status\": \"continue\",\n \"actions\": [\n {\n \"action\": \"launch_app\",\n \"parameters\": {\n \"app_name\": \"Excel\"\n }\n },\n {\n \"action\": \"open_file\",\n \"parameters\": {\n \"file_path\": \"sales_report.xlsx\"\n }\n }\n ],\n \"session_id\": \"sess_task_abc123\",\n \"response_id\": \"cmd_001\"\n}\n```\n\n**COMMAND_RESULTS Message (client→server):**\n```json\n{\n \"type\": \"command_results\",\n \"client_type\": \"device\",\n \"client_id\": \"device_agent_id\",\n \"status\": \"continue\",\n \"action_results\": [\n {\n \"action\": \"launch_app\",\n \"status\": \"completed\",\n \"result\": \"Excel launched successfully\"\n },\n {\n \"action\": \"open_file\",\n \"status\": \"completed\",\n \"result\": \"File opened: sales_report.xlsx\"\n }\n ],\n \"session_id\": \"sess_task_abc123\",\n \"prev_response_id\": \"cmd_001\"\n}\n```\n\n**TASK_END Message (client→server):**\n```json\n{\n \"type\": \"task_end\",\n \"client_type\": \"device\",\n \"client_id\": \"device_agent_id\",\n \"status\": \"completed\",\n \"result\": {\n \"success\": true,\n \"output\": \"Created bar chart showing quarterly sales\",\n \"artifacts\": [\n {\n \"type\": \"file\",\n \"path\": \"sales_report_with_chart.xlsx\"\n }\n ]\n },\n \"session_id\": \"sess_task_abc123\",\n \"timestamp\": \"2025-11-06T10:40:15Z\"\n}\n```\n\nThis multi-message pattern allows streaming execution updates and early error detection.\n\n---\n\n## Error Handling\n\nAIP uses ERROR messages for protocol-level errors:\n\n### Error Types\n\n**Connection Errors**: WebSocket closed, network failure\n- Handled by: WebSocketConnectionManager\n- Recovery: Reconnection with exponential backoff\n\n**Protocol Errors**: Invalid message format, unknown message type\n- Handled by: AIPProtocol\n- Recovery: Send ERROR message, log warning, continue\n\n**Task Errors**: Command execution failure, task timeout\n- Handled by: Device agent\n- Recovery: Send TASK_END with status=failed\n\n**Application Errors**: Device not found, capability mismatch\n- Handled by: DeviceManager, ConstellationClient\n- Recovery: Application-specific (queue task, fail request, etc.)\n\n### ERROR Message Format\n\n```json\n{\n \"type\": \"error\",\n \"status\": \"error\",\n \"error\": \"Task execution exceeded 300 second timeout\",\n \"session_id\": \"sess_task_abc123\",\n \"timestamp\": \"2025-11-06T10:45:00Z\",\n \"response_id\": \"err_001\",\n \"metadata\": {\n \"error_code\": \"TASK_TIMEOUT\",\n \"elapsed_time\": 315.2,\n \"last_command\": \"create_chart\"\n }\n}\n```\n\n**Error Codes:**\n\n- `CONNECTION_FAILED`: WebSocket connection failed\n- `REGISTRATION_FAILED`: Device registration rejected\n- `TASK_TIMEOUT`: Task execution exceeded timeout\n- `COMMAND_FAILED`: Individual command failed\n- `PROTOCOL_ERROR`: Invalid message format or type\n- `DEVICE_NOT_FOUND`: Target device doesn't exist\n- `CAPABILITY_MISMATCH`: Device lacks required capability\n\n### Error Handling Example\n\n```python\ntry:\n # Send TASK message\n await message_processor.send_message(\n device_id=device_id,\n message_type=\"TASK\",\n payload=task_data\n )\n \n # Wait for TASK_END\n result = await message_processor.wait_for_response(\n device_id=device_id,\n message_type=\"TASK_END\",\n timeout=300.0\n )\n \n if result.status == TaskStatus.FAILED:\n # Task failed on device\n error_info = result.payload.get(\"error\")\n logger.error(f\"Task failed: {error_info}\")\n # Application-specific recovery\n \nexcept TimeoutError:\n # No response within timeout\n logger.error(\"Task timeout, marking device as failed\")\n await device_manager._handle_device_disconnection(\n device_id,\n reason=\"task_timeout\"\n )\n \nexcept ConnectionError:\n # Connection lost during execution\n logger.error(\"Connection lost during task\")\n await device_manager._handle_device_disconnection(\n device_id,\n reason=\"connection_lost\"\n )\n```\n\n---\n\n## Message Processing Implementation\n\n### MessageProcessor Component\n\nMessageProcessor (from DeviceManager components) implements AIP message handling:\n\n**Key Responsibilities:**\n\n1. **Message Sending**: Serialize and send messages via AIPProtocol\n2. **Message Receiving**: Deserialize and route incoming messages\n3. **Handler Registration**: Allow components to register callbacks for message types\n4. **Request-Response Pattern**: Implement synchronous request-response over async WebSocket\n\n**Internal Architecture:**\n\n```python\nclass MessageProcessor:\n def __init__(self):\n self._protocols: Dict[str, AIPProtocol] = {} # device_id → protocol\n self._handlers: Dict[str, Dict[str, Callable]] = {} # device_id → {msg_type → handler}\n self._response_queues: Dict[str, asyncio.Queue] = {} # (device_id, msg_type) → queue\n \n async def send_message(\n self,\n device_id: str,\n message_type: str,\n payload: Dict[str, Any]\n ):\n \"\"\"Send message to device.\"\"\"\n protocol = self._protocols[device_id]\n \n # Create message\n msg = ClientMessage(\n type=message_type,\n payload=payload,\n client_type=ClientType.CONSTELLATION,\n status=TaskStatus.OK\n )\n \n # Send via protocol\n await protocol.send_message(msg)\n \n async def wait_for_response(\n self,\n device_id: str,\n message_type: str,\n timeout: float = 30.0\n ) -> ServerMessage:\n \"\"\"Wait for specific message type from device.\"\"\"\n queue_key = (device_id, message_type)\n \n # Create queue if not exists\n if queue_key not in self._response_queues:\n self._response_queues[queue_key] = asyncio.Queue()\n \n # Wait for message with timeout\n try:\n message = await asyncio.wait_for(\n self._response_queues[queue_key].get(),\n timeout=timeout\n )\n return message\n except asyncio.TimeoutError:\n raise TimeoutError(f\"No {message_type} received from {device_id} within {timeout}s\")\n \n async def start_message_handler(self, device_id: str):\n \"\"\"Start background loop to receive and route messages.\"\"\"\n protocol = self._protocols[device_id]\n \n while True:\n try:\n # Receive message\n message = await protocol.receive_message(ServerMessage)\n \n # Route to handler\n msg_type = message.type\n if msg_type in self._handlers.get(device_id, {}):\n handler = self._handlers[device_id][msg_type]\n await handler(message)\n \n # Also add to response queue\n queue_key = (device_id, msg_type)\n if queue_key in self._response_queues:\n await self._response_queues[queue_key].put(message)\n \n except ConnectionError:\n # Connection closed, exit loop\n break\n except Exception as e:\n logger.error(f\"Error processing message: {e}\")\n```\n\nThis implementation supports both callback-based handlers and synchronous request-response patterns.\n\n---\n\n## AIP Extensions and Middleware\n\n### Protocol Middleware\n\nAIPProtocol supports middleware for cross-cutting concerns:\n\n```python\nfrom aip.protocol.base import ProtocolMiddleware\n\nclass LoggingMiddleware(ProtocolMiddleware):\n \"\"\"Log all messages for debugging.\"\"\"\n \n async def process_outgoing(self, message: Any) -> Any:\n \"\"\"Called before sending message.\"\"\"\n logger.debug(f\"Sending: {message.type} to {message.device_id}\")\n return message\n \n async def process_incoming(self, message: Any) -> Any:\n \"\"\"Called after receiving message.\"\"\"\n logger.debug(f\"Received: {message.type} from device\")\n return message\n\nclass MetricsMiddleware(ProtocolMiddleware):\n \"\"\"Track message statistics.\"\"\"\n \n def __init__(self):\n self.sent_count = 0\n self.received_count = 0\n \n async def process_outgoing(self, message: Any) -> Any:\n self.sent_count += 1\n metrics.increment(\"aip.messages.sent\", tags={\"type\": message.type})\n return message\n \n async def process_incoming(self, message: Any) -> Any:\n self.received_count += 1\n metrics.increment(\"aip.messages.received\", tags={\"type\": message.type})\n return message\n\n# Add middleware to protocol\nprotocol.middleware_chain.append(LoggingMiddleware())\nprotocol.middleware_chain.append(MetricsMiddleware())\n```\n\nMiddleware runs for every message, allowing logging, metrics, validation, transformation, etc.\n\n### Custom Message Types\n\nExtend AIP with custom message types:\n\n```python\nfrom enum import Enum\nfrom pydantic import BaseModel\n\n# Define custom message type\nclass CustomMessageType(str, Enum):\n DEVICE_SCREENSHOT = \"device_screenshot\"\n PERFORMANCE_METRICS = \"performance_metrics\"\n\n# Define message structure\nclass ScreenshotRequest(BaseModel):\n type: Literal[\"device_screenshot\"]\n payload: Dict[str, Any]\n\n# Register handler\nmessage_processor.register_handler(\n message_type=\"device_screenshot\",\n handler=handle_screenshot_request\n)\n\n# Send custom message\nawait message_processor.send_message(\n device_id=device_id,\n message_type=\"device_screenshot\",\n payload={\"region\": \"full_screen\", \"format\": \"png\"}\n)\n```\n\n---\n\n## Complete Message Flow: ConstellationClient to Device Agent\n\nThis section shows the complete end-to-end message flow from when a ConstellationClient assigns a task through the Agent Server to the final execution on a Device Agent.\n\n### Architecture Overview\n\nThe message routing follows a three-tier architecture:\n\n```\nConstellationClient (Galaxy Client)\n ↓ WebSocket + AIP\nUFOWebSocketHandler (Agent Server)\n ↓ WebSocket + AIP \nDevice Agent Client\n```\n\n**Related Documentation:**\n\n- [AIP Overview](../../aip/overview.md) - Protocol specification\n\n### Task Execution End-to-End Flow\n\nWhen ConstellationClient assigns a task to a device, the message passes through multiple layers:\n\n```mermaid\nsequenceDiagram\n participant CC as ConstellationClient\n participant DM as DeviceManager\n participant MP as MessageProcessor\n participant WS1 as WebSocket(Client→Server)\n participant Server as UFOWebSocketHandler\n participant WS2 as WebSocket(Server→Device)\n participant Device as Device Agent\n \n Note over CC,Device: 1. Task Assignment Request\n CC->>DM: assign_task_to_device(task_id, device_id, ...)\n DM->>DM: Check device status (IDLE/BUSY)\n DM->>MP: send_message(TASK)\n \n Note over CC,Device: 2. Client → Server\n MP->>MP: Create ClientMessage(type=TASK, client_type=CONSTELLATION)\n MP->>WS1: Send TASK via WebSocket\n \n Note over CC,Device: 3. Server Receives & Routes\n WS1->>Server: TASK message arrives\n Server->>Server: handle_message() parses ClientMessage\n Server->>Server: handle_task_request()\n Server->>Server: client_type=CONSTELLATION, extract target_id\n Server->>Server: Create session_id, register constellation session\n Server->>Server: Track device session mapping\n \n Note over CC,Device: 4. Server → Device\n Server->>WS2: Forward TASK to target device via AIP\n WS2->>Device: TASK message\n Device->>Device: Execute task (multiple rounds)\n \n Note over CC,Device: 5. Task Execution on Device\n loop For each action step\n Device->>WS2: Request COMMAND\n WS2->>Server: COMMAND request\n Server->>Server: Generate action commands\n Server->>WS2: COMMAND response\n WS2->>Device: Action commands\n Device->>Device: Execute commands\n Device->>WS2: COMMAND_RESULTS\n WS2->>Server: Command results\n end\n \n Note over CC,Device: 6. Task Completion Device → Server\n Device->>WS2: TASK_END (status=completed)\n WS2->>Server: Task completion\n Server->>Server: Invoke callback send_result()\n \n Note over CC,Device: 7. Server → Client (Dual Send)\n Server->>WS1: TASK_END to ConstellationClient\n Server->>WS2: TASK_END to Device Agent\n WS1->>MP: TASK_END message\n MP->>DM: Task result\n DM->>CC: ExecutionResult\n```\n\n### Message Details at Each Layer\n\n#### Layer 1: ConstellationClient to Server\n\n**ConstellationClient sends:**\n\n```json\n{\n \"type\": \"task\",\n \"client_type\": \"constellation\",\n \"client_id\": \"constellation_abc123\",\n \"target_id\": \"windows_pc\",\n \"session_id\": \"sess_xyz789\",\n \"task_name\": \"open_excel_task\",\n \"request\": \"Open Excel and create a chart\",\n \"payload\": {\n \"task_id\": \"task_001\",\n \"description\": \"Open Excel and create a chart\",\n \"data\": {\n \"file_path\": \"sales_report.xlsx\",\n \"chart_type\": \"bar\"\n }\n },\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:40:00Z\"\n}\n```\n\n**Key Fields:**\n\n- `client_type: \"constellation\"`: Identifies this as a constellation client (not device)\n- `target_id`: The device that should execute this task (e.g., \"windows_pc\")\n- `session_id`: Constellation session identifier for tracking\n- `task_name`: Human-readable task identifier\n\n#### Layer 2: Server Processing\n\nThe `UFOWebSocketHandler` receives the message and processes it:\n\n**handle_task_request() Logic:**\n\n```python\nasync def handle_task_request(self, data: ClientMessage) -> None:\n client_type = data.client_type\n client_id = data.client_id\n target_device_id = None\n \n if client_type == ClientType.CONSTELLATION:\n # Extract target device from constellation request\n target_device_id = data.target_id\n self.logger.info(f\"🌟 Constellation task for device {target_device_id}\")\n \n # Track session mapping\n self.client_manager.add_constellation_session(client_id, session_id)\n self.client_manager.add_device_session(target_device_id, session_id)\n \n # Get target device's task protocol\n target_protocol = self.client_manager.get_task_protocol(target_device_id)\n \n else:\n # Direct device request\n target_protocol = self.client_manager.get_task_protocol(client_id)\n \n # Start task in background (non-blocking)\n await self.session_manager.execute_task_async(\n session_id=session_id,\n task_protocol=target_protocol, # Send to target device\n callback=send_result # Called when task completes\n )\n```\n\n**Session Tracking:**\n\nThe server maintains two mappings:\n\n1. **Constellation Sessions**: Maps constellation_client_id → [session_ids]\n2. **Device Sessions**: Maps device_id → [session_ids]\n\nThis allows the server to:\n\n- Cancel all sessions when a constellation client disconnects\n- Cancel all sessions when a device disconnects\n- Send results to both constellation client AND device\n\n#### Layer 3: Server to Device\n\nThe server forwards the task to the target device via its WebSocket connection:\n\n**Message sent to device:**\n\n```json\n{\n \"type\": \"task\",\n \"session_id\": \"sess_xyz789\",\n \"payload\": {\n \"request\": \"Open Excel and create a chart\",\n \"task_data\": {\n \"file_path\": \"sales_report.xlsx\",\n \"chart_type\": \"bar\"\n }\n },\n \"status\": \"ok\",\n \"timestamp\": \"2025-11-06T10:40:01Z\"\n}\n```\n\nThe device receives this via its own WebSocket connection and begins execution.\n\n#### Layer 4: Task Execution and Results\n\nDuring execution, the device exchanges multiple messages with the server:\n\n**Device requests commands:**\n```json\n{\n \"type\": \"command_request\",\n \"session_id\": \"sess_xyz789\",\n \"round\": 1\n}\n```\n\n**Server responds with commands:**\n```json\n{\n \"type\": \"command\",\n \"payload\": {\n \"commands\": [\n {\"action\": \"launch_app\", \"parameters\": {\"app_name\": \"Excel\"}},\n {\"action\": \"open_file\", \"parameters\": {\"file_path\": \"sales_report.xlsx\"}}\n ]\n }\n}\n```\n\n**Device sends results:**\n```json\n{\n \"type\": \"command_results\",\n \"client_type\": \"device\",\n \"client_id\": \"windows_pc\",\n \"session_id\": \"sess_xyz789\",\n \"payload\": {\n \"results\": [\n {\"action\": \"launch_app\", \"status\": \"completed\"},\n {\"action\": \"open_file\", \"status\": \"completed\"}\n ]\n }\n}\n```\n\n**Device signals completion:**\n```json\n{\n \"type\": \"task_end\",\n \"client_type\": \"device\",\n \"client_id\": \"windows_pc\",\n \"session_id\": \"sess_xyz789\",\n \"status\": \"completed\",\n \"payload\": {\n \"result\": {\n \"success\": true,\n \"output\": \"Created bar chart in sales_report.xlsx\"\n }\n }\n}\n```\n\n#### Layer 5: Dual Result Delivery\n\nWhen the task completes, the server's callback `send_result()` sends TASK_END to **both**:\n\n1. **ConstellationClient** (the requester):\n```python\nrequester_protocol = self.client_manager.get_task_protocol(client_id)\nawait requester_protocol.send_task_end(\n session_id=session_id,\n status=result_msg.status,\n result=result_msg.result\n)\n```\n\n2. **Device Agent** (the executor):\n```python\nif client_type == ClientType.CONSTELLATION and target_device_id:\n target_protocol = self.client_manager.get_task_protocol(target_device_id)\n await target_protocol.send_task_end(\n session_id=session_id,\n status=result_msg.status,\n result=result_msg.result\n )\n```\n\nThis ensures both parties know the task completed.\n\n### Disconnection Handling\n\nThe server handles disconnections at multiple levels:\n\n**Constellation Client Disconnects:**\n\n```python\n# Cancel all sessions started by this constellation\nsession_ids = self.client_manager.get_constellation_sessions(client_id)\nfor session_id in session_ids:\n await self.session_manager.cancel_task(\n session_id, \n reason=\"constellation_disconnected\"\n )\n```\n\n**Device Disconnects:**\n\n```python\n# Cancel all sessions running on this device\nsession_ids = self.client_manager.get_device_sessions(device_id)\nfor session_id in session_ids:\n await self.session_manager.cancel_task(\n session_id,\n reason=\"device_disconnected\"\n )\n```\n\nOn the ConstellationClient side, DeviceManager detects disconnection via:\n\n- Heartbeat timeout (no response to HEARTBEAT within 10s)\n- WebSocket connection closed\n- Message send failure\n\nAnd triggers automatic reconnection with exponential backoff.\n\n### Client Type Distinction\n\nThe server handles two client types differently:\n\n| Aspect | CONSTELLATION Client | DEVICE Client |\n|--------|---------------------|---------------|\n| Task Request | Includes `target_id` field | No `target_id`, executes locally |\n| Session Tracking | Tracked in constellation_sessions | Tracked in device_sessions |\n| Result Delivery | Receives TASK_END | Receives TASK_END |\n| Disconnection | Cancels all its sessions | Cancels sessions on this device |\n\nThis allows the same server to support both direct device connections and constellation-mediated connections.\n\n---\n\n## Summary\n\nAIP integration in Galaxy Client follows a layered architecture:\n\n1. **Transport**: WebSocketConnectionManager handles raw WebSocket I/O via AIP WebSocketTransport\n2. **Protocol**: AIP protocol classes (RegistrationProtocol, TaskExecutionProtocol, HeartbeatProtocol, DeviceInfoProtocol) handle message serialization and protocol logic\n3. **Message Processing**: MessageProcessor routes messages to handlers\n4. **Application**: DeviceManager and ConstellationClient use messages for coordination\n5. **Server Routing**: UFOWebSocketHandler routes messages between constellation clients and devices\n6. **Device Execution**: Device agents execute tasks and return results\n\n**Key Message Flows:**\n\n- **Registration**: REGISTER → HEARTBEAT (OK) → DEVICE_INFO_REQUEST → DEVICE_INFO_RESPONSE\n- **Heartbeat**: HEARTBEAT (request) → HEARTBEAT (response), every 30 seconds\n- **Task Execution (Constellation)**: ConstellationClient TASK → Server routes → Device executes → Server routes → ConstellationClient TASK_END\n- **Task Execution (Direct)**: Device TASK → Server orchestrates → Device TASK_END\n\n**Error Handling:**\n\n- Connection errors trigger reconnection\n- Protocol errors send ERROR messages\n- Task errors return TASK_END with status=failed\n- Application errors use application-specific recovery\n- Disconnections cancel all associated sessions\n\n**Complete Architecture:**\n\n```\nUser Request\n ↓\nGalaxyClient (session management)\n ↓\nConstellationClient (device coordination)\n ↓\nDeviceManager (connection orchestration)\n ↓\nMessageProcessor (AIP messaging)\n ↓\nWebSocket → UFOWebSocketHandler (server routing)\n ↓\nWebSocket → Device Agent (task execution)\n```\n\nAIP provides a robust, extensible protocol for agent communication with strong typing, clear message flows, comprehensive error handling, and intelligent routing between constellation clients and devices.\n\n## Next Steps\n\n- See [DeviceManager](./device_manager.md) for connection management details\n- See [Components](./components.md) for MessageProcessor and WebSocketConnectionManager implementation\n- See [ConstellationClient](./constellation_client.md) for device coordination API\n- See [AIP Protocol Specification](../../aip/overview.md) for complete protocol reference\n- See [AIP Message Reference](../../aip/messages.md) for detailed message structures and examples\n- See [Server Documentation](../../server/websocket_handler.md) for server-side routing details\n"
},
{
"path": "documents/docs/galaxy/client/components.md",
"content": "# Galaxy Client Components\n\nGalaxy Client is built from focused, single-responsibility components that work together to provide device management capabilities. This document explains how these components interact and what each one does.\n\n## Related Documentation\n\n- [Overview](./overview.md) - Overall Galaxy Client architecture\n- [DeviceManager](./device_manager.md) - How DeviceManager orchestrates these components\n- [ConstellationClient](./constellation_client.md) - How components are used in the coordination layer\n- [AIP Integration](./aip_integration.md) - Message protocol used by components\n\n---\n\n## Component Architecture Overview\n\nGalaxy Client uses 8 modular components divided into three categories: **Device Management**, **Display & UI**, and **Support Components**. Understanding how these components work together is key to understanding Galaxy Client's design.\n\n### The Big Picture: How Components Collaborate\n\nWhen DeviceManager needs to manage a device connection, it doesn't do everything itself. Instead, it delegates specific responsibilities to specialized components:\n\n```mermaid\ngraph TB\n DM[DeviceManager Orchestrator]\n \n subgraph \"State Management\"\n DR[DeviceRegistry Device State Storage]\n end\n \n subgraph \"Connection Layer\"\n WS[WebSocketConnectionManager Network Communication]\n HM[HeartbeatManager Health Monitoring]\n MP[MessageProcessor Message Handling]\n end\n \n subgraph \"Task Layer\"\n TQ[TaskQueueManager Task Scheduling]\n end\n \n DM --> DR\n DM --> WS\n DM --> HM\n DM --> MP\n DM --> TQ\n \n WS -.->|updates| DR\n HM -.->|reads| DR\n HM -.->|uses| WS\n MP -.->|updates| DR\n MP -.->|uses| WS\n \n style DM fill:#e1f5ff\n style DR fill:#fff4e1\n```\n\nThis diagram shows the component relationships. DeviceManager acts as the orchestrator, creating and coordinating all other components. DeviceRegistry serves as the single source of truth for device state. WebSocketConnectionManager, HeartbeatManager, and MessageProcessor all depend on both DeviceRegistry (for state) and each other (for operations). TaskQueueManager works independently, managing task queues.\n\n**Key Design Principles:**\n\n1. **Single Source of Truth**: DeviceRegistry is the only component that stores device state. All other components read from or write to DeviceRegistry, never maintaining their own state.\n\n2. **Dependency Injection**: DeviceManager creates all components and injects dependencies. For example, HeartbeatManager receives references to both WebSocketConnectionManager (to send heartbeats) and DeviceRegistry (to update timestamps).\n\n3. **Background Services**: HeartbeatManager and MessageProcessor run as independent asyncio tasks. They operate continuously in the background without blocking the main execution flow.\n\n4. **Component Independence**: Each component can be tested and understood in isolation. Changing one component's implementation doesn't affect others as long as the interface remains the same.\n\n---\n\n## Device Management Components\n\nThese components handle the core device lifecycle: registration, connection, monitoring, and task execution.\n\n### DeviceRegistry: The Single Source of Truth\n\n**Purpose**: DeviceRegistry is the central repository for all device information. Every component that needs to know about device state queries DeviceRegistry.\n\n**What It Stores**: Each device is represented by an `AgentProfile` object containing:\n\n```python\n@dataclass\nclass AgentProfile:\n device_id: str # Unique device identifier\n server_url: str # WebSocket endpoint\n os: str # Operating system (windows/linux/mac)\n status: DeviceStatus # Current state (DISCONNECTED/CONNECTING/CONNECTED/IDLE/BUSY/FAILED)\n capabilities: List[str] # What the device can do ([\"office\", \"web\", \"email\"])\n metadata: Dict[str, Any] # Custom device properties\n last_heartbeat: datetime # Last successful heartbeat timestamp\n connection_attempts: int # Number of connection attempts made\n max_retries: int # Maximum reconnection attempts allowed\n current_task_id: str # Task being executed (None if idle)\n system_info: Dict # Hardware/software details from device\n```\n\nThe `status` field is particularly important as it drives the system's behavior. When a device is IDLE, it can accept new tasks. When BUSY, tasks are queued. When DISCONNECTED, reconnection is attempted.\n\n**Key Operations**:\n\n```python\n# Registration and lookup\nregistry.register_device(device_id, server_url, os, capabilities, metadata)\nprofile = registry.get_device(device_id)\nall_devices = registry.get_all_devices(connected=True)\n\n# Status management\nregistry.update_device_status(device_id, DeviceStatus.CONNECTED)\nis_busy = registry.is_device_busy(device_id)\nregistry.set_device_busy(device_id, task_id)\nregistry.set_device_idle(device_id)\n\n# Health tracking\nregistry.update_heartbeat(device_id)\nregistry.increment_connection_attempts(device_id)\nregistry.reset_connection_attempts(device_id)\n```\n\n**Why It Matters**: Having a single registry prevents state inconsistencies. Without DeviceRegistry, each component might have its own view of device state, leading to race conditions and bugs. For example, HeartbeatManager might think a device is connected while MessageProcessor thinks it's disconnected.\n\n### WebSocketConnectionManager: Network Communication Handler\n\n**Purpose**: Manages the low-level WebSocket connections to Agent Server and handles message transmission.\n\n**Connection Lifecycle**:\n\nWhen `connect_to_device()` is called, WebSocketConnectionManager performs these steps:\n\n1. **Establish WebSocket**: Creates an AIP `WebSocketTransport` and connects to the device's server_url. This is an async operation that may timeout or fail due to network issues.\n\n2. **Start Message Handler BEFORE Registration**: Crucially, this happens *before* sending REGISTER to prevent race conditions. The message handler is started via MessageProcessor to ensure we don't miss the server's response.\n\n3. **Send REGISTER**: Uses `RegistrationProtocol` to send an AIP REGISTER message identifying this client to the server. The server responds with a HEARTBEAT message with OK status to confirm registration.\n\n4. **Store Transport**: Saves the WebSocketTransport object and initializes AIP protocol handlers (`RegistrationProtocol`, `TaskExecutionProtocol`, `DeviceInfoProtocol`) for this connection.\n\n**Task Execution**:\n\nWhen sending a task to a device, WebSocketConnectionManager:\n\n```python\nasync def send_task_to_device(device_id, task_request):\n # 1. Get Transport and TaskExecutionProtocol\n transport = self._transports[device_id]\n task_protocol = self._task_protocols[device_id]\n \n # 2. Create AIP ClientMessage for task execution\n task_message = ClientMessage(\n type=ClientMessageType.TASK,\n client_type=ClientType.CONSTELLATION,\n client_id=task_client_id,\n target_id=device_id,\n task_name=f\"galaxy/{task_name}/{task_request.task_name}\",\n request=task_request.request,\n session_id=constellation_task_id,\n status=TaskStatus.CONTINUE,\n ...\n )\n \n # 3. Send message via AIP transport\n await transport.send(task_message.model_dump_json().encode(\"utf-8\"))\n \n # 4. Wait for response (handled via future)\n result = await self._wait_for_task_response(device_id, constellation_task_id)\n \n return ExecutionResult(...)\n```\n\nThe `_wait_for_task_completion()` method creates an asyncio.Future that MessageProcessor will complete when it receives the TASK_END message from the device.\n\n**Error Handling**: WebSocketConnectionManager catches connection errors (InvalidURI, WebSocketException, OSError, TimeoutError) and returns False, allowing DeviceManager to trigger reconnection logic.\n\n### HeartbeatManager: Connection Health Monitor\n\n**Purpose**: Continuously monitors device health by sending periodic heartbeat messages. This detects connection failures faster than waiting for a task to timeout.\n\n**How It Works**:\n\nFor each connected device, HeartbeatManager starts an independent background task that uses AIP `HeartbeatProtocol` to send HEARTBEAT messages periodically and verify the device is still responsive.\n\n**Timeout Detection**: Uses a timeout mechanism to detect when devices stop responding. If no heartbeat response arrives within the expected timeframe, the device is considered disconnected and HeartbeatManager triggers the disconnection handler.\n\n**Why Not Just Use TCP Keepalive?**: WebSocket runs over TCP, which has its own keepalive mechanism. However, TCP keepalive operates at a much longer timescale (typically 2 hours by default) and only detects network-level failures, not application-level hangs. HeartbeatManager detects if the device agent is responsive, not just if the TCP connection is alive.\n\n### MessageProcessor: Message Router and Handler\n\n**Purpose**: Runs a continuous message receiving loop for each device, dispatching incoming AIP messages to appropriate handlers.\n\n**The Message Loop**:\n\nMessageProcessor runs a background task that receives messages from the AIP transport and routes them based on message type. It handles `TASK_END` messages by completing the corresponding future that WebSocketConnectionManager is waiting on, enabling async task execution patterns.\n\n**Task Completion Handling**: When a TASK_END message arrives, MessageProcessor uses the `complete_task_response()` method in WebSocketConnectionManager to resolve the pending future for that task.\n\n**Why Run in Background**: The message loop runs continuously as an asyncio task. This allows it to receive messages asynchronously while the main execution flow (e.g., sending tasks) continues unblocked. Without this, we'd need to alternate between sending and receiving, making the code much more complex.\n\n### TaskQueueManager: Task Scheduling and Queuing\n\n**Purpose**: Manages per-device task queues, ensuring tasks execute sequentially when devices are busy.\n\n**Queue Behavior**:\n\nWhen a task is assigned to a device that's already executing another task:\n\n```python\n# In DeviceManager.assign_task_to_device()\nif self.device_registry.is_device_busy(device_id):\n # Device is BUSY - enqueue task\n future = self.task_queue_manager.enqueue_task(device_id, task_request)\n # Wait for task to complete\n result = await future\n return result\nelse:\n # Device is IDLE - execute immediately\n return await self._execute_task_on_device(device_id, task_request)\n```\n\n**How Queuing Works**:\n\nTaskQueueManager maintains a dictionary of queues: `{device_id: queue}`. Each queue is a list of `(task_request, future)` tuples. When a task is enqueued:\n\n```python\ndef enqueue_task(device_id, task_request):\n # Create a future for this task\n future = asyncio.Future()\n \n # Add to device's queue\n self.queues[device_id].append((task_request, future))\n \n # Return future so caller can await result\n return future\n```\n\nWhen a device completes a task and becomes IDLE, DeviceManager calls:\n\n```python\nasync def _process_next_queued_task(device_id):\n if self.task_queue_manager.has_queued_tasks(device_id):\n task_request = self.task_queue_manager.dequeue_task(device_id)\n # Execute next task (don't await to avoid blocking)\n asyncio.create_task(self._execute_task_on_device(device_id, task_request))\n```\n\n**Why Futures?**: Using asyncio.Future allows the calling code to await task completion even though the task is queued. The caller doesn't need to know whether the task executed immediately or was queued—it just awaits the future and gets the result when ready.\n\n---\n\n## Display Component\n\n### ClientDisplay: User Interface and Console Output\n\n**Purpose**: Provides Rich-based console output for interactive mode and status reporting. This component is only used by GalaxyClient, not by ConstellationClient or DeviceManager.\n\n**Key Features**:\n\n**Banner and Branding**: Shows ASCII art banner when GalaxyClient starts, creating a visual identity for the framework.\n\n**Progress Indication**: Uses Rich Progress bars for long-running operations like initialization:\n\n```python\nwith display.show_initialization_progress() as progress:\n task = progress.add_task(\"[cyan]Initializing...\", total=None)\n # ... initialization work ...\n progress.update(task, description=\"[green]Complete!\")\n```\n\n**Result Display**: Formats execution results in readable tables:\n\n```python\ndisplay.display_result({\n \"status\": \"completed\",\n \"execution_time\": 23.45,\n \"rounds\": 2,\n \"constellation\": {\"task_count\": 5}\n})\n```\n\nThis creates a formatted table showing status, time, rounds, and task count in color-coded output.\n\n**Interactive Input**: Provides user input prompts with styling:\n\n```python\nuser_input = display.get_user_input(\"UFO[0]\")\n```\n\n**Colored Messages**: Semantic color coding for different message types:\n- Green (success): Task completed, connection established\n- Red (error): Task failed, connection error\n- Yellow (warning): Device disconnected, timeout\n- Cyan (info): Status updates, progress\n\n**Why Separate Component?**: Keeping display logic separate from business logic makes it easy to replace or disable. For example, a web-based frontend could replace ClientDisplay without touching any other components.\n\n---\n\n## Support Components\n\nThese components support higher-level client operations by providing status aggregation and configuration management capabilities.\n\n### StatusManager: System-Wide Status Aggregation\n\n**Purpose**: Provides consolidated views of system health and performance across all devices. While DeviceRegistry stores individual device status, StatusManager aggregates this into system-wide metrics.\n\n**Health Summary Example**:\n\n```python\nsummary = status_manager.get_device_health_summary()\n# Returns:\n{\n \"total_devices\": 5,\n \"connected_devices\": 3,\n \"disconnected_devices\": 2,\n \"connection_rate\": 0.6, # 60% connected\n \"devices_by_status\": {\n \"CONNECTED\": 2,\n \"IDLE\": 1,\n \"DISCONNECTED\": 1,\n \"FAILED\": 1\n },\n \"devices_with_issues\": [\n {\n \"device_id\": \"device_3\",\n \"issue\": \"multiple_connection_attempts\",\n \"attempts\": 4,\n \"max_retries\": 5\n }\n ]\n}\n```\n\n**Task Statistics**:\n\n```python\nstats = status_manager.get_task_statistics()\n# Returns:\n{\n \"total_tasks_executed\": 127,\n \"successful_tasks\": 120,\n \"failed_tasks\": 7,\n \"success_rate\": 0.945,\n \"average_execution_time\": 15.3, # seconds\n \"tasks_by_device\": {\n \"windows_pc\": 65,\n \"linux_server\": 62\n }\n}\n```\n\n**Why This Matters**: In production, you need to monitor system health. StatusManager provides the data needed for dashboards, alerts, and capacity planning. For example, if connection_rate drops below 80%, you might trigger an alert.\n\n---\n\n## How Components Work Together: A Complete Example\n\nLet's trace what happens when you call `device_manager.connect_device(\"windows_pc\")`:\n\n**Step 1: DeviceManager Initiates Connection**\n\n```python\n# DeviceManager.connect_device()\ndevice_info = self.device_registry.get_device(device_id) # Get device details\nself.device_registry.update_device_status(device_id, DeviceStatus.CONNECTING) # Update status\n```\n\n**Step 2: WebSocketConnectionManager Establishes Connection**\n\n```python\n# WebSocketConnectionManager.connect_to_device()\ntransport = WebSocketTransport(...)\nawait transport.connect(device_info.server_url) # Create AIP transport\nself._transports[device_id] = transport # Store transport\n\n# Initialize AIP protocols for this connection\nself._registration_protocols[device_id] = RegistrationProtocol(transport)\nself._task_protocols[device_id] = TaskExecutionProtocol(transport)\nself._device_info_protocols[device_id] = DeviceInfoProtocol(transport)\n\n# ⚠️ CRITICAL: Start message handler BEFORE sending registration\n# This ensures we don't miss the server's registration response\nself.message_processor.start_message_handler(device_id, transport)\nawait asyncio.sleep(0.05) # Small delay to ensure handler is listening\n\n# Register as constellation client using AIP RegistrationProtocol\nawait self._register_constellation_client(device_info)\n```\n\n**Step 3: MessageProcessor Starts Background Loop**\n\n```python\n# MessageProcessor.start_message_handler()\ntask = asyncio.create_task(self._handle_device_messages(device_id, transport))\nself._message_handlers[device_id] = task # Store task for later cancellation\n```\n\nNow MessageProcessor is running in the background, ready to receive messages via the AIP transport.\n\n**Step 4: Device Registration Completes**\n\nThe device sends back HEARTBEAT with OK status (which serves as registration confirmation). Then WebSocketConnectionManager requests device info via `DeviceInfoProtocol`.\n\n**Step 5: DeviceRegistry Updated with System Info**\n\n```python\n# DeviceManager.connect_device() continues\nself.device_registry.update_device_system_info(device_id, device_system_info)\nself.device_registry.update_device_status(device_id, DeviceStatus.CONNECTED)\nself.device_registry.set_device_idle(device_id) # Ready for tasks\n```\n\n**Step 6: HeartbeatManager Starts Monitoring**\n\n```python\n# HeartbeatManager.start_heartbeat()\ntask = asyncio.create_task(self._send_heartbeat_loop(device_id))\nself.heartbeat_tasks[device_id] = task\n```\n\nNow HeartbeatManager is running in the background, sending heartbeats every 30 seconds.\n\n**Step 7: Connection Complete**\n\nAll components are now working together:\n- DeviceRegistry knows the device is IDLE and ready\n- WebSocketConnectionManager has an active AIP Transport with initialized protocols\n- MessageProcessor is listening for incoming messages via the transport\n- HeartbeatManager is monitoring connection health\n- TaskQueueManager is ready to queue tasks if device becomes busy\n\nThis coordinated setup ensures reliable device communication.\n\n---\n\n## Component Dependencies\n\nUnderstanding component dependencies helps when debugging or extending the system:\n\n```\nDeviceManager (creates all components)\n├── DeviceRegistry (no dependencies - foundational)\n├── WebSocketConnectionManager (depends on: DeviceRegistry for task name)\n├── HeartbeatManager (depends on: WebSocketConnectionManager, DeviceRegistry)\n├── MessageProcessor (depends on: DeviceRegistry, HeartbeatManager, WebSocketConnectionManager)\n└── TaskQueueManager (no dependencies - independent)\n```\n\n**Construction Order**: DeviceManager must create components in dependency order:\n\n```python\ndef __init__(self, task_name, heartbeat_interval, reconnect_delay):\n # 1. DeviceRegistry first (no dependencies)\n self.device_registry = DeviceRegistry()\n \n # 2. WebSocketConnectionManager (needs task_name only)\n self.connection_manager = WebSocketConnectionManager(task_name)\n \n # 3. HeartbeatManager (depends on connection_manager and device_registry)\n self.heartbeat_manager = HeartbeatManager(\n self.connection_manager,\n self.device_registry,\n heartbeat_interval\n )\n \n # 4. MessageProcessor (depends on all previous components)\n self.message_processor = MessageProcessor(\n self.device_registry,\n self.heartbeat_manager,\n self.connection_manager\n )\n \n # 5. TaskQueueManager (independent)\n self.task_queue_manager = TaskQueueManager()\n```\n\n**Why This Order Matters**: If we created MessageProcessor before HeartbeatManager, we'd get an error because MessageProcessor's constructor expects HeartbeatManager to exist. The dependency graph dictates construction order.\n\n---\n\n## Testing Components\n\nThe modular design makes components easy to test in isolation:\n\n**Testing DeviceRegistry**:\n\n```python\n# No external dependencies needed\nregistry = DeviceRegistry()\nregistry.register_device(\"test_device\", \"ws://localhost:5000\", \"windows\", [\"test\"])\nassert registry.is_device_registered(\"test_device\")\n```\n\n**Testing WebSocketConnectionManager**:\n\n```python\n# Mock the WebSocket connection\nmock_websocket = AsyncMock()\nconnection_manager = WebSocketConnectionManager(\"test\")\nconnection_manager.connections[\"test_device\"] = mock_websocket\n\n# Test message sending\nawait connection_manager.send_task_to_device(\"test_device\", task_request)\nmock_websocket.send.assert_called_once()\n```\n\n**Testing HeartbeatManager**:\n\n```python\n# Inject mock dependencies\nmock_connection_manager = Mock()\nmock_registry = Mock()\nheartbeat_manager = HeartbeatManager(mock_connection_manager, mock_registry, 30.0)\n\n# Test heartbeat loop\nheartbeat_manager.start_heartbeat(\"test_device\")\nawait asyncio.sleep(0.1) # Let loop run\nassert mock_connection_manager.get_connection.called\n```\n\n**Why Testability Matters**: Complex systems are hard to test. By breaking DeviceManager into 5 focused components, we can write targeted unit tests for each component's specific behavior, making bugs easier to find and fix.\n\n---\n\n## Summary\n\nGalaxy Client's component architecture demonstrates several important design principles:\n\n**Single Responsibility**: Each component does one thing well. DeviceRegistry stores state, WebSocketConnectionManager handles networking, HeartbeatManager monitors health, MessageProcessor routes messages, TaskQueueManager manages queues.\n\n**Dependency Injection**: DeviceManager creates components and injects dependencies, making the system flexible and testable. Want to replace WebSocketConnectionManager with a different implementation? Just swap it out while keeping the interface.\n\n**Separation of Concerns**: Business logic (in DeviceManager) is separate from display logic (in ClientDisplay) and orchestration support (in StatusManager). Each layer can evolve independently.\n\n**Asynchronous Background Services**: HeartbeatManager and MessageProcessor run as independent asyncio tasks, enabling concurrent operations without blocking the main execution flow.\n\nThis design makes Galaxy Client maintainable, extensible, and testable. When you understand how components collaborate, you can confidently modify or extend the system.\n\n## Related Documentation\n\n- [DeviceManager Reference](./device_manager.md) - See how DeviceManager orchestrates these components\n- [ConstellationClient](./constellation_client.md) - Learn how components are used in the coordination layer\n- [Overview](./overview.md) - Understand the broader Galaxy Client architecture\n- [AIP Integration](./aip_integration.md) - Learn about the message protocol components use\n- [DeviceRegistry Details](../agent_registration/device_registry.md) - Deep dive into device state management\n"
},
{
"path": "documents/docs/galaxy/client/constellation_client.md",
"content": "# ConstellationClient Reference\n\nConstellationClient is the device coordination layer in Galaxy Client. It provides a clean API for registering devices, managing connections, and assigning tasks. Most applications interact with ConstellationClient rather than the lower-level DeviceManager.\n\n## Related Documentation\n\n- [Overview](./overview.md) - Overall architecture and workflow\n- [DeviceManager](./device_manager.md) - Internal connection management\n- [Components](./components.md) - Modular component details\n- [Configuration](../../configuration/system/galaxy_constellation.md) - Device configuration\n- [GalaxyClient](./galaxy_client.md) - Session wrapper on top of ConstellationClient\n\n## What ConstellationClient Does\n\nConstellationClient implements the Facade pattern, providing a simplified interface to the complex device management system underneath. Think of it as the \"device management API\" for Galaxy.\n\n**Core Responsibilities:**\n\n**Device Lifecycle Management**: ConstellationClient handles the complete lifecycle of device connections. When you register a device, it stores the device information (ID, server URL, capabilities) in DeviceRegistry. When you connect, it coordinates with DeviceManager to establish WebSocket connections, perform AIP registration, and start health monitoring. When you disconnect, it cleanly tears down all resources.\n\n**Task Assignment**: When you have a task to execute, ConstellationClient determines which device should run it (based on capabilities), checks if the device is available, and delegates to DeviceManager for actual execution. It abstracts away details like task queuing when devices are busy or handling connection failures during execution.\n\n**Configuration Management**: ConstellationClient loads device configurations from YAML files or programmatic APIs, validates settings, and maintains the runtime configuration. This centralizes all configuration logic so other components don't need to worry about it.\n\n**Status Reporting**: Applications need to know what's happening with devices. ConstellationClient provides methods to query device status, get health summaries, and retrieve execution statistics. This information is aggregated from multiple components (DeviceRegistry, DeviceManager, TaskQueueManager) and presented in a unified format.\n\n**What ConstellationClient Does NOT Do:**\n\n- **DAG Planning**: Task decomposition is handled by ConstellationAgent\n- **DAG Execution**: Coordinating task dependencies is handled by TaskConstellationOrchestrator \n- **Session Management**: Multi-round interactions are handled by GalaxySession\n- **Low-Level Connection Management**: WebSocket lifecycle is handled by DeviceManager\n\nThis separation of concerns keeps ConstellationClient focused on device-level operations.\n\n## Initialization\n\n### Constructor\n\n```python\ndef __init__(\n self,\n config: Optional[ConstellationConfig] = None,\n task_name: Optional[str] = None,\n):\n \"\"\"\n Initialize ConstellationClient with configuration.\n \n Args:\n config: Device configuration (creates default if None)\n task_name: Override task name from config\n \"\"\"\n```\n\nWhen you create a ConstellationClient, it performs these initialization steps:\n\n1. **Load or Create Configuration**: If you provide a `config` parameter, it uses that. Otherwise, it creates a default `ConstellationConfig` object. This config contains device information, heartbeat settings, and other parameters.\n\n2. **Override Task Name**: If you provide `task_name`, it overrides the task name from the configuration. The task name identifies this constellation instance in logs and messages.\n\n3. **Create DeviceManager**: ConstellationClient creates an internal DeviceManager instance, passing the task name and connection settings (heartbeat interval, reconnect delay). DeviceManager is the component that actually manages connections.\n\n**Initialization Examples:**\n\n```python\n# Simple: Use default configuration\nclient = ConstellationClient()\n\n# Load configuration from YAML\nconfig = ConstellationConfig.from_yaml(\"config/devices.yaml\")\nclient = ConstellationClient(config=config)\n\n# Override task name for this instance\nclient = ConstellationClient(\n config=config,\n task_name=\"data_processing_pipeline\"\n)\n```\n\nThe task name appears in logs and helps identify which constellation instance generated which messages, which is useful when running multiple constellations simultaneously.\n\n### Async Initialize Method\n\n```python\nasync def initialize(self) -> Dict[str, bool]:\n \"\"\"\n Register and optionally connect all devices from configuration.\n \n Returns:\n Dictionary mapping device_id to registration success status\n \"\"\"\n```\n\nAfter creating a ConstellationClient, you must call `initialize()` before using it. This method processes all devices defined in the configuration:\n\n**Registration Process:**\n\nFor each device in the configuration, `initialize()` calls `register_device_from_config()`, which:\n\n1. Extracts device parameters (device_id, server_url, os, capabilities, metadata)\n2. Calls DeviceManager to register the device\n3. If `auto_connect: true` is set, immediately connects to the device\n\n**Auto-Connect Behavior:**\n\nThe `auto_connect` flag in configuration determines whether devices connect during initialization or wait for explicit `connect_device()` calls. Auto-connect is convenient for simple scenarios but may not be suitable if you need fine-grained control over connection timing.\n\n**Return Value:**\n\nThe method returns a dictionary showing which devices successfully registered:\n\n```python\nresults = await client.initialize()\n# Example: {\"windows_pc\": True, \"linux_server\": True, \"failed_device\": False}\n\n# Check for failures\nfailed = [device_id for device_id, success in results.items() if not success]\nif failed:\n print(f\"Failed to register: {failed}\")\n```\n\n**Typical Initialization Flow:**\n\n```mermaid\nsequenceDiagram\n participant App\n participant CC as ConstellationClient\n participant DM as DeviceManager\n participant Server as Agent Server\n \n App->>CC: ConstellationClient(config)\n CC->>CC: Create DeviceManager\n \n App->>CC: initialize()\n \n loop For each device in config\n CC->>DM: register_device()\n DM->>DM: Store in DeviceRegistry\n \n alt auto_connect = true\n DM->>Server: WebSocket connect\n Server-->>DM: Connection established\n DM->>Server: REGISTER (AIP)\n Server-->>DM: REGISTER_CONFIRMATION\n DM->>Server: DEVICE_INFO_REQUEST\n Server-->>DM: Device telemetry\n DM->>DM: Start heartbeat & message handler\n end\n \n DM-->>CC: Success/failure\n end\n \n CC-->>App: {\"device1\": true, \"device2\": true}\n```\n\nThis diagram shows the initialization sequence. For each configured device, ConstellationClient delegates to DeviceManager, which handles the low-level connection setup if auto-connect is enabled.\n\n## Device Management Methods\n\n### Register Device\n\n```python\nasync def register_device(\n self,\n device_id: str,\n server_url: str,\n capabilities: Optional[List[str]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n auto_connect: bool = True,\n) -> bool:\n```\n\nThis method registers a device programmatically (outside of configuration). It's useful for dynamically adding devices at runtime.\n\n!!! warning \"Known Limitation\"\n The current implementation does not pass the OS parameter to the underlying `DeviceManager`. For proper device registration with OS information, use configuration-based registration via `register_device_from_config()` or ensure the OS is included in the device metadata.\n\n**Parameters Explained:**\n\n- **device_id**: Unique identifier for the device. Used in all subsequent operations.\n- **server_url**: WebSocket endpoint of the Agent Server (e.g., `ws://192.168.1.100:5000/ws`)\n- **capabilities**: List of capabilities this device provides (e.g., `[\"office\", \"web\", \"email\"]`)\n- **metadata**: Additional device properties (e.g., `{\"location\": \"datacenter\", \"gpu\": \"RTX 4090\"}`)\n- **auto_connect**: Whether to immediately connect after registration\n\n**Usage Example:**\n\n```python\n# Register a Windows device with Office capabilities\nsuccess = await client.register_device(\n device_id=\"workstation_001\",\n server_url=\"ws://192.168.1.50:5000/ws\",\n capabilities=[\"office\", \"web\", \"email\"],\n metadata={\"location\": \"office\", \"user\": \"john\"},\n auto_connect=True\n)\n\nif success:\n print(\"Device registered and connected\")\nelse:\n print(\"Registration failed\")\n```\n\n### Connect and Disconnect\n\n```python\nasync def connect_device(self, device_id: str) -> bool:\n \"\"\"Connect to a registered device.\"\"\"\n\nasync def disconnect_device(self, device_id: str) -> bool:\n \"\"\"Disconnect from a device.\"\"\"\n \nasync def connect_all_devices(self) -> Dict[str, bool]:\n \"\"\"Connect to all registered devices.\"\"\"\n \nasync def disconnect_all_devices(self) -> None:\n \"\"\"Disconnect from all devices.\"\"\"\n```\n\nThese methods control device connections. You might disconnect devices to save resources or reconnect after configuration changes.\n\n**Connection Example:**\n\n```python\n# Connect to specific device\nawait client.connect_device(\"windows_pc\")\n\n# Connect to all registered devices\nresults = await client.connect_all_devices()\nprint(f\"Connected to {sum(results.values())} devices\")\n\n# Disconnect when done\nawait client.disconnect_device(\"windows_pc\")\n```\n\nConnection establishment involves WebSocket handshake, AIP registration, device info exchange, and starting background monitoring services (heartbeat and message processing).\n\n## Task Execution\n\n### Assign Task to Device\n\nWhile ConstellationClient doesn't expose a direct `assign_task_to_device()` method in its public API (that's internal to DeviceManager), it's used by higher-level orchestrators like TaskConstellationOrchestrator. Understanding how task assignment works helps you understand the system:\n\n**Task Assignment Process:**\n\n1. **Device Status Check**: DeviceManager checks if the target device is IDLE or BUSY\n2. **Immediate Execution**: If IDLE, the task executes immediately\n3. **Queuing**: If BUSY, the task enters the device's queue\n4. **Task Transmission**: WebSocketConnectionManager sends TASK message via AIP\n5. **Result Waiting**: MessageProcessor waits for TASK_END message\n6. **Completion**: Device returns to IDLE, next queued task starts\n\n**Why Task Assignment is Internal:**\n\nConstellationClient focuses on device management, not task orchestration. Task assignment is exposed through higher-level APIs:\n\n- TaskConstellationOrchestrator assigns tasks based on DAG dependencies\n- GalaxySession coordinates multi-round task execution\n- Direct device-level task assignment is available through DeviceManager if needed\n\nThis layering ensures each component has a clear responsibility.\n\n## Status and Information\n\n### Get Device Status\n\n```python\ndef get_device_status(self, device_id: Optional[str] = None) -> Dict[str, Any]:\n \"\"\"\n Get device status information.\n \n If device_id is provided, returns status for that device.\n If device_id is None, returns status for all connected devices.\n \"\"\"\n```\n\nDevice status includes:\n\n```python\n{\n \"device_id\": \"windows_pc\",\n \"status\": \"IDLE\", # DISCONNECTED/CONNECTING/CONNECTED/IDLE/BUSY/FAILED\n \"server_url\": \"ws://192.168.1.100:5000/ws\",\n \"capabilities\": [\"office\", \"web\"],\n \"last_heartbeat\": \"2025-11-06T10:30:45\",\n \"connection_attempts\": 1,\n \"max_retries\": 5,\n \"current_task_id\": None, # Task ID if device is BUSY\n \"queued_tasks\": 0, # Number of queued tasks\n \"system_info\": { # From device telemetry\n \"cpu_count\": 8,\n \"memory_gb\": 32,\n \"os_version\": \"Windows 11\",\n ...\n }\n}\n```\n\nThe status provides a comprehensive view of device health and activity, useful for monitoring dashboards or debugging connection issues.\n\n### Get Connected Devices\n\n```python\ndef get_connected_devices(self) -> List[str]:\n \"\"\"Get list of device IDs that are currently connected.\"\"\"\n```\n\nReturns a list of device IDs in CONNECTED, IDLE, or BUSY status. Useful for determining which devices are available for task assignment.\n\n```python\nconnected = client.get_connected_devices()\nprint(f\"Available devices: {', '.join(connected)}\")\n\n# Check if specific device is connected\nif \"windows_pc\" in connected:\n # Assign task to this device\n ...\n```\n\n### Get Constellation Info\n\n```python\ndef get_constellation_info(self) -> Dict[str, Any]:\n \"\"\"Get overall constellation status and configuration.\"\"\"\n```\n\nReturns constellation-level information:\n\n```python\n{\n \"constellation_id\": \"production_constellation\",\n \"connected_devices\": 3, # Number currently connected\n \"total_devices\": 5, # Total registered devices\n \"configuration\": {\n \"heartbeat_interval\": 30.0,\n \"reconnect_delay\": 5.0,\n \"max_concurrent_tasks\": 10\n }\n}\n```\n\nThis provides a high-level view of the entire constellation, useful for monitoring overall system health.\n\n## Configuration Management\n\n### Validate Configuration\n\n```python\ndef validate_config(self, config: Optional[ConstellationConfig] = None) -> Dict[str, Any]:\n \"\"\"\n Validate constellation configuration.\n \n Checks:\n - task_name is provided\n - devices are configured\n - settings are in valid ranges\n \"\"\"\n```\n\nValidation catches configuration errors early:\n\n```python\nresult = client.validate_config()\n\nif not result[\"valid\"]:\n print(\"Configuration errors:\")\n for error in result[\"errors\"]:\n print(f\" - {error}\")\n \nif result[\"warnings\"]:\n print(\"Warnings:\")\n for warning in result[\"warnings\"]:\n print(f\" - {warning}\")\n```\n\n### Get Configuration Summary\n\n```python\ndef get_config_summary(self) -> Dict[str, Any]:\n \"\"\"Get summary of current configuration.\"\"\"\n```\n\nReturns a human-readable configuration summary:\n\n```python\n{\n \"task_name\": \"production_constellation\",\n \"devices_count\": 3,\n \"devices\": [\n {\n \"device_id\": \"windows_pc\",\n \"server_url\": \"ws://192.168.1.100:5000/ws\",\n \"capabilities\": [\"office\", \"web\"],\n \"auto_connect\": true\n },\n ...\n ],\n \"settings\": {\n \"heartbeat_interval\": 30.0,\n \"reconnect_delay\": 5.0,\n \"max_concurrent_tasks\": 10\n }\n}\n```\n\n### Add Device to Configuration\n\n```python\nasync def add_device_to_config(\n self,\n device_id: str,\n server_url: str,\n capabilities: Optional[List[str]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n auto_connect: bool = True,\n register_immediately: bool = True,\n) -> bool:\n```\n\nDynamically adds a device to the configuration and optionally registers it:\n\n```python\n# Add device to config and register\nawait client.add_device_to_config(\n device_id=\"new_device\",\n server_url=\"ws://192.168.1.200:5000/ws\",\n capabilities=[\"database\"],\n register_immediately=True # Register right away\n)\n\n# Add to config only, register later\nawait client.add_device_to_config(\n device_id=\"staging_device\",\n server_url=\"ws://staging.example.com:5000/ws\",\n register_immediately=False # Just update config\n)\n```\n\nThis is useful for dynamic device discovery scenarios where devices are added at runtime.\n\n## Lifecycle Management\n\n### Shutdown\n\n```python\nasync def shutdown(self) -> None:\n \"\"\"\n Gracefully shutdown the constellation client.\n \n Stops all background services and disconnects all devices.\n \"\"\"\n```\n\nShutdown performs cleanup in this order:\n\n1. **Stop Task Queues**: Cancel all queued tasks across all devices\n2. **Stop Message Handlers**: Stop MessageProcessor loops for all devices\n3. **Stop Heartbeats**: Stop HeartbeatManager loops for all devices\n4. **Disconnect Devices**: Close WebSocket connections to all devices\n5. **Cancel Reconnection Tasks**: Cancel any pending reconnection attempts\n\n**Proper Shutdown Example:**\n\n```python\ntry:\n client = ConstellationClient(config)\n await client.initialize()\n \n # Use the client\n ...\n \nfinally:\n # Always shutdown to cleanup resources\n await client.shutdown()\n```\n\nWithout proper shutdown, background tasks continue running, WebSocket connections remain open, and resources leak.\n\n## Usage Patterns\n\n### Basic Device Management\n\n```python\n# Create and initialize client\nclient = ConstellationClient()\nawait client.initialize()\n\n# Check which devices connected\nconnected = client.get_connected_devices()\nprint(f\"Connected: {connected}\")\n\n# Get status for specific device\nstatus = client.get_device_status(\"windows_pc\")\nprint(f\"Status: {status['status']}, Tasks queued: {status['queued_tasks']}\")\n\n# Shutdown when done\nawait client.shutdown()\n```\n\n### Dynamic Device Addition\n\n```python\n# Start with base configuration\nclient = ConstellationClient(base_config)\nawait client.initialize()\n\n# Discover new device at runtime\nnew_device_info = await discover_device()\n\n# Add and connect\nawait client.add_device_to_config(\n device_id=new_device_info[\"id\"],\n server_url=new_device_info[\"url\"],\n capabilities=new_device_info[\"capabilities\"],\n register_immediately=True\n)\n\n# Verify connection\nif new_device_info[\"id\"] in client.get_connected_devices():\n print(\"New device ready\")\n```\n\n### Health Monitoring\n\n```python\nimport asyncio\n\nasync def monitor_health(client):\n \"\"\"Continuously monitor device health.\"\"\"\n while True:\n info = client.get_constellation_info()\n \n # Check connection rate\n connection_rate = info[\"connected_devices\"] / info[\"total_devices\"]\n if connection_rate < 0.8: # Less than 80% connected\n print(f\"Warning: Only {connection_rate:.0%} devices connected\")\n \n # Check individual device health\n for device_id in client.get_connected_devices():\n status = client.get_device_status(device_id)\n \n # Check heartbeat freshness\n last_hb = datetime.fromisoformat(status[\"last_heartbeat\"])\n age = datetime.now() - last_hb\n if age.total_seconds() > 60: # No heartbeat in 60 seconds\n print(f\"Warning: {device_id} heartbeat stale\")\n \n await asyncio.sleep(30) # Check every 30 seconds\n```\n\n## Integration with Other Components\n\n### Used by GalaxyClient\n\nGalaxyClient wraps ConstellationClient for session management:\n\n```python\nclass GalaxyClient:\n def __init__(self, ...):\n # Create internal ConstellationClient\n self._client = ConstellationClient(config, task_name)\n \n async def initialize(self):\n # Initialize ConstellationClient\n await self._client.initialize()\n \n async def process_request(self, request):\n # Use ConstellationClient for device coordination\n # while GalaxySession handles task orchestration\n session = GalaxySession(client=self._client, ...)\n await session.run()\n```\n\n### Used by TaskConstellationOrchestrator\n\nTaskConstellationOrchestrator uses ConstellationClient's DeviceManager for task assignment:\n\n```python\n# Orchestrator assigns tasks to devices based on capabilities\nfor task in dag.tasks:\n device_id = select_device_for_task(task)\n \n # Assign through DeviceManager (internal to ConstellationClient)\n result = await constellation_client.device_manager.assign_task_to_device(\n task_id=task.id,\n device_id=device_id,\n task_description=task.description,\n task_data=task.data\n )\n```\n\n## Summary\n\nConstellationClient is the primary interface for device management in Galaxy Client. It provides:\n\n- **Simple API**: Clean methods for registration, connection, status queries\n- **Configuration Management**: Load from files, validate, modify at runtime\n- **Delegation**: Hides complexity of DeviceManager and its components\n- **Focused Scope**: Device management only, not DAG planning or session management\n\nFor most applications, ConstellationClient (or GalaxyClient which wraps it) is all you need. Only advanced scenarios require working directly with DeviceManager or its components.\n\n**Next Steps:**\n\n- See [DeviceManager](./device_manager.md) for low-level connection management details\n- See [Components](./components.md) for modular component architecture\n- See [Overview](./overview.md) for overall system architecture\n- See [GalaxyClient](./galaxy_client.md) for session-level API\n"
},
{
"path": "documents/docs/galaxy/client/device_manager.md",
"content": "# DeviceManager Reference\n\nDeviceManager is the connection orchestration layer in Galaxy Client. While ConstellationClient provides the high-level device management API, DeviceManager handles the low-level details of WebSocket connections, health monitoring, message routing, and task queuing.\n\n## Related Documentation\n\n- [Overview](./overview.md) - Overall Galaxy Client architecture and workflow\n- [ConstellationClient](./constellation_client.md) - High-level device management API\n- [Components](./components.md) - Detailed documentation for each DeviceManager component\n- [AIP Integration](./aip_integration.md) - Protocol details and message flows\n\n---\n\n## What DeviceManager Does\n\nDeviceManager acts as the orchestration coordinator, managing the lifecycle of device connections from initial registration through task execution to disconnection. It doesn't perform these operations itself; instead, it coordinates five specialized components to handle different aspects of device management.\n\n**Orchestration Philosophy:**\n\nDeviceManager follows the Coordinator pattern. When you call `register_device()`, DeviceManager doesn't directly store device information—it delegates to DeviceRegistry. When you call `connect_device()`, DeviceManager doesn't create WebSocket connections itself—it delegates to WebSocketConnectionManager. When a device sends a message, DeviceManager doesn't process it—MessageProcessor handles that.\n\nThis separation of concerns makes each component focused and testable. DeviceManager simply coordinates the flow of operations across components.\n\n**Core Responsibilities:**\n\n**Device Registration**: When a device registers, DeviceManager creates an AgentProfile containing device metadata (ID, server URL, capabilities, OS) and delegates to DeviceRegistry for storage. DeviceRegistry becomes the single source of truth for device state.\n\n**Connection Establishment**: When you connect to a device, DeviceManager coordinates multiple steps: WebSocketConnectionManager establishes the WebSocket connection, MessageProcessor sends the REGISTER message per AIP protocol, DeviceManager requests device telemetry, and HeartbeatManager starts background health monitoring.\n\n**Disconnection Handling**: When a device disconnects (intentionally or due to failure), DeviceManager coordinates cleanup: HeartbeatManager stops health checks, MessageProcessor stops the message handling loop, WebSocketConnectionManager closes the WebSocket, TaskQueueManager clears pending tasks, and DeviceRegistry updates device status.\n\n**Reconnection Logic**: For network failures, DeviceManager implements exponential backoff reconnection. It tracks connection attempts, waits progressively longer between retries (5s, 10s, 20s, ...), and gives up after max retries. Reconnection happens automatically without user intervention.\n\n**Task Assignment Coordination**: When assigning a task, DeviceManager checks device status via DeviceRegistry, queues tasks via TaskQueueManager if the device is busy, and delegates execution to MessageProcessor when the device becomes available.\n\n**What DeviceManager Does NOT Do:**\n\n- **WebSocket I/O**: Handled by WebSocketConnectionManager\n- **Health Monitoring**: Handled by HeartbeatManager \n- **Message Processing**: Handled by MessageProcessor\n- **Device State Storage**: Handled by DeviceRegistry\n- **Task Queuing**: Handled by TaskQueueManager\n\nDeviceManager coordinates these components but doesn't duplicate their functionality.\n\n---\n\n## Component Architecture\n\nDeviceManager uses a modular architecture with five components, each responsible for a specific aspect of device management:\n\n```\nDeviceManager (Orchestrator)\n |\n +-- DeviceRegistry (Device State)\n | Stores AgentProfiles, device status\n |\n +-- WebSocketConnectionManager (Connection Lifecycle)\n | Establishes/closes WebSocket connections\n |\n +-- HeartbeatManager (Health Monitoring)\n | Sends periodic heartbeats, detects failures\n |\n +-- MessageProcessor (Message Routing)\n | Routes AIP messages, handles responses\n |\n +-- TaskQueueManager (Task Queuing)\n Queues tasks when devices busy\n```\n\n**Why This Architecture?**\n\n**Single Responsibility**: Each component has one job. DeviceRegistry manages state, WebSocketConnectionManager manages connections, HeartbeatManager monitors health. This makes each component easy to understand, test, and modify.\n\n**Testability**: You can test each component in isolation. Mock DeviceRegistry to test connection logic. Mock WebSocketConnectionManager to test message processing. This simplifies unit testing.\n\n**Extensibility**: Adding new functionality means adding or modifying a single component. Need different health monitoring? Replace HeartbeatManager. Need different queuing strategies? Modify TaskQueueManager. Other components remain unchanged.\n\n**Clarity**: When debugging, you know where to look. Connection failures? Check WebSocketConnectionManager. Missed heartbeats? Check HeartbeatManager. Status inconsistencies? Check DeviceRegistry.\n\n**Component Interactions:**\n\nComponents interact through DeviceManager as the coordinator:\n\n1. **Registration Flow**: DeviceManager → DeviceRegistry (store profile)\n2. **Connection Flow**: DeviceManager → WebSocketConnectionManager (connect) → MessageProcessor (send REGISTER) → DeviceRegistry (update status) → HeartbeatManager (start monitoring)\n3. **Task Assignment Flow**: DeviceManager → DeviceRegistry (check status) → TaskQueueManager (queue if busy) → MessageProcessor (send TASK)\n4. **Disconnection Flow**: DeviceManager → HeartbeatManager (stop) → MessageProcessor (stop) → WebSocketConnectionManager (close) → TaskQueueManager (clear) → DeviceRegistry (update status)\n\nThe coordinator pattern ensures components don't directly depend on each other, reducing coupling.\n\n---\n\n## Initialization\n\n### Constructor\n\n```python\ndef __init__(\n self,\n task_name: str = \"test_task\",\n heartbeat_interval: float = 30.0,\n reconnect_delay: float = 5.0,\n):\n \"\"\"\n Initialize DeviceManager.\n \n Args:\n task_name: Identifier for this constellation instance (default \"test_task\")\n heartbeat_interval: Seconds between heartbeat checks (default 30s)\n reconnect_delay: Initial delay before reconnection attempt (default 5s)\n \"\"\"\n```\n\nWhen you create a DeviceManager, it initializes the five components:\n\n1. **Create DeviceRegistry**: Initializes empty device storage\n2. **Create WebSocketConnectionManager**: Prepares connection handling infrastructure\n3. **Create HeartbeatManager**: Creates heartbeat scheduler with specified interval\n4. **Create MessageProcessor**: Creates message routing infrastructure \n5. **Create TaskQueueManager**: Creates per-device task queues\n6. **Store Configuration**: Saves task_name, reconnect settings for later use\n\n**Parameter Explanations:**\n\n**task_name**: This identifier appears in log messages and helps distinguish between multiple constellation instances running simultaneously. For example, \"production_constellation\" vs \"test_constellation\".\n\n**heartbeat_interval**: How often (in seconds) HeartbeatManager checks device health. Lower values (e.g., 10s) detect failures faster but increase network traffic. Higher values (e.g., 60s) reduce overhead but delay failure detection. Default 30s balances responsiveness and efficiency.\n\n**reconnect_delay**: Initial delay before first reconnection attempt. DeviceManager uses exponential backoff, so subsequent delays double: 5s, 10s, 20s, 40s, 80s. Lower values reconnect faster but may overwhelm unstable networks. Higher values give networks more recovery time.\n\n**max_retries**: The maximum number of reconnection attempts is configured per-device during registration via the `max_retries` parameter (default 5) in `AgentProfile`. This allows different devices to have different retry limits based on their reliability characteristics.\n\n---\n\n## Device Lifecycle Methods\n\n### Register Device\n\n```python\nasync def register_device(\n self,\n device_id: str,\n server_url: str,\n os: str,\n capabilities: Optional[List[str]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n max_retries: int = 5,\n auto_connect: bool = True,\n) -> bool:\n \"\"\"\n Register a device for management.\n \n Creates an AgentProfile and stores it in DeviceRegistry.\n Does NOT establish connection; use connect_device() for that.\n \"\"\"\n```\n\nRegistration stores device information without connecting. This separation allows you to register all devices at startup but connect selectively based on runtime conditions.\n\n**Registration Process:**\n\n1. **Create AgentProfile**: DeviceManager creates an AgentProfile object containing:\n - `device_id`: Unique identifier\n - `server_url`: WebSocket endpoint \n - `os`: Operating system (Windows, Linux, macOS)\n - `capabilities`: List of capability tags (e.g., [\"office\", \"web\", \"email\"])\n - `metadata`: Arbitrary key-value data (e.g., {\"location\": \"datacenter\", \"gpu\": \"RTX 4090\"})\n - `status`: Initially set to DISCONNECTED\n\n2. **Store in DeviceRegistry**: DeviceManager delegates to DeviceRegistry, which:\n - Validates device_id is unique\n - Stores the AgentProfile\n - Initializes device status to DISCONNECTED\n\n3. **Return Success**: Returns True if registration succeeds, False if device_id already exists\n\n**When Registration Fails:**\n\nRegistration fails if:\n- Device ID already registered (must use unique IDs)\n- Invalid server URL format\n- Validation errors in AgentProfile creation\n\n**Example:**\n\n```python\n# Register device without connecting\nsuccess = await device_manager.register_device(\n device_id=\"office_pc\",\n server_url=\"ws://192.168.1.100:5000/ws\",\n os=\"Windows\",\n capabilities=[\"office\", \"web\"],\n metadata={\"location\": \"office_building_a\", \"user\": \"john\"}\n)\n\nif success:\n print(\"Device registered, ready to connect\")\nelse:\n print(\"Registration failed (ID already exists?)\")\n```\n\n### Connect Device\n\n```python\nasync def connect_device(self, device_id: str, is_reconnection: bool = False) -> bool:\n \"\"\"\n Establish connection to a registered device.\n \n Performs WebSocket handshake, AIP registration, device info exchange,\n and starts background monitoring services.\n \"\"\"\n```\n\nConnection is a multi-step process involving several components working together:\n\n**Step 1: Verify Registration**\n\nDeviceManager queries DeviceRegistry to verify the device is registered. If not registered, connection fails immediately.\n\n**Step 2: WebSocket Connection**\n\nDeviceManager delegates to WebSocketConnectionManager, passing the MessageProcessor to start message handling before registration (to avoid race conditions):\n\n```python\n# Connect and automatically start message handler\nawait connection_manager.connect_to_device(\n device_info, \n message_processor=self.message_processor\n)\n```\n\nWebSocketConnectionManager creates an AIP `WebSocketTransport`, establishes the connection, starts the message handler (via MessageProcessor), and performs AIP registration using `RegistrationProtocol`.\n\n**Step 3: Update Status and Start Heartbeat**\n\nAfter WebSocket connects successfully:\n\n```python\n# Update status to CONNECTED\ndevice_registry.update_device_status(device_id, DeviceStatus.CONNECTED)\ndevice_registry.update_heartbeat(device_id)\n\n# Start heartbeat monitoring\nheartbeat_manager.start_heartbeat(device_id)\n```\n\nNote: The message handler was already started in `connect_to_device()` to prevent race conditions.\n\n**Step 4: Device Info Exchange**\n\nDeviceManager requests device system information from the server (the device pushes its info during registration, server stores it):\n\n```python\ndevice_system_info = await connection_manager.request_device_info(device_id)\nif device_system_info:\n device_registry.update_device_system_info(device_id, device_system_info)\n```\n\nDevice info includes CPU count, memory, OS version, screen resolution, and other system details stored in the AgentProfile.\n\n**Step 5: Set Device to IDLE**\n\nDeviceManager updates device status to ready for tasks:\n\n```python\ndevice_registry.set_device_idle(device_id)\n```\n\nDevice is now ready to accept tasks. Note that HeartbeatManager was already started in Step 3, and MessageProcessor's message handler was started automatically during the WebSocket connection in Step 2.\n\n**Connection Sequence Diagram:**\n\n```mermaid\nsequenceDiagram\n participant DM as DeviceManager\n participant DR as DeviceRegistry\n participant WSM as WebSocketConnectionManager\n participant MP as MessageProcessor\n participant HM as HeartbeatManager\n participant Server as Agent Server\n \n DM->>DR: Get device profile\n DR-->>DM: AgentProfile\n \n DM->>WSM: connect_to_device(device_info, message_processor)\n WSM->>Server: WebSocket handshake (via AIP Transport)\n Server-->>WSM: Connection established\n \n Note over WSM,MP: CRITICAL: Start message handler BEFORE registration\n WSM->>MP: start_message_handler(device_id, transport)\n MP-->>MP: Start background message listener\n \n WSM->>Server: REGISTER (via RegistrationProtocol)\n Server-->>WSM: HEARTBEAT (OK status = registration confirmed)\n WSM-->>DM: Connection successful\n \n DM->>DR: update_device_status(CONNECTED)\n DM->>DR: update_heartbeat()\n \n DM->>HM: start_heartbeat(device_id)\n HM-->>HM: Start background heartbeat loop\n \n DM->>WSM: request_device_info(device_id)\n WSM->>Server: DEVICE_INFO_REQUEST\n Server-->>WSM: DEVICE_INFO_RESPONSE\n WSM-->>DM: Device system info\n \n DM->>DR: update_device_system_info()\n DM->>DR: set_device_idle()\n \n DM-->>DM: Connection complete\n```\n\nThis diagram shows the entire connection flow, from initial WebSocket handshake through AIP registration to background service startup.\n\n**When Connection Fails:**\n\nConnection can fail at multiple points:\n\n- **WebSocket Failure**: Network unreachable, server not running, firewall blocking\n- **Registration Failure**: Server rejects device (invalid credentials, server full)\n- **Timeout**: Server doesn't respond within timeout period\n- **Protocol Error**: Server sends unexpected message format\n\nWhen connection fails, DeviceManager:\n\n1. Closes WebSocket if partially connected\n2. Updates device status to FAILED\n3. Schedules reconnection attempt (if retries remain)\n\n### Disconnect Device\n\n```python\nasync def disconnect_device(self, device_id: str) -> None:\n \"\"\"\n Disconnect from a device and cleanup resources.\n \n Stops background services, closes WebSocket, and updates status.\n \"\"\"\n```\n\nDisconnection performs cleanup in reverse order of connection:\n\n**Step 1: Stop Heartbeat**\n\n```python\nawait heartbeat_manager.stop_heartbeat(device_id)\n```\n\nThis cancels the background heartbeat task, preventing further heartbeat messages.\n\n**Step 2: Stop Message Handler**\n\n```python\nawait message_processor.stop_message_handler(device_id)\n```\n\nThis cancels the background message listener task, preventing further message processing.\n\n**Step 3: Clear Task Queue**\n\n```python\ntask_queue_manager.clear_queue(device_id)\n```\n\nAny queued tasks are cancelled. In-progress tasks are allowed to complete (graceful shutdown).\n\n**Step 4: Close WebSocket**\n\n```python\nawait websocket_connection_manager.disconnect(device_id)\n```\n\nThis sends WebSocket CLOSE frame and closes the connection.\n\n**Step 5: Update Status**\n\n```python\ndevice_registry.update_status(device_id, DeviceStatus.DISCONNECTED)\n```\n\nDevice status becomes DISCONNECTED, indicating it's no longer available.\n\n**Graceful vs Forceful Disconnection:**\n\nCurrent implementation is graceful: it waits for in-progress tasks to complete before closing the connection. For forceful disconnection (immediate shutdown), you would:\n\n1. Cancel in-progress tasks\n2. Clear task queue\n3. Close WebSocket immediately without waiting\n\n---\n\n## Task Assignment\n\n### Assign Task to Device\n\n```python\nasync def assign_task_to_device(\n self,\n task_id: str,\n device_id: str,\n task_description: str,\n task_data: Dict[str, Any],\n timeout: float = 1000,\n) -> ExecutionResult:\n \"\"\"\n Assign a task to a device for execution.\n \n If device is IDLE, executes immediately.\n If device is BUSY, queues task for later execution.\n \"\"\"\n```\n\nTask assignment involves checking device status, potentially queuing, and sending the TASK message:\n\n**Step 1: Check Device Status**\n\n```python\nprofile = device_registry.get_device(device_id)\nstatus = profile.status\n```\n\nDevice must be CONNECTED, IDLE, or BUSY. If DISCONNECTED or FAILED, task assignment fails immediately.\n\n**Step 2: Queue if Busy**\n\n```python\nif status == DeviceStatus.BUSY:\n # Add to queue\n task_queue_manager.add_task(\n device_id=device_id,\n task_id=task_id,\n task_description=task_description,\n task_data=task_data\n )\n return {\"status\": \"queued\", \"task_id\": task_id}\n```\n\nTaskQueueManager maintains per-device FIFO queues. When the device completes its current task, TaskQueueManager automatically assigns the next queued task.\n\n**Step 3: Execute Immediately**\n\n```python\nif status == DeviceStatus.IDLE:\n # Update status to BUSY\n device_registry.update_status(device_id, DeviceStatus.BUSY)\n \n # Send TASK message\n await message_processor.send_message(\n device_id=device_id,\n message_type=\"TASK\",\n payload={\n \"task_id\": task_id,\n \"description\": task_description,\n \"data\": task_data\n }\n )\n \n # Wait for TASK_END\n result = await message_processor.wait_for_response(\n device_id=device_id,\n message_type=\"TASK_END\",\n timeout=1000.0 # Default timeout\n )\n \n # Update status back to IDLE\n device_registry.update_status(device_id, DeviceStatus.IDLE)\n \n # Execute next queued task if any\n next_task = task_queue_manager.get_next_task(device_id)\n if next_task:\n await self.assign_task_to_device(**next_task)\n \n return result\n```\n\nThis flow ensures devices never have more than one task executing at a time, preventing resource contention.\n\n**Task Assignment Sequence:**\n\n```mermaid\nsequenceDiagram\n participant App\n participant DM as DeviceManager\n participant DR as DeviceRegistry\n participant TQM as TaskQueueManager\n participant MP as MessageProcessor\n participant Device\n \n App->>DM: assign_task_to_device(task_id, device_id, ...)\n \n DM->>DR: get_device(device_id)\n DR-->>DM: AgentProfile (status=IDLE)\n \n DM->>DR: update_status(BUSY)\n \n DM->>MP: send_message(TASK)\n MP->>Device: TASK message\n \n Device-->>Device: Execute task\n \n Device->>MP: TASK_END\n MP-->>DM: Task result\n \n DM->>DR: update_status(IDLE)\n \n DM->>TQM: get_next_task(device_id)\n \n alt Queue has tasks\n TQM-->>DM: Next task\n DM->>DM: assign_task_to_device (recursive)\n else Queue empty\n TQM-->>DM: None\n end\n \n DM-->>App: Task result\n```\n\nThis diagram shows the complete task assignment flow, including automatic processing of queued tasks after completion.\n\n**Task Timeout Handling:**\n\nIf a task doesn't complete within the timeout period (default 1000 seconds):\n\n1. MessageProcessor raises TimeoutError\n2. DeviceManager marks device as FAILED\n3. DeviceManager attempts reconnection\n4. Queued tasks remain in queue and execute after reconnection\n\n---\n\n## Disconnection and Reconnection\n\n### Handle Device Disconnection\n\n```python\nasync def _handle_device_disconnection(\n self,\n device_id: str,\n reason: str = \"unknown\",\n) -> None:\n \"\"\"\n Internal handler for unexpected disconnections.\n \n Performs cleanup and initiates reconnection if retries remain.\n \"\"\"\n```\n\nWhen a device disconnects unexpectedly (network failure, server crash, heartbeat timeout), DeviceManager performs cleanup and attempts reconnection:\n\n**Step 1: Log Disconnection**\n\n```python\nlogger.warning(f\"Device {device_id} disconnected: {reason}\")\n```\n\nReason indicates why disconnection occurred: \"heartbeat_timeout\", \"websocket_error\", \"protocol_error\", etc.\n\n**Step 2: Cleanup Resources**\n\nSame as `disconnect_device()`:\n- Stop heartbeat\n- Stop message handler \n- Close WebSocket\n- Update status to FAILED\n\n**Step 3: Check Reconnection Eligibility**\n\n```python\nprofile = device_registry.get_device(device_id)\nattempts = profile.connection_attempts\n\nif attempts < max_retries:\n # Schedule reconnection\n await self._schedule_reconnection(device_id)\nelse:\n # Give up\n logger.error(f\"Device {device_id} exceeded max retries ({max_retries})\")\n device_registry.update_status(device_id, DeviceStatus.FAILED)\n```\n\nDeviceRegistry tracks connection attempts per device. If max retries exceeded, DeviceManager gives up and marks device as permanently failed.\n\n**Step 4: Schedule Reconnection**\n\n```python\nasync def _schedule_reconnection(self, device_id: str) -> None:\n \"\"\"Schedule reconnection with exponential backoff.\"\"\"\n profile = device_registry.get_device(device_id)\n attempts = profile.connection_attempts\n \n # Calculate delay: 5s, 10s, 20s, 40s, 80s\n delay = reconnect_delay * (2 ** attempts)\n \n logger.info(f\"Reconnecting to {device_id} in {delay}s (attempt {attempts+1}/{max_retries})\")\n \n # Wait\n await asyncio.sleep(delay)\n \n # Increment attempt counter\n device_registry.increment_attempts(device_id)\n \n # Try to reconnect\n success = await self.connect_device(device_id)\n \n if success:\n # Reset attempt counter on success\n device_registry.reset_attempts(device_id)\n logger.info(f\"Device {device_id} reconnected successfully\")\n else:\n # Reconnection failed, will retry again\n await self._handle_device_disconnection(device_id, \"reconnection_failed\")\n```\n\nExponential backoff prevents overwhelming unstable networks with rapid reconnection attempts.\n\n**Reconnection Flow:**\n\n```mermaid\nsequenceDiagram\n participant HM as HeartbeatManager\n participant DM as DeviceManager\n participant DR as DeviceRegistry\n participant Device\n \n HM->>HM: Send heartbeat\n Note over HM,Device: No response (timeout)\n \n HM->>DM: _handle_device_disconnection(\"heartbeat_timeout\")\n \n DM->>DM: Stop heartbeat\n DM->>DM: Stop message handler\n DM->>DM: Close WebSocket\n DM->>DR: update_status(FAILED)\n \n DM->>DR: get connection_attempts\n DR-->>DM: attempts = 1\n \n alt attempts < max_retries\n DM->>DM: Calculate delay (5s * 2^1 = 10s)\n DM->>DM: await asyncio.sleep(10)\n \n DM->>DR: increment_attempts (now 2)\n \n DM->>Device: connect_device()\n \n alt Connection succeeds\n Device-->>DM: Success\n DM->>DR: reset_attempts (back to 0)\n DM->>DR: update_status(IDLE)\n else Connection fails\n Device-->>DM: Failure\n DM->>DM: _handle_device_disconnection (recursive)\n Note over DM: Next attempt in 20s\n end\n else attempts >= max_retries\n DM->>DR: update_status(FAILED)\n Note over DM: Give up\n end\n```\n\nThis diagram shows the reconnection loop with exponential backoff.\n\n**Queued Task Handling During Reconnection:**\n\nTasks queued when a device disconnects remain in the queue. After successful reconnection, TaskQueueManager automatically starts processing queued tasks. This ensures no task loss during temporary network failures.\n\n---\n\n## Component Integration Example\n\nHere's a complete example showing how all components work together during a typical device lifecycle:\n\n```python\n# 1. Create DeviceManager\nmanager = DeviceManager(\n task_name=\"production_constellation\",\n heartbeat_interval=30.0,\n reconnect_delay=5.0\n)\n\n# This creates all five components:\n# - DeviceRegistry (stores device state)\n# - WebSocketConnectionManager (handles connections)\n# - HeartbeatManager (monitors health)\n# - MessageProcessor (routes messages)\n# - TaskQueueManager (manages queues)\n\n# 2. Register device\nawait manager.register_device(\n device_id=\"office_pc\",\n server_url=\"ws://192.168.1.100:5000/ws\",\n os=\"Windows\",\n capabilities=[\"office\", \"web\"],\n max_retries=5,\n auto_connect=True # Will automatically connect after registration\n)\n# DeviceManager → DeviceRegistry (store AgentProfile)\n# If auto_connect=True → DeviceManager → connect_device()\n\n# 3. Connect device (if auto_connect was False)\n# await manager.connect_device(\"office_pc\")\n# DeviceManager → WebSocketConnectionManager (connect, start message handler)\n# → DeviceRegistry (update status to CONNECTED, then IDLE)\n# → HeartbeatManager (start heartbeat loop)\n\n# 4. Assign first task (device is IDLE)\nresult1 = await manager.assign_task_to_device(\n task_id=\"task_1\",\n device_id=\"office_pc\",\n task_description=\"Open Excel\",\n task_data={\"file\": \"report.xlsx\"},\n timeout=300\n)\n# DeviceManager → DeviceRegistry (check status: IDLE)\n# → DeviceRegistry (update status to BUSY via set_device_busy)\n# → WebSocketConnectionManager (send TASK via TaskExecutionProtocol)\n# [wait for TASK_END]\n# → DeviceRegistry (update status to IDLE via set_device_idle)\n\n# 5. Assign second task while first is running (device is BUSY)\n# Note: This happens concurrently with task_1\nasyncio.create_task(\n manager.assign_task_to_device(\n task_id=\"task_2\",\n device_id=\"office_pc\",\n task_description=\"Send email\",\n task_data={\"to\": \"john@example.com\"},\n timeout=300\n )\n)\n# DeviceManager → DeviceRegistry (check status: BUSY)\n# → TaskQueueManager (add to queue)\n# [returns immediately with \"queued\" status]\n\n# When task_1 completes:\n# MessageProcessor → DeviceManager (TASK_END received)\n# DeviceManager → DeviceRegistry (update status to IDLE)\n# → TaskQueueManager (get_next_task)\n# → TaskQueueManager (returns task_2)\n# → DeviceManager (assign_task_to_device recursively for task_2)\n\n# 6. Simulate network failure\n# HeartbeatManager → [send heartbeat]\n# → [timeout waiting for response]\n# → DeviceManager (_handle_device_disconnection)\n\n# DeviceManager → HeartbeatManager (stop)\n# → MessageProcessor (stop)\n# → WebSocketConnectionManager (disconnect)\n# → TaskQueueManager (tasks remain queued)\n# → DeviceRegistry (update status to FAILED)\n# → [schedule reconnection attempt]\n# → [wait reconnect_delay seconds]\n# → connect_device (reconnection attempt with is_reconnection=True)\n\n# 7. Reconnection succeeds\n# After reconnection:\n# DeviceManager → DeviceRegistry (reset attempts, update status to IDLE)\n# → TaskQueueManager (get_next_task)\n# [if tasks queued, automatically start execution]\n\n# 8. Disconnect device\nawait manager.disconnect_device(\"office_pc\")\n# DeviceManager → HeartbeatManager (stop)\n# → MessageProcessor (stop)\n# → WebSocketConnectionManager (disconnect)\n# → TaskQueueManager (clear queue)\n# → DeviceRegistry (update status to DISCONNECTED)\n```\n\nThis complete example demonstrates how DeviceManager coordinates all five components throughout the device lifecycle.\n\n---\n\n## Internal Architecture Details\n\n### Component Responsibilities\n\n**DeviceRegistry:**\n\n- Stores AgentProfile objects (one per device)\n- Manages device status transitions (DISCONNECTED → CONNECTED → IDLE → BUSY → FAILED)\n- Tracks connection attempts for reconnection logic\n- Provides thread-safe access to device state\n\nDeviceRegistry is the single source of truth. All other components query DeviceRegistry for device information rather than maintaining their own state copies.\n\n**WebSocketConnectionManager:**\n\n- Establishes WebSocket connections using `websockets` library\n- Maintains WebSocket object per device\n- Sends messages over WebSocket\n- Handles WebSocket-level errors (connection refused, SSL errors, etc.)\n- Closes connections gracefully\n\nWebSocketConnectionManager knows nothing about AIP protocol or device status. It's purely a WebSocket I/O layer.\n\n**HeartbeatManager:**\n\n- Runs background loop per device (every `heartbeat_interval` seconds)\n- Sends HEARTBEAT message via MessageProcessor\n- Waits for HEARTBEAT response\n- Calls DeviceManager's disconnection handler on timeout\n- Cancellable via `stop_heartbeat()`\n\nHeartbeatManager detects connection failures that WebSocket layer might miss (e.g., server hangs without closing connection).\n\n**MessageProcessor:**\n\n- Routes incoming messages by type (REGISTER_CONFIRMATION, DEVICE_INFO, TASK_END, HEARTBEAT)\n- Implements request-response pattern for synchronous messaging\n- Runs background message listener loop per device\n- Queues responses for `wait_for_response()` calls\n- Handles protocol-level errors\n\nMessageProcessor implements the AIP protocol message routing. It's the component that \"speaks AIP\".\n\n**TaskQueueManager:**\n\n- Maintains FIFO queue per device\n- Adds tasks when device is BUSY\n- Returns next task when device becomes IDLE \n- Clears queue on disconnection\n- Thread-safe for concurrent access\n\nTaskQueueManager ensures tasks execute in order and prevents task loss when devices are busy.\n\n### Component Communication Pattern\n\nComponents communicate exclusively through DeviceManager as the coordinator. They do NOT directly call each other:\n\n**Wrong (direct component communication):**\n```python\n# DON'T do this\nwebsocket_manager.connect(device_id)\nmessage_processor.send_message(device_id, \"REGISTER\")\ndevice_registry.update_status(device_id, DeviceStatus.IDLE)\n```\n\n**Correct (through DeviceManager):**\n```python\n# DO this\nawait device_manager.connect_device(device_id)\n# DeviceManager internally coordinates:\n# websocket_manager.connect()\n# message_processor.send_message()\n# device_registry.update_status()\n```\n\nThis pattern enforces proper coordination and ensures all necessary steps happen in the correct order.\n\n---\n\n## Advanced Usage Patterns\n\n### Custom Reconnection Logic\n\nOverride disconnection handler for custom reconnection behavior:\n\n```python\nclass CustomDeviceManager(DeviceManager):\n async def _handle_device_disconnection(self, device_id: str, reason: str):\n # Custom logic: Only reconnect for specific reasons\n if reason == \"heartbeat_timeout\":\n # Network glitch, reconnect immediately\n await self.connect_device(device_id)\n elif reason == \"protocol_error\":\n # Protocol mismatch, don't reconnect\n logger.error(f\"Protocol error on {device_id}, not reconnecting\")\n self.device_registry.update_status(device_id, DeviceStatus.FAILED)\n else:\n # Use default exponential backoff\n await super()._handle_device_disconnection(device_id, reason)\n```\n\n### Priority Task Queue\n\nExtend TaskQueueManager for priority queuing:\n\n```python\nclass PriorityTaskQueueManager(TaskQueueManager):\n def add_task(self, device_id: str, task_id: str, priority: int, **kwargs):\n \"\"\"Add task with priority (lower number = higher priority).\"\"\"\n if device_id not in self._queues:\n self._queues[device_id] = []\n \n # Insert in priority order\n task = {\"task_id\": task_id, \"priority\": priority, **kwargs}\n queue = self._queues[device_id]\n \n # Find insertion point\n insert_idx = 0\n for i, queued_task in enumerate(queue):\n if queued_task[\"priority\"] > priority:\n insert_idx = i\n break\n else:\n insert_idx = len(queue)\n \n queue.insert(insert_idx, task)\n \n def get_next_task(self, device_id: str):\n \"\"\"Get highest priority task.\"\"\"\n if device_id in self._queues and self._queues[device_id]:\n return self._queues[device_id].pop(0) # First is highest priority\n return None\n\n# Use custom queue manager\nmanager = DeviceManager(task_name=\"production\")\nmanager.task_queue_manager = PriorityTaskQueueManager()\n```\n\n### Connection Pool Management\n\nLimit concurrent connections:\n\n```python\nclass PooledDeviceManager(DeviceManager):\n def __init__(self, *args, max_concurrent_connections: int = 10, **kwargs):\n super().__init__(*args, **kwargs)\n self.max_concurrent = max_concurrent_connections\n self.connection_semaphore = asyncio.Semaphore(max_concurrent_connections)\n \n async def connect_device(self, device_id: str) -> bool:\n async with self.connection_semaphore:\n # Only max_concurrent connections can proceed\n return await super().connect_device(device_id)\n\n# Limit to 5 concurrent connections\nmanager = PooledDeviceManager(\n task_name=\"production\",\n max_concurrent_connections=5\n)\n```\n\n---\n\n## Summary\n\nDeviceManager is the orchestration layer that coordinates five specialized components to manage device connections. It doesn't perform low-level operations itself; instead, it delegates to components and ensures they work together correctly.\n\n**Key Concepts:**\n\n- **Orchestrator Pattern**: DeviceManager coordinates components but doesn't duplicate their functionality\n- **Modular Architecture**: Five components with single responsibilities (DeviceRegistry, WebSocketConnectionManager, HeartbeatManager, MessageProcessor, TaskQueueManager)\n- **Lifecycle Management**: Register → Connect → Execute → Disconnect → Reconnect\n- **Automatic Reconnection**: Exponential backoff with configurable retries per device\n- **Task Queuing**: Automatic queuing when devices are busy\n\n**When to Use DeviceManager Directly:**\n\nMost applications should use ConstellationClient, which wraps DeviceManager. Use DeviceManager directly only for:\n\n- Custom reconnection strategies\n- Custom task queuing logic\n- Fine-grained control over component behavior\n- Advanced monitoring and debugging\n\n**Next Steps:**\n\n- See [Components](./components.md) for detailed component documentation\n- See [ConstellationClient](./constellation_client.md) for high-level API\n- See [AIP Integration](./aip_integration.md) for protocol details and message flows\n- See [Overview](./overview.md) for overall Galaxy Client architecture\n- See [Agent Registration](../agent_registration/overview.md) for device registration details\n"
},
{
"path": "documents/docs/galaxy/client/galaxy_client.md",
"content": "# GalaxyClient Reference\n\nGalaxyClient is an optional session management wrapper on top of ConstellationClient. It provides a convenient high-level API for initializing the system, processing user requests through GalaxySession, and running interactive sessions. Most applications use GalaxyClient as the main entry point.\n\n## Related Documentation\n\n- [Overview](./overview.md) - Overall architecture and workflow\n- [ConstellationClient](./constellation_client.md) - Device coordination layer\n\n## What GalaxyClient Does\n\nGalaxyClient is the \"easy mode\" API for Galaxy. While you can use ConstellationClient directly for device management, GalaxyClient adds session management, request processing, and interactive mode on top.\n\n**Think of it this way:**\n\n- **ConstellationClient**: \"I need to register devices and assign tasks\"\n- **GalaxyClient**: \"I have a user request, please execute it across my devices\"\n\nGalaxyClient handles the entire request lifecycle: parsing the request, creating a GalaxySession, coordinating with ConstellationAgent for task planning, executing the DAG across devices, and returning results to the user.\n\n**Core Responsibilities:**\n\n**Session Management**: GalaxyClient creates and manages GalaxySession objects. Each session represents one user request and contains the conversation history, task planning, and execution state. Sessions are isolated—failures in one session don't affect others.\n\n**Request Processing**: When you call `process_request()`, GalaxyClient:\n1. Creates a GalaxySession with the request\n2. Passes the session to ConstellationAgent for DAG planning\n3. Uses TaskConstellationOrchestrator to execute the DAG across devices\n4. Collects results and returns them to you\n\n**Interactive Mode**: GalaxyClient provides an interactive CLI loop where users can type requests, see execution progress, and view results. This is useful for demos, debugging, and manual testing.\n\n**Configuration Integration**: GalaxyClient loads configurations from YAML files, validates settings, and passes them to ConstellationClient. This centralizes configuration management.\n\n**What GalaxyClient Does NOT Do:**\n\n- **Device Connection Management**: Handled by ConstellationClient → DeviceManager\n- **Task Planning**: Handled by ConstellationAgent \n- **DAG Execution**: Handled by TaskConstellationOrchestrator\n- **Multi-round Interaction Logic**: Handled by GalaxySession\n\nGalaxyClient is the orchestrator at the highest level, delegating to specialized components for each concern.\n\n## When to Use GalaxyClient\n\n**Use GalaxyClient when:**\n\n- You want a simple API for processing user requests\n- You need session management for multi-round interactions\n- You want interactive mode for demos or debugging\n- You're building a conversational agent or task automation system\n\n**Use ConstellationClient directly when:**\n\n- You only need device management without session/request processing\n- You're building a custom orchestrator \n- You need fine-grained control over task assignment\n- Sessions are managed by your own higher-level system\n\n**Example Use Cases:**\n\n**GalaxyClient**: Chatbot that processes natural language requests (\"Open PowerPoint and create a presentation about AI\")\n\n**ConstellationClient**: Monitoring system that assigns health check tasks to devices every 5 minutes\n\n## Initialization\n\n### Constructor\n\n```python\ndef __init__(\n self,\n session_name: Optional[str] = None,\n task_name: Optional[str] = None,\n max_rounds: int = 10,\n log_level: str = \"INFO\",\n output_dir: Optional[str] = None,\n):\n \"\"\"\n Initialize GalaxyClient.\n \n Args:\n session_name: Name for the Galaxy session (auto-generated if None)\n task_name: Name for the task (auto-generated if None)\n max_rounds: Maximum number of rounds per session (default: 10)\n log_level: Logging level (default: \"INFO\")\n output_dir: Output directory for logs and results\n \"\"\"\n```\n\nGalaxyClient initialization automatically loads device configuration from the Galaxy config system:\n\n**Automatic Configuration Loading:**\n\nGalaxyClient loads device configuration from the centralized config system:\n\n```python\n# Configuration is loaded automatically\nclient = GalaxyClient(\n session_name=\"production_session\",\n task_name=\"email_automation\",\n max_rounds=10\n)\n```\n\nInternally, GalaxyClient:\n\n1. Loads Galaxy configuration using `get_galaxy_config()`\n2. Extracts device info path from `galaxy_config.constellation.DEVICE_INFO`\n3. Loads ConstellationConfig from the YAML file\n4. Creates internal ConstellationClient with this configuration\n\n**Session and Task Names:**\n\n```python\n# Use custom names\nclient = GalaxyClient(\n session_name=\"production_session\",\n task_name=\"email_task\"\n)\n\n# Auto-generate names with timestamps\nclient = GalaxyClient()\n# session_name: \"galaxy_session_20251106_103045\"\n# task_name: \"request_20251106_103045\"\n```\n\nSession name identifies the overall session, while task name identifies individual tasks within the session.\n\n**Max Rounds:**\n\n```python\n# Limit conversation rounds\nclient = GalaxyClient(max_rounds=5)\n```\n\nMax rounds controls how many back-and-forth exchanges the agent can have during task execution. Higher values allow more complex tasks but take longer.\n\n**Output Directory:**\n\n```python\n# Custom output directory\nclient = GalaxyClient(output_dir=\"./custom_logs\")\n```\n\nIf not specified, uses the default session log path from configuration.\n\n**Internal ConstellationClient Creation:**\n\nAfter loading configuration, GalaxyClient creates an internal ConstellationClient:\n\n```python\nself._constellation_client = ConstellationClient(\n config=self.config,\n task_name=self.task_name\n)\n```\n\nAll device management operations delegate to this internal client.\n\n### Async Initialize Method\n\n```python\nasync def initialize(self) -> None:\n \"\"\"\n Initialize the Galaxy Client and connect to devices.\n \n This calls ConstellationClient.initialize() to register and\n optionally connect to all configured devices.\n \"\"\"\n```\n\nAfter creating a GalaxyClient, you must call `initialize()`:\n\n```python\nclient = GalaxyClient(session_name=\"my_session\")\nawait client.initialize()\n\n# Now ready to process requests\nresult = await client.process_request(\"Open Excel and create a chart\")\n```\n\nInitialization creates and initializes the internal ConstellationClient, which:\n\n1. Registers all devices from configuration\n2. Connects to devices with `auto_connect: true`\n3. Starts heartbeat monitoring\n4. Starts message handlers\n\n**Initialization Failures:**\n\nIf some devices fail to connect during initialization, `initialize()` logs warnings but continues. You can check connection status after initialization:\n\n```python\nawait client.initialize()\n\n# Check which devices connected\nconnected = client._constellation_client.get_connected_devices()\nif len(connected) == 0:\n raise RuntimeError(\"No devices connected\")\n```\n\n## Request Processing\n\n### Process Request\n\n```python\nasync def process_request(\n self,\n request: str,\n context: Optional[Dict[str, Any]] = None,\n) -> Dict[str, Any]:\n \"\"\"\n Process a user request end-to-end.\n \n Args:\n request: Natural language user request\n context: Additional context (previous results, user preferences, etc.)\n \n Returns:\n Dictionary containing execution results, session info, and metadata\n \"\"\"\n```\n\nThis is the primary method you'll use. It handles the entire request lifecycle:\n\n**Step 1: Create Session**\n\n```python\nsession = GalaxySession(\n task=task_name,\n should_evaluate=False,\n id=session_id,\n client=self._constellation_client,\n initial_request=request\n)\n```\n\nGalaxySession encapsulates one request execution, including conversation history, task planning, and execution state.\n\n**Step 2: Execute Session**\n\n```python\nresult = await session.run()\n```\n\nSession execution involves:\n\n1. **ConstellationAgent Planning**: Agent analyzes the request, determines required capabilities, and creates a DAG (Directed Acyclic Graph) of tasks\n2. **Device Selection**: For each task, select a device with matching capabilities\n3. **DAG Execution**: TaskConstellationOrchestrator executes tasks respecting dependencies\n4. **Result Collection**: Gather results from all tasks\n\n**Step 3: Return Results**\n\n```python\nreturn {\n \"success\": result.success,\n \"output\": result.output,\n \"session_id\": session.session_id,\n \"task_count\": len(session.dag.tasks),\n \"execution_time\": result.execution_time,\n \"errors\": result.errors\n}\n```\n\n**Complete Request Processing Flow:**\n\n```mermaid\nsequenceDiagram\n participant User\n participant GC as GalaxyClient\n participant Session as GalaxySession\n participant Agent as ConstellationAgent\n participant Orch as TaskConstellationOrchestrator\n participant CC as ConstellationClient\n participant Devices\n \n User->>GC: process_request(\"Create PowerPoint about AI\")\n \n GC->>Session: Create GalaxySession\n GC->>Session: run()\n \n Session->>Agent: Analyze request\n Agent->>Agent: Create DAG\n Agent-->>Session: DAG (tasks + dependencies)\n \n Session->>Orch: execute_dag()\n \n loop For each task in topological order\n Orch->>Orch: Select device by capabilities\n Orch->>CC: assign_task_to_device()\n CC->>Devices: Send TASK (AIP)\n Devices-->>CC: TASK_END (results)\n CC-->>Orch: Task result\n end\n \n Orch-->>Session: All task results\n \n Session-->>GC: Execution result\n GC-->>User: {\"success\": true, \"output\": \"...\"}\n```\n\n**Example Usage:**\n\n```python\n# Simple request\nresult = await client.process_request(\n request=\"Open Excel and create a chart showing quarterly sales\"\n)\n\nif result[\"success\"]:\n print(f\"Completed {result['task_count']} tasks in {result['execution_time']:.2f}s\")\n print(f\"Output: {result['output']}\")\nelse:\n print(f\"Errors: {result['errors']}\")\n\n# Request with context\nresult = await client.process_request(\n request=\"Update the chart with new data\",\n context={\n \"previous_file\": \"Q1_sales.xlsx\",\n \"user_preferences\": {\"chart_type\": \"bar\"}\n }\n)\n```\n\nContext is useful for multi-round conversations where later requests reference earlier results.\n\n## Interactive Mode\n\n### Interactive Mode\n\n```python\nasync def interactive_mode(self) -> None:\n \"\"\"\n Start an interactive CLI loop for processing user requests.\n \n Users can type requests, see execution progress, and view results.\n Type 'quit' or 'exit' to stop.\n \"\"\"\n```\n\nInteractive mode provides a REPL (Read-Eval-Print Loop) for manual testing:\n\n```python\nclient = GalaxyClient(config_path=\"config/devices.yaml\")\nawait client.initialize()\n\n# Start interactive loop\nawait client.interactive_mode()\n```\n\n**Interactive Session Example:**\n\n```\n=== Galaxy Client Interactive Mode ===\nConnected to 3 devices: windows_pc, linux_server, mac_laptop\nType 'quit' or 'exit' to stop.\n\n> Open PowerPoint and create a presentation about AI\n\n[ConstellationAgent] Analyzing request...\n[ConstellationAgent] Created DAG with 3 tasks:\n - Task 1: Open PowerPoint\n - Task 2: Create new presentation \n - Task 3: Add slides about AI\n\n[TaskOrchestrator] Executing task 1 on windows_pc...\n[TaskOrchestrator] Task 1 completed successfully\n\n[TaskOrchestrator] Executing task 2 on windows_pc...\n[TaskOrchestrator] Task 2 completed successfully\n\n[TaskOrchestrator] Executing task 3 on windows_pc...\n[TaskOrchestrator] Task 3 completed successfully\n\n✓ Request completed successfully (3 tasks, 15.3s)\nOutput: Created presentation \"AI_Overview.pptx\" with 5 slides\n\n> Send the presentation via email to john@example.com\n\n[ConstellationAgent] Analyzing request...\n[ConstellationAgent] Using context from previous task\n\n[TaskOrchestrator] Executing task 1 on windows_pc...\n[TaskOrchestrator] Task 1 completed successfully\n\n✓ Request completed successfully (1 task, 3.2s)\nOutput: Email sent to john@example.com with attachment AI_Overview.pptx\n\n> quit\n\nShutting down Galaxy Client...\nDisconnected from all devices.\nGoodbye!\n```\n\n**Interactive Mode Features:**\n\n**Persistent Session Context**: Interactive mode maintains context across requests, so later requests can reference earlier results (\"Send the presentation\" knows which presentation).\n\n**Real-time Progress**: Shows task execution progress as it happens, useful for understanding what's happening during long-running requests.\n\n**Error Display**: Shows detailed error messages if tasks fail, helpful for debugging.\n\n**Device Status**: Shows which devices are connected at startup.\n\n## Lifecycle Management\n\n### Shutdown\n\n```python\nasync def shutdown(self) -> None:\n \"\"\"\n Gracefully shutdown the Galaxy Client.\n \n Stops all sessions, disconnects all devices, and cleans up resources.\n \"\"\"\n```\n\nAlways call `shutdown()` to cleanup resources:\n\n```python\ntry:\n client = GalaxyClient(config_path=\"config.yaml\")\n await client.initialize()\n \n # Use the client\n await client.process_request(\"...\")\n \nfinally:\n # Always shutdown\n await client.shutdown()\n```\n\nShutdown delegates to ConstellationClient, which:\n\n1. Stops all task queues\n2. Stops message handlers \n3. Stops heartbeat monitoring\n4. Closes WebSocket connections\n5. Cancels background tasks\n\nWithout proper shutdown, background tasks continue running, connections stay open, and resources leak.\n\n**Context Manager Pattern** (recommended):\n\n```python\nasync with GalaxyClient(config_path=\"config.yaml\") as client:\n await client.initialize()\n result = await client.process_request(\"Open Excel\")\n \n# Automatically calls shutdown() on exit\n```\n\n## Configuration Management\n\n### Get Device Status\n\n```python\ndef get_device_status(self, device_id: Optional[str] = None) -> Dict[str, Any]:\n \"\"\"Get device status from underlying ConstellationClient.\"\"\"\n return self._constellation_client.get_device_status(device_id)\n```\n\nGalaxyClient exposes device status from ConstellationClient:\n\n```python\n# Get all device statuses\nall_status = client.get_device_status()\n\n# Get specific device status\npc_status = client.get_device_status(\"windows_pc\")\nprint(f\"Status: {pc_status['status']}\")\nprint(f\"Current task: {pc_status['current_task_id']}\")\nprint(f\"Queued tasks: {pc_status['queued_tasks']}\")\n```\n\n### Get Connected Devices\n\n```python\ndef get_connected_devices(self) -> List[str]:\n \"\"\"Get list of connected device IDs.\"\"\"\n return self._constellation_client.get_connected_devices()\n```\n\nCheck which devices are available:\n\n```python\nconnected = client.get_connected_devices()\n\nif \"windows_pc\" not in connected:\n print(\"Warning: Windows PC not connected\")\n```\n\n### Add Device\n\n```python\nasync def add_device(\n self,\n device_id: str,\n server_url: str,\n capabilities: Optional[List[str]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n) -> bool:\n \"\"\"Add and connect a new device at runtime.\"\"\"\n```\n\nDynamically add devices:\n\n```python\n# Add new device discovered at runtime\nsuccess = await client.add_device(\n device_id=\"new_workstation\",\n server_url=\"ws://192.168.1.200:5000/ws\",\n capabilities=[\"office\", \"web\", \"design\"],\n metadata={\"location\": \"design_team\", \"gpu\": \"RTX 4090\"}\n)\n\nif success:\n print(\"New device ready for tasks\")\n```\n\nThis delegates to ConstellationClient, which registers and connects the device.\n\n## Usage Patterns\n\n### Basic Request Processing\n\n```python\nasync def main():\n # Initialize client\n client = GalaxyClient(session_name=\"automation_session\")\n await client.initialize()\n \n try:\n # Process single request\n result = await client.process_request(\n request=\"Open Word and create a document about machine learning\"\n )\n \n if result[\"success\"]:\n print(f\"Completed in {result['execution_time']:.1f}s\")\n else:\n print(f\"Failed: {result['errors']}\")\n \n finally:\n await client.shutdown()\n\nasyncio.run(main())\n```\n\n### Multi-Round Conversation\n\n```python\nasync def multi_round_conversation():\n client = GalaxyClient(session_name=\"conversation\", max_rounds=15)\n await client.initialize()\n \n try:\n # First request\n result1 = await client.process_request(\n request=\"Create a sales report spreadsheet\"\n )\n \n # Second request references first\n result2 = await client.process_request(\n request=\"Add a pie chart showing regional distribution\"\n )\n \n # Third request references both\n result3 = await client.process_request(\n request=\"Email the report to the team\"\n )\n \n finally:\n await client.shutdown()\n```\n\n### Error Handling\n\n```python\nasync def robust_processing():\n client = GalaxyClient(session_name=\"robust\")\n \n try:\n await client.initialize()\n except Exception as e:\n print(f\"Initialization failed: {e}\")\n return\n \n try:\n result = await client.process_request(\"Open Excel\")\n \n if not result[\"success\"]:\n # Handle execution errors\n for error in result[\"errors\"]:\n print(f\"Task {error['task_id']} failed: {error['message']}\")\n \n # Retry specific tasks\n if \"connection\" in error[\"message\"].lower():\n print(\"Retrying due to connection error...\")\n result = await client.process_request(\"Open Excel\")\n \n except Exception as e:\n # Handle unexpected errors\n print(f\"Unexpected error: {e}\")\n \n finally:\n await client.shutdown()\n```\n\n### Dynamic Device Management\n\n```python\nasync def adaptive_constellation():\n client = GalaxyClient(session_name=\"adaptive\")\n await client.initialize()\n \n try:\n # Monitor device health\n while True:\n connected = client.get_connected_devices()\n \n if len(connected) < 2:\n # Not enough devices, add more\n print(\"Adding fallback device...\")\n await client.add_device(\n device_id=\"fallback_device\",\n server_url=\"ws://backup.example.com:5000/ws\",\n capabilities=[\"office\", \"web\"]\n )\n \n # Process request\n result = await client.process_request(\"Create report\")\n \n # Sleep before next iteration\n await asyncio.sleep(60)\n \n finally:\n await client.shutdown()\n```\n\n## Integration with Other Components\n\n### GalaxyClient vs ConstellationClient\n\n```python\n# GalaxyClient: High-level request processing\ngalaxy_client = GalaxyClient(session_name=\"production\")\nawait galaxy_client.initialize()\n\nresult = await galaxy_client.process_request(\"Open PowerPoint\")\n# Internally:\n# 1. Creates GalaxySession\n# 2. ConstellationAgent plans DAG\n# 3. TaskOrchestrator executes DAG\n# 4. ConstellationClient assigns tasks to devices\n\n# ConstellationClient: Device management only\nconstellation_client = ConstellationClient(config)\nawait constellation_client.initialize()\n\nawait constellation_client.connect_device(\"windows_pc\")\n# No automatic task planning, you control everything\n```\n\n### Using GalaxyClient in Web Applications\n\n```python\nfrom fastapi import FastAPI, HTTPException\n\napp = FastAPI()\n\n# Global GalaxyClient instance\ngalaxy_client = None\n\n@app.on_event(\"startup\")\nasync def startup():\n global galaxy_client\n galaxy_client = GalaxyClient(session_name=\"api_server\")\n await galaxy_client.initialize()\n\n@app.on_event(\"shutdown\")\nasync def shutdown():\n global galaxy_client\n if galaxy_client:\n await galaxy_client.shutdown()\n\n@app.post(\"/execute\")\nasync def execute_request(request: str):\n \"\"\"Execute user request via Galaxy.\"\"\"\n if not galaxy_client:\n raise HTTPException(status_code=500, detail=\"Galaxy not initialized\")\n \n result = await galaxy_client.process_request(request)\n \n if result[\"success\"]:\n return {\"status\": \"completed\", \"output\": result[\"output\"]}\n else:\n raise HTTPException(\n status_code=500,\n detail={\"status\": \"failed\", \"errors\": result[\"errors\"]}\n )\n\n@app.get(\"/devices\")\nasync def list_devices():\n \"\"\"Get connected device status.\"\"\"\n if not galaxy_client:\n raise HTTPException(status_code=500, detail=\"Galaxy not initialized\")\n \n return {\n \"connected\": galaxy_client.get_connected_devices(),\n \"status\": galaxy_client.get_device_status()\n }\n```\n\n## Summary\n\nGalaxyClient is the high-level entry point for Galaxy Client, providing:\n\n- **Simple API**: Single method (`process_request`) for end-to-end execution\n- **Session Management**: Creates and manages GalaxySession objects \n- **Interactive Mode**: CLI loop for demos and debugging\n- **Configuration Management**: Loads and validates configurations\n- **Delegation**: Wraps ConstellationClient for device management\n\n**When to Use:**\n\n- **GalaxyClient**: Processing natural language requests, multi-round conversations, interactive demos\n- **ConstellationClient**: Direct device management, custom orchestration, fine-grained control\n\nFor most applications, GalaxyClient provides the right level of abstraction. Use ConstellationClient directly only when you need custom orchestration or don't need session management.\n\n**Next Steps:**\n\n- See [ConstellationClient](./constellation_client.md) for device management details\n- See [Overview](./overview.md) for overall architecture\n"
},
{
"path": "documents/docs/galaxy/client/overview.md",
"content": "# Galaxy Client Overview\n\nGalaxy Client is the client-side layer responsible for multi-device coordination in the UFO³ framework. At its core is **ConstellationClient**, which manages device registration, connection, and task assignment. **GalaxyClient** provides a lightweight wrapper offering convenient session management interfaces.\n\n## Related Documentation\n\n- [ConstellationClient](./constellation_client.md) - Core device coordination component\n- [DeviceManager](./device_manager.md) - Low-level connection management\n- [Components](./components.md) - Modular component architecture\n- [AIP Integration](./aip_integration.md) - Communication protocol integration\n- [GalaxyClient](./galaxy_client.md) - Session wrapper API\n- [Configuration](../../configuration/system/galaxy_constellation.md) - Device configuration guide\n\n## The Complete Path: From User Request to Device Execution\n\nTo understand Galaxy Client, we first need to see the entire system workflow. When a user submits a task request, the system processes it through several layers:\n\n### 1. User Interaction Layer (Optional)\n\nUsers can interact with the Galaxy system in two ways:\n\n**Interactive Mode**: Users input natural language requests through a command-line interface (CLI), which are received and processed by GalaxyClient. This mode is primarily used for rapid prototyping and manual testing.\n\n**Programmatic Mode**: Developers directly call the Python API of ConstellationClient or GalaxyClient, integrating Galaxy into their applications. This is the recommended approach for production environments.\n\n### 2. Session Management Layer (GalaxyClient)\n\nGalaxyClient's role is to manage the lifecycle of task sessions. It doesn't handle specific device operations but instead:\n\n- Initializes and holds a ConstellationClient instance\n- Creates a GalaxySession for each user request\n- Passes requests to ConstellationAgent for DAG planning (task decomposition)\n- Coordinates TaskConstellationOrchestrator to execute the DAG\n- Collects and aggregates execution results\n\n**GalaxyClient is optional**. If your application doesn't need session management, you can use ConstellationClient directly.\n\n### 3. Device Coordination Layer (ConstellationClient)\n\nConstellationClient is the heart of Galaxy Client. It is responsible for:\n\n**Device Management**: Registering devices (each device has a unique ID, server URL, capability list, etc.), connecting to devices (via WebSocket), disconnecting devices, and monitoring device health status.\n\n**Task Assignment**: Receiving task requests from upper layers (TaskConstellationOrchestrator), selecting appropriate devices based on capabilities, sending tasks to devices via the AIP protocol, and waiting for and collecting task execution results.\n\nConstellationClient doesn't concern itself with how tasks are decomposed (that's ConstellationAgent's responsibility) or how DAGs are executed (that's TaskConstellationOrchestrator's responsibility). It focuses on \"device-level matters.\"\n\n### 4. Connection Management Layer (DeviceManager)\n\nDeviceManager is the core internal component of ConstellationClient, responsible for all low-level connection management:\n\n**WebSocket Connection Establishment**: Establishes WebSocket connections with Agent Server, sends AIP REGISTER messages to register device identity, and requests device system information (DEVICE_INFO_REQUEST).\n\n**Connection Monitoring**: Sends HEARTBEAT messages every 20-30 seconds to check if devices are online. If a timeout occurs with no response, it triggers disconnection handling and automatically attempts reconnection (up to max_retries times).\n\n**Message Routing**: Starts a background message processing loop, receives messages returned by devices (TASK_END, COMMAND_RESULTS, etc.), and dispatches messages to appropriate handlers.\n\n**Task Queuing**: If a device is busy executing another task, new tasks are queued and automatically dequeued when the device becomes idle.\n\n### 5. Protocol Layer (AIP)\n\nAll communication with devices goes through the [Agent Interaction Protocol (AIP)](../../aip/overview.md). AIP is a WebSocket-based messaging protocol that defines standard message types and interaction flows. Main message types used by Galaxy Client include:\n\n- `REGISTER`: Register device identity with Agent Server\n- `DEVICE_INFO_REQUEST/RESPONSE`: Request and return device system information\n- `TASK`: Assign task to device\n- `TASK_END`: Device reports task completion\n- `HEARTBEAT/HEARTBEAT_ACK`: Heartbeat health check\n- `COMMAND_RESULTS`: Device reports intermediate execution results\n- `ERROR`: Error reporting\n\nFor detailed AIP explanation, see [AIP Integration](./aip_integration.md).\n\n## Component Responsibilities\n\nHaving understood the overall flow, let's examine the specific responsibilities of each component:\n\n### ConstellationClient: The Device Coordination Facade\n\nConstellationClient implements the Facade pattern. It provides simple device management APIs externally while delegating actual work to DeviceManager internally.\n\n**What it does:**\n\n```python\n# Register device\nawait client.register_device(\n device_id=\"windows_pc\",\n server_url=\"ws://192.168.1.100:5000/ws\",\n os=\"windows\",\n capabilities=[\"office\", \"web\", \"email\"]\n)\n\n# Connect device\nsuccess = await client.connect_device(\"windows_pc\")\n\n# Assign task\nresult = await client.assign_task_to_device(\n device_id=\"windows_pc\",\n task_request=TaskRequest(...)\n)\n\n# Query status\nstatus = client.get_device_status(\"windows_pc\")\n```\n\n**What it doesn't do:**\n\n- DAG planning (handled by ConstellationAgent)\n- DAG execution (handled by TaskConstellationOrchestrator)\n- Session management (handled by GalaxySession)\n\nSee [ConstellationClient documentation](./constellation_client.md) for detailed API reference.\n\n### DeviceManager: The Connection Management Engine\n\nDeviceManager is the \"engine\" of ConstellationClient. It uses 5 modular components to accomplish connection management:\n\n**DeviceRegistry**: Stores AgentProfiles for all registered devices (including device ID, URL, status, capabilities, metadata, etc.). This component maintains the single source of truth for device state. When a device connects, disconnects, or changes status, DeviceRegistry is updated. Other components query DeviceRegistry to make decisions.\n\n**WebSocketConnectionManager**: Manages WebSocket connection lifecycle (connect, disconnect, send messages). This component handles the low-level WebSocket operations, including establishing connections, handling connection errors, and sending AIP messages. It maintains a mapping from device_id to WebSocket objects.\n\n**HeartbeatManager**: Background heartbeat loop that periodically sends HEARTBEAT to check device health. This runs as an independent asyncio task for each connected device. If a device fails to respond within the timeout period (2 × heartbeat_interval), HeartbeatManager triggers the disconnection handler, allowing the system to detect and respond to connection failures quickly.\n\n**MessageProcessor**: Background message processing loop that receives and routes AIP messages. This component runs a continuous loop for each device, receiving messages from the WebSocket and dispatching them to appropriate handlers. For example, TASK_END messages are used to complete task futures, COMMAND_RESULTS are logged for progress tracking, and ERROR messages trigger error handling.\n\n**TaskQueueManager**: Manages task queue for each device, queuing tasks when device is busy. When a task is assigned to a busy device, it's placed in that device's queue. When the device completes its current task and becomes IDLE, TaskQueueManager automatically dequeues the next task and executes it. This ensures tasks are never lost even when devices are overloaded.\n\nThis modular design ensures each component has a single responsibility, making testing and maintenance easier. See [DeviceManager documentation](./device_manager.md) for details.\n\n### GalaxyClient: Session Management Wrapper\n\nGalaxyClient provides a higher-level abstraction on top of ConstellationClient:\n\n```python\nclient = GalaxyClient()\nawait client.initialize() # Initialize ConstellationClient and connect devices\n\n# Process user request (internally creates GalaxySession, calls ConstellationAgent for DAG planning)\nresult = await client.process_request(\"Open Excel and create a sales chart\")\n\nawait client.shutdown() # Cleanup resources\n```\n\nGalaxyClient's main value lies in:\n\n- Simplifying initialization flow (automatically loads device info from config)\n- Providing session management (creates independent GalaxySession for each request)\n- Integrating display components (Rich console output, progress bars, etc.)\n- Supporting interactive mode (command-line interface)\n\nIf your application already has its own session management logic, you can skip GalaxyClient and use ConstellationClient directly. See [GalaxyClient documentation](./galaxy_client.md) for detailed API.\n\n## Typical Workflow Example\n\nLet's walk through a complete example, from user request to device execution:\n\n### Scenario: Processing a Multi-Device Task\n\nSuppose a user submits: \"Download sales.xlsx from email, analyze it in Excel on Windows, then generate a report PDF on Linux\".\n\n**Step 1: Initialize GalaxyClient**\n\n```python\nclient = GalaxyClient()\nawait client.initialize()\n```\n\nWhat happens inside `initialize()`:\n\n1. GalaxyClient loads device information from config file (`device_info.yaml`)\n2. Creates ConstellationClient instance and passes configuration\n3. ConstellationClient calls `device_manager.register_device()` to register each device\n4. If `auto_connect: true` is configured, automatically calls `device_manager.connect_device()`\n5. DeviceManager executes connection flow for each device (detailed below)\n\n**Step 2: Device Connection Flow (Inside DeviceManager)**\n\nFor each device (e.g., \"windows_pc\" and \"linux_server\"), DeviceManager executes:\n\n```mermaid\nsequenceDiagram\n participant DM as DeviceManager\n participant WS as WebSocketConnectionManager\n participant Server as Agent Server\n participant Device as Device Agent\n \n Note over DM,Device: 1. Establish WebSocket Connection\n DM->>WS: connect_to_device(device_info)\n WS->>Server: WebSocket handshake\n Server-->>WS: Connection established\n \n Note over DM,Device: 2. Register Device Identity (AIP REGISTER)\n WS->>Server: REGISTER message\n Server->>Device: Forward registration\n Device-->>Server: Service manifest (available MCP servers)\n Server-->>WS: REGISTER_CONFIRMATION\n \n Note over DM,Device: 3. Request Device System Info\n WS->>Server: DEVICE_INFO_REQUEST\n Server->>Device: Request system info\n Device-->>Server: System info (CPU, memory, OS, etc.)\n Server-->>WS: DEVICE_INFO_RESPONSE\n WS->>DM: Update AgentProfile\n \n Note over DM,Device: 4. Start Background Services\n DM->>DM: Start MessageProcessor (message handling loop)\n DM->>DM: Start HeartbeatManager (heartbeat loop)\n DM->>DM: Set device status to IDLE\n```\n\nThis sequence diagram shows the connection establishment process. First, a WebSocket connection is established with the Agent Server. Then, the device registers its identity through the AIP REGISTER message, allowing the server to know which device is connecting and what capabilities it offers. Next, the client requests detailed system information from the device to populate the AgentProfile with actual hardware and software details. Finally, background services are started to maintain the connection and handle incoming messages.\n\n**Step 3: User Request Processing**\n\n```python\nresult = await client.process_request(\"Download sales.xlsx...\")\n```\n\nInside `process_request()`:\n\n1. GalaxyClient creates a GalaxySession\n2. GalaxySession calls ConstellationAgent for task planning\n3. ConstellationAgent (LLM-powered) decomposes task into DAG:\n - Task 1: Download sales.xlsx from email (requires \"email\" capability)\n - Task 2: Analyze in Excel (requires \"office\" capability, depends on Task 1)\n - Task 3: Generate PDF on Linux (requires \"pdf_generation\" capability, depends on Task 2)\n4. TaskConstellationOrchestrator executes DAG:\n - Based on capability matching, Task 1 assigned to device with \"email\" capability\n - Task 2 assigned to \"windows_pc\" (has \"office\" capability)\n - Task 3 assigned to \"linux_server\" (has \"pdf_generation\" capability)\n\nThe DAG structure ensures tasks execute in the correct order respecting dependencies, while allowing independent tasks to run in parallel across different devices.\n\n**Step 4: Task Assignment and Execution (ConstellationClient/DeviceManager)**\n\nFor each task, ConstellationClient calls:\n\n```python\nresult = await client.assign_task_to_device(\n device_id=\"windows_pc\",\n task_request=TaskRequest(\n task_id=\"task_2\",\n request=\"Analyze sales.xlsx in Excel\",\n ...\n )\n)\n```\n\nInside `assign_task_to_device()`:\n\n1. DeviceManager checks device status (via DeviceRegistry)\n2. If device is IDLE, execute task immediately\n3. If device is BUSY, task enters queue (TaskQueueManager)\n4. WebSocketConnectionManager sends TASK message to device via AIP\n5. MessageProcessor waits in background for device to return COMMAND_RESULTS and TASK_END\n6. When task completes, DeviceManager changes device status back to IDLE\n7. If there are queued tasks, automatically dequeue and execute next task\n\nThe queuing mechanism ensures no tasks are lost when devices are busy, and tasks are executed in order as devices become available.\n\n**Step 5: Connection Monitoring (Continuous Background Process)**\n\nThroughout task execution, HeartbeatManager continuously monitors each device:\n\n- Sends HEARTBEAT message every 20-30 seconds\n- If device responds, updates `last_heartbeat` timestamp\n- If timeout with no response (2 × heartbeat_interval), triggers disconnection handling:\n - Stops MessageProcessor and HeartbeatManager\n - Sets device status to DISCONNECTED\n - If device was executing a task, marks task as failed\n - Attempts automatic reconnection (up to max_retries times)\n\nThis continuous monitoring ensures the system quickly detects and responds to connection failures, maintaining reliable communication with devices.\n\n**Step 6: Result Collection and Return**\n\nAfter all tasks complete:\n\n1. TaskConstellationOrchestrator aggregates all task results\n2. GalaxySession generates session results (including execution time, rounds, DAG statistics)\n3. GalaxyClient returns results to user\n4. Results are automatically saved to log directory\n\nThe complete execution trace is preserved in logs for debugging and analysis.\n\n## Relationships with Other System Components\n\nGalaxy Client is not an isolated system—it closely collaborates with other UFO³ components:\n\n### Depends on Agent Server for Message Routing\n\nGalaxy Client doesn't connect directly to devices but routes through [Agent Server](../../server/overview.md). Agent Server's role is to:\n\n**Maintain Device Registry**: Tracks which devices are online and their connection details. When a device connects, Agent Server registers it in the central registry.\n\n**Route Messages**: Forwards TASK messages from Galaxy Client to the correct device based on device_id. The server acts as a message broker, decoupling clients from devices.\n\n**Broadcast Device Status**: Notifies clients when devices come online or go offline, enabling clients to maintain accurate device availability information.\n\n**Load Balancing**: If multiple clients connect to the same device, Agent Server can distribute load and prevent conflicts.\n\n### Used by ConstellationAgent for Task Planning\n\nWhen GalaxyClient receives a user request, it calls [ConstellationAgent](../constellation_agent/overview.md) to decompose the request into a DAG (Directed Acyclic Graph). ConstellationAgent is LLM-powered and can:\n\n**Understand Natural Language**: Parses user requests to identify subtasks and their relationships. For example, \"Download file and then analyze it\" is recognized as two sequential tasks.\n\n**Identify Task Dependencies**: Determines which tasks must complete before others can start, constructing a proper dependency graph.\n\n**Suggest Device Assignments**: Based on device capabilities, recommends which device should execute each task. If a task requires \"office\" capability, it's assigned to devices that advertise this capability.\n\n**Dynamically Adjust DAG**: If issues arise during execution (e.g., a device fails), ConstellationAgent can replan and modify the DAG to adapt to the new situation.\n\nFor more details, see [ConstellationAgent Documentation](../constellation_agent/overview.md).\n\n### Coordinates with TaskConstellationOrchestrator for DAG Execution\n\nOnce ConstellationAgent creates the DAG, [TaskConstellationOrchestrator](../constellation_orchestrator/overview.md) executes it across devices. The orchestrator:\n\n- **Respects Dependencies**: Ensures tasks execute in the correct order based on the DAG structure\n- **Selects Devices**: Chooses appropriate devices based on capability matching\n- **Parallel Execution**: Runs independent tasks concurrently across different devices\n- **Handles Failures**: Manages task failures and triggers replanning if needed\n\nFor more details, see [TaskConstellationOrchestrator Documentation](../constellation_orchestrator/overview.md).\n\n### Collaborates with Device Agents for Task Execution\n\nThe actual task execution happens on [Device Agents](../../client/overview.md) running on each device (such as UFO² Desktop Agent, Linux Agent, etc.). Device Agents are responsible for:\n\n**Receiving Tasks**: Accepts tasks from Agent Server and parses task requirements. Each task specifies what action to perform and what parameters to use.\n\n**Invoking MCP Servers**: Calls local MCP servers to perform specific operations (such as opening Excel, running commands, etc.). MCP servers provide the actual execution capabilities.\n\n**Reporting Progress**: Sends intermediate execution results through COMMAND_RESULTS messages, allowing clients to track progress in real-time.\n\n**Handling Errors**: Deals with local errors and exceptions, reporting them back to the client through ERROR messages for proper error handling.\n\n### Unified Communication through AIP Protocol\n\nAll cross-component communication follows the [AIP protocol](../../aip/overview.md). AIP provides:\n\n**Standardized Message Formats**: Uses Pydantic models to define message structure, ensuring type safety and validation at both ends of communication.\n\n**Type-Safe Message Validation**: Automatically validates message fields using Pydantic, catching errors early before they propagate through the system.\n\n**Request-Response Correlation**: Uses request_id/response_id fields to match requests with their responses, enabling proper async handling.\n\n**Error Handling Mechanism**: Defines standard ERROR message types for reporting and handling failures consistently across all components.\n\n## Configuration and Deployment\n\n### Device Configuration\n\nDevice information is defined through configuration files. See [Galaxy Configuration](../../configuration/system/galaxy_constellation.md) for complete configuration options.\n\nA typical configuration example:\n\n```yaml\n# config/galaxy/constellation.yaml\ntask_name: \"production_constellation\"\nheartbeat_interval: 30.0 # Heartbeat interval (seconds)\nreconnect_delay: 5.0 # Reconnection delay (seconds)\nmax_concurrent_tasks: 5 # Max concurrent tasks per device\n\ndevices:\n - device_id: \"windows_pc\"\n server_url: \"ws://192.168.1.100:5000/ws\"\n os: \"windows\"\n capabilities: [\"office\", \"email\", \"web\"]\n auto_connect: true\n max_retries: 5 # Maximum reconnection attempts\n \n - device_id: \"linux_server\"\n server_url: \"ws://192.168.1.101:5000/ws\"\n os: \"linux\"\n capabilities: [\"database\", \"api\", \"pdf_generation\"]\n auto_connect: true\n max_retries: 10\n```\n\nConfiguration fields explained:\n\n- **task_name**: Unique identifier for this constellation, used in logs and debugging\n- **heartbeat_interval**: How often to check device health (recommended: 20-30 seconds)\n- **reconnect_delay**: Wait time between reconnection attempts (recommended: 3-5 seconds)\n- **max_concurrent_tasks**: Maximum tasks a device can execute simultaneously\n- **capabilities**: List of capabilities each device provides, used for task assignment\n- **auto_connect**: Whether to automatically connect when client initializes\n- **max_retries**: Maximum reconnection attempts before giving up\n\n### Development vs Production Environment\n\n**Development Recommendations:**\n\n- Use interactive mode for quick testing: `python -m galaxy --interactive`\n- Enable DEBUG log level for detailed information\n- Single-device configuration to simplify debugging\n- Use local Agent Server (`ws://127.0.0.1:5000/ws`)\n- Lower heartbeat_interval (e.g., 10 seconds) for faster failure detection\n\n**Production Recommendations:**\n\n- Use WSS (secure WebSocket) instead of WS for encrypted communication\n- Configure reasonable heartbeat_interval (20-30 seconds) to balance responsiveness and network overhead\n- Set appropriate max_retries (5-10 attempts) based on network reliability\n- Enable automatic reconnection (`auto_connect: true`) for resilience\n- Monitor device health status via `get_device_status()` API and set up alerts\n- Configure log rotation and archiving to prevent disk space issues\n- Use connection pooling if connecting to many devices\n- Implement circuit breaker pattern for failing devices\n\n## Detailed Component Documentation\n\n- [ConstellationClient API Reference](./constellation_client.md) - Complete device coordination API\n- [DeviceManager Internals](./device_manager.md) - Detailed connection management mechanisms\n- [Components Module](./components.md) - Detailed explanation of 5 core components\n- [AIP Integration](./aip_integration.md) - How to use the communication protocol\n- [GalaxyClient Session Wrapper](./galaxy_client.md) - Session management API\n\n## Summary\n\nGalaxy Client provides the core multi-device coordination capabilities in UFO³. Through layered design, it simplifies complex distributed system management into clear APIs:\n\n- **ConstellationClient** is the core of device management, handling device registration, connection, and task assignment\n- **DeviceManager** is the underlying engine, processing WebSocket, heartbeat, message routing, and task queuing\n- **GalaxyClient** is an optional session wrapper, providing more convenient high-level APIs\n\nIf you're new to Galaxy Client, we recommend reading the documentation in this order:\n\n1. This Overview (understand overall architecture and workflow)\n2. [ConstellationClient](./constellation_client.md) (learn core API)\n3. [Components](./components.md) (understand modular components)\n4. [DeviceManager](./device_manager.md) (dive deep into connection management)\n5. [AIP Integration](./aip_integration.md) (master communication protocol)\n\nIf you need to get started quickly, jump directly to [GalaxyClient](./galaxy_client.md) example code.\n"
},
{
"path": "documents/docs/galaxy/constellation/constellation_editor.md",
"content": "# ConstellationEditor — Interactive DAG Editor\n\n---\n\n## 📋 Overview\n\n**ConstellationEditor** provides a high-level, command pattern-based interface for safe and comprehensive TaskConstellation manipulation. It offers undo/redo capabilities, batch operations, validation, and observer patterns for building, modifying, and managing complex workflow DAGs interactively.\n\nThe editor uses the **Command Pattern** to encapsulate all operations as reversible command objects, enabling undo/redo with full command history, transactional safety with atomic operations, complete operation tracking for auditability, and easy extensibility for new command types.\n\n**Usage in Galaxy**: The ConstellationEditor is primarily used by the [Constellation Agent](../constellation_agent/overview.md) to programmatically build task workflows, but can also be used directly for manual constellation creation and debugging.\n\n---\n\n## 🏗️ Architecture\n\n### Core Components\n\n```mermaid\ngraph TD\n A[ConstellationEditor] -->|manages| B[TaskConstellation]\n A -->|uses| C[CommandInvoker]\n C -->|executes| D[Commands]\n D -->|modifies| B\n A -->|notifies| E[Observers]\n \n style A fill:#87CEEB\n style B fill:#90EE90\n style C fill:#FFD700\n style D fill:#FFB6C1\n style E fill:#DDA0DD\n```\n\n| Component | Purpose |\n|-----------|---------|\n| **ConstellationEditor** | High-level interface for constellation editing |\n| **CommandInvoker** | Manages command execution, history, undo/redo |\n| **Commands** | Encapsulated operations (Add, Remove, Update, etc.) |\n| **Observers** | Callback functions notified on changes |\n\n---\n\n## 💻 Basic Usage\n\n### Creating an Editor\n\n```python\nfrom galaxy.constellation import TaskConstellation\nfrom galaxy.constellation.editor import ConstellationEditor\n\n# Create editor with new constellation\neditor = ConstellationEditor()\n\n# Create editor with existing constellation\nexisting = TaskConstellation(name=\"my_workflow\")\neditor = ConstellationEditor(\n constellation=existing,\n enable_history=True, # Enable undo/redo\n max_history_size=100 # Keep last 100 commands\n)\n\n# Access constellation\nprint(f\"Editing: {editor.constellation.name}\")\n```\n\n---\n\n## 🎯 Task Operations\n\n### Adding Tasks\n\n```python\nfrom galaxy.constellation import TaskStar\n\n# Method 1: Add existing TaskStar\ntask = TaskStar(\n task_id=\"fetch_data\",\n description=\"Download dataset from S3\",\n target_device_id=\"linux_server_1\"\n)\nadded_task = editor.add_task(task)\n\n# Method 2: Add from dictionary\ntask_dict = {\n \"task_id\": \"preprocess\",\n \"description\": \"Clean and normalize data\",\n \"target_device_id\": \"linux_server_2\",\n \"timeout\": 300.0\n}\nadded_task = editor.add_task(task_dict)\n\n# Method 3: Create and add in one step\ntask = editor.create_and_add_task(\n task_id=\"train_model\",\n description=\"Train neural network on preprocessed data\",\n name=\"Model Training\",\n target_device_id=\"gpu_server\",\n priority=\"HIGH\",\n timeout=3600.0,\n retry_count=2\n)\n```\n\n### Updating Tasks\n\n```python\n# Update task properties\nupdated_task = editor.update_task(\n task_id=\"train_model\",\n description=\"Train BERT model on preprocessed text data\",\n timeout=7200.0,\n priority=\"CRITICAL\"\n)\n\n# Update with task_data\neditor.update_task(\n task_id=\"train_model\",\n task_data={\n \"model_type\": \"BERT\",\n \"epochs\": 10,\n \"batch_size\": 32\n }\n)\n```\n\n### Removing Tasks\n\n```python\n# Remove task (also removes related dependencies)\nremoved_id = editor.remove_task(\"preprocess\")\n\nprint(f\"Removed task: {removed_id}\")\n```\n\n### Querying Tasks\n\n```python\n# Get specific task\ntask = editor.get_task(\"fetch_data\")\n\n# List all tasks\nall_tasks = editor.list_tasks()\n\nfor task in all_tasks:\n print(f\"{task.name}: {task.status.value}\")\n\n# Get ready tasks\nready = editor.get_ready_tasks()\n```\n\n---\n\n## 🔗 Dependency Operations\n\n### Adding Dependencies\n\n```python\nfrom galaxy.constellation import TaskStarLine\n\n# Method 1: Add existing TaskStarLine\ndep = TaskStarLine.create_success_only(\n from_task_id=\"fetch_data\",\n to_task_id=\"preprocess\",\n description=\"Preprocess after successful download\"\n)\nadded_dep = editor.add_dependency(dep)\n\n# Method 2: Add from dictionary\ndep_dict = {\n \"from_task_id\": \"preprocess\",\n \"to_task_id\": \"train_model\",\n \"dependency_type\": \"SUCCESS_ONLY\",\n \"condition_description\": \"Train on preprocessed data\"\n}\nadded_dep = editor.add_dependency(dep_dict)\n\n# Method 3: Create and add in one step\ndep = editor.create_and_add_dependency(\n from_task_id=\"train_model\",\n to_task_id=\"evaluate_model\",\n dependency_type=\"UNCONDITIONAL\",\n condition_description=\"Evaluate after training completes\"\n)\n```\n\n### Updating Dependencies\n\n```python\n# Update dependency properties\nupdated_dep = editor.update_dependency(\n dependency_id=dep.line_id,\n dependency_type=\"CONDITIONAL\",\n condition_description=\"Evaluate only if training accuracy > 90%\"\n)\n```\n\n### Removing Dependencies\n\n```python\n# Remove dependency\nremoved_id = editor.remove_dependency(dep.line_id)\n```\n\n### Querying Dependencies\n\n```python\n# Get specific dependency\ndep = editor.get_dependency(dep_id)\n\n# List all dependencies\nall_deps = editor.list_dependencies()\n\n# Get dependencies for specific task\ntask_deps = editor.get_task_dependencies(\"train_model\")\n```\n\n---\n\n## 🔄 Undo/Redo Operations\n\n### Basic Undo/Redo\n\n```python\n# Add a task\ntask = editor.create_and_add_task(\n task_id=\"test_task\",\n description=\"Run unit tests\"\n)\n\n# Oops, didn't mean to add that\nif editor.can_undo():\n editor.undo()\n print(\"✅ Task addition undone\")\n\n# Actually, let's keep it\nif editor.can_redo():\n editor.redo()\n print(\"✅ Task addition redone\")\n```\n\n### Checking Undo/Redo Availability\n\n```python\n# Check if undo/redo is available\nprint(f\"Can undo: {editor.can_undo()}\")\nprint(f\"Can redo: {editor.can_redo()}\")\n\n# Get description of what would be undone/redone\nif editor.can_undo():\n print(f\"Undo: {editor.get_undo_description()}\")\n\nif editor.can_redo():\n print(f\"Redo: {editor.get_redo_description()}\")\n```\n\n### Command History\n\n```python\n# Get command history\nhistory = editor.get_history()\nfor i, cmd_desc in enumerate(history):\n print(f\"{i+1}. {cmd_desc}\")\n\n# Example output:\n# 1. Add task: fetch_data\n# 2. Add task: preprocess\n# 3. Add dependency: fetch_data → preprocess\n# 4. Update task: preprocess\n\n# Clear history (cannot undo after this)\neditor.clear_history()\n```\n\n---\n\n## 🏗️ Bulk Operations\n\n### Building from Configuration\n\n```python\nfrom galaxy.agents.schema import TaskConstellationSchema\n\n# Build constellation from schema\nconfig = TaskConstellationSchema(\n name=\"ml_pipeline\",\n tasks=[\n {\n \"task_id\": \"fetch\",\n \"description\": \"Fetch data\",\n \"target_device_id\": \"server_1\"\n },\n {\n \"task_id\": \"process\",\n \"description\": \"Process data\",\n \"target_device_id\": \"server_2\"\n }\n ],\n dependencies=[\n {\n \"from_task_id\": \"fetch\",\n \"to_task_id\": \"process\",\n \"dependency_type\": \"SUCCESS_ONLY\"\n }\n ]\n)\n\nconstellation = editor.build_constellation(\n config=config,\n clear_existing=True # Clear current constellation first\n)\n```\n\n### Building from Lists\n\n```python\n# Build from task and dependency lists\ntasks = [\n {\n \"task_id\": \"a\",\n \"description\": \"Task A\",\n \"target_device_id\": \"device_1\"\n },\n {\n \"task_id\": \"b\",\n \"description\": \"Task B\",\n \"target_device_id\": \"device_2\"\n }\n]\n\ndependencies = [\n {\n \"from_task_id\": \"a\",\n \"to_task_id\": \"b\",\n \"dependency_type\": \"UNCONDITIONAL\"\n }\n]\n\nconstellation = editor.build_from_tasks_and_dependencies(\n tasks=tasks,\n dependencies=dependencies,\n clear_existing=True,\n metadata={\"version\": \"1.0\", \"author\": \"system\"}\n)\n```\n\n### Clearing Constellation\n\n```python\n# Remove all tasks and dependencies\ncleared = editor.clear_constellation()\n\nprint(f\"Constellation cleared: {cleared.task_count == 0}\")\n```\n\n---\n\n## 💾 File Operations\n\n### Saving Constellation\n\n```python\n# Save to JSON file\nfile_path = editor.save_constellation(\"my_workflow.json\")\n\nprint(f\"Saved to: {file_path}\")\n```\n\n### Loading Constellation\n\n```python\n# Load from JSON file\nloaded = editor.load_constellation(\"my_workflow.json\")\n\nprint(f\"Loaded: {loaded.name}\")\nprint(f\"Tasks: {loaded.task_count}\")\nprint(f\"Dependencies: {loaded.dependency_count}\")\n```\n\n### Loading from Data\n\n```python\n# Load from dictionary\ndata = {\n \"name\": \"test_workflow\",\n \"tasks\": {...},\n \"dependencies\": {...}\n}\nconstellation = editor.load_from_dict(data)\n\n# Load from JSON string\njson_string = '{\"name\": \"workflow\", \"tasks\": {...}}'\nconstellation = editor.load_from_json_string(json_string)\n```\n\n---\n\n## 🔍 Validation and Analysis\n\n### DAG Validation\n\n```python\n# Validate constellation structure\nis_valid, errors = editor.validate_constellation()\n\nif not is_valid:\n print(\"❌ Validation errors:\")\n for error in errors:\n print(f\" - {error}\")\nelse:\n print(\"✅ Constellation is valid\")\n\n# Check for cycles\nif editor.has_cycles():\n print(\"❌ Constellation contains cycles\")\n```\n\n### Topological Analysis\n\n```python\n# Get topological order\ntry:\n order = editor.get_topological_order()\n print(f\"Execution order: {' → '.join(order)}\")\nexcept ValueError as e:\n print(f\"Cannot get order: {e}\")\n```\n\n### Statistics\n\n```python\n# Get comprehensive statistics\nstats = editor.get_statistics()\n\nprint(f\"Constellation: {stats['constellation_id']}\")\nprint(f\"Tasks: {stats['total_tasks']}\")\nprint(f\"Dependencies: {stats['total_dependencies']}\")\nprint(f\"Longest path: {stats['longest_path_length']}\")\nprint(f\"Max width: {stats['max_width']}\")\nprint(f\"Parallelism ratio: {stats['parallelism_ratio']:.2f}\")\n\n# Editor-specific stats\nprint(f\"Commands executed: {stats['editor_execution_count']}\")\nprint(f\"History size: {stats['editor_history_size']}\")\nprint(f\"Can undo: {stats['editor_can_undo']}\")\nprint(f\"Can redo: {stats['editor_can_redo']}\")\n```\n\n---\n\n## 👀 Observer Pattern\n\n### Adding Observers\n\n```python\n# Define observer callback\ndef on_change(editor, command, result):\n print(f\"Operation: {command}\")\n print(f\"Result: {result}\")\n print(f\"Constellation state: {editor.constellation.state.value}\")\n\n# Add observer\neditor.add_observer(on_change)\n\n# Now all operations trigger the observer\ntask = editor.create_and_add_task(\n task_id=\"observed_task\",\n description=\"This triggers the observer\"\n)\n# Output:\n# Operation: add_task\n# Result: \n# Constellation state: ready\n```\n\n### Removing Observers\n\n```python\n# Remove specific observer\neditor.remove_observer(on_change)\n\n# Operations no longer trigger this observer\n```\n\n### Multiple Observers\n\n```python\ndef log_observer(editor, command, result):\n with open(\"constellation_log.txt\", \"a\") as f:\n f.write(f\"{command}: {result}\\n\")\n\ndef metrics_observer(editor, command, result):\n stats = editor.get_statistics()\n print(f\"Current metrics: P={stats['parallelism_ratio']:.2f}\")\n\n# Add multiple observers\neditor.add_observer(log_observer)\neditor.add_observer(metrics_observer)\n\n# All observers are notified on each operation\n```\n\n---\n\n## 🎨 Advanced Features\n\n### Batch Operations\n\n```python\n# Execute multiple operations in sequence\noperations = [\n lambda e: e.create_and_add_task(\"task_a\", \"Task A\"),\n lambda e: e.create_and_add_task(\"task_b\", \"Task B\"),\n lambda e: e.create_and_add_dependency(\"task_a\", \"task_b\", \"UNCONDITIONAL\"),\n]\n\nresults = editor.batch_operations(operations)\n\nfor i, result in enumerate(results):\n if isinstance(result, Exception):\n print(f\"Operation {i+1} failed: {result}\")\n else:\n print(f\"Operation {i+1} succeeded: {result}\")\n```\n\n### Creating Subgraphs\n\n```python\n# Extract subgraph with specific tasks\ntask_ids = [\"fetch_data\", \"preprocess\", \"train_model\"]\nsubgraph_editor = editor.create_subgraph(task_ids)\n\nprint(f\"Subgraph tasks: {subgraph_editor.constellation.task_count}\")\nprint(f\"Subgraph deps: {subgraph_editor.constellation.dependency_count}\")\n\n# Subgraph includes only dependencies between included tasks\n```\n\n### Merging Constellations\n\n```python\n# Create two separate workflows\neditor1 = ConstellationEditor()\neditor1.create_and_add_task(\"task_a\", \"Task A from editor1\")\n\neditor2 = ConstellationEditor()\neditor2.create_and_add_task(\"task_b\", \"Task B from editor2\")\n\n# Merge editor2 into editor1 with prefix\neditor1.merge_constellation(\n other_editor=editor2,\n prefix=\"imported_\"\n)\n\n# editor1 now contains: task_a, imported_task_b\n```\n\n---\n\n## 🛡️ Error Handling\n\n### Validation Errors\n\n```python\ntry:\n # Try to add task with duplicate ID\n editor.create_and_add_task(\"existing_id\", \"Duplicate task\")\nexcept Exception as e:\n print(f\"❌ Error: {e}\")\n # Can undo to previous valid state\n if editor.can_undo():\n editor.undo()\n```\n\n### Cyclic Dependency Detection\n\n```python\n# Create cycle: A → B → C → A\neditor.create_and_add_task(\"a\", \"Task A\")\neditor.create_and_add_task(\"b\", \"Task B\")\neditor.create_and_add_task(\"c\", \"Task C\")\n\neditor.create_and_add_dependency(\"a\", \"b\", \"UNCONDITIONAL\")\neditor.create_and_add_dependency(\"b\", \"c\", \"UNCONDITIONAL\")\n\ntry:\n # This creates a cycle\n editor.create_and_add_dependency(\"c\", \"a\", \"UNCONDITIONAL\")\nexcept Exception as e:\n print(f\"❌ Cycle detected: {e}\")\n # Undo the failed operation\n # (Actually, the operation fails before execution, so nothing to undo)\n```\n\n---\n\n## 📊 Complete Example Workflow\n\n```python\nfrom galaxy.constellation.editor import ConstellationEditor\n\n# Create editor\neditor = ConstellationEditor(enable_history=True)\n\n# Build ML training pipeline\n# Step 1: Add tasks\nfetch = editor.create_and_add_task(\n task_id=\"fetch_data\",\n description=\"Download dataset from S3\",\n target_device_id=\"linux_server_1\",\n timeout=300.0\n)\n\npreprocess = editor.create_and_add_task(\n task_id=\"preprocess\",\n description=\"Clean and normalize data\",\n target_device_id=\"linux_server_2\",\n timeout=600.0\n)\n\ntrain = editor.create_and_add_task(\n task_id=\"train_model\",\n description=\"Train BERT model\",\n target_device_id=\"gpu_server_a100\",\n priority=\"HIGH\",\n timeout=7200.0,\n retry_count=2\n)\n\nevaluate = editor.create_and_add_task(\n task_id=\"evaluate\",\n description=\"Evaluate model on test set\",\n target_device_id=\"linux_server_3\"\n)\n\n# Step 2: Add dependencies\neditor.create_and_add_dependency(\n \"fetch_data\", \"preprocess\", \"SUCCESS_ONLY\"\n)\neditor.create_and_add_dependency(\n \"preprocess\", \"train_model\", \"SUCCESS_ONLY\"\n)\neditor.create_and_add_dependency(\n \"train_model\", \"evaluate\", \"UNCONDITIONAL\"\n)\n\n# Step 3: Validate\nis_valid, errors = editor.validate_constellation()\nassert is_valid, f\"Validation failed: {errors}\"\n\n# Step 4: Analyze\nstats = editor.get_statistics()\nprint(f\"Pipeline: {stats['total_tasks']} tasks, {stats['total_dependencies']} dependencies\")\nprint(f\"Critical path: {stats['longest_path_length']}\")\nprint(f\"Parallelism: {stats['parallelism_ratio']:.2f}\")\n\n# Step 5: Save\neditor.save_constellation(\"ml_training_pipeline.json\")\n\n# Step 6: Execute (via orchestrator)\nconstellation = editor.constellation\n# Pass to ConstellationOrchestrator for distributed execution\n# See: ../constellation_orchestrator/overview.md for execution details\n```\n\nFor details on executing the built constellation, see the [Constellation Orchestrator documentation](../constellation_orchestrator/overview.md).\n\n---\n\n## 🎯 Best Practices\n\n### Editor Usage Guidelines\n\n1. **Enable history**: Always enable undo/redo for interactive editing sessions\n2. **Validate frequently**: Run `validate_constellation()` after major structural changes\n3. **Use observers**: Add observers for logging, metrics tracking, or UI updates\n4. **Batch operations**: Use `batch_operations()` for multiple related changes to improve efficiency\n5. **Save incrementally**: Create constellation checkpoints during complex editing workflows\n\n### Command Pattern Benefits\n\nThe command pattern architecture provides several key advantages:\n\n- **Undo/Redo**: Full operation history with rollback capabilities\n- **Audit trail**: Every change is recorded and traceable\n- **Transaction safety**: Operations are atomic and validated\n- **Extensibility**: New operation types can be added easily\n\n!!!warning \"Common Pitfalls\"\n - **Forgetting to validate**: Always validate before passing to orchestrator for execution\n - **Clearing history prematurely**: Cannot undo operations after calling `clear_history()`\n - **Modifying running constellations**: Editor operations will fail if constellation is currently executing\n - **Ignoring observer errors**: Observers should handle their own exceptions to avoid breaking the editor\n\n---\n\n## 📚 Command Registry\n\n### Available Commands\n\n```python\n# List all available commands\ncommands = editor.list_available_commands()\n\nfor name, metadata in commands.items():\n print(f\"{name}: {metadata['description']}\")\n print(f\" Category: {metadata['category']}\")\n\n# Get command categories\ncategories = editor.get_command_categories()\nprint(f\"Categories: {categories}\")\n\n# Get metadata for specific command\nmetadata = editor.get_command_metadata(\"add_task\")\nprint(metadata)\n```\n\n### Executing Commands by Name\n\n```python\n# Execute command using registry\nresult = editor.execute_command_by_name(\n \"add_task\",\n task_data={\"task_id\": \"new_task\", \"description\": \"New task\"}\n)\n\n# This is equivalent to:\n# editor.add_task({\"task_id\": \"new_task\", \"description\": \"New task\"})\n```\n\n---\n\n## 🔗 Related Components\n\n- **[TaskStar](task_star.md)** — Individual tasks that can be edited and managed\n- **[TaskStarLine](task_star_line.md)** — Dependencies between tasks that define execution order\n- **[TaskConstellation](task_constellation.md)** — The constellation DAG being edited\n- **[Overview](overview.md)** — Task Constellation framework overview\n\n### Related Documentation\n\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Learn how edited constellations are scheduled and executed\n- **[Constellation Agent](../constellation_agent/overview.md)** — Understand how agents use the editor to build constellations\n- **[Command Pattern](https://en.wikipedia.org/wiki/Command_pattern)** — More about the command design pattern\n\n---\n\n## 📚 API Reference\n\n### Constructor\n\n```python\nConstellationEditor(\n constellation: Optional[TaskConstellation] = None,\n enable_history: bool = True,\n max_history_size: int = 100\n)\n```\n\n### Task Operations\n\n| Method | Description |\n|--------|-------------|\n| `add_task(task)` | Add task (TaskStar or dict), returns TaskStar |\n| `create_and_add_task(task_id, description, name, **kwargs)` | Create and add new task, returns TaskStar |\n| `update_task(task_id, **updates)` | Update task properties, returns updated TaskStar |\n| `remove_task(task_id)` | Remove task and related dependencies, returns removed task ID (str) |\n| `get_task(task_id)` | Get task by ID, returns Optional[TaskStar] |\n| `list_tasks()` | Get all tasks, returns List[TaskStar] |\n\n### Dependency Operations\n\n| Method | Description |\n|--------|-------------|\n| `add_dependency(dependency)` | Add dependency (TaskStarLine or dict), returns TaskStarLine |\n| `create_and_add_dependency(from_id, to_id, type, **kwargs)` | Create and add dependency, returns TaskStarLine |\n| `update_dependency(dependency_id, **updates)` | Update dependency properties, returns updated TaskStarLine |\n| `remove_dependency(dependency_id)` | Remove dependency, returns removed dependency ID (str) |\n| `get_dependency(dependency_id)` | Get dependency by ID, returns Optional[TaskStarLine] |\n| `list_dependencies()` | Get all dependencies, returns List[TaskStarLine] |\n| `get_task_dependencies(task_id)` | Get dependencies for specific task, returns List[TaskStarLine] |\n\n### Bulk Operations\n\n| Method | Description |\n|--------|-------------|\n| `build_constellation(config, clear_existing)` | Build constellation from TaskConstellationSchema |\n| `build_from_tasks_and_dependencies(tasks, deps, ...)` | Build constellation from task and dependency lists (returns TaskConstellation) |\n| `clear_constellation()` | Remove all tasks and dependencies from constellation |\n| `batch_operations(operations)` | Execute multiple operations in sequence, returning list of results |\n\n### File Operations\n\n| Method | Description |\n|--------|-------------|\n| `save_constellation(file_path)` | Save constellation to JSON file, returns file path |\n| `load_constellation(file_path)` | Load constellation from JSON file, returns TaskConstellation |\n| `load_from_dict(data)` | Load constellation from dictionary, returns TaskConstellation |\n| `load_from_json_string(json_string)` | Load constellation from JSON string, returns TaskConstellation |\n\n### History Operations\n\n| Method | Description |\n|--------|-------------|\n| `undo()` | Undo last command, returns True if successful, False if no undo available |\n| `redo()` | Redo next command, returns True if successful, False if no redo available |\n| `can_undo()` | Check if undo is available (returns bool) |\n| `can_redo()` | Check if redo is available (returns bool) |\n| `get_undo_description()` | Get description of operation that would be undone (returns Optional[str]) |\n| `get_redo_description()` | Get description of operation that would be redone (returns Optional[str]) |\n| `clear_history()` | Clear command history (no return value) |\n| `get_history()` | Get list of command descriptions (returns List[str]) |\n\n### Validation\n\n| Method | Description |\n|--------|-------------|\n| `validate_constellation()` | Validate DAG structure, returns tuple of (is_valid: bool, errors: List[str]) |\n| `has_cycles()` | Check for cycles in the DAG, returns bool |\n| `get_topological_order()` | Get topological ordering of tasks, returns List[str] of task IDs |\n| `get_ready_tasks()` | Get tasks ready to execute (no pending dependencies), returns List[TaskStar] |\n| `get_statistics()` | Get comprehensive constellation and editor statistics, returns Dict[str, Any] |\n\n### Observers\n\n| Method | Description |\n|--------|-------------|\n| `add_observer(observer)` | Add change observer callable that receives (editor, command, result) |\n| `remove_observer(observer)` | Remove previously added observer |\n\n### Advanced\n\n| Method | Description |\n|--------|-------------|\n| `create_subgraph(task_ids)` | Extract subgraph with specific tasks |\n| `merge_constellation(other_editor, prefix)` | Merge another constellation with optional ID prefix |\n| `display_constellation(mode)` | Display visualization (modes: 'overview', 'topology', 'details', 'execution') |\n\nFor interactive web-based visualization and editing, see the [Galaxy WebUI](../webui.md).\n\n---\n\n**ConstellationEditor** — Safe, interactive, and reversible constellation manipulation\n"
},
{
"path": "documents/docs/galaxy/constellation/overview.md",
"content": "# Task Constellation — Overview\n\n
\n \n
Example of a Task Constellation illustrating both sequential and parallel dependencies
\n
\n\n---\n\n## 🌌 Introduction\n\nThe **Task Constellation** is the central abstraction in Galaxy that captures the concurrent and asynchronous structure of distributed task execution. It provides a formal, directed acyclic graph (DAG) representation of complex workflows, enabling consistent scheduling, fault-tolerant orchestration, and runtime dynamism across heterogeneous devices.\n\nAt its core, a Task Constellation decomposes complex user requests into interdependent subtasks connected through explicit dependency edges. This formalism not only enables correct distributed execution but also supports runtime adaptation—allowing new tasks or dependencies to be introduced as the workflow evolves.\n\nFor information on how Task Constellations are orchestrated and scheduled, see the [Constellation Orchestrator](../constellation_orchestrator/overview.md) documentation. To understand how agents interact with constellations, refer to the [Constellation Agent](../constellation_agent/overview.md) guide.\n\n---\n\n## 🎯 Core Components\n\nThe Task Constellation framework consists of four primary components:\n\n| Component | Purpose | Key Features |\n|-----------|---------|--------------|\n| **[TaskStar](task_star.md)** | Atomic execution unit | Self-contained task with description, device assignment, execution state, dependencies |\n| **[TaskStarLine](task_star_line.md)** | Dependency relationship | Directed edge with conditional logic, success-only, completion-only, or unconditional execution |\n| **[TaskConstellation](task_constellation.md)** | DAG orchestrator | Complete workflow graph with validation, scheduling, and dynamic modification |\n| **[ConstellationEditor](constellation_editor.md)** | Interactive editor | Command pattern-based interface with undo/redo for safe constellation manipulation |\n\n---\n\n## 📐 Formal Model\n\n### Mathematical Foundation\n\nA Task Constellation $\\mathcal{C}$ is formally defined as a directed acyclic graph (DAG):\n\n$$\n\\mathcal{C} = (\\mathcal{T}, \\mathcal{E})\n$$\n\nwhere:\n- $\\mathcal{T}$ is the set of all **TaskStars** (task nodes)\n- $\\mathcal{E}$ is the set of **TaskStarLines** (dependency edges)\n\n### TaskStar Representation\n\nEach TaskStar $t_i \\in \\mathcal{T}$ encapsulates a complete task specification:\n\n$$\nt_i = (\\text{name}_ i, \\text{description}_ i, \\text{target\\_device\\_id}_ i, \\text{tips}_ i, \\text{status}_ i, \\text{dependencies}_ i)\n$$\n\n**Components:**\n- **name**: Short name for the task\n- **description**: Natural-language specification sent to the device agent\n- **target_device_id**: ID of the device agent responsible for execution\n- **tips**: List of guidance hints to help the device agent complete the task\n- **status**: Current execution state (pending, running, completed, failed, cancelled, waiting_dependency)\n- **dependencies**: Set of prerequisite task IDs that must complete first\n\n### TaskStarLine Representation\n\nEach TaskStarLine $e_{i \\rightarrow j} \\in \\mathcal{E}$ represents a dependency from task $t_i$ to task $t_j$.\n\n**Dependency Types:**\n\n| Type | Behavior |\n|------|----------|\n| **Unconditional** | $t_j$ always waits for $t_i$ to complete |\n| **Success-only** | $t_j$ proceeds only if $t_i$ succeeds |\n| **Completion-only** | $t_j$ proceeds when $t_i$ completes (regardless of success/failure) |\n| **Conditional** | $t_j$ proceeds based on a user-defined or runtime condition |\n\n---\n\n## ✨ Key Advantages\n\n### 1. Explicit Task Ordering\nTask dependencies are explicitly captured in the DAG structure, ensuring correctness across distributed execution without ambiguity.\n\n### 2. Natural Parallelism\nThe DAG topology naturally exposes parallelizable tasks, enabling efficient concurrent execution across heterogeneous devices.\n\n### 3. Runtime Dynamism\nUnlike static DAG schedulers, Task Constellations are **mutable objects**. Tasks and dependency edges can be:\n- **Added**: Introduce new subtasks or diagnostic tasks\n- **Removed**: Prune completed or redundant nodes\n- **Modified**: Rewire dependencies, update conditions, change device assignments\n\nThis enables adaptive execution without restarting the entire workflow.\n\n### 4. Formal Guarantees\nThe DAG representation provides formal properties:\n- **Acyclicity**: No circular dependencies\n- **Causal consistency**: Execution respects logical ordering\n- **Safe concurrency**: Parallel execution without race conditions\n\n---\n\n## 🔄 Lifecycle States\n\nThe Task Constellation progresses through several states during its lifecycle:\n\n```mermaid\nstateDiagram-v2\n [*] --> CREATED: Initialize\n CREATED --> READY: Add tasks & dependencies\n READY --> EXECUTING: Start execution\n EXECUTING --> EXECUTING: Tasks running\n EXECUTING --> COMPLETED: All tasks succeed\n EXECUTING --> FAILED: All tasks fail\n EXECUTING --> PARTIALLY_FAILED: Some succeed, some fail\n COMPLETED --> [*]\n FAILED --> [*]\n PARTIALLY_FAILED --> [*]\n```\n\n| State | Description |\n|-------|-------------|\n| **CREATED** | Constellation initialized, no tasks added |\n| **READY** | Tasks and dependencies configured, ready to execute |\n| **EXECUTING** | At least one task is running or completed |\n| **COMPLETED** | All tasks completed successfully |\n| **FAILED** | All tasks failed |\n| **PARTIALLY_FAILED** | Some tasks succeeded, some failed |\n\n---\n\n## 📊 DAG Metrics\n\n### Parallelism Analysis\n\nThe Task Constellation provides several metrics to analyze workflow parallelism:\n\n#### Critical Path Length ($L$)\nThe longest serial dependency chain in the constellation:\n\n$$\nL = \\max_{p \\in \\text{paths}} |p|\n$$\n\nwhere $|p|$ is the length of path $p$ from any root to any leaf node.\n\n#### Total Work ($W$)\nSum of all task execution durations:\n\n$$\nW = \\sum_{t_i \\in \\mathcal{T}} \\text{duration}(t_i)\n$$\n\n#### Parallelism Ratio ($P$)\nMeasure of achievable parallelism:\n\n$$\nP = \\frac{W}{L}\n$$\n\n- $P = 1$: Completely serial execution\n- $P > 1$: Parallel execution possible\n- Higher $P$ indicates more parallelism\n\n#### Maximum Width\nMaximum number of tasks that can execute concurrently:\n\n$$\n\\text{MaxWidth} = \\max_{\\text{level}} |\\text{tasks at level}|\n$$\n\n!!!info \"Calculation Modes\"\n The constellation supports two calculation modes:\n \n - **Node Count Mode**: Uses task counts when execution is incomplete\n - **Actual Time Mode**: Uses real execution durations when all tasks are terminal\n\n---\n\n## 🛠️ Core Operations\n\n### DAG Construction\n\n```python\nfrom galaxy.constellation import TaskConstellation, TaskStar, TaskStarLine\n\n# Create constellation\nconstellation = TaskConstellation(name=\"my_workflow\")\n\n# Add tasks\ntask_a = TaskStar(name=\"task_a\", description=\"Checkout code on laptop\")\ntask_b = TaskStar(name=\"task_b\", description=\"Build on GPU server\")\ntask_c = TaskStar(name=\"task_c\", description=\"Deploy to staging\")\n\nconstellation.add_task(task_a)\nconstellation.add_task(task_b)\nconstellation.add_task(task_c)\n\n# Add dependencies\ndep_ab = TaskStarLine.create_success_only(\n from_task_id=task_a.task_id,\n to_task_id=task_b.task_id,\n description=\"Build depends on successful checkout\"\n)\n\ndep_bc = TaskStarLine.create_unconditional(\n from_task_id=task_b.task_id,\n to_task_id=task_c.task_id,\n description=\"Deploy after build\"\n)\n\nconstellation.add_dependency(dep_ab)\nconstellation.add_dependency(dep_bc)\n```\n\n### DAG Validation\n\n```python\n# Validate structure\nis_valid, errors = constellation.validate_dag()\nif not is_valid:\n print(f\"Validation errors: {errors}\")\n\n# Check for cycles\nhas_cycles = constellation.has_cycle()\n\n# Get topological order\norder = constellation.get_topological_order()\nprint(f\"Execution order: {order}\")\n```\n\n### Parallelism Analysis\n\n```python\n# Get parallelism metrics\nmetrics = constellation.get_parallelism_metrics()\n\nprint(f\"Critical Path Length: {metrics['critical_path_length']}\")\nprint(f\"Total Work: {metrics['total_work']}\")\nprint(f\"Parallelism Ratio: {metrics['parallelism_ratio']}\")\nprint(f\"Critical Path: {metrics['critical_path_tasks']}\")\n\n# Get maximum width\nmax_width = constellation.get_max_width()\nprint(f\"Maximum concurrent tasks: {max_width}\")\n```\n\n---\n\n## 🔧 Dynamic Modification\n\n### Safe Editing with ConstellationEditor\n\n```python\nfrom galaxy.constellation.editor import ConstellationEditor\n\n# Create editor with undo/redo support\neditor = ConstellationEditor(constellation)\n\n# Add a new diagnostic task\ndiagnostic_task = editor.create_and_add_task(\n task_id=\"diag_1\",\n description=\"Check server health\",\n name=\"Server Health Check\"\n)\n\n# Add conditional dependency\neditor.create_and_add_dependency(\n from_task_id=task_b.task_id,\n to_task_id=diagnostic_task.task_id,\n dependency_type=\"CONDITIONAL\",\n condition_description=\"Run diagnostic if build fails\"\n)\n\n# Undo if needed\nif something_wrong:\n editor.undo()\n\n# Get modifiable components\nmodifiable_tasks = constellation.get_modifiable_tasks()\nmodifiable_deps = constellation.get_modifiable_dependencies()\n```\n\n!!!warning \"Modification Safety\"\n Tasks and dependencies can only be modified if they are in `PENDING` or `WAITING_DEPENDENCY` status. Running or completed tasks cannot be modified to ensure execution consistency.\n\n---\n\n## 📈 Example Workflows\n\n### Sequential Workflow\n\n```mermaid\ngraph LR\n A[Task A] --> B[Task B]\n B --> C[Task C]\n```\n\n- **Parallelism Ratio**: 1.0 (completely serial)\n- **Maximum Width**: 1\n\n### Parallel Workflow\n\n```mermaid\ngraph LR\n A[Task A] --> B[Task B]\n A --> C[Task C]\n B --> D[Task D]\n C --> D\n```\n\n- **Parallelism Ratio**: 2.0 (B and C can run in parallel)\n- **Maximum Width**: 2\n\n### Complex Workflow\n\n```mermaid\ngraph LR\n A[Task A] --> B[Task B]\n A --> C[Task C]\n B --> D[Task D]\n C --> E[Task E]\n D --> F[Task F]\n E --> F\n```\n\n- **Parallelism Ratio**: ~1.67\n- **Maximum Width**: 3 (B, C, E can run concurrently after A completes)\n\n---\n\n## 🎨 Visualization\n\nThe Task Constellation provides multiple visualization modes for monitoring and debugging:\n\n### Overview Mode\nHigh-level constellation structure with task counts and state\n\n### Topology Mode\nDAG graph showing task relationships and dependencies\n\n### Details Mode\nDetailed task information including execution times and status\n\n### Execution Mode\nReal-time execution flow with progress tracking\n\n```python\n# Display constellation\nconstellation.display_dag(mode=\"overview\") # or \"topology\", \"details\", \"execution\"\n```\n\nFor interactive web-based visualization, check out the [Galaxy WebUI](../webui.md).\n\n---\n\n## 📚 Component Documentation\n\nExplore detailed documentation for each component:\n\n- **[TaskStar](task_star.md)** — Atomic execution units representing individual tasks in the constellation\n- **[TaskStarLine](task_star_line.md)** — Dependency relationships connecting tasks with conditional logic\n- **[TaskConstellation](task_constellation.md)** — Complete DAG orchestrator managing workflow execution and coordination\n- **[ConstellationEditor](constellation_editor.md)** — Interactive editor with command pattern and undo/redo capabilities\n\n### Related Documentation\n\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Learn how constellations are scheduled and executed across devices\n- **[Constellation Agent](../constellation_agent/overview.md)** — Understand how agents plan and manage constellation lifecycles\n- **[Evaluation & Metrics](../evaluation/performance_metrics.md)** — Monitor constellation performance and analyze execution patterns\n\n---\n\n## 🔬 Research Background\n\nThe Task Constellation model is grounded in formal DAG theory and distributed systems research. Key properties include:\n\n- **Acyclicity guarantees** through Kahn's algorithm for topological sorting\n- **Topological ordering** for consistent execution\n- **Critical path analysis** for performance optimization\n- **Dynamic graph evolution** without compromising consistency\n\nFor more on Galaxy's architecture and design principles, see the [Galaxy Overview](../overview.md).\n\n---\n\n## 💡 Best Practices\n\n!!!tip \"Designing Effective Constellations\"\n 1. **Keep tasks atomic**: Each TaskStar should represent a single, well-defined operation\n 2. **Minimize dependencies**: Reduce unnecessary dependencies to maximize parallelism\n 3. **Use appropriate dependency types**: Choose conditional dependencies for error handling\n 4. **Validate early**: Run `validate_dag()` before execution\n 5. **Monitor metrics**: Track parallelism ratio to optimize workflow design\n\n**Common Patterns:**\n\n- **Fan-out**: One task spawns multiple independent parallel tasks\n- **Fan-in**: Multiple parallel tasks converge to a single task\n- **Pipeline**: Sequential stages with parallel tasks within each stage\n- **Conditional branching**: Use conditional dependencies for error handling paths\n\n---\n\n## 🚀 Next Steps\n\n- Learn about **[TaskStar](task_star.md)** — Atomic task execution units\n- Explore **[TaskStarLine](task_star_line.md)** — Dependency relationships\n- Master **[TaskConstellation](task_constellation.md)** — DAG orchestration\n- Try **[ConstellationEditor](constellation_editor.md)** — Interactive editing\n"
},
{
"path": "documents/docs/galaxy/constellation/task_constellation.md",
"content": "# TaskConstellation — DAG Orchestrator\n\n## Overview\n\n**TaskConstellation** is the complete DAG (Directed Acyclic Graph) orchestration system that manages distributed workflows across heterogeneous devices. It provides comprehensive task management, dependency validation, execution scheduling, and runtime dynamism for complex cross-device orchestration.\n\n**Formal Definition:** A TaskConstellation $\\mathcal{C}$ is a DAG defined as:\n\n$$\n\\mathcal{C} = (\\mathcal{T}, \\mathcal{E})\n$$\n\nwhere $\\mathcal{T}$ is the set of TaskStars and $\\mathcal{E}$ is the set of TaskStarLines.\n\n---\n\n## Architecture\n\n### Core Components\n\n| Component | Type | Description |\n|-----------|------|-------------|\n| **constellation_id** | `str` | Unique identifier for the constellation |\n| **name** | `str` | Human-readable constellation name |\n| **state** | `ConstellationState` | Current execution state |\n| **tasks** | `Dict[str, TaskStar]` | All tasks in the constellation |\n| **dependencies** | `Dict[str, TaskStarLine]` | All dependency relationships |\n| **metadata** | `Dict[str, Any]` | Additional constellation metadata |\n\n### Execution Tracking\n\n| Property | Type | Description |\n|----------|------|-------------|\n| **execution_start_time** | `datetime` | When execution started |\n| **execution_end_time** | `datetime` | When execution completed |\n| **execution_duration** | `float` | Total execution time in seconds |\n| **created_at** | `datetime` | Constellation creation timestamp |\n| **updated_at** | `datetime` | Last modification timestamp |\n\n---\n\n## Constellation Lifecycle\n\n```mermaid\nstateDiagram-v2\n [*] --> CREATED: Initialize\n CREATED --> READY: Add tasks & validate\n READY --> EXECUTING: Start execution\n EXECUTING --> EXECUTING: Tasks running\n EXECUTING --> COMPLETED: All succeed\n EXECUTING --> FAILED: All fail\n EXECUTING --> PARTIALLY_FAILED: Mixed results\n COMPLETED --> [*]\n FAILED --> [*]\n PARTIALLY_FAILED --> [*]\n```\n\n### State Definitions\n\n| State | Description | Transition Trigger |\n|-------|-------------|-------------------|\n| **CREATED** | Empty constellation, no tasks added | Initialization |\n| **READY** | Tasks added, validated, ready to execute | Tasks added, no running tasks |\n| **EXECUTING** | At least one task running or completed | First task starts |\n| **COMPLETED** | All tasks completed successfully | Last task succeeds |\n| **FAILED** | All tasks failed | Last task fails, no successes |\n| **PARTIALLY_FAILED** | Some tasks succeeded, some failed | Mixed terminal states |\n\n---\n\n## Core Operations\n\n### Creating a Constellation\n\n```python\nfrom galaxy.constellation import TaskConstellation\n\n# Create with auto-generated ID\nconstellation = TaskConstellation()\nprint(f\"ID: {constellation.constellation_id}\")\n# Output: constellation_20251106_143052_a1b2c3d4\n\n# Create with custom name\nconstellation = TaskConstellation(\n name=\"ml_training_pipeline\",\n constellation_id=\"pipeline_001\"\n)\n```\n\n---\n\n### Adding Tasks\n\n```python\nfrom galaxy.constellation import TaskStar\n\n# Create tasks\ntask_a = TaskStar(\n task_id=\"fetch_data\",\n description=\"Download training dataset\",\n target_device_id=\"linux_server_1\"\n)\n\ntask_b = TaskStar(\n task_id=\"preprocess\",\n description=\"Preprocess and normalize data\",\n target_device_id=\"linux_server_2\"\n)\n\ntask_c = TaskStar(\n task_id=\"train_model\",\n description=\"Train neural network\",\n target_device_id=\"gpu_server_1\"\n)\n\n# Add to constellation\nconstellation.add_task(task_a)\nconstellation.add_task(task_b)\nconstellation.add_task(task_c)\n\nprint(f\"Total tasks: {constellation.task_count}\")\n# Output: Total tasks: 3\n```\n\n---\n\n### Adding Dependencies\n\n```python\nfrom galaxy.constellation import TaskStarLine\n\n# Create dependencies\ndep1 = TaskStarLine.create_success_only(\n from_task_id=\"fetch_data\",\n to_task_id=\"preprocess\",\n description=\"Preprocess after successful download\"\n)\n\ndep2 = TaskStarLine.create_success_only(\n from_task_id=\"preprocess\",\n to_task_id=\"train_model\",\n description=\"Train on preprocessed data\"\n)\n\n# Add to constellation\nconstellation.add_dependency(dep1)\nconstellation.add_dependency(dep2)\n\nprint(f\"Total dependencies: {constellation.dependency_count}\")\n# Output: Total dependencies: 2\n```\n\n---\n\n### Removing Tasks and Dependencies\n\n```python\n# Remove a task (also removes related dependencies)\nconstellation.remove_task(\"preprocess\")\n\n# Remove a dependency\nconstellation.remove_dependency(dep1.line_id)\n\n# Get specific task or dependency\ntask = constellation.get_task(\"fetch_data\")\ndep = constellation.get_dependency(dep1.line_id)\n```\n\n---\n\n## DAG Validation\n\n### Cycle Detection\n\n```python\n# Check for cycles\nhas_cycles = constellation.has_cycle()\n\nif has_cycles:\n print(\"❌ Constellation contains cycles!\")\nelse:\n print(\"✅ DAG is acyclic\")\n\n# Comprehensive validation\nis_valid, errors = constellation.validate_dag()\n\nif not is_valid:\n for error in errors:\n print(f\"❌ {error}\")\nelse:\n print(\"✅ Constellation is valid\")\n```\n\n### Topological Ordering\n\n```python\ntry:\n # Get topological order (throws if cyclic)\n order = constellation.get_topological_order()\n print(f\"Execution order: {' → '.join(order)}\")\n # Output: fetch_data → preprocess → train_model\n \nexcept ValueError as e:\n print(f\"Cannot get topological order: {e}\")\n```\n\n---\n\n## Scheduling and Execution\n\n### Getting Ready Tasks\n\n```python\n# Get tasks ready to execute (no pending dependencies)\nready_tasks = constellation.get_ready_tasks()\n\nfor task in ready_tasks:\n print(f\"Ready: {task.name} (priority: {task.priority.value})\")\n # Tasks are sorted by priority (highest first)\n```\n\n### Execution Flow\n\n```python\n# Start constellation execution\nconstellation.start_execution()\n\n# Start a specific task\nconstellation.start_task(\"fetch_data\")\n\n# Mark task as completed\nnewly_ready = constellation.mark_task_completed(\n task_id=\"fetch_data\",\n success=True,\n result={\"rows\": 10000, \"status\": \"success\"}\n)\n\n# newly_ready contains tasks that became ready after this completion\nfor task in newly_ready:\n print(f\"Now ready: {task.name}\")\n```\n\n### Querying Task Status\n\n```python\n# Get tasks by status\nrunning = constellation.get_running_tasks()\ncompleted = constellation.get_completed_tasks()\nfailed = constellation.get_failed_tasks()\npending = constellation.get_pending_tasks()\n\nprint(f\"Running: {len(running)}\")\nprint(f\"Completed: {len(completed)}\")\nprint(f\"Failed: {len(failed)}\")\nprint(f\"Pending: {len(pending)}\")\n\n# Check if entire constellation is complete\nif constellation.is_complete():\n constellation.complete_execution()\n print(f\"State: {constellation.state}\")\n```\n\n---\n\n## Parallelism Analysis\n\n### DAG Metrics\n\n```python\n# Get longest path (critical path) using node counts\nlongest_path_length, longest_path = constellation.get_longest_path()\n\nprint(f\"Critical path length: {longest_path_length}\")\nprint(f\"Critical path: {' → '.join(longest_path)}\")\n\n# Get maximum width (max concurrent tasks)\nmax_width = constellation.get_max_width()\nprint(f\"Maximum parallelism: {max_width} tasks\")\n```\n\n### Parallelism Ratio\n\n```python\n# Calculate parallelism metrics (L, W, P)\nmetrics = constellation.get_parallelism_metrics()\n\nprint(f\"Critical Path Length (L): {metrics['critical_path_length']}\")\nprint(f\"Total Work (W): {metrics['total_work']}\")\nprint(f\"Parallelism Ratio (P): {metrics['parallelism_ratio']:.2f}\")\nprint(f\"Calculation Mode: {metrics['calculation_mode']}\")\n\n# Interpretation:\n# P = 1.0 → Completely serial\n# P = 2.0 → 2x parallelism on average\n# P = 3.5 → 3.5x parallelism on average\n```\n\n**Note:** Calculation modes depend on task completion status:\n- **node_count**: Used when tasks are incomplete (counts each task as 1 unit)\n- **actual_time**: Used when all tasks are terminal (uses real execution durations)\n\n### Time-Based Critical Path\n\n```python\n# Get critical path using actual execution times\n# Only valid when all tasks are completed or failed\ncritical_time, critical_path_tasks = constellation.get_critical_path_length_with_time()\n\nprint(f\"Critical path duration: {critical_time:.2f} seconds\")\nprint(f\"Tasks on critical path: {critical_path_tasks}\")\n\n# Get total work\ntotal_work = constellation.get_total_work()\nprint(f\"Total work: {total_work:.2f} seconds\")\n\n# Calculate speedup\nspeedup = total_work / critical_time if critical_time > 0 else 0\nprint(f\"Speedup: {speedup:.2f}x\")\n```\n\n---\n\n## Statistics and Monitoring\n\n### Comprehensive Statistics\n\n```python\nstats = constellation.get_statistics()\n\nprint(f\"Constellation: {stats['name']}\")\nprint(f\"State: {stats['state']}\")\nprint(f\"Tasks: {stats['total_tasks']}\")\nprint(f\"Dependencies: {stats['total_dependencies']}\")\nprint(f\"Longest Path: {stats['longest_path_length']}\")\nprint(f\"Max Width: {stats['max_width']}\")\nprint(f\"Parallelism Ratio: {stats['parallelism_ratio']:.2f}\")\n\n# Task status breakdown\nstatus_counts = stats['task_status_counts']\nfor status, count in status_counts.items():\n print(f\" {status}: {count}\")\n\n# Execution duration\nif stats['execution_duration']:\n print(f\"Duration: {stats['execution_duration']:.2f} seconds\")\n```\n\n---\n\n## Dynamic Modification\n\n### Modifiable Components\n\n```python\n# Get tasks that can be safely modified\nmodifiable_tasks = constellation.get_modifiable_tasks()\n# Only tasks in PENDING or WAITING_DEPENDENCY status\n\n# Get modifiable dependencies\nmodifiable_deps = constellation.get_modifiable_dependencies()\n# Dependencies whose target task hasn't started\n\n# Check specific task/dependency\ncan_modify_task = constellation.is_task_modifiable(\"task_a\")\ncan_modify_dep = constellation.is_dependency_modifiable(\"dep_1\")\n```\n\n### Runtime Graph Evolution\n\n```python\n# Add diagnostic task during execution\ndiagnostic_task = TaskStar(\n task_id=\"health_check\",\n description=\"Check server health after failure\"\n)\nconstellation.add_task(diagnostic_task)\n\n# Add conditional fallback dependency\nfallback_dep = TaskStarLine.create_conditional(\n from_task_id=\"train_model\",\n to_task_id=\"health_check\",\n condition_description=\"Run health check if training fails\",\n condition_evaluator=lambda result: result is None\n)\nconstellation.add_dependency(fallback_dep)\n\n# Update constellation state\nconstellation.update_state()\n```\n\n!!! warning \"Modification Safety\"\n The constellation enforces safe modification:\n \n - **RUNNING tasks**: Cannot be modified\n - **Completed/Failed tasks**: Cannot be modified\n - **Dependencies to running tasks**: Cannot be modified\n \n This ensures execution consistency and prevents race conditions.\n\n---\n\n## Serialization and Persistence\n\n### JSON Export/Import\n\n```python\n# Export to JSON string\njson_string = constellation.to_json()\n\n# Save to file\nconstellation.to_json(save_path=\"constellation_backup.json\")\n\n# Load from JSON string\nrestored = TaskConstellation.from_json(json_data=json_string)\n\n# Load from file\nloaded = TaskConstellation.from_json(file_path=\"constellation_backup.json\")\n```\n\n### Dictionary Conversion\n\n```python\n# Convert to dictionary\nconstellation_dict = constellation.to_dict()\n\n# Create from dictionary\nnew_constellation = TaskConstellation.from_dict(constellation_dict)\n\n# Dictionary structure includes:\n# - constellation_id, name, state\n# - tasks (dict of task_id -> TaskStar dict)\n# - dependencies (dict of line_id -> TaskStarLine dict)\n# - metadata, timestamps\n```\n\n### Pydantic Schema\n\n```python\n# Convert to Pydantic BaseModel\nschema = constellation.to_basemodel()\n\n# Create from schema\nconstellation_from_schema = TaskConstellation.from_basemodel(schema)\n```\n\n---\n\n## Visualization\n\n### Display Modes\n\n```python\n# Overview mode - high-level structure\nconstellation.display_dag(mode=\"overview\")\n\n# Topology mode - detailed DAG graph\nconstellation.display_dag(mode=\"topology\")\n\n# Details mode - task execution details\nconstellation.display_dag(mode=\"details\")\n\n# Execution mode - real-time flow\nconstellation.display_dag(mode=\"execution\")\n```\n\n---\n\n## Querying Dependencies\n\n### Task-Specific Dependencies\n\n```python\n# Get all dependencies for a specific task\ntask_deps = constellation.get_task_dependencies(\"train_model\")\n\nfor dep in task_deps:\n print(f\"{dep.from_task_id} → {dep.to_task_id} ({dep.dependency_type.value})\")\n\n# Get all dependencies in constellation\nall_deps = constellation.get_all_dependencies()\n```\n\n---\n\n## Example Workflows\n\n### Simple Linear Pipeline\n\n```mermaid\ngraph LR\n A[Task A] --> B[Task B]\n B --> C[Task C]\n```\n\n```python\n# Create: A → B → C\nconstellation = TaskConstellation(name=\"linear_pipeline\")\n\ntask_a = TaskStar(task_id=\"a\", description=\"Task A\")\ntask_b = TaskStar(task_id=\"b\", description=\"Task B\")\ntask_c = TaskStar(task_id=\"c\", description=\"Task C\")\n\nconstellation.add_task(task_a)\nconstellation.add_task(task_b)\nconstellation.add_task(task_c)\n\ndep_ab = TaskStarLine.create_unconditional(\"a\", \"b\")\ndep_bc = TaskStarLine.create_unconditional(\"b\", \"c\")\n\nconstellation.add_dependency(dep_ab)\nconstellation.add_dependency(dep_bc)\n\n# Validate\nis_valid, errors = constellation.validate_dag()\nassert is_valid\n\n# Get metrics\nmetrics = constellation.get_parallelism_metrics()\nassert metrics['parallelism_ratio'] == 1.0 # Completely serial\n```\n\n### Parallel Fan-Out\n\n```mermaid\ngraph LR\n A[Task A] --> B[Task B]\n A --> C[Task C]\n A --> D[Task D]\n```\n\n```python\n# Create: A → [B, C, D]\nconstellation = TaskConstellation(name=\"fan_out\")\n\ntask_a = TaskStar(task_id=\"a\", description=\"Root task\")\ntask_b = TaskStar(task_id=\"b\", description=\"Parallel task 1\")\ntask_c = TaskStar(task_id=\"c\", description=\"Parallel task 2\")\ntask_d = TaskStar(task_id=\"d\", description=\"Parallel task 3\")\n\nfor task in [task_a, task_b, task_c, task_d]:\n constellation.add_task(task)\n\n# All three tasks depend on A, can run in parallel\nfor target_id in [\"b\", \"c\", \"d\"]:\n dep = TaskStarLine.create_success_only(\"a\", target_id)\n constellation.add_dependency(dep)\n\n# Get metrics\nmetrics = constellation.get_parallelism_metrics()\nassert metrics['max_width'] >= 3 # Can run 3 tasks in parallel\n```\n\n### Complex Diamond Pattern\n\n```mermaid\ngraph LR\n A[Task A] --> B[Task B]\n A --> C[Task C]\n B --> D[Task D]\n C --> D\n```\n\n```python\n# Create: A → [B, C] → D\nconstellation = TaskConstellation(name=\"diamond\")\n\ntasks = {\n \"a\": TaskStar(task_id=\"a\", description=\"Start\"),\n \"b\": TaskStar(task_id=\"b\", description=\"Path 1\"),\n \"c\": TaskStar(task_id=\"c\", description=\"Path 2\"),\n \"d\": TaskStar(task_id=\"d\", description=\"Merge\")\n}\n\nfor task in tasks.values():\n constellation.add_task(task)\n\n# Fan-out: A → B, A → C\nconstellation.add_dependency(TaskStarLine.create_success_only(\"a\", \"b\"))\nconstellation.add_dependency(TaskStarLine.create_success_only(\"a\", \"c\"))\n\n# Fan-in: B → D, C → D\nconstellation.add_dependency(TaskStarLine.create_success_only(\"b\", \"d\"))\nconstellation.add_dependency(TaskStarLine.create_success_only(\"c\", \"d\"))\n\n# Analyze\norder = constellation.get_topological_order()\nprint(f\"Possible order: {order}\") # ['a', 'b', 'c', 'd'] or ['a', 'c', 'b', 'd']\n\nlongest_path_length, path = constellation.get_longest_path()\nassert longest_path_length == 3 # A → B/C → D\n```\n\n---\n\n## Error Handling\n\n### Cycle Detection\n\n```python\n# Attempt to create a cycle\ntry:\n # This would create A → B → C → A\n constellation.add_dependency(\n TaskStarLine.create_unconditional(\"c\", \"a\")\n )\nexcept ValueError as e:\n print(f\"❌ {e}\")\n # Output: Adding dependency would create a cycle\n```\n\n### Missing Task References\n\n```python\n# Try to add dependency with non-existent task\ntry:\n dep = TaskStarLine.create_unconditional(\n \"nonexistent_task\",\n \"task_b\"\n )\n constellation.add_dependency(dep)\nexcept ValueError as e:\n print(f\"❌ {e}\")\n # Output: Source task nonexistent_task not found\n```\n\n### Modifying Running Tasks\n\n```python\n# Try to remove a running task\ntask.start_execution()\n\ntry:\n constellation.remove_task(task.task_id)\nexcept ValueError as e:\n print(f\"❌ {e}\")\n # Output: Cannot remove running task\n```\n\n---\n\n## Best Practices\n\n### Constellation Design Guidelines\n\n1. **Validate early**: Run `validate_dag()` before execution\n2. **Minimize dependencies**: Reduce unnecessary edges to maximize parallelism\n3. **Use appropriate dependency types**: Match dependency type to workflow logic\n4. **Monitor metrics**: Track parallelism ratio to optimize design\n5. **Handle failures**: Use conditional dependencies for error recovery\n\n### Optimization Patterns\n\n**Before (Serial):**\n\n```mermaid\ngraph LR\n A[A] --> B[B]\n B --> C[C]\n C --> D[D]\n D --> E[E]\n E --> F[F]\n```\n\nParallelism Ratio: 1.0\n\n**After (Optimized):**\n\n```mermaid\ngraph LR\n A[A] --> B[B]\n A --> C[C]\n A --> D[D]\n B --> F[F]\n C --> F\n D --> E[E]\n```\n\nParallelism Ratio: 1.67\n\n!!! warning \"Common Pitfalls\"\n - **Over-parallelization**: Too many parallel tasks can overwhelm resources\n - **Tight coupling**: Excessive dependencies reduce parallelism\n - **Missing validation**: Always validate before execution\n - **Ignoring state**: Check constellation state before modifications\n\n---\n\n## Formal Properties\n\n### Acyclicity Guarantee\n\nThe TaskConstellation enforces **acyclicity** through:\n\n1. **DFS-based cycle detection** before adding dependencies\n2. **Topological ordering** validation using Kahn's algorithm\n3. **Runtime validation** during DAG modification\n\n### Causal Consistency\n\nTask dependencies ensure **causal consistency**:\n\n- If task $t_j$ depends on $t_i$, then $t_i$ must complete before $t_j$ starts\n- Transitive dependencies are preserved\n- Concurrent tasks have no causal ordering\n\n### Concurrency Safety\n\nThe constellation provides **safe concurrent execution**:\n\n- **Read-only queries** are always safe\n- **Modifications** are protected by state checks\n- **Assignment locking** prevents race conditions (handled by orchestrator)\n\n---\n\n## Related Components\n\n- **[TaskStar](task_star.md)** — Atomic task execution units\n- **[TaskStarLine](task_star_line.md)** — Dependency relationships\n- **[ConstellationEditor](constellation_editor.md)** — Safe editing with undo/redo\n- **[Overview](overview.md)** — Framework overview\n\n---\n\n## API Reference\n\n### Constructor\n\n```python\nTaskConstellation(\n constellation_id: Optional[str] = None,\n name: Optional[str] = None\n)\n```\n\n### Task Management\n\n| Method | Description |\n|--------|-------------|\n| `add_task(task)` | Add task to constellation |\n| `remove_task(task_id)` | Remove task and related dependencies |\n| `get_task(task_id)` | Get task by ID |\n| `get_all_tasks()` | Get all tasks |\n| `get_ready_tasks()` | Get tasks ready to execute |\n| `get_running_tasks()` | Get currently running tasks |\n| `get_completed_tasks()` | Get completed tasks |\n| `get_failed_tasks()` | Get failed tasks |\n| `get_pending_tasks()` | Get pending tasks |\n| `get_modifiable_tasks()` | Get tasks safe to modify |\n\n### Dependency Management\n\n| Method | Description |\n|--------|-------------|\n| `add_dependency(dependency)` | Add dependency edge |\n| `remove_dependency(dependency_id)` | Remove dependency |\n| `get_dependency(dependency_id)` | Get dependency by ID |\n| `get_all_dependencies()` | Get all dependencies |\n| `get_task_dependencies(task_id)` | Get dependencies for specific task |\n| `get_modifiable_dependencies()` | Get dependencies safe to modify |\n\n### Validation\n\n| Method | Description |\n|--------|-------------|\n| `validate_dag()` | Validate DAG structure, returns `(bool, List[str])` with validation errors |\n| `has_cycle()` | Check for cycles (returns `bool`) |\n| `get_topological_order()` | Get topological ordering (returns `List[str]`, raises `ValueError` if cyclic) |\n\n### Execution\n\n| Method | Description |\n|--------|-------------|\n| `start_execution()` | Mark constellation as started |\n| `start_task(task_id)` | Start specific task |\n| `mark_task_completed(task_id, success, result, error)` | Mark task done, returns `List[TaskStar]` of newly ready tasks |\n| `complete_execution()` | Mark constellation as completed |\n| `is_complete()` | Check if all tasks are terminal (returns `bool`) |\n| `update_state()` | Update constellation state based on task states |\n\n### Analysis\n\n| Method | Description |\n|--------|-------------|\n| `get_longest_path()` | Get critical path using node count, returns `(int, List[str])` |\n| `get_critical_path_length_with_time()` | Get critical path using actual time, returns `(float, List[str])` |\n| `get_max_width()` | Get maximum parallelism (returns `int`) |\n| `get_total_work()` | Get sum of execution durations (returns `float`) |\n| `get_parallelism_metrics()` | Get comprehensive parallelism metrics (returns `Dict[str, Any]`) |\n| `get_statistics()` | Get all constellation statistics (returns `Dict[str, Any]`) |\n\n### Serialization\n\n| Method | Description |\n|--------|-------------|\n| `to_dict()` | Convert to dictionary |\n| `to_json(save_path)` | Export to JSON string or file |\n| `from_dict(data)` | Create from dictionary (classmethod) |\n| `from_json(json_data, file_path)` | Create from JSON (classmethod) |\n| `to_basemodel()` | Convert to Pydantic schema |\n| `from_basemodel(schema)` | Create from Pydantic schema (classmethod) |\n\n### Visualization\n\n| Method | Description |\n|--------|-------------|\n| `display_dag(mode)` | Display constellation (modes: overview, topology, details, execution) |\n\n---\n\n*TaskConstellation — Orchestrating distributed workflows across the digital galaxy*\n"
},
{
"path": "documents/docs/galaxy/constellation/task_star.md",
"content": "# TaskStar — Atomic Execution Unit\n\n## Overview\n\n**TaskStar** represents the atomic unit of computation in UFO Galaxy—the smallest indivisible task scheduled on a device agent. Each TaskStar encapsulates complete context necessary for autonomous execution, including semantic description, assigned device, execution state, and dependency relationships.\n\n**Formal Definition:** A TaskStar $t_i$ is formally defined as:\n\n$$\nt_i = (\\text{name}_i, \\text{description}_i, \\text{device}_i, \\text{tips}_i, \\text{status}_i, \\text{dependencies}_i)\n$$\n\n---\n\n## Architecture\n\n### Core Properties\n\n| Property | Type | Description |\n|----------|------|-------------|\n| **task_id** | `str` | Unique identifier (auto-generated UUID if not provided) |\n| **name** | `str` | Short, human-readable task name |\n| **description** | `str` | Natural-language specification of what the task should do |\n| **tips** | `List[str]` | Guidance list to help device agent complete the task |\n| **target_device_id** | `str` | ID of the device agent responsible for execution |\n| **device_type** | `DeviceType` | Type of target device (Windows, Linux, Android, etc.) |\n| **status** | `TaskStatus` | Current execution state |\n| **priority** | `TaskPriority` | Priority level for scheduling (LOW, MEDIUM, HIGH, CRITICAL) |\n| **timeout** | `float` | Maximum execution time in seconds |\n| **retry_count** | `int` | Number of allowed retries on failure |\n| **task_data** | `Dict[str, Any]` | Additional data needed for task execution |\n| **expected_output_type** | `str` | Expected type/format of the output |\n\n**Note:** The property `task_description` is available as a backward compatibility alias for `description`.\n\n### Execution Tracking\n\n| Property | Type | Description |\n|----------|------|-------------|\n| **result** | `Any` | Task execution result (if completed successfully) |\n| **error** | `Exception` | Error information (if failed) |\n| **execution_start_time** | `datetime` | Timestamp when execution started |\n| **execution_end_time** | `datetime` | Timestamp when execution ended |\n| **execution_duration** | `float` | Duration in seconds (calculated) |\n| **created_at** | `datetime` | Task creation timestamp |\n| **updated_at** | `datetime` | Last modification timestamp |\n\n**Note:** All execution tracking properties are read-only and automatically managed by the TaskStar lifecycle methods.\n\n### Computed Properties\n\n| Property | Type | Description |\n|----------|------|-------------|\n| **is_terminal** | `bool` | True if task is in a terminal state (COMPLETED, FAILED, or CANCELLED) |\n| **is_ready_to_execute** | `bool` | True if task is PENDING and has no pending dependencies |\n\n---\n\n## Task Status Lifecycle\n\n```mermaid\nstateDiagram-v2\n [*] --> PENDING: Create\n PENDING --> WAITING_DEPENDENCY: Has dependencies\n WAITING_DEPENDENCY --> PENDING: Dependencies satisfied\n PENDING --> RUNNING: Start execution\n RUNNING --> COMPLETED: Success\n RUNNING --> FAILED: Error\n RUNNING --> CANCELLED: User cancels\n FAILED --> PENDING: Retry\n COMPLETED --> [*]\n FAILED --> [*]\n CANCELLED --> [*]\n```\n\n### Status Definitions\n\n| Status | Description | Terminal |\n|--------|-------------|----------|\n| **PENDING** | Task is ready to execute (no pending dependencies) | ❌ |\n| **WAITING_DEPENDENCY** | Task is waiting for prerequisite tasks | ❌ |\n| **RUNNING** | Task is currently executing on device | ❌ |\n| **COMPLETED** | Task finished successfully | ✅ |\n| **FAILED** | Task encountered an error | ✅ |\n| **CANCELLED** | Task was cancelled by user | ✅ |\n\n**Note:** Terminal states (COMPLETED, FAILED, CANCELLED) are final—tasks in these states cannot transition to other states without explicit retry.\n\n---\n\n## Priority Levels\n\nTasks are scheduled based on priority when multiple tasks are ready to execute:\n\n| Priority | Value | Use Case |\n|----------|-------|----------|\n| **LOW** | 1 | Background tasks, cleanup operations |\n| **MEDIUM** | 2 | Standard tasks (default) |\n| **HIGH** | 3 | Important tasks requiring quick execution |\n| **CRITICAL** | 4 | Time-sensitive tasks, system health checks |\n\n---\n\n## Usage Examples\n\n### Creating a TaskStar\n\n```python\nfrom galaxy.constellation import TaskStar\nfrom galaxy.constellation.enums import DeviceType, TaskPriority\n\n# Basic task creation\ntask = TaskStar(\n task_id=\"build_docker_image\",\n name=\"Docker Build\",\n description=\"Build the Docker image from Dockerfile in the current directory\",\n tips=[\n \"Use docker build command\",\n \"Tag the image as 'myapp:latest'\",\n \"Check for build errors in output\"\n ],\n target_device_id=\"linux_gpu_server\",\n device_type=DeviceType.LINUX,\n priority=TaskPriority.HIGH,\n timeout=300.0, # 5 minutes\n retry_count=2\n)\n```\n\n### Task with Additional Data\n\n```python\n# Task with custom data payload\ntask = TaskStar(\n task_id=\"process_dataset\",\n description=\"Preprocess the dataset and save to output directory\",\n task_data={\n \"input_path\": \"/data/raw/dataset.csv\",\n \"output_path\": \"/data/processed/dataset_clean.csv\",\n \"columns_to_drop\": [\"temp_col1\", \"temp_col2\"],\n \"normalization\": \"min-max\"\n },\n target_device_id=\"linux_cpu_1\",\n device_type=DeviceType.LINUX\n)\n```\n\n### Auto-Generated Task\n\n```python\n# Minimal creation with auto-generated ID and defaults\ntask = TaskStar(\n description=\"Run unit tests\",\n target_device_id=\"windows_desktop\"\n)\n\nprint(task.task_id) # Auto-generated UUID\nprint(task.name) # Auto-generated: \"task_{first 8 chars of UUID}\"\nprint(task.priority) # Default: TaskPriority.MEDIUM\n```\n\n---\n\n## Core Operations\n\n### Execution Management\n\n```python\n# Start execution\ntask.start_execution()\nprint(f\"Started at: {task.execution_start_time}\")\n\n# Mark as completed (success)\nresult = {\"status\": \"success\", \"output\": \"Tests passed: 45/45\"}\ntask.complete_with_success(result)\nprint(f\"Duration: {task.execution_duration} seconds\")\n\n# Mark as failed\ntry:\n # ... execution code ...\n raise Exception(\"Docker build failed\")\nexcept Exception as e:\n task.complete_with_failure(e)\n print(f\"Error: {task.error}\")\n```\n\n### Retry Logic\n\n```python\n# Check if task should retry\nif task.should_retry():\n task.retry()\n print(f\"Retry attempt {task._current_retry}/{task._retry_count}\")\n # Task status is now PENDING again\n```\n\n### Validation\n\n```python\n# Validate task configuration\nif task.validate():\n print(\"Task configuration is valid\")\nelse:\n errors = task.get_validation_errors()\n print(f\"Validation errors: {errors}\")\n```\n\n---\n\n## State Queries\n\n### Checking Task State\n\n```python\n# Check if task is ready to execute\nif task.is_ready_to_execute:\n print(\"Task can be started\")\n\n# Check if task is in terminal state\nif task.is_terminal:\n print(\"Task has finished executing\")\n\n# Query specific status\nif task.status == TaskStatus.RUNNING:\n elapsed = datetime.now(timezone.utc) - task.execution_start_time\n print(f\"Running for {elapsed.total_seconds()} seconds\")\n```\n\n### Accessing Results\n\n```python\n# Access execution results\nif task.status == TaskStatus.COMPLETED:\n print(f\"Result: {task.result}\")\n print(f\"Duration: {task.execution_duration}s\")\n \nelif task.status == TaskStatus.FAILED:\n print(f\"Error: {task.error}\")\n print(f\"Failed at: {task.execution_end_time}\")\n```\n\n---\n\n## Serialization\n\n### JSON Export/Import\n\n```python\n# Export to JSON\njson_string = task.to_json()\nprint(json_string)\n\n# Save to file\ntask.to_json(save_path=\"task_backup.json\")\n\n# Load from JSON string\nrestored_task = TaskStar.from_json(json_data=json_string)\n\n# Load from file\nloaded_task = TaskStar.from_json(file_path=\"task_backup.json\")\n```\n\n### Dictionary Conversion\n\n```python\n# Convert to dictionary\ntask_dict = task.to_dict()\n\n# Create from dictionary\nnew_task = TaskStar.from_dict(task_dict)\n```\n\n### Pydantic Schema Conversion\n\n```python\n# Convert to Pydantic BaseModel\nschema = task.to_basemodel()\n\n# Create from Pydantic schema\ntask_from_schema = TaskStar.from_basemodel(schema)\n```\n\n---\n\n## Advanced Features\n\n### Request String Formatting\n\nThe `to_request_string()` method formats the task for device agent consumption:\n\n```python\nrequest = task.to_request_string()\n\n# Output:\n# Task Description: Build the Docker image from Dockerfile\n# Tips for Completion:\n# - Use docker build command\n# - Tag the image as 'myapp:latest'\n# - Check for build errors in output\n```\n\nThis formatted string is sent to device agents for execution.\n\n### Dynamic Data Updates\n\n```python\n# Update task data\ntask.update_task_data({\n \"additional_flags\": [\"--no-cache\", \"--pull\"],\n \"build_args\": {\"VERSION\": \"1.2.3\"}\n})\n\n# Access task data\ndata = task.task_data\nprint(data[\"additional_flags\"])\n```\n\n!!! warning \"Modification Restrictions\"\n Task properties cannot be modified while the task is in `RUNNING` status. This prevents race conditions and ensures execution consistency.\n\n---\n\n## Dependency Management\n\n### Internal Dependency Tracking\n\nTaskStar maintains internal sets of dependencies and dependents:\n\n```python\n# Add dependency (internal use by TaskConstellation)\ntask.add_dependency(\"prerequisite_task_id\")\n\n# Remove dependency\ntask.remove_dependency(\"prerequisite_task_id\")\n\n# Add dependent task\ntask.add_dependent(\"dependent_task_id\")\n\n# Check dependencies\nprint(f\"Dependencies: {task._dependencies}\")\nprint(f\"Dependents: {task._dependents}\")\n```\n\n!!! note \"Managed by TaskConstellation\"\n Dependency management methods are primarily used internally by `TaskConstellation`. Direct manipulation is not recommended—use `ConstellationEditor` for safe editing with undo/redo support.\n\n---\n\n## Integration with Constellation\n\n### Adding to Constellation\n\n```python\nfrom galaxy.constellation import TaskConstellation\n\nconstellation = TaskConstellation(name=\"my_workflow\")\n\n# Add task to constellation\nconstellation.add_task(task)\n\n# Task is now managed by constellation\nready_tasks = constellation.get_ready_tasks()\n```\n\n### Execution via Device Manager\n\n```python\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Execute task using device manager\ndevice_manager = ConstellationDeviceManager()\n\n# Execute returns an ExecutionResult object\nexecution_result = await task.execute(device_manager)\n\nprint(f\"Status: {execution_result.status}\")\nprint(f\"Result: {execution_result.result}\")\nprint(f\"Execution Time: {execution_result.execution_time}s\")\n```\n\n---\n\n## Error Handling\n\n### Validation Errors\n\n```python\ntask = TaskStar(\n task_id=\"\", # Invalid: empty ID\n name=\"\", # Invalid: empty name\n description=\"\", # Invalid: empty description\n timeout=-1.0 # Invalid: negative timeout\n)\n\nif not task.validate():\n for error in task.get_validation_errors():\n print(f\"❌ {error}\")\n\n# Output:\n# ❌ Task ID must be a non-empty string\n# ❌ Task name must be a non-empty string\n# ❌ Task description must be a non-empty string\n# ❌ Timeout must be a positive number\n```\n\n### Execution Errors\n\n```python\ntry:\n task.start_execution()\nexcept ValueError as e:\n print(f\"Cannot start: {e}\")\n # Example: \"Cannot start task in status RUNNING\"\n\ntry:\n task.complete_with_success(result)\nexcept ValueError as e:\n print(f\"Cannot complete: {e}\")\n # Example: \"Cannot complete task in status PENDING\"\n```\n\n---\n\n## Example Workflows\n\n### Simple Task Execution\n\n```python\n# Create task\ntask = TaskStar(\n description=\"Run Python script\",\n target_device_id=\"linux_server_1\",\n timeout=60.0\n)\n\n# Execute\ntask.start_execution()\ntry:\n # ... actual execution ...\n result = {\"output\": \"Script completed\", \"exit_code\": 0}\n task.complete_with_success(result)\nexcept Exception as e:\n task.complete_with_failure(e)\n\n# Check result\nif task.status == TaskStatus.COMPLETED:\n print(f\"✅ Success: {task.result}\")\nelse:\n print(f\"❌ Failed: {task.error}\")\n```\n\n### Retry on Failure\n\n```python\nmax_attempts = 3\nattempt = 0\n\nwhile attempt < max_attempts:\n attempt += 1\n task.start_execution()\n \n try:\n # ... execution code ...\n task.complete_with_success(result)\n break\n except Exception as e:\n task.complete_with_failure(e)\n \n if task.should_retry():\n task.retry()\n print(f\"Retry {attempt}/{max_attempts}\")\n else:\n print(\"Max retries exceeded\")\n break\n```\n\n---\n\n## Best Practices\n\n### Task Design Guidelines\n\n1. **Keep tasks atomic**: Each task should represent a single, well-defined operation\n2. **Provide clear descriptions**: Use natural language that device agents can understand\n3. **Include helpful tips**: Guide the agent with specific instructions or common pitfalls\n4. **Set appropriate timeouts**: Prevent hanging tasks with realistic timeout values\n5. **Use retry wisely**: Enable retries for transient failures, not logic errors\n\n### Good vs. Bad Task Descriptions\n\n✅ **Good**: \"Build the Docker image from the Dockerfile in /app directory and tag it as 'myapp:v1.2.3'\"\n\n❌ **Bad**: \"Build stuff\"\n\n✅ **Good**: \"Run pytest on the test/ directory and generate a coverage report in HTML format\"\n\n❌ **Bad**: \"Test the code\"\n\n!!! warning \"Common Pitfalls\"\n - **Don't modify running tasks**: Attempting to change properties during execution raises `ValueError`\n - **Don't forget validation**: Always validate tasks before adding to constellation\n - **Don't ignore timeouts**: Set realistic timeouts to prevent resource exhaustion\n\n---\n\n## Related Components\n\n- **[TaskStarLine](task_star_line.md)** — Dependency relationships between tasks\n- **[TaskConstellation](task_constellation.md)** — DAG orchestration and execution\n- **[ConstellationEditor](constellation_editor.md)** — Safe task editing with undo/redo\n- **[ConstellationDeviceManager](../client/device_manager.md)** — Device management and task assignment\n- **[Overview](overview.md)** — Task Constellation framework overview\n\n---\n\n## API Reference\n\n### Constructor\n\n```python\nTaskStar(\n task_id: Optional[str] = None,\n name: str = \"\",\n description: str = \"\",\n tips: List[str] = None,\n target_device_id: Optional[str] = None,\n device_type: Optional[DeviceType] = None,\n priority: TaskPriority = TaskPriority.MEDIUM,\n timeout: Optional[float] = None,\n retry_count: int = 0,\n task_data: Optional[Dict[str, Any]] = None,\n expected_output_type: Optional[str] = None,\n config: Optional[TaskConfiguration] = None\n)\n```\n\n### Key Methods\n\n| Method | Description |\n|--------|-------------|\n| `execute(device_manager)` | Execute task using device manager (async, returns `ExecutionResult`) |\n| `validate()` | Validate task configuration (returns `bool`) |\n| `get_validation_errors()` | Get list of validation errors (returns `List[str]`) |\n| `start_execution()` | Mark task as started |\n| `complete_with_success(result)` | Mark task as completed successfully |\n| `complete_with_failure(error)` | Mark task as failed |\n| `retry()` | Reset task for retry attempt |\n| `cancel()` | Cancel the task |\n| `should_retry()` | Check if task should be retried (returns `bool`) |\n| `to_dict()` | Convert to dictionary |\n| `to_json(save_path)` | Export to JSON string or file |\n| `from_dict(data)` | Create from dictionary (classmethod) |\n| `from_json(json_data, file_path)` | Create from JSON (classmethod) |\n| `to_basemodel()` | Convert to Pydantic BaseModel schema |\n| `from_basemodel(schema)` | Create from Pydantic schema (classmethod) |\n\n---\n\n*TaskStar — The atomic building block of distributed workflows*\n"
},
{
"path": "documents/docs/galaxy/constellation/task_star_line.md",
"content": "# TaskStarLine — Dependency Relationship\n\n## Overview\n\n**TaskStarLine** represents a directed dependency relationship between two TaskStars, forming an edge in the task constellation DAG. Each TaskStarLine defines how tasks depend on each other, with support for conditional logic, success-only execution, and custom condition evaluation.\n\n**Formal Definition:** A TaskStarLine $e_{i \\rightarrow j}$ specifies a dependency from task $t_i$ to task $t_j$:\n\n$$\ne_{i \\rightarrow j} = (\\text{from\\_task}_i, \\text{to\\_task}_j, \\text{type}, \\text{description})\n$$\n\nTask $t_j$ cannot begin until certain conditions on $t_i$ are satisfied, based on the dependency type.\n\n---\n\n## Architecture\n\n### Core Properties\n\n| Property | Type | Description |\n|----------|------|-------------|\n| **line_id** | `str` | Unique identifier (auto-generated UUID if not provided) |\n| **from_task_id** | `str` | ID of the prerequisite task (source) |\n| **to_task_id** | `str` | ID of the dependent task (target) |\n| **dependency_type** | `DependencyType` | Type of dependency relationship |\n| **condition_description** | `str` | Natural language description of the condition |\n| **condition_evaluator** | `Callable` | Function to evaluate if condition is met |\n| **metadata** | `Dict[str, Any]` | Additional metadata for the dependency |\n\n**Note:** The properties `source_task_id` and `target_task_id` are available as aliases for `from_task_id` and `to_task_id` respectively (for IDependency interface compatibility).\n\n### State Tracking\n\n| Property | Type | Description |\n|----------|------|-------------|\n| **is_satisfied** | `bool` | Whether the dependency condition is currently satisfied |\n| **last_evaluation_result** | `bool` | Result of the most recent condition evaluation |\n| **last_evaluation_time** | `datetime` | Timestamp of last condition evaluation |\n| **created_at** | `datetime` | Dependency creation timestamp |\n| **updated_at** | `datetime` | Last modification timestamp |\n\n**Note:** All state tracking properties are read-only and automatically managed by TaskStarLine methods.\n\n---\n\n## Dependency Types\n\nTaskStarLine supports four types of dependency relationships:\n\n### 1. Unconditional (`UNCONDITIONAL`)\n\nTask $t_j$ **always** waits for $t_i$ to complete, regardless of success or failure.\n\n```mermaid\ngraph LR\n A[Task A] -->|UNCONDITIONAL| B[Task B]\n style A fill:#90EE90\n style B fill:#87CEEB\n```\n\n**Use Cases:**\n- Sequential pipeline stages\n- Resource cleanup after any task completion\n- Logging or notification tasks\n\n**Example:**\n```python\n# Task B always runs after Task A completes\ndep = TaskStarLine.create_unconditional(\n from_task_id=\"task_a\",\n to_task_id=\"task_b\",\n description=\"B runs after A regardless of outcome\"\n)\n```\n\n---\n\n### 2. Success-Only (`SUCCESS_ONLY`)\n\nTask $t_j$ proceeds **only if** $t_i$ completes successfully (result is not `None`).\n\n```mermaid\ngraph LR\n A[Task A] -->|SUCCESS_ONLY| B[Task B]\n A -->|FAILED| C[Skip B]\n style A fill:#90EE90\n style B fill:#87CEEB\n style C fill:#FFB6C1\n```\n\n**Use Cases:**\n- Build pipeline (deploy only if build succeeds)\n- Multi-step data processing\n- Conditional workflow branches\n\n**Example:**\n```python\n# Task B only runs if Task A succeeds\ndep = TaskStarLine.create_success_only(\n from_task_id=\"build_task\",\n to_task_id=\"deploy_task\",\n description=\"Deploy only if build succeeds\"\n)\n```\n\n**Note:** Success is determined by the prerequisite task returning a non-`None` result.\n\n---\n\n### 3. Completion-Only (`COMPLETION_ONLY`)\n\nTask $t_j$ proceeds when $t_i$ completes, **regardless of success or failure**.\n\n```mermaid\ngraph LR\n A[Task A] -->|COMPLETION_ONLY| B[Task B]\n A -->|SUCCESS or FAIL| B\n style A fill:#90EE90\n style B fill:#87CEEB\n```\n\n**Use Cases:**\n- Cleanup tasks\n- Notification tasks\n- Audit logging\n\n**Example:**\n```python\n# Task B runs after Task A finishes, regardless of outcome\ndep = TaskStarLine(\n from_task_id=\"main_task\",\n to_task_id=\"cleanup_task\",\n dependency_type=DependencyType.COMPLETION_ONLY,\n condition_description=\"Cleanup runs regardless of main task outcome\"\n)\n```\n\n---\n\n### 4. Conditional (`CONDITIONAL`)\n\nTask $t_j$ proceeds based on a **user-defined condition** evaluated on $t_i$'s result.\n\n```mermaid\ngraph LR\n A[Task A] -->|CONDITIONAL| B{Evaluate}\n B -->|True| C[Task B runs]\n B -->|False| D[Task B skipped]\n style A fill:#90EE90\n style B fill:#FFD700\n style C fill:#87CEEB\n style D fill:#FFB6C1\n```\n\n**Use Cases:**\n- Error handling branches\n- Result-based routing\n- Performance-based optimization\n\n**Example:**\n```python\n# Define custom condition evaluator\ndef check_coverage_threshold(result):\n \"\"\"Run next task only if test coverage > 80%\"\"\"\n if result and isinstance(result, dict):\n coverage = result.get(\"coverage_percent\", 0)\n return coverage > 80\n return False\n\n# Create conditional dependency\ndep = TaskStarLine.create_conditional(\n from_task_id=\"test_task\",\n to_task_id=\"quality_gate_task\",\n condition_description=\"Proceed if test coverage > 80%\",\n condition_evaluator=check_coverage_threshold\n)\n```\n\n**Note:** If no `condition_evaluator` is provided for a CONDITIONAL dependency, it defaults to SUCCESS_ONLY behavior (checks if result is not `None`).\n\n---\n\n## Dependency Lifecycle\n\n```mermaid\nstateDiagram-v2\n [*] --> Created: Initialize\n Created --> Waiting: Prerequisite running\n Waiting --> Evaluating: Prerequisite completes\n Evaluating --> Satisfied: Condition met\n Evaluating --> Unsatisfied: Condition not met\n Satisfied --> [*]: Dependent can run\n Unsatisfied --> [*]: Dependent blocked\n```\n\n---\n\n## Usage Examples\n\n### Creating Dependencies\n\n```python\nfrom galaxy.constellation import TaskStarLine\nfrom galaxy.constellation.enums import DependencyType\n\n# 1. Unconditional dependency\ndep1 = TaskStarLine.create_unconditional(\n from_task_id=\"checkout_code\",\n to_task_id=\"build_project\",\n description=\"Build after checkout\"\n)\n\n# 2. Success-only dependency\ndep2 = TaskStarLine.create_success_only(\n from_task_id=\"build_project\",\n to_task_id=\"deploy_staging\",\n description=\"Deploy only if build succeeds\"\n)\n\n# 3. Conditional dependency with custom logic\ndef check_test_results(result):\n return result.get(\"tests_passed\", 0) == result.get(\"total_tests\", 0)\n\ndep3 = TaskStarLine.create_conditional(\n from_task_id=\"run_tests\",\n to_task_id=\"deploy_production\",\n condition_description=\"Deploy to production only if all tests pass\",\n condition_evaluator=check_test_results\n)\n\n# 4. Manual construction\ndep4 = TaskStarLine(\n from_task_id=\"task_a\",\n to_task_id=\"task_b\",\n dependency_type=DependencyType.COMPLETION_ONLY,\n condition_description=\"Task B runs after Task A completes\",\n metadata={\"priority\": \"high\", \"category\": \"cleanup\"}\n)\n```\n\n---\n\n## Core Operations\n\n### Condition Evaluation\n\n```python\n# Evaluate condition with prerequisite result\nprerequisite_result = {\n \"status\": \"success\",\n \"coverage_percent\": 85,\n \"tests_passed\": 120,\n \"total_tests\": 120\n}\n\nis_satisfied = dep.evaluate_condition(prerequisite_result)\n\nif is_satisfied:\n print(\"✅ Dependency satisfied, dependent task can run\")\n print(f\"Evaluated at: {dep.last_evaluation_time}\")\nelse:\n print(\"❌ Dependency not satisfied, dependent task blocked\")\n\n# Check evaluation history\nprint(f\"Last result: {dep.last_evaluation_result}\")\n```\n\n### Manual Satisfaction Control\n\n```python\n# Manually mark dependency as satisfied (override)\ndep.mark_satisfied()\n\n# Reset satisfaction status\ndep.reset_satisfaction()\n\n# Check satisfaction\nif dep.is_satisfied():\n print(\"Dependency is satisfied\")\n```\n\n---\n\n## State Queries\n\n### Checking Dependency State\n\n```python\n# Method 1: Check using completed tasks list (for IDependency interface)\n# Returns True if from_task_id is in the completed_tasks list\ncompleted_tasks = [\"task_a\", \"task_b\", \"task_c\"]\nif dep.is_satisfied(completed_tasks):\n print(\"Prerequisite task is completed\")\n\n# Method 2: Check internal satisfaction state (without parameter)\n# Returns the internal _is_satisfied flag set by evaluate_condition\nif dep.is_satisfied():\n print(\"Dependency condition is satisfied\")\n\n# Get last evaluation details\nprint(f\"Last evaluated: {dep.last_evaluation_time}\")\nprint(f\"Result: {dep.last_evaluation_result}\")\n\n# Access metadata\nprint(f\"Metadata: {dep.metadata}\")\n```\n\n---\n\n## Modification\n\n### Updating Dependency Properties\n\n```python\n# Change dependency type\ndep.dependency_type = DependencyType.SUCCESS_ONLY\n\n# Update condition description\ndep.condition_description = \"Updated: Deploy only after successful validation\"\n\n# Set new condition evaluator\ndef new_evaluator(result):\n return result.get(\"validation_score\", 0) > 0.95\n\ndep.set_condition_evaluator(new_evaluator)\n\n# Update metadata\ndep.update_metadata({\n \"updated_by\": \"admin\",\n \"reason\": \"Stricter validation threshold\"\n})\n```\n\n!!! warning \"Modification During Execution\"\n Changing `dependency_type` or `condition_evaluator` resets the satisfaction status. Be cautious when modifying dependencies during active constellation execution.\n\n---\n\n## Serialization\n\n### JSON Export/Import\n\n```python\n# Export to JSON\njson_string = dep.to_json()\nprint(json_string)\n\n# Save to file\ndep.to_json(save_path=\"dependency_backup.json\")\n\n# Load from JSON string\nrestored_dep = TaskStarLine.from_json(json_data=json_string)\n\n# Load from file\nloaded_dep = TaskStarLine.from_json(file_path=\"dependency_backup.json\")\n```\n\n### Dictionary Conversion\n\n```python\n# Convert to dictionary\ndep_dict = dep.to_dict()\n\n# Create from dictionary\nnew_dep = TaskStarLine.from_dict(dep_dict)\n\n# Dictionary structure\nprint(dep_dict)\n# {\n# \"line_id\": \"uuid-string\",\n# \"from_task_id\": \"task_a\",\n# \"to_task_id\": \"task_b\",\n# \"dependency_type\": \"success_only\",\n# \"condition_description\": \"...\",\n# \"metadata\": {...},\n# \"is_satisfied\": false,\n# \"last_evaluation_result\": null,\n# \"created_at\": \"2025-11-06T...\",\n# \"updated_at\": \"2025-11-06T...\"\n# }\n```\n\n### Pydantic Schema Conversion\n\n```python\n# Convert to Pydantic BaseModel\nschema = dep.to_basemodel()\n\n# Create from Pydantic schema\ndep_from_schema = TaskStarLine.from_basemodel(schema)\n```\n\n---\n\n## Integration with Constellation\n\n### Adding to Constellation\n\n```python\nfrom galaxy.constellation import TaskConstellation\n\nconstellation = TaskConstellation(name=\"my_workflow\")\n\n# Add tasks first\nconstellation.add_task(task_a)\nconstellation.add_task(task_b)\n\n# Add dependency\ntry:\n constellation.add_dependency(dep)\n print(\"✅ Dependency added successfully\")\nexcept ValueError as e:\n print(f\"❌ Failed to add dependency: {e}\")\n```\n\n### Dependency Validation\n\n```python\n# TaskConstellation validates dependencies automatically\ntry:\n # This would fail if it creates a cycle\n constellation.add_dependency(cyclic_dep)\nexcept ValueError as e:\n print(f\"Validation error: {e}\")\n # Output: \"Adding dependency would create a cycle\"\n\n# Check DAG validity\nis_valid, errors = constellation.validate_dag()\nif not is_valid:\n for error in errors:\n print(f\"❌ {error}\")\n```\n\n---\n\n## Advanced Patterns\n\n### Conditional Error Handling\n\n```python\n# Main task\nmain_task = TaskStar(\n task_id=\"main_process\",\n description=\"Process data\"\n)\n\n# Success path\nsuccess_task = TaskStar(\n task_id=\"success_notification\",\n description=\"Send success notification\"\n)\n\n# Error path\nerror_task = TaskStar(\n task_id=\"error_recovery\",\n description=\"Attempt recovery\"\n)\n\n# Success-only dependency\nsuccess_dep = TaskStarLine.create_success_only(\n from_task_id=\"main_process\",\n to_task_id=\"success_notification\"\n)\n\n# Failure-only dependency (using conditional)\ndef on_failure(result):\n return result is None # Task failed if result is None\n\nfailure_dep = TaskStarLine.create_conditional(\n from_task_id=\"main_process\",\n to_task_id=\"error_recovery\",\n condition_description=\"Run recovery if main task fails\",\n condition_evaluator=on_failure\n)\n```\n\n### Performance-Based Routing\n\n```python\n# Route to different processing paths based on data size\ndef route_large_dataset(result):\n data_size = result.get(\"row_count\", 0)\n return data_size > 1_000_000 # Route to GPU if > 1M rows\n\n# Route to GPU for large datasets\ngpu_dep = TaskStarLine.create_conditional(\n from_task_id=\"analyze_dataset\",\n to_task_id=\"process_on_gpu\",\n condition_description=\"Use GPU for datasets > 1M rows\",\n condition_evaluator=route_large_dataset\n)\n\n# Route to CPU for small datasets\ndef route_small_dataset(result):\n data_size = result.get(\"row_count\", 0)\n return data_size <= 1_000_000\n\ncpu_dep = TaskStarLine.create_conditional(\n from_task_id=\"analyze_dataset\",\n to_task_id=\"process_on_cpu\",\n condition_description=\"Use CPU for datasets <= 1M rows\",\n condition_evaluator=route_small_dataset\n)\n```\n\n---\n\n## Error Handling\n\n### Validation\n\n```python\n# TaskStarLine validates on creation\ntry:\n invalid_dep = TaskStarLine(\n from_task_id=\"task_a\",\n to_task_id=\"task_a\", # Self-loop!\n dependency_type=DependencyType.UNCONDITIONAL\n )\n constellation.add_dependency(invalid_dep)\nexcept ValueError as e:\n print(f\"Validation error: {e}\")\n # TaskConstellation will detect cycle\n```\n\n### Evaluation Errors\n\n```python\ndef risky_evaluator(result):\n # This might raise an exception\n return result[\"complex_calculation\"] / result[\"divisor\"]\n\ndep = TaskStarLine.create_conditional(\n from_task_id=\"task_a\",\n to_task_id=\"task_b\",\n condition_description=\"Conditional with potential error\",\n condition_evaluator=risky_evaluator\n)\n\n# evaluate_condition catches exceptions and returns False\nresult = {\"complex_calculation\": 100} # Missing \"divisor\"\nis_satisfied = dep.evaluate_condition(result)\nprint(is_satisfied) # False (evaluator raised KeyError, caught internally)\nprint(dep.last_evaluation_result) # False\n```\n\n---\n\n## Example Workflows\n\n### Build Pipeline\n\n```python\n# checkout → build → test → deploy\ncheckout = TaskStar(task_id=\"checkout\", description=\"Checkout code\")\nbuild = TaskStar(task_id=\"build\", description=\"Build project\")\ntest = TaskStar(task_id=\"test\", description=\"Run tests\")\ndeploy = TaskStar(task_id=\"deploy\", description=\"Deploy to production\")\n\n# Sequential success-only dependencies\ndep1 = TaskStarLine.create_success_only(\"checkout\", \"build\")\ndep2 = TaskStarLine.create_success_only(\"build\", \"test\")\ndep3 = TaskStarLine.create_success_only(\"test\", \"deploy\")\n```\n\n### Fan-Out Pattern\n\n```python\n# analyze → [process_gpu, process_cpu, process_edge]\nanalyze = TaskStar(task_id=\"analyze\", description=\"Analyze data\")\nprocess_gpu = TaskStar(task_id=\"gpu\", description=\"Process on GPU\")\nprocess_cpu = TaskStar(task_id=\"cpu\", description=\"Process on CPU\")\nprocess_edge = TaskStar(task_id=\"edge\", description=\"Process on edge device\")\n\n# All three can start after analyze completes\ndep1 = TaskStarLine.create_unconditional(\"analyze\", \"gpu\")\ndep2 = TaskStarLine.create_unconditional(\"analyze\", \"cpu\")\ndep3 = TaskStarLine.create_unconditional(\"analyze\", \"edge\")\n```\n\n### Fan-In Pattern\n\n```python\n# [task_a, task_b, task_c] → aggregate\ntask_a = TaskStar(task_id=\"task_a\", description=\"Process batch A\")\ntask_b = TaskStar(task_id=\"task_b\", description=\"Process batch B\")\ntask_c = TaskStar(task_id=\"task_c\", description=\"Process batch C\")\naggregate = TaskStar(task_id=\"aggregate\", description=\"Aggregate results\")\n\n# Aggregate waits for all three to complete\ndep1 = TaskStarLine.create_success_only(\"task_a\", \"aggregate\")\ndep2 = TaskStarLine.create_success_only(\"task_b\", \"aggregate\")\ndep3 = TaskStarLine.create_success_only(\"task_c\", \"aggregate\")\n```\n\n---\n\n## Best Practices\n\n### Dependency Design Guidelines\n\n1. **Use the right type**: Choose the dependency type that matches your workflow logic\n2. **Keep conditions simple**: Condition evaluators should be fast and deterministic\n3. **Handle evaluator errors**: Ensure evaluators don't raise uncaught exceptions (they're caught internally but logged)\n4. **Document conditions**: Use clear `condition_description` for debugging\n5. **Avoid cycles**: TaskConstellation validates, but design carefully to avoid attempts\n\n### Good vs. Bad Condition Evaluators\n\n✅ **Good**: Simple, fast, defensive\n\n```python\ndef check_success(result):\n return result is not None and result.get(\"status\") == \"success\"\n```\n\n❌ **Bad**: Complex, slow, error-prone\n\n```python\ndef check_success(result):\n # Slow database query\n db_status = query_database(result[\"task_id\"])\n # Complex logic with potential errors\n return eval(result[\"complex_expression\"]) and db_status\n```\n\n!!! warning \"Common Pitfalls\"\n - **Cyclic dependencies**: Always validate DAG before execution\n - **Missing tasks**: Ensure both `from_task_id` and `to_task_id` exist in constellation\n - **Stateful evaluators**: Avoid evaluators that depend on external state\n - **Slow evaluators**: Keep evaluation fast; avoid I/O or expensive computation\n\n---\n\n## Related Components\n\n- **[TaskStar](task_star.md)** — Atomic execution units that TaskStarLines connect\n- **[TaskConstellation](task_constellation.md)** — DAG manager that validates and executes dependencies\n- **[ConstellationEditor](constellation_editor.md)** — Safe dependency editing with undo/redo\n- **[Overview](overview.md)** — Task Constellation framework overview\n\n---\n\n## API Reference\n\n### Constructor\n\n```python\nTaskStarLine(\n from_task_id: str,\n to_task_id: str,\n dependency_type: DependencyType = DependencyType.UNCONDITIONAL,\n condition_description: Optional[str] = None,\n condition_evaluator: Optional[Callable[[Any], bool]] = None,\n line_id: Optional[str] = None,\n metadata: Optional[Dict[str, Any]] = None\n)\n```\n\n### Factory Methods\n\n| Method | Description |\n|--------|-------------|\n| `create_unconditional(from_id, to_id, desc)` | Create unconditional dependency (classmethod) |\n| `create_success_only(from_id, to_id, desc)` | Create success-only dependency (classmethod) |\n| `create_conditional(from_id, to_id, desc, evaluator)` | Create conditional dependency (classmethod) |\n\n### Key Methods\n\n| Method | Description |\n|--------|-------------|\n| `evaluate_condition(result)` | Evaluate if condition is satisfied (returns `bool`) |\n| `mark_satisfied()` | Manually mark as satisfied |\n| `reset_satisfaction()` | Reset satisfaction status |\n| `is_satisfied(completed_tasks=None)` | Check if dependency is satisfied (returns `bool`); with parameter checks if from_task is completed, without checks internal state |\n| `set_condition_evaluator(evaluator)` | Set new condition evaluator |\n| `update_metadata(metadata)` | Update metadata |\n| `to_dict()` | Convert to dictionary |\n| `to_json(save_path)` | Export to JSON |\n| `from_dict(data)` | Create from dictionary (classmethod) |\n| `from_json(json_data, file_path)` | Create from JSON (classmethod) |\n| `to_basemodel()` | Convert to Pydantic BaseModel schema |\n| `from_basemodel(schema)` | Create from Pydantic schema (classmethod) |\n\n---\n\n*TaskStarLine — Connecting tasks with intelligent dependency logic*\n"
},
{
"path": "documents/docs/galaxy/constellation_agent/command.md",
"content": "# Constellation MCP Server — Structured Task Management\n\n## Overview\n\nThe **Constellation MCP Server** provides a standardized, idempotent interface for manipulating Task Constellations. Through Model Context Protocol (MCP), it exposes task and dependency management primitives that bridge LLM-level reasoning and concrete execution state, ensuring reproducibility and auditability.\n\nThe Constellation MCP Server is a lightweight component that operationalizes dynamic graph construction for the Constellation Agent. It serves as the **structured manipulation layer** between LLM reasoning and the Task Constellation data structure.\n\n### Design Principles\n\n| Principle | Description |\n|-----------|-------------|\n| **Idempotency** | Each operation can be safely retried without side effects |\n| **Atomicity** | Single operation per tool call with clear success/failure |\n| **Consistency** | Returns globally valid constellation snapshots after each operation |\n| **Auditability** | All operations are logged and traceable |\n| **Type Safety** | Pydantic schema validation for all inputs/outputs |\n\n### Architecture\n\n```mermaid\ngraph TB\n subgraph \"Constellation Agent\"\n Agent[Agent Logic]\n Prompter[Prompter]\n end\n \n subgraph \"MCP Server\"\n MCP[FastMCP Server]\n Editor[ConstellationEditor]\n Constellation[TaskConstellation]\n end\n \n Agent --> Prompter\n Prompter -->|Tool Descriptions| Agent\n Agent -->|Execute Command| MCP\n MCP --> Editor\n Editor --> Constellation\n Constellation -->|JSON Response| MCP\n MCP -->|Updated State| Agent\n \n style MCP fill:#e1f5ff\n style Editor fill:#fff4e1\n style Constellation fill:#e8f5e9\n```\n\n---\n\n## 🛠️ Core Tools\n\nThe MCP server exposes **7 core tools** organized into three categories:\n\n### Tool Categories\n\n```mermaid\nmindmap\n root((MCP Tools))\n Task Management\n add_task\n remove_task\n update_task\n Dependency Management\n add_dependency\n remove_dependency\n update_dependency\n Bulk Operations\n build_constellation\n```\n\n---\n\n## 📦 Task Management Tools\n\n### add_task\n\nAdd a new atomic task (TaskStar) to the constellation.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `task_id` | `str` | ✅ Yes | Unique identifier for the task (e.g., `\"open_browser\"`, `\"login_system\"`) |\n| `name` | `str` | ✅ Yes | Human-readable name (e.g., `\"Open Browser\"`, `\"Login to System\"`) |\n| `description` | `str` | ✅ Yes | Detailed task specification including steps and expected outcomes |\n| `target_device_id` | `str` | ❌ No (default: `None`) | Device where task executes (e.g., `\"DESKTOP-ABC123\"`, `\"iPhone-001\"`) |\n| `tips` | `List[str]` | ❌ No (default: `None`) | Critical hints for successful execution |\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of complete updated TaskConstellation after adding task\"\n}\n```\n\n#### Example Usage\n\n```python\n# Add a task to download data\nresult = await mcp_client.call_tool(\n tool_name=\"add_task\",\n parameters={\n \"task_id\": \"download_dataset\",\n \"name\": \"Download MNIST Dataset\",\n \"description\": \"Download MNIST dataset from official source, verify checksums, extract to data/ directory\",\n \"target_device_id\": \"laptop_001\",\n \"tips\": [\n \"Ensure stable internet connection\",\n \"Verify disk space > 500MB\",\n \"Resume download if interrupted\"\n ]\n }\n)\n\n# Returns complete constellation JSON\nconstellation = json.loads(result)\n```\n\n#### Validation\n\n- **Unique task_id**: Must not conflict with existing tasks\n- **Auto-timestamps**: `created_at` and `updated_at` are automatically set\n- **Default values**: `status=PENDING`, `priority=MEDIUM` if not specified\n\n**Task ID Naming Best Practice**: Use descriptive, action-oriented identifiers:\n\n✅ Good: `\"fetch_user_data\"`, `\"train_model\"`, `\"send_notification\"` \n❌ Avoid: `\"task1\"`, `\"t\"`, `\"temp\"`\n\n---\n\n### remove_task\n\nRemove a task and all associated dependencies from the constellation.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `task_id` | `str` | ✅ Yes | Unique identifier of task to remove |\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of complete updated TaskConstellation after removing task\"\n}\n```\n\n#### Example Usage\n\n```python\n# Remove a task\nresult = await mcp_client.call_tool(\n tool_name=\"remove_task\",\n parameters={\"task_id\": \"download_dataset\"}\n)\n\n# Returns updated constellation without the task\nconstellation = json.loads(result)\n```\n\n#### Side Effects\n\n**Cascade Deletion**: Removing a task automatically removes:\n\n- All **incoming dependencies** (edges pointing to this task)\n- All **outgoing dependencies** (edges from this task)\n\nThis maintains DAG integrity by preventing dangling references.\n\n#### Validation\n\n- **Task exists**: `task_id` must exist in constellation\n- **Modifiable status**: Task must not be in `RUNNING`, `COMPLETED`, or `FAILED` states\n\n---\n\n### update_task\n\nModify specific fields of an existing task.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `task_id` | `str` | ✅ Yes | Task identifier |\n| `name` | `str` | ❌ No (default: `None`) | New human-readable name |\n| `description` | `str` | ❌ No (default: `None`) | New detailed description |\n| `target_device_id` | `str` | ❌ No (default: `None`) | New target device |\n| `tips` | `List[str]` | ❌ No (default: `None`) | New tips list |\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of complete updated TaskConstellation after updating task\"\n}\n```\n\n#### Example Usage\n\n```python\n# Update task device assignment\nresult = await mcp_client.call_tool(\n tool_name=\"update_task\",\n parameters={\n \"task_id\": \"train_model\",\n \"target_device_id\": \"gpu_server_002\", # Switch to different GPU\n \"tips\": [\n \"Use mixed precision training\",\n \"Monitor GPU memory usage\",\n \"Save checkpoints every 1000 steps\"\n ]\n }\n)\n```\n\n#### Partial Updates\n\nOnly provided fields are modified — others remain unchanged:\n\n```python\n# Update only description\nresult = await mcp_client.call_tool(\n tool_name=\"update_task\",\n parameters={\n \"task_id\": \"process_data\",\n \"description\": \"Process data with enhanced validation and error handling\"\n # name, target_device_id, tips remain unchanged\n }\n)\n```\n\n#### Validation\n\n- **At least one field**: Must provide at least one field to update\n- **Modifiable status**: Task must be in modifiable state\n- **Auto-update timestamp**: `updated_at` is automatically refreshed\n\n---\n\n## 🔗 Dependency Management Tools\n\n### add_dependency\n\nCreate a dependency relationship (TaskStarLine) between two tasks.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `dependency_id` | `str` | ✅ Yes | Unique line identifier (e.g., `\"task_a->task_b\"`, `\"line_001\"`) |\n| `from_task_id` | `str` | ✅ Yes | Source/prerequisite task that must complete first |\n| `to_task_id` | `str` | ✅ Yes | Target/dependent task that waits for source |\n| `condition_description` | `str` | ❌ No (default: `None`) | Human-readable explanation of dependency logic |\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of complete updated TaskConstellation after adding dependency\"\n}\n```\n\n#### Example Usage\n\n```python\n# Add unconditional dependency\nresult = await mcp_client.call_tool(\n tool_name=\"add_dependency\",\n parameters={\n \"dependency_id\": \"download->process\",\n \"from_task_id\": \"download_dataset\",\n \"to_task_id\": \"process_data\",\n \"condition_description\": \"Processing requires dataset to be fully downloaded and verified\"\n }\n)\n```\n\n#### Dependency Types\n\nCurrently defaults to **UNCONDITIONAL** dependency:\n\n```python\n{\n \"dependency_type\": \"unconditional\" # Always wait for source to complete\n}\n```\n\nFuture extensions may support:\n- `SUCCESS_ONLY`: Wait only if source succeeds\n- `CONDITIONAL`: Evaluate custom condition\n- `COMPLETION_ONLY`: Wait regardless of success/failure\n\n#### Validation\n\n- **Both tasks exist**: `from_task_id` and `to_task_id` must exist in constellation\n- **No cycles**: Adding dependency cannot create cycles in the DAG\n- **Unique line_id**: `dependency_id` must be unique\n- **No self-loops**: `from_task_id != to_task_id`\n\n**Cycle Detection**: The server validates DAG acyclicity after adding each dependency:\n\n```\nA → B → C\n ↓\n A ❌ Creates cycle!\n```\n\n---\n\n### remove_dependency\n\nRemove a specific dependency relationship.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `dependency_id` | `str` | ✅ Yes | Line identifier to remove |\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of complete updated TaskConstellation after removing dependency\"\n}\n```\n\n#### Example Usage\n\n```python\n# Remove a dependency\nresult = await mcp_client.call_tool(\n tool_name=\"remove_dependency\",\n parameters={\"dependency_id\": \"download->process\"}\n)\n\n# Now process_data can run independently of download_dataset\n```\n\n#### Side Effects\n\n- Removing dependency does **NOT** affect the tasks themselves\n- Target task may become immediately ready if no other dependencies remain\n\n---\n\n### update_dependency\n\nModify the condition description of an existing dependency.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `dependency_id` | `str` | ✅ Yes | Line identifier |\n| `condition_description` | `str` | ✅ Yes | New explanation of dependency logic |\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of complete updated TaskConstellation after updating dependency\"\n}\n```\n\n#### Example Usage\n\n```python\n# Update dependency description\nresult = await mcp_client.call_tool(\n tool_name=\"update_dependency\",\n parameters={\n \"dependency_id\": \"train->evaluate\",\n \"condition_description\": \"Evaluation requires model training to complete successfully with validation loss < 0.5\"\n }\n)\n```\n\n---\n\n## 🏗️ Bulk Operations\n\n### build_constellation\n\nBatch-create a complete constellation from structured configuration.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `config` | `TaskConstellationSchema` | ✅ Yes | Constellation configuration with tasks and dependencies |\n| `clear_existing` | `bool` | ❌ No (default: `True`) | Clear existing constellation before building |\n\n#### Configuration Schema\n\n```python\n{\n \"tasks\": [\n {\n \"task_id\": \"string (required)\",\n \"name\": \"string (optional)\",\n \"description\": \"string (required)\",\n \"target_device_id\": \"string (optional)\",\n \"tips\": [\"string\", ...] (optional),\n \"priority\": int (1-4, optional),\n \"status\": \"string (optional)\",\n \"task_data\": dict (optional)\n }\n ],\n \"dependencies\": [\n {\n \"from_task_id\": \"string (required)\",\n \"to_task_id\": \"string (required)\",\n \"dependency_type\": \"string (optional)\",\n \"condition_description\": \"string (optional)\"\n }\n ],\n \"metadata\": dict (optional)\n}\n```\n\n#### Return Value\n\n```json\n{\n \"type\": \"string\",\n \"description\": \"JSON string of built TaskConstellation with all tasks, dependencies, and metadata\"\n}\n```\n\n#### Example Usage\n\n```python\n# Build complete ML training pipeline\nconfig = {\n \"tasks\": [\n {\n \"task_id\": \"fetch_data\",\n \"name\": \"Fetch Training Data\",\n \"description\": \"Download CIFAR-10 dataset from S3\",\n \"target_device_id\": \"laptop_001\"\n },\n {\n \"task_id\": \"preprocess\",\n \"name\": \"Preprocess Data\",\n \"description\": \"Normalize images, augment with rotations\",\n \"target_device_id\": \"server_001\"\n },\n {\n \"task_id\": \"train\",\n \"name\": \"Train Model\",\n \"description\": \"Train ResNet-50 for 100 epochs\",\n \"target_device_id\": \"gpu_server_001\",\n \"tips\": [\"Use mixed precision\", \"Save checkpoints every 10 epochs\"]\n },\n {\n \"task_id\": \"evaluate\",\n \"name\": \"Evaluate Model\",\n \"description\": \"Run inference on test set, compute metrics\",\n \"target_device_id\": \"test_server_001\"\n }\n ],\n \"dependencies\": [\n {\n \"from_task_id\": \"fetch_data\",\n \"to_task_id\": \"preprocess\",\n \"condition_description\": \"Preprocessing requires raw data\"\n },\n {\n \"from_task_id\": \"preprocess\",\n \"to_task_id\": \"train\",\n \"condition_description\": \"Training requires preprocessed data\"\n },\n {\n \"from_task_id\": \"train\",\n \"to_task_id\": \"evaluate\",\n \"condition_description\": \"Evaluation requires trained model\"\n }\n ],\n \"metadata\": {\n \"project\": \"image_classification\",\n \"version\": \"1.0\"\n }\n}\n\nresult = await mcp_client.call_tool(\n tool_name=\"build_constellation\",\n parameters={\n \"config\": config,\n \"clear_existing\": True\n }\n)\n```\n\n#### Execution Order\n\n1. **Clear existing** (if `clear_existing=True`)\n2. **Create all tasks** sequentially\n3. **Create all dependencies** sequentially\n4. **Validate DAG** structure (acyclicity, task references)\n5. **Return constellation** snapshot\n\n#### Validation\n\n- **Task references**: All `from_task_id` and `to_task_id` in dependencies must exist in tasks\n- **DAG acyclicity**: Final graph must have no cycles\n- **Schema compliance**: Pydantic validation ensures type correctness\n\n**Creation Mode Usage**: In creation mode, the Constellation Agent uses `build_constellation` to generate the initial constellation in a single operation, which is more efficient than incremental `add_task` calls.\n\n---\n\n## 📊 Tool Comparison Table\n\n| Tool | Category | Granularity | Creates | Modifies | Deletes | Returns |\n|------|----------|-------------|---------|----------|---------|---------|\n| `add_task` | Task | Single | ✅ Task | ❌ | ❌ | Full constellation |\n| `remove_task` | Task | Single | ❌ | ❌ | ✅ Task + deps | Full constellation |\n| `update_task` | Task | Single | ❌ | ✅ Task | ❌ | Full constellation |\n| `add_dependency` | Dependency | Single | ✅ Dependency | ❌ | ❌ | Full constellation |\n| `remove_dependency` | Dependency | Single | ❌ | ❌ | ✅ Dependency | Full constellation |\n| `update_dependency` | Dependency | Single | ❌ | ✅ Dependency | ❌ | Full constellation |\n| `build_constellation` | Bulk | Batch | ✅ Many | ✅ Full | ✅ All (if clear) | Full constellation |\n\n---\n\n## 🔄 Usage Patterns\n\n### Creation Mode Pattern\n\n```python\n# Agent creates initial constellation via build_constellation\nconfig = {\n \"tasks\": [...],\n \"dependencies\": [...]\n}\n\nconstellation_json = await mcp_client.call_tool(\n \"build_constellation\",\n {\"config\": config, \"clear_existing\": True}\n)\n\n# Parse and start orchestration\nconstellation = TaskConstellation.from_json(constellation_json)\n```\n\n### Editing Mode Pattern\n\n```python\n# Agent edits constellation incrementally based on events\n\n# Scenario: Training failed, add diagnostic task\ndiagnostic_json = await mcp_client.call_tool(\n \"add_task\",\n {\n \"task_id\": \"diagnose_failure\",\n \"name\": \"Diagnose Training Failure\",\n \"description\": \"Check logs, GPU memory, data integrity\",\n \"target_device_id\": \"gpu_server_001\"\n }\n)\n\n# Add dependency from failed task to diagnostic\ndep_json = await mcp_client.call_tool(\n \"add_dependency\",\n {\n \"dependency_id\": \"train->diagnose\",\n \"from_task_id\": \"train\",\n \"to_task_id\": \"diagnose_failure\",\n \"condition_description\": \"Run diagnostics after training failure\"\n }\n)\n\n# Remove original deployment task (no longer needed)\nfinal_json = await mcp_client.call_tool(\n \"remove_task\",\n {\"task_id\": \"deploy_model\"}\n)\n```\n\n### Modification Constraints\n\n```python\n# Check if task is modifiable before editing\nmodifiable_tasks = constellation.get_modifiable_tasks()\nmodifiable_task_ids = {t.task_id for t in modifiable_tasks}\n\nif \"train_model\" in modifiable_task_ids:\n # Safe to modify\n await mcp_client.call_tool(\"update_task\", {...})\nelse:\n # Task is RUNNING, COMPLETED, or FAILED - read-only\n print(\"Task cannot be modified in current state\")\n```\n\n---\n\n## 🛡️ Error Handling\n\n### Common Errors\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| `Task not found` | Invalid `task_id` | Verify task exists in constellation |\n| `Dependency creates cycle` | Adding edge violates DAG | Remove conflicting dependencies |\n| `Task not modifiable` | Task is running/completed | Wait or skip modification |\n| `Duplicate task_id` | ID already exists | Use unique identifier |\n| `Invalid device` | `target_device_id` not in registry | Choose from available devices |\n| `At least one field required` | Empty `update_task` call | Provide fields to update |\n\n### Exception Handling\n\n```python\nfrom fastmcp.exceptions import ToolError\n\ntry:\n result = await mcp_client.call_tool(\n \"add_dependency\",\n {\n \"dependency_id\": \"c->a\",\n \"from_task_id\": \"task_c\",\n \"to_task_id\": \"task_a\"\n }\n )\nexcept ToolError as e:\n print(f\"Operation failed: {e}\")\n # Output: \"Failed to add dependency: Adding edge would create cycle\"\n```\n\n---\n\n## 📈 Performance Characteristics\n\n### Operation Complexity\n\n| Tool | Time Complexity | Space Complexity | Notes |\n|------|----------------|------------------|-------|\n| `add_task` | $O(1)$ | $O(1)$ | Constant time insertion |\n| `remove_task` | $O(e)$ | $O(1)$ | Must remove $e$ dependencies |\n| `update_task` | $O(1)$ | $O(1)$ | In-place field update |\n| `add_dependency` | $O(n + e)$ | $O(n)$ | Cycle detection via DFS |\n| `remove_dependency` | $O(1)$ | $O(1)$ | Direct deletion |\n| `update_dependency` | $O(1)$ | $O(1)$ | In-place update |\n| `build_constellation` | $O(n + e)$ | $O(n + e)$ | Full constellation rebuild |\n\nWhere:\n- $n$ = number of tasks\n- $e$ = number of dependencies\n\n### Scalability\n\n| Metric | Typical | Maximum Tested |\n|--------|---------|----------------|\n| Tasks per constellation | 5-20 | 100+ |\n| Dependencies per constellation | 4-30 | 200+ |\n| build_constellation latency | 50-200ms | 1s |\n| add_task latency | 10-50ms | 100ms |\n| Constellation JSON size | 5-50 KB | 500 KB |\n\n---\n\n## 💡 Best Practices\n\n### Tool Selection\n\n**Creation Mode:** Use `build_constellation` for initial synthesis\n\n**Editing Mode:** Use granular tools (`add_task`, `update_task`, etc.)\n\n**Bulk Edits:** Accumulate changes and apply via `build_constellation` with `clear_existing=False`\n\n### Modification Safety\n\nAlways check task/dependency modifiability before calling update/remove tools:\n\n```python\nmodifiable = constellation.get_modifiable_tasks()\nif task in modifiable:\n await mcp_client.call_tool(\"update_task\", ...)\n```\n\n### Idempotent Operations\n\nDesign agent logic to be idempotent:\n\n```python\n# Safe to retry - will fail gracefully if task exists\ntry:\n await mcp_client.call_tool(\"add_task\", {...})\nexcept ToolError:\n # Task already exists, continue\n pass\n```\n\n---\n\n## 🔗 Related Documentation\n\n- [Constellation Agent Overview](overview.md) — Architecture and weaving modes\n- [Constellation Agent State Machine](state.md) — FSM lifecycle and transitions\n- [Constellation Agent Strategy Pattern](strategy.md) — Processing strategies and prompters\n- [Constellation Editor MCP Server](../../mcp/servers/constellation_editor.md) — Detailed MCP server reference\n- [Task Constellation Overview](../constellation/overview.md) — DAG model and data structures\n- [Processor Framework](../../infrastructure/agents/design/processor.md) — Agent processing architecture\n\n---\n\n## 📋 API Reference\n\n### Tool Signatures\n\n```python\n# Task Management\ndef add_task(\n task_id: str,\n name: str,\n description: str,\n target_device_id: Optional[str] = None,\n tips: Optional[List[str]] = None\n) -> str # JSON string\n\ndef remove_task(task_id: str) -> str # JSON string\n\ndef update_task(\n task_id: str,\n name: Optional[str] = None,\n description: Optional[str] = None,\n target_device_id: Optional[str] = None,\n tips: Optional[List[str]] = None\n) -> str # JSON string\n\n# Dependency Management\ndef add_dependency(\n dependency_id: str,\n from_task_id: str,\n to_task_id: str,\n condition_description: Optional[str] = None\n) -> str # JSON string\n\ndef remove_dependency(dependency_id: str) -> str # JSON string\n\ndef update_dependency(\n dependency_id: str,\n condition_description: str\n) -> str # JSON string\n\n# Bulk Operations\ndef build_constellation(\n config: TaskConstellationSchema,\n clear_existing: bool = True\n) -> str # JSON string\n```\n\n---\n\n**Constellation MCP Server — Structured, idempotent task manipulation for adaptive orchestration**\n"
},
{
"path": "documents/docs/galaxy/constellation_agent/overview.md",
"content": "# Constellation Agent — The Centralized Constellation Weaver\n\nThe **Constellation Agent** serves as the central intelligence of UFO³ Galaxy, acting as both a planner and replanner. It interprets user intent, constructs executable Task Constellations, and dynamically steers their evolution across heterogeneous devices. By bridging high-level natural-language goals and concrete multi-agent execution, the Constellation Agent provides unified orchestration through a feedback-driven control loop.\n\nFor an overview of the Galaxy system architecture, see [Galaxy Overview](../overview.md).\n\n## 🌟 Introduction\n\n\n**Figure:** An overview of the Constellation Agent showing the dual-mode control cycle between creation and editing phases.\n\nThe Constellation Agent extends the abstract [Task Constellation](../constellation/overview.md) model into runtime execution. Residing within the **ConstellationClient** (see [Galaxy Client](../client/overview.md)), it transforms user requests into structured DAG workflows and continuously refines them as distributed agents provide feedback.\n\nUnlike traditional static DAG schedulers, the Constellation Agent operates as a **dynamic orchestrator** powered by an LLM-driven architecture and governed by a finite-state machine (FSM). This design enables it to alternate between two complementary operating modes:\n\n- **Creation Mode**: Synthesizes initial Task Constellations from user instructions\n- **Editing Mode**: Incrementally refines constellations based on runtime feedback\n\nThis feedback-driven control loop achieves tight coupling between symbolic reasoning and distributed execution, maintaining global consistency while adapting to changing device conditions.\n\n## 🎯 Core Responsibilities\n\nThe Constellation Agent orchestrates distributed workflows through structured feedback loops, alternating between creation and editing phases with explicit operational boundaries. For details on task execution, see [Constellation Orchestrator](../constellation_orchestrator/overview.md).\n\n### Primary Functions\n\n| Function | Description | Mode |\n|----------|-------------|------|\n| **Request Interpretation** | Parse user goals and context into actionable requirements | Creation |\n| **DAG Synthesis** | Decompose requests into structured Task Constellations with dependencies | Creation |\n| **Device Assignment** | Map tasks to appropriate devices based on AgentProfile capabilities | Creation |\n| **Runtime Monitoring** | Track task completion events and constellation state | Editing |\n| **Dynamic Adaptation** | Add, remove, or modify tasks/dependencies based on feedback | Editing |\n| **Consistency Maintenance** | Ensure DAG validity and execution correctness throughout lifecycle | Both |\n\n## 🏗️ Architecture\n\n### Dual-Mode Control System\n\nThe Constellation Agent implements a **dual-mode control pattern** that separates planning from replanning:\n\n```mermaid\ngraph LR\n A[User Request] --> B[Creation Mode]\n B --> C[Initial Constellation]\n C --> D[Orchestrator]\n D --> E[Task Execution]\n E --> F{Event Queue}\n F -->|Task Completed| G[Editing Mode]\n G --> H[Updated Constellation]\n H --> D\n F -->|All Complete| I[Finish]\n \n style B fill:#e1f5ff\n style G fill:#fff4e1\n style I fill:#e8f5e9\n```\n\n### Component Integration\n\n```mermaid\ngraph TB\n subgraph \"Constellation Agent\"\n FSM[Finite State Machine]\n Prompter[Prompter]\n Processor[Agent Processor]\n end\n \n subgraph \"MCP Layer\"\n Dispatcher[Command Dispatcher]\n MCP[MCP Server Manager]\n Editor[Constellation Editor MCP]\n end\n \n subgraph \"Execution Layer\"\n Orchestrator[Task Orchestrator]\n EventBus[Event Bus]\n end\n \n FSM --> Prompter\n Prompter --> Processor\n Processor --> Dispatcher\n Dispatcher --> MCP\n MCP --> Editor\n Editor --> Orchestrator\n Orchestrator --> EventBus\n EventBus -->|Task Events| FSM\n \n style FSM fill:#e1f5ff\n style MCP fill:#fff4e1\n style Orchestrator fill:#e8f5e9\n```\n\n## 🔄 Creation Mode\n\nIn creation mode, the Constellation Agent receives a user request and generates the initial Task Constellation.\n\n### Inputs\n\n| Input | Type | Description |\n|-------|------|-------------|\n| **User Request** | `str` | Natural language goal or structured command |\n| **AgentProfile Registry** | `Dict[str, AgentProfile]` | Available device agents with capabilities and metadata |\n| **Demonstration Examples** | `List[Example]` | In-context learning examples for task decomposition |\n\n### Processing Flow\n\n```mermaid\nsequenceDiagram\n participant User\n participant Agent as Constellation Agent\n participant Prompter\n participant LLM\n participant Dispatcher as Command Dispatcher\n participant MCP as MCP Server Manager\n participant Editor as Constellation Editor MCP\n participant Orchestrator\n \n User->>Agent: Submit Request\n Agent->>Prompter: Format Creation Prompt\n Prompter->>LLM: Send Prompt + Examples\n LLM->>Agent: Return Constellation JSON\n Agent->>Dispatcher: Execute build_constellation\n Dispatcher->>MCP: Route Command\n MCP->>Editor: Call build_constellation\n Editor->>MCP: Return Built Constellation\n MCP->>Dispatcher: Return Result\n Dispatcher->>Agent: Constellation Ready\n Agent->>Orchestrator: Start Execution\n Orchestrator-->>Agent: Constellation Started\n Agent->>User: Display Initial Plan\n```\n\n### Outputs\n\n| Output | Type | Description |\n|--------|------|-------------|\n| **Task Constellation** | `TaskConstellation` | Structured DAG with tasks and dependencies |\n| **Observation** | `str` | Analysis of input context and device profiles |\n| **Thought** | `str` | Reasoning trace explaining decomposition logic |\n| **State** | `ConstellationAgentStatus` | Next FSM state (typically `CONTINUE`) |\n| **Result** | `Any` | Summary for user or error message |\n\n**Example: Creation Mode Response**\n\n**User Request:** \"Download dataset on laptop, preprocess on server, train model on GPU\"\n\n**Generated Constellation:**\n\n- Task 1: `fetch_data` → Device: laptop\n- Task 2: `preprocess` → Device: linux_server (depends on Task 1)\n- Task 3: `train_model` → Device: gpu_server (depends on Task 2)\n\n**Thought:** \"Decomposed into 3 sequential tasks based on computational requirements. Laptop handles download, server preprocesses data, GPU server trains model.\"\n\n## ✏️ Editing Mode\n\nDuring execution, the Constellation Agent enters editing mode to process task completion events and adapt the constellation.\n\n### Inputs\n\n| Input | Type | Description |\n|-------|------|-------------|\n| **Original Request** | `str` | The initial user request for context |\n| **AgentProfile Registry** | `Dict[str, AgentProfile]` | Current device availability |\n| **Current Constellation** | `TaskConstellation` | Serialized constellation snapshot |\n| **Task Events** | `List[TaskEvent]` | Completion/failure events from orchestrator |\n| **Demonstration Examples** | `List[Example]` | In-context learning examples for editing |\n\n### Processing Flow\n\n```mermaid\nsequenceDiagram\n participant Orchestrator\n participant EventBus\n participant Agent as Constellation Agent\n participant Prompter\n participant LLM\n participant Dispatcher as Command Dispatcher\n participant MCP as MCP Server Manager\n participant Editor as Constellation Editor MCP\n \n Orchestrator->>EventBus: Task Completed Event\n EventBus->>Agent: Queue Event\n Agent->>Agent: Collect Pending Events\n Agent->>Dispatcher: Sync Constellation State\n Dispatcher->>MCP: build_constellation (sync)\n MCP->>Editor: Update State\n Agent->>Prompter: Format Editing Prompt\n Prompter->>LLM: Send Current State + Events\n LLM->>Agent: Return Modification Actions\n Agent->>Dispatcher: Execute Modification Commands\n Dispatcher->>MCP: Route Commands\n MCP->>Editor: Apply Modifications\n Editor->>MCP: Return Updated Constellation\n MCP->>Dispatcher: Return Results\n Dispatcher->>Agent: Constellation Updated\n Agent->>EventBus: Publish Modified Event\n Agent->>Orchestrator: Continue Execution\n```\n\n### Editing Operations\n\nThe agent can perform the following modifications through the MCP-based Constellation Editor:\n\n| Operation | Use Case | Example |\n|-----------|----------|---------|\n| **Add Task** | Introduce follow-up or diagnostic tasks | Add health check after training fails |\n| **Remove Task** | Prune redundant or obsolete tasks | Remove preprocessing if data is pre-processed |\n| **Update Task** | Modify description, device, or tips | Switch training to different GPU |\n| **Add Dependency** | Establish new task relationships | Make validation depend on training |\n| **Remove Dependency** | Decouple independent tasks | Remove unnecessary sequential constraint |\n| **Update Dependency** | Change conditional logic | Update success criteria for task trigger |\n\n> **Note:** Only tasks in `PENDING` or `WAITING_DEPENDENCY` status can be modified. Running or completed tasks are **read-only** to ensure execution consistency.\n\n### Outputs\n\n| Output | Type | Description |\n|--------|------|-------------|\n| **Updated Constellation** | `TaskConstellation` | Modified DAG with new tasks/dependencies |\n| **Thought** | `str` | Reasoning explaining modifications or no-op |\n| **State** | `ConstellationAgentStatus` | Next FSM state (`CONTINUE`, `FINISH`, or `FAIL`) |\n| **Result** | `Any` | Summary of changes or completion status |\n\n## 🔁 Finite-State Machine Lifecycle\n\n\n**Figure:** Lifecycle state transitions of the Constellation Agent FSM.\n\nThe Constellation Agent's behavior is governed by a **4-state finite-state machine**:\n\n| State | Description | Triggers |\n|-------|-------------|----------|\n| **START** | Initialize constellation, begin orchestration | Agent instantiation, restart after completion |\n| **CONTINUE** | Monitor events, process feedback, update constellation | Task completion/failure events |\n| **FINISH** | Successful termination, aggregate results | All tasks completed successfully |\n| **FAIL** | Terminal error state, abort execution | Irrecoverable errors, validation failures |\n\n### State Transition Rules\n\n```mermaid\nstateDiagram-v2\n [*] --> START: Initialize Agent\n START --> CONTINUE: Constellation Created\n START --> FAIL: Creation Failed\n \n CONTINUE --> CONTINUE: Process Events\n CONTINUE --> FINISH: All Tasks Complete\n CONTINUE --> FAIL: Critical Error\n CONTINUE --> START: New Constellation Needed\n \n FINISH --> [*]\n FAIL --> [*]\n \n note right of START\n Creation Mode:\n - Generate initial constellation\n - Validate DAG structure\n - Start orchestration\n end note\n \n note right of CONTINUE\n Editing Mode:\n - Wait for task events\n - Process completion feedback\n - Apply modifications\n end note\n```\n\nFor detailed state machine documentation, see [State Machine Details](state.md).\n\n## 🛠️ MCP-Based Constellation Editor\n\nThe Constellation Agent interacts with the **Constellation Editor** through the **Model Context Protocol (MCP)** layer. The architecture uses:\n\n- **MCP Server Manager**: Routes commands to appropriate MCP servers\n- **Command Dispatcher**: Provides a unified interface for executing MCP commands\n- **Constellation Editor MCP Server**: Implements the actual constellation manipulation operations\n\nThis MCP-based architecture provides:\n\n- **Protocol Standardization**: Consistent interface across all agent types\n- **Loose Coupling**: Agent logic decoupled from editor implementation\n- **Extensibility**: Easy to add new operations or alternative editors\n- **Tool Discovery**: Dynamic tool listing via `list_tools` command\n\n### Core MCP Operations\n\nThe Constellation Editor MCP Server exposes the following operations:\n\n| Operation | Purpose | Inputs | Output |\n|------|---------|--------|--------|\n| `build_constellation` | Batch-create constellation from config | Configuration dict, clear flag | Built constellation |\n| `add_task` | Add atomic task node | Task ID, name, description, device, tips | Updated constellation |\n| `remove_task` | Remove task and dependencies | Task ID | Updated constellation |\n| `update_task` | Modify task fields | Task ID + updated fields | Updated constellation |\n| `add_dependency` | Create dependency edge | From/to task IDs, type, condition | Updated constellation |\n| `remove_dependency` | Delete dependency | Dependency ID | Updated constellation |\n| `update_dependency` | Update dependency logic | Dependency ID, condition | Updated constellation |\n\nAll operations are:\n\n- **Idempotent**: Safe to retry without side effects\n- **Atomic**: Single operation per command\n- **Validated**: Ensures DAG consistency after each modification\n- **Auditable**: All changes are logged and traceable\n\nFor complete MCP command specifications and examples, see [Command Reference](command.md). For details on the underlying Task Constellation structure, see [Task Constellation Overview](../constellation/overview.md).\n\n## 📋 Processing Pipeline\n\nThe Constellation Agent follows a **4-phase processing pipeline** for both creation and editing modes:\n\n### Phase 1: Context Provision\n\n```python\n# Load available MCP tools from Constellation Editor\nawait agent.context_provision(context=context)\n# Queries MCP server for available operations via list_tools\n# Formats tools into LLM-compatible prompt\n```\n\n### Phase 2: LLM Interaction\n\n```python\n# Construct prompt based on mode\nprompt = agent.message_constructor(\n request=user_request,\n device_info=agent_profiles,\n constellation=current_constellation\n)\n\n# Get LLM response\nresponse = await llm.query(prompt)\n# Returns: ConstellationAgentResponse with thought, status, actions\n```\n\n### Phase 3: Action Execution\n\n```python\n# Execute MCP commands via Command Dispatcher\nfor command in response.actions:\n result = await context.command_dispatcher.execute_commands([command])\n \n# Validate constellation\nis_valid, errors = constellation.validate_dag()\n```\n\n### Phase 4: Memory Update\n\n```python\n# Update global context\ncontext.set(ContextNames.CONSTELLATION, updated_constellation)\ncontext.set(ContextNames.ROUND_RESULT, results)\n\n# Log to memory\nmemory.add_round_log(\n step=step,\n weaving_mode=mode,\n request=request,\n constellation=constellation,\n response=response\n)\n```\n\n## 🎭 Prompter Architecture\n\nThe Constellation Agent uses the **Factory Pattern** to create appropriate prompters for different weaving modes (creation and editing).\n\n### Prompter Hierarchy\n\n```mermaid\nclassDiagram\n class BaseConstellationPrompter {\n <>\n +format_agent_profile()\n +format_constellation()\n +user_content_construction()\n +system_prompt_construction()\n }\n \n class ConstellationCreationPrompter {\n +user_prompt_construction()\n +examples_prompt_helper()\n }\n \n class ConstellationEditingPrompter {\n +user_prompt_construction()\n +examples_prompt_helper()\n }\n \n class ConstellationPrompterFactory {\n +create_prompter(mode)\n +get_supported_modes()\n }\n \n BaseConstellationPrompter <|-- ConstellationCreationPrompter\n BaseConstellationPrompter <|-- ConstellationEditingPrompter\n ConstellationPrompterFactory --> BaseConstellationPrompter\n```\n\n### Factory Pattern Benefits\n\n| Benefit | Description |\n|---------|-------------|\n| **Mode Isolation** | Creation and editing prompts remain independent |\n| **Extensibility** | New modes can be added without modifying existing code |\n| **Type Safety** | Compile-time checking for prompter selection |\n| **Testability** | Each prompter can be unit tested independently |\n\nFor complete prompter architecture documentation, see [Prompter Details](strategy.md).\n\n## 💡 Key Design Benefits\n\n### 1. Unified Reasoning and Control\n\nHigh-level task synthesis and low-level execution coordination are decoupled yet tightly synchronized through the Task Constellation abstraction. The agent focuses on semantic reasoning while the orchestrator handles distributed execution.\n\n### 2. Dynamic Adaptability\n\nThe editable constellation enables:\n- **Failure Recovery**: Add diagnostic tasks after failures\n- **Resource Reallocation**: Switch tasks to available devices\n- **Opportunistic Execution**: Insert new tasks as conditions permit\n\n### 3. End-to-End Observability\n\nComplete lineage tracking of:\n- **State Transitions**: FSM state changes logged with timestamps\n- **Modifications**: All edits tracked with before/after snapshots\n- **Events**: Task completion events queued and processed\n- **Reasoning Traces**: LLM thought processes captured in memory\n\n### 4. Safe Modification Guarantees\n\nThe FSM + MCP Server architecture ensures:\n- **Acyclicity**: DAG validation prevents circular dependencies\n- **Consistency**: Only modifiable tasks can be edited\n- **Atomicity**: Each MCP operation is atomic and idempotent\n- **Auditability**: Full modification history maintained\n\n## 🔍 Example Workflow\n\n### User Request\n```\n\"Download MNIST dataset on laptop, train CNN on GPU server, \nevaluate on test server, deploy to production if accuracy > 95%\"\n```\n\n### Creation Mode Output\n\n```json\n{\n \"thought\": \"Decomposed into 4 tasks: (1) download on laptop, (2) train on GPU, (3) evaluate on test server, (4) conditional deploy based on accuracy\",\n \"status\": \"CONTINUE\",\n \"constellation\": {\n \"tasks\": [\n {\"task_id\": \"task_001\", \"name\": \"download_mnist\", \"device\": \"laptop\"},\n {\"task_id\": \"task_002\", \"name\": \"train_cnn\", \"device\": \"gpu_server\"},\n {\"task_id\": \"task_003\", \"name\": \"evaluate\", \"device\": \"test_server\"},\n {\"task_id\": \"task_004\", \"name\": \"deploy\", \"device\": \"prod_server\"}\n ],\n \"dependencies\": [\n {\"from\": \"task_001\", \"to\": \"task_002\", \"type\": \"SUCCESS_ONLY\"},\n {\"from\": \"task_002\", \"to\": \"task_003\", \"type\": \"SUCCESS_ONLY\"},\n {\"from\": \"task_003\", \"to\": \"task_004\", \"type\": \"CONDITIONAL\", \n \"condition\": \"accuracy > 0.95\"}\n ]\n }\n}\n```\n\n### Editing Mode Event\n\n```\nTask task_003 (evaluate) completed with result: {\"accuracy\": 0.92}\n```\n\n### Editing Mode Output\n\n```json\n{\n \"thought\": \"Evaluation accuracy (92%) did not meet deployment threshold (95%). Adding retraining task with adjusted hyperparameters. Removing original deployment task.\",\n \"status\": \"CONTINUE\",\n \"actions\": [\n {\"tool\": \"add_task\", \"parameters\": {\n \"task_id\": \"task_005\", \n \"name\": \"retrain_with_tuning\",\n \"device\": \"gpu_server\",\n \"description\": \"Retrain with learning rate decay and data augmentation\"\n }},\n {\"tool\": \"add_dependency\", \"parameters\": {\n \"from\": \"task_003\", \"to\": \"task_005\", \"type\": \"SUCCESS_ONLY\"\n }},\n {\"tool\": \"remove_task\", \"parameters\": {\"task_id\": \"task_004\"}}\n ]\n}\n```\n\n## 📊 Performance Characteristics\n\n### Creation Complexity\n\n- **Time**: $O(n \\cdot m)$ where $n$ is task count, $m$ is LLM inference time\n- **Space**: $O(n + e)$ for $n$ tasks and $e$ edges\n- **Validation**: $O(n + e)$ for DAG cycle detection (DFS)\n\n### Editing Complexity\n\n- **Event Processing**: $O(k)$ for $k$ queued events (batched)\n- **Modification**: $O(1)$ per MCP command (constant time)\n- **Re-validation**: $O(n + e)$ for modified constellation\n\n### Scalability\n\n| Metric | Typical | Maximum Tested |\n|--------|---------|----------------|\n| Tasks per Constellation | 5-20 | 100+ |\n| Dependencies per Constellation | 4-30 | 200+ |\n| Editing Events per Session | 1-10 | 50+ |\n| LLM Response Time | 2-5s | 15s |\n\n## 🔗 Related Components\n\n- **[Task Constellation](../constellation/overview.md)** — Abstract DAG model\n- **[TaskStar](../constellation/task_star.md)** — Atomic execution units\n- **[TaskStarLine](../constellation/task_star_line.md)** — Dependency relationships\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Distributed executor\n- **[State Machine](state.md)** — FSM lifecycle details\n- **[Prompter Details](strategy.md)** — Prompter architecture\n- **[Command Reference](command.md)** — Editor operation specifications\n\n## 🎯 Summary\n\nThe Constellation Agent serves as the **central weaver** of distributed intelligence in UFO³ Galaxy. Through its dual-mode control loop, finite-state machine governance, and MCP-based constellation manipulation, it transforms abstract user goals into live, evolving constellations—maintaining both rigor and adaptability across the complete lifecycle of multi-device orchestration.\n\n**Key Capabilities:**\n\n- **Semantic Decomposition**: Natural language → structured DAG \n- **Dynamic Adaptation**: Runtime graph evolution based on feedback \n- **MCP Integration**: Protocol-based tool invocation for extensibility\n- **Formal Guarantees**: DAG validity + safe concurrent modification \n- **Complete Observability**: Full lineage tracking and reasoning traces \n- **Modular Design**: Clean separation between reasoning and execution\n"
},
{
"path": "documents/docs/galaxy/constellation_agent/state.md",
"content": "# Constellation Agent State Machine\n\nThe Constellation Agent's finite-state machine provides deterministic lifecycle management while enabling dynamic constellation evolution. This FSM governs how the agent transitions between creation, monitoring, success, and failure states—ensuring predictable behavior in complex distributed workflows.\n\nFor an overview of the Constellation Agent architecture, see [Overview](overview.md).\n\n## 📐 State Machine Overview\n\n\n**Figure:** Lifecycle state transitions of the Constellation Agent showing the 4-state FSM.\n\nThe Constellation Agent implements a **4-state finite-state machine (FSM)** that provides clear, enforceable structure for task lifecycle management. This design separates LLM reasoning from deterministic control logic, improving safety and debuggability.\n\n### State Space\n\n```mermaid\nstateDiagram-v2\n [*] --> START: Agent Initialization\n START --> CONTINUE: Constellation Created Successfully\n START --> FAIL: Creation Failed\n \n CONTINUE --> CONTINUE: Process Task Events\n CONTINUE --> FINISH: All Tasks Complete\n CONTINUE --> FAIL: Critical Error\n CONTINUE --> START: Restart Needed\n \n FINISH --> [*]: Success\n FAIL --> [*]: Abort\n```\n\n## 🎯 State Definitions\n\n### State Enumeration\n\n```python\nclass ConstellationAgentStatus(Enum):\n \"\"\"Constellation Agent states\"\"\"\n START = \"START\"\n CONTINUE = \"CONTINUE\"\n FINISH = \"FINISH\"\n FAIL = \"FAIL\"\n```\n\n| State | Type | Description | Entry Conditions |\n|-------|------|-------------|------------------|\n| **START** | Initial | Initialize and create constellation | Agent instantiation, restart after completion |\n| **CONTINUE** | Steady-State | Monitor events and process feedback | Constellation created successfully |\n| **FINISH** | Terminal | Successful termination | All tasks completed, no edits needed |\n| **FAIL** | Terminal | Error termination | Irrecoverable errors, validation failures |\n\n## 🚀 START State\n\n### Purpose\n\nThe START state is the **initialization and creation phase** where the agent:\n1. Generates the initial Task Constellation from user request\n2. Validates DAG structure for correctness\n3. Launches background orchestration\n4. Transitions to monitoring mode\n\n### State Handler Implementation\n\n```python\n@ConstellationAgentStateManager.register\nclass StartConstellationAgentState(ConstellationAgentState):\n \"\"\"Start state - create and execute constellation\"\"\"\n \n async def handle(self, agent: \"ConstellationAgent\", context: Context) -> None:\n # Skip if already in terminal state\n if agent.status in [\n ConstellationAgentStatus.FINISH.value,\n ConstellationAgentStatus.FAIL.value,\n ]:\n return\n \n # Initialize timing_info\n timing_info = {}\n \n # Create constellation if not exists\n if not agent.current_constellation:\n context.set(ContextNames.WEAVING_MODE, WeavingMode.CREATION)\n \n agent._current_constellation, timing_info = (\n await agent.process_creation(context)\n )\n \n # Start orchestration in background\n if agent.current_constellation:\n asyncio.create_task(\n agent.orchestrator.orchestrate_constellation(\n agent.current_constellation, \n metadata=timing_info\n )\n )\n agent.status = ConstellationAgentStatus.CONTINUE.value\n elif agent.status == ConstellationAgentStatus.CONTINUE.value:\n agent.status = ConstellationAgentStatus.FAIL.value\n```\n\n### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant FSM as State Machine\n participant Agent\n participant Creation as Creation Process\n participant Validator\n participant Orchestrator\n \n FSM->>Agent: handle(START)\n Agent->>Agent: Check if constellation exists\n \n alt No Constellation\n Agent->>Creation: process_creation(context)\n Creation->>Agent: Return constellation + timing\n Agent->>Validator: validate_dag()\n \n alt Valid DAG\n Validator-->>Agent: Success\n Agent->>Orchestrator: orchestrate_constellation()\n Note over Orchestrator: Background task started\n Agent->>FSM: Set status = CONTINUE\n else Invalid DAG\n Validator-->>Agent: Errors\n Agent->>FSM: Set status = FAIL\n end\n else Constellation Exists\n Agent->>Orchestrator: orchestrate_constellation()\n Agent->>FSM: Set status = CONTINUE\n end\n```\n\n### Behaviors\n\n| Scenario | Action | Next State |\n|----------|--------|------------|\n| **First Execution** | Generate constellation via LLM | `CONTINUE` (success) / `FAIL` (error) |\n| **Restart Trigger** | Use existing constellation | `CONTINUE` |\n| **Creation Failure** | Log error, no constellation created | `FAIL` |\n| **Validation Failure** | DAG contains cycles or invalid structure | `FAIL` |\n| **Already Terminal** | No-op, return immediately | Same state |\n\n> **Tip:** Orchestration is launched as a **non-blocking** background task using `asyncio.create_task()`. This allows the agent to transition to CONTINUE state immediately and begin monitoring for events.\n\n### Error Handling\n\n```python\ntry:\n # Creation logic\n agent._current_constellation, timing_info = (\n await agent.process_creation(context)\n )\nexcept AttributeError as e:\n agent.logger.error(f\"Attribute error: {traceback.format_exc()}\")\n agent.status = ConstellationAgentStatus.FAIL.value\nexcept KeyError as e:\n agent.logger.error(f\"Missing key: {traceback.format_exc()}\")\n agent.status = ConstellationAgentStatus.FAIL.value\nexcept Exception as e:\n agent.logger.error(f\"Unexpected error: {traceback.format_exc()}\")\n agent.status = ConstellationAgentStatus.FAIL.value\n```\n\n## 🔄 CONTINUE State\n\n### Purpose\n\nThe CONTINUE state is the **steady-state monitoring and editing phase** where the agent:\n1. Waits for task completion/failure events from orchestrator\n2. Collects batched events from the queue\n3. Merges constellation state with latest modifications\n4. Processes events and applies edits\n5. Loops until all tasks complete or critical error occurs\n\n### State Handler Implementation\n\n```python\n@ConstellationAgentStateManager.register\nclass ContinueConstellationAgentState(ConstellationAgentState):\n \"\"\"Continue state - wait for task completion events\"\"\"\n \n async def handle(self, agent: \"ConstellationAgent\", context=None) -> None:\n # Set editing mode\n context.set(ContextNames.WEAVING_MODE, WeavingMode.EDITING)\n \n # Collect task completion events (batched)\n completed_task_events = []\n \n # Wait for at least one event (blocking)\n first_event = await agent.task_completion_queue.get()\n completed_task_events.append(first_event)\n \n # Collect other pending events (non-blocking)\n while not agent.task_completion_queue.empty():\n try:\n event = agent.task_completion_queue.get_nowait()\n completed_task_events.append(event)\n except asyncio.QueueEmpty:\n break\n \n # Get latest constellation and merge states\n latest_constellation = completed_task_events[-1].data.get(\"constellation\")\n merged_constellation = await self._get_merged_constellation(\n agent, latest_constellation\n )\n \n # Process editing with all collected events\n await agent.process_editing(\n context=context,\n task_ids=[e.task_id for e in completed_task_events],\n before_constellation=merged_constellation\n )\n```\n\n### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant FSM as State Machine\n participant Agent\n participant Queue as Event Queue\n participant Sync as State Synchronizer\n participant Editing as Editing Process\n \n FSM->>Agent: handle(CONTINUE)\n Agent->>Queue: Wait for event (blocking)\n Queue-->>Agent: Task Event 1\n \n loop Collect Pending\n Agent->>Queue: Get nowait()\n Queue-->>Agent: Task Event N\n end\n \n Agent->>Sync: Merge constellation states\n Sync-->>Agent: Merged constellation\n \n Agent->>Editing: process_editing(events, constellation)\n Editing->>Agent: Updated constellation\n \n Agent->>FSM: Update status\n```\n\n### Event Batching\n\n**Why Batch Events?**\n\nIf multiple tasks complete simultaneously (e.g., parallel execution), the agent collects **all pending events** before processing. This enables:\n\n- **Single LLM call** instead of multiple sequential calls\n- **Atomic modifications** reflecting multiple completions\n- **Reduced latency** and lower API costs\n\n```python\n# Example: 3 tasks complete in quick succession\n# Without batching: 3 LLM calls, 3 editing sessions\n# With batching: 1 LLM call, 1 editing session processing all 3 events\n```\n\n### State Merging\n\nThe **state synchronizer** merges the orchestrator's constellation with agent modifications:\n\n```python\nasync def _get_merged_constellation(\n self, agent: \"ConstellationAgent\", orchestrator_constellation\n):\n \"\"\"\n Get real-time merged constellation from synchronizer.\n \n Ensures agent processes with most up-to-date state, including\n structural modifications from previous editing sessions.\n \"\"\"\n synchronizer = agent.orchestrator._modification_synchronizer\n \n if not synchronizer:\n return orchestrator_constellation\n \n merged_constellation = synchronizer.merge_and_sync_constellation_states(\n orchestrator_constellation=orchestrator_constellation\n )\n \n agent.logger.info(\n f\"Merged constellation for editing. \"\n f\"Tasks before: {len(orchestrator_constellation.tasks)}, \"\n f\"Tasks after merge: {len(merged_constellation.tasks)}\"\n )\n \n return merged_constellation\n```\n\n> **Warning:** State synchronization is critical. Consider this scenario:\n> \n> 1. Task A completes → Agent edits constellation (adds Task C)\n> 2. Task B completes **while editing is happening**\n> 3. Without merging: Task B editing sees **old state** (no Task C)\n> 4. With merging: Task B editing sees **merged state** (includes Task C)\n\n### Behaviors\n\n| Scenario | Action | Next State |\n|----------|--------|------------|\n| **Task Completed** | Process event, apply edits | `CONTINUE` |\n| **Multiple Tasks Completed** | Batch process, single edit session | `CONTINUE` |\n| **All Tasks Done** | Agent decides to finish | `FINISH` |\n| **Critical Error** | Exception during processing | `FAIL` |\n| **Restart Needed** | New constellation required | `START` |\n\n### Transition Logic\n\n```python\n# Agent's editing process sets status based on analysis:\n\nif constellation.is_complete() and no_more_edits_needed:\n agent.status = ConstellationAgentStatus.FINISH.value\nelif critical_error_occurred:\n agent.status = ConstellationAgentStatus.FAIL.value\nelif new_constellation_needed:\n agent.status = ConstellationAgentStatus.START.value\nelse:\n agent.status = ConstellationAgentStatus.CONTINUE.value # Keep monitoring\n```\n\n## ✅ FINISH State\n\n### Purpose\n\nThe FINISH state represents **successful termination** when:\n- All tasks in the constellation have completed successfully\n- No further edits are necessary\n- User goal has been achieved\n\n### State Handler Implementation\n\n```python\n@ConstellationAgentStateManager.register\nclass FinishConstellationAgentState(ConstellationAgentState):\n \"\"\"Finish state - task completed successfully\"\"\"\n \n async def handle(self, agent: \"ConstellationAgent\", context=None) -> None:\n agent.logger.info(\"Galaxy task completed successfully\")\n agent._status = ConstellationAgentStatus.FINISH.value\n \n def next_state(self, agent: \"ConstellationAgent\") -> AgentState:\n return self # Terminal state - no transitions\n \n def is_round_end(self) -> bool:\n return True\n \n def is_subtask_end(self) -> bool:\n return True\n```\n\n### Characteristics\n\n| Property | Value | Description |\n|----------|-------|-------------|\n| **Terminal** | Yes | No outgoing transitions |\n| **Round End** | Yes | Marks execution round complete |\n| **Subtask End** | Yes | Marks all subtasks complete |\n\n### Entry Conditions\n\n```python\n# LLM decides to finish based on constellation state\n{\n \"thought\": \"All tasks completed successfully. No further actions needed.\",\n \"status\": \"FINISH\",\n \"result\": {\n \"summary\": \"Dataset downloaded, model trained, deployed to production\",\n \"total_tasks\": 5,\n \"completed\": 5,\n \"failed\": 0\n }\n}\n```\n\n**Clean Termination:**\n\nThe FINISH state ensures graceful shutdown with:\n\n- All resources released\n- Final results aggregated\n- Memory logs persisted\n- Success metrics recorded\n\n## ❌ FAIL State\n\n### Purpose\n\nThe FAIL state represents **error termination** when:\n- Irrecoverable errors occur during creation or editing\n- DAG validation fails\n- Critical system failures prevent continuation\n\n### State Handler Implementation\n\n```python\n@ConstellationAgentStateManager.register\nclass FailConstellationAgentState(ConstellationAgentState):\n \"\"\"Fail state - task failed\"\"\"\n \n async def handle(self, agent: \"ConstellationAgent\", context=None) -> None:\n agent.logger.error(\"Galaxy task failed\")\n agent._status = ConstellationAgentStatus.FAIL.value\n \n def next_state(self, agent: \"ConstellationAgent\") -> AgentState:\n return self # Terminal state - no transitions\n \n def is_round_end(self) -> bool:\n return True\n \n def is_subtask_end(self) -> bool:\n return True\n```\n\n### Failure Scenarios\n\n| Scenario | Trigger | Recovery |\n|----------|---------|----------|\n| **Creation Failure** | LLM cannot decompose request | User reformulates request |\n| **Validation Failure** | Generated DAG has cycles | Agent retries or manual fix |\n| **Critical Exception** | Unexpected system error | Check logs, restart agent |\n| **Timeout** | Processing exceeds limits | Increase timeout or simplify task |\n\n### Error Propagation\n\n```python\n# Example error chain:\ntry:\n constellation = await agent.process_creation(context)\nexcept Exception as e:\n agent.logger.error(f\"Creation failed: {e}\")\n agent.status = ConstellationAgentStatus.FAIL.value\n # State machine handles transition to FAIL state\n```\n\n> **Important:** Both FINISH and FAIL states are **terminal** — they have no outgoing transitions. This ensures the agent cannot accidentally resume execution after completion or failure.\n\n## 🔀 State Transitions\n\n### Transition Matrix\n\n| From ↓ / To → | START | CONTINUE | FINISH | FAIL |\n|---------------|-------|----------|--------|------|\n| **START** | ❌ | ✅ (success) | ❌ | ✅ (error) |\n| **CONTINUE** | ✅ (restart) | ✅ (loop) | ✅ (done) | ✅ (error) |\n| **FINISH** | ❌ | ❌ | ✅ (stay) | ❌ |\n| **FAIL** | ❌ | ❌ | ❌ | ✅ (stay) |\n\n### Transition Rules\n\n```python\nclass ConstellationAgentState(AgentState):\n \"\"\"Base state for Constellation Agent\"\"\"\n \n def next_state(self, agent: \"ConstellationAgent\") -> AgentState:\n \"\"\"Determine next state based on agent status\"\"\"\n status = agent.status\n state = ConstellationAgentStateManager().get_state(status)\n return state\n```\n\n### State Manager\n\n```python\nclass ConstellationAgentStateManager(AgentStateManager):\n \"\"\"State manager for Constellation Agent\"\"\"\n \n _state_mapping: Dict[str, Type[AgentState]] = {}\n \n @property\n def none_state(self) -> AgentState:\n return StartConstellationAgentState()\n```\n\nThe state manager uses the **@register decorator** pattern to automatically register state classes. For more details on the overall agent architecture, see [Constellation Agent Overview](overview.md).\n\n```python\n@ConstellationAgentStateManager.register\nclass StartConstellationAgentState(ConstellationAgentState):\n @classmethod\n def name(cls) -> str:\n return ConstellationAgentStatus.START.value\n```\n\n## 📊 State Metrics\n\n### Execution Timeline\n\n```mermaid\ngantt\n title Constellation Agent State Timeline\n dateFormat YYYY-MM-DD\n section States\n START :start1, 2024-01-01, 3s\n CONTINUE :cont1, after start1, 30s\n CONTINUE :cont2, after cont1, 25s\n CONTINUE :cont3, after cont2, 20s\n FINISH :finish1, after cont3, 1s\n```\n\n### Typical Duration\n\n| State | Typical Duration | Factors |\n|-------|------------------|---------|\n| **START** | 2-5 seconds | LLM response time, validation complexity |\n| **CONTINUE** | Variable (10s - 10min) | Task execution time, parallelism |\n| **FINISH** | < 1 second | Logging and cleanup |\n| **FAIL** | < 1 second | Error logging |\n\n## 🛡️ Error Handling\n\n### Exception Hierarchy\n\n```python\n# START State Error Handling\ntry:\n constellation, timing = await agent.process_creation(context)\nexcept AttributeError as e:\n # Missing attribute (e.g., context field)\n agent.logger.error(f\"Attribute error: {e}\")\n agent.status = ConstellationAgentStatus.FAIL.value\nexcept KeyError as e:\n # Missing key in dictionary\n agent.logger.error(f\"Missing key: {e}\")\n agent.status = ConstellationAgentStatus.FAIL.value\nexcept Exception as e:\n # Catch-all for unexpected errors\n agent.logger.error(f\"Unexpected error: {e}\")\n agent.status = ConstellationAgentStatus.FAIL.value\n```\n\n### Recovery Strategies\n\n| Error Type | State | Recovery Action |\n|------------|-------|-----------------|\n| **Temporary Network Failure** | CONTINUE | Retry with backoff |\n| **Invalid LLM Response** | CONTINUE | Re-prompt with examples |\n| **DAG Cycle Detected** | START | Fail fast, require user intervention |\n| **Task Execution Timeout** | CONTINUE | Mark task failed, continue constellation |\n| **Critical System Error** | Any | Transition to FAIL immediately |\n\n## 🔍 State Inspection\n\n### Agent State Query\n\n```python\n# Check current state\ncurrent_state = agent.current_state\nprint(f\"State: {current_state.name()}\")\n\n# Check if terminal\nif current_state.is_round_end():\n print(\"Agent execution completed\")\n\n# Get status\nstatus = agent.status\nprint(f\"Status: {status}\") # \"START\", \"CONTINUE\", \"FINISH\", or \"FAIL\"\n```\n\n### State History\n\nThe agent maintains state transition history in memory logs:\n\n```python\n{\n \"step\": 1,\n \"state\": \"START\",\n \"timestamp\": \"2024-01-01T10:00:00\",\n \"constellation_id\": \"constellation_abc123\"\n}\n```\n\n## 💡 Best Practices\n\n**State Machine Design:**\n\n1. **Keep states focused**: Each state should have a single, clear responsibility\n2. **Minimize transitions**: Fewer transitions = simpler debugging\n3. **Log all transitions**: Record state changes with context\n4. **Handle errors explicitly**: Don't rely on implicit error propagation\n5. **Use terminal states**: Ensure execution cannot resume accidentally\n\n**Common Pitfalls to Avoid:**\n\n- **Infinite loops in CONTINUE**: Always check termination conditions\n- **Missing error handling**: Unhandled exceptions → unpredictable state\n- **Blocking operations**: Use async/await to prevent deadlocks\n- **State pollution**: Don't modify agent state outside state handlers\n\n**Example: State Transition Logging**\n\n```python\nagent.logger.info(\n f\"State transition: {old_state.name()} → {new_state.name()}\"\n)\n```\n\n## 🔗 Related Documentation\n\n- **[Overview](overview.md)** — Constellation Agent architecture\n- **[Prompter Details](strategy.md)** — Prompter implementation\n- **[Command Reference](command.md)** — MCP tool specifications\n- **[Task Constellation](../constellation/overview.md)** — DAG model\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Task execution engine\n\n## 📋 State Interface Reference\n\n### AgentState Base Class\n\n```python\nclass AgentState(ABC):\n \"\"\"Base interface for agent states\"\"\"\n \n @abstractmethod\n async def handle(self, agent, context) -> None:\n \"\"\"Execute state-specific logic\"\"\"\n pass\n \n def next_state(self, agent) -> AgentState:\n \"\"\"Determine next state based on agent status\"\"\"\n pass\n \n def next_agent(self, agent):\n \"\"\"Get next agent (for multi-agent systems)\"\"\"\n return agent\n \n @abstractmethod\n def is_round_end(self) -> bool:\n \"\"\"Check if this state marks round end\"\"\"\n pass\n \n @abstractmethod\n def is_subtask_end(self) -> bool:\n \"\"\"Check if this state marks subtask end\"\"\"\n pass\n \n @classmethod\n @abstractmethod\n def name(cls) -> str:\n \"\"\"State identifier\"\"\"\n pass\n```\n"
},
{
"path": "documents/docs/galaxy/constellation_agent/strategy.md",
"content": "# Processing Strategy Pattern\n\n## Overview\n\nThe Constellation Agent employs a sophisticated **multi-phase processing architecture** based on the [`ProcessorTemplate`](../../infrastructure/agents/design/processor.md) framework. The core orchestrator `ConstellationAgentProcessor` assembles different processing strategies for three distinct phases: **LLM Interaction**, **Action Execution**, and **Memory Update**. This modular design separates concerns, enables mode-specific behaviors, and provides robust error handling across the processing pipeline.\n\nThe Constellation Agent uses `ConstellationAgentProcessor` as the central orchestrator, which dynamically creates and configures processing strategies based on the weaving mode (CREATION vs. EDITING). This follows the Template Method pattern with Strategy composition.\n\n### Core Architecture\n\n```mermaid\nclassDiagram\n class ProcessorTemplate {\n <>\n +process()*\n +_setup_strategies()*\n +_setup_middleware()*\n -strategies: Dict\n -middleware_chain: List\n }\n \n class ConstellationAgentProcessor {\n +_setup_strategies()\n +_setup_middleware()\n +_get_processor_specific_context_data()\n }\n \n class ConstellationStrategyFactory {\n +create_llm_interaction_strategy()\n +create_action_execution_strategy(mode)\n +create_memory_update_strategy()\n }\n \n class ConstellationLLMInteractionStrategy {\n +execute()\n -_build_comprehensive_prompt()\n -_get_llm_response_with_retry()\n -_parse_and_validate_response()\n }\n \n class BaseConstellationActionExecutionStrategy {\n <>\n +execute()\n +_create_mode_specific_action_info()*\n +publish_actions()*\n +sync_constellation()*\n -_execute_constellation_action()\n }\n \n class ConstellationCreationActionExecutionStrategy {\n +_create_mode_specific_action_info()\n +publish_actions()\n +sync_constellation()\n }\n \n class ConstellationEditingActionExecutionStrategy {\n +_create_mode_specific_action_info()\n +publish_actions()\n +sync_constellation()\n }\n \n class ConstellationMemoryUpdateStrategy {\n +execute()\n -_create_additional_memory_data()\n -_create_and_populate_memory_item()\n }\n \n ProcessorTemplate <|-- ConstellationAgentProcessor\n ConstellationAgentProcessor --> ConstellationStrategyFactory : uses\n ConstellationStrategyFactory --> ConstellationLLMInteractionStrategy : creates\n ConstellationStrategyFactory --> BaseConstellationActionExecutionStrategy : creates\n ConstellationStrategyFactory --> ConstellationMemoryUpdateStrategy : creates\n BaseConstellationActionExecutionStrategy <|-- ConstellationCreationActionExecutionStrategy\n BaseConstellationActionExecutionStrategy <|-- ConstellationEditingActionExecutionStrategy\n```\n\n### Processing Phases\n\n| Phase | Strategy | Purpose | Mode-Specific |\n|-------|----------|---------|---------------|\n| **LLM Interaction** | `ConstellationLLMInteractionStrategy` | Prompt construction, LLM response parsing | ❌ Shared |\n| **Action Execution** | `ConstellationCreation/EditingActionExecutionStrategy` | Action generation and execution | ✅ Mode-specific |\n| **Memory Update** | `ConstellationMemoryUpdateStrategy` | Memory logging and state tracking | ❌ Shared |\n\n---\n\n## Processor Framework\n\n### ConstellationAgentProcessor\n\nThe `ConstellationAgentProcessor` extends `ProcessorTemplate` to orchestrate the entire processing workflow. It assembles strategies based on weaving mode and manages the execution pipeline.\n\n#### Initialization\n\n```python\nclass ConstellationAgentProcessor(ProcessorTemplate):\n \"\"\"Enhanced processor for Constellation Agent.\"\"\"\n \n processor_context_class: Type[ConstellationProcessorContext] = (\n ConstellationProcessorContext\n )\n \n def __init__(\n self,\n agent: \"ConstellationAgent\",\n global_context: Context\n ) -> None:\n \"\"\"Initialize with agent and global context.\"\"\"\n super().__init__(agent, global_context)\n```\n\n#### Strategy Assembly\n\nThe processor creates appropriate strategies based on weaving mode:\n\n```python\ndef _setup_strategies(self) -> None:\n \"\"\"Configure processing strategies using factory pattern.\"\"\"\n \n # Get weaving mode from context\n weaving_mode = self.global_context.get(ContextNames.WEAVING_MODE)\n \n if not weaving_mode:\n raise ValueError(\"Weaving mode must be specified in global context\")\n \n # Create strategies via factory\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n ConstellationStrategyFactory.create_llm_interaction_strategy(\n fail_fast=True, # LLM interaction failure should trigger recovery\n )\n )\n \n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n ConstellationStrategyFactory.create_action_execution_strategy(\n weaving_mode=weaving_mode,\n fail_fast=False, # Action failures can be handled gracefully\n )\n )\n \n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n ConstellationStrategyFactory.create_memory_update_strategy(\n fail_fast=False # Memory update failures shouldn't stop the process\n )\n )\n```\n\n#### Middleware Configuration\n\n```python\ndef _setup_middleware(self) -> None:\n \"\"\"Set up enhanced middleware chain with comprehensive monitoring.\"\"\"\n self.middleware_chain = [\n ConstellationLoggingMiddleware() # Specialized logging for Constellation Agent\n ]\n```\n\n#### Context Management\n\n```python\ndef _get_processor_specific_context_data(self) -> Dict[str, Any]:\n \"\"\"Provide Constellation-specific context initialization.\"\"\"\n \n before_constellation = self.global_context.get(\n ContextNames.CONSTELLATION\n )\n \n return {\n \"weaving_mode\": self.global_context.get(ContextNames.WEAVING_MODE),\n \"device_info\": self.global_context.get(ContextNames.DEVICE_INFO),\n \"constellation_before\": (\n before_constellation.to_json() if before_constellation else None\n ),\n }\n```\n\n### Processing Context\n\nThe `ConstellationProcessorContext` extends `BasicProcessorContext` with constellation-specific data:\n\n```python\n@dataclass\nclass ConstellationProcessorContext(BasicProcessorContext):\n \"\"\"Constellation-specific processor context.\"\"\"\n \n # Agent metadata\n agent_type: str = \"ConstellationAgent\"\n weaving_mode: str = \"CREATION\"\n \n # Device and constellation state\n device_info: List[Dict] = field(default_factory=list)\n constellation_before: Optional[str] = None\n constellation_after: Optional[str] = None\n \n # Action information\n action_info: Optional[ActionCommandInfo] = None\n target: Optional[TargetInfo] = None\n \n # Performance tracking\n llm_cost: float = 0.0\n prompt_tokens: int = 0\n completion_tokens: int = 0\n```\n\n---\n\n## Strategy Factory\n\n### ConstellationStrategyFactory\n\nThe factory provides centralized strategy creation with mode-aware instantiation.\n\n#### Factory Methods\n\n```python\nclass ConstellationStrategyFactory:\n \"\"\"Factory for creating Constellation processing strategies.\"\"\"\n \n _action_execution_strategies: Dict[WeavingMode, Type[BaseProcessingStrategy]] = {\n WeavingMode.CREATION: ConstellationCreationActionExecutionStrategy,\n WeavingMode.EDITING: ConstellationEditingActionExecutionStrategy,\n }\n \n @classmethod\n def create_llm_interaction_strategy(\n cls,\n fail_fast: bool = True\n ) -> BaseProcessingStrategy:\n \"\"\"Create LLM interaction strategy (shared across modes).\"\"\"\n return ConstellationLLMInteractionStrategy(fail_fast)\n \n @classmethod\n def create_action_execution_strategy(\n cls,\n weaving_mode: WeavingMode,\n fail_fast: bool = False\n ) -> BaseProcessingStrategy:\n \"\"\"Create mode-specific action execution strategy.\"\"\"\n \n if weaving_mode not in cls._action_execution_strategies:\n raise ValueError(f\"Unsupported mode: {weaving_mode}\")\n \n strategy_class = cls._action_execution_strategies[weaving_mode]\n return strategy_class(fail_fast=fail_fast)\n \n @classmethod\n def create_memory_update_strategy(\n cls,\n fail_fast: bool = False\n ) -> BaseProcessingStrategy:\n \"\"\"Create memory update strategy (shared across modes).\"\"\"\n return ConstellationMemoryUpdateStrategy(fail_fast=fail_fast)\n```\n\n#### Batch Strategy Creation\n\n```python\n@classmethod\ndef create_all_strategies(\n cls,\n weaving_mode: WeavingMode,\n llm_fail_fast: bool = True,\n action_fail_fast: bool = False,\n memory_fail_fast: bool = False,\n) -> Dict[str, BaseProcessingStrategy]:\n \"\"\"Create all required strategies for a weaving mode.\"\"\"\n \n return {\n \"llm_interaction\": cls.create_llm_interaction_strategy(llm_fail_fast),\n \"action_execution\": cls.create_action_execution_strategy(\n weaving_mode, action_fail_fast\n ),\n \"memory_update\": cls.create_memory_update_strategy(memory_fail_fast),\n }\n```\n\n**Note:** The `create_llm_interaction_strategy()` returns a shared `ConstellationLLMInteractionStrategy` (not mode-specific), as LLM interaction logic is the same across creation and editing modes.\n\n---\n\n## LLM Interaction Strategy\n\n### ConstellationLLMInteractionStrategy\n\nHandles prompt construction, LLM communication, and response parsing. This strategy is **shared across both creation and editing modes**, with mode-specific prompt generation delegated to the agent's prompter.\n\n#### Strategy Execution\n\n```python\n@provides(\n \"parsed_response\",\n \"response_text\",\n \"llm_cost\",\n \"prompt_message\",\n \"status\",\n)\nclass ConstellationLLMInteractionStrategy(BaseProcessingStrategy):\n \"\"\"LLM interaction strategy for Constellation Agent.\"\"\"\n \n async def execute(\n self,\n agent: \"ConstellationAgent\",\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Execute LLM interaction with retry logic.\"\"\"\n \n try:\n # Extract context\n session_step = context.get_local(\"session_step\", 0)\n device_info = context.get_local(\"device_info\", {})\n constellation = context.get_global(\"CONSTELLATION\")\n request = context.get(\"request\", \"\")\n \n # Build prompt (delegates to agent's prompter)\n prompt_message = await self._build_comprehensive_prompt(\n agent, device_info, constellation, request, ...\n )\n \n # Get LLM response with retry\n response_text, llm_cost = await self._get_llm_response_with_retry(\n agent, prompt_message\n )\n \n # Parse and validate\n parsed_response = self._parse_and_validate_response(\n agent, response_text\n )\n \n return ProcessingResult(\n success=True,\n data={\n \"parsed_response\": parsed_response,\n \"response_text\": response_text,\n \"llm_cost\": llm_cost,\n **parsed_response.model_dump(),\n },\n phase=ProcessingPhase.LLM_INTERACTION,\n )\n \n except Exception as e:\n return self.handle_error(e, ProcessingPhase.LLM_INTERACTION, context)\n```\n\n#### Prompt Construction\n\nThe strategy delegates mode-specific prompt building to the agent's prompter:\n\n```python\nasync def _build_comprehensive_prompt(\n self,\n agent: \"ConstellationAgent\",\n device_info: Dict,\n constellation: TaskConstellation,\n request: str,\n ...\n) -> Dict[str, Any]:\n \"\"\"Build prompt using agent's mode-specific prompter.\"\"\"\n \n # Agent's message_constructor uses the appropriate prompter\n # (ConstellationCreationPrompter or ConstellationEditingPrompter)\n prompt_message = agent.message_constructor(\n request=request,\n device_info=device_info,\n constellation=constellation\n )\n \n # Log request for debugging\n self._log_request_data(...)\n \n return prompt_message\n```\n\nThe LLM strategy doesn't implement prompt construction directly. Instead, it calls `agent.message_constructor()`, which delegates to the appropriate prompter based on weaving mode. For details on prompter design, see the [Prompter Framework](../../infrastructure/agents/design/prompter.md). The prompters are responsible for mode-specific prompt formatting.\n\n#### Retry Logic\n\n```python\nasync def _get_llm_response_with_retry(\n self,\n agent: \"ConstellationAgent\",\n prompt_message: Dict[str, Any]\n) -> tuple[str, float]:\n \"\"\"Get LLM response with retry for JSON parsing failures.\"\"\"\n \n max_retries = ufo_config.system.JSON_PARSING_RETRY\n \n for retry_count in range(max_retries):\n try:\n # Get response from LLM\n response_text, cost = await asyncio.get_event_loop().run_in_executor(\n None,\n agent.get_response,\n prompt_message,\n AgentType.CONSTELLATION,\n True # use_backup_engine\n )\n \n # Validate JSON parsing\n agent.response_to_dict(response_text)\n \n return response_text, cost\n \n except Exception as e:\n if retry_count < max_retries - 1:\n self.logger.warning(f\"Retry {retry_count + 1}/{max_retries}\")\n else:\n raise Exception(f\"Failed after {max_retries} attempts: {e}\")\n```\n\n#### Response Validation\n\n```python\ndef _parse_and_validate_response(\n self,\n agent: \"ConstellationAgent\",\n response_text: str\n) -> ConstellationAgentResponse:\n \"\"\"Parse and validate LLM response.\"\"\"\n \n response_dict = agent.response_to_dict(response_text)\n parsed_response = ConstellationAgentResponse.model_validate(response_dict)\n \n # Validate required fields\n if not parsed_response.thought:\n raise ValueError(\"Missing 'thought' field\")\n if not parsed_response.status:\n raise ValueError(\"Missing 'status' field\")\n \n agent.print_response(parsed_response)\n return parsed_response\n```\n\n---\n\n## Action Execution Strategies\n\n### Base Action Execution Strategy\n\nThe `BaseConstellationActionExecutionStrategy` provides shared logic for action execution, with abstract methods for mode-specific behaviors.\n\n```python\n@depends_on(\"parsed_response\")\n@provides(\"execution_result\", \"action_info\", \"status\")\nclass BaseConstellationActionExecutionStrategy(BaseProcessingStrategy):\n \"\"\"Base strategy for executing Constellation actions.\"\"\"\n \n def __init__(self, weaving_mode: WeavingMode, fail_fast: bool = False):\n super().__init__(\n name=f\"constellation_action_execution_{weaving_mode.value}\",\n fail_fast=fail_fast\n )\n self.weaving_mode = weaving_mode\n \n async def execute(\n self,\n agent: \"ConstellationAgent\",\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Execute constellation actions with mode-specific logic.\"\"\"\n \n parsed_response = context.get_local(\"parsed_response\")\n command_dispatcher = context.global_context.command_dispatcher\n \n # Create mode-specific action info (abstract method)\n action_info = await self._create_mode_specific_action_info(\n agent, parsed_response\n )\n \n # Execute actions via dispatcher\n execution_results = await self._execute_constellation_action(\n command_dispatcher, action_info\n )\n \n # Sync constellation state (abstract method)\n self.sync_constellation(execution_results, context)\n \n # Create action info for memory\n actions = self._create_action_info(action_info, execution_results)\n \n # Publish actions (abstract method)\n action_list_info = ListActionCommandInfo(actions)\n await self.publish_actions(agent, action_list_info)\n \n return ProcessingResult(\n success=True,\n data={\n \"execution_result\": execution_results,\n \"action_info\": action_list_info,\n \"status\": parsed_response.status,\n },\n phase=ProcessingPhase.ACTION_EXECUTION,\n )\n \n @abstractmethod\n async def _create_mode_specific_action_info(\n self, agent, parsed_response\n ) -> ActionCommandInfo | List[ActionCommandInfo]:\n \"\"\"Must be implemented by subclasses.\"\"\"\n pass\n \n @abstractmethod\n async def publish_actions(\n self, agent, actions\n ) -> None:\n \"\"\"Must be implemented by subclasses.\"\"\"\n pass\n \n @abstractmethod\n def sync_constellation(self, results, context) -> None:\n \"\"\"Must be implemented by subclasses.\"\"\"\n pass\n```\n\n#### Shared Action Execution\n\n```python\nasync def _execute_constellation_action(\n self,\n command_dispatcher: BasicCommandDispatcher,\n actions: ActionCommandInfo | List[ActionCommandInfo],\n) -> List[Result]:\n \"\"\"Execute actions via command dispatcher.\"\"\"\n \n if isinstance(actions, ActionCommandInfo):\n actions = [actions]\n \n commands = [\n Command(\n tool_name=action.function,\n parameters=action.arguments or {},\n tool_type=\"action\"\n )\n for action in actions if action.function\n ]\n \n return await command_dispatcher.execute_commands(commands)\n```\n\n### Creation Mode Strategy\n\nThe `ConstellationCreationActionExecutionStrategy` implements creation-specific action generation.\n\n```python\nclass ConstellationCreationActionExecutionStrategy(\n BaseConstellationActionExecutionStrategy\n):\n \"\"\"Action execution for constellation creation mode.\"\"\"\n \n def __init__(self, fail_fast: bool = False):\n super().__init__(weaving_mode=WeavingMode.CREATION, fail_fast=fail_fast)\n \n async def _create_mode_specific_action_info(\n self,\n agent: \"ConstellationAgent\",\n parsed_response: ConstellationAgentResponse\n ) -> List[ActionCommandInfo]:\n \"\"\"Create constellation building action.\"\"\"\n \n if not parsed_response.constellation:\n self.logger.warning(\"No constellation in response\")\n return []\n \n return [\n ActionCommandInfo(\n function=agent._constellation_creation_tool_name, # \"build_constellation\"\n arguments={\"config\": parsed_response.constellation},\n )\n ]\n \n def sync_constellation(\n self,\n results: List[Result],\n context: ProcessingContext\n ) -> None:\n \"\"\"Sync newly created constellation to context.\"\"\"\n \n constellation_json = results[0].result if results else None\n if constellation_json:\n constellation = TaskConstellation.from_json(constellation_json)\n context.global_context.set(ContextNames.CONSTELLATION, constellation)\n \n async def publish_actions(\n self, agent, actions: ListActionCommandInfo\n ) -> None:\n \"\"\"Publish constellation creation actions as events.\"\"\"\n # Publishes simplified event for WebUI display\n pass\n```\n\n### Editing Mode Strategy\n\nThe `ConstellationEditingActionExecutionStrategy` implements editing-specific action extraction and constellation synchronization.\n\n```python\nclass ConstellationEditingActionExecutionStrategy(\n BaseConstellationActionExecutionStrategy\n):\n \"\"\"Action execution for constellation editing mode.\"\"\"\n \n def __init__(self, fail_fast: bool = False):\n super().__init__(weaving_mode=WeavingMode.EDITING, fail_fast=fail_fast)\n \n async def _create_mode_specific_action_info(\n self,\n agent: \"ConstellationAgent\",\n parsed_response: ConstellationAgentResponse\n ) -> List[ActionCommandInfo]:\n \"\"\"Extract editing actions from LLM response.\"\"\"\n \n if parsed_response.action:\n return parsed_response.action\n else:\n return []\n \n def sync_constellation(\n self,\n results: List[Result],\n context: ProcessingContext\n ) -> None:\n \"\"\"Sync modified constellation from MCP tool results.\"\"\"\n \n # Find last successful result with constellation data\n constellation_json = None\n for result in reversed(results):\n if result.status == ResultStatus.SUCCESS and result.result:\n if isinstance(result.result, str):\n if '\"constellation_id\"' in result.result or '\"tasks\"' in result.result:\n constellation_json = result.result\n break\n elif isinstance(result.result, dict):\n if \"constellation_id\" in result.result or \"tasks\" in result.result:\n constellation_json = result.result\n break\n \n if constellation_json:\n if isinstance(constellation_json, str):\n constellation = TaskConstellation.from_json(constellation_json)\n else:\n constellation = TaskConstellation.from_dict(constellation_json)\n \n context.global_context.set(ContextNames.CONSTELLATION, constellation)\n self.logger.info(f\"Synced constellation: {constellation.constellation_id}\")\n \n async def publish_actions(self, agent, actions: ListActionCommandInfo) -> None:\n \"\"\"Publish editing actions as events for WebUI display.\"\"\"\n # Publishes detailed action events\n pass\n```\n\n---\n\n## Memory Update Strategy\n\n### ConstellationMemoryUpdateStrategy\n\nThe memory update strategy is **shared across both modes** and handles comprehensive memory logging.\n\n```python\n@depends_on(\"parsed_response\")\n@provides(\"additional_memory\", \"memory_item\", \"memory_keys_count\")\nclass ConstellationMemoryUpdateStrategy(BaseProcessingStrategy):\n \"\"\"Memory update strategy (shared across modes).\"\"\"\n \n async def execute(\n self,\n agent: \"ConstellationAgent\",\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Execute comprehensive memory update.\"\"\"\n \n parsed_response = context.get_local(\"parsed_response\")\n \n # Create additional memory data\n additional_memory = self._create_additional_memory_data(agent, context)\n \n # Create and populate memory item\n memory_item = self._create_and_populate_memory_item(\n parsed_response, additional_memory\n )\n \n # Add to agent memory\n agent.add_memory(memory_item)\n \n # Update structural logs\n self._update_structural_logs(memory_item, context.global_context)\n \n return ProcessingResult(\n success=True,\n data={\n \"additional_memory\": additional_memory,\n \"memory_item\": memory_item,\n \"memory_keys_count\": len(memory_item.to_dict()),\n },\n phase=ProcessingPhase.MEMORY_UPDATE,\n )\n```\n\n#### Memory Data Creation\n\n```python\ndef _create_additional_memory_data(\n self,\n agent: \"ConstellationAgent\",\n context: ProcessingContext\n) -> ConstellationProcessorContext:\n \"\"\"Create comprehensive memory data from processing context.\"\"\"\n \n constellation_context = context.local_context\n \n # Update with current state\n constellation_context.session_step = context.get_global(\"SESSION_STEP\", 0)\n constellation_context.round_step = context.get_global(\"CURRENT_ROUND_STEP\", 0)\n constellation_context.round_num = context.get_global(\"CURRENT_ROUND_ID\", 0)\n constellation_context.agent_step = agent.step\n \n # Update action information\n action_info = constellation_context.action_info\n if action_info:\n constellation_context.action = [info.model_dump() for info in action_info.actions]\n constellation_context.function_call = [info.function for info in action_info.actions]\n constellation_context.arguments = [info.arguments for info in action_info.actions]\n \n # Update constellation_after\n constellation_after = context.get_global(\"CONSTELLATION\")\n if constellation_after:\n constellation_context.constellation_after = constellation_after.to_json()\n \n return constellation_context\n```\n\n---\n\n## Mode Comparison\n\n### Strategy Differences by Mode\n\n| Aspect | Creation Mode | Editing Mode |\n|--------|---------------|--------------|\n| **LLM Interaction** | Shared strategy | Shared strategy |\n| **Prompt Generation** | `ConstellationCreationPrompter` | `ConstellationEditingPrompter` |\n| **Action Generation** | `build_constellation` with JSON | Extract `action` field from response |\n| **Action Execution** | Single bulk creation | Multiple MCP commands |\n| **Constellation Sync** | Set from creation result | Extract from last successful MCP result |\n| **Action Publishing** | Simplified event for WebUI | Detailed action events for WebUI |\n| **Memory Update** | Shared strategy | Shared strategy |\n\n### Processing Pipeline Comparison\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant Processor\n participant Factory\n participant LLMStrat\n participant ActionStrat\n participant MemStrat\n \n Note over Agent,MemStrat: CREATION MODE\n Agent->>Processor: process()\n Processor->>Factory: create_action_execution_strategy(CREATION)\n Factory->>Processor: ConstellationCreationActionExecutionStrategy\n Processor->>LLMStrat: execute() [shared]\n LLMStrat->>Processor: parsed_response with constellation JSON\n Processor->>ActionStrat: execute()\n ActionStrat->>ActionStrat: Create build_constellation command\n ActionStrat->>Processor: execution_result\n Processor->>MemStrat: execute() [shared]\n MemStrat->>Processor: memory_item\n \n Note over Agent,MemStrat: EDITING MODE\n Agent->>Processor: process()\n Processor->>Factory: create_action_execution_strategy(EDITING)\n Factory->>Processor: ConstellationEditingActionExecutionStrategy\n Processor->>LLMStrat: execute() [shared]\n LLMStrat->>Processor: parsed_response with action list\n Processor->>ActionStrat: execute()\n ActionStrat->>ActionStrat: Extract MCP commands\n ActionStrat->>Processor: execution_result\n Processor->>MemStrat: execute() [shared]\n MemStrat->>Processor: memory_item\n```\n\n---\n\n## Error Handling\n\n### Fail-Fast Configuration\n\nEach strategy can be configured with `fail_fast` to control error propagation:\n\n```python\n# LLM failures should trigger recovery\nConstellationStrategyFactory.create_llm_interaction_strategy(\n fail_fast=True\n)\n\n# Action failures can be handled gracefully\nConstellationStrategyFactory.create_action_execution_strategy(\n weaving_mode=mode,\n fail_fast=False\n)\n\n# Memory failures shouldn't stop the process\nConstellationStrategyFactory.create_memory_update_strategy(\n fail_fast=False\n)\n```\n\n### Strategy-Level Error Handling\n\n```python\nclass BaseProcessingStrategy:\n def handle_error(\n self,\n error: Exception,\n phase: ProcessingPhase,\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Handle strategy execution errors.\"\"\"\n \n error_msg = f\"{self.name} failed: {str(error)}\"\n self.logger.error(error_msg)\n \n if self.fail_fast:\n raise error\n \n return ProcessingResult(\n success=False,\n data={\"error\": error_msg},\n phase=phase\n )\n```\n\n---\n\n## Best Practices\n\n### Strategy Design\n\n1. **Keep strategies focused**: Each strategy handles one processing phase\n2. **Use dependencies**: Declare data dependencies with `@depends_on` and `@provides`\n3. **Handle errors gracefully**: Configure `fail_fast` appropriately per strategy\n4. **Log comprehensively**: Use structured logging for debugging\n5. **Validate outputs**: Ensure each strategy produces expected data structures\n\n### Mode Selection\n\n```python\ndef determine_strategy_mode(constellation: Optional[TaskConstellation]) -> WeavingMode:\n \"\"\"Determine appropriate mode based on constellation state.\"\"\"\n \n if constellation is None or len(constellation.tasks) == 0:\n return WeavingMode.CREATION\n else:\n return WeavingMode.EDITING\n```\n\n### Testing Strategies\n\n```python\nclass TestConstellationStrategies(unittest.TestCase):\n def test_creation_action_strategy(self):\n \"\"\"Test creation strategy generates build_constellation action.\"\"\"\n \n strategy = ConstellationCreationActionExecutionStrategy()\n response = ConstellationAgentResponse(\n constellation={\"tasks\": [...], \"dependencies\": [...]}\n )\n \n actions = await strategy._create_mode_specific_action_info(\n agent, response\n )\n \n self.assertEqual(len(actions), 1)\n self.assertEqual(actions[0].function, \"build_constellation\")\n \n def test_editing_action_strategy(self):\n \"\"\"Test editing strategy extracts actions from response.\"\"\"\n \n strategy = ConstellationEditingActionExecutionStrategy()\n response = ConstellationAgentResponse(\n action=[\n ActionCommandInfo(function=\"add_task\", arguments={...}),\n ActionCommandInfo(function=\"add_dependency\", arguments={...})\n ]\n )\n \n actions = await strategy._create_mode_specific_action_info(\n agent, response\n )\n \n self.assertEqual(len(actions), 2)\n```\n\n---\n\n## Summary\n\nThe Constellation Agent's processing strategy pattern provides:\n\n- **Modular Processing**: Three distinct phases (LLM, Action, Memory) with dedicated strategies assembled by `ConstellationAgentProcessor`\n- **Mode Flexibility**: Factory-based strategy creation adapts to CREATION vs. EDITING modes\n- **Shared Logic**: LLM interaction and memory update strategies are mode-agnostic\n- **Targeted Customization**: Only action execution varies by mode (creation builds entire constellation, editing applies MCP commands)\n- **Robust Error Handling**: Per-strategy fail-fast configuration\n- **Clean Architecture**: ProcessorTemplate provides the orchestration framework, strategies implement phase-specific logic\n- **Testability**: Each strategy can be tested in isolation\n\nThis architecture enables the Constellation Agent to handle both initial constellation creation and subsequent modifications with appropriate processing strategies while maintaining clean separation of concerns. The processor assembles these strategies dynamically based on weaving mode, making the prompters support components rather than the primary focus of the strategy pattern.\n\n## Related Documentation\n\n- [Constellation Agent Overview](overview.md) - Learn about constellation creation and editing modes\n- [Constellation Agent State Machine](state.md) - Understand the state transitions and lifecycle\n- [Processor Framework Design](../../infrastructure/agents/design/processor.md) - Deep dive into the ProcessorTemplate architecture\n- [Prompter Framework](../../infrastructure/agents/design/prompter.md) - Mode-specific prompt generation framework\n- [Constellation Editor MCP Server](../../mcp/servers/constellation_editor.md) - MCP commands for constellation manipulation\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/api_reference.md",
"content": "# API Reference\n\n## Overview\n\nThis document provides comprehensive API documentation for the Constellation Orchestrator system. The API is organized into three main components:\n\n- **TaskConstellationOrchestrator** - Main orchestration engine\n- **ConstellationManager** - Device assignment and resource management \n- **ConstellationModificationSynchronizer** - Safe concurrent editing\n\n## TaskConstellationOrchestrator\n\nThe main orchestration engine that coordinates task execution across devices.\n\n**Module**: `galaxy.constellation.orchestrator.orchestrator`\n\n### Constructor\n\n```python\nTaskConstellationOrchestrator(\n device_manager: Optional[ConstellationDeviceManager] = None,\n enable_logging: bool = True,\n event_bus = None\n)\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Default |\n|-----------|------|-------------|---------|\n| `device_manager` | `ConstellationDeviceManager` or `None` | Device manager for communication | `None` |\n| `enable_logging` | `bool` | Enable logging output | `True` |\n| `event_bus` | `EventBus` or `None` | Custom event bus instance | `None` (uses global) |\n\n**Example**:\n```python\nfrom galaxy.constellation.orchestrator import TaskConstellationOrchestrator\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\ndevice_manager = ConstellationDeviceManager()\norchestrator = TaskConstellationOrchestrator(\n device_manager=device_manager,\n enable_logging=True\n)\n```\n\n### Core Methods\n\n#### orchestrate_constellation()\n\nMain entry point for orchestrating a constellation's execution.\n\n```python\nasync def orchestrate_constellation(\n self,\n constellation: TaskConstellation,\n device_assignments: Optional[Dict[str, str]] = None,\n assignment_strategy: Optional[str] = None,\n metadata: Optional[Dict] = None,\n) -> Dict[str, Any]\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `constellation` | `TaskConstellation` | The constellation to orchestrate | Yes |\n| `device_assignments` | `Dict[str, str]` or `None` | Manual task→device mapping | No |\n| `assignment_strategy` | `str` or `None` | Strategy: `\"round_robin\"`, `\"capability_match\"`, or `\"load_balance\"` | No |\n| `metadata` | `Dict` or `None` | Additional orchestration metadata | No |\n\n**Returns**: `Dict[str, Any]` with keys:\n```python\n{\n \"results\": {}, # Task results\n \"status\": \"completed\", # Overall status\n \"total_tasks\": int, # Number of tasks\n \"statistics\": {} # Execution statistics\n}\n```\n\n**Raises**:\n- `ValueError`: Invalid DAG structure or device assignments\n- `RuntimeError`: Orchestration execution error\n- `asyncio.CancelledError`: Orchestration cancelled\n\n**Example**:\n```python\n# With automatic assignment\nresults = await orchestrator.orchestrate_constellation(\n constellation=my_constellation,\n assignment_strategy=\"capability_match\"\n)\n\n# With manual assignments\ndevice_assignments = {\n \"task_1\": \"windows_main\",\n \"task_2\": \"android_device\",\n \"task_3\": \"windows_main\"\n}\nresults = await orchestrator.orchestrate_constellation(\n constellation=my_constellation,\n device_assignments=device_assignments\n)\n```\n\n#### execute_single_task()\n\nExecute a single task independently (without constellation context).\n\n```python\nasync def execute_single_task(\n self,\n task: TaskStar,\n target_device_id: Optional[str] = None,\n) -> Any\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `task` | `TaskStar` | Task to execute | Yes |\n| `target_device_id` | `str` or `None` | Device for execution | No (auto-assigned if None) |\n\n**Returns**: Task execution result content (extracts `result.result` from task execution)\n\n**Raises**:\n- `ValueError`: No available devices for task execution\n\n**Example**:\n```python\ntask = TaskStar(\n task_id=\"standalone_task\",\n description=\"Collect system information\"\n)\n\nresult = await orchestrator.execute_single_task(\n task=task,\n target_device_id=\"windows_main\"\n)\n```\n\n#### get_constellation_status()\n\nGet detailed status of a constellation during execution.\n\n```python\nasync def get_constellation_status(\n self, \n constellation: TaskConstellation\n) -> Dict[str, Any]\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `constellation` | `TaskConstellation` | Constellation to query | Yes |\n\n**Returns**: Status dictionary from ConstellationManager\n\n**Note**: This method delegates to `ConstellationManager.get_constellation_status()` using the constellation's ID.\n\n**Example**:\n```python\nstatus = await orchestrator.get_constellation_status(constellation)\nif status:\n print(f\"State: {status['state']}\")\n print(f\"Running: {len(status['running_tasks'])}\")\n```\n\n#### get_available_devices()\n\nGet list of available devices from device manager.\n\n```python\nasync def get_available_devices(self) -> List[Dict[str, Any]]\n```\n\n**Returns**: List of device info dictionaries\n\n**Example**:\n```python\ndevices = await orchestrator.get_available_devices()\nfor device in devices:\n print(f\"{device['device_id']}: {device['device_type']}\")\n```\n\n### Configuration Methods\n\n#### set_device_manager()\n\nSet or update the device manager.\n\n```python\ndef set_device_manager(\n self, \n device_manager: ConstellationDeviceManager\n) -> None\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `device_manager` | `ConstellationDeviceManager` | Device manager instance | Yes |\n\n**Example**:\n```python\nnew_device_manager = ConstellationDeviceManager()\norchestrator.set_device_manager(new_device_manager)\n```\n\n#### set_modification_synchronizer()\n\nAttach a modification synchronizer for safe concurrent editing.\n\n```python\ndef set_modification_synchronizer(\n self, \n synchronizer: ConstellationModificationSynchronizer\n) -> None\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `synchronizer` | `ConstellationModificationSynchronizer` | Synchronizer instance | Yes |\n\n**Example**:\n```python\nfrom galaxy.session.observers.constellation_sync_observer import (\n ConstellationModificationSynchronizer\n)\n\nsynchronizer = ConstellationModificationSynchronizer(orchestrator)\norchestrator.set_modification_synchronizer(synchronizer)\n```\n\n---\n\n## ConstellationManager\n\nManages device assignments, resource allocation, and constellation lifecycle.\n\n**Module**: `galaxy.constellation.orchestrator.constellation_manager`\n\n### Constructor\n\n```python\nConstellationManager(\n device_manager: Optional[ConstellationDeviceManager] = None,\n enable_logging: bool = True\n)\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Default |\n|-----------|------|-------------|---------|\n| `device_manager` | `ConstellationDeviceManager` or `None` | Device manager instance | `None` |\n| `enable_logging` | `bool` | Enable logging | `True` |\n\n### Device Assignment Methods\n\n#### assign_devices_automatically()\n\nAutomatically assign devices to all tasks using a strategy.\n\n```python\nasync def assign_devices_automatically(\n self,\n constellation: TaskConstellation,\n strategy: str = \"round_robin\",\n device_preferences: Optional[Dict[str, str]] = None,\n) -> Dict[str, str]\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Default |\n|-----------|------|-------------|---------|\n| `constellation` | `TaskConstellation` | Constellation to assign | Required |\n| `strategy` | `str` | Assignment strategy | `\"round_robin\"` |\n| `device_preferences` | `Dict[str, str]` or `None` | Preferred task→device mappings | `None` |\n\n**Strategies**:\n- `\"round_robin\"`: Distribute tasks evenly\n- `\"capability_match\"`: Match device types to task requirements\n- `\"load_balance\"`: Minimize maximum device load\n\nFor more details on device assignment strategies, see [Constellation Manager](constellation_manager.md).\n\n**Returns**: `Dict[str, str]` mapping task_id → device_id\n\n**Raises**:\n- `ValueError`: No available devices or invalid strategy\n\n**Example**:\n```python\nassignments = await manager.assign_devices_automatically(\n constellation,\n strategy=\"capability_match\",\n device_preferences={\"critical_task\": \"windows_main\"}\n)\n```\n\n#### reassign_task_device()\n\nReassign a single task to a different device.\n\n```python\ndef reassign_task_device(\n self,\n constellation: TaskConstellation,\n task_id: str,\n new_device_id: str,\n) -> bool\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `constellation` | `TaskConstellation` | Constellation containing task | Yes |\n| `task_id` | `str` | ID of task to reassign | Yes |\n| `new_device_id` | `str` | New device ID | Yes |\n\n**Returns**: `True` if successful, `False` if task not found\n\n**Example**:\n```python\nsuccess = manager.reassign_task_device(\n constellation,\n task_id=\"task_5\",\n new_device_id=\"android_backup\"\n)\n```\n\n#### clear_device_assignments()\n\nClear all device assignments from a constellation.\n\n```python\ndef clear_device_assignments(\n self, \n constellation: TaskConstellation\n) -> int\n```\n\n**Returns**: Number of assignments cleared\n\n### Validation Methods\n\n#### validate_constellation_assignments()\n\nValidate that all tasks have valid device assignments.\n\n```python\ndef validate_constellation_assignments(\n self, \n constellation: TaskConstellation\n) -> tuple[bool, List[str]]\n```\n\n**Returns**: `(is_valid, errors)` tuple\n\n**Example**:\n```python\nis_valid, errors = manager.validate_constellation_assignments(constellation)\nif not is_valid:\n for error in errors:\n print(f\"Error: {error}\")\n```\n\n### Lifecycle Methods\n\n#### register_constellation()\n\nRegister a constellation for management tracking.\n\n```python\ndef register_constellation(\n self,\n constellation: TaskConstellation,\n metadata: Optional[Dict[str, Any]] = None,\n) -> str\n```\n\n**Returns**: Constellation ID\n\n#### unregister_constellation()\n\nUnregister and clean up a constellation.\n\n```python\ndef unregister_constellation(\n self, \n constellation_id: str\n) -> bool\n```\n\n**Returns**: `True` if unregistered, `False` if not found\n\n#### get_constellation()\n\nGet a managed constellation by ID.\n\n```python\ndef get_constellation(\n self, \n constellation_id: str\n) -> Optional[TaskConstellation]\n```\n\n#### list_constellations()\n\nList all managed constellations.\n\n```python\ndef list_constellations(self) -> List[Dict[str, Any]]\n```\n\n**Returns**: List of constellation info dictionaries\n\n### Status Methods\n\n#### get_constellation_status()\n\nGet detailed status of a constellation.\n\n```python\nasync def get_constellation_status(\n self, \n constellation_id: str\n) -> Optional[Dict[str, Any]]\n```\n\n**Returns**: Status dictionary with keys:\n```python\n{\n \"constellation_id\": str,\n \"name\": str,\n \"state\": str,\n \"statistics\": dict,\n \"ready_tasks\": List[str],\n \"running_tasks\": List[str],\n \"completed_tasks\": List[str],\n \"failed_tasks\": List[str],\n \"metadata\": dict\n}\n```\n\n#### get_available_devices()\n\nGet list of available devices.\n\n```python\nasync def get_available_devices(self) -> List[Dict[str, Any]]\n```\n\n**Returns**: List of device info dictionaries:\n```python\n[\n {\n \"device_id\": str,\n \"device_type\": str,\n \"capabilities\": List[str],\n \"status\": str,\n \"metadata\": dict\n },\n ...\n]\n```\n\n#### get_device_utilization()\n\nGet device utilization statistics for a constellation.\n\n```python\ndef get_device_utilization(\n self, \n constellation: TaskConstellation\n) -> Dict[str, int]\n```\n\n**Returns**: `Dict[device_id, task_count]`\n\n#### get_task_device_info()\n\nGet device information for a specific task.\n\n```python\ndef get_task_device_info(\n self,\n constellation: TaskConstellation,\n task_id: str\n) -> Optional[Dict[str, Any]]\n```\n\n**Returns**: Device info dictionary or `None`\n\n---\n\n## ConstellationModificationSynchronizer\n\nSynchronizes constellation modifications with orchestrator execution to prevent race conditions.\n\n**Module**: `galaxy.session.observers.constellation_sync_observer`\n\n### Constructor\n\n```python\nConstellationModificationSynchronizer(\n orchestrator: TaskConstellationOrchestrator,\n logger: Optional[logging.Logger] = None\n)\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `orchestrator` | `TaskConstellationOrchestrator` | Orchestrator instance | Yes |\n| `logger` | `logging.Logger` or `None` | Custom logger | No |\n\n**Example**:\n```python\nsynchronizer = ConstellationModificationSynchronizer(\n orchestrator=orchestrator,\n logger=logging.getLogger(__name__)\n)\n```\n\n### Core Methods\n\n#### on_event()\n\nHandle orchestration events (implements `IEventObserver`).\n\n```python\nasync def on_event(self, event: Event) -> None\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `event` | `Event` | Event to process | Yes |\n\n**Events handled**:\n- `TASK_COMPLETED`: Register pending modification\n- `TASK_FAILED`: Register pending modification\n- `CONSTELLATION_MODIFIED`: Complete pending modifications\n\n#### wait_for_pending_modifications()\n\nWait for all pending modifications to complete.\n\n```python\nasync def wait_for_pending_modifications(\n self, \n timeout: Optional[float] = None\n) -> bool\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Default |\n|-----------|------|-------------|---------|\n| `timeout` | `float` or `None` | Timeout in seconds | `None` (uses default: 600s) |\n\n**Returns**: `True` if all completed, `False` if timeout\n\n**Example**:\n```python\n# In orchestration loop\ncompleted = await synchronizer.wait_for_pending_modifications(timeout=300.0)\nif not completed:\n logger.warning(\"Modifications timed out\")\n```\n\n#### merge_and_sync_constellation_states()\n\nMerge agent's structural changes with orchestrator's execution state.\n\n```python\ndef merge_and_sync_constellation_states(\n self,\n orchestrator_constellation: TaskConstellation,\n) -> TaskConstellation\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `orchestrator_constellation` | `TaskConstellation` | Orchestrator's constellation | Yes |\n\n**Returns**: Merged constellation with consistent state\n\n**Example**:\n```python\nmerged = synchronizer.merge_and_sync_constellation_states(\n orchestrator_constellation=current_constellation\n)\n```\n\n### Configuration Methods\n\n#### set_modification_timeout()\n\nSet the timeout for modifications.\n\n```python\ndef set_modification_timeout(self, timeout: float) -> None\n```\n\n**Parameters**:\n\n| Parameter | Type | Description | Required |\n|-----------|------|-------------|----------|\n| `timeout` | `float` | Timeout in seconds (must be > 0) | Yes |\n\n**Raises**: `ValueError` if timeout ≤ 0\n\n**Example**:\n```python\n# Increase timeout for slow LLM responses\nsynchronizer.set_modification_timeout(1800.0) # 30 minutes\n```\n\n### Query Methods\n\n#### has_pending_modifications()\n\nCheck if any modifications are pending.\n\n```python\ndef has_pending_modifications(self) -> bool\n```\n\n**Returns**: `True` if modifications pending\n\n#### get_pending_count()\n\nGet number of pending modifications.\n\n```python\ndef get_pending_count(self) -> int\n```\n\n#### get_pending_task_ids()\n\nGet list of task IDs with pending modifications.\n\n```python\ndef get_pending_task_ids(self) -> list\n```\n\n#### get_current_constellation()\n\nGet the constellation currently being modified.\n\n```python\ndef get_current_constellation(self) -> Optional[TaskConstellation]\n```\n\n#### get_statistics()\n\nGet synchronization statistics.\n\n```python\ndef get_statistics(self) -> Dict[str, int]\n```\n\n**Returns**:\n```python\n{\n \"total_modifications\": int,\n \"completed_modifications\": int,\n \"timeout_modifications\": int\n}\n```\n\n### Utility Methods\n\n#### clear_pending_modifications()\n\n⚠️ **Emergency use only**: Forcefully clear all pending modifications.\n\n```python\ndef clear_pending_modifications(self) -> None\n```\n\n---\n\n## Common Usage Patterns\n\n### Basic Orchestration\n\n```python\nfrom galaxy.constellation.orchestrator import TaskConstellationOrchestrator\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Setup\ndevice_manager = ConstellationDeviceManager()\norchestrator = TaskConstellationOrchestrator(device_manager)\n\n# Create constellation\nconstellation = TaskConstellation(name=\"MyWorkflow\")\n# ... add tasks and dependencies ...\n\n# Orchestrate\nresults = await orchestrator.orchestrate_constellation(\n constellation,\n assignment_strategy=\"round_robin\"\n)\n\nprint(f\"Status: {results['status']}\")\nprint(f\"Total tasks: {results['total_tasks']}\")\n```\n\n### With Synchronization\n\n```python\nfrom galaxy.session.observers.constellation_sync_observer import (\n ConstellationModificationSynchronizer\n)\nfrom galaxy.core.events import get_event_bus\n\n# Setup orchestrator\norchestrator = TaskConstellationOrchestrator(device_manager)\n\n# Attach synchronizer\nsynchronizer = ConstellationModificationSynchronizer(orchestrator)\norchestrator.set_modification_synchronizer(synchronizer)\n\n# Subscribe to events\nevent_bus = get_event_bus()\nevent_bus.subscribe(synchronizer)\n\n# Orchestrate with automatic synchronization\nresults = await orchestrator.orchestrate_constellation(constellation)\n```\n\nFor details on the synchronization protocol, see [Safe Assignment Locking](safe_assignment_locking.md).\n\n### Custom Event Handling\n\n```python\nfrom galaxy.core.events import IEventObserver, Event, EventType\n\nclass ProgressTracker(IEventObserver):\n async def on_event(self, event: Event):\n if event.event_type == EventType.TASK_COMPLETED:\n print(f\"✓ {event.task_id} completed\")\n elif event.event_type == EventType.TASK_FAILED:\n print(f\"✗ {event.task_id} failed\")\n\n# Subscribe\ntracker = ProgressTracker()\nevent_bus.subscribe(tracker, {\n EventType.TASK_COMPLETED,\n EventType.TASK_FAILED\n})\n\n# Orchestrate with tracking\nresults = await orchestrator.orchestrate_constellation(constellation)\n```\n\nFor more details on event handling, see [Event-Driven Coordination](event_driven_coordination.md).\n\n### Manual Device Assignment\n\n```python\n# Method 1: Pre-assign in tasks\nfor task in constellation.get_all_tasks():\n if \"windows\" in task.description.lower():\n task.target_device_id = \"windows_main\"\n elif \"android\" in task.description.lower():\n task.target_device_id = \"android_device\"\n\n# Method 2: Manual assignment dict\ndevice_assignments = {\n task.task_id: determine_device(task)\n for task in constellation.get_all_tasks()\n}\n\nresults = await orchestrator.orchestrate_constellation(\n constellation,\n device_assignments=device_assignments\n)\n```\n\n## Type Definitions\n\n### TaskConstellation\n\nSee [TaskConstellation documentation](../constellation/task_constellation.md)\n\n### TaskStar\n\nSee [TaskStar documentation](../constellation/task_star.md)\n\n### Event Types\n\n```python\nfrom galaxy.core.events import EventType\n\nEventType.TASK_STARTED # Task execution begins\nEventType.TASK_COMPLETED # Task completes successfully\nEventType.TASK_FAILED # Task fails\nEventType.CONSTELLATION_STARTED # Orchestration begins\nEventType.CONSTELLATION_COMPLETED # All tasks finished\nEventType.CONSTELLATION_FAILED # Orchestration failed\nEventType.CONSTELLATION_MODIFIED # DAG structure updated\n```\n\n## Error Handling\n\n### Common Exceptions\n\n| Exception | Cause | Handling |\n|-----------|-------|----------|\n| `ValueError` | Invalid DAG, missing assignments | Validate before orchestration |\n| `RuntimeError` | Execution error | Check device connectivity |\n| `asyncio.TimeoutError` | Task timeout | Increase task timeout |\n| `asyncio.CancelledError` | Orchestration cancelled | Cleanup resources |\n\n### Example Error Handling\n\n```python\ntry:\n results = await orchestrator.orchestrate_constellation(\n constellation,\n assignment_strategy=\"capability_match\"\n )\nexcept ValueError as e:\n logger.error(f\"Invalid constellation: {e}\")\n # Fix validation errors\nexcept RuntimeError as e:\n logger.error(f\"Execution failed: {e}\")\n # Retry or alert\nexcept asyncio.CancelledError:\n logger.warning(\"Orchestration cancelled\")\n # Cleanup\nfinally:\n # Always cleanup\n await device_manager.disconnect_all()\n```\n\n## Related Documentation\n\n- **[Overview](overview.md)** - System architecture and design\n- **[Event-Driven Coordination](event_driven_coordination.md)** - Event system details\n- **[Asynchronous Scheduling](asynchronous_scheduling.md)** - Execution model\n- **[Safe Assignment Locking](safe_assignment_locking.md)** - Synchronization protocol\n- **[Consistency Guarantees](consistency_guarantees.md)** - Invariants and validation\n- **[Batched Editing](batched_editing.md)** - Efficiency optimizations\n- **[Constellation Manager](constellation_manager.md)** - Resource management\n\n---\n\n## Getting Help\n\nCheck the examples directory for complete code samples or see [GitHub issues](https://github.com/microsoft/UFO/issues) for known problems.\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/asynchronous_scheduling.md",
"content": "# Asynchronous Scheduling\n\n## Overview\n\nAt the core of the Constellation Orchestrator lies a fully **asynchronous scheduling loop** that maximizes parallelism across heterogeneous devices. Unlike traditional schedulers that alternate between discrete planning and execution phases, the orchestrator continuously monitors the evolving DAG to identify ready tasks and dispatches them concurrently.\n\nMost critically, **task execution and constellation editing can proceed concurrently**, allowing the system to adapt in real-time as results stream in while computation continues uninterrupted.\n\nFor more on the DAG structure being scheduled, see the [TaskConstellation documentation](../constellation/task_constellation.md).\n\n\n\n*Illustration of asynchronous scheduling and concurrent constellation editing. Task execution overlaps with DAG modifications, reducing end-to-end latency.*\n\n## Core Scheduling Loop\n\nThe orchestration workflow is driven by a continuous asynchronous loop that coordinates task execution, constellation synchronization, and event handling:\n\n```python\nasync def _run_execution_loop(self, constellation: TaskConstellation) -> None:\n \"\"\"Main execution loop for processing constellation tasks.\"\"\"\n \n while not constellation.is_complete():\n # 1. Wait for pending modifications and refresh constellation\n constellation = await self._sync_constellation_modifications(constellation)\n \n # 2. Validate device assignments\n self._validate_existing_device_assignments(constellation)\n \n # 3. Get ready tasks and schedule them\n ready_tasks = constellation.get_ready_tasks()\n await self._schedule_ready_tasks(ready_tasks, constellation)\n \n # 4. Wait for task completion\n await self._wait_for_task_completion()\n \n # Wait for all remaining tasks\n await self._wait_for_all_tasks()\n```\n\nThis loop embodies several key design principles:\n\n### 1. Continuous Monitoring\n\nThe loop runs continuously until all tasks reach terminal states (`COMPLETED`, `FAILED`, or `CANCELLED`). Each iteration:\n\n- Checks for constellation modifications from the agent\n- Identifies newly ready tasks (dependencies satisfied)\n- Dispatches tasks to devices\n- Waits for at least one task to complete before repeating\n\n### 2. Non-Blocking Execution\n\nAll operations use `async/await` to avoid blocking:\n\n```python\n# Schedule tasks without waiting for completion\nawait self._schedule_ready_tasks(ready_tasks, constellation)\n\n# Wait for ANY task to complete (not all)\nawait self._wait_for_task_completion()\n```\n\nThis enables maximum concurrency - new tasks can be scheduled while others are still executing.\n\n### 3. Dynamic Adaptation\n\nThe constellation can be modified during execution:\n\n```python\n# Synchronization point: merge agent's edits with runtime progress\nconstellation = await self._sync_constellation_modifications(constellation)\n```\n\nAfter synchronization, the orchestrator immediately identifies and schedules newly ready tasks based on the updated DAG structure.\n\nThe orchestrator treats the TaskConstellation as a **living data structure** that evolves during execution, not a static plan fixed at the start.\n\n## Task Scheduling Mechanism\n\n### Ready Task Identification\n\nTasks become \"ready\" when all their dependencies are satisfied:\n\n```python\nready_tasks = constellation.get_ready_tasks()\n```\n\nThe `TaskConstellation` determines readiness by checking:\n\n1. **Status**: Task must be in `PENDING` state\n2. **Dependencies**: All prerequisite tasks must be completed\n3. **Conditions**: Any conditional dependencies must evaluate to `True`\n\n**Implementation in TaskConstellation:**\n\n```python\ndef get_ready_tasks(self) -> List[TaskStar]:\n \"\"\"Get all tasks ready to execute.\"\"\"\n ready_tasks = []\n for task in self._tasks.values():\n if task.is_ready_to_execute:\n # Double-check dependencies satisfied\n if self._are_dependencies_satisfied(task.task_id):\n ready_tasks.append(task)\n \n # Sort by priority (higher first)\n ready_tasks.sort(key=lambda t: t.priority.value, reverse=True)\n return ready_tasks\n```\n\n!!!tip \"Priority Scheduling\"\n Ready tasks are sorted by priority before dispatching, ensuring critical tasks execute first when multiple tasks are ready simultaneously.\n\n### Asynchronous Task Dispatch\n\nOnce ready tasks are identified, they're dispatched concurrently:\n\n```python\nasync def _schedule_ready_tasks(\n self, ready_tasks: List[TaskStar], constellation: TaskConstellation\n) -> None:\n \"\"\"Schedule ready tasks for execution.\"\"\"\n \n for task in ready_tasks:\n if task.task_id not in self._execution_tasks:\n # Create async task (non-blocking)\n task_future = asyncio.create_task(\n self._execute_task_with_events(task, constellation)\n )\n self._execution_tasks[task.task_id] = task_future\n```\n\n**Key aspects:**\n\n- **Non-blocking dispatch**: `asyncio.create_task()` schedules the task without waiting\n- **Deduplication**: Only schedule if not already in `_execution_tasks` dict\n- **Tracking**: Store task futures for later completion detection\n\n### Task Execution Lifecycle\n\nEach task executes within its own coroutine that encapsulates the full lifecycle:\n\n```mermaid\nstateDiagram-v2\n [*] --> PENDING\n PENDING --> RUNNING: start_execution()\n RUNNING --> COMPLETED: Success\n RUNNING --> FAILED: Error\n COMPLETED --> [*]: Publish event\n FAILED --> [*]: Publish event\n \n note right of RUNNING\n Task executes on device\n via device_manager\n end note\n \n note right of COMPLETED\n Mark in constellation\n Identify newly ready tasks\n Publish TASK_COMPLETED\n end note\n```\n\n**Execution implementation:**\n\n```python\nasync def _execute_task_with_events(\n self, task: TaskStar, constellation: TaskConstellation\n) -> None:\n \"\"\"Execute a single task and publish events.\"\"\"\n \n try:\n # Publish TASK_STARTED event\n start_event = TaskEvent(\n event_type=EventType.TASK_STARTED,\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\"constellation_id\": constellation.constellation_id},\n task_id=task.task_id,\n status=TaskStatus.RUNNING.value,\n )\n await self._event_bus.publish_event(start_event)\n \n # Mark task as started\n task.start_execution()\n \n # Execute on device\n result = await task.execute(self._device_manager)\n \n is_success = result.status == TaskStatus.COMPLETED.value\n \n # Mark task as completed in constellation\n newly_ready = constellation.mark_task_completed(\n task.task_id, success=is_success, result=result\n )\n \n # Publish TASK_COMPLETED or TASK_FAILED event\n completed_event = TaskEvent(\n event_type=(\n EventType.TASK_COMPLETED if is_success \n else EventType.TASK_FAILED\n ),\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\n \"constellation_id\": constellation.constellation_id,\n \"newly_ready_tasks\": [t.task_id for t in newly_ready],\n \"constellation\": constellation,\n },\n task_id=task.task_id,\n status=result.status,\n result=result,\n )\n await self._event_bus.publish_event(completed_event)\n \n except Exception as e:\n # Handle failure (mark task failed, publish event)\n newly_ready = constellation.mark_task_completed(\n task.task_id, success=False, error=e\n )\n \n failed_event = TaskEvent(\n event_type=EventType.TASK_FAILED,\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\n \"constellation_id\": constellation.constellation_id,\n \"newly_ready_tasks\": [t.task_id for t in newly_ready],\n },\n task_id=task.task_id,\n status=TaskStatus.FAILED.value,\n error=e,\n )\n await self._event_bus.publish_event(failed_event)\n raise\n```\n\n## Concurrent Execution Model\n\n### Parallel Task Execution\n\nMultiple tasks execute concurrently across devices:\n\n```python\n# Track active execution tasks\nself._execution_tasks: Dict[str, asyncio.Task] = {}\n\n# Schedule multiple ready tasks at once\nfor task in ready_tasks:\n task_future = asyncio.create_task(\n self._execute_task_with_events(task, constellation)\n )\n self._execution_tasks[task.task_id] = task_future\n```\n\n**Concurrency characteristics:**\n\n| Aspect | Behavior | Benefit |\n|--------|----------|---------|\n| **Device parallelism** | Independent devices execute tasks simultaneously | Maximize resource utilization |\n| **Dependency-based** | Only independent tasks (no dependency path) run concurrently | Maintain correctness |\n| **Heterogeneous** | Different device types (Windows, Android, iOS, etc.) in parallel | Cross-platform orchestration |\n| **Unbounded** | No artificial limit on concurrent tasks | Scale with available devices |\n\n### Completion Detection\n\nThe orchestrator waits for at least one task to complete before continuing:\n\n```python\nasync def _wait_for_task_completion(self) -> None:\n \"\"\"Wait for at least one task to complete and clean up.\"\"\"\n \n if self._execution_tasks:\n # Wait for first completion\n done, _ = await asyncio.wait(\n self._execution_tasks.values(), \n return_when=asyncio.FIRST_COMPLETED\n )\n \n # Clean up completed tasks\n await self._cleanup_completed_tasks(done)\n else:\n # No running tasks, wait briefly\n await asyncio.sleep(0.1)\n```\n\n**Why wait for first completion?**\n\n1. **Responsiveness**: React immediately to any task completion\n2. **Event publishing**: Trigger constellation modifications as soon as possible\n3. **Resource efficiency**: Avoid busy-waiting when no tasks are running\n4. **Fairness**: Give equal opportunity for any task to trigger next iteration\n\n### Task Cleanup\n\nCompleted tasks are removed from tracking:\n\n```python\nasync def _cleanup_completed_tasks(self, done_futures: set) -> None:\n \"\"\"Clean up completed task futures from tracking.\"\"\"\n \n completed_task_ids = []\n for task_future in done_futures:\n for task_id, future in self._execution_tasks.items():\n if future == task_future:\n completed_task_ids.append(task_id)\n break\n \n for task_id in completed_task_ids:\n del self._execution_tasks[task_id]\n```\n\nThis prevents memory leaks and ensures `_execution_tasks` reflects only actively running tasks.\n\n## Concurrent Constellation Editing\n\n### The Challenge\n\nTraditional schedulers treat DAG structure as **immutable** during execution. But in UFO, the LLM-based Constellation Agent can modify the DAG based on task results:\n\n- Add new tasks when decomposition is needed\n- Remove unnecessary tasks when shortcuts are found\n- Modify dependencies when task relationships change\n- Update task descriptions or parameters\n\nThis creates a **race condition**: tasks may be executing while the agent modifies the constellation.\n\n### The Solution: Overlapping Execution and Editing\n\nThe orchestrator allows task execution and constellation editing to **proceed concurrently**:\n\n```mermaid\ngantt\n title Concurrent Execution and Editing Timeline\n dateFormat X\n axisFormat %L\n \n section Tasks\n Task A executes :a1, 0, 100\n Task B executes :b1, 50, 150\n Task C executes :c1, 100, 200\n \n section Editing\n Edit on A completion :e1, 100, 130\n Edit on B completion :e2, 150, 180\n \n section Sync\n Sync after Edit A :s1, 130, 135\n Sync after Edit B :s2, 180, 185\n```\n\nIn the diagram:\n\n- **Task A** completes at t=100, triggering an edit\n- **Task B** continues executing during the edit (100-130)\n- Edit completes and syncs at t=135\n- **Task C** starts at t=135 based on updated constellation\n- **Task B** completes at t=150, triggering another edit\n- **Task C** continues executing during this second edit\n\nBy overlapping execution and editing, end-to-end latency is reduced by up to 30% compared to sequential edit-then-execute approaches.\n\n### Synchronization Points\n\nThe orchestrator synchronizes constellation state at the start of each scheduling iteration:\n\n```python\nasync def _sync_constellation_modifications(\n self, constellation: TaskConstellation\n) -> TaskConstellation:\n \"\"\"Synchronize pending constellation modifications.\"\"\"\n \n if self._modification_synchronizer:\n # Wait for agent to finish any pending edits\n await self._modification_synchronizer.wait_for_pending_modifications()\n \n # Merge agent's structural changes with orchestrator's execution state\n constellation = self._modification_synchronizer \\\n .merge_and_sync_constellation_states(\n orchestrator_constellation=constellation,\n )\n \n return constellation\n```\n\n**What gets synchronized:**\n\n1. **Structural changes** from agent (new tasks, dependencies, modifications)\n2. **Execution state** from orchestrator (task statuses, results, errors)\n3. **Consistency validation** (check invariants I1-I3)\n\nThe `merge_and_sync_constellation_states` method ensures:\n\n- Agent's constellation has latest structural modifications\n- Orchestrator's execution progress is preserved\n- More advanced task states (e.g., COMPLETED) take precedence over stale states (e.g., RUNNING)\n\n[Learn more about synchronization →](safe_assignment_locking.md#constellation-state-merging)\n\n## Performance Optimizations\n\n### 1. Lazy Evaluation\n\nReady tasks are computed only when needed:\n\n```python\n# Only compute when scheduling\nready_tasks = constellation.get_ready_tasks()\n```\n\nAvoids repeated expensive graph traversals when no tasks complete.\n\n### 2. Priority-Based Scheduling\n\nHigher priority tasks execute first:\n\n```python\n# Sort by priority before dispatching\nready_tasks.sort(key=lambda t: t.priority.value, reverse=True)\n```\n\nEnsures critical-path tasks don't wait behind low-priority tasks.\n\n### 3. Incremental Completion Detection\n\nUse `asyncio.wait(..., return_when=FIRST_COMPLETED)` instead of waiting for all:\n\n```python\ndone, pending = await asyncio.wait(\n self._execution_tasks.values(), \n return_when=asyncio.FIRST_COMPLETED\n)\n```\n\nMinimizes latency between task completion and next scheduling iteration.\n\n### 4. Batched Synchronization\n\nModifications are batched during agent editing:\n\n```python\n# Agent may modify multiple tasks before publishing CONSTELLATION_MODIFIED\n# Orchestrator waits once for all modifications\nawait self._modification_synchronizer.wait_for_pending_modifications()\n```\n\nReduces synchronization overhead from O(N) to O(1) per editing cycle.\n\n[Learn more about batching →](batched_editing.md)\n\n## Execution Timeline Example\n\nHere's a concrete example showing how asynchronous scheduling works:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator Loop\n participant C as Constellation\n participant T1 as Task A (Device 1)\n participant T2 as Task B (Device 2)\n participant T3 as Task C (Device 1)\n participant A as Agent\n \n Note over O: Iteration 1\n O->>C: get_ready_tasks()\n C-->>O: [Task A, Task B]\n O->>T1: Schedule Task A (async)\n O->>T2: Schedule Task B (async)\n O->>O: wait_for_task_completion()\n \n Note over T1,T2: Both execute concurrently\n \n T1-->>O: Task A completes\n O->>A: Trigger editing (async)\n \n Note over O: Iteration 2\n O->>O: sync_constellation_modifications()\n \n par Agent editing\n A->>A: Modify constellation\n A-->>O: Publish CONSTELLATION_MODIFIED\n and Task B continues\n Note over T2: Still executing\n end\n \n O->>C: get_ready_tasks()\n C-->>O: [Task C]\n O->>T3: Schedule Task C (async)\n O->>O: wait_for_task_completion()\n \n T2-->>O: Task B completes\n \n Note over O: Iteration 3\n O->>O: sync_constellation_modifications()\n O->>C: get_ready_tasks()\n C-->>O: []\n \n T3-->>O: Task C completes\n \n Note over O: Constellation complete\n```\n\n**Key observations:**\n\n1. **Iteration 1**: Tasks A and B scheduled concurrently\n2. **Concurrent editing**: Agent modifies constellation while Task B executes\n3. **Iteration 2**: Task C scheduled immediately after sync, Task B still running\n4. **No blocking**: Orchestrator never waits idle; always scheduling or executing\n\n## Error Handling\n\n### Task Failure\n\nWhen a task fails, the orchestrator:\n\n1. Publishes `TASK_FAILED` event\n2. Marks task as failed in constellation\n3. Identifies newly ready tasks (if any dependencies allow failure)\n4. Continues scheduling remaining tasks\n\n```python\nexcept Exception as e:\n newly_ready = constellation.mark_task_completed(\n task.task_id, success=False, error=e\n )\n \n failed_event = TaskEvent(\n event_type=EventType.TASK_FAILED,\n ...\n error=e,\n )\n await self._event_bus.publish_event(failed_event)\n```\n\n### Cancellation\n\nIf orchestration is cancelled:\n\n```python\nexcept asyncio.CancelledError:\n if self._logger:\n self._logger.info(\n f\"Orchestration cancelled for constellation {constellation.constellation_id}\"\n )\n raise\n```\n\nAll running tasks are automatically cancelled via `asyncio` cancellation propagation.\n\n### Cleanup\n\nCleanup always happens, even on error:\n\n```python\nfinally:\n await self._cleanup_constellation(constellation)\n```\n\n## Usage Patterns\n\n### Basic Orchestration\n\n```python\norchestrator = TaskConstellationOrchestrator(device_manager)\n\nresults = await orchestrator.orchestrate_constellation(\n constellation=my_constellation,\n assignment_strategy=\"round_robin\"\n)\n```\n\n### With Custom Event Handlers\n\n```python\nclass ProgressTracker(IEventObserver):\n async def on_event(self, event: Event):\n if event.event_type == EventType.TASK_COMPLETED:\n print(f\"✓ Task {event.task_id} completed\")\n\nevent_bus.subscribe(ProgressTracker())\n\nresults = await orchestrator.orchestrate_constellation(constellation)\n```\n\n### With Modification Synchronizer\n\n```python\nsynchronizer = ConstellationModificationSynchronizer(orchestrator)\norchestrator.set_modification_synchronizer(synchronizer)\nevent_bus.subscribe(synchronizer)\n\n# Now edits are synchronized automatically\nresults = await orchestrator.orchestrate_constellation(constellation)\n```\n\n## Performance Characteristics\n\n| Metric | Typical Value | Notes |\n|--------|--------------|-------|\n| **Scheduling latency** | < 10ms | Time from task ready to dispatch |\n| **Completion detection** | < 5ms | Time from task done to next iteration |\n| **Sync overhead** | 10-50ms | Per constellation modification |\n| **Max concurrent tasks** | Limited by devices | No artificial orchestrator limit |\n| **Throughput** | 10-100 tasks/sec | Depends on task duration |\n\n*Performance measured on: Intel i7, 16GB RAM, 5 connected devices, tasks averaging 2-5 seconds each*\n\n## Related Documentation\n\n- **[Event-Driven Coordination](event_driven_coordination.md)** - Event system enabling async scheduling\n- **[Safe Assignment Locking](safe_assignment_locking.md)** - How editing synchronizes with execution\n- **[Consistency Guarantees](consistency_guarantees.md)** - Invariants preserved during async execution\n- **[API Reference](api_reference.md)** - Orchestrator API details\n\n---\n\n!!!tip \"Next Steps\"\n To understand how concurrent editing is made safe, continue to [Safe Assignment Locking](safe_assignment_locking.md).\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/batched_editing.md",
"content": "# Batched Constellation Editing\n\n## Overview\n\nFrequent LLM-driven edits can introduce significant overhead if processed individually. Each modification requires:\n\n- LLM invocation (100-1000ms latency)\n- Lock acquisition and release\n- Validation of invariants I1-I3\n- Constellation state synchronization\n- Event publishing and notification\n\nTo balance **responsiveness** with **efficiency**, the orchestrator supports **batched constellation editing**: during a reasoning round, multiple task completion events are aggregated and their resulting modifications applied atomically in a single cycle.\n\nFor more on the synchronization mechanism, see [Safe Assignment Locking](safe_assignment_locking.md).\n\n## The Batching Problem\n\n### Without Batching\n\nConsider three tasks completing nearly simultaneously:\n\n```mermaid\ngantt\n title Sequential Editing (No Batching)\n dateFormat X\n axisFormat %L\n \n section Events\n Task A completes :e1, 0, 5\n Task B completes :e2, 10, 15\n Task C completes :e3, 20, 25\n \n section Editing\n Lock + Edit A :l1, 5, 155\n Lock + Edit B :l2, 155, 305\n Lock + Edit C :l3, 305, 455\n \n section Overhead\n Total overhead :o1, 5, 455\n```\n\n**Overhead**: 3 × 150ms = **450ms total**\n\n- 3 lock acquisitions\n- 3 LLM invocations\n- 3 validations\n- 3 synchronizations\n\n### With Batching\n\nSame scenario with batched editing:\n\n```mermaid\ngantt\n title Batched Editing\n dateFormat X\n axisFormat %L\n \n section Events\n Task A completes :e1, 0, 5\n Task B completes :e2, 10, 15\n Task C completes :e3, 20, 25\n \n section Editing\n Lock + Edit A,B,C :l1, 25, 175\n \n section Overhead\n Total overhead :o1, 25, 175\n```\n\n**Overhead**: 1 × 150ms = **150ms total**\n\n- 1 lock acquisition\n- 1 LLM invocation (potentially processing multiple tasks)\n- 1 validation\n- 1 synchronization\n\n**Improvement**: **3× reduction** in overhead!\n\nBatching reduces orchestration overhead from O(N) to O(1) per reasoning round, where N = number of completed tasks.\n\n## Batching Mechanism\n\n### Event Queuing\n\nWhen tasks complete, their IDs are queued for batch processing:\n\n```python\n# In safe assignment lock algorithm\nwhile system is running:\n foreach event e ∈ E do\n if e is TASK_COMPLETED or TASK_FAILED then\n async enqueue(e) # ← Queue instead of immediate processing\n end\n end\n \n acquire(assign_lock)\n \n # Process ALL queued events in one batch\n while queue not empty do\n e ← dequeue()\n Δ ← invoke(ConstellationAgent, edit(C, e))\n C ← apply(C, Δ)\n end\n \n validate(C) # ← Single validation for entire batch\n publish(CONSTELLATION_MODIFIED, all_task_ids)\n C ← synchronize(C, T_C)\n \n release(assign_lock)\n```\n\n### Implementation in Synchronizer\n\nThe `ConstellationModificationSynchronizer` batches pending modifications:\n\n```python\nasync def wait_for_pending_modifications(\n self, timeout: Optional[float] = None\n) -> bool:\n \"\"\"Wait for all pending modifications to complete.\"\"\"\n \n if not self._pending_modifications:\n return True\n \n try:\n while self._pending_modifications:\n # Get current pending tasks (snapshot)\n pending_tasks = list(self._pending_modifications.keys())\n pending_futures = list(self._pending_modifications.values())\n \n self.logger.info(\n f\"⏳ Waiting for {len(pending_tasks)} pending modification(s): \"\n f\"{pending_tasks}\"\n )\n \n # Wait for ALL current pending modifications (batching)\n await asyncio.wait_for(\n asyncio.gather(*pending_futures, return_exceptions=True),\n timeout=remaining_timeout,\n )\n \n # Check if new modifications were added during wait\n if not self._pending_modifications:\n break\n \n # Small delay to allow new registrations to settle\n await asyncio.sleep(0.01)\n \n self.logger.info(\"✅ All pending modifications completed\")\n return True\n \n except asyncio.TimeoutError:\n ...\n```\n\n**Key aspects:**\n\n1. **Snapshot pending modifications** - Capture current batch\n2. **Wait for all in batch** - Use `asyncio.gather()` for parallel completion\n3. **Check for new arrivals** - Handle dynamic additions during wait\n4. **Iterate until empty** - Process all batches\n\n### Agent-Side Batching\n\nThe Constellation Agent receives multiple task IDs and processes them together:\n\n```python\nasync def process_editing(\n self,\n context: Context = None,\n task_ids: Optional[List[str]] = None, # ← Multiple task IDs\n before_constellation: Optional[TaskConstellation] = None,\n) -> TaskConstellation:\n \"\"\"Process task completion events and update constellation.\"\"\"\n \n task_ids = task_ids or []\n \n # Agent can see multiple completed tasks at once\n self.logger.debug(\n f\"Tasks {task_ids} marked as completed, processing modifications...\"\n )\n \n # Potentially make decisions based on multiple task outcomes\n # e.g., \"Task A and B both succeeded, skip Task C\"\n after_constellation = await self._create_and_process(context)\n \n # Publish single CONSTELLATION_MODIFIED event for entire batch\n await self._publish_constellation_modified_event(\n before_constellation,\n after_constellation,\n task_ids, # ← All modified tasks\n self._create_timing_info(start_time, end_time, duration),\n )\n \n return after_constellation\n```\n\n## Batching Timeline Example\n\nHere's a detailed timeline showing how batching works:\n\n```\nt=100ms: Task A completes\n → Synchronizer registers pending modification for A\n → Task A's event added to queue\n\nt=150ms: Task B completes (during A's queueing)\n → Synchronizer registers pending modification for B\n → Task B's event added to queue\n\nt=200ms: Task C completes\n → Synchronizer registers pending modification for C\n → Task C's event added to queue\n\nt=205ms: Orchestrator reaches synchronization point\n → Calls wait_for_pending_modifications()\n → Sees pending: [A, B, C]\n → Waits for all three futures\n\nt=210ms: Agent starts processing (lock acquired)\n → Receives task_ids = ['A', 'B', 'C']\n → Makes unified editing decision\n\nt=350ms: Agent completes editing\n → Publishes CONSTELLATION_MODIFIED with on_task_id = ['A', 'B', 'C']\n\nt=355ms: Synchronizer receives event\n → Completes futures for A, B, C\n → wait_for_pending_modifications() returns\n\nt=360ms: Orchestrator merges states and continues\n → Single validation\n → Single synchronization\n → Resume scheduling\n```\n\n**Total overhead**: 360ms - 100ms = **260ms** for 3 tasks\n\nCompare to sequential: 3 × 150ms = **450ms** (ignoring event queueing)\n\n## Efficiency Analysis\n\n### Overhead Breakdown\n\nPer-task overhead without batching:\n\n| Operation | Cost (ms) | Frequency |\n|-----------|-----------|-----------|\n| Lock acquisition | 1-2 | Per task |\n| LLM invocation | 100-1000 | Per task |\n| Validation (I1-I3) | 5-10 | Per task |\n| State synchronization | 10-20 | Per task |\n| Event publishing | 1-2 | Per task |\n| **Total** | **117-1034** | **Per task** |\n\nPer-batch overhead with batching:\n\n| Operation | Cost (ms) | Frequency |\n|-----------|-----------|-----------|\n| Lock acquisition | 1-2 | Per batch |\n| LLM invocation | 100-1000 | Per batch |\n| Validation (I1-I3) | 5-10 | Per batch |\n| State synchronization | 10-20 | Per batch |\n| Event publishing | 1-2 | Per batch |\n| **Total** | **117-1034** | **Per batch** |\n\n**Savings with batch size N**: (N - 1) × overhead\n\n!!!example \"Concrete Example\"\n With N=5 tasks completing simultaneously and 200ms average overhead:\n \n - **Without batching**: 5 × 200ms = 1000ms\n - **With batching**: 1 × 200ms = 200ms\n - **Savings**: 800ms (80% reduction)\n\n### Throughput Improvement\n\nBatching improves task throughput:\n\n$$\\text{Throughput}_{\\text{batched}} = \\frac{N \\times \\text{Throughput}_{\\text{unbatched}}}{1 + (N-1) \\times \\frac{\\text{overhead}}{\\text{task\\_duration}}}$$\n\nFor tasks averaging 5 seconds with 200ms overhead:\n\n- N=1: 0.20 tasks/sec\n- N=3: 0.55 tasks/sec (**2.75× improvement**)\n- N=5: 0.83 tasks/sec (**4.15× improvement**)\n- N=10: 1.35 tasks/sec (**6.75× improvement**)\n\n### Latency Trade-Off\n\nBatching may slightly increase latency for individual tasks:\n\n- **Best case**: Task completes, is first in batch → minimal additional latency\n- **Average case**: Task waits for 1-2 other tasks to complete → ~50-200ms additional latency\n- **Worst case**: Task waits for full batch to accumulate → ~500ms additional latency\n\n**Acceptable trade-off** for significantly improved overall throughput.\n\n## Dynamic Batch Size\n\nThe orchestrator uses **dynamic batching** - batch size adapts to task completion patterns:\n\n### Natural Batching\n\nTasks completing within a short window are naturally batched:\n\n```python\n# In wait_for_pending_modifications()\nwhile self._pending_modifications:\n # Snapshot current pending tasks\n pending_tasks = list(self._pending_modifications.keys())\n \n # Wait for all of them\n await asyncio.gather(*pending_futures)\n \n # Check for new arrivals during processing\n if not self._pending_modifications:\n break\n \n # If new tasks arrived, include them in next iteration\n```\n\n**Batch size**: Determined by task completion timing, not fixed parameter\n\n### Adaptive Grouping\n\nThe synchronizer automatically groups tasks:\n\n- **Slow periods**: Small batches (1-2 tasks)\n- **Burst periods**: Large batches (5-10+ tasks)\n- **Mixed patterns**: Variable batch sizes\n\nThis provides **optimal efficiency** without manual tuning.\n\n## Atomicity of Batched Edits\n\n### Single Edit Cycle\n\nAll modifications in a batch are applied in a single atomic edit cycle:\n\n```python\nacquire(assign_lock)\n\n# Apply all modifications together\nforeach event in batch:\n Δ ← invoke(ConstellationAgent, edit(C, event))\n C ← apply(C, Δ)\nend\n\nvalidate(C) # ← Validates combined result\npublish(CONSTELLATION_MODIFIED, batch_task_ids)\n\nrelease(assign_lock)\n```\n\n**Atomicity guarantee**: Either all modifications in the batch are applied, or none are.\n\n### Confluence Property\n\nThe paper proves an **Edit-Sync Confluence Lemma**:\n\n**Lemma**: Folding runtime events commutes with lock-bounded edits within the same window.\n\n**Formally**: Given events $e_1, e_2, \\ldots, e_n$ arriving within a lock window:\n\n$$\\text{apply}(C, \\Delta_{e_1} \\circ \\Delta_{e_2} \\circ \\cdots \\circ \\Delta_{e_n}) \\equiv \\text{apply}(\\cdots\\text{apply}(\\text{apply}(C, \\Delta_{e_1}), \\Delta_{e_2})\\cdots, \\Delta_{e_n})$$\n\nBatched application produces the same result as sequential application.\n\n**Proof sketch**: \n\n1. Each $\\Delta_i$ is a pure function of $C$ and $e_i$\n2. Lock ensures no intermediate states are visible\n3. Validation enforces invariants on final state\n4. Synchronization merges all runtime progress atomically\n\n[See Appendix A.4 in paper for complete proof]\n\nBatching is a pure **performance optimization** - it doesn't change the semantics of constellation evolution.\n\n## Implementation Patterns\n\n### Enabling Batching\n\nBatching is enabled automatically when using the synchronizer:\n\n```python\nfrom galaxy.session.observers.constellation_sync_observer import (\n ConstellationModificationSynchronizer\n)\n\n# Create and attach synchronizer\nsynchronizer = ConstellationModificationSynchronizer(orchestrator)\norchestrator.set_modification_synchronizer(synchronizer)\n\n# Subscribe to events\nevent_bus.subscribe(synchronizer)\n\n# Batching happens automatically\nresults = await orchestrator.orchestrate_constellation(constellation)\n```\n\n### Monitoring Batch Sizes\n\nTrack batching statistics:\n\n```python\n# After orchestration\nstats = synchronizer.get_statistics()\n\nprint(f\"Total modifications: {stats['total_modifications']}\")\nprint(f\"Completed: {stats['completed_modifications']}\")\n\n# Infer average batch size\navg_batch_size = stats['total_modifications'] / number_of_edit_cycles\nprint(f\"Average batch size: {avg_batch_size:.2f}\")\n```\n\n### Tuning Batch Timeout\n\nAdjust timeout for slower LLM responses:\n\n```python\n# Increase timeout for complex reasoning\nsynchronizer.set_modification_timeout(1800.0) # 30 minutes\n\n# Or decrease for simple tasks\nsynchronizer.set_modification_timeout(120.0) # 2 minutes\n```\n\n## Performance Best Practices\n\n### 1. Group Related Tasks\n\nDesign constellations with tasks that complete around the same time:\n\n```python\n# Good: Tasks with similar durations\nTask A: 5 seconds\nTask B: 6 seconds # ← Likely completes near Task A\nTask C: 5 seconds # ← Likely completes near Task A\n\n# Bad: Widely varying durations\nTask X: 1 second\nTask Y: 30 seconds # ← Won't batch with X\nTask Z: 2 seconds # ← Won't batch with Y\n```\n\n### 2. Minimize LLM Overhead\n\nReduce individual modification latency:\n\n- Use efficient prompts\n- Cache common editing patterns\n- Pre-compute possible modifications\n\n### 3. Balance Batch Size\n\nToo small: Frequent overhead\nToo large: Increased latency\n\n**Sweet spot**: 3-7 tasks per batch for most workloads\n\n### 4. Monitor and Adjust\n\nTrack metrics:\n\n```python\nclass BatchMetricsObserver(IEventObserver):\n def __init__(self):\n self.batch_sizes = []\n \n async def on_event(self, event: Event):\n if event.event_type == EventType.CONSTELLATION_MODIFIED:\n task_ids = event.data.get(\"on_task_id\", [])\n batch_size = len(task_ids)\n self.batch_sizes.append(batch_size)\n \n if batch_size > 1:\n print(f\"✓ Batched {batch_size} modifications\")\n```\n\n## Comparison with Alternatives\n\n### Micro-Batching\n\n**Alternative**: Fixed small batches (e.g., always wait for 2-3 tasks)\n\n**Drawback**: \n- Adds artificial delay even when single task completes\n- May miss larger natural batches\n\n**UFO's approach**: Dynamic batching with no artificial delays\n\n### Window-Based Batching\n\n**Alternative**: Fixed time window (e.g., batch every 1 second)\n\n**Drawback**:\n- Adds latency even when editing is fast\n- May split natural batches across windows\n\n**UFO's approach**: Event-driven batching without fixed windows\n\n### No Batching\n\n**Alternative**: Process each modification immediately\n\n**Drawback**:\n- High overhead for concurrent completions\n- Redundant LLM invocations\n\n**UFO's approach**: Automatic batching when beneficial\n\n| Approach | Latency | Throughput | Complexity |\n|----------|---------|------------|------------|\n| **No batching** | Low (best) | Low | Low |\n| **Fixed window** | Medium | Medium | Medium |\n| **Fixed size** | High | Medium | Medium |\n| **Dynamic (UFO)** | Low-Medium | High (best) | Low |\n\n## Related Documentation\n\n- **[Safe Assignment Locking](safe_assignment_locking.md)** - How batching integrates with locking\n- **[Asynchronous Scheduling](asynchronous_scheduling.md)** - Concurrent execution enabling batching\n- **[Event-Driven Coordination](event_driven_coordination.md)** - Event system for batching\n\n---\n\n!!!tip \"Next Steps\"\n To understand device assignment and resource management, continue to [Constellation Manager](constellation_manager.md).\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/consistency_guarantees.md",
"content": "# Consistency and Safety Guarantees\n\n## Overview\n\nSince the TaskConstellation may be dynamically rewritten by an LLM-based agent, the orchestrator must enforce runtime invariants to preserve correctness even under partial or invalid updates. Without these guarantees, the system could execute invalid DAGs, violate dependencies, or enter inconsistent states.\n\nThe Constellation Orchestrator enforces three critical invariants (I1-I3) that together ensure safety, consistency, and semantic validity throughout execution.\n\n## The Three Invariants\n\n### I1: Single Assignment\n\n**Invariant**: Each TaskStar has at most one active device assignment at any time.\n\n**Rationale**: A task cannot execute on multiple devices simultaneously - this would lead to duplicate execution, wasted resources, inconsistent results, and ambiguous state (which device's result is authoritative?).\n\n**Enforcement**:\n\n```python\n# In TaskStar\n@property\ndef target_device_id(self) -> Optional[str]:\n \"\"\"Get the target device ID.\"\"\"\n return self._target_device_id\n\n@target_device_id.setter\ndef target_device_id(self, value: Optional[str]) -> None:\n \"\"\"Set the target device ID.\"\"\"\n if self._status == TaskStatus.RUNNING:\n raise ValueError(\n f\"Cannot modify device assignment of running task {self._task_id}\"\n )\n self._target_device_id = value\n```\n\n**Validation**:\n\n```python\ndef validate_constellation_assignments(\n self, constellation: TaskConstellation\n) -> tuple[bool, List[str]]:\n \"\"\"Validate that all tasks have valid device assignments.\"\"\"\n \n errors = []\n for task_id, task in constellation.tasks.items():\n if not task.target_device_id:\n errors.append(f\"Task '{task_id}' has no device assignment\")\n \n return len(errors) == 0, errors\n```\n\n**Warning:** The setter explicitly prevents reassignment of running tasks, ensuring I1 cannot be violated during execution.\n\n### I2: Acyclic Consistency\n\n**Invariant**: Edits must preserve DAG acyclicity - the constellation remains a valid directed acyclic graph after all modifications.\n\n**Rationale**: Cycles in the task graph create deadlocks (tasks wait for each other indefinitely), undefined execution order, and inability to determine ready tasks.\n\n**Enforcement**:\n\n```python\ndef add_dependency(self, dependency: TaskStarLine) -> None:\n \"\"\"Add a dependency to the constellation.\"\"\"\n \n # Validate tasks exist\n if dependency.from_task_id not in self._tasks:\n raise ValueError(f\"Source task {dependency.from_task_id} not found\")\n if dependency.to_task_id not in self._tasks:\n raise ValueError(f\"Target task {dependency.to_task_id} not found\")\n \n # Check for cycle BEFORE adding\n if self._would_create_cycle(dependency.from_task_id, dependency.to_task_id):\n raise ValueError(\n f\"Adding dependency {dependency.from_task_id} -> {dependency.to_task_id} would create a cycle\"\n )\n \n # Safe to add\n self._dependencies[dependency.line_id] = dependency\n```\n\n**Cycle Detection Algorithm**:\n\n```python\ndef _would_create_cycle(self, from_task_id: str, to_task_id: str) -> bool:\n \"\"\"Check if adding a dependency would create a cycle.\"\"\"\n \n # Use DFS to check if path exists from to_task_id to from_task_id\n visited = set()\n \n def has_path(current: str, target: str) -> bool:\n if current == target:\n return True\n if current in visited:\n return False\n \n visited.add(current)\n \n # Check all dependencies where current is the source\n for dependency in self._dependencies.values():\n if dependency.from_task_id == current:\n if has_path(dependency.to_task_id, target):\n return True\n \n return False\n \n # If path exists from to_task → from_task, adding from_task → to_task creates cycle\n return has_path(to_task_id, from_task_id)\n```\n\n**Validation**:\n\n```python\ndef validate_dag(self) -> Tuple[bool, List[str]]:\n \"\"\"Validate the DAG structure.\"\"\"\n \n errors = []\n \n # Check for cycles\n if self.has_cycle():\n errors.append(\"DAG contains cycles\")\n \n # Check for invalid dependencies\n for dependency in self._dependencies.values():\n if dependency.from_task_id not in self._tasks:\n errors.append(\n f\"Dependency references non-existent source task \"\n f\"{dependency.from_task_id}\"\n )\n if dependency.to_task_id not in self._tasks:\n errors.append(\n f\"Dependency references non-existent target task \"\n f\"{dependency.to_task_id}\"\n )\n \n return len(errors) == 0, errors\n```\n\nThe orchestrator uses `get_topological_order()` which raises `ValueError` if cycles exist, providing an additional check.\n\n### I3: Valid Update\n\n**Invariant**: Only `PENDING` and `WAITING_DEPENDENCY` tasks may be modified; `RUNNING`, `COMPLETED`, and `FAILED` tasks are immutable.\n\n**Rationale**: Modifying tasks that have started or finished execution could invalidate already-collected results, create inconsistencies between device state and constellation state, or violate causal dependencies.\n\n**Enforcement**:\n\n```python\ndef get_modifiable_tasks(self) -> List[TaskStar]:\n \"\"\"Get all tasks that can be modified.\"\"\"\n \n modifiable_statuses = {TaskStatus.PENDING, TaskStatus.WAITING_DEPENDENCY}\n return [\n task for task in self._tasks.values() \n if task.status in modifiable_statuses\n ]\n\ndef is_task_modifiable(self, task_id: str) -> bool:\n \"\"\"Check if a specific task can be modified.\"\"\"\n \n task = self._tasks.get(task_id)\n if not task:\n return False\n return task.status in {TaskStatus.PENDING, TaskStatus.WAITING_DEPENDENCY}\n```\n\n**Task-Level Protection**:\n\n```python\n# In TaskStar\n@name.setter\ndef name(self, value: str) -> None:\n if self._status == TaskStatus.RUNNING:\n raise ValueError(f\"Cannot modify name of running task {self._task_id}\")\n self._name = value\n\n@description.setter\ndef description(self, value: str) -> None:\n if self._status == TaskStatus.RUNNING:\n raise ValueError(\n f\"Cannot modify description of running task {self._task_id}\"\n )\n self._description = value\n```\n\n**Dependency Validation**:\n\n```python\ndef get_modifiable_dependencies(self) -> List[TaskStarLine]:\n \"\"\"Get all dependencies that can be modified.\"\"\"\n \n modifiable_deps = []\n modifiable_statuses = {TaskStatus.PENDING, TaskStatus.WAITING_DEPENDENCY}\n \n for dep in self._dependencies.values():\n target_task = self._tasks.get(dep.to_task_id)\n if target_task and target_task.status in modifiable_statuses:\n modifiable_deps.append(dep)\n \n return modifiable_deps\n```\n\nOnce a task starts execution, its core properties (description, dependencies, device assignment) become immutable, ensuring execution integrity.\n\n## Invariant Verification\n\n### Pre-Execution Validation\n\nBefore orchestration begins, the orchestrator validates all invariants:\n\n```python\nasync def _validate_and_prepare_constellation(\n self, constellation: TaskConstellation, \n device_assignments: Optional[Dict[str, str]],\n assignment_strategy: Optional[str] = None,\n) -> None:\n \"\"\"Validate DAG structure and prepare device assignments.\"\"\"\n \n # Validate I2: Acyclic Consistency\n is_valid, errors = constellation.validate_dag()\n if not is_valid:\n raise ValueError(f\"Invalid DAG: {errors}\")\n \n # Handle device assignments\n await self._assign_devices_to_tasks(\n constellation, device_assignments, assignment_strategy\n )\n \n # Validate I1: Single Assignment\n is_valid, errors = self._constellation_manager \\\n .validate_constellation_assignments(constellation)\n if not is_valid:\n raise ValueError(f\"Device assignment validation failed: {errors}\")\n```\n\n### Runtime Validation\n\nDuring execution, the orchestrator validates before each scheduling iteration:\n\n```python\nasync def _run_execution_loop(self, constellation: TaskConstellation) -> None:\n \"\"\"Main execution loop with validation.\"\"\"\n \n while not constellation.is_complete():\n # Sync and validate after modifications\n constellation = await self._sync_constellation_modifications(constellation)\n \n # Validate I1: Single Assignment\n self._validate_existing_device_assignments(constellation)\n \n # I2 checked implicitly by TaskConstellation.add_dependency()\n # I3 checked by TaskStar property setters\n \n # Schedule ready tasks\n ready_tasks = constellation.get_ready_tasks()\n await self._schedule_ready_tasks(ready_tasks, constellation)\n await self._wait_for_task_completion()\n```\n\n### Post-Modification Validation\n\nAfter the agent modifies the constellation, validation occurs before releasing the lock:\n\n```python\n# In safe assignment lock algorithm\nwhile queue not empty:\n e ← dequeue()\n Δ ← invoke(ConstellationAgent, edit(C, e))\n C ← apply(C, Δ)\n validate(C) # ← Verify I1, I2, I3\n publish(CONSTELLATION_MODIFIED, t)\n C ← synchronize(C, T_C)\nend\n```\n\n## Consistency Under Concurrent Modification\n\n### The Challenge\n\nConcurrent task execution and constellation editing create multiple consistency challenges:\n\n| Challenge | Without Invariants | With Invariants |\n|-----------|-------------------|-----------------|\n| **Duplicate execution** | Task assigned to multiple devices | I1 prevents multiple assignments |\n| **Cyclic dependencies** | Deadlocked tasks | I2 prevents cycle introduction |\n| **Stale modifications** | Running task gets edited | I3 prevents editing running tasks |\n| **Lost results** | Completed task gets removed | I3 makes completed tasks immutable |\n\n### Consistency Model\n\nThe orchestrator maintains eventual consistency with strong isolation:\n\n```mermaid\ngraph TD\n A[Task Completes] -->|Event| B[Agent Starts Editing]\n B -->|Lock| C[Modification Phase]\n C -->|Validate I1-I3| D{Valid?}\n D -->|Yes| E[Apply Changes]\n D -->|No| F[Reject Changes]\n E -->|Sync| G[Merge States]\n F -->|Error| H[Log & Continue]\n G -->|Release Lock| I[Orchestrator Sees Update]\n```\n\n**Properties:**\n\n1. **Isolation**: Modifications occur atomically within lock\n2. **Validation**: All changes checked against I1-I3 before commit\n3. **Rejection**: Invalid modifications are discarded with error logging\n4. **Consistency**: Orchestrator only sees valid constellation states\n\nDuring the lock period (modification + validation + sync), the orchestrator has a strongly consistent view of the constellation. Learn more about [safe assignment locking](safe_assignment_locking.md).\n\n## Formal Invariant Definitions\n\n### Mathematical Formulation\n\nLet C = (T, D) be a constellation with tasks T and dependencies D, where:\n- τ ∈ T is a task with status σ(τ) and device assignment δ(τ)\n- d = (t₁ → t₂) ∈ D is a dependency from task t₁ to task t₂\n\n**I1 (Single Assignment)**: Each task has at most one device assignment.\n\n**I2 (Acyclic Consistency)**: No cyclic paths exist in the dependency graph.\n\n**I3 (Valid Update)**: If σ(τ) ∈ {RUNNING, COMPLETED, FAILED} then τ is immutable.\n\n### State Transition Rules\n\nValid state transitions preserve invariants:\n\n```mermaid\nstateDiagram-v2\n [*] --> PENDING: Create task\n PENDING --> WAITING_DEPENDENCY: Has dependencies\n WAITING_DEPENDENCY --> PENDING: Dependencies satisfied\n PENDING --> RUNNING: Execute (I1, I2 checked)\n RUNNING --> COMPLETED: Success\n RUNNING --> FAILED: Error\n RUNNING --> CANCELLED: Cancel\n```\n\n**Note:** Tasks become immutable (I3) upon entering RUNNING state. PENDING and WAITING_DEPENDENCY tasks remain modifiable.\n\n## Error Handling\n\n### Invariant Violation Responses\n\nWhen invariants are violated, the orchestrator takes appropriate action:\n\n#### I1 Violation: Multiple Assignments\n\n```python\n# Detected during validation\nif not task.target_device_id:\n errors.append(f\"Task '{task_id}' has no device assignment\")\n\n# Or during reassignment attempt\nif self._status == TaskStatus.RUNNING:\n raise ValueError(\n f\"Cannot modify device assignment of running task {self._task_id}\"\n )\n```\n\n**Response**: Reject modification, log error, continue with existing assignment\n\n#### I2 Violation: Cycle Detected\n\n```python\nif self._would_create_cycle(dependency.from_task_id, dependency.to_task_id):\n raise ValueError(\n f\"Adding dependency {dependency.from_task_id} -> {dependency.to_task_id} would create a cycle\"\n )\n```\n\n**Response**: Reject dependency addition, log error, constellation remains acyclic\n\n#### I3 Violation: Modifying Running Task\n\n```python\nif self._status == TaskStatus.RUNNING:\n raise ValueError(f\"Cannot modify name of running task {self._task_id}\")\n```\n\n**Response**: Reject modification, log error, task properties unchanged\n\n### Graceful Degradation\n\nIf the agent produces invalid modifications:\n\n```python\ntry:\n constellation.add_dependency(new_dependency)\nexcept ValueError as e:\n self.logger.error(f\"Invalid dependency rejected: {e}\")\n # Continue with existing constellation structure\n # Don't block orchestration on agent errors\n```\n\nThe orchestrator continues execution with the last valid constellation state.\n\n**Warning:** The orchestrator prioritizes safety (correctness) over liveness (progress). If the agent produces invalid modifications, orchestration may slow or stall, but will never execute an invalid DAG.\n\n## Performance Impact\n\n### Validation Overhead\n\n| Invariant | Check Complexity | Per-Operation Cost | When Checked |\n|-----------|-----------------|-------------------|--------------|\n| I1 | O(1) | < 1ms | Per task assignment |\n| I2 | O(V + E) (DFS) | 1-10ms | Per dependency add |\n| I3 | O(1) | < 1ms | Per task modification |\n\nWhere V = number of tasks, E = number of dependencies.\n\n### Optimization Strategies\n\n**1. Lazy Validation**:\n```python\n# Only validate when needed\ndef validate_dag(self) -> Tuple[bool, List[str]]:\n # Cache validation results if DAG hasn't changed\n if self._last_validation_time == self._updated_at:\n return self._cached_validation\n```\n\n**2. Incremental Checking**:\n```python\n# Check only affected subgraph for cycles\ndef _would_create_cycle(self, from_task_id: str, to_task_id: str) -> bool:\n # Only traverse from to_task → from_task\n # Don't re-check entire graph\n```\n\n**3. Batch Validation**:\n```python\n# Validate once after applying all modifications in a batch\nwhile queue not empty:\n # Apply all modifications\n pass\nvalidate(C) # Single validation for entire batch\n```\n\nLearn more about [batched editing strategies](batched_editing.md).\n\n## Testing Invariants\n\n### Unit Tests\n\nEach invariant has dedicated test coverage:\n\n```python\ndef test_single_assignment_invariant():\n \"\"\"Test I1: Single Assignment.\"\"\"\n task = TaskStar(task_id=\"test_task\")\n task.target_device_id = \"device_1\"\n \n # Assignment succeeds\n assert task.target_device_id == \"device_1\"\n \n # Reassignment before execution succeeds\n task.target_device_id = \"device_2\"\n assert task.target_device_id == \"device_2\"\n \n # Start execution\n task.start_execution()\n \n # Reassignment after execution fails\n with pytest.raises(ValueError):\n task.target_device_id = \"device_3\"\n\ndef test_acyclic_consistency_invariant():\n \"\"\"Test I2: Acyclic Consistency.\"\"\"\n constellation = TaskConstellation()\n task_a = TaskStar(task_id=\"A\")\n task_b = TaskStar(task_id=\"B\")\n task_c = TaskStar(task_id=\"C\")\n \n constellation.add_task(task_a)\n constellation.add_task(task_b)\n constellation.add_task(task_c)\n \n # Add A → B\n dep_ab = TaskStarLine(\"dep1\", \"A\", \"B\")\n constellation.add_dependency(dep_ab)\n \n # Add B → C\n dep_bc = TaskStarLine(\"dep2\", \"B\", \"C\")\n constellation.add_dependency(dep_bc)\n \n # Try to add C → A (creates cycle)\n dep_ca = TaskStarLine(\"dep3\", \"C\", \"A\")\n with pytest.raises(ValueError, match=\"would create a cycle\"):\n constellation.add_dependency(dep_ca)\n\ndef test_valid_update_invariant():\n \"\"\"Test I3: Valid Update.\"\"\"\n task = TaskStar(task_id=\"test_task\", description=\"Original\")\n \n # Modification before execution succeeds\n task.description = \"Modified\"\n assert task.description == \"Modified\"\n \n # Start execution\n task.start_execution()\n \n # Modification after execution fails\n with pytest.raises(ValueError):\n task.description = \"Invalid modification\"\n```\n\n### Integration Tests\n\nTest invariants during full orchestration:\n\n```python\nasync def test_invariants_during_orchestration():\n \"\"\"Test that invariants hold during concurrent orchestration.\"\"\"\n \n # Create constellation with potential for violations\n constellation = create_complex_constellation()\n \n # Attach synchronizer and validators\n orchestrator = TaskConstellationOrchestrator(device_manager)\n synchronizer = ConstellationModificationSynchronizer(orchestrator)\n orchestrator.set_modification_synchronizer(synchronizer)\n \n # Run orchestration\n results = await orchestrator.orchestrate_constellation(constellation)\n \n # Verify invariants held throughout\n assert results[\"status\"] == \"completed\"\n \n # Check I1: No duplicate assignments\n assignments = {}\n for task in constellation.get_all_tasks():\n device = task.target_device_id\n assert device not in assignments.values()\n assignments[task.task_id] = device\n \n # Check I2: No cycles\n is_valid, errors = constellation.validate_dag()\n assert is_valid\n \n # Check I3: Terminal tasks are immutable\n for task in constellation.get_completed_tasks() + constellation.get_failed_tasks():\n with pytest.raises(ValueError):\n task.description = \"Should fail\"\n```\n\n## Related Documentation\n\n- [Safe Assignment Locking](safe_assignment_locking.md) - How invariants are enforced during locking\n- [Asynchronous Scheduling](asynchronous_scheduling.md) - Concurrent execution preserving invariants\n- [Batched Editing](batched_editing.md) - Efficient modification batching while maintaining invariants\n- [API Reference](api_reference.md) - API methods for validation\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/constellation_manager.md",
"content": "# Constellation Manager\n\n## Overview\n\nThe `ConstellationManager` is a companion component to the `TaskConstellationOrchestrator` that handles device assignment, resource management, and constellation lifecycle tracking. While the orchestrator focuses on execution flow and coordination, the manager provides the infrastructure for device operations and state management.\n\nThis separation of concerns follows the Single Responsibility Principle: orchestration logic remains independent of device management details.\n\n## Architecture\n\n```mermaid\ngraph TB\n O[TaskConstellationOrchestrator] -->|uses| CM[ConstellationManager]\n CM -->|communicates| DM[ConstellationDeviceManager]\n DM -->|manages| D1[Device 1]\n DM -->|manages| D2[Device 2]\n DM -->|manages| D3[Device N]\n \n CM -->|tracks| MD[(Constellation Metadata)]\n CM -->|validates| AS[Device Assignments]\n```\n\nLearn more about the [orchestrator architecture](overview.md) and [asynchronous scheduling](asynchronous_scheduling.md).\n\n## Core Responsibilities\n\nThe ConstellationManager handles four primary responsibilities:\n\n### 1. Device Assignment\n\nAssigns tasks to appropriate devices using configurable strategies:\n\n| Strategy | Description | Use Case |\n|----------|-------------|----------|\n| Round Robin | Distributes tasks evenly across devices | Load balancing for homogeneous devices |\n| Capability Match | Matches task requirements to device capabilities | Heterogeneous device types (Windows, Android, iOS) |\n| Load Balance | Assigns to device with lowest current load | Dynamic workload distribution |\n\n### 2. Resource Management\n\nTracks and manages constellation resources:\n\n- Device availability and status\n- Constellation registration and metadata\n- Device utilization statistics\n- Assignment validation\n\n### 3. Lifecycle Management\n\nManages constellation lifecycle:\n\n- Registration when orchestration begins\n- Metadata tracking during execution\n- Unregistration after completion\n- Status querying\n\n### 4. Validation\n\nValidates device assignments against constraints:\n\n- All tasks have assigned devices\n- Assigned devices exist and are connected\n- Device capabilities match task requirements\n\n## Device Assignment Strategies\n\n### Round Robin\n\nDistributes tasks cyclically across available devices:\n\n```python\nasync def _assign_round_robin(\n self,\n constellation: TaskConstellation,\n available_devices: List[Dict[str, Any]],\n preferences: Optional[Dict[str, str]] = None,\n) -> Dict[str, str]:\n \"\"\"Round robin device assignment strategy.\"\"\"\n \n assignments = {}\n device_index = 0\n \n for task_id, task in constellation.tasks.items():\n # Check preferences first\n if preferences and task_id in preferences:\n preferred_device = preferences[task_id]\n if any(d[\"device_id\"] == preferred_device for d in available_devices):\n assignments[task_id] = preferred_device\n continue\n \n # Round robin assignment\n device = available_devices[device_index % len(available_devices)]\n assignments[task_id] = device[\"device_id\"]\n device_index += 1\n \n return assignments\n```\n\n**Characteristics:**\n\n- Fairness: Each device gets approximately equal number of tasks\n- Simplicity: No complex decision-making\n- Overhead: O(N) where N = number of tasks\n- Best for: Homogeneous devices with similar capabilities\n\n**Example**:\n```python\n# 3 devices, 7 tasks\nTask 1 → Device A\nTask 2 → Device B\nTask 3 → Device C\nTask 4 → Device A\nTask 5 → Device B\nTask 6 → Device C\nTask 7 → Device A\n```\n\n### Capability Match\n\nMatches tasks to devices based on device type and capabilities:\n\n```python\nasync def _assign_capability_match(\n self,\n constellation: TaskConstellation,\n available_devices: List[Dict[str, Any]],\n preferences: Optional[Dict[str, str]] = None,\n) -> Dict[str, str]:\n \"\"\"Capability-based device assignment strategy.\"\"\"\n \n assignments = {}\n \n for task_id, task in constellation.tasks.items():\n # Check preferences first\n if preferences and task_id in preferences:\n preferred_device = preferences[task_id]\n if any(d[\"device_id\"] == preferred_device for d in available_devices):\n assignments[task_id] = preferred_device\n continue\n \n # Find devices matching task requirements\n matching_devices = []\n \n if task.device_type:\n matching_devices = [\n d for d in available_devices\n if d.get(\"device_type\") == task.device_type.value\n ]\n \n # Fall back to any available device if no matches\n if not matching_devices:\n matching_devices = available_devices\n \n # Choose first matching device\n if matching_devices:\n assignments[task_id] = matching_devices[0][\"device_id\"]\n \n return assignments\n```\n\n**Characteristics:**\n\n- Type-aware: Respects task's `device_type` requirement\n- Fallback: Uses any device if no type match found\n- Overhead: O(N × D) where N = tasks, D = devices\n- Best for: Heterogeneous device ecosystems\n\n**Example**:\n```python\n# Mixed device types\nTask A (requires Windows) → Windows Device 1\nTask B (requires Android) → Android Device 1\nTask C (requires Windows) → Windows Device 1\nTask D (no requirement) → Any available device\n```\n\n### Load Balance\n\nAssigns tasks to minimize device load:\n\n```python\nasync def _assign_load_balance(\n self,\n constellation: TaskConstellation,\n available_devices: List[Dict[str, Any]],\n preferences: Optional[Dict[str, str]] = None,\n) -> Dict[str, str]:\n \"\"\"Load-balanced device assignment strategy.\"\"\"\n \n assignments = {}\n device_load = {d[\"device_id\"]: 0 for d in available_devices}\n \n for task_id, task in constellation.tasks.items():\n # Check preferences first\n if preferences and task_id in preferences:\n preferred_device = preferences[task_id]\n if any(d[\"device_id\"] == preferred_device for d in available_devices):\n assignments[task_id] = preferred_device\n device_load[preferred_device] += 1\n continue\n \n # Find device with lowest load\n min_load_device = min(device_load.keys(), key=lambda d: device_load[d])\n assignments[task_id] = min_load_device\n device_load[min_load_device] += 1\n \n return assignments\n```\n\n**Characteristics:**\n\n- Balanced: Minimizes maximum device load\n- Dynamic: Adapts to varying task counts\n- Overhead: O(N × log D) with priority queue optimization\n- Best for: Constellations with varying task complexity\n\n**Example**:\n```python\n# 2 devices, 5 tasks with varying complexity\nTask 1 (simple) → Device A [load: 1]\nTask 2 (complex) → Device B [load: 1]\nTask 3 (simple) → Device A [load: 2]\nTask 4 (simple) → Device B [load: 2]\nTask 5 (complex) → Device A [load: 3]\n```\n\n## Constellation Lifecycle Management\n\n### Registration\n\nRegister a constellation for management:\n\n```python\ndef register_constellation(\n self,\n constellation: TaskConstellation,\n metadata: Optional[Dict[str, Any]] = None,\n) -> str:\n \"\"\"Register a constellation for management.\"\"\"\n \n constellation_id = constellation.constellation_id\n self._managed_constellations[constellation_id] = constellation\n self._constellation_metadata[constellation_id] = metadata or {}\n \n if self._logger:\n self._logger.info(\n f\"Registered constellation '{constellation.name}' ({constellation_id})\"\n )\n \n return constellation_id\n```\n\n**Purpose**: Track active constellations and their metadata\n\n**Metadata examples**:\n```python\nmetadata = {\n \"user_id\": \"user123\",\n \"session_id\": \"session_456\",\n \"priority\": \"high\",\n \"created_by\": \"automation_pipeline\",\n}\n```\n\n### Status Querying\n\nGet detailed status of a managed constellation:\n\n```python\nasync def get_constellation_status(\n self, constellation_id: str\n) -> Optional[Dict[str, Any]]:\n \"\"\"Get detailed status of a managed constellation.\"\"\"\n \n constellation = self._managed_constellations.get(constellation_id)\n if not constellation:\n return None\n \n metadata = self._constellation_metadata.get(constellation_id, {})\n \n return {\n \"constellation_id\": constellation_id,\n \"name\": constellation.name,\n \"state\": constellation.state.value,\n \"statistics\": constellation.get_statistics(),\n \"ready_tasks\": [task.task_id for task in constellation.get_ready_tasks()],\n \"running_tasks\": [task.task_id for task in constellation.get_running_tasks()],\n \"completed_tasks\": [task.task_id for task in constellation.get_completed_tasks()],\n \"failed_tasks\": [task.task_id for task in constellation.get_failed_tasks()],\n \"metadata\": metadata,\n }\n```\n\n**Returns**:\n```json\n{\n \"constellation_id\": \"constellation_20251106_143052_a1b2c3d4\",\n \"name\": \"Multi-Device Data Collection\",\n \"state\": \"executing\",\n \"statistics\": {\n \"total_tasks\": 10,\n \"task_status_counts\": {\n \"completed\": 3,\n \"running\": 2,\n \"pending\": 5\n },\n \"parallelism_ratio\": 2.5\n },\n \"ready_tasks\": [\"task_6\", \"task_7\"],\n \"running_tasks\": [\"task_4\", \"task_5\"],\n \"completed_tasks\": [\"task_1\", \"task_2\", \"task_3\"],\n \"failed_tasks\": [],\n \"metadata\": {\n \"user_id\": \"user123\",\n \"priority\": \"high\"\n }\n}\n```\n\n### Unregistration\n\nRemove a constellation from management:\n\n```python\ndef unregister_constellation(self, constellation_id: str) -> bool:\n \"\"\"Unregister a constellation from management.\"\"\"\n \n if constellation_id in self._managed_constellations:\n constellation = self._managed_constellations[constellation_id]\n del self._managed_constellations[constellation_id]\n del self._constellation_metadata[constellation_id]\n \n if self._logger:\n self._logger.info(\n f\"Unregistered constellation '{constellation.name}' ({constellation_id})\"\n )\n return True\n \n return False\n```\n\n**Purpose**: Clean up resources after orchestration completes\n\n## Device Operations\n\n### Getting Available Devices\n\nRetrieve list of connected devices:\n\n```python\nasync def get_available_devices(self) -> List[Dict[str, Any]]:\n \"\"\"Get list of available devices from device manager.\"\"\"\n \n if not self._device_manager:\n return []\n \n try:\n connected_device_ids = self._device_manager.get_connected_devices()\n devices = []\n \n for device_id in connected_device_ids:\n device_info = self._device_manager.device_registry.get_device_info(\n device_id\n )\n if device_info:\n devices.append({\n \"device_id\": device_id,\n \"device_type\": getattr(device_info, \"device_type\", \"unknown\"),\n \"capabilities\": getattr(device_info, \"capabilities\", []),\n \"status\": \"connected\",\n \"metadata\": getattr(device_info, \"metadata\", {}),\n })\n \n return devices\n except Exception as e:\n if self._logger:\n self._logger.error(f\"Failed to get available devices: {e}\")\n return []\n```\n\n**Returns**:\n```python\n[\n {\n \"device_id\": \"windows_main\",\n \"device_type\": \"windows\",\n \"capabilities\": [\"file_ops\", \"browser\", \"office\"],\n \"status\": \"connected\",\n \"metadata\": {\"os_version\": \"Windows 11\"}\n },\n {\n \"device_id\": \"android_pixel\",\n \"device_type\": \"android\",\n \"capabilities\": [\"touch\", \"camera\", \"gps\"],\n \"status\": \"connected\",\n \"metadata\": {\"android_version\": \"14\"}\n }\n]\n```\n\n### Device Assignment\n\nAutomatically assign devices to all tasks:\n\n```python\nasync def assign_devices_automatically(\n self,\n constellation: TaskConstellation,\n strategy: str = \"round_robin\",\n device_preferences: Optional[Dict[str, str]] = None,\n) -> Dict[str, str]:\n \"\"\"Automatically assign devices to tasks in a constellation.\"\"\"\n \n if not self._device_manager:\n raise ValueError(\"Device manager not available for device assignment\")\n \n available_devices = await self._get_available_devices()\n if not available_devices:\n raise ValueError(\"No available devices for assignment\")\n \n if self._logger:\n self._logger.info(\n f\"Assigning devices to constellation '{constellation.name}' \"\n f\"using strategy '{strategy}'\"\n )\n \n # Select strategy\n if strategy == \"round_robin\":\n assignments = await self._assign_round_robin(\n constellation, available_devices, device_preferences\n )\n elif strategy == \"capability_match\":\n assignments = await self._assign_capability_match(\n constellation, available_devices, device_preferences\n )\n elif strategy == \"load_balance\":\n assignments = await self._assign_load_balance(\n constellation, available_devices, device_preferences\n )\n else:\n raise ValueError(f\"Unknown assignment strategy: {strategy}\")\n \n # Apply assignments to tasks\n for task_id, device_id in assignments.items():\n task = constellation.get_task(task_id)\n if task:\n task.target_device_id = device_id\n \n if self._logger:\n self._logger.info(f\"Assigned {len(assignments)} tasks to devices\")\n \n return assignments\n```\n\n### Manual Reassignment\n\nReassign a single task to a different device:\n\n```python\ndef reassign_task_device(\n self,\n constellation: TaskConstellation,\n task_id: str,\n new_device_id: str,\n) -> bool:\n \"\"\"Reassign a task to a different device.\"\"\"\n \n task = constellation.get_task(task_id)\n if not task:\n return False\n \n old_device_id = task.target_device_id\n task.target_device_id = new_device_id\n \n if self._logger:\n self._logger.info(\n f\"Reassigned task '{task_id}' from device '{old_device_id}' \"\n f\"to '{new_device_id}'\"\n )\n \n return True\n```\n\n## Validation\n\n### Assignment Validation\n\nValidate that all tasks have valid device assignments:\n\n```python\ndef validate_constellation_assignments(\n self, constellation: TaskConstellation\n) -> tuple[bool, List[str]]:\n \"\"\"Validate that all tasks have valid device assignments.\"\"\"\n \n errors = []\n \n for task_id, task in constellation.tasks.items():\n if not task.target_device_id:\n errors.append(f\"Task '{task_id}' has no device assignment\")\n \n is_valid = len(errors) == 0\n \n if self._logger:\n if is_valid:\n self._logger.info(\n f\"All tasks in constellation '{constellation.name}' have \"\n f\"valid assignments\"\n )\n else:\n self._logger.warning(\n f\"Constellation '{constellation.name}' has {len(errors)} \"\n f\"assignment errors\"\n )\n \n return is_valid, errors\n```\n\n### Device Information\n\nGet device information for a specific task:\n\n```python\ndef get_task_device_info(\n self, constellation: TaskConstellation, task_id: str\n) -> Optional[Dict[str, Any]]:\n \"\"\"Get device information for a specific task.\"\"\"\n \n task = constellation.get_task(task_id)\n if not task or not task.target_device_id:\n return None\n \n # Get device info from device manager\n if self._device_manager:\n try:\n device_info = self._device_manager.device_registry.get_device_info(\n task.target_device_id\n )\n if device_info:\n return {\n \"device_id\": task.target_device_id,\n \"device_type\": getattr(device_info, \"device_type\", \"unknown\"),\n \"capabilities\": getattr(device_info, \"capabilities\", []),\n \"metadata\": getattr(device_info, \"metadata\", {}),\n }\n except Exception as e:\n if self._logger:\n self._logger.error(\n f\"Failed to get device info for task '{task_id}': {e}\"\n )\n \n return None\n```\n\n## Utilization Tracking\n\n### Device Utilization Statistics\n\nGet device utilization across constellation:\n\n```python\ndef get_device_utilization(\n self, constellation: TaskConstellation\n) -> Dict[str, int]:\n \"\"\"Get device utilization statistics for a constellation.\"\"\"\n \n utilization = {}\n \n for task in constellation.tasks.values():\n if task.target_device_id:\n utilization[task.target_device_id] = (\n utilization.get(task.target_device_id, 0) + 1\n )\n \n return utilization\n```\n\n**Example output**:\n```python\n{\n \"windows_main\": 5,\n \"android_pixel\": 3,\n \"ios_iphone\": 2\n}\n```\n\n### Listing All Constellations\n\nList all managed constellations:\n\n```python\ndef list_constellations(self) -> List[Dict[str, Any]]:\n \"\"\"List all managed constellations with basic information.\"\"\"\n \n result = []\n for constellation_id, constellation in self._managed_constellations.items():\n metadata = self._constellation_metadata.get(constellation_id, {})\n result.append({\n \"constellation_id\": constellation_id,\n \"name\": constellation.name,\n \"state\": constellation.state.value,\n \"task_count\": constellation.task_count,\n \"dependency_count\": constellation.dependency_count,\n \"metadata\": metadata,\n })\n \n return result\n```\n\n## Usage Patterns\n\n### Basic Setup\n\n```python\nfrom galaxy.constellation.orchestrator import ConstellationManager\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Create device manager\ndevice_manager = ConstellationDeviceManager()\n\n# Create constellation manager\nmanager = ConstellationManager(device_manager, enable_logging=True)\n\n# Register constellation\nconstellation_id = manager.register_constellation(\n constellation,\n metadata={\"priority\": \"high\"}\n)\n```\n\n### Automatic Assignment\n\n```python\n# Assign devices using capability matching\nassignments = await manager.assign_devices_automatically(\n constellation,\n strategy=\"capability_match\"\n)\n\nprint(f\"Assigned {len(assignments)} tasks\")\n```\n\n### With Preferences\n\n```python\n# Specify preferred devices for specific tasks\npreferences = {\n \"critical_task_1\": \"windows_main\",\n \"gpu_task_2\": \"windows_gpu\",\n}\n\nassignments = await manager.assign_devices_automatically(\n constellation,\n strategy=\"load_balance\",\n device_preferences=preferences\n)\n```\n\n### Manual Override\n\n```python\n# Reassign specific task\nmanager.reassign_task_device(\n constellation,\n task_id=\"task_5\",\n new_device_id=\"android_backup\"\n)\n```\n\n### Validation\n\n```python\n# Validate assignments before orchestration\nis_valid, errors = manager.validate_constellation_assignments(constellation)\n\nif not is_valid:\n print(f\"Validation errors: {errors}\")\n # Fix assignments...\n```\n\n### Monitoring\n\n```python\n# Check constellation status during execution\nstatus = await manager.get_constellation_status(constellation_id)\n\nprint(f\"State: {status['state']}\")\nprint(f\"Running tasks: {len(status['running_tasks'])}\")\nprint(f\"Completed tasks: {len(status['completed_tasks'])}\")\n\n# Get device utilization\nutilization = manager.get_device_utilization(constellation)\nfor device_id, task_count in utilization.items():\n print(f\"{device_id}: {task_count} tasks\")\n```\n\n## Integration with Orchestrator\n\nThe orchestrator uses the manager internally:\n\n```python\nclass TaskConstellationOrchestrator:\n def __init__(self, device_manager, enable_logging=True):\n self._device_manager = device_manager\n self._constellation_manager = ConstellationManager(\n device_manager, enable_logging\n )\n \n async def orchestrate_constellation(self, constellation, ...):\n # Use manager for assignment\n await self._constellation_manager.assign_devices_automatically(\n constellation, assignment_strategy\n )\n \n # Use manager for validation\n is_valid, errors = self._constellation_manager \\\n .validate_constellation_assignments(constellation)\n \n if not is_valid:\n raise ValueError(f\"Device assignment validation failed: {errors}\")\n \n # Continue orchestration...\n```\n\n## Related Documentation\n\n- [Overview](overview.md) - Orchestrator architecture and design\n- [Asynchronous Scheduling](asynchronous_scheduling.md) - Task execution model\n- [Consistency Guarantees](consistency_guarantees.md) - Device assignment validation\n- [API Reference](api_reference.md) - Complete API documentation\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/event_driven_coordination.md",
"content": "# Event-Driven Coordination\n\n## Overview\n\nTraditional DAG schedulers rely on **polling** or **global checkpoints** to detect task completion, introducing latency and synchronization overhead. In contrast, the Constellation Orchestrator operates as a fully **event-driven** system built on an internal event bus and observer design pattern.\n\nThis architecture enables immediate, fine-grained reactions to runtime signals without centralized coordination delays, providing the foundation for adaptive orchestration in UFO. For an overview of how events drive the orchestrator, see the [Orchestrator Overview](overview.md).\n\n## Event System Architecture\n\nThe event-driven coordination system consists of three core components:\n\n```mermaid\ngraph LR\n A[Event Publishers] -->|publish| B[Event Bus]\n B -->|notify| C[Event Observers]\n \n D[Orchestrator] -.->|implements| A\n D -.->|implements| C\n \n E[Synchronizer] -.->|implements| C\n F[Agent] -.->|implements| A\n \n style B fill:#4a90e2,stroke:#333,stroke-width:3px,color:#fff\n style D fill:#ffa726,stroke:#333,stroke-width:2px\n style E fill:#66bb6a,stroke:#333,stroke-width:2px\n style F fill:#ab47bc,stroke:#333,stroke-width:2px\n```\n\n### Event Bus\n\nThe `EventBus` class serves as the central message broker, managing subscriptions and distributing events throughout the system.\n\n**Key Features:**\n\n- **Type-based subscription**: Observers subscribe to specific event types\n- **Wildcard subscription**: Observers can subscribe to all events\n- **Concurrent notification**: All observers are notified asynchronously in parallel\n- **Error isolation**: Exceptions in one observer don't affect others\n\n**Implementation** (`galaxy/core/events.py`):\n\n```python\nclass EventBus(IEventPublisher):\n \"\"\"Central event bus for Galaxy framework.\"\"\"\n \n def subscribe(self, observer: IEventObserver, \n event_types: Set[EventType] = None) -> None:\n \"\"\"Subscribe observer to specific events or all events.\"\"\"\n if event_types is None:\n self._all_observers.add(observer)\n else:\n for event_type in event_types:\n if event_type not in self._observers:\n self._observers[event_type] = set()\n self._observers[event_type].add(observer)\n \n async def publish_event(self, event: Event) -> None:\n \"\"\"Publish event to all relevant subscribers.\"\"\"\n observers_to_notify = set()\n \n # Add type-specific observers\n if event.event_type in self._observers:\n observers_to_notify.update(self._observers[event.event_type])\n \n # Add wildcard observers\n observers_to_notify.update(self._all_observers)\n \n # Notify all concurrently\n if observers_to_notify:\n tasks = [observer.on_event(event) for observer in observers_to_notify]\n await asyncio.gather(*tasks, return_exceptions=True)\n```\n\n!!!tip \"Design Pattern\"\n The event bus implements the **Observer** (or Publish-Subscribe) pattern, decoupling event producers from consumers and enabling extensible system behavior.\n\n## Event Types\n\nThe orchestrator uses four primary event types that capture the complete lifecycle of tasks and constellations:\n\n### Task-Level Events\n\nThese events track individual task state transitions during execution:\n\n| Event Type | Trigger | Published By | Data Payload |\n|------------|---------|--------------|--------------|\n| `TASK_STARTED` | Task assigned to device and execution begins | Orchestrator | `task_id`, `status`, `constellation_id` |\n| `TASK_COMPLETED` | Task finishes successfully | Orchestrator | `task_id`, `status`, `result`, `newly_ready_tasks`, `constellation` |\n| `TASK_FAILED` | Task execution fails | Orchestrator | `task_id`, `status`, `error`, `newly_ready_tasks` |\n\n**Event Structure:**\n\n```python\n@dataclass\nclass TaskEvent(Event):\n \"\"\"Task-specific event.\"\"\"\n task_id: str\n status: str\n result: Any = None\n error: Optional[Exception] = None\n```\n\n### Constellation-Level Events\n\nThese events track macro-level constellation lifecycle and structural changes:\n\n| Event Type | Trigger | Published By | Data Payload |\n|------------|---------|--------------|--------------|\n| `CONSTELLATION_STARTED` | Orchestration begins | Orchestrator | `total_tasks`, `assignment_strategy`, `constellation` |\n| `CONSTELLATION_COMPLETED` | All tasks finished | Orchestrator | `total_tasks`, `statistics`, `execution_duration` |\n| `CONSTELLATION_MODIFIED` | DAG structure updated by agent | Agent | `on_task_id`, `new_constellation`, `modifications` |\n\n**Event Structure:**\n\n```python\n@dataclass\nclass ConstellationEvent(Event):\n \"\"\"Constellation-specific event.\"\"\"\n constellation_id: str\n constellation_state: str\n new_ready_tasks: List[str] = None\n```\n\nAll events inherit from the base `Event` class which provides common fields: `event_type`, `source_id`, `timestamp`, and `data`.\n\n## Observer Pattern Implementation\n\nThe orchestrator and related components implement the `IEventObserver` interface to react to events:\n\n```python\nclass IEventObserver(ABC):\n \"\"\"Interface for event observers.\"\"\"\n \n @abstractmethod\n async def on_event(self, event: Event) -> None:\n \"\"\"Handle an event.\"\"\"\n pass\n```\n\n### Key Observers in the System\n\n#### 1. ConstellationModificationSynchronizer\n\nEnsures proper synchronization between task completion and constellation modifications:\n\n```python\nclass ConstellationModificationSynchronizer(IEventObserver):\n \"\"\"Synchronizes constellation modifications with orchestrator execution.\"\"\"\n \n async def on_event(self, event: Event) -> None:\n if isinstance(event, TaskEvent):\n await self._handle_task_event(event)\n elif isinstance(event, ConstellationEvent):\n await self._handle_constellation_event(event)\n```\n\n**Responsibilities:**\n\n- Register pending modifications when tasks complete\n- Mark modifications as complete when agent finishes editing\n- Provide synchronization point for orchestrator\n\n[Learn more →](safe_assignment_locking.md#modification-synchronizer)\n\n#### 2. Visualization Observers\n\nHandle real-time visualization updates as constellation evolves:\n\n- `DAGVisualizationObserver` - Updates DAG topology visualization\n- `TaskVisualizationHandler` - Updates task status displays\n- `ConstellationVisualizationHandler` - Updates overall constellation state\n\n!!!example \"Observer Subscription\"\n ```python\n # Subscribe synchronizer to task and constellation events\n event_bus = get_event_bus()\n synchronizer = ConstellationModificationSynchronizer(orchestrator)\n event_bus.subscribe(synchronizer, {\n EventType.TASK_COMPLETED,\n EventType.TASK_FAILED,\n EventType.CONSTELLATION_MODIFIED\n })\n ```\n\n## Event Flow in Orchestration\n\nThe following sequence diagram illustrates how events flow through the system during orchestration:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant EB as Event Bus\n participant S as Synchronizer\n participant V as Visualizer\n participant A as Agent\n\n Note over O: Task A execution starts\n O->>EB: Publish TASK_STARTED(A)\n EB-->>V: Notify observers\n V->>V: Update visualization\n \n Note over O: Task A completes\n O->>EB: Publish TASK_COMPLETED(A)\n EB-->>S: Notify observers\n EB-->>V: Notify observers\n \n S->>S: Register pending modification\n V->>V: Update task status\n \n Note over A: Agent processes completion\n A->>A: Edit constellation\n A->>EB: Publish CONSTELLATION_MODIFIED\n \n EB-->>S: Notify observers\n EB-->>V: Notify observers\n \n S->>S: Complete modification future\n S->>O: Sync constellation state\n V->>V: Update constellation view\n```\n\nThis flow demonstrates several key aspects:\n\n1. **Immediate notification**: Events are published as soon as state changes occur\n2. **Parallel processing**: Multiple observers react concurrently\n3. **Decoupled components**: Publishers don't know about subscribers\n4. **Asynchronous coordination**: No blocking waits or polling\n\n## Event Publishing in Orchestrator\n\nThe orchestrator publishes events at critical execution points:\n\n### Task Execution Events\n\nWhen executing a task, the orchestrator wraps execution in event publishing:\n\n```python\nasync def _execute_task_with_events(\n self, task: TaskStar, constellation: TaskConstellation\n) -> None:\n \"\"\"Execute a single task and publish events.\"\"\"\n \n try:\n # Publish task started event\n start_event = TaskEvent(\n event_type=EventType.TASK_STARTED,\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\"constellation_id\": constellation.constellation_id},\n task_id=task.task_id,\n status=TaskStatus.RUNNING.value,\n )\n await self._event_bus.publish_event(start_event)\n \n # Execute task\n task.start_execution()\n result = await task.execute(self._device_manager)\n \n is_success = result.status == TaskStatus.COMPLETED.value\n \n # Mark as completed and get newly ready tasks\n newly_ready = constellation.mark_task_completed(\n task.task_id, success=is_success, result=result\n )\n \n # Publish task completed or failed event\n completed_event = TaskEvent(\n event_type=(\n EventType.TASK_COMPLETED if is_success \n else EventType.TASK_FAILED\n ),\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\n \"constellation_id\": constellation.constellation_id,\n \"newly_ready_tasks\": [t.task_id for t in newly_ready],\n \"constellation\": constellation,\n },\n task_id=task.task_id,\n status=result.status,\n result=result,\n )\n await self._event_bus.publish_event(completed_event)\n \n except Exception as e:\n # Mark task as failed and get newly ready tasks\n newly_ready = constellation.mark_task_completed(\n task.task_id, success=False, error=e\n )\n \n # Publish task failed event\n failed_event = TaskEvent(\n event_type=EventType.TASK_FAILED,\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\n \"constellation_id\": constellation.constellation_id,\n \"newly_ready_tasks\": [t.task_id for t in newly_ready],\n },\n task_id=task.task_id,\n status=TaskStatus.FAILED.value,\n error=e,\n )\n await self._event_bus.publish_event(failed_event)\n raise\n```\n\n!!!warning \"Critical Section\"\n Event publishing happens **immediately** after state transitions but **before** any dependent operations, ensuring observers have the latest state.\n\n### Constellation Lifecycle Events\n\nThe orchestrator also publishes constellation-level events:\n\n```python\n# At orchestration start\nstart_event = ConstellationEvent(\n event_type=EventType.CONSTELLATION_STARTED,\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\n \"total_tasks\": len(constellation.tasks),\n \"assignment_strategy\": assignment_strategy,\n \"constellation\": constellation,\n },\n constellation_id=constellation.constellation_id,\n constellation_state=\"executing\",\n)\nawait self._event_bus.publish_event(start_event)\n\n# At orchestration completion\ncompletion_event = ConstellationEvent(\n event_type=EventType.CONSTELLATION_COMPLETED,\n source_id=f\"orchestrator_{id(self)}\",\n timestamp=time.time(),\n data={\n \"total_tasks\": len(constellation.tasks),\n \"statistics\": constellation.get_statistics(),\n \"execution_duration\": time.time() - start_event.timestamp,\n },\n constellation_id=constellation.constellation_id,\n constellation_state=\"completed\",\n)\nawait self._event_bus.publish_event(completion_event)\n```\n\n## Benefits of Event-Driven Architecture\n\nThe event-driven design provides several critical advantages:\n\n### 1. High Responsiveness\n\nEvents are processed **immediately** upon publication with no polling delay:\n\n- Task completion → Agent notified instantly\n- Constellation modified → Orchestrator syncs immediately\n- Failure detected → Recovery triggered without delay\n\n### 2. Loose Coupling\n\nComponents interact through events rather than direct calls:\n\n- Orchestrator doesn't know about visualization\n- Agent doesn't know about synchronizer\n- New observers can be added without modifying publishers\n\n### 3. Extensibility\n\nNew functionality can be added by creating new observers:\n\n```python\nclass MetricsCollector(IEventObserver):\n \"\"\"Collect orchestration metrics.\"\"\"\n \n async def on_event(self, event: Event) -> None:\n if event.event_type == EventType.TASK_COMPLETED:\n self._record_task_completion(event)\n elif event.event_type == EventType.CONSTELLATION_COMPLETED:\n self._record_constellation_metrics(event)\n\n# Subscribe to event bus\nevent_bus.subscribe(MetricsCollector())\n```\n\n### 4. Concurrent Processing\n\nMultiple observers process events in parallel:\n\n- Visualization updates don't block synchronization\n- Logging doesn't delay task scheduling\n- Metrics collection happens asynchronously\n\n### 5. Error Isolation\n\nExceptions in one observer don't affect others:\n\n```python\n# In EventBus.publish_event()\nawait asyncio.gather(*tasks, return_exceptions=True)\n```\n\nIf a visualization observer crashes, the synchronizer still processes the event correctly.\n\n## Performance Characteristics\n\n| Aspect | Measurement | Impact |\n|--------|-------------|--------|\n| **Event Latency** | < 1ms (in-memory) | Negligible overhead |\n| **Notification Overhead** | O(N) where N = observers | Scales linearly with observers |\n| **Concurrency** | Unlimited parallel observers | No bottleneck from sequential processing |\n| **Memory** | Event objects garbage collected | No long-term accumulation |\n\nThe event system has been battle-tested in production with up to 50+ concurrent observers, 1000+ events per second, complex multi-device constellations, and long-running orchestration sessions.\n\n## Usage Patterns\n\n### Creating Custom Observers\n\nTo create a custom observer for orchestration events:\n\n```python\nfrom galaxy.core.events import IEventObserver, Event, EventType\n\nclass CustomOrchestrationObserver(IEventObserver):\n \"\"\"Custom observer for orchestration events.\"\"\"\n \n def __init__(self):\n self.task_count = 0\n self.completion_times = []\n \n async def on_event(self, event: Event) -> None:\n \"\"\"Handle events of interest.\"\"\"\n \n if event.event_type == EventType.TASK_COMPLETED:\n self.task_count += 1\n duration = event.data.get(\"result\").end_time - \\\n event.data.get(\"result\").start_time\n self.completion_times.append(duration.total_seconds())\n \n print(f\"Task {event.task_id} completed in \"\n f\"{duration.total_seconds():.2f}s\")\n \n elif event.event_type == EventType.CONSTELLATION_COMPLETED:\n avg_time = sum(self.completion_times) / len(self.completion_times)\n print(f\"Constellation completed! \"\n f\"Average task time: {avg_time:.2f}s\")\n\n# Register observer\nfrom galaxy.core.events import get_event_bus\n\nobserver = CustomOrchestrationObserver()\nevent_bus = get_event_bus()\nevent_bus.subscribe(observer, {\n EventType.TASK_COMPLETED,\n EventType.CONSTELLATION_COMPLETED\n})\n```\n\n### Event Filtering\n\nObservers can filter events based on custom criteria:\n\n```python\nclass FailureMonitor(IEventObserver):\n \"\"\"Monitor and log only failure events.\"\"\"\n \n async def on_event(self, event: Event) -> None:\n # Only process failure events\n if event.event_type != EventType.TASK_FAILED:\n return\n \n # Log failure details\n self.logger.error(\n f\"Task {event.task_id} failed: {event.error}\"\n )\n \n # Optionally trigger alerts or recovery\n await self._handle_task_failure(event)\n```\n\n## Related Documentation\n\n- **[Asynchronous Scheduling](asynchronous_scheduling.md)** - How events trigger task scheduling\n- **[Safe Assignment Locking](safe_assignment_locking.md)** - Event-driven synchronization\n- **[API Reference](api_reference.md)** - Event classes and interfaces\n\n---\n\n!!!tip \"Next Steps\"\n To understand how events drive concurrent task execution, continue to [Asynchronous Scheduling](asynchronous_scheduling.md).\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/overview.md",
"content": "# Constellation Orchestrator Overview\n\n## Introduction\n\nThe **Constellation Orchestrator** is the execution engine at the heart of UFO's multi-device orchestration system. While the Constellation Agent handles reasoning and task graph evolution, the orchestrator transforms these declarative plans into concrete execution across heterogeneous devices.\n\nUnlike traditional DAG schedulers that execute static task graphs, the Constellation Orchestrator operates as a **living execution fabric** where tasks evolve concurrently, react to runtime signals, and adapt to new decisions from the reasoning agent in real-time.\n\n\n\n*The Constellation Orchestrator bridges TaskConstellation and execution, enabling asynchronous, adaptive task orchestration across devices.*\n\n## Key Capabilities\n\nThe orchestrator achieves three critical goals that traditional schedulers struggle to balance:\n\n| Capability | Description | Benefit |\n|------------|-------------|---------|\n| **Asynchronous Parallelism** | Execute independent tasks concurrently across heterogeneous devices | Maximize device utilization and minimize idle time |\n| **Safety & Consistency** | Maintain correctness under concurrent DAG updates from LLM reasoning | Prevent race conditions and invalid execution states |\n| **Runtime Adaptivity** | React to feedback from both devices and LLM reasoning dynamically | Enable intelligent re-planning and error recovery |\n\n## Architecture Overview\n\nThe Constellation Orchestrator is built on five fundamental design pillars:\n\n```mermaid\ngraph TB\n A[Event-Driven Coordination] --> B[Orchestrator Core]\n C[Asynchronous Scheduling] --> B\n D[Safe Assignment Locking] --> B\n E[Consistency Enforcement] --> B\n F[Batched Editing] --> B\n \n B --> G[Device Execution]\n B --> H[Constellation Evolution]\n \n style B fill:#4a90e2,stroke:#333,stroke-width:3px,color:#fff\n style A fill:#7cb342,stroke:#333,stroke-width:2px\n style C fill:#7cb342,stroke:#333,stroke-width:2px\n style D fill:#7cb342,stroke:#333,stroke-width:2px\n style E fill:#7cb342,stroke:#333,stroke-width:2px\n style F fill:#7cb342,stroke:#333,stroke-width:2px\n```\n\n### 1. Event-Driven Coordination\n\nThe orchestrator operates as a fully event-driven system using an observer pattern and internal event bus. Instead of polling or global checkpoints, it reacts immediately to four primary event types:\n\n- `TASK_STARTED` - Task assigned and execution begins\n- `TASK_COMPLETED` - Task finishes successfully\n- `TASK_FAILED` - Task execution fails\n- `CONSTELLATION_MODIFIED` - DAG structure updated by agent\n\nThis design provides **high responsiveness** and eliminates synchronization overhead.\n\n[Learn more →](event_driven_coordination.md)\n\n### 2. Asynchronous Scheduling\n\nA continuous scheduling loop monitors the evolving TaskConstellation, identifies ready tasks (dependencies satisfied), and dispatches them concurrently to available devices. Critically, **task execution and constellation editing proceed in parallel**, overlapping computation with orchestration.\n\nThis enables **maximum parallelism** and **minimal latency** in cross-device workflows.\n\n[Learn more →](asynchronous_scheduling.md)\n\n### 3. Safe Assignment Locking\n\nTo prevent race conditions when LLM-driven edits overlap with task execution, the orchestrator employs a safe assignment lock protocol. During edit cycles, new task assignments are suspended while modifications are applied atomically and synchronized with runtime progress.\n\nThis guarantees **atomicity** and **prevents conflicts** between execution and modification.\n\n[Learn more →](safe_assignment_locking.md)\n\n### 4. Consistency Enforcement\n\nThe orchestrator enforces three runtime invariants to preserve correctness even under partial or invalid LLM updates:\n\n- **I1 (Single Assignment)**: Each task has at most one active device assignment\n- **I2 (Acyclic Consistency)**: Edits preserve DAG acyclicity (no cycles)\n- **I3 (Valid Update)**: Only PENDING tasks and their dependents can be modified\n\nThese invariants ensure **structural integrity** and **semantic validity** of the constellation.\n\n[Learn more →](consistency_guarantees.md)\n\n### 5. Batched Constellation Editing\n\nTo balance responsiveness with efficiency, the orchestrator batches multiple task completion events and applies their resulting modifications atomically. This amortizes LLM invocation overhead while preserving atomicity and consistency.\n\nThis achieves both **efficiency** and **adaptivity** without excessive micro-edits.\n\n[Learn more →](batched_editing.md)\n\n## System Components\n\nThe orchestrator consists of two primary components working in tandem:\n\n### TaskConstellationOrchestrator\n\nThe main execution orchestrator focused on flow control and coordination. It manages:\n\n- Event-driven task lifecycle (start, complete, fail)\n- Asynchronous scheduling loop\n- Safe assignment locking protocol\n- Integration with modification synchronizer\n\n[API Reference →](api_reference.md)\n\n### ConstellationManager\n\nHandles device assignment, resource management, and constellation lifecycle. It provides:\n\n- Multiple assignment strategies (round-robin, capability-match, load-balance)\n- Device validation and status tracking\n- Constellation registration and metadata management\n\n[Learn more →](constellation_manager.md)\n\n## Execution Flow\n\nThe orchestration workflow follows this sequence:\n\n```mermaid\nsequenceDiagram\n participant U as User Request\n participant O as Orchestrator\n participant C as Constellation\n participant S as Synchronizer\n participant D as Devices\n participant A as Agent\n\n U->>O: orchestrate_constellation()\n O->>C: Validate DAG\n O->>C: Assign devices\n \n loop Execution Loop\n O->>C: Get ready tasks\n O->>D: Dispatch tasks (async)\n D-->>O: Task completed event\n O->>S: Wait for modifications\n S->>A: Trigger editing\n A->>C: Update constellation\n A-->>S: Modification complete\n S->>O: Sync constellation state\n end\n \n O->>U: Return results\n```\n\nThe orchestrator treats task execution as an **open-world process** - continuously evolving, reacting, and converging toward user intent rather than executing a fixed plan.\n\n## Design Highlights\n\n### Asynchronous by Default\n\nEvery operation runs asynchronously using Python's `asyncio`, enabling:\n\n- Concurrent task execution across devices\n- Non-blocking event handling\n- Parallel constellation editing\n\n### LLM-Aware Orchestration\n\nUnlike traditional schedulers, the orchestrator is designed for **reasoning-aware execution**:\n\n- Expects and handles dynamic graph modifications\n- Synchronizes LLM reasoning with runtime execution\n- Validates and enforces safety under AI-driven changes\n\n### Production-Ready Safeguards\n\n- Timeout protection for modifications (default: 600s)\n- Automatic validation before every execution cycle\n- Device assignment verification\n- Cycle detection on every edit\n- Comprehensive error handling and logging\n\n## Performance Characteristics\n\n| Metric | Description | Implementation |\n|--------|-------------|----------------|\n| **Latency** | Time from task ready to execution start | Minimized via event-driven dispatch |\n| **Throughput** | Tasks completed per unit time | Maximized via async parallelism |\n| **Overhead** | Orchestration cost per task | Reduced via batched editing |\n| **Scalability** | Performance with increasing tasks/devices | Linear with async coordination |\n\n## Getting Started\n\n### Basic Usage\n\n```python\nfrom galaxy.constellation import TaskConstellationOrchestrator\nfrom galaxy.client.device_manager import ConstellationDeviceManager\n\n# Create orchestrator\ndevice_manager = ConstellationDeviceManager()\norchestrator = TaskConstellationOrchestrator(device_manager)\n\n# Orchestrate constellation\nresults = await orchestrator.orchestrate_constellation(\n constellation=my_constellation,\n assignment_strategy=\"capability_match\"\n)\n```\n\n### With Modification Synchronizer\n\n```python\nfrom galaxy.session.observers.constellation_sync_observer import (\n ConstellationModificationSynchronizer\n)\n\n# Create synchronizer\nsynchronizer = ConstellationModificationSynchronizer(orchestrator)\norchestrator.set_modification_synchronizer(synchronizer)\n\n# Subscribe to events\nevent_bus.subscribe(synchronizer)\n\n# Now orchestrator will wait for LLM edits to complete\n```\n\n## Related Documentation\n\n- **[Event-Driven Coordination](event_driven_coordination.md)** - Event system and observer pattern\n- **[Asynchronous Scheduling](asynchronous_scheduling.md)** - Concurrent task execution\n- **[Safe Assignment Locking](safe_assignment_locking.md)** - Race condition prevention\n- **[Consistency Guarantees](consistency_guarantees.md)** - Runtime invariants\n- **[Batched Editing](batched_editing.md)** - Efficient constellation updates\n- **[Constellation Manager](constellation_manager.md)** - Device and resource management\n- **[API Reference](api_reference.md)** - Complete API documentation\n\n## Further Reading\n\n- [TaskConstellation Documentation](../constellation/overview.md) - Understand the DAG structure\n- [Constellation Agent](../constellation_agent/overview.md) - LLM-based reasoning\n- [Device Manager](../client/device_manager.md) - Device communication layer\n\n---\n\n!!!tip \"Next Steps\"\n To understand how events drive orchestration, continue to [Event-Driven Coordination](event_driven_coordination.md).\n"
},
{
"path": "documents/docs/galaxy/constellation_orchestrator/safe_assignment_locking.md",
"content": "# Safe Assignment Locking\n\n## Overview\n\nWhile asynchronous execution maximizes efficiency, it introduces correctness challenges when task execution overlaps with DAG updates. The orchestrator must prevent race conditions where the Constellation Agent dynamically adds, removes, or rewires tasks during execution.\n\nWithout safeguards, a task could be dispatched based on a stale DAG, leading to duplicated execution, missed dependencies, or invalid state transitions.\n\nTo ensure atomicity, the orchestrator employs a safe assignment lock protocol combined with constellation state synchronization.\n\n\n\n*An example of the safe assignment locking and event synchronization workflow. When multiple tasks complete simultaneously, the orchestrator locks assignments, batches modifications, and releases after synchronization.*\n\nLearn more about how this integrates with [asynchronous scheduling](asynchronous_scheduling.md) and [batched editing](batched_editing.md).\n\n## The Race Condition Problem\n\n### Scenario Without Locking\n\nConsider this problematic sequence:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant C as Constellation\n participant A as Agent\n \n Note over O: Task A completes\n O->>C: mark_task_completed(A)\n O->>A: Trigger editing\n \n par Agent modifies DAG\n A->>C: Remove Task B\n A->>C: Add Task B'\n and Orchestrator continues\n O->>C: get_ready_tasks()\n C-->>O: [Task B, Task C]\n Note over O: Task B dispatched (but removed!)\n end\n \n Note over O,A: Task B' never executed\n```\n\n**Problems:**\n\n1. **Stale dispatch**: Task B dispatched after being removed\n2. **Missing tasks**: Task B' never identified as ready\n3. **Inconsistent state**: Constellation doesn't reflect actual execution\n\n### Root Cause\n\nThe orchestrator's scheduling loop and agent's editing process are **concurrent and unsynchronized**:\n\n```python\n# Orchestrator loop (simplified)\nwhile not constellation.is_complete():\n ready_tasks = constellation.get_ready_tasks() # ← May see stale state\n await schedule_ready_tasks(ready_tasks) # ← Dispatch based on stale view\n await wait_for_task_completion()\n```\n\nMeanwhile, the agent modifies the same constellation object concurrently.\n\n## Safe Assignment Lock Protocol\n\n### The Solution\n\nThe orchestrator uses a lock-bounded editing regime: during edit cycles, new task assignments are suspended until modifications are complete and synchronized.\n\n```python\nasync def wait_for_pending_modifications(\n self, timeout: Optional[float] = None\n) -> bool:\n \"\"\"Wait for all pending modifications to complete.\"\"\"\n \n if not self._pending_modifications:\n return True\n \n timeout = timeout or self._modification_timeout\n start_time = asyncio.get_event_loop().time()\n \n try:\n while self._pending_modifications:\n # Get current pending tasks\n pending_tasks = list(self._pending_modifications.keys())\n pending_futures = list(self._pending_modifications.values())\n \n self.logger.info(\n f\"⏳ Waiting for {len(pending_tasks)} pending modification(s): {pending_tasks}\"\n )\n \n # Calculate remaining timeout\n elapsed = asyncio.get_event_loop().time() - start_time\n remaining_timeout = timeout - elapsed\n \n if remaining_timeout <= 0:\n raise asyncio.TimeoutError()\n \n # Wait for all current pending modifications\n await asyncio.wait_for(\n asyncio.gather(*pending_futures, return_exceptions=True),\n timeout=remaining_timeout,\n )\n \n # Check if new modifications were added during the wait\n if not self._pending_modifications:\n break\n \n # Small delay to allow new registrations to settle\n await asyncio.sleep(0.01)\n \n self.logger.info(\"✅ All pending modifications completed\")\n return True\n \n except asyncio.TimeoutError:\n pending = list(self._pending_modifications.keys())\n self.logger.warning(\n f\"⚠️ Timeout waiting for modifications after {timeout}s. \"\n f\"Proceeding anyway. Pending: {pending}\"\n )\n # Clear all pending modifications to prevent permanent deadlock\n self._pending_modifications.clear()\n return False\n```\n\n### Edit Cycle Lifecycle\n\nAn edit cycle is bounded by two events:\n\n1. **Start**: `TASK_COMPLETED` or `TASK_FAILED` event published\n2. **End**: `CONSTELLATION_MODIFIED` event published\n\n```mermaid\nstateDiagram-v2\n [*] --> Normal_Execution\n Normal_Execution --> Edit_Cycle_Started: TASK_COMPLETED/FAILED\n Edit_Cycle_Started --> Editing_In_Progress: Register pending\n Editing_In_Progress --> Edit_Cycle_Complete: CONSTELLATION_MODIFIED\n Edit_Cycle_Complete --> Normal_Execution: Merge & release\n Normal_Execution --> [*]\n```\n\nDuring the editing phase, new task assignments are suspended to prevent race conditions.\n\n### Safe Locking Protocol\n\nThe complete protocol ensures atomic constellation updates:\n\n```\nAlgorithm: Safe Assignment Locking and Asynchronous Rescheduling Protocol\n\nInput: Event stream E, current TaskConstellation C\nOutput: Consistent and updated C with newly scheduled ready tasks\n\nwhile system is running do\n foreach event e ∈ E do\n if e is TASK_COMPLETED or TASK_FAILED then\n async enqueue(e) // Record for processing\n end\n end\n \n acquire(assign_lock) // Suspend new assignments\n \n while queue not empty do\n e ← dequeue()\n Δ ← invoke(ConstellationAgent, edit(C, e)) // Propose DAG edits\n C ← apply(C, Δ) // Update structure\n validate(C) // Ensure invariants I1-I3\n publish(CONSTELLATION_MODIFIED, t)\n C ← synchronize(C, T_C) // Merge completed tasks\n end\n \n release(assign_lock) // Resume orchestration\n \n // Rescheduling Phase (outside lock)\n T_R ← get_ready_tasks(C)\n foreach t ∈ T_R do\n async dispatch(t)\n async publish(TASK_STARTED, t)\n end\nend\n```\n\n**Key properties:**\n\n- **Atomicity**: All edits within a queue batch are applied together\n- **Validation**: Constellation consistency checked before releasing\n- **Synchronization**: Runtime progress merged before rescheduling\n- **Non-blocking**: Lock only held during modification, not execution\n\n## Modification Synchronizer\n\nThe `ConstellationModificationSynchronizer` component implements the locking protocol by coordinating between the orchestrator and agent.\n\n### Tracking Pending Modifications\n\nWhen a task completes, the synchronizer registers a pending modification:\n\n```python\nasync def _handle_task_event(self, event: TaskEvent) -> None:\n \"\"\"Handle task completion/failure events.\"\"\"\n \n if event.event_type not in [EventType.TASK_COMPLETED, EventType.TASK_FAILED]:\n return\n \n constellation_id = event.data.get(\"constellation_id\")\n if not constellation_id:\n return\n \n # Register pending modification\n if event.task_id not in self._pending_modifications:\n modification_future = asyncio.Future()\n self._pending_modifications[event.task_id] = modification_future\n self._stats[\"total_modifications\"] += 1\n \n self.logger.info(\n f\"🔒 Registered pending modification for task '{event.task_id}'\"\n )\n \n # Set timeout to auto-complete if modification takes too long\n asyncio.create_task(\n self._auto_complete_on_timeout(event.task_id, modification_future)\n )\n```\n\n**Data structure:**\n\n```python\n# task_id -> Future mapping\nself._pending_modifications: Dict[str, asyncio.Future] = {}\n```\n\nEach future represents an edit cycle that will be completed when `CONSTELLATION_MODIFIED` is received.\n\n### Completing Modifications\n\nWhen the agent publishes `CONSTELLATION_MODIFIED`, the synchronizer completes the future:\n\n```python\nasync def _handle_constellation_event(self, event: ConstellationEvent) -> None:\n \"\"\"Handle constellation modification events.\"\"\"\n \n if event.event_type != EventType.CONSTELLATION_MODIFIED:\n return\n \n task_ids = event.data.get(\"on_task_id\")\n if not task_ids:\n return\n \n new_constellation = event.data.get(\"new_constellation\")\n if new_constellation:\n self._current_constellation = new_constellation\n \n # Mark modifications as complete\n for task_id in task_ids:\n if task_id in self._pending_modifications:\n future = self._pending_modifications[task_id]\n if not future.done():\n future.set_result(True) # Unblocks wait_for_pending_modifications\n self._stats[\"completed_modifications\"] += 1\n self.logger.info(\n f\"✅ Completed modification for task '{task_id}'\"\n )\n del self._pending_modifications[task_id]\n```\n\n### Timeout Protection\n\nTo prevent deadlocks if the agent fails to publish `CONSTELLATION_MODIFIED`:\n\n```python\nasync def _auto_complete_on_timeout(\n self, task_id: str, future: asyncio.Future\n) -> None:\n \"\"\"Auto-complete a pending modification if it times out.\"\"\"\n \n try:\n await asyncio.sleep(self._modification_timeout) # Default: 600s\n \n if not future.done():\n self._stats[\"timeout_modifications\"] += 1\n self.logger.warning(\n f\"⚠️ Modification for task '{task_id}' timed out. \"\n f\"Auto-completing to prevent deadlock.\"\n )\n future.set_result(False)\n if task_id in self._pending_modifications:\n del self._pending_modifications[task_id]\n except asyncio.CancelledError:\n raise\n```\n\n**Warning:** Timeout protection ensures the orchestrator never permanently blocks, even if the agent encounters an error.\n\n## Constellation State Merging\n\nAfter modifications complete, the synchronizer must merge two potentially conflicting views:\n\n1. **Agent's constellation**: Has latest structural changes (new tasks, modified dependencies)\n2. **Orchestrator's constellation**: Has latest execution state (task statuses, results)\n\n### The Challenge\n\nDuring editing, tasks may complete:\n\n```\nt0: Task A completes → Agent starts editing\nt1: Agent modifies constellation (Task A still RUNNING in agent's copy)\nt2: Task B completes (orchestrator marks as COMPLETED)\nt3: Agent publishes CONSTELLATION_MODIFIED\nt4: Orchestrator syncs...\n```\n\n**Problem**: Direct replacement would lose Task B's COMPLETED status!\n\n### State Merging Algorithm\n\nThe synchronizer preserves the most advanced state for each task:\n\n```python\ndef merge_and_sync_constellation_states(\n self, orchestrator_constellation: TaskConstellation\n) -> TaskConstellation:\n \"\"\"Merge constellation states: structural changes + execution state.\"\"\"\n \n if not self._current_constellation:\n return orchestrator_constellation\n \n # Use agent's constellation as base (has structural modifications)\n merged = self._current_constellation\n \n # Preserve execution state from orchestrator for existing tasks\n for task_id, orchestrator_task in orchestrator_constellation.tasks.items():\n if task_id in merged.tasks:\n agent_task = merged.tasks[task_id]\n \n # If orchestrator's state is more advanced, preserve it\n if self._is_state_more_advanced(\n orchestrator_task.status, agent_task.status\n ):\n # Preserve orchestrator's state and results\n agent_task._status = orchestrator_task.status\n agent_task._result = orchestrator_task.result\n agent_task._error = orchestrator_task.error\n agent_task._execution_start_time = orchestrator_task.execution_start_time\n agent_task._execution_end_time = orchestrator_task.execution_end_time\n \n # Update constellation state\n merged.update_state()\n \n return merged\n```\n\n### State Advancement Hierarchy\n\nStates are ordered by execution progression:\n\n```python\ndef _is_state_more_advanced(self, state1, state2) -> bool:\n \"\"\"Check if state1 is more advanced than state2.\"\"\"\n \n state_levels = {\n TaskStatus.PENDING: 0,\n TaskStatus.WAITING_DEPENDENCY: 1,\n TaskStatus.RUNNING: 2,\n TaskStatus.COMPLETED: 3,\n TaskStatus.FAILED: 3, # Terminal states equally advanced\n TaskStatus.CANCELLED: 3,\n }\n \n level1 = state_levels.get(state1, 0)\n level2 = state_levels.get(state2, 0)\n \n return level1 > level2\n```\n\n**Examples:**\n\n- `COMPLETED > RUNNING`: Preserve orchestrator's COMPLETED status\n- `FAILED > PENDING`: Preserve orchestrator's FAILED status\n- `RUNNING > PENDING`: Preserve orchestrator's RUNNING status\n- `COMPLETED = FAILED`: Both terminal, don't override\n\nState merging ensures no execution progress is lost during concurrent editing.\n\n## Synchronization in Orchestration Loop\n\nThe orchestrator syncs at the start of each iteration:\n\n```python\nasync def _sync_constellation_modifications(\n self, constellation: TaskConstellation\n) -> TaskConstellation:\n \"\"\"Synchronize pending constellation modifications.\"\"\"\n \n if self._modification_synchronizer:\n # Wait for agent to finish any pending edits\n await self._modification_synchronizer.wait_for_pending_modifications()\n \n # Merge agent's structural changes with orchestrator's execution state\n constellation = self._modification_synchronizer \\\n .merge_and_sync_constellation_states(\n orchestrator_constellation=constellation,\n )\n \n return constellation\n```\n\nThe synchronization flow ensures the orchestrator always works with the latest merged state that includes both structural changes from the agent and execution progress from the orchestrator.\n\n## Batched Event Processing\n\nWhen multiple tasks complete simultaneously, their modifications are batched:\n\n```python\n# Process ALL pending modifications in one cycle\nwhile self._pending_modifications:\n # Wait for all to complete\n ...\n```\n\n**Timeline with batching:**\n\n```\nt0: Task A completes → enqueue(A)\nt3: Task B completes → enqueue(B)\nt4: Task C completes → enqueue(C)\n\nt5: acquire(lock)\nt6: Process A → Δ_A\nt7: Process B → Δ_B\nt8: Process C → Δ_C\nt9: Apply all Δs atomically\nt10: release(lock)\n```\n\n**Benefits:**\n\n- **Reduced overhead**: One lock acquisition for multiple edits\n- **Atomicity**: All modifications visible together\n- **Efficiency**: Amortize validation and synchronization costs\n\nLearn more about [batched editing strategies](batched_editing.md).\n\n## Correctness Properties\n\nThe safe assignment lock protocol guarantees:\n\n**1. Atomicity**: Edit cycles are atomic - either all modifications in a batch are applied, or none are. Lock held during entire edit-validate-sync sequence.\n\n**2. Consistency**: Constellation always satisfies invariants after edits. Validation performed before releasing. See [consistency guarantees](consistency_guarantees.md) for details.\n\n**3. Progress**: The system never permanently blocks (liveness). Ensured by timeout protection (600s default), auto-completion on timeout, and exception handling in observers.\n\n## Performance Impact\n\n### Lock Overhead\n\n| Scenario | Lock Duration | Impact |\n|----------|--------------|---------|\n| Single task completion | 10-50ms | Negligible - concurrent tasks unaffected |\n| Batched completions | 50-200ms | Amortized over multiple edits |\n| Complex editing | 200-500ms | Depends on LLM response time |\n\n### Throughput Analysis\n\nThe lock does not block task execution - while the lock is held for constellation modification, already-dispatched tasks continue executing concurrently.\n\n**Impact on throughput**: Minimal - only affects scheduling of new tasks, not execution of running tasks.\n\n### Latency Analysis\n\nAdditional latency per task completion:\n\n- Without synchronizer: ~5ms (direct scheduling)\n- With synchronizer: ~10-50ms (wait for edit + merge)\n\nThis is an acceptable tradeoff for correctness in dynamic orchestration.\n\n## Usage Patterns\n\n### Setting Up Synchronization\n\n```python\nfrom galaxy.constellation.orchestrator import TaskConstellationOrchestrator\nfrom galaxy.session.observers.constellation_sync_observer import (\n ConstellationModificationSynchronizer\n)\n\n# Create orchestrator\norchestrator = TaskConstellationOrchestrator(device_manager)\n\n# Create and attach synchronizer\nsynchronizer = ConstellationModificationSynchronizer(orchestrator)\norchestrator.set_modification_synchronizer(synchronizer)\n\n# Subscribe to events\nfrom galaxy.core.events import get_event_bus\nevent_bus = get_event_bus()\nevent_bus.subscribe(synchronizer)\n\n# Orchestrate with synchronization\nresults = await orchestrator.orchestrate_constellation(constellation)\n```\n\n### Custom Timeout\n\n```python\n# Increase timeout for slow LLM responses\nsynchronizer.set_modification_timeout(1200.0) # 20 minutes\n```\n\n### Monitoring Synchronization\n\n```python\n# Check pending modifications\nif synchronizer.has_pending_modifications():\n pending = synchronizer.get_pending_task_ids()\n print(f\"Waiting for modifications: {pending}\")\n\n# Get statistics\nstats = synchronizer.get_statistics()\nprint(f\"Total: {stats['total_modifications']}\")\nprint(f\"Completed: {stats['completed_modifications']}\")\nprint(f\"Timeouts: {stats['timeout_modifications']}\")\n```\n\n## Related Documentation\n\n- [Asynchronous Scheduling](asynchronous_scheduling.md) - Concurrent execution model\n- [Consistency Guarantees](consistency_guarantees.md) - Invariants enforced by locking\n- [Batched Editing](batched_editing.md) - Efficient modification batching\n- [Event-Driven Coordination](event_driven_coordination.md) - Event system foundation\n"
},
{
"path": "documents/docs/galaxy/evaluation/performance_metrics.md",
"content": "# Performance Metrics and Logging\n\nGalaxy provides comprehensive performance monitoring and metrics collection throughout multi-device workflow execution. The system tracks task execution times, constellation modifications, and overall session metrics to enable analysis and optimization of distributed workflows.\n\n## Overview\n\nGalaxy uses an **event-driven observer pattern** to collect real-time performance metrics without impacting execution flow. The `SessionMetricsObserver` automatically captures timing data, task statistics, constellation modifications, and parallelism metrics.\n\n### Key Metrics Categories\n\n| Category | Description | Use Cases |\n|----------|-------------|-----------|\n| **Task Metrics** | Individual task execution times and outcomes | Identify slow tasks, success rates |\n| **Constellation Metrics** | DAG-level statistics and parallelism analysis | Optimize workflow structure |\n| **Modification Metrics** | Dynamic constellation editing during execution | Understand adaptability patterns |\n| **Session Metrics** | Overall session duration and resource usage | End-to-end performance analysis |\n\n## Metrics Collection System\n\n### SessionMetricsObserver\n\nThe `SessionMetricsObserver` is automatically initialized for every Galaxy session and listens to events from the orchestration system.\n\n**Architecture:**\n\n```mermaid\ngraph LR\n A[Task Execution] -->|Task Events| B[SessionMetricsObserver]\n C[Constellation Operations] -->|Constellation Events| B\n B -->|Collect & Aggregate| D[Metrics Dictionary]\n D -->|Save on Completion| E[result.json]\n \n style B fill:#e1f5ff\n style D fill:#fff4e1\n style E fill:#c8e6c9\n```\n\n**Event Types Tracked:**\n\n| Event Type | Trigger | Metrics Captured |\n|-----------|---------|------------------|\n| `TASK_STARTED` | Task begins execution | Start timestamp, task count |\n| `TASK_COMPLETED` | Task finishes successfully | Duration, end timestamp |\n| `TASK_FAILED` | Task encounters error | Duration, failure count |\n| `CONSTELLATION_STARTED` | New DAG created | Initial statistics, task count |\n| `CONSTELLATION_COMPLETED` | DAG fully executed | Final statistics, total duration |\n| `CONSTELLATION_MODIFIED` | DAG edited during execution | Changes, modification type |\n\n---\n\n## Collected Metrics\n\n### 1. Task Metrics\n\n**Raw Task Data:**\n\n```python\n{\n \"task_timings\": {\n \"t1\": {\n \"start\": 1761388508.9484463,\n \"duration\": 11.852121591567993,\n \"end\": 1761388520.8005679\n },\n \"t2\": {\n \"start\": 1761388508.9494512,\n \"duration\": 12.128723621368408,\n \"end\": 1761388521.0781748\n },\n # ... more tasks\n }\n}\n```\n\n**Computed Task Statistics:**\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `total_tasks` | int | Total number of tasks created | `5` |\n| `completed_tasks` | int | Successfully completed tasks | `5` |\n| `failed_tasks` | int | Failed tasks | `0` |\n| `success_rate` | float | Completion rate (0.0-1.0) | `1.0` |\n| `failure_rate` | float | Failure rate (0.0-1.0) | `0.0` |\n| `average_task_duration` | float | Mean task execution time (seconds) | `134.91` |\n| `min_task_duration` | float | Fastest task duration | `11.85` |\n| `max_task_duration` | float | Slowest task duration | `369.05` |\n| `total_task_execution_time` | float | Sum of all task durations | `674.55` |\n\n### 2. Constellation Metrics\n\n**Raw Constellation Data:**\n\n```python\n{\n \"constellation_timings\": {\n \"constellation_b0864385_20251025_183508\": {\n \"start_time\": 1761388508.9061587,\n \"initial_statistics\": {\n \"total_tasks\": 5,\n \"total_dependencies\": 4,\n \"longest_path_length\": 2,\n \"max_width\": 4,\n \"parallelism_ratio\": 2.5\n },\n \"processing_start_time\": 1761388493.1049807,\n \"processing_end_time\": 1761388508.9061587,\n \"processing_duration\": 15.801177978515625,\n \"end_time\": 1761389168.8877504,\n \"duration\": 659.9815917015076,\n \"final_statistics\": {\n \"total_tasks\": 5,\n \"task_status_counts\": {\n \"completed\": 5\n },\n \"critical_path_length\": 638.134632,\n \"total_work\": 674.4709760000001,\n \"parallelism_ratio\": 1.0569415013350976\n }\n }\n }\n}\n```\n\n**Computed Constellation Statistics:**\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `total_constellations` | int | Number of DAGs created | `1` |\n| `completed_constellations` | int | Successfully completed DAGs | `1` |\n| `failed_constellations` | int | Failed DAGs | `0` |\n| `success_rate` | float | Completion rate | `1.0` |\n| `average_constellation_duration` | float | Mean DAG execution time | `659.98` |\n| `min_constellation_duration` | float | Fastest DAG completion | `659.98` |\n| `max_constellation_duration` | float | Slowest DAG completion | `659.98` |\n| `average_tasks_per_constellation` | float | Mean tasks per DAG | `5.0` |\n\n**Key Constellation Metrics:**\n\n| Metric | Description | Formula | Interpretation |\n|--------|-------------|---------|----------------|\n| **Critical Path Length** | Duration of longest task chain | `max(path_durations)` | Minimum possible execution time |\n| **Total Work** | Sum of all task durations | `Σ task_durations` | Total computational effort |\n| **Parallelism Ratio** | Efficiency of parallel execution | `total_work / critical_path_length` | >1.0 indicates parallelism benefit |\n| **Max Width** | Maximum concurrent tasks | `max(concurrent_tasks_at_time_t)` | Peak resource utilization |\n\n!!! note \"Parallelism Calculation Modes\"\n The system uses two calculation modes:\n \n - **`node_count`**: Used when tasks are incomplete. Uses task count and path length.\n - **`actual_time`**: Used when all tasks are completed. Uses real execution times for accurate parallelism analysis.\n\n**Example from result.json:**\n\n```json\n{\n \"critical_path_length\": 638.134632,\n \"total_work\": 674.4709760000001,\n \"parallelism_ratio\": 1.0569415013350976\n}\n```\n\n**Analysis:** Parallelism ratio of `1.057` indicates minimal parallelism benefit (5.7% reduction in execution time). This suggests most tasks executed sequentially due to dependencies.\n\n### 3. Constellation Modification Metrics\n\n**Modification Records:**\n\n```python\n{\n \"constellation_modifications\": {\n \"constellation_b0864385_20251025_183508\": [\n {\n \"timestamp\": 1761388539.3350308,\n \"modification_type\": \"Edited by constellation_agent\",\n \"on_task_id\": [\"t1\"],\n \"changes\": {\n \"modification_type\": \"task_properties_updated\",\n \"added_tasks\": [],\n \"removed_tasks\": [],\n \"modified_tasks\": [\"t5\", \"t3\"],\n \"added_dependencies\": [],\n \"removed_dependencies\": []\n },\n \"new_statistics\": {\n \"total_tasks\": 5,\n \"task_status_counts\": {\n \"completed\": 2,\n \"running\": 2,\n \"pending\": 1\n }\n },\n \"processing_start_time\": 1761388521.482895,\n \"processing_end_time\": 1761388537.9989598,\n \"processing_duration\": 16.516064882278442\n }\n # ... more modifications\n ]\n }\n}\n```\n\n**Computed Modification Statistics:**\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `total_modifications` | int | Total constellation edits | `4` |\n| `constellations_modified` | int | Number of DAGs modified | `1` |\n| `average_modifications_per_constellation` | float | Mean edits per DAG | `4.0` |\n| `max_modifications_for_single_constellation` | int | Most edits to one DAG | `4` |\n| `most_modified_constellation` | str | Constellation ID with most edits | `constellation_...` |\n| `modifications_per_constellation` | dict | Edit count per DAG | `{\"constellation_...\": 4}` |\n| `modification_types_breakdown` | dict | Count by modification type | `{\"Edited by constellation_agent\": 4}` |\n\n**Modification Types:**\n\n| Type | Description | Trigger |\n|------|-------------|---------|\n| `Edited by constellation_agent` | ConstellationAgent refined DAG | Task completion, feedback |\n| `task_properties_updated` | Task details modified | Result refinement |\n| `constellation_updated` | DAG structure changed | Dependency updates |\n| `tasks_added` | New tasks inserted | Workflow expansion |\n| `tasks_removed` | Tasks deleted | Optimization |\n\n## Session Results Structure\n\nThe complete session results are saved to `logs/galaxy//result.json` with the following structure:\n\n```json\n{\n \"session_name\": \"galaxy_session_20251025_183449\",\n \"request\": \"User's original request text\",\n \"task_name\": \"task_32\",\n \"status\": \"completed\",\n \"execution_time\": 684.864645,\n \"rounds\": 1,\n \"start_time\": \"2025-10-25T18:34:52.641877\",\n \"end_time\": \"2025-10-25T18:46:17.506522\",\n \"trajectory_path\": \"logs/galaxy/task_32/\",\n \n \"session_results\": {\n \"total_execution_time\": 684.8532314300537,\n \"final_constellation_stats\": { /* ... */ },\n \"status\": \"FINISH\",\n \"final_results\": [ /* ... */ ],\n \"metrics\": { /* ... */ }\n },\n \n \"constellation\": {\n \"id\": \"constellation_b0864385_20251025_183508\",\n \"name\": \"constellation_b0864385_20251025_183508\",\n \"task_count\": 5,\n \"dependency_count\": 4,\n \"state\": \"completed\"\n }\n}\n```\n\n**Top-Level Fields:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `session_name` | str | Unique session identifier |\n| `request` | str | Original user request |\n| `task_name` | str | Task identifier |\n| `status` | str | Session outcome: `\"completed\"`, `\"failed\"`, `\"timeout\"` |\n| `execution_time` | float | Total session duration (seconds) |\n| `rounds` | int | Number of orchestration rounds |\n| `start_time` | str | ISO 8601 session start timestamp |\n| `end_time` | str | ISO 8601 session end timestamp |\n| `trajectory_path` | str | Path to session logs |\n\n## Performance Analysis\n\n### Reading Metrics Programmatically\n\n```python\nimport json\nfrom pathlib import Path\n\ndef analyze_session_performance(result_path: str):\n \"\"\"\n Analyze Galaxy session performance from result.json.\n \n :param result_path: Path to result.json file\n \"\"\"\n with open(result_path, 'r', encoding='utf-8') as f:\n result = json.load(f)\n \n metrics = result[\"session_results\"][\"metrics\"]\n \n # Task performance\n task_stats = metrics[\"task_statistics\"]\n print(f\"✅ Tasks completed: {task_stats['completed_tasks']}/{task_stats['total_tasks']}\")\n print(f\"⏱️ Average task duration: {task_stats['average_task_duration']:.2f}s\")\n print(f\"📊 Success rate: {task_stats['success_rate'] * 100:.1f}%\")\n \n # Constellation performance\n const_stats = metrics[\"constellation_statistics\"]\n print(f\"\\n🌌 Constellations: {const_stats['completed_constellations']}/{const_stats['total_constellations']}\")\n print(f\"⏱️ Average constellation duration: {const_stats['average_constellation_duration']:.2f}s\")\n \n # Parallelism analysis\n final_stats = result[\"session_results\"][\"final_constellation_stats\"]\n parallelism = final_stats.get(\"parallelism_ratio\", 1.0)\n print(f\"\\n🔀 Parallelism ratio: {parallelism:.2f}\")\n \n if parallelism > 1.5:\n print(\" → High parallelism: tasks executed concurrently\")\n elif parallelism > 1.0:\n print(\" → Moderate parallelism: some concurrent execution\")\n else:\n print(\" → Sequential execution: limited parallelism\")\n \n # Modification analysis\n mod_stats = metrics[\"modification_statistics\"]\n print(f\"\\n✏️ Total modifications: {mod_stats['total_modifications']}\")\n print(f\" Average per constellation: {mod_stats['average_modifications_per_constellation']:.1f}\")\n \n return metrics\n\n# Example usage\nmetrics = analyze_session_performance(\"logs/galaxy/task_32/result.json\")\n```\n\n**Expected Output:**\n\n```\n✅ Tasks completed: 5/5\n⏱️ Average task duration: 134.91s\n📊 Success rate: 100.0%\n\n🌌 Constellations: 1/1\n⏱️ Average constellation duration: 659.98s\n\n🔀 Parallelism ratio: 1.06\n → Sequential execution: limited parallelism\n\n✏️ Total modifications: 4\n Average per constellation: 4.0\n```\n\n### Identifying Performance Bottlenecks\n\n```python\ndef identify_bottlenecks(result_path: str):\n \"\"\"\n Identify performance bottlenecks from session metrics.\n \n :param result_path: Path to result.json file\n \"\"\"\n with open(result_path, 'r', encoding='utf-8') as f:\n result = json.load(f)\n \n metrics = result[\"session_results\"][\"metrics\"]\n task_timings = metrics[\"task_timings\"]\n task_stats = metrics[\"task_statistics\"]\n \n # Find slowest tasks\n avg_duration = task_stats[\"average_task_duration\"]\n threshold = avg_duration * 2 # 2x average = bottleneck\n \n bottlenecks = []\n for task_id, timing in task_timings.items():\n if \"duration\" in timing and timing[\"duration\"] > threshold:\n bottlenecks.append({\n \"task_id\": task_id,\n \"duration\": timing[\"duration\"],\n \"factor\": timing[\"duration\"] / avg_duration\n })\n \n if bottlenecks:\n print(\"⚠️ Performance Bottlenecks Detected:\")\n for task in sorted(bottlenecks, key=lambda x: x[\"duration\"], reverse=True):\n print(f\" • {task['task_id']}: {task['duration']:.2f}s ({task['factor']:.1f}x average)\")\n else:\n print(\"✅ No significant bottlenecks detected\")\n \n return bottlenecks\n\n# Example usage\nbottlenecks = identify_bottlenecks(\"logs/galaxy/task_32/result.json\")\n```\n\n**Example Output:**\n\n```\n⚠️ Performance Bottlenecks Detected:\n • t5: 369.05s (2.7x average)\n • t4: 269.11s (2.0x average)\n```\n\n### Visualizing Task Timeline\n\n```python\nimport matplotlib.pyplot as plt\nfrom datetime import datetime\n\ndef visualize_task_timeline(result_path: str):\n \"\"\"\n Visualize task execution timeline.\n \n :param result_path: Path to result.json file\n \"\"\"\n with open(result_path, 'r', encoding='utf-8') as f:\n result = json.load(f)\n \n metrics = result[\"session_results\"][\"metrics\"]\n task_timings = metrics[\"task_timings\"]\n \n # Prepare data\n tasks = []\n for task_id, timing in task_timings.items():\n if \"start\" in timing and \"end\" in timing:\n tasks.append({\n \"task_id\": task_id,\n \"start\": timing[\"start\"],\n \"end\": timing[\"end\"],\n \"duration\": timing[\"duration\"]\n })\n \n # Sort by start time\n tasks.sort(key=lambda x: x[\"start\"])\n \n # Create Gantt chart\n fig, ax = plt.subplots(figsize=(12, 6))\n \n for i, task in enumerate(tasks):\n start_offset = task[\"start\"] - tasks[0][\"start\"]\n ax.barh(i, task[\"duration\"], left=start_offset, height=0.5)\n ax.text(start_offset + task[\"duration\"] / 2, i, \n f\"{task['task_id']} ({task['duration']:.1f}s)\", \n ha='center', va='center')\n \n ax.set_yticks(range(len(tasks)))\n ax.set_yticklabels([t[\"task_id\"] for t in tasks])\n ax.set_xlabel(\"Time (seconds)\")\n ax.set_title(\"Task Execution Timeline\")\n ax.grid(axis='x', alpha=0.3)\n \n plt.tight_layout()\n plt.savefig(\"task_timeline.png\")\n print(\"📊 Timeline saved to task_timeline.png\")\n\n# Example usage\nvisualize_task_timeline(\"logs/galaxy/task_32/result.json\")\n```\n\n---\n\n## Optimization Strategies\n\n### 1. Improve Parallelism\n\n**Goal:** Increase parallelism ratio by reducing dependencies\n\n```python\n# Analyze dependency structure\ndef analyze_dependencies(result_path: str):\n with open(result_path, 'r', encoding='utf-8') as f:\n result = json.load(f)\n \n final_stats = result[\"session_results\"][\"final_constellation_stats\"]\n \n max_width = final_stats[\"max_width\"]\n total_tasks = final_stats[\"total_tasks\"]\n parallelism = final_stats[\"parallelism_ratio\"]\n \n print(f\"Current parallelism: {parallelism:.2f}\")\n print(f\"Max concurrent tasks: {max_width}/{total_tasks}\")\n \n if parallelism < 1.5:\n print(\"\\n💡 Recommendations:\")\n print(\" • Reduce task dependencies where possible\")\n print(\" • Break large sequential tasks into parallel subtasks\")\n print(\" • Use more device agents for concurrent execution\")\n\n# Example usage\nanalyze_dependencies(\"logs/galaxy/task_32/result.json\")\n```\n\n### 2. Reduce Task Duration**Goal:** Optimize slow tasks identified as bottlenecks\n\n```python\n# Generate optimization report\ndef generate_optimization_report(result_path: str):\n with open(result_path, 'r', encoding='utf-8') as f:\n result = json.load(f)\n \n metrics = result[\"session_results\"][\"metrics\"]\n task_stats = metrics[\"task_statistics\"]\n \n avg_duration = task_stats[\"average_task_duration\"]\n max_duration = task_stats[\"max_task_duration\"]\n \n potential_savings = max_duration - avg_duration\n \n print(f\"📈 Optimization Potential:\")\n print(f\" Current slowest task: {max_duration:.2f}s\")\n print(f\" Average task duration: {avg_duration:.2f}s\")\n print(f\" Potential time savings: {potential_savings:.2f}s ({potential_savings/max_duration*100:.1f}%)\")\n\n# Example usage\ngenerate_optimization_report(\"logs/galaxy/task_32/result.json\")\n```\n\n### 3. Reduce Constellation Modifications\n\n**Goal:** Minimize dynamic editing overhead\n\n```python\n# Analyze modification overhead\ndef analyze_modification_overhead(result_path: str):\n with open(result_path, 'r', encoding='utf-8') as f:\n result = json.load(f)\n \n metrics = result[\"session_results\"][\"metrics\"]\n modifications = metrics[\"constellation_modifications\"]\n \n total_processing_time = 0\n modification_count = 0\n \n for const_mods in modifications.values():\n for mod in const_mods:\n if \"processing_duration\" in mod:\n total_processing_time += mod[\"processing_duration\"]\n modification_count += 1\n \n if modification_count > 0:\n avg_overhead = total_processing_time / modification_count\n print(f\"✏️ Modification Overhead:\")\n print(f\" Total modifications: {modification_count}\")\n print(f\" Total overhead: {total_processing_time:.2f}s\")\n print(f\" Average per modification: {avg_overhead:.2f}s\")\n \n if modification_count > 10:\n print(\"\\n💡 Recommendations:\")\n print(\" • Provide more detailed initial request\")\n print(\" • Use device capabilities metadata for better planning\")\n\n# Example usage\nanalyze_modification_overhead(\"logs/galaxy/task_32/result.json\")\n```\n\n## Best Practices\n\n### 1. Regular Analysis\n\nAnalyze every session to identify trends:\n\n```python\nfrom pathlib import Path\n\n# Analyze every session to identify trends\nfor session_dir in Path(\"logs/galaxy\").iterdir():\n result_file = session_dir / \"result.json\"\n if result_file.exists():\n analyze_session_performance(str(result_file))\n```\n\n### 2. Baseline Metrics\n\nEstablish baseline performance for common task types:\n\n| Task Type | Baseline Duration | Acceptable Range |\n|-----------|-------------------|------------------|\n| Simple data query | 10-30s | <60s |\n| Document generation | 30-60s | <120s |\n| Multi-device workflow | 60-180s | <300s |\n\n### 3. Track Trends\n\nMonitor performance over time to detect degradation:\n\n```python\nimport pandas as pd\nfrom pathlib import Path\n\ndef track_performance_trends(log_dir: str):\n \"\"\"Track performance metrics over time.\"\"\"\n results = []\n for session_dir in Path(log_dir).iterdir():\n result_file = session_dir / \"result.json\"\n if result_file.exists():\n with open(result_file, 'r') as f:\n data = json.load(f)\n results.append({\n \"session_name\": data[\"session_name\"],\n \"execution_time\": data[\"execution_time\"],\n \"task_count\": data[\"session_results\"][\"metrics\"][\"task_count\"],\n \"parallelism\": data[\"session_results\"][\"final_constellation_stats\"].get(\"parallelism_ratio\", 1.0)\n })\n \n df = pd.DataFrame(results)\n print(df.describe())\n\n# Example usage\ntrack_performance_trends(\"logs/galaxy\")\n```\n\n## Related Documentation\n\n- **[Result JSON Format](./result_json.md)** - Complete result.json schema reference\n- **[Galaxy Overview](../overview.md)** - Main Galaxy framework documentation\n- **[Task Constellation](../constellation/task_constellation.md)** - DAG-based task planning and parallelism metrics\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** - Execution coordination and event handling\n\n## Summary\n\nGalaxy's performance metrics system provides comprehensive monitoring capabilities:\n\n- **Real-time monitoring** - Event-driven metrics collection through `SessionMetricsObserver`\n- **Comprehensive coverage** - Tasks, constellations, and modifications tracking\n- **Parallelism analysis** - Critical path and efficiency metrics with two calculation modes\n- **Bottleneck identification** - Statistical analysis to find performance outliers\n- **Optimization insights** - Data-driven improvement recommendations\n- **Programmatic access** - Structured JSON format for automated analysis\n\nUse these metrics to optimize workflow design, analyze task dependencies, and enhance overall system performance.\n"
},
{
"path": "documents/docs/galaxy/evaluation/result_json.md",
"content": "# Result JSON Format Reference\n\nGalaxy automatically saves comprehensive execution results to `result.json` after each session completes. This file contains the complete execution history, performance metrics, constellation statistics, and final outcomes of multi-device workflows.\n\n## Overview\n\nThe `result.json` file provides a **complete audit trail** and **performance analysis** of Galaxy session execution. It combines session metadata, execution metrics, constellation statistics, and final results into a single structured document.\n\n### File Location\n\n```\nlogs/galaxy//result.json\n```\n\n**Example:**\n\n```\nlogs/galaxy/request_20251111_140216_1/result.json\n```\n\n## File Structure\n\n### Top-Level Schema\n\n```json\n{\n \"session_name\": \"string\", // Unique session identifier\n \"request\": \"string\", // Original user request\n \"task_name\": \"string\", // Task identifier\n \"status\": \"string\", // Session outcome\n \"execution_time\": \"float\", // Total duration (seconds)\n \"rounds\": \"integer\", // Number of orchestration rounds\n \"start_time\": \"string\", // ISO 8601 start timestamp\n \"end_time\": \"string\", // ISO 8601 end timestamp\n \"trajectory_path\": \"string\", // Path to session logs\n \"session_results\": { /* ... */ }, // Detailed execution results\n \"constellation\": { /* ... */ } // Final constellation summary\n}\n```\n\n---\n\n## Field Reference\n\n### Session Metadata\n\n#### `session_name` (string)\n\nUnique identifier for the Galaxy session, generated automatically.\n\n**Format:** `galaxy_session_YYYYMMDD_HHMMSS`\n\n**Example:**\n\n```json\n{\n \"session_name\": \"galaxy_session_20251025_183449\"\n}\n```\n\n#### `request` (string)\n\nThe original natural language request provided by the user.\n\n**Example:**\n\n```json\n{\n \"request\": \"For all linux, get their disk usage statistics. Then, from Windows browser, search for the top 3 recommended ways to reduce high disk usage for Linux systems and document these in a report on notepad.\"\n}\n```\n\n#### `task_name` (string)\n\nInternal task identifier assigned to the session.\n\n**Format:** `task_` or custom name\n\n**Example:**\n\n```json\n{\n \"task_name\": \"task_32\"\n}\n```\n\n#### `status` (string)\n\nFinal session outcome status.\n\n**Possible Values:**\n\n| Status | Description | Meaning |\n|--------|-------------|---------|\n| `\"completed\"` | Session finished successfully | All tasks completed |\n| `\"failed\"` | Session encountered unrecoverable error | Task failure or system error |\n| `\"timeout\"` | Session exceeded time limit | Max execution time reached |\n| `\"cancelled\"` | Session manually stopped by user | User interruption |\n\n**Example:**\n\n```json\n{\n \"status\": \"completed\"\n}\n```\n\n#### `execution_time` (float)\n\nTotal session duration in seconds, from start to completion.\n\n**Example:**\n\n```json\n{\n \"execution_time\": 684.864645\n}\n```\n\n#### `rounds` (integer)\n\nNumber of orchestration rounds executed during the session. Each round represents a full constellation creation or modification cycle.\n\n**Example:**\n\n```json\n{\n \"rounds\": 1\n}\n```\n\n!!! tip \"Understanding Rounds\"\n Multiple rounds indicate a complex request requiring iterative refinement. Most sessions complete in 1-2 rounds.\n\n#### `start_time` (string)\n\nISO 8601 formatted timestamp when the session started.\n\n**Format:** `YYYY-MM-DDTHH:MM:SS.ssssss`\n\n**Example:**\n\n```json\n{\n \"start_time\": \"2025-10-25T18:34:52.641877\"\n}\n```\n\n#### `end_time` (string)\n\nISO 8601 formatted timestamp when the session completed.\n\n**Example:**\n\n```json\n{\n \"end_time\": \"2025-10-25T18:46:17.506522\"\n}\n```\n\n#### `trajectory_path` (string)\n\nFile system path to the directory containing all session logs and artifacts.\n\n**Example:**\n\n```json\n{\n \"trajectory_path\": \"logs/galaxy/request_20251111_140216_1/\"\n}\n```\n\n**Directory Contents:**\n\n```\nlogs/galaxy/request_20251111_140216_1/\n├── result.json # This file\n├── output.md # Trajectory report\n├── response.log # JSONL execution log\n├── request.log # Request details\n├── evaluation.log # Optional evaluation\n└── topology_images/ # DAG visualizations\n └── *.png\n```\n\n### Session Results\n\nThe `session_results` object contains detailed execution information and metrics.\n\n```json\n{\n \"session_results\": {\n \"total_execution_time\": \"float\",\n \"final_constellation_stats\": { /* ... */ },\n \"status\": \"string\",\n \"final_results\": [ /* ... */ ],\n \"metrics\": { /* ... */ }\n }\n}\n```\n\n#### `total_execution_time` (float)\n\nTotal time spent executing tasks (excludes planning/overhead).\n\n**Example:**\n\n```json\n{\n \"total_execution_time\": 684.8532314300537\n}\n```\n\n#### `final_constellation_stats` (object)\n\nStatistics for the final constellation after all tasks completed.\n\n**Schema:**\n\n```json\n{\n \"constellation_id\": \"string\", // Unique constellation ID\n \"name\": \"string\", // Constellation name\n \"state\": \"string\", // \"completed\", \"failed\", \"executing\"\n \"total_tasks\": \"integer\", // Total task count\n \"total_dependencies\": \"integer\", // Dependency count\n \"task_status_counts\": { // Task states\n \"completed\": \"integer\",\n \"failed\": \"integer\",\n \"pending\": \"integer\",\n \"running\": \"integer\"\n },\n \"longest_path_length\": \"integer\", // Max depth (levels)\n \"longest_path_tasks\": [\"string\"], // Task IDs in longest path\n \"max_width\": \"integer\", // Max concurrent tasks\n \"critical_path_length\": \"float\", // Critical path duration (seconds)\n \"total_work\": \"float\", // Sum of all task durations\n \"parallelism_ratio\": \"float\", // total_work / critical_path_length\n \"parallelism_calculation_mode\": \"string\", // \"actual_time\" or \"node_count\"\n \"critical_path_tasks\": [\"string\"], // Task IDs in critical path\n \"execution_duration\": \"float\", // Constellation total duration\n \"created_at\": \"string\", // ISO 8601 creation timestamp\n \"updated_at\": \"string\" // ISO 8601 last update timestamp\n}\n```\n\n**Example:**\n\n```json\n{\n \"final_constellation_stats\": {\n \"constellation_id\": \"constellation_b0864385_20251025_183508\",\n \"name\": \"constellation_b0864385_20251025_183508\",\n \"state\": \"completed\",\n \"total_tasks\": 5,\n \"total_dependencies\": 4,\n \"task_status_counts\": {\n \"completed\": 5\n },\n \"longest_path_length\": 2,\n \"longest_path_tasks\": [\"t1\", \"t5\"],\n \"max_width\": 4,\n \"critical_path_length\": 638.134632,\n \"total_work\": 674.4709760000001,\n \"parallelism_ratio\": 1.0569415013350976,\n \"parallelism_calculation_mode\": \"actual_time\",\n \"critical_path_tasks\": [\"t4\", \"t5\"],\n \"execution_duration\": null,\n \"created_at\": \"2025-10-25T10:35:08.777663+00:00\",\n \"updated_at\": \"2025-10-25T10:46:08.625716+00:00\"\n }\n}\n```\n\n**Key Metrics:**\n\n| Field | Description | Use Case |\n|-------|-------------|----------|\n| `critical_path_length` | Minimum possible execution time | Theoretical performance limit |\n| `total_work` | Total computational effort | Resource utilization |\n| `parallelism_ratio` | Efficiency of parallel execution | Optimization target |\n| `max_width` | Peak concurrent tasks | Capacity planning |\n\n!!! note \"Parallelism Ratio Interpretation\"\n - **1.0**: Sequential execution (no parallelism)\n - **1.5**: 50% time reduction through parallelism\n - **2.0**: 2x speedup from parallel execution\n - **>2.0**: High parallelism efficiency\n\n#### `status` (string)\n\nFinal status from ConstellationAgent.\n\n**Possible Values:**\n\n- `\"FINISH\"`: Successful completion\n- `\"FAIL\"`: Execution failure\n- `\"PENDING\"`: Incomplete (should not appear in final result)\n\n**Example:**\n\n```json\n{\n \"status\": \"FINISH\"\n}\n```\n\n#### `final_results` (array)\n\nArray of result objects containing request-result pairs.\n\n**Schema:**\n\n```json\n{\n \"final_results\": [\n {\n \"request\": \"string\", // User request (may be same as top-level)\n \"result\": \"string\" // Final outcome description\n }\n ]\n}\n```\n\n**Example:**\n\n```json\n{\n \"final_results\": [\n {\n \"request\": \"For all linux, get their disk usage statistics. Then, from Windows browser, search for the top 3 recommended ways to reduce high disk usage for Linux systems and document these in a report on notepad.\",\n \"result\": \"User request fully completed. Final artifact: 'Documents\\\\\\\\Linux_Disk_Usage_Report.txt' on windows_agent, containing full disk usage summaries for linux_agent_1, linux_agent_2, and linux_agent_3, and top 3 recommendations for reducing high disk usage (from Tecmint). All tasks completed successfully; no further constellation updates required.\"\n }\n ]\n}\n```\n\n#### `metrics` (object)\n\nComprehensive performance metrics collected during execution. See **[Performance Metrics](./performance_metrics.md)** for detailed documentation.\n\n**Schema:**\n\n```json\n{\n \"metrics\": {\n \"session_id\": \"string\",\n \"task_count\": \"integer\",\n \"completed_tasks\": \"integer\",\n \"failed_tasks\": \"integer\",\n \"total_execution_time\": \"float\",\n \"task_timings\": { /* ... */ },\n \"constellation_count\": \"integer\",\n \"completed_constellations\": \"integer\",\n \"failed_constellations\": \"integer\",\n \"total_constellation_time\": \"float\",\n \"constellation_timings\": { /* ... */ },\n \"constellation_modifications\": { /* ... */ },\n \"task_statistics\": { /* ... */ },\n \"constellation_statistics\": { /* ... */ },\n \"modification_statistics\": { /* ... */ }\n }\n}\n```\n\n**See:** [Performance Metrics Documentation](./performance_metrics.md)\n\n### Constellation Summary\n\nThe `constellation` object provides a high-level summary of the final constellation.\n\n**Schema:**\n\n```json\n{\n \"constellation\": {\n \"id\": \"string\", // Constellation ID\n \"name\": \"string\", // Constellation name\n \"task_count\": \"integer\", // Total tasks\n \"dependency_count\": \"integer\", // Total dependencies\n \"state\": \"string\" // Final state\n }\n}\n```\n\n**Example:**\n\n```json\n{\n \"constellation\": {\n \"id\": \"constellation_b0864385_20251025_183508\",\n \"name\": \"constellation_b0864385_20251025_183508\",\n \"task_count\": 5,\n \"dependency_count\": 4,\n \"state\": \"completed\"\n }\n}\n```\n\n---\n\n## Complete Example\n\nHere's a complete `result.json` file from an actual Galaxy session:\n\n```json\n{\n \"session_name\": \"galaxy_session_20251025_183449\",\n \"request\": \"For all linux, get their disk usage statistics. Then, from Windows browser, search for the top 3 recommended ways to reduce high disk usage for Linux systems and document these in a report on notepad.\",\n \"task_name\": \"task_32\",\n \"status\": \"completed\",\n \"execution_time\": 684.864645,\n \"rounds\": 1,\n \"start_time\": \"2025-10-25T18:34:52.641877\",\n \"end_time\": \"2025-10-25T18:46:17.506522\",\n \"trajectory_path\": \"logs/galaxy/task_32/\",\n \n \"session_results\": {\n \"total_execution_time\": 684.8532314300537,\n \n \"final_constellation_stats\": {\n \"constellation_id\": \"constellation_b0864385_20251025_183508\",\n \"name\": \"constellation_b0864385_20251025_183508\",\n \"state\": \"completed\",\n \"total_tasks\": 5,\n \"total_dependencies\": 4,\n \"task_status_counts\": {\n \"completed\": 5\n },\n \"longest_path_length\": 2,\n \"longest_path_tasks\": [\"t1\", \"t5\"],\n \"max_width\": 4,\n \"critical_path_length\": 638.134632,\n \"total_work\": 674.4709760000001,\n \"parallelism_ratio\": 1.0569415013350976,\n \"parallelism_calculation_mode\": \"actual_time\",\n \"critical_path_tasks\": [\"t4\", \"t5\"],\n \"execution_duration\": null,\n \"created_at\": \"2025-10-25T10:35:08.777663+00:00\",\n \"updated_at\": \"2025-10-25T10:46:08.625716+00:00\"\n },\n \n \"status\": \"FINISH\",\n \n \"final_results\": [\n {\n \"request\": \"For all linux, get their disk usage statistics. Then, from Windows browser, search for the top 3 recommended ways to reduce high disk usage for Linux systems and document these in a report on notepad.\",\n \"result\": \"User request fully completed. Final artifact: 'Documents\\\\\\\\Linux_Disk_Usage_Report.txt' on windows_agent, containing full disk usage summaries for linux_agent_1, linux_agent_2, and linux_agent_3, and top 3 recommendations for reducing high disk usage (from Tecmint). All tasks completed successfully; no further constellation updates required.\"\n }\n ],\n \n \"metrics\": {\n \"session_id\": \"galaxy_session_galaxy_session_20251025_183449_task_32\",\n \"task_count\": 5,\n \"completed_tasks\": 5,\n \"failed_tasks\": 0,\n \"total_execution_time\": 674.547759771347,\n \n \"task_timings\": {\n \"t1\": {\n \"start\": 1761388508.9484463,\n \"duration\": 11.852121591567993,\n \"end\": 1761388520.8005679\n },\n \"t2\": {\n \"start\": 1761388508.9494512,\n \"duration\": 12.128723621368408,\n \"end\": 1761388521.0781748\n },\n \"t3\": {\n \"start\": 1761388508.9494512,\n \"duration\": 12.409801721572876,\n \"end\": 1761388521.359253\n },\n \"t4\": {\n \"start\": 1761388508.9494512,\n \"duration\": 269.1103162765503,\n \"end\": 1761388778.0597675\n },\n \"t5\": {\n \"start\": 1761388799.57892,\n \"duration\": 369.0467965602875,\n \"end\": 1761389168.6257164\n }\n },\n \n \"constellation_count\": 1,\n \"completed_constellations\": 1,\n \"failed_constellations\": 0,\n \"total_constellation_time\": 0.0,\n \n \"task_statistics\": {\n \"total_tasks\": 5,\n \"completed_tasks\": 5,\n \"failed_tasks\": 0,\n \"success_rate\": 1.0,\n \"failure_rate\": 0.0,\n \"average_task_duration\": 134.9095519542694,\n \"min_task_duration\": 11.852121591567993,\n \"max_task_duration\": 369.0467965602875,\n \"total_task_execution_time\": 674.547759771347\n },\n \n \"constellation_statistics\": {\n \"total_constellations\": 1,\n \"completed_constellations\": 1,\n \"failed_constellations\": 0,\n \"success_rate\": 1.0,\n \"average_constellation_duration\": 659.9815917015076,\n \"min_constellation_duration\": 659.9815917015076,\n \"max_constellation_duration\": 659.9815917015076,\n \"total_constellation_time\": 0.0,\n \"average_tasks_per_constellation\": 5.0\n },\n \n \"modification_statistics\": {\n \"total_modifications\": 4,\n \"constellations_modified\": 1,\n \"average_modifications_per_constellation\": 4.0,\n \"max_modifications_for_single_constellation\": 4,\n \"most_modified_constellation\": \"constellation_b0864385_20251025_183508\",\n \"modifications_per_constellation\": {\n \"constellation_b0864385_20251025_183508\": 4\n },\n \"modification_types_breakdown\": {\n \"Edited by constellation_agent\": 4\n }\n }\n }\n },\n \n \"constellation\": {\n \"id\": \"constellation_b0864385_20251025_183508\",\n \"name\": \"constellation_b0864385_20251025_183508\",\n \"task_count\": 5,\n \"dependency_count\": 4,\n \"state\": \"completed\"\n }\n}\n```\n\n---\n\n## Programmatic Access\n\n### Reading Result JSON\n\n```python\nimport json\nfrom pathlib import Path\n\ndef load_session_result(task_name: str) -> dict:\n \"\"\"\n Load Galaxy session result.\n \n :param task_name: Task identifier (e.g., \"task_32\")\n :return: Result dictionary\n \"\"\"\n result_path = Path(\"logs/galaxy\") / task_name / \"result.json\"\n \n with open(result_path, 'r', encoding='utf-8') as f:\n return json.load(f)\n\n# Example usage\nresult = load_session_result(\"task_32\")\nprint(f\"Session: {result['session_name']}\")\nprint(f\"Status: {result['status']}\")\nprint(f\"Duration: {result['execution_time']:.2f}s\")\n```\n\n### Extracting Key Information\n\n```python\ndef extract_summary(result: dict) -> dict:\n \"\"\"\n Extract key summary information from result.json.\n \n :param result: Result dictionary from load_session_result()\n :return: Summary dictionary\n \"\"\"\n metrics = result[\"session_results\"][\"metrics\"]\n task_stats = metrics[\"task_statistics\"]\n const_stats = result[\"session_results\"][\"final_constellation_stats\"]\n \n return {\n \"session_name\": result[\"session_name\"],\n \"request\": result[\"request\"],\n \"status\": result[\"status\"],\n \"total_duration\": result[\"execution_time\"],\n \"task_count\": task_stats[\"total_tasks\"],\n \"success_rate\": task_stats[\"success_rate\"],\n \"parallelism_ratio\": const_stats.get(\"parallelism_ratio\", 1.0),\n \"final_result\": result[\"session_results\"][\"final_results\"][0][\"result\"] \n if result[\"session_results\"][\"final_results\"] else None\n }\n\n# Example usage\nresult = load_session_result(\"task_32\")\nsummary = extract_summary(result)\n\nprint(f\"✅ Success Rate: {summary['success_rate'] * 100:.1f}%\")\nprint(f\"⏱️ Duration: {summary['total_duration']:.2f}s\")\nprint(f\"🔀 Parallelism: {summary['parallelism_ratio']:.2f}\")\n```\n\n**Expected Output:**\n\n```\n✅ Success Rate: 100.0%\n⏱️ Duration: 684.86s\n🔀 Parallelism: 1.06\n```\n\n### Batch Analysis\n\n```python\ndef analyze_multiple_sessions(log_dir: str = \"logs/galaxy\"):\n \"\"\"\n Analyze multiple Galaxy sessions from log directory.\n \n :param log_dir: Path to Galaxy log directory\n :return: DataFrame with session analysis\n \"\"\"\n import pandas as pd\n \n sessions = []\n \n for task_dir in Path(log_dir).iterdir():\n result_file = task_dir / \"result.json\"\n \n if result_file.exists():\n with open(result_file, 'r', encoding='utf-8') as f:\n result = json.load(f)\n summary = extract_summary(result)\n sessions.append(summary)\n \n df = pd.DataFrame(sessions)\n \n print(\"📊 Session Analysis Summary:\")\n print(f\" Total sessions: {len(df)}\")\n print(f\" Average duration: {df['total_duration'].mean():.2f}s\")\n print(f\" Average success rate: {df['success_rate'].mean() * 100:.1f}%\")\n print(f\" Average parallelism: {df['parallelism_ratio'].mean():.2f}\")\n \n return df\n\n# Example usage\ndf = analyze_multiple_sessions()\n```\n\n### Generating Reports\n\n```python\ndef generate_performance_report(task_name: str, output_file: str = \"report.md\"):\n \"\"\"\n Generate Markdown performance report from result.json.\n \n :param task_name: Task identifier\n :param output_file: Output Markdown file path\n \"\"\"\n result = load_session_result(task_name)\n metrics = result[\"session_results\"][\"metrics\"]\n \n # Generate Markdown report\n report = f\"\"\"# Galaxy Session Performance Report\n```\n\n## Session Information\n\n- **Session Name:** {result['session_name']}\n- **Task Name:** {result['task_name']}\n- **Status:** {result['status']}\n- **Start Time:** {result['start_time']}\n- **End Time:** {result['end_time']}\n- **Total Duration:** {result['execution_time']:.2f}s\n\n\n## Task Performance\n\n| Metric | Value |\n|--------|-------|\n| Total Tasks | `{metrics['task_count']}` |\n| Completed Tasks | `{metrics['completed_tasks']}` |\n| Failed Tasks | `{metrics['failed_tasks']}` |\n| Success Rate | `{metrics['task_statistics']['success_rate'] * 100:.1f}%` |\n| Average Task Duration | `{metrics['task_statistics']['average_task_duration']:.2f}s` |\n| Min Task Duration | `{metrics['task_statistics']['min_task_duration']:.2f}s` |\n| Max Task Duration | `{metrics['task_statistics']['max_task_duration']:.2f}s` |\n\n## Constellation Performance\n\n| Metric | Value |\n|--------|-------|\n| Parallelism Ratio | `{result['session_results']['final_constellation_stats']['parallelism_ratio']:.2f}` |\n| Critical Path Length | `{result['session_results']['final_constellation_stats']['critical_path_length']:.2f}s` |\n| Total Work | `{result['session_results']['final_constellation_stats']['total_work']:.2f}s` |\n| Max Width | `{result['session_results']['final_constellation_stats']['max_width']}` |\n\n\n# Example usage\n\n```python\n generate_performance_report(\"task_32\", \"task_32_report.md\")\n```\n\n## Use Cases\n\n### 1. Debugging Failed Sessions\n\n```python\ndef debug_failed_session(task_name: str):\n \"\"\"\n Analyze failed session for debugging.\n \n :param task_name: Task identifier\n \"\"\"\n result = load_session_result(task_name)\n \n if result[\"status\"] != \"completed\":\n print(f\"⚠️ Session Failed: {result['status']}\")\n \n metrics = result[\"session_results\"][\"metrics\"]\n failed_tasks = []\n \n for task_id, timing in metrics[\"task_timings\"].items():\n # Check if task is in failed list\n if task_id in [f\"t{i}\" for i in range(metrics[\"failed_tasks\"])]:\n failed_tasks.append(task_id)\n \n if failed_tasks:\n print(f\"\\n❌ Failed Tasks:\")\n for task_id in failed_tasks:\n print(f\" • {task_id}\")\n \n # Check logs for more details\n log_dir = Path(result[\"trajectory_path\"])\n print(f\"\\n📁 Check logs in: {log_dir}\")\n```\n\n### 2. Comparing Session Performance\n\n```python\ndef compare_sessions(task_name_1: str, task_name_2: str):\n \"\"\"\n Compare performance of two Galaxy sessions.\n \n :param task_name_1: First task identifier\n :param task_name_2: Second task identifier\n \"\"\"\n result1 = load_session_result(task_name_1)\n result2 = load_session_result(task_name_2)\n \n summary1 = extract_summary(result1)\n summary2 = extract_summary(result2)\n \n print(f\"📊 Session Comparison:\")\n print(f\"\\n{'Metric':<30} {task_name_1:<20} {task_name_2:<20}\")\n print(\"-\" * 70)\n print(f\"{'Duration (s)':<30} {summary1['total_duration']:<20.2f} {summary2['total_duration']:<20.2f}\")\n print(f\"{'Task Count':<30} {summary1['task_count']:<20} {summary2['task_count']:<20}\")\n print(f\"{'Success Rate':<30} {summary1['success_rate']*100:<20.1f}% {summary2['success_rate']*100:<20.1f}%\")\n print(f\"{'Parallelism Ratio':<30} {summary1['parallelism_ratio']:<20.2f} {summary2['parallelism_ratio']:<20.2f}\")\n```\n\n```python\nimport matplotlib.pyplot as plt\nfrom datetime import datetime\n\ndef plot_performance_trend(log_dir: str = \"logs/galaxy\"):\n \"\"\"\n Plot performance trends across sessions.\n \n :param log_dir: Path to Galaxy log directory\n \"\"\"\n sessions = []\n \n for task_dir in sorted(Path(log_dir).iterdir()):\n result_file = task_dir / \"result.json\"\n \n if result_file.exists():\n with open(result_file, 'r') as f:\n result = json.load(f)\n sessions.append({\n \"timestamp\": datetime.fromisoformat(result[\"start_time\"]),\n \"duration\": result[\"execution_time\"],\n \"task_count\": result[\"session_results\"][\"metrics\"][\"task_count\"],\n \"parallelism\": result[\"session_results\"][\"final_constellation_stats\"].get(\"parallelism_ratio\", 1.0)\n })\n \n if not sessions:\n print(\"No sessions found\")\n return\n \n # Plot duration trend\n fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 8))\n \n timestamps = [s[\"timestamp\"] for s in sessions]\n durations = [s[\"duration\"] for s in sessions]\n parallelism = [s[\"parallelism\"] for s in sessions]\n \n ax1.plot(timestamps, durations, marker='o')\n ax1.set_xlabel(\"Session Timestamp\")\n ax1.set_ylabel(\"Duration (seconds)\")\n ax1.set_title(\"Session Duration Trend\")\n ax1.grid(True, alpha=0.3)\n \n ax2.plot(timestamps, parallelism, marker='o', color='green')\n ax2.set_xlabel(\"Session Timestamp\")\n ax2.set_ylabel(\"Parallelism Ratio\")\n ax2.set_title(\"Parallelism Efficiency Trend\")\n ax2.axhline(y=1.0, color='red', linestyle='--', label='Sequential (no parallelism)')\n ax2.grid(True, alpha=0.3)\n ax2.legend()\n \n plt.tight_layout()\n plt.savefig(\"performance_trend.png\")\n print(\"📈 Trend plot saved to performance_trend.png\")\n\n# Example usage\nplot_performance_trend()\n```\n\n## Related Documentation\n\n- **[Performance Metrics](./performance_metrics.md)** - Detailed metrics documentation and analysis\n- **[Trajectory Report](./trajectory_report.md)** - Human-readable execution log with DAG visualizations\n- **[Galaxy Overview](../overview.md)** - Main Galaxy framework documentation\n- **[Task Constellation](../constellation/task_constellation.md)** - DAG structure and parallelism metrics\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** - Execution coordination\n\n## Summary\n\nThe `result.json` file provides comprehensive session analysis:\n\n- **Complete execution history** - All session details in structured format\n- **Performance metrics** - Comprehensive timing and statistics via `SessionMetricsObserver`\n- **Constellation analysis** - DAG structure and parallelism data\n- **Programmatic access** - JSON format for automated analysis and reporting\n- **Debugging support** - Failed task identification and detailed execution logs\n- **Trend analysis** - Compare sessions over time for performance monitoring\n\nUse `result.json` for debugging, performance optimization, reporting, and automated analysis of Galaxy workflows.\n"
},
{
"path": "documents/docs/galaxy/evaluation/trajectory_report.md",
"content": "# Galaxy Trajectory Report\n\n## Overview\n\nThe **Galaxy Trajectory Report** (`output.md`) is an automatically generated comprehensive execution log that documents the complete lifecycle of a multi-device task execution session in Galaxy. This human-readable Markdown report provides step-by-step visualization of constellation evolution, task execution, and device coordination.\n\n### Report Location\n\nAfter each Galaxy session completes, the trajectory report is automatically generated:\n\n```\nlogs/galaxy//output.md\nlogs/galaxy//topology_images/ # DAG visualizations\n```\n\n**Example:**\n```\nlogs/galaxy/request_20251111_140216_1/\n├── output.md # Main trajectory report\n├── response.log # Raw JSONL execution log\n├── request.log # Request details \n├── evaluation.log # Optional evaluation\n├── result.json # Performance metrics\n└── topology_images/ # Generated DAG topology graphs\n ├── step1_after_constellation_xxx.png\n ├── step2_after_constellation_xxx.png\n └── step999_final_constellation_xxx.png\n```\n\n## Report Structure\n\n### 1. Executive Summary\n\nHigh-level session overview:\n\n```markdown\n## Executive Summary\n\n- **User Request**: type hi on all linux and write results to windows notepad\n- **Total Steps**: 4\n- **Total Time**: 31.54s\n```\n\n**Components:**\n- **User Request**: Original natural language task description\n- **Total Steps**: Number of orchestration steps (DAG creation + execution rounds)\n- **Total Time**: End-to-end session duration in seconds\n\n### 2. Step-by-Step Execution\n\nDetailed breakdown of each orchestration step with:\n\n#### Step Metadata\n\n```markdown\n### Step 2\n\n- **Agent**: constellation_agent (ConstellationAgent)\n- **Status**: CONTINUE\n- **Round**: 0 | **Round Step**: 0\n- **Execution Time**: 9.27s\n- **Time Breakdown**:\n - LLM_INTERACTION: 8.96s\n - ACTION_EXECUTION: 0.29s\n - MEMORY_UPDATE: 0.00s\n```\n\n**Fields:**\n- **Agent**: Agent name and type (ConstellationAgent for orchestration)\n- **Status**: Step outcome (`CONTINUE`, `FINISH`, `ERROR`)\n- **Round/Round Step**: ReAct iteration counters\n- **Execution Time**: Total step duration\n- **Time Breakdown**: Profiling data for LLM calls, action execution, memory updates\n\n#### Actions Performed\n\nDocuments agent actions with collapsible argument details:\n\n```markdown\n#### Actions Performed\n\n**Function**: `build_constellation`\n\n\nArguments (click to expand)\n\n```json\n{\n \"config\": {\n \"constellation_id\": \"constellation_xxx\",\n \"tasks\": { ... },\n \"dependencies\": { ... }\n }\n}\n```\n\n\n```\n\n**Common Functions:**\n- `build_constellation`: Initial DAG creation\n- `edit_constellation`: Dynamic DAG modification\n- `execute_constellation`: Trigger task execution\n\n#### Constellation Evolution\n\nVisualizes DAG state changes with interactive topology graphs:\n\n```markdown\n#### Constellation Evolution\n\n\nConstellation AFTER (click to expand)\n\n**Constellation ID**: constellation_bcd1726e_20251105_134526\n**State**: created\n\n##### Dependency Graph (Topology)\n\n\n\n##### Task Summary Table\n\n| Task ID | Name | Status | Device | Duration |\n|---------|------|--------|--------|----------|\n| task-1 | Type hi on linux_agent_1 | pending | linux_agent_1 | N/A |\n| task-2 | Type hi on linux_agent_2 | pending | linux_agent_2 | N/A |\n| task-3 | Type hi on linux_agent_3 | pending | linux_agent_3 | N/A |\n```\n\n**Topology Visualization Features:**\n- **Color-coded nodes** by task status:\n - 🟢 Green: Completed\n - 🔵 Cyan: Running\n - ⚫ Gray: Pending\n - 🔴 Red: Failed/Error\n- **Edge styles** for dependencies:\n - Solid green: Satisfied dependencies\n - Dashed orange: Pending dependencies\n- **Automatic layout** with hierarchical spring algorithm\n- **Legend** showing node/edge meanings\n\n##### Detailed Task Information\n\nComprehensive task metadata with execution details:\n\n```markdown\n#### Task task-1: Type hi on linux_agent_1\n\n- **Status**: completed\n- **Target Device**: linux_agent_1\n- **Priority**: 2\n- **Description**: On device linux_agent_1 (Linux), open a terminal and execute the command: echo 'hi'. Return the output text.\n- **Tips**:\n - Ensure CLI access is available.\n - Expected textual result: Return the exact output of the command, which should be 'hi'.\n- **Result**: \n ```\n hi\n ```\n- **Started**: 2025-11-05T05:45:26.395208+00:00\n- **Ended**: 2025-11-05T05:45:42.981859+00:00\n- **Duration**: 16.59s\n```\n\n**Task Fields:**\n- **Status**: Current execution state (`pending`, `running`, `completed`, `failed`, `cancelled`)\n- **Target Device**: Assigned device agent ID\n- **Priority**: Task scheduling priority (1=HIGH, 2=MEDIUM, 3=LOW)\n- **Description**: Natural language task specification for device agent\n- **Tips**: Execution hints and expected output guidance\n- **Result**: Task execution output (truncated if large)\n- **Error**: Error message if task failed\n- **Timing**: Start/end timestamps and duration\n\n##### Dependency Details\n\nShows task relationships and satisfaction status:\n\n```markdown\n| Line ID | From Task | To Task | Type | Satisfied | Condition |\n|---------|-----------|---------|------|-----------|----------|\n| l1 | t1 | t4 | unconditional | [PENDING] | Output from linux_agent_1 collected successfully. |\n| l2 | t2 | t4 | unconditional | [OK] | Output from linux_agent_2 collected successfully. |\n```\n\n**Dependency Types:**\n- `unconditional`: Always active when source task completes\n- `conditional`: Activated based on result evaluation\n\n#### Connected Devices\n\nDevice registry snapshot at step completion:\n\n```markdown\n\nConnected Devices\n\n| Device ID | OS | Status | Last Heartbeat |\n|-----------|----|---------|--------------|\n| windowsagent | windows | idle | 2025-11-05T05:45:43 |\n| linux_agent_1 | linux | idle | 2025-11-05T05:45:43 |\n| linux_agent_2 | linux | idle | 2025-11-05T05:45:43 |\n| linux_agent_3 | linux | idle | 2025-11-05T05:45:43 |\n```\n\n**Device Statuses:**\n- `idle`: Connected and available\n- `busy`: Executing task\n- `disconnected`: WebSocket connection lost\n\n### 3. Final Constellation State\n\nComplete final DAG with all task results:\n\n```markdown\n## Final Constellation State\n\n**ID**: constellation_bcd1726e_20251105_134526\n**State**: completed\n**Created**: 2025-11-05T05:45:26.230930+00:00\n**Updated**: 2025-11-05T05:45:42.981859+00:00\n\n### Task Details\n[Full task information with results]\n\n### Task Summary Table\n[Aggregated task status table]\n\n### Final Dependency Graph\n[Final topology visualization]\n```\n\n## Generation Process\n\n### Automatic Generation\n\nThe trajectory report is generated automatically by `GalaxySession` upon completion:\n\n```python\n# galaxy/session/galaxy_session.py\nasync def close_session(self):\n \"\"\"Generate trajectory report on session close\"\"\"\n trajectory = GalaxyTrajectory(self.log_path)\n trajectory.to_markdown(self.log_path + \"output.md\")\n```\n\n**Trigger Points:**\n1. Normal session completion (`GalaxyClient.shutdown()`)\n2. User termination (Ctrl+C in interactive mode)\n3. Error-induced session end\n\n### Manual Generation\n\nYou can regenerate reports manually using the CLI tool:\n\n```bash\n# Generate report for specific session\npython -m galaxy.trajectory.generate_report logs/galaxy/test1\n\n# Custom output path\npython -m galaxy.trajectory.generate_report logs/galaxy/test1 -o custom_report.md\n\n# Minimal report (exclude details)\npython -m galaxy.trajectory.generate_report logs/galaxy/test1 \\\n --no-constellation --no-tasks --no-devices\n```\n\n**CLI Options:**\n- `--no-constellation`: Exclude constellation evolution details\n- `--no-tasks`: Exclude detailed task information\n- `--no-devices`: Exclude device connection information\n- `-o, --output`: Custom output file path\n\n### Batch Generation\n\nProcess multiple sessions at once:\n\n```python\n# galaxy/trajectory/galaxy_parser.py\nif __name__ == \"__main__\":\n \"\"\"Process all Galaxy task logs and generate markdown reports.\"\"\"\n \n galaxy_logs_dir = Path(\"logs/galaxy\")\n task_dirs = sorted([d for d in galaxy_logs_dir.iterdir() if d.is_dir()])\n \n for task_dir in task_dirs:\n trajectory = GalaxyTrajectory(str(task_dir))\n output_path = task_dir / \"trajectory_report.md\"\n trajectory.to_markdown(str(output_path))\n```\n\nRun batch processing:\n\n```bash\ncd c:\\Users\\chaoyunzhang\\OneDrive - Microsoft\\Desktop\\research\\GPTV\\UFO-windows\\github\\saber\\UFO2\npython -m galaxy.trajectory.galaxy_parser\n```\n\n**Output:**\n```\n[BOLD BLUE] Galaxy Trajectory Parser - Batch Mode\nFound 42 task directories\n\nProcessing task_1... [OK]\nProcessing task_2... [OK]\nProcessing test1... [OK]\n...\n\n=====================================================\nSummary:\n Total: 42\n Success: 40\n Skipped: 2\n Failed: 0\n=====================================================\n```\n\n## Programmatic Access\n\n### Loading Trajectory Data\n\n```python\nfrom galaxy.trajectory import GalaxyTrajectory\n\n# Load trajectory from log directory\ntrajectory = GalaxyTrajectory(\"logs/galaxy/test1\")\n\n# Access metadata\nprint(f\"Request: {trajectory.request}\")\nprint(f\"Steps: {trajectory.total_steps}\")\nprint(f\"Cost: ${trajectory.total_cost:.4f}\")\nprint(f\"Time: {trajectory.total_time:.2f}s\")\n\n# Iterate through steps\nfor idx, step in enumerate(trajectory.step_log, 1):\n agent = step.get(\"agent_name\")\n status = step.get(\"status\")\n time = step.get(\"total_time\", 0)\n print(f\"Step {idx}: {agent} - {status} ({time:.2f}s)\")\n```\n\n### Extracting Constellation Data\n\n```python\n# Get final constellation state\nlast_step = trajectory.step_log[-1]\nfinal_constellation = trajectory._parse_constellation(\n last_step.get(\"constellation_after\")\n)\n\nif final_constellation:\n constellation_id = final_constellation.get(\"constellation_id\")\n state = final_constellation.get(\"state\")\n tasks = final_constellation.get(\"tasks\", {})\n \n print(f\"Constellation {constellation_id}: {state}\")\n print(f\"Tasks: {len(tasks)}\")\n \n # Analyze task outcomes\n completed = sum(1 for t in tasks.values() if t.get(\"status\") == \"completed\")\n failed = sum(1 for t in tasks.values() if t.get(\"status\") == \"failed\")\n \n print(f\"Completed: {completed}/{len(tasks)}\")\n print(f\"Failed: {failed}/{len(tasks)}\")\n```\n\n### Custom Report Generation\n\n```python\n# Generate custom report with specific options\ntrajectory.to_markdown(\n output_path=\"custom_report.md\",\n include_constellation_details=True, # Show DAG evolution\n include_task_details=True, # Show task results\n include_device_info=False # Hide device info\n)\n```\n\n## Visualization Features\n\n### Topology Graph Generation\n\nThe trajectory report includes dynamically generated DAG topology images:\n\n**Implementation:**\n```python\ndef _generate_topology_image(\n self,\n dependencies: Dict[str, Any],\n tasks: Dict[str, Any],\n constellation_id: str,\n step_number: int,\n state: str = \"before\"\n) -> Optional[str]:\n \"\"\"Generate beautiful topology graph using networkx and matplotlib\"\"\"\n \n # Create directed graph\n G = nx.DiGraph()\n \n # Add all tasks as nodes\n for task_id in tasks.keys():\n G.add_node(task_id)\n \n # Add dependency edges\n for dep in dependencies.values():\n from_task = dep[\"from_task_id\"]\n to_task = dep[\"to_task_id\"]\n G.add_edge(from_task, to_task)\n \n # Color nodes by status\n status_colors = {\n \"completed\": \"#28A745\", # Green\n \"running\": \"#17A2B8\", # Cyan\n \"pending\": \"#6C757D\", # Gray\n \"failed\": \"#DC3545\", # Red\n }\n \n # Generate layout and save image\n pos = nx.spring_layout(G, k=1.5, iterations=100)\n # ... [matplotlib rendering code]\n```\n\n**Graph Features:**\n- **Hierarchical Layout**: Spring algorithm with optimized spacing (`k=1.5`)\n- **Adaptive Node Size**: Ellipses scale with task ID length\n- **Color-Coded Status**: Bootstrap-inspired color scheme\n- **Edge Differentiation**: Solid (satisfied) vs dashed (pending)\n- **Legend**: Automatic status and dependency type legend\n- **High Quality**: 120 DPI PNG with antialiasing\n\n### Image Organization\n\n```\ntopology_images/\n├── step1_after_constellation_7b3c0f47_20251104_182305.png\n├── step2_before_constellation_bcd1726e_20251105_134526.png\n├── step2_after_constellation_bcd1726e_20251105_134526.png\n├── step3_before_constellation_bcd1726e_20251105_134526.png\n├── step3_after_constellation_bcd1726e_20251105_134526.png\n└── step999_final_constellation_bcd1726e_20251105_134526.png\n```\n\n**Naming Convention:**\n- `step{N}_{state}_{constellation_id}.png`\n- `state`: `before`, `after`, or `final`\n- `step999`: Reserved for final summary graph\n\n## Use Cases\n\n### 1. Debugging Failed Sessions\n\nIdentify which task failed and why:\n\n```python\ntrajectory = GalaxyTrajectory(\"logs/galaxy/failed_session\")\n\nfor step in trajectory.step_log:\n constellation = trajectory._parse_constellation(step.get(\"constellation_after\"))\n if not constellation:\n continue\n \n tasks = constellation.get(\"tasks\", {})\n for task_id, task in tasks.items():\n if task.get(\"status\") == \"failed\":\n print(f\"❌ Task {task_id}: {task.get('name')}\")\n print(f\" Device: {task.get('target_device_id')}\")\n print(f\" Error: {task.get('error')}\")\n```\n\n### 2. Performance Analysis\n\nCorrelate with `result.json` for bottleneck identification:\n\n```python\nimport json\n\n# Load trajectory for execution timeline\ntrajectory = GalaxyTrajectory(\"logs/galaxy/task_32\")\n\n# Load metrics for performance data\nwith open(\"logs/galaxy/task_32/result.json\") as f:\n result = json.load(f)\n\nmetrics = result[\"session_results\"][\"metrics\"]\ntask_stats = metrics[\"task_statistics\"]\n\n# Find slowest tasks\nslow_tasks = [\n (tid, task.get(\"execution_duration\", 0))\n for step in trajectory.step_log\n for tid, task in trajectory._parse_constellation(\n step.get(\"constellation_after\")\n ).get(\"tasks\", {}).items()\n]\n\nslow_tasks.sort(key=lambda x: x[1], reverse=True)\nprint(f\"Top 5 slowest tasks:\")\nfor tid, duration in slow_tasks[:5]:\n print(f\" {tid}: {duration:.2f}s\")\n```\n\n### 3. Constellation Evolution Analysis\n\nTrack DAG modifications across steps:\n\n```python\ntrajectory = GalaxyTrajectory(\"logs/galaxy/adaptive_session\")\n\nfor idx, step in enumerate(trajectory.step_log, 1):\n before = trajectory._parse_constellation(step.get(\"constellation_before\"))\n after = trajectory._parse_constellation(step.get(\"constellation_after\"))\n \n if before and after:\n tasks_before = len(before.get(\"tasks\", {}))\n tasks_after = len(after.get(\"tasks\", {}))\n \n if tasks_after > tasks_before:\n print(f\"Step {idx}: Added {tasks_after - tasks_before} tasks\")\n elif tasks_after < tasks_before:\n print(f\"Step {idx}: Removed {tasks_before - tasks_after} tasks\")\n```\n\n### 4. Device Utilization Tracking\n\nAnalyze device workload distribution:\n\n```python\ntrajectory = GalaxyTrajectory(\"logs/galaxy/multi_device\")\n\n# Count tasks per device\ndevice_tasks = {}\nfor step in trajectory.step_log:\n constellation = trajectory._parse_constellation(step.get(\"constellation_after\"))\n if not constellation:\n continue\n \n for task in constellation.get(\"tasks\", {}).values():\n device = task.get(\"target_device_id\")\n device_tasks[device] = device_tasks.get(device, 0) + 1\n\nprint(\"Task distribution:\")\nfor device, count in sorted(device_tasks.items(), key=lambda x: x[1], reverse=True):\n print(f\" {device}: {count} tasks\")\n```\n\n### 5. Session Comparison\n\nCompare multiple sessions for regression testing:\n\n```python\ndef compare_sessions(session1_path, session2_path):\n t1 = GalaxyTrajectory(session1_path)\n t2 = GalaxyTrajectory(session2_path)\n \n print(f\"Session 1 vs Session 2:\")\n print(f\" Steps: {t1.total_steps} vs {t2.total_steps}\")\n print(f\" Time: {t1.total_time:.2f}s vs {t2.total_time:.2f}s\")\n print(f\" Cost: ${t1.total_cost:.4f} vs ${t2.total_cost:.4f}\")\n \n speedup = (t1.total_time - t2.total_time) / t1.total_time * 100\n print(f\" Performance: {speedup:+.1f}%\")\n\ncompare_sessions(\"logs/galaxy/test_v1\", \"logs/galaxy/test_v2\")\n```\n\n## Data Sources\n\nThe trajectory report aggregates data from multiple log sources:\n\n### 1. response.log (Primary Source)\n\nJSONL file with per-step execution records:\n\n```json\n{\n \"request\": \"type hi on all linux devices\",\n \"agent_name\": \"constellation_agent\",\n \"agent_type\": \"ConstellationAgent\",\n \"status\": \"CONTINUE\",\n \"round_num\": 0,\n \"round_step\": 0,\n \"total_time\": 9.27,\n \"cost\": 0.0042,\n \"execution_times\": {\n \"LLM_INTERACTION\": 8.96,\n \"ACTION_EXECUTION\": 0.29,\n \"MEMORY_UPDATE\": 0.00\n },\n \"action\": [\n {\n \"function\": \"build_constellation\",\n \"arguments\": { ... }\n }\n ],\n \"constellation_before\": \"{...}\",\n \"constellation_after\": \"{...}\",\n \"device_info\": { ... }\n}\n```\n\n### 2. result.json (Performance Metrics)\n\nAggregated session-level metrics:\n\n```json\n{\n \"session_results\": {\n \"request\": \"type hi on all linux devices\",\n \"status\": \"completed\",\n \"total_cost\": 0.0156,\n \"total_rounds\": 1,\n \"total_steps\": 4,\n \"total_time\": 31.54,\n \"metrics\": {\n \"task_statistics\": { ... },\n \"constellation_statistics\": { ... }\n }\n }\n}\n```\n\n### 3. evaluation.log (Optional)\n\nUser-provided evaluation results:\n\n```json\n{\n \"task_success\": true,\n \"evaluation_score\": 5,\n \"comments\": \"All tasks completed successfully\"\n}\n```\n\n## Configuration\n\n### Customizing Report Content\n\nControl report verbosity via generation parameters:\n\n```python\ntrajectory.to_markdown(\n output_path=\"output.md\",\n include_constellation_details=True, # DAG evolution (default: True)\n include_task_details=True, # Task execution logs (default: True)\n include_device_info=True # Device status (default: True)\n)\n```\n\n**Report Size Impact:**\n- Full report (all options enabled): ~200KB for 10-task session\n- Minimal report (all options disabled): ~20KB\n- Topology images: ~50KB each\n\n### Topology Graph Styling\n\nCustomize graph appearance by modifying `_generate_topology_image()`:\n\n```python\n# Adjust node colors\nstatus_colors = {\n \"completed\": \"#28A745\", # Change to custom color\n \"running\": \"#17A2B8\",\n # ...\n}\n\n# Adjust layout parameters\npos = nx.spring_layout(\n G,\n k=1.5, # Node spacing (higher = more spread)\n iterations=100, # Layout quality (higher = better but slower)\n seed=42 # Deterministic layout\n)\n\n# Adjust image quality\nplt.savefig(\n image_path,\n dpi=120, # Resolution (higher = larger files)\n bbox_inches=\"tight\",\n facecolor=\"white\"\n)\n```\n\n## Best Practices\n\n### 1. Regular Report Review\n\nMonitor trajectory reports to catch issues early:\n\n```bash\n# Generate reports for recent sessions\nfor dir in logs/galaxy/*/; do\n python -m galaxy.trajectory.generate_report \"$dir\"\ndone\n\n# Open reports in browser for visual inspection\nstart logs/galaxy/test1/output.md\n```\n\n### 2. Archive Trajectory Reports\n\nStore reports with version control for reproducibility:\n\n```bash\n# Create timestamped archive\nmkdir -p trajectory_archives/$(date +%Y-%m-%d)\ncp logs/galaxy/*/output.md trajectory_archives/$(date +%Y-%m-%d)/\ncp logs/galaxy/*/result.json trajectory_archives/$(date +%Y-%m-%d)/\n```\n\n### 3. Automated Analysis\n\nIntegrate trajectory parsing into CI/CD pipelines:\n\n```python\n# test/analyze_trajectory.py\ndef validate_trajectory(log_dir):\n trajectory = GalaxyTrajectory(log_dir)\n \n # Check for failures\n for step in trajectory.step_log:\n if step.get(\"status\") == \"ERROR\":\n raise AssertionError(f\"Session failed at step {step.get('_line_number')}\")\n \n # Check performance thresholds\n if trajectory.total_time > 60.0:\n print(f\"WARNING: Session took {trajectory.total_time:.2f}s (>60s threshold)\")\n \n return True\n```\n\n### 4. Compare Before/After States\n\nUse constellation evolution to verify correctness:\n\n```python\n# Verify DAG grows monotonically (no premature task deletion)\ntrajectory = GalaxyTrajectory(\"logs/galaxy/session\")\n\nprev_task_count = 0\nfor step in trajectory.step_log:\n constellation = trajectory._parse_constellation(step.get(\"constellation_after\"))\n if constellation:\n task_count = len(constellation.get(\"tasks\", {}))\n if task_count < prev_task_count:\n print(f\"WARNING: Task count decreased from {prev_task_count} to {task_count}\")\n prev_task_count = task_count\n```\n\n## Related Documentation\n\n- **[Performance Metrics](./performance_metrics.md)** - Quantitative session analysis with `result.json`\n- **[Result JSON Reference](./result_json.md)** - Complete `result.json` schema documentation\n- **[Galaxy Overview](../overview.md)** - Main Galaxy framework documentation\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** - DAG execution engine\n- **[Task Constellation](../constellation/overview.md)** - DAG data structure and validation\n\n## Troubleshooting\n\n### Empty or Missing Report\n\n**Problem:** `output.md` not generated after session\n\n**Solutions:**\n\n1. Check for `response.log` existence:\n ```bash\n ls logs/galaxy//response.log\n ```\n\n2. Manually trigger generation:\n ```bash\n python -m galaxy.trajectory.generate_report logs/galaxy/\n ```\n\n3. Verify session closed properly (check for exception in terminal)\n\n### Parse Errors in Report\n\n**Problem:** `⚠️ Parse Error` warnings in report\n\n**Cause:** Legacy log format with serialization bugs (tasks as Python strings instead of JSON)\n\n**Solution:** This is a known issue fixed in current versions. Reports will display:\n```markdown\n##### ⚠️ Parse Error\n\n**Error Type**: `legacy_serialization_bug`\n**Message**: Tasks field contains Python object representations (not pure JSON). \nThis is due to a serialization bug in older versions.\n```\n\n**Workaround:** Re-run session with updated codebase to generate proper logs.\n\n### Missing Topology Images\n\n**Problem:** Broken image links in report\n\n**Solutions:**\n\n1. Check `topology_images/` directory exists:\n ```bash\n ls logs/galaxy//topology_images/\n ```\n\n2. Verify matplotlib backend:\n ```python\n import matplotlib\n matplotlib.use(\"Agg\") # Non-interactive backend required\n ```\n\n3. Regenerate report to recreate images:\n ```bash\n python -m galaxy.trajectory.generate_report logs/galaxy/\n ```\n\n### Large Report Files\n\n**Problem:** `output.md` exceeds 10MB\n\n**Solutions:**\n\n1. Generate minimal report:\n ```bash\n python -m galaxy.trajectory.generate_report logs/galaxy/ \\\n --no-constellation --no-tasks\n ```\n\n2. Reduce topology image quality (edit `galaxy_parser.py`):\n ```python\n plt.savefig(image_path, dpi=80) # Lower DPI\n ```\n\n3. Archive and compress:\n ```bash\n gzip logs/galaxy//output.md\n ```\n\n## API Reference\n\n### GalaxyTrajectory Class\n\n```python\nclass GalaxyTrajectory:\n \"\"\"Parser for Galaxy agent logs with constellation visualization\"\"\"\n \n def __init__(self, folder_path: str) -> None:\n \"\"\"\n Initialize trajectory parser.\n \n Args:\n folder_path: Path to Galaxy log directory (e.g., logs/galaxy/task_1)\n \n Raises:\n ValueError: If response.log file not found\n \"\"\"\n \n @property\n def step_log(self) -> List[Dict[str, Any]]:\n \"\"\"Get all step logs from response.log\"\"\"\n \n @property\n def evaluation_log(self) -> Dict[str, Any]:\n \"\"\"Get evaluation results from evaluation.log\"\"\"\n \n @property\n def request(self) -> Optional[str]:\n \"\"\"Get original user request\"\"\"\n \n @property\n def total_steps(self) -> int:\n \"\"\"Get total number of steps\"\"\"\n \n @property\n def total_cost(self) -> float:\n \"\"\"Calculate total LLM cost\"\"\"\n \n @property\n def total_time(self) -> float:\n \"\"\"Calculate total execution time\"\"\"\n \n def to_markdown(\n self,\n output_path: str,\n include_constellation_details: bool = True,\n include_task_details: bool = True,\n include_device_info: bool = True\n ) -> None:\n \"\"\"\n Export trajectory to Markdown file.\n \n Args:\n output_path: Path to save markdown file\n include_constellation_details: Include DAG evolution details\n include_task_details: Include task execution logs\n include_device_info: Include device status information\n \"\"\"\n```\n\n---\n\n**Next Steps:**\n- Combine trajectory reports with `result.json` metrics for comprehensive analysis\n- Automate report generation in CI/CD pipelines\n- Visualize execution timelines with custom scripts\n- Compare session trajectories for performance regression testing\n"
},
{
"path": "documents/docs/galaxy/observer/agent_output_observer.md",
"content": "# Agent Output Observer\n\nThe **AgentOutputObserver** handles real-time display of agent responses and actions. It listens for agent interaction events and delegates the actual presentation logic to specialized presenters, providing a clean separation between event handling and output formatting.\n\n**Location:** `galaxy/session/observers/agent_output_observer.py`\n\n## Purpose\n\nThe Agent Output Observer enables:\n\n- **Real-time Feedback** — Display agent thinking and decision-making process\n- **Action Visibility** — Show what actions the agent is taking\n- **Debugging** — Understand agent behavior during constellation execution\n- **User Engagement** — Keep users informed of progress and decisions\n\n## Architecture\n\nThe observer uses a **presenter pattern** for flexible output formatting:\n\n```mermaid\ngraph TB\n subgraph \"Agent Layer\"\n A[ConstellationAgent]\n end\n \n subgraph \"Event System\"\n EB[EventBus]\n end\n \n subgraph \"Observer Layer\"\n AOO[AgentOutputObserver]\n ER[Event Router]\n end\n \n subgraph \"Presenter Layer\"\n P[Presenter Factory]\n RP[RichPresenter]\n TP[TextPresenter]\n end\n \n subgraph \"Output\"\n O[Terminal/Console]\n end\n \n A -->|publish| EB\n EB -->|notify| AOO\n AOO --> ER\n ER -->|agent_response| RP\n ER -->|agent_action| RP\n \n P --> RP\n P --> TP\n \n RP --> O\n TP --> O\n \n style AOO fill:#66bb6a,stroke:#333,stroke-width:3px\n style P fill:#ffa726,stroke:#333,stroke-width:2px\n style EB fill:#4a90e2,stroke:#333,stroke-width:2px,color:#fff\n```\n\n**Component Responsibilities:**\n\n| Component | Role | Description |\n|-----------|------|-------------|\n| **Agent** | Event publisher | Publishes AGENT_RESPONSE and AGENT_ACTION events |\n| **AgentOutputObserver** | Event handler | Receives and routes agent events |\n| **Presenter** | Output formatter | Formats and displays agent output |\n| **PresenterFactory** | Creator | Creates appropriate presenter based on type |\n\n## Handled Events\n\nThe observer handles two types of agent events:\n\n### 1. AGENT_RESPONSE\n\nTriggered when agent generates responses (thoughts, plans, reasoning):\n\n**Event Data Structure:**\n\n```python\n{\n \"agent_name\": \"constellation_agent\",\n \"agent_type\": \"constellation\",\n \"output_type\": \"response\",\n \"output_data\": {\n # ConstellationAgentResponse fields\n \"thought\": \"Task 1 completed successfully...\",\n \"plan\": \"Next, I will process the results...\",\n \"operation\": \"EDIT\",\n \"observation\": \"Task result shows...\",\n # ... other fields\n },\n \"print_action\": False # Whether to print action details\n}\n```\n\n### 2. AGENT_ACTION\n\nTriggered when agent executes actions (constellation editing):\n\n**Event Data Structure:**\n\n```python\n{\n \"agent_name\": \"constellation_agent\",\n \"agent_type\": \"constellation\",\n \"output_type\": \"action\",\n \"output_data\": {\n \"action_type\": \"constellation_editing\",\n \"actions\": [\n {\n \"name\": \"add_task\",\n \"arguments\": {\n \"task_id\": \"new_task_1\",\n \"description\": \"Process attachment\",\n # ...\n }\n },\n # ... more actions\n ]\n }\n}\n```\n\n## Implementation\n\n### Initialization\n\n```python\nfrom galaxy.session.observers import AgentOutputObserver\n\n# Create agent output observer with default Rich presenter\nagent_output_observer = AgentOutputObserver(presenter_type=\"rich\")\n\n# Subscribe to event bus\nfrom galaxy.core.events import get_event_bus\nevent_bus = get_event_bus()\nevent_bus.subscribe(agent_output_observer)\n```\n\n**Constructor Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `presenter_type` | `str` | `\"rich\"` | Type of presenter (\"rich\", \"text\", etc.) |\n\n### Presenter Types\n\nThe observer supports different presenter types for various output formats:\n\n| Presenter Type | Description | Use Case |\n|----------------|-------------|----------|\n| `\"rich\"` | Rich terminal formatting with colors and boxes | Interactive terminal use |\n| `\"text\"` | Plain text output | Log files, CI/CD, simple terminals |\n\n## Output Examples\n\n### Agent Response Display\n\nWhen the agent generates a response, the Rich presenter displays:\n\n```\n╭─────────────────────────────────────────────────────────────╮\n│ 🤖 Agent Response │\n├─────────────────────────────────────────────────────────────┤\n│ Thought: │\n│ Task 'fetch_emails' has completed successfully. I need to │\n│ analyze the results and determine next steps. │\n│ │\n│ Plan: │\n│ I will extract the email count from the result and create │\n│ parallel parsing tasks for each email. │\n│ │\n│ Operation: EDIT │\n│ │\n│ Observation: │\n│ Result shows 3 emails were fetched. I will create 3 │\n│ parsing tasks with dependencies on the fetch task. │\n╰─────────────────────────────────────────────────────────────╯\n```\n\n### Agent Action Display\n\nWhen the agent performs constellation editing:\n\n```\n╭─────────────────────────────────────────────────────────────╮\n│ 🛠️ Agent Actions: Constellation Editing │\n├─────────────────────────────────────────────────────────────┤\n│ Action 1: add_task │\n│ ├─ task_id: parse_email_1 │\n│ ├─ description: Parse the first email │\n│ ├─ target_device_id: windows_pc_001 │\n│ └─ priority: MEDIUM │\n│ │\n│ Action 2: add_task │\n│ ├─ task_id: parse_email_2 │\n│ ├─ description: Parse the second email │\n│ ├─ target_device_id: windows_pc_001 │\n│ └─ priority: MEDIUM │\n│ │\n│ Action 3: add_dependency │\n│ ├─ from_task_id: fetch_emails │\n│ ├─ to_task_id: parse_email_1 │\n│ └─ dependency_type: SUCCESS_ONLY │\n│ │\n│ Action 4: add_dependency │\n│ ├─ from_task_id: fetch_emails │\n│ ├─ to_task_id: parse_email_2 │\n│ └─ dependency_type: SUCCESS_ONLY │\n╰─────────────────────────────────────────────────────────────╯\n```\n\n## Event Processing Flow\n\n```mermaid\nsequenceDiagram\n participant A as ConstellationAgent\n participant EB as EventBus\n participant AOO as AgentOutputObserver\n participant P as Presenter\n participant C as Console\n \n Note over A: Agent generates response\n A->>EB: publish(AGENT_RESPONSE)\n EB->>AOO: on_event(event)\n AOO->>AOO: _handle_agent_response()\n AOO->>AOO: Reconstruct ConstellationAgentResponse\n AOO->>P: present_constellation_agent_response()\n P->>C: Display formatted response\n \n Note over A: Agent performs actions\n A->>EB: publish(AGENT_ACTION)\n EB->>AOO: on_event(event)\n AOO->>AOO: _handle_agent_action()\n AOO->>AOO: Reconstruct ActionCommandInfo\n AOO->>P: present_constellation_editing_actions()\n P->>C: Display formatted actions\n```\n\n## API Reference\n\n### Constructor\n\n```python\ndef __init__(self, presenter_type: str = \"rich\")\n```\n\nInitialize the agent output observer with specified presenter type.\n\n**Parameters:**\n\n- `presenter_type` — Type of presenter to use (\"rich\", \"text\", etc.)\n\n**Example:**\n\n```python\n# Use Rich presenter (default)\nrich_observer = AgentOutputObserver(presenter_type=\"rich\")\n\n# Use plain text presenter\ntext_observer = AgentOutputObserver(presenter_type=\"text\")\n```\n\n### Event Handler\n\n```python\nasync def on_event(self, event: Event) -> None\n```\n\nHandle agent output events.\n\n**Parameters:**\n\n- `event` — Event instance (must be AgentEvent)\n\n**Behavior:**\n\n- Filters for `AgentEvent` instances\n- Routes to appropriate handler based on event type\n- Reconstructs response/action objects from event data\n- Delegates display to presenter\n\n## Usage Examples\n\n### Example 1: Basic Setup\n\n```python\nfrom galaxy.core.events import get_event_bus\nfrom galaxy.session.observers import AgentOutputObserver\n\n# Create and subscribe agent output observer\nagent_output_observer = AgentOutputObserver(presenter_type=\"rich\")\nevent_bus = get_event_bus()\nevent_bus.subscribe(agent_output_observer)\n\n# Agent events will now be displayed automatically\nawait orchestrator.execute_constellation(constellation)\n\n# Clean up\nevent_bus.unsubscribe(agent_output_observer)\n```\n\n### Example 2: Conditional Display\n\n```python\nasync def execute_with_agent_feedback(show_agent_output: bool = True):\n \"\"\"Execute constellation with optional agent output display.\"\"\"\n \n event_bus = get_event_bus()\n \n if show_agent_output:\n agent_output_observer = AgentOutputObserver(presenter_type=\"rich\")\n event_bus.subscribe(agent_output_observer)\n \n try:\n await orchestrator.execute_constellation(constellation)\n finally:\n if show_agent_output:\n event_bus.unsubscribe(agent_output_observer)\n```\n\n### Example 3: Different Presenters for Different Modes\n\n```python\nimport sys\n\ndef create_agent_observer():\n \"\"\"Create appropriate agent observer based on environment.\"\"\"\n \n # Use Rich presenter for interactive terminal\n if sys.stdout.isatty():\n return AgentOutputObserver(presenter_type=\"rich\")\n \n # Use text presenter for logs/CI\n else:\n return AgentOutputObserver(presenter_type=\"text\")\n\n# Usage\nagent_output_observer = create_agent_observer()\nevent_bus.subscribe(agent_output_observer)\n```\n\n### Example 4: Custom Filtering\n\n```python\nfrom galaxy.core.events import EventType\n\n# Subscribe only to specific agent events\nevent_bus.subscribe(\n agent_output_observer,\n {EventType.AGENT_ACTION} # Only show actions, not responses\n)\n```\n\n## Implementation Details\n\n### Response Handling\n\nThe observer reconstructs `ConstellationAgentResponse` from event data:\n\n```python\nasync def _handle_agent_response(self, event: AgentEvent) -> None:\n \"\"\"Handle agent response event.\"\"\"\n \n try:\n output_data = event.output_data\n \n if event.agent_type == \"constellation\":\n # Reconstruct ConstellationAgentResponse from output data\n response = ConstellationAgentResponse.model_validate(output_data)\n print_action = output_data.get(\"print_action\", False)\n \n # Use presenter to display the response\n self.presenter.present_constellation_agent_response(\n response, \n print_action=print_action\n )\n \n except Exception as e:\n self.logger.error(f\"Error handling agent response: {e}\")\n```\n\n### Action Handling\n\nThe observer reconstructs action command objects:\n\n```python\nasync def _handle_agent_action(self, event: AgentEvent) -> None:\n \"\"\"Handle agent action event.\"\"\"\n \n try:\n output_data = event.output_data\n \n if output_data.get(\"action_type\") == \"constellation_editing\":\n actions_data = output_data.get(\"actions\", [])\n \n # Convert each action dict to ActionCommandInfo\n action_objects = []\n for action_dict in actions_data:\n action_obj = ActionCommandInfo.model_validate(action_dict)\n action_objects.append(action_obj)\n \n # Create ListActionCommandInfo with reconstructed actions\n actions = ListActionCommandInfo(actions=action_objects)\n \n # Use presenter to display the actions\n self.presenter.present_constellation_editing_actions(actions)\n \n except Exception as e:\n self.logger.error(f\"Error handling agent action: {e}\")\n```\n\n## Best Practices\n\n### 1. Match Presenter to Environment\n\n```python\n# ✅ Good: Choose presenter based on context\nif running_in_jupyter:\n presenter_type = \"rich\" # Good for notebooks\nelif running_in_ci:\n presenter_type = \"text\" # Good for logs\nelif is_interactive_terminal:\n presenter_type = \"rich\" # Good for terminal\nelse:\n presenter_type = \"text\" # Safe default\n```\n\n### 2. Selective Event Subscription\n\n```python\n# Only show actions (skip verbose responses)\nevent_bus.subscribe(\n agent_output_observer,\n {EventType.AGENT_ACTION}\n)\n\n# Show everything (responses + actions)\nevent_bus.subscribe(agent_output_observer)\n```\n\n### 3. Handle Errors Gracefully\n\nThe observer includes comprehensive error handling:\n\n```python\ntry:\n # Process agent event\n await self._handle_agent_response(event)\nexcept Exception as e:\n self.logger.error(f\"Error handling agent output event: {e}\")\n # Don't re-raise - continue observing other events\n```\n\n## Integration with Agent\n\nThe observer integrates with the ConstellationAgent's state machine:\n\n### Agent Publishes Events\n\nThe agent publishes events at key points:\n\n```python\nclass ConstellationAgent:\n async def generate_response(self):\n \"\"\"Generate agent response and publish event.\"\"\"\n \n # Generate response using LLM\n response = await self._llm_call(...)\n \n # Publish AGENT_RESPONSE event\n await self._publish_agent_response_event(response)\n \n return response\n \n async def execute_actions(self, actions):\n \"\"\"Execute actions and publish event.\"\"\"\n \n # Publish AGENT_ACTION event\n await self._publish_agent_action_event(actions)\n \n # Actually execute the actions\n result = await self._execute_constellation_editing(actions)\n \n return result\n```\n\n## Performance Considerations\n\n### Display Overhead\n\nThe observer adds minimal overhead:\n\n- **Event processing**: < 1ms per event\n- **Rich rendering**: 5-10ms per display\n- **Text rendering**: < 1ms per display\n\n### Optimization for Large Outputs\n\n```python\n# For very verbose agents, consider:\n\n# 1. Use text presenter instead of rich\nagent_output_observer = AgentOutputObserver(presenter_type=\"text\")\n\n# 2. Subscribe only to actions\nevent_bus.subscribe(\n agent_output_observer,\n {EventType.AGENT_ACTION}\n)\n\n# 3. Disable in production\nif not debug_mode:\n # Don't create or subscribe observer\n pass\n```\n\n## Related Documentation\n\n- **[Observer System Overview](overview.md)** — Architecture and design\n- **[Progress Observer](progress_observer.md)** — Task completion coordination\n- **[Constellation Agent](../constellation_agent/overview.md)** — Agent implementation and state machine\n\n## Summary\n\nThe Agent Output Observer:\n\n- **Displays** agent responses and actions in real-time\n- **Delegates** to presenters for flexible formatting\n- **Supports** multiple output formats (Rich, text)\n- **Provides** transparency into agent decision-making\n- **Enables** debugging and user engagement\n\nThis observer is essential for understanding agent behavior during constellation execution, providing visibility into the AI's thought process and actions.\n"
},
{
"path": "documents/docs/galaxy/observer/event_system.md",
"content": "# Event System Core\n\nThe Event System Core provides the foundational infrastructure for event-driven communication in the Galaxy framework. It implements the Observer pattern through a central event bus, type-safe event classes, and well-defined interfaces.\n\n**Location:** `galaxy/core/events.py`\n\n---\n\n## 📦 Core Components\n\n### EventBus — Central Message Broker\n\nThe `EventBus` class is the heart of the event system, managing subscriptions and distributing events to all registered observers.\n\n```mermaid\ngraph LR\n A[Publisher 1] -->|publish| B[EventBus]\n C[Publisher 2] -->|publish| B\n D[Publisher 3] -->|publish| B\n \n B -->|notify| E[Observer 1]\n B -->|notify| F[Observer 2]\n B -->|notify| G[Observer 3]\n B -->|notify| H[Observer 4]\n \n style B fill:#4a90e2,stroke:#333,stroke-width:3px,color:#fff\n style E fill:#66bb6a,stroke:#333,stroke-width:2px\n style F fill:#66bb6a,stroke:#333,stroke-width:2px\n style G fill:#66bb6a,stroke:#333,stroke-width:2px\n style H fill:#66bb6a,stroke:#333,stroke-width:2px\n```\n\n**Key Features:**\n\n- **Singleton Pattern**: Single global instance accessed via `get_event_bus()`\n- **Type-based Filtering**: Observers can subscribe to specific event types or all events\n- **Concurrent Notification**: All observers notified in parallel using `asyncio.gather()`\n- **Error Isolation**: Exceptions in one observer don't affect others\n\n### Event Types\n\n`EventType` enumeration defines all possible events in the system:\n\n```python\nclass EventType(Enum):\n # Task-level events\n TASK_STARTED = \"task_started\"\n TASK_COMPLETED = \"task_completed\"\n TASK_FAILED = \"task_failed\"\n \n # Constellation lifecycle events\n CONSTELLATION_STARTED = \"constellation_started\"\n CONSTELLATION_COMPLETED = \"constellation_completed\"\n CONSTELLATION_FAILED = \"constellation_failed\"\n \n # Structure modification events\n CONSTELLATION_MODIFIED = \"constellation_modified\"\n \n # Agent output events\n AGENT_RESPONSE = \"agent_response\"\n AGENT_ACTION = \"agent_action\"\n \n # Device events\n DEVICE_CONNECTED = \"device_connected\"\n DEVICE_DISCONNECTED = \"device_disconnected\"\n DEVICE_STATUS_CHANGED = \"device_status_changed\"\n```\n\n### Event Classes\n\nFive specialized event types provide type-safe event handling:\n\n| Event Class | Extends | Additional Fields | Use Case |\n|-------------|---------|-------------------|----------|\n| `Event` | (base) | `event_type`, `source_id`, `timestamp`, `data` | Generic events |\n| `TaskEvent` | `Event` | `task_id`, `status`, `result`, `error` | Task execution events |\n| `ConstellationEvent` | `Event` | `constellation_id`, `constellation_state`, `new_ready_tasks` | Constellation lifecycle events |\n| `AgentEvent` | `Event` | `agent_name`, `agent_type`, `output_type`, `output_data` | Agent interaction events |\n| `DeviceEvent` | `Event` | `device_id`, `device_status`, `device_info`, `all_devices` | Device management events |\n\n---\n\n## 🔌 Interfaces\n\n### IEventObserver\n\nDefines the contract for all observer implementations:\n\n```python\nfrom abc import ABC, abstractmethod\nfrom galaxy.core.events import Event\n\nclass IEventObserver(ABC):\n \"\"\"Interface for event observers.\"\"\"\n \n @abstractmethod\n async def on_event(self, event: Event) -> None:\n \"\"\"\n Handle an event.\n \n :param event: The event object containing type, source, timestamp and data\n \"\"\"\n pass\n```\n\n**Implementation Pattern:**\n\n```python\nclass MyCustomObserver(IEventObserver):\n \"\"\"Custom observer implementation.\"\"\"\n \n async def on_event(self, event: Event) -> None:\n \"\"\"Handle events of interest.\"\"\"\n \n # Type-safe handling using isinstance\n if isinstance(event, TaskEvent):\n await self._handle_task_event(event)\n elif isinstance(event, ConstellationEvent):\n await self._handle_constellation_event(event)\n \n async def _handle_task_event(self, event: TaskEvent) -> None:\n \"\"\"Process task events.\"\"\"\n if event.event_type == EventType.TASK_COMPLETED:\n print(f\"Task {event.task_id} completed with status: {event.status}\")\n \n async def _handle_constellation_event(self, event: ConstellationEvent) -> None:\n \"\"\"Process constellation events.\"\"\"\n if event.event_type == EventType.CONSTELLATION_STARTED:\n print(f\"Constellation {event.constellation_id} started\")\n```\n\n### IEventPublisher\n\nDefines the contract for event publishing:\n\n```python\nclass IEventPublisher(ABC):\n \"\"\"Interface for event publishers.\"\"\"\n \n @abstractmethod\n def subscribe(self, observer: IEventObserver, \n event_types: Set[EventType] = None) -> None:\n \"\"\"Subscribe an observer to events.\"\"\"\n pass\n \n @abstractmethod\n def unsubscribe(self, observer: IEventObserver) -> None:\n \"\"\"Unsubscribe an observer.\"\"\"\n pass\n \n @abstractmethod\n async def publish_event(self, event: Event) -> None:\n \"\"\"Publish an event to subscribers.\"\"\"\n pass\n```\n\n---\n\n## 📖 EventBus API Reference\n\n### Subscription Management\n\n#### subscribe()\n\nSubscribe an observer to receive event notifications:\n\n```python\ndef subscribe(\n self, \n observer: IEventObserver, \n event_types: Set[EventType] = None\n) -> None\n```\n\n**Parameters:**\n\n- `observer`: The observer object implementing `IEventObserver`\n- `event_types`: Optional set of event types to subscribe to (None = all events)\n\n**Examples:**\n\n```python\nfrom galaxy.core.events import get_event_bus, EventType\n\nevent_bus = get_event_bus()\n\n# Subscribe to all events\nevent_bus.subscribe(my_observer)\n\n# Subscribe to specific event types\nevent_bus.subscribe(my_observer, {\n EventType.TASK_COMPLETED,\n EventType.TASK_FAILED\n})\n\n# Subscribe to constellation events only\nevent_bus.subscribe(constellation_observer, {\n EventType.CONSTELLATION_STARTED,\n EventType.CONSTELLATION_COMPLETED,\n EventType.CONSTELLATION_MODIFIED\n})\n```\n\n#### unsubscribe()\n\nRemove an observer from all event subscriptions:\n\n```python\ndef unsubscribe(self, observer: IEventObserver) -> None\n```\n\n**Parameters:**\n\n- `observer`: The observer object to unsubscribe\n\n**Example:**\n\n```python\n# Clean up observer when done\nevent_bus.unsubscribe(my_observer)\n```\n\n### Event Publishing\n\n#### publish_event()\n\nPublish an event to all subscribed observers:\n\n```python\nasync def publish_event(self, event: Event) -> None\n```\n\n**Parameters:**\n\n- `event`: The event object to publish\n\n**Example:**\n\n```python\nfrom galaxy.core.events import TaskEvent, EventType\nimport time\n\n# Create and publish a task event\nevent = TaskEvent(\n event_type=EventType.TASK_COMPLETED,\n source_id=\"orchestrator\",\n timestamp=time.time(),\n data={\n \"execution_time\": 2.5,\n \"newly_ready_tasks\": [\"task_2\", \"task_3\"]\n },\n task_id=\"task_1\",\n status=\"COMPLETED\",\n result={\"output\": \"success\"}\n)\n\nawait event_bus.publish_event(event)\n```\n\n**Concurrent Notification**: The event bus notifies all observers concurrently using `asyncio.gather()` with `return_exceptions=True`. This means:\n\n- All observers receive events in parallel\n- Slow observers don't block fast ones\n- Exceptions in one observer don't affect others\n- The `publish_event()` call returns after all observers have processed the event\n\n---\n\n## 🔄 Event Flow Patterns\n\n### Pattern 1: Task Execution Flow\n\nThis pattern shows how task events flow through the system:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant EB as EventBus\n participant PO as ProgressObserver\n participant MO as MetricsObserver\n participant VO as VizObserver\n \n Note over O: Start task execution\n O->>EB: publish(TASK_STARTED)\n \n par Concurrent Notification\n EB->>PO: on_event(event)\n EB->>MO: on_event(event)\n EB->>VO: on_event(event)\n end\n \n Note over PO: Track progress\n Note over MO: Record start time\n Note over VO: Display task start\n \n Note over O: Task completes\n O->>EB: publish(TASK_COMPLETED)\n \n par Concurrent Notification\n EB->>PO: on_event(event)\n EB->>MO: on_event(event)\n EB->>VO: on_event(event)\n end\n \n Note over PO: Queue for agent\n Note over MO: Calculate duration\n Note over VO: Update display\n```\n\n### Pattern 2: Constellation Modification Flow\n\nThis pattern shows how modification events coordinate agent and orchestrator:\n\n```mermaid\nsequenceDiagram\n participant A as Agent\n participant EB as EventBus\n participant S as Synchronizer\n participant M as MetricsObserver\n participant V as VizObserver\n \n Note over A: Modify constellation\n A->>EB: publish(CONSTELLATION_MODIFIED)\n \n par Concurrent Notification\n EB->>S: on_event(event)\n EB->>M: on_event(event)\n EB->>V: on_event(event)\n end\n \n Note over S: Complete pending modification\n Note over M: Track modification\n Note over V: Display changes\n```\n\n---\n\n## 💻 Usage Examples\n\n### Example 1: Basic Event Publishing\n\n```python\nimport asyncio\nimport time\nfrom galaxy.core.events import (\n get_event_bus, Event, EventType, IEventObserver\n)\n\nclass SimpleLogger(IEventObserver):\n \"\"\"Simple observer that logs all events.\"\"\"\n \n async def on_event(self, event: Event) -> None:\n print(f\"[{event.timestamp}] {event.event_type.value} from {event.source_id}\")\n\nasync def main():\n # Get event bus and subscribe observer\n event_bus = get_event_bus()\n logger = SimpleLogger()\n event_bus.subscribe(logger)\n \n # Publish some events\n for i in range(3):\n event = Event(\n event_type=EventType.TASK_STARTED,\n source_id=\"test_publisher\",\n timestamp=time.time(),\n data={\"iteration\": i}\n )\n await event_bus.publish_event(event)\n await asyncio.sleep(0.1)\n \n # Clean up\n event_bus.unsubscribe(logger)\n\nasyncio.run(main())\n```\n\n### Example 2: Type-Specific Subscription\n\n```python\nfrom galaxy.core.events import (\n get_event_bus, TaskEvent, ConstellationEvent, \n EventType, IEventObserver\n)\n\nclass TaskOnlyObserver(IEventObserver):\n \"\"\"Observer that only handles task events.\"\"\"\n \n def __init__(self):\n self.task_count = 0\n self.completed_tasks = []\n \n async def on_event(self, event: Event) -> None:\n if isinstance(event, TaskEvent):\n self.task_count += 1\n \n if event.event_type == EventType.TASK_COMPLETED:\n self.completed_tasks.append(event.task_id)\n print(f\"Task {event.task_id} completed. \"\n f\"Total: {len(self.completed_tasks)}\")\n\n# Subscribe only to task events\nobserver = TaskOnlyObserver()\nevent_bus = get_event_bus()\nevent_bus.subscribe(observer, {\n EventType.TASK_STARTED,\n EventType.TASK_COMPLETED,\n EventType.TASK_FAILED\n})\n```\n\n### Example 3: Custom Metrics Collection\n\n```python\nfrom typing import Dict, List\nfrom galaxy.core.events import (\n TaskEvent, ConstellationEvent, EventType, IEventObserver\n)\n\nclass CustomMetricsCollector(IEventObserver):\n \"\"\"Collect custom domain-specific metrics.\"\"\"\n \n def __init__(self):\n self.task_durations: Dict[str, float] = {}\n self.task_start_times: Dict[str, float] = {}\n self.constellation_tasks: Dict[str, List[str]] = {}\n \n async def on_event(self, event: Event) -> None:\n if isinstance(event, TaskEvent):\n await self._handle_task_event(event)\n elif isinstance(event, ConstellationEvent):\n await self._handle_constellation_event(event)\n \n async def _handle_task_event(self, event: TaskEvent) -> None:\n if event.event_type == EventType.TASK_STARTED:\n self.task_start_times[event.task_id] = event.timestamp\n \n elif event.event_type == EventType.TASK_COMPLETED:\n if event.task_id in self.task_start_times:\n duration = event.timestamp - self.task_start_times[event.task_id]\n self.task_durations[event.task_id] = duration\n \n async def _handle_constellation_event(self, event: ConstellationEvent) -> None:\n if event.event_type == EventType.CONSTELLATION_STARTED:\n const_id = event.constellation_id\n self.constellation_tasks[const_id] = []\n \n def get_average_duration(self) -> float:\n \"\"\"Calculate average task duration.\"\"\"\n if not self.task_durations:\n return 0.0\n return sum(self.task_durations.values()) / len(self.task_durations)\n \n def get_slowest_tasks(self, n: int = 5) -> List[tuple]:\n \"\"\"Get the n slowest tasks.\"\"\"\n sorted_tasks = sorted(\n self.task_durations.items(),\n key=lambda x: x[1],\n reverse=True\n )\n return sorted_tasks[:n]\n```\n\n---\n\n## ⚙️ Implementation Details\n\n### Internal Observer Storage\n\nThe EventBus maintains two internal data structures:\n\n```python\nclass EventBus(IEventPublisher):\n def __init__(self):\n # Type-specific observers: EventType -> Set[IEventObserver]\n self._observers: Dict[EventType, Set[IEventObserver]] = {}\n \n # Observers subscribed to all events\n self._all_observers: Set[IEventObserver] = set()\n```\n\n**Storage Strategy:**\n\n| Subscription Type | Storage | Lookup Time | Use Case |\n|-------------------|---------|-------------|----------|\n| All events | `_all_observers` set | O(1) | General monitoring |\n| Specific types | `_observers` dict | O(1) | Targeted handling |\n\n### Concurrent Notification Logic\n\nWhen an event is published, the bus:\n\n1. **Collects relevant observers**: Combines type-specific and all-event observers\n2. **Creates async tasks**: One task per observer\n3. **Executes concurrently**: Uses `asyncio.gather()` with `return_exceptions=True`\n4. **Isolates errors**: Exceptions don't propagate to other observers\n\n```python\nasync def publish_event(self, event: Event) -> None:\n observers_to_notify: Set[IEventObserver] = set()\n \n # Add type-specific observers\n if event.event_type in self._observers:\n observers_to_notify.update(self._observers[event.event_type])\n \n # Add wildcard observers\n observers_to_notify.update(self._all_observers)\n \n # Notify concurrently\n if observers_to_notify:\n tasks = [observer.on_event(event) for observer in observers_to_notify]\n await asyncio.gather(*tasks, return_exceptions=True)\n```\n\n---\n\n## 🎯 Best Practices\n\n### 1. Use Type-Specific Subscriptions\n\nSubscribe only to events you care about:\n\n```python\n# ❌ Bad: Receives all events, must filter manually\nevent_bus.subscribe(observer)\n\n# ✅ Good: Receives only relevant events\nevent_bus.subscribe(observer, {\n EventType.TASK_COMPLETED,\n EventType.CONSTELLATION_MODIFIED\n})\n```\n\n### 2. Handle Errors Gracefully\n\nAlways catch exceptions in observer implementations:\n\n```python\nclass RobustObserver(IEventObserver):\n async def on_event(self, event: Event) -> None:\n try:\n await self._process_event(event)\n except Exception as e:\n self.logger.error(f\"Error processing event: {e}\")\n # Don't re-raise - other observers should continue\n```\n\n### 3. Clean Up Subscriptions\n\nUnsubscribe observers when done to prevent memory leaks:\n\n```python\nclass SessionManager:\n def __init__(self):\n self.observers = []\n \n def setup_observers(self):\n # Create and subscribe observers\n observer = MyObserver()\n event_bus.subscribe(observer)\n self.observers.append(observer)\n \n def cleanup(self):\n # Unsubscribe all observers\n event_bus = get_event_bus()\n for observer in self.observers:\n event_bus.unsubscribe(observer)\n self.observers.clear()\n```\n\n### 4. Use Type Guards\n\nLeverage Python's type system for safer event handling:\n\n```python\nfrom typing import cast\n\nasync def on_event(self, event: Event) -> None:\n if isinstance(event, TaskEvent):\n # Type checker now knows event is TaskEvent\n task_event = cast(TaskEvent, event)\n task_id = task_event.task_id # Type-safe access\n status = task_event.status\n```\n\n---\n\n## 🔗 Related Documentation\n\n- **[Observer System Overview](overview.md)** — High-level architecture and design\n- **[Session Metrics Observer](metrics_observer.md)** — Performance metrics collection\n\n!!! note \"Additional Observer Documentation\"\n For documentation on `ConstellationProgressObserver`, `DAGVisualizationObserver`, `ConstellationModificationSynchronizer`, and `AgentOutputObserver`, refer to their implementation in `galaxy/session/observers/`.\n\n---\n\n## 📋 Summary\n\nThe Event System Core provides:\n\n- **EventBus**: Singleton message broker for system-wide communication\n- **EventType**: Enumeration of all system events\n- **Event Classes**: Type-safe event data structures\n- **Interfaces**: Clear contracts for observers and publishers\n- **Concurrent Execution**: Efficient parallel event processing\n- **Error Isolation**: Robust error handling\n\nThis foundation enables the Galaxy framework to implement a loosely coupled, extensible event-driven architecture.\n"
},
{
"path": "documents/docs/galaxy/observer/metrics_observer.md",
"content": "# Session Metrics Observer\n\nThe **SessionMetricsObserver** collects comprehensive performance metrics and statistics during constellation execution. It tracks task execution times, constellation lifecycle, modifications, and computes detailed statistics for performance analysis.\n\n**Location:** `galaxy/session/observers/base_observer.py`\n\nThe metrics observer is essential for evaluating Galaxy performance, identifying bottlenecks, and analyzing constellation modification patterns for research and optimization.\n\n---\n\n## 🎯 Purpose\n\nThe Metrics Observer provides:\n\n1. **Performance Tracking** — Measure task and constellation execution times\n2. **Success Rate Monitoring** — Track completion and failure rates\n3. **Modification Analytics** — Monitor constellation structural changes\n4. **Statistical Summaries** — Compute aggregated metrics for analysis\n\n---\n\n## 🏗️ Architecture\n\n```mermaid\ngraph TB\n subgraph \"Event Sources\"\n O[Orchestrator]\n A[Agent]\n end\n \n subgraph \"Event System\"\n EB[EventBus]\n end\n \n subgraph \"Metrics Observer\"\n SMO[SessionMetricsObserver]\n TE[Task Events Handler]\n CE[Constellation Events Handler]\n MS[Metrics Storage]\n SC[Statistics Computer]\n end\n \n subgraph \"Outputs\"\n R[result.json]\n L[Logs]\n end\n \n O -->|task events| EB\n A -->|constellation events| EB\n EB -->|notify| SMO\n \n SMO --> TE\n SMO --> CE\n TE --> MS\n CE --> MS\n MS --> SC\n SC --> R\n SC --> L\n \n style SMO fill:#66bb6a,stroke:#333,stroke-width:3px\n style MS fill:#fff4e1,stroke:#333,stroke-width:2px\n style SC fill:#ffa726,stroke:#333,stroke-width:2px\n style EB fill:#4a90e2,stroke:#333,stroke-width:2px,color:#fff\n```\n\n---\n\n## 📊 Metrics Collected\n\nThe observer collects metrics across three categories:\n\n### Task Metrics\n\nTrack individual task execution:\n\n| Metric | Description | Computed |\n|--------|-------------|----------|\n| **task_count** | Total number of tasks started | Real-time |\n| **completed_tasks** | Number of successfully completed tasks | Real-time |\n| **failed_tasks** | Number of failed tasks | Real-time |\n| **total_execution_time** | Sum of all task execution times | Real-time |\n| **task_timings** | Dict mapping task_id → {start, end, duration} | Real-time |\n| **success_rate** | completed / total tasks | Computed |\n| **failure_rate** | failed / total tasks | Computed |\n| **average_task_duration** | Average execution time per task | Computed |\n| **min_task_duration** | Fastest task execution time | Computed |\n| **max_task_duration** | Slowest task execution time | Computed |\n\n### Constellation Metrics\n\nMonitor constellation lifecycle:\n\n| Metric | Description | Computed |\n|--------|-------------|----------|\n| **constellation_count** | Total constellations processed | Real-time |\n| **completed_constellations** | Successfully completed constellations | Real-time |\n| **failed_constellations** | Failed constellations | Real-time |\n| **total_constellation_time** | Total constellation execution time | Real-time |\n| **constellation_timings** | Dict mapping constellation_id → timing data | Real-time |\n| **constellation_success_rate** | completed / total constellations | Computed |\n| **average_constellation_duration** | Average constellation execution time | Computed |\n| **min_constellation_duration** | Fastest constellation | Computed |\n| **max_constellation_duration** | Slowest constellation | Computed |\n| **average_tasks_per_constellation** | Average number of tasks | Computed |\n\n### Modification Metrics\n\nTrack constellation structural changes:\n\n| Metric | Description | Computed |\n|--------|-------------|----------|\n| **constellation_modifications** | Dict mapping constellation_id → modification list | Real-time |\n| **total_modifications** | Total number of modifications | Computed |\n| **constellations_modified** | Number of constellations with modifications | Computed |\n| **average_modifications_per_constellation** | Average modifications per constellation | Computed |\n| **max_modifications_for_single_constellation** | Most-modified constellation | Computed |\n| **most_modified_constellation** | ID of most-modified constellation | Computed |\n| **modification_types_breakdown** | Count by modification type | Computed |\n\n---\n\n## 💻 Implementation\n\n### Initialization\n\n```python\nfrom galaxy.session.observers import SessionMetricsObserver\nimport logging\n\n# Create metrics observer\nmetrics_observer = SessionMetricsObserver(\n session_id=\"galaxy_session_20231113\",\n logger=logging.getLogger(__name__)\n)\n\n# Subscribe to event bus\nfrom galaxy.core.events import get_event_bus\nevent_bus = get_event_bus()\nevent_bus.subscribe(metrics_observer)\n```\n\n**Constructor Parameters:**\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `session_id` | `str` | Yes | Unique identifier for the session |\n| `logger` | `logging.Logger` | No | Logger instance (creates default if None) |\n\n### Internal Metrics Structure\n\nThe observer maintains a comprehensive metrics dictionary:\n\n```python\nself.metrics: Dict[str, Any] = {\n \"session_id\": session_id,\n \n # Task metrics\n \"task_count\": 0,\n \"completed_tasks\": 0,\n \"failed_tasks\": 0,\n \"total_execution_time\": 0.0,\n \"task_timings\": {}, # task_id -> {start, end, duration}\n \n # Constellation metrics\n \"constellation_count\": 0,\n \"completed_constellations\": 0,\n \"failed_constellations\": 0,\n \"total_constellation_time\": 0.0,\n \"constellation_timings\": {}, # constellation_id -> timing data\n \n # Modification tracking\n \"constellation_modifications\": {} # constellation_id -> [modifications]\n}\n```\n\n---\n\n## 🔄 Event Processing\n\n### Task Event Handling\n\nThe observer tracks task lifecycle events:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant EB as EventBus\n participant MO as MetricsObserver\n participant MS as Metrics Storage\n \n O->>EB: TASK_STARTED\n EB->>MO: on_event(event)\n MO->>MS: Increment task_count Record start_time\n \n Note over O: Task executes\n \n O->>EB: TASK_COMPLETED\n EB->>MO: on_event(event)\n MO->>MS: Increment completed_tasks Calculate duration Update total_execution_time\n```\n\n**Processing Logic:**\n\n```python\ndef _handle_task_started(self, event: TaskEvent) -> None:\n \"\"\"Handle TASK_STARTED event.\"\"\"\n self.metrics[\"task_count\"] += 1\n self.metrics[\"task_timings\"][event.task_id] = {\n \"start\": event.timestamp\n }\n\ndef _handle_task_completed(self, event: TaskEvent) -> None:\n \"\"\"Handle TASK_COMPLETED event.\"\"\"\n self.metrics[\"completed_tasks\"] += 1\n \n if event.task_id in self.metrics[\"task_timings\"]:\n duration = (\n event.timestamp - \n self.metrics[\"task_timings\"][event.task_id][\"start\"]\n )\n self.metrics[\"task_timings\"][event.task_id][\"duration\"] = duration\n self.metrics[\"task_timings\"][event.task_id][\"end\"] = event.timestamp\n self.metrics[\"total_execution_time\"] += duration\n\ndef _handle_task_failed(self, event: TaskEvent) -> None:\n \"\"\"Handle TASK_FAILED event.\"\"\"\n self.metrics[\"failed_tasks\"] += 1\n # Also calculate duration for failed tasks\n if event.task_id in self.metrics[\"task_timings\"]:\n duration = (\n event.timestamp - \n self.metrics[\"task_timings\"][event.task_id][\"start\"]\n )\n self.metrics[\"task_timings\"][event.task_id][\"duration\"] = duration\n self.metrics[\"total_execution_time\"] += duration\n```\n\n### Constellation Event Handling\n\nTracks constellation lifecycle and modifications:\n\n```python\ndef _handle_constellation_started(self, event: ConstellationEvent) -> None:\n \"\"\"Handle CONSTELLATION_STARTED event.\"\"\"\n self.metrics[\"constellation_count\"] += 1\n constellation_id = event.constellation_id\n constellation = event.data.get(\"constellation\")\n \n # Store initial statistics\n self.metrics[\"constellation_timings\"][constellation_id] = {\n \"start_time\": event.timestamp,\n \"initial_statistics\": (\n constellation.get_statistics() if constellation else {}\n ),\n \"processing_start_time\": event.data.get(\"processing_start_time\"),\n \"processing_end_time\": event.data.get(\"processing_end_time\"),\n \"processing_duration\": event.data.get(\"processing_duration\"),\n }\n\ndef _handle_constellation_completed(self, event: ConstellationEvent) -> None:\n \"\"\"Handle CONSTELLATION_COMPLETED event.\"\"\"\n self.metrics[\"completed_constellations\"] += 1\n constellation_id = event.constellation_id\n constellation = event.data.get(\"constellation\")\n \n # Calculate duration and store final statistics\n duration = (\n event.timestamp - \n self.metrics[\"constellation_timings\"][constellation_id][\"start_time\"]\n if constellation_id in self.metrics[\"constellation_timings\"]\n else None\n )\n \n if constellation_id in self.metrics[\"constellation_timings\"]:\n self.metrics[\"constellation_timings\"][constellation_id].update({\n \"end_time\": event.timestamp,\n \"duration\": duration,\n \"final_statistics\": (\n constellation.get_statistics() if constellation else {}\n ),\n })\n```\n\n### Modification Tracking\n\nTracks constellation structural changes with detailed change detection:\n\n```python\ndef _handle_constellation_modified(self, event: ConstellationEvent) -> None:\n \"\"\"Handle CONSTELLATION_MODIFIED event.\"\"\"\n constellation_id = event.constellation_id\n \n # Initialize modifications list if needed\n if constellation_id not in self.metrics[\"constellation_modifications\"]:\n self.metrics[\"constellation_modifications\"][constellation_id] = []\n \n if hasattr(event, \"data\") and event.data:\n old_constellation = event.data.get(\"old_constellation\")\n new_constellation = event.data.get(\"new_constellation\")\n \n # Calculate changes using VisualizationChangeDetector\n changes = None\n if old_constellation and new_constellation:\n changes = VisualizationChangeDetector.calculate_constellation_changes(\n old_constellation, new_constellation\n )\n \n # Store modification record\n modification_record = {\n \"timestamp\": event.timestamp,\n \"modification_type\": event.data.get(\"modification_type\", \"unknown\"),\n \"on_task_id\": event.data.get(\"on_task_id\", []),\n \"changes\": changes,\n \"new_statistics\": (\n new_constellation.get_statistics() if new_constellation else {}\n ),\n \"processing_start_time\": event.data.get(\"processing_start_time\"),\n \"processing_end_time\": event.data.get(\"processing_end_time\"),\n \"processing_duration\": event.data.get(\"processing_duration\"),\n }\n \n self.metrics[\"constellation_modifications\"][constellation_id].append(\n modification_record\n )\n```\n\n---\n\n## 📖 API Reference\n\n### Constructor\n\n```python\ndef __init__(self, session_id: str, logger: Optional[logging.Logger] = None)\n```\n\nInitialize the metrics observer.\n\n**Parameters:**\n\n- `session_id` — Unique identifier for the session\n- `logger` — Optional logger instance (creates default if None)\n\n### get_metrics()\n\n```python\ndef get_metrics(self) -> Dict[str, Any]\n```\n\nGet collected metrics with computed statistics.\n\n**Returns:**\n\nDictionary containing:\n- All raw metrics (counts, timings, etc.)\n- `task_statistics` — Computed task metrics\n- `constellation_statistics` — Computed constellation metrics\n- `modification_statistics` — Computed modification metrics\n\n**Example:**\n\n```python\n# After constellation execution\nmetrics = metrics_observer.get_metrics()\n\n# Access task statistics\nprint(f\"Total tasks: {metrics['task_statistics']['total_tasks']}\")\nprint(f\"Success rate: {metrics['task_statistics']['success_rate']:.2%}\")\nprint(f\"Avg duration: {metrics['task_statistics']['average_task_duration']:.2f}s\")\n\n# Access constellation statistics\nprint(f\"Total constellations: {metrics['constellation_statistics']['total_constellations']}\")\nprint(f\"Avg tasks per constellation: {metrics['constellation_statistics']['average_tasks_per_constellation']:.1f}\")\n\n# Access modification statistics\nprint(f\"Total modifications: {metrics['modification_statistics']['total_modifications']}\")\nprint(f\"Modification types: {metrics['modification_statistics']['modification_types_breakdown']}\")\n```\n\n---\n\n## 📊 Computed Statistics\n\nThe observer computes three categories of statistics:\n\n### Task Statistics\n\n```python\n{\n \"total_tasks\": 10,\n \"completed_tasks\": 8,\n \"failed_tasks\": 2,\n \"success_rate\": 0.8,\n \"failure_rate\": 0.2,\n \"average_task_duration\": 2.5,\n \"min_task_duration\": 0.5,\n \"max_task_duration\": 5.2,\n \"total_task_execution_time\": 25.0\n}\n```\n\n### Constellation Statistics\n\n```python\n{\n \"total_constellations\": 1,\n \"completed_constellations\": 1,\n \"failed_constellations\": 0,\n \"success_rate\": 1.0,\n \"average_constellation_duration\": 30.5,\n \"min_constellation_duration\": 30.5,\n \"max_constellation_duration\": 30.5,\n \"total_constellation_time\": 30.5,\n \"average_tasks_per_constellation\": 10.0\n}\n```\n\n### Modification Statistics\n\n```python\n{\n \"total_modifications\": 3,\n \"constellations_modified\": 1,\n \"average_modifications_per_constellation\": 3.0,\n \"max_modifications_for_single_constellation\": 3,\n \"most_modified_constellation\": \"const_123\",\n \"modifications_per_constellation\": {\n \"const_123\": 3\n },\n \"modification_types_breakdown\": {\n \"add_tasks\": 2,\n \"modify_dependencies\": 1\n }\n}\n```\n\n---\n\n## 🔍 Usage Examples\n\n### Example 1: Basic Metrics Collection\n\n```python\nimport asyncio\nfrom galaxy.core.events import get_event_bus\nfrom galaxy.session.observers import SessionMetricsObserver\n\nasync def collect_metrics():\n \"\"\"Collect and display metrics for constellation execution.\"\"\"\n \n # Create and subscribe metrics observer\n metrics_observer = SessionMetricsObserver(session_id=\"demo_session\")\n event_bus = get_event_bus()\n event_bus.subscribe(metrics_observer)\n \n # Execute constellation (orchestrator will publish events)\n await orchestrator.execute_constellation(constellation)\n \n # Retrieve metrics\n metrics = metrics_observer.get_metrics()\n \n # Display summary\n print(\"\\n=== Execution Summary ===\")\n print(f\"Session: {metrics['session_id']}\")\n print(f\"Tasks: {metrics['task_count']} total, \"\n f\"{metrics['completed_tasks']} completed, \"\n f\"{metrics['failed_tasks']} failed\")\n print(f\"Total execution time: {metrics['total_execution_time']:.2f}s\")\n \n # Display task statistics\n task_stats = metrics['task_statistics']\n print(f\"\\nTask Success Rate: {task_stats['success_rate']:.1%}\")\n print(f\"Average Task Duration: {task_stats['average_task_duration']:.2f}s\")\n print(f\"Fastest Task: {task_stats['min_task_duration']:.2f}s\")\n print(f\"Slowest Task: {task_stats['max_task_duration']:.2f}s\")\n \n # Clean up\n event_bus.unsubscribe(metrics_observer)\n\nasyncio.run(collect_metrics())\n```\n\n### Example 2: Performance Analysis\n\n```python\ndef analyze_performance(metrics_observer: SessionMetricsObserver):\n \"\"\"Analyze performance metrics and identify bottlenecks.\"\"\"\n \n metrics = metrics_observer.get_metrics()\n task_timings = metrics['task_timings']\n \n # Find slowest tasks\n sorted_tasks = sorted(\n task_timings.items(),\n key=lambda x: x[1].get('duration', 0),\n reverse=True\n )\n \n print(\"\\n=== Top 5 Slowest Tasks ===\")\n for task_id, timing in sorted_tasks[:5]:\n duration = timing.get('duration', 0)\n print(f\"{task_id}: {duration:.2f}s\")\n \n # Analyze modification patterns\n mod_stats = metrics['modification_statistics']\n if mod_stats['total_modifications'] > 0:\n print(f\"\\n=== Modification Analysis ===\")\n print(f\"Total Modifications: {mod_stats['total_modifications']}\")\n print(f\"Average per Constellation: \"\n f\"{mod_stats['average_modifications_per_constellation']:.1f}\")\n print(f\"Most Modified: {mod_stats['most_modified_constellation']}\")\n print(\"\\nModification Types:\")\n for mod_type, count in mod_stats['modification_types_breakdown'].items():\n print(f\" {mod_type}: {count}\")\n```\n\n### Example 3: Export Metrics to JSON\n\n```python\nimport json\nfrom pathlib import Path\n\ndef export_metrics(metrics_observer: SessionMetricsObserver, output_path: str):\n \"\"\"Export metrics to JSON file for analysis.\"\"\"\n \n metrics = metrics_observer.get_metrics()\n \n # Convert to JSON-serializable format\n output_data = {\n \"session_id\": metrics[\"session_id\"],\n \"task_statistics\": metrics[\"task_statistics\"],\n \"constellation_statistics\": metrics[\"constellation_statistics\"],\n \"modification_statistics\": metrics[\"modification_statistics\"],\n \"raw_metrics\": {\n \"task_count\": metrics[\"task_count\"],\n \"completed_tasks\": metrics[\"completed_tasks\"],\n \"failed_tasks\": metrics[\"failed_tasks\"],\n \"total_execution_time\": metrics[\"total_execution_time\"],\n \"constellation_count\": metrics[\"constellation_count\"],\n }\n }\n \n # Write to file\n output_file = Path(output_path)\n output_file.parent.mkdir(parents=True, exist_ok=True)\n \n with open(output_file, 'w') as f:\n json.dump(output_data, f, indent=2)\n \n print(f\"Metrics exported to: {output_file}\")\n```\n\n---\n\n## 🎓 Best Practices\n\n### 1. Session ID Naming\n\nUse descriptive session IDs for easier analysis:\n\n```python\n# ✅ Good: Descriptive session ID\nsession_id = f\"galaxy_session_{task_type}_{timestamp}\"\n\n# ❌ Bad: Generic session ID\nsession_id = \"session_1\"\n```\n\n### 2. Metrics Export\n\nExport metrics immediately after execution:\n\n```python\ntry:\n await orchestrator.execute_constellation(constellation)\nfinally:\n # Always export metrics, even if execution failed\n metrics = metrics_observer.get_metrics()\n export_metrics(metrics, \"results/metrics.json\")\n```\n\n### 3. Memory Management\n\nClear large timing dictionaries for long-running sessions:\n\n```python\n# After processing metrics\nmetrics_observer.metrics[\"task_timings\"].clear()\nmetrics_observer.metrics[\"constellation_timings\"].clear()\n```\n\n---\n\n## 🔗 Related Documentation\n\n- **[Observer System Overview](overview.md)** — Architecture and design\n- **[Event System Core](event_system.md)** — Event types and EventBus\n\n!!! note \"Additional Resources\"\n For information on constellation execution and orchestration, see the constellation orchestrator documentation in `galaxy/constellation/orchestrator/`.\n\n---\n\n## 📋 Summary\n\nThe Session Metrics Observer:\n\n- **Collects** comprehensive performance metrics\n- **Tracks** task and constellation execution times\n- **Monitors** modification patterns\n- **Computes** statistical summaries\n- **Exports** data for analysis\n\nThis observer is essential for performance evaluation, bottleneck identification, and research analysis of Galaxy's constellation execution.\n"
},
{
"path": "documents/docs/galaxy/observer/overview.md",
"content": "# Observer System — Overview\n\nThe **Observer System** in UFO Galaxy implements an event-driven architecture that enables real-time monitoring, visualization, and coordination of constellation execution. It provides a decoupled, extensible mechanism for components to react to system events without tight coupling.\n\nThe system implements the classic **Observer Pattern** (also known as Publish-Subscribe), enabling loose coupling between event producers and consumers. This allows the system to be extended with new observers without modifying existing code.\n\n---\n\n## 🎯 Purpose and Design Goals\n\nThe observer system serves several critical functions in the Galaxy framework:\n\n1. **Real-time Monitoring** — Track task execution, constellation lifecycle, and system events\n2. **Visualization** — Provide live updates for DAG topology and execution progress\n3. **Metrics Collection** — Gather performance statistics and execution data\n4. **Synchronization** — Coordinate between agent modifications and orchestrator execution\n5. **Agent Output Handling** — Display agent responses and actions in real-time\n\n---\n\n## 🏗️ Architecture Overview\n\nThe observer system consists of three main layers:\n\n```mermaid\ngraph TB\n subgraph \"Event Publishers\"\n A1[Orchestrator]\n A2[Agent]\n A3[Device Manager]\n end\n \n subgraph \"Event Bus Layer\"\n B[EventBus Singleton]\n end\n \n subgraph \"Observer Layer\"\n C1[ConstellationProgressObserver]\n C2[SessionMetricsObserver]\n C3[DAGVisualizationObserver]\n C4[ConstellationModificationSynchronizer]\n C5[AgentOutputObserver]\n end\n \n subgraph \"Handler Layer\"\n D1[TaskVisualizationHandler]\n D2[ConstellationVisualizationHandler]\n end\n \n A1 -->|publish events| B\n A2 -->|publish events| B\n A3 -->|publish events| B\n \n B -->|notify| C1\n B -->|notify| C2\n B -->|notify| C3\n B -->|notify| C4\n B -->|notify| C5\n \n C3 -->|delegate| D1\n C3 -->|delegate| D2\n \n style B fill:#4a90e2,stroke:#333,stroke-width:3px,color:#fff\n style C1 fill:#66bb6a,stroke:#333,stroke-width:2px\n style C2 fill:#66bb6a,stroke:#333,stroke-width:2px\n style C3 fill:#66bb6a,stroke:#333,stroke-width:2px\n style C4 fill:#ffa726,stroke:#333,stroke-width:2px\n style C5 fill:#66bb6a,stroke:#333,stroke-width:2px\n```\n\n**Architecture Layers:**\n\n| Layer | Component | Responsibility |\n|-------|-----------|----------------|\n| **Event Publishers** | Orchestrator, Agent, Device Manager | Generate events during system operation |\n| **Event Bus** | `EventBus` singleton | Central message broker, manages subscriptions and routing |\n| **Observers** | 5 specialized observers | React to specific event types and perform actions |\n| **Handlers** | Task & Constellation handlers | Delegate visualization logic for specific components |\n\n---\n\n## 📊 Core Components\n\n### Event System Core\n\nThe foundation of the observer system consists of:\n\n| Component | Location | Description |\n|-----------|----------|-------------|\n| **EventBus** | `galaxy/core/events.py` | Central message broker managing subscriptions |\n| **EventType** | `galaxy/core/events.py` | Enumeration of all system event types |\n| **Event Classes** | `galaxy/core/events.py` | Base (`Event`) and specialized (`TaskEvent`, `ConstellationEvent`, `AgentEvent`, `DeviceEvent`) event data structures |\n| **Interfaces** | `galaxy/core/events.py` | `IEventObserver`, `IEventPublisher` contracts |\n\nFor detailed documentation of the event system core components, see the **[Event System Core](event_system.md)** documentation.\n\n### Observer Implementations\n\nFive specialized observers handle different aspects of system monitoring:\n\n| Observer | File Location | Primary Role | Key Features |\n|----------|---------------|--------------|--------------|\n| **ConstellationProgressObserver** | `galaxy/session/observers/base_observer.py` | Task progress tracking | Queues completion events for agent, coordinates task lifecycle |\n| **SessionMetricsObserver** | `galaxy/session/observers/base_observer.py` | Performance metrics | Collects timing, success rates, modification statistics |\n| **DAGVisualizationObserver** | `galaxy/session/observers/dag_visualization_observer.py` | Real-time visualization | Displays constellation topology and execution flow |\n| **ConstellationModificationSynchronizer** | `galaxy/session/observers/constellation_sync_observer.py` | Modification coordination | Prevents race conditions between agent and orchestrator |\n| **AgentOutputObserver** | `galaxy/session/observers/agent_output_observer.py` | Agent interaction display | Shows agent responses and actions in real-time |\n\n---\n\n## 🔄 Event Flow\n\nThe following diagram illustrates how events flow through the system:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant EB as EventBus\n participant CPO as ProgressObserver\n participant SMO as MetricsObserver\n participant DVO as VisualizationObserver\n participant A as Agent\n \n O->>EB: publish(TASK_STARTED)\n EB->>CPO: on_event(event)\n EB->>SMO: on_event(event)\n EB->>DVO: on_event(event)\n \n Note over DVO: Display task start\n Note over SMO: Increment task count\n \n O->>EB: publish(TASK_COMPLETED)\n EB->>CPO: on_event(event)\n EB->>SMO: on_event(event)\n EB->>DVO: on_event(event)\n \n CPO->>A: add_task_completion_event()\n Note over A: Process result, modify constellation\n \n A->>EB: publish(CONSTELLATION_MODIFIED)\n EB->>SMO: on_event(event)\n EB->>DVO: on_event(event)\n \n Note over DVO: Display updated DAG\n Note over SMO: Track modification\n```\n\nThe event flow demonstrates how a single action (task completion) triggers multiple observers, each performing their specialized function without interfering with others.\n\n---\n\n## 📋 Event Types\n\nThe system defines the following event types:\n\n### Task Events\n\nTrack individual task execution lifecycle:\n\n| Event Type | Trigger | Data Includes |\n|------------|---------|---------------|\n| `TASK_STARTED` | Task begins execution | task_id, status, constellation_id |\n| `TASK_COMPLETED` | Task finishes successfully | task_id, result, execution_time, newly_ready_tasks |\n| `TASK_FAILED` | Task encounters error | task_id, error, retry_info |\n\n### Constellation Events\n\nMonitor constellation-level operations:\n\n| Event Type | Trigger | Data Includes |\n|------------|---------|---------------|\n| `CONSTELLATION_STARTED` | Constellation begins processing | constellation, initial_statistics, processing_time |\n| `CONSTELLATION_COMPLETED` | All tasks finished | constellation, final_statistics, execution_time |\n| `CONSTELLATION_FAILED` | Constellation encounters error | constellation, error |\n| `CONSTELLATION_MODIFIED` | Structure changed by agent | old_constellation, new_constellation, on_task_id, modification_type, changes |\n\n### Agent Events\n\nDisplay agent interactions:\n\n| Event Type | Trigger | Data Includes |\n|------------|---------|---------------|\n| `AGENT_RESPONSE` | Agent generates response | agent_name, agent_type, response_data |\n| `AGENT_ACTION` | Agent executes action | agent_name, action_type, actions |\n\n### Device Events\n\nMonitor device status (used by client):\n\n| Event Type | Trigger | Data Includes |\n|------------|---------|---------------|\n| `DEVICE_CONNECTED` | Device joins pool | device_id, device_status, device_info |\n| `DEVICE_DISCONNECTED` | Device leaves pool | device_id, device_status |\n| `DEVICE_STATUS_CHANGED` | Device state changes | device_id, device_status, all_devices |\n\n---\n\n## 🚀 Usage Example\n\nHere's a complete example showing how observers are initialized and used in a Galaxy session:\n\n```python\nfrom galaxy.core.events import get_event_bus, EventType\nfrom galaxy.session.observers import (\n ConstellationProgressObserver,\n SessionMetricsObserver,\n DAGVisualizationObserver,\n ConstellationModificationSynchronizer,\n AgentOutputObserver\n)\n\n# Get the global event bus\nevent_bus = get_event_bus()\n\n# 1. Create progress observer for agent coordination\nprogress_observer = ConstellationProgressObserver(agent=constellation_agent)\nevent_bus.subscribe(progress_observer)\n\n# 2. Create metrics observer for performance tracking\nmetrics_observer = SessionMetricsObserver(\n session_id=\"my_session\",\n logger=logger\n)\nevent_bus.subscribe(metrics_observer)\n\n# 3. Create visualization observer for real-time display\nviz_observer = DAGVisualizationObserver(enable_visualization=True)\nevent_bus.subscribe(viz_observer)\n\n# 4. Create synchronizer to prevent race conditions\nsynchronizer = ConstellationModificationSynchronizer(\n orchestrator=orchestrator,\n logger=logger\n)\nevent_bus.subscribe(synchronizer)\n\n# 5. Create agent output observer for displaying interactions\nagent_output_observer = AgentOutputObserver(presenter_type=\"rich\")\nevent_bus.subscribe(agent_output_observer)\n\n# Execute constellation\nawait orchestrator.execute_constellation(constellation)\n\n# Retrieve collected metrics\nmetrics = metrics_observer.get_metrics()\nprint(f\"Tasks completed: {metrics['completed_tasks']}\")\nprint(f\"Total execution time: {metrics['total_execution_time']:.2f}s\")\nprint(f\"Modifications: {metrics['constellation_modifications']}\")\n```\n\n---\n\n## 🔑 Key Benefits\n\n### 1. Decoupling\n\nEvents decouple components — publishers don't need to know about observers:\n\n- **Orchestrator** publishes task events without knowing who's listening\n- **Agent** publishes modification events without coordinating with orchestrator\n- **New observers** can be added without changing existing code\n\n### 2. Extensibility\n\nAdd custom observers for new functionality:\n\n```python\nfrom galaxy.core.events import IEventObserver, Event, EventType\n\nclass CustomMetricsObserver(IEventObserver):\n \"\"\"Custom observer for domain-specific metrics.\"\"\"\n \n def __init__(self):\n self.custom_metrics = {}\n \n async def on_event(self, event: Event) -> None:\n if event.event_type == EventType.TASK_COMPLETED:\n # Collect custom metrics\n task_type = event.data.get(\"task_type\")\n if task_type not in self.custom_metrics:\n self.custom_metrics[task_type] = []\n \n self.custom_metrics[task_type].append({\n \"duration\": event.data.get(\"execution_time\"),\n \"result\": event.result\n })\n\n# Subscribe to specific events\nevent_bus = get_event_bus()\ncustom_observer = CustomMetricsObserver()\nevent_bus.subscribe(custom_observer, {EventType.TASK_COMPLETED})\n```\n\n### 3. Concurrent Execution\n\nAll observers are notified concurrently using `asyncio.gather()`:\n\n- No observer blocks another\n- Exceptions in one observer don't affect others\n- Efficient parallel processing\n\n### 4. Type-Safe Event Handling\n\nSpecialized event classes provide type safety:\n\n```python\nasync def on_event(self, event: Event) -> None:\n if isinstance(event, TaskEvent):\n # TaskEvent-specific handling\n task_id = event.task_id # Type-safe access\n status = event.status\n \n elif isinstance(event, ConstellationEvent):\n # ConstellationEvent-specific handling\n constellation_id = event.constellation_id\n state = event.constellation_state\n```\n\n---\n\n## 📚 Component Documentation\n\nExplore detailed documentation for each observer:\n\n- **[Session Metrics Observer](metrics_observer.md)** — Performance metrics and statistics collection\n- **[Event System Core](event_system.md)** — Event bus, event types, and interfaces\n\n!!! note \"Additional Observers\"\n Documentation for `ConstellationProgressObserver`, `DAGVisualizationObserver`, `ConstellationModificationSynchronizer`, and `AgentOutputObserver` is available in their source code files. These observers handle task progress tracking, real-time visualization, modification synchronization, and agent output display respectively.\n\n---\n\n## 🔗 Related Documentation\n\n- **[Constellation Orchestrator](../constellation_orchestrator/overview.md)** — Event publishers for task execution\n- **[Constellation Agent](../constellation_agent/overview.md)** — Event publishers for agent operations\n- **[Performance Metrics](../evaluation/performance_metrics.md)** — How metrics are collected and analyzed\n- **[Event-Driven Coordination](../constellation_orchestrator/event_driven_coordination.md)** — Deep dive into event system architecture\n\n---\n\n## 💡 Best Practices\n\n### Observer Lifecycle Management\n\nProperly manage observer subscriptions to prevent memory leaks:\n\n```python\n# Subscribe observers\nobservers = [progress_observer, metrics_observer, viz_observer]\nfor observer in observers:\n event_bus.subscribe(observer)\n\ntry:\n # Execute constellation\n await orchestrator.execute_constellation(constellation)\nfinally:\n # Clean up observers\n for observer in observers:\n event_bus.unsubscribe(observer)\n```\n\n### Event-Specific Subscription\n\nSubscribe only to relevant events for efficiency:\n\n```python\n# Instead of subscribing to all events\nevent_bus.subscribe(observer) # Receives ALL events\n\n# Subscribe to specific event types\nevent_bus.subscribe(observer, {\n EventType.TASK_COMPLETED,\n EventType.TASK_FAILED,\n EventType.CONSTELLATION_MODIFIED\n})\n```\n\n### Error Handling in Observers\n\nAlways handle exceptions gracefully:\n\n```python\nasync def on_event(self, event: Event) -> None:\n try:\n # Process event\n await self._handle_event(event)\n except Exception as e:\n self.logger.error(f\"Error processing event: {e}\")\n # Don't re-raise - let other observers continue\n```\n\n---\n\n## 🎓 Summary\n\nThe Observer System provides a robust, event-driven foundation for monitoring and coordinating Galaxy's constellation execution:\n\n- **Event Bus** acts as central message broker\n- **5 specialized observers** handle different aspects of monitoring\n- **Loose coupling** enables extensibility and maintainability\n- **Concurrent execution** ensures efficient event processing\n- **Type-safe events** provide clear contracts and error prevention\n\nFor implementation details of specific observers, refer to the individual component documentation pages linked above.\n"
},
{
"path": "documents/docs/galaxy/observer/progress_observer.md",
"content": "# Constellation Progress Observer\n\nThe **ConstellationProgressObserver** is responsible for tracking task execution progress and coordinating between the orchestrator and the agent. It acts as the bridge that enables the agent to react to task completion events and make necessary constellation modifications.\n\n**Location:** `galaxy/session/observers/base_observer.py`\n\n## Purpose\n\nThe Progress Observer serves two critical functions:\n\n- **Task Completion Coordination** — Queues task completion events for the agent to process\n- **Constellation Event Handling** — Notifies the agent when constellation execution completes\n\n## Architecture\n\n```mermaid\ngraph TB\n subgraph \"Orchestrator Layer\"\n O[TaskConstellationOrchestrator]\n end\n \n subgraph \"Event System\"\n EB[EventBus]\n end\n \n subgraph \"Observer Layer\"\n CPO[ConstellationProgressObserver]\n end\n \n subgraph \"Agent Layer\"\n A[ConstellationAgent]\n Q[Task Completion Queue]\n end\n \n O -->|publish events| EB\n EB -->|notify| CPO\n CPO -->|queue events| Q\n A -->|process from| Q\n \n style CPO fill:#66bb6a,stroke:#333,stroke-width:3px\n style Q fill:#fff4e1,stroke:#333,stroke-width:2px\n style EB fill:#4a90e2,stroke:#333,stroke-width:2px,color:#fff\n```\n\n**Component Interaction:**\n\n| Component | Role | Communication |\n|-----------|------|---------------|\n| **Orchestrator** | Executes tasks, publishes events | → EventBus |\n| **EventBus** | Distributes events | → Progress Observer |\n| **Progress Observer** | Filters & queues relevant events | → Agent Queue |\n| **Agent** | Processes completions, modifies constellation | ← Agent Queue |\n\n## Event Handling\n\nThe Progress Observer handles two types of events:\n\n### Task Events\n\nMonitors task execution lifecycle and queues completion events:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant EB as EventBus\n participant PO as ProgressObserver\n participant Q as Agent Queue\n participant A as Agent\n \n O->>EB: TASK_STARTED\n EB->>PO: on_event(event)\n Note over PO: Store task result Log progress\n \n O->>EB: TASK_COMPLETED\n EB->>PO: on_event(event)\n Note over PO: Store result Queue for agent\n PO->>Q: add_task_completion_event()\n \n Note over A: Agent in Continue state waiting for events\n A->>Q: get event\n Note over A: Process result Modify constellation\n```\n\n**Handled Event Types:**\n\n| Event Type | Action | Data Stored |\n|------------|--------|-------------|\n| `TASK_STARTED` | Store task result placeholder | task_id, status, timestamp |\n| `TASK_COMPLETED` | Store result, queue for agent | task_id, status, result, timestamp |\n| `TASK_FAILED` | Store error, queue for agent | task_id, status, error, timestamp |\n\n### Constellation Events\n\nHandles constellation lifecycle events:\n\n| Event Type | Action | Effect |\n|------------|--------|--------|\n| `CONSTELLATION_COMPLETED` | Queue completion event for agent | Wakes up agent's Continue state to process final results |\n\n## Implementation\n\n### Initialization\n\n```python\nfrom galaxy.session.observers import ConstellationProgressObserver\nfrom galaxy.agents import ConstellationAgent\n\n# Create progress observer with agent reference\nagent = ConstellationAgent(orchestrator=orchestrator)\nprogress_observer = ConstellationProgressObserver(agent=agent)\n\n# Subscribe to event bus\nfrom galaxy.core.events import get_event_bus\nevent_bus = get_event_bus()\nevent_bus.subscribe(progress_observer)\n```\n\n**Constructor Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `agent` | `ConstellationAgent` | The agent that will process queued events |\n\n### Internal Data Structures\n\nThe observer maintains:\n\n```python\nclass ConstellationProgressObserver(IEventObserver):\n def __init__(self, agent: ConstellationAgent):\n self.agent = agent\n \n # Task results storage: task_id -> result dict\n self.task_results: Dict[str, Dict[str, Any]] = {}\n \n self.logger = logging.getLogger(__name__)\n```\n\n**Task Result Structure:**\n\n```python\n{\n \"task_id\": \"task_123\",\n \"status\": \"COMPLETED\", # or \"FAILED\"\n \"result\": {...}, # Task execution result\n \"error\": None, # Exception if failed\n \"timestamp\": 1234567890.123\n}\n```\n\n## Event Processing Flow\n\n### Task Event Processing\n\n```python\nasync def _handle_task_event(self, event: TaskEvent) -> None:\n \"\"\"Handle task progress events and queue them for agent processing.\"\"\"\n \n try:\n self.logger.info(\n f\"Task progress: {event.task_id} -> {event.status}. \"\n f\"Event Type: {event.event_type}\"\n )\n \n # 1. Store task result for tracking\n self.task_results[event.task_id] = {\n \"task_id\": event.task_id,\n \"status\": event.status,\n \"result\": event.result,\n \"error\": event.error,\n \"timestamp\": event.timestamp,\n }\n \n # 2. Queue completion/failure events for agent\n if event.event_type in [EventType.TASK_COMPLETED, EventType.TASK_FAILED]:\n await self.agent.add_task_completion_event(event)\n \n except Exception as e:\n self.logger.error(f\"Error handling task event: {e}\", exc_info=True)\n```\n\n**Processing Steps:**\n\n1. **Log Progress**: Record task status change\n2. **Store Result**: Update internal task_results dictionary\n3. **Queue for Agent**: If completion/failure, add to agent's queue\n4. **Error Handling**: Catch and log any exceptions\n\n### Constellation Event Processing\n\n```python\nasync def _handle_constellation_event(self, event: ConstellationEvent) -> None:\n \"\"\"Handle constellation update events.\"\"\"\n \n try:\n if event.event_type == EventType.CONSTELLATION_COMPLETED:\n # Queue completion event for agent\n await self.agent.add_constellation_completion_event(event)\n \n except Exception as e:\n self.logger.error(\n f\"Error handling constellation event: {e}\", \n exc_info=True\n )\n```\n\n## API Reference\n\n### Constructor\n\n```python\ndef __init__(self, agent: ConstellationAgent)\n```\n\nInitialize the progress observer with a reference to the agent.\n\n**Parameters:**\n\n- `agent` — `ConstellationAgent` instance that will process queued events\n\n**Example:**\n\n```python\nfrom galaxy.agents import ConstellationAgent\nfrom galaxy.session.observers import ConstellationProgressObserver\n\nagent = ConstellationAgent(orchestrator=orchestrator)\nprogress_observer = ConstellationProgressObserver(agent=agent)\n```\n\n### Event Handler\n\n```python\nasync def on_event(self, event: Event) -> None\n```\n\nHandle constellation-related events (TaskEvent or ConstellationEvent).\n\n**Parameters:**\n\n- `event` — Event instance (TaskEvent or ConstellationEvent)\n\n**Behavior:**\n\n- Filters events by type (TaskEvent vs ConstellationEvent)\n- Delegates to appropriate handler method\n- Logs progress and stores results\n- Queues completion events for agent\n\n## Usage Examples\n\n### Example 1: Basic Setup\n\n```python\nimport asyncio\nfrom galaxy.core.events import get_event_bus\nfrom galaxy.agents import ConstellationAgent\nfrom galaxy.constellation import TaskConstellationOrchestrator\nfrom galaxy.session.observers import ConstellationProgressObserver\n\nasync def setup_progress_tracking():\n \"\"\"Set up progress tracking for constellation execution.\"\"\"\n \n # Create orchestrator and agent\n orchestrator = TaskConstellationOrchestrator()\n agent = ConstellationAgent(orchestrator=orchestrator)\n \n # Create and subscribe progress observer\n progress_observer = ConstellationProgressObserver(agent=agent)\n event_bus = get_event_bus()\n event_bus.subscribe(progress_observer)\n \n # Now orchestrator events will be tracked and queued for agent\n return agent, orchestrator, progress_observer\n```\n\n### Example 2: Monitoring Task Results\n\n```python\nasync def monitor_task_progress(observer: ConstellationProgressObserver):\n \"\"\"Monitor task execution progress.\"\"\"\n \n # Wait for some tasks to complete\n await asyncio.sleep(5)\n \n # Access stored results\n for task_id, result in observer.task_results.items():\n status = result[\"status\"]\n timestamp = result[\"timestamp\"]\n \n if status == \"COMPLETED\":\n print(f\"✅ Task {task_id} completed at {timestamp}\")\n print(f\" Result: {result['result']}\")\n elif status == \"FAILED\":\n print(f\"❌ Task {task_id} failed at {timestamp}\")\n print(f\" Error: {result['error']}\")\n```\n\n### Example 3: Custom Progress Observer\n\n```python\nfrom galaxy.core.events import IEventObserver, TaskEvent, EventType\n\nclass CustomProgressObserver(IEventObserver):\n \"\"\"Custom observer with additional progress tracking.\"\"\"\n \n def __init__(self, agent, on_progress_callback=None):\n self.agent = agent\n self.on_progress_callback = on_progress_callback\n \n # Track progress statistics\n self.total_tasks = 0\n self.completed_tasks = 0\n self.failed_tasks = 0\n \n async def on_event(self, event: Event) -> None:\n if isinstance(event, TaskEvent):\n # Update statistics\n if event.event_type == EventType.TASK_STARTED:\n self.total_tasks += 1\n elif event.event_type == EventType.TASK_COMPLETED:\n self.completed_tasks += 1\n elif event.event_type == EventType.TASK_FAILED:\n self.failed_tasks += 1\n \n # Call custom callback\n if self.on_progress_callback:\n progress = self.completed_tasks / self.total_tasks if self.total_tasks > 0 else 0\n self.on_progress_callback(progress, event)\n \n # Queue for agent\n if event.event_type in [EventType.TASK_COMPLETED, EventType.TASK_FAILED]:\n await self.agent.add_task_completion_event(event)\n\n# Usage\ndef progress_callback(progress, event):\n print(f\"Progress: {progress*100:.1f}% - {event.task_id} {event.status}\")\n\ncustom_observer = CustomProgressObserver(\n agent=agent,\n on_progress_callback=progress_callback\n)\nevent_bus.subscribe(custom_observer)\n```\n\n## Integration with Agent\n\nThe Progress Observer integrates tightly with the ConstellationAgent's state machine:\n\n### Agent Queue Interface\n\nThe observer calls these agent methods:\n\n```python\n# Queue task completion event\nawait self.agent.add_task_completion_event(task_event)\n\n# Queue constellation completion event\nawait self.agent.add_constellation_completion_event(constellation_event)\n```\n\n### Agent Processing\n\nThe agent processes queued events in its `Continue` state:\n\n```mermaid\nstateDiagram-v2\n [*] --> Continue: Task completes\n Continue --> ProcessEvent: Get event from queue\n ProcessEvent --> UpdateConstellation: Event is TASK_COMPLETED\n ProcessEvent --> HandleFailure: Event is TASK_FAILED\n UpdateConstellation --> Continue: More tasks pending\n UpdateConstellation --> Finish: All tasks done\n HandleFailure --> Continue: Retry task\n HandleFailure --> Finish: Max retries exceeded\n Finish --> [*]\n```\n\n**Agent State Machine States:**\n\n| State | Description | Trigger |\n|-------|-------------|---------|\n| **Continue** | Wait for task completion events | Events queued by Progress Observer |\n| **ProcessEvent** | Extract event from queue | Event available |\n| **UpdateConstellation** | Modify constellation based on result | Task completed successfully |\n| **HandleFailure** | Handle task failure, retry if needed | Task failed |\n| **Finish** | Complete constellation execution | All tasks done or unrecoverable error |\n\n## Performance Considerations\n\n### Memory Management\n\nThe observer stores all task results in memory:\n\n```python\nself.task_results: Dict[str, Dict[str, Any]] = {}\n```\n\n**Best Practices:**\n\n- **Clear results** after constellation completion to free memory\n- **Limit result size** by storing only essential data\n- **Use weak references** for large result objects if needed\n\n### Queue Management\n\nEvents are queued to the agent's asyncio queue:\n\n```python\nawait self.agent.add_task_completion_event(event)\n```\n\n**Considerations:**\n\n- **Queue size** is unbounded by default\n- **Back pressure** may occur if agent processes slowly\n- **Memory growth** possible with many rapid completions\n\n!!! warning \"Memory Usage\"\n For long-running sessions with many tasks, consider periodically clearing the `task_results` dictionary to prevent memory growth.\n\n## Best Practices\n\n### 1. Clean Up After Completion\n\nClear task results after constellation execution:\n\n```python\nasync def execute_with_cleanup(orchestrator, constellation, progress_observer):\n \"\"\"Execute constellation and clean up observer.\"\"\"\n \n try:\n await orchestrator.execute_constellation(constellation)\n finally:\n # Clear stored results\n progress_observer.task_results.clear()\n```\n\n### 2. Handle Errors Gracefully\n\nThe observer includes comprehensive error handling:\n\n```python\ntry:\n # Process event\n await self._handle_task_event(event)\nexcept AttributeError as e:\n self.logger.error(f\"Attribute error: {e}\", exc_info=True)\nexcept KeyError as e:\n self.logger.error(f\"Missing key: {e}\", exc_info=True)\nexcept Exception as e:\n self.logger.error(f\"Unexpected error: {e}\", exc_info=True)\n```\n\n### 3. Monitor Queue Size\n\nCheck agent queue size periodically:\n\n```python\n# Access agent's internal queue\nqueue_size = self.agent.task_completion_queue.qsize()\nif queue_size > 100:\n logger.warning(f\"Task completion queue growing large: {queue_size}\")\n```\n\n## Related Documentation\n\n- **[Observer System Overview](overview.md)** — Architecture and design principles\n- **[Agent Output Observer](agent_output_observer.md)** — Agent response and action display\n- **[Constellation Agent](../constellation_agent/overview.md)** — Agent state machine and event processing\n- **[Constellation Modification Synchronizer](synchronizer.md)** — Coordination between agent and orchestrator\n\n## Summary\n\nThe Constellation Progress Observer:\n\n- **Tracks** task execution progress\n- **Stores** task results for historical reference\n- **Queues** completion events for agent processing\n- **Coordinates** between orchestrator and agent\n- **Enables** event-driven constellation modification\n\nThis observer is essential for the agent-orchestrator coordination pattern in Galaxy, replacing complex callback mechanisms with a clean event-driven interface.\n"
},
{
"path": "documents/docs/galaxy/observer/synchronizer.md",
"content": "# Constellation Modification Synchronizer\n\nThe **ConstellationModificationSynchronizer** prevents race conditions between constellation modifications by the agent and task execution by the orchestrator. It ensures proper synchronization so the orchestrator doesn't execute newly ready tasks before the agent finishes updating the constellation structure.\n\n**Location:** `galaxy/session/observers/constellation_sync_observer.py`\n\n## Problem Statement\n\nWithout synchronization, the following race condition can occur:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant T as Task A\n participant A as Agent\n participant C as Constellation\n \n T->>O: Task A completes\n O->>A: Publish TASK_COMPLETED\n O->>C: Get ready tasks\n Note over O: Task B appears ready!\n O->>T: Execute Task B\n \n Note over A: Slow: Processing Task A completion...\n A->>C: Modify Task B (changes dependencies!)\n \n Note over T: ERROR: Task B executing with outdated state!\n```\n\n**The Race Condition:**\n\n- **Task A completes** → triggers constellation update\n- **Orchestrator immediately** gets ready tasks → might execute Task B\n- **Agent is still** modifying Task B or its dependencies\n- **Result**: Task B executes with outdated/incorrect configuration\n\n!!! danger \"Critical Issue\"\n Executing tasks with outdated constellation state can lead to incorrect task parameters, wrong dependency chains, data inconsistencies, and unpredictable workflow behavior.\n\n## Solution: Synchronization Pattern\n\nThe Synchronizer implements a **wait-before-execute** pattern:\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant T as Task A\n participant S as Synchronizer\n participant A as Agent\n participant C as Constellation\n \n T->>O: Task A completes\n O->>S: Publish TASK_COMPLETED\n S->>S: Register pending modification\n O->>A: Forward to Agent\n \n Note over O: Before getting ready tasks\n O->>S: wait_for_pending_modifications()\n Note over S: Block until agent done\n \n A->>C: Modify constellation\n A->>S: Publish CONSTELLATION_MODIFIED\n S->>S: Mark modification complete\n \n Note over S: Unblock orchestrator\n O->>C: Get ready tasks\n Note over C: Now safe to execute!\n O->>T: Execute Task B\n```\n\n## Architecture\n\n```mermaid\ngraph TB\n subgraph \"Orchestrator Loop\"\n OL[Execute Task Loop]\n WF[Wait for Modifications]\n GT[Get Ready Tasks]\n ET[Execute Tasks]\n end\n \n subgraph \"Synchronizer\"\n PM[Pending Modifications Dict]\n TC[Task Completion Handler]\n MC[Modification Complete Handler]\n WP[Wait Point]\n end\n \n subgraph \"Agent\"\n A[Agent Process Results]\n M[Modify Constellation]\n end\n \n OL --> WF\n WF --> WP\n WP -->|all modifications complete| GT\n GT --> ET\n ET --> OL\n \n TC --> PM\n MC --> PM\n PM --> WP\n \n A --> M\n M -->|CONSTELLATION_MODIFIED| MC\n \n style WP fill:#ffa726,stroke:#333,stroke-width:3px\n style PM fill:#fff4e1,stroke:#333,stroke-width:2px\n style WF fill:#4a90e2,stroke:#333,stroke-width:2px,color:#fff\n```\n\n## Synchronization Flow\n\n### Step-by-Step Process\n\n1. **Task Completes** → `TASK_COMPLETED` event published\n2. **Synchronizer Registers** → Creates pending modification Future\n3. **Orchestrator Waits** → Calls `wait_for_pending_modifications()`\n4. **Agent Processes** → Modifies constellation structure\n5. **Agent Publishes** → `CONSTELLATION_MODIFIED` event published\n6. **Synchronizer Completes** → Sets Future result, unblocks orchestrator\n7. **Orchestrator Continues** → Gets ready tasks with updated constellation\n\n### Event Flow\n\n```mermaid\nstateDiagram-v2\n [*] --> WaitingForCompletion: Task executing\n WaitingForCompletion --> PendingModification: TASK_COMPLETED event\n PendingModification --> AgentProcessing: Registered in synchronizer\n AgentProcessing --> ModificationComplete: CONSTELLATION_MODIFIED event\n ModificationComplete --> Ready: Future completed\n Ready --> WaitingForCompletion: Next task\n \n note right of PendingModification\n Orchestrator blocks here\n until modification completes\n end note\n```\n\n## Implementation\n\n### Initialization\n\n```python\nfrom galaxy.session.observers import ConstellationModificationSynchronizer\nfrom galaxy.constellation import TaskConstellationOrchestrator\n\n# Create synchronizer with orchestrator reference\nsynchronizer = ConstellationModificationSynchronizer(\n orchestrator=orchestrator,\n logger=logger\n)\n\n# Subscribe to event bus\nfrom galaxy.core.events import get_event_bus\nevent_bus = get_event_bus()\nevent_bus.subscribe(synchronizer)\n\n# Attach to orchestrator (for easy access)\norchestrator.set_modification_synchronizer(synchronizer)\n```\n\n### Constructor Parameters\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `orchestrator` | `TaskConstellationOrchestrator` | Orchestrator to synchronize with |\n| `logger` | `logging.Logger` | Optional logger instance |\n\n### Internal State\n\nThe synchronizer maintains:\n\n```python\nclass ConstellationModificationSynchronizer(IEventObserver):\n def __init__(self, orchestrator, logger=None):\n self.orchestrator = orchestrator\n \n # Pending modifications: task_id -> asyncio.Future\n self._pending_modifications: Dict[str, asyncio.Future] = {}\n \n # Current constellation being modified\n self._current_constellation_id: Optional[str] = None\n self._current_constellation: Optional[TaskConstellation] = None\n \n # Timeout for modifications (safety measure)\n self._modification_timeout = 600.0 # 10 minutes\n \n # Statistics\n self._stats = {\n \"total_modifications\": 0,\n \"completed_modifications\": 0,\n \"timeout_modifications\": 0,\n }\n```\n\n## API Reference\n\n### Main Wait Point\n\n#### wait_for_pending_modifications()\n\nWait for all pending modifications to complete before proceeding.\n\n```python\nasync def wait_for_pending_modifications(\n self, \n timeout: Optional[float] = None\n) -> bool\n```\n\n**Parameters:**\n\n- `timeout` — Optional timeout in seconds (uses default 600s if None)\n\n**Returns:**\n\n- `True` if all modifications completed successfully\n- `False` if timeout occurred\n\n**Usage in Orchestrator:**\n\n```python\nasync def execute_constellation(self, constellation):\n \"\"\"Execute constellation with synchronized modifications.\"\"\"\n \n while True:\n # Wait for any pending modifications\n await self.synchronizer.wait_for_pending_modifications()\n \n # Now safe to get ready tasks\n ready_tasks = constellation.get_ready_tasks()\n \n if not ready_tasks:\n break # All tasks complete\n \n # Execute ready tasks\n await self._execute_tasks(ready_tasks)\n```\n\n### State Management Methods\n\n#### get_current_constellation()\n\nGet the most recent constellation state after modifications.\n\n```python\ndef get_current_constellation(self) -> Optional[TaskConstellation]\n```\n\n**Returns:** Latest constellation instance or None\n\n#### has_pending_modifications()\n\nCheck if any modifications are pending.\n\n```python\ndef has_pending_modifications(self) -> bool\n```\n\n**Returns:** `True` if modifications pending, `False` otherwise\n\n#### get_pending_count()\n\nGet number of pending modifications.\n\n```python\ndef get_pending_count(self) -> int\n```\n\n**Returns:** Count of pending modifications\n\n### Constellation State Merging\n\n#### merge_and_sync_constellation_states()\n\nMerge constellation states to preserve both structural changes and execution state.\n\n```python\ndef merge_and_sync_constellation_states(\n self,\n orchestrator_constellation: TaskConstellation\n) -> TaskConstellation\n```\n\n**Purpose:** Prevents loss of execution state when agent modifies constellation structure.\n\n**Merge Strategy:**\n\n1. **Use agent's constellation as base** (has structural modifications)\n2. **Preserve orchestrator's execution state** for existing tasks\n3. **Priority rule**: More advanced state wins (COMPLETED > RUNNING > PENDING)\n4. **Update constellation state** after merging\n\n**Example Scenario:**\n\n```\nBefore Merge:\n- Orchestrator's Task A: COMPLETED (execution state)\n- Agent's Task A: RUNNING (structural changes applied)\n\nAfter Merge:\n- Task A: COMPLETED (preserved from orchestrator)\n + structural changes from agent\n```\n\n## Usage Examples\n\n### Example 1: Basic Integration\n\n```python\nfrom galaxy.core.events import get_event_bus\nfrom galaxy.session.observers import ConstellationModificationSynchronizer\n\nasync def setup_synchronized_execution():\n \"\"\"Set up synchronized constellation execution.\"\"\"\n \n # Create orchestrator\n orchestrator = TaskConstellationOrchestrator()\n \n # Create and attach synchronizer\n synchronizer = ConstellationModificationSynchronizer(\n orchestrator=orchestrator,\n logger=logger\n )\n \n # Subscribe to events\n event_bus = get_event_bus()\n event_bus.subscribe(synchronizer)\n \n # Attach to orchestrator\n orchestrator.set_modification_synchronizer(synchronizer)\n \n # Execute constellation (now synchronized)\n await orchestrator.execute_constellation(constellation)\n```\n\n### Example 2: Monitor Synchronization\n\n```python\nasync def monitor_synchronization(synchronizer):\n \"\"\"Monitor synchronization status during execution.\"\"\"\n \n while True:\n await asyncio.sleep(1)\n \n if synchronizer.has_pending_modifications():\n count = synchronizer.get_pending_count()\n pending = synchronizer.get_pending_task_ids()\n print(f\"⏳ Waiting for {count} modifications: {pending}\")\n else:\n print(\"✅ No pending modifications\")\n \n # Check statistics\n stats = synchronizer.get_statistics()\n print(f\"Stats: {stats['completed_modifications']} completed, \"\n f\"{stats['timeout_modifications']} timeouts\")\n```\n\n### Example 3: Custom Timeout Handling\n\n```python\n# Set custom timeout (default is 600 seconds)\nsynchronizer.set_modification_timeout(300.0) # 5 minutes\n\n# Wait with custom timeout\nsuccess = await synchronizer.wait_for_pending_modifications(timeout=120.0)\n\nif not success:\n print(\"⚠️ Modifications timed out, proceeding anyway\")\n # Handle timeout scenario\n synchronizer.clear_pending_modifications() # Emergency cleanup\n```\n\n## Advanced Features\n\n### Automatic Timeout Handling\n\nThe synchronizer automatically times out stuck modifications:\n\n```python\nasync def _auto_complete_on_timeout(\n self, \n task_id: str, \n future: asyncio.Future\n) -> None:\n \"\"\"Auto-complete a pending modification if it times out.\"\"\"\n \n await asyncio.sleep(self._modification_timeout)\n \n if not future.done():\n self._stats[\"timeout_modifications\"] += 1\n self.logger.warning(\n f\"⚠️ Modification for task '{task_id}' timed out after \"\n f\"{self._modification_timeout}s. Auto-completing to prevent deadlock.\"\n )\n future.set_result(False)\n del self._pending_modifications[task_id]\n```\n\n**Timeout Benefits:**\n\n- Prevents deadlocks if agent fails\n- Allows execution to continue\n- Logs timeout for debugging\n- Tracks timeout statistics\n\n### Dynamic Modification Tracking\n\nHandles new modifications registered during wait:\n\n```python\nasync def wait_for_pending_modifications(self, timeout=None) -> bool:\n \"\"\"Wait for all pending modifications, including those added during wait.\"\"\"\n \n while self._pending_modifications:\n # Get snapshot of current pending modifications\n pending_tasks = list(self._pending_modifications.keys())\n pending_futures = list(self._pending_modifications.values())\n \n # Wait for current batch\n await asyncio.wait_for(\n asyncio.gather(*pending_futures, return_exceptions=True),\n timeout=remaining_timeout\n )\n \n # Check if new modifications were added during wait\n # If yes, loop again; if no, we're done\n if not self._pending_modifications:\n break\n \n return True\n```\n\n## Statistics and Monitoring\n\n### Available Statistics\n\n```python\nstats = synchronizer.get_statistics()\n\n{\n \"total_modifications\": 10, # Total registered\n \"completed_modifications\": 9, # Successfully completed\n \"timeout_modifications\": 1 # Timed out\n}\n```\n\n### Monitoring Points\n\n| Metric | Method | Description |\n|--------|--------|-------------|\n| Pending count | `get_pending_count()` | Number of pending modifications |\n| Pending tasks | `get_pending_task_ids()` | List of task IDs with pending modifications |\n| Has pending | `has_pending_modifications()` | Boolean check |\n| Statistics | `get_statistics()` | Complete stats dictionary |\n\n## Performance Considerations\n\n### Memory Usage\n\nThe synchronizer stores futures for each pending modification:\n\n```python\nself._pending_modifications: Dict[str, asyncio.Future] = {}\n```\n\n**Memory Impact:**\n\n- **Low overhead**: Only stores Future objects (small)\n- **Temporary**: Cleared after completion\n- **Bounded**: Limited by concurrent task completions\n\n### Timeout Configuration\n\nChoose appropriate timeout based on constellation complexity:\n\n```python\n# Simple constellations\nsynchronizer.set_modification_timeout(60.0) # 1 minute\n\n# Complex constellations with slow LLM\nsynchronizer.set_modification_timeout(600.0) # 10 minutes\n\n# Very complex multi-device scenarios\nsynchronizer.set_modification_timeout(1800.0) # 30 minutes\n```\n\n## Best Practices\n\n### 1. Always Attach to Orchestrator\n\nThe orchestrator needs to call `wait_for_pending_modifications()`:\n\n```python\n# ✅ Good: Orchestrator can access synchronizer\norchestrator.set_modification_synchronizer(synchronizer)\n\n# ❌ Bad: No way for orchestrator to wait\n# synchronizer exists but orchestrator doesn't use it\n```\n\n### 2. Handle Timeouts Gracefully\n\n```python\nsuccess = await synchronizer.wait_for_pending_modifications()\n\nif not success:\n # Log timeout\n logger.warning(\"Modifications timed out\")\n \n # Get current state anyway (may be partially updated)\n constellation = synchronizer.get_current_constellation()\n \n # Continue execution (with caution)\n```\n\n### 3. Monitor Statistics\n\nTrack synchronization health:\n\n```python\nstats = synchronizer.get_statistics()\n\ntimeout_rate = (\n stats[\"timeout_modifications\"] / stats[\"total_modifications\"]\n if stats[\"total_modifications\"] > 0\n else 0\n)\n\nif timeout_rate > 0.1: # More than 10% timing out\n logger.warning(f\"High timeout rate: {timeout_rate:.1%}\")\n # Consider increasing timeout or investigating agent performance\n```\n\n## Related Documentation\n\n- **[Observer System Overview](overview.md)** — Architecture and design\n- **[Constellation Progress Observer](progress_observer.md)** — Task completion events\n- **[Constellation Agent](../constellation_agent/overview.md)** — Agent modification process\n\n## Summary\n\nThe Constellation Modification Synchronizer:\n\n- **Prevents** race conditions between agent and orchestrator\n- **Synchronizes** constellation modifications with task execution\n- **Blocks** orchestrator until modifications complete\n- **Handles** timeouts to prevent deadlocks\n- **Merges** constellation states to preserve execution data\n\nThis observer is critical for ensuring correct constellation execution when the agent dynamically modifies workflow structure during execution.\n"
},
{
"path": "documents/docs/galaxy/observer/visualization_observer.md",
"content": "# DAG Visualization Observer\n\nThe **DAGVisualizationObserver** provides real-time visual feedback during constellation execution. It displays DAG topology, task progress, and constellation modifications using rich terminal graphics.\n\n**Location:** `galaxy/session/observers/dag_visualization_observer.py`\n\n## Purpose\n\nThe Visualization Observer enables developers and users to:\n\n- **See DAG Structure** — View constellation topology and task dependencies\n- **Monitor Progress** — Track task execution in real-time\n- **Observe Modifications** — Visualize how the constellation changes\n- **Debug Issues** — Identify bottlenecks and failed tasks visually\n\n## Architecture\n\nThe observer uses a **delegation pattern** with specialized handlers:\n\n```mermaid\ngraph TB\n subgraph \"Main Observer\"\n DVO[DAGVisualizationObserver]\n CE[Constellation Events]\n TE[Task Events]\n end\n \n subgraph \"Specialized Handlers\"\n CVH[ConstellationVisualizationHandler]\n TVH[TaskVisualizationHandler]\n end\n \n subgraph \"Display Components\"\n CD[ConstellationDisplay]\n TD[TaskDisplay]\n DV[DAGVisualizer]\n end\n \n DVO --> CE\n DVO --> TE\n CE --> CVH\n TE --> TVH\n \n CVH --> CD\n CVH --> DV\n TVH --> TD\n TVH --> DV\n \n style DVO fill:#66bb6a,stroke:#333,stroke-width:3px\n style CVH fill:#ffa726,stroke:#333,stroke-width:2px\n style TVH fill:#ffa726,stroke:#333,stroke-width:2px\n```\n\n**Component Responsibilities:**\n\n| Component | Role | Handled Events |\n|-----------|------|----------------|\n| **DAGVisualizationObserver** | Main coordinator, routes events | All constellation and task events |\n| **ConstellationVisualizationHandler** | Handles constellation-level displays | CONSTELLATION_STARTED, COMPLETED, MODIFIED |\n| **TaskVisualizationHandler** | Handles task-level displays | TASK_STARTED, COMPLETED, FAILED |\n| **DAGVisualizer** | Renders complex DAG visualizations | Used by handlers for topology |\n| **ConstellationDisplay** | Renders constellation information | Used by handler for constellation events |\n| **TaskDisplay** | Renders task information | Used by handler for task events |\n\n## Implementation\n\n### Initialization\n\n```python\nfrom galaxy.session.observers import DAGVisualizationObserver\nfrom rich.console import Console\n\n# Create visualization observer\nviz_observer = DAGVisualizationObserver(\n enable_visualization=True,\n console=Console() # Optional: provide custom console\n)\n\n# Subscribe to event bus\nfrom galaxy.core.events import get_event_bus\nevent_bus = get_event_bus()\nevent_bus.subscribe(viz_observer)\n```\n\n**Constructor Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `enable_visualization` | `bool` | `True` | Whether to enable visualization |\n| `console` | `rich.Console` | `None` | Optional rich console for output |\n\n### Disabling Visualization\n\nVisualization can be toggled at runtime:\n\n```python\n# Disable visualization temporarily\nviz_observer.set_visualization_enabled(False)\n\n# Re-enable\nviz_observer.set_visualization_enabled(True)\n```\n\n## Visualization Types\n\nThe observer produces several types of visualizations:\n\n### 1. Constellation Started\n\nDisplays when a constellation begins execution:\n\n```\n╭──────────────────────────────────────────────────────────────╮\n│ 🌟 Constellation Started: email_batch_constellation │\n├──────────────────────────────────────────────────────────────┤\n│ ID: const_abc123 │\n│ Total Tasks: 8 │\n│ Status: ACTIVE │\n│ Parallel Capacity: 3 │\n╰──────────────────────────────────────────────────────────────╯\n```\n\nFollowed by DAG topology:\n\n```mermaid\ngraph TD\n fetch_emails[Fetch Emails]\n parse_1[Parse Email 1]\n parse_2[Parse Email 2]\n parse_3[Parse Email 3]\n reply_1[Reply Email 1]\n reply_2[Reply Email 2]\n reply_3[Reply Email 3]\n summarize[Summarize Results]\n \n fetch_emails --> parse_1\n fetch_emails --> parse_2\n fetch_emails --> parse_3\n parse_1 --> reply_1\n parse_2 --> reply_2\n parse_3 --> reply_3\n reply_1 --> summarize\n reply_2 --> summarize\n reply_3 --> summarize\n```\n\n### 2. Task Progress\n\nDisplays task execution events:\n\n**Task Started:**\n```\n▶ Task Started: parse_email_1\n └─ Type: parse_email\n └─ Device: windows_pc_001\n └─ Priority: MEDIUM\n```\n\n**Task Completed:**\n```\n✅ Task Completed: parse_email_1\n Duration: 2.3s\n Result: Parsed 1 email with 2 attachments\n Newly Ready: [reply_email_1]\n```\n\n**Task Failed:**\n```\n❌ Task Failed: parse_email_2\n Duration: 1.8s\n Error: NetworkTimeout: Failed to connect to email server\n Retry: 1/3\n Newly Ready: []\n```\n\n### 3. Constellation Modified\n\nShows structural changes to the constellation:\n\n```\n🔄 Constellation Modified: email_batch_constellation\n Modification Type: add_tasks\n On Task: parse_email_1\n \n Changes:\n ├─ Tasks Added: 2\n │ └─ extract_attachment_1\n │ └─ extract_attachment_2\n ├─ Dependencies Added: 2\n │ └─ parse_email_1 → extract_attachment_1\n │ └─ parse_email_1 → extract_attachment_2\n └─ Tasks Modified: 1\n └─ reply_email_1 (dependencies updated)\n```\n\nFollowed by updated DAG topology showing new tasks.\n\n### 4. Execution Flow\n\nShows current execution state (for smaller constellations):\n\n```\nExecution Flow:\n┏━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┓\n┃ Task ID ┃ Status ┃ Device ┃ Duration ┃\n┡━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━┩\n│ fetch_emails │ COMPLETED │ win_001 │ 1.2s │\n│ parse_email_1 │ RUNNING │ win_001 │ 0.8s... │\n│ parse_email_2 │ RUNNING │ mac_002 │ 0.5s... │\n│ parse_email_3 │ PENDING │ - │ - │\n│ reply_email_1 │ PENDING │ - │ - │\n└─────────────────┴───────────┴─────────┴──────────┘\n```\n\n## Event Handling Flow\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant EB as EventBus\n participant DVO as DAGVisualizationObserver\n participant CVH as ConstellationHandler\n participant TVH as TaskHandler\n participant D as Display Components\n \n O->>EB: CONSTELLATION_STARTED\n EB->>DVO: on_event(event)\n DVO->>CVH: handle_constellation_event()\n CVH->>D: Display constellation start\n CVH->>D: Display DAG topology\n \n O->>EB: TASK_STARTED\n EB->>DVO: on_event(event)\n DVO->>TVH: handle_task_event()\n TVH->>D: Display task start\n \n O->>EB: TASK_COMPLETED\n EB->>DVO: on_event(event)\n DVO->>TVH: handle_task_event()\n TVH->>D: Display task completion\n TVH->>D: Display execution flow\n \n Note over O: Agent modifies constellation\n \n O->>EB: CONSTELLATION_MODIFIED\n EB->>DVO: on_event(event)\n DVO->>CVH: handle_constellation_event()\n CVH->>D: Display modifications\n CVH->>D: Display updated topology\n```\n\n## API Reference\n\n### Main Observer Methods\n\n#### Constructor\n\n```python\ndef __init__(\n self, \n enable_visualization: bool = True, \n console=None\n)\n```\n\n**Parameters:**\n\n- `enable_visualization` — Enable/disable visualization output\n- `console` — Optional `rich.Console` for output control\n\n#### set_visualization_enabled()\n\nToggle visualization at runtime:\n\n```python\ndef set_visualization_enabled(self, enabled: bool) -> None\n```\n\n**Example:**\n\n```python\n# Disable during bulk operations\nviz_observer.set_visualization_enabled(False)\nawait orchestrator.execute_constellation(constellation)\n\n# Re-enable for interactive use\nviz_observer.set_visualization_enabled(True)\n```\n\n### Constellation Management\n\n#### register_constellation()\n\nManually register a constellation for visualization:\n\n```python\ndef register_constellation(\n self,\n constellation_id: str, \n constellation: TaskConstellation\n) -> None\n```\n\n**Use Case:** Pre-register constellations before execution starts.\n\n#### get_constellation()\n\nRetrieve stored constellation reference:\n\n```python\ndef get_constellation(self, constellation_id: str) -> Optional[TaskConstellation]\n```\n\n#### clear_constellations()\n\nClear all stored constellation references:\n\n```python\ndef clear_constellations(self) -> None\n```\n\n## Customization\n\n### Custom Console\n\nProvide custom Rich console for output control:\n\n```python\nfrom rich.console import Console\n\n# Console with custom width and theme\ncustom_console = Console(\n width=120,\n theme=my_custom_theme,\n record=True # Enable recording for export\n)\n\nviz_observer = DAGVisualizationObserver(\n enable_visualization=True,\n console=custom_console\n)\n```\n\n### Selective Visualization\n\nVisualize only specific event types:\n\n```python\nfrom galaxy.core.events import EventType\n\n# Subscribe to specific events only\nevent_bus.subscribe(viz_observer, {\n EventType.CONSTELLATION_STARTED,\n EventType.CONSTELLATION_MODIFIED,\n EventType.TASK_FAILED # Only show failures\n})\n```\n\n## Usage Examples\n\n### Example 1: Basic Visualization\n\n```python\nfrom galaxy.session.observers import DAGVisualizationObserver\nfrom galaxy.core.events import get_event_bus\n\nasync def visualize_execution():\n \"\"\"Execute constellation with visualization.\"\"\"\n \n # Create and subscribe visualization observer\n viz_observer = DAGVisualizationObserver(enable_visualization=True)\n event_bus = get_event_bus()\n event_bus.subscribe(viz_observer)\n \n # Execute constellation (visualization happens automatically)\n await orchestrator.execute_constellation(constellation)\n \n # Clean up\n event_bus.unsubscribe(viz_observer)\n```\n\n### Example 2: Conditional Visualization\n\n```python\nasync def execute_with_conditional_viz(constellation, verbose: bool = False):\n \"\"\"Execute with visualization only if verbose mode enabled.\"\"\"\n \n viz_observer = DAGVisualizationObserver(enable_visualization=verbose)\n event_bus = get_event_bus()\n \n if verbose:\n event_bus.subscribe(viz_observer)\n \n try:\n await orchestrator.execute_constellation(constellation)\n finally:\n if verbose:\n event_bus.unsubscribe(viz_observer)\n```\n\n### Example 3: Export Visualization\n\n```python\nfrom rich.console import Console\n\nasync def execute_and_export_visualization():\n \"\"\"Execute constellation and export visualization to HTML.\"\"\"\n \n # Create console with recording enabled\n console = Console(record=True, width=120)\n viz_observer = DAGVisualizationObserver(\n enable_visualization=True,\n console=console\n )\n \n event_bus = get_event_bus()\n event_bus.subscribe(viz_observer)\n \n try:\n await orchestrator.execute_constellation(constellation)\n finally:\n event_bus.unsubscribe(viz_observer)\n \n # Export recorded output to HTML\n console.save_html(\"execution_visualization.html\")\n print(\"Visualization saved to execution_visualization.html\")\n```\n\n### Example 4: Multiple Constellations\n\n```python\nasync def visualize_multiple_constellations():\n \"\"\"Visualize multiple constellation executions.\"\"\"\n \n viz_observer = DAGVisualizationObserver(enable_visualization=True)\n event_bus = get_event_bus()\n event_bus.subscribe(viz_observer)\n \n try:\n for constellation in constellations:\n print(f\"\\n{'='*60}\")\n print(f\"Executing: {constellation.name}\")\n print(f\"{'='*60}\\n\")\n \n await orchestrator.execute_constellation(constellation)\n \n # Clear constellation references between executions\n viz_observer.clear_constellations()\n finally:\n event_bus.unsubscribe(viz_observer)\n```\n\n## Performance Considerations\n\n### Visualization Overhead\n\nVisualization adds minimal overhead:\n\n- **Small DAGs** (< 10 tasks): Negligible impact\n- **Medium DAGs** (10-50 tasks): < 1% overhead\n- **Large DAGs** (> 50 tasks): Topology rendering may be slow\n\n### Optimization Strategies\n\n```python\n# Strategy 1: Disable for large constellations\nif constellation.task_count > 50:\n viz_observer.set_visualization_enabled(False)\n\n# Strategy 2: Subscribe to fewer events\nevent_bus.subscribe(viz_observer, {\n EventType.CONSTELLATION_STARTED,\n EventType.CONSTELLATION_COMPLETED,\n EventType.TASK_FAILED # Only show problems\n})\n\n# Strategy 3: Conditional topology display\n# (Handler automatically skips topology for constellations > 10 tasks)\n```\n\n## Best Practices\n\n### 1. Enable for Interactive Sessions\n\n```python\n# ✅ Good: Interactive development/debugging\nif __name__ == \"__main__\":\n viz_observer = DAGVisualizationObserver(enable_visualization=True)\n # ...\n\n# ✅ Good: Batch processing\nif running_in_batch_mode:\n viz_observer = DAGVisualizationObserver(enable_visualization=False)\n```\n\n### 2. Clean Up Constellation References\n\n```python\n# After processing many constellations\nfor constellation in constellation_list:\n await orchestrator.execute_constellation(constellation)\n viz_observer.clear_constellations() # Free memory\n```\n\n### 3. Export for Documentation\n\n```python\n# Record visualization for documentation/reports\nconsole = Console(record=True)\nviz_observer = DAGVisualizationObserver(console=console)\n\n# ... execute constellation ...\n\n# Export\nconsole.save_html(\"docs/execution_example.html\")\nconsole.save_text(\"logs/execution.txt\")\n```\n\n## Related Documentation\n\n- **[Observer System Overview](overview.md)** — Architecture and design\n- **[Progress Observer](progress_observer.md)** — Task completion tracking\n\n## Summary\n\nThe DAG Visualization Observer:\n\n- **Displays** constellation structure and execution progress\n- **Delegates** to specialized handlers for clean separation\n- **Uses** Rich terminal graphics for beautiful output\n- **Supports** conditional enabling/disabling\n- **Exports** visualization for documentation\n\nThis observer is essential for understanding and debugging constellation execution, providing intuitive visual feedback for complex DAG workflows.\n"
},
{
"path": "documents/docs/galaxy/overview.md",
"content": "# UFO³ — Weaving the Digital Agent Galaxy\n\n
\n \n
From isolated device agents to interconnected constellations — Building the Digital Agent Galaxy
\n
\n\n---\n\n## 🚀 What is UFO³ Galaxy?\n\n**UFO³ Galaxy** is a revolutionary **cross-device orchestration framework** that transforms isolated device agents into a unified digital ecosystem. It models complex user requests as **Task Constellations** (星座) — dynamic distributed DAGs where nodes represent executable subtasks and edges capture dependencies across heterogeneous devices.\n\n### 🎯 The Vision\n\nBuilding truly ubiquitous intelligent agents requires moving beyond single-device automation. UFO³ Galaxy addresses four fundamental challenges in cross-device agent orchestration:\n\n**🔄 Asynchronous Parallelism** \nEnabling concurrent task execution across multiple devices while maintaining correctness through event-driven coordination and safe concurrency control\n\n**⚡ Dynamic Adaptation** \nReal-time workflow evolution in response to intermediate results, transient failures, and runtime observations without workflow abortion\n\n**🌐 Distributed Coordination** \nReliable, low-latency communication across heterogeneous devices via WebSocket-based Agent Interaction Protocol with fault tolerance\n\n**🛡️ Safety Guarantees** \nFormal invariants ensuring DAG consistency during concurrent modifications and parallel execution, verified through rigorous proofs\n\n---\n\n## 🏗️ Architecture\n\n
\n \n
UFO³ Galaxy Layered Architecture — From natural language to distributed execution
\n
\n\n\n### Layered Design\n\nUFO³ Galaxy follows a **hierarchical orchestration model** that separates global coordination from local execution. This architecture enables scalable cross-device orchestration while maintaining consistent control and responsiveness across diverse operating systems and network environments.\n\n#### 🎛️ Hierarchical Control Plane\n\n**ConstellationClient** serves as the **global control plane**, maintaining a live registry of all connected device agents with their:\n- Capability profiles and system specifications\n- Runtime health metrics and availability status\n- Current load and resource utilization\n\nThis registry enables intelligent task placement based on device capabilities, avoiding mismatches between task requirements and device capacity.\n\nEach device hosts a **device agent server** that manages local orchestration through persistent WebSocket sessions with ConstellationClient. The server:\n- Maintains execution contexts on the host\n- Provides unified interface to underlying tools via MCP servers\n- Handles task execution, telemetry streaming, and resource monitoring\n\n**Clean separation**: Global orchestration policies are decoupled from device-specific heterogeneity, providing consistent abstraction across endpoints with different OS, hardware, or network conditions.\n\n#### 🔄 Orchestration Flow\n\n1. **DAG Synthesis**: ConstellationClient invokes the **Constellation Agent** to construct a TaskConstellation—a dynamic DAG encoding task decomposition, dependencies, and device mappings\n2. **Device Assignment**: Each TaskStar (DAG node) is assigned to suitable device agents based on capability profiles and system load\n3. **Asynchronous Execution**: The **Constellation Orchestrator** executes the DAG in an event-driven manner:\n - Task completions trigger dependent nodes\n - Failures prompt retry, migration, or partial DAG rewrites\n - Workflows adapt to real-time system dynamics (device churn, network variability)\n\n**Result**: Highly parallel and resilient execution that sustains workflow completion even as subsets of devices fail or reconnect.\n\n#### 🔌 Cross-Agent Communication\n\nThe **Agent Interaction Protocol (AIP)** handles all cross-agent interactions:\n- Agent registration and capability synchronization\n- Task dispatch and progress reporting \n- Result aggregation and telemetry streaming\n\nBuilt on persistent WebSocket channels, AIP provides:\n- **Lightweight**: Minimal overhead for control messages\n- **Bidirectional**: Full-duplex communication between client and agents\n- **Multiplexed**: Concurrent message streams over single connection\n- **Low-latency**: Fast propagation of control signals and state updates\n- **Resilient**: Maintains global consistency despite intermittent connectivity\n\nTogether, these design elements form a cohesive foundation for orchestrating large-scale, heterogeneous, and adaptive workflows across a resilient multi-device execution fabric.\n\n---\n\n## ✨ Core Design Principles\n\nUFO³ Galaxy realizes cross-device orchestration through **five tightly integrated design principles**:\n\n### 1. 🌟 Declarative Decomposition into Dynamic DAG (Task Constellation)\n\nNatural-language or programmatic requests are decomposed by the **Constellation Agent** into a structured DAG of **TaskStars** (nodes) and **TaskStarLines** (edges) that encode workflow logic, dependencies, and device assignments. This declarative structure is amenable to automated scheduling, introspection, and dynamic modification throughout execution.\n\n**Key Benefits:**\n- 📋 **Declarative structure** for automated scheduling\n- 🔍 **Runtime introspection** for workflow visibility\n- ✏️ **Dynamic rewriting** throughout execution\n- 🔄 **Automated orchestration** across heterogeneous devices\n\n```mermaid\ngraph LR\n A[User Intent] --> B[Constellation Agent]\n B --> C[Task Constellation DAG]\n C --> D[TaskStar 1 Windows]\n C --> E[TaskStar 2 Linux GPU]\n C --> F[TaskStar 3 Linux CPU]\n C --> G[TaskStar 4 Mobile]\n E --> H[TaskStar 5]\n F --> H\n G --> H\n```\n\n[Learn more →](constellation/overview.md)\n\n### 2. 🔄 Continuous, Result-Driven Graph Evolution\n\nThe **Task Constellation** is a **living data structure** that evolves in response to execution feedback. Intermediate outputs, transient failures, and new observations trigger controlled rewrites—adding diagnostic TaskStars, creating fallbacks, rewiring dependencies, or pruning completed nodes—so the system adapts dynamically instead of aborting on errors.\n\n**Adaptation Mechanisms:**\n- 🩺 **Diagnostic TaskStars** added for debugging\n- 🛡️ **Fallback creation** for error recovery\n- 🔗 **Dependency rewiring** for workflow optimization\n- ✂️ **Node pruning** after completion\n\nThe **Constellation Agent** operates in two modes:\n- **Creation Mode**: Synthesizes initial DAG from user request with device-aware task decomposition\n- **Editing Mode**: Incrementally refines constellation based on task completion events and runtime feedback\n\n[Learn more →](constellation_agent/overview.md)\n\n### 3. 🎯 Heterogeneous, Asynchronous, and Safe Orchestration\n\nEach **Task Star** is matched to the most suitable device agent via rich **Agent Profiles** reflecting OS, hardware capabilities, and installed tools. The **Constellation Orchestrator** executes tasks asynchronously, allowing multiple TaskStars to progress in parallel.\n\n**Safety Guarantees:**\n- 🔒 **Safe assignment locking** prevents race conditions\n- 📅 **Event-driven scheduling** monitors DAG readiness\n- ✅ **DAG consistency checks** maintain structural integrity\n- 🔄 **Batched edits** ensure atomicity\n- 📐 **Formal verification** reinforces correctness\n- ⏱️ **Timeout protection** prevents deadlocks\n\nThese mechanisms collectively ensure **high efficiency without compromising reliability**.\n\n[Learn more →](constellation_orchestrator/overview.md)\n\n### 4. 🔌 Unified Agent Interaction Protocol (AIP)\n\nBuilt atop persistent **WebSocket channels**, AIP provides a unified, secure, and fault-tolerant layer for the entire agent ecosystem.\n\n**Core Capabilities:**\n- 📝 **Agent registry** with capability profiles\n- 🔐 **Session management** for secure communication\n- 📤 **Task dispatch** with intelligent routing\n- 🎯 **Coordination primitives** for distributed workflows\n- 💓 **Heartbeat monitoring** for health tracking\n- 🔌 **Automatic reconnection** under network fluctuations\n- 🔄 **Retry mechanisms** for reliability\n\n**Architecture Benefits:**\n- 🪶 **Lightweight interface** for easy integration\n- 🧩 **Extensible design** supports new agent types\n- 🛡️ **Fault tolerance** ensures continuous operation\n\nThis protocol **abstracts OS and network heterogeneity**, enabling seamless collaboration among agents across desktops, servers, and edge devices, while allowing new agents to integrate seamlessly into the UFO³ ecosystem.\n\n[Learn more →](../aip/overview.md)\n\n### 5. 🛠️ Template-Driven Framework for Device Agents\n\nTo **democratize agent creation**, UFO³ provides a **lightweight development template and toolkit** for rapidly building new device agents.\n\n**Development Framework:**\n- 📄 **Capability declaration** defines agent profiles\n- 🔗 **Environment binding** connects to local systems\n- 🧩 **MCP server integration** for tool augmentation\n- 🔧 **Modular design** accelerates development\n\n**Model Context Protocol (MCP) Integration:**\n- 🎁 **Tool packages** via MCP servers\n- 🔌 **Plug-and-play** capability extension\n- 🌐 **Cross-platform** tool standardization\n- 🚀 **Rapid prototyping** of new agents\n\nThis modular architecture maintains consistency across the constellation while enabling developers to extend UFO³ to new platforms (mobile, web, IoT, embedded systems, etc.) with minimal effort.\n\n**🔌 Extensibility:** UFO³ is designed as a **universal framework** that supports developing new device agents for different platforms (mobile, web, IoT, embedded systems, etc.) and applications. Through the **Agent Interaction Protocol (AIP)**, custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device automation. **Want to build your own device agent?** See our [Creating Custom Device Agents tutorial](../tutorials/creating_device_agent/overview.md) to learn how to extend UFO³ to new platforms.\n\n[Learn more →](agent_registration/overview.md) | [MCP Integration →](../mcp/overview.md)\n\n---\n\n## 🎯 Key Capabilities\n\n### 🌐 Cross-Device Collaboration\nExecute workflows that span Windows desktops, Linux servers, GPU clusters, mobile devices, and edge nodes—all from a single natural language request.\n\n### ⚡ Asynchronous Parallelism\nAutomatically identify parallelizable subtasks and execute them concurrently across devices through:\n- **Event-driven scheduling** that continuously monitors DAG topology for ready tasks\n- **Non-blocking execution** with Python `asyncio` for maximum concurrency\n- **Dynamic adaptation** that integrates new tasks without interrupting running execution\n\nResult: Dramatically reduced end-to-end latency compared to sequential execution.\n\n### 🛡️ Safety & Consistency\n- **Three formal invariants** (I1-I3) enforced at runtime for DAG correctness\n- **Safe assignment locking** prevents race conditions during concurrent modifications\n- **Acyclicity validation** ensures no circular dependencies\n- **State merging** algorithm preserves execution progress during dynamic edits\n- **Timeout protection** prevents deadlocks from agent failures\n\n### 🔄 Dynamic Workflow Evolution\n- **Dual-mode operation**: Separate creation and editing phases with controlled transitions\n- **Feedback-driven adaptation**: Task completion events trigger intelligent constellation refinement\n- **LLM-powered reasoning**: ReAct architecture for context-aware DAG modifications\n- **Undo/redo support**: ConstellationEditor with command pattern for safe interactive editing\n\n### 👁️ Rich Observability\n- Real-time constellation visualization with DAG topology updates\n- Event bus with publish-subscribe pattern for monitoring task progress\n- Detailed execution logs with markdown trajectory support\n- Task status tracking (pending, running, completed, failed, cancelled)\n- Dependency graph inspection and validation tools\n\n---\n\n## 🎨 Use Cases\n\n### 🖥️ Software Development & Deployment\n*\"Clone the repo on my laptop, build the Docker image on the GPU server, deploy to staging, and run the test suite on the CI cluster.\"*\n\n**Workflow DAG:**\n```mermaid\ngraph LR\n A[Clone Windows] --> B[Build Linux GPU]\n B --> C[Deploy Linux Server]\n C --> D[Test Linux CI]\n```\n\n### 📊 Data Science Workflows\n*\"Fetch the dataset from cloud storage, preprocess on the Linux workstation, train the model on the A100 node, and generate a visualization dashboard on my Windows machine.\"*\n\n**Workflow DAG:**\n```mermaid\ngraph LR\n A[Fetch Any] --> B[Preprocess Linux]\n B --> C[Train Linux GPU]\n C --> D[Visualize Windows]\n```\n\n### 📝 Cross-Platform Document Processing\n*\"Extract data from Excel on Windows, process with Python scripts on Linux, generate PDF reports, and send summary emails.\"*\n\n**Workflow DAG:**\n```mermaid\ngraph LR\n A[Extract Windows] --> B[Process Linux]\n B --> C[Generate PDF Windows]\n B --> D[Send Email Windows]\n```\n\n### 🔬 Distributed System Monitoring\n*\"Collect server logs from all Linux machines, analyze for errors, generate alerts, and create a consolidated report.\"*\n\n**Workflow DAG:**\n```mermaid\ngraph LR\n A[Collect Logs Linux 1] --> D[Analyze Errors Any]\n B[Collect Logs Linux 2] --> D\n C[Collect Logs Linux 3] --> D\n D --> E[Generate Report Windows]\n```\n\n### 🏢 Enterprise Automation\n*\"Query the database on the server, process the results, update Excel spreadsheets on Windows, and generate PowerPoint presentations.\"*\n\n**Workflow DAG:**\n```mermaid\ngraph LR\n A[Query DB Linux] --> B[Process Data Any]\n B --> C[Update Excel Windows]\n B --> D[Create PPT Windows]\n```\n\n---\n\n## 🗺️ Documentation Structure\n\n### 🚀 [Quick Start](../getting_started/quick_start_galaxy.md)\nGet UFO³ Galaxy up and running in minutes with our step-by-step guide\n\n### 👥 [Galaxy Client](client/overview.md)\nDevice coordination, connection management, and ConstellationClient API\n\n### 🧠 [Constellation Agent](constellation_agent/overview.md)\nLLM-driven task decomposition, DAG creation, and dynamic workflow evolution\n\n### ⚙️ [Constellation Orchestrator](constellation_orchestrator/overview.md)\nAsynchronous execution engine, event-driven coordination, and safety guarantees\n\n### 📊 [Task Constellation](constellation/overview.md)\nDAG structure, TaskStar nodes, TaskStarLine edges, and constellation editor\n\n### 🆔 [Agent Registration](agent_registration/overview.md)\nDevice registry, agent profiles, and registration flow\n\n### 🌐 [Agent Interaction Protocol](../aip/overview.md)\nWebSocket messaging, protocol specification, and communication patterns\n\n### ⚙️ [Configuration](../configuration/system/galaxy_devices.md)\nDevice pools, capabilities, and orchestration policies\n\n---\n\n## 🚦 Getting Started\n\nReady to build your Digital Agent Galaxy? Follow these steps:\n\n### 1. Install UFO³\n```bash\n# Clone the repository\ngit clone https://github.com/microsoft/UFO.git\ncd UFO\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### 2. Configure Device Pool\n\nCreate configuration files in `config/galaxy/`:\n\n**`config/galaxy/devices.yaml`** - Define your devices:\n\n```yaml\ndevices:\n - device_id: \"windowsagent\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"web_browsing\"\n - \"office_applications\"\n - \"file_management\"\n metadata:\n location: \"home_office\"\n os: \"windows\"\n performance: \"medium\"\n max_retries: 5\n \n - device_id: \"linux_agent_1\"\n server_url: \"ws://localhost:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server\"\n - \"python\"\n - \"docker\"\n metadata:\n os: \"linux\"\n performance: \"high\"\n logs_file_path: \"/root/log/log1.txt\"\n auto_connect: true\n max_retries: 5\n \n - device_id: \"mobile_agent_1\"\n server_url: \"ws://localhost:5002/ws\"\n os: \"android\"\n capabilities:\n - \"mobile\"\n - \"adb\"\n - \"ui_automation\"\n metadata:\n os: \"android\"\n performance: \"medium\"\n device_type: \"smartphone\"\n auto_connect: true\n max_retries: 5\n```\n\n**`config/galaxy/constellation.yaml`** - Configure runtime settings:\n\n```yaml\n# Constellation Runtime Settings\nCONSTELLATION_ID: \"my_constellation\"\nHEARTBEAT_INTERVAL: 30.0 # Heartbeat interval in seconds\nRECONNECT_DELAY: 5.0 # Delay before reconnecting in seconds\nMAX_CONCURRENT_TASKS: 6 # Maximum concurrent tasks\nMAX_STEP: 15 # Maximum steps per session\n\n# Device Configuration\nDEVICE_INFO: \"config/galaxy/devices.yaml\"\n\n# Logging Configuration\nLOG_TO_MARKDOWN: true\n```\n\nSee [Galaxy Configuration](../configuration/system/galaxy_devices.md) for complete documentation.\n\n### 3. Start Device Agents\n\nOn each device, launch the Agent Server. For detailed setup instructions, see the respective quick start guides:\n\n**On Windows:**\n\nSee [Windows Agent (UFO²) Quick Start →](../getting_started/quick_start_ufo2.md)\n\n**On Linux:**\n\nSee [Linux Agent Quick Start →](../getting_started/quick_start_linux.md)\n\n**On Mobile (Android):**\n\nSee [Mobile Agent Quick Start →](../getting_started/quick_start_mobile.md)\n\n### 4. Launch Galaxy Client\n\n**Interactive Mode:**\n```bash\npython -m galaxy --interactive\n```\n\n**Direct Request:**\n```bash\npython -m galaxy \"Your cross-device task here\"\n```\n\n**Programmatic API:**\n```python\nfrom galaxy.galaxy_client import GalaxyClient\n\nasync def main():\n client = GalaxyClient(session_name=\"my_session\")\n await client.initialize()\n result = await client.process_request(\"Your task request\")\n await client.shutdown()\n```\n\nFor detailed instructions, see the [Quick Start Guide](../getting_started/quick_start_galaxy.md).\n\n---\n\n## 🔧 System Components\n\nUFO³ Galaxy consists of several integrated components working together:\n\n### Core Components\n\n| Component | Location | Responsibility |\n|-----------|----------|----------------|\n| **GalaxyClient** | `galaxy/galaxy_client.py` | Session management, user interaction, orchestration coordination |\n| **ConstellationClient** | `galaxy/client/constellation_client.py` | Device management, connection lifecycle, task assignment |\n| **ConstellationAgent** | `galaxy/agents/constellation_agent.py` | LLM-driven DAG synthesis and evolution, state machine control |\n| **TaskConstellationOrchestrator** | `galaxy/constellation/orchestrator/` | Asynchronous execution, event coordination, safety enforcement |\n| **TaskConstellation** | `galaxy/constellation/task_constellation.py` | DAG data structure, validation, and modification APIs |\n| **DeviceManager** | `galaxy/client/device_manager.py` | WebSocket connections, heartbeat monitoring, message routing |\n\n### Supporting Infrastructure\n\n| Component | Purpose |\n|-----------|---------|\n| **Event Bus** | Publish-subscribe system for constellation events |\n| **Observer Pattern** | Event listeners for visualization and synchronization |\n| **Device Registry** | Centralized device information and capability tracking |\n| **Agent Profile** | Device metadata and capability declarations |\n| **MCP Servers** | Tool augmentation via Model Context Protocol |\n\nFor detailed component documentation, see the respective sections in [Documentation Structure](#documentation-structure).\n\n### Technology Stack\n\n| Layer | Technologies |\n|-------|-------------|\n| **Programming** | Python 3.10+, asyncio, dataclasses |\n| **Communication** | WebSockets, JSON-RPC |\n| **LLM Integration** | OpenAI API, Azure OpenAI, Gemini, Claude, Custom Models |\n| **Tool Augmentation** | Model Context Protocol (MCP) |\n| **Configuration** | YAML, Pydantic models |\n| **Logging** | Python logging, Rich console, Markdown trajectory |\n| **Testing** | pytest, mock agents |\n\n---\n\n## 🌟 From Devices to Constellations to Galaxy\n\nUFO³ represents a paradigm shift in intelligent automation:\n\n- **Single Device** → Isolated agents operating within one OS\n- **Task Constellation** → Coordinated multi-device workflows for one task\n- **Digital Agent Galaxy** → Interconnected constellations spanning your entire digital estate\n\nOver time, multiple constellations can interconnect, weaving together agents, devices, and capabilities into a self-organizing **Digital Agent Galaxy**. This design elevates cross-device automation from a brittle engineering challenge to a unified orchestration paradigm, where multi-device workflows become naturally expressive, paving the way for large-scale, adaptive, and resilient intelligent ubiquitous computing systems.\n\n---\n\n## 📊 Performance Monitoring & Evaluation\n\nUFO³ Galaxy provides comprehensive performance monitoring and evaluation tools to analyze multi-device workflow execution:\n\n### Automated Metrics Collection\n\nGalaxy automatically collects detailed performance metrics during execution through an event-driven observer pattern:\n\n- **Task Metrics**: Execution times, success rates, bottleneck identification\n- **Constellation Metrics**: DAG statistics, parallelism analysis, critical path computation\n- **Modification Metrics**: Dynamic editing patterns and adaptation frequency\n- **Device Metrics**: Per-device performance and resource utilization\n\nAll metrics are captured in real-time without impacting execution performance and saved to structured JSON files for programmatic analysis.\n\n### Trajectory Report\n\nGalaxy automatically generates a comprehensive Markdown trajectory report (`output.md`) documenting the complete execution lifecycle:\n\n```\nlogs/galaxy//output.md\n```\n\nThis human-readable report includes:\n- Step-by-step execution timeline with agent actions\n- Interactive DAG topology visualizations showing constellation evolution\n- Detailed task execution logs with results and errors\n- Device connection status and coordination events\n- Complete before/after constellation states at each step\n\nThe trajectory report provides visual debugging and workflow understanding, complementing the quantitative `result.json` metrics.\n\n### Result JSON Format\n\nAfter each session, Galaxy also generates a comprehensive `result.json` file containing:\n\n```\nlogs/galaxy//result.json\n```\n\nThis file includes:\n- Complete session metadata and execution timeline\n- Task-by-task performance breakdown\n- Constellation statistics (parallelism ratio, critical path, max concurrency)\n- Modification history showing DAG evolution\n- Final results and outcomes\n\n**Example Key Metrics:**\n\n| Metric | Description | Use Case |\n|--------|-------------|----------|\n| `parallelism_ratio` | Efficiency of parallel execution (total_work / critical_path) | Optimization target |\n| `critical_path_length` | Minimum possible execution time | Theoretical performance limit |\n| `average_task_duration` | Mean task execution time | Baseline performance |\n| `modification_count` | Number of dynamic DAG edits | Adaptability analysis |\n\n### Performance Analysis Tools\n\n```python\nimport json\n\n# Load session results\nwith open(\"logs/galaxy/task_32/result.json\", 'r') as f:\n result = json.load(f)\n\n# Extract key metrics\nmetrics = result[\"session_results\"][\"metrics\"]\ntask_stats = metrics[\"task_statistics\"]\nconst_stats = result[\"session_results\"][\"final_constellation_stats\"]\n\nprint(f\"✅ Success Rate: {task_stats['success_rate'] * 100:.1f}%\")\nprint(f\"⏱️ Avg Task Duration: {task_stats['average_task_duration']:.2f}s\")\nprint(f\"🔀 Parallelism Ratio: {const_stats['parallelism_ratio']:.2f}\")\n```\n\n**Documentation:**\n\n- **[Trajectory Report Guide](./evaluation/trajectory_report.md)** - Complete guide to the human-readable execution log with DAG visualizations\n- **[Performance Metrics Guide](./evaluation/performance_metrics.md)** - Comprehensive metrics documentation with analysis examples\n- **[Result JSON Reference](./evaluation/result_json.md)** - Complete schema reference and programmatic access guide\n\n---\n\n## 📚 Learn More\n\n- **Research Paper**: [UFO³: Weaving the Digital Agent Galaxy](https://arxiv.org/) *(Coming Soon)*\n- **UFO² (Desktop AgentOS)**: [Documentation](../ufo2/overview.md)\n- **UFO (Original)**: [GitHub Repository](https://github.com/microsoft/UFO)\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! Whether you're building new device agents, improving orchestration algorithms, or enhancing the protocol, check out our Contributing Guide on GitHub.\n\n---\n\n## 📄 License\n\nUFO³ Galaxy is released under the MIT License.\n\n---\n\n
\n
Transform your distributed devices into a unified digital collective.
\n
UFO³ Galaxy — Where every device is a star, and every task is a constellation.
\n
\n"
},
{
"path": "documents/docs/galaxy/webui.md",
"content": "# Galaxy WebUI\n\nThe **Galaxy WebUI** is a modern, interactive web interface for the UFO³ Galaxy Framework. It provides real-time visualization of task constellations, device status, agent interactions, and execution flow through an elegant, space-themed interface.\n\n
\n \n
Galaxy WebUI - Interactive constellation visualization and real-time monitoring
\n
\n\n---\n\n## 🌟 Overview\n\nThe Galaxy WebUI transforms the command-line Galaxy experience into a rich, visual interface where you can:\n\n- **🗣️ Chat with Galaxy**: Submit natural language requests through an intuitive chat interface\n- **📊 Visualize Constellations**: Watch task constellations form and execute as interactive DAG graphs\n- **🎯 Monitor Execution**: Track task status, device assignments, and real-time progress\n- **🔄 See Agent Reasoning**: Observe agent thoughts, plans, and decision-making processes\n- **🖥️ Manage Devices**: View, monitor, and **add new devices** through the UI\n- **➕ Add Device Agents**: Register new device agents dynamically without restarting\n- **📡 Stream Events**: Follow the event log to understand system behavior in real-time\n\n---\n\n## 🚀 Quick Start\n\n### Starting the WebUI\n\n```powershell\n# Launch Galaxy with WebUI\npython -m galaxy --webui\n```\n\nThe WebUI will automatically:\n1. Start the backend server on `http://localhost:8000` (or next available port)\n2. Open your default browser to the interface\n3. Establish WebSocket connection for real-time updates\n\n!!!tip \"Custom Session Name\"\n ```powershell\n python -m galaxy --webui --session-name \"data_pipeline_demo\"\n ```\n\n### First Request\n\n1. **Enter your request** in the chat input at the bottom\n2. **Press Enter** or click Send\n3. **Watch the constellation form** in the DAG visualization panel\n4. **Monitor task execution** as devices process their assigned tasks\n5. **See results** displayed in the chat window\n\n---\n\n## 🏗️ Architecture\n\n### Design Principles\n\nThe Galaxy WebUI backend follows **software engineering best practices**:\n\n**Separation of Concerns:**\n- **Models Layer**: Pydantic models ensure type safety and validation\n- **Services Layer**: Business logic isolated from presentation\n- **Handlers Layer**: WebSocket message processing logic\n- **Routers Layer**: HTTP endpoint definitions\n\n**Dependency Injection:**\n- `AppState` class provides centralized state management\n- `get_app_state()` dependency injection function\n- Replaces global variables with type-safe properties\n\n**Type Safety:**\n- Pydantic models for all API requests/responses\n- Enums for constants (`WebSocketMessageType`, `RequestStatus`)\n- `TYPE_CHECKING` pattern for forward references\n- Comprehensive type annotations throughout\n\n**Modularity:**\n- Clear module boundaries\n- Easy to test individual components\n- Simple to extend with new features\n- Better code organization and maintainability\n\n### System Architecture\n\nThe Galaxy WebUI follows a modern client-server architecture with real-time event streaming:\n\n```mermaid\ngraph TB\n subgraph \"Galaxy WebUI Stack\"\n subgraph Frontend[\"Frontend (React + TypeScript + Vite)\"]\n F1[Chat Interface]\n F2[DAG Visualization ReactFlow]\n F3[Device Management]\n F4[Event Log]\n F5[State Management Zustand]\n end\n \n subgraph Backend[\"Backend (FastAPI + WebSocket)\"]\n subgraph Presentation[\"Presentation Layer\"]\n B1[FastAPI App server.py]\n B2[Routers health/devices/websocket]\n end\n \n subgraph Business[\"Business Logic Layer\"]\n B3[Services Config/Device/Galaxy]\n B4[Handlers WebSocket Message Handler]\n end\n \n subgraph Data[\"Data & Models Layer\"]\n B5[Models Requests/Responses]\n B6[Enums MessageType/Status]\n B7[Dependencies AppState]\n end\n \n subgraph Events[\"Event Processing\"]\n B8[WebSocketObserver]\n B9[EventSerializer]\n end\n end\n \n subgraph Core[\"Galaxy Core\"]\n C1[ConstellationAgent]\n C2[Task Orchestrator]\n C3[Device Manager]\n C4[Event System]\n end\n \n Frontend <-->|WebSocket| B2\n B2 --> B4\n B4 --> B3\n B3 --> B7\n B2 --> B5\n B8 --> B9\n B8 -->|Broadcast| Frontend\n C4 -->|Publish Events| B8\n B3 <-->|State Access| B7\n Backend <-->|Event Bus| Core\n end\n \n style Frontend fill:#1a1a2e,stroke:#00d4ff,stroke-width:2px,color:#fff\n style Presentation fill:#16213e,stroke:#7b2cbf,stroke-width:2px,color:#fff\n style Business fill:#1a1a2e,stroke:#00d4ff,stroke-width:2px,color:#fff\n style Data fill:#0f1419,stroke:#10b981,stroke-width:2px,color:#fff\n style Events fill:#16213e,stroke:#ff006e,stroke-width:2px,color:#fff\n style Core fill:#0a0e27,stroke:#ff006e,stroke-width:2px,color:#fff\n```\n\n### Component Overview\n\n#### Backend Components\n\nThe Galaxy WebUI backend follows a **modular architecture** with clear separation of concerns:\n\n| Component | File/Directory | Responsibility |\n|-----------|----------------|----------------|\n| **FastAPI Server** | `galaxy/webui/server.py` | Application initialization, middleware, router registration, lifespan management |\n| **Models** | `galaxy/webui/models/` | Pydantic models for requests/responses, enums for type safety |\n| **Services** | `galaxy/webui/services/` | Business logic layer (config, device, galaxy operations) |\n| **Handlers** | `galaxy/webui/handlers/` | WebSocket message processing and routing |\n| **Routers** | `galaxy/webui/routers/` | FastAPI endpoint definitions organized by feature |\n| **Dependencies** | `galaxy/webui/dependencies.py` | Dependency injection for state management (AppState) |\n| **WebSocket Observer** | `galaxy/webui/websocket_observer.py` | Event subscription and broadcasting to WebSocket clients |\n| **Event Serializer** | Built into observer | Converts Python objects to JSON-compatible format |\n\n**Detailed Backend Structure:**\n\n```\ngalaxy/webui/\n├── server.py # Main FastAPI application\n├── dependencies.py # AppState and dependency injection\n├── websocket_observer.py # EventSerializer + WebSocketObserver\n├── models/\n│ ├── __init__.py # Export all models\n│ ├── enums.py # WebSocketMessageType, RequestStatus enums\n│ ├── requests.py # Pydantic request models\n│ └── responses.py # Pydantic response models\n├── services/\n│ ├── __init__.py\n│ ├── config_service.py # Configuration management\n│ ├── device_service.py # Device operations and snapshots\n│ └── galaxy_service.py # Galaxy client interactions\n├── handlers/\n│ ├── __init__.py\n│ └── websocket_handlers.py # WebSocket message handler\n├── routers/\n│ ├── __init__.py\n│ ├── health.py # Health check endpoint\n│ ├── devices.py # Device management endpoints\n│ └── websocket.py # WebSocket endpoint\n└── templates/\n └── index.html # Fallback HTML page\n```\n\n**Architecture Benefits:**\n\n✅ **Maintainability**: Each module has a single, clear responsibility \n✅ **Testability**: Services and handlers can be unit tested independently \n✅ **Type Safety**: Pydantic models validate all inputs/outputs \n✅ **Extensibility**: Easy to add new endpoints, message types, or services \n✅ **Readability**: Clear module boundaries improve code comprehension \n✅ **Reusability**: Services can be shared across multiple endpoints \n\n#### Frontend Components\n\n| Component | Location | Purpose |\n|-----------|----------|---------|\n| **App** | `src/App.tsx` | Main layout, connection status, theme management |\n| **ChatWindow** | `src/components/chat/ChatWindow.tsx` | Message display and input interface |\n| **DagPreview** | `src/components/constellation/DagPreview.tsx` | Interactive constellation graph visualization |\n| **DevicePanel** | `src/components/devices/DevicePanel.tsx` | Device status cards, search, and add button |\n| **DeviceCard** | `src/components/devices/DeviceCard.tsx` | Individual device status display |\n| **AddDeviceModal** | `src/components/devices/AddDeviceModal.tsx` | Modal dialog for adding new devices |\n| **RightPanel** | `src/components/layout/RightPanel.tsx` | Tabbed panel for constellation, tasks, details |\n| **EventLog** | `src/components/EventLog.tsx` | Real-time event stream display |\n| **GalaxyStore** | `src/store/galaxyStore.ts` | Zustand state management |\n| **WebSocket Client** | `src/services/websocket.ts` | WebSocket connection with auto-reconnect |\n\n---\n\n## 🔌 Communication Protocol\n\n### HTTP API Endpoints\n\n#### Health Check\n\n```http\nGET /health\n```\n\n**Response:**\n```json\n{\n \"status\": \"healthy\",\n \"connections\": 3,\n \"events_sent\": 1247\n}\n```\n\n#### Add Device\n\n```http\nPOST /api/devices\nContent-Type: application/json\n```\n\n**Request Body:**\n```json\n{\n \"device_id\": \"windows-laptop-1\",\n \"server_url\": \"ws://192.168.1.100:8080\",\n \"os\": \"Windows\",\n \"capabilities\": [\"excel\", \"outlook\", \"browser\"],\n \"metadata\": {\n \"region\": \"us-west-2\",\n \"owner\": \"data-team\"\n },\n \"auto_connect\": true,\n \"max_retries\": 5\n}\n```\n\n**Success Response (200):**\n```json\n{\n \"status\": \"success\",\n \"message\": \"Device 'windows-laptop-1' added successfully\",\n \"device\": {\n \"device_id\": \"windows-laptop-1\",\n \"server_url\": \"ws://192.168.1.100:8080\",\n \"os\": \"Windows\",\n \"capabilities\": [\"excel\", \"outlook\", \"browser\"],\n \"auto_connect\": true,\n \"max_retries\": 5,\n \"metadata\": {\n \"region\": \"us-west-2\",\n \"owner\": \"data-team\"\n }\n }\n}\n```\n\n**Error Responses:**\n\n- **404 Not Found**: `devices.yaml` configuration file not found\n ```json\n {\n \"detail\": \"devices.yaml not found\"\n }\n ```\n\n- **409 Conflict**: Device ID already exists\n ```json\n {\n \"detail\": \"Device ID 'windows-laptop-1' already exists\"\n }\n ```\n\n- **500 Internal Server Error**: Failed to add device\n ```json\n {\n \"detail\": \"Failed to add device: \"\n }\n ```\n\n### WebSocket Connection\n\nThe WebUI maintains a persistent WebSocket connection to the Galaxy backend for bidirectional real-time communication.\n\n**Connection URL:** `ws://localhost:8000/ws`\n\n### Message Types\n\n#### Client → Server\n\n**1. User Request**\n```json\n{\n \"type\": \"request\",\n \"text\": \"Extract sales data and create an Excel report\",\n \"timestamp\": 1234567890\n}\n```\n\n**2. Session Reset**\n```json\n{\n \"type\": \"reset\",\n \"timestamp\": 1234567890\n}\n```\n\n**3. Ping (Keepalive)**\n```json\n{\n \"type\": \"ping\",\n \"timestamp\": 1234567890\n}\n```\n\n#### Server → Client\n\n**1. Welcome Message**\n```json\n{\n \"type\": \"welcome\",\n \"message\": \"Connected to Galaxy Web UI\",\n \"timestamp\": 1234567890\n}\n```\n\n**2. Device Snapshot (on connect)**\n```json\n{\n \"event_type\": \"device_snapshot\",\n \"source_id\": \"webui.server\",\n \"timestamp\": 1234567890,\n \"data\": {\n \"event_name\": \"device_snapshot\",\n \"device_count\": 2\n },\n \"all_devices\": {\n \"windows_device_1\": {\n \"device_id\": \"windows_device_1\",\n \"status\": \"connected\",\n \"os\": \"windows\",\n \"capabilities\": [\"desktop_automation\", \"excel\"],\n \"metadata\": {},\n \"last_heartbeat\": \"2025-11-09T10:30:00\",\n \"current_task_id\": null\n }\n }\n}\n```\n\n**3. Galaxy Events**\n\nAll Galaxy events are forwarded to the WebUI in real-time:\n\n```json\n{\n \"event_type\": \"agent_response\",\n \"source_id\": \"ConstellationAgent\",\n \"timestamp\": 1234567890,\n \"agent_name\": \"ConstellationAgent\",\n \"agent_type\": \"constellation\",\n \"output_type\": \"response\",\n \"output_data\": {\n \"thought\": \"I need to decompose this task...\",\n \"plan\": [\"Analyze requirements\", \"Create DAG\", \"Assign devices\"],\n \"response\": \"Creating constellation with 3 tasks\"\n }\n}\n```\n\n```json\n{\n \"event_type\": \"constellation_created\",\n \"source_id\": \"TaskConstellation\",\n \"timestamp\": 1234567890,\n \"constellation_id\": \"constellation_123\",\n \"constellation_state\": \"planning\",\n \"data\": {\n \"constellation\": {\n \"constellation_id\": \"constellation_123\",\n \"name\": \"Sales Report Pipeline\",\n \"state\": \"planning\",\n \"tasks\": {\n \"task_1\": {\n \"task_id\": \"task_1\",\n \"name\": \"Extract Data\",\n \"status\": \"pending\",\n \"target_device_id\": \"linux_device_1\"\n }\n },\n \"dependencies\": {\n \"task_2\": [\"task_1\"]\n }\n }\n }\n}\n```\n\n```json\n{\n \"event_type\": \"task_status_changed\",\n \"source_id\": \"TaskOrchestrator\",\n \"timestamp\": 1234567890,\n \"task_id\": \"task_1\",\n \"status\": \"running\",\n \"result\": null,\n \"error\": null\n}\n```\n\n```json\n{\n \"event_type\": \"device_status_changed\",\n \"source_id\": \"DeviceManager\",\n \"timestamp\": 1234567890,\n \"device_id\": \"windows_device_1\",\n \"device_status\": \"busy\",\n \"device_info\": {\n \"current_task_id\": \"task_2\"\n }\n}\n```\n\n---\n\n## 🎨 User Interface\n\n### Main Layout\n\nThe WebUI uses a three-panel layout:\n\n```mermaid\ngraph LR\n subgraph UI[\"Galaxy WebUI Layout\"]\n subgraph Header[\"🌌 Header Bar\"]\n H1[Galaxy Logo]\n H2[Connection Status]\n H3[Settings]\n end\n \n subgraph Left[\"📱 Left Panel: Devices\"]\n L1[Device Card 1 Windows 🟢 Connected]\n L2[Device Card 2 Linux 🔵 Busy]\n L3[Device Card 3 macOS 🟢 Idle]\n end\n \n subgraph Center[\"💬 Center Panel: Chat\"]\n C1[Message History User/Agent/Actions]\n C2[Action Trees Collapsible]\n C3[Input Box Type request...]\n end\n \n subgraph Right[\"📊 Right Panel: Tabs\"]\n R1[🌟 Constellation DAG Graph]\n R2[📋 Tasks Task List]\n R3[📝 Details Selected Info]\n end\n \n Header -.-> Left\n Header -.-> Center\n Header -.-> Right\n Left -.-> Center\n Center -.-> Right\n end\n \n style Header fill:#1a1a2e,stroke:#00d4ff,stroke-width:2px,color:#fff\n style Left fill:#0f1419,stroke:#10b981,stroke-width:2px,color:#fff\n style Center fill:#16213e,stroke:#7b2cbf,stroke-width:2px,color:#fff\n style Right fill:#1a1a2e,stroke:#ff006e,stroke-width:2px,color:#fff\n```\n\n### Key Features\n\n#### 🗣️ Chat Interface\n\n**Location:** Center panel\n\n**Features:**\n- Natural language input for requests\n- Message history with agent responses\n- Collapsible action trees showing execution details\n- Thought, plan, and response display\n- Status indicators (pending, running, completed, failed)\n- Markdown rendering for rich text\n- Code block syntax highlighting\n\n**Message Types:**\n- **User Messages**: Your requests to Galaxy\n- **Agent Responses**: ConstellationAgent thoughts, plans, and responses\n- **Action Messages**: Individual constellation operations (add_task, build_constellation, etc.)\n- **System Messages**: Status updates and notifications\n\n#### 📊 DAG Visualization\n\n**Location:** Right panel → Constellation tab\n\n**Features:**\n- Interactive node-and-edge graph\n- Real-time task status updates\n- Color-coded status indicators:\n - 🔵 Pending: Gray\n - 🟡 Running: Blue (animated)\n - 🟢 Completed: Green\n - 🔴 Failed: Red\n - ⚫ Skipped: Orange\n- Dependency edges showing task relationships\n- Pan and zoom controls\n- Automatic layout optimization\n- Node click to view task details\n\n**Interaction:**\n- **Click node**: Select task and show details\n- **Pan**: Click and drag background\n- **Zoom**: Mouse wheel or pinch gesture\n- **Fit view**: Click fit-to-screen button\n\n#### 🖥️ Device Management\n\n**Location:** Left sidebar\n\n**Features:**\n- Device status cards with real-time updates\n- Color-coded status indicators:\n - 🟢 Connected/Idle: Green\n - 🔵 Busy: Blue\n - 🟡 Connecting: Yellow\n - 🔴 Disconnected/Failed: Red\n- Capability badges\n- Current task assignment\n- Last heartbeat timestamp\n- Connection metrics\n- Click to view device details\n- **➕ Add Device Button**: Manually add new devices through UI\n\n**Device Information:**\n- OS type and version\n- Server URL\n- Installed applications\n- Performance tier\n- Custom metadata\n\n**Adding a New Device:**\n\nClick the **\"+\"** button in the Device Panel header to open the Add Device Modal:\n\n
\n \n
Add Device Modal - Register new device agents through the UI
\n
\n\n1. **Basic Information:**\n - **Device ID**: Unique identifier for the device (required)\n - **Server URL**: WebSocket endpoint URL (must start with `ws://` or `wss://`)\n - **Operating System**: Select from Windows, Linux, macOS, or enter custom OS\n\n2. **Capabilities:**\n - Add capabilities one by one (e.g., `excel`, `outlook`, `browser`)\n - Remove capabilities by clicking the ✕ icon\n - At least one capability is required\n\n3. **Advanced Options:**\n - **Auto-connect**: Automatically connect to device after registration (default: enabled)\n - **Max Retries**: Maximum connection retry attempts (default: 5)\n\n4. **Metadata (Optional):**\n - Add custom key-value pairs for additional device information\n - Examples: `region: us-east-1`, `tier: premium`, `owner: team-a`\n\n**API Endpoint:**\n\n```http\nPOST /api/devices\nContent-Type: application/json\n\n{\n \"device_id\": \"my-device-1\",\n \"server_url\": \"ws://192.168.1.100:8080\",\n \"os\": \"Windows\",\n \"capabilities\": [\"excel\", \"outlook\", \"powerpoint\"],\n \"metadata\": {\n \"region\": \"us-east-1\",\n \"tier\": \"standard\"\n },\n \"auto_connect\": true,\n \"max_retries\": 5\n}\n```\n\n**Response:**\n\n```json\n{\n \"status\": \"success\",\n \"message\": \"Device 'my-device-1' added successfully\",\n \"device\": {\n \"device_id\": \"my-device-1\",\n \"server_url\": \"ws://192.168.1.100:8080\",\n \"os\": \"Windows\",\n \"capabilities\": [\"excel\", \"outlook\", \"powerpoint\"],\n \"auto_connect\": true,\n \"max_retries\": 5,\n \"metadata\": {\n \"region\": \"us-east-1\",\n \"tier\": \"standard\"\n }\n }\n}\n```\n\n**Device Registration Process:**\n\nWhen a device is added through the UI:\n\n1. **Validation**: Form data is validated (required fields, URL format, duplicate device_id)\n2. **Configuration**: Device is saved to `config/galaxy/devices.yaml`\n3. **Registration**: Device is registered with the Galaxy Device Manager\n4. **Connection**: If `auto_connect` is enabled, connection is initiated automatically\n5. **Event Broadcast**: Device status updates are broadcast to all WebSocket clients\n6. **UI Update**: Device card appears in the Device Panel with real-time status\n\n#### 📋 Task Details\n\n**Location:** Right panel → Tasks tab / Details tab\n\n**Features:**\n- Task name and description\n- Current status with icon\n- Assigned device\n- Dependencies and dependents\n- Input and output data\n- Execution results\n- Error messages (if failed)\n- Execution timeline\n- Retry information\n\n#### 📡 Event Log\n\n**Location:** Right panel (optional view)\n\n**Features:**\n- Real-time event stream\n- Event type filtering\n- Timestamp display\n- JSON payload viewer\n- Search and filter\n- Auto-scroll option\n- Export to JSON\n\n---\n\n## 🎨 Theme and Styling\n\n### Design System\n\nThe Galaxy WebUI uses a **space-themed design** with a dark color palette and vibrant accents.\n\n#### Color Palette\n\n```typescript\n// Primary Colors\ngalaxy-dark: #0a0e27 // Deep space background\ngalaxy-blue: #00d4ff // Cyan accent (primary actions)\ngalaxy-purple: #7b2cbf // Purple accent (secondary)\ngalaxy-pink: #ff006e // Pink accent (tertiary)\n\n// Status Colors\nemerald: #10b981 // Success/Completed\ncyan: #06b6d4 // Running/Active\namber: #f59e0b // Warning/Pending\nrose: #f43f5e // Error/Failed\nslate: #64748b // Neutral/Disabled\n```\n\n#### Visual Effects\n\n- **Starfield Background**: Animated particle system with depth layers\n- **Glassmorphism**: Frosted glass panels with backdrop blur\n- **Glow Effects**: Neon-style glows on interactive elements\n- **Smooth Animations**: Framer Motion for transitions\n- **Gradient Accents**: Multi-color gradients on headers and buttons\n\n#### Accessibility\n\n- **High Contrast Mode**: Toggle for improved readability\n- **Keyboard Navigation**: Full keyboard support\n- **Screen Reader**: ARIA labels and semantic HTML\n- **Focus Indicators**: Clear focus rings on interactive elements\n\n---\n\n## 🔧 Configuration\n\n### Server Configuration\n\nThe WebUI server is configured through command-line arguments:\n\n```powershell\npython -m galaxy --webui [OPTIONS]\n```\n\n**Options:**\n\n| Flag | Description | Default |\n|------|-------------|---------|\n| `--webui` | Enable WebUI mode | `False` |\n| `--session-name` | Session display name | `\"Galaxy Session\"` |\n| `--log-level` | Logging level | `INFO` |\n| `--port` | Server port (if implemented) | `8000` |\n\n### Frontend Configuration\n\n**Development Mode:**\n\n```bash\ncd galaxy/webui/frontend\nnpm run dev\n```\n\nAccess at: `http://localhost:5173` (Vite dev server with HMR)\n\n**Environment Variables:**\n\n```bash\n# .env.development\nVITE_WS_URL=ws://localhost:8000/ws\nVITE_API_URL=http://localhost:8000\n```\n\n**Build Configuration:**\n\n```bash\ncd galaxy/webui/frontend\nnpm run build\n```\n\nBuilds production-ready frontend to `galaxy/webui/frontend/dist/`\n\n---\n\n## 🔍 Event Handling\n\n### Event Flow\n\n```mermaid\nflowchart TD\n A[Galaxy Core Event] --> B[Event Bus publish]\n B --> C[WebSocketObserver on_event]\n C --> D[EventSerializer serialize_event]\n D --> D1[Type-specific field extraction]\n D --> D2[Recursive value serialization]\n D2 --> D3[Python → JSON]\n D3 --> E[WebSocket Broadcast to all clients]\n E --> F[Frontend Clients receive message]\n F --> G[Store Update Zustand]\n G --> H[UI Re-render React Components]\n \n style A fill:#0a0e27,stroke:#ff006e,stroke-width:2px,color:#fff\n style C fill:#16213e,stroke:#7b2cbf,stroke-width:2px,color:#fff\n style D fill:#1a1a2e,stroke:#f59e0b,stroke-width:2px,color:#fff\n style E fill:#1a1a2e,stroke:#00d4ff,stroke-width:2px,color:#fff\n style G fill:#0f1419,stroke:#10b981,stroke-width:2px,color:#fff\n style H fill:#1a1a2e,stroke:#f59e0b,stroke-width:2px,color:#fff\n```\n\n### Event Serialization\n\nThe `EventSerializer` class handles conversion of complex Python objects to JSON-compatible format:\n\n**Features:**\n- **Type Handler Registry**: Pre-registered handlers for Galaxy-specific types (TaskStarLine, TaskConstellation)\n- **Type Caching**: Cached imports to avoid repeated import attempts\n- **Recursive Serialization**: Handles nested structures (dicts, lists, dataclasses, Pydantic models)\n- **Polymorphic Event Handling**: Different serialization logic for TaskEvent, ConstellationEvent, AgentEvent, DeviceEvent\n- **Fallback Strategies**: Multiple serialization attempts with graceful fallback to string representation\n\n**Serialization Chain:**\n1. Handle primitives (str, int, float, bool, None)\n2. Handle datetime objects → ISO format\n3. Handle collections (dict, list, tuple) → recursive serialization\n4. Check registered type handlers (TaskStarLine, TaskConstellation)\n5. Try dataclass serialization (`asdict()`)\n6. Try Pydantic model serialization (`model_dump()`)\n7. Try generic `to_dict()` method\n8. Fallback to `str()` representation\n\n### Event Types\n\nThe WebUI subscribes to all Galaxy event types:\n\n| Event Type | Source | Description |\n|------------|--------|-------------|\n| `agent_request` | ConstellationAgent | User request received |\n| `agent_response` | ConstellationAgent | Agent thought/plan/response |\n| `constellation_created` | TaskConstellation | New constellation formed |\n| `constellation_updated` | TaskConstellation | Constellation modified |\n| `constellation_completed` | TaskConstellation | All tasks finished |\n| `task_created` | TaskOrchestrator | New task added |\n| `task_assigned` | TaskOrchestrator | Task assigned to device |\n| `task_started` | TaskOrchestrator | Task execution started |\n| `task_status_changed` | TaskOrchestrator | Task status updated |\n| `task_completed` | TaskOrchestrator | Task finished successfully |\n| `task_failed` | TaskOrchestrator | Task encountered error |\n| `device_connected` | DeviceManager | Device came online |\n| `device_disconnected` | DeviceManager | Device went offline |\n| `device_status_changed` | DeviceManager | Device status updated |\n| `device_heartbeat` | DeviceManager | Device health check |\n\n### State Management\n\nThe frontend uses **Zustand** for centralized state management:\n\n```typescript\n// Store Structure\ninterface GalaxyStore {\n // Connection\n connectionStatus: ConnectionStatus;\n connected: boolean;\n \n // Session\n session: {\n id: string | null;\n displayName: string;\n startedAt: number | null;\n };\n \n // Data\n messages: Message[];\n constellations: Record;\n tasks: Record;\n devices: Record;\n notifications: NotificationItem[];\n \n // UI State\n ui: {\n activeConstellationId: string | null;\n activeTaskId: string | null;\n activeDeviceId: string | null;\n rightPanelTab: 'constellation' | 'tasks' | 'details';\n showDeviceDrawer: boolean;\n };\n}\n```\n\n---\n\n## 📱 Responsive Design\n\nThe WebUI is designed to work on various screen sizes:\n\n### Desktop (1920px+)\n- Three-panel layout with full sidebar\n- Large DAG visualization\n- Expanded device cards\n\n### Laptop (1280px - 1919px)\n- Standard three-panel layout\n- Medium DAG visualization\n- Compact device cards\n\n### Tablet (768px - 1279px)\n- Collapsible sidebar\n- Simplified DAG view\n- Stacked layout option\n\n### Mobile (< 768px)\n- Single-panel navigation\n- Tab-based interface\n- Touch-optimized controls\n\n!!!warning \"Recommended Resolution\"\n For the best experience, use a desktop or laptop with at least **1280px width**. The DAG visualization requires adequate screen space for clear readability.\n\n---\n\n## 🐛 Troubleshooting\n\n### Connection Issues\n\n**Problem:** WebSocket connection fails\n\n**Solutions:**\n\n1. **Verify backend is running:**\n ```powershell\n # Check health endpoint\n curl http://localhost:8000/health\n ```\n\n2. **Check firewall settings:**\n - Allow incoming connections on port 8000\n - Check corporate firewall/proxy settings\n\n3. **Verify WebSocket URL:**\n - Browser console should show: `WebSocket connection established`\n - Check for CORS errors in console\n\n4. **Try different port:**\n ```powershell\n python -m galaxy --webui --port 8080\n ```\n\n### Frontend Not Loading\n\n**Problem:** Blank page or \"Server is running\" placeholder\n\n**Solutions:**\n\n1. **Build the frontend:**\n ```bash\n cd galaxy/webui/frontend\n npm install\n npm run build\n ```\n\n2. **Check build output:**\n - Verify `galaxy/webui/frontend/dist/` exists\n - Check for TypeScript errors: `npm run build`\n\n3. **Clear browser cache:**\n - Hard refresh: `Ctrl+Shift+R` (Windows) or `Cmd+Shift+R` (Mac)\n - Clear site data in DevTools\n\n### Events Not Appearing\n\n**Problem:** No events shown in UI, DAG not updating\n\n**Solutions:**\n\n1. **Check event system:**\n - Look for \"WebSocket observer registered\" in backend logs\n - Verify connection count: `curl http://localhost:8000/health`\n\n2. **Check browser console:**\n - Look for WebSocket message logs\n - Check for JavaScript errors\n\n3. **Enable debug mode:**\n ```powershell\n python -m galaxy --webui --log-level DEBUG\n ```\n\n### Performance Issues\n\n**Problem:** UI slow or unresponsive\n\n**Solutions:**\n\n1. **Limit event log size:**\n - Event log keeps last 200 events\n - Messages limited to 500\n\n2. **Reduce DAG complexity:**\n - Large constellations (>50 tasks) may be slow\n - Consider viewport culling for very large graphs\n\n3. **Check browser performance:**\n - Close unnecessary tabs\n - Use Chrome/Edge for best performance\n - Disable browser extensions\n\n### Device Addition Issues\n\n**Problem:** Cannot add device through UI\n\n**Solutions:**\n\n1. **Check `devices.yaml` exists:**\n ```powershell\n # Verify configuration file\n Test-Path config/galaxy/devices.yaml\n ```\n\n2. **Verify device ID uniqueness:**\n - Device ID must be unique across all devices\n - Check existing devices in the Device Panel\n\n3. **Validate server URL format:**\n - Must start with `ws://` or `wss://`\n - Example: `ws://192.168.1.100:8080` or `wss://device.example.com`\n - Ensure device server is actually running at that URL\n\n4. **Check backend logs:**\n ```powershell\n # Look for error messages\n python -m galaxy --webui --log-level DEBUG\n ```\n\n**Problem:** Device added but not connecting\n\n**Solutions:**\n\n1. **Verify device server is running:**\n - Check that the device agent is running at the specified URL\n - Test connection: `curl ws://your-device-url/`\n\n2. **Check firewall/network:**\n - Ensure WebSocket port is open\n - Verify no proxy/firewall blocking connection\n\n3. **Check device logs:**\n - Look at the device agent logs for connection errors\n - Verify device can reach the Galaxy server\n\n4. **Manual connection:**\n - If `auto_connect` failed, devices will retry automatically\n - Check `connection_attempts` in device details\n - Increase `max_retries` if needed\n\n**Problem:** Validation errors when adding device\n\n**Common Validation Issues:**\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| \"Device ID is required\" | Empty device_id field | Provide a unique identifier |\n| \"Device ID already exists\" | Duplicate device_id | Choose a different ID |\n| \"Server URL is required\" | Empty server_url | Provide WebSocket URL |\n| \"Invalid WebSocket URL\" | Wrong URL format | Use `ws://` or `wss://` prefix |\n| \"OS is required\" | No OS selected | Select or enter OS type |\n| \"At least one capability required\" | No capabilities added | Add at least one capability |\n\n---\n\n## 🧪 Development\n\n### Prerequisites\n\n- **Node.js** >= 18\n- **npm** >= 9\n- **Python** >= 3.10\n- **Galaxy** installed and configured\n\n### Development Setup\n\n```bash\n# 1. Install frontend dependencies\ncd galaxy/webui/frontend\nnpm install\n\n# 2. Start development server\nnpm run dev\n\n# 3. In another terminal, start Galaxy backend\ncd ../../..\npython -m galaxy --webui\n```\n\n**Development URL:** `http://localhost:5173`\n\n### Project Structure\n\n```\ngalaxy/webui/\n├── server.py # FastAPI application entry point\n├── dependencies.py # AppState and dependency injection\n├── websocket_observer.py # EventSerializer + WebSocketObserver\n├── __init__.py\n├── models/ # Data models and validation\n│ ├── __init__.py # Export all models\n│ ├── enums.py # WebSocketMessageType, RequestStatus\n│ ├── requests.py # WebSocketMessage, DeviceAddRequest, etc.\n│ └── responses.py # WelcomeMessage, DeviceSnapshot, etc.\n├── services/ # Business logic layer\n│ ├── __init__.py\n│ ├── config_service.py # Configuration management\n│ ├── device_service.py # Device operations and snapshots\n│ └── galaxy_service.py # Galaxy client interaction\n├── handlers/ # Request/message processing\n│ ├── __init__.py\n│ └── websocket_handlers.py # WebSocketMessageHandler class\n├── routers/ # API endpoint definitions\n│ ├── __init__.py\n│ ├── health.py # GET /health\n│ ├── devices.py # POST /api/devices\n│ └── websocket.py # WebSocket /ws\n├── templates/ # HTML templates\n│ └── index.html # Fallback page when frontend not built\n└── frontend/ # React frontend application\n ├── src/\n │ ├── main.tsx # Entry point\n │ ├── App.tsx # Main layout\n │ ├── components/ # React components\n │ │ ├── chat/ # Chat interface\n │ │ ├── constellation/ # DAG visualization\n │ │ ├── devices/ # Device management\n │ │ ├── layout/ # Layout components\n │ │ ├── session/ # Session management\n │ │ └── tasks/ # Task details\n │ ├── services/ # WebSocket client\n │ └── store/ # Zustand store\n ├── public/ # Static assets\n ├── dist/ # Build output (gitignored)\n ├── package.json # Dependencies\n ├── vite.config.ts # Vite configuration\n ├── tailwind.config.js # Tailwind CSS\n └── tsconfig.json # TypeScript config\n```\n\n### Building for Production\n\n```bash\ncd galaxy/webui/frontend\nnpm run build\n```\n\nOutput: `galaxy/webui/frontend/dist/`\n\n### Code Quality\n\n**Frontend:**\n\n```bash\n# Lint\nnpm run lint\n\n# Type check\nnpm run type-check\n\n# Format\nnpm run format\n```\n\n**Backend:**\n\nThe modular architecture improves testability. Example unit tests:\n\n```python\n# tests/webui/test_event_serializer.py\nimport pytest\nfrom galaxy.webui.websocket_observer import EventSerializer\nfrom galaxy.core.events import TaskEvent\n\ndef test_serialize_task_event():\n \"\"\"Test serialization of TaskEvent.\"\"\"\n serializer = EventSerializer()\n \n event = TaskEvent(\n event_type=EventType.TASK_STARTED,\n source_id=\"test\",\n timestamp=1234567890,\n task_id=\"task_1\",\n status=\"running\",\n result=None,\n error=None\n )\n \n result = serializer.serialize_event(event)\n \n assert result[\"event_type\"] == \"task_started\"\n assert result[\"task_id\"] == \"task_1\"\n assert result[\"status\"] == \"running\"\n\ndef test_serialize_nested_dict():\n \"\"\"Test recursive serialization of nested structures.\"\"\"\n serializer = EventSerializer()\n \n data = {\n \"level1\": {\n \"level2\": {\n \"value\": 42\n }\n }\n }\n \n result = serializer.serialize_value(data)\n assert result[\"level1\"][\"level2\"][\"value\"] == 42\n```\n\n```python\n# tests/webui/test_services.py\nimport pytest\nfrom galaxy.webui.services.device_service import DeviceService\nfrom galaxy.webui.dependencies import AppState\n\ndef test_build_device_snapshot():\n \"\"\"Test device snapshot building.\"\"\"\n app_state = AppState()\n # Setup mock galaxy_client with devices\n \n service = DeviceService(app_state)\n snapshot = service.build_device_snapshot()\n \n assert \"device_count\" in snapshot\n assert \"all_devices\" in snapshot\n```\n\n```python\n# tests/webui/test_handlers.py\nimport pytest\nfrom unittest.mock import AsyncMock, MagicMock\nfrom galaxy.webui.handlers.websocket_handlers import WebSocketMessageHandler\nfrom galaxy.webui.models.enums import WebSocketMessageType\n\n@pytest.mark.asyncio\nasync def test_handle_ping():\n \"\"\"Test ping message handling.\"\"\"\n websocket = AsyncMock()\n app_state = MagicMock()\n \n handler = WebSocketMessageHandler(websocket, app_state)\n \n response = await handler.handle_message({\n \"type\": WebSocketMessageType.PING,\n \"timestamp\": 1234567890\n })\n \n assert response[\"type\"] == \"pong\"\n```\n\n---\n\n## 🚀 Advanced Usage\n\n### Extending the Backend\n\nThe modular architecture makes it easy to extend the Galaxy WebUI backend:\n\n#### Adding a New API Endpoint\n\n**1. Define Pydantic models:**\n\n```python\n# galaxy/webui/models/requests.py\nfrom pydantic import BaseModel, Field\n\nclass TaskQueryRequest(BaseModel):\n \"\"\"Request to query task status.\"\"\"\n task_id: str = Field(..., description=\"The task ID to query\")\n include_history: bool = Field(default=False)\n```\n\n```python\n# galaxy/webui/models/responses.py\nfrom pydantic import BaseModel\n\nclass TaskQueryResponse(BaseModel):\n \"\"\"Response with task details.\"\"\"\n task_id: str\n status: str\n result: dict | None = None\n```\n\n**2. Create a service method:**\n\n```python\n# galaxy/webui/services/task_service.py\nfrom typing import Dict, Any\nfrom galaxy.webui.dependencies import AppState\n\nclass TaskService:\n \"\"\"Service for task-related operations.\"\"\"\n \n def __init__(self, app_state: AppState):\n self.app_state = app_state\n \n def get_task_details(self, task_id: str, include_history: bool) -> Dict[str, Any]:\n \"\"\"Get details for a specific task.\"\"\"\n galaxy_session = self.app_state.galaxy_session\n if not galaxy_session:\n raise ValueError(\"No active Galaxy session\")\n \n # Your business logic here\n task = galaxy_session.get_task(task_id)\n return {\n \"task_id\": task.task_id,\n \"status\": task.status.value,\n \"result\": task.result if include_history else None\n }\n```\n\n**3. Add a router endpoint:**\n\n```python\n# galaxy/webui/routers/tasks.py\nfrom fastapi import APIRouter, Depends\nfrom galaxy.webui.dependencies import get_app_state\nfrom galaxy.webui.models.requests import TaskQueryRequest\nfrom galaxy.webui.models.responses import TaskQueryResponse\nfrom galaxy.webui.services.task_service import TaskService\n\nrouter = APIRouter(prefix=\"/api/tasks\", tags=[\"tasks\"])\n\n@router.post(\"/query\", response_model=TaskQueryResponse)\nasync def query_task(\n request: TaskQueryRequest,\n app_state = Depends(get_app_state)\n):\n \"\"\"Query task status and details.\"\"\"\n service = TaskService(app_state)\n result = service.get_task_details(request.task_id, request.include_history)\n return TaskQueryResponse(**result)\n```\n\n**4. Register the router:**\n\n```python\n# galaxy/webui/server.py\nfrom galaxy.webui.routers import tasks_router\n\napp.include_router(tasks_router)\n```\n\n#### Adding a New WebSocket Message Type\n\n**1. Add enum value:**\n\n```python\n# galaxy/webui/models/enums.py\nclass WebSocketMessageType(str, Enum):\n \"\"\"Types of messages exchanged via WebSocket.\"\"\"\n # ... existing types ...\n CUSTOM_ACTION = \"custom_action\"\n```\n\n**2. Add request model:**\n\n```python\n# galaxy/webui/models/requests.py\nclass CustomActionMessage(BaseModel):\n \"\"\"Custom action message.\"\"\"\n action_name: str\n parameters: Dict[str, Any] = Field(default_factory=dict)\n```\n\n**3. Add handler method:**\n\n```python\n# galaxy/webui/handlers/websocket_handlers.py\nasync def _handle_custom_action(self, data: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"Handle custom action messages.\"\"\"\n message = CustomActionMessage(**data)\n \n # Your logic here\n result = await self.service.perform_custom_action(\n message.action_name,\n message.parameters\n )\n \n return {\n \"type\": \"custom_action_completed\",\n \"result\": result\n }\n```\n\n**4. Register handler:**\n\n```python\n# galaxy/webui/handlers/websocket_handlers.py\ndef __init__(self, websocket: WebSocket, app_state: AppState):\n # ... existing code ...\n self._handlers[WebSocketMessageType.CUSTOM_ACTION] = self._handle_custom_action\n```\n\n#### Customizing Event Serialization\n\nAdd custom serialization for new types:\n\n```python\n# galaxy/webui/websocket_observer.py\n\nclass EventSerializer:\n def _register_handlers(self) -> None:\n \"\"\"Register type-specific serialization handlers.\"\"\"\n # ... existing handlers ...\n \n # Add custom type handler\n try:\n from your_module import CustomType\n self._cached_types[\"CustomType\"] = CustomType\n self._type_handlers[CustomType] = self._serialize_custom_type\n except ImportError:\n self._cached_types[\"CustomType\"] = None\n \n def _serialize_custom_type(self, value: Any) -> Dict[str, Any]:\n \"\"\"Serialize a CustomType object.\"\"\"\n try:\n return {\n \"id\": value.id,\n \"data\": self.serialize_value(value.data),\n \"metadata\": value.get_metadata()\n }\n except Exception as e:\n self.logger.warning(f\"Failed to serialize CustomType: {e}\")\n return str(value)\n```\n\n### Custom Event Handlers\n\nYou can extend the WebUI with custom event handlers:\n\n```typescript\n// src/services/customHandlers.ts\nimport { GalaxyEvent } from './websocket';\n\nexport function handleCustomEvent(event: GalaxyEvent) {\n if (event.event_type === 'custom_event') {\n // Your custom logic\n console.log('Custom event:', event);\n }\n}\n```\n\n### Programmatic Device Management\n\nAdd devices programmatically using the API:\n\n```typescript\n// Add a device via API\nasync function addDevice(deviceConfig: {\n device_id: string;\n server_url: string;\n os: string;\n capabilities: string[];\n metadata?: Record;\n auto_connect?: boolean;\n max_retries?: number;\n}) {\n const response = await fetch('http://localhost:8000/api/devices', {\n method: 'POST',\n headers: {\n 'Content-Type': 'application/json',\n },\n body: JSON.stringify(deviceConfig),\n });\n\n if (!response.ok) {\n const error = await response.json();\n throw new Error(error.detail || 'Failed to add device');\n }\n\n return await response.json();\n}\n\n// Usage example\ntry {\n const result = await addDevice({\n device_id: 'production-server-1',\n server_url: 'wss://prod-device.company.com',\n os: 'Linux',\n capabilities: ['docker', 'kubernetes', 'python'],\n metadata: {\n region: 'us-east-1',\n environment: 'production',\n tier: 'premium',\n },\n auto_connect: true,\n max_retries: 10,\n });\n \n console.log('Device added:', result.device);\n} catch (error) {\n console.error('Failed to add device:', error);\n}\n```\n\n**Batch Device Addition:**\n\n```python\n# Python script to add multiple devices\nimport requests\nimport json\n\ndevices = [\n {\n \"device_id\": \"win-desktop-1\",\n \"server_url\": \"ws://192.168.1.10:8080\",\n \"os\": \"Windows\",\n \"capabilities\": [\"office\", \"excel\", \"outlook\"],\n },\n {\n \"device_id\": \"linux-server-1\",\n \"server_url\": \"ws://192.168.1.20:8080\",\n \"os\": \"Linux\",\n \"capabilities\": [\"python\", \"docker\", \"git\"],\n },\n {\n \"device_id\": \"mac-laptop-1\",\n \"server_url\": \"ws://192.168.1.30:8080\",\n \"os\": \"macOS\",\n \"capabilities\": [\"safari\", \"xcode\", \"python\"],\n }\n]\n\nfor device in devices:\n response = requests.post(\n \"http://localhost:8000/api/devices\",\n json=device,\n headers={\"Content-Type\": \"application/json\"}\n )\n \n if response.status_code == 200:\n result = response.json()\n print(f\"✅ Added: {result['device']['device_id']}\")\n else:\n error = response.json()\n print(f\"❌ Failed: {device['device_id']} - {error.get('detail')}\")\n```\n\n**Checking Device Status:**\n\nAfter adding devices, monitor their connection status through WebSocket events:\n\n```typescript\n// Listen for device connection events\nwebsocket.onmessage = (event) => {\n const data = JSON.parse(event.data);\n \n if (data.event_type === 'device_status_changed') {\n console.log(`Device ${data.device_id} status: ${data.device_status}`);\n \n if (data.device_status === 'connected') {\n console.log('✅ Device connected successfully');\n } else if (data.device_status === 'failed') {\n console.log('❌ Device connection failed');\n }\n }\n};\n```\n\n### Custom Components\n\nAdd custom visualization components:\n\n```tsx\n// src/components/custom/MyVisualization.tsx\nimport React from 'react';\nimport { useGalaxyStore } from '../../store/galaxyStore';\n\nexport const MyVisualization: React.FC = () => {\n const constellation = useGalaxyStore(s => \n s.constellations[s.ui.activeConstellationId || '']\n );\n \n return (\n
\n {/* Your custom visualization */}\n
\n );\n};\n```\n\n### Theming\n\nCreate custom themes by extending Tailwind configuration:\n\n```javascript\n// tailwind.config.js\nmodule.exports = {\n theme: {\n extend: {\n colors: {\n 'custom-primary': '#your-color',\n 'custom-secondary': '#your-color',\n },\n },\n },\n};\n```\n\n---\n\n## 📊 Monitoring and Analytics\n\n### Health Check\n\n**Endpoint:** `GET /health`\n\n```json\n{\n \"status\": \"healthy\",\n \"connections\": 3,\n \"events_sent\": 1247\n}\n```\n\n### Metrics\n\nThe WebUI tracks:\n- Active WebSocket connections\n- Total events broadcasted\n- Device online/offline status\n- Task execution statistics\n- Session duration\n\n### Logging\n\n**Backend Logs:**\n```\nINFO - WebSocket connection established from ('127.0.0.1', 54321)\nINFO - Broadcasting event #42: agent_response to 2 clients\nINFO - WebSocket client disconnected. Total connections: 1\n```\n\n**Frontend Console:**\n```javascript\n🌌 Connected to Galaxy WebSocket\n📨 Raw WebSocket message received\n📦 Parsed event data: {event_type: 'constellation_created', ...}\n```\n\n---\n\n## 🔒 Security Considerations\n\n### Production Deployment\n\nWhen deploying to production:\n\n1. **Use HTTPS/WSS:**\n ```python\n # Use secure WebSocket\n wss://your-domain.com/ws\n ```\n\n2. **Configure CORS:**\n ```python\n # server.py\n app.add_middleware(\n CORSMiddleware,\n allow_origins=[\"https://your-domain.com\"], # Specific origins\n allow_credentials=True,\n allow_methods=[\"GET\", \"POST\"],\n allow_headers=[\"*\"],\n )\n ```\n\n3. **Add Authentication:**\n - Implement JWT tokens\n - Validate WebSocket connections\n - Secure API endpoints\n\n4. **Rate Limiting:**\n - Limit request frequency\n - Throttle WebSocket messages\n - Prevent DoS attacks\n\n---\n\n## 📚 Additional Resources\n\n### Documentation\n- [FastAPI WebSocket Documentation](https://fastapi.tiangolo.com/advanced/websockets/)\n- [React Documentation](https://react.dev/)\n- [ReactFlow Documentation](https://reactflow.dev/)\n- [Zustand Documentation](https://github.com/pmndrs/zustand)\n- [Tailwind CSS Documentation](https://tailwindcss.com/)\n- [Vite Documentation](https://vitejs.dev/)\n\n### Galaxy Framework\n- [Galaxy Overview](overview.md)\n- [Constellation Agent](constellation_agent/overview.md)\n- [Task Orchestrator](constellation_orchestrator/overview.md)\n- [Device Manager](client/device_manager.md)\n\n### Community\n- [GitHub Issues](https://github.com/microsoft/UFO/issues)\n- [GitHub Discussions](https://github.com/microsoft/UFO/discussions)\n- [Email Support](mailto:ufo-agent@microsoft.com)\n\n---\n\n## 🎯 Next Steps\n\nNow that you understand the Galaxy WebUI:\n\n1. **[Quick Start Guide](../getting_started/quick_start_galaxy.md)** - Set up your first Galaxy session\n2. **[Constellation Agent](constellation_agent/overview.md)** - Learn about task decomposition\n3. **[Task Orchestrator](constellation_orchestrator/overview.md)** - Understand task execution\n4. **[Device Manager](client/device_manager.md)** - Configure and monitor devices\n\nHappy orchestrating with Galaxy WebUI! 🌌✨\n"
},
{
"path": "documents/docs/getting_started/migration_ufo2_to_galaxy.md",
"content": "# Migration Guide: UFO² to UFO³ Galaxy\n\nThis guide helps you understand the evolution from **UFO²** (Desktop AgentOS) to **UFO³ Galaxy** (Multi-Device AgentOS), and provides practical steps for migrating your workflows to leverage Galaxy's cross-device orchestration capabilities.\n\n---\n\n## 🌟 Understanding the UFO Evolution\n\n### The UFO Journey\n\nThe UFO project has evolved through three major iterations, each addressing increasingly complex automation challenges:\n\n```mermaid\ngraph LR\n A[UFO v1 2024-02] -->|Desktop Agent| B[UFO² 2025-04]\n B -->|Multi-Device| C[UFO³ Galaxy 2025-11]\n \n style A fill:#e3f2fd\n style B fill:#c8e6c9\n style C fill:#fff9c4\n```\n\n#### **UFO (v1.0)** — The Beginning\n📅 *Released: February 2024*\n\n- **Vision**: Screenshot-based Windows automation\n- **Architecture**: Multi-agent (HostAgent + AppAgents)\n- **Approach**: GPT-4V + pure GUI automation (click/type)\n- **Scope**: Single Windows desktop, cross-app workflows\n- **Limitation**: No deep OS integration\n\n**Key Innovation:** First LLM-powered multi-agent GUI automation framework\n\n---\n\n#### **UFO² (v2.0)** — Desktop AgentOS\n📅 *Released: April 2025* \n📄 *Paper:* [UFO²: A Windows Agent for Seamless OS Interaction](https://arxiv.org/abs/2504.14603)\n\n- **Vision**: Deep OS integration for robust automation\n- **Architecture**: Two-tier hierarchy (HostAgent + AppAgents)\n- **Innovations**:\n - ✅ **Hybrid GUI–API execution** (51% fewer LLM calls)\n - ✅ **Windows UIA + Win32 + WinCOM APIs**\n - ✅ **Continuous knowledge learning** from docs & experience\n - ✅ **Picture-in-Picture desktop** (non-disruptive automation)\n - ✅ **MCP server integration** for tool augmentation\n- **Scope**: Single Windows desktop\n- **Success**: 10%+ better than state-of-the-art CUAs\n\n**Key Innovation:** First agent to deeply integrate with Windows OS internals\n\n---\n\n#### **UFO³ Galaxy** — Multi-Device AgentOS\n📅 *Released: November 2025* \n📄 *Paper:* UFO³: Weaving the Digital Agent Galaxy *(Coming Soon)*\n\n- **Vision**: Cross-device orchestration at scale\n- **Architecture**: Constellation-based distributed DAG orchestration\n- **Innovations**:\n - ✅ **Task Constellation** (dynamic DAG decomposition)\n - ✅ **Asynchronous parallel execution** across devices\n - ✅ **Event-driven coordination** with formal safety guarantees\n - ✅ **Dual-mode DAG evolution** (creation + editing)\n - ✅ **Agent Interaction Protocol** (persistent WebSocket)\n - ✅ **Heterogeneous device support** (Windows, Linux, macOS)\n- **Scope**: Multi-device workflows across platforms\n- **Capability**: Orchestrate 10+ devices simultaneously\n\n**Key Innovation:** First LLM-powered multi-device orchestration framework with provable correctness\n\n---\n\n### Architecture Evolution\n\n#### UFO v1 Architecture\n\n**Multi-Agent (GUI-Only)**\n\n```\nUser Request\n ↓\nHostAgent\n ↓\nAppAgent 1, 2, 3...\n ↓\nWindows Apps (GUI)\n```\n\n**Capabilities:**\n\n- Multi-app workflows\n- Pure screenshot + click/type\n- No API integration\n- Single device\n\n#### UFO² Architecture\n\n**Two-Tier Hierarchy (Hybrid)**\n\n```\nUser Request\n ↓\nHostAgent\n ↓\nAppAgent 1, 2, 3...\n ↓\nWindows Apps (GUI + API)\n```\n\n**Capabilities:**\n\n- Multi-app workflows\n- Desktop orchestration\n- Hybrid GUI–API execution\n- Deep OS integration\n- Single device\n\n#### UFO³ Galaxy Architecture\n\n**Constellation Model (Distributed)**\n\n```\nUser Request\n ↓\nConstellationAgent\n ↓\nTask Constellation (DAG)\n ↓\nDevice 1, 2, 3... (UFO² instances)\n ↓\nCross-Platform Apps\n```\n\n**Capabilities:**\n\n- Multi-device workflows\n- Parallel execution\n- Dynamic adaptation\n- Heterogeneous platforms\n\n---\n\n## 🎯 When to Use Which?\n\n### Use **UFO²** (Desktop AgentOS) When:\n\n✅ You're automating tasks on a **single Windows desktop** \n✅ You need **deep Windows integration** (Office, File Explorer, etc.) \n✅ You want **fast, simple execution** without network overhead \n✅ You're learning agent automation basics \n✅ Your workflow is entirely **local** (no cross-device dependencies)\n\n**Examples:**\n- \"Create a PowerPoint presentation from this Excel data\"\n- \"Organize my Downloads folder by file type\"\n- \"Send emails to all contacts in this spreadsheet\"\n\n---\n\n### Use **UFO³ Galaxy** When:\n\n✅ Your workflow spans **multiple devices** (Windows, Linux, servers) \n✅ You need **parallel task execution** for performance \n✅ You have **complex dependencies** between subtasks \n✅ You want **dynamic workflow adaptation** based on results \n✅ You need **fault tolerance** and automatic recovery \n✅ You're orchestrating **heterogeneous systems** (desktop + server + cloud)\n\n**Examples:**\n- \"Clone repo on my laptop, build Docker image on GPU server, deploy to staging, run tests on CI cluster\"\n- \"Fetch data from cloud storage, preprocess on Linux workstation, train model on A100 node, visualize on my Windows machine\"\n- \"Collect logs from all Linux servers, analyze for errors, generate report on Windows\"\n\n---\n\n### Can You Use Both?\n\n**Yes!** UFO² can run as a **device agent** in the Galaxy:\n\n```\nGalaxy (Orchestrator)\n ├── Windows Device (UFO² instance)\n ├── Linux Device (UFO² instance)\n └── Server Device (UFO² instance)\n```\n\nThis is the **recommended hybrid approach** for complex workflows.\n\n---\n\n## 🔄 Key Concept Mapping\n\nUnderstanding how UFO² concepts map to Galaxy:\n\n| UFO² Concept | Galaxy Equivalent | Relationship |\n|--------------|-------------------|--------------|\n| **HostAgent** | **ConstellationAgent** | Global orchestrator (but across devices) |\n| **AppAgent** | **Device Agent (HostAgent)** | Local executor on each device |\n| **Session** | **GalaxySession** | Workflow execution context |\n| **Round** | **Constellation Round** | Orchestration iteration |\n| **Action** | **TaskStar** | Executable unit (but on specific device) |\n| **Blackboard** | **Task Results** | Inter-task communication |\n| **Config File** | `config/ufo/` → `config/galaxy/` | Configuration location |\n| **Execution Mode** | `python -m ufo.server.app --port ` | Device runs as WebSocket server |\n\n### Architecture Translation\n\n**UFO² (Single Device):**\n```python\n# UFO² executes locally\npython -m ufo --task \"Create report from data.xlsx\"\n\n# HostAgent coordinates AppAgents on one desktop\nHostAgent\n ├── ExcelAgent (data.xlsx)\n ├── WordAgent (report.docx)\n └── OutlookAgent (send email)\n```\n\n**Galaxy (Multi-Device):**\n```python\n# Galaxy orchestrates across devices\npython -m galaxy --request \"Create report from data on Server, generate PDF on Windows\"\n\n# ConstellationAgent creates DAG, assigns to devices\nConstellationAgent\n └── TaskConstellation (DAG)\n ├── TaskStar-1: Fetch data → Linux Server\n ├── TaskStar-2: Process → GPU Workstation\n └── TaskStar-3: Generate PDF → Windows Desktop\n```\n\n---\n\n## ⚙️ Configuration Migration\n\n### Step 1: Preserve UFO² Configuration\n\n**Keep your existing UFO² config** — you'll use it for device agents:\n\n```\nconfig/ufo/\n├── agents.yaml # LLM config for device agents\n├── app_agent.yaml # AppAgent settings\n├── host_agent.yaml # HostAgent settings\n└── ...\n```\n\n**No changes needed** — each Galaxy device will use its own UFO² config.\n\n---\n\n### Step 2: Create Galaxy Configuration\n\nGalaxy adds **new orchestration-level config**:\n\n#### A. ConstellationAgent LLM Config\n\n```bash\n# Copy template\ncopy config\\galaxy\\agent.yaml.template config\\galaxy\\agent.yaml\n```\n\nEdit `config/galaxy/agent.yaml`:\n\n```yaml\n# ConstellationAgent LLM (orchestrator)\nCONSTELLATION_AGENT:\n API_TYPE: \"openai\" # or \"azure\", \"qwen\", etc.\n API_BASE: \"https://api.openai.com/v1\"\n API_KEY: \"sk-your-api-key-here\"\n API_MODEL: \"gpt-4o\"\n API_VERSION: null\n\n# Optional: Use different model for orchestration\n# Recommended: Use GPT-4o or Claude for complex DAG reasoning\n```\n\n---\n\n#### B. Device Pool Configuration\n\n**New in Galaxy:** Define all available devices\n\n```bash\n# Create device registry\nnotepad config\\galaxy\\devices.yaml\n```\n\n```yaml\ndevices:\n # Your Windows desktop (existing UFO² instance)\n - device_id: \"my_windows_desktop\"\n server_url: \"ws://localhost:5005/ws\"\n os: \"windows\"\n capabilities:\n - \"office_applications\" # Excel, Word, PowerPoint\n - \"web_browsing\"\n - \"file_management\"\n metadata:\n location: \"local\"\n os: \"windows\"\n performance: \"high\"\n auto_connect: true\n max_retries: 5\n\n # Linux workstation\n - device_id: \"linux_workstation\"\n server_url: \"ws://192.168.1.100:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"python\"\n - \"docker\"\n - \"server\"\n metadata:\n location: \"office\"\n os: \"ubuntu_22.04\"\n performance: \"high\"\n gpu: \"nvidia_a100\"\n auto_connect: true\n\n # GPU server\n - device_id: \"gpu_server\"\n server_url: \"ws://192.168.1.200:5002/ws\"\n os: \"linux\"\n capabilities:\n - \"machine_learning\"\n - \"cuda\"\n - \"docker\"\n metadata:\n os: \"centos_7\"\n gpu: \"nvidia_v100\"\n performance: \"ultra\"\n```\n\n**Capability Matching:** ConstellationAgent uses these capabilities to assign tasks intelligently.\n\n---\n\n#### C. Constellation Runtime Config\n\n```bash\nnotepad config\\galaxy\\constellation.yaml\n```\n\n```yaml\n# Constellation Orchestration Settings\nCONSTELLATION_ID: \"my_constellation\"\nHEARTBEAT_INTERVAL: 30.0 # Device health check (seconds)\nRECONNECT_DELAY: 5.0 # Auto-reconnect delay\nMAX_CONCURRENT_TASKS: 6 # Parallel task limit\nMAX_STEP: 15 # Max orchestration rounds\n\n# Device Configuration\nDEVICE_INFO: \"config/galaxy/devices.yaml\"\n\n# Logging\nLOG_TO_MARKDOWN: true # Generate trajectory reports\n```\n\n---\n\n## 🚀 Migration Steps\n\n### Option 1: Keep UFO² for Local, Add Galaxy for Multi-Device\n\n**Best for:** Gradual adoption, maintaining existing workflows\n\n1. **Continue using UFO² for single-device tasks**\n ```bash\n python -m ufo --task \"Your local task\"\n ```\n\n2. **Use Galaxy only when you need multi-device orchestration**\n ```bash\n python -m galaxy --request \"Your cross-device task\"\n ```\n\n3. **No migration required** — both coexist independently\n\n---\n\n### Option 2: Convert UFO² Instance to Galaxy Device\n\n**Best for:** Leveraging Galaxy's orchestration for all workflows\n\n#### Step 1: Start UFO² as Agent Server\n\n**On each device** (Windows, Linux, etc.), run UFO² server:\n\n```bash\n# Windows Desktop\npython -m ufo.server.app --port 5005\n\n# Linux Workstation \npython -m ufo.server.app --port 5001\n\n# GPU Server\npython -m ufo.server.app --port 5002\n```\n\n**What this does:**\n- Starts WebSocket server on the device\n- Listens for task assignments from Galaxy\n- Uses existing UFO² agents (HostAgent/AppAgent) for local execution\n- Reports results back to ConstellationClient\n\n---\n\n#### Step 2: Configure Galaxy Client\n\nCreate `config/galaxy/devices.yaml` with all your devices (see Configuration section above).\n\n---\n\n#### Step 3: Launch Galaxy Client\n\n```bash\n# Interactive mode\npython -m galaxy --interactive\n\n# Direct request\npython -m galaxy --request \"Clone repo on laptop, build on server, test on Windows\"\n```\n\n**What happens:**\n1. ConstellationAgent decomposes request into DAG\n2. TaskStars assigned to devices based on capabilities\n3. Devices execute tasks using their local UFO² agents\n4. Results aggregated and presented to user\n\n---\n\n### Option 3: Programmatic Migration\n\n**Best for:** Custom workflows, CI/CD integration\n\n#### UFO² API (Before):\n\n```python\nfrom ufo.module.session_pool import SessionFactory, SessionPool\nimport asyncio\n\nasync def main():\n # Create UFO² session on local device\n sessions = SessionFactory().create_session(\n task=\"my_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Create a presentation from data.xlsx\"\n )\n \n # Run session\n pool = SessionPool(sessions)\n await pool.run_all()\n\nasyncio.run(main())\n```\n\n#### Galaxy API (After):\n\n```python\nfrom galaxy import GalaxyClient\nimport asyncio\n\nasync def main():\n # Galaxy session coordinating multiple devices\n client = GalaxyClient(session_name=\"my_workflow\")\n await client.initialize()\n \n result = await client.process_request(\n \"Clone repo on laptop, build on server, test on Windows\"\n )\n \n print(f\"Workflow completed: {result}\")\n await client.shutdown()\n\nasyncio.run(main())\n```\n\n**Key Differences:**\n- Both are **async** (UFO² v2.0+ uses asyncio)\n- UFO²: Uses `SessionFactory` + `SessionPool` pattern\n- Galaxy: Uses `GalaxyClient` for multi-device orchestration\n- Galaxy returns **constellation results** (multi-device)\n- Galaxy requires **device registration** first\n\n---\n\n## 📊 Feature Comparison\n\n### Preserved UFO² Features in Galaxy\n\nWhen running UFO² as a Galaxy device, you **keep all UFO² capabilities**:\n\n| UFO² Feature | Available in Galaxy Device? | Notes |\n|--------------|----------------------------|-------|\n| ✅ Hybrid GUI–API execution | ✅ Yes | Each device uses its native UFO² agent |\n| ✅ Windows UIA/Win32/COM | ✅ Yes | Full OS integration preserved |\n| ✅ MCP server integration | ✅ Yes | Devices can use custom MCP servers |\n| ✅ Continuous learning | ✅ Yes | Each device maintains its own RAG |\n| ✅ Picture-in-Picture | ✅ Yes | Non-disruptive execution on each device |\n| ✅ AppAgent specialization | ✅ Yes | HostAgent manages local AppAgents |\n\n---\n\n### New Galaxy-Only Features\n\n| Feature | Description | Benefit |\n|---------|-------------|---------|\n| **Task Constellation** | DAG-based task decomposition | Complex workflow planning |\n| **Parallel Execution** | Asynchronous multi-device tasks | 3-5x faster for parallelizable work |\n| **Dynamic Adaptation** | Runtime DAG modification | Self-healing workflows |\n| **Device Assignment** | Capability-based task placement | Optimal resource utilization |\n| **Cross-Platform** | Windows + Linux + macOS support | Heterogeneous orchestration |\n| **Event-Driven Coordination** | Observer pattern for task events | Reactive workflow control |\n| **Formal Safety Guarantees** | I1-I3 invariants | Provably correct concurrent execution |\n\n---\n\n## 🛠️ Practical Examples\n\n### Example 1: Simple Local Task\n\n**UFO² (Before):**\n```bash\npython -m ufo --task \"Create a presentation from data.xlsx\"\n```\n\n**Galaxy (After) — Option A: Keep UFO²**\n```bash\n# No change needed — continue using UFO² for local tasks\npython -m ufo --task \"Create a presentation from data.xlsx\"\n```\n\n**Galaxy (After) — Option B: Use Galaxy**\n```bash\n# Galaxy will assign to local Windows device automatically\npython -m galaxy --request \"Create a presentation from data.xlsx on my desktop\"\n```\n\n**When to use which?**\n- Use UFO² if you only have one Windows desktop (simpler)\n- Use Galaxy if you want logging/monitoring features\n\n---\n\n### Example 2: Cross-Device Workflow\n\n**UFO² (Before):**\n```bash\n# ❌ Not possible — UFO² is single-device only\n# You'd need to manually:\n# 1. SSH to server\n# 2. Run build command\n# 3. Copy results back\n# 4. Open locally\n```\n\n**Galaxy (After):**\n```bash\npython -m galaxy --request \\\n \"Clone https://github.com/myrepo on laptop, \\\n build Docker image on gpu_server, \\\n deploy to staging server, \\\n open logs on my Windows desktop\"\n```\n\n**Galaxy automatically:**\n1. Creates 4-task DAG\n2. Assigns tasks to capable devices\n3. Executes in parallel where possible\n4. Streams results back\n\n---\n\n### Example 3: Data Pipeline\n\n**UFO² (Before):**\n```python\n# UFO² requires manual orchestration across multiple steps\nfrom ufo.module.session_pool import SessionFactory, SessionPool\nimport asyncio\n\nasync def main():\n # Step 1: Fetch data (local)\n sessions_1 = SessionFactory().create_session(\n task=\"fetch_data\",\n mode=\"normal\",\n plan=\"\",\n request=\"Download dataset from cloud storage\"\n )\n pool_1 = SessionPool(sessions_1)\n await pool_1.run_all()\n\n # Step 2: Manually transfer to server\n # scp data.csv user@server:/data/\n\n # Step 3: SSH and run processing\n # ssh server \"python process.py\"\n\n # Step 4: Manually copy results back\n # scp server:/output/results.csv .\n\n # Step 5: Visualize locally\n sessions_2 = SessionFactory().create_session(\n task=\"visualize\",\n mode=\"normal\",\n plan=\"\",\n request=\"Create charts from results.csv\"\n )\n pool_2 = SessionPool(sessions_2)\n await pool_2.run_all()\n\nasyncio.run(main())\n```\n\n**Galaxy (After):**\n```python\nimport asyncio\nfrom galaxy import GalaxyClient\n\nasync def main():\n client = GalaxyClient(session_name=\"data_pipeline\")\n await client.initialize()\n \n # Single request — Galaxy handles orchestration\n await client.process_request(\n \"Fetch dataset from cloud to laptop, \"\n \"preprocess on linux_workstation, \"\n \"train model on gpu_server, \"\n \"visualize results on my Windows desktop\"\n )\n \n await client.shutdown()\n\nasyncio.run(main())\n```\n\n**Galaxy automatically:**\n- Creates dependency chain\n- Transfers data between devices\n- Executes pipeline stages in order\n- Handles failures with retries\n\n---\n\n## 🎓 Learning Path\n\n### For UFO² Users\n\n1. **Week 1: Understand Concepts**\n - Read [Galaxy Overview](../galaxy/overview.md)\n - Understand Task Constellation and DAG model\n - Compare with UFO² two-tier hierarchy\n\n2. **Week 2: Hands-On**\n - Set up one Windows device as Galaxy agent\n - Run simple multi-step workflow\n - Compare logs: UFO² vs Galaxy\n\n3. **Week 3: Multi-Device**\n - Add Linux device to pool\n - Create cross-platform workflow\n - Monitor with trajectory reports\n\n4. **Week 4: Advanced**\n - Build custom device capabilities\n - Integrate MCP servers across devices\n - Optimize task assignment logic\n\n---\n\n## 📚 Related Documentation\n\n### Migration Resources\n\n- **[Galaxy Quick Start](./quick_start_galaxy.md)** — Step-by-step Galaxy setup\n- **[UFO² Quick Start](./quick_start_ufo2.md)** — UFO² reference\n- **[Device Configuration](../configuration/system/galaxy_devices.md)** — Device pool setup\n- **[Agent Registration](../galaxy/agent_registration/overview.md)** — How devices join Galaxy\n\n### Architecture Deep Dives\n\n- **[Galaxy Overview](../galaxy/overview.md)** — Constellation architecture\n- **[UFO² Overview](../ufo2/overview.md)** — Desktop AgentOS design\n- **[Constellation Agent](../galaxy/constellation_agent/overview.md)** — DAG orchestration\n- **[Task Constellation](../galaxy/constellation/overview.md)** — DAG structure\n\n### Operational Guides\n\n- **[Trajectory Report](../galaxy/evaluation/trajectory_report.md)** — Execution logs\n- **[Performance Metrics](../galaxy/evaluation/performance_metrics.md)** — Monitoring\n- **[AIP Protocol](../aip/overview.md)** — Device communication\n\n---\n\n## 🤝 Getting Help\n\n### Common Questions\n\n**Q: Can I still use UFO² after migrating to Galaxy?** \nA: Yes! They coexist. Use UFO² for simple local tasks, Galaxy for multi-device workflows.\n\n**Q: Do I need to rewrite my custom agents?** \nA: No. Existing UFO² agents work as-is when running as Galaxy devices.\n\n**Q: Is Galaxy production-ready?** \nA: Galaxy is in active development. UFO² is more mature for mission-critical single-device workflows.\n\n**Q: Can I mix Windows and Linux devices?** \nA: Yes! That's Galaxy's key feature. Each device uses its native UFO² implementation.\n\n**Q: How do I debug failed cross-device workflows?** \nA: Check `logs/galaxy//output.md` for step-by-step execution details and DAG visualizations.\n\n---\n\n## 🚦 Migration Checklist\n\nUse this checklist to track your migration progress:\n\n- [ ] **Understand UFO evolution** (v1 → UFO² → Galaxy)\n- [ ] **Decide migration strategy** (hybrid vs full Galaxy)\n- [ ] **Preserve UFO² config** (`config/ufo/` untouched)\n- [ ] **Create Galaxy config** (`config/galaxy/agent.yaml`, `devices.yaml`)\n- [ ] **Start devices as servers** (each device runs `python -m ufo.server.app --port `)\n- [ ] **Test single-device workflow** (verify connectivity)\n- [ ] **Test multi-device workflow** (cross-platform task)\n- [ ] **Review trajectory reports** (`logs/galaxy/*/output.md`)\n- [ ] **Compare performance** (UFO² vs Galaxy for your use cases)\n- [ ] **Update automation scripts** (if using programmatic API)\n- [ ] **Train team** (share this guide!)\n\n---\n\n**🎉 Congratulations!** You're now ready to leverage the full power of UFO³ Galaxy's multi-device orchestration while preserving your existing UFO² workflows.\n\nFor questions or issues, please open an issue on [GitHub](https://github.com/microsoft/UFO) or check the [documentation](https://microsoft.github.io/UFO/).\n"
},
{
"path": "documents/docs/getting_started/more_guidance.md",
"content": "# More Guidance\n\nThis page provides additional guidance and resources for different user types and use cases.\n\n---\n\n## 🎯 For End Users\n\nIf you want to use UFO³ to automate your tasks on Windows, Linux, or across multiple devices, here's your learning path:\n\n### 1. Getting Started (5-10 minutes)\n\nChoose your path based on your needs:\n\n| Your Goal | Start Here | Time |\n|-----------|-----------|------|\n| **Automate Windows desktop tasks** | [UFO² Quick Start](quick_start_ufo2.md) | 5 min |\n| **Manage Linux servers** | [Linux Quick Start](quick_start_linux.md) | 10 min |\n| **Orchestrate multiple devices** | [Galaxy Quick Start](quick_start_galaxy.md) | 10 min |\n\n### 2. Configure Your Environment (10-20 minutes)\n\nAfter installation, customize UFO³ to your needs:\n\n**Essential Configuration:**\n\n- **[Agent Configuration](../configuration/system/agents_config.md)** - Set up LLM API keys (OpenAI, Azure, Gemini, Claude, etc.)\n- **[System Configuration](../configuration/system/system_config.md)** - Adjust runtime settings (step limits, timeouts, logging)\n\n**Optional Enhancements:**\n\n- **[RAG Configuration](../configuration/system/rag_config.md)** - Add external knowledge sources:\n - Offline help documents\n - Bing search integration\n - Experience learning from past tasks\n - User demonstrations\n- **[MCP Configuration](../configuration/system/mcp_reference.md)** - Enable tool servers for:\n - Better Office automation\n - Linux command execution\n - Custom tool integration\n\n> **💡 Configuration Tip:** Start with default settings and adjust only what you need. See [Configuration Overview](../configuration/system/overview.md) for the big picture.\n\n### 3. Learn Core Features (20-30 minutes)\n\n**For UFO² Users (Windows Desktop Automation):**\n\n| Feature | Documentation | What It Does |\n|---------|---------------|--------------|\n| **Hybrid GUI-API Execution** | [Hybrid Actions](../ufo2/core_features/hybrid_actions.md) | Combines UI automation with native API calls for faster, more reliable execution |\n| **Knowledge Substrate** | [Knowledge Overview](../ufo2/core_features/knowledge_substrate/overview.md) | Augments agents with external knowledge (docs, search, experience) |\n| **MCP Integration** | [MCP Overview](../mcp/overview.md) | Extends capabilities with custom tools and Office APIs |\n\n**For Galaxy Users (Multi-Device Orchestration):**\n\n| Feature | Documentation | What It Does |\n|---------|---------------|--------------|\n| **Task Constellation** | [Constellation Overview](../galaxy/constellation_orchestrator/overview.md) | Decomposes tasks into parallel DAGs across devices |\n| **Device Capabilities** | [Galaxy Devices Config](../configuration/system/galaxy_devices.md) | Routes tasks based on device capabilities and metadata |\n| **Asynchronous Execution** | [Constellation Overview](../galaxy/constellation/overview.md) | Executes subtasks in parallel for faster completion |\n| **Agent Interaction Protocol** | [AIP Overview](../aip/overview.md) | Enables persistent WebSocket communication between devices |\n\n### 4. Troubleshooting & Support\n\n**When Things Go Wrong:**\n\n1. **Check the [FAQ](../faq.md)** - Common issues and solutions\n2. **Review logs** - Located in `logs//`:\n ```\n logs/my-task-2025-11-11/\n ├── request.log # Request logs\n ├── response.log # Response logs\n ├── action_step*.png # Screenshots at each step\n └── action_step*_annotated.png # Annotated screenshots\n ```\n3. **Validate configuration:**\n ```bash\n python -m ufo.tools.validate_config ufo --show-config\n ```\n4. **Enable debug logging:**\n ```yaml\n # config/ufo/system.yaml\n LOG_LEVEL: \"DEBUG\"\n ```\n\n**Get Help:**\n\n- **[GitHub Discussions](https://github.com/microsoft/UFO/discussions)** - Ask questions, share tips\n- **[GitHub Issues](https://github.com/microsoft/UFO/issues)** - Report bugs, request features\n- **Email:** ufo-agent@microsoft.com\n\n---\n\n## 👨💻 For Developers\n\nIf you want to contribute to UFO³ or build extensions, here's your development guide:\n\n### 1. Understand the Architecture (30-60 minutes)\n\n**Start with the big picture:**\n\n- **[Project Structure](../project_directory_structure.md)** - Codebase organization and component roles\n- **[Configuration Architecture](../configuration/system/overview.md)** - New modular config system design\n\n**Deep dive into core components:**\n\n| Component | Documentation | What to Learn |\n|-----------|---------------|---------------|\n| **Session** | [Session Module](../infrastructure/modules/session.md) | Task lifecycle management, state tracking |\n| **Round** | [Round Module](../infrastructure/modules/round.md) | Single agent reasoning cycle |\n| **HostAgent** | [HostAgent](../ufo2/host_agent/overview.md) | High-level task planning and app selection |\n| **AppAgent** | [AppAgent](../ufo2/app_agent/overview.md) | Low-level action execution |\n| **ConstellationAgent** | [ConstellationAgent](../galaxy/constellation_agent/overview.md) | Multi-device task orchestration |\n\n### 2. Set Up Development Environment (15-30 minutes)\n\n**Installation:**\n\n```bash\n# Clone the repository\ngit clone https://github.com/microsoft/UFO.git\ncd UFO\n\n# Create development environment\nconda create -n ufo-dev python=3.10\nconda activate ufo-dev\n\n# Install dependencies (including dev tools)\npip install -r requirements.txt\npip install pytest pytest-cov black flake8 # Testing & linting\n```\n\n**Configuration:**\n\n```bash\n# Create config files from templates\ncp config/ufo/agents.yaml.template config/ufo/agents.yaml\ncp config/galaxy/agent.yaml.template config/galaxy/agent.yaml\n\n# Edit with your development API keys\n# (Consider using lower-cost models for testing)\n```\n\n### 3. Explore the Codebase (1-2 hours)\n\n**Key Directories:**\n\n```\nUFO/\n├── ufo/ # Core UFO² implementation\n│ ├── agents/ # HostAgent, AppAgent\n│ ├── automator/ # UI automation engines\n│ ├── prompter/ # Prompt management\n│ └── module/ # Core modules (Session, Round)\n├── galaxy/ # Galaxy orchestration framework\n│ ├── agents/ # ConstellationAgent\n│ ├── constellation/ # DAG orchestration\n│ └── core/ # Core Galaxy infrastructure\n├── aip/ # Agent Interaction Protocol\n│ ├── protocol/ # Message definitions\n│ └── transport/ # WebSocket transport\n├── ufo/client/ # Device agents (Windows, Linux)\n│ ├── client.py # Generic client\n│ └── mcp/ # MCP integration\n├── ufo/server/ # Device agent server\n│ └── app.py # FastAPI server\n└── config/ # Configuration system\n ├── ufo/ # UFO² configs\n └── galaxy/ # Galaxy configs\n```\n\n**Entry Points:**\n\n- **UFO² Main:** `ufo/__main__.py`\n- **Galaxy Main:** `galaxy/__main__.py`\n- **Server:** `ufo/server/app.py`\n- **Client:** `ufo/client/client.py`\n\n### 4. Development Workflows\n\n#### Adding a New Feature\n\n1. **Identify the component** to modify (Agent, Module, Automator, etc.)\n2. **Read existing code** in that component\n3. **Check related tests** in `tests/` directory\n4. **Implement your feature** following existing patterns\n5. **Add tests** for your feature\n6. **Update documentation** if needed\n\n#### Extending Configuration\n\nSee **[Extending Configuration](../configuration/system/extending.md)** for:\n- Adding custom fields\n- Creating new config modules\n- Environment-specific overrides\n- Plugin configuration patterns\n\n#### Creating Custom MCP Servers\n\nSee **[Creating MCP Servers Tutorial](../tutorials/creating_mcp_servers.md)** for:\n- MCP server architecture\n- Tool definition and registration\n- HTTP vs. local vs. stdio servers\n- Integration with UFO³\n\n### 5. Testing & Debugging\n\n**Run Tests:**\n\n```bash\n# Run all tests\npytest\n\n# Run specific test file\npytest tests/config/test_config_system.py\n\n# Run with coverage\npytest --cov=ufo --cov-report=html\n```\n\n**Debug Logging:**\n\n```python\n# Add debug logs to your code\nimport logging\nlogger = logging.getLogger(__name__)\n\nlogger.debug(\"Debug message with context: %s\", variable)\nlogger.info(\"Informational message\")\nlogger.warning(\"Warning message\")\nlogger.error(\"Error message\")\n```\n\n**Interactive Debugging:**\n\n```python\n# Add breakpoint in code\nimport pdb; pdb.set_trace()\n\n# Or use VS Code debugger with launch.json\n```\n\n### 6. Code Style & Best Practices\n\n**Formatting:**\n\n```bash\n# Auto-format with black\nblack ufo/ galaxy/\n\n# Check style with flake8\nflake8 ufo/ galaxy/\n```\n\n**Best Practices:**\n\n- ✅ Use type hints: `def process(data: Dict[str, Any]) -> Optional[str]:`\n- ✅ Write docstrings for public functions\n- ✅ Follow existing code patterns\n- ✅ Add comments for complex logic\n- ✅ Keep functions focused and modular\n- ✅ Handle errors gracefully\n- ✅ Write tests for new features\n\n**Configuration Best Practices:**\n\n- ✅ Use typed config access: `config.system.max_step`\n- ✅ Provide `.template` files for sensitive configs\n- ✅ Document custom fields in YAML comments\n- ✅ Use environment variables for secrets: `${OPENAI_API_KEY}`\n- ✅ Validate configurations early: `ConfigValidator.validate()`\n\n### 7. Contributing Guidelines\n\n**Before Submitting a PR:**\n\n1. **Test your changes** thoroughly\n2. **Update documentation** if needed\n3. **Follow code style** (black + flake8)\n4. **Write clear commit messages**\n5. **Reference related issues** in PR description\n\n**PR Template:**\n\n```markdown\n## Description\nBrief description of changes\n\n## Type of Change\n- [ ] Bug fix\n- [ ] New feature\n- [ ] Documentation update\n- [ ] Refactoring\n\n## Testing\n- [ ] Added tests for new functionality\n- [ ] All tests pass locally\n- [ ] Manual testing completed\n\n## Checklist\n- [ ] Code follows project style\n- [ ] Documentation updated\n- [ ] No breaking changes (or documented)\n```\n\n### 8. Advanced Topics\n\n**For Deep Customization:**\n\n- **[Prompt Engineering](../ufo2/prompts/overview.md)** - Customize agent prompts\n- **[State Management](../galaxy/constellation/overview.md)** - Constellation state machine internals\n- **[Protocol Extensions](../aip/messages.md)** - Extend AIP message types\n- **[Custom Automators](../ufo2/core_features/control_detection/overview.md)** - Implement new automation backends\n\n---\n\n## 🎓 Learning Paths\n\n### Path 1: Basic User → Power User\n\n1. ✅ Complete quick start for your platform\n2. ✅ Run 5-10 simple automation tasks\n3. ✅ Configure RAG for your organization's docs\n4. ✅ Enable MCP for better Office automation\n5. ✅ Set up experience learning for common tasks\n6. ✅ Create custom device configurations (Galaxy)\n\n**Time Investment:** 2-4 hours \n**Outcome:** Efficient automation of daily tasks\n\n### Path 2: Power User → Developer\n\n1. ✅ Understand project structure and architecture\n2. ✅ Read Session and Round module code\n3. ✅ Create a custom MCP server\n4. ✅ Add custom metadata to device configs\n5. ✅ Contribute documentation improvements\n6. ✅ Submit your first bug fix PR\n\n**Time Investment:** 10-20 hours \n**Outcome:** Ability to extend and customize UFO³\n\n### Path 3: Developer → Core Contributor\n\n1. ✅ Deep dive into agent implementations\n2. ✅ Understand Galaxy orchestration internals\n3. ✅ Study AIP protocol and transport layer\n4. ✅ Implement a new agent capability\n5. ✅ Add support for a new LLM provider\n6. ✅ Contribute major features or refactorings\n\n**Time Investment:** 40+ hours \n**Outcome:** Core contributor to UFO³ project\n\n---\n\n## 📚 Additional Resources\n\n### Documentation Hubs\n\n| Topic | Link | Description |\n|-------|------|-------------|\n| **Getting Started** | [Getting Started Index](../index.md#getting-started) | All quick start guides |\n| **Configuration** | [Configuration Overview](../configuration/system/overview.md) | Complete config system documentation |\n| **Architecture** | [Galaxy Overview](../galaxy/overview.md), [UFO² Overview](../ufo2/overview.md) | System architecture and design |\n| **API Reference** | [Agent APIs](../infrastructure/agents/overview.md) | Agent interfaces and APIs |\n| **Tutorials** | [Creating Device Agents](../tutorials/creating_device_agent/index.md) | Step-by-step guides |\n\n### Community Resources\n\n- **[GitHub Repository](https://github.com/microsoft/UFO)** - Source code and releases\n- **[GitHub Discussions](https://github.com/microsoft/UFO/discussions)** - Q&A and community\n- **[GitHub Issues](https://github.com/microsoft/UFO/issues)** - Bug reports and features\n- **[Project Website](https://microsoft.github.io/UFO/)** - Official website\n\n### Research Papers\n\n- **UFO v1** (Feb 2024): [A UI-Focused Agent for Windows OS Interaction](https://arxiv.org/abs/2402.07939)\n- **UFO² v2** (Apr 2025): [A Windows Agent for Seamless OS Interaction](https://arxiv.org/abs/2504.14603)\n- **UFO³ Galaxy** (Nov 2025): UFO³: Weaving the Digital Agent Galaxy *(Coming Soon)*\n\n---\n\n## 🆘 Need More Help?\n\n- **Can't find what you're looking for?** Check the [FAQ](../faq.md)\n- **Still stuck?** Ask on [GitHub Discussions](https://github.com/microsoft/UFO/discussions)\n- **Found a bug?** Open an issue on [GitHub Issues](https://github.com/microsoft/UFO/issues)\n- **Want to contribute?** Read the [Contributing Guidelines](https://github.com/microsoft/UFO/blob/main/CONTRIBUTING.md)\n\n**Happy automating!** 🚀\n"
},
{
"path": "documents/docs/getting_started/quick_start_galaxy.md",
"content": "# Quick Start Guide - UFO³ Galaxy\n\nWelcome to **UFO³ Galaxy** – the Multi-Device AgentOS! This guide will help you orchestrate complex cross-platform workflows across multiple devices in just a few steps.\n\n**What is UFO³ Galaxy?**\n\nUFO³ Galaxy is a multi-tier orchestration framework that coordinates distributed agents across Windows and Linux devices. It enables complex workflows that span multiple machines, combining desktop automation, server operations, and heterogeneous device capabilities into unified task execution.\n\n---\n\n## 🛠️ Step 1: Installation\n\n### Requirements\n\n- **Python** >= 3.10\n- **Windows OS** >= 10 (for Windows agents)\n- **Linux** (for Linux agents)\n- **Git** (for cloning the repository)\n- **Network connectivity** between all devices\n\n### Installation Steps\n\n```powershell\n# [Optional] Create conda environment\nconda create -n ufo python=3.10\nconda activate ufo\n\n# Clone the repository\ngit clone https://github.com/microsoft/UFO.git\ncd UFO\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n> **💡 Tip:** If you want to use Qwen as your LLM, uncomment the related libraries in `requirements.txt` before installing.\n\n---\n\n## ⚙️ Step 2: Configure ConstellationAgent LLM\n\nUFO³ Galaxy uses a **ConstellationAgent** that orchestrates all device agents. You need to configure its LLM settings.\n\n### Configure Constellation Agent\n\n```powershell\n# Copy template to create constellation agent config\ncopy config\\galaxy\\agent.yaml.template config\\galaxy\\agent.yaml\nnotepad config\\galaxy\\agent.yaml # Edit your LLM API credentials\n```\n\n**Configuration File Location:**\n```\nconfig/galaxy/\n├── agent.yaml.template # Template - COPY THIS\n├── agent.yaml # Your config with API keys (DO NOT commit)\n└── devices.yaml # Device pool configuration (Step 4)\n```\n\n### LLM Configuration Examples\n\n#### Azure OpenAI Configuration\n\n**Edit `config/galaxy/agent.yaml`:**\n\n```yaml\nCONSTELLATION_AGENT:\n REASONING_MODEL: false\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o\"\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n```\n\n> **ℹ️ More LLM Options:** Galaxy supports various LLM providers including Qwen, Gemini, Claude, DeepSeek, and more. See the [Model Configuration Guide](../configuration/models/overview.md) for complete details.\n\n---\n \n # Prompt configurations (use defaults)\n CONSTELLATION_CREATION_PROMPT: \"galaxy/prompts/constellation/share/constellation_creation.yaml\"\n CONSTELLATION_EDITING_PROMPT: \"galaxy/prompts/constellation/share/constellation_editing.yaml\"\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_creation_example.yaml\"\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_editing_example.yaml\"\n```\n\n#### OpenAI Configuration\n\n```yaml\nCONSTELLATION_AGENT:\n REASONING_MODEL: false\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-4o\"\n \n # Prompt configurations (use defaults)\n CONSTELLATION_CREATION_PROMPT: \"galaxy/prompts/constellation/share/constellation_creation.yaml\"\n CONSTELLATION_EDITING_PROMPT: \"galaxy/prompts/constellation/share/constellation_editing.yaml\"\n CONSTELLATION_CREATION_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_creation_example.yaml\"\n CONSTELLATION_EDITING_EXAMPLE_PROMPT: \"galaxy/prompts/constellation/examples/constellation_editing_example.yaml\"\n```\n\n!!!info \"More LLM Options\"\n Galaxy supports various LLM providers including **Qwen**, **Gemini**, **Claude**, **DeepSeek**, and more. See the **[Model Configuration Guide](../configuration/models/overview.md)** for complete details.\n\n---\n\n## 🖥️ Step 3: Set Up Device Agents\n\nGalaxy orchestrates **device agents** that execute tasks on individual machines. You need to start the appropriate device agents based on your needs.\n\n### Supported Device Agents\n\n| Device Agent | Platform | Documentation | Use Cases |\n|--------------|----------|---------------|-----------|\n| **WindowsAgent (UFO²)** | Windows 10/11 | [UFO² as Galaxy Device](../ufo2/as_galaxy_device.md) | Desktop automation, Office apps, GUI operations |\n| **LinuxAgent** | Linux | [Linux as Galaxy Device](../linux/as_galaxy_device.md) | Server management, CLI operations, log analysis |\n| **MobileAgent** | Android | [Mobile as Galaxy Device](../mobile/as_galaxy_device.md) | Mobile app automation, UI testing, device control |\n\n> **💡 Choose Your Devices:** You can use any combination of Windows, Linux, and Mobile agents. Galaxy will intelligently route tasks based on device capabilities.\n\n### Quick Setup Overview\n\nFor each device agent you want to use, you need to:\n\n1. **Start the Device Agent Server** (manages tasks)\n2. **Start the Device Agent Client** (executes commands)\n3. **Start MCP Services** (provides automation tools, if needed)\n\n**Detailed Setup Instructions:**\n\n- **For Windows devices (UFO²):** See [UFO² as Galaxy Device](../ufo2/as_galaxy_device.md) for complete step-by-step instructions.\n- **For Linux devices:** See [Linux as Galaxy Device](../linux/as_galaxy_device.md) for complete step-by-step instructions.\n- **For Mobile devices:** See [Mobile as Galaxy Device](../mobile/as_galaxy_device.md) for complete step-by-step instructions.\n\n### Example: Quick Windows Device Setup\n\n**On your Windows machine:**\n\n```powershell\n# Terminal 1: Start UFO² Server\npython -m ufo.server.app --port 5000\n\n# Terminal 2: Start UFO² Client (connect to server)\npython -m ufo.client.client `\n --ws `\n --ws-server ws://localhost:5000/ws `\n --client-id windows_device_1 `\n --platform windows\n```\n\n> **💡 Important:** Always include `--platform windows` for Windows devices and `--platform linux` for Linux devices!\n\n### Example: Quick Linux Device Setup\n\n**On your Linux machine:**\n\n```bash\n# Terminal 1: Start Device Agent Server\npython -m ufo.server.app --port 5001\n\n# Terminal 2: Start Linux Client (connect to server)\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id linux_device_1 \\\n --platform linux\n\n# Terminal 3: Start HTTP MCP Server (for Linux tools)\npython -m ufo.client.mcp.http_servers.linux_mcp_server\n```\n\n> **💡 Note:** For detailed Mobile Agent setup with ADB and Android device configuration, see [Mobile Quick Start](quick_start_mobile.md).\n\n---\n\n## 🔌 Step 4: Configure Device Pool\n\nAfter starting your device agents, register them in Galaxy's device pool configuration.\n\n### Option 1: Add Devices via Configuration File\n\n### Edit Device Configuration\n\n```powershell\nnotepad config\\galaxy\\devices.yaml\n```\n\n### Example Device Pool Configuration\n\n```yaml\n# Device Configuration for Galaxy\n# Each device agent must be registered here\n\ndevices:\n # Windows Device (UFO²)\n - device_id: \"windows_device_1\" # Must match --client-id\n server_url: \"ws://localhost:5000/ws\" # Must match server WebSocket URL\n os: \"windows\"\n capabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"excel\"\n - \"word\"\n - \"outlook\"\n - \"email\"\n - \"web_browsing\"\n metadata:\n os: \"windows\"\n version: \"11\"\n performance: \"high\"\n installed_apps:\n - \"Microsoft Excel\"\n - \"Microsoft Word\"\n - \"Microsoft Outlook\"\n - \"Google Chrome\"\n description: \"Primary Windows desktop for office automation\"\n auto_connect: true\n max_retries: 5\n\n # Linux Device\n - device_id: \"linux_device_1\" # Must match --client-id\n server_url: \"ws://localhost:5001/ws\" # Must match server WebSocket URL\n os: \"linux\"\n capabilities:\n - \"server_management\"\n - \"log_analysis\"\n - \"file_operations\"\n - \"database_operations\"\n metadata:\n os: \"linux\"\n performance: \"medium\"\n logs_file_path: \"/var/log/myapp/app.log\"\n dev_path: \"/home/user/projects/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n description: \"Development server for backend operations\"\n auto_connect: true\n max_retries: 5\n\n # Mobile Device (Android)\n - device_id: \"mobile_phone_1\" # Must match --client-id\n server_url: \"ws://localhost:5001/ws\" # Must match server WebSocket URL\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"ui_automation\"\n - \"messaging\"\n - \"camera\"\n - \"location\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"13\"\n screen_size: \"1080x2400\"\n installed_apps:\n - \"com.android.chrome\"\n - \"com.google.android.apps.maps\"\n - \"com.whatsapp\"\n description: \"Android phone for mobile automation and testing\"\n auto_connect: true\n max_retries: 5\n```\n\n> **⚠️ Critical:** IDs and URLs must match exactly:\n> \n> - `device_id` must exactly match the `--client-id` flag\n> - `server_url` must exactly match the server WebSocket URL\n> - Otherwise, Galaxy cannot control the device!\n\n**Complete Configuration Guide:** For detailed information about all configuration options, capabilities, and metadata, see [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md).\n\n### Option 2: Add Devices via WebUI (When Using --webui Mode)\n\nIf you start Galaxy with the `--webui` flag (see Step 5), you can add new device agents directly through the web interface without editing configuration files.\n\n**Steps to Add Device via WebUI:**\n\n1. **Launch Galaxy with WebUI** (as shown in Step 5):\n ```powershell\n python -m galaxy --webui\n ```\n\n2. **Click the \"+\" button** in the top-right corner of the Device Agent panel (left sidebar)\n\n3. **Fill in the device information** in the Add Device Modal:\n\n
\n \n
➕ Add Device Modal - Register new device agents through the WebUI
\n
\n\n**Required Fields:**\n- **Device ID**: Unique identifier (must match `--client-id` in device agent)\n- **Server URL**: WebSocket endpoint (e.g., `ws://localhost:5000/ws`)\n- **Operating System**: Select Windows, Linux, macOS, or enter custom OS\n- **Capabilities**: Add at least one capability (e.g., `excel`, `outlook`, `log_analysis`)\n\n**Optional Fields:**\n- **Auto-connect**: Enable to automatically connect after registration (default: enabled)\n- **Max Retries**: Maximum connection attempts (default: 5)\n- **Metadata**: Add custom key-value pairs (e.g., `region: us-east-1`)\n\n**Benefits of WebUI Device Management:**\n- ✅ No need to manually edit YAML files\n- ✅ Real-time validation of device ID uniqueness\n- ✅ Automatic connection after registration\n- ✅ Immediate visual feedback on device status\n- ✅ Form validation prevents configuration errors\n\n**After Adding:**\nThe device will be:\n1. Saved to `config/galaxy/devices.yaml` automatically\n2. Registered with Galaxy's Device Manager\n3. Connected automatically (if auto-connect is enabled)\n4. Displayed in the Device Agent panel with real-time status\n\n> **💡 Tip:** You can add devices while Galaxy is running! No need to restart the server.\n\n---\n\n## 🎉 Step 5: Start UFO³ Galaxy\n\nWith all device agents running and configured, you can now launch Galaxy!\n\n### Pre-Launch Checklist\n\nBefore starting Galaxy, ensure:\n\n1. ✅ All Device Agent Servers are running\n2. ✅ All Device Agent Clients are connected\n3. ✅ MCP Services are running (for Linux devices)\n4. ✅ LLM configured in `config/galaxy/agent.yaml`\n5. ✅ Devices configured in `config/galaxy/devices.yaml`\n6. ✅ Network connectivity between all components\n\n### 🎨 Launch Galaxy - WebUI Mode (Recommended)\n\nStart Galaxy with an interactive web interface for real-time constellation visualization and monitoring:\n\n```powershell\n# Assume you are in the cloned UFO folder\npython -m galaxy --webui\n```\n\nThis will start the Galaxy server with WebUI and automatically open your browser to the interactive interface:\n\n
\n \n
🎨 Galaxy WebUI - Interactive constellation visualization and chat interface
\n
\n\n**WebUI Features:**\n\n- 🗣️ **Chat Interface**: Submit requests and interact with ConstellationAgent in real-time\n- 📊 **Live DAG Visualization**: Watch task constellation formation and execution\n- 🎯 **Task Status Tracking**: Monitor each TaskStar's progress and completion\n- 🔄 **Dynamic Updates**: See constellation evolution as tasks complete\n- 📱 **Responsive Design**: Works on desktop and tablet devices\n\n**Default URL:** `http://localhost:8000` (automatically finds next available port if 8000 is occupied)\n\n---\n\n### 💬 Launch Galaxy - Interactive Terminal Mode\n\nStart Galaxy in interactive mode where you can enter requests dynamically:\n\n```powershell\n# Assume you are in the cloned UFO folder\npython -m galaxy --interactive\n```\n\n**Expected Output:**\n\n```\n🌌 Welcome to UFO³ Galaxy Framework\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\nMulti-Device AI Orchestration System\n\n📡 Initializing Galaxy...\n✅ ConstellationAgent initialized\n✅ Connected to device: windows_device_1 (windows)\n✅ Connected to device: linux_device_1 (linux)\n\n🌟 Galaxy Ready - 2 devices online\n\nPlease enter your request 🛸:\n```\n\n---\n\n### ⚡ Launch Galaxy - Direct Request Mode\n\nInvoke Galaxy with a specific request directly:\n\n```powershell\npython -m galaxy --request \"Your task description here\"\n```\n\n**Example:**\n\n```powershell\npython -m galaxy --request \"Generate a sales report from the database and create an Excel dashboard\"\n```\n\n---\n\n### 🎬 Launch Galaxy - Demo Mode\n\nRun Galaxy in demo mode to see example workflows:\n\n```powershell\npython -m galaxy --demo\n```\n\n---\n\n## 🎯 Step 6: Try Your First Multi-Device Workflow\n\n### Example 1: Simple Cross-Platform Task\n\n**User Request:**\n> \"Check the server logs for errors and email me a summary\"\n\n**Galaxy orchestrates:**\n\n1. **Linux Device**: Analyze server logs for error patterns\n2. **Windows Device**: Open Outlook, create email with log summary\n3. **Windows Device**: Send email\n\n**How to run:**\n\n```powershell\npython -m galaxy --request \"Check the server logs for errors and email me a summary\"\n```\n\n### Example 2: Data Processing Pipeline\n\n**User Request:**\n> \"Export sales data from the database, create an Excel report with charts, and email it to the team\"\n\n**Galaxy orchestrates:**\n\n1. **Linux Device**: Query database, export CSV\n2. **Windows Device**: Open Excel, import CSV, create charts\n3. **Windows Device**: Open Outlook, attach Excel file, send email\n\n**How to run:**\n\n```powershell\npython -m galaxy --request \"Export sales data from the database, create an Excel report with charts, and email it to the team\"\n```\n\n### Example 3: Multi-Server Monitoring\n\n**User Request:**\n> \"Check all servers for disk usage and alert if any are above 80%\"\n\n**Galaxy orchestrates:**\n\n1. **Linux Device 1**: Check disk usage on server 1\n2. **Linux Device 2**: Check disk usage on server 2\n3. **Galaxy**: Aggregate results, check thresholds\n4. **Windows Device**: Send alert email if needed\n\n---\n\n## 📔 Step 7: Understanding Device Routing\n\nGalaxy uses **capability-based routing** to intelligently assign tasks to appropriate devices.\n\n### How Galaxy Selects Devices\n\n| Factor | Description | Example |\n|--------|-------------|---------|\n| **Capabilities** | Matches task requirements | `\"excel\"` → Windows device with Excel |\n| **OS Requirement** | Platform-specific tasks | Linux commands → Linux device |\n| **Metadata** | Device-specific context | Email task → device with Outlook |\n| **Status** | Online and healthy devices only | Skips offline devices |\n\n### Example Task Decomposition\n\n**User Request:**\n> \"Prepare monthly reports and distribute to team\"\n\n**Galaxy Decomposition:**\n\n```yaml\nSubtask 1:\n Description: \"Extract monthly data from database\"\n Target Device: linux_device_1\n Reason: Has \"database_operations\" capability\n\nSubtask 2:\n Description: \"Create Excel report with visualizations\"\n Target Device: windows_device_1\n Reason: Has \"excel\" capability\n\nSubtask 3:\n Description: \"Email reports to distribution list\"\n Target Device: windows_device_1\n Reason: Has \"email\" and \"outlook\" capabilities\n```\n\n---\n\n## 🔄 Step 8: Execution Logs\n\nGalaxy automatically saves execution logs, task graphs, and device traces for debugging and analysis.\n\n**Log Location:**\n\n```\n./logs//\n```\n\n**Log Contents:**\n\n| File/Folder | Description |\n|-------------|-------------|\n| `constellation/` | DAG visualization and task decomposition |\n| `device_logs/` | Individual device execution logs |\n| `screenshots/` | Screenshots from Windows devices (if enabled) |\n| `task_results/` | Task execution results |\n| `request_response.log` | Complete LLM request/response logs |\n\n> **Analyzing Logs:** Use the logs to debug task routing, identify bottlenecks, replay execution flow, and analyze orchestration decisions.\n\n---\n\n## 🔧 Advanced Configuration\n\n### Custom Session Name\n\n```powershell\npython -m galaxy --request \"Your task\" --session-name \"my_project\"\n```\n\n### Custom Output Directory\n\n```powershell\npython -m galaxy --request \"Your task\" --output-dir \"./custom_results\"\n```\n\n### Debug Mode\n\n```powershell\npython -m galaxy --interactive --log-level DEBUG\n```\n\n### Limit Maximum Rounds\n\n```powershell\npython -m galaxy --interactive --max-rounds 20\n```\n\n---\n\n## ❓ Troubleshooting\n\n### Issue 1: Device Not Appearing in Galaxy\n\n**Error:** Device not found in configuration\n\n```log\nERROR - Device 'windows_device_1' not found in configuration\n```\n\n**Solutions:**\n\n1. Verify `devices.yaml` configuration:\n ```powershell\n notepad config\\galaxy\\devices.yaml\n ```\n\n2. Check device ID matches:\n - In `devices.yaml`: `device_id: \"windows_device_1\"`\n - In client command: `--client-id windows_device_1`\n\n3. Check server URL matches:\n - In `devices.yaml`: `server_url: \"ws://localhost:5000/ws\"`\n - In client command: `--ws-server ws://localhost:5000/ws`\n\n### Issue 2: Device Agent Not Connecting\n\n**Error:** Connection refused\n\n```log\nERROR - [WS] Failed to connect to ws://localhost:5000/ws\nConnection refused\n```\n\n**Solutions:**\n\n1. Verify server is running:\n ```powershell\n curl http://localhost:5000/api/health\n ```\n\n2. Check port number is correct:\n - Server: `--port 5000`\n - Client: `ws://localhost:5000/ws`\n\n3. Ensure platform flag is set:\n ```powershell\n # For Windows devices\n --platform windows\n \n # For Linux devices\n --platform linux\n ```\n\n### Issue 3: Galaxy Cannot Find Constellation Agent Config\n\n**Error:** Configuration file not found\n\n```log\nERROR - Cannot find config/galaxy/agent.yaml\n```\n\n**Solution:**\n```powershell\n# Copy template to create configuration file\ncopy config\\galaxy\\agent.yaml.template config\\galaxy\\agent.yaml\n\n# Edit with your LLM credentials\nnotepad config\\galaxy\\agent.yaml\n```\n\n### Issue 4: Task Not Routed to Expected Device\n\n**Issue:** Wrong device selected for task\n\n**Diagnosis:** Check device capabilities in `devices.yaml`:\n\n```yaml\ncapabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"excel\" # Required for Excel tasks\n - \"outlook\" # Required for email tasks\n```\n\n**Solution:** Add appropriate capabilities to your device configuration.\n\n---\n\n## 📚 Additional Resources\n\n### Core Documentation\n\n**Architecture & Concepts:**\n\n- [Galaxy Overview](../galaxy/overview.md) - System architecture and design principles\n- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Task orchestration and DAG management\n- [Agent Interaction Protocol (AIP)](../aip/overview.md) - Communication substrate\n\n### Device Agent Setup\n\n**Device Agent Guides:**\n\n- [UFO² as Galaxy Device](../ufo2/as_galaxy_device.md) - Complete Windows device setup\n- [Linux as Galaxy Device](../linux/as_galaxy_device.md) - Complete Linux device setup\n- [Mobile as Galaxy Device](../mobile/as_galaxy_device.md) - Complete Android device setup\n- [UFO² Overview](../ufo2/overview.md) - Windows desktop automation capabilities\n- [Linux Agent Overview](../linux/overview.md) - Linux server automation capabilities\n- [Mobile Agent Overview](../mobile/overview.md) - Android mobile automation capabilities\n\n### Configuration\n\n**Configuration Guides:**\n\n- [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md) - Complete device pool configuration\n- [Galaxy Constellation Configuration](../configuration/system/galaxy_constellation.md) - Runtime settings\n- [Agents Configuration](../configuration/system/agents_config.md) - LLM settings for all agents\n- [Model Configuration](../configuration/models/overview.md) - Supported LLM providers\n\n### Advanced Features\n\n**Advanced Topics:**\n\n- [Task Constellation](../galaxy/constellation/task_constellation.md) - DAG-based task planning\n- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Multi-device orchestration\n- [Device Registry](../galaxy/agent_registration/device_registry.md) - Device management\n- [Agent Profiles](../galaxy/agent_registration/agent_profile.md) - Multi-source profiling\n\n---\n\n## ❓ Getting Help\n\n- 📖 **Documentation**: [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)\n- 🐛 **GitHub Issues**: [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues) (preferred)\n- 📧 **Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)\n\n---\n\n## 🎯 Next Steps\n\nNow that Galaxy is set up, explore these guides to unlock its full potential:\n\n1. **[Add More Devices](../configuration/system/galaxy_devices.md)** - Expand your device pool\n2. **[Configure Capabilities](../configuration/system/galaxy_devices.md)** - Optimize task routing\n3. **[Constellation Agent](../galaxy/constellation_agent/overview.md)** - Deep dive into orchestration agent\n4. **[Advanced Orchestration](../galaxy/constellation_orchestrator/overview.md)** - Deep dive into DAG planning\n\nHappy orchestrating with UFO³ Galaxy! 🌌🚀\n"
},
{
"path": "documents/docs/getting_started/quick_start_linux.md",
"content": "# ⚡ Quick Start: Linux Agent\n\nGet your Linux device running as a UFO³ device agent in 5 minutes. This guide walks you through server/client configuration and MCP service initialization.\n\n---\n\n## 📋 Prerequisites\n\nBefore you begin, ensure you have:\n\n- **Python 3.10+** installed on both server and client machines\n- **UFO repository** cloned\n- **Network connectivity** between server and client machines\n- **Linux machine** for task execution (client)\n- **Terminal access** (bash, ssh, etc.)\n- **LLM configured** in `config/ufo/agents.yaml` (same as AppAgent)\n\n| Component | Minimum Version | Verification Command |\n|-----------|----------------|---------------------|\n| Python | 3.10 | `python3 --version` |\n| Git | 2.0+ | `git --version` |\n| Network | N/A | `ping ` |\n| LLM API Key | N/A | Check `config/ufo/agents.yaml` |\n\n> **⚠️ LLM Configuration Required:** The Linux Agent shares the same LLM configuration with the AppAgent. Before starting, ensure you have configured your LLM provider (OpenAI, Azure OpenAI, Gemini, Claude, etc.) and added your API keys to `config/ufo/agents.yaml`. See [Model Setup Guide](../configuration/models/overview.md) for detailed instructions.\n\n---\n\n## 📦 Step 1: Install Dependencies\n\nInstall all dependencies from the requirements file:\n\n```bash\npip install -r requirements.txt\n```\n\n**Verify installation:**\n\n```bash\npython3 -c \"import ufo; print('✅ UFO² installed successfully')\"\n```\n\n> **Tip:** For production deployments, use a virtual environment to isolate dependencies:\n> \n> ```bash\n> python3 -m venv venv\n> source venv/bin/activate # Linux/macOS\n> pip install -r requirements.txt\n> ```\n\n---\n\n## 🖥️ Step 2: Start Device Agent Server\n\n**Server Component:** The Device Agent Server is the central hub that manages connections from client devices and dispatches tasks. It can run on any machine (Linux, Windows, or remote server).\n\n### Server Machine Setup\n\nYou can run the server on:\n\n- ✅ Same machine as the client (localhost setup for testing)\n- ✅ Different machine on the same network\n- ✅ Remote server (requires proper network routing/SSH tunneling)\n\n### Basic Server Startup\n\nOn the server machine, run:\n\n```bash\npython -m ufo.server.app --port 5001\n```\n\n**Expected Output:**\n\n```console\n2024-11-06 10:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5001\nINFO: Started server process [12345]\nINFO: Waiting for application startup.\nINFO: Application startup complete.\nINFO: Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)\n```\n\nOnce you see \"Uvicorn running\", the server is ready at `ws://0.0.0.0:5001/ws`.\n\n### Server Configuration Options\n\n| Argument | Default | Description | Example |\n|----------|---------|-------------|---------|\n| `--port` | `5000` | Server listening port | `--port 5001` |\n| `--host` | `0.0.0.0` | Bind address (0.0.0.0 = all interfaces) | `--host 127.0.0.1` |\n| `--log-level` | `INFO` | Logging verbosity | `--log-level DEBUG` |\n\n**Custom Server Configuration:**\n\n**Custom Port:**\n```bash\npython -m ufo.server.app --port 8080\n```\n\n**Specific IP Binding:**\n```bash\npython -m ufo.server.app --host 192.168.1.100 --port 5001\n```\n\n**Debug Mode:**\n```bash\npython -m ufo.server.app --port 5001 --log-level DEBUG\n```\n\n### Verify Server is Running\n\n```bash\n# Test server health endpoint\ncurl http://localhost:5001/api/health\n```\n\n**Expected Response:**\n\n```json\n{\n \"status\": \"healthy\",\n \"online_clients\": []\n}\n```\n\n> **Documentation Reference:** For detailed server configuration and advanced features, see [Server Quick Start Guide](../server/quick_start.md).\n\n---\n\n## 🐧 Step 3: Start Device Agent Client (Linux Machine)\n\n**Client Component:** The Device Agent Client runs on the Linux machine where you want to execute tasks. It connects to the server via WebSocket and receives task commands.\n\n### Basic Client Startup\n\nOn the Linux machine where you want to execute tasks:\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://172.23.48.1:5001/ws \\\n --client-id linux_agent_1 \\\n --platform linux\n```\n\n### Client Parameters Explained\n\n| Parameter | Required | Description | Example |\n|-----------|----------|-------------|---------|\n| `--ws` | ✅ Yes | Enable WebSocket mode | `--ws` |\n| `--ws-server` | ✅ Yes | Server WebSocket URL | `ws://172.23.48.1:5001/ws` |\n| `--client-id` | ✅ Yes | **Unique** device identifier | `linux_agent_1` |\n| `--platform` | ✅ Yes (Linux) | Platform type (must be `linux` for Linux Agent) | `--platform linux` |\n\n> **⚠️ Critical Requirements:**\n> \n> 1. `--client-id` must be globally unique - No two devices can share the same ID\n> 2. `--platform linux` is mandatory - Without this flag, the Linux Agent won't work correctly\n> 3. Server address must be correct - Replace `172.23.48.1:5001` with your actual server IP and port\n\n### Understanding the WebSocket URL\n\nThe `--ws-server` parameter format is:\n\n```\nws://:/ws\n```\n\nExamples:\n\n| Scenario | WebSocket URL | Description |\n|----------|---------------|-------------|\n| **Localhost** | `ws://localhost:5001/ws` | Server and client on same machine |\n| **Same Network** | `ws://192.168.1.100:5001/ws` | Server on local network |\n| **Remote Server** | `ws://203.0.113.50:5001/ws` | Server on internet (public IP) |\n| **SSH Tunnel** | `ws://localhost:5001/ws` | After SSH reverse tunnel setup |\n\n### Connection Success Indicators\n\n**Client Logs:**\n\n```log\nINFO - Platform detected/specified: linux\nINFO - UFO Client initialized for platform: linux\nINFO - [WS] Connecting to ws://172.23.48.1:5001/ws (attempt 1/5)\nINFO - [WS] [AIP] Successfully registered as linux_agent_1\nINFO - [WS] Heartbeat loop started (interval: 30s)\n```\n\n**Server Logs:**\n\n```log\nINFO - [WS] ✅ Registered device client: linux_agent_1\nINFO - [WS] Device linux_agent_1 platform: linux\n```\n\nClient is connected and ready to receive tasks when you see \"Successfully registered\"!\n\n### Verify Connection\n\n```bash\n# Check connected clients on server\ncurl http://172.23.48.1:5001/api/clients\n```\n\n**Expected Response:**\n\n```json\n{\n \"clients\": [\n {\n \"client_id\": \"linux_agent_1\",\n \"type\": \"device\",\n \"platform\": \"linux\",\n \"connected_at\": 1730899822.0,\n \"uptime_seconds\": 45\n }\n ]\n}\n```\n\n> **Documentation Reference:** For detailed client configuration, see [Client Quick Start Guide](../client/quick_start.md).\n\n---\n\n## 🔌 Step 4: Start MCP Service (Linux Machine)\n\n**MCP Service Component:** The MCP (Model Context Protocol) Service provides the execution layer for CLI commands. It must be running on the same Linux machine as the client to handle command execution requests.\n\n### Start the MCP Server\n\nOn the Linux machine (same machine as the client):\n\n```bash\npython -m ufo.client.mcp.http_servers.linux_mcp_server\n```\n\n**Expected Output:**\n\n```console\nINFO: Started server process [23456]\nINFO: Waiting for application startup.\nINFO: Application startup complete.\nINFO: Uvicorn running on http://127.0.0.1:8010 (Press CTRL+C to quit)\n```\n\nThe MCP service is now ready to execute CLI commands at `http://127.0.0.1:8010`.\n\n### What is the MCP Service?\n\nThe **Linux MCP Server** provides two main functionalities:\n\n| Command | Purpose | Example Use Case |\n|---------|---------|------------------|\n| `EXEC_CLI` | Execute shell commands | `ls -la`, `grep pattern file.txt`, `ps aux` |\n| `SYS_INFO` | Retrieve system information | CPU usage, memory stats, disk space |\n\n**Architecture:**\n\n```mermaid\nsequenceDiagram\n participant Agent as Linux Agent\n participant MCP as Linux MCP Server\n participant Shell as Bash Shell\n \n Agent->>MCP: EXEC_CLI {command: \"ls -la\"}\n MCP->>Shell: Execute command\n Shell-->>MCP: stdout, stderr, exit_code\n MCP-->>Agent: {result, output}\n```\n\n### MCP Service Configuration\n\nThe MCP server typically runs on `localhost:8010` by default. The client automatically connects to it when configured properly.\n\n> **⚠️ MCP Service Must Be Running:** If the MCP service is not running, the Linux Agent cannot execute commands and will fail with:\n> ```\n> ERROR: Cannot connect to MCP server at http://127.0.0.1:8010\n> ```\n\n**Documentation Reference:** For detailed MCP command specifications, see [MCP Overview](../mcp/overview.md), [Linux MCP Commands](../linux/commands.md), and [BashExecutor Server](../mcp/servers/bash_executor.md).\n\n---\n\n## 🎯 Step 5: Dispatch Tasks via HTTP API\n\nOnce the server, client, and MCP service are all running, you can dispatch tasks to the Linux agent through the server's HTTP API.\n\n### API Endpoint\n\n```\nPOST http://:/api/dispatch\n```\n\n### Request Format\n\n```json\n{\n \"client_id\": \"linux_agent_1\",\n \"request\": \"Your natural language task description\",\n \"task_name\": \"optional_task_identifier\"\n}\n```\n\n### Example: Simple File Listing\n\n**Using cURL:**\n```bash\ncurl -X POST http://172.23.48.1:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"linux_agent_1\",\n \"request\": \"List all files in the /tmp directory\",\n \"task_name\": \"list_tmp_files\"\n }'\n```\n\n**Using Python:**\n```python\nimport requests\n\nresponse = requests.post(\n \"http://172.23.48.1:5001/api/dispatch\",\n json={\n \"client_id\": \"linux_agent_1\",\n \"request\": \"List all files in the /tmp directory\",\n \"task_name\": \"list_tmp_files\"\n }\n)\nprint(response.json())\n```\n\n**Using HTTPie:**\n```bash\nhttp POST http://172.23.48.1:5001/api/dispatch \\\n client_id=linux_agent_1 \\\n request=\"List all files in the /tmp directory\" \\\n task_name=list_tmp_files\n```\n\n**Successful Response:**\n\n```json\n{\n \"status\": \"dispatched\",\n \"task_name\": \"list_tmp_files\",\n \"client_id\": \"linux_agent_1\",\n \"session_id\": \"550e8400-e29b-41d4-a716-446655440000\"\n}\n```\n\n### Example: System Information Query\n\n```bash\ncurl -X POST http://172.23.48.1:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"linux_agent_1\",\n \"request\": \"Show disk usage for all mounted filesystems\",\n \"task_name\": \"check_disk_usage\"\n }'\n```\n\n### Example: Log File Analysis\n\n```bash\ncurl -X POST http://172.23.48.1:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"linux_agent_1\",\n \"request\": \"Find all ERROR or FATAL entries in /var/log/app.log from the last hour\",\n \"task_name\": \"analyze_error_logs\"\n }'\n```\n\n### Task Execution Flow\n\n```mermaid\nsequenceDiagram\n participant API as HTTP Client\n participant Server as Agent Server\n participant Client as Linux Client\n participant MCP as MCP Service\n participant Shell as Bash\n \n Note over API,Server: 1. Task Submission\n API->>Server: POST /api/dispatch {client_id, request}\n Server->>Server: Generate session_id\n Server-->>API: {status: dispatched, session_id}\n \n Note over Server,Client: 2. Task Assignment\n Server->>Client: TASK_ASSIGNMENT (via WebSocket)\n Client->>Client: Parse request Plan actions\n \n Note over Client,MCP: 3. Command Execution\n Client->>MCP: EXEC_CLI {command: \"ls -la /tmp\"}\n MCP->>Shell: Execute command\n Shell-->>MCP: stdout, stderr, exit_code\n MCP-->>Client: {result, output}\n \n Note over Client,Server: 4. Result Reporting\n Client->>Server: TASK_RESULT {status, result}\n```\n\n### Request Parameters\n\n| Field | Required | Type | Description | Example |\n|-------|----------|------|-------------|---------|\n| `client_id` | ✅ Yes | string | Target Linux agent ID (must match `--client-id`) | `\"linux_agent_1\"` |\n| `request` | ✅ Yes | string | Natural language task description | `\"List files in /var/log\"` |\n| `task_name` | ❌ Optional | string | Unique task identifier (auto-generated if omitted) | `\"task_001\"` |\n\n> **⚠️ Client Must Be Online:** If the `client_id` is not connected, you'll receive:\n> ```json\n> {\n> \"detail\": \"Client not online\"\n> }\n> ```\n> \n> Verify the client is connected:\n> ```bash\n> curl http://172.23.48.1:5001/api/clients\n> ```\n\n---\n\n## 🌉 Network Connectivity & SSH Tunneling\n\nWhen the server and client are on different networks or behind firewalls, you may need SSH tunneling to establish connectivity.\n\n### Scenario 1: Same Network (No Tunnel Needed)\n\n**Setup:**\n- Server: `192.168.1.100:5001`\n- Client: `192.168.1.50` (same LAN)\n\n**Client Command:**\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5001/ws \\\n --client-id linux_agent_1 \\\n --platform linux\n```\n\n**No additional configuration needed** ✅\n\n### Scenario 2: Client Behind Firewall (Reverse SSH Tunnel)\n\n**Problem:**\n- Server: `203.0.113.50:5001` (public IP, accessible)\n- Client: `192.168.1.50` (private network, behind NAT/firewall)\n- **Client cannot directly reach server**\n\n**Solution: SSH Reverse Tunnel**\n\nOn the **client machine**, create an SSH reverse tunnel:\n\n```bash\nssh -N -R 5001:localhost:5001 user@203.0.113.50\n```\n\n**Parameters:**\n- `-N`: No remote command execution (tunnel only)\n- `-R 5001:localhost:5001`: Forward remote port 5001 to local port 5001\n- `user@203.0.113.50`: SSH server address (where the UFO server runs)\n\n**What This Does:**\n\n```mermaid\ngraph LR\n Client[Client Machine 192.168.1.50]\n SSH[SSH Tunnel]\n Server[Server Machine 203.0.113.50]\n \n Client -->|SSH Reverse Tunnel| SSH\n SSH -->|Port 5001| Server\n \n style Client fill:#e1f5ff\n style Server fill:#ffe1e1\n style SSH fill:#fffacd\n```\n\n**After tunnel is established:**\n\n```bash\n# Client can now connect to localhost:5001\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id linux_agent_1 \\\n --platform linux\n```\n\n### Scenario 3: Server Behind Firewall (Forward SSH Tunnel)\n\n**Problem:**\n- Server: `192.168.1.100:5001` (private network)\n- Client: `203.0.113.75` (public network)\n- **Client cannot directly reach server**\n\n**Solution: SSH Forward Tunnel**\n\nOn the **client machine**, create an SSH forward tunnel to the server's network:\n\n```bash\nssh -N -L 5001:192.168.1.100:5001 gateway-user@vpn.company.com\n```\n\n**Parameters:**\n- `-N`: No remote command execution\n- `-L 5001:192.168.1.100:5001`: Forward local port 5001 to remote 192.168.1.100:5001\n- `gateway-user@vpn.company.com`: SSH gateway that can access the server\n\n**After tunnel is established:**\n\n```bash\n# Client connects to localhost, which forwards to server\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id linux_agent_1 \\\n --platform linux\n```\n\n### Example: Complex Tunnel Setup\n\n**Situation:**\n- Server IP: `10.0.0.50:5001` (corporate network)\n- Client IP: `192.168.1.75` (home network)\n- SSH Gateway: `vpn.company.com` (accessible from internet)\n\n**Step 1: Create SSH Tunnel**\n```bash\n# On client machine\nssh -N -L 5001:10.0.0.50:5001 myuser@vpn.company.com\n```\n\n**Step 2: Start Client (in another terminal)**\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id linux_agent_home_1 \\\n --platform linux\n```\n\n### SSH Tunnel Best Practices\n\nFor production use, add these flags to your SSH tunnel:\n\n```bash\nssh -N \\\n -L 5001:server:5001 \\\n -o ServerAliveInterval=60 \\\n -o ServerAliveCountMax=3 \\\n -o ExitOnForwardFailure=yes \\\n user@gateway\n```\n\n**Flags explained:**\n- `ServerAliveInterval=60`: Send keep-alive every 60 seconds\n- `ServerAliveCountMax=3`: Disconnect after 3 failed keep-alives\n- `ExitOnForwardFailure=yes`: Exit if port forwarding fails\n\n### Persistent SSH Tunnel with Autossh\n\nFor production, use `autossh` to automatically restart the tunnel if it fails:\n\n```bash\n# Install autossh\nsudo apt-get install autossh # Debian/Ubuntu\n\n# Start persistent tunnel\nautossh -M 0 \\\n -N \\\n -L 5001:server:5001 \\\n -o ServerAliveInterval=60 \\\n -o ServerAliveCountMax=3 \\\n user@gateway\n```\n\n> **ℹ️ Network Configuration:** For more network configuration details, see [Server Quick Start - Troubleshooting](../server/quick_start.md#common-issues-troubleshooting).\n\n---\n\n## 🌌 Step 6: Configure as UFO³ Galaxy Device\n\nTo use the Linux Agent as a managed device within the **UFO³ Galaxy** multi-tier framework, you need to register it in the `devices.yaml` configuration file.\n\n### Device Configuration File\n\nThe Galaxy configuration is located at:\n\n```\nconfig/galaxy/devices.yaml\n```\n\n### Add Linux Agent Configuration\n\nEdit `config/galaxy/devices.yaml` and add your Linux agent under the `devices` section:\n\n```yaml\ndevices:\n - device_id: \"linux_agent_1\"\n server_url: \"ws://172.23.48.1:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server\"\n - \"log_analysis\"\n - \"file_operations\"\n metadata:\n os: \"linux\"\n performance: \"medium\"\n logs_file_path: \"/var/log/myapp/app.log\"\n dev_path: \"/home/user/development/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n auto_connect: true\n max_retries: 5\n```\n\n### Configuration Fields Explained\n\n| Field | Required | Type | Description | Example |\n|-------|----------|------|-------------|---------|\n| `device_id` | ✅ Yes | string | **Must match client `--client-id`** | `\"linux_agent_1\"` |\n| `server_url` | ✅ Yes | string | **Must match server WebSocket URL** | `\"ws://172.23.48.1:5001/ws\"` |\n| `os` | ✅ Yes | string | Operating system | `\"linux\"` |\n| `capabilities` | ❌ Optional | list | Device capabilities (for task routing) | `[\"server\", \"log_analysis\"]` |\n| `metadata` | ❌ Optional | dict | Custom metadata for task context | See below |\n| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup | `true` |\n| `max_retries` | ❌ Optional | integer | Connection retry attempts | `5` |\n\n### Metadata Fields (Custom)\n\nThe `metadata` section can contain any custom fields relevant to your Linux agent:\n\n| Field | Purpose | Example |\n|-------|---------|---------|\n| `logs_file_path` | Path to application logs | `\"/var/log/app.log\"` |\n| `dev_path` | Development directory | `\"/home/user/dev/\"` |\n| `warning_log_pattern` | Regex pattern for warnings | `\"WARN\"` |\n| `error_log_pattern` | Regex pattern for errors | `\"ERROR\\|FATAL\"` |\n| `performance` | Performance tier | `\"high\"`, `\"medium\"`, `\"low\"` |\n| `description` | Human-readable description | `\"Production database server\"` |\n\n### Multiple Linux Agents Example\n\n```yaml\ndevices:\n - device_id: \"linux_agent_1\"\n server_url: \"ws://172.23.48.1:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"web_server\"\n metadata:\n logs_file_path: \"/var/log/nginx/access.log\"\n dev_path: \"/var/www/html/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n auto_connect: true\n max_retries: 5\n\n - device_id: \"linux_agent_2\"\n server_url: \"ws://172.23.48.2:5002/ws\"\n os: \"linux\"\n capabilities:\n - \"database_server\"\n metadata:\n logs_file_path: \"/var/log/postgresql/postgresql.log\"\n dev_path: \"/var/lib/postgresql/\"\n warning_log_pattern: \"WARNING\"\n error_log_pattern: \"ERROR|FATAL|PANIC\"\n auto_connect: true\n max_retries: 5\n\n - device_id: \"linux_agent_3\"\n server_url: \"ws://172.23.48.3:5003/ws\"\n os: \"linux\"\n capabilities:\n - \"monitoring\"\n metadata:\n logs_file_path: \"/var/log/prometheus/prometheus.log\"\n dev_path: \"/opt/prometheus/\"\n warning_log_pattern: \"level=warn\"\n error_log_pattern: \"level=error\"\n auto_connect: true\n max_retries: 5\n```\n\n### Critical Requirements\n\n> **⚠️ Configuration Validation - These fields MUST match exactly:**\n> \n> 1. **`device_id` in YAML** ↔ **`--client-id` in client command**\n> ```yaml\n> device_id: \"linux_agent_1\" # In devices.yaml\n> ```\n> ```bash\n> --client-id linux_agent_1 # In client command\n> ```\n> \n> 2. **`server_url` in YAML** ↔ **`--ws-server` in client command**\n> ```yaml\n> server_url: \"ws://172.23.48.1:5001/ws\" # In devices.yaml\n> ```\n> ```bash\n> --ws-server ws://172.23.48.1:5001/ws # In client command\n> ```\n> \n> **If these don't match, Galaxy cannot control the device!**\n\n### Using Galaxy to Control Linux Agents\n\nOnce configured, you can launch Galaxy and it will automatically manage the Linux agents:\n\n```bash\npython -m galaxy --interactive\n```\n\n**Galaxy will:**\n1. ✅ Automatically load device configuration from `config/galaxy/devices.yaml`\n2. ✅ Connect to all configured devices\n3. ✅ Orchestrate multi-device tasks\n4. ✅ Route tasks based on capabilities\n5. ✅ Monitor device health\n\n> **ℹ️ Galaxy Documentation:** For detailed Galaxy configuration and usage, see:\n> \n> - [Galaxy Overview](../galaxy/overview.md)\n> - [Galaxy Quick Start](quick_start_galaxy.md)\n> - [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md)\n\n---\n\n## 🐛 Common Issues & Troubleshooting\n\n### Issue 1: Client Cannot Connect to Server\n\n**Error: Connection Refused**\n\nSymptoms:\n```log\nERROR - [WS] Failed to connect to ws://172.23.48.1:5001/ws\nConnection refused\n```\n\n**Diagnosis Checklist:**\n\n- [ ] Is the server running? (`curl http://172.23.48.1:5001/api/health`)\n- [ ] Is the port correct? (Check server startup logs)\n- [ ] Can client reach server IP? (`ping 172.23.48.1`)\n- [ ] Is firewall blocking port 5001?\n- [ ] Is SSH tunnel established (if needed)?\n\n**Solutions:**\n\nVerify Server:\n```bash\n# On server machine\ncurl http://localhost:5001/api/health\n\n# From client machine\ncurl http://172.23.48.1:5001/api/health\n```\n\nCheck Network:\n```bash\n# Test connectivity\nping 172.23.48.1\n\n# Test port accessibility\nnc -zv 172.23.48.1 5001\ntelnet 172.23.48.1 5001\n```\n\nCheck Firewall:\n```bash\n# On server machine (Ubuntu/Debian)\nsudo ufw status\nsudo ufw allow 5001/tcp\n\n# On server machine (RHEL/CentOS)\nsudo firewall-cmd --list-ports\nsudo firewall-cmd --add-port=5001/tcp --permanent\nsudo firewall-cmd --reload\n```\n\n### Issue 2: MCP Service Not Responding\n\n**Error: Cannot Execute Commands**\n\nSymptoms:\n```log\nERROR - Cannot connect to MCP server at http://127.0.0.1:8010\nERROR - Command execution failed\n```\n\n**Diagnosis:**\n\n- [ ] Is the MCP service running?\n- [ ] Is it running on the correct port?\n- [ ] Are there any startup errors in MCP logs?\n\n**Solutions:**\n\nVerify MCP Service:\n```bash\n# Check if MCP service is running\ncurl http://localhost:8010/health\n\n# Or check process\nps aux | grep linux_mcp_server\n```\n\nRestart MCP Service:\n```bash\n# Kill existing process (if hung)\npkill -f linux_mcp_server\n\n# Start fresh\npython -m ufo.client.mcp.http_servers.linux_mcp_server\n```\n\nCheck Port Conflict:\n```bash\n# See if something else is using port 8010\nlsof -i :8010\nnetstat -tuln | grep 8010\n\n# If port is taken, start MCP on different port\npython -m ufo.client.mcp.http_servers.linux_mcp_server --port 8011\n```\n\n### Issue 3: Missing `--platform linux` Flag\n\n**Error: Incorrect Agent Type**\n\nSymptoms:\n- Client connects but cannot execute Linux commands\n- Server logs show wrong platform type\n- Tasks fail with \"unsupported operation\" errors\n\n**Cause:** Forgot to add `--platform linux` flag when starting the client.\n\n**Solution:**\n```bash\n# Wrong (missing platform)\npython -m ufo.client.client --ws --client-id linux_agent_1\n\n# Correct\npython -m ufo.client.client \\\n --ws \\\n --client-id linux_agent_1 \\\n --platform linux\n```\n\n### Issue 4: Duplicate Client ID\n\n**Error: Registration Failed**\n\nSymptoms:\n```log\nERROR - [WS] Registration failed: client_id already exists\nERROR - Another device is using ID 'linux_agent_1'\n```\n\n**Cause:** Multiple clients trying to use the same `client_id`.\n\n**Solutions:**\n\n1. **Use unique client IDs:**\n ```bash\n # Device 1\n --client-id linux_agent_1\n \n # Device 2\n --client-id linux_agent_2\n \n # Device 3\n --client-id linux_agent_3\n ```\n\n2. **Check currently connected clients:**\n ```bash\n curl http://172.23.48.1:5001/api/clients\n ```\n\n### Issue 5: Galaxy Cannot Find Device\n\n**Error: Device Not Configured**\n\nSymptoms:\n```log\nERROR - Device 'linux_agent_1' not found in configuration\nWARNING - Cannot dispatch task to unknown device\n```\n\n**Cause:** Mismatch between `devices.yaml` configuration and actual client setup.\n\n**Diagnosis:**\n\nCheck that these match **exactly**:\n\n| Location | Field | Example |\n|----------|-------|---------|\n| `devices.yaml` | `device_id` | `\"linux_agent_1\"` |\n| Client command | `--client-id` | `linux_agent_1` |\n| `devices.yaml` | `server_url` | `\"ws://172.23.48.1:5001/ws\"` |\n| Client command | `--ws-server` | `ws://172.23.48.1:5001/ws` |\n\n**Solution:** Update `devices.yaml` to match your client configuration, or vice versa.\n\n### Issue 6: SSH Tunnel Keeps Disconnecting\n\n**Error: Tunnel Connection Lost**\n\nSymptoms:\n- Client disconnects after a few minutes\n- SSH tunnel closes unexpectedly\n- \"Connection reset by peer\" errors\n\n**Solutions:**\n\nUse ServerAliveInterval:\n```bash\nssh -N \\\n -L 5001:server:5001 \\\n -o ServerAliveInterval=60 \\\n -o ServerAliveCountMax=3 \\\n user@gateway\n```\n\nUse Autossh:\n```bash\nautossh -M 0 \\\n -N \\\n -L 5001:server:5001 \\\n -o ServerAliveInterval=60 \\\n user@gateway\n```\n\nRun in Screen/Tmux:\n```bash\n# Start screen session\nscreen -S ssh-tunnel\n\n# Run SSH tunnel\nssh -N -L 5001:server:5001 user@gateway\n\n# Detach: Ctrl+A, then D\n# Reattach: screen -r ssh-tunnel\n```\n\n---\n\n## 📚 Next Steps\n\nYou've successfully set up a Linux Agent! Explore these topics to deepen your understanding:\n\n### Immediate Next Steps\n\n| Priority | Topic | Time | Link |\n|----------|-------|------|------|\n| 🥇 | **Linux Agent Architecture** | 10 min | [Overview](../linux/overview.md) |\n| 🥈 | **State Machine & Processing** | 15 min | [State Machine](../linux/state.md) |\n| 🥉 | **MCP Commands Reference** | 10 min | [Commands](../linux/commands.md) |\n\n### Advanced Topics\n\n| Topic | Description | Link |\n|-------|-------------|------|\n| **Processing Strategy** | 3-phase pipeline (LLM, Action, Memory) | [Strategy](../linux/strategy.md) |\n| **Galaxy Integration** | Multi-device orchestration | [Galaxy Overview](../galaxy/overview.md) |\n| **MCP Protocol** | Deep dive into command execution | [MCP Overview](../mcp/overview.md) |\n| **Server Architecture** | Understanding the server internals | [Server Overview](../server/overview.md) |\n\n### Production Deployment\n\n| Best Practice | Description | Link |\n|---------------|-------------|------|\n| **Systemd Service** | Run client as Linux service | [Client Guide](../client/quick_start.md#running-as-background-service) |\n| **Log Management** | Structured logging and rotation | [Server Monitoring](../server/monitoring.md) |\n| **Security Hardening** | SSL/TLS, authentication, firewalls | [Server Guide](../server/quick_start.md#production-deployment) |\n\n---\n\n## ✅ Summary\n\n## ✅ What You've Accomplished\n\nCongratulations! You've successfully:\n\n✅ Switched to the `linux-client` branch \n✅ Installed all dependencies \n✅ Started the Device Agent Server \n✅ Connected a Linux Device Agent Client \n✅ Launched the MCP service for command execution \n✅ Dispatched tasks via HTTP API \n✅ (Optional) Configured SSH tunneling for remote access \n✅ (Optional) Registered the device in Galaxy configuration \n\n**Your Linux Agent is Ready**\n\nYou can now:\n\n- 🎯 Execute CLI commands on Linux machines remotely\n- 📊 Analyze log files across multiple servers\n- 🔧 Manage development environments\n- 🌌 Integrate with UFO³ Galaxy for multi-device workflows\n\n**Start exploring and automating your Linux infrastructure!** 🚀\n"
},
{
"path": "documents/docs/getting_started/quick_start_mobile.md",
"content": "# ⚡ Quick Start: Mobile Agent\n\nGet your Android device running as a UFO³ device agent in 10 minutes. This guide walks you through ADB setup, server/client configuration, and MCP service initialization for Android automation.\n\n> **📚 Documentation Navigation:**\n> \n> - **Architecture & Concepts:** [Mobile Agent Overview](../mobile/overview.md)\n> - **State Management:** [State Machine](../mobile/state.md)\n> - **Processing Pipeline:** [Processing Strategy](../mobile/strategy.md)\n> - **Available Commands:** [MCP Commands Reference](../mobile/commands.md)\n> - **Galaxy Integration:** [As Galaxy Device](../mobile/as_galaxy_device.md)\n\n---\n\n## 📋 Prerequisites\n\nBefore you begin, ensure you have:\n\n- **Python 3.10+** installed on your computer\n- **UFO repository** cloned from [GitHub](https://github.com/microsoft/UFO)\n- **Android device** (physical device or emulator) with Android 5.0+ (API 21+)\n- **ADB (Android Debug Bridge)** installed and accessible\n- **USB debugging enabled** on your Android device (for physical devices)\n- **Network connectivity** between server and client machines\n- **LLM configured** in `config/ufo/agents.yaml` (see [Model Configuration](../configuration/models/overview.md))\n\n| Component | Minimum Version | Verification Command |\n|-----------|----------------|---------------------|\n| Python | 3.10 | `python --version` |\n| Android OS | 5.0 (API 21) | Check device settings |\n| ADB | Latest | `adb --version` |\n| LLM API Key | N/A | Check `config/ufo/agents.yaml` |\n\n> **⚠️ LLM Configuration Required:** The Mobile Agent shares the same LLM configuration with the AppAgent. Before starting, ensure you have configured your LLM provider (OpenAI, Azure OpenAI, Gemini, Claude, etc.) and added your API keys to `config/ufo/agents.yaml`. See [Model Setup Guide](../configuration/models/overview.md) for detailed instructions.\n\n---\n\n## 📱 Step 0: Android Device Setup\n\nYou can use either a **physical Android device** or an **Android emulator**. Choose the setup method that fits your needs.\n\n### Option A: Physical Android Device Setup\n\n#### 1. Enable Developer Options\n\nOn your Android device:\n\n1. Open **Settings** → **About phone**\n2. Tap **Build number** 7 times\n3. You'll see \"You are now a developer!\"\n\n#### 2. Enable USB Debugging\n\n1. Go to **Settings** → **System** → **Developer options**\n2. Turn on **USB debugging**\n3. (Optional) Turn on **Stay awake** (device won't sleep while charging)\n\n#### 3. Connect Device to Computer\n\n**Via USB Cable:**\n\n```bash\n# Connect device via USB\n# On device, allow USB debugging when prompted\n\n# Verify connection\nadb devices\n```\n\n**Expected Output:**\n```\nList of devices attached\nXXXXXXXXXXXXXX device\n```\n\n**Via Wireless ADB (Android 11+):**\n\n```bash\n# On device: Settings → Developer options → Wireless debugging\n# Get IP address and port (e.g., 192.168.1.100:5555)\n\n# On computer: Connect to device\nadb connect 192.168.1.100:5555\n\n# Verify connection\nadb devices\n```\n\n**Expected Output:**\n```\nList of devices attached\n192.168.1.100:5555 device\n```\n\n### Option B: Android Emulator Setup\n\n#### Option B1: Using Android Studio Emulator (Recommended)\n\n**Step 1: Install Android Studio**\n\nDownload from: https://developer.android.com/studio\n\n**Windows:**\n```powershell\n# Download Android Studio installer\n# Run: android-studio-xxx.exe\n# Follow installation wizard\n```\n\n**macOS:**\n```bash\n# Download Android Studio DMG\n# Drag to Applications folder\n# Open Android Studio\n```\n\n**Linux:**\n```bash\n# Download Android Studio tarball\ntar -xzf android-studio-*.tar.gz\ncd android-studio/bin\n./studio.sh\n```\n\n**Step 2: Install Android SDK Components**\n\n1. Open Android Studio\n2. Go to **Tools** → **SDK Manager**\n3. Install:\n - ✅ Android SDK Platform (API 33 or higher)\n - ✅ Android SDK Platform-Tools\n - ✅ Android SDK Build-Tools\n - ✅ Android Emulator\n\n**Step 3: Create Virtual Device**\n\n1. In Android Studio, click **Device Manager** (phone icon)\n2. Click **Create Device**\n3. Select hardware:\n - **Phone** category\n - Choose **Pixel 6** or **Pixel 7** (recommended)\n - Click **Next**\n\n4. Select system image:\n - Choose **Release Name**: **Tiramisu** (Android 13, API 33) or newer\n - Click **Download** if not installed\n - Click **Next**\n\n5. Configure AVD:\n - **AVD Name**: `Pixel_6_API_33` (or your choice)\n - **Startup orientation**: Portrait\n - **Graphics**: Automatic or Hardware\n - Click **Finish**\n\n**Step 4: Start Emulator**\n\n**From Android Studio:**\n1. Open **Device Manager**\n2. Click ▶️ (Play button) next to your AVD\n\n**From Command Line:**\n```bash\n# List available emulators\nemulator -list-avds\n\n# Start emulator\nemulator -avd Pixel_6_API_33 &\n```\n\n**Step 5: Verify ADB Connection**\n\n```bash\n# Wait for emulator to fully boot (~1-2 minutes)\nadb devices\n```\n\n**Expected Output:**\n```\nList of devices attached\nemulator-5554 device\n```\n\n#### Option B2: Using Genymotion (Alternative)\n\n**Step 1: Install Genymotion**\n\nDownload from: https://www.genymotion.com/download/\n\n```bash\n# Free personal edition available\n# Requires VirtualBox (auto-installed)\n```\n\n**Step 2: Create Virtual Device**\n\n1. Open Genymotion\n2. Click **+** (Add new device)\n3. Sign in with Genymotion account (free)\n4. Select device:\n - **Google Pixel 6** or similar\n - **Android 13.0** or newer\n5. Click **Install**\n6. Click **Start**\n\n**Step 3: Verify ADB Connection**\n\n```bash\nadb devices\n```\n\n**Expected Output:**\n```\nList of devices attached\n192.168.56.101:5555 device\n```\n\n### Verify Device is Ready\n\nRun this test to ensure device is accessible:\n\n```bash\n# Get device model\nadb shell getprop ro.product.model\n\n# Get Android version\nadb shell getprop ro.build.version.release\n\n# Test screenshot capability\nadb shell screencap -p /sdcard/test.png\nadb pull /sdcard/test.png .\n```\n\nIf all commands succeed, your device is ready! ✅\n\n---\n\n## 🔧 Step 1: Install ADB (Android Debug Bridge)\n\nADB is essential for communicating with Android devices. Choose your platform:\n\n### Windows\n\n**Option 1: Install via Android Studio (Recommended)**\n\nADB is included with Android Studio (see Step 0 Option B1).\n\nAfter installation, add to PATH:\n\n```powershell\n# Add Android SDK platform-tools to PATH\n# Default location:\n$env:PATH += \";C:\\Users\\\\AppData\\Local\\Android\\Sdk\\platform-tools\"\n\n# Test\nadb --version\n```\n\n**Option 2: Standalone ADB Installation**\n\n```powershell\n# Download platform-tools\n# https://developer.android.com/studio/releases/platform-tools\n\n# Extract to C:\\adb\n# Add to PATH:\n$env:PATH += \";C:\\adb\"\n\n# Test\nadb --version\n```\n\n**Make PATH Permanent (Optional):**\n\n1. Open **System Properties** → **Environment Variables**\n2. Under **User variables**, edit **Path**\n3. Add: `C:\\Users\\\\AppData\\Local\\Android\\Sdk\\platform-tools`\n4. Click **OK**\n\n### macOS\n\n**Option 1: Via Homebrew (Recommended)**\n\n```bash\n# Install Homebrew (if not installed)\n/bin/bash -c \"$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)\"\n\n# Install ADB\nbrew install android-platform-tools\n\n# Verify\nadb --version\n```\n\n**Option 2: Via Android Studio**\n\nADB is included with Android Studio. Add to PATH:\n\n```bash\n# Add to ~/.zshrc or ~/.bash_profile\nexport PATH=\"$PATH:$HOME/Library/Android/sdk/platform-tools\"\n\n# Reload\nsource ~/.zshrc\n\n# Test\nadb --version\n```\n\n### Linux\n\n**Ubuntu/Debian:**\n\n```bash\nsudo apt update\nsudo apt install -y adb\n\n# Verify\nadb --version\n```\n\n**Fedora/RHEL:**\n\n```bash\nsudo dnf install android-tools\n\n# Verify\nadb --version\n```\n\n**Arch Linux:**\n\n```bash\nsudo pacman -S android-tools\n\n# Verify\nadb --version\n```\n\n### Verify ADB Installation\n\n```bash\nadb version\n```\n\n**Expected Output:**\n```\nAndroid Debug Bridge version 1.0.41\nVersion 34.0.5-10900879\n```\n\n---\n\n## 📦 Step 2: Install Python Dependencies\n\nInstall all UFO dependencies:\n\n```bash\ncd /path/to/UFO\npip install -r requirements.txt\n```\n\n**Verify installation:**\n\n```bash\npython -c \"import ufo; print('✅ UFO installed successfully')\"\n```\n\n> **Tip:** For production deployments, use a virtual environment:\n> \n> ```bash\n> python -m venv venv\n> \n> # Windows\n> venv\\Scripts\\activate\n> \n> # macOS/Linux\n> source venv/bin/activate\n> \n> pip install -r requirements.txt\n> ```\n\n---\n\n## 🖥️ Step 3: Start Device Agent Server\n\n**Server Component:** The Device Agent Server manages connections from Android devices and dispatches tasks.\n\n### Basic Server Startup\n\nOn your computer (where Python is installed):\n\n```bash\npython -m ufo.server.app --port 5001 --platform mobile\n```\n\n**Expected Output:**\n\n```console\nINFO - Starting UFO Server on 0.0.0.0:5001\nINFO - Platform: mobile\nINFO - Log level: WARNING\nINFO: Started server process [12345]\nINFO: Waiting for application startup.\nINFO: Application startup complete.\nINFO: Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)\n```\n\nOnce you see \"Uvicorn running\", the server is ready at `ws://0.0.0.0:5001/ws`.\n\n### Server Configuration Options\n\n| Argument | Default | Description | Example |\n|----------|---------|-------------|---------|\n| `--port` | `5000` | Server listening port | `--port 5001` |\n| `--host` | `0.0.0.0` | Bind address | `--host 127.0.0.1` |\n| `--platform` | Auto | Platform override | `--platform mobile` |\n| `--log-level` | `WARNING` | Logging verbosity | `--log-level DEBUG` |\n\n**Custom Configuration Examples:**\n\n```bash\n# Different port\npython -m ufo.server.app --port 8080 --platform mobile\n\n# Localhost only\npython -m ufo.server.app --host 127.0.0.1 --port 5001 --platform mobile\n\n# Debug mode\npython -m ufo.server.app --port 5001 --platform mobile --log-level DEBUG\n```\n\n### Verify Server is Running\n\n```bash\ncurl http://localhost:5001/api/health\n```\n\n**Expected Response (when no clients connected):**\n\n```json\n{\n \"status\": \"healthy\",\n \"online_clients\": []\n}\n```\n\n> **💡 Tip:** The `online_clients` list will be empty until you start and connect the Mobile Client in Step 5.\n\n---\n\n## 🔌 Step 4: Start MCP Services (Android Machine)\n\n**MCP Service Component:** Two MCP servers provide Android device interaction capabilities. They must be running before starting the client.\n\n> **💡 Learn More:** For detailed documentation on all available MCP commands and their usage, see the [MCP Commands Reference](../mobile/commands.md).\n\n### Understanding the Two MCP Servers\n\nMobileAgent uses **two separate MCP servers** for different responsibilities:\n\n| Server | Port | Purpose | Tools |\n|--------|------|---------|-------|\n| **Data Collection** | 8020 | Screenshot, UI tree, device info, apps list | 5 read-only tools |\n| **Action** | 8021 | Touch actions, typing, app launching | 8 control tools |\n\n### Start Both MCP Servers\n\n**Recommended: Start Both Servers Together**\n\nOn the machine with ADB access to your Android device:\n\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --data-port 8020 \\\n --action-port 8021 \\\n --server both\n```\n\n**Expected Output:**\n\n```console\n====================================================================\nUFO Mobile MCP Servers (Android)\nAndroid device control via ADB and Model Context Protocol\n====================================================================\n\nUsing ADB: adb\nChecking ADB connection...\n\nList of devices attached\nemulator-5554 device\n\n✅ Found 1 connected device(s)\n====================================================================\n\n🚀 Starting both servers on localhost (shared state)\n - Data Collection Server: localhost:8020\n - Action Server: localhost:8021\n\nNote: Both servers share the same MobileServerState for caching\n\n✅ Starting both servers in same process (shared MobileServerState)\n - Data Collection Server: localhost:8020\n - Action Server: localhost:8021\n\n======================================================================\nBoth servers share MobileServerState cache. Press Ctrl+C to stop.\n======================================================================\n\n✅ Data Collection Server thread started\n✅ Action Server thread started\n\n======================================================================\nBoth servers are running. Press Ctrl+C to stop.\n======================================================================\n```\n\n**Alternative: Start Servers Separately**\n\nIf needed, you can start each server in separate terminals:\n\n**Terminal 1: Data Collection Server**\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --data-port 8020 \\\n --server data\n```\n\n**Terminal 2: Action Server**\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --action-port 8021 \\\n --server action\n```\n\n> **⚠️ Important:** When running servers separately, they won't share cached state, which may impact performance. Running both together is recommended.\n\n### MCP Server Configuration Options\n\n| Argument | Default | Description | Example |\n|----------|---------|-------------|---------|\n| `--host` | `localhost` | Server host | `--host 127.0.0.1` |\n| `--data-port` | `8020` | Data collection server port | `--data-port 8020` |\n| `--action-port` | `8021` | Action server port | `--action-port 8021` |\n| `--server` | `both` | Which server(s) to start | `--server both` |\n| `--adb-path` | `adb` | Path to ADB executable | `--adb-path /path/to/adb` |\n\n### Verify MCP Servers are Running\n\n**Check Data Collection Server:**\n```bash\ncurl http://localhost:8020/health\n```\n\n**Check Action Server:**\n```bash\ncurl http://localhost:8021/health\n```\n\nBoth should return a health status response indicating the server is operational.\n\n### What if ADB is not in PATH?\n\nIf ADB is not in your system PATH, specify the full path:\n\n**Windows:**\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --adb-path \"C:\\Users\\YourUsername\\AppData\\Local\\Android\\Sdk\\platform-tools\\adb.exe\" \\\n --server both\n```\n\n**macOS:**\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --adb-path \"$HOME/Library/Android/sdk/platform-tools/adb\" \\\n --server both\n```\n\n**Linux:**\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --adb-path /usr/bin/adb \\\n --server both\n```\n\n---\n\n## 📱 Step 5: Start Device Agent Client\n\n**Client Component:** The Device Agent Client connects your Android device to the server and executes mobile automation tasks.\n\n### Basic Client Startup\n\nOn your computer (same machine as MCP servers):\n\n```bash\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id mobile_phone_1 \\\n --platform mobile\n```\n\n### Client Parameters Explained\n\n| Parameter | Required | Description | Example |\n|-----------|----------|-------------|---------|\n| `--ws` | ✅ Yes | Enable WebSocket mode | `--ws` |\n| `--ws-server` | ✅ Yes | Server WebSocket URL | `ws://localhost:5001/ws` |\n| `--client-id` | ✅ Yes | **Unique** device identifier | `mobile_phone_1` |\n| `--platform` | ✅ Yes | Platform type (must be `mobile`) | `--platform mobile` |\n\n> **⚠️ Critical Requirements:**\n> \n> 1. `--client-id` must be globally unique - No two devices can share the same ID\n> 2. `--platform mobile` is mandatory - Without this flag, the Mobile Agent won't work correctly\n> 3. Server address must be correct - Use actual server IP if not on localhost\n\n### Understanding the WebSocket URL\n\nThe `--ws-server` parameter format is:\n\n```\nws://:/ws\n```\n\nExamples:\n\n| Scenario | WebSocket URL | Description |\n|----------|---------------|-------------|\n| **Same Machine** | `ws://localhost:5001/ws` | Server and client on same computer |\n| **Same Network** | `ws://192.168.1.100:5001/ws` | Server on local network |\n| **Remote Server** | `ws://203.0.113.50:5001/ws` | Server on internet (public IP) |\n\n### Connection Success Indicators\n\n**Client Logs:**\n\n```log\nINFO - Platform detected/specified: mobile\nINFO - UFO Client initialized for platform: mobile\nINFO - [WS] Connecting to ws://localhost:5001/ws (attempt 1/5)\nINFO - [WS] [AIP] Successfully registered as mobile_phone_1\nINFO - [WS] Heartbeat loop started (interval: 30s)\n```\n\n**Server Logs:**\n\n```log\nINFO - [WS] ✅ Registered device client: mobile_phone_1\nINFO - [WS] Device mobile_phone_1 platform: mobile\n```\n\nClient is connected and ready to receive tasks when you see \"Successfully registered\"! ✅\n\n### Verify Connection\n\n```bash\n# Check connected clients on server\ncurl http://localhost:5001/api/clients\n```\n\n**Expected Response:**\n\n```json\n{\n \"online_clients\": [\"mobile_phone_1\"]\n}\n```\n\n> **Note:** The response shows only client IDs. For detailed information about each client, check the server logs.\n\n---\n\n## 🎯 Step 6: Dispatch Tasks via HTTP API\n\nOnce the server, client, and MCP services are all running, you can dispatch tasks to your Android device through the server's HTTP API.\n\n### API Endpoint\n\n```\nPOST http://:/api/dispatch\n```\n\n### Request Format\n\n```json\n{\n \"client_id\": \"mobile_phone_1\",\n \"request\": \"Your natural language task description\",\n \"task_name\": \"optional_task_identifier\"\n}\n```\n\n### Example 1: Launch an App\n\n**Using cURL:**\n```bash\ncurl -X POST http://localhost:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"mobile_phone_1\",\n \"request\": \"Open Google Chrome browser\",\n \"task_name\": \"launch_chrome\"\n }'\n```\n\n**Using Python:**\n```python\nimport requests\n\nresponse = requests.post(\n \"http://localhost:5001/api/dispatch\",\n json={\n \"client_id\": \"mobile_phone_1\",\n \"request\": \"Open Google Chrome browser\",\n \"task_name\": \"launch_chrome\"\n }\n)\nprint(response.json())\n```\n\n**Successful Response:**\n\n```json\n{\n \"status\": \"dispatched\",\n \"task_name\": \"launch_chrome\",\n \"client_id\": \"mobile_phone_1\",\n \"session_id\": \"550e8400-e29b-41d4-a716-446655440000\"\n}\n```\n\n### Example 2: Search on Maps\n\n```bash\ncurl -X POST http://localhost:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"mobile_phone_1\",\n \"request\": \"Open Google Maps and search for coffee shops nearby\",\n \"task_name\": \"search_coffee\"\n }'\n```\n\n### Example 3: Type and Submit Text\n\n```bash\ncurl -X POST http://localhost:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"mobile_phone_1\",\n \"request\": \"Open Chrome, search for weather forecast, and show me the results\",\n \"task_name\": \"check_weather\"\n }'\n```\n\n### Example 4: Take Screenshot\n\n```bash\ncurl -X POST http://localhost:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"mobile_phone_1\",\n \"request\": \"Take a screenshot of the current screen\",\n \"task_name\": \"capture_screen\"\n }'\n```\n\n### Task Execution Flow\n\n```mermaid\nsequenceDiagram\n participant API as HTTP Client\n participant Server as Agent Server\n participant Client as Mobile Client\n participant MCP as MCP Services\n participant Device as Android Device\n \n Note over API,Server: 1. Task Submission\n API->>Server: POST /api/dispatch {client_id, request}\n Server->>Server: Generate session_id\n Server-->>API: {status: dispatched, session_id}\n \n Note over Server,Client: 2. Task Assignment\n Server->>Client: TASK_ASSIGNMENT (via WebSocket)\n Client->>Client: Initialize Mobile Agent\n \n Note over Client,MCP: 3. Data Collection\n Client->>MCP: Capture screenshot\n Client->>MCP: Get installed apps\n Client->>MCP: Get UI controls\n MCP->>Device: ADB commands\n Device-->>MCP: Screenshot + Apps + Controls\n MCP-->>Client: Visual context\n \n Note over Client: 4. LLM Decision\n Client->>Client: Construct prompt with screenshots\n Client->>Client: Get action from LLM\n \n Note over Client,MCP: 5. Action Execution\n Client->>MCP: Execute mobile action (tap, swipe, launch_app, etc.)\n MCP->>Device: ADB input commands\n Device-->>MCP: Action result\n MCP-->>Client: Success/Failure\n \n Note over Client,Server: 6. Result Reporting\n Client->>Server: TASK_RESULT {status, screenshots, actions}\n Server-->>API: Task completed\n```\n\n### Request Parameters\n\n| Field | Required | Type | Description | Example |\n|-------|----------|------|-------------|---------|\n| `client_id` | ✅ Yes | string | Target mobile device ID (must match `--client-id`) | `\"mobile_phone_1\"` |\n| `request` | ✅ Yes | string | Natural language task description | `\"Open Chrome\"` |\n| `task_name` | ❌ Optional | string | Unique task identifier (auto-generated if omitted) | `\"task_001\"` |\n\n> **⚠️ Client Must Be Online:** If the `client_id` is not connected, you'll receive:\n> ```json\n> {\n> \"detail\": \"Client not online\"\n> }\n> ```\n> \n> Verify the client is connected:\n> ```bash\n> curl http://localhost:5001/api/clients\n> ```\n\n---\n\n## 🌌 Step 7: Configure as UFO³ Galaxy Device\n\nTo use the Mobile Agent as a managed device within the **UFO³ Galaxy** multi-tier framework, you need to register it in the `devices.yaml` configuration file.\n\n> **📖 Detailed Guide:** For comprehensive information on using Mobile Agent in Galaxy, including multi-device workflows and advanced configuration, see [Using Mobile Agent as Galaxy Device](../mobile/as_galaxy_device.md).\n\n### Device Configuration File\n\nThe Galaxy configuration is located at:\n\n```\nconfig/galaxy/devices.yaml\n```\n\n### Add Mobile Agent Configuration\n\nEdit `config/galaxy/devices.yaml` and add your Mobile agent:\n\n```yaml\ndevices:\n - device_id: \"mobile_phone_1\"\n server_url: \"ws://localhost:5001/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"messaging\"\n - \"maps\"\n - \"camera\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"13\"\n screen_size: \"1080x2400\"\n installed_apps:\n - \"com.android.chrome\"\n - \"com.google.android.apps.maps\"\n - \"com.whatsapp\"\n description: \"Android phone for mobile automation\"\n auto_connect: true\n max_retries: 5\n```\n\n### Configuration Fields Explained\n\n| Field | Required | Type | Description | Example |\n|-------|----------|------|-------------|---------|\n| `device_id` | ✅ Yes | string | **Must match client `--client-id`** | `\"mobile_phone_1\"` |\n| `server_url` | ✅ Yes | string | **Must match server WebSocket URL** | `\"ws://localhost:5001/ws\"` |\n| `os` | ✅ Yes | string | Operating system | `\"mobile\"` |\n| `capabilities` | ❌ Optional | list | Device capabilities | `[\"mobile\", \"android\"]` |\n| `metadata` | ❌ Optional | dict | Custom metadata | See below |\n| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup | `true` |\n| `max_retries` | ❌ Optional | integer | Connection retry attempts | `5` |\n\n### Metadata Fields (Custom)\n\nThe `metadata` section provides context to the LLM:\n\n| Field | Purpose | Example |\n|-------|---------|---------|\n| `device_type` | Phone, tablet, emulator | `\"phone\"` |\n| `android_version` | OS version | `\"13\"` |\n| `screen_size` | Resolution | `\"1080x2400\"` |\n| `installed_apps` | Available apps | `[\"com.android.chrome\", ...]` |\n| `description` | Human-readable description | `\"Personal phone\"` |\n\n### Multiple Mobile Devices Example\n\n```yaml\ndevices:\n # Personal Phone\n - device_id: \"mobile_phone_personal\"\n server_url: \"ws://192.168.1.100:5001/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"messaging\"\n - \"whatsapp\"\n - \"maps\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"13\"\n installed_apps:\n - \"com.whatsapp\"\n - \"com.google.android.apps.maps\"\n description: \"Personal Android phone\"\n auto_connect: true\n max_retries: 5\n\n # Work Phone\n - device_id: \"mobile_phone_work\"\n server_url: \"ws://192.168.1.101:5002/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"email\"\n - \"teams\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"12\"\n installed_apps:\n - \"com.microsoft.office.outlook\"\n - \"com.microsoft.teams\"\n description: \"Work Android phone\"\n auto_connect: true\n max_retries: 5\n\n # Tablet\n - device_id: \"mobile_tablet_home\"\n server_url: \"ws://192.168.1.102:5003/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"tablet\"\n - \"media\"\n metadata:\n os: \"mobile\"\n device_type: \"tablet\"\n android_version: \"13\"\n screen_size: \"2560x1600\"\n installed_apps:\n - \"com.netflix.mediaclient\"\n description: \"Home tablet for media\"\n auto_connect: true\n max_retries: 5\n```\n\n### Critical Requirements\n\n> **⚠️ Configuration Validation - These fields MUST match exactly:**\n> \n> 1. **`device_id` in YAML** ↔ **`--client-id` in client command**\n> 2. **`server_url` in YAML** ↔ **`--ws-server` in client command**\n> \n> **If these don't match, Galaxy cannot control the device!**\n\n### Using Galaxy to Control Mobile Agents\n\nOnce configured, launch Galaxy:\n\n```bash\npython -m galaxy --interactive\n```\n\n**Galaxy will:**\n1. ✅ Load device configuration from `config/galaxy/devices.yaml`\n2. ✅ Connect to all configured Android devices\n3. ✅ Orchestrate multi-device tasks\n4. ✅ Route tasks based on capabilities\n\n> **ℹ️ Galaxy Documentation:** For detailed Galaxy usage, see:\n> \n> - [Galaxy Overview](../galaxy/overview.md)\n> - [Galaxy Quick Start](quick_start_galaxy.md)\n> - [Mobile Agent as Galaxy Device](../mobile/as_galaxy_device.md)\n\n---\n\n## 🔍 Understanding Mobile Agent Internals\n\nNow that you have Mobile Agent running, you may want to understand how it works under the hood:\n\n### State Machine\n\nMobile Agent uses a **3-state finite state machine** to manage task execution:\n\n- **CONTINUE** - Active execution, processing user requests\n- **FINISH** - Task completed successfully\n- **FAIL** - Unrecoverable error occurred\n\nLearn more: [State Machine Documentation](../mobile/state.md)\n\n### Processing Pipeline\n\nDuring the CONTINUE state, Mobile Agent executes a **4-phase pipeline**:\n\n1. **Data Collection** - Capture screenshots, get apps, collect UI controls\n2. **LLM Interaction** - Send visual context to LLM for decision making\n3. **Action Execution** - Execute mobile actions (tap, swipe, type, etc.)\n4. **Memory Update** - Record actions and results for context\n\nLearn more: [Processing Strategy Documentation](../mobile/strategy.md)\n\n### Available Commands\n\nMobile Agent uses **13 MCP commands** across two servers:\n\n- **Data Collection Server (8020)**: 5 read-only commands\n- **Action Server (8021)**: 8 control commands\n\nLearn more: [MCP Commands Reference](../mobile/commands.md)\n\n---\n\n## 🐛 Common Issues & Troubleshooting\n\n### Issue 1: ADB Device Not Found\n\n**Error: No Devices Detected**\n\nSymptoms:\n```bash\n$ adb devices\nList of devices attached\n# Empty list\n```\n\n**Solutions:**\n\n**For Physical Devices:**\n\n1. **Check USB connection:**\n - Use a different USB cable (some cables are charge-only)\n - Try a different USB port on your computer\n - Ensure USB debugging is enabled on device\n\n2. **Authorize computer on device:**\n - Disconnect and reconnect USB\n - On device, tap \"Allow USB debugging\" when prompted\n - Check \"Always allow from this computer\"\n\n3. **Restart ADB server:**\n ```bash\n adb kill-server\n adb start-server\n adb devices\n ```\n\n4. **Check USB driver (Windows):**\n - Install Google USB Driver via Android Studio SDK Manager\n - Or install device-specific driver from manufacturer\n\n**For Emulators:**\n\n1. **Wait for emulator to fully boot** (can take 1-2 minutes)\n\n2. **Restart emulator:**\n - Close emulator completely\n - Start emulator again from Android Studio or command line\n\n3. **Check emulator is running:**\n ```bash\n emulator -list-avds\n emulator -avd Pixel_6_API_33\n ```\n\n### Issue 2: MCP Server Cannot Connect to Device\n\n**Error: ADB Connection Failed**\n\nSymptoms:\n```log\nERROR - Failed to execute ADB command\nERROR - Device not accessible\n```\n\n**Solutions:**\n\n1. **Verify ADB connection first:**\n ```bash\n adb devices\n ```\n Device should show \"device\" status (not \"offline\" or \"unauthorized\")\n\n2. **Test ADB commands manually:**\n ```bash\n adb shell getprop ro.product.model\n adb shell screencap -p /sdcard/test.png\n ```\n\n3. **Restart MCP servers with debug output:**\n ```bash\n # Kill existing servers\n pkill -f mobile_mcp_server\n \n # Start with explicit ADB path\n python -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --adb-path $(which adb) \\\n --server both\n ```\n\n4. **Check device permissions:**\n - Ensure USB debugging is still authorized\n - Revoke and re-grant USB debugging authorization on device\n\n### Issue 3: Client Cannot Connect to Server\n\n**Error: Connection Refused or Failed**\n\nSymptoms:\n```log\nERROR - [WS] Failed to connect to ws://localhost:5001/ws\nConnection refused\n```\n\n**Solutions:**\n\n1. **Verify server is running:**\n ```bash\n curl http://localhost:5001/api/health\n ```\n \n Should return:\n ```json\n {\n \"status\": \"healthy\",\n \"online_clients\": []\n }\n ```\n\n2. **Check server address:**\n - If server and client are on different machines, use server's IP address\n - Replace `localhost` with actual IP address (e.g., `ws://192.168.1.100:5001/ws`)\n - Ensure the port number matches the server's `--port` argument\n\n3. **Check firewall settings:**\n ```bash\n # Windows: Allow port 5001\n netsh advfirewall firewall add rule name=\"UFO Server\" dir=in action=allow protocol=TCP localport=5001\n \n # macOS: System Preferences → Security & Privacy → Firewall → Firewall Options\n \n # Linux (Ubuntu):\n sudo ufw allow 5001/tcp\n ```\n\n### Issue 4: Missing `--platform mobile` Flag\n\n**Error: Incorrect Agent Type**\n\nSymptoms:\n- Client connects but cannot execute mobile commands\n- Server logs show wrong platform type\n- Tasks fail with \"unsupported operation\" errors\n\n**Solution:**\n\nAlways include `--platform mobile` when starting the client:\n\n```bash\n# Wrong (missing platform)\npython -m ufo.client.client --ws --client-id mobile_phone_1\n\n# Correct\npython -m ufo.client.client \\\n --ws \\\n --client-id mobile_phone_1 \\\n --platform mobile\n```\n\n### Issue 5: Screenshot Capture Fails\n\n**Error: Cannot Capture Screenshot**\n\nSymptoms:\n```log\nERROR - Failed to capture screenshot\nERROR - screencap command failed\n```\n\n**Solutions:**\n\n1. **Test screenshot manually:**\n ```bash\n adb shell screencap -p /sdcard/test.png\n adb pull /sdcard/test.png .\n ```\n\n2. **Check device storage:**\n ```bash\n adb shell df -h /sdcard\n ```\n Ensure sufficient space on device\n\n3. **Check permissions:**\n ```bash\n adb shell ls -l /sdcard\n ```\n\n4. **Try alternative screenshot method:**\n ```bash\n adb exec-out screencap -p > screenshot.png\n ```\n\n### Issue 6: UI Controls Not Found\n\n**Error: Control Information Missing**\n\nSymptoms:\n```log\nWARNING - Failed to get UI controls\nWARNING - UI tree dump failed\n```\n\n**Solutions:**\n\n1. **Test UI dump manually:**\n ```bash\n adb shell uiautomator dump /sdcard/window_dump.xml\n adb shell cat /sdcard/window_dump.xml\n ```\n\n2. **Enable accessibility services:**\n - Some apps require accessibility services for UI automation\n - Settings → Accessibility → Enable required services\n\n3. **Update Android WebView:**\n - Old WebView versions may cause UI dump issues\n - Update via Play Store: Android System WebView\n\n4. **Restart device:**\n ```bash\n adb reboot\n # Wait for device to restart\n adb wait-for-device\n ```\n\n### Issue 7: Emulator Too Slow\n\n**Error: Performance Issues**\n\nSymptoms:\n- Emulator lags or freezes\n- Actions take very long to execute\n- Timeouts occur frequently\n\n**Solutions:**\n\n1. **Enable Hardware Acceleration:**\n - **Windows:** Ensure Hyper-V or Intel HAXM is enabled\n - **macOS:** Hypervisor.framework is used automatically\n - **Linux:** Install KVM\n\n2. **Allocate More Resources:**\n - In Android Studio AVD Manager, edit AVD\n - Increase RAM to 2048 MB or higher\n - Increase VM heap to 512 MB\n - Set Graphics to \"Hardware - GLES 2.0\"\n\n3. **Use x86_64 System Image:**\n - Faster than ARM images\n - Download x86_64 image in SDK Manager\n\n4. **Reduce Screen Resolution:**\n - Edit AVD settings\n - Choose lower resolution (e.g., 720x1280 instead of 1080x2400)\n\n### Issue 8: Multiple Devices Connected\n\n**Error: More Than One Device**\n\nSymptoms:\n```bash\n$ adb devices\nList of devices attached\nemulator-5554 device\n192.168.1.100:5555 device\n```\n\n**Solutions:**\n\n1. **Specify device for ADB:**\n ```bash\n # Use emulator\n export ANDROID_SERIAL=emulator-5554\n \n # Use physical device\n export ANDROID_SERIAL=192.168.1.100:5555\n ```\n\n2. **Disconnect other devices:**\n ```bash\n # Disconnect wireless device\n adb disconnect 192.168.1.100:5555\n ```\n\n3. **Run separate MCP servers:**\n ```bash\n # Server for emulator\n ANDROID_SERIAL=emulator-5554 python -m ufo.client.mcp.http_servers.mobile_mcp_server --data-port 8020 --action-port 8021 --server both\n \n # Server for physical device\n ANDROID_SERIAL=192.168.1.100:5555 python -m ufo.client.mcp.http_servers.mobile_mcp_server --data-port 8022 --action-port 8023 --server both\n ```\n\n---\n\n## 📚 Next Steps\n\nYou've successfully set up a Mobile Agent! Explore these topics to deepen your understanding:\n\n### Immediate Next Steps\n\n| Priority | Topic | Time | Link |\n|----------|-------|------|------|\n| 🥇 | **Mobile Agent Architecture** | 10 min | [Overview](../mobile/overview.md) |\n| 🥈 | **State Machine & Processing** | 15 min | [State Machine](../mobile/state.md) |\n| 🥉 | **MCP Commands Reference** | 15 min | [Commands](../mobile/commands.md) |\n\n### Advanced Topics\n\n| Topic | Description | Link |\n|-------|-------------|------|\n| **Processing Strategy** | 4-phase pipeline (Data, LLM, Action, Memory) | [Strategy](../mobile/strategy.md) |\n| **Galaxy Integration** | Multi-device orchestration with UFO³ | [As Galaxy Device](../mobile/as_galaxy_device.md) |\n| **MCP Protocol Details** | Deep dive into mobile interaction protocol | [Commands](../mobile/commands.md) |\n\n### Production Deployment\n\n| Best Practice | Description |\n|---------------|-------------|\n| **Persistent ADB** | Keep ADB connection stable for physical devices |\n| **Emulator Management** | Automate emulator lifecycle (start/stop/reset) |\n| **Screenshot Storage** | Configure log paths and cleanup policies in `config/ufo/system.yaml` |\n| **Security** | Use secure WebSocket (wss://) for remote deployments |\n\n> **💡 Learn More:** For comprehensive understanding of the Mobile Agent architecture and processing flow, see the [Mobile Agent Overview](../mobile/overview.md).\n\n---\n\n## ✅ Summary\n\nCongratulations! You've successfully:\n\n✅ Set up Android device (physical or emulator) \n✅ Installed ADB (Android Debug Bridge) \n✅ Installed Python dependencies \n✅ Started the Device Agent Server \n✅ Launched MCP services (data collection + action) \n✅ Connected Mobile Device Agent Client \n✅ Dispatched mobile automation tasks via HTTP API \n✅ (Optional) Configured device in Galaxy \n\n**Your Mobile Agent is Ready**\n\nYou can now:\n\n- 📱 Automate Android apps remotely\n- 🖼️ Capture and analyze screenshots\n- 🎯 Interact with UI controls precisely\n- 🌌 Integrate with UFO³ Galaxy for cross-platform workflows\n\n**Start exploring mobile automation!** 🚀\n\n---\n\n## 💡 Pro Tips\n\n### Quick Start Command Summary\n\n**Start everything in order:**\n\n```bash\n# Terminal 1: Start server\npython -m ufo.server.app --port 5001 --platform mobile\n\n# Terminal 2: Start MCP services\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server both\n\n# Terminal 3: Start client\npython -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id mobile_phone_1 --platform mobile\n\n# Terminal 4: Dispatch task\ncurl -X POST http://localhost:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\"client_id\": \"mobile_phone_1\", \"request\": \"Open Chrome browser\"}'\n```\n\n### Development Shortcuts\n\n**Create shell scripts for common operations:**\n\n**Windows (PowerShell):**\n```powershell\n# start-mobile-agent.ps1\nStart-Process powershell -ArgumentList \"-NoExit\", \"-Command\", \"python -m ufo.server.app --port 5001 --platform mobile\"\nStart-Sleep 2\nStart-Process powershell -ArgumentList \"-NoExit\", \"-Command\", \"python -m ufo.client.mcp.http_servers.mobile_mcp_server --server both\"\nStart-Sleep 2\nStart-Process powershell -ArgumentList \"-NoExit\", \"-Command\", \"python -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id mobile_phone_1 --platform mobile\"\n```\n\n**macOS/Linux (Bash):**\n```bash\n#!/bin/bash\n# start-mobile-agent.sh\n\n# Start server in background\npython -m ufo.server.app --port 5001 --platform mobile &\nsleep 2\n\n# Start MCP services in background\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server both &\nsleep 2\n\n# Start client in foreground\npython -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id mobile_phone_1 --platform mobile\n```\n\nMake executable:\n```bash\nchmod +x start-mobile-agent.sh\n./start-mobile-agent.sh\n```\n\n### Testing Your Setup\n\n**Quick test to verify everything works:**\n\n```bash\n# Test 1: Check ADB\nadb devices\n# Should show your device\n\n# Test 2: Check Server\ncurl http://localhost:5001/api/health\n# Should return {\"status\": \"healthy\"}\n\n# Test 3: Check MCP\ncurl http://localhost:8020/health\ncurl http://localhost:8021/health\n# Should return health status\n\n# Test 4: Check Client\ncurl http://localhost:5001/api/clients\n# Should show mobile_phone_1\n\n# Test 5: Dispatch simple task\ncurl -X POST http://localhost:5001/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\"client_id\": \"mobile_phone_1\", \"request\": \"Take a screenshot\"}'\n# Should return dispatched status\n```\n\n**Happy Mobile Automation! 🎉**\n"
},
{
"path": "documents/docs/getting_started/quick_start_ufo2.md",
"content": "# Quick Start Guide\n\nWelcome to **UFO²** – the Desktop AgentOS! This guide will help you get started with UFO² in just a few minutes.\n\n**What is UFO²?**\n\nUFO² is a Desktop AgentOS that turns natural-language requests into automatic, reliable, multi-application workflows on Windows. It goes beyond UI-focused automation by combining GUI actions with native API calls for faster and more robust execution.\n\n---\n\n## 🛠️ Step 1: Installation\n\n### Requirements\n\n- **Python** >= 3.10\n- **Windows OS** >= 10\n- **Git** (for cloning the repository)\n\n### Installation Steps\n\n```powershell\n# [Optional] Create conda environment\nconda create -n ufo python=3.10\nconda activate ufo\n\n# Clone the repository\ngit clone https://github.com/microsoft/UFO.git\ncd UFO\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n> **💡 Tip:** If you want to use Qwen as your LLM, uncomment the related libraries in `requirements.txt` before installing.\n\n---\n\n---\n\n## ⚙️ Step 2: Configure LLMs\n\n> **📢 New Configuration System (Recommended)** \n> UFO² now uses a **new modular config system** located in `config/ufo/` with auto-discovery and type validation. While the legacy `ufo/config/config.yaml` is still supported for backward compatibility, we strongly recommend migrating to the new system for better maintainability.\n\n### Option 1: New Config System (Recommended)\n\nThe new config files are organized in `config/ufo/` with separate YAML files for different components:\n\n```powershell\n# Copy template to create your agent config file (contains API keys)\ncopy config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\nnotepad config\\ufo\\agents.yaml # Edit your LLM API credentials\n```\n\n**Directory Structure:**\n```\nconfig/ufo/\n├── agents.yaml.template # Template: Agent configs (HOST_AGENT, APP_AGENT) - COPY & EDIT THIS\n├── agents.yaml # Your agent configs with API keys (DO NOT commit to git)\n├── rag.yaml # RAG and knowledge settings (default values, edit if needed)\n├── system.yaml # System settings (default values, edit if needed)\n├── mcp.yaml # MCP integration settings (default values, edit if needed)\n└── ... # Other modular configs with defaults\n```\n\n> **Configuration Files:** `agents.yaml` contains sensitive information (API keys) and must be configured. Other config files have default values and only need editing for customization.\n\n**Migration Benefits:**\n\n- ✅ **Type Safety**: Automatic validation with Pydantic schemas\n- ✅ **Auto-Discovery**: No manual config loading needed\n- ✅ **Modular**: Separate concerns into individual files\n- ✅ **IDE Support**: Better autocomplete and error detection\n\n### Option 2: Legacy Config (Backward Compatible)\n\nFor existing users, the old config path still works:\n\n```powershell\ncopy ufo\\config\\config.yaml.template ufo\\config\\config.yaml\nnotepad ufo\\config\\config.yaml # Paste your key & endpoint\n```\n\n> **Config Precedence:** If both old and new configs exist, the new config in `config/ufo/` takes precedence. A warning will be displayed during startup.\n\n---\n\n### LLM Configuration Examples\n\n#### OpenAI Configuration\n\n**New Config (`config/ufo/agents.yaml`):**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: true\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\" # Replace with your actual API key\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-4o\"\n\nAPP_AGENT:\n VISUAL_MODE: true\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\" # Replace with your actual API key\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-4o\"\n```\n\n**Legacy Config (`ufo/config/config.yaml`):**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: True\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o\"\n```\n\n#### Azure OpenAI (AOAI) Configuration\n\n**New Config (`config/ufo/agents.yaml`):**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: true\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o\"\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n\nAPP_AGENT:\n VISUAL_MODE: true\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-4o\"\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n```\n\n> **ℹ️ More LLM Options:** UFO² supports various LLM providers including Qwen, Gemini, Claude, DeepSeek, and more. See the [Model Configuration Guide](../configuration/models/overview.md) for complete details.\n\n---\n\n---\n\n## 📔 Step 3: Additional Settings (Optional)\n\n### RAG Configuration\n\nEnhance UFO's capabilities with external knowledge through Retrieval Augmented Generation (RAG):\n\n**For New Config**: Edit `config/ufo/rag.yaml` (already exists with default values) \n**For Legacy Config**: Edit `ufo/config/config.yaml`\n\n**Available RAG Options:**\n\n| Feature | Documentation | Description |\n|---------|--------------|-------------|\n| **Offline Help Documents** | [Learning from Help Documents](../ufo2/core_features/knowledge_substrate/learning_from_help_document.md) | Retrieve information from offline help documentation |\n| **Online Bing Search** | [Learning from Bing Search](../ufo2/core_features/knowledge_substrate/learning_from_bing_search.md) | Utilize up-to-date online search results |\n| **Self-Experience** | [Experience Learning](../ufo2/core_features/knowledge_substrate/experience_learning.md) | Save task trajectories into memory for future reference |\n| **User Demonstrations** | [Learning from Demonstrations](../ufo2/core_features/knowledge_substrate/learning_from_demonstration.md) | Learn from user-provided demonstrations |\n\n**Example RAG Config (`config/ufo/rag.yaml`):**\n```yaml\n# Enable Bing search\nRAG_ONLINE_SEARCH: true\nBING_API_KEY: \"YOUR_BING_API_KEY\" # Get from https://www.microsoft.com/en-us/bing/apis\n\n# Enable experience learning\nRAG_EXPERIENCE: true\n```\n\n> **ℹ️ RAG Resources:** See [Knowledge Substrate Overview](../ufo2/core_features/knowledge_substrate/overview.md) for complete RAG configuration and best practices.\n\n---\n\n---\n\n## 🎉 Step 4: Start UFO²\n\n### Interactive Mode\n\nStart UFO² in interactive mode where you can enter requests dynamically:\n\n```powershell\n# Assume you are in the cloned UFO folder\npython -m ufo --task \n```\n\n**Expected Output:**\n```\nWelcome to use UFO🛸, A UI-focused Agent for Windows OS Interaction. \n _ _ _____ ___\n| | | || ___| / _ \\\n| | | || |_ | | | |\n| |_| || _| | |_| |\n \\___/ |_| \\___/\nPlease enter your request to be completed🛸:\n```\n\n### Direct Request Mode\n\nInvoke UFO² with a specific task and request directly:\n\n```powershell\npython -m ufo --task -r \"\"\n```\n\n**Example:**\n```powershell\npython -m ufo --task email_demo -r \"Send an email to john@example.com with subject 'Meeting Reminder'\"\n```\n\n---\n\n\n---\n\n## 🎥 Step 5: Execution Logs\n\nUFO² automatically saves execution logs, screenshots, and traces for debugging and analysis.\n\n**Log Location:**\n```\n./logs//\n```\n\n**Log Contents:**\n\n| File/Folder | Description |\n|-------------|-------------|\n| `screenshots/` | Screenshots captured during execution |\n| `action_*.json` | Agent actions and responses |\n| `ui_trees/` | UI control tree snapshots (if enabled) |\n| `request_response.log` | Complete LLM request/response logs |\n\n> **Analyzing Logs:** Use the logs to debug agent behavior, replay execution flow, and analyze agent decision-making patterns.\n\n> **Privacy Notice:** Screenshots may contain sensitive or confidential information. Ensure no private data is visible during execution. See [DISCLAIMER.md](https://github.com/microsoft/UFO/blob/main/DISCLAIMER.md) for details.\n\n---\n\n## 🔄 Migrating from Legacy Config\n\nIf you're upgrading from an older version that used `ufo/config/config.yaml`, UFO² provides an **automated conversion tool**.\n\n### Automatic Conversion (Recommended)\n\n```powershell\n# Interactive conversion with automatic backup\npython -m ufo.tools.convert_config\n\n# Preview changes first (dry run)\npython -m ufo.tools.convert_config --dry-run\n\n# Force conversion without confirmation\npython -m ufo.tools.convert_config --force\n```\n\n**What the tool does:**\n\n- ✅ Splits monolithic `config.yaml` into modular files\n- ✅ Converts flow-style YAML (with braces) to block-style YAML\n- ✅ Maps legacy file names to new structure\n- ✅ Preserves all configuration values\n- ✅ Creates timestamped backup for rollback\n- ✅ Validates output files\n\n**Conversion Mapping:**\n\n| Legacy File | → | New File(s) | Transformation |\n|-------------|---|-------------|----------------|\n| `config.yaml` (monolithic) | → | `agents.yaml` + `rag.yaml` + `system.yaml` | Smart field splitting |\n| `agent_mcp.yaml` | → | `mcp.yaml` | Rename + format conversion |\n| `config_prices.yaml` | → | `prices.yaml` | Rename + format conversion |\n\n> **Migration Guide:** For detailed migration instructions, rollback procedures, and troubleshooting, see the [Configuration Migration Guide](../configuration/system/migration.md).\n\n---\n\n## 📚 Additional Resources\n\n### Core Documentation\n\n**Architecture & Concepts:**\n\n- [UFO² Overview](../ufo2/overview.md) - System architecture and design principles\n- [HostAgent](../ufo2/host_agent/overview.md) - Desktop-level coordination agent\n- [AppAgent](../ufo2/app_agent/overview.md) - Application-level execution agent\n\n### Configuration\n\n**Configuration Guides:**\n\n- [Configuration Overview](../configuration/system/overview.md) - Configuration system architecture\n- [Agents Configuration](../configuration/system/agents_config.md) - LLM and agent settings\n- [System Configuration](../configuration/system/system_config.md) - Runtime and execution settings\n- [MCP Configuration](../configuration/system/mcp_reference.md) - MCP server settings\n- [Model Configuration](../configuration/models/overview.md) - Supported LLM providers\n\n### Advanced Features\n\n**Advanced Topics:**\n\n- [Hybrid Actions](../ufo2/core_features/hybrid_actions.md) - GUI + API automation\n- [Control Detection](../ufo2/core_features/control_detection/overview.md) - UIA + Vision detection\n- [Knowledge Substrate](../ufo2/core_features/knowledge_substrate/overview.md) - RAG and learning\n- [Multi-Action Execution](../ufo2/core_features/multi_action.md) - Speculative action batching\n\n### Evaluation & Benchmarks\n\n**Benchmarking:**\n\n- [Benchmark Overview](../ufo2/evaluation/benchmark/overview.md) - Evaluation framework and datasets\n- [Windows Agent Arena](../ufo2/evaluation/benchmark/windows_agent_arena.md) - 154 real Windows tasks\n- [OSWorld](../ufo2/evaluation/benchmark/osworld.md) - Cross-application benchmarks\n\n---\n\n## ❓ Getting Help\n\n- 📖 **Documentation**: [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)\n- 🐛 **GitHub Issues**: [https://github.com/microsoft/UFO/issues](https://github.com/microsoft/UFO/issues) (preferred)\n- 📧 **Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)\n\n---\n\n## 🎯 Next Steps\n\nNow that UFO² is set up, explore these guides to unlock its full potential:\n\n1. **[Configuration Customization](../configuration/system/overview.md)** - Fine-tune UFO² behavior\n2. **[Knowledge Substrate Setup](../ufo2/core_features/knowledge_substrate/overview.md)** - Enable RAG capabilities\n3. **[Creating Custom Agents](../tutorials/creating_app_agent/overview.md)** - Build specialized agents\n4. **[MCP Integration](../mcp/overview.md)** - Extend with custom MCP servers\n\nHappy automating with UFO²! 🛸"
},
{
"path": "documents/docs/index.md",
"content": "# Welcome to UFO³ Documentation\n\n
\n
\n UFO³ : Weaving the Digital Agent Galaxy\n
\n
A Multi-Device Orchestration Framework for Cross-Platform Intelligent Automation
\n\n\n## 📖 About This Documentation\n\nWelcome to the official documentation for **UFO³**, Microsoft's open-source framework for intelligent automation across devices and platforms. Whether you're looking to automate Windows applications or orchestrate complex workflows across multiple devices, this documentation will guide you through every step.\n\n**What you'll find here:**\n\n- 🚀 **[Quick Start Guides](getting_started/quick_start_galaxy.md)** – Get up and running in minutes\n- 📚 **[Core Concepts](galaxy/overview.md)** – Understand the architecture and key components \n- ⚙️ **[Configuration](configuration/system/agents_config.md)** – Set up your agents and models\n- 🔧 **[Advanced Features](ufo2/core_features/multi_action.md)** – Deep dive into powerful capabilities\n- 💡 **[FAQ](faq.md)** – Common questions and troubleshooting\n\n---\n\n## 🎯 Choose Your Path\n\nUFO³ consists of two complementary frameworks. Choose the one that best fits your needs, or use both together!\n\n| Framework | Best For | Key Strength | Get Started |\n|-----------|----------|--------------|-------------|\n| **🌌 Galaxy** ✨ NEW & RECOMMENDED | Cross-device workflows Complex automation Parallel execution | Multi-device orchestration DAG-based planning Real-time monitoring | [Quick Start →](getting_started/quick_start_galaxy.md) |\n| **🪟 UFO²** ⚡ STABLE & LTS | Windows automation Quick tasks Learning basics | Deep Windows integration Hybrid GUI + API Stable & reliable | [Quick Start →](getting_started/quick_start_ufo2.md) |\n\n### 🤔 Decision Guide\n\n| Question | Galaxy | UFO² |\n|----------|:------:|:----:|\n| Need cross-device collaboration? | ✅ | ❌ |\n| Complex multi-step workflows? | ✅ | ⚠️ Limited |\n| Windows-only automation? | ✅ | ✅ Optimized |\n| Quick setup & learning? | ⚠️ Moderate | ✅ Easy |\n| Stable & reliable? | 🚧 Active Dev | ✅ LTS |\n\n---\n\n## 🌟 What's New in UFO³?\n\n**UFO³ is a scalable, universal cross-device agent framework** that enables you to develop new device agents for different platforms and applications. Through the **Agent Interaction Protocol (AIP)**, custom device agents can seamlessly integrate into UFO³ Galaxy for coordinated multi-device orchestration.\n\n### Evolution Timeline\n\n```mermaid\n%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E8F4F8','primaryTextColor':'#1A1A1A','primaryBorderColor':'#7CB9E8','lineColor':'#A8D5E2','secondaryColor':'#B8E6F0','tertiaryColor':'#D4F1F4','fontSize':'16px','fontFamily':'Segoe UI, Arial, sans-serif'}}}%%\ngraph LR\n A[\"🎈 UFO February 2024 GUI Agent for Windows\"] \n B[\"🖥️ UFO² April 2025 Desktop AgentOS\"]\n C[\"🌌 UFO³ Galaxy November 2025 Multi-Device Orchestration\"]\n \n A -->|Evolve| B\n B -->|Scale| C\n \n style A fill:#E8F4F8,stroke:#7CB9E8,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15\n style B fill:#C5E8F5,stroke:#5BA8D0,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15\n style C fill:#A4DBF0,stroke:#3D96BE,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15\n```\n\n### 🚀 UFO³ = **Galaxy** (Multi-Device Orchestration) + **UFO²** (Device Agent)\n\nUFO³ introduces **Galaxy**, a revolutionary multi-device orchestration framework that coordinates intelligent agents across heterogeneous platforms. Built on five tightly integrated design principles:\n\n1. **🌟 Declarative Decomposition into Dynamic DAG** - Requests decomposed into structured DAG with TaskStars and dependencies for automated scheduling and runtime rewriting\n\n2. **🔄 Continuous Result-Driven Graph Evolution** - Living constellation that adapts to execution feedback through controlled rewrites and dynamic adjustments\n\n3. **⚡ Heterogeneous, Asynchronous & Safe Orchestration** - Capability-based device matching with async execution, safe locking, and formally verified correctness\n\n4. **🔌 Unified Agent Interaction Protocol (AIP)** - WebSocket-based secure coordination layer with fault tolerance and automatic reconnection\n\n5. **🛠️ Template-Driven MCP-Empowered Device Agents** - Lightweight toolkit for rapid agent development with MCP integration for tool augmentation\n\n| Aspect | UFO² | UFO³ Galaxy |\n|--------|------|-------------|\n| **Architecture** | Single Windows Agent | Multi-Device Orchestration |\n| **Task Model** | Sequential ReAct Loop | DAG-based Constellation Workflows |\n| **Scope** | Single device, multi-app | Multi-device, cross-platform |\n| **Coordination** | HostAgent + AppAgents | ConstellationAgent + TaskOrchestrator |\n| **Device Support** | Windows Desktop | Windows, Linux, macOS, Android, Web |\n| **Task Planning** | Application-level | Device-level with dependencies |\n| **Execution** | Sequential | Parallel DAG execution |\n| **Device Agent Role** | Standalone | Can serve as Galaxy device agent |\n| **Complexity** | Simple to Moderate | Simple to Very Complex |\n| **Learning Curve** | Low | Moderate |\n| **Cross-Device Collaboration** | ❌ Not Supported | ✅ Core Feature |\n| **Setup Difficulty** | ✅ Easy | ⚠️ Moderate |\n| **Status** | ✅ LTS (Long-Term Support) | ⚡ Active Development |\n\n### 🎓 Migration Path\n\n**For UFO² Users:**\n1. ✅ **Keep using UFO²** – Fully supported, actively maintained\n2. 🔄 **Gradual adoption** – Galaxy can use UFO² as Windows device agent\n3. 📈 **Scale up** – Move to Galaxy when you need multi-device capabilities\n4. 📚 **Learning resources** – [Migration Guide](./getting_started/migration_ufo2_to_galaxy.md)\n\n---\n\n## ✨ Capabilities at a Glance\n\n### 🌌 Galaxy Framework – What's Different?\n\n#### 🌟 Constellation Planning\n\n```\nUser Request\n ↓\nConstellationAgent\n ↓\n [Task DAG]\n / | \\\nTask1 Task2 Task3\n(Win) (Linux)(Mac)\n```\n\n**Benefits:**\n- Cross-device dependency tracking\n- Parallel execution optimization\n- Cross-device dataflow management\n\n#### 🎯 Device Assignment\n\n```\nSelection Criteria\n • Platform\n • Resource\n • Task requirements\n • Performance history\n ↓\n Auto-Assignment\n ↓\n Optimal Devices\n```\n\n**Smart Matching:**\n- Capability-based selection\n- Real-time resource monitoring\n- Dynamic reallocation\n\n#### 📊 Orchestration\n\n```\nTask1 → Running ✅\nTask2 → Pending ⏸️\nTask3 → Running 🔄\n ↓\n Completion\n ↓\n Final Report\n```\n\n**Orchestration:**\n- Real-time status updates\n- Automatic error recovery\n- Progress tracking with feedback\n\n---\n\n### 🪟 UFO² Desktop AgentOS – Core Strengths\n\nUFO² serves dual roles: **standalone Windows automation** and **Galaxy device agent** for Windows platforms.\n\n| Feature | Description | Documentation |\n|---------|-------------|---------------|\n| **Deep OS Integration** | Windows UIA, Win32, WinCOM native control | [Learn More](ufo2/overview.md) |\n| **Hybrid Actions** | GUI clicks + API calls for optimal performance | [Learn More](ufo2/core_features/hybrid_actions.md) |\n| **Speculative Multi-Action** | Batch predictions → **51% fewer LLM calls** | [Learn More](ufo2/core_features/multi_action.md) |\n| **Visual + UIA Detection** | Hybrid control detection for robustness | [Learn More](ufo2/core_features/control_detection/hybrid_detection.md) |\n| **Knowledge Substrate** | RAG with docs, demos, execution traces | [Learn More](ufo2/core_features/knowledge_substrate/overview.md) |\n| **Device Agent Role** | Can serve as Windows executor in Galaxy orchestration | [Learn More](galaxy/overview.md) |\n\n**As Galaxy Device Agent:**\n- Receives tasks from ConstellationAgent through Galaxy orchestration layer\n- Executes Windows-specific operations using proven UFO² capabilities\n- Reports status and results back to TaskOrchestrator\n- Seamlessly participates in cross-device workflows\n\n---\n\n## 🏗️ Architecture\n\n### UFO³ Galaxy – Multi-Device Orchestration\n\n
\n \n
\n\n| Component | Role |\n|-----------|------|\n| **ConstellationAgent** | Plans and decomposes tasks into DAG workflows |\n| **TaskConstellation** | DAG representation with TaskStar nodes and dependencies |\n| **Device Pool Manager** | Matches tasks to capable devices dynamically |\n| **TaskOrchestrator** | Coordinates parallel execution and handles data flow |\n| **Event System** | Real-time monitoring with observer pattern |\n\n[📖 Learn More →](galaxy/overview.md)\n\n### UFO² – Desktop AgentOS\n\n
\n \n
\n\n| Component | Role |\n|-----------|------|\n| **HostAgent** | Desktop orchestrator, application lifecycle management |\n| **AppAgents** | Per-application executors with hybrid GUI–API actions |\n| **Knowledge Substrate** | RAG-enhanced learning from docs & execution history |\n| **Speculative Executor** | Multi-action prediction for efficiency |\n\n[📖 Learn More →](ufo2/overview.md)\n\n---\n\n## 🚀 Quick Start\n\nReady to dive in? Follow these guides to get started with your chosen framework:\n\n### 🌌 Galaxy Quick Start (Multi-Device Orchestration)\n\nPerfect for complex workflows across multiple devices and platforms.\n\n```bash\n# 1. Install dependencies\npip install -r requirements.txt\n\n# 2. Configure agents (see detailed guide for API key setup)\ncopy config\\galaxy\\agent.yaml.template config\\galaxy\\agent.yaml\ncopy config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n\n# 3. Start device agents\npython -m ufo.server.app --port 5000\npython -m ufo.client.client --ws --ws-server ws://localhost:5000/ws --client-id device_1 --platform windows\n\n# 4. Launch Galaxy\npython -m galaxy --interactive\n```\n\n**📖 [Complete Galaxy Quick Start Guide →](getting_started/quick_start_galaxy.md)** \n**⚙️ [Galaxy Configuration Details →](configuration/system/galaxy_devices.md)**\n\n### 🪟 UFO² Quick Start (Windows Automation)\n\nPerfect for Windows-only automation tasks with quick setup.\n\n```bash\n# 1. Install\npip install -r requirements.txt\n\n# 2. Configure (add your API keys)\ncopy config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n\n# 3. Run\npython -m ufo --task \n```\n\n**📖 [Complete UFO² Quick Start Guide →](getting_started/quick_start_ufo2.md)** \n**⚙️ [UFO² Configuration Details →](configuration/system/agents_config.md)**\n\n---\n\n## 📚 Documentation Navigation\n\n### 🎯 Getting Started\n\nStart here if you're new to UFO³:\n\n| Guide | Description | Framework |\n|-------|-------------|-----------|\n| [Galaxy Quick Start](getting_started/quick_start_galaxy.md) | Set up multi-device orchestration in 10 minutes | 🌌 Galaxy |\n| [UFO² Quick Start](getting_started/quick_start_ufo2.md) | Start automating Windows in 5 minutes | 🪟 UFO² |\n| [Linux Agent Quick Start](getting_started/quick_start_linux.md) | Automate Linux systems | 🐧 Linux |\n| [Mobile Agent Quick Start](getting_started/quick_start_mobile.md) | Automate Android devices via ADB | 📱 Mobile |\n| [Choosing Your Path](choose_path.md) | Decision guide for selecting the right framework | Both |\n\n### 🏗️ Core Architecture\n\nUnderstand how UFO³ works under the hood:\n\n| Topic | Description | Framework |\n|-------|-------------|-----------|\n| [Galaxy Overview](galaxy/overview.md) | Multi-device orchestration architecture | 🌌 Galaxy |\n| [UFO² Overview](ufo2/overview.md) | Desktop AgentOS architecture and concepts | 🪟 UFO² |\n| [Task Constellation](galaxy/constellation/overview.md) | DAG-based workflow representation | 🌌 Galaxy |\n| [ConstellationAgent](galaxy/constellation_agent/overview.md) | Intelligent task planner and decomposer | 🌌 Galaxy |\n| [Task Orchestrator](galaxy/constellation_orchestrator/overview.md) | Execution engine and coordinator | 🌌 Galaxy |\n| [AIP Protocol](aip/overview.md) | Agent communication protocol | 🌌 Galaxy |\n\n### ⚙️ Configuration & Setup\n\nConfigure your agents, models, and environments:\n\n| Topic | Description | Framework |\n|-------|-------------|-----------|\n| [Agent Configuration](configuration/system/agents_config.md) | LLM and agent settings | Both |\n| [Galaxy Devices](configuration/system/galaxy_devices.md) | Device pool and capability management | 🌌 Galaxy |\n| [Model Providers](configuration/models/overview.md) | Supported LLMs (OpenAI, Azure, Qwen, etc.) | Both |\n\n### 🎓 Tutorials & Examples\n\nLearn through practical examples in the documentation:\n\n| Topic | Description | Framework |\n|-------|-------------|-----------|\n| [Creating App Agents](tutorials/creating_app_agent/overview.md) | Build custom application agents | 🪟 UFO² |\n| [Multi-Action Prediction](ufo2/core_features/multi_action.md) | Efficient batch predictions | 🪟 UFO² |\n| [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) | RAG-enhanced learning | 🪟 UFO² |\n\n### 🔧 Advanced Topics\n\nDeep dive into powerful features:\n\n| Topic | Description | Framework |\n|-------|-------------|-----------|\n| [Multi-Action Prediction](ufo2/core_features/multi_action.md) | Batch actions for 51% fewer LLM calls | 🪟 UFO² |\n| [Hybrid Detection](ufo2/core_features/control_detection/hybrid_detection.md) | Visual + UIA control detection | 🪟 UFO² |\n| [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) | RAG-enhanced learning | 🪟 UFO² |\n| [Constellation Agent](galaxy/constellation_agent/overview.md) | Task planning and decomposition | 🌌 Galaxy |\n| [Task Orchestrator](galaxy/constellation_orchestrator/overview.md) | Execution coordination | 🌌 Galaxy |\n\n### 🛠️ Development & Extension\n\nCustomize and extend UFO³:\n\n| Topic | Description |\n|-------|-------------|\n| [Project Structure](project_directory_structure.md) | Understand the codebase layout |\n| [Creating Custom Device Agents](tutorials/creating_device_agent/overview.md) | Build device agents for new platforms (mobile, web, IoT, etc.) |\n| [Creating App Agents](tutorials/creating_app_agent/overview.md) | Build custom application agents |\n| [Contributing Guide](about/CONTRIBUTING.md) | How to contribute to UFO³ |\n\n### ❓ Support & Troubleshooting\n\nGet help when you need it:\n\n| Resource | What You'll Find |\n|----------|------------------|\n| [FAQ](faq.md) | Common questions and answers |\n| [GitHub Discussions](https://github.com/microsoft/UFO/discussions) | Community Q&A |\n| [GitHub Issues](https://github.com/microsoft/UFO/issues) | Bug reports and feature requests |\n\n---\n\n## 📊 Feature Matrix\n\n| Feature | UFO² Desktop AgentOS | UFO³ Galaxy | Winner |\n|---------|:--------------------:|:-----------:|:------:|\n| **Windows Automation** | ⭐⭐⭐⭐⭐ Optimized | ⭐⭐⭐⭐ Supported | UFO² |\n| **Cross-Device Tasks** | ❌ Not supported | ⭐⭐⭐⭐⭐ Core feature | Galaxy |\n| **Setup Complexity** | ⭐⭐⭐⭐⭐ Very easy | ⭐⭐⭐ Moderate | UFO² |\n| **Learning Curve** | ⭐⭐⭐⭐⭐ Gentle | ⭐⭐⭐ Moderate | UFO² |\n| **Task Complexity** | ⭐⭐⭐ Good | ⭐⭐⭐⭐⭐ Excellent | Galaxy |\n| **Parallel Execution** | ❌ Sequential | ⭐⭐⭐⭐⭐ Native DAG | Galaxy |\n| **Stability** | ⭐⭐⭐⭐⭐ Stable | ⭐⭐⭐ Active dev | UFO² |\n| **Monitoring Tools** | ⭐⭐⭐ Logs | ⭐⭐⭐⭐⭐ Real-time viz | Galaxy |\n| **API Flexibility** | ⭐⭐⭐ Good | ⭐⭐⭐⭐⭐ Extensive | Galaxy |\n\n---\n\n## 🎯 Use Cases & Examples\n\nExplore what you can build with UFO³:\n\n### 🌌 Galaxy Use Cases (Cross-Device)\n\nPerfect for complex, multi-device workflows:\n\n- **Cross-Platform Data Pipelines**: Extract from Windows Excel → Process on Linux → Visualize on Mac\n- **Distributed Testing**: Run tests on Windows → Deploy to Linux → Update mobile app\n- **Multi-Device Monitoring**: Collect logs from multiple devices → Aggregate centrally\n- **Complex Automation**: Orchestrate workflows across heterogeneous platforms\n\n### 🪟 UFO² Use Cases (Windows)\n\nPerfect for Windows automation and rapid task execution:\n\n- **Office Automation**: Excel/Word/PowerPoint report generation and data processing\n- **Web Automation**: Browser-based research, form filling, data extraction\n- **File Management**: Organize, rename, convert files based on rules\n- **System Tasks**: Windows configuration, software installation, backups\n\n---\n\n## 🌐 Community & Resources\n\n### 📺 Media & Videos\n\nCheck out our official deep dive of UFO on [YouTube](https://www.youtube.com/watch?v=QT_OhygMVXU).\n\n### Media Coverage:\n- [微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」](https://www.jiqizhixin.com/articles/2025-05-06-13)\n- [Microsoft's UFO: Smarter Windows Experience](https://the-decoder.com/microsofts-ufo-abducts-traditional-user-interfaces-for-a-smarter-windows-experience/)\n- [下一代Windows系统曝光：基于GPT-4V](https://baijiahao.baidu.com/s?id=1790938358152188625)\n\n### 💬 Get Help & Connect\n- **📖 Documentation**: You're here! Browse the navigation above\n- **💬 Discussions**: [GitHub Discussions](https://github.com/microsoft/UFO/discussions)\n- **🐛 Issues**: [GitHub Issues](https://github.com/microsoft/UFO/issues)\n- **📧 Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)\n\n### 🎨 Related Projects\n- **[TaskWeaver](https://github.com/microsoft/TaskWeaver)** – Code-first LLM agent framework\n- **[Windows Agent Arena](https://github.com/nice-mee/WindowsAgentArena)** – Evaluation benchmark\n- **[GUI Agents Survey](https://vyokky.github.io/LLM-Brained-GUI-Agents-Survey/)** – Latest research\n\n---\n\n## 📚 Research & Citation\n\nUFO³ is built on cutting-edge research in multi-agent systems and GUI automation.\n\n### Papers\n\nIf you use UFO³ in your research, please cite:\n\n**UFO³ Galaxy Framework (2025)**\n```bibtex\n@article{zhang2025ufo3,\n title={UFO$^3$: Weaving the Digital Agent Galaxy}, \n author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\n journal = {arXiv preprint arXiv:2511.11332},\n year = {2025},\n}\n```\n\n**UFO² Desktop AgentOS (2025)**\n```bibtex\n@article{zhang2025ufo2,\n title = {{UFO2: The Desktop AgentOS}},\n author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\n journal = {arXiv preprint arXiv:2504.14603},\n year = {2025}\n}\n```\n\n**Original UFO (2024)**\n```bibtex\n@article{zhang2024ufo,\n title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},\n author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},\n journal = {arXiv preprint arXiv:2402.07939},\n year = {2024}\n}\n```\n\n**📖 [Read the Papers →](https://arxiv.org/abs/2504.14603)**\n\n---\n\n\n## 🗺️ Roadmap & Future\n\n### UFO² Desktop AgentOS (Stable/LTS)\n- ✅ Long-term support and maintenance \n- ✅ Windows device agent integration\n- 🔜 Enhanced device capabilities\n- 🔜 Picture-in-Picture mode\n\n### UFO³ Galaxy (Active Development)\n- ✅ Constellation Framework\n- ✅ Multi-device coordination\n- 🔄 Mobile, Web, IoT agents\n- 🔄 Interactive visualization\n- 🔜 Advanced fault tolerance\n\n**Legend:** ✅ Done | 🔄 In Progress | 🔜 Planned\n\n---\n\n## ⚖️ License & Legal\n\n- **License**: [MIT License](https://github.com/microsoft/UFO/blob/main/LICENSE)\n- **Disclaimer**: [Read our disclaimer](https://github.com/microsoft/UFO/blob/main/DISCLAIMER.md)\n- **Trademarks**: [Microsoft Trademark Guidelines](https://www.microsoft.com/legal/intellectualproperty/trademarks)\n- **Contributing**: [Contribution Guidelines](about/CONTRIBUTING.md)\n\n---\n\n\n## 🚀 Ready to Start?\n\nChoose your framework and begin your automation journey:\n\n\n### 🌌 Start with Galaxy\n**For multi-device orchestration**\n\n[](getting_started/quick_start_galaxy.md)\n\n\n### 🪟 Start with UFO²\n**For Windows automation**\n\n[](getting_started/quick_start_ufo2.md)\n\n\n### 📖 Explore the Documentation\n\n[Core Concepts](galaxy/overview.md) | [Configuration](configuration/system/agents_config.md) | [FAQ](faq.md) | [GitHub](https://github.com/microsoft/UFO)\n\n\n---\n\n
\n \n \n From Single Agent to Digital Galaxy\n \n UFO³ - Weaving the Future of Intelligent Automation\n
\n\n---\n\n\n"
},
{
"path": "documents/docs/infrastructure/agents/agent_types.md",
"content": "# Platform-Specific Agent Implementations\n\nThis document describes how the unified three-layer Device Agent architecture is implemented across different platforms. While the core framework (State, Processor, Command layers) remains consistent, each platform implements specialized agents optimized for their native control mechanisms and hierarchies. Understanding these implementations is essential for extending UFO3 to new platforms or customizing existing agents.\n\n## Overview\n\nUFO3's Device Agent architecture achieves cross-platform compatibility through **platform-specific agent implementations** that inherit from a common abstract framework. Each platform's agents implement the same `BasicAgent` interface while adapting the three-layer architecture to their unique execution environments:\n\n```mermaid\ngraph TB\n subgraph \"Unified Framework\"\n Framework[Three-Layer Architecture State + Processor + Command]\n end\n \n subgraph \"Windows Platform\"\n HostAgent[HostAgent Application Selection]\n AppAgent[AppAgent Application Control]\n \n HostAgent -->|Delegates| AppAgent\n end\n \n subgraph \"Linux Platform\"\n LinuxAgent[LinuxAgent Shell Commands]\n end\n \n subgraph \"Future Platforms\"\n MacAgent[macOS Agent]\n AndroidAgent[Android Agent]\n IOSAgent[iOS Agent]\n end\n \n Framework -.Implements.-> HostAgent\n Framework -.Implements.-> AppAgent\n Framework -.Implements.-> LinuxAgent\n Framework -.Extends to.-> MacAgent\n Framework -.Extends to.-> AndroidAgent\n Framework -.Extends to.-> IOSAgent\n \n style Framework fill:#fff4e1\n style HostAgent fill:#e1f5ff\n style AppAgent fill:#e1f5ff\n style LinuxAgent fill:#c8e6c9\n style MacAgent fill:#f0f0f0\n style AndroidAgent fill:#f0f0f0\n style IOSAgent fill:#f0f0f0\n```\n\n**Unified Framework Benefits:**\n\n- **Code Reuse**: State management, strategy orchestration, and command dispatch logic shared across platforms\n- **Consistent Interface**: All agents implement BasicAgent interface with same lifecycle (handle, next_state, next_agent)\n- **Extensibility**: New platforms inherit three-layer architecture, only implementing platform-specific strategies and commands\n- **Multi-Platform Coordination**: HostAgent on Windows can coordinate with LinuxAgent on Linux device via Blackboard\n\n---\n\n## Platform Comparison\n\n| Feature | Windows (Two-Tier) | Linux (Single-Tier) | Future (macOS, Mobile) |\n|---------|-------------------|---------------------|------------------------|\n| **Agent Hierarchy** | HostAgent ?AppAgent delegation | LinuxAgent (flat) | Platform-specific (TBD) |\n| **Observation Method** | UI Automation API (COM) | Shell output, accessibility tree | Platform APIs (Accessibility, Screen) |\n| **Action Mechanism** | UI element manipulation (click, type) | Shell command execution | Platform-specific controls |\n| **Application Model** | Windowed applications | Command-line tools, X11 apps | Application frameworks |\n| **State Complexity** | 7 states (CONTINUE, FINISH, CONFIRM, etc.) | Simplified state set | Platform-dependent |\n| **Multi-Agent Coordination** | HostAgent ?AppAgent via Blackboard | N/A (single agent per device) | Cross-device via Blackboard |\n| **Primary Use Cases** | Office automation, GUI apps | Server management, DevOps | Mobile apps, embedded systems |\n\n!!! info \"Platform Selection Strategy\"\n - **Windows**: Use HostAgent + AppAgent for GUI-based applications requiring multi-step workflows (e.g., Excel data analysis, Word document editing)\n - **Linux**: Use LinuxAgent for command-line tasks, server administration, scripting workflows\n - **Cross-Platform**: Coordinate Windows and Linux agents via Blackboard for hybrid tasks (e.g., Windows collects data, Linux processes on server)\n\n---\n\n## Windows Platform: Two-Tier Agent Hierarchy\n\nWindows implements a **two-tier hierarchy** where HostAgent manages application selection and task decomposition, delegating execution to AppAgent instances for specific applications.\n\n### Architecture Overview\n\n```mermaid\ngraph TB\n subgraph \"Windows Two-Tier Hierarchy\"\n User[User Request: 'Create chart in Excel from Word data']\n \n HostAgent[HostAgent Task Orchestrator]\n \n AppAgent1[AppAgent Microsoft Word]\n AppAgent2[AppAgent Microsoft Excel]\n \n User --> HostAgent\n HostAgent -->|1. Extract data| AppAgent1\n HostAgent -->|2. Create chart| AppAgent2\n \n AppAgent1 -.Result via Blackboard.-> HostAgent\n AppAgent2 -.Result via Blackboard.-> HostAgent\n end\n \n subgraph \"HostAgent Responsibilities\"\n TaskDecomp[Task Decomposition]\n AppSelect[Application Selection]\n SubtaskDist[Subtask Distribution]\n ResultAgg[Result Aggregation]\n end\n \n subgraph \"AppAgent Responsibilities\"\n UIObserve[UI Observation]\n ActionExec[Action Execution]\n AppControl[Application Control]\n end\n \n HostAgent --> TaskDecomp\n HostAgent --> AppSelect\n HostAgent --> SubtaskDist\n HostAgent --> ResultAgg\n \n AppAgent1 --> UIObserve\n AppAgent1 --> ActionExec\n AppAgent1 --> AppControl\n \n style HostAgent fill:#fff4e1\n style AppAgent1 fill:#e1f5ff\n style AppAgent2 fill:#e1f5ff\n```\n\n**Two-Tier Execution Flow Example:**\n\n**User Request**: \"Extract data from sales.docx and create a bar chart in Excel\"\n\n**HostAgent**:\n1. Analyzes request → Identifies need for Word + Excel\n2. Creates subtask 1: \"Extract sales data from Word document\"\n3. Delegates to AppAgent (Word) via `next_agent()`\n\n**AppAgent (Word)**:\n1. Observes Word UI, locates sales data table\n2. Executes `select_text` + `copy_to_clipboard` actions\n3. Writes result to Blackboard: `blackboard.add_data(data, blackboard.trajectories)`\n4. Returns to HostAgent via `next_agent(HostAgent)`\n\n**HostAgent**:\n1. Reads result from Blackboard\n2. Creates subtask 2: \"Create bar chart in Excel from extracted data\"\n3. Delegates to AppAgent (Excel) via `next_agent()`\n\n**AppAgent (Excel)**:\n1. Reads data from Blackboard trajectories\n2. Executes actions: `paste_data` → `select_data_range` → `insert_chart`\n3. Returns to HostAgent with `AgentStatus.FINISH`\n\n---\n\n## HostAgent: Application Selection and Task Orchestration\n\nThe **HostAgent** is the top-level coordinator in the Windows two-tier hierarchy, responsible for **application selection**, **task decomposition**, and **subtask distribution**.\n\n### HostAgent Architecture\n\n```python\n@AgentRegistry.register(agent_name=\"hostagent\")\nclass HostAgent(BasicAgent):\n \"\"\"\n The HostAgent class is the manager of AppAgents.\n Coordinates multi-application workflows on Windows.\n \"\"\"\n \n def __init__(\n self,\n name: str,\n is_visual: bool,\n main_prompt: str,\n example_prompt: str,\n api_prompt: str,\n ) -> None:\n super().__init__(name=name)\n self.prompter = HostAgentPrompter(is_visual, main_prompt, example_prompt, api_prompt)\n self.agent_factory = AgentFactory()\n self.appagent_dict = {} # Cache of created AppAgent instances\n self._active_appagent = None\n self._blackboard = Blackboard() # Shared coordination space\n self.set_state(self.default_state)\n```\n\n### Key Responsibilities\n\n| Responsibility | Implementation | Example |\n|----------------|----------------|---------|\n| **Task Decomposition** | LLM analyzes user request, breaks into subtasks | \"Create report\" ?[\"Extract data\", \"Generate chart\", \"Format document\"] |\n| **Application Selection** | Identifies required applications for each subtask | Subtask \"Extract data\" ?Microsoft Word |\n| **AppAgent Creation** | Factory pattern creates AppAgent instances on-demand | `agent_factory.create_agent(\"app\", process=\"WINWORD.EXE\")` |\n| **Subtask Delegation** | Routes subtasks to appropriate AppAgent | `next_agent() ?AppAgent(Word)` |\n| **Result Aggregation** | Collects results from AppAgents via Blackboard | `blackboard.get_value(\"appagent/word/result\")` |\n| **Multi-App Coordination** | Sequences actions across multiple applications | Word → Excel → PowerPoint workflow |\n\n### HostAgent Processor\n\n```python\nclass HostAgentProcessor(ProcessorTemplate):\n \"\"\"\n Processor for HostAgent with specialized strategies.\n \"\"\"\n \n def __init__(self, agent, context):\n super().__init__(agent, context)\n \n # DATA_COLLECTION: Get list of running applications\n self.register_strategy(\n ProcessingPhase.DATA_COLLECTION,\n HostDataCollectionStrategy(agent, context)\n )\n \n # LLM_INTERACTION: Application selection and task planning\n self.register_strategy(\n ProcessingPhase.LLM_INTERACTION,\n HostLLMInteractionStrategy(agent, context)\n )\n \n # ACTION_EXECUTION: Create AppAgent, delegate subtask\n self.register_strategy(\n ProcessingPhase.ACTION_EXECUTION,\n HostActionExecutionStrategy(agent, context)\n )\n```\n\n**HostAgent Strategy Specializations:**\n\n- **DATA_COLLECTION**: Uses MCP tools to observe available Windows apps\n- **LLM_INTERACTION**: Specialized prompt template for application selection:\n - Input: User request + list of running apps\n - Output: Selected application + decomposed subtask\n- **ACTION_EXECUTION**: Instead of executing UI commands, creates/retrieves AppAgent instance and delegates via `next_agent()`\n\n### HostAgent State Transitions\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE: User request received\n \n CONTINUE --> CONTINUE: Select app, delegate to AppAgent\n CONTINUE --> FINISH: All subtasks completed\n CONTINUE --> CONFIRM: Need user confirmation\n CONTINUE --> ERROR: Application selection failed\n \n CONFIRM --> CONTINUE: User confirms\n CONFIRM --> FINISH: User rejects\n \n ERROR --> FINISH: Unrecoverable error\n FINISH --> [*]\n \n note right of CONTINUE\n HostAgent delegates to AppAgent\n via next_agent() method\n end note\n \n note right of FINISH\n HostAgent aggregates results\n from all AppAgents\n end note\n```\n\n**HostAgent Delegation Pattern Example:**\n\n```python\nclass HostAgent(BasicAgent):\n def handle(self, context: Context) -> Tuple[AgentStatus, Optional[BasicAgent]]:\n \"\"\"\n Handle HostAgent state: select application and delegate.\n \"\"\"\n # Execute processor strategies\n processor = HostAgentProcessor(self, context)\n result = processor.process()\n \n # Get selected application from LLM response\n selected_app = result.parsed_response.get(\"application\")\n subtask = result.parsed_response.get(\"subtask\")\n \n # Create or retrieve AppAgent for selected application\n appagent = self.get_or_create_appagent(selected_app)\n \n # Write subtask to Blackboard for AppAgent to read\n self._blackboard.add_data(\n {\"subtask\": subtask, \"app\": selected_app},\n self._blackboard.requests\n )\n \n # Delegate to AppAgent\n return AgentStatus.CONTINUE, appagent\n \n def get_or_create_appagent(self, app_name: str) -> AppAgent:\n \"\"\"\n Factory method: Create AppAgent if not exists, otherwise return cached instance.\n \"\"\"\n if app_name not in self.appagent_dict:\n self.appagent_dict[app_name] = self.agent_factory.create_agent(\n agent_type=\"app\",\n name=f\"AppAgent/{app_name}\",\n process_name=app_name,\n app_root_name=app_name\n )\n return self.appagent_dict[app_name]\n```\n\n---\n\n## AppAgent: Application-Specific Control\n\nThe **AppAgent** is responsible for **direct control of a specific Windows application**, executing UI-based actions through Windows UI Automation APIs.\n\n### AppAgent Architecture\n\n```python\n@AgentRegistry.register(agent_name=\"appagent\", processor_cls=AppAgentProcessor)\nclass AppAgent(BasicAgent):\n \"\"\"\n The AppAgent class manages interaction with a specific Windows application.\n \"\"\"\n \n def __init__(\n self,\n name: str,\n process_name: str,\n app_root_name: str,\n is_visual: bool,\n main_prompt: str,\n example_prompt: str,\n mode: str = \"normal\",\n ) -> None:\n super().__init__(name=name)\n self.prompter = AppAgentPrompter(is_visual, main_prompt, example_prompt)\n self._process_name = process_name # e.g., \"WINWORD.EXE\"\n self._app_root_name = app_root_name # e.g., \"Microsoft Word\"\n self._mode = mode\n self.set_state(self.default_state)\n```\n\n### Key Responsibilities\n\n| Responsibility | Implementation | Example |\n|----------------|----------------|---------|\n| **UI Observation** | Screenshot + UI Automation tree capture | `get_ui_tree` returns hierarchical element structure |\n| **Element Identification** | Parse UI tree to locate target elements | Find \"Save\" button by name, control type, bounding box |\n| **Action Execution** | Execute UI commands via MCP tools | `click_element(element_id=\"save_button\")` |\n| **Application Context** | Maintain application-specific state | Current document, active window, focus element |\n| **Error Handling** | Detect and recover from UI failures | Retry on stale element, fallback to keyboard shortcuts |\n| **Result Reporting** | Write results to Blackboard for HostAgent | `blackboard.add_key_value(\"result\", \"Document saved\")` |\n\n### AppAgent Processor\n\n```python\nclass AppAgentProcessor(ProcessorTemplate):\n \"\"\"\n Processor for AppAgent with UI-focused strategies.\n \"\"\"\n \n def __init__(self, agent, context):\n super().__init__(agent, context)\n \n # DATA_COLLECTION: Screenshot + UI tree\n self.register_strategy(\n ProcessingPhase.DATA_COLLECTION,\n ComposedStrategy([\n ScreenshotStrategy(agent, context),\n UITreeStrategy(agent, context)\n ])\n )\n \n # LLM_INTERACTION: UI element selection\n self.register_strategy(\n ProcessingPhase.LLM_INTERACTION,\n AppAgentLLMStrategy(agent, context)\n )\n \n # ACTION_EXECUTION: Execute UI commands\n self.register_strategy(\n ProcessingPhase.ACTION_EXECUTION,\n UIActionExecutionStrategy(agent, context)\n )\n```\n\n### Windows UI Automation Integration\n\nAppAgent leverages **Windows UI Automation (UIA)** for robust UI control:\n\n```mermaid\ngraph LR\n subgraph \"AppAgent Observation\"\n AppAgent[AppAgent]\n \n Screenshot[Screenshot Visual Context]\n UITree[UI Automation Tree Element Hierarchy]\n \n AppAgent --> Screenshot\n AppAgent --> UITree\n end\n \n subgraph \"Windows UI Automation\"\n UIA[UI Automation API]\n \n Elements[UI Elements Button, TextBox, etc.]\n Properties[Element Properties Name, Type, BoundingBox]\n Patterns[Control Patterns Invoke, Value, Selection]\n \n UIA --> Elements\n UIA --> Properties\n UIA --> Patterns\n end\n \n UITree -.Query.-> UIA\n \n subgraph \"Action Execution\"\n Commands[MCP Commands]\n \n Click[click_element]\n Type[type_text]\n Select[select_item]\n \n Commands --> Click\n Commands --> Type\n Commands --> Select\n end\n \n AppAgent --> Commands\n Commands -.Invoke.-> Patterns\n \n style AppAgent fill:#e1f5ff\n style UIA fill:#fff4e1\n style Commands fill:#ffe1e1\n```\n\n**UI Automation Capabilities:**\n\n- **Element Discovery**: Traverse UI tree to find controls by name, type, automation ID\n- **Property Access**: Read element properties (text, state, position, visibility)\n- **Pattern Invocation**: Execute control-specific actions:\n - InvokePattern: Click buttons, menu items\n - ValuePattern: Set text in textboxes\n - SelectionPattern: Select items in lists, dropdowns\n - TogglePattern: Toggle checkboxes, radio buttons\n\n### AppAgent Commands\n\n| Command Category | Commands | Description |\n|-----------------|----------|-------------|\n| **Observation** | `screenshot`, `get_ui_tree`, `get_accessibility_tree` | Capture visual and structural UI information |\n| **Navigation** | `click_element`, `double_click`, `right_click` | Navigate UI through mouse interactions |\n| **Text Input** | `type_text`, `set_value`, `clear_text` | Input and modify text in UI controls |\n| **Selection** | `select_item`, `select_dropdown`, `toggle_checkbox` | Manipulate selection controls |\n| **Scrolling** | `scroll`, `scroll_to_element` | Navigate large UI areas |\n| **Window Management** | `activate_window`, `close_window`, `maximize_window` | Control window state |\n| **File Operations** | `open_file`, `save_file`, `save_as` | Application-specific file actions |\n\n**AppAgent UI Control Pattern Example:**\n\n```python\nclass AppAgent(BasicAgent):\n def handle(self, context: Context) -> Tuple[AgentStatus, Optional[BasicAgent]]:\n \"\"\"\n Handle AppAgent state: Control application UI.\n \"\"\"\n # Read subtask from Blackboard (written by HostAgent)\n subtask_memory = self._blackboard.requests.to_list_of_dicts()\n if subtask_memory:\n subtask = subtask_memory[-1].get(\"subtask\")\n \n # Execute processor strategies\n processor = AppAgentProcessor(self, context)\n context.set(ContextNames.REQUEST, subtask)\n result = processor.process()\n \n # Check if subtask completed\n if result.status == AgentStatus.FINISH:\n # Write result to Blackboard\n self._blackboard.add_data(\n {\"result\": result.parsed_response.get(\"result\")},\n self._blackboard.trajectories\n )\n \n # Return to HostAgent\n return AgentStatus.FINISH, self.parent_agent\n \n return result.status, None\n```\n\n---\n\n## Linux Platform: Single-Tier Agent System\n\nLinux implements a **single-tier architecture** where LinuxAgent directly executes shell commands without hierarchical delegation.\n\n### LinuxAgent Architecture\n\n```mermaid\ngraph TB\n subgraph \"Linux Single-Tier System\"\n User[User Request: 'Check server logs and restart service']\n \n LinuxAgent[LinuxAgent Shell Command Executor]\n \n Shell[Linux Shell bash, zsh, etc.]\n \n User --> LinuxAgent\n LinuxAgent -->|1. Execute: tail /var/log/app.log| Shell\n Shell -.Output.-> LinuxAgent\n LinuxAgent -->|2. Execute: systemctl restart app| Shell\n Shell -.Status.-> LinuxAgent\n end\n \n subgraph \"LinuxAgent Capabilities\"\n ShellExec[Shell Command Execution]\n OutputParse[Output Parsing]\n ChainCmd[Command Chaining]\n ErrorHandle[Error Detection]\n end\n \n LinuxAgent --> ShellExec\n LinuxAgent --> OutputParse\n LinuxAgent --> ChainCmd\n LinuxAgent --> ErrorHandle\n \n style LinuxAgent fill:#c8e6c9\n style Shell fill:#fff4e1\n```\n\n```python\n@AgentRegistry.register(\n agent_name=\"LinuxAgent\",\n third_party=True,\n processor_cls=LinuxAgentProcessor\n)\nclass LinuxAgent(CustomizedAgent):\n \"\"\"\n LinuxAgent is a specialized agent that interacts with Linux systems.\n Executes shell commands and parses output.\n \"\"\"\n \n def __init__(\n self,\n name: str,\n main_prompt: str,\n example_prompt: str,\n ) -> None:\n super().__init__(\n name=name,\n main_prompt=main_prompt,\n example_prompt=example_prompt,\n process_name=None,\n app_root_name=None,\n is_visual=None # LinuxAgent typically operates without visual mode\n )\n self._blackboard = Blackboard()\n self.set_state(ContinueLinuxAgentState())\n```\n\n### Key Differences from Windows Agents\n\n| Aspect | Windows (HostAgent + AppAgent) | Linux (LinuxAgent) |\n|--------|--------------------------------|-------------------|\n| **Hierarchy** | Two-tier (delegation pattern) | Single-tier (direct execution) |\n| **Observation** | Screenshot + UI Automation tree | Shell command output (stdout/stderr) |\n| **Action Mechanism** | UI element manipulation | Shell command execution |\n| **Context Tracking** | Application windows, UI state | Command history, working directory |\n| **Error Detection** | UI element not found, timeout | Exit code ?0, stderr output |\n| **Coordination** | Via Blackboard between HostAgent and AppAgent | Via Blackboard with other devices (cross-device) |\n\n### LinuxAgent Processor\n\n```python\nclass LinuxAgentProcessor(ProcessorTemplate):\n \"\"\"\n Processor for LinuxAgent with shell-focused strategies.\n \"\"\"\n \n def __init__(self, agent, context):\n super().__init__(agent, context)\n \n # DATA_COLLECTION: No visual observation, use command output from previous step\n self.register_strategy(\n ProcessingPhase.DATA_COLLECTION,\n LinuxDataCollectionStrategy(agent, context) # Collects shell output\n )\n \n # LLM_INTERACTION: Command generation from request\n self.register_strategy(\n ProcessingPhase.LLM_INTERACTION,\n LinuxLLMStrategy(agent, context) # Generates shell commands\n )\n \n # ACTION_EXECUTION: Execute shell commands\n self.register_strategy(\n ProcessingPhase.ACTION_EXECUTION,\n ShellExecutionStrategy(agent, context) # Executes via shell_execute\n )\n```\n\n### LinuxAgent Commands\n\n| Command | Function | Example |\n|---------|----------|---------|\n| `shell_execute` | Execute shell command (non-blocking) | `shell_execute(command=\"ls -la /home/user\")` |\n| `shell_execute_read` | Execute command and capture output | `shell_execute_read(command=\"cat /var/log/app.log\")` |\n| `get_accessibility_tree` | Get GUI app accessibility tree (X11) | `get_accessibility_tree()` for GUI apps |\n| `screenshot` | Capture screen (optional, for GUI) | `screenshot()` |\n\n**LinuxAgent Best Practices:**\n\n- **Command Chaining**: Use `&&` and `||` for robust workflows:\n ```bash\n cd /app && ./deploy.sh || echo \"Deployment failed\"\n ```\n- **Output Parsing**: Parse stdout for structured data:\n ```python\n output = shell_execute_read(\"systemctl status app\")\n if \"active (running)\" in output:\n # Service is running\n ```\n- **Error Handling**: Check exit codes and stderr:\n ```python\n result = shell_execute(\"restart_service.sh\")\n if result.status == ResultStatus.FAILURE:\n # Handle error from stderr\n ```\n- **Idempotency**: Design commands to be safely re-runnable:\n ```bash\n # Good: Check before creating\n [ -d /app/backup ] || mkdir -p /app/backup\n \n # Bad: Fails if directory exists\n mkdir /app/backup\n ```\n\n**LinuxAgent Cross-Device Coordination Example:**\n\n```python\n# Windows HostAgent prepares data for Linux processing\nwindows_blackboard.add_data(\n {\"data_file\": \"C:/export/data.csv\", \"ready\": True},\n windows_blackboard.requests\n)\n\n# LinuxAgent polls Blackboard for task availability\nrequests = linux_blackboard.requests.to_list_of_dicts()\nif requests and requests[-1].get(\"ready\"):\n # Download data from Windows device (via network share or AIP)\n await linux_agent.execute_command(\n \"scp user@windows-pc:/c/export/data.csv /tmp/data.csv\"\n )\n \n # Process data\n await linux_agent.execute_command(\n \"python3 /app/process.py /tmp/data.csv\"\n )\n \n # Report completion\n linux_blackboard.add_data(\n {\"status\": \"completed\"},\n linux_blackboard.trajectories\n )\n```\n\n---\n\n## Multi-Agent Coordination Patterns\n\nThe three-layer architecture enables seamless coordination across different agent types through **Blackboard-based communication**.\n\n### Pattern 1: Windows Multi-App Workflow\n\n```mermaid\nsequenceDiagram\n participant User\n participant HostAgent\n participant AppAgentWord\n participant AppAgentExcel\n participant Blackboard\n \n User->>HostAgent: \"Extract data from Word, create chart in Excel\"\n \n HostAgent->>HostAgent: Decompose task\n HostAgent->>Blackboard: Write subtask_1: \"Extract data\"\n HostAgent->>AppAgentWord: Delegate (next_agent)\n \n AppAgentWord->>AppAgentWord: Observe Word UI\n AppAgentWord->>AppAgentWord: Execute: select_text, copy\n AppAgentWord->>Blackboard: Write result: extracted_data\n AppAgentWord->>HostAgent: Return (next_agent)\n \n HostAgent->>Blackboard: Read extracted_data\n HostAgent->>Blackboard: Write subtask_2: \"Create chart\"\n HostAgent->>AppAgentExcel: Delegate (next_agent)\n \n AppAgentExcel->>Blackboard: Read extracted_data\n AppAgentExcel->>AppAgentExcel: Execute: paste, select_range, insert_chart\n AppAgentExcel->>Blackboard: Write result: chart_created\n AppAgentExcel->>HostAgent: Return (next_agent)\n \n HostAgent->>User: Task completed\n```\n\n### Pattern 2: Cross-Device Linux-Windows Coordination\n\n```mermaid\nsequenceDiagram\n participant WindowsHost\n participant WindowsApp\n participant Blackboard\n participant LinuxAgent\n \n WindowsHost->>WindowsApp: \"Export sales data to CSV\"\n WindowsApp->>WindowsApp: Execute: export to C:/data/sales.csv\n WindowsApp->>Blackboard: Write: data_ready=true, path=C:/data/sales.csv\n \n LinuxAgent->>Blackboard: Poll: data_ready?\n Blackboard-->>LinuxAgent: data_ready=true\n \n LinuxAgent->>LinuxAgent: Execute: scp windows-pc:/c/data/sales.csv /tmp/\n LinuxAgent->>LinuxAgent: Execute: python3 analyze.py /tmp/sales.csv\n LinuxAgent->>Blackboard: Write: analysis_complete=true, results=/tmp/report.pdf\n \n WindowsHost->>Blackboard: Read: analysis_complete?\n Blackboard-->>WindowsHost: analysis_complete=true, results=/tmp/report.pdf\n WindowsHost->>LinuxAgent: Request: Download /tmp/report.pdf\n```\n\n### Pattern 3: Parallel Multi-Device Tasks\n\n```mermaid\ngraph TB\n subgraph \"Orchestrator (HostAgent)\"\n Orchestrator[HostAgent Task Coordinator]\n end\n \n subgraph \"Device 1: Windows Desktop\"\n AppAgent1[AppAgent PowerPoint]\n end\n \n subgraph \"Device 2: Windows Laptop\"\n AppAgent2[AppAgent Excel]\n end\n \n subgraph \"Device 3: Linux Server\"\n LinuxAgent1[LinuxAgent Data Processing]\n end\n \n subgraph \"Shared Blackboard\"\n BB[Blackboard Coordination Space]\n end\n \n Orchestrator -->|Subtask 1: Create slides| AppAgent1\n Orchestrator -->|Subtask 2: Generate charts| AppAgent2\n Orchestrator -->|Subtask 3: Process data| LinuxAgent1\n \n AppAgent1 -.Write result.-> BB\n AppAgent2 -.Write result.-> BB\n LinuxAgent1 -.Write result.-> BB\n \n BB -.Aggregate results.-> Orchestrator\n \n style Orchestrator fill:#fff4e1\n style AppAgent1 fill:#e1f5ff\n style AppAgent2 fill:#e1f5ff\n style LinuxAgent1 fill:#c8e6c9\n style BB fill:#ffe1e1\n```\n\n---\n\n## Platform Extensibility: Adding New Platforms\n\nThe three-layer architecture provides a clear path for extending UFO3 to new platforms:\n\n### Extension Checklist\n\n**Steps to Add a New Platform:**\n\n1. **Define Agent Class**\n ```python\n @AgentRegistry.register(\n agent_name=\"MacOSAgent\",\n processor_cls=MacOSAgentProcessor\n )\n class MacOSAgent(BasicAgent):\n # Implement platform-specific initialization\n ```\n\n2. **Implement Platform-Specific Strategies**\n - **DATA_COLLECTION**: How to observe system state (screenshots, accessibility tree, shell output)\n - **LLM_INTERACTION**: Adapt prompt template for platform capabilities\n - **ACTION_EXECUTION**: Map actions to platform APIs (AppKit, Accessibility API, etc.)\n - **MEMORY_UPDATE**: Standard implementation (usually no changes needed)\n\n3. **Define Platform Commands (MCP Tools)**\n ```python\n # macOS-specific commands\n commands = [\n \"applescript_execute\", # Execute AppleScript\n \"accessibility_tree\", # macOS Accessibility API\n \"click_element\", # macOS UI control\n \"type_text\" # Text input\n ]\n ```\n\n4. **Implement AgentState Subclasses** (if needed)\n ```python\n class ContinueMacOSAgentState(AgentState):\n def handle(self, agent, context):\n # macOS-specific state handling\n ```\n\n5. **Create Platform-Specific Processor**\n ```python\n class MacOSAgentProcessor(ProcessorTemplate):\n def __init__(self, agent, context):\n super().__init__(agent, context)\n self.register_strategy(\n ProcessingPhase.DATA_COLLECTION,\n MacOSDataCollectionStrategy(agent, context)\n )\n # Register other strategies...\n ```\n\n6. **Configure MCP Server** (on device client)\n - Implement MCP tools for platform-specific operations\n - Register tools with MCP server manager\n - Ensure AIP client routes commands correctly\n\n### Platform-Specific Considerations\n\n| Platform | Key Considerations | Suggested Implementation |\n|----------|-------------------|--------------------------|\n| **macOS** | Accessibility API, AppleScript, window management | MacOSAgent (single-tier), AppleScript execution strategy |\n| **Android** | Activity lifecycle, UI Automator, touch gestures | AndroidAgent (single-tier), UI Automator integration |\n| **iOS** | Accessibility, XCTest, limited automation | iOSAgent (single-tier), XCTest framework |\n| **Embedded** | Limited resources, no GUI, command-line only | EmbeddedAgent (minimal strategies, shell-based) |\n| **Web** | Browser automation, DOM manipulation | WebAgent (Selenium/Playwright integration) |\n\n**Example: Adding macOS Support**\n\n```python\n# 1. Define macOS Agent\n@AgentRegistry.register(\n agent_name=\"MacOSAgent\",\n processor_cls=MacOSAgentProcessor\n)\nclass MacOSAgent(BasicAgent):\n def __init__(self, name: str, main_prompt: str, example_prompt: str):\n super().__init__(name=name)\n self.prompter = MacOSAgentPrompter(main_prompt, example_prompt)\n self.set_state(ContinueMacOSAgentState())\n\n# 2. Implement macOS-specific DATA_COLLECTION strategy\nclass MacOSDataCollectionStrategy(ProcessingStrategy):\n def execute(self, context: ProcessingContext):\n # Use macOS Accessibility API\n commands = [\n Command(tool_name=\"get_accessibility_tree\", parameters={}, tool_type=\"data_collection\"),\n Command(tool_name=\"screenshot\", parameters={}, tool_type=\"data_collection\")\n ]\n results = self.dispatcher.execute_commands(commands)\n \n context.set_local(\"accessibility_tree\", results[0].result)\n context.set_local(\"screenshot\", results[1].result)\n\n# 3. Implement macOS-specific ACTION_EXECUTION strategy\nclass MacOSActionExecutionStrategy(ProcessingStrategy):\n def execute(self, context: ProcessingContext):\n action = context.get_global(\"action\")\n \n if action == \"click_element\":\n # Use macOS Accessibility API via MCP tool\n command = Command(\n tool_name=\"macos_click_element\",\n parameters={\"element_id\": context.get_global(\"element_id\")},\n tool_type=\"action\"\n )\n elif action == \"applescript_execute\":\n # Execute AppleScript via MCP tool\n command = Command(\n tool_name=\"applescript_execute\",\n parameters={\"script\": context.get_global(\"applescript\")},\n tool_type=\"action\"\n )\n \n results = self.dispatcher.execute_commands([command])\n context.set_local(\"execution_results\", results)\n\n# 4. Configure MCP tools on macOS device client\n# In device client code:\nmcp_server_manager.register_tool(\n MCPToolInfo(\n tool_name=\"macos_click_element\",\n description=\"Click element via macOS Accessibility API\",\n input_schema={\n \"element_id\": {\"type\": \"string\", \"description\": \"Accessibility element ID\"}\n },\n # ... other fields\n ),\n handler=macos_accessibility_click_handler\n)\n```\n\n---\n\n## Agent Lifecycle Comparison\n\n### Windows HostAgent Lifecycle\n\n```mermaid\nsequenceDiagram\n participant Session\n participant HostAgent\n participant AppAgent\n participant Blackboard\n \n Session->>HostAgent: Initialize (user request)\n HostAgent->>HostAgent: Set state = ContinueHostAgentState\n \n loop Until Task Complete\n HostAgent->>HostAgent: Execute HostAgentProcessor\n HostAgent->>HostAgent: LLM selects application\n HostAgent->>HostAgent: Create/retrieve AppAgent\n HostAgent->>Blackboard: Write subtask\n HostAgent->>AppAgent: Delegate (next_agent)\n \n AppAgent->>Blackboard: Read subtask\n AppAgent->>AppAgent: Execute AppAgentProcessor\n AppAgent->>Blackboard: Write result\n AppAgent->>HostAgent: Return (next_agent)\n \n HostAgent->>Blackboard: Read result\n HostAgent->>HostAgent: Update task status\n end\n \n HostAgent->>Session: Return AgentStatus.FINISH\n```\n\n### Linux LinuxAgent Lifecycle\n\n```mermaid\nsequenceDiagram\n participant Session\n participant LinuxAgent\n participant Shell\n \n Session->>LinuxAgent: Initialize (user request)\n LinuxAgent->>LinuxAgent: Set state = ContinueLinuxAgentState\n \n loop Until Task Complete\n LinuxAgent->>LinuxAgent: Execute LinuxAgentProcessor\n LinuxAgent->>LinuxAgent: LLM generates shell command\n LinuxAgent->>Shell: Execute command\n Shell-->>LinuxAgent: Return output (stdout/stderr)\n LinuxAgent->>LinuxAgent: Parse output\n LinuxAgent->>LinuxAgent: Update context with result\n LinuxAgent->>LinuxAgent: Check task completion\n end\n \n LinuxAgent->>Session: Return AgentStatus.FINISH\n```\n\n---\n\n## Performance and Scalability\n\n| Metric | Windows (Two-Tier) | Linux (Single-Tier) | Notes |\n|--------|-------------------|---------------------|-------|\n| **Agent Initialization** | ~500ms (HostAgent) + ~300ms per AppAgent | ~200ms (LinuxAgent) | AppAgent creation overhead for each application |\n| **Observation Latency** | ~1-2s (screenshot + UI tree) | ~100-500ms (shell output) | UI Automation API slower than shell |\n| **Action Execution** | ~200-500ms per UI action | ~50-200ms per shell command | UI actions require element discovery |\n| **Memory Footprint** | ~50MB (HostAgent) + ~30MB per AppAgent | ~20MB (LinuxAgent) | UI Automation increases memory usage |\n| **Scalability** | Limited by number of AppAgents | Handles many parallel commands | HostAgent manages AppAgent pool |\n| **Coordination Overhead** | Blackboard read/write per delegation | Minimal (only cross-device) | Two-tier hierarchy increases communication |\n\n**Performance Optimization:**\n\n- **Windows**: Reuse AppAgent instances across subtasks (cached in `appagent_dict`)\n- **Linux**: Batch multiple shell commands with `&&` to reduce round trips\n- **Cross-Platform**: Minimize Blackboard writes; use hierarchical keys for efficient reads\n\n---\n\n## Best Practices\n\n### Windows Agent Best Practices\n\n**HostAgent:**\n\n- **AppAgent Caching**: Reuse AppAgent instances for same application to avoid recreation overhead\n- **Task Decomposition**: Break complex tasks into independent subtasks for parallel execution\n- **Blackboard Namespacing**: Use clear keys within appropriate memory sections\n- **Error Propagation**: Detect AppAgent failures and retry with different strategy\n\n**AppAgent:**\n\n- **Element Stability**: Wait for UI elements to stabilize before interaction (use `wait_for_element`)\n- **Fallback Actions**: If UI Automation fails, fallback to keyboard shortcuts (e.g., Ctrl+S instead of clicking Save button)\n- **Context Awareness**: Track active window and focus to ensure actions target correct application\n- **Idempotent Actions**: Design actions to be safely retryable (e.g., check if file exists before creating)\n\n### Linux Agent Best Practices\n\n**LinuxAgent:**\n\n- **Command Validation**: Validate commands before execution to prevent injection attacks\n- **Output Parsing**: Use structured output formats (JSON, CSV) instead of parsing raw text\n- **Error Detection**: Check exit codes (`$?`) and stderr for failure detection\n- **Idempotency**: Use conditional commands (`[ -f file ] || create_file`) to safely re-run workflows\n- **Resource Cleanup**: Always clean up temporary files and processes after task completion\n\n### Cross-Platform Best Practices\n\n**Multi-Agent Coordination:**\n\n- **Blackboard Keys**: Use appropriate memory sections to separate agent-specific data:\n ```python\n # Good - using structured memory sections\n blackboard.add_data({\"status\": \"ready\"}, blackboard.requests)\n blackboard.add_data({\"status\": \"processing\"}, blackboard.trajectories)\n \n # Bad - unclear categorization\n blackboard.add_data({\"status\": \"ready\"}, blackboard.questions)\n ```\n\n- **Synchronization**: Use polling or event-based patterns for cross-device synchronization:\n ```python\n # Polling pattern\n while not any(r.get(\"task_complete\") for r in blackboard.requests.to_list_of_dicts()):\n await asyncio.sleep(1)\n \n # Event-based (via AIP custom messages)\n # Linux device sends completion event\n aip_client.send_event(\"task_complete\", {...})\n ```\n\n- **Data Transfer**: For large data, use shared storage (network drive, S3) instead of Blackboard:\n ```python\n # Bad: Store large data in Blackboard\n blackboard.add_data({\"dataset\": [1000000 rows]}, blackboard.trajectories)\n \n # Good: Store reference to shared storage\n blackboard.add_data({\"dataset_path\": \"s3://bucket/data.csv\"}, blackboard.requests)\n ```\n\n---\n\n## Related Documentation\n\n- [Device Agent Overview](overview.md) - Three-layer architecture and design principles\n- [Server-Client Architecture](server_client_architecture.md) - Server and client separation\n- [State Layer](design/state.md) - AgentState interface and state machine\n- [Processor and Strategy Layer](design/processor.md) - ProcessorTemplate and strategy implementations\n- [Command Layer](design/command.md) - CommandDispatcher and MCP integration\n- [Memory System](design/memory.md) - Memory and Blackboard for agent coordination\n- [Server Architecture](../../server/overview.md) - Server-side orchestration\n- [Client Architecture](../../client/overview.md) - Device client MCP execution\n- [AIP Protocol](../../aip/overview.md) - Agent Interaction Protocol for communication\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n- **Windows Two-Tier Hierarchy**: HostAgent (orchestration) + AppAgent (application control) for GUI workflows\n- **Linux Single-Tier System**: LinuxAgent executes shell commands directly for command-line tasks\n- **Unified Framework**: Both platforms leverage same three-layer architecture (State, Processor, Command)\n- **Multi-Agent Coordination**: Blackboard enables seamless coordination across HostAgent → AppAgent and cross-device communication\n- **Platform Extensibility**: Clear extension path for macOS, Android, iOS, embedded systems\n- **HostAgent Responsibilities**: Task decomposition, application selection, AppAgent creation, subtask delegation\n- **AppAgent Capabilities**: UI observation (screenshot + UI Automation), element identification, UI action execution\n- **LinuxAgent Characteristics**: Shell command execution, output parsing, idempotent workflows\n- **Best Practices**: AppAgent caching, appropriate Blackboard usage, idempotent commands, structured output parsing\n- **Performance**: Windows UI Automation slower but more robust; Linux shell commands faster but less structured\n\nUFO3's platform-specific agent implementations demonstrate the flexibility and extensibility of the three-layer architecture, enabling cross-platform and cross-device task automation while maintaining consistent design principles and coordination mechanisms.\n"
},
{
"path": "documents/docs/infrastructure/agents/design/blackboard.md",
"content": "# Agent Blackboard\n\nThe `Blackboard` is a shared memory space visible to all agents in the UFO framework. It stores information required for agents to interact with the user and applications at every step. The `Blackboard` enables agents to share information and collaborate to fulfill user requests.\n\n## Components\n\nThe `Blackboard` consists of the following data components:\n\n| Component | Description |\n| --- | --- |\n| `questions` | A list of questions that UFO asks the user, along with their corresponding answers. |\n| `requests` | A list of historical user requests received in previous rounds. |\n| `trajectories` | A list of step-wise trajectories that record the agent's actions and decisions at each step. |\n| `screenshots` | A list of screenshots taken by the agent when it believes the current state is important for future reference. |\n\nThe keys stored in the `trajectories` are configured as `HISTORY_KEYS` in the `config/ufo/system.yaml` file. You can customize the keys based on your requirements and the agent's logic.\n\nWhether to save the screenshots is determined by the `AppAgent`. You can enable or disable screenshot capture by setting the `SCREENSHOT_TO_MEMORY` flag in the `config/ufo/system.yaml` file.\n\n## Blackboard to Prompt\n\nData in the `Blackboard` is based on the `MemoryItem` class. It has a method `blackboard_to_prompt` that converts the information stored in the `Blackboard` to a list of prompt content objects. Agents call this method to construct the prompt for the LLM's inference. The `blackboard_to_prompt` method is defined as follows:\n\n```python\ndef blackboard_to_prompt(self) -> List[str]:\n \"\"\"\n Convert the blackboard to a prompt.\n :return: The prompt.\n \"\"\"\n prefix = [\n {\n \"type\": \"text\",\n \"text\": \"[Blackboard:]\",\n }\n ]\n\n blackboard_prompt = (\n prefix\n + self.texts_to_prompt(self.questions, \"[Questions & Answers:]\")\n + self.texts_to_prompt(self.requests, \"[Request History:]\")\n + self.texts_to_prompt(\n self.trajectories, \"[Step Trajectories Completed Previously:]\"\n )\n + self.screenshots_to_prompt()\n )\n\n return blackboard_prompt\n```\n\n## Reference\n\n:::agents.memory.blackboard.Blackboard\n\nYou can customize the class to tailor the `Blackboard` to your requirements."
},
{
"path": "documents/docs/infrastructure/agents/design/command.md",
"content": "# Command Layer (Level-3 System Interface)\n\nThe **Command Layer** provides atomic, deterministic system operations that bridge device agents with underlying platform capabilities. Each command encapsulates a **tool** and its **parameters**, mapping directly to MCP tools on the device client. This layer ensures reliable, auditable, and extensible execution across heterogeneous devices.\n\n## Overview\n\nThe Command Layer implements **Level-3** of the [three-layer device agent architecture](../overview.md#three-layer-architecture). It provides:\n\n- **Atomic Commands**: Self-contained execution units with tool + parameters\n- **MCP Integration**: Commands map to Model Context Protocol tools on device client\n- **CommandDispatcher**: Routes commands from agent server to device client via AIP\n- **Deterministic Execution**: Same inputs → same outputs, fully auditable\n- **Dynamic Discovery**: LLM queries available tools and selects appropriate commands\n\n```mermaid\ngraph TB\n subgraph \"Command Layer Architecture\"\n Strategy[ProcessingStrategy Level-2] -->|creates| Commands[List of Commands tool_name + parameters]\n Commands -->|executes via| Dispatcher[CommandDispatcher]\n \n Dispatcher -->|routes| AIP[AIP Protocol WebSocket]\n AIP -->|sends| Client[Device Client]\n \n Client -->|dispatches to| MCP[MCP Server Manager]\n MCP -->|invokes| Tool1[MCP Tool 1 click_element]\n MCP -->|invokes| Tool2[MCP Tool 2 type_text]\n MCP -->|invokes| Tool3[MCP Tool 3 run_command]\n \n Tool1 -->|result| MCP\n Tool2 -->|result| MCP\n Tool3 -->|result| MCP\n \n MCP -->|aggregates| Client\n Client -->|returns| AIP\n AIP -->|results| Dispatcher\n Dispatcher -->|List of Results| Strategy\n end\n```\n\n## Design Philosophy\n\nThe Command Layer follows the **Command Pattern**:\n\n## Command Structure\n\n## Design Philosophy\n\nThe Command Layer follows the **Command Pattern**:\n\n- **Encapsulation**: Each command encapsulates request as object\n- **Decoupling**: Invoker (strategy) decoupled from executor (MCP tool)\n- **Extensibility**: New commands added without changing invoker code\n- **Auditability**: Command history provides complete execution trace\n\n## Command Structure\n\nEach command is represented by the `Command` Pydantic model:\n\n```python\nclass Command(BaseModel):\n \"\"\"\n Represents a command to be executed by an agent.\n Commands are atomic units of work dispatched by the orchestrator.\n \"\"\"\n \n tool_name: str = Field(..., description=\"Name of the tool to execute\")\n parameters: Optional[Dict[str, Any]] = Field(\n default=None, description=\"Parameters for the tool\"\n )\n tool_type: Literal[\"data_collection\", \"action\"] = Field(\n ..., description=\"Type of tool: data_collection or action\"\n )\n call_id: Optional[str] = Field(\n default=None, description=\"Unique identifier for this command call\"\n )\n```\n\n### Command Properties\n\n| Property | Type | Purpose | Example |\n|----------|------|---------|---------|\n| **tool_name** | `str` | MCP tool name to invoke | `\"click_element\"`, `\"type_text\"`, `\"shell_execute\"` |\n| **parameters** | `Optional[Dict[str, Any]]` | Tool parameters | `{\"control_id\": \"Button_123\"}`, `{\"text\": \"Hello\"}` |\n| **tool_type** | `Literal[\"data_collection\", \"action\"]` | Tool category | `\"data_collection\"` (observation), `\"action\"` (modification) |\n| **call_id** | `Optional[str]` | Unique execution identifier | `\"uuid-1234-5678\"` (auto-generated) |\n\n### Command Examples\n\n**Windows UI Automation Command**:\n\n```python\nCommand(\n tool_name=\"click_element\",\n parameters={\n \"control_id\": \"Button_InsertChart\",\n \"process_name\": \"EXCEL.EXE\",\n \"app_root_name\": \"Microsoft Excel\"\n },\n tool_type=\"action\"\n)\n```\n\n**Linux Shell Command**:\n\n```python\nCommand(\n tool_name=\"shell_execute\",\n parameters={\n \"command\": \"ls -la /home/user/documents\",\n \"timeout\": 30\n },\n tool_type=\"action\"\n)\n```\n\n**File Operation Command**:\n\n```python\nCommand(\n tool_name=\"read_file\",\n parameters={\n \"file_path\": \"/home/user/data.csv\",\n \"encoding\": \"utf-8\"\n },\n tool_type=\"data_collection\"\n)\n```\n\n## Result Structure\n\nEach command execution returns a `Result` object:\n\n```python\nclass Result(BaseModel):\n \"\"\"\n Represents the result of a command execution.\n Contains status, error information, and the actual result payload.\n \"\"\"\n \n status: ResultStatus = Field(..., description=\"Execution status\")\n error: Optional[str] = Field(default=None, description=\"Error message if failed\")\n result: Any = Field(default=None, description=\"Result payload\")\n namespace: Optional[str] = Field(\n default=None, description=\"Namespace of the executed tool\"\n )\n call_id: Optional[str] = Field(\n default=None, description=\"ID matching the Command.call_id\"\n )\n```\n\n### ResultStatus Enum\n\n```python\nclass ResultStatus(str, Enum):\n \"\"\"Represents the status of a command execution result.\"\"\"\n SUCCESS = \"success\" # Command executed successfully\n FAILURE = \"failure\" # Command failed with error\n SKIPPED = \"skipped\" # Command was skipped\n NONE = \"none\" # No result available\n```\n\n### Result Examples\n\n**Successful Click**:\n\n```python\nResult(\n status=ResultStatus.SUCCESS,\n result={\"clicked\": True, \"control_name\": \"Insert Chart\"},\n call_id=\"uuid-1234-5678\"\n)\n```\n\n**Failed Command**:\n\n```python\nResult(\n status=ResultStatus.FAILURE,\n result=None,\n error=\"Control 'Button_123' not found in UI tree\",\n call_id=\"uuid-1234-5678\"\n)\n```\n\n**Shell Execution**:\n\n```python\nResult(\n status=ResultStatus.SUCCESS,\n result={\n \"stdout\": \"total 24\\ndrwxr-xr-x 5 user user 4096...\",\n \"stderr\": \"\",\n \"exit_code\": 0\n },\n call_id=\"uuid-1234-5678\"\n)\n```\n\n## CommandDispatcher Interface\n\nThe `BasicCommandDispatcher` provides the abstract interface for command execution:\n\n```python\nclass BasicCommandDispatcher(ABC):\n \"\"\"\n Abstract base class for command dispatcher.\n \n Responsibilities:\n - Send commands to device client\n - Wait for execution results\n - Handle errors and timeouts\n \"\"\"\n \n @abstractmethod\n async def execute_commands(\n self, commands: List[Command], timeout: float = 6000\n ) -> Optional[List[Result]]:\n \"\"\"\n Execute commands and return results.\n \n :param commands: List of commands to execute\n :param timeout: Execution timeout in seconds\n :return: List of results, or None if timeout\n \"\"\"\n pass\n \n def generate_error_results(\n self, commands: List[Command], error: Exception\n ) -> List[Result]:\n \"\"\"\n Generate error results for failed commands.\n \n :param commands: Commands that failed\n :param error: The error that occurred\n :return: List of error results\n \"\"\"\n result_list = []\n for command in commands:\n error_msg = f\"Error executing {command}: {error}\"\n result = Result(\n status=ResultStatus.FAILURE,\n error=error_msg,\n result=error_msg,\n call_id=command.call_id\n )\n result_list.append(result)\n return result_list\n```\n\n## Command Execution Flow\n\nThe following sequence diagram shows the complete command execution flow:\n\n```mermaid\nsequenceDiagram\n participant Strategy as ProcessingStrategy (ACTION_EXECUTION)\n participant Dispatcher as CommandDispatcher\n participant AIP as AIP Protocol\n participant Client as Device Client\n participant MCP as MCP Server Manager\n participant Tool as MCP Tool\n\n Note over Strategy: Step 1: Create Commands\n Strategy->>Strategy: Build Command objects from LLM response\n \n Note over Strategy: Step 2: Execute via Dispatcher\n Strategy->>Dispatcher: execute_commands([cmd1, cmd2])\n \n Note over Dispatcher: Step 3: Add Call IDs\n Dispatcher->>Dispatcher: Assign unique call_id to each command\n \n Note over Dispatcher: Step 4: Send via AIP\n Dispatcher->>AIP: Send ServerMessage (COMMAND type)\n AIP->>Client: WebSocket message (serialized commands)\n \n Note over Client: Step 5: Route to MCP\n Client->>MCP: Route commands to appropriate MCP server\n \n Note over MCP: Step 6: Execute Tools\n loop For each command\n MCP->>Tool: Invoke tool function with arguments\n Tool->>Tool: Execute operation (click, type, shell, etc.)\n Tool->>MCP: Return result\n end\n \n Note over Client: Step 7: Aggregate Results\n MCP->>Client: List[Result]\n Client->>AIP: Send ClientMessage (RESULT type)\n AIP->>Dispatcher: WebSocket message (serialized results)\n \n Note over Dispatcher: Step 8: Return Results\n Dispatcher->>Strategy: List[Result]\n \n Note over Strategy: Step 9: Process Results\n Strategy->>Strategy: Update context with execution results\n```\n\n### Execution Phases\n\n1. **Command Creation**: Strategy builds `Command` objects from LLM response or predefined logic\n2. **Dispatcher Invocation**: Strategy calls `dispatcher.execute_commands()`\n3. **Call ID Assignment**: Dispatcher assigns unique `call_id` to each command\n4. **AIP Transmission**: Commands serialized and sent via WebSocket to device client\n5. **MCP Routing**: Client routes commands to appropriate MCP server\n6. **Tool Execution**: MCP server invokes tool functions with arguments\n7. **Result Aggregation**: Results collected and sent back via AIP\n8. **Result Handling**: Dispatcher returns results to strategy\n9. **Context Update**: Strategy updates processing context with results\n\n!!! warning \"Timeout Handling\"\n If execution exceeds timeout:\n \n 1. Dispatcher raises `asyncio.TimeoutError`\n 2. Error results generated via `generate_error_results()`\n 3. Strategy receives error results (status = FAILURE)\n 4. Processor can retry or fail based on `fail_fast` setting\n\n---\n\n## Dispatcher Implementations\n\nUFO provides two dispatcher implementations for different deployment scenarios:\n\n### 1. LocalCommandDispatcher\n\n**Purpose**: Direct local execution (server and client on same machine)\n\n**Use Case**: Development, testing, single-device deployments\n\n```python\nclass LocalCommandDispatcher(BasicCommandDispatcher):\n \"\"\"\n Local command dispatcher - executes commands directly.\n \n No network communication - calls MCP tools locally.\n \"\"\"\n \n def __init__(self, session: BaseSession, mcp_server_manager: MCPServerManager):\n self.session = session\n self.mcp_server_manager = mcp_server_manager\n \n # Direct local execution\n self.computer_manager = ComputerManager(configs, mcp_server_manager)\n self.command_router = CommandRouter(self.computer_manager)\n \n async def execute_commands(\n self, commands: List[Command], timeout=6000\n ) -> Optional[List[Result]]:\n \"\"\"Execute commands locally via CommandRouter\"\"\"\n try:\n # Direct invocation (no network)\n action_results = await asyncio.wait_for(\n self.command_router.execute(\n agent_name=self.session.current_agent_class,\n root_name=self.session.context.get(ContextNames.APPLICATION_ROOT_NAME),\n process_name=self.session.context.get(ContextNames.APPLICATION_PROCESS_NAME),\n commands=commands\n ),\n timeout=timeout\n )\n return action_results\n except Exception as e:\n return self.generate_error_results(commands, e)\n```\n\n### 2. WebSocketCommandDispatcher\n\n**Purpose**: Remote execution via AIP (server and client on different machines)\n\n**Use Case**: Production, multi-device deployments, distributed systems\n\n```python\nclass WebSocketCommandDispatcher(BasicCommandDispatcher):\n \"\"\"\n WebSocket command dispatcher - executes commands remotely via AIP.\n \n Uses AIP's TaskExecutionProtocol for structured messaging.\n \"\"\"\n \n def __init__(self, session: BaseSession, protocol: TaskExecutionProtocol):\n self.session = session\n self.protocol = protocol # AIP protocol instance\n self.pending: Dict[str, asyncio.Future] = {}\n self.logger = logging.getLogger(__name__)\n \n async def execute_commands(\n self, commands: List[Command], timeout=6000\n ) -> Optional[List[Result]]:\n \"\"\"Execute commands remotely via AIP WebSocket\"\"\"\n try:\n # Build ServerMessage\n server_msg = self.make_server_response(commands)\n \n # Send via AIP\n await self.protocol.send_command(server_msg)\n \n # Wait for results\n results = await asyncio.wait_for(\n self._wait_for_results(server_msg.response_id),\n timeout=timeout\n )\n \n return results\n except Exception as e:\n return self.generate_error_results(commands, e)\n \n def make_server_response(self, commands: List[Command]) -> ServerMessage:\n \"\"\"Create ServerMessage for commands\"\"\"\n # Assign call_ids\n for command in commands:\n command.call_id = str(uuid.uuid4())\n \n return ServerMessage(\n type=ServerMessageType.COMMAND,\n status=TaskStatus.CONTINUE,\n agent_name=self.session.current_agent_class,\n process_name=self.session.context.get(ContextNames.APPLICATION_PROCESS_NAME),\n root_name=self.session.context.get(ContextNames.APPLICATION_ROOT_NAME),\n actions=commands,\n session_id=self.session.id,\n task_name=self.session.task,\n response_id=str(uuid.uuid4()),\n timestamp=datetime.datetime.now(datetime.timezone.utc).isoformat()\n )\n```\n\n### Dispatcher Selection\n\nThe dispatcher is selected at session initialization:\n\n- **Local Mode**: `LocalCommandDispatcher` (no AIP client connection)\n- **Remote Mode**: `WebSocketCommandDispatcher` (AIP client connected)\n\nThis is transparent to strategies - they call `dispatcher.execute_commands()` regardless.\n\n## MCP Integration\n\nCommands map to **Model Context Protocol (MCP)** tools on the device client:\n\n```mermaid\ngraph TB\n subgraph \"Device Client\"\n Router[Command Router]\n Manager[MCP Server Manager]\n \n Router -->|routes by agent/app| Manager\n end\n \n subgraph \"MCP Servers\"\n Win[Windows MCP Server ufo_windows]\n Linux[Linux MCP Server ufo_linux]\n Custom[Custom MCP Server user_defined]\n end\n \n Manager -->|manages| Win\n Manager -->|manages| Linux\n Manager -->|manages| Custom\n \n subgraph \"MCP Tools (Windows)\"\n Win -->|provides| T1[click_element]\n Win -->|provides| T2[type_text]\n Win -->|provides| T3[get_ui_tree]\n Win -->|provides| T4[screenshot]\n end\n \n subgraph \"MCP Tools (Linux)\"\n Linux -->|provides| T5[shell_execute]\n Linux -->|provides| T6[read_file]\n Linux -->|provides| T7[write_file]\n end\n \n Command[Command function + arguments] -->|routed to| Router\n```\n\n### MCP Tool Registration\n\nMCP tools are registered on the device client at initialization:\n\n```python\n# Windows MCP Server registration\nmcp_server_manager.register_server(\n server_name=\"ufo_windows\",\n tools=[\n MCPToolInfo(\n name=\"click_element\",\n description=\"Click UI element by control ID\",\n arguments_schema={\n \"control_id\": {\"type\": \"string\", \"required\": True},\n \"process_name\": {\"type\": \"string\", \"required\": True}\n }\n ),\n MCPToolInfo(\n name=\"type_text\",\n description=\"Type text into focused element\",\n arguments_schema={\n \"text\": {\"type\": \"string\", \"required\": True}\n }\n ),\n # ... more tools\n ]\n)\n```\n\n### Tool Discovery\n\nThe LLM can query available tools via the `get_mcp_tools` command:\n\n```python\n# Strategy requests available tools\ntools_cmd = Command(\n function=\"get_mcp_tools\",\n arguments={\"server_name\": \"ufo_windows\"}\n)\n\nresults = await dispatcher.execute_commands([tools_cmd])\n\n# LLM receives tool registry\navailable_tools = results[0].result # List[MCPToolInfo]\n```\n\n### Dynamic Tool Selection\n\nThe LLM dynamically selects appropriate tools based on:\n\n1. **Tool Descriptions**: Natural language descriptions of tool capabilities\n2. **Input Schemas**: Required/optional parameters with types\n3. **Context**: Current task requirements and device state\n\nThis enables **adaptive behavior** without hardcoded command sequences.\n\nSee [MCP Documentation](../../../mcp/overview.md) for complete MCP integration details.\n\n## Command Categories\n\nCommands can be categorized by their purpose:\n\n### 1. Observation Commands (DATA_COLLECTION)\n\n**Purpose**: Gather information from device without modifying state\n\n| Command | Platform | Description | Arguments | Result |\n|---------|----------|-------------|-----------|--------|\n| `screenshot` | All | Capture screen image | `{\"region\": \"fullscreen\"}` | Base64 image |\n| `get_ui_tree` | Windows | Extract UI Automation tree | `{\"process_name\": \"...\"}` | XML/JSON tree |\n| `shell_execute_read` | Linux | Execute shell command (read-only) | `{\"command\": \"ls -la\"}` | stdout/stderr |\n| `get_accessibility_tree` | macOS | Extract Accessibility tree | `{\"app_name\": \"...\"}` | Tree structure |\n\n### 2. Action Commands (ACTION_EXECUTION)\n\n**Purpose**: Modify device state through interactions\n\n| Command | Platform | Description | Arguments | Result |\n|---------|----------|-------------|-----------|--------|\n| `click_element` | Windows | Click UI element | `{\"control_id\": \"...\"}` | Click success |\n| `type_text` | Windows | Type text into element | `{\"text\": \"...\"}` | Typing success |\n| `scroll` | Windows | Scroll element | `{\"direction\": \"down\", \"amount\": 3}` | Scroll success |\n| `shell_execute` | Linux | Execute shell command | `{\"command\": \"...\", \"timeout\": 30}` | stdout/stderr/exit_code |\n| `press_key` | All | Press keyboard key | `{\"key\": \"Enter\"}` | Key press success |\n\n### 3. System Commands\n\n**Purpose**: Interact with OS or hardware\n\n| Command | Platform | Description | Arguments | Result |\n|---------|----------|-------------|-----------|--------|\n| `launch_application` | All | Start application | `{\"app_name\": \"...\", \"args\": [...]}` | PID |\n| `close_application` | All | Terminate application | `{\"process_name\": \"...\"}` | Close success |\n| `read_file` | All | Read file contents | `{\"file_path\": \"...\", \"encoding\": \"utf-8\"}` | File contents |\n| `write_file` | All | Write file contents | `{\"file_path\": \"...\", \"content\": \"...\"}` | Write success |\n| `get_system_info` | All | Query system status | `{\"info_type\": \"cpu\"}` | CPU/memory/disk stats |\n\n### Command Naming Convention\n\n- Use **snake_case** for function names\n- Use **verb_noun** pattern (e.g., `click_element`, `read_file`)\n- Keep names **concise** but **descriptive**\n- Prefix with platform if platform-specific (e.g., `windows_get_registry`)\n\n## Command Validation\n\nCommands are validated at multiple stages:\n\n### 1. Client-Side Validation\n\nThe device client validates commands before execution:\n\n```python\nclass CommandRouter:\n \"\"\"Routes and validates commands on device client\"\"\"\n \n async def execute(\n self,\n agent_name: str,\n root_name: str,\n process_name: str,\n commands: List[Command]\n ) -> List[Result]:\n \"\"\"Execute commands with validation\"\"\"\n results = []\n \n for command in commands:\n # Validate command\n validation_error = self._validate_command(command)\n if validation_error:\n results.append(Result(\n status=ResultStatus.FAILURE,\n error=validation_error,\n call_id=command.call_id\n ))\n continue\n \n # Execute command\n try:\n result = await self._execute_single_command(command)\n results.append(result)\n except Exception as e:\n results.append(Result(\n status=ResultStatus.FAILURE,\n error=str(e),\n call_id=command.call_id\n ))\n \n return results\n \n def _validate_command(self, command: Command) -> Optional[str]:\n \"\"\"Validate command structure and arguments\"\"\"\n # Check function exists\n tool_info = self.mcp_server_manager.get_tool(command.function)\n if not tool_info:\n return f\"Unknown command: {command.function}\"\n \n # Check required arguments\n schema = tool_info.arguments_schema\n for arg_name, arg_spec in schema.items():\n if arg_spec.get(\"required\") and arg_name not in command.arguments:\n return f\"Missing required argument: {arg_name}\"\n \n # Check argument types\n for arg_name, arg_value in command.arguments.items():\n expected_type = schema.get(arg_name, {}).get(\"type\")\n if expected_type and not self._check_type(arg_value, expected_type):\n return f\"Argument '{arg_name}' has wrong type\"\n \n return None # Valid\n```\n\n### 2. MCP Schema Validation\n\nMCP tools define argument schemas that are enforced:\n\n```python\n{\n \"name\": \"click_element\",\n \"description\": \"Click UI element by control ID\",\n \"arguments_schema\": {\n \"control_id\": {\n \"type\": \"string\",\n \"required\": True,\n \"description\": \"Unique identifier of control to click\"\n },\n \"process_name\": {\n \"type\": \"string\",\n \"required\": True,\n \"description\": \"Process name of application\"\n },\n \"double_click\": {\n \"type\": \"boolean\",\n \"required\": False,\n \"default\": False,\n \"description\": \"Whether to double-click\"\n }\n }\n}\n```\n\n### Validation Benefits\n\n- **Early Error Detection**: Invalid commands caught before execution\n- **Clear Error Messages**: Specific validation failures reported\n- **Type Safety**: Argument types validated against schema\n- **Security**: Prevents injection attacks and malformed requests\n\n## Command Execution Patterns\n\nCommon patterns for command execution in strategies:\n\n### 1. Single Command Execution\n\n```python\n# Execute single command\nasync def execute(self, agent, context):\n command = Command(\n function=\"screenshot\",\n arguments={\"region\": \"fullscreen\"}\n )\n \n results = await context.command_dispatcher.execute_commands([command])\n \n if results[0].status == ResultStatus.SUCCESS:\n screenshot = results[0].result\n context.update_local({\"screenshot\": screenshot})\n return ProcessingResult(success=True, data={\"screenshot\": screenshot})\n else:\n return ProcessingResult(success=False, error=results[0].error)\n```\n\n### 2. Batch Command Execution\n\n```python\n# Execute multiple commands in parallel\nasync def execute(self, agent, context):\n commands = [\n Command(function=\"screenshot\", arguments={}),\n Command(function=\"get_ui_tree\", arguments={\"process_name\": \"EXCEL.EXE\"}),\n Command(function=\"get_system_info\", arguments={\"info_type\": \"memory\"})\n ]\n \n results = await context.command_dispatcher.execute_commands(commands)\n \n # Process results\n data = {\n \"screenshot\": results[0].result if results[0].status == ResultStatus.SUCCESS else None,\n \"ui_tree\": results[1].result if results[1].status == ResultStatus.SUCCESS else None,\n \"memory_info\": results[2].result if results[2].status == ResultStatus.SUCCESS else None\n }\n \n return ProcessingResult(success=True, data=data)\n```\n\n### 3. Conditional Command Execution\n\n```python\n# Execute command based on LLM decision\nasync def execute(self, agent, context):\n parsed_response = context.require_local(\"parsed_response\")\n action = parsed_response.get(\"ControlText\")\n \n if action == \"click\":\n command = Command(\n function=\"click_element\",\n arguments={\"control_id\": parsed_response.get(\"ControlID\")}\n )\n elif action == \"type\":\n command = Command(\n function=\"type_text\",\n arguments={\"text\": parsed_response.get(\"InputText\")}\n )\n else:\n return ProcessingResult(success=False, error=f\"Unknown action: {action}\")\n \n results = await context.command_dispatcher.execute_commands([command])\n return ProcessingResult(success=results[0].status == ResultStatus.SUCCESS)\n```\n\n### 4. Retry Pattern\n\n```python\n# Retry command on failure\nasync def execute(self, agent, context):\n command = Command(\n function=\"click_element\",\n arguments={\"control_id\": \"Button_123\"}\n )\n \n max_retries = 3\n for attempt in range(max_retries):\n results = await context.command_dispatcher.execute_commands([command])\n \n if results[0].status == ResultStatus.SUCCESS:\n return ProcessingResult(success=True, data=results[0].result)\n \n # Retry with exponential backoff\n await asyncio.sleep(2 ** attempt)\n \n return ProcessingResult(success=False, error=\"Max retries exceeded\")\n```\n\n## Best Practices\n\n### Command Design Guidelines\n\n**1. Atomic Operations**: Each command should perform one well-defined operation\n\n- ✅ Good: `click_element(control_id=\"Button_123\")`\n- ❌ Bad: `click_and_wait_and_validate(...)` (too many responsibilities)\n\n**2. Idempotency**: Commands should be safe to retry\n\n- ✅ Good: `read_file(path=\"/data.csv\")` (idempotent)\n- ⚠️ Caution: `append_to_file(path=\"/log.txt\", text=\"...\")` (not idempotent)\n\n**3. Clear Arguments**: Use descriptive argument names\n\n- ✅ Good: `{\"file_path\": \"...\", \"encoding\": \"utf-8\"}`\n- ❌ Bad: `{\"p\": \"...\", \"e\": \"utf-8\"}` (unclear)\n\n**4. Structured Results**: Return structured data, not just strings\n\n- ✅ Good: `{\"stdout\": \"...\", \"stderr\": \"...\", \"exit_code\": 0}`\n- ❌ Bad: `\"output: ... error: ... code: 0\"` (unstructured)\n\n### Security Considerations\n\n!!! warning \"Security Best Practices\"\n **Validate All Inputs**: Never trust command arguments from LLM without validation\n \n **Limit Command Scope**: Restrict commands to necessary operations only\n \n - Use MCP tool permissions to limit file access\n - Sandbox shell command execution\n - Validate file paths against allowed directories\n \n **Audit Command History**: Log all commands for compliance\n \n ```python\n self.logger.info(f\"Executing command: {command.tool_name} with args: {command.parameters}\")\n ```\n \n **Timeout All Commands**: Prevent runaway execution\n \n ```python\n results = await dispatcher.execute_commands(commands, timeout=30)\n ```\n\n!!! danger \"Dangerous Commands\"\n Some commands require extra caution:\n \n **Shell Execution**: Risk of command injection\n \n - Use argument escaping/sanitization\n - Whitelist allowed commands\n - Never concatenate user input directly\n \n **File Operations**: Risk of unauthorized access\n \n - Validate paths against allowed directories\n - Check file permissions before access\n - Never allow arbitrary path traversal\n \n **System Modification**: Risk of breaking system state\n \n - Require explicit user confirmation\n - Implement undo/rollback mechanisms\n - Never allow destructive ops without safeguards\n\n## Integration with Other Layers\n\nThe Command Layer integrates with other components:\n\n```mermaid\ngraph TB\n subgraph \"Strategy Layer (Level-2)\"\n AE[ACTION_EXECUTION Strategy]\n DC[DATA_COLLECTION Strategy]\n end\n \n subgraph \"Command Layer (Level-3)\"\n Dispatcher[CommandDispatcher]\n Commands[Commands]\n Results[Results]\n end\n \n subgraph \"Communication\"\n AIP[AIP Protocol]\n end\n \n subgraph \"Device Client\"\n MCP[MCP Servers]\n Tools[MCP Tools]\n end\n \n AE -->|creates| Commands\n DC -->|creates| Commands\n Commands -->|via| Dispatcher\n Dispatcher -->|via| AIP\n AIP -->|routes to| MCP\n MCP -->|invokes| Tools\n Tools -->|results via| MCP\n MCP -->|via| AIP\n AIP -->|via| Dispatcher\n Dispatcher -->|returns| Results\n Results -->|used by| AE\n Results -->|used by| DC\n```\n\n| Integration Point | Layer/Component | Relationship |\n|-------------------|-----------------|--------------|\n| **ProcessingStrategy** | Level-2 Strategy | Strategies create and execute commands via dispatcher |\n| **AIP Protocol** | Communication | Dispatcher uses AIP to send commands to client |\n| **Device Client** | Execution | Client receives commands, routes to MCP servers |\n| **MCP Servers** | Tool Registry | MCP servers execute tool functions, return results |\n| **Global Context** | Module System | Command dispatcher accessed via processing context |\n\nSee [Strategy Layer](processor.md), [AIP Protocol](../../../aip/overview.md), and [MCP Integration](../../../mcp/overview.md) for integration details.\n\n## API Reference\n\nBelow is the complete API reference for the Command Layer:\n\n**BasicCommandDispatcher** (Abstract Base Class)\n```python\n# Location: ufo/module/dispatcher.py\nclass BasicCommandDispatcher(ABC):\n \"\"\"Abstract base class for command dispatcher handling.\"\"\"\n \n @abstractmethod\n async def execute_commands(\n self, commands: List[Command], timeout: float = 6000\n ) -> Optional[List[Result]]:\n \"\"\"Execute commands and return results.\"\"\"\n pass\n```\n\n**LocalCommandDispatcher** (Local Execution)\n```python\n# Location: ufo/module/dispatcher.py\nclass LocalCommandDispatcher(BasicCommandDispatcher):\n \"\"\"Command dispatcher for local execution (testing/development).\"\"\"\n pass\n```\n\n**WebSocketCommandDispatcher** (Server-Client Communication)\n```python\n# Location: ufo/module/dispatcher.py\nclass WebSocketCommandDispatcher(BasicCommandDispatcher):\n \"\"\"Command dispatcher using WebSocket/AIP protocol.\"\"\"\n pass\n```\n\n## Summary\n\n**Key Takeaways**:\n\n- **Atomic Execution**: Commands are self-contained units with tool_name + parameters\n- **MCP Integration**: Commands map to Model Context Protocol tools on device client\n- **CommandDispatcher**: Routes commands from server to client via AIP\n- **Deterministic**: Same inputs → same outputs, fully auditable\n- **Dynamic Discovery**: LLM queries and selects appropriate tools at runtime\n- **Validation**: Multi-stage validation (client + MCP schema) ensures safety\n- **Extensibility**: New commands added via MCP tool registration without code changes\n\nThe Command Layer completes the three-layer device agent architecture, providing **reliable, auditable, and extensible system interfaces** that bridge high-level reasoning with low-level device operations across heterogeneous platforms.\n"
},
{
"path": "documents/docs/infrastructure/agents/design/memory.md",
"content": "# Memory System\n\nThe Memory System provides both short-term and long-term memory capabilities for Device Agents in UFO3. The system consists of two primary components: **Memory** (agent-specific execution history) and **Blackboard** (shared multi-agent communication). This dual-memory architecture enables agents to maintain their own execution context while coordinating seamlessly across devices and sessions.\n\n## Overview\n\nThe Memory System supports the Device Agent architecture through two distinct but complementary mechanisms:\n\n```mermaid\ngraph TB\n subgraph \"Memory System Architecture\"\n Agent1[Agent Instance]\n Agent2[Agent Instance]\n AgentN[Agent Instance N]\n \n Memory1[Memory Short-term]\n Memory2[Memory Short-term]\n MemoryN[Memory Short-term]\n \n Blackboard[Blackboard Long-term Shared]\n \n Agent1 --> Memory1\n Agent2 --> Memory2\n AgentN --> MemoryN\n \n Agent1 -.Share.-> Blackboard\n Agent2 -.Share.-> Blackboard\n AgentN -.Share.-> Blackboard\n \n Blackboard -.Read.-> Agent1\n Blackboard -.Read.-> Agent2\n Blackboard -.Read.-> AgentN\n end\n \n style Memory1 fill:#e1f5ff\n style Memory2 fill:#e1f5ff\n style MemoryN fill:#e1f5ff\n style Blackboard fill:#fff4e1\n```\n\n| Component | Scope | Persistence | Primary Use Case |\n|-----------|-------|-------------|------------------|\n| **Memory** | Agent-specific | Session lifetime | Execution history, context tracking |\n| **Blackboard** | Multi-agent shared | Configurable (file-backed) | Cross-agent coordination, information sharing |\n\n**Design Benefits:**\n- **Separation of Concerns**: Agent-specific history isolated from shared state\n- **Scalability**: Each agent manages own memory independently\n- **Coordination**: Blackboard enables multi-agent communication without tight coupling\n- **Persistence**: Blackboard can survive session restarts (file-backed storage)\n\n---\n\n## Memory (Short-term Agent Memory)\n\nThe `Memory` class manages the **short-term execution history** of a single agent. Each agent instance has its own `Memory` that records every interaction step, forming a chronological execution trace.\n\n### Memory Architecture\n\n```mermaid\ngraph LR\n subgraph \"Memory Lifecycle\"\n Step1[Step 1 MemoryItem]\n Step2[Step 2 MemoryItem]\n Step3[Step 3 MemoryItem]\n StepN[Step N MemoryItem]\n \n Step1 --> Step2\n Step2 --> Step3\n Step3 --> StepN\n end\n \n subgraph \"MemoryItem Contents\"\n Screenshot[Screenshot]\n Action[Action Taken]\n Result[Execution Result]\n Observation[UI Observation]\n Cost[LLM Cost]\n end\n \n StepN --> Screenshot\n StepN --> Action\n StepN --> Result\n StepN --> Observation\n StepN --> Cost\n \n style Step1 fill:#e1f5ff\n style Step2 fill:#e1f5ff\n style Step3 fill:#e1f5ff\n style StepN fill:#e1f5ff\n```\n\n### MemoryItem Structure\n\nA `MemoryItem` is a flexible dataclass that represents a **single execution step** in the agent's history. The structure is customizable to accommodate different agent types and platforms.\n\n::: agents.memory.memory.MemoryItem\n\n#### Common MemoryItem Fields\n\n| Field | Type | Description | Usage in Strategies |\n|-------|------|-------------|---------------------|\n| `step` | `int` | Execution step number | Tracking execution progress |\n| `screenshot` | `str` (path) | Screenshot file path | Visual context for LLM reasoning |\n| `action` | `str` | Action function name | Execution history, replay |\n| `arguments` | `Dict[str, Any]` | Action arguments | Debugging, audit logging |\n| `results` | `List[Result]` | Command execution results | Success/failure tracking |\n| `observation` | `str` | UI element descriptions | LLM prompt context |\n| `control_text` | `str` | UI text content | Element identification |\n| `request` | `str` | User request at this step | Task context |\n| `response` | `str` | LLM raw response | Debugging LLM decisions |\n| `parsed_response` | `Dict` | Parsed LLM output | Structured action extraction |\n| `cost` | `float` | LLM API cost | Budget tracking |\n| `error` | `Optional[str]` | Error message if failed | Error recovery |\n\n**Example: Creating a MemoryItem**\n ```python\n from ufo.agents.memory.memory import MemoryItem\n \n # After executing a step, create memory item\n memory_item = MemoryItem(\n step=3,\n screenshot=\"screenshots/step_3.png\",\n action=\"click_element\",\n arguments={\"element_id\": \"submit_button\"},\n results=[Result(status=ResultStatus.SUCCESS, result=\"Button clicked\")],\n observation=\"Submit button located at (500, 300)\",\n request=\"Submit the form\",\n response='{\"action\": \"click_element\", \"element\": \"submit_button\"}',\n parsed_response={\"action\": \"click_element\", \"element\": \"submit_button\"},\n cost=0.0023\n )\n ```\n\n**Note on Flexible Schema:**\n`MemoryItem` uses a flexible dataclass structure. Agent implementations can add custom fields based on their specific requirements. For example, Windows agents might add `ui_automation_info`, while Linux agents might add `shell_output`.\n\n### Memory Class\n\nThe `Memory` class manages a **list of MemoryItem instances**, providing methods to add, retrieve, and filter execution history.\n\n::: agents.memory.memory.Memory\n\n#### Key Methods\n\n| Method | Purpose | Usage |\n|--------|---------|-------|\n| `add_memory_item(item)` | Append new execution step | Called by `MEMORY_UPDATE` strategy after each step |\n| `get_latest_item()` | Retrieve the most recent item | Get the last execution step |\n| `filter_memory_from_keys(keys)` | Filter items by specific keys | Build LLM prompt with selected fields |\n| `filter_memory_from_steps(steps)` | Filter items by step numbers | Retrieve specific execution steps |\n| `clear()` | Reset memory | New session initialization |\n| `is_empty()` | Check if memory is empty | Validate memory state |\n\n**Example: Using Memory in Processor**\n ```python\n from ufo.agents.processors.strategies.memory_strategies import MemoryUpdateStrategy\n from ufo.agents.memory.memory import Memory, MemoryItem\n \n class AppAgentProcessor(ProcessorTemplate):\n def __init__(self, agent, context):\n super().__init__(agent, context)\n self.memory = Memory() # Agent-specific memory\n \n # MEMORY_UPDATE strategy adds items to memory\n self.register_strategy(\n ProcessingPhase.MEMORY_UPDATE,\n MemoryUpdateStrategy(agent, context, self.memory)\n )\n \n def get_prompt_context(self) -> str:\n \"\"\"Build LLM prompt with recent execution history.\"\"\"\n # Get recent steps using content property\n all_steps = self.memory.content\n recent_steps = all_steps[-5:] if len(all_steps) > 5 else all_steps\n \n context = \"## Recent Execution History:\\n\"\n for item in recent_steps:\n context += f\"Step {item.get_value('step')}: {item.get_value('action')}\"\n context += f\"({item.get_value('arguments')}) -> {item.get_value('results')}\\n\"\n \n return context\n ```\n\n#### Memory Lifecycle\n\n```mermaid\nsequenceDiagram\n participant Processor\n participant Memory\n participant MemoryUpdateStrategy\n \n Note over Processor: Agent starts session\n Processor->>Memory: Initialize Memory()\n \n loop Each Execution Step\n Note over Processor: Execute strategies\n Processor->>MemoryUpdateStrategy: execute()\n MemoryUpdateStrategy->>MemoryUpdateStrategy: Create MemoryItem from context\n MemoryUpdateStrategy->>Memory: add_memory_item(item)\n Memory->>Memory: Append to internal list\n end\n \n Note over Processor: Need prompt context\n Processor->>Memory: content property (get all)\n Memory-->>Processor: List[MemoryItem]\n \n Note over Processor: Session ends\n Processor->>Memory: clear()\n```\n\n**Memory Management Best Practices:**\n- **Limited Context**: When building LLM prompts, use the `content` property and slice for recent items to avoid token limits\n- **Selective Fields**: Only include relevant MemoryItem fields in prompts (e.g., action + results, not raw screenshots)\n- **Error Analysis**: Use `filter_memory_from_keys()` to extract specific information patterns\n\n---\n\n## Blackboard (Long-term Shared Memory)\n\nThe `Blackboard` class implements the **Blackboard Pattern** for multi-agent coordination. It provides a shared memory space where agents can read and write information that persists across sessions and is accessible to all agents.\n\n### Blackboard Pattern\n\nThe Blackboard Pattern is a well-known architectural pattern for multi-agent systems:\n\n```mermaid\ngraph TB\n subgraph \"Blackboard Pattern\"\n BB[Blackboard Shared Knowledge Space]\n \n HostAgent[HostAgent Windows]\n AppAgent[AppAgent Windows]\n \n HostAgent -->|Write: questions| BB\n HostAgent -->|Write: requests| BB\n AppAgent -->|Read: requests| BB\n AppAgent -->|Write: trajectories| BB\n \n BB -.Persist.-> FileStorage[(JSON/JSONL)]\n end\n \n style BB fill:#fff4e1\n style HostAgent fill:#e1f5ff\n style AppAgent fill:#e1f5ff\n```\n\n**Blackboard Pattern Characteristics:**\n- **Centralized Knowledge**: All agents read/write from a single shared space\n- **Loose Coupling**: Agents don't directly communicate; they interact via blackboard\n- **Opportunistic Scheduling**: Agents can act when relevant information appears on blackboard\n- **Persistence**: Knowledge survives agent restarts and session boundaries\n\n### Blackboard Architecture\n\nThe Blackboard is organized with four main memory components, each storing a list of `MemoryItem` objects:\n\n```python\n# Blackboard internal structure\nclass Blackboard:\n _questions: Memory # Q&A pairs with user\n _requests: Memory # Historical user requests\n _trajectories: Memory # Step-wise execution history\n _screenshots: Memory # Important screenshots\n```\n\nEach component is a `Memory` object that stores `MemoryItem` instances with flexible key-value pairs.\n\n### Blackboard Class\n\n::: agents.memory.blackboard.Blackboard\n\n#### Key Methods\n\n| Method | Purpose | Example Usage |\n|--------|---------|---------------|\n| `add_questions(item)` | Add Q&A with user | Store user clarification dialogs |\n| `add_requests(item)` | Add user request | Track historical user requests |\n| `add_trajectories(item)` | Add execution step | Record agent actions and decisions |\n| `add_image(path, metadata)` | Add screenshot | Save important UI states |\n| `blackboard_to_prompt()` | Convert to LLM prompt | Build context for agent inference |\n| `blackboard_to_dict()` | Export as dictionary | Serialize for persistence |\n| `blackboard_from_dict(data)` | Import from dictionary | Restore from persistence |\n| `clear()` | Reset blackboard | New session initialization |\n| `is_empty()` | Check if empty | Validate blackboard state |\n\n**Example: Multi-Agent Coordination via Blackboard**\n ```python\n from ufo.agents.memory.blackboard import Blackboard\n \n # Initialize shared blackboard\n blackboard = Blackboard()\n \n # HostAgent adds user request to blackboard\n class HostAgent:\n def handle(self, context):\n # ... process user request ...\n user_request = \"Create a presentation about AI\"\n \n # Write to blackboard for AppAgent to read\n blackboard.add_requests({\"request\": user_request, \"timestamp\": \"2025-11-12\"})\n blackboard.add_trajectories({\n \"step\": 1,\n \"agent\": \"HostAgent\",\n \"action\": \"delegate_task\",\n \"app\": \"PowerPoint\"\n })\n \n # Delegate to AppAgent\n return AgentStatus.CONTINUE, AppAgent\n \n # AppAgent reads from blackboard and performs task\n class AppAgent:\n def handle(self, context):\n # Read from blackboard\n recent_requests = blackboard.requests.content\n if recent_requests:\n last_request = recent_requests[-1]\n print(f\"AppAgent working on: {last_request.get_value('request')}\")\n \n # ... perform actions ...\n \n # Write task result back to blackboard\n blackboard.add_trajectories({\n \"step\": 2,\n \"agent\": \"AppAgent\",\n \"action\": \"create_presentation\",\n \"status\": \"completed\"\n })\n \n return AgentStatus.FINISH, None\n ```\n\n### Blackboard Persistence\n\nThe Blackboard supports serialization for session recovery:\n\n```mermaid\ngraph LR\n subgraph \"Session Lifecycle\"\n Start[Session Start]\n Execute[Agent Execution]\n End[Session End]\n \n Start --> Execute\n Execute --> End\n end\n \n subgraph \"Serialization\"\n Dict[blackboard_to_dict]\n JSON[JSON Format]\n end\n \n Execute --> Dict\n Dict --> JSON\n \n style Execute fill:#ffe1e1\n```\n\n**Example: Blackboard Serialization**\n ```python\n from ufo.agents.memory.blackboard import Blackboard\n import json\n \n # Create and use blackboard\n blackboard = Blackboard()\n blackboard.add_requests({\"request\": \"Create chart\", \"priority\": \"high\"})\n blackboard.add_trajectories({\"step\": 1, \"action\": \"open_excel\"})\n \n # Serialize to dictionary\n blackboard_dict = blackboard.blackboard_to_dict()\n \n # Save to file\n with open(\"blackboard_state.json\", \"w\") as f:\n json.dump(blackboard_dict, f)\n \n # Later, restore from file\n new_blackboard = Blackboard()\n with open(\"blackboard_state.json\", \"r\") as f:\n loaded_dict = json.load(f)\n new_blackboard.blackboard_from_dict(loaded_dict)\n ```\n\n---\n\n## Memory Types and Usage Patterns\n\nThe Memory System supports different types of information storage based on use cases:\n\n| Memory Type | Storage Location | Persistence | Access Pattern | Primary Use Case |\n|-------------|------------------|-------------|----------------|------------------|\n| **Execution History** | Memory (agent-specific) | Session lifetime | Sequential, recent-first | LLM context, debugging |\n| **Shared State** | Blackboard | File-backed | Key-value lookup | Multi-agent coordination |\n| **Session Context** | Blackboard | File-backed | Hierarchical access | Session recovery, checkpoints |\n| **Global Trajectories** | Blackboard | JSONL append | Sequential log | Audit trail, analytics |\n\n### Common Memory Patterns\n\n#### Pattern 1: Recent Context for LLM Prompts\n\n```python\n# Use Memory.content property to get recent execution context\nall_items = agent.memory.content\nrecent_steps = all_items[-5:] if len(all_items) > 5 else all_items\nprompt_context = \"\\n\".join([\n f\"Step {item.get_value('step')}: {item.get_value('action')}\"\n for item in recent_steps\n])\n```\n\n#### Pattern 2: Multi-Agent Information Sharing\n\n```python\n# HostAgent writes to Blackboard\nblackboard.add_requests({\"request\": \"Create Excel chart\", \"app\": \"Excel\"})\n\n# AppAgent reads from Blackboard\nrequests = blackboard.requests.content\nif requests:\n latest_request = requests[-1]\n app = latest_request.get_value(\"app\")\n```\n\n#### Pattern 3: Execution History Tracking\n\n```python\n# Record each step in trajectories\nblackboard.add_trajectories({\n \"step\": 1,\n \"agent\": \"AppAgent\",\n \"action\": \"click_button\",\n \"target\": \"Save\",\n \"status\": \"success\"\n})\n\n# Later, review execution history\nhistory = blackboard.trajectories.content\nfor item in history:\n print(f\"Step {item.get_value('step')}: {item.get_value('action')}\")\n```\n\n#### Pattern 4: Screenshot Memory\n\n```python\n# Save important UI state with metadata\nblackboard.add_image(\n screenshot_path=\"screenshots/step_5.png\",\n metadata={\"step\": 5, \"description\": \"Before form submission\"}\n)\n\n# Access screenshots for review\nscreenshots = blackboard.screenshots.content\nfor screenshot in screenshots:\n metadata = screenshot.get_value(\"metadata\")\n path = screenshot.get_value(\"image_path\")\n```\n\n---\n\n## Integration with Agent Architecture\n\nThe Memory System integrates with all three architectural layers:\n\n```mermaid\ngraph TB\n subgraph \"Level-1: State Layer\"\n State[AgentState.handle]\n end\n \n subgraph \"Level-2: Strategy Layer\"\n DataCollection[DATA_COLLECTION Strategy]\n LLMInteraction[LLM_INTERACTION Strategy]\n ActionExecution[ACTION_EXECUTION Strategy]\n MemoryUpdate[MEMORY_UPDATE Strategy]\n end\n \n subgraph \"Memory System\"\n Memory[Memory Short-term]\n Blackboard[Blackboard Long-term]\n end\n \n State --> DataCollection\n DataCollection --> LLMInteraction\n LLMInteraction --> ActionExecution\n ActionExecution --> MemoryUpdate\n \n MemoryUpdate -->|Write| Memory\n LLMInteraction -.Read Context.-> Memory\n \n State -.Read/Write.-> Blackboard\n MemoryUpdate -.Write Trajectories.-> Blackboard\n \n style Memory fill:#e1f5ff\n style Blackboard fill:#fff4e1\n style MemoryUpdate fill:#ffe1e1\n```\n\n### Integration Points\n\n| Component | Interaction with Memory | Interaction with Blackboard |\n|-----------|-------------------------|----------------------------|\n| **AgentState.handle()** | - | Read shared state, write delegation info |\n| **DATA_COLLECTION Strategy** | Read recent steps for context | - |\n| **LLM_INTERACTION Strategy** | Read history for prompt building | - |\n| **ACTION_EXECUTION Strategy** | - | - |\n| **MEMORY_UPDATE Strategy** | Write MemoryItem after each step | Write execution trajectories |\n| **ProcessorTemplate** | Maintain agent-specific Memory instance | Access shared Blackboard instance |\n\n**Memory vs Blackboard Decision Guide:**\n\nUse Memory when:\n- Information is agent-specific (execution history)\n- Data is only needed during current session\n- Building LLM prompts with recent context\n- Tracking agent's own performance\n\nUse Blackboard when:\n- Information needs to be shared across agents\n- Data should persist across session restarts\n- Coordinating multi-agent workflows\n- Implementing handoffs between agents\n- Storing global task state\n\n---\n\n## Best Practices\n\n### Memory Management\n\n**Limit Memory Size:**\n```python\n# Prevent unbounded memory growth\nclass Memory:\n MAX_ITEMS = 100\n \n def add_memory_item(self, item):\n self._content.append(item)\n if len(self._content) > self.MAX_ITEMS:\n self._content = self._content[-self.MAX_ITEMS:] # Keep latest 100\n```\n\n**Selective Context for LLM:**\n```python\n# Don't send full MemoryItem objects to LLM\ndef build_prompt_context(memory):\n all_items = memory.content\n recent = all_items[-5:] if len(all_items) > 5 else all_items\n return \"\\n\".join([\n f\"Step {item.get_value('step')}: {item.get_value('action')} -> \"\n f\"{item.get_value('status')}\"\n for item in recent\n ])\n```\n\n**Avoid Storing Large Binary Data:**\nStore file paths instead of file contents in MemoryItem:\n```python\n# Good: Store path\nmemory_item.set_value(\"screenshot\", \"screenshots/step_3.png\")\n\n# Bad: Store binary data\n# memory_item.set_value(\"screenshot\", )\n```\n\n### Blackboard Management\n\n**Organize with Descriptive Keys:**\n```python\n# Use descriptive keys in MemoryItem dictionaries\nblackboard.add_trajectories({\n \"step\": 1,\n \"agent\": \"HostAgent\",\n \"action\": \"select_app\",\n \"app_name\": \"Word\",\n \"timestamp\": \"2025-11-12T10:00:00\"\n})\n```\n\n**Regular Serialization:**\n```python\n# Periodically save blackboard state\nclass Session:\n def __init__(self):\n self.blackboard = Blackboard()\n self.save_interval = 10 # Every 10 steps\n \n def execute_step(self, step_num):\n # ... execute step ...\n \n if step_num % self.save_interval == 0:\n state = self.blackboard.blackboard_to_dict()\n with open(\"blackboard_backup.json\", \"w\") as f:\n json.dump(state, f)\n```\n\n**Clean Up Appropriately:**\n```python\n# Clear blackboard when starting new session\nif new_session:\n blackboard.clear()\n```\n\n---\n\n## Common Pitfalls\n\n**Pitfall 1: Confusing Memory and Blackboard Scope**\n\nProblem: Storing agent-specific data in Blackboard or shared data in Memory.\n\nSolution: Follow the scope principle:\n- Memory = agent-specific, session-lifetime\n- Blackboard = multi-agent shared, persistent\n\n```python\n# Correct\nagent.memory.add_memory_item(...) # Agent's own history\nblackboard.add_trajectories({...}) # Shared execution history\n```\n\n**Pitfall 2: Memory Leaks in Long Sessions**\n\nProblem: Memory grows unbounded in long-running sessions.\n\nSolution: Implement memory size limits or periodic cleanup:\n```python\n# Add size limit\nif len(memory.content) > 1000:\n memory._content = memory.content[-500:] # Keep recent half\n```\n\n**Pitfall 3: Not Preserving Important State**\n\nProblem: Losing important state during crashes.\n\nSolution: Periodically serialize critical Blackboard state:\n```python\n# After critical operations\nstate = blackboard.blackboard_to_dict()\nwith open(\"checkpoint.json\", \"w\") as f:\n json.dump(state, f)\n```\n\n## Related Documentation\n\n- [Device Agent Overview](../overview.md) - Memory system in overall architecture\n- [Strategy Layer](processor.md) - `MEMORY_UPDATE` strategy implementation\n- [State Layer](state.md) - States reading/writing Blackboard for coordination\n- [Module System - Round](../../modules/round.md) - Round-level memory management\n- [Module System - Context](../../modules/context.md) - Context data vs Memory data separation\n\n## API Reference\n\nFor complete API documentation, see:\n\n::: agents.memory.memory.Memory\n::: agents.memory.memory.MemoryItem\n::: agents.memory.blackboard.Blackboard\n\n## Summary\n\n**Key Takeaways:**\n- **Dual-Memory Architecture**: Memory (short-term, agent-specific) + Blackboard (long-term, shared)\n- **Memory for Execution History**: Stores chronological MemoryItem instances for LLM context and debugging\n- **Blackboard for Coordination**: Implements Blackboard Pattern for multi-agent communication\n- **Flexible Schema**: MemoryItem supports custom fields for platform-specific requirements\n- **Persistence Support**: Blackboard can serialize/deserialize via dictionaries for session recovery\n- **Integration**: MEMORY_UPDATE strategy writes to Memory, states coordinate via Blackboard\n- **Best Practices**: Limit memory size, organize Blackboard with descriptive keys, periodically serialize state\n- **Scope Awareness**: Use Memory for agent-specific data, Blackboard for shared coordination\n\nThe Memory System provides the foundation for both individual agent intelligence (through execution history) and collective multi-agent coordination (through shared knowledge space), enabling UFO3 to orchestrate complex cross-device tasks effectively."
},
{
"path": "documents/docs/infrastructure/agents/design/processor.md",
"content": "# Strategy Layer: Processor (Level-2)\n\nThe **Processor** is the core component of the **Strategy Layer (Level-2)**, providing a configurable framework that orchestrates **ProcessingStrategies** through defined phases. Each agent state encapsulates a **ProcessorTemplate** that manages strategy registration, middleware chains, dependency validation, and context management. Together with modular strategies, the processor enables agents to compose complex execution workflows from reusable components.\n\n## Overview\n\nThe Processor implements the orchestration framework within **Level-2: Strategy Layer** of the [three-layer device agent architecture](../overview.md#three-layer-architecture). The Strategy Layer handles:\n\n- **Processor Framework** (This Document): Orchestrates strategy execution workflow\n- **Processing Strategies** (See [strategy.md](strategy.md)): Modular execution units\n- **Middleware System**: Cross-cutting concerns (logging, metrics, error handling)\n- **Dependency Validation**: Ensure strategies execute in correct order\n- **Context Management**: Unified data access across strategies\n\n```mermaid\ngraph TB\n subgraph \"Level-2: Strategy Layer\"\n State[AgentState Level-1 FSM] -->|encapsulates| Processor[ProcessorTemplate Strategy Orchestrator]\n \n Processor -->|registers| Registry[Strategy Registry Phase → Strategy mapping]\n Processor -->|configures| Middleware[Middleware Chain Logging, Metrics, etc.]\n \n Processor -->|Phase 1| DC[DATA_COLLECTION Strategy/Strategies]\n Processor -->|Phase 2| LLM[LLM_INTERACTION Strategy/Strategies]\n Processor -->|Phase 3| AE[ACTION_EXECUTION Strategy/Strategies]\n Processor -->|Phase 4| MU[MEMORY_UPDATE Strategy/Strategies]\n \n DC -->|provides data| Context[ProcessingContext]\n Context -->|consumed by| LLM\n LLM -->|provides actions| Context\n Context -->|consumed by| AE\n AE -->|provides results| Context\n Context -->|consumed by| MU\n end\n \n Strategies[ProcessingStrategy Implementations] -.registered by.-> Processor\n Middleware -.wraps.-> DC\n Middleware -.wraps.-> LLM\n Middleware -.wraps.-> AE\n Middleware -.wraps.-> MU\n```\n\n**Design Philosophy:** The Processor framework follows the **Template Method Pattern** where `ProcessorTemplate.process()` defines the workflow skeleton, subclasses configure phase-specific strategies, and middleware applies cross-cutting concerns uniformly. Strategies and middleware are injected at initialization, enabling extensibility without modifying the core framework.\n\n---\n\n## ProcessorTemplate Framework\n\nThe `ProcessorTemplate` is an **abstract base class** that defines the execution workflow. Platform-specific processors (AppAgentProcessor, HostAgentProcessor, LinuxAgentProcessor) subclass it to configure platform-specific strategies and middleware.\n\n### ProcessorTemplate Structure\n\n```python\nfrom abc import ABC, abstractmethod\nfrom typing import Dict, List, Optional, Type\nfrom enum import Enum\n\nclass ProcessingPhase(Enum):\n \"\"\"Enumeration of processor execution phases\"\"\"\n SETUP = \"setup\" # Initialization (optional)\n DATA_COLLECTION = \"data_collection\" # Gather context from device\n LLM_INTERACTION = \"llm_interaction\" # LLM reasoning and decision\n ACTION_EXECUTION = \"action_execution\" # Execute commands on device\n MEMORY_UPDATE = \"memory_update\" # Update memory and blackboard\n CLEANUP = \"cleanup\" # Cleanup (optional)\n\n\nclass ProcessorTemplate(ABC):\n \"\"\"\n Abstract processor template defining workflow orchestration framework.\n \n Responsibilities:\n 1. Strategy Registration: Configure strategies for each phase\n 2. Middleware Management: Setup cross-cutting concern handlers\n 3. Dependency Validation: Ensure strategy data flow is valid\n 4. Workflow Execution: Orchestrate strategy execution in phase order\n 5. Context Management: Create and manage ProcessingContext\n \n Subclasses must implement:\n - _setup_strategies(): Register strategies for processing phases\n - _setup_middleware(): Register middleware (optional)\n \"\"\"\n \n # Subclasses can override to use custom context class\n processor_context_class: Type[BasicProcessorContext] = BasicProcessorContext\n \n def __init__(self, agent: BasicAgent, global_context: Context):\n \"\"\"\n Initialize processor.\n \n :param agent: The agent instance\n :param global_context: Shared global context (session-wide)\n \"\"\"\n self.agent = agent\n self.global_context = global_context\n \n # Strategy registry: phase -> strategy mapping\n self.strategies: Dict[ProcessingPhase, ProcessingStrategy] = {}\n \n # Middleware chain (executed in order)\n self.middleware_chain: List[ProcessorMiddleware] = []\n \n # Logging\n self.logger = logging.getLogger(self.__class__.__name__)\n \n # Dependency validator\n self.dependency_validator = StrategyDependencyValidator()\n \n # Lifecycle\n self._setup_strategies() # Subclass configures strategies\n self._setup_middleware() # Subclass configures middleware\n self._validate_strategy_chain() # Validate dependencies\n \n # Create processing context (local data store)\n self.processing_context = self._create_processing_context()\n \n @abstractmethod\n def _setup_strategies(self) -> None:\n \"\"\"\n Setup strategies for each processing phase.\n \n Subclasses must implement this method to configure their strategy workflow.\n \n Example:\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([\n ScreenshotStrategy(),\n UITreeStrategy()\n ])\n self.strategies[ProcessingPhase.LLM_INTERACTION] = LLMStrategy()\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = ActionStrategy()\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = MemoryStrategy()\n \"\"\"\n pass\n \n def _setup_middleware(self) -> None:\n \"\"\"\n Setup middleware for cross-cutting concerns.\n \n Subclasses can override to add middleware (logging, metrics, etc.).\n Default: no middleware.\n \n Example:\n self.middleware_chain = [\n LoggingMiddleware(),\n MetricsMiddleware(),\n ErrorHandlingMiddleware()\n ]\n \"\"\"\n pass\n \n def _validate_strategy_chain(self) -> None:\n \"\"\"\n Validate that strategy dependencies are satisfied.\n \n Raises ProcessingException if validation fails.\n \"\"\"\n errors = self.dependency_validator.validate_chain(self.strategies)\n if errors:\n error_msg = \"Strategy chain validation failed:\\n\" + \"\\n\".join(errors)\n self.logger.error(error_msg)\n raise ProcessingException(error_msg)\n \n def _create_processing_context(self) -> ProcessingContext:\n \"\"\"\n Create processing context with local and global data separation.\n \n :return: ProcessingContext instance\n \"\"\"\n local_context = self.processor_context_class()\n return ProcessingContext(\n global_context=self.global_context,\n local_context=local_context\n )\n \n async def process(self) -> None:\n \"\"\"\n Main execution method - orchestrates workflow execution.\n \n Workflow:\n 1. Execute strategies in phase order (DATA_COLLECTION → LLM → ACTION → MEMORY)\n 2. Apply middleware before/after each strategy\n 3. Validate dependencies before each strategy execution\n 4. Update context with strategy outputs\n 5. Handle errors according to strategy fail_fast setting\n \n :raises ProcessingException: If critical error occurs\n \"\"\"\n try:\n self.logger.info(f\"Starting processor execution: {self.__class__.__name__}\")\n \n # Define execution order\n execution_order = [\n ProcessingPhase.SETUP,\n ProcessingPhase.DATA_COLLECTION,\n ProcessingPhase.LLM_INTERACTION,\n ProcessingPhase.ACTION_EXECUTION,\n ProcessingPhase.MEMORY_UPDATE,\n ProcessingPhase.CLEANUP\n ]\n \n # Execute each phase\n for phase in execution_order:\n strategy = self.strategies.get(phase)\n if not strategy:\n self.logger.debug(f\"No strategy registered for phase {phase.value}, skipping\")\n continue\n \n self.logger.info(f\"Executing phase: {phase.value} with strategy: {strategy.name}\")\n \n # Validate dependencies\n missing_deps = strategy.validate_dependencies(self.processing_context)\n if missing_deps:\n raise ProcessingException(\n f\"Strategy {strategy.name} missing required dependencies: {missing_deps}\"\n )\n \n # Apply middleware (before)\n await self._apply_middleware_before(phase, strategy)\n \n # Execute strategy\n result = await strategy.execute(self.agent, self.processing_context)\n \n # Handle result\n if result.success:\n self.logger.info(f\"Strategy {strategy.name} succeeded\")\n # Update context with strategy outputs\n self.processing_context.update_local(result.data)\n else:\n self.logger.error(f\"Strategy {strategy.name} failed: {result.error}\")\n if strategy.fail_fast:\n raise ProcessingException(\n f\"Strategy {strategy.name} failed in phase {phase.value}: {result.error}\"\n )\n else:\n self.logger.warning(f\"Continuing despite failure in {strategy.name}\")\n \n # Apply middleware (after)\n await self._apply_middleware_after(phase, strategy, result)\n \n # Finalize context (promote local data to global if needed)\n self._finalize_processing_context()\n \n self.logger.info(\"Processor execution completed successfully\")\n \n except Exception as e:\n self.logger.error(f\"Processor execution failed: {e}\", exc_info=True)\n raise\n \n async def _apply_middleware_before(\n self,\n phase: ProcessingPhase,\n strategy: ProcessingStrategy\n ) -> None:\n \"\"\"\n Apply middleware before strategy execution.\n \n :param phase: Current processing phase\n :param strategy: Strategy about to execute\n \"\"\"\n for middleware in self.middleware_chain:\n await middleware.before_execute(phase, strategy, self.processing_context)\n \n async def _apply_middleware_after(\n self,\n phase: ProcessingPhase,\n strategy: ProcessingStrategy,\n result: ProcessingResult\n ) -> None:\n \"\"\"\n Apply middleware after strategy execution.\n \n :param phase: Current processing phase\n :param strategy: Strategy that just executed\n :param result: Strategy execution result\n \"\"\"\n for middleware in self.middleware_chain:\n await middleware.after_execute(phase, strategy, result, self.processing_context)\n \n def _finalize_processing_context(self) -> None:\n \"\"\"\n Finalize processing context after workflow completion.\n \n Subclasses can override to customize context finalization.\n Default: Promote selected local data to global context.\n \"\"\"\n # Example: Promote final action status to global context\n if self.processing_context.get_local(\"action_success\") is not None:\n self.global_context.set(\n \"last_action_success\",\n self.processing_context.get_local(\"action_success\")\n raise\n```\n\n### ProcessorTemplate Benefits\n\n**Consistent Workflow:** All processors follow the same execution pattern, ensuring predictable behavior across platforms.\n\n**Platform Customization:** Subclasses configure platform-specific strategies without modifying the core framework.\n\n**Reusable Framework:** Core orchestration logic is shared across all processors, reducing code duplication.\n\n**Middleware Support:** Cross-cutting concerns (logging, metrics, error handling) are applied uniformly to all strategy executions.\n\n**Testable:** Each phase can be tested independently with mock strategies and contexts.\n\n---\n\n## Strategy Registration\n\nProcessors configure their workflow by **registering strategies** for each processing phase:\n\n```mermaid\ngraph TB\n subgraph \"Strategy Registration\"\n Processor[ProcessorTemplate Subclass]\n \n Processor -->|_setup_strategies()| Registry[Strategy Registry]\n \n Registry -->|ProcessingPhase.DATA_COLLECTION| DC[ScreenshotStrategy + UITreeStrategy]\n Registry -->|ProcessingPhase.LLM_INTERACTION| LLM[LLMStrategy]\n Registry -->|ProcessingPhase.ACTION_EXECUTION| AE[ActionStrategy]\n Registry -->|ProcessingPhase.MEMORY_UPDATE| MU[MemoryStrategy]\n end\n```\n\n### Example: Windows AppAgent Processor\n\n```python\nfrom ufo.agents.processors.core.processor_framework import ProcessorTemplate, ProcessingPhase\nfrom ufo.agents.processors.strategies.processing_strategy import ComposedStrategy\n\nclass AppAgentProcessor(ProcessorTemplate):\n \"\"\"Processor for Windows AppAgent (UI Automation)\"\"\"\n \n processor_context_class = AppAgentProcessorContext # Custom context type\n \n def _setup_strategies(self):\n \"\"\"Configure strategies for Windows UI automation workflow\"\"\"\n \n # Phase 1: DATA_COLLECTION - Compose multiple strategies\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(\n strategies=[\n AppScreenshotCaptureStrategy(), # Capture application screenshot\n AppControlInfoStrategy() # Extract UI Automation tree\n ],\n name=\"AppDataCollection\",\n phase=ProcessingPhase.DATA_COLLECTION\n )\n \n # Phase 2: LLM_INTERACTION - Single strategy\n self.strategies[ProcessingPhase.LLM_INTERACTION] = AppLLMInteractionStrategy()\n \n # Phase 3: ACTION_EXECUTION - Execute UI commands\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = AppActionExecutionStrategy()\n \n # Phase 4: MEMORY_UPDATE - Update memory and blackboard\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy()\n \n def _setup_middleware(self):\n \"\"\"Configure middleware for logging and metrics\"\"\"\n self.middleware_chain = [\n EnhancedLoggingMiddleware()\n ]\n```\n\n### Example: Linux Agent Processor\n\n```python\nclass LinuxAgentProcessor(ProcessorTemplate):\n \"\"\"Processor for Linux Agent (Shell Commands)\"\"\"\n \n processor_context_class = LinuxAgentProcessorContext\n \n def _setup_strategies(self):\n \"\"\"Configure strategies for Linux shell workflow\"\"\"\n \n # Phase 1: DATA_COLLECTION - Screenshot + shell output\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([\n CustomizedScreenshotCaptureStrategy(),\n ShellOutputStrategy()\n ])\n \n # Phase 2: LLM_INTERACTION - Generate shell commands\n self.strategies[ProcessingPhase.LLM_INTERACTION] = CustomizedLLMInteractionStrategy()\n \n # Phase 3: ACTION_EXECUTION - Execute shell commands\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = LinuxActionExecutionStrategy()\n \n # Phase 4: MEMORY_UPDATE - Record command history\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = LinuxMemoryUpdateStrategy()\n```\n\n**Registration Best Practices:**\n\n- Use ComposedStrategy for phases requiring multiple data sources (e.g., DATA_COLLECTION)\n- Use single strategy for phases with focused responsibility (e.g., LLM_INTERACTION)\n- Don't register SETUP/CLEANUP phases unless needed for initialization/cleanup\n- Override `processor_context_class` for platform-specific data structures\n\n---\n\n## Middleware System\n\nMiddleware provides cross-cutting concerns that apply uniformly across all strategy executions. The middleware chain executes before/after processing and handles errors.\n\n```mermaid\ngraph LR\n subgraph \"Middleware Chain\"\n MW1[EnhancedLogging Middleware]\n end\n \n Processor[ProcessorTemplate]\n \n MW1 -.before_process.-> Processor\n Processor -.after_process.-> MW1\n Processor -.on_error.-> MW1\n```\n\n### ProcessorMiddleware Interface\n\n```python\nfrom abc import ABC, abstractmethod\nfrom typing import Optional\n\nclass ProcessorMiddleware(ABC):\n \"\"\"\n Abstract base for processor middleware.\n \n Middleware wraps strategy execution to provide cross-cutting concerns\n such as logging, metrics collection, error handling, caching, etc.\n \"\"\"\n \n @abstractmethod\n async def before_process(\n self,\n processor: ProcessorTemplate,\n context: ProcessingContext\n ) -> None:\n \"\"\"\n Called before processing starts.\n \n :param processor: The processor instance\n :param context: Processing context\n \"\"\"\n pass\n \n @abstractmethod\n async def after_process(\n self,\n processor: ProcessorTemplate,\n result: ProcessingResult\n ) -> None:\n \"\"\"\n Called after processing completes.\n \n :param processor: The processor instance\n :param result: Processing execution result\n \"\"\"\n pass\n \n @abstractmethod\n async def on_error(\n self,\n processor: ProcessorTemplate,\n error: Exception\n ) -> None:\n \"\"\"\n Called when an error occurs during processing.\n \n :param processor: The processor instance\n :param error: The error that occurred\n \"\"\"\n pass\n```\n\n### Built-in Middleware: EnhancedLoggingMiddleware\n\nThe framework provides `EnhancedLoggingMiddleware` for comprehensive logging during processor execution:\n\n```python\nclass EnhancedLoggingMiddleware(ProcessorMiddleware):\n \"\"\"Enhanced logging middleware that handles different types of errors appropriately\"\"\"\n \n def __init__(self, log_level: int = logging.INFO, name: Optional[str] = None):\n super().__init__(name)\n self.logger = logging.getLogger(f\"{self.__class__.__name__}.{self.name}\")\n self.log_level = log_level\n \n async def before_process(self, processor, context):\n \"\"\"Log processing start with context information\"\"\"\n round_num = context.get(\"round_num\", 0)\n round_step = context.get(\"round_step\", 0)\n \n self.logger.log(\n self.log_level,\n f\"Starting processing: Round {round_num + 1}, Step {round_step + 1}, \"\n f\"Processor: {processor.__class__.__name__}\"\n )\n \n async def after_process(self, processor, result):\n \"\"\"Log processing completion with result summary and save to file\"\"\"\n if result.success:\n self.logger.log(\n self.log_level,\n f\"Processing completed successfully in {result.execution_time:.2f}s\"\n )\n \n # Log phase execution times if available\n data_keys = list(result.data.keys())\n if data_keys:\n self.logger.debug(f\"Result data keys: {data_keys}\")\n else:\n self.logger.warning(f\"Processing completed with failure: {result.error}\")\n \n # Save local context to log file\n local_logger = processor.processing_context.global_context.get(ContextNames.LOGGER)\n local_context = processor.processing_context.local_context\n \n local_context.total_time = result.execution_time\n \n # Record phase time costs\n phrase_time_cost = {}\n for phrase, phrase_result in processor.processing_context.phase_results.items():\n phrase_time_cost[phrase.name] = phrase_result.execution_time\n \n local_context.execution_times = phrase_time_cost\n \n # Write to log file\n safe_obj = to_jsonable_python(local_context.to_dict(selective=True))\n local_context_string = json.dumps(safe_obj, ensure_ascii=False)\n local_logger.write(local_context_string)\n \n self.logger.info(\"Log saved successfully.\")\n \n async def on_error(self, processor, error):\n \"\"\"Enhanced error logging with context information\"\"\"\n if isinstance(error, ProcessingException):\n self.logger.error(\n f\"ProcessingException in {processor.__class__.__name__}:\\n\"\n f\" Phase: {error.phase}\\n\"\n f\" Message: {str(error)}\\n\"\n f\" Context: {error.context_data}\\n\"\n f\" Original Exception: {error.original_exception}\"\n )\n \n if error.original_exception:\n self.logger.info(\n f\"Original traceback:\\n{traceback.format_exception(error.original_exception)}\"\n )\n else:\n self.logger.error(\n f\"Unexpected error in {processor.__class__.__name__}: {str(error)}\\n\"\n f\"Error type: {type(error).__name__}\\n\"\n f\"Traceback:\\n{traceback.format_exception(error)}\"\n )\n```\n\n**Key Features:**\n\n- **Context-Aware Logging**: Logs round/step information for traceability\n- **Result Summary**: Logs execution time and phase breakdown\n- **Persistent Logging**: Saves structured context data to log files\n- **Enhanced Error Handling**: Distinguishes ProcessingException from general errors\n- **Traceback Capture**: Full stack traces for debugging\n\n### Configuring Middleware\n\nProcessors configure middleware in `_setup_middleware()`:\n\n```python\nclass AppAgentProcessor(ProcessorTemplate):\n def _setup_middleware(self):\n \"\"\"Setup middleware chain\"\"\"\n self.middleware_chain = [\n EnhancedLoggingMiddleware(log_level=logging.INFO, name=\"AppAgent\")\n ]\n```\n\n**Middleware Execution Order:**\n\n1. **Before Processing**: `before_process()` called for each middleware in order\n2. **Strategy Execution**: Strategies execute through phases\n3. **After Processing**: `after_process()` called for each middleware in reverse order\n4. **On Error**: `on_error()` called for all middleware if exception occurs\n\n**Middleware Benefits:**\n\n- **Separation of Concerns**: Cross-cutting logic separated from strategy logic\n- **Reusability**: Same middleware can be used across different processors\n- **Non-invasive**: Add/remove middleware without modifying strategies\n\n---\n\n## Workflow Execution\n\nThe processor executes the workflow by orchestrating strategies through defined phases:\n\n```mermaid\nsequenceDiagram\n participant State as AgentState\n participant Processor as ProcessorTemplate\n participant MW as Middleware Chain\n participant Strategy as ProcessingStrategy\n participant Context as ProcessingContext\n \n State->>Processor: process()\n \n Processor->>MW: before_process(processor, context)\n MW-->>Processor: Ready\n \n loop For each Phase\n Processor->>Processor: Get strategy for phase\n Processor->>Strategy: validate_dependencies(context)\n Strategy-->>Processor: [] (no missing deps)\n \n Processor->>Strategy: execute(agent, context)\n Strategy->>Context: get_local(\"screenshot\")\n Context-->>Strategy: screenshot data\n Strategy->>Strategy: Process data\n Strategy->>Context: update_local({\"parsed_response\": ...})\n Strategy-->>Processor: ProcessingResult(success=True, data={...})\n \n Processor->>Context: Update with strategy outputs\n end\n \n Processor->>MW: after_process(processor, result)\n MW-->>Processor: (middleware processing)\n \n Processor->>Processor: _finalize_processing_context()\n Processor-->>State: ProcessingResult\n```\n\n### Execution Order\n\n```python\n# Defined in ProcessorTemplate.process()\nexecution_order = [\n ProcessingPhase.SETUP, # Optional: Initialize resources\n ProcessingPhase.DATA_COLLECTION, # Gather device/environment context\n ProcessingPhase.LLM_INTERACTION, # LLM reasoning and decision-making\n ProcessingPhase.ACTION_EXECUTION, # Execute actions on device\n ProcessingPhase.MEMORY_UPDATE, # Update memory and blackboard\n ProcessingPhase.CLEANUP # Optional: Cleanup resources\n]\n```\n\n**Phase Execution Rules:**\n\n- **Optional Phases**: SETUP and CLEANUP are optional (skipped if no strategy registered)\n- **Sequential Execution**: Phases execute in fixed order (no parallelization)\n- **Dependency Validation**: Validated before each strategy execution using `StrategyDependencyValidator`\n- **Fail-Fast vs. Continue**: Strategy `fail_fast` setting determines error handling\n- **Context Updates**: Each strategy's outputs immediately available to next strategy via `ProcessingContext`\n\n---\n\n## ProcessingContext\n\nThe `ProcessingContext` provides unified data access across strategies, separating local (processor-specific) and global (session-wide) data:\n\n```python\n@dataclass\nclass ProcessingContext:\n \"\"\"\n Processing context with local and global data separation.\n \n :param global_context: Global context (shared across all components)\n :param local_context: Local context (processor-specific data)\n \"\"\"\n global_context: Context # Module system global context\n local_context: BasicProcessorContext # Processor local data\n \n def get_local(self, key: str, default=None) -> Any:\n \"\"\"\n Get value from local context.\n \n :param key: Field name\n :param default: Default value if not found\n :return: Field value or default\n \"\"\"\n return getattr(self.local_context, key, default)\n \n def get_global(self, key: str, default=None) -> Any:\n \"\"\"\n Get value from global context.\n \n :param key: Context key\n :param default: Default value if not found\n :return: Context value or default\n \"\"\"\n return self.global_context.get(key, default)\n \n def update_local(self, data: Dict[str, Any]) -> None:\n \"\"\"\n Update local context with strategy outputs.\n \n :param data: Dictionary of field name -> value pairs\n \"\"\"\n self.local_context.update_from_dict(data)\n \n def require_local(self, field_name: str, expected_type: Type = None) -> Any:\n \"\"\"\n Get required field from local context.\n \n :param field_name: Required field name\n :param expected_type: Expected Python type (optional)\n :return: Field value\n :raises ProcessingException: If field missing or wrong type\n \"\"\"\n value = self.get_local(field_name)\n if value is None:\n raise ProcessingException(f\"Required field '{field_name}' not found in local context\")\n if expected_type and not isinstance(value, expected_type):\n raise ProcessingException(\n f\"Field '{field_name}' has type {type(value).__name__}, \"\n f\"expected {expected_type.__name__}\"\n )\n return value\n```\n\n**Context Separation Rationale:**\n\n**Global Context** (session-wide, shared across all components):\n\n- User request (`REQUEST`)\n- Session ID, round number, step number\n- Configuration settings\n- Command dispatcher reference\n- Blackboard reference\n\n**Local Context** (processor-specific, temporary):\n\n- Screenshot data (`screenshot`, `screenshot_path`)\n- UI control information (`control_info`)\n- LLM parsed response (`parsed_response`)\n- Action execution results (`results`)\n- Temporary processing data\n\n---\n\n## Platform-Specific Processors\n\nDifferent agent types implement platform-specific processors:\n\n| Platform | Processor Class | DATA_COLLECTION | LLM_INTERACTION | ACTION_EXECUTION | MEMORY_UPDATE |\n|----------|----------------|-----------------|-----------------|------------------|---------------|\n| **Windows AppAgent** | `AppAgentProcessor` | Screenshot + UI tree | UI element selection | UI Automation commands | UI interaction history |\n| **Windows HostAgent** | `HostAgentProcessor` | Desktop screenshot + app list | Application selection | Launch app, create AppAgent | App selection history |\n| **Linux** | `LinuxAgentProcessor` | Screenshot + shell output | Shell command generation | Shell command execution | Command history |\n\nSee the [Agent Types documentation](../agent_types.md) for platform-specific processor implementations.\n\n---\n\n## Best Practices\n\n### Processor Design Guidelines\n\n**1. Clear Phase Separation**: Each phase should have distinct responsibility\n\n- DATA_COLLECTION gathers raw data\n- LLM_INTERACTION performs reasoning\n- ACTION_EXECUTION executes commands\n- MEMORY_UPDATE persists state\n\n**2. Appropriate Strategy Composition**: Use `ComposedStrategy` for multi-source data collection\n\n```python\nself.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([\n AppScreenshotCaptureStrategy(),\n AppControlInfoStrategy()\n])\n```\n\n**3. Middleware for Cross-Cutting Concerns**: Don't implement logging/metrics in strategies\n\n**4. Dependency Validation**: Leverage automatic validation via `StrategyDependencyValidator`\n\n**5. Custom Context Classes**: Define platform-specific context classes when needed\n\n```python\n@dataclass\nclass AppAgentProcessorContext(BasicProcessorContext):\n \"\"\"Extended context for Windows AppAgent\"\"\"\n agent_type: str = \"AppAgent\"\n screenshot: str = \"\"\n screenshot_path: str = \"\"\n control_info: str = \"\"\n control_elements: List[Dict] = field(default_factory=list)\n parsed_response: Dict = field(default_factory=dict)\n action: List[Dict[str, Any]] = field(default_factory=list)\n arguments: Dict = field(default_factory=dict)\n results: str = \"\"\n```\n\n!!! warning \"Common Pitfalls\"\n **Skipping Phases**: Don't skip required phases (DATA_COLLECTION → LLM → ACTION → MEMORY)\n \n **Phase Order Changes**: Don't reorder phases (breaks dependency chain)\n \n **Strategy State**: Don't store state in strategy instances (use context instead)\n \n **Direct Agent Modification**: Don't modify agent attributes in processor (use proper channels like memory system)\n\n---\n\n## Integration with Other Layers\n\n```mermaid\ngraph TB\n subgraph \"State Layer (Level-1)\"\n State[AgentState]\n end\n \n subgraph \"Strategy Layer (Level-2)\"\n Processor[ProcessorTemplate]\n Strategies[ProcessingStrategies]\n Middleware[Middleware Chain]\n end\n \n subgraph \"Command Layer (Level-3)\"\n Dispatcher[CommandDispatcher]\n end\n \n subgraph \"Supporting Systems\"\n Memory[Memory System]\n Context[Global Context]\n end\n \n State -->|calls process()| Processor\n Processor -->|orchestrates| Strategies\n Processor -->|applies| Middleware\n Strategies -->|uses| Dispatcher\n Strategies -->|updates| Memory\n Strategies -->|reads/writes| Context\n```\n\n| Integration Point | Layer/Component | Relationship |\n|-------------------|-----------------|--------------|\n| **AgentState** | Level-1 State | State calls `processor.process()` to execute workflow |\n| **ProcessingStrategy** | Level-2 Strategy | Processor registers and executes strategies |\n| **CommandDispatcher** | Level-3 Command | ACTION_EXECUTION strategies use dispatcher |\n| **Memory/Blackboard** | Memory System | MEMORY_UPDATE strategies update agent memory |\n| **Global Context** | Module System | Processor reads request, writes results via context |\n\nSee [State Layer](state.md), [Strategy Layer](strategy.md), and [Command Layer](command.md) for integration details.\n\n---\n\n## API Reference\n\nThe following classes are documented via docstrings:\n\n- `ProcessorTemplate`: Abstract processor framework base class\n- `ProcessingPhase`: Enum defining processor execution phases\n- `ProcessingContext`: Unified context with local/global data separation\n- `ProcessorMiddleware`: Abstract middleware base class\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n- **ProcessorTemplate**: Abstract framework for workflow orchestration\n- **Strategy Registration**: Configure phase-specific strategies via `_setup_strategies()`\n- **Middleware System**: Cross-cutting concerns (logging, error handling) applied uniformly\n- **Workflow Execution**: Orchestrates DATA_COLLECTION → LLM → ACTION → MEMORY phases\n- **Dependency Validation**: Ensures strategies execute with required data available via `StrategyDependencyValidator`\n- **Context Management**: Separates local (processor) and global (session) data\n- **Platform Extensibility**: Subclass to create platform-specific processors\n- **Template Method Pattern**: Defines workflow skeleton, subclasses customize details\n\nThe Processor provides the orchestration framework within the Strategy Layer that coordinates strategy execution, middleware application, and context management, enabling agents to execute complex workflows reliably and efficiently across diverse platforms.\n"
},
{
"path": "documents/docs/infrastructure/agents/design/prompter.md",
"content": "# Agent Prompter\n\nThe `Prompter` is a key component of the UFO framework, responsible for constructing prompts for the LLM to generate responses. Each agent has its own `Prompter` class that defines the structure of the prompt and the information to be fed to the LLM.\n\n## Overview\n\nThe Prompter system follows a hierarchical design pattern:\n\n```\nBasicPrompter (Abstract Base Class)\n├── HostAgentPrompter\n├── AppAgentPrompter\n├── EvaluationAgentPrompter\n├── ExperiencePrompter\n├── DemonstrationPrompter\n└── customized/\n └── LinuxAgentPrompter (extends AppAgentPrompter)\n```\n\nEach prompter is responsible for:\n\n1. **Loading templates** from YAML configuration files\n2. **Constructing system prompts** with instructions, APIs, and examples\n3. **Building user prompts** from agent observations and context\n4. **Formatting multimodal content** (text + images for visual models)\n\nYou can find all prompter implementations in the `ufo/prompter` folder.\n\n## Prompt Message Structure\n\nA prompt fed to the LLM is a list of dictionaries, where each dictionary represents a message with the following structure:\n\n| Key | Description | Example Values |\n| --- | --- | --- |\n| `role` | The role of the message | `system`, `user`, `assistant` |\n| `content` | The message content | String or list of content objects |\n\nFor **visual models**, the `content` field can contain multiple elements:\n\n```python\n[\n {\"type\": \"text\", \"text\": \"Current Screenshots:\"},\n {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/png;base64,...\"}}\n]\n```\n\n## Prompt Construction Workflow\n\nThe final prompt is constructed through a multi-step process:\n\n```mermaid\ngraph LR\n A[Load Templates] --> B[Build System Prompt]\n B --> C[Build User Prompt]\n C --> D[Combine into Message List]\n D --> E[Send to LLM]\n \n B --> B1[Base Instructions]\n B --> B2[API Documentation]\n B --> B3[Examples]\n \n C --> C1[Observation]\n C --> C2[Retrieved Knowledge]\n C --> C3[Blackboard State]\n```\n\n### Step 1: Template Loading\n\nTemplates are loaded from YAML files during initialization:\n\n```python\ndef __init__(self, is_visual: bool, prompt_template: str, example_prompt_template: str):\n self.is_visual = is_visual\n self.prompt_template = self.load_prompt_template(prompt_template, is_visual)\n self.example_prompt_template = self.load_prompt_template(example_prompt_template, is_visual)\n```\n\nThe `is_visual` parameter determines which template variant to load:\n- **Visual models**: Use templates with screenshot handling\n- **Non-visual models**: Use text-only templates\n\n### Step 2: System Prompt Construction\n\nThe system prompt is built using the `system_prompt_construction()` method, which combines:\n\n1. **Base instructions** from the template\n2. **API documentation** via `api_prompt_helper()`\n3. **Demonstration examples** via `examples_prompt_helper()`\n4. **Third-party agent instructions** (for HostAgent)\n\nExample for HostAgent:\n\n```python\ndef system_prompt_construction(self) -> str:\n apis = self.api_prompt_helper(verbose=0)\n examples = self.examples_prompt_helper()\n third_party_instructions = self.third_party_agent_instruction()\n \n system_key = \"system\" if self.is_visual else \"system_nonvisual\"\n \n return self.prompt_template[system_key].format(\n apis=apis,\n examples=examples,\n third_party_instructions=third_party_instructions,\n )\n```\n\n### Step 3: User Prompt Construction\n\nThe user prompt is constructed using the `user_prompt_construction()` method with agent-specific parameters:\n\n**HostAgent Parameters:**\n```python\ndef user_prompt_construction(\n self,\n control_item: List[str], # Available applications/windows\n prev_subtask: List[Dict], # Previous subtask history\n prev_plan: List[str], # Previous plan steps\n user_request: str, # Original user request\n retrieved_docs: str = \"\", # Retrieved knowledge\n) -> str\n```\n\n**AppAgent Parameters:**\n```python\ndef user_prompt_construction(\n self,\n control_item: List[str], # Available UI controls\n prev_subtask: List[Dict], # Previous subtask history\n prev_plan: List[str], # Previous plan steps\n user_request: str, # Original user request\n subtask: str, # Current subtask\n current_application: str, # Current app name\n host_message: List[str], # Messages from HostAgent\n retrieved_docs: str = \"\", # Retrieved knowledge\n last_success_actions: List = [], # Last successful actions\n) -> str\n```\n\n### Step 4: User Content Construction\n\nFor multimodal models, the `user_content_construction()` method builds a list of content objects:\n\n```python\ndef user_content_construction(self, image_list: List[str], ...) -> List[Dict]:\n user_content = []\n \n if self.is_visual:\n # Add screenshots\n for i, image in enumerate(image_list):\n user_content.append({\"type\": \"text\", \"text\": f\"Screenshot {i+1}:\"})\n user_content.append({\"type\": \"image_url\", \"image_url\": {\"url\": image}})\n \n # Add text prompt\n user_content.append({\n \"type\": \"text\",\n \"text\": self.user_prompt_construction(...)\n })\n \n return user_content\n```\n\n### Step 5: Final Assembly\n\nThe `prompt_construction()` static method combines system and user prompts:\n\n```python\n@staticmethod\ndef prompt_construction(system_prompt: str, user_content: List[Dict]) -> List:\n return [\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": user_content}\n ]\n```\n\n## Prompt Components\n\n### System Prompt\n\nThe system prompt defines the agent's role, capabilities, and output format. It is loaded from YAML templates configured in the system configuration.\n\n**Template Locations:**\n- HostAgent: `ufo/prompts/share/base/host_agent.yaml`\n- AppAgent: `ufo/prompts/share/base/app_agent.yaml`\n- EvaluationAgent: `ufo/prompts/evaluation/evaluate.yaml`\n\nThe system prompt is constructed by the `system_prompt_construction()` method and typically includes:\n\n| Component | Description | Method |\n| --- | --- | --- |\n| **Base Instructions** | Role definition, action guidelines, output format | Loaded from YAML template |\n| **API Documentation** | Available tools and their usage | `api_prompt_helper()` |\n| **Examples** | Demonstration examples for in-context learning | `examples_prompt_helper()` |\n| **Special Instructions** | Third-party agent integration (HostAgent only) | `third_party_agent_instruction()` |\n\n#### API Documentation\n\nThe `api_prompt_helper()` method formats tool information for the LLM:\n\n```python\ndef api_prompt_helper(self, verbose: int = 1) -> str:\n \"\"\"Construct formatted API documentation.\"\"\"\n return self.api_prompt_template\n```\n\nTools are converted to LLM-readable format using `tool_to_llm_prompt()`:\n\n```\nTool name: click_input\nDescription: Click on a control item\n\nParameters:\n- id (string, required): The ID of the control item\n- button (string, optional): Mouse button to click. Default: left\n- double (boolean, optional): Whether to double-click. Default: false\n\nReturns: Result of the click action\n\nExample usage:\nclick_input(id=\"42\", button=\"left\", double=false)\n```\n\n#### Demonstration Examples\n\nThe `examples_prompt_helper()` method constructs in-context learning examples:\n\n```python\ndef examples_prompt_helper(\n self, \n header: str = \"## Response Examples\",\n separator: str = \"Example\",\n additional_examples: List[str] = []\n) -> str:\n \"\"\"Construct examples from YAML template.\"\"\"\n template = \"\"\"\n [User Request]:\n {request}\n [Response]:\n {response}\"\"\"\n \n example_list = []\n for key, values in self.example_prompt_template.items():\n if key.startswith(\"example\"):\n example = template.format(\n request=values.get(\"Request\"),\n response=json.dumps(values.get(\"Response\"))\n )\n example_list.append(example)\n \n return self.retrieved_documents_prompt_helper(header, separator, example_list)\n```\n\nExamples are loaded from:\n- `ufo/prompts/examples/visual/` - For visual models\n- `ufo/prompts/examples/nonvisual/` - For text-only models\n\n### User Prompt\n\nThe user prompt is constructed from the agent's current context and observations. It is built by the `user_prompt_construction()` method using information from:\n\n| Component | Description | Method |\n| --- | --- | --- |\n| **Observation** | Current state (UI controls, screenshots) | Passed as parameters |\n| **Retrieved Knowledge** | Documents from RAG system | `retrieved_documents_prompt_helper()` |\n| **Blackboard State** | Shared memory across agents | `blackboard_to_prompt()` |\n| **Task Context** | User request, subtask, plans | Passed as parameters |\n\n#### Retrieved Documents\n\nExternal knowledge is formatted using the `retrieved_documents_prompt_helper()` method:\n\n```python\n@staticmethod\ndef retrieved_documents_prompt_helper(\n header: str, # Section header\n separator: str, # Document separator\n documents: List[str] # Retrieved documents\n) -> str:\n \"\"\"Format retrieved documents for the prompt.\"\"\"\n if header:\n prompt = f\"\\n<{header}:>\\n\"\n else:\n prompt = \"\"\n \n for i, document in enumerate(documents):\n if separator:\n prompt += f\"[{separator} {i+1}:]\\n\"\n prompt += document + \"\\n\\n\"\n \n return prompt\n```\n\n**Example Output:**\n```\n\n[Document 1:]\nTo create a new email in Outlook, click the \"New Email\" button...\n\n[Document 2:]\nThe email composition window has three main fields: To, Subject, and Body...\n```\n\n#### Blackboard Integration\n\nThe Blackboard system allows agents to share information. Prompters can access this through:\n\n```python\ndef blackboard_to_prompt(self) -> str:\n \"\"\"Convert Blackboard state to prompt text.\"\"\"\n # Implementation depends on specific agent needs\n pass\n```\n\n## Specialized Prompters\n\n### HostAgentPrompter\n\nSpecialized for desktop-level orchestration:\n\n**Key Features:**\n- Application selection and window management\n- Third-party agent integration support\n- Desktop-wide task planning\n\n**Unique Method:**\n```python\ndef third_party_agent_instruction(self) -> str:\n \"\"\"Generate instructions for enabled third-party agents.\"\"\"\n enabled_agents = config.system.enabled_third_party_agents\n instructions = []\n \n for agent_name in enabled_agents:\n config = get_third_party_config(agent_name)\n instructions.append(f\"{agent_name}: {config['INTRODUCTION']}\")\n \n return \"\\n\".join(instructions)\n```\n\n### AppAgentPrompter\n\nSpecialized for application-level interactions:\n\n**Key Features:**\n- UI control interaction\n- Multi-action sequence support\n- Application-specific API integration\n\n**Template Variants:**\n- `system`: Standard single-action mode\n- `system_as`: Action sequence mode (multi-action)\n- `system_nonvisual`: Text-only mode\n\n**Usage:**\n```python\ndef system_prompt_construction(self, additional_examples: List[str] = []) -> str:\n apis = self.api_prompt_helper(verbose=1)\n examples = self.examples_prompt_helper(additional_examples=additional_examples)\n \n # Select template based on configuration\n if config.system.action_sequence:\n system_key = \"system_as\"\n else:\n system_key = \"system\"\n \n if not self.is_visual:\n system_key += \"_nonvisual\"\n \n return self.prompt_template[system_key].format(apis=apis, examples=examples)\n```\n\n### EvaluationAgentPrompter\n\nSpecialized for task evaluation:\n\n**Purpose:** Assesses whether a Session or Round was successfully completed\n\n**Configuration:** Uses `ufo/prompts/evaluation/evaluate.yaml`\n\n### ExperiencePrompter\n\nSpecialized for learning from execution traces:\n\n**Purpose:** Summarizes task completion trajectories for future reference\n\n**Use Case:** Self-experience learning in the Knowledge Substrate\n\n### DemonstrationPrompter\n\nSpecialized for learning from human demonstrations:\n\n**Purpose:** Processes Step Recorder outputs into learnable examples\n\n**Use Case:** User demonstration learning in the Knowledge Substrate\n\n## Configuration\n\nPrompter behavior is controlled through system configuration:\n\n```yaml\n# config/ufo/system.yaml\n# Prompt template paths\nHOSTAGENT_PROMPT: \"./ufo/prompts/share/base/host_agent.yaml\"\nAPPAGENT_PROMPT: \"./ufo/prompts/share/base/app_agent.yaml\"\nEVALUATION_PROMPT: \"./ufo/prompts/evaluation/evaluate.yaml\"\n\n# Example prompt paths (visual vs. non-visual)\nHOSTAGENT_EXAMPLE_PROMPT: \"./ufo/prompts/examples/{mode}/host_agent_example.yaml\"\nAPPAGENT_EXAMPLE_PROMPT: \"./ufo/prompts/examples/{mode}/app_agent_example.yaml\"\n\n# Feature flags\nACTION_SEQUENCE: False # Enable multi-action mode for AppAgent\n```\n\nThe `{mode}` placeholder is automatically replaced with `visual` or `nonvisual` based on the LLM's capabilities.\n\n## Custom Prompters\n\nYou can create custom prompters by extending `BasicPrompter` or existing specialized prompters:\n\n```python\nfrom ufo.prompter.agent_prompter import AppAgentPrompter\n\nclass CustomAppPrompter(AppAgentPrompter):\n \"\"\"Custom prompter for specialized application.\"\"\"\n \n def system_prompt_construction(self, **kwargs) -> str:\n # Add custom logic\n base_prompt = super().system_prompt_construction(**kwargs)\n custom_instructions = self.load_custom_instructions()\n return base_prompt + \"\\n\" + custom_instructions\n \n def load_custom_instructions(self) -> str:\n \"\"\"Load application-specific instructions.\"\"\"\n return \"Custom instructions for specialized app...\"\n```\n\n\n# Reference\n\n## Class Hierarchy\n\nThe `Prompter` system is implemented in the `ufo/prompter` folder with the following structure:\n\n```\nufo/prompter/\n├── basic.py # BasicPrompter abstract base class\n├── agent_prompter.py # HostAgentPrompter, AppAgentPrompter\n├── eva_prompter.py # EvaluationAgentPrompter\n├── experience_prompter.py # ExperiencePrompter\n├── demonstration_prompter.py # DemonstrationPrompter\n└── customized/\n └── linux_agent_prompter.py # LinuxAgentPrompter (custom)\n```\n\n## BasicPrompter API\n\nBelow is the complete API reference for the `BasicPrompter` class:\n\n:::prompter.basic.BasicPrompter\n\n## Key Methods\n\n| Method | Purpose | Return Type |\n| --- | --- | --- |\n| `load_prompt_template()` | Load YAML template file | `Dict[str, str]` |\n| `system_prompt_construction()` | Build system prompt | `str` |\n| `user_prompt_construction()` | Build user text prompt | `str` |\n| `user_content_construction()` | Build full user content (text + images) | `List[Dict]` |\n| `prompt_construction()` | Combine system and user into message list | `List[Dict]` |\n| `api_prompt_helper()` | Format API documentation | `str` |\n| `examples_prompt_helper()` | Format demonstration examples | `str` |\n| `retrieved_documents_prompt_helper()` | Format retrieved knowledge | `str` |\n| `tool_to_llm_prompt()` | Convert single tool to LLM format | `str` |\n| `tools_to_llm_prompt()` | Convert multiple tools to LLM format | `str` |\n\n## See Also\n\n- [Prompts Overview](../../../ufo2/prompts/overview.md) - Prompt template structure\n- [Basic Template](../../../ufo2/prompts/basic_template.md) - YAML template format\n- [Example Prompts](../../../ufo2/prompts/examples_prompts.md) - Demonstration examples\n\nYou can customize the `Prompter` class to tailor the prompt to your requirements. Start by extending `BasicPrompter` or one of the specialized prompters."
},
{
"path": "documents/docs/infrastructure/agents/design/state.md",
"content": "# State Layer (Level-1 FSM)\n\nThe **State Layer** is the top-level control structure governing device agent lifecycle. It implements a Finite State Machine (FSM) that determines **when** and **what** to execute, delegating the **how** to the Strategy layer. Each state encapsulates transition logic, processor binding, and multi-agent coordination.\n\n## Overview\n\nThe State Layer implements the **Level-1** of the [three-layer device agent architecture](../overview.md#three-layer-architecture). It provides:\n\n- **Finite State Machine (FSM)**: Governs agent execution lifecycle through state transitions\n- **State Management**: Singleton registry for state classes with lazy loading\n- **Transition Logic**: Rule-based and LLM-driven state transitions\n- **Multi-Agent Coordination**: State-level agent handoff for hierarchical workflows\n\n```mermaid\ngraph TB\n subgraph \"State Layer Components\"\n Status[AgentStatus Enum 7 possible states]\n Manager[AgentStateManager Singleton Registry]\n State[AgentState Interface handle, next_state, next_agent]\n Concrete[Concrete States ContinueState, FinishState, etc.]\n \n Status --> Manager\n Manager -->|lazy loads| Concrete\n Concrete -.->|implements| State\n end\n \n Agent[BasicAgent] -->|current_state| State\n State -->|delegates to| Processor[ProcessorTemplate Level-2 Strategy Layer]\n State -->|transitions to| State\n State -->|hands off to| Agent2[Next Agent]\n```\n\n## Design Philosophy\n\nThe State Layer follows the **State Pattern** from Gang of Four design patterns:\n\n- **Encapsulation**: Each state encapsulates state-specific behavior\n- **Polymorphism**: States share common `AgentState` interface\n- **Dynamic Behavior**: Agent behavior changes dynamically as state changes\n- **Open/Closed Principle**: New states can be added via registration without modifying existing code\n\n## AgentStatus Enum\n\nThe `AgentStatus` enum defines the **seven possible states** that a device agent can be in:\n\n```python\nclass AgentStatus(Enum):\n \"\"\"Enumeration of agent states\"\"\"\n ERROR = \"ERROR\" # Critical error occurred\n FINISH = \"FINISH\" # Task completed successfully\n CONTINUE = \"CONTINUE\" # Normal execution, continue processing\n FAIL = \"FAIL\" # Task failed, cannot proceed\n PENDING = \"PENDING\" # Waiting for external event (user input, async operation)\n CONFIRM = \"CONFIRM\" # Awaiting user confirmation before proceeding\n SCREENSHOT = \"SCREENSHOT\" # Capture observation data (screenshot, UI tree)\n```\n\n### State Characteristics\n\n| State | Type | Description | Typical Next States | Processor Executed |\n|-------|------|-------------|---------------------|-------------------|\n| **CONTINUE** | Active | Normal execution flow, agent processes next step | CONTINUE, FINISH, FAIL, ERROR, PENDING, CONFIRM | Yes ✅ |\n| **FINISH** | Terminal | Task completed successfully, agent stops | (none - end state) | No ❌ |\n| **FAIL** | Terminal | Task failed, agent stops with error | (none - end state) | No ❌ |\n| **ERROR** | Terminal | Critical error, agent stops immediately | (none - end state) | No ❌ |\n| **PENDING** | Waiting | Waiting for external event (user input, callback) | CONTINUE, FAIL | No ❌ |\n| **CONFIRM** | Waiting | Awaiting user confirmation (safety check) | CONTINUE, FAIL | Yes ✅ (collect confirmation) |\n| **SCREENSHOT** | Data Collection | Capture observation without action | CONTINUE | Yes ✅ (capture only) |\n\n### State Categories\n\nStates can be categorized into three groups:\n\n- **Active States** (CONTINUE): Agent actively executing tasks\n- **Waiting States** (PENDING, CONFIRM, SCREENSHOT): Agent waiting for external input or data\n- **Terminal States** (FINISH, FAIL, ERROR): Agent execution completed (success or failure)\n\n## State Machine Diagram\n\nThe following diagram shows the state machine transitions for a typical device agent:\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE: Agent initialized\n \n CONTINUE --> CONTINUE: Step executed successfully (LLM decides to continue)\n CONTINUE --> PENDING: Waiting for external event (async operation, callback)\n CONTINUE --> CONFIRM: User confirmation needed (safety check)\n CONTINUE --> SCREENSHOT: Capture observation (data collection only)\n CONTINUE --> FINISH: Task complete (LLM determines completion)\n CONTINUE --> FAIL: Action failed (error handling)\n CONTINUE --> ERROR: Critical failure (unrecoverable error)\n \n PENDING --> CONTINUE: Event received (user input, callback returned)\n PENDING --> FAIL: Timeout or error (event never received)\n \n CONFIRM --> CONTINUE: User confirmed (approved to proceed)\n CONFIRM --> FAIL: User rejected (operation cancelled)\n \n SCREENSHOT --> CONTINUE: Screenshot captured (observation complete)\n \n FINISH --> [*]: Success\n FAIL --> [*]: Failure\n ERROR --> [*]: Critical Error\n \n note right of CONTINUE\n Active state\n Processor executes all strategies\n end note\n \n note right of PENDING\n Waiting state\n No processor execution\n end note\n \n note right of FINISH\n Terminal state\n Agent lifecycle ends\n end note\n```\n\n### Transition Determination\n\nState transitions are determined by:\n\n1. **LLM Reasoning**: Agent analyzes results and decides next status (e.g., CONTINUE vs FINISH)\n2. **Rule-Based Logic**: Predefined rules trigger transitions (e.g., error → ERROR)\n3. **User Input**: User confirms or rejects → CONFIRM → CONTINUE/FAIL\n4. **External Events**: Async callback received → PENDING → CONTINUE\n\n## AgentStateManager (Singleton Registry)\n\nThe `AgentStateManager` is a **singleton** that manages the registry of state classes. It provides:\n\n- **State Registration**: `@AgentStateManager.register` decorator to register state classes\n- **Lazy Loading**: State instances created only when first accessed\n- **Centralized Management**: Single source of truth for all agent states\n\n```mermaid\ngraph TB\n subgraph \"AgentStateManager (Singleton)\"\n Registry[_state_mapping Dict[str, Type[AgentState]]]\n Instances[_state_instance_mapping Dict[str, AgentState]]\n \n Registry -->|lazy load on first access| Instances\n end\n \n Register[@register decorator] -->|adds class| Registry\n GetState[get_state(status)] -->|creates/retrieves| Instances\n \n Agent1[AppAgent] -->|requests| GetState\n Agent2[HostAgent] -->|requests| GetState\n Agent3[LinuxAgent] -->|requests| GetState\n \n GetState -->|returns| State[AgentState instance]\n```\n\n### AgentStateManager Implementation\n\n```python\nclass AgentStateManager(ABC, metaclass=SingletonABCMeta):\n \"\"\"\n Singleton state manager for agent states.\n \n Responsibilities:\n - Register state classes via decorator\n - Lazy load state instances on demand\n - Provide centralized state access\n \"\"\"\n\n _state_mapping: Dict[str, Type[AgentState]] = {} # Class registry\n\n def __init__(self):\n self._state_instance_mapping: Dict[str, AgentState] = {} # Instance cache\n\n def get_state(self, status: str) -> AgentState:\n \"\"\"\n Get state instance for the given status string.\n \n :param status: The status string (e.g., \"CONTINUE\")\n :return: The state instance\n \n Note: Uses lazy loading - instances created on first access\n \"\"\"\n # Lazy load: create instance only when first requested\n if status not in self._state_instance_mapping:\n state_class = self._state_mapping.get(status)\n if state_class:\n self._state_instance_mapping[status] = state_class()\n else:\n # Fallback to none_state if status not registered\n self._state_instance_mapping[status] = self.none_state\n\n return self._state_instance_mapping.get(status, self.none_state)\n\n def add_state(self, status: str, state: AgentState) -> None:\n \"\"\"\n Add a state instance at runtime (advanced usage).\n \n :param status: The status string\n :param state: The state instance\n \"\"\"\n self._state_instance_mapping[status] = state\n\n @property\n def state_map(self) -> Dict[str, AgentState]:\n \"\"\"\n The state mapping of status to state.\n :return: The state mapping.\n \"\"\"\n return self._state_instance_mapping\n\n @classmethod\n def register(cls, state_class: Type[AgentState]) -> Type[AgentState]:\n \"\"\"\n Decorator to register state class.\n \n Usage:\n @AgentStateManager.register\n class ContinueAppAgentState(AgentState):\n @staticmethod\n def name() -> str:\n return AgentStatus.CONTINUE.value\n \n :param state_class: The state class to register\n :return: The state class (unchanged)\n \"\"\"\n cls._state_mapping[state_class.name()] = state_class\n return state_class\n\n @property\n @abstractmethod\n def none_state(self) -> AgentState:\n \"\"\"\n Fallback state when requested state not found.\n \n :return: Default/fallback state instance\n \"\"\"\n pass\n```\n\n### State Registration Pattern\n\nEach agent type (AppAgent, HostAgent, LinuxAgent) has its own `StateManager` subclass:\n\n```python\n# AppAgent states\nclass AppAgentStateManager(AgentStateManager):\n @property\n def none_state(self):\n return NoneAppAgentState()\n\n@AppAgentStateManager.register\nclass ContinueAppAgentState(AgentState):\n @classmethod\n def name(cls):\n return AgentStatus.CONTINUE.value\n```\n\n**Benefits of Singleton + Lazy Loading**:\n\n- **Memory Efficiency**: State instances created only when needed\n- **Single Source of Truth**: All agents share same state instances\n- **Thread-Safe**: Singleton metaclass ensures thread-safe instantiation\n- **Extensibility**: New states registered without modifying existing code\n\n## AgentState Interface\n\nAll state classes implement the `AgentState` abstract interface:\n\n```python\nclass AgentState(ABC):\n \"\"\"\n Abstract base class for agent states.\n \"\"\"\n\n @abstractmethod\n async def handle(\n self, agent: BasicAgent, context: Optional[Context] = None\n ) -> None:\n \"\"\"\n Handle the agent for the current step.\n :param agent: The agent to handle.\n :param context: The context for the agent and session.\n \"\"\"\n pass\n\n @abstractmethod\n def next_agent(self, agent: BasicAgent) -> BasicAgent:\n \"\"\"\n Get the agent for the next step.\n :param agent: The agent for the current step.\n :return: The agent for the next step.\n \"\"\"\n return agent\n\n @abstractmethod\n def next_state(self, agent: BasicAgent) -> AgentState:\n \"\"\"\n Get the state for the next step.\n :param agent: The agent for the current step.\n :return: The state for the next step.\n \"\"\"\n pass\n\n @abstractmethod\n def is_round_end(self) -> bool:\n \"\"\"\n Check if the round ends.\n :return: True if the round ends, False otherwise.\n \"\"\"\n pass\n\n @abstractmethod\n def is_subtask_end(self) -> bool:\n \"\"\"\n Check if the subtask ends.\n :return: True if the subtask ends, False otherwise.\n \"\"\"\n pass\n\n @classmethod\n @abstractmethod\n def agent_class(cls) -> Type[BasicAgent]:\n \"\"\"\n The class of the agent.\n :return: The class of the agent.\n \"\"\"\n pass\n\n @classmethod\n @abstractmethod\n def name(cls) -> str:\n \"\"\"\n The class name of the state.\n :return: The class name of the state.\n \"\"\"\n return \"\"\n```\n\n### Method Responsibilities\n\n| Method | Purpose | Called By | Returns | Side Effects |\n|--------|---------|-----------|---------|--------------|\n| **handle()** | Execute state-specific logic | Round manager | None | Updates agent status, context, memory |\n| **next_state()** | FSM state transition | Round manager | Next `AgentState` instance | None (pure function) |\n| **next_agent()** | Multi-agent coordination | Round manager | Next `BasicAgent` instance | May create new agent instances |\n| **is_round_end()** | Check if round ends | Round manager | Boolean | None (pure function) |\n| **is_subtask_end()** | Check if subtask ends | Round manager | Boolean | None (pure function) |\n| **agent_class()** | Get agent class | State manager | Agent class type | None (class method) |\n| **name()** | State identifier | State manager registration | State name string | None (class method) |\n\n### Concrete State Example\n\nHere's an example of a concrete state for AppAgent's CONTINUE status:\n\n```python\n@AppAgentStateManager.register\nclass ContinueAppAgentState(AgentState):\n \"\"\"\n Continue state for AppAgent - normal execution flow.\n \"\"\"\n \n async def handle(self, agent: AppAgent, context: Context):\n \"\"\"Execute AppAgent processor strategies.\"\"\"\n # Get processor (Level-2 Strategy Layer)\n processor = agent.processor\n \n # Execute all strategies in sequence\n await processor.process(agent, context)\n \n # Processor updates agent.status based on LLM response\n # Possible status: CONTINUE, FINISH, FAIL, ERROR, CONFIRM, etc.\n \n def next_state(self, agent: AppAgent) -> AgentState:\n \"\"\"Transition to next state based on agent status.\"\"\"\n state_manager = AppAgentStateManager()\n return state_manager.get_state(agent.status)\n \n def next_agent(self, agent: AppAgent) -> BasicAgent:\n \"\"\"For AppAgent, typically stays on same agent.\"\"\"\n # AppAgent continues executing unless delegating back to HostAgent\n if agent.status == AgentStatus.FINISH:\n return agent.host # Return to HostAgent\n return agent # Continue with current agent\n \n @classmethod\n def name(cls) -> str:\n \"\"\"State name for registration\"\"\"\n return AgentStatus.CONTINUE.value # \"CONTINUE\"\n```\n\n## State Lifecycle\n\nThe following sequence diagram shows how states orchestrate agent execution:\n\n```mermaid\nsequenceDiagram\n participant Round as Round Manager\n participant Agent as BasicAgent\n participant State as AgentState\n participant Processor as ProcessorTemplate\n participant LLM\n participant Context\n\n Round->>Agent: Get current_state\n Agent-->>Round: Return state instance\n \n Round->>State: handle(agent, context)\n activate State\n \n State->>Processor: process(agent, context)\n activate Processor\n \n Note over Processor: DATA_COLLECTION strategy\n Processor->>Context: Store screenshot, UI info\n \n Note over Processor: LLM_INTERACTION strategy\n Processor->>LLM: Send prompt with context\n LLM-->>Processor: Return action decision\n \n Note over Processor: ACTION_EXECUTION strategy\n Processor->>Processor: Execute commands\n \n Note over Processor: MEMORY_UPDATE strategy\n Processor->>Agent: Update memory, blackboard\n \n Processor->>Agent: Set status (CONTINUE/FINISH/FAIL/etc)\n deactivate Processor\n \n deactivate State\n \n Round->>State: next_state(agent)\n State-->>Round: Return next state instance\n \n Round->>State: next_agent(agent)\n State-->>Round: Return next agent (may be same or different)\n \n Round->>Round: Update current state, current agent\n Round->>Round: Repeat until terminal state\n```\n\n### Execution Flow\n\n1. **Round Manager** calls `state.handle(agent, context)`\n2. **State** delegates to `processor.process(agent, context)` (Level-2 Strategy Layer)\n3. **Processor** executes strategies (DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE)\n4. **Processor** sets `agent.status` based on LLM response or error handling\n5. **Round Manager** calls `state.next_state(agent)` to get next state\n6. **Round Manager** calls `state.next_agent(agent)` to check for agent handoff\n7. **Round Manager** updates `agent.current_state` and repeats until terminal state\n\n## State-Specific Behaviors\n\nDifferent state types implement different behaviors in their `handle()` method:\n\n### Active State (CONTINUE)\n\n```python\nasync def handle(self, agent, context):\n \"\"\"Execute full processor workflow\"\"\"\n # Run all four strategy phases\n await agent.processor.process(agent, context)\n # Status updated by LLM response parsing\n```\n\n### Waiting State (PENDING)\n\n```python\nasync def handle(self, agent, context):\n \"\"\"Wait for external event\"\"\"\n # Do not execute processor\n # Wait for callback, user input, or timeout\n event = await wait_for_event(timeout=60)\n if event:\n agent.status = AgentStatus.CONTINUE\n else:\n agent.status = AgentStatus.FAIL\n```\n\n### Confirmation State (CONFIRM)\n\n```python\nasync def handle(self, agent, context):\n \"\"\"Request user confirmation\"\"\"\n # Execute DATA_COLLECTION to show current state\n await agent.processor.execute_phase(ProcessingPhase.DATA_COLLECTION)\n \n # Prompt user for confirmation\n confirmed = await prompt_user_confirmation()\n \n if confirmed:\n agent.status = AgentStatus.CONTINUE\n else:\n agent.status = AgentStatus.FAIL\n```\n\n### Terminal State (FINISH/FAIL/ERROR)\n\n```python\nasync def handle(self, agent, context):\n \"\"\"No action - state is terminal\"\"\"\n # Terminal states do not execute processor\n # Round manager will detect terminal status and end execution\n pass\n```\n\n### Processor Execution by State Type\n\n| State Type | Executes Processor? | Which Phases? | Purpose |\n|------------|---------------------|---------------|---------|\n| CONTINUE | ✅ Yes | All phases | Full execution cycle |\n| SCREENSHOT | ✅ Yes | DATA_COLLECTION only | Observation without action |\n| CONFIRM | ✅ Yes | DATA_COLLECTION + custom | Show state, request confirmation |\n| PENDING | ❌ No | None | Wait for external event |\n| FINISH/FAIL/ERROR | ❌ No | None | Terminal states |\n\n## Multi-Agent Coordination\n\nThe State Layer enables **multi-agent coordination** through the `next_agent()` method. This is critical for Windows agents (HostAgent → AppAgent hierarchy).\n\n```mermaid\ngraph TB\n subgraph \"Multi-Agent State Transitions\"\n HS1[HostAgent: CONTINUE]\n HS2[HostAgent: DELEGATE_TO_APP]\n AS1[AppAgent: CONTINUE]\n AS2[AppAgent: FINISH]\n HS3[HostAgent: CONTINUE]\n \n HS1 -->|next_state| HS2\n HS2 -->|next_agent| AS1\n AS1 -->|next_state| AS1\n AS1 -->|next_state| AS2\n AS2 -->|next_agent| HS3\n end\n```\n\n### HostAgent → AppAgent Delegation\n\n```python\nclass ContinueHostAgentState(AgentState):\n def next_agent(self, agent: HostAgent) -> BasicAgent:\n \"\"\"Delegate to AppAgent when task decomposed\"\"\"\n if agent.status == \"DELEGATE_TO_APP\":\n # Create AppAgent for selected application\n app_agent = AgentFactory.create_agent(\n agent_type=\"app\",\n name=f\"AppAgent/{agent.selected_app}\",\n process_name=agent.selected_process,\n app_root_name=agent.selected_app,\n is_visual=True,\n main_prompt=config.appagent_prompt,\n example_prompt=config.appagent_example_prompt\n )\n \n # Set HostAgent as host (for returning)\n app_agent.host = agent\n \n # Transfer context via blackboard\n app_agent.blackboard = agent.blackboard\n \n return app_agent\n \n # No delegation, continue with HostAgent\n return agent\n```\n\n### AppAgent → HostAgent Return\n\n```python\nclass FinishAppAgentState(AgentState):\n def next_agent(self, agent: AppAgent) -> BasicAgent:\n \"\"\"Return to HostAgent when app task complete\"\"\"\n if agent.host:\n # Update HostAgent's blackboard with results\n agent.host.blackboard = agent.blackboard\n \n # Set HostAgent status to continue\n agent.host.status = AgentStatus.CONTINUE\n \n return agent.host\n \n # No host, AppAgent finishes independently\n return agent\n```\n\n## Best Practices\n\n### State Design Guidelines\n\n**1. Single Responsibility**: Each state should have one clear purpose\n\n- ✅ Good: `ContinueState` (normal execution), `ErrorState` (error handling)\n- ❌ Bad: `ContinueOrErrorState` (mixed responsibilities)\n\n**2. Minimal State Logic**: Keep `handle()` simple, delegate to processor\n\n- ✅ Good: `await processor.process(agent, context)`\n- ❌ Bad: Implementing strategy logic directly in state\n\n**3. Predictable Transitions**: Make `next_state()` deterministic when possible\n\n- ✅ Good: Map status string to state instance\n- ❌ Bad: Complex conditional logic in `next_state()`\n\n**4. Document Invariants**: Clearly state what conditions trigger state\n\n- Example: \"PENDING state entered when async operation started\"\n\n### Common Pitfalls\n\n!!! warning \"Stateful State Classes\"\n States should be stateless (data in agent/context, not state)\n \n ```python\n # BAD: Storing state-specific data in state class\n class ContinueState(AgentState):\n def __init__(self):\n self.step_count = 0 # ❌ Don't do this\n \n # GOOD: Store data in agent or context\n class ContinueState(AgentState):\n async def handle(self, agent, context):\n step_count = context.get(\"step_count\", 0) # ✅ Store in context\n ```\n\n!!! warning \"Tight Coupling\"\n States should not depend on specific processor implementations\n \n ```python\n # BAD: Directly calling strategy methods\n async def handle(self, agent, context):\n strategy = AppScreenshotCaptureStrategy() # ❌\n await strategy.execute(agent, context)\n \n # GOOD: Use processor abstraction\n async def handle(self, agent, context):\n await agent.processor.process(agent, context) # ✅\n ```\n\n## Platform-Specific States\n\nDifferent agent types may define platform-specific states:\n\n### Windows AppAgent States\n\n```python\n@AppAgentStateManager.register\nclass ContinueAppAgentState(AgentState):\n \"\"\"Continue state for Windows AppAgent\"\"\"\n # Implements UI Automation-specific logic\n\n@AppAgentStateManager.register\nclass ScreenshotAppAgentState(AgentState):\n \"\"\"Screenshot state for Windows AppAgent\"\"\"\n # Captures Windows UI tree + screenshot\n```\n\n### Linux Agent States\n\n```python\n@LinuxAgentStateManager.register\nclass ContinueLinuxAgentState(AgentState):\n \"\"\"Continue state for Linux Agent\"\"\"\n # Implements shell command execution logic\n\n@LinuxAgentStateManager.register\nclass FinishLinuxAgentState(AgentState):\n \"\"\"Finish state for Linux Agent\"\"\"\n # Terminal state for Linux workflow\n```\n\nWhile state **names** (CONTINUE, FINISH, etc.) are consistent across platforms, state **implementations** (`handle()` method) differ based on:\n\n- Platform-specific processors (Windows UI Automation vs Linux shell)\n- Platform-specific strategies (screenshot+UI tree vs shell output)\n- Platform-specific MCP tools (Win32 API vs shell commands)\n\n## Integration with Other Layers\n\nThe State Layer integrates with other components:\n\n```mermaid\ngraph TB\n subgraph \"State Layer (Level-1)\"\n State[AgentState]\n Manager[AgentStateManager]\n end\n \n subgraph \"Strategy Layer (Level-2)\"\n Processor[ProcessorTemplate]\n Strategies[ProcessingStrategies]\n end\n \n subgraph \"Command Layer (Level-3)\"\n Dispatcher[CommandDispatcher]\n MCP[MCP Tools]\n end\n \n subgraph \"Module System\"\n Round[Round Manager]\n Context[Global Context]\n end\n \n State -->|delegates to| Processor\n Processor -->|executes| Strategies\n Strategies -->|uses| Dispatcher\n Dispatcher -->|calls| MCP\n \n Round -->|orchestrates| State\n Round -->|provides| Context\n State -->|reads/writes| Context\n```\n\n| Integration Point | Layer/Component | Relationship |\n|-------------------|-----------------|--------------|\n| **Round Manager** | Module System | Round calls `handle()`, `next_state()`, `next_agent()` |\n| **ProcessorTemplate** | Level-2 Strategy | State delegates execution to processor |\n| **Global Context** | Module System | State reads request, writes results, shares data |\n| **Agent** | Agent System | State accesses agent properties (memory, blackboard, host) |\n\nSee [Strategy Layer](processor.md), [Command Layer](command.md), and [Round Documentation](../../modules/round.md) for integration details.\n\n## API Reference\n\nBelow is the complete API reference for the State Layer classes:\n\n::: agents.states.basic.AgentState\n::: agents.states.basic.AgentStateManager\n::: agents.states.basic.AgentStatus\n\n## Summary\n\n**Key Takeaways**:\n\n- **Finite State Machine**: State Layer implements FSM with 7 states (CONTINUE, FINISH, FAIL, ERROR, PENDING, CONFIRM, SCREENSHOT)\n- **Singleton Registry**: AgentStateManager provides centralized, lazy-loaded state management\n- **Core Methods**: `handle()` (execute), `next_state()` (FSM transition), `next_agent()` (multi-agent), `is_round_end()`, `is_subtask_end()`, `agent_class()`, `name()`\n- **State Pattern**: Encapsulates state-specific behavior, enables dynamic transitions\n- **Multi-Agent Coordination**: `next_agent()` enables HostAgent ↔ AppAgent delegation\n- **Platform Extensibility**: Same state names, different implementations per platform\n- **Clean Separation**: State controls **when/what**, Processor controls **how**\n\nThe State Layer provides the **control structure** for device agent execution, orchestrating the transition between different behavioral modes while delegating actual execution to the Strategy layer.\n"
},
{
"path": "documents/docs/infrastructure/agents/design/strategy.md",
"content": "# Processing Strategies\n\n**ProcessingStrategy** classes are the fundamental building blocks of agent execution logic. Each strategy encapsulates a specific unit of work (data collection, LLM reasoning, action execution, memory update) with explicit dependencies and outputs. Strategies are composed by Processors to form complete execution workflows.\n\n## Overview\n\nProcessing Strategies implement the **Strategy Pattern**, providing interchangeable algorithms for different aspects of agent behavior. Each strategy:\n\n- Implements a **unified `execute()` interface**\n- Declares **explicit dependencies** (required inputs)\n- Declares **explicit outputs** (provided data)\n- Can be **composed** with other strategies\n- Operates on a **shared ProcessingContext**\n\n```mermaid\ngraph TB\n subgraph \"Strategy Ecosystem\"\n Interface[ProcessingStrategy Protocol]\n \n Base[BaseProcessingStrategy Abstract Base]\n Composed[ComposedStrategy Multiple Strategies]\n \n Interface -.implements.-> Base\n Interface -.implements.-> Composed\n \n subgraph \"Concrete Strategies\"\n DC[DATA_COLLECTION Strategies]\n LLM[LLM_INTERACTION Strategies]\n AE[ACTION_EXECUTION Strategies]\n MU[MEMORY_UPDATE Strategies]\n end\n \n Base -.extends.-> DC\n Base -.extends.-> LLM\n Base -.extends.-> AE\n Base -.extends.-> MU\n end\n \n Processor[ProcessorTemplate] -->|registers & executes| Interface\n Context[ProcessingContext] <-->|read/write data| Interface\n```\n\n**Strategy Benefits:**\n\n- **Modularity**: Each strategy does one thing well, can be tested independently\n- **Reusability**: Same strategy can be used across different processors\n- **Composability**: Combine multiple strategies within a phase via `ComposedStrategy`\n- **Extensibility**: Add new strategies without modifying processor framework\n- **Type Safety**: Explicit dependency declarations prevent runtime errors\n\n---\n\n## ProcessingStrategy Interface\n\nAll strategies implement the `ProcessingStrategy` protocol:\n\n```python\nfrom typing import Protocol\nfrom ufo.agents.agent.basic import BasicAgent\nfrom ufo.agents.processors.context.processing_context import ProcessingContext, ProcessingResult\n\nclass ProcessingStrategy(Protocol):\n \"\"\"\n Protocol for processing strategies.\n \n All strategies must implement the execute() method and provide\n a name attribute for logging/debugging.\n \"\"\"\n \n name: str # Strategy identifier for logging\n \n async def execute(\n self,\n agent: BasicAgent,\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"\n Execute strategy logic.\n \n :param agent: The agent instance (access to memory, blackboard, prompter)\n :param context: Processing context with local/global data\n :return: ProcessingResult with success status and output data\n \"\"\"\n ...\n```\n\n**Minimal Interface:** The protocol defines only what's essential - a `name` for logging/debugging and an `execute()` method for unified execution.\n\n---\n\n## BaseProcessingStrategy\n\nMost concrete strategies extend `BaseProcessingStrategy`, which provides:\n\n- Dependency declaration and validation\n- Output declaration\n- Error handling infrastructure\n- Logging utilities\n\n```python\nfrom abc import ABC, abstractmethod\nfrom typing import List, Optional\nfrom ufo.agents.processors.strategies.dependency import StrategyDependency\n\nclass BaseProcessingStrategy(ABC):\n \"\"\"\n Abstract base class for processing strategies.\n \n Features:\n - Dependency declaration via get_dependencies()\n - Output declaration via get_provides()\n - Dependency validation\n - Standardized error handling\n - Logging integration\n \"\"\"\n \n def __init__(\n self,\n name: Optional[str] = None,\n fail_fast: bool = True\n ):\n \"\"\"\n Initialize strategy.\n \n :param name: Strategy name (defaults to class name)\n :param fail_fast: Raise exception immediately on error vs. return error result\n \"\"\"\n self.name = name or self.__class__.__name__\n self.fail_fast = fail_fast\n self.logger = logging.getLogger(f\"Strategy.{self.name}\")\n \n def get_dependencies(self) -> List[StrategyDependency]:\n \"\"\"\n Declare required dependencies.\n \n Override to specify what data this strategy needs from context.\n \n Example:\n return [\n StrategyDependency(\"screenshot\", required=True, expected_type=str),\n StrategyDependency(\"control_info\", required=False, expected_type=str)\n ]\n \n :return: List of dependency declarations\n \"\"\"\n return []\n \n def get_provides(self) -> List[str]:\n \"\"\"\n Declare provided outputs.\n \n Override to specify what data this strategy writes to context.\n \n Example:\n return [\"parsed_response\", \"action\", \"arguments\"]\n \n :return: List of output field names\n \"\"\"\n return []\n \n def validate_dependencies(self, context: ProcessingContext) -> List[str]:\n \"\"\"\n Validate that all required dependencies are available in context.\n \n :param context: Processing context to validate against\n :return: List of missing required dependency names\n \"\"\"\n missing = []\n for dependency in self.get_dependencies():\n value = context.get_local(dependency.field_name)\n if dependency.required and value is None:\n missing.append(dependency.field_name)\n return missing\n \n def handle_error(\n self,\n error: Exception,\n phase: ProcessingPhase,\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"\n Standardized error handling.\n \n :param error: The exception that occurred\n :param phase: Processing phase where error occurred\n :param context: Current processing context\n :return: ProcessingResult with error information\n \"\"\"\n self.logger.error(f\"Strategy {self.name} failed: {error}\", exc_info=True)\n \n if self.fail_fast:\n raise error\n else:\n return ProcessingResult(\n success=False,\n data={},\n error=str(error),\n phase=phase\n )\n \n @abstractmethod\n async def execute(\n self,\n agent: BasicAgent,\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"\n Execute strategy logic.\n \n Subclasses must implement this method.\n \n :param agent: Agent instance\n :param context: Processing context\n :return: ProcessingResult with outputs\n \"\"\"\n pass\n```\n\n**Creating a Concrete Strategy Example:**\n\n```python\nfrom ufo.agents.processors.strategies.processing_strategy import BaseProcessingStrategy\nfrom ufo.agents.processors.strategies.strategy_dependency import StrategyDependency\nfrom ufo.agents.processors.context.processing_context import ProcessingResult, ProcessingPhase\n\nclass AppScreenshotCaptureStrategy(BaseProcessingStrategy):\n \"\"\"Capture screenshot of Windows application\"\"\"\n \n def __init__(self):\n super().__init__(name=\"AppScreenshotCapture\")\n \n def get_dependencies(self) -> List[StrategyDependency]:\n # No dependencies - runs first in DATA_COLLECTION phase\n return []\n \n def get_provides(self) -> List[str]:\n return [\"screenshot\", \"screenshot_path\"]\n \n async def execute(\n self,\n agent,\n context: ProcessingContext\n ) -> ProcessingResult:\n try:\n # Capture screenshot\n screenshot_path = await self._capture_screenshot(agent)\n screenshot_str = self._encode_image(screenshot_path)\n \n # Return result with provided data\n return ProcessingResult(\n success=True,\n data={\n \"screenshot\": screenshot_str,\n \"screenshot_path\": screenshot_path\n },\n phase=ProcessingPhase.DATA_COLLECTION\n )\n except Exception as e:\n return self.handle_error(e, ProcessingPhase.DATA_COLLECTION, context)\n \n async def _capture_screenshot(self, agent):\n # Platform-specific screenshot logic\n ...\n \n def _encode_image(self, path):\n # Base64 encoding for LLM\n ...\n```\n\n---\n\n## Strategy Dependency System\n\nThe dependency system ensures strategies execute in correct order with required data available.\n\n### StrategyDependency\n\n```python\nfrom dataclasses import dataclass\nfrom typing import Optional, Type\n\n@dataclass\nclass StrategyDependency:\n \"\"\"\n Represents a data dependency for a strategy.\n \n :param field_name: Name of required field in ProcessingContext\n :param required: Whether dependency is mandatory (vs. optional)\n :param expected_type: Expected Python type (for validation)\n :param description: Human-readable description\n \"\"\"\n field_name: str\n required: bool = True\n expected_type: Optional[Type] = None\n description: str = \"\"\n```\n\n### Dependency Declaration\n\nStrategies declare dependencies in two ways:\n\n#### Method 1: Override `get_dependencies()`\n\n```python\nclass LLMInteractionStrategy(BaseProcessingStrategy):\n def get_dependencies(self) -> List[StrategyDependency]:\n return [\n StrategyDependency(\n field_name=\"screenshot\",\n required=True,\n expected_type=str,\n description=\"Base64-encoded screenshot for LLM visual input\"\n ),\n StrategyDependency(\n field_name=\"control_info\",\n required=True,\n expected_type=str,\n description=\"UI control information from UI Automation\"\n ),\n StrategyDependency(\n field_name=\"request\",\n required=True,\n expected_type=str,\n description=\"User's task request\"\n )\n ]\n \n def get_provides(self) -> List[str]:\n return [\"parsed_response\", \"action\", \"arguments\"]\n```\n\n#### Method 2: Use Decorators\n\n```python\nfrom ufo.agents.processors.strategies.strategy_dependency import depends_on, provides\n\n@depends_on(\"screenshot\", \"control_info\", \"request\")\n@provides(\"parsed_response\", \"action\", \"arguments\")\nclass LLMInteractionStrategy(BaseProcessingStrategy):\n async def execute(self, agent, context):\n # Dependency validation automatic via StrategyDependencyValidator\n screenshot = context.require_local(\"screenshot\")\n control_info = context.require_local(\"control_info\")\n request = context.get_global(\"REQUEST\")\n \n # ... LLM interaction logic ...\n \n return ProcessingResult(\n success=True,\n data={\n \"parsed_response\": parsed,\n \"action\": action,\n \"arguments\": arguments\n }\n )\n```\n\n**Dependency Validation:** The processor validates dependencies before executing each strategy using `StrategyDependencyValidator`:\n\n```python\n# In ProcessorTemplate.process()\nfor phase in execution_order:\n strategy = self.strategies.get(phase)\n if strategy:\n # Validate dependencies at runtime\n self._validate_strategy_dependencies_runtime(strategy, self.processing_context)\n \n # Execute strategy\n result = await strategy.execute(agent, self.processing_context)\n```\n\n---\n\n## Four Core Strategy Types\n\nStrategies are organized by **ProcessingPhase**, with four core types:\n\n```mermaid\ngraph LR\n subgraph \"Strategy Types by Phase\"\n DC[DATA_COLLECTION Strategies]\n LLM[LLM_INTERACTION Strategies]\n AE[ACTION_EXECUTION Strategies]\n MU[MEMORY_UPDATE Strategies]\n \n DC -->|provides data| LLM\n LLM -->|provides decisions| AE\n AE -->|provides results| MU\n end\n```\n\n### 1. DATA_COLLECTION Strategies\n\n**Purpose**: Gather contextual information from the device/environment\n\n**Common Implementations**:\n- `AppScreenshotCaptureStrategy`: Capture application screenshot (Windows)\n- `AppControlInfoStrategy`: Extract UI Automation tree (Windows)\n- `LinuxShellOutputStrategy`: Capture shell command output (Linux)\n- `SystemStatusStrategy`: Gather system metrics (CPU, memory, disk)\n\n**Dependencies**: None (typically first in execution chain)\n\n**Provides**: `screenshot`, `control_info`, `observation`, `system_status`\n\n```python\nclass AppControlInfoStrategy(BaseProcessingStrategy):\n \"\"\"Extract UI Automation tree from Windows application\"\"\"\n \n def get_dependencies(self) -> List[StrategyDependency]:\n return [] # No dependencies\n \n def get_provides(self) -> List[str]:\n return [\"control_info\", \"control_elements\"]\n \n async def execute(self, agent, context):\n # Get UI Automation tree via MCP tool\n command = Command(function=\"get_ui_tree\", arguments={})\n results = agent.dispatcher.execute_commands([command])\n \n control_tree = results[0].result\n \n return ProcessingResult(\n success=True,\n data={\n \"control_info\": control_tree,\n \"control_elements\": self._parse_tree(control_tree)\n },\n phase=ProcessingPhase.DATA_COLLECTION\n )\n```\n\n**Platform Differences:**\n\n- **Windows**: Screenshot + UI Automation tree\n- **Linux**: Screenshot + shell output + accessibility tree (X11/Wayland)\n- **macOS**: Screenshot + Accessibility API tree (future)\n\n### 2. LLM_INTERACTION Strategies\n\n**Purpose**: Construct prompts, call LLM, parse responses\n\n**Common Implementations**:\n- `AppLLMInteractionStrategy`: UI element selection for AppAgent (Windows)\n- `HostLLMInteractionStrategy`: Application selection for HostAgent (Windows)\n- `LinuxLLMInteractionStrategy`: Shell command generation for LinuxAgent\n\n**Dependencies**: `screenshot`, `control_info`, `request`, `memory`\n\n**Provides**: `parsed_response`, `action`, `arguments`, `function_call`\n\n```python\nclass AppLLMInteractionStrategy(BaseProcessingStrategy):\n \"\"\"LLM reasoning for Windows AppAgent\"\"\"\n \n def get_dependencies(self) -> List[StrategyDependency]:\n return [\n StrategyDependency(\"screenshot\", required=True, expected_type=str),\n StrategyDependency(\"control_info\", required=True, expected_type=str),\n StrategyDependency(\"request\", required=True, expected_type=str)\n ]\n \n def get_provides(self) -> List[str]:\n return [\"parsed_response\", \"action\", \"arguments\", \"function_call\"]\n \n async def execute(self, agent, context):\n # 1. Build prompt with screenshot + UI elements\n prompt = agent.prompter.construct_prompt(\n screenshot=context.get_local(\"screenshot\"),\n control_info=context.get_local(\"control_info\"),\n request=context.get_global(\"request\"),\n memory=agent.memory.get_latest(5)\n )\n \n # 2. Call LLM\n response = await agent.llm_client.get_response(prompt)\n \n # 3. Parse JSON response\n parsed = agent.prompter.parse_response(response)\n \n # 4. Extract action details\n return ProcessingResult(\n success=True,\n data={\n \"parsed_response\": parsed,\n \"action\": parsed.get(\"ControlText\"),\n \"arguments\": parsed.get(\"Plan\"),\n \"function_call\": parsed.get(\"Function\")\n },\n phase=ProcessingPhase.LLM_INTERACTION\n )\n```\n\n!!! warning \"LLM Response Validation\"\n Always validate and sanitize LLM outputs to prevent errors and security issues:\n \n ```python\n # Validate required fields\n if \"Function\" not in parsed:\n raise ProcessingException(\"LLM response missing 'Function' field\")\n \n # Sanitize dangerous operations\n if parsed[\"Function\"] == \"shell_execute\":\n command = parsed.get(\"Plan\", \"\")\n if any(danger in command for danger in [\"rm -rf\", \"del /f /q\"]):\n raise ProcessingException(\"Dangerous command detected\")\n ```\n\n### 3. ACTION_EXECUTION Strategies\n\n**Purpose**: Execute commands via CommandDispatcher\n\n**Common Implementations**:\n- `AppActionExecutionStrategy`: Execute UI Automation commands (Windows)\n- `HostActionExecutionStrategy`: Launch applications, create AppAgents (Windows)\n- `LinuxActionExecutionStrategy`: Execute shell commands (Linux)\n\n**Dependencies**: `action`, `arguments`, `function_call`, `command_dispatcher`\n\n**Provides**: `results`, `execution_status`, `action_success`\n\n```python\nclass AppActionExecutionStrategy(BaseProcessingStrategy):\n \"\"\"Execute UI Automation commands for Windows AppAgent\"\"\"\n \n def get_dependencies(self) -> List[StrategyDependency]:\n return [\n StrategyDependency(\"action\", required=True, expected_type=str),\n StrategyDependency(\"arguments\", required=True, expected_type=dict),\n StrategyDependency(\"function_call\", required=True, expected_type=str)\n ]\n \n def get_provides(self) -> List[str]:\n return [\"results\", \"execution_status\", \"action_success\"]\n \n async def execute(self, agent, context):\n # 1. Build command from LLM output\n command = Command(\n function=context.get_local(\"function_call\"),\n arguments=context.get_local(\"arguments\")\n )\n \n # 2. Execute via dispatcher (routes to device client)\n dispatcher = context.get_global(\"command_dispatcher\")\n results = await dispatcher.execute_commands([command])\n \n # 3. Check execution success\n success = all(r.status == ResultStatus.SUCCESS for r in results)\n \n return ProcessingResult(\n success=True,\n data={\n \"results\": results,\n \"execution_status\": results[0].status,\n \"action_success\": success\n },\n phase=ProcessingPhase.ACTION_EXECUTION\n )\n```\n\nSee the [Command Layer documentation](command.md) for details on command execution.\n\n### 4. MEMORY_UPDATE Strategies\n\n**Purpose**: Update agent memory and shared blackboard\n\n**Common Implementations**:\n- `AppMemoryUpdateStrategy`: Record UI interactions (Windows AppAgent)\n- `HostMemoryUpdateStrategy`: Record application selections (Windows HostAgent)\n- `LinuxMemoryUpdateStrategy`: Record shell command history (Linux)\n\n**Dependencies**: `action`, `results`, `observation`, `screenshot`\n\n**Provides**: `memory_item`, `updated_blackboard`\n\n```python\nclass AppMemoryUpdateStrategy(BaseProcessingStrategy):\n \"\"\"Update memory for Windows AppAgent\"\"\"\n \n def get_dependencies(self) -> List[StrategyDependency]:\n return [\n StrategyDependency(\"action\", required=True),\n StrategyDependency(\"results\", required=True),\n StrategyDependency(\"screenshot_path\", required=False)\n ]\n \n def get_provides(self) -> List[str]:\n return [\"memory_item\", \"updated_blackboard\"]\n \n async def execute(self, agent, context):\n # 1. Create memory item for agent's short-term memory\n memory_item = MemoryItem()\n memory_item.add_values_from_dict({\n \"step\": context.get_global(\"session_step\"),\n \"action\": context.get_local(\"action\"),\n \"results\": context.get_local(\"results\"),\n \"screenshot\": context.get_local(\"screenshot_path\"),\n \"observation\": context.get_local(\"control_info\")\n })\n \n # 2. Add to agent memory\n agent.memory.add_memory_item(memory_item)\n \n # 3. Update blackboard (shared multi-agent memory)\n if context.get_local(\"action_success\"):\n agent.blackboard.add_trajectories({\n \"step\": context.get_global(\"session_step\"),\n \"action\": context.get_local(\"action\"),\n \"status\": \"success\"\n })\n \n return ProcessingResult(\n success=True,\n data={\n \"memory_item\": memory_item,\n \"updated_blackboard\": True\n },\n phase=ProcessingPhase.MEMORY_UPDATE\n )\n```\n\nSee the [Memory System documentation](memory.md) for details on Memory and Blackboard.\n\n---\n\n## ComposedStrategy\n\nThe `ComposedStrategy` class enables **combining multiple strategies** within a single processing phase:\n\n```python\nclass ComposedStrategy(BaseProcessingStrategy):\n \"\"\"\n Compose multiple strategies into a single execution flow.\n \n Features:\n - Sequential execution of component strategies\n - Aggregated dependency/provides metadata\n - Flexible error handling (fail-fast or continue-on-error)\n - Shared processing context across components\n \"\"\"\n \n def __init__(\n self,\n strategies: List[BaseProcessingStrategy],\n name: str = \"\",\n fail_fast: bool = True,\n phase: ProcessingPhase = ProcessingPhase.DATA_COLLECTION\n ):\n \"\"\"\n Initialize composed strategy.\n \n :param strategies: List of strategies to execute sequentially\n :param name: Composed strategy name\n :param fail_fast: Stop on first error vs. continue execution\n :param phase: Processing phase this composition belongs to\n \"\"\"\n super().__init__(name=name or \"ComposedStrategy\", fail_fast=fail_fast)\n \n if not strategies:\n raise ValueError(\"ComposedStrategy requires at least one strategy\")\n \n self.strategies = strategies\n self.execution_phase = phase\n \n # Collect metadata from component strategies\n self._collect_metadata()\n \n def _collect_metadata(self):\n \"\"\"Aggregate dependencies and provides from component strategies\"\"\"\n all_deps = []\n all_provides = set()\n \n for strategy in self.strategies:\n all_deps.extend(strategy.get_dependencies())\n all_provides.update(strategy.get_provides())\n \n # Remove internal dependencies (provided by earlier strategies in composition)\n external_deps = [\n dep for dep in all_deps\n if dep.field_name not in all_provides\n ]\n \n self._dependencies = external_deps\n self._provides = list(all_provides)\n \n def get_dependencies(self) -> List[StrategyDependency]:\n return self._dependencies\n \n def get_provides(self) -> List[str]:\n return self._provides\n \n async def execute(\n self,\n agent: BasicAgent,\n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Execute all component strategies sequentially\"\"\"\n combined_data = {}\n \n for i, strategy in enumerate(self.strategies):\n self.logger.debug(\n f\"Executing component strategy {i+1}/{len(self.strategies)}: {strategy.name}\"\n )\n \n # Execute component strategy\n result = await strategy.execute(agent, context)\n \n if result.success:\n # Update context for next strategy\n context.update_local(result.data)\n combined_data.update(result.data)\n else:\n # Handle failure\n self.logger.error(f\"Component strategy {strategy.name} failed: {result.error}\")\n \n if self.fail_fast:\n return result # Propagate failure immediately\n else:\n # Continue with remaining strategies\n self.logger.warning(f\"Continuing despite failure in {strategy.name}\")\n \n return ProcessingResult(\n success=True,\n data=combined_data,\n phase=self.execution_phase\n )\n```\n\n### Using ComposedStrategy\n\n```mermaid\ngraph TB\n subgraph \"DATA_COLLECTION Phase\"\n Composed[ComposedStrategy]\n \n S1[ScreenshotStrategy]\n S2[UITreeStrategy]\n S3[SystemStatusStrategy]\n \n Composed -->|1. execute| S1\n Composed -->|2. execute| S2\n Composed -->|3. execute| S3\n \n S1 -->|provides: screenshot| Context[ProcessingContext]\n S2 -->|provides: control_info| Context\n S3 -->|provides: system_status| Context\n end\n```\n\n**Composing DATA_COLLECTION Strategies Example:**\n\n```python\n# In AppAgentProcessor._setup_strategies()\n\n# Compose multiple data collection strategies\ndata_collection = ComposedStrategy(\n strategies=[\n AppScreenshotCaptureStrategy(),\n AppControlInfoStrategy(),\n SystemStatusStrategy()\n ],\n name=\"AppDataCollection\",\n fail_fast=False, # Continue even if SystemStatus fails (optional)\n phase=ProcessingPhase.DATA_COLLECTION\n)\n\n# Register composed strategy\nself.strategies[ProcessingPhase.DATA_COLLECTION] = data_collection\n```\n\n**Composition Benefits:**\n\n- **Modularity**: Build complex workflows from simple, testable components\n- **Reusability**: Mix and match strategies across different processors\n- **Flexibility**: Easily reorder or replace component strategies\n- **Error Handling**: Choose fail-fast or continue-on-error per composition\n- **Metadata Aggregation**: Dependencies and provides automatically computed\n\n---\n\n## Best Practices\n\n### Strategy Design Guidelines\n\n**1. Single Responsibility:** Each strategy should do one thing well.\n\n- ✅ Good: `ScreenshotCaptureStrategy` (captures screenshot)\n- ❌ Bad: `ScreenshotAndLLMStrategy` (mixed concerns)\n\n**2. Explicit Dependencies:** Always declare what you need.\n\n```python\ndef get_dependencies(self) -> List[StrategyDependency]:\n return [\n StrategyDependency(\"screenshot\", required=True),\n StrategyDependency(\"system_status\", required=False) # Optional\n ]\n```\n\n**3. Clear Outputs:** Document what you provide.\n\n```python\ndef get_provides(self) -> List[str]:\n return [\"parsed_response\", \"action\", \"arguments\"]\n```\n\n**4. Appropriate Error Handling:**\n\n- Use `fail_fast=True` for critical strategies (LLM_INTERACTION, ACTION_EXECUTION)\n- Use `fail_fast=False` for optional strategies (system metrics, logging)\n\n**5. Platform Agnostic:** Strategies shouldn't assume specific agent types.\n\n```python\n# ❌ BAD: Type-checking agent\nasync def execute(self, agent, context):\n if isinstance(agent, AppAgent): # Tight coupling\n ...\n\n# ✅ GOOD: Use context data\nasync def execute(self, agent, context):\n control_info = context.require_local(\"control_info\") # Generic\n ...\n```\n\n### Common Pitfalls\n\n!!! warning \"Pitfall 1: Stateful Strategies\"\n Strategies should be stateless (no instance variables modified during execution):\n \n ```python\n # ❌ BAD: Stateful\n class BadStrategy(BaseProcessingStrategy):\n def __init__(self):\n super().__init__()\n self.counter = 0 # State\n \n async def execute(self, agent, context):\n self.counter += 1 # Modifying state\n ...\n \n # ✅ GOOD: Stateless\n class GoodStrategy(BaseProcessingStrategy):\n async def execute(self, agent, context):\n counter = context.get_local(\"counter\", 0) # Read from context\n context.update_local({\"counter\": counter + 1}) # Write to context\n ...\n ```\n\n!!! warning \"Pitfall 2: Hidden Dependencies\"\n Don't access context data without declaring dependencies:\n \n ```python\n # ❌ BAD: Hidden dependency\n class BadStrategy(BaseProcessingStrategy):\n def get_dependencies(self):\n return [] # Claims no dependencies\n \n async def execute(self, agent, context):\n screenshot = context.get_local(\"screenshot\") # But uses screenshot!\n ...\n \n # ✅ GOOD: Explicit dependency\n class GoodStrategy(BaseProcessingStrategy):\n def get_dependencies(self):\n return [StrategyDependency(\"screenshot\", required=True)]\n \n async def execute(self, agent, context):\n screenshot = context.require_local(\"screenshot\")\n ...\n ```\n\n!!! warning \"Pitfall 3: Side Effects\"\n Strategies shouldn't modify global state or agent attributes directly:\n \n ```python\n # ❌ BAD: Side effects\n async def execute(self, agent, context):\n agent.custom_attribute = \"value\" # Modifying agent\n global_config[\"setting\"] = \"new\" # Modifying global\n ...\n \n # ✅ GOOD: Update through proper channels\n async def execute(self, agent, context):\n context.update_local({\"custom_value\": \"value\"}) # Context\n agent.memory.add_memory_item(...) # Memory system\n ...\n ```\n\n---\n\n## Integration with Processor\n\nStrategies are **registered and executed** by ProcessorTemplate:\n\n```mermaid\nsequenceDiagram\n participant Processor as ProcessorTemplate\n participant Strategy as ProcessingStrategy\n participant Context as ProcessingContext\n \n Note over Processor: Initialization\n Processor->>Processor: _setup_strategies()\n Processor->>Processor: Register strategies by phase\n \n Note over Processor: Execution\n Processor->>Strategy: validate_dependencies(context)\n Strategy-->>Processor: [] (no missing deps)\n \n Processor->>Strategy: execute(agent, context)\n Strategy->>Context: get_local(\"screenshot\")\n Context-->>Strategy: screenshot data\n \n Strategy->>Strategy: Process data\n \n Strategy->>Context: update_local({\"parsed_response\": ...})\n Strategy-->>Processor: ProcessingResult(success=True, data={...})\n \n Processor->>Context: Update with strategy outputs\n```\n\n**See [Processor Documentation](processor.md) for details on how processors orchestrate strategies.**\n\n---\n\n## Platform-Specific Strategies\n\nDifferent platforms implement platform-specific strategies while following the same interface:\n\n| Platform | DATA_COLLECTION | LLM_INTERACTION | ACTION_EXECUTION | MEMORY_UPDATE |\n|----------|-----------------|-----------------|------------------|---------------|\n| **Windows AppAgent** | Screenshot + UI tree | UI element selection | UI Automation commands | UI interaction history |\n| **Windows HostAgent** | Desktop screenshot + app list | Application selection | Launch app, create AppAgent | App selection history |\n| **Linux** | Screenshot + shell output | Shell command generation | Shell command execution | Command history |\n| **macOS** (future) | Screenshot + Accessibility tree | Accessibility element selection | Accessibility API commands | Interaction history |\n\n!!! example \"Platform-Specific Implementation\"\n ```python\n # Windows AppAgent\n class AppAgentProcessor(ProcessorTemplate):\n def _setup_strategies(self):\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([\n AppScreenshotCaptureStrategy(), # Windows-specific\n AppControlInfoStrategy() # UI Automation specific\n ])\n \n # Linux Agent\n class LinuxAgentProcessor(ProcessorTemplate):\n def _setup_strategies(self):\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([\n CustomizedScreenshotCaptureStrategy(), # Linux-specific\n ShellOutputStrategy() # Shell-specific\n ])\n ```\n\n---\n\n## Related Documentation\n\n- [Strategy Layer - Processor](processor.md): How ProcessorTemplate orchestrates strategies\n- [Command Layer](command.md): How ACTION_EXECUTION strategies dispatch commands \n- [Memory System](memory.md): How MEMORY_UPDATE strategies use Memory and Blackboard\n- [State Layer](state.md): How AgentState delegates to Processor\n- [Agent Types](../agent_types.md): Platform-specific strategy implementations\n\n---\n\n## API Reference\n\nThe following classes are documented via docstrings:\n\n- `ProcessingStrategy`: Protocol defining strategy interface\n- `BaseProcessingStrategy`: Abstract base class for strategies\n- `ComposedStrategy`: Compose multiple strategies within a phase\n- `StrategyDependency`: Dependency declaration dataclass\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n- **ProcessingStrategy**: Unified interface with `execute()` method\n- **BaseProcessingStrategy**: Abstract base with dependency management and error handling\n- **Four Strategy Types**: DATA_COLLECTION, LLM_INTERACTION, ACTION_EXECUTION, MEMORY_UPDATE\n- **Dependency System**: Explicit declarations ensure correct execution order via `StrategyDependency`\n- **ComposedStrategy**: Combine multiple strategies within a phase\n- **Platform Agnostic**: Same interface, platform-specific implementations\n- **Modular & Reusable**: Strategies can be mixed, matched, and tested independently\n- **Processor Integration**: Strategies are registered and orchestrated by ProcessorTemplate\n\nProcessing Strategies are the fundamental building blocks of agent execution logic, providing modularity, reusability, and extensibility across diverse platforms and task requirements.\n"
},
{
"path": "documents/docs/infrastructure/agents/overview.md",
"content": "# Device Agent Architecture\n\nDevice Agents are the execution engines of UFO3's multi-device orchestration system. Each device agent operates as an autonomous, intelligent controller that translates high-level user intentions into low-level system commands. The architecture is designed for **extensibility**, **safety**, and **scalability** across heterogeneous computing environments.\n\n## Overview\n\nUFO3 orchestrates tasks across multiple devices through a network of **Device Agents**. Originally designed as a Windows automation framework (UFO2), the architecture has evolved to support diverse platforms including Linux, macOS, and embedded systems. This document describes the abstract design principles and interfaces that enable this multi-platform capability.\n\n**Key Capabilities:**\n\n- **Multi-Platform**: Windows agents (HostAgent, AppAgent), Linux agent, extensible to macOS and embedded systems\n- **Safe Execution**: Server-client separation isolates reasoning from system-level operations\n- **Scalable Architecture**: Hierarchical agent coordination supports complex cross-device workflows\n- **LLM-Driven Reasoning**: Dynamic decision-making using large language models\n- **Modular Design**: Three-layer architecture (State, Strategy, Command) enables customization\n\n---\n\n## Three-Layer Architecture\n\nDevice agents implement a **three-layer framework** that separates concerns, promotes modularity, and enables extensibility:\n\n```mermaid\ngraph TB\n subgraph \"Device Agent Architecture\"\n subgraph \"Level-1: State Layer (FSM)\"\n S1[AgentState]\n S2[State Machine]\n S3[State Transitions]\n S1 --> S2 --> S3\n end\n \n subgraph \"Level-2: Strategy Layer (Execution Logic)\"\n P1[ProcessorTemplate Strategy Orchestrator]\n P2[DATA_COLLECTION Strategies]\n P3[LLM_INTERACTION Strategies]\n P4[ACTION_EXECUTION Strategies]\n P5[MEMORY_UPDATE Strategies]\n P1 -->|manages & executes| P2\n P2 --> P3 --> P4 --> P5\n end\n \n subgraph \"Level-3: Command Layer (System Interface)\"\n C1[CommandDispatcher]\n C2[MCP Tools]\n C3[Atomic Commands]\n C1 --> C2 --> C3\n end\n \n S3 -->|delegates to| P1\n P5 -->|executes via| C1\n end\n \n LLM[Large Language Model]\n P3 -.->|reasoning| LLM\n LLM -.->|decisions| P4\n```\n\n### Layer Responsibilities\n\n| Layer | Level | Responsibility | Key Components | Extensibility |\n|-------|-------|----------------|----------------|---------------|\n| **State** | Level-1 | Finite State Machine governing agent lifecycle | `AgentState`, `AgentStateManager`, `AgentStatus` | Register new states via `@AgentStateManager.register` |\n| **Strategy** | Level-2 | Execution logic layer: processor manages sequence of modular strategies | `ProcessorTemplate`, `ProcessingStrategy`, `ProcessingPhase`, `Middleware` | Compose custom strategies via `ComposedStrategy`, add middleware |\n| **Command** | Level-3 | Atomic system operations mapped to MCP tools | `BasicCommandDispatcher`, `Command`, MCP integration | Add new tools via client-side MCP server registration |\n\n**Design Rationale:**\n\nThe three-layer separation ensures:\n\n- **State Layer (Level-1)**: Controls *when* and *what* to execute (state transitions, agent handoff)\n- **Strategy Layer (Level-2)**: Defines *how* to execute (processor orchestrates modular strategies)\n- **Command Layer (Level-3)**: Performs *actual* execution (deterministic system operations)\n\nThis separation allows replacing individual layers without affecting others.\n\n---\n\n## Level-1: State Layer (FSM)\n\nThe **State Layer** implements a Finite State Machine (FSM) that governs the agent's execution lifecycle. Each state encapsulates:\n\n- A **processor** (strategy execution logic)\n- **Transition rules** (to next state)\n- **Agent handoff logic** (for multi-agent workflows)\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE\n CONTINUE --> CONTINUE: Success\n CONTINUE --> PENDING: Wait for external event\n CONTINUE --> CONFIRM: User confirmation needed\n CONTINUE --> SCREENSHOT: Capture observation\n CONTINUE --> FINISH: Task complete\n CONTINUE --> FAIL: Error occurred\n CONTINUE --> ERROR: Critical failure\n \n PENDING --> CONTINUE: Event received\n CONFIRM --> CONTINUE: User confirmed\n CONFIRM --> FAIL: User rejected\n SCREENSHOT --> CONTINUE: Screenshot captured\n \n FINISH --> [*]\n FAIL --> [*]\n ERROR --> [*]\n```\n\n### AgentStatus Enum\n\n```python\nclass AgentStatus(Enum):\n \"\"\"Agent status enumeration\"\"\"\n ERROR = \"ERROR\" # Critical error occurred\n FINISH = \"FINISH\" # Task completed successfully\n CONTINUE = \"CONTINUE\" # Normal execution\n FAIL = \"FAIL\" # Task failed\n PENDING = \"PENDING\" # Waiting for external event\n CONFIRM = \"CONFIRM\" # Awaiting user confirmation\n SCREENSHOT = \"SCREENSHOT\" # Screenshot capture needed\n```\n\n**State Registration:**\n\nNew states can be registered dynamically using the `@AgentStateManager.register` decorator:\n\n```python\n@AgentStateManager.register\nclass CustomState(AgentState):\n async def handle(self, agent, context):\n # Custom state logic\n pass\n \n def next_state(self, agent):\n return AgentStateManager.get_state(\"CONTINUE\")\n```\n\n**See [State Layer Documentation](design/state.md) for complete details.**\n\n---\n\n## Level-2: Strategy Layer (Execution Logic)\n\nThe **Strategy Layer** implements the execution logic within each state. Each state encapsulates a **processor** that manages a sequence of **strategies** to implement step-level workflow. This layer consists of two key components:\n\n### Processor: Strategy Orchestrator\n\nThe **ProcessorTemplate** orchestrates the execution of strategies:\n\n- **Registers Strategies**: Configures which strategies execute in each phase\n- **Manages Middleware**: Wraps strategy execution with logging, metrics, error handling\n- **Validates Dependencies**: Ensures strategies have required data before execution\n- **Controls Execution**: Sequences strategies through fixed workflow phases\n\n```mermaid\ngraph TB\n State[AgentState] -->|encapsulates| Processor[ProcessorTemplate Strategy Orchestrator]\n \n Processor -->|1. Register| Strategies[ProcessingStrategies]\n Processor -->|2. Wrap| Middleware[Middleware Chain]\n Processor -->|3. Validate| Dependencies[Strategy Dependencies]\n Processor -->|4. Execute| Workflow[Workflow Phases]\n \n Workflow --> DC[DATA_COLLECTION]\n Workflow --> LLM[LLM_INTERACTION]\n Workflow --> AE[ACTION_EXECUTION]\n Workflow --> MU[MEMORY_UPDATE]\n \n DC & LLM & AE & MU -.->|implements| Strategies\n```\n\n**Processor and Strategy Relationship:**\n\n- **Processor**: Framework that manages the sequence of strategies\n- **Strategy**: Modular, reusable execution units\n\nTogether they form **Level-2: Strategy Layer**, which handles:\n- Data collection and environment inspection\n- Prompt construction and LLM reasoning\n- Action planning and tool invocation\n- Memory updates and context synchronization\n\n### Strategy: Modular Execution Units\n\n**ProcessingStrategies** are modular execution units with a unified `execute()` interface:\n\n```mermaid\ngraph LR\n A[DATA_COLLECTION] --> B[LLM_INTERACTION]\n B --> C[ACTION_EXECUTION]\n C --> D[MEMORY_UPDATE]\n \n A1[Screenshots UI Info System Status] --> A\n B1[Prompt Construction LLM Call Response Parsing] --> B\n C1[Command Dispatch MCP Execution Result Handling] --> C\n D1[Memory Items Blackboard Update Context Sync] --> D\n```\n\n### Four Core Strategy Types\n\n| Strategy Type | ProcessingPhase | Purpose | Examples |\n|---------------|-----------------|---------|----------|\n| **DATA_COLLECTION** | `data_collection` | Gather contextual information | Screenshot capture, UI tree extraction, system info |\n| **LLM_INTERACTION** | `llm_interaction` | Construct prompts, interact with LLM, parse responses | Prompt building, LLM reasoning, JSON parsing |\n| **ACTION_EXECUTION** | `action_execution` | Execute commands from LLM/toolkits | Click, type, scroll, API calls |\n| **MEMORY_UPDATE** | `memory_update` | Update short-term/long-term memory | Add memory items, update blackboard, sync context |\n\n**Strategy Layer Configuration Example:**\n\nEach state configures its processor with strategies and middleware:\n\n```python\nclass AppAgentProcessor(ProcessorTemplate):\n def _setup_strategies(self):\n # Register strategies for each phase\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy([\n AppScreenshotCaptureStrategy(),\n AppControlInfoStrategy()\n ])\n self.strategies[ProcessingPhase.LLM_INTERACTION] = AppLLMInteractionStrategy()\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = AppActionExecutionStrategy()\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy()\n \n def _setup_middleware(self):\n # Add middleware for logging, metrics, error handling\n self.middleware_chain = [\n LoggingMiddleware(),\n PerformanceMetricsMiddleware(),\n ErrorHandlingMiddleware()\n ]\n```\n\n**See [Processor Documentation](design/processor.md) and [Strategy Documentation](design/strategy.md) for complete details.**\n\n---\n\n## Level-3: Command Layer (System Interface)\n\nThe **Command Layer** provides atomic, deterministic system operations. Each command maps to an **MCP tool** that executes on the device client.\n\n```mermaid\nsequenceDiagram\n participant Agent as Device Agent (Server)\n participant Dispatcher as CommandDispatcher\n participant Protocol as AIP Protocol\n participant Client as Device Client\n participant MCP as MCP Tool\n\n Agent->>Dispatcher: execute_commands([command1, command2])\n Dispatcher->>Protocol: Send ServerMessage (COMMAND)\n Protocol->>Client: WebSocket (AIP)\n Client->>MCP: Route to MCP server\n MCP->>MCP: Execute tool function\n MCP->>Client: Return result\n Client->>Protocol: Send ClientMessage (RESULT)\n Protocol->>Dispatcher: Receive results\n Dispatcher->>Agent: Return List[Result]\n```\n\n### Command Structure\n\n```python\n@dataclass\nclass Command:\n \"\"\"Atomic command to be executed on device client\"\"\"\n tool_name: str # MCP tool name (e.g., \"click_element\")\n parameters: Dict[str, Any] # Tool arguments\n tool_type: str # \"data_collection\" or \"action\"\n call_id: str # Unique identifier\n```\n\n!!! warning \"Deterministic Execution\"\n Commands are designed to be:\n \n - **Atomic**: Single, indivisible operation\n - **Deterministic**: Same inputs → same outputs\n - **Auditable**: Full command history logged\n - **Reversible**: Where possible, support undo operations\n\n**Extensibility:**\n\nNew commands can be added by:\n\n1. Registering MCP tool on device client\n2. LLM dynamically selects tool from available MCP registry\n3. No server-side code changes required\n\n**See [Command Layer Documentation](design/command.md) for complete details.**\n\n---\n\n## Server-Client Architecture\n\nDevice agents use a **server-client separation** to balance safety, scalability, and functionality:\n\n```mermaid\ngraph TB\n subgraph \"Server Side (UFO3 Orchestrator)\"\n Server[Device Agent Server]\n State[State Machine]\n Processor[Strategy Processor]\n LLM[LLM Service]\n Memory[Memory & Context]\n \n Server --> State\n Server --> Processor\n Server --> Memory\n Processor -.-> LLM\n end\n \n subgraph \"Communication Layer\"\n AIP[AIP Protocol WebSocket]\n end\n \n subgraph \"Client Side (Device)\"\n Client[Device Client]\n Dispatcher[Command Dispatcher]\n MCP[MCP Server Manager]\n Tools[MCP Tools]\n OS[Operating System]\n \n Client --> Dispatcher\n Dispatcher --> MCP\n MCP --> Tools\n Tools --> OS\n end\n \n Server <-->|Commands/Results| AIP\n AIP <-->|Commands/Results| Client\n```\n\n### Separation of Concerns\n\n| Component | Location | Responsibilities | Security Boundary |\n|-----------|----------|------------------|-------------------|\n| **Agent Server** | Orchestrator | State management, reasoning, planning, memory | Untrusted (LLM-driven decisions) |\n| **Device Client** | Device | Command execution, MCP tool calls, resource access | Trusted (validated operations) |\n| **AIP Protocol** | Communication | Message serialization, WebSocket transport, error handling | Secure channel (authentication, encryption) |\n\n**Why Server-Client Separation?**\n\n**Safety**: Isolates potentially unsafe LLM-generated decisions from direct system access. Clients validate all commands before execution.\n\n**Scalability**: Single orchestrator server manages multiple device clients. Reduces per-device resource requirements.\n\n**Flexibility**: Device clients can run on resource-constrained devices (embedded systems, mobile) while heavy reasoning occurs on server.\n\n**See [Server-Client Architecture](server_client_architecture.md) for complete details.**\n\n---\n\n## Supported Device Platforms\n\nUFO3 currently supports **Windows** and **Linux** device agents, with architecture designed for extensibility to other platforms.\n\n### Windows Agents\n\n```mermaid\ngraph TB\n subgraph \"Windows Device (Two-Tier Hierarchy)\"\n Host[HostAgent Application Selection]\n App1[AppAgent Word]\n App2[AppAgent Excel]\n App3[AppAgent Browser]\n \n Host -->|delegates| App1\n Host -->|delegates| App2\n Host -->|delegates| App3\n end\n \n User[User Request] --> Host\n```\n\n**HostAgent** (Application-Level Coordinator):\n- Selects appropriate application(s) for user request\n- Decomposes tasks into application-specific subtasks\n- Coordinates multiple AppAgents\n- Manages application switching and data transfer\n\n**AppAgent** (Application-Level Executor):\n- Controls specific Windows application (Word, Excel, browser, etc.)\n- Uses UI Automation for control element discovery\n- Executes application-specific actions (type, click, scroll)\n- Maintains application context and memory\n\n!!! example \"Windows Agent Example\"\n **User Request**: \"Create a chart from sales.xlsx and insert into report.docx\"\n \n 1. **HostAgent** decomposes: \n - Open Excel → Create chart → Copy chart\n - Open Word → Paste chart\n 2. **AppAgent (Excel)**: Opens `sales.xlsx`, creates chart, copies to clipboard\n 3. **AppAgent (Word)**: Opens `report.docx`, pastes chart at cursor\n\n### Linux Agent\n\n```mermaid\ngraph TB\n subgraph \"Linux Device (Single-Tier Architecture)\"\n Linux[LinuxAgent Direct System Control]\n Shell[Shell Commands]\n Files[File Operations]\n Apps[Application Launch]\n \n Linux --> Shell\n Linux --> Files\n Linux --> Apps\n end\n \n User[User Request] --> Linux\n```\n\n**LinuxAgent** (System-Level Executor):\n- Direct shell command execution\n- File system operations\n- Application launch and management\n- Single-tier architecture (no application-level hierarchy)\n\n!!! info \"Architecture Difference\"\n **Windows** uses two-tier hierarchy (HostAgent → AppAgent) due to:\n \n - UI Automation framework's application-centric model\n - Distinct application contexts requiring specialized agents\n \n **Linux** uses single-tier architecture because:\n \n - Shell provides unified interface to all system operations\n - Application control occurs through same command-line interface\n\n### Platform Comparison\n\n| Feature | Windows (UFO2) | Linux | macOS (Future) | Embedded (Future) |\n|---------|----------------|-------|----------------|-------------------|\n| **Agent Hierarchy** | Two-tier (Host → App) | Single-tier | TBD | Single-tier |\n| **UI Control** | UI Automation | X11/Wayland | Accessibility API | Platform-specific |\n| **Command Interface** | MCP tools (Win32 API) | MCP tools (Shell) | MCP tools (AppleScript) | MCP tools (Custom) |\n| **Observation** | Screenshot + UI tree | Screenshot + Shell output | Screenshot + UI tree | Sensor data |\n| **State Management** | Shared FSM | Shared FSM | Shared FSM | Shared FSM |\n| **Strategy Layer** | Processor framework | Processor framework | Processor framework | Processor framework |\n| **Current Status** | ✅ Production | ✅ Production | 🔜 Planned | 🔜 Planned |\n\n**Extensibility Path:**\n\nAdding a new platform requires:\n\n1. **Implement Agent Class**: Extend `BasicAgent` (inherit State layer, Processor framework)\n2. **Create Processor**: Subclass `ProcessorTemplate`, implement platform-specific strategies\n3. **Define MCP Tools**: Register platform-specific MCP tools on device client\n4. **Register Agent**: Use `@AgentRegistry.register` decorator\n\nNo changes to core State layer, Processor framework, or AIP protocol required.\n\n**See [Agent Types Documentation](agent_types.md) for complete implementation details.**\n\n---\n\n## Agent Lifecycle\n\nA typical device agent execution follows this lifecycle:\n\n```mermaid\nsequenceDiagram\n participant User\n participant Orchestrator\n participant Agent\n participant State\n participant Processor\n participant LLM\n participant Dispatcher\n participant Client\n\n User->>Orchestrator: Submit task\n Orchestrator->>Agent: Initialize agent (CONTINUE state)\n \n loop Until FINISH/FAIL/ERROR\n Agent->>State: handle(agent, context)\n State->>Processor: execute strategies\n \n Processor->>Processor: DATA_COLLECTION\n Note over Processor: Screenshot, UI info\n \n Processor->>LLM: LLM_INTERACTION\n LLM-->>Processor: Action decision\n \n Processor->>Dispatcher: ACTION_EXECUTION\n Dispatcher->>Client: Execute commands\n Client-->>Dispatcher: Results\n Dispatcher-->>Processor: Results\n \n Processor->>Processor: MEMORY_UPDATE\n Note over Processor: Update memory, blackboard\n \n State->>State: next_state(agent)\n State->>Agent: Update agent status\n end\n \n Agent->>Orchestrator: Task complete/failed\n Orchestrator->>User: Return result\n```\n\n### Execution Phases\n\n1. **Initialization**: Agent created with default state (`CONTINUE`), processor, memory\n2. **State Handling**: Current state's `handle()` method invoked with agent and context\n3. **Strategy Execution**: Processor runs strategies in sequence (DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE)\n4. **State Transition**: State's `next_state()` determines next FSM state\n5. **Repeat/Terminate**: Loop continues until terminal state (`FINISH`, `FAIL`, `ERROR`)\n\n!!! tip \"Multi-Agent Handoff\"\n For multi-agent scenarios (e.g., Windows HostAgent → AppAgent), states implement `next_agent()`:\n \n ```python\n def next_agent(self, agent: BasicAgent) -> BasicAgent:\n # HostAgent delegates to AppAgent\n if agent.status == \"DELEGATE_TO_APP\":\n return agent.create_app_agent(...)\n return agent\n ```\n\n---\n\n## Memory and Context Management\n\nDevice agents maintain two types of memory:\n\n### Short-Term Memory (Agent Memory)\n\n**Purpose**: Track agent's execution history within a session\n\n**Implementation**: `Memory` class with `MemoryItem` entries\n\n```python\nclass Memory:\n \"\"\"Agent's short-term memory\"\"\"\n _content: List[MemoryItem]\n \n def add_memory_item(self, memory_item: MemoryItem):\n \"\"\"Add new memory entry\"\"\"\n self._content.append(memory_item)\n```\n\n**Content**: Actions taken, observations made, results received\n\n**Lifetime**: Single session (cleared between tasks)\n\n### Long-Term Memory (Blackboard)\n\n**Purpose**: Share information across agents and sessions\n\n**Implementation**: `Blackboard` class with multiple memory types\n\n```python\nclass Blackboard:\n \"\"\"Multi-agent shared memory\"\"\"\n _questions: Memory # Q&A history\n _requests: Memory # Request history\n _trajectories: Memory # Action trajectories\n _screenshots: Memory # Visual observations\n```\n\n**Content**: Common knowledge, successful action patterns, user preferences\n\n**Lifetime**: Persistent across sessions (can be saved/loaded)\n\n**Blackboard Usage Example:**\n\n**Scenario**: HostAgent delegates to AppAgent (Excel)\n\n1. HostAgent adds to blackboard:\n - Request: \"Create sales chart\"\n - Context: Previous analysis results\n2. AppAgent reads from blackboard:\n - Retrieves request and context\n - Adds action trajectories as executed\n - Adds screenshot after chart creation\n3. HostAgent reads updated blackboard:\n - Verifies chart creation\n - Continues to next step (insert to Word)\n\n**See [Memory System Documentation](design/memory.md) for complete details.**\n\n---\n\n## Integration with UFO3 Components\n\nDevice agents integrate with other UFO3 components:\n\n```mermaid\ngraph TB\n subgraph \"UFO3 Architecture\"\n Session[Session/Round Manager]\n Context[Global Context]\n Agent[Device Agent]\n Dispatcher[Command Dispatcher]\n AIP[AIP Protocol]\n Client[Device Client]\n MCP[MCP Servers]\n end\n \n Session -->|manages lifecycle| Agent\n Session -->|provides| Context\n Agent -->|reads/writes| Context\n Agent -->|sends commands| Dispatcher\n Dispatcher -->|uses| AIP\n AIP <-->|WebSocket| Client\n Client -->|calls| MCP\n```\n\n### Integration Points\n\n| Component | Relationship | Description |\n|-----------|--------------|-------------|\n| **Session Manager** | Parent | Creates agents, manages agent lifecycle, coordinates multi-agent workflows |\n| **Round Manager** | Sibling | Manages round-based execution, tracks round state, synchronizes with agent steps |\n| **Global Context** | Shared State | Agent reads request/config, writes results/status, shares data across components |\n| **Command Dispatcher** | Execution Interface | Agent sends commands, dispatcher routes to client, returns results |\n| **AIP Protocol** | Communication | Serializes commands/results, manages WebSocket, handles errors/timeouts |\n| **Device Client** | Executor | Receives commands, invokes MCP tools, returns results |\n| **MCP Servers** | Tool Registry | Provides available tools, executes tool functions, returns structured results |\n\n**See [Session Documentation](../modules/session.md), [Context Documentation](../modules/context.md), and [AIP Protocol](../../aip/overview.md) for integration details.**\n\n---\n\n## Design Patterns\n\nDevice agent architecture leverages several design patterns:\n\n### 1. State Pattern (FSM Layer)\n\n**Purpose**: Encapsulate state-specific behavior, enable dynamic state transitions\n\n**Implementation**: `AgentState` abstract class, concrete state classes\n\n```python\nclass AgentState(ABC):\n @abstractmethod\n async def handle(self, agent, context):\n \"\"\"Execute state-specific logic\"\"\"\n pass\n \n @abstractmethod\n def next_state(self, agent):\n \"\"\"Determine next state\"\"\"\n pass\n```\n\n### 2. Strategy Pattern (Strategy Layer)\n\n**Purpose**: Define family of algorithms (strategies), make them interchangeable\n\n**Implementation**: `ProcessingStrategy` protocol, concrete strategy classes\n\n```python\nclass ProcessingStrategy(Protocol):\n async def execute(self, agent, context) -> ProcessingResult:\n \"\"\"Execute strategy logic\"\"\"\n pass\n```\n\n### 3. Template Method Pattern (Processor Framework)\n\n**Purpose**: Define skeleton of algorithm, let subclasses override specific steps\n\n**Implementation**: `ProcessorTemplate` abstract class\n\n```python\nclass ProcessorTemplate(ABC):\n @abstractmethod\n def _setup_strategies(self):\n \"\"\"Subclass defines which strategies to use\"\"\"\n pass\n \n async def process(self, agent, context):\n \"\"\"Template method - runs strategies in sequence\"\"\"\n for phase, strategy in self.strategies.items():\n result = await strategy.execute(agent, context)\n # Handle result, update context\n```\n\n### 4. Singleton Pattern (State Manager)\n\n**Purpose**: Ensure single instance of state registry\n\n**Implementation**: `AgentStateManager` with metaclass\n\n```python\nclass AgentStateManager(ABC, metaclass=SingletonABCMeta):\n _state_mapping: Dict[str, Type[AgentState]] = {}\n \n def get_state(self, status: str) -> AgentState:\n \"\"\"Lazy load and return state instance\"\"\"\n pass\n```\n\n### 5. Registry Pattern (Agent Registration)\n\n**Purpose**: Register agent types, enable dynamic agent creation\n\n**Implementation**: `AgentRegistry` decorator\n\n```python\n@AgentRegistry.register(agent_name=\"appagent\", processor_cls=AppAgentProcessor)\nclass AppAgent(BasicAgent):\n pass\n```\n\n### 6. Blackboard Pattern (Multi-Agent Coordination)\n\n**Purpose**: Share data across multiple agents\n\n**Implementation**: `Blackboard` class\n\n```python\nclass Blackboard:\n _questions: Memory\n _requests: Memory\n _trajectories: Memory\n _screenshots: Memory\n```\n\n---\n\n## Best Practices\n\n### State Design\n\n- Keep states **focused**: Each state should have single, clear responsibility\n- Use **rule-based transitions** for deterministic flows, **LLM-driven transitions** for adaptive behavior\n- Implement **error states** for graceful degradation\n- Document **state invariants** and **transition conditions**\n\n### Strategy Design\n\n- Keep strategies **atomic**: Each strategy should perform one cohesive task\n- Declare **dependencies explicitly** using `get_dependencies()`\n- Use **ComposedStrategy** to combine multiple strategies within a phase\n- Implement **fail-fast** for critical errors, **continue-on-error** for optional operations\n\n### Command Design\n\n- Keep commands **atomic**: Single, indivisible operation\n- Design commands to be **idempotent** where possible\n- Validate **arguments** on client side before execution\n- Return **structured results** with success/failure status\n\n### Memory Management\n\n- Use **short-term memory** for agent-specific execution history\n- Use **blackboard** for multi-agent coordination and persistent knowledge\n- **Clear memory** between sessions to avoid context pollution\n- Implement **memory pruning** for long-running sessions\n\n!!! warning \"Security Considerations\"\n - **Validate all commands** on client side before execution\n - **Sanitize LLM outputs** before converting to commands\n - **Limit command scope** via MCP tool permissions\n - **Audit all actions** for compliance and debugging\n - **Isolate agents** to prevent unauthorized cross-agent access\n\n---\n\n## Related Documentation\n\n**Deep Dive Into Layers:**\n\n- [State Layer Documentation](design/state.md): FSM, AgentState, transitions, state registration\n- [Processor and Strategy Documentation](design/processor.md): ProcessorTemplate, strategies, dependency management\n- [Command Layer Documentation](design/command.md): CommandDispatcher, MCP integration, atomic commands\n\n**Supporting Systems:**\n\n- [Memory System Documentation](design/memory.md): Memory, MemoryItem, Blackboard patterns\n- [Agent Types Documentation](agent_types.md): Windows agents, Linux agent, platform-specific implementations\n\n**Integration Points:**\n\n- [Server-Client Architecture](server_client_architecture.md): Server and client separation, communication patterns\n- [Server Architecture](../../server/overview.md): Agent server, WebSocket manager, orchestration\n- [Client Architecture](../../client/overview.md): Device client, MCP servers, command execution\n- [AIP Protocol](../../aip/overview.md): Agent Interaction Protocol for server-client communication\n- [MCP Integration](../../mcp/overview.md): Model Context Protocol for tool execution\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n✅ **Three-Layer Architecture**: State (FSM) → Strategy (Execution Logic) → Command (System Interface)\n\n✅ **Server-Client Separation**: Safe isolation of reasoning (server) from execution (client)\n\n✅ **Multi-Platform Support**: Windows (two-tier), Linux (single-tier), extensible to macOS and embedded\n\n✅ **LLM-Driven Reasoning**: Dynamic decision-making with structured command output\n\n✅ **Modular & Extensible**: Register new states, compose strategies, add MCP tools without core changes\n\n✅ **Memory Systems**: Short-term (agent memory) and long-term (blackboard) for coordination\n\n✅ **Design Patterns**: State, Strategy, Template Method, Singleton, Registry, Blackboard\n\nThe Device Agent architecture provides a **robust, extensible foundation** for multi-device automation. By separating concerns across three layers and isolating reasoning from execution, UFO3 achieves both **safety** and **flexibility** for orchestrating complex cross-device workflows.\n\n---\n\n## Reference\n\nBelow is the reference for the `BasicAgent` class. All device agents inherit from `BasicAgent` and implement platform-specific processors and states:\n\n::: agents.agent.basic.BasicAgent\n\n"
},
{
"path": "documents/docs/infrastructure/agents/server_client_architecture.md",
"content": "# Server-Client Architecture\n\nDevice agents in UFO are partitioned into **server** and **client** components, separating high-level orchestration from low-level execution. This architecture enables safe, scalable, and flexible task execution across heterogeneous devices through the Agent Interaction Protocol (AIP).\n\n---\n\n## Overview\n\nTo support safe, scalable, and flexible execution across heterogeneous devices, each **device agent** is partitioned into two distinct components: a **server** and a **client**. This separation of responsibilities aligns naturally with the [layered FSM architecture](./overview.md#three-layer-architecture) and leverages [AIP](../../aip/overview.md) for reliable, low-latency communication.\n\n\n \n The server-client architecture of a device agent. The server handles orchestration, state management, and LLM-driven decision-making, while the client executes commands through MCP tools and reports results back.\n\n\n### Architecture Benefits\n\n| Benefit | Description |\n|---------|-------------|\n| **🔒 Safe Execution** | Separates reasoning (server) from system operations (client), reducing risk |\n| **📈 Scalable Orchestration** | Single server can manage multiple clients concurrently |\n| **🔧 Independent Updates** | Server logic and client tools can be updated independently |\n| **🌐 Multi-Device Support** | Clients can be rapidly deployed on new devices with minimal configuration |\n| **🛡️ Fault Isolation** | Client failures don't crash the server's reasoning logic |\n| **📡 Real-Time Communication** | Persistent WebSocket connections enable low-latency bidirectional messaging |\n\n**Design Philosophy:**\n\nThe server-client architecture embodies the **separation of concerns** principle: the server focuses on **what** to do (strategy), while the client focuses on **how** to do it (execution). This clear division enhances maintainability, security, and scalability.\n\n---\n\n## Server: Orchestration and State Management\n\nThe **agent server** is responsible for managing the agent's state machine lifecycle, executing high-level strategies, and interacting with the Constellation Agent or orchestrator. It handles task decomposition, prompt construction, decision-making, and command sequencing.\n\n**Server Responsibilities:**\n\n- 🧠 **State Machine Management**: Controls agent lifecycle through the [FSM](./overview.md#level-1-state-layer-fsm)\n- 🎯 **Strategy Execution**: Implements the [Strategy Layer](./overview.md#level-2-strategy-layer-execution-logic)\n- 🤖 **LLM Interaction**: Constructs prompts, parses responses, makes decisions\n- 📋 **Task Decomposition**: Breaks down high-level tasks into executable commands\n- 🔀 **Command Sequencing**: Determines execution order and dependencies\n- 👥 **Multi-Client Coordination**: Manages multiple device clients concurrently\n\n### Server Architecture\n\n```mermaid\ngraph TB\n subgraph \"Agent Server\"\n subgraph \"State Layer\"\n FSM[Finite State Machine]\n SM[State Manager]\n end\n \n subgraph \"Strategy Layer\"\n PROC[ProcessorTemplate]\n LLM[LLM Interaction]\n CMD[Command Generation]\n end\n \n subgraph \"Communication Layer\"\n WS[WebSocket Handler]\n AIP_S[AIP Protocol]\n end\n \n subgraph \"Metadata\"\n PROFILE[AgentProfile]\n CAP[Capabilities]\n STATUS[Runtime Status]\n end\n \n FSM --> SM\n SM --> PROC\n PROC --> LLM\n PROC --> CMD\n CMD --> WS\n WS --> AIP_S\n \n PROFILE --> CAP\n PROFILE --> STATUS\n end\n \n subgraph \"External Interfaces\"\n ORCHESTRATOR[Constellation Agent/ Orchestrator]\n CLIENTS[Multiple Device Clients]\n end\n \n ORCHESTRATOR <-->|Task Assignment| FSM\n ORCHESTRATOR <-->|Profile Query| PROFILE\n AIP_S <-->|Commands/Results| CLIENTS\n \n style FSM fill:#e1f5ff\n style PROC fill:#fff4e1\n style WS fill:#ffe1f5\n style PROFILE fill:#f0ffe1\n```\n\n### AgentProfile\n\nEach server instance exposes its capabilities and status through metadata. This information allows the orchestrator to dynamically select suitable agents for specific subtasks, improving task distribution efficiency.\n\nNote: The AgentProfile concept is part of the design for multi-agent coordination in Galaxy (constellation-level orchestration). In UFO3's current implementation, agent metadata is managed through the session context and WebSocket handler registration.\n\n### Multi-Client Management\n\nA **single server instance** can manage **multiple agent clients concurrently**, maintaining isolation across devices while supporting centralized supervision and coordination.\n\n```mermaid\nsequenceDiagram\n participant O as Orchestrator\n participant S as Agent Server\n participant C1 as Client 1 (Desktop)\n participant C2 as Client 2 (Laptop)\n participant C3 as Client 3 (Server)\n \n Note over S: Server manages multiple clients\n \n C1->>S: Connect & Register (Device Info)\n C2->>S: Connect & Register (Device Info)\n C3->>S: Connect & Register (Device Info)\n \n S->>S: Update AgentProfile (3 clients available)\n \n O->>S: Query AgentProfile\n S->>O: Profile (3 devices, capabilities)\n \n O->>S: Assign Task (requires GPU)\n S->>S: Select Client 1 (has GPU)\n S->>C1: Execute Command\n C1->>S: Result\n S->>O: Task Complete\n \n Note over S,C1: Client 2 & 3 remain available\n```\n\n**Benefits of centralized server management:**\n\n- **Session Isolation**: Each client maintains independent state\n- **Load Balancing**: Server distributes tasks across available clients\n- **Fault Tolerance**: Client failures don't affect other clients\n- **Unified Monitoring**: Centralized view of all client statuses\n\n### Server Flexibility\n\nCrucially, the server maintains **full control** over the agent's workflow logic, enabling **updates to decision strategies** without impacting low-level execution on the device.\n\n**Update Scenarios:**\n\n- **Prompt Engineering**: Modify LLM prompts to improve decision quality\n- **Strategy Changes**: Switch between different processing strategies\n- **State Transitions**: Adjust FSM logic for new workflows\n- **API Integration**: Add new orchestrator interfaces\n\nAll these updates happen **server-side only**, without redeploying clients.\n\nFor detailed server implementation, see the [Server Documentation](../../server/overview.md).\n\n---\n\n## Client: Command Execution and Resource Access\n\nThe **agent client** runs on the target device and manages a collection of MCP servers or tool interfaces. These MCP servers can operate locally (via direct invocation) or remotely (through HTTP requests), and each client may register multiple MCP servers to access diverse tool sources.\n\n**Client Responsibilities:**\n\n- ⚙️ **Command Execution**: Translates server commands into MCP tool calls\n- 🛠️ **Tool Management**: Registers and orchestrates local/remote MCP servers\n- 📊 **Device Profiling**: Reports hardware and software configuration\n- 📡 **Result Reporting**: Returns structured execution results via AIP\n- 🔍 **Self-Checks**: Performs diagnostics (disk, CPU, memory, GPU, network)\n- 🚫 **Stateless Operation**: Executes directives without high-level reasoning\n\n### Client Architecture\n\n```mermaid\ngraph TB\n subgraph \"Agent Client\"\n subgraph \"Communication Layer\"\n WS_C[WebSocket Client]\n AIP_C[AIP Protocol Handler]\n end\n \n subgraph \"Orchestration Layer\"\n UFC[UFO Client]\n CM[Computer Manager]\n end\n \n subgraph \"Execution Layer\"\n COMP[Computer Instance]\n DISP[Command Dispatcher]\n end\n \n subgraph \"Tool Layer\"\n MCP_MGR[MCP Server Manager]\n LOCAL_MCP[Local MCP Servers]\n REMOTE_MCP[Remote MCP Servers]\n end\n \n subgraph \"Device Layer\"\n TOOLS[System Tools]\n HW[Hardware Access]\n FS[File System]\n UI[UI Automation]\n end\n \n WS_C --> AIP_C\n AIP_C --> UFC\n UFC --> CM\n CM --> COMP\n COMP --> DISP\n DISP --> MCP_MGR\n MCP_MGR --> LOCAL_MCP\n MCP_MGR --> REMOTE_MCP\n LOCAL_MCP --> TOOLS\n REMOTE_MCP -.->|HTTP| TOOLS\n TOOLS --> HW\n TOOLS --> FS\n TOOLS --> UI\n end\n \n subgraph \"Agent Server\"\n SERVER[Server Process]\n end\n \n SERVER <-->|AIP over WebSocket| WS_C\n \n style WS_C fill:#e1f5ff\n style COMP fill:#fff4e1\n style MCP_MGR fill:#ffe1f5\n style TOOLS fill:#f0ffe1\n```\n\n### Command Execution Pipeline\n\nUpon receiving commands from the agent server—such as collecting telemetry, invoking system utilities, or interacting with hardware components—the client follows this execution pipeline:\n\n```mermaid\nsequenceDiagram\n participant S as Agent Server\n participant C as Agent Client\n participant D as Dispatcher\n participant M as MCP Manager\n participant T as MCP Tool\n \n S->>C: Command via AIP (function, parameters)\n C->>D: Parse command\n D->>M: Resolve MCP tool\n M->>M: Select server (local/remote)\n M->>T: Invoke tool\n T->>T: Execute operation\n T->>M: Raw result\n M->>D: Structured output\n D->>C: Aggregate results\n C->>S: Result via AIP (status, data)\n```\n\n**Pipeline stages:**\n\n1. **Command Reception**: Client receives AIP message with command metadata\n2. **Parsing**: Extract function name and parameters\n3. **Tool Resolution**: Map command to registered MCP tool\n4. **Server Selection**: Choose local or remote MCP server\n5. **Execution**: Invoke tool deterministically\n6. **Result Aggregation**: Structure output according to schema\n7. **Response Transmission**: Return results via AIP\n\n### MCP Server Management\n\nEach client may **register multiple MCP servers** to access diverse tool sources. MCP servers provide standardized interfaces for:\n\n| Tool Category | Examples | Local/Remote |\n|---------------|----------|--------------|\n| **UI Automation** | Click, type, screenshot, select controls | Local |\n| **File Operations** | Read, write, copy, delete files | Local |\n| **System Utilities** | Process management, network config | Local |\n| **Application APIs** | Excel, Word, Browser automation | Local |\n| **Remote Services** | Cloud APIs, external databases | Remote (HTTP) |\n| **Hardware Control** | Camera, sensors, GPIO | Local |\n\n```python\n# Example: Client registers multiple MCP servers\nclient.register_mcp_server(\n name=\"ui_automation\",\n type=\"local\",\n tools=[\"click\", \"type\", \"screenshot\"]\n)\n\nclient.register_mcp_server(\n name=\"file_operations\",\n type=\"local\",\n tools=[\"read_file\", \"write_file\", \"list_dir\"]\n)\n\nclient.register_mcp_server(\n name=\"cloud_api\",\n type=\"remote\",\n endpoint=\"https://api.example.com/mcp\",\n tools=[\"query_database\", \"send_notification\"]\n)\n```\n\nFor detailed MCP integration, see [MCP Integration](../../client/mcp_integration.md).\n\n### Device Initialization and Registration\n\nDuring initialization, each client connects to the agent server through the AIP endpoint, performs **self-checks**, and **registers its hardware-software profile**.\n\n```mermaid\nsequenceDiagram\n participant C as Agent Client\n participant S as Agent Server\n \n Note over C: Client Startup\n \n C->>C: Load configuration\n C->>C: Initialize MCP servers\n \n C->>C: Self-Check: - Disk space - CPU info - Memory - GPU availability - Network config\n \n C->>S: Connect (WebSocket)\n S->>C: Connection Acknowledged\n \n C->>S: Register Device Info (hardware profile)\n S->>S: Update AgentProfile\n S->>C: Registration Confirmed\n \n Note over C,S: Ready for task execution\n```\n\n**Self-checks performed during initialization:**\n\n```python\ndevice_info = {\n # Hardware\n \"cpu\": {\n \"model\": \"Intel Core i7-12700K\",\n \"cores\": 12,\n \"threads\": 20,\n \"frequency_mhz\": 3600\n },\n \"memory\": {\n \"total_gb\": 32,\n \"available_gb\": 24\n },\n \"disk\": {\n \"total_gb\": 1024,\n \"free_gb\": 512\n },\n \"gpu\": {\n \"available\": True,\n \"model\": \"NVIDIA RTX 4090\",\n \"vram_gb\": 24\n },\n \n # Network\n \"network\": {\n \"hostname\": \"desktop-001\",\n \"ip_address\": \"192.168.1.100\",\n \"bandwidth_mbps\": 1000\n },\n \n # Software\n \"os\": {\n \"platform\": \"windows\",\n \"version\": \"11\",\n \"build\": \"22621\"\n },\n \"installed_apps\": [\n \"Microsoft Excel\",\n \"Google Chrome\",\n \"Visual Studio Code\"\n ],\n \"mcp_servers\": [\n \"ui_automation\",\n \"file_operations\",\n \"system_utilities\"\n ]\n}\n```\n\nThis profile is integrated into the server's **AgentProfile**, giving the orchestrator **complete visibility** into system topology and resource availability for informed task assignment and scheduling.\n\nFor client implementation details, see the [Client Documentation](../../client/overview.md).\n\n### Stateless Design\n\nThe client remains **stateless with respect to reasoning**: it faithfully executes directives without engaging in high-level decision-making.\n\n**Client Does NOT:**\n\n- ❌ Construct prompts for LLMs\n- ❌ Make strategic decisions\n- ❌ Manage state transitions\n- ❌ Decompose tasks into subtasks\n- ❌ Coordinate with other agents\n\n**Client DOES:**\n\n- ✅ Execute commands deterministically\n- ✅ Manage MCP tool lifecycle\n- ✅ Report execution results\n- ✅ Monitor device health\n- ✅ Handle tool failures gracefully\n\nThis separation ensures that **updates to one layer do not interfere with the other**, enhancing maintainability and reducing risk of disruption.\n\n---\n\n## Server-Client Communication\n\nAll communication between the server and client is routed through the **Agent Interaction Protocol (AIP)**, leveraging **persistent WebSocket connections**. This allows bidirectional, low-latency messaging that supports both synchronous command execution and asynchronous event reporting.\n\n**Why AIP over WebSocket?**\n\n- **Low Latency**: Real-time command dispatch and result streaming\n- **Bidirectional**: Server sends commands, client sends results/events\n- **Persistent**: Maintains connection across multiple commands\n- **Event-Driven**: Supports async notifications (progress updates, errors)\n- **Protocol Abstraction**: Hides network complexity from application logic\n\n### Communication Patterns\n\n#### 1. Synchronous Command Execution\n\n```mermaid\nsequenceDiagram\n participant S as Server\n participant C as Client\n \n S->>C: Command (request_id=123) function: screenshot\n \n Note over C: Execute tool\n \n C->>S: Result (request_id=123) status: success data: image_base64\n```\n\n**Flow:**\n1. Server sends command with unique `request_id`\n2. Client executes MCP tool synchronously\n3. Client returns result with matching `request_id`\n4. Server matches result to pending request\n\n#### 2. Asynchronous Event Reporting\n\n```mermaid\nsequenceDiagram\n participant S as Server\n participant C as Client\n \n S->>C: Command: long_running_task\n \n C->>S: Event: progress (25%)\n C->>S: Event: progress (50%)\n C->>S: Event: progress (75%)\n C->>S: Result: complete (100%)\n```\n\n**Use cases:**\n- Progress updates for long-running operations\n- Error notifications during execution\n- Resource utilization alerts\n- Device state changes\n\n#### 3. Multi-Command Pipeline\n\n```mermaid\nsequenceDiagram\n participant S as Server\n participant C as Client\n \n S->>C: Command 1: screenshot\n S->>C: Command 2: click(x, y)\n S->>C: Command 3: screenshot\n \n Note over C: Execute in order\n \n C->>S: Result 1: image_before\n C->>S: Result 2: click_success\n C->>S: Result 3: image_after\n```\n\n**Benefits:**\n- Reduces round-trip latency\n- Enables atomic operation sequences\n- Supports transaction-like semantics\n\n### AIP Message Format\n\nCommands and results follow the AIP message schema:\n\n```json\n{\n \"type\": \"command\",\n \"request_id\": \"abc-123\",\n \"timestamp\": \"2025-11-06T10:30:00Z\",\n \"payload\": {\n \"function\": \"screenshot\",\n \"arguments\": {\n \"region\": \"active_window\"\n }\n }\n}\n```\n\n```json\n{\n \"type\": \"result\",\n \"request_id\": \"abc-123\",\n \"timestamp\": \"2025-11-06T10:30:01Z\",\n \"payload\": {\n \"status\": \"success\",\n \"data\": {\n \"image\": \"base64_encoded_data\",\n \"dimensions\": {\"width\": 1920, \"height\": 1080}\n }\n }\n}\n```\n\nFor complete AIP specification, see [AIP Documentation](../../aip/overview.md).\n\n### Connection Management\n\nThe server and client maintain persistent connections with automatic reconnection logic:\n\n```mermaid\nstateDiagram-v2\n [*] --> Disconnected\n Disconnected --> Connecting: Client Start\n Connecting --> Connected: Handshake Success\n Connected --> Disconnected: Network Error\n Connected --> Reconnecting: Connection Lost\n Reconnecting --> Connected: Reconnect Success\n Reconnecting --> Disconnected: Max Retries Exceeded\n Connected --> [*]: Shutdown\n```\n\n**Connection lifecycle:**\n\n1. **Initial Connection**: Client initiates WebSocket connection to server\n2. **Registration**: Client sends device info, receives confirmation\n3. **Active Communication**: Bidirectional message exchange\n4. **Heartbeat**: Periodic pings to detect connection loss\n5. **Reconnection**: Automatic retry with exponential backoff\n6. **Graceful Shutdown**: Clean disconnection on exit\n\n**Resilience features:**\n\n- **Heartbeat Monitoring**: Detects silent connection failures\n- **Automatic Reconnection**: Exponential backoff with jitter\n- **Message Queuing**: Buffers messages during disconnection\n- **Session Recovery**: Restores context after reconnection\n\n---\n\n## Design Considerations\n\nThis server-client architecture offers several key advantages:\n\n### 1. Rapid Device Deployment\n\nDevice clients can be **rapidly deployed** on new devices with minimal configuration, immediately becoming execution endpoints within UFO.\n\n```bash\n# Deploy client on new device (example)\n# 1. Install client package\npip install ufo-client\n\n# 2. Configure server endpoint\ncat > client_config.yaml < Dict[str, Any]:\n \"\"\"Execute command on remote client.\"\"\"\n \n # Create command message\n message = {\n \"type\": \"command\",\n \"request_id\": generate_request_id(),\n \"payload\": {\n \"function\": command,\n \"arguments\": arguments\n }\n }\n \n # Send via AIP\n result = await server.send_command(client_id, message)\n \n return result\n```\n\n### Client: Executing Commands\n\n```python\n# Client receives and executes command\nasync def handle_command(\n client: AgentClient,\n command_message: Dict[str, Any]\n) -> Dict[str, Any]:\n \"\"\"Handle incoming command from server.\"\"\"\n \n # Extract command details\n function = command_message[\"payload\"][\"function\"]\n arguments = command_message[\"payload\"][\"arguments\"]\n request_id = command_message[\"request_id\"]\n \n try:\n # Execute via MCP tool\n result = await client.computer.execute_tool(\n tool_name=function,\n parameters=arguments\n )\n \n # Return success result\n return {\n \"type\": \"result\",\n \"request_id\": request_id,\n \"payload\": {\n \"status\": \"success\",\n \"data\": result\n }\n }\n \n except Exception as e:\n # Return error result\n return {\n \"type\": \"result\",\n \"request_id\": request_id,\n \"payload\": {\n \"status\": \"error\",\n \"error\": str(e)\n }\n }\n```\n\n---\n\n## Summary\n\nThe server-client architecture is a foundational design pattern in UFO's distributed agent system:\n\n**Key Takeaways:**\n\n- 🏗️ **Separation of Concerns**: Server handles reasoning, client handles execution\n- 📡 **AIP Communication**: Persistent WebSocket connections enable real-time bidirectional messaging\n- 🔧 **Independent Updates**: Server logic and client tools evolve independently\n- 📈 **Scalable Management**: Single server orchestrates multiple clients\n- 🛡️ **Fault Isolation**: Client failures don't crash server reasoning\n- 🌐 **Multi-Device Ready**: Supports heterogeneous device orchestration\n\n**Related Documentation:**\n\n- [Device Agent Overview](overview.md) - Three-layer FSM framework\n- [Agent Types](agent_types.md) - Platform-specific implementations\n- [Server Overview](../../server/overview.md) - Detailed server architecture and APIs\n- [Client Overview](../../client/overview.md) - Detailed client architecture and tools\n- [AIP Protocol](../../aip/overview.md) - Communication protocol specification\n- [MCP Integration](../../mcp/overview.md) - Tool management and execution\n\nBy decoupling high-level reasoning from low-level execution, the server-client architecture enables UFO to safely orchestrate complex workflows across diverse computing environments while maintaining flexibility, reliability, and ease of maintenance.\n"
},
{
"path": "documents/docs/infrastructure/modules/context.md",
"content": "# Context\n\nThe **Context** object is a type-safe shared state container that persists conversation state across all Rounds within a Session, providing centralized access to logs, costs, application state, and execution metadata.\n\n**Quick Reference:**\n\n- Get value? `context.get(ContextNames.REQUEST)`\n- Set value? `context.set(ContextNames.REQUEST, \"new value\")`\n- Auto-sync? See [Auto-Syncing Properties](#auto-syncing-properties)\n- All attributes? See [Complete Attribute Reference](#complete-attribute-reference)\n\n---\n\n## Overview\n\nThe `Context` object serves as the central state store for sessions:\n\n1. **Type Safety**: Enum-based attribute names with type definitions\n2. **Default Values**: Automatic initialization with sensible defaults\n3. **Auto-Syncing**: Current round values sync automatically\n4. **Serialization**: Convert to/from dict for persistence\n5. **Dispatcher Attachment**: Command execution integration\n\n### Architecture\n\n```mermaid\ngraph TB\n subgraph \"Context Container\"\n CTX[Context Dataclass]\n VALUES[Attribute Values Dict]\n NAMES[ContextNames Enum]\n end\n \n subgraph \"Access Patterns\"\n GET[get method]\n SET[set method]\n UPDATE[update_dict method]\n TO_DICT[to_dict method]\n FROM_DICT[from_dict method]\n end\n \n subgraph \"Auto-Sync Properties\"\n PROP_STEP[current_round_step]\n PROP_COST[current_round_cost]\n PROP_SUBTASK[current_round_subtask_amount]\n end\n \n subgraph \"Shared Across\"\n SESS[Session]\n R1[Round 1]\n R2[Round 2]\n R3[Round 3]\n AGENTHost[HostAgent]\n AGENTApp[AppAgent]\n end\n \n CTX --> VALUES\n NAMES --> VALUES\n \n GET --> VALUES\n SET --> VALUES\n UPDATE --> VALUES\n \n PROP_STEP -.auto-updates.-> VALUES\n PROP_COST -.auto-updates.-> VALUES\n PROP_SUBTASK -.auto-updates.-> VALUES\n \n SESS -.shares.-> CTX\n R1 -.shares.-> CTX\n R2 -.shares.-> CTX\n R3 -.shares.-> CTX\n AGENTHost -.reads/writes.-> CTX\n AGENTApp -.reads/writes.-> CTX\n \n style CTX fill:#e1f5ff\n style VALUES fill:#fff4e1\n style PROP_STEP fill:#f0ffe1\n style SESS fill:#ffe1f5\n```\n\n---\n\n## ContextNames Enum\n\nAll context attributes are defined in the `ContextNames` enum for type safety:\n\n```python\nfrom ufo.module.context import ContextNames\n\n# Type-safe attribute names\nrequest = context.get(ContextNames.REQUEST)\ncontext.set(ContextNames.SESSION_COST, 0.42)\n```\n\n### Attribute Categories\n\n!!!info \"30+ Context Attributes\"\n Context attributes are organized into 7 logical categories.\n\n#### 1. Identifiers & Mode\n\nContext attributes for session and mode identification.\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `ID` | `int` | `0` | Session ID |\n| `MODE` | `str` | `\"\"` | Execution mode (normal, service, etc.) |\n| `CURRENT_ROUND_ID` | `int` | `0` | Current round number |\n\n#### 2. Execution State\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `REQUEST` | `str` | `\"\"` | Current user request |\n| `SUBTASK` | `str` | `\"\"` | Current subtask for AppAgent |\n| `PREVIOUS_SUBTASKS` | `List` | `[]` | Previous subtasks history |\n| `HOST_MESSAGE` | `List` | `[]` | HostAgent → AppAgent messages |\n| `ROUND_RESULT` | `str` | `\"\"` | Current round result |\n\n#### 3. Cost Tracking\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `SESSION_COST` | `float` | `0.0` | Total session cost ($) |\n| `ROUND_COST` | `Dict[int, float]` | `{}` | Cost per round |\n| `CURRENT_ROUND_COST` | `float` | `0.0` | Current round cost (auto-sync) |\n\n#### 4. Step Counting\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `SESSION_STEP` | `int` | `0` | Total steps in session |\n| `ROUND_STEP` | `Dict[int, int]` | `{}` | Steps per round |\n| `CURRENT_ROUND_STEP` | `int` | `0` | Current round steps (auto-sync) |\n| `ROUND_SUBTASK_AMOUNT` | `Dict[int, int]` | `{}` | Subtasks per round |\n| `CURRENT_ROUND_SUBTASK_AMOUNT` | `int` | `0` | Current subtasks (auto-sync) |\n\n#### 5. Application Context\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `APPLICATION_WINDOW` | `UIAWrapper` | `None` | Current application window |\n| `APPLICATION_WINDOW_INFO` | `Any` | - | Window metadata |\n| `APPLICATION_PROCESS_NAME` | `str` | `\"\"` | Process name (e.g., \"WINWORD.EXE\") |\n| `APPLICATION_ROOT_NAME` | `str` | `\"\"` | Root UI element name |\n| `CONTROL_REANNOTATION` | `List` | `[]` | Control re-annotations |\n\n#### 6. Logging\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `LOG_PATH` | `str` | `\"\"` | Log directory path |\n| `LOGGER` | `Logger` | `None` | Session logger |\n| `REQUEST_LOGGER` | `Logger` | `None` | LLM request logger |\n| `EVALUATION_LOGGER` | `Logger` | `None` | Evaluation logger |\n| `STRUCTURAL_LOGS` | `defaultdict` | `defaultdict(...)` | Structured logs |\n\n#### 7. Tools & Communication\n\n| Attribute | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `TOOL_INFO` | `Dict` | `{}` | Available tools metadata |\n| `DEVICE_INFO` | `List` | `[]` | Connected device information (Galaxy) |\n| `CONSTELLATION` | `TaskConstellation` | `None` | Task constellation (Galaxy) |\n| `WEAVING_MODE` | `WeavingMode` | `CREATION` | Weaving mode (Galaxy) |\n\n---\n\n## Complete Attribute Reference\n\nAll 30+ attributes with types and defaults.\n\n```python\nclass ContextNames(Enum):\n # Identifiers\n ID = \"ID\" # int, default: 0\n MODE = \"MODE\" # str, default: \"\"\n CURRENT_ROUND_ID = \"CURRENT_ROUND_ID\" # int, default: 0\n \n # Requests & Tasks\n REQUEST = \"REQUEST\" # str, default: \"\"\n SUBTASK = \"SUBTASK\" # str, default: \"\"\n PREVIOUS_SUBTASKS = \"PREVIOUS_SUBTASKS\" # List, default: []\n HOST_MESSAGE = \"HOST_MESSAGE\" # List, default: []\n ROUND_RESULT = \"ROUND_RESULT\" # str, default: \"\"\n \n # Costs\n SESSION_COST = \"SESSION_COST\" # float, default: 0.0\n ROUND_COST = \"ROUND_COST\" # Dict, default: {}\n CURRENT_ROUND_COST = \"CURRENT_ROUND_COST\" # float, default: 0.0\n \n # Steps\n SESSION_STEP = \"SESSION_STEP\" # int, default: 0\n ROUND_STEP = \"ROUND_STEP\" # Dict, default: {}\n CURRENT_ROUND_STEP = \"CURRENT_ROUND_STEP\" # int, default: 0\n ROUND_SUBTASK_AMOUNT = \"ROUND_SUBTASK_AMOUNT\" # Dict, default: {}\n CURRENT_ROUND_SUBTASK_AMOUNT = \"CURRENT_ROUND_SUBTASK_AMOUNT\" # int, default: 0\n \n # Application\n APPLICATION_WINDOW = \"APPLICATION_WINDOW\" # UIAWrapper, default: None\n APPLICATION_WINDOW_INFO = \"APPLICATION_WINDOW_INFO\" # Any\n APPLICATION_PROCESS_NAME = \"APPLICATION_PROCESS_NAME\" # str, default: \"\"\n APPLICATION_ROOT_NAME = \"APPLICATION_ROOT_NAME\" # str, default: \"\"\n CONTROL_REANNOTATION = \"CONTROL_REANNOTATION\" # List, default: []\n \n # Logging\n LOG_PATH = \"LOG_PATH\" # str, default: \"\"\n LOGGER = \"LOGGER\" # Logger, default: None\n REQUEST_LOGGER = \"REQUEST_LOGGER\" # Logger, default: None\n EVALUATION_LOGGER = \"EVALUATION_LOGGER\" # Logger, default: None\n STRUCTURAL_LOGS = \"STRUCTURAL_LOGS\" # defaultdict\n \n # Tools & Devices\n TOOL_INFO = \"TOOL_INFO\" # Dict, default: {}\n DEVICE_INFO = \"DEVICE_INFO\" # List, default: []\n CONSTELLATION = \"CONSTELLATION\" # TaskConstellation, default: None\n WEAVING_MODE = \"WEAVING_MODE\" # WeavingMode, default: CREATION\n```\n\n---\n\n## Context Methods\n\n### get()\n\nRetrieve a value from context:\n\n```python\ndef get(self, name: ContextNames, default: Any = None) -> Any\n```\n\n**Example:**\n\n```python\nrequest = context.get(ContextNames.REQUEST)\n# Returns \"\" if not set\n\ncost = context.get(ContextNames.SESSION_COST, 0.0)\n# Returns 0.0 if not set or uses provided default\n```\n\n### set()\n\nSet a context value:\n\n```python\ndef set(self, name: ContextNames, value: Any) -> None\n```\n\n**Example:**\n\n```python\ncontext.set(ContextNames.REQUEST, \"Send an email to John\")\ncontext.set(ContextNames.SESSION_COST, 0.42)\ncontext.set(ContextNames.APPLICATION_PROCESS_NAME, \"WINWORD.EXE\")\n```\n\n### update_dict()\n\nBatch update multiple values:\n\n```python\ndef update_dict(self, updates: Dict[ContextNames, Any]) -> None\n```\n\n**Example:**\n\n```python\ncontext.update_dict({\n ContextNames.REQUEST: \"New task\",\n ContextNames.MODE: \"normal\",\n ContextNames.SESSION_STEP: 10\n})\n```\n\n### to_dict()\n\nSerialize context to dictionary:\n\n```python\ndef to_dict(self) -> Dict[str, Any]\n```\n\n**Returns**: Dictionary with only JSON-serializable values\n\n**Example:**\n\n```python\ncontext_dict = context.to_dict()\n# Save to file\njson.dump(context_dict, open(\"context.json\", \"w\"))\n```\n\n**Excluded from serialization:**\n- Loggers (`LOGGER`, `REQUEST_LOGGER`, `EVALUATION_LOGGER`)\n- Window objects (`APPLICATION_WINDOW`)\n- Non-serializable objects\n\n### from_dict()\n\nRestore context from dictionary:\n\n```python\n@staticmethod\ndef from_dict(data: Dict[str, Any]) -> \"Context\"\n```\n\n**Example:**\n\n```python\n# Load from file\ndata = json.load(open(\"context.json\"))\ncontext = Context.from_dict(data)\n```\n\n### attach_command_dispatcher()\n\nAttach dispatcher for command execution:\n\n```python\ndef attach_command_dispatcher(self, dispatcher: BasicCommandDispatcher) -> None\n```\n\n**Example:**\n\n```python\nfrom ufo.module.dispatcher import LocalCommandDispatcher\n\ndispatcher = LocalCommandDispatcher(session, mcp_manager)\ncontext.attach_command_dispatcher(dispatcher)\n\n# Now rounds can execute commands via context\n```\n\n---\n\n## Auto-Syncing Properties\n\nThese properties automatically sync with current round values in dictionaries.\n\n### current_round_step\n\n```python\n@property\ndef current_round_step(self) -> int:\n \"\"\"Get current round step.\"\"\"\n return self.attributes.get(ContextNames.ROUND_STEP, {}).get(\n self.attributes.get(ContextNames.CURRENT_ROUND_ID, 0), 0\n )\n\n@current_round_step.setter\ndef current_round_step(self, value: int) -> None:\n \"\"\"Set current round step and update dict.\"\"\"\n round_id = self.attributes.get(ContextNames.CURRENT_ROUND_ID, 0)\n self.attributes[ContextNames.ROUND_STEP][round_id] = value\n self.attributes[ContextNames.CURRENT_ROUND_STEP] = value\n```\n\n**Usage:**\n\n```python\n# Reading\nsteps = context.current_round_step\n\n# Writing (updates both ROUND_STEP dict and CURRENT_ROUND_STEP)\ncontext.current_round_step = 5\n```\n\n### current_round_cost\n\nAuto-syncs cost tracking:\n\n```python\n# Reading\ncost = context.current_round_cost\n\n# Writing (updates both ROUND_COST dict and CURRENT_ROUND_COST)\ncontext.current_round_cost += 0.01\n```\n\n### current_round_subtask_amount\n\nAuto-syncs subtask counting:\n\n```python\n# Reading\nsubtasks = context.current_round_subtask_amount\n\n# Writing\ncontext.current_round_subtask_amount += 1\n```\n\n---\n\n## Usage Patterns\n\n### Pattern 1: Session Initialization\n\n```python\nfrom ufo.module.context import Context, ContextNames\n\n# Create context\ncontext = Context()\n\n# Initialize session metadata\ncontext.set(ContextNames.ID, 0)\ncontext.set(ContextNames.MODE, \"normal\")\ncontext.set(ContextNames.LOG_PATH, \"./logs/task_001/\")\ncontext.set(ContextNames.REQUEST, \"Send an email\")\n```\n\n### Pattern 2: Round Execution\n\n```python\n# At round start\ncontext.set(ContextNames.CURRENT_ROUND_ID, round_id)\n\n# During round\ncontext.current_round_step += 1\ncontext.current_round_cost += agent_cost\n\n# Agent reads state\nrequest = context.get(ContextNames.REQUEST)\nprocess_name = context.get(ContextNames.APPLICATION_PROCESS_NAME)\n```\n\n### Pattern 3: Cost Tracking\n\n```python\n# Agent incurs cost\nagent_cost = llm_call_cost()\ncontext.current_round_cost += agent_cost\n\n# Session total auto-updates\ncontext.set(\n ContextNames.SESSION_COST,\n context.get(ContextNames.SESSION_COST, 0.0) + agent_cost\n)\n\n# Print summary\nprint(f\"Round cost: ${context.current_round_cost:.4f}\")\nprint(f\"Session total: ${context.get(ContextNames.SESSION_COST):.4f}\")\n```\n\n### Pattern 4: Application Tracking\n\n```python\n# Agent selects application\ncontext.set(ContextNames.APPLICATION_PROCESS_NAME, \"WINWORD.EXE\")\ncontext.set(ContextNames.APPLICATION_ROOT_NAME, \"Document1 - Word\")\ncontext.set(ContextNames.APPLICATION_WINDOW, word_window)\n\n# Later rounds access same app\napp_window = context.get(ContextNames.APPLICATION_WINDOW)\nif app_window:\n app_window.set_focus()\n```\n\n### Pattern 5: Logging\n\n```python\n# Setup loggers\ncontext.set(ContextNames.LOGGER, session_logger)\ncontext.set(ContextNames.REQUEST_LOGGER, request_logger)\n\n# Use throughout session\nlogger = context.get(ContextNames.LOGGER)\nlogger.info(\"Round started\")\n\nrequest_logger = context.get(ContextNames.REQUEST_LOGGER)\nrequest_logger.log_request(prompt, response)\n```\n\n### Pattern 6: Persistence\n\n```python\n# Save context state\ncontext_dict = context.to_dict()\nwith open(\"checkpoint.json\", \"w\") as f:\n json.dump(context_dict, f, indent=2)\n\n# Resume from checkpoint\nwith open(\"checkpoint.json\") as f:\n data = json.load(f)\nrestored_context = Context.from_dict(data)\n```\n\n---\n\n## Best Practices\n\n### Type Safety\n\n!!!tip \"Use Enum Names\"\n Always use `ContextNames` enum instead of strings:\n \n ```python\n # ✅ Good\n context.get(ContextNames.REQUEST)\n \n # ❌ Bad\n context.attributes[\"REQUEST\"]\n ```\n\n### Default Values\n\n!!!success \"Leverage Defaults\"\n ContextNames provides sensible defaults:\n \n ```python\n # No need to check for None\n cost = context.get(ContextNames.SESSION_COST) # Returns 0.0 if unset\n \n # Explicit default\n steps = context.get(ContextNames.SESSION_STEP, 0)\n ```\n\n### Auto-Sync\n\n!!!warning \"Use Auto-Sync Properties\"\n For current round values, use auto-sync properties:\n \n ```python\n # ✅ Good - auto-syncs both dicts\n context.current_round_cost += 0.01\n \n # ❌ Manual - must update both\n round_id = context.get(ContextNames.CURRENT_ROUND_ID)\n context.attributes[ContextNames.ROUND_COST][round_id] += 0.01\n context.attributes[ContextNames.CURRENT_ROUND_COST] += 0.01\n ```\n\n---\n\n## Reference\n\n### Context Dataclass\n\n::: module.context.Context\n\n### ContextNames Enum\n\n::: module.context.ContextNames\n\n---\n\n## See Also\n\n- [Session](./session.md) - Session lifecycle and context usage\n- [Round](./round.md) - Round execution with context\n- [Overview](./overview.md) - Module system architecture"
},
{
"path": "documents/docs/infrastructure/modules/dispatcher.md",
"content": "# Command Dispatcher\n\nThe **Command Dispatcher** is the bridge between agent decisions and actual execution, routing commands to the appropriate execution environment (local MCP tools or remote WebSocket clients) and managing result delivery with timeout and error handling.\n\n**Quick Reference:**\n\n- Local execution? Use [LocalCommandDispatcher](#localcommanddispatcher)\n- Remote control? Use [WebSocketCommandDispatcher](#websocketcommanddispatcher)\n- Error handling? See [Error Handling](#error-handling)\n- Custom dispatcher? Extend [BasicCommandDispatcher](#basiccommanddispatcher-abstract-base)\n\n---\n\n## Architecture Overview\n\nThe dispatcher system implements the **Command Pattern** with async execution and comprehensive error handling:\n\n```mermaid\ngraph TB\n subgraph \"Agent Layer\"\n A[Agent Decision Engine]\n CMD[Generate Command Objects]\n end\n \n subgraph \"Dispatcher Interface\"\n BD[BasicCommandDispatcher Abstract Base]\n EXEC[execute_commands async method]\n ERR[generate_error_results error handler]\n end\n \n subgraph \"Local Execution Path\"\n LCD[LocalCommandDispatcher]\n CR[CommandRouter]\n CM[ComputerManager]\n MCP[MCP Server Manager]\n TOOLS[Local Tool Execution]\n end\n \n subgraph \"Remote Execution Path\"\n WSD[WebSocketCommandDispatcher]\n AIP[AIP Protocol]\n WS[WebSocket Transport]\n CLIENT[Remote Client]\n end\n \n subgraph \"Result Handling\"\n RES[Result Objects List~Result~]\n SUCCESS[ResultStatus.SUCCESS]\n FAILURE[ResultStatus.FAILURE]\n end\n \n A --> CMD\n CMD --> EXEC\n EXEC -.inherits.-> BD\n \n BD --> LCD\n BD --> WSD\n \n LCD --> CR\n CR --> CM\n CM --> MCP\n MCP --> TOOLS\n TOOLS --> RES\n \n WSD --> AIP\n AIP --> WS\n WS --> CLIENT\n CLIENT --> RES\n \n ERR --> FAILURE\n RES --> SUCCESS\n RES --> FAILURE\n \n style A fill:#e1f5ff\n style BD fill:#fff4e1\n style LCD fill:#f0ffe1\n style WSD fill:#ffe1f5\n style RES fill:#e1ffe1\n style ERR fill:#ffe1e1\n```\n\n---\n\n## BasicCommandDispatcher (Abstract Base)\n\n`BasicCommandDispatcher` defines the interface that all concrete dispatchers must implement.\n\n### Core Methods\n\n#### `execute_commands()` (Abstract)\n\n```python\nasync def execute_commands(\n self, \n commands: List[Command], \n timeout: float = 6000\n) -> Optional[List[Result]]\n```\n\n**Purpose**: Execute a list of commands and return results.\n\n**Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `commands` | `List[Command]` | Required | Commands to execute |\n| `timeout` | `float` | `6000` | Timeout in seconds |\n\n**Returns:**\n- `List[Result]`: Results from command execution\n- `None`: If execution timed out\n\n!!!warning \"Must Override\"\n Concrete dispatchers **must** implement this method with platform-specific logic.\n\n#### `generate_error_results()`\n\n```python\ndef generate_error_results(\n self, \n commands: List[Command], \n error: Exception\n) -> Optional[List[Result]]\n```\n\n**Purpose**: Convert exceptions into structured error Results.\n\n**Error Handling Logic:**\n\n```mermaid\nsequenceDiagram\n participant D as Dispatcher\n participant E as Exception Handler\n participant R as Result Factory\n \n D->>D: execute_commands()\n D-xD: Exception raised\n D->>E: generate_error_results(commands, error)\n \n loop For each command\n E->>R: Create Result object\n R->>R: status = FAILURE\n R->>R: error = error message\n R->>R: result = error description\n R->>R: call_id = command.call_id\n R-->>E: Error Result\n end\n \n E-->>D: List[Result] (all failures)\n D-->>Agent: Return error results\n```\n\n**Generated Error Result:**\n\n```python\nResult(\n status=ResultStatus.FAILURE,\n error=f\"Error occurred while executing command {command}: {error}\",\n result=f\"Error occurred while executing command {command}: {error}, \"\n f\"please retry or execute a different command.\",\n call_id=command.call_id\n)\n```\n\n!!!example \"Error Result Structure\"\n ```python\n from aip.messages import Result, ResultStatus\n \n # Example error result\n error_result = Result(\n status=ResultStatus.FAILURE,\n error=\"ConnectionRefusedError: [WinError 10061]\",\n result=\"Error occurred while executing command click_element: \"\n \"ConnectionRefusedError, please retry or execute a different command.\",\n call_id=\"cmd_12345\"\n )\n \n # Check in agent code\n if result.status == ResultStatus.FAILURE:\n print(f\"Action failed: {result.error}\")\n # Agent can retry or use alternative approach\n ```\n\n---\n\n## LocalCommandDispatcher\n\n`LocalCommandDispatcher` routes commands to local MCP tool servers for direct execution on the current machine. Used for interactive and standalone sessions.\n\n### Architecture\n\n```mermaid\ngraph TB\n subgraph \"LocalCommandDispatcher\"\n LCD[LocalCommandDispatcher]\n SESSION[session: BaseSession]\n PENDING[pending: Dict~str, Future~]\n MCP_MGR[mcp_server_manager: MCPServerManager]\n CM[computer_manager: ComputerManager]\n CR[command_router: CommandRouter]\n end\n \n subgraph \"Execution Flow\"\n CMD[Receive Commands]\n ID[Assign call_id to each]\n ROUTE[CommandRouter.execute]\n EXEC[ComputerManager → MCP]\n WAIT[asyncio.wait_for]\n RES[Return Results]\n end\n \n subgraph \"Error Paths\"\n TIMEOUT[asyncio.TimeoutError]\n EXCEPTION[Exception]\n ERR_RES[generate_error_results]\n end\n \n LCD --> SESSION\n LCD --> MCP_MGR\n LCD --> CM\n LCD --> CR\n \n CMD --> ID\n ID --> ROUTE\n ROUTE --> EXEC\n EXEC --> WAIT\n WAIT --> RES\n \n WAIT -.timeout.-> TIMEOUT\n EXEC -.exception.-> EXCEPTION\n TIMEOUT --> ERR_RES\n EXCEPTION --> ERR_RES\n ERR_RES --> RES\n \n style LCD fill:#e1f5ff\n style CMD fill:#fff4e1\n style RES fill:#e1ffe1\n style ERR_RES fill:#ffe1e1\n```\n\n### Initialization\n\n```python\nfrom ufo.module.dispatcher import LocalCommandDispatcher\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\ndef _init_context(self) -> None:\n \"\"\"Initialize context with local dispatcher.\"\"\"\n super()._init_context()\n \n # Create MCP server manager\n mcp_server_manager = MCPServerManager()\n \n # Create local dispatcher\n command_dispatcher = LocalCommandDispatcher(\n session=self,\n mcp_server_manager=mcp_server_manager\n )\n \n # Attach to context\n self.context.attach_command_dispatcher(command_dispatcher)\n```\n\n**Initialization Parameters:**\n\n| Parameter | Type | Purpose |\n|-----------|------|---------|\n| `session` | `BaseSession` | Current session instance |\n| `mcp_server_manager` | `MCPServerManager` | MCP server lifecycle manager |\n\n**Internal Components Created:**\n\n- `ComputerManager`: Manages computer-level operations\n- `CommandRouter`: Routes commands to appropriate MCP tools\n\n### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant Dispatcher as LocalCommandDispatcher\n participant Router as CommandRouter\n participant Computer as ComputerManager\n participant MCP as MCP Servers\n \n Agent->>Dispatcher: execute_commands([cmd1, cmd2])\n Dispatcher->>Dispatcher: Assign call_id to each command\n \n Dispatcher->>Router: execute(agent_name, root_name, process_name, commands)\n \n Router->>Computer: Route based on tool_type\n \n par Execute cmd1\n Computer->>MCP: Tool server 1\n MCP-->>Computer: Result 1\n and Execute cmd2\n Computer->>MCP: Tool server 2\n MCP-->>Computer: Result 2\n end\n \n Computer-->>Router: Results [res1, res2]\n Router-->>Dispatcher: Results\n Dispatcher-->>Agent: Results\n \n alt Timeout\n Dispatcher-xDispatcher: asyncio.TimeoutError\n Dispatcher->>Dispatcher: generate_error_results()\n Dispatcher-->>Agent: Error Results\n end\n \n alt Exception\n Router-xRouter: Exception\n Dispatcher->>Dispatcher: generate_error_results()\n Dispatcher-->>Agent: Error Results\n end\n```\n\n### Command Routing Context\n\nThe dispatcher provides execution context to the CommandRouter:\n\n| Context | Source | Purpose |\n|---------|--------|---------|\n| `agent_name` | `session.current_agent_class` | Track which agent issued command |\n| `root_name` | `context.APPLICATION_ROOT_NAME` | Application root for UI operations |\n| `process_name` | `context.APPLICATION_PROCESS_NAME` | Process name for targeting |\n| `commands` | Command list | Actions to execute |\n\n!!!example \"Local Execution Example\"\n ```python\n from aip.messages import Command, ResultStatus\n \n # Commands for local execution\n commands = [\n Command(\n tool_name=\"click_element\",\n parameters={\"control_label\": \"1\", \"button\": \"left\"},\n tool_type=\"windows\", # Routed to Windows MCP server\n call_id=\"\" # Will be auto-assigned\n ),\n Command(\n tool_name=\"type_text\",\n parameters={\"text\": \"Hello World\"},\n tool_type=\"windows\",\n call_id=\"\"\n )\n ]\n \n # Execute locally\n results = await context.command_dispatcher.execute_commands(\n commands=commands,\n timeout=30.0\n )\n \n # Process results\n for i, result in enumerate(results):\n if result.status == ResultStatus.SUCCESS:\n print(f\"Command {i+1} succeeded: {result.result}\")\n else:\n print(f\"Command {i+1} failed: {result.error}\")\n ```\n\n### Error Scenarios\n\n| Error Type | Trigger | Handling | Result |\n|------------|---------|----------|--------|\n| **TimeoutError** | Execution exceeds `timeout` | `generate_error_results()` | Error Results with timeout message |\n| **ConnectionError** | MCP server unreachable | `generate_error_results()` | Error Results with connection error |\n| **ValidationError** | Invalid command parameters | `generate_error_results()` | Error Results with validation error |\n| **RuntimeError** | Tool execution failure | `generate_error_results()` | Error Results with execution error |\n\n!!!warning \"Timeout Considerations\"\n - Default timeout: **6000 seconds** (100 minutes)\n - For UI operations: Consider **30-60 seconds**\n - For network operations: May need longer timeouts\n - Always handle timeout gracefully in agent code\n\n---\n\n## WebSocketCommandDispatcher\n\n`WebSocketCommandDispatcher` uses the AIP protocol to send commands to remote clients over WebSocket connections. Used for service sessions and remote control.\n\n### Architecture\n\n```mermaid\ngraph TB\n subgraph \"WebSocketCommandDispatcher\"\n WSD[WebSocketCommandDispatcher]\n SESSION[session: BaseSession]\n PROTOCOL[protocol: TaskExecutionProtocol]\n PENDING[pending: Dict~str, Future~]\n QUEUE[send_queue: asyncio.Queue]\n end\n \n subgraph \"AIP Protocol Layer\"\n MSG[ServerMessage Factory]\n SEND[protocol.send_command]\n RECV[protocol.receive_result]\n end\n \n subgraph \"WebSocket Transport\"\n WS[WebSocket Connection]\n CLIENT[Remote Client]\n end\n \n subgraph \"Result Management\"\n FUT[asyncio.Future]\n WAIT[await with timeout]\n RES[Results]\n end\n \n WSD --> SESSION\n WSD --> PROTOCOL\n WSD --> PENDING\n \n WSD --> MSG\n MSG --> SEND\n SEND --> WS\n WS --> CLIENT\n \n CLIENT --> RECV\n RECV --> FUT\n FUT --> WAIT\n WAIT --> RES\n \n style WSD fill:#e1f5ff\n style PROTOCOL fill:#fff4e1\n style WS fill:#f0ffe1\n style RES fill:#e1ffe1\n```\n\n### Initialization\n\n```python\nfrom ufo.module.dispatcher import WebSocketCommandDispatcher\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\ndef _init_context(self) -> None:\n \"\"\"Initialize context with WebSocket dispatcher.\"\"\"\n super()._init_context()\n \n # Create WebSocket dispatcher with AIP protocol\n command_dispatcher = WebSocketCommandDispatcher(\n session=self,\n protocol=self.task_protocol # TaskExecutionProtocol instance\n )\n \n # Attach to context\n self.context.attach_command_dispatcher(command_dispatcher)\n```\n\n**Initialization Parameters:**\n\n| Parameter | Type | Purpose |\n|-----------|------|---------|\n| `session` | `BaseSession` | Current service session |\n| `protocol` | `TaskExecutionProtocol` | AIP protocol handler |\n\n!!!danger \"Protocol Required\"\n WebSocketCommandDispatcher **requires** a `TaskExecutionProtocol` instance. It will raise `ValueError` if protocol is `None`.\n\n### Message Construction\n\nThe dispatcher creates structured AIP ServerMessages:\n\n```python\ndef make_server_response(self, commands: List[Command]) -> ServerMessage:\n \"\"\"\n Create a server response message for the given commands.\n \"\"\"\n # Assign unique IDs\n for command in commands:\n command.call_id = str(uuid.uuid4())\n \n # Extract context\n agent_name = self.session.current_agent_class\n process_name = self.session.context.get(ContextNames.APPLICATION_PROCESS_NAME)\n root_name = self.session.context.get(ContextNames.APPLICATION_ROOT_NAME)\n session_id = self.session.id\n response_id = str(uuid.uuid4())\n \n # Build AIP message\n return ServerMessage(\n type=ServerMessageType.COMMAND,\n status=TaskStatus.CONTINUE,\n agent_name=agent_name,\n process_name=process_name,\n root_name=root_name,\n actions=commands,\n session_id=session_id,\n task_name=self.session.task,\n timestamp=datetime.datetime.now(datetime.timezone.utc).isoformat(),\n response_id=response_id\n )\n```\n\n**ServerMessage Fields:**\n\n| Field | Source | Purpose |\n|-------|--------|---------|\n| `type` | `ServerMessageType.COMMAND` | Indicates command message |\n| `status` | `TaskStatus.CONTINUE` | Task in progress |\n| `agent_name` | Current agent class | Track agent issuing command |\n| `process_name` | Context | Target process |\n| `root_name` | Context | Application root |\n| `actions` | Command list | Commands to execute |\n| `session_id` | Session ID | Session tracking |\n| `task_name` | Session task | Task identification |\n| `timestamp` | Current UTC time | Message timing |\n| `response_id` | UUID | Correlate request/response |\n\n### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant Dispatcher as WebSocketCommandDispatcher\n participant Protocol as TaskExecutionProtocol\n participant WS as WebSocket\n participant Client as Remote Client\n \n Agent->>Dispatcher: execute_commands([cmd1, cmd2])\n Dispatcher->>Dispatcher: Assign call_id to each\n Dispatcher->>Dispatcher: make_server_response()\n Dispatcher->>Dispatcher: Create Future for response_id\n \n Dispatcher->>Protocol: send_command(ServerMessage)\n Protocol->>WS: Send via WebSocket\n WS->>Client: Transmit message\n \n Note over Dispatcher: await Future with timeout\n \n Client->>Client: Execute commands locally\n Client->>WS: Send ClientMessage with results\n WS->>Protocol: Receive message\n Protocol->>Dispatcher: set_result(response_id, ClientMessage)\n Dispatcher->>Dispatcher: Resolve Future\n \n Dispatcher-->>Agent: Return action_results\n \n alt Timeout\n Dispatcher-xDispatcher: asyncio.TimeoutError\n Dispatcher->>Dispatcher: generate_error_results()\n Dispatcher-->>Agent: Error Results\n end\n \n alt Send Error\n Protocol-xProtocol: Exception\n Dispatcher->>Dispatcher: generate_error_results()\n Dispatcher-->>Agent: Error Results\n end\n```\n\n### Result Handling\n\nThe `set_result()` method is called by the WebSocket handler when a client response arrives:\n\n```python\nasync def set_result(self, response_id: str, result: ClientMessage) -> None:\n \"\"\"\n Called by WebSocket handler when client returns a message.\n :param response_id: The ID of the response.\n :param result: The result from the client.\n \"\"\"\n fut = self.pending.get(response_id)\n if fut and not fut.done():\n fut.set_result(result.action_results)\n```\n\n**Pending Future Management:**\n\n```mermaid\ngraph LR\n subgraph \"Request Side\"\n REQ[execute_commands]\n FUT[Create Future]\n PEND[Store in pending dict]\n WAIT[Await Future]\n end\n \n subgraph \"Response Side\"\n RECV[WebSocket receives result]\n LOOKUP[Lookup Future by response_id]\n RESOLVE[set_result on Future]\n end\n \n REQ --> FUT\n FUT --> PEND\n PEND --> WAIT\n \n RECV --> LOOKUP\n LOOKUP --> RESOLVE\n RESOLVE -.resolves.-> WAIT\n \n style REQ fill:#e1f5ff\n style RECV fill:#fff4e1\n style WAIT fill:#e1ffe1\n```\n\n!!!example \"WebSocket Execution Example\"\n ```python\n from aip.messages import Command\n \n # Session is ServiceSession with WebSocketCommandDispatcher\n commands = [\n Command(\n tool_name=\"capture_window_screenshot\",\n parameters={},\n tool_type=\"data_collection\"\n )\n ]\n \n # Execute remotely via WebSocket\n results = await context.command_dispatcher.execute_commands(\n commands=commands,\n timeout=60.0 # Screenshot may take time\n )\n \n # Results came from remote client\n if results:\n screenshot_base64 = results[0].result\n # Process screenshot...\n ```\n\n### Error Scenarios\n\n| Error Type | Trigger | Handling | Result |\n|------------|---------|----------|--------|\n| **TimeoutError** | Client doesn't respond in time | `generate_error_results()` | Error Results |\n| **ProtocolError** | AIP protocol violation | `generate_error_results()` | Error Results |\n| **ConnectionError** | WebSocket disconnected | `generate_error_results()` | Error Results |\n| **ClientError** | Client reports execution failure | Return client's error Result | Propagate client error |\n\n!!!warning \"WebSocket-Specific Considerations\"\n - **Network latency**: Add buffer to timeouts\n - **Client state**: Client may be busy with other tasks\n - **Connection loss**: Implement reconnection logic\n - **Message ordering**: AIP ensures ordered delivery\n\n---\n\n## Error Handling\n\nAll dispatchers convert exceptions into structured `Result` objects to maintain consistent error handling.\n\n### Error Flow\n\n```mermaid\ngraph TB\n START[Command Execution Starts]\n \n TRY{Try Block}\n SUCCESS[Commands Execute Successfully]\n RETURN_OK[Return Results]\n \n TIMEOUT{Timeout?}\n EXCEPTION{Other Exception?}\n \n GEN_ERR[generate_error_results]\n CREATE_RESULTS[Create Result for each command]\n SET_FAILURE[Set status = FAILURE]\n ADD_ERROR[Add error message]\n RETURN_ERR[Return Error Results]\n \n START --> TRY\n TRY -->|Success| SUCCESS\n TRY -->|Failure| TIMEOUT\n SUCCESS --> RETURN_OK\n \n TIMEOUT -->|Yes| GEN_ERR\n TIMEOUT -->|No| EXCEPTION\n \n EXCEPTION -->|Yes| GEN_ERR\n \n GEN_ERR --> CREATE_RESULTS\n CREATE_RESULTS --> SET_FAILURE\n SET_FAILURE --> ADD_ERROR\n ADD_ERROR --> RETURN_ERR\n \n style START fill:#e1f5ff\n style SUCCESS fill:#e1ffe1\n style GEN_ERR fill:#ffe1e1\n style RETURN_OK fill:#f0ffe1\n style RETURN_ERR fill:#fff4e1\n```\n\n### Error Result Format\n\n```python\n{\n \"status\": \"failure\", # ResultStatus.FAILURE\n \"error\": \"asyncio.TimeoutError: Command execution timed out\",\n \"result\": \"Error occurred while executing command : TimeoutError, \"\n \"please retry or execute a different command.\",\n \"call_id\": \"cmd_abc123\"\n}\n```\n\n### Agent Error Handling\n\nAgents should handle error results appropriately:\n\n```python\nasync def execute_action(self, context: Context) -> None:\n \"\"\"Execute action with error handling.\"\"\"\n commands = self.generate_commands()\n \n results = await context.command_dispatcher.execute_commands(\n commands=commands,\n timeout=30.0\n )\n \n for command, result in zip(commands, results):\n if result.status == ResultStatus.FAILURE:\n # Log error\n self.logger.error(f\"Command {command.tool_name} failed: {result.error}\")\n \n # Decision logic\n if \"timeout\" in result.error.lower():\n # Retry with longer timeout\n self.retry_count += 1\n if self.retry_count < 3:\n return await self.execute_action(context)\n \n elif \"connection\" in result.error.lower():\n # Switch to alternative approach\n return self.fallback_strategy()\n \n else:\n # Escalate to error state\n self.transition_to_error_state(result.error)\n else:\n # Process successful result\n self.process_result(result.result)\n```\n\n!!!tip \"Error Handling Best Practices\"\n - ✅ Always check `result.status` before using `result.result`\n - ✅ Log errors with context (command, parameters, error message)\n - ✅ Implement retry logic for transient errors\n - ✅ Provide fallback strategies for permanent failures\n - ✅ Include helpful error messages for users\n - ❌ Don't ignore error results\n - ❌ Don't assume all commands succeed\n - ❌ Don't retry indefinitely without backoff\n\n---\n\n## Usage Patterns\n\n### Pattern 1: Sequential Execution\n\nExecute commands one at a time:\n\n```python\nfor command in command_list:\n results = await context.command_dispatcher.execute_commands(\n commands=[command],\n timeout=30.0\n )\n \n if results[0].status == ResultStatus.SUCCESS:\n # Process result and decide next command\n next_command = self.decide_next_action(results[0])\n else:\n # Handle error and possibly abort\n break\n```\n\n### Pattern 2: Batch Execution\n\nExecute multiple related commands together:\n\n```python\n# All commands for a subtask\ncommands = [\n Command(tool_name=\"click_element\", ...),\n Command(tool_name=\"type_text\", ...),\n Command(tool_name=\"press_key\", ...)\n]\n\nresults = await context.command_dispatcher.execute_commands(\n commands=commands,\n timeout=60.0\n)\n\n# Process all results\nfor command, result in zip(commands, results):\n if result.status == ResultStatus.FAILURE:\n # One failure might invalidate the whole subtask\n self.handle_subtask_failure(command, result)\n```\n\n### Pattern 3: Conditional Execution\n\nExecute commands based on previous results:\n\n```python\n# Check state first\ncheck_cmd = Command(tool_name=\"get_ui_tree\", ...)\ncheck_results = await dispatcher.execute_commands([check_cmd])\n\nif check_results[0].status == ResultStatus.SUCCESS:\n ui_tree = check_results[0].result\n \n # Decide action based on UI state\n if \"Login\" in ui_tree:\n action_cmd = Command(tool_name=\"click_element\", parameters={\"label\": \"Login\"})\n else:\n action_cmd = Command(tool_name=\"type_text\", parameters={\"text\": \"username\"})\n \n # Execute decided action\n await dispatcher.execute_commands([action_cmd])\n```\n\n### Pattern 4: Retry with Backoff\n\nRetry failed commands with exponential backoff:\n\n```python\nimport asyncio\n\nasync def execute_with_retry(\n dispatcher, \n commands, \n max_retries=3, \n base_delay=1.0\n):\n \"\"\"Execute commands with exponential backoff retry.\"\"\"\n \n for attempt in range(max_retries):\n results = await dispatcher.execute_commands(commands, timeout=30.0)\n \n # Check if all succeeded\n all_success = all(r.status == ResultStatus.SUCCESS for r in results)\n \n if all_success:\n return results\n \n # Not last attempt - retry with backoff\n if attempt < max_retries - 1:\n delay = base_delay * (2 ** attempt)\n logger.warning(f\"Retry attempt {attempt + 1} after {delay}s\")\n await asyncio.sleep(delay)\n \n # All retries exhausted\n return results # Return last attempt results\n```\n\n---\n\n## Performance Considerations\n\n### Timeout Configuration\n\nChoose timeouts based on operation type:\n\n| Operation Type | Recommended Timeout | Reason |\n|----------------|---------------------|--------|\n| **UI clicks** | 10-30s | Fast but may wait for animations |\n| **Text input** | 5-15s | Usually fast |\n| **Screenshots** | 30-60s | May need rendering time |\n| **File operations** | 60-120s | I/O dependent |\n| **Network calls** | 120-300s | Network latency + processing |\n| **Batch operations** | Sum of individual + 20% | Account for overhead |\n\n### Command Batching\n\n**When to batch:**\n- ✅ Related actions in same context (e.g., fill form fields)\n- ✅ Commands with no dependencies between them\n- ✅ All commands target same application\n\n**When not to batch:**\n- ❌ Commands with dependencies (need sequential execution)\n- ❌ Mix of fast and slow operations (one timeout for all)\n- ❌ Need intermediate results to decide next action\n\n### Resource Management\n\n```python\n# Good: Reuse dispatcher attached to context\nresults1 = await context.command_dispatcher.execute_commands(commands1)\nresults2 = await context.command_dispatcher.execute_commands(commands2)\n\n# Bad: Creating new dispatchers\ndispatcher1 = LocalCommandDispatcher(session, mcp_manager)\ndispatcher2 = LocalCommandDispatcher(session, mcp_manager)\n```\n\n---\n\n## Advanced Topics\n\n### Custom Dispatcher Implementation\n\nExtend `BasicCommandDispatcher` for custom execution logic:\n\n```python\nfrom ufo.module.dispatcher import BasicCommandDispatcher\nfrom aip.messages import Command, Result, ResultStatus\nfrom typing import List, Optional\n\nclass CustomCommandDispatcher(BasicCommandDispatcher):\n \"\"\"\n Custom dispatcher that logs all commands and results.\n \"\"\"\n \n def __init__(self, session, log_file: str):\n self.session = session\n self.log_file = log_file\n \n async def execute_commands(\n self, \n commands: List[Command], \n timeout: float = 6000\n ) -> Optional[List[Result]]:\n \"\"\"Execute with logging.\"\"\"\n \n # Log commands\n with open(self.log_file, 'a') as f:\n f.write(f\"Executing {len(commands)} commands\\n\")\n for cmd in commands:\n f.write(f\" {cmd.tool_name}: {cmd.parameters}\\n\")\n \n try:\n # Your custom execution logic here\n results = await self.custom_execute(commands, timeout)\n \n # Log results\n with open(self.log_file, 'a') as f:\n for result in results:\n f.write(f\" Result: {result.status}\\n\")\n \n return results\n \n except Exception as e:\n # Log error\n with open(self.log_file, 'a') as f:\n f.write(f\" ERROR: {e}\\n\")\n \n return self.generate_error_results(commands, e)\n \n async def custom_execute(\n self, \n commands: List[Command], \n timeout: float\n ) -> List[Result]:\n \"\"\"Implement custom execution logic.\"\"\"\n # Your implementation here\n pass\n```\n\n### Dispatcher Selection Logic\n\nChoose dispatcher based on session type:\n\n```python\nfrom ufo.module.dispatcher import LocalCommandDispatcher, WebSocketCommandDispatcher\n\ndef attach_appropriate_dispatcher(session, context):\n \"\"\"Attach correct dispatcher based on session type.\"\"\"\n \n if isinstance(session, ServiceSession):\n # Service session uses WebSocket\n dispatcher = WebSocketCommandDispatcher(\n session=session,\n protocol=session.task_protocol\n )\n else:\n # Interactive session uses local execution\n mcp_manager = MCPServerManager()\n dispatcher = LocalCommandDispatcher(\n session=session,\n mcp_server_manager=mcp_manager\n )\n \n context.attach_command_dispatcher(dispatcher)\n```\n\n---\n\n## Troubleshooting\n\n### Issue: Commands Timeout\n\n**Symptoms:**\n- Commands consistently timeout\n- `asyncio.TimeoutError` in logs\n- Error results with timeout messages\n\n**Diagnosis:**\n```python\n# Check timeout value\nresults = await dispatcher.execute_commands(commands, timeout=30.0)\n\n# Enable debug logging\nlogging.getLogger('ufo.module.dispatcher').setLevel(logging.DEBUG)\n```\n\n**Solutions:**\n1. Increase timeout for slow operations\n2. Check MCP server health (local dispatcher)\n3. Verify WebSocket connection (WebSocket dispatcher)\n4. Split batch into smaller groups\n\n### Issue: Connection Errors\n\n**Symptoms:**\n- Connection refused errors\n- WebSocket disconnection\n- MCP server not responding\n\n**Diagnosis:**\n```python\n# For LocalCommandDispatcher\n# Check MCP server status\nmcp_manager.check_server_health()\n\n# For WebSocketCommandDispatcher\n# Check WebSocket connection\nif protocol.is_connected():\n print(\"WebSocket connected\")\nelse:\n print(\"WebSocket disconnected\")\n```\n\n**Solutions:**\n1. Restart MCP servers\n2. Reconnect WebSocket\n3. Check firewall/network settings\n4. Verify client is running\n\n### Issue: Wrong Dispatcher Used\n\n**Symptoms:**\n- Commands routed incorrectly\n- MCP tools called in service session\n- WebSocket messages in local session\n\n**Diagnosis:**\n```python\n# Check dispatcher type\nprint(type(context.command_dispatcher))\n# Should be LocalCommandDispatcher or WebSocketCommandDispatcher\n\n# Check session type\nprint(type(session))\n```\n\n**Solution:**\nEnsure correct dispatcher initialization in session `_init_context()`.\n\n---\n\n## Reference\n\n### BasicCommandDispatcher\n\n::: module.dispatcher.BasicCommandDispatcher\n\n### LocalCommandDispatcher\n\n::: module.dispatcher.LocalCommandDispatcher\n\n### WebSocketCommandDispatcher\n\n::: module.dispatcher.WebSocketCommandDispatcher\n\n---\n\n\n## See Also\n\n- [Context](./context.md) - State management and dispatcher attachment\n- [Session](./session.md) - Session lifecycle and dispatcher initialization\n- [AIP Protocol](../../aip/overview.md) - WebSocket message protocol\n- [MCP Integration](../../mcp/overview.md) - Local tool execution\n\n"
},
{
"path": "documents/docs/infrastructure/modules/overview.md",
"content": "# Module System Overview\n\nThe **Module System** is the core execution engine of UFO, orchestrating the complete lifecycle of user interactions from initial request to final completion. It manages sessions, rounds, context state, and command dispatch across both Windows and Linux platforms.\n\n**Quick Navigation:**\n\n- New to modules? Start with [Session](./session.md) and [Round](./round.md) basics\n- Understanding state? See [Context](./context.md) management\n- Command execution? Check [Dispatcher](./dispatcher.md) patterns\n\n---\n\n## Architecture Overview\n\nThe module system implements a **hierarchical execution model** with clear separation of concerns:\n\n```mermaid\ngraph TB\n subgraph \"User Interaction Layer\"\n UI[Interactor User I/O]\n end\n \n subgraph \"Session Management Layer\"\n SF[SessionFactory Creates sessions]\n SP[SessionPool Manages multiple sessions]\n S[Session Conversation lifecycle]\n end\n \n subgraph \"Execution Layer\"\n R[Round Single request handler]\n C[Context Shared state]\n end\n \n subgraph \"Command Layer\"\n D[Dispatcher Command routing]\n LCD[LocalCommandDispatcher]\n WSD[WebSocketCommandDispatcher]\n end\n \n subgraph \"Platform Layer\"\n WS[WindowsBaseSession]\n LS[LinuxBaseSession]\n SS[ServiceSession]\n end\n \n UI -.Request.-> SF\n SF --> SP\n SP --> S\n S --> R\n R --> C\n R --> D\n D --> LCD\n D --> WSD\n \n S -.inherits.-> WS\n S -.inherits.-> LS\n S -.inherits.-> SS\n \n style UI fill:#e1f5ff\n style SF fill:#fff4e1\n style SP fill:#f0ffe1\n style S fill:#ffe1f5\n style R fill:#e1ffe1\n style C fill:#ffe1e1\n style D fill:#f5e1ff\n```\n\n---\n\n## Core Components\n\n### 1. Session Management\n\nA **Session** represents a complete conversation between the user and UFO, potentially spanning multiple requests and rounds.\n\n**Session Hierarchy:**\n\n```mermaid\nclassDiagram\n class BaseSession {\n <>\n +task: str\n +context: Context\n +rounds: Dict[int, BaseRound]\n +run()\n +create_new_round()\n +is_finished()\n }\n \n class WindowsBaseSession {\n +host_agent: HostAgent\n +_init_agents()\n }\n \n class LinuxBaseSession {\n +agent: LinuxAgent\n +_init_agents()\n }\n \n class Session {\n +mode: str\n +next_request()\n }\n \n class ServiceSession {\n +task_protocol: TaskExecutionProtocol\n +_init_context()\n }\n \n class LinuxSession {\n +next_request()\n }\n \n class FollowerSession {\n +plan_reader: PlanReader\n }\n \n BaseSession <|-- WindowsBaseSession\n BaseSession <|-- LinuxBaseSession\n WindowsBaseSession <|-- Session\n WindowsBaseSession <|-- ServiceSession\n LinuxBaseSession <|-- LinuxSession\n Session <|-- FollowerSession\n```\n\n**Session Types:**\n\n| Session Type | Platform | Use Case | Communication |\n|--------------|----------|----------|---------------|\n| **Session** | Windows | Interactive mode | Local |\n| **ServiceSession** | Windows | Server-controlled | WebSocket (AIP) |\n| **LinuxSession** | Linux | Interactive mode | Local |\n| **LinuxServiceSession** | Linux | Server-controlled | WebSocket (AIP) |\n| **FollowerSession** | Windows | Plan execution | Local |\n| **FromFileSession** | Windows | Batch processing | Local |\n| **OpenAIOperatorSession** | Windows | Operator mode | Local |\n\n!!!example \"Session Creation\"\n ```python\n from ufo.module.session_pool import SessionFactory\n \n # Create interactive Windows session\n factory = SessionFactory()\n sessions = factory.create_session(\n task=\"email_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Open Outlook and send an email\"\n )\n \n # Create Linux service session\n linux_session = factory.create_service_session(\n task=\"data_task\",\n should_evaluate=True,\n id=\"session_001\",\n request=\"Process CSV files\",\n platform_override=\"linux\"\n )\n ```\n\n---\n\n### 2. Round Execution\n\nA **Round** handles a single user request by orchestrating agents through a state machine, executing actions until completion.\n\n**Round Lifecycle:**\n\n```mermaid\nstateDiagram-v2\n [*] --> Created: Initialize Round\n Created --> AgentHandle: agent.handle(context)\n AgentHandle --> StateTransition: Determine next state\n StateTransition --> AgentSwitch: Switch agent if needed\n AgentSwitch --> SubtaskCheck: Check if subtask ends\n \n SubtaskCheck --> CaptureSnapshot: Subtask complete\n SubtaskCheck --> AgentHandle: Continue\n \n CaptureSnapshot --> AgentHandle: Next subtask\n \n AgentHandle --> RoundComplete: is_finished() = True\n RoundComplete --> Evaluation: should_evaluate = True\n RoundComplete --> [*]: should_evaluate = False\n Evaluation --> [*]\n \n note right of AgentHandle\n Agent processes current state\n Updates context\n Executes actions\n end note\n \n note right of StateTransition\n State pattern determines:\n - Next state\n - Next agent\n - Round completion\n end note\n```\n\n**Key Round Operations:**\n\n| Operation | Purpose | Trigger |\n|-----------|---------|---------|\n| `agent.handle(context)` | Process current state | Each iteration |\n| `state.next_state(agent)` | Determine next state | After handle |\n| `state.next_agent(agent)` | Switch agent if needed | After state transition |\n| `capture_last_snapshot()` | Save UI state | Subtask/Round end |\n| `evaluation()` | Assess completion | Round end (if enabled) |\n\n!!!warning \"Round Termination Conditions\"\n A round finishes when:\n - `state.is_round_end()` returns `True`\n - Session step exceeds `ufo_config.system.max_step`\n - Agent enters ERROR state\n\n---\n\n### 3. Context State Management\n\n**Context** is a type-safe key-value store that maintains state across all rounds in a session.\n\n**Context Architecture:**\n\n```mermaid\ngraph LR\n subgraph \"Context Storage\"\n CN[ContextNames Enum]\n CV[Context Values Dict]\n end\n \n subgraph \"Tracked Data\"\n ID[Session/Round IDs]\n ST[Steps & Costs]\n LOG[Loggers]\n APP[Application State]\n CMD[Command Dispatcher]\n end\n \n subgraph \"Access Patterns\"\n GET[context.get(key)]\n SET[context.set(key, value)]\n UPD[context.update_dict(key, dict)]\n end\n \n CN -.defines.-> CV\n CV --> ID\n CV --> ST\n CV --> LOG\n CV --> APP\n CV --> CMD\n \n GET -.reads.-> CV\n SET -.writes.-> CV\n UPD -.merges.-> CV\n \n style CN fill:#e1f5ff\n style CV fill:#fff4e1\n style GET fill:#f0ffe1\n style SET fill:#ffe1f5\n style UPD fill:#f5e1ff\n```\n\n**Context Categories:**\n\n| Category | Context Names | Type | Purpose |\n|----------|---------------|------|---------|\n| **Identifiers** | `ID`, `CURRENT_ROUND_ID` | `int` | Session/round tracking |\n| **Execution State** | `SESSION_STEP`, `ROUND_STEP` | `int/dict` | Progress tracking |\n| **Cost Tracking** | `SESSION_COST`, `ROUND_COST` | `float/dict` | LLM API costs |\n| **Requests** | `REQUEST`, `SUBTASK`, `PREVIOUS_SUBTASKS` | `str/list` | Task information |\n| **Application** | `APPLICATION_WINDOW`, `APPLICATION_PROCESS_NAME` | `UIAWrapper/str` | UI automation |\n| **Logging** | `LOGGER`, `REQUEST_LOGGER`, `EVALUATION_LOGGER` | `FileWriter` | Log outputs |\n| **Communication** | `HOST_MESSAGE`, `CONTROL_REANNOTATION` | `list` | Agent messages |\n| **Infrastructure** | `command_dispatcher` | `BasicCommandDispatcher` | Command execution |\n\n!!!example \"Context Usage Patterns\"\n ```python\n from ufo.module.context import Context, ContextNames\n \n # Initialize context\n context = Context()\n \n # Set values\n context.set(ContextNames.REQUEST, \"Open Notepad\")\n context.set(ContextNames.SESSION_STEP, 0)\n \n # Get values\n request = context.get(ContextNames.REQUEST) # \"Open Notepad\"\n step = context.get(ContextNames.SESSION_STEP) # 0\n \n # Update dictionaries (for round-specific tracking)\n round_costs = {1: 0.05, 2: 0.03}\n context.update_dict(ContextNames.ROUND_COST, round_costs)\n \n # Auto-sync current round values\n current_cost = context.current_round_cost # Auto-synced\n ```\n\n---\n\n### 4. Command Dispatching\n\n**Dispatchers** route commands to execution environments (local MCP tools or remote WebSocket clients) and handle result delivery.\n\n**Dispatcher Architecture:**\n\n```mermaid\ngraph TB\n subgraph \"Agent Layer\"\n AG[Agent generates commands]\n end\n \n subgraph \"Dispatcher Layer\"\n BD[BasicCommandDispatcher Abstract base]\n LCD[LocalCommandDispatcher MCP tools]\n WSD[WebSocketCommandDispatcher AIP protocol]\n end\n \n subgraph \"Execution Layer\"\n CR[CommandRouter]\n CM[ComputerManager]\n MCP[MCP Servers]\n WS[WebSocket Client]\n end\n \n subgraph \"Result Handling\"\n RES[Results: List~Result~]\n ERR[Error Results]\n end\n \n AG --> BD\n BD -.implements.-> LCD\n BD -.implements.-> WSD\n \n LCD --> CR\n CR --> CM\n CM --> MCP\n \n WSD --> WS\n \n MCP --> RES\n WS --> RES\n LCD --> ERR\n WSD --> ERR\n \n style AG fill:#e1f5ff\n style BD fill:#fff4e1\n style LCD fill:#f0ffe1\n style WSD fill:#ffe1f5\n style RES fill:#e1ffe1\n```\n\n**Dispatcher Comparison:**\n\n| Dispatcher | Use Case | Communication | Error Handling | Timeout |\n|------------|----------|---------------|----------------|---------|\n| **LocalCommandDispatcher** | Interactive sessions | Direct MCP calls | Generates error Results | 6000s |\n| **WebSocketCommandDispatcher** | Service sessions | AIP protocol messages | Generates error Results | 6000s |\n\n!!!example \"Command Dispatch Flow\"\n ```python\n from aip.messages import Command\n \n # Create commands\n commands = [\n Command(\n tool_name=\"click_element\",\n parameters={\"control_label\": \"1\", \"button\": \"left\"},\n tool_type=\"windows\"\n )\n ]\n \n # Execute via dispatcher (attached to context)\n results = await context.command_dispatcher.execute_commands(\n commands=commands,\n timeout=30.0\n )\n \n # Process results\n for result in results:\n if result.status == ResultStatus.SUCCESS:\n print(f\"Action succeeded: {result.result}\")\n else:\n print(f\"Action failed: {result.error}\")\n ```\n\n---\n\n### 5. User Interaction\n\n**Interactor** provides rich CLI experiences for user input with styled prompts, panels, and confirmations.\n\n**Interaction Flows:**\n\n```mermaid\nsequenceDiagram\n participant U as User\n participant I as Interactor\n participant S as Session\n participant R as Round\n \n U->>I: Start UFO\n I->>I: first_request()\n I-->>U: 🛸 Welcome Panel\n U->>I: \"Open Notepad\"\n I->>S: Initial request\n \n S->>R: Create Round 1\n R->>R: Execute...\n R-->>S: Round complete\n \n S->>I: new_request()\n I-->>U: 🛸 Next Request Panel\n U->>I: \"Type hello\"\n I->>S: Next request\n \n S->>R: Create Round 2\n R->>R: Execute...\n R-->>S: Round complete\n \n S->>I: new_request()\n I-->>U: 🛸 Next Request Panel\n U->>I: \"N\"\n I-->>U: 👋 Goodbye Panel\n I->>S: complete=True\n S->>S: Terminate\n \n S->>I: experience_asker()\n I-->>U: 💾 Save Experience Panel\n U->>I: Yes\n I->>S: Save experience\n```\n\n**Interactor Functions:**\n\n| Function | Purpose | Returns | Example UI |\n|----------|---------|---------|-----------|\n| `first_request()` | Initial request prompt | `str` | 🛸 Welcome Panel with examples |\n| `new_request()` | Subsequent requests | `Tuple[str, bool]` | 🛸 Next Request Panel |\n| `experience_asker()` | Save experience prompt | `bool` | 💾 Learning & Memory Panel |\n| `question_asker()` | Collect information | `str` | 🤔 Numbered Question Panel |\n| `sensitive_step_asker()` | Security confirmation | `bool` | 🔒 Security Check Panel |\n\n!!!example \"Styled User Prompts\"\n ```python\n from ufo.module import interactor\n \n # First interaction with rich welcome\n request = interactor.first_request()\n # Shows:\n # ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n # ┃ 🛸 UFO Assistant ┃\n # ┃ 🚀 Welcome to UFO - Your AI Assistant ┃\n # ┃ ...examples... ┃\n # ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛\n \n # Get next request\n request, complete = interactor.new_request()\n if complete:\n print(\"User exited\")\n \n # Ask for permission on sensitive actions\n proceed = interactor.sensitive_step_asker(\n action=\"Delete file\",\n control_text=\"important.docx\"\n )\n if not proceed:\n print(\"Action cancelled by user\")\n ```\n\n---\n\n### 6. Session Factory & Pool\n\n**SessionFactory** creates platform-specific sessions based on mode and configuration, while **SessionPool** manages batch execution.\n\n**Factory Creation Logic:**\n\n```mermaid\ngraph TB\n START[SessionFactory.create_session]\n \n PLATFORM{Platform?}\n MODE{Mode?}\n \n WNORMAL[Windows Session]\n WSERVICE[Windows ServiceSession]\n WFOLLOWER[Windows FollowerSession]\n WBATCH[Windows FromFileSession]\n WOPERATOR[Windows OpenAIOperatorSession]\n \n LNORMAL[Linux Session]\n LSERVICE[Linux ServiceSession]\n \n START --> PLATFORM\n PLATFORM -->|Windows| MODE\n PLATFORM -->|Linux| MODE\n \n MODE -->|normal| WNORMAL\n MODE -->|service| WSERVICE\n MODE -->|follower| WFOLLOWER\n MODE -->|batch_normal| WBATCH\n MODE -->|operator| WOPERATOR\n \n MODE -->|normal Linux| LNORMAL\n MODE -->|service Linux| LSERVICE\n \n style START fill:#e1f5ff\n style PLATFORM fill:#fff4e1\n style MODE fill:#f0ffe1\n style WNORMAL fill:#ffe1f5\n style LNORMAL fill:#ffe1f5\n```\n\n**Session Modes:**\n\n| Mode | Platform | Description | Input | Evaluation |\n|------|----------|-------------|-------|------------|\n| **normal** | Both | Interactive single-task | User input | Optional |\n| **service** | Both | WebSocket-controlled | Remote request | Optional |\n| **follower** | Windows | Replay recorded plan | Plan JSON file | Optional |\n| **batch_normal** | Windows | Multiple tasks from files | JSON folder | Per-task |\n| **operator** | Windows | OpenAI Operator API | User input | Optional |\n| **normal_operator** | Both | Interactive with operator | User input | Optional |\n\n!!!example \"SessionFactory Usage\"\n ```python\n from ufo.module.session_pool import SessionFactory, SessionPool\n \n factory = SessionFactory()\n \n # Interactive Windows session\n sessions = factory.create_session(\n task=\"task1\",\n mode=\"normal\",\n plan=\"\",\n request=\"Open calculator\"\n )\n \n # Batch Windows sessions from folder\n batch_sessions = factory.create_session(\n task=\"batch_task\",\n mode=\"batch_normal\",\n plan=\"./plans/\", # Folder with multiple .json files\n request=\"\"\n )\n \n # Run all sessions\n pool = SessionPool(batch_sessions)\n await pool.run_all()\n ```\n\n---\n\n## Cross-Platform Support\n\nThe module system provides a unified API while allowing platform-specific behavior through inheritance.\n\n**Platform Differences:**\n\n| Aspect | Windows | Linux |\n|--------|---------|-------|\n| **Agent Architecture** | HostAgent → AppAgent (two-tier) | LinuxAgent (single-tier) |\n| **HostAgent** | ✅ Used for planning | ❌ Not used |\n| **Session Base** | `WindowsBaseSession` | `LinuxBaseSession` |\n| **UI Automation** | UIA (pywinauto) | Custom automation |\n| **Service Mode** | `ServiceSession` | `LinuxServiceSession` |\n| **Evaluation** | ✅ Full support | ⚠️ Limited |\n| **Markdown Logs** | ✅ Supported | ⚠️ Planned |\n\n!!!example \"Platform Detection\"\n ```python\n import platform\n \n # Auto-detect platform\n current_platform = platform.system().lower() # 'windows' or 'linux'\n \n # Override platform\n sessions = factory.create_session(\n task=\"cross_platform_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"List files\",\n platform_override=\"linux\" # Force Linux session\n )\n ```\n\n---\n\n## Execution Flow\n\nUnderstanding how components interact during a complete user request:\n\n```mermaid\nsequenceDiagram\n participant User\n participant Interactor\n participant SessionFactory\n participant Session\n participant Round\n participant Context\n participant Agent\n participant Dispatcher\n participant MCP\n \n User->>Interactor: Start UFO\n Interactor->>User: Show welcome, ask request\n User->>Interactor: \"Open Notepad and type Hello\"\n \n Interactor->>SessionFactory: create_session(request)\n SessionFactory->>Session: __init__(task, request)\n Session->>Context: Initialize context\n Session->>Agent: Initialize agents\n \n Session->>Session: run()\n loop Until is_finished()\n Session->>Round: create_new_round()\n Round->>Context: Initialize round context\n \n loop Until round.is_finished()\n Round->>Agent: handle(context)\n Agent->>Agent: Process current state\n Agent->>Dispatcher: execute_commands([cmd1, cmd2])\n Dispatcher->>MCP: Route to MCP tools\n MCP-->>Dispatcher: Results\n Dispatcher-->>Agent: Results\n Agent->>Context: Update state\n \n Round->>Agent: Transition to next state\n Round->>Agent: Switch agent if needed\n end\n \n Round->>Round: capture_last_snapshot()\n Round-->>Session: Round complete\n \n Session->>Interactor: new_request()\n Interactor->>User: Continue or exit?\n User->>Interactor: \"N\"\n end\n \n Session->>Session: evaluation()\n Session->>Interactor: experience_asker()\n Interactor->>User: Save experience?\n User->>Interactor: Yes\n Session->>Session: experience_saver()\n Session-->>User: Session complete\n```\n\n---\n\n## File Structure\n\n```\nufo/module/\n├── __init__.py\n├── basic.py # BaseSession, BaseRound, FileWriter\n├── context.py # Context, ContextNames\n├── dispatcher.py # Command dispatchers\n├── interactor.py # User interaction functions\n├── session_pool.py # SessionFactory, SessionPool\n└── sessions/\n ├── __init__.py\n ├── platform_session.py # WindowsBaseSession, LinuxBaseSession\n ├── session.py # Session, FollowerSession, FromFileSession\n ├── service_session.py # ServiceSession\n ├── linux_session.py # LinuxSession, LinuxServiceSession\n └── plan_reader.py # PlanReader for follower mode\n```\n\n---\n\n## Key Design Patterns\n\n### 1. State Pattern\n\nAgents use the State pattern to manage transitions and determine control flow.\n\n```python\n# Agent state determines:\nnext_state = agent.state.next_state(agent)\nnext_agent = agent.state.next_agent(agent)\nis_done = agent.state.is_round_end()\n```\n\n### 2. Factory Pattern\n\nSessionFactory creates appropriate session types based on platform and mode.\n\n### 3. Command Pattern\n\nCommands encapsulate actions with parameters, enabling async execution and result tracking.\n\n### 4. Observer Pattern\n\nContext changes notify dependent components (implicit through shared state).\n\n---\n\n## Best Practices\n\n!!!tip \"Session Management\"\n - ✅ Always initialize context before creating rounds\n - ✅ Use `SessionFactory` for session creation (handles platform differences)\n - ✅ Attach command dispatcher to context early\n - ✅ Call `context._sync_round_values()` before accessing round-specific data\n - ❌ Don't access round context before round initialization\n\n!!!tip \"Round Execution\"\n - ✅ Let the state machine control agent transitions\n - ✅ Capture snapshots at subtask boundaries\n - ✅ Check `is_finished()` before each iteration\n - ❌ Don't bypass state transitions\n - ❌ Don't manually manipulate agent states\n\n!!!tip \"Context Usage\"\n - ✅ Use `ContextNames` enum for type-safe access\n - ✅ Update dictionaries with `update_dict()` for merging\n - ✅ Use properties (`current_round_cost`) for auto-synced values\n - ❌ Don't directly access `_context` dictionary\n - ❌ Don't store non-serializable objects without marking them\n\n!!!tip \"Command Dispatch\"\n - ✅ Always await `execute_commands()` (async)\n - ✅ Handle timeout exceptions gracefully\n - ✅ Check `ResultStatus` before using results\n - ❌ Don't ignore error results\n - ❌ Don't assume commands succeed\n\n---\n\n## Configuration\n\nKey configuration options from `ufo_config`:\n\n| Setting | Location | Default | Purpose |\n|---------|----------|---------|---------|\n| `max_step` | `system.max_step` | 50 | Max steps per session |\n| `max_round` | `system.max_round` | 10 | Max rounds per session |\n| `eva_session` | `system.eva_session` | `True` | Evaluate session |\n| `eva_round` | `system.eva_round` | `False` | Evaluate each round |\n| `save_experience` | `system.save_experience` | `\"ask\"` | When to save experience |\n| `log_to_markdown` | `system.log_to_markdown` | `True` | Generate markdown logs |\n| `save_ui_tree` | `system.save_ui_tree` | `True` | Save UI tree snapshots |\n\n---\n\n## Documentation Index\n\n| Document | Description |\n|----------|-------------|\n| [Session](./session.md) | Session lifecycle and management |\n| [Round](./round.md) | Round execution and orchestration |\n| [Context](./context.md) | State management and context names |\n| [Dispatcher](./dispatcher.md) | Command routing and execution |\n| [Session Pool](./session_pool.md) | Factory and batch management |\n| [Platform Sessions](./platform_sessions.md) | Windows/Linux implementations |\n\n---\n\n## Next Steps\n\n**Learning Path:**\n\n1. **Understand Sessions**: Read [Session](./session.md) to grasp the conversation model\n2. **Learn Rounds**: Study [Round](./round.md) to understand action execution\n3. **Master Context**: Review [Context](./context.md) for state management\n4. **Explore Dispatch**: Check [Dispatcher](./dispatcher.md) for command execution\n5. **Platform Specifics**: See [Platform Sessions](./platform_sessions.md) for Windows/Linux differences\n"
},
{
"path": "documents/docs/infrastructure/modules/platform_sessions.md",
"content": "# Platform-Specific Sessions\n\n**WindowsBaseSession** and **LinuxBaseSession** provide platform-specific base classes with fundamentally different agent architectures: Windows uses two-tier (HostAgent + AppAgent), while Linux uses single-tier (LinuxAgent only).\n\n**Quick Reference:**\n\n- Windows sessions? See [WindowsBaseSession](#windowsbasesession)\n- Linux sessions? See [LinuxBaseSession](#linuxbasesession)\n- Differences? See [Architecture Comparison](#architecture-comparison)\n- Choosing platform? See [Platform Selection](#platform-selection)\n\n---\n\n## Overview\n\nPlatform-specific base classes abstract OS-level differences:\n\n- **WindowsBaseSession**: Two-tier agent architecture with HostAgent coordination\n- **LinuxBaseSession**: Single-tier architecture with direct LinuxAgent control\n\n### Inheritance Hierarchy\n\n```mermaid\ngraph TB\n BASE[BaseSession Abstract Base]\n \n WIN_BASE[WindowsBaseSession Windows Platform]\n LINUX_BASE[LinuxBaseSession Linux Platform]\n \n SESSION[Session]\n SERVICE[ServiceSession]\n FOLLOWER[FollowerSession]\n FROMFILE[FromFileSession]\n OPERATOR[OpenAIOperatorSession]\n \n LINUX_SESS[LinuxSession]\n LINUX_SERVICE[LinuxServiceSession]\n \n BASE --> WIN_BASE\n BASE --> LINUX_BASE\n \n WIN_BASE --> SESSION\n WIN_BASE --> SERVICE\n WIN_BASE --> FOLLOWER\n WIN_BASE --> FROMFILE\n WIN_BASE --> OPERATOR\n \n LINUX_BASE --> LINUX_SESS\n LINUX_BASE --> LINUX_SERVICE\n \n style BASE fill:#e1f5ff\n style WIN_BASE fill:#fff4e1\n style LINUX_BASE fill:#f0ffe1\n style SESSION fill:#e1ffe1\n style LINUX_SESS fill:#e1ffe1\n```\n\n---\n\n## WindowsBaseSession\n\nWindows sessions use **HostAgent** for application selection and task planning, then **AppAgent** for in-application execution. This provides a two-tier agent architecture.\n\n### Agent Initialization\n\n```python\ndef _init_agents(self) -> None:\n \"\"\"Initialize Windows-specific agents, including the HostAgent.\"\"\"\n \n self._host_agent: HostAgent = AgentFactory.create_agent(\n \"host\",\n \"HostAgent\",\n ufo_config.host_agent.visual_mode,\n ufo_config.system.HOSTAGENT_PROMPT,\n ufo_config.system.HOSTAGENT_EXAMPLE_PROMPT,\n ufo_config.system.API_PROMPT,\n )\n```\n\n**What's Created:**\n\n| Component | Type | Purpose |\n|-----------|------|---------|\n| `_host_agent` | `HostAgent` | Application selection and task coordination |\n| Visual Mode | `bool` | Enable screenshot-based reasoning |\n| Prompts | `str` | HostAgent behavior templates |\n\n### Two-Tier Execution Flow\n\n```mermaid\nsequenceDiagram\n participant U as User\n participant S as WindowsBaseSession\n participant H as HostAgent\n participant A as AppAgent\n participant UI as Windows UI\n \n U->>S: Request: \"Send email to John\"\n S->>H: Initialize HostAgent\n H->>H: Observe desktop\n H->>UI: Screenshot desktop\n UI-->>H: Desktop image\n \n H->>H: LLM Decision\n Note over H: \"Best app: Outlook\"\n \n H->>S: Select application: Outlook\n S->>A: Create AppAgent for Outlook\n \n A->>UI: Observe Outlook window\n UI-->>A: Outlook screenshot + controls\n \n A->>A: LLM Planning\n Note over A: Plan: Click \"New Email\" Type recipient Type subject Click \"Send\"\n \n loop Execute plan steps\n A->>UI: Execute command\n UI-->>A: Result\n end\n \n A->>S: Task complete\n S->>U: Email sent\n```\n\n### Agent Switching Logic\n\n**HostAgent selects applications:**\n\n```python\n# HostAgent decision\nselected_app = host_agent.handle(context)\n# Result: \"Outlook\"\n\n# Session switches to AppAgent\napp_agent = create_app_agent(\"Outlook\")\ncontext.set(ContextNames.APPLICATION_PROCESS_NAME, \"OUTLOOK.EXE\")\n```\n\n**AppAgent may request HostAgent:**\n\n```python\n# AppAgent realizes need different app\nif need_different_app:\n # Switch back to HostAgent\n agent = host_agent\n # HostAgent selects new app\n```\n\n### Reset Behavior\n\n```python\ndef reset(self):\n \"\"\"Reset the session state for a new session.\"\"\"\n self._host_agent.set_state(self._host_agent.default_state)\n```\n\n**Reset restores:**\n- HostAgent to initial state\n- Clears previous application selections\n- Ready for new task\n\n---\n\n## LinuxBaseSession\n\nLinux sessions use **LinuxAgent** directly without HostAgent intermediary, providing simpler but less flexible architecture. This is a single-tier model.\n\n### Agent Initialization\n\n```python\ndef _init_agents(self) -> None:\n \"\"\"Initialize Linux-specific agents.\"\"\"\n \n # No host agent for Linux\n self._host_agent = None\n \n # Create LinuxAgent directly\n self._agent: LinuxAgent = AgentFactory.create_agent(\n \"LinuxAgent\",\n \"LinuxAgent\",\n ufo_config.system.third_party_agent_config[\"LinuxAgent\"][\"APPAGENT_PROMPT\"],\n ufo_config.system.third_party_agent_config[\"LinuxAgent\"][\"APPAGENT_EXAMPLE_PROMPT\"],\n )\n```\n\n**What's Created:**\n\n| Component | Type | Purpose |\n|-----------|------|---------|\n| `_host_agent` | `None` | **Not used in Linux** |\n| `_agent` | `LinuxAgent` | Direct application control |\n| Prompts | `str` | LinuxAgent behavior templates |\n\n### Single-Tier Execution Flow\n\n```mermaid\nsequenceDiagram\n participant U as User\n participant S as LinuxBaseSession\n participant L as LinuxAgent\n participant UI as Linux UI\n \n U->>S: Request: \"Open gedit and type Hello\"\n S->>L: Initialize LinuxAgent\n \n L->>UI: Observe desktop\n UI-->>L: Desktop state\n \n L->>L: LLM Decision\n Note over L: \"Launch gedit Type text\"\n \n L->>UI: Execute: launch gedit\n UI-->>L: gedit opened\n \n L->>UI: Execute: type \"Hello\"\n UI-->>L: Text typed\n \n L->>S: Task complete\n S->>U: Done\n```\n\n**No Agent Switching:**\n\n- LinuxAgent handles entire workflow\n- Application specified upfront or agent decides\n- Simpler execution model\n\n### Feature Limitations\n\nSome methods are not yet implemented:\n\n```python\ndef evaluation(self) -> None:\n \"\"\"Evaluation logic for Linux sessions.\"\"\"\n self.logger.warning(\"Evaluation not yet implemented for Linux sessions.\")\n pass\n\ndef save_log_to_markdown(self) -> None:\n \"\"\"Save the log of the session to markdown file.\"\"\"\n self.logger.warning(\"Markdown logging not yet implemented for Linux sessions.\")\n pass\n```\n\n!!!warning \"Coming Soon\"\n Full evaluation and markdown logging support for Linux sessions is planned for future releases.\n\n### Reset Behavior\n\n```python\ndef reset(self) -> None:\n \"\"\"Reset the session state for a new session.\"\"\"\n self._agent.set_state(self._agent.default_state)\n```\n\n**Reset restores:**\n- LinuxAgent to initial state\n- Ready for new task\n\n---\n\n## Architecture Comparison\n\n### High-Level Differences\n\n```mermaid\ngraph TB\n subgraph \"Windows Architecture (Two-Tier)\"\n WIN_USER[User Request]\n WIN_HOST[HostAgent Application Selector]\n WIN_APP1[AppAgent Word]\n WIN_APP2[AppAgent Excel]\n WIN_APP3[AppAgent Outlook]\n \n WIN_USER --> WIN_HOST\n WIN_HOST -->|Select app| WIN_APP1\n WIN_HOST -->|Switch app| WIN_APP2\n WIN_HOST -->|Switch app| WIN_APP3\n end\n \n subgraph \"Linux Architecture (Single-Tier)\"\n LINUX_USER[User Request]\n LINUX_AGENT[LinuxAgent Direct Control]\n LINUX_APP[gedit/firefox/etc]\n \n LINUX_USER --> LINUX_AGENT\n LINUX_AGENT --> LINUX_APP\n end\n \n style WIN_HOST fill:#fff4e1\n style WIN_APP1 fill:#e1ffe1\n style LINUX_AGENT fill:#f0ffe1\n```\n\n### Feature Matrix\n\n| Feature | Windows | Linux | Notes |\n|---------|---------|-------|-------|\n| **HostAgent** | ✅ Yes | ❌ No | Windows uses HostAgent for app selection |\n| **AppAgent** | ✅ Yes | ❌ No | Windows creates AppAgent per application |\n| **LinuxAgent** | ❌ No | ✅ Yes | Linux uses LinuxAgent directly |\n| **Agent Switching** | ✅ Yes | ❌ No | Windows can switch between apps mid-task |\n| **Multi-App Tasks** | ✅ Native | ⚠️ Limited | Windows handles multi-app naturally |\n| **Execution Modes** | ✅ All 7 | ⚠️ 3 modes | Windows supports all modes |\n| **Evaluation** | ✅ Yes | 🚧 Planned | Linux evaluation in development |\n| **Markdown Logs** | ✅ Yes | 🚧 Planned | Linux markdown logging in development |\n| **UI Automation** | UIA | Platform tools | Different automation backends |\n\n### Execution Comparison\n\n**Windows Multi-Application Task:**\n\n```python\n# Request: \"Copy data from Excel to Word\"\n\n# Round 1\nHostAgent: Select Excel → AppAgent(Excel): Copy data\n# Round 2 \nHostAgent: Select Word → AppAgent(Word): Paste data\n\n# Agent switching handled automatically\n```\n\n**Linux Single-Application Task:**\n\n```python\n# Request: \"Open gedit and type text\"\n\n# Single round\nLinuxAgent: Launch gedit → Type text\n\n# No agent switching, direct execution\n```\n\n---\n\n## Platform Selection\n\n### Automatic Detection\n\nSessionFactory automatically detects platform:\n\n```python\nfrom ufo.module.session_pool import SessionFactory\nimport platform\n\nfactory = SessionFactory()\n\n# Auto-detects: \"windows\" or \"linux\"\nsessions = factory.create_session(\n task=\"cross_platform_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Open text editor\"\n)\n\n# Correct base class automatically selected:\n# - Windows: Session extends WindowsBaseSession\n# - Linux: LinuxSession extends LinuxBaseSession\n```\n\n### Manual Override\n\nFor testing or special cases:\n\n```python\n# Force Windows session on Linux machine\nsessions = factory.create_session(\n task=\"test_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Test request\",\n platform_override=\"windows\"\n)\n\n# Force Linux session on Windows machine\nsessions = factory.create_session(\n task=\"test_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Test request\",\n platform_override=\"linux\"\n)\n```\n\n!!!warning \"Override Use Cases\"\n Only use `platform_override` for:\n - Testing cross-platform code\n - Development without target OS\n - Generating plans for other platforms\n \n Never use in production!\n\n---\n\n## Migration Guide\n\n### Porting Tasks Windows → Linux\n\n**Considerations:**\n\n1. **No HostAgent**: Specify application upfront or in request\n2. **Single-tier**: Cannot switch applications mid-task\n3. **Limited modes**: Only `normal`, `normal_operator`, `service`\n\n**Example:**\n\n**Windows Request:**\n```python\n\"Send an email to John and create a calendar event\"\n# HostAgent selects Outlook → AppAgent sends email\n# HostAgent switches to Calendar → AppAgent creates event\n```\n\n**Linux Request (Split):**\n```python\n# Request 1: Email only\n\"Send an email to John using Thunderbird\"\n# LinuxAgent(Thunderbird): Send email\n\n# Request 2: Calendar separately\n\"Create a calendar event in GNOME Calendar\"\n# LinuxAgent(Calendar): Create event\n```\n\n### Configuration Differences\n\n**Windows Configuration:**\n\n```yaml\n# config/ufo/config.yaml\nhost_agent:\n visual_mode: true\nsystem:\n HOSTAGENT_PROMPT: \"prompts/host_agent.yaml\"\n APPAGENT_PROMPT: \"prompts/app_agent.yaml\"\n```\n\n**Linux Configuration:**\n\n```yaml\n# config/ufo/config.yaml \nsystem:\n third_party_agent_config:\n LinuxAgent:\n APPAGENT_PROMPT: \"prompts/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"prompts/linux_examples.yaml\"\n```\n\n---\n\n## Best Practices\n\n### Windows Sessions\n\n!!!tip \"Leverage Two-Tier Architecture\"\n - ✅ Use HostAgent for complex multi-app workflows\n - ✅ Let HostAgent decide application selection\n - ✅ Design tasks that benefit from app switching\n - ❌ Don't micromanage app selection\n - ❌ Don't bypass HostAgent for multi-app tasks\n\n### Linux Sessions\n\n!!!success \"Work Within Single-Tier Model\"\n - ✅ Specify application in request if known\n - ✅ Keep tasks focused on single application\n - ✅ Split multi-app workflows into multiple sessions\n - ❌ Don't expect automatic app switching\n - ❌ Don't assume HostAgent features available\n\n### Cross-Platform Development\n\n!!!warning \"Platform Awareness\"\n - ✅ Test on both platforms if deploying cross-platform\n - ✅ Use platform detection, not hardcoded assumptions\n - ✅ Handle platform-specific features gracefully\n - ✅ Document platform limitations\n - ❌ Don't assume identical behavior\n - ❌ Don't use platform_override in production\n\n---\n\n## Reference\n\n### WindowsBaseSession\n\n::: module.sessions.platform_session.WindowsBaseSession\n\n### LinuxBaseSession\n\n::: module.sessions.platform_session.LinuxBaseSession\n\n---\n\n## See Also\n\n- [Session](./session.md) - Session lifecycle and types\n- [Session Factory](./session_pool.md) - Platform-aware session creation\n- [Overview](./overview.md) - Module system architecture\n- [Round](./round.md) - Agent orchestration in rounds\n"
},
{
"path": "documents/docs/infrastructure/modules/round.md",
"content": "# Round\n\nA **Round** is a single request-response cycle within a Session, orchestrating agents through a state machine to execute commands until the user's request is fulfilled.\n\n**Quick Reference:**\n\n- Lifecycle? See [Round Lifecycle](#round-lifecycle)\n- State machine? See [State Machine](#state-machine)\n- Agent switching? See [Agent Orchestration](#agent-orchestration)\n- Snapshots? See [Snapshot Capture](#snapshot-capture)\n\n---\n\n## Overview\n\nA `Round` represents one complete request-response interaction:\n\n- **Input**: User request (e.g., \"Send an email to John\")\n- **Processing**: Agent state machine execution\n- **Output**: Request fulfilled or error state\n\n### Round in Context\n\n```mermaid\ngraph TB\n subgraph \"Session Scope\"\n SESS[Session]\n REQ1[Request 1]\n REQ2[Request 2]\n REQ3[Request 3]\n end\n \n subgraph \"Round Scope (One Request)\"\n ROUND[Round Instance]\n CTX[Shared Context]\n INIT[Initialize]\n LOOP[Execution Loop]\n FINISH[Finish Condition]\n end\n \n subgraph \"Execution Loop Detail\"\n HANDLE[agent.handle Generate & Execute]\n NEXT_STATE[next_state State Transition]\n NEXT_AGENT[next_agent Agent Switching]\n SUBTASK{Subtask End?}\n SNAPSHOT[capture_last_snapshot]\n end\n \n SESS --> REQ1\n SESS --> REQ2\n SESS --> REQ3\n \n REQ1 --> ROUND\n ROUND --> CTX\n ROUND --> INIT\n INIT --> LOOP\n \n LOOP --> HANDLE\n HANDLE --> NEXT_STATE\n NEXT_STATE --> NEXT_AGENT\n NEXT_AGENT --> SUBTASK\n \n SUBTASK -->|Yes| SNAPSHOT\n SNAPSHOT --> FINISH\n SUBTASK -->|No| FINISH\n \n FINISH -->|Not finished| HANDLE\n FINISH -->|Finished| REQ2\n \n style ROUND fill:#e1f5ff\n style HANDLE fill:#f0ffe1\n style SNAPSHOT fill:#fff4e1\n style FINISH fill:#ffe1f5\n```\n\n---\n\n## Round Lifecycle\n\n### State Machine Overview\n\n```mermaid\nstateDiagram-v2\n [*] --> Initialized: create_new_round()\n Initialized --> Running: run()\n \n Running --> AgentHandle: agent.handle(context)\n AgentHandle --> StateTransition: generate actions\n StateTransition --> AgentSwitch: determine next\n AgentSwitch --> SubtaskCheck: update agent\n \n SubtaskCheck --> CaptureSnapshot: if subtask_end\n SubtaskCheck --> FinishCheck: if not subtask_end\n CaptureSnapshot --> FinishCheck: snapshot saved\n \n FinishCheck --> AgentHandle: not finished\n FinishCheck --> FinalSnapshot: finished\n \n FinalSnapshot --> Evaluation: if enabled\n Evaluation --> [*]: round complete\n FinalSnapshot --> [*]: skip evaluation\n```\n\n### Core Execution Loop\n\n```python\nasync def run(self) -> None:\n \"\"\"\n Run the round asynchronously.\n \"\"\"\n \n while not self.is_finished():\n # 1. Agent processes current state\n await self.agent.handle(self.context)\n \n # 2. State machine transitions\n self.state = self.agent.state.next_state(self.agent)\n \n # 3. Agent switching (HostAgent ↔ AppAgent)\n self.agent = self.agent.state.next_agent(self.agent)\n self.agent.set_state(self.state)\n \n # 4. Snapshot capture at subtask boundaries\n if self.state.is_subtask_end():\n time.sleep(configs[\"SLEEP_TIME\"])\n await self.capture_last_snapshot(sub_round_id=self.subtask_amount)\n self.subtask_amount += 1\n \n # 5. Add request to blackboard\n self.agent.blackboard.add_requests(\n {f\"request_{self.id}\": self.request}\n )\n \n # 6. Final snapshot\n if self.application_window is not None:\n await self.capture_last_snapshot()\n \n # 7. Evaluation (optional)\n if self._should_evaluate:\n await self.evaluation()\n```\n\n---\n\n## Lifecycle Stages\n\n### 1. Initialization\n\nCreated by session's `create_new_round()`:\n\n```python\nround = Round(\n task=\"email_task\",\n context=session.context,\n request=\"Send an email to John\",\n id=0 # Round number\n)\n```\n\n**Initialization sets:**\n\n| Property | Source | Description |\n|----------|--------|-------------|\n| `task` | Session | Task name for logging |\n| `context` | Session | Shared context object |\n| `request` | User input | Natural language request |\n| `id` | Round counter | Sequential round number |\n| `agent` | Initial agent | Usually HostAgent (Windows) or LinuxAgent |\n| `state` | Initial state | Usually START state |\n\n### 2. Agent Handle\n\nEach loop iteration calls `agent.handle(context)`:\n\n```python\nawait self.agent.handle(self.context)\n```\n\n**What happens:**\n\n1. **Observation**: Agent observes UI state\n2. **Reasoning**: LLM generates plan and actions\n3. **Action**: Commands sent to dispatcher\n4. **Execution**: Commands executed locally or remotely\n5. **Results**: Results stored in context\n\n**Example Flow:**\n\n```mermaid\nsequenceDiagram\n participant R as Round\n participant A as Agent (HostAgent)\n participant LLM as Language Model\n participant D as Dispatcher\n participant UI as UI System\n \n R->>A: handle(context)\n A->>UI: Observe desktop\n UI-->>A: Screenshot + control tree\n \n A->>LLM: Generate plan\n Note over LLM: Request: \"Send email to John\" Observation: Desktop with Outlook icon\n LLM-->>A: Action: open_application(\"Outlook\")\n \n A->>D: execute_commands([open_app_cmd])\n D->>UI: Click Outlook icon\n UI-->>D: Result: Outlook opened\n D-->>A: ResultStatus.SUCCESS\n \n A->>R: Update context with results\n```\n\n### 3. State Transition\n\nAfter agent handling, state machine transitions:\n\n```python\nself.state = self.agent.state.next_state(self.agent)\n```\n\n**State Transitions:**\n\n| Current State | Condition | Next State |\n|---------------|-----------|------------|\n| **START** | Initial | **CONTINUE** |\n| **CONTINUE** | More actions needed | **CONTINUE** |\n| **CONTINUE** | Task complete | **FINISH** |\n| **CONTINUE** | Error occurred | **ERROR** |\n| **FINISH** | Always | Round ends |\n| **ERROR** | Always | Round ends |\n\n**State Diagram:**\n\n```mermaid\nstateDiagram-v2\n [*] --> START\n START --> CONTINUE: First action\n CONTINUE --> CONTINUE: More actions\n CONTINUE --> FINISH: Task complete\n CONTINUE --> ERROR: Error occurred\n FINISH --> [*]\n ERROR --> [*]\n```\n\n### 4. Agent Switching\n\nDetermine which agent handles next step:\n\n```python\nself.agent = self.agent.state.next_agent(self.agent)\nself.agent.set_state(self.state)\n```\n\n**Agent Switching Logic (Windows):**\n\n| Current Agent | Condition | Next Agent |\n|---------------|-----------|------------|\n| **HostAgent** | Application selected | **AppAgent** |\n| **AppAgent** | Need different app | **HostAgent** |\n| **AppAgent** | Same app continues | **AppAgent** |\n| **HostAgent** | Task complete | **HostAgent** (finish) |\n\n**Agent Switching Logic (Linux):**\n\n| Current Agent | Condition | Next Agent |\n|---------------|-----------|------------|\n| **LinuxAgent** | Always | **LinuxAgent** (no switching) |\n\n**Switching Example:**\n\n```mermaid\nsequenceDiagram\n participant R as Round\n participant H as HostAgent\n participant A as AppAgent\n \n R->>H: handle() - Select app\n H-->>R: Application: Outlook\n \n Note over R: Agent switch: HostAgent → AppAgent\n \n R->>A: handle() - Compose email\n A-->>R: Commands executed\n \n R->>A: handle() - Send email\n A-->>R: Task complete\n \n Note over R: State: FINISH\n```\n\n### 5. Subtask Boundary Capture\n\nCapture snapshot when subtask ends:\n\n```python\nif self.state.is_subtask_end():\n time.sleep(configs[\"SLEEP_TIME\"]) # Let UI settle\n await self.capture_last_snapshot(sub_round_id=self.subtask_amount)\n self.subtask_amount += 1\n```\n\n**Subtask End Conditions:**\n\n- Agent switched (HostAgent ↔ AppAgent)\n- Major UI change detected\n- Explicit subtask boundary in plan\n\n**Captured Data:**\n\n1. **Window screenshot**: `action_round_{id}_sub_round_{sub_id}_final.png`\n2. **UI tree** (if enabled): `ui_tree_round_{id}_sub_round_{sub_id}_final.json`\n3. **Desktop screenshot** (if enabled): `desktop_round_{id}_sub_round_{sub_id}_final.png`\n\n### 6. Finish Check\n\n```python\ndef is_finished(self) -> bool:\n \"\"\"Check if round is complete.\"\"\"\n return self.state in [AgentState.FINISH, AgentState.ERROR]\n```\n\nLoop continues until state is `FINISH` or `ERROR`.\n\n### 7. Final Snapshot\n\nAfter loop exits:\n\n```python\nif self.application_window is not None:\n await self.capture_last_snapshot()\n```\n\n**Final snapshot** captures the end state of the application for logging and evaluation.\n\n### 8. Evaluation\n\nOptional evaluation of round success:\n\n```python\nif self._should_evaluate:\n await self.evaluation()\n```\n\n**Evaluation checks:**\n- Was the request fulfilled?\n- Quality of actions taken\n- Efficiency metrics\n\n---\n\n## State Machine\n\n### AgentState Enum\n\n```python\nclass AgentState(Enum):\n START = \"START\"\n CONTINUE = \"CONTINUE\"\n FINISH = \"FINISH\"\n ERROR = \"ERROR\"\n```\n\n### State Behaviors\n\n| State | Meaning | Transitions To |\n|-------|---------|----------------|\n| **START** | Initial state | CONTINUE |\n| **CONTINUE** | Actively processing | CONTINUE, FINISH, ERROR |\n| **FINISH** | Successfully complete | Round ends |\n| **ERROR** | Fatal error occurred | Round ends |\n\n### State Methods\n\nEach state implements:\n\n```python\nclass StateInterface:\n def next_state(self, agent) -> AgentState:\n \"\"\"Determine next state based on agent's decision.\"\"\"\n pass\n \n def next_agent(self, agent) -> Agent:\n \"\"\"Determine next agent to handle the request.\"\"\"\n pass\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if current state marks subtask boundary.\"\"\"\n pass\n```\n\n---\n\n## Agent Orchestration\n\n### Windows Two-Tier Architecture\n\n```mermaid\nsequenceDiagram\n participant U as User Request\n participant R as Round\n participant H as HostAgent\n participant A as AppAgent\n participant UI as UI System\n \n U->>R: \"Send email to John\"\n R->>H: handle() - Select application\n H->>UI: Observe desktop\n UI-->>H: Screenshot of desktop\n H->>H: Decide: Outlook\n H-->>R: Switch to AppAgent for Outlook\n \n R->>A: handle() - Compose email\n A->>UI: Observe Outlook window\n UI-->>A: Screenshot + control tree\n A->>A: Plan: Click \"New Email\"\n A->>UI: Click command\n UI-->>A: New email window opened\n A-->>R: Continue\n \n R->>A: handle() - Fill recipient\n A->>UI: Type \"john@example.com\"\n UI-->>A: Recipient filled\n A-->>R: Continue\n \n R->>A: handle() - Click Send\n A->>UI: Click \"Send\" button\n UI-->>A: Email sent\n A-->>R: Finish\n \n R-->>U: Request complete\n```\n\n### Linux Single-Tier Architecture\n\n```mermaid\nsequenceDiagram\n participant U as User Request\n participant R as Round\n participant L as LinuxAgent\n participant UI as UI System\n \n U->>R: \"Open gedit and type Hello\"\n R->>L: handle() - Open application\n L->>UI: Observe desktop\n UI-->>L: Desktop state\n L->>L: Plan: Open gedit\n L->>UI: Launch gedit command\n UI-->>L: gedit opened\n L-->>R: Continue\n \n R->>L: handle() - Type text\n L->>UI: Type \"Hello\"\n UI-->>L: Text typed\n L-->>R: Finish\n \n R-->>U: Request complete\n```\n\n---\n\n## Snapshot Capture\n\n### capture_last_snapshot()\n\n```python\nasync def capture_last_snapshot(self, sub_round_id: Optional[int] = None) -> None\n```\n\n**Purpose**: Capture UI state for logging, debugging, and evaluation.\n\n**Captured Artifacts:**\n\n| Artifact | File Pattern | Purpose |\n|----------|--------------|---------|\n| **Window Screenshot** | `action_round_{id}_final.png` | Visual state |\n| **Subtask Screenshot** | `action_round_{id}_sub_round_{sub_id}_final.png` | Subtask boundary |\n| **UI Tree** | `ui_tree_round_{id}_final.json` | Control structure |\n| **Desktop Screenshot** | `desktop_round_{id}_final.png` | Full desktop (if enabled) |\n\n**Example Output:**\n\n```\nlogs/task_name/\n├── action_round_0_sub_round_0_final.png ← After HostAgent selects Outlook\n├── action_round_0_sub_round_1_final.png ← After AppAgent composes email\n├── action_round_0_final.png ← Final state after sending\n├── ui_trees/\n│ ├── ui_tree_round_0_sub_round_0_final.json\n│ ├── ui_tree_round_0_sub_round_1_final.json\n│ └── ui_tree_round_0_final.json\n└── desktop_round_0_final.png\n```\n\n### save_ui_tree()\n\n```python\nasync def save_ui_tree(self, save_path: str)\n```\n\nSaves the control tree as JSON for analysis:\n\n```json\n{\n \"root\": {\n \"control_type\": \"Window\",\n \"name\": \"Outlook\",\n \"children\": [\n {\n \"control_type\": \"Button\",\n \"name\": \"New Email\",\n \"automation_id\": \"btn_new_email\",\n \"bounding_box\": [100, 50, 150, 30]\n }\n ]\n }\n}\n```\n\n---\n\n## Properties\n\n### Auto-Syncing Properties\n\nProperties that sync with context automatically:\n\n```python\n@property\ndef step(self) -> int:\n \"\"\"Current step number in this round.\"\"\"\n return self._context.get(ContextNames.ROUND_STEP).get(self.id, 0)\n\n@property\ndef cost(self) -> float:\n \"\"\"Total cost for this round.\"\"\"\n return self._context.get(ContextNames.ROUND_COST).get(self.id, 0)\n\n@property\ndef subtask_amount(self) -> int:\n \"\"\"Number of subtasks completed.\"\"\"\n return self._context.get(ContextNames.ROUND_SUBTASK_AMOUNT).get(self.id, 0)\n\n@subtask_amount.setter\ndef subtask_amount(self, value: int) -> None:\n \"\"\"Set subtask amount in context.\"\"\"\n self._context.current_round_subtask_amount = value\n```\n\n### Static Properties\n\n```python\n@property\ndef request(self) -> str:\n \"\"\"User request for this round.\"\"\"\n return self._request\n\n@property\ndef id(self) -> int:\n \"\"\"Round number (sequential).\"\"\"\n return self._id\n\n@property\ndef context(self) -> Context:\n \"\"\"Shared context object.\"\"\"\n return self._context\n```\n\n---\n\n## Cost Tracking\n\n### print_cost()\n\nDisplay round cost after completion:\n\n```python\ndef print_cost(self) -> None:\n \"\"\"Print the total cost of the round.\"\"\"\n \n total_cost = self.cost\n if isinstance(total_cost, float):\n formatted_cost = \"${:.2f}\".format(total_cost)\n console.print(\n f\"💰 Request total cost for current round is {formatted_cost}\",\n style=\"yellow\",\n )\n```\n\n**Output Example:**\n\n```\n💰 Request total cost for current round is $0.42\n```\n\n**Cost Components:**\n\n- LLM API calls (HostAgent + AppAgent)\n- Vision model calls (screenshot analysis)\n- Embedding model calls (if used)\n\n---\n\n## Error Handling\n\n### Error States\n\nRounds can end in error state:\n\n```python\nif agent_fails:\n self.state = AgentState.ERROR\n # Round exits loop with ERROR state\n```\n\n### Common Error Scenarios\n\n| Error Type | Trigger | Handling |\n|------------|---------|----------|\n| **Timeout** | Command execution timeout | Set ERROR state |\n| **Agent Failure** | LLM returns invalid plan | Set ERROR state |\n| **UI Not Found** | Element doesn't exist | Retry or ERROR |\n| **Connection Lost** | Dispatcher disconnected | Set ERROR state |\n\n### Error Recovery\n\n```python\ntry:\n await self.agent.handle(self.context)\nexcept AgentError as e:\n logger.error(f\"Agent handle failed: {e}\")\n self.state = AgentState.ERROR\n # Loop exits\n```\n\n---\n\n## Configuration\n\n### Round Behavior Settings\n\n| Setting | Type | Purpose |\n|---------|------|---------|\n| `eva_round` | `bool` | Enable round evaluation |\n| `SLEEP_TIME` | `float` | Wait time before snapshot (seconds) |\n| `save_ui_tree` | `bool` | Save UI trees |\n| `save_full_screen` | `bool` | Save desktop screenshots |\n\n**Example Configuration:**\n\n```yaml\n# config/ufo/config.yaml\nsystem:\n eva_round: true\n SLEEP_TIME: 0.5\n save_ui_tree: true\n save_full_screen: false\n```\n\n---\n\n## Best Practices\n\n### Efficient Round Execution\n\n!!!tip \"Performance Tips\"\n - ✅ Keep agent prompts concise\n - ✅ Use appropriate timeouts for commands\n - ✅ Disable full desktop screenshots unless needed\n - ✅ Capture UI trees only for debugging\n - ❌ Don't set SLEEP_TIME too high\n - ❌ Don't enable all logging in production\n\n### State Machine Design\n\n!!!success \"Clean State Management\"\n - ✅ Each state should have clear purpose\n - ✅ Transitions should be deterministic\n - ✅ Error states should be terminal\n - ✅ Subtask boundaries should be meaningful\n - ❌ Don't create circular state loops\n - ❌ Don't mix state logic with business logic\n\n---\n\n## Reference\n\n### BaseRound\n\n::: module.basic.BaseRound\n\n---\n\n## See Also\n\n- [Session](./session.md) - Multi-round conversation management\n- [Context](./context.md) - Shared state across rounds\n- [Dispatcher](./dispatcher.md) - Command execution\n- [Overview](./overview.md) - Module system architecture"
},
{
"path": "documents/docs/infrastructure/modules/session.md",
"content": "# Session\n\nA **Session** is a continuous conversation instance between the user and UFO, managing multiple rounds of interaction from initial request to task completion across different execution modes and platforms.\n\n**Quick Reference:**\n\n- Session types? See [Session Types](#session-types)\n- Lifecycle? See [Session Lifecycle](#session-lifecycle)\n- Mode differences? See [Execution Modes](#execution-modes)\n- Platform differences? See [Platform-Specific Sessions](#platform-specific-sessions)\n\n---\n\n## Overview\n\nA `Session` represents a complete conversation workflow, containing one or more `Rounds` of agent execution. Sessions manage:\n\n1. **Context**: Shared state across all rounds\n2. **Agents**: HostAgent and AppAgent (or LinuxAgent)\n3. **Rounds**: Individual request-response cycles\n4. **Evaluation**: Optional task completion assessment\n5. **Experience**: Learning from successful workflows\n\n### Relationship: Session vs Round\n\n```mermaid\ngraph TB\n subgraph \"Session (Conversation)\"\n S[Session Instance]\n CTX[Context Shared State]\n R1[Round 1 Request 1]\n R2[Round 2 Request 2]\n R3[Round 3 Request 3]\n EVAL[Evaluation Optional]\n end\n \n subgraph \"Round 1 Details\"\n HOST1[HostAgent]\n APP1[AppAgent]\n CMD1[Commands]\n end\n \n subgraph \"Round 2 Details\"\n HOST2[HostAgent]\n APP2[AppAgent]\n CMD2[Commands]\n end\n \n S --> CTX\n S --> R1\n S --> R2\n S --> R3\n S --> EVAL\n \n R1 -.shares.-> CTX\n R2 -.shares.-> CTX\n R3 -.shares.-> CTX\n \n R1 --> HOST1\n HOST1 --> APP1\n APP1 --> CMD1\n \n R2 --> HOST2\n HOST2 --> APP2\n APP2 --> CMD2\n \n style S fill:#e1f5ff\n style CTX fill:#fff4e1\n style R1 fill:#f0ffe1\n style R2 fill:#f0ffe1\n style R3 fill:#f0ffe1\n style EVAL fill:#ffe1f5\n```\n\n---\n\n## Session Types\n\nUFO supports **7 session types** across Windows and Linux platforms:\n\n| Session Type | Platform | Mode | Description |\n|--------------|----------|------|-------------|\n| **Session** | Windows | `normal`, `normal_operator` | Interactive with HostAgent |\n| **ServiceSession** | Windows | `service` | WebSocket-controlled via AIP |\n| **FollowerSession** | Windows | `follower` | Replays saved plans |\n| **FromFileSession** | Windows | `batch_normal` | Executes from request files |\n| **OpenAIOperatorSession** | Windows | `operator` | Pure operator mode |\n| **LinuxSession** | Linux | `normal`, `normal_operator` | Interactive without HostAgent |\n| **LinuxServiceSession** | Linux | `service` | WebSocket-controlled on Linux |\n\n### Class Hierarchy\n\n```mermaid\ngraph TB\n BASE[BaseSession Abstract]\n \n WIN_BASE[WindowsBaseSession with HostAgent]\n LINUX_BASE[LinuxBaseSession without HostAgent]\n \n SESSION[Session Interactive]\n SERVICE[ServiceSession WebSocket]\n FOLLOWER[FollowerSession Plan Replay]\n FROMFILE[FromFileSession Batch]\n OPERATOR[OpenAIOperatorSession Operator]\n \n LINUX_SESS[LinuxSession Interactive]\n LINUX_SERVICE[LinuxServiceSession WebSocket]\n \n BASE --> WIN_BASE\n BASE --> LINUX_BASE\n \n WIN_BASE --> SESSION\n WIN_BASE --> SERVICE\n WIN_BASE --> FOLLOWER\n WIN_BASE --> FROMFILE\n WIN_BASE --> OPERATOR\n \n LINUX_BASE --> LINUX_SESS\n LINUX_BASE --> LINUX_SERVICE\n \n style BASE fill:#e1f5ff\n style WIN_BASE fill:#fff4e1\n style LINUX_BASE fill:#f0ffe1\n style SESSION fill:#e1ffe1\n style LINUX_SESS fill:#e1ffe1\n```\n\n!!!note \"Platform Base Classes\"\n - `WindowsBaseSession`: Creates HostAgent, supports two-tier architecture\n - `LinuxBaseSession`: Single-tier architecture with LinuxAgent only\n\n---\n\n## Session Lifecycle\n\n### Standard Lifecycle\n\n```mermaid\nstateDiagram-v2\n [*] --> Initialized: __init__\n Initialized --> ContextReady: _init_context\n ContextReady --> Running: run()\n \n Running --> RoundCreate: create_new_round\n RoundCreate --> RoundExecute: round.run()\n RoundExecute --> RoundComplete: Round finishes\n \n RoundComplete --> CheckMore: is_finished?\n CheckMore --> RoundCreate: More requests\n CheckMore --> Snapshot: No more requests\n \n Snapshot --> Evaluation: capture_last_snapshot\n Evaluation --> CostPrint: evaluation() if enabled\n CostPrint --> [*]: Session complete\n```\n\n### Core Execution Loop\n\nThe main session logic:\n\n```python\nasync def run(self) -> None:\n \"\"\"\n Run the session.\n \"\"\"\n \n while not self.is_finished():\n # Create new round for each request\n round = self.create_new_round()\n if round is None:\n break\n \n # Execute the round\n await round.run()\n \n # Capture final state\n if self.application_window is not None:\n await self.capture_last_snapshot()\n \n # Evaluate if configured\n if self._should_evaluate and not self.is_error():\n await self.evaluation()\n \n # Print cost summary\n self.print_cost()\n```\n\n### Lifecycle Stages\n\n#### 1. Initialization\n\n```python\nsession = Session(\n task=\"email_task\",\n should_evaluate=True,\n id=0,\n request=\"Send an email to John\",\n mode=\"normal\"\n)\n```\n\n**What happens:**\n- Task name assigned\n- Session ID set\n- Initial request stored\n- Mode configured\n\n#### 2. Context Initialization\n\n```python\ndef _init_context(self) -> None:\n \"\"\"Initialize the session context.\"\"\"\n super()._init_context()\n \n # Create MCP server manager\n mcp_server_manager = MCPServerManager()\n \n # Create local dispatcher\n command_dispatcher = LocalCommandDispatcher(\n session=self,\n mcp_server_manager=mcp_server_manager\n )\n \n # Attach to context\n self.context.attach_command_dispatcher(command_dispatcher)\n```\n\n**What happens:**\n- Context object created\n- Command dispatcher attached (Local or WebSocket)\n- MCP servers initialized (if applicable)\n- Application window tracked\n\n#### 3. Round Creation\n\n```python\ndef create_new_round(self):\n \"\"\"Create a new round.\"\"\"\n \n # Get request (first or new)\n if not self.context.get(ContextNames.REQUEST):\n request = first_request()\n else:\n request, complete = new_request()\n if complete:\n return None\n \n # Create round with request\n round = Round(\n task=self.task,\n context=self.context,\n request=request,\n id=self._round_num\n )\n \n self._round_num += 1\n return round\n```\n\n**What happens:**\n- User prompted for request (interactive modes)\n- Or request read from file/plan (non-interactive)\n- Round object created with shared context\n- Round counter incremented\n\n#### 4. Round Execution\n\n```python\nawait round.run()\n```\n\n**What happens:**\n- HostAgent selects application (Windows)\n- AppAgent executes in application (or LinuxAgent directly)\n- Commands dispatched and executed\n- Results captured in context\n- Experience logged\n\n#### 5. Continuation Check\n\n```python\ndef is_finished(self) -> bool:\n \"\"\"Check if session is complete.\"\"\"\n return self.context.get(ContextNames.SESSION_FINISH, False)\n```\n\n**What happens:**\n- Check if user wants another request\n- Check if error occurred\n- Check if plan is complete (follower/batch modes)\n\n#### 6. Final Snapshot\n\n```python\nasync def capture_last_snapshot(self) -> None:\n \"\"\"Capture the last snapshot of the application.\"\"\"\n \n last_round = self.context.get(ContextNames.ROUND_STEP)\n subtask_amount = self.context.get(ContextNames.SUBTASK_AMOUNT)\n \n # Capture screenshot\n screenshot = self.application_window.capture_screenshot_infor()\n \n # Save to logs\n self.file_writer.save_screenshot(\n screenshot,\n last_round,\n subtask_amount,\n \"last\"\n )\n```\n\n**What happens:**\n- Screenshot captured\n- Control tree logged\n- Final state preserved\n\n#### 7. Evaluation\n\n```python\nasync def evaluation(self) -> None:\n \"\"\"Evaluate the session.\"\"\"\n \n evaluator = EvaluationAgent(\n name=\"evaluation\",\n process_name=self.context.get(ContextNames.APPLICATION_PROCESS_NAME),\n app_root_name=self.context.get(ContextNames.APPLICATION_ROOT_NAME),\n is_visual=self.configs[\"EVA_SESSION\"][\"VIS_EVAL\"],\n main_prompt=self.configs[\"EVA_SESSION\"][\"MAIN_PROMPT\"],\n api_prompt=self.configs[\"EVA_SESSION\"][\"API_PROMPT\"]\n )\n \n score = await evaluator.evaluate(\n request=self.context.get(ContextNames.REQUEST),\n trajectory=self.context.get(ContextNames.TRAJECTORY)\n )\n \n self.file_writer.save_evaluation(score)\n```\n\n**What happens:**\n- EvaluationAgent created\n- Task completion assessed\n- Score logged\n- Feedback saved\n\n#### 8. Cost Summary\n\n```python\ndef print_cost(self) -> None:\n \"\"\"Print the session cost.\"\"\"\n \n total_cost = self.context.get(ContextNames.TOTAL_COST, 0.0)\n total_tokens = self.context.get(ContextNames.TOTAL_TOKENS, 0)\n \n console.print(f\"[bold green]Session Complete[/bold green]\")\n console.print(f\"Total Cost: ${total_cost:.4f}\")\n console.print(f\"Total Tokens: {total_tokens}\")\n```\n\n---\n\n## Execution Modes\n\n### Normal Mode\n\n**Interactive execution with user in the loop:**\n\n```python\nsession = Session(\n task=\"document_edit\",\n should_evaluate=True,\n id=0,\n request=\"\", # Will prompt user\n mode=\"normal\"\n)\n\nawait session.run()\n```\n\n**Features:**\n- User prompted for initial request via `first_request()`\n- User prompted for each new request via `new_request()`\n- Commands executed locally via `LocalCommandDispatcher`\n- User can exit anytime by typing \"N\"\n\n**Flow:**\n```\n1. Display welcome panel\n2. User enters: \"Open Word\"\n3. HostAgent selects Word application\n4. AppAgent types content\n5. User asked: \"What next?\"\n6. User enters: \"Save document\"\n7. AppAgent saves file\n8. User asked: \"What next?\"\n9. User enters: \"N\" (exit)\n10. Session ends\n```\n\n### Normal_Operator Mode\n\n**Normal mode with operator capabilities:**\n\n```python\nsession = Session(\n task=\"complex_workflow\",\n should_evaluate=True,\n id=0,\n request=\"Organize my files by date\",\n mode=\"normal_operator\"\n)\n```\n\n**Differences from Normal:**\n- Agent can use operator-level actions\n- More powerful command set\n- Same interactive workflow\n\n### Service Mode\n\n**WebSocket-controlled remote execution:**\n\n```python\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\nprotocol = TaskExecutionProtocol(websocket_connection)\n\nsession = ServiceSession(\n task=\"remote_automation\",\n should_evaluate=True,\n id=\"session_abc123\",\n request=\"Click Submit button\",\n task_protocol=protocol\n)\n\nawait session.run()\n```\n\n**Features:**\n- No user interaction prompts\n- Single request per session\n- Commands sent via WebSocket\n- Results returned to server\n- Uses `WebSocketCommandDispatcher`\n\n**Flow:**\n```\n1. Server sends request via WebSocket\n2. ServiceSession created\n3. Agent generates commands\n4. Commands sent to client via WebSocket\n5. Client executes locally\n6. Results sent back\n7. Session finishes immediately\n```\n\n**Key Difference:**\n\n```python\ndef is_finished(self) -> bool:\n \"\"\"Service session finishes after one round.\"\"\"\n return self._round_num > 0\n```\n\n### Follower Mode\n\n**Replay saved action plans:**\n\n```python\nsession = FollowerSession(\n task=\"email_replay\",\n plan_file=\"/plans/send_email.json\",\n should_evaluate=True,\n id=0\n)\n\nawait session.run()\n```\n\n**Features:**\n- No user prompts\n- Reads actions from plan file\n- Deterministic execution\n- Good for testing/demos\n\n**Plan File Format:**\n\n```json\n{\n \"request\": \"Send an email to John\",\n \"actions\": [\n {\n \"agent\": \"HostAgent\",\n \"action\": \"select_application\",\n \"parameters\": {\"app_name\": \"Outlook\"}\n },\n {\n \"agent\": \"AppAgent\",\n \"action\": \"click_element\",\n \"parameters\": {\"label\": \"New Email\"}\n }\n ]\n}\n```\n\n### Batch_Normal Mode\n\n**Execute multiple requests from files:**\n\n```python\nsession = FromFileSession(\n task=\"batch_task\",\n plan_file=\"/requests/task1.json\",\n should_evaluate=True,\n id=0\n)\n\nawait session.run()\n```\n\n**Features:**\n- Request loaded from file\n- No user interaction\n- Can batch multiple files with SessionPool\n- Task status tracking available\n\n**Request File:**\n\n```json\n{\n \"request\": \"Create a spreadsheet with sales data\"\n}\n```\n\n### Operator Mode\n\n**Pure operator-level execution:**\n\n```python\nsession = OpenAIOperatorSession(\n task=\"system_automation\",\n should_evaluate=True,\n id=0,\n request=\"Install and configure software\"\n)\n\nawait session.run()\n```\n\n**Features:**\n- Operator-level permissions\n- Can modify system settings\n- More powerful than AppAgent\n- Same interactive prompts as normal mode\n\n---\n\n## Platform-Specific Sessions\n\n### Windows Sessions\n\n**Characteristics:**\n- **Two-tier architecture**: HostAgent → AppAgent\n- **Base class**: `WindowsBaseSession`\n- **Agent flow**: HostAgent selects app, AppAgent controls it\n- **Automation**: Uses UIA (UI Automation)\n\n**Example:**\n\n```python\nclass Session(WindowsBaseSession):\n \"\"\"Windows interactive session.\"\"\"\n \n def _init_context(self):\n \"\"\"Initialize with HostAgent.\"\"\"\n super()._init_context()\n \n # HostAgent created by WindowsBaseSession\n self.host_agent = self.create_host_agent()\n \n # MCP and LocalCommandDispatcher\n self.setup_command_dispatcher()\n```\n\n### Linux Sessions\n\n**Characteristics:**\n- **Single-tier architecture**: LinuxAgent only (no HostAgent)\n- **Base class**: `LinuxBaseSession`\n- **Agent flow**: LinuxAgent controls application directly\n- **Automation**: Platform-specific tools\n\n**Example:**\n\n```python\nclass LinuxSession(LinuxBaseSession):\n \"\"\"Linux interactive session.\"\"\"\n \n def _init_context(self):\n \"\"\"Initialize without HostAgent.\"\"\"\n super()._init_context()\n \n # No HostAgent - direct LinuxAgent usage\n self.linux_agent = self.create_linux_agent(\n application_name=self.application_name\n )\n```\n\n**Comparison:**\n\n| Aspect | Windows | Linux |\n|--------|---------|-------|\n| **Architecture** | Two-tier (HostAgent + AppAgent) | Single-tier (LinuxAgent) |\n| **Application Selection** | HostAgent decides | Pre-specified or LinuxAgent decides |\n| **Agent Switching** | Yes (HostAgent ↔ AppAgent) | No |\n| **Modes Supported** | All 7 modes | normal, normal_operator, service |\n| **UI Automation** | UIA (UIAutomation) | Platform tools |\n\nSee [Platform Sessions](./platform_sessions.md) for detailed comparison.\n\n---\n\n## Experience Saving\n\nSessions can save successful workflows for future learning:\n\n```python\n# After successful task completion\nif self.configs[\"SAVE_EXPERIENCE\"] == \"ask\":\n save = experience_asker()\n \n if save:\n self.save_experience()\n```\n\n**Save Modes:**\n\n| Mode | Behavior |\n|------|----------|\n| `always` | Auto-save every successful session |\n| `ask` | Prompt user after each session |\n| `auto` | Save if evaluation score > threshold |\n| `always_not` | Never save |\n\n**Saved Experience Structure:**\n\n```json\n{\n \"task\": \"Send email\",\n \"request\": \"Send an email to John about the meeting\",\n \"trajectory\": [\n {\n \"round\": 0,\n \"agent\": \"HostAgent\",\n \"observation\": \"Desktop with Outlook icon\",\n \"action\": \"select_application\",\n \"parameters\": {\"app_name\": \"Outlook\"}\n },\n {\n \"round\": 0,\n \"agent\": \"AppAgent\",\n \"observation\": \"Outlook main window\",\n \"action\": \"click_element\",\n \"parameters\": {\"label\": \"New Email\"}\n }\n ],\n \"outcome\": \"success\",\n \"evaluation_score\": 0.95,\n \"cost\": 0.0234,\n \"tokens\": 1542\n}\n```\n\n---\n\n## Error Handling\n\n### Error States\n\nSessions track errors through context:\n\n```python\ndef is_error(self) -> bool:\n \"\"\"Check if session encountered error.\"\"\"\n return self.context.get(ContextNames.ERROR, False)\n\ndef set_error(self, error_message: str):\n \"\"\"Set error state.\"\"\"\n self.context.set(ContextNames.ERROR, True)\n self.context.set(ContextNames.ERROR_MESSAGE, error_message)\n```\n\n### Error Recovery\n\n```python\ntry:\n await round.run()\nexcept AgentError as e:\n self.set_error(str(e))\n logger.error(f\"Round {self._round_num} failed: {e}\")\n \n # Decide whether to continue or abort\n if self.can_recover(e):\n # Try next round\n continue\n else:\n # Abort session\n break\n```\n\n### Common Errors\n\n| Error Type | Cause | Handling |\n|------------|-------|----------|\n| **TimeoutError** | Command execution timeout | Retry or skip |\n| **ConnectionError** | WebSocket/MCP disconnection | Reconnect or abort |\n| **AgentError** | Agent decision failure | Log and retry |\n| **ValidationError** | Invalid command parameters | Skip command |\n\n---\n\n## Best Practices\n\n### Session Creation\n\n!!!tip \"Efficient Sessions\"\n - ✅ Use `SessionFactory.create_session()` for platform-aware creation\n - ✅ Enable evaluation for quality tracking\n - ✅ Choose appropriate mode for use case\n - ✅ Set meaningful task names for logging\n - ❌ Don't create sessions directly (use factory)\n - ❌ Don't mix modes (each session has one mode)\n\n### Interactive Sessions\n\n!!!success \"User Experience\"\n - ✅ Provide clear initial requests\n - ✅ Allow users to exit gracefully (\"N\" option)\n - ✅ Show progress and confirmations\n - ✅ Handle sensitive actions with confirmation\n - ❌ Don't prompt excessively\n - ❌ Don't hide errors from users\n\n### Service Sessions\n\n!!!warning \"WebSocket Considerations\"\n - ✅ Always provide `task_protocol`\n - ✅ Handle connection loss gracefully\n - ✅ Set appropriate timeouts\n - ✅ Validate requests before execution\n - ❌ Don't assume connection is stable\n - ❌ Don't block waiting for results indefinitely\n\n### Batch Sessions\n\n!!!tip \"Batch Processing\"\n - ✅ Enable task status tracking\n - ✅ Use descriptive file names\n - ✅ Group similar tasks\n - ✅ Log failures for retry\n - ❌ Don't stop batch on first failure\n - ❌ Don't run too many sessions in parallel\n\n---\n\n## Examples\n\n### Example 1: Basic Interactive Session\n\n```python\nfrom ufo.module.sessions.session import Session\n\n# Create session\nsession = Session(\n task=\"word_editing\",\n should_evaluate=True,\n id=0,\n request=\"\", # Will prompt user\n mode=\"normal\"\n)\n\n# Run session\nawait session.run()\n\n# User interaction:\n# 1. Welcome panel shown\n# 2. User enters: \"Open Word and type Hello World\"\n# 3. HostAgent selects Word\n# 4. AppAgent types text\n# 5. User asked for next request\n# 6. User enters: \"N\" to exit\n# 7. Session evaluates and ends\n```\n\n### Example 2: Service Session\n\n```python\nfrom ufo.module.sessions.service_session import ServiceSession\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\n# WebSocket established\nprotocol = TaskExecutionProtocol(websocket)\n\n# Create service session\nsession = ServiceSession(\n task=\"remote_click\",\n should_evaluate=False, # Server evaluates\n id=\"sess_12345\",\n request=\"Click the Submit button\",\n task_protocol=protocol\n)\n\n# Run (non-blocking for client)\nawait session.run()\n\n# Session finishes after one request\n```\n\n### Example 3: Follower Session\n\n```python\nfrom ufo.module.sessions.session import FollowerSession\n\n# Replay saved plan\nsession = FollowerSession(\n task=\"email_demo\",\n plan_file=\"./plans/send_email.json\",\n should_evaluate=True,\n id=0\n)\n\nawait session.run()\n\n# Executes exactly as recorded in plan file\n# No user prompts\n# Deterministic execution\n```\n\n### Example 4: Linux Session\n\n```python\nfrom ufo.module.sessions.linux_session import LinuxSession\n\n# Linux interactive session\nsession = LinuxSession(\n task=\"linux_task\",\n should_evaluate=True,\n id=0,\n request=\"Open gedit and type Hello Linux\",\n mode=\"normal\",\n application_name=\"gedit\"\n)\n\nawait session.run()\n\n# Single-tier architecture\n# No HostAgent\n# LinuxAgent controls gedit directly\n```\n\n---\n\n## Reference\n\n### BaseSession\n\n::: module.basic.BaseSession\n\n### Session (Windows)\n\n::: module.sessions.session.Session\n\n### LinuxSession\n\n::: module.sessions.linux_session.LinuxSession\n\n---\n\n## See Also\n\n- [Round](./round.md) - Individual request-response cycles\n- [Context](./context.md) - Shared state management\n- [Session Factory](./session_pool.md) - Session creation\n- [Platform Sessions](./platform_sessions.md) - Windows vs Linux"
},
{
"path": "documents/docs/infrastructure/modules/session_pool.md",
"content": "# Session Factory & Pool\n\nThe **SessionFactory** and **SessionPool** classes provide platform-aware session creation and batch execution management, supporting 7 different session modes across Windows and Linux platforms.\n\n**Quick Reference:**\n\n- Create single session? Use [SessionFactory.create_session()](#create_session)\n- Create service session? Use [SessionFactory.create_service_session()](#create_service_session)\n- Batch execution? Use [SessionPool](#sessionpool)\n- Platform detection? Automatic or override with `platform_override`\n\n---\n\n## Overview\n\nThe session factory and pool system provides:\n\n1. **Platform Abstraction**: Automatically creates the correct session type for Windows or Linux\n2. **Mode Support**: Handles 7 different execution modes with appropriate session classes\n3. **Batch Management**: Executes multiple sessions sequentially with status tracking\n4. **Service Integration**: Creates WebSocket-controlled sessions with AIP protocol\n\n### Architecture\n\n```mermaid\ngraph TB\n subgraph \"Client Code\"\n REQ[User Request]\n MODE[Execution Mode]\n PLATFORM[Platform Detection]\n end\n \n subgraph \"SessionFactory\"\n FACTORY[SessionFactory]\n DETECT[Platform Detection]\n WINDOWS[_create_windows_session]\n LINUX[_create_linux_session]\n SERVICE[create_service_session]\n end\n \n subgraph \"Session Types\"\n S1[Session Windows Normal]\n S2[ServiceSession Windows Service]\n S3[FollowerSession Windows Follower]\n S4[FromFileSession Windows Batch]\n S5[OpenAIOperatorSession Windows Operator]\n S6[LinuxSession Linux Normal]\n S7[LinuxServiceSession Linux Service]\n end\n \n subgraph \"SessionPool\"\n POOL[SessionPool]\n LIST[session_list]\n RUN[run_all]\n NEXT[next_session]\n end\n \n REQ --> FACTORY\n MODE --> FACTORY\n PLATFORM --> DETECT\n \n FACTORY --> DETECT\n DETECT -->|Windows| WINDOWS\n DETECT -->|Linux| LINUX\n DETECT -->|Service| SERVICE\n \n WINDOWS --> S1\n WINDOWS --> S2\n WINDOWS --> S3\n WINDOWS --> S4\n WINDOWS --> S5\n \n LINUX --> S6\n LINUX --> S7\n \n SERVICE -->|Windows| S2\n SERVICE -->|Linux| S7\n \n S1 --> POOL\n S2 --> POOL\n S3 --> POOL\n S4 --> POOL\n S5 --> POOL\n S6 --> POOL\n S7 --> POOL\n \n POOL --> LIST\n POOL --> RUN\n POOL --> NEXT\n \n style FACTORY fill:#e1f5ff\n style POOL fill:#f0ffe1\n style DETECT fill:#fff4e1\n style SERVICE fill:#ffe1f5\n```\n\n---\n\n## SessionFactory\n\n`SessionFactory` is the central factory for creating all session types with automatic platform detection.\n\n### Class Overview\n\n```python\nfrom ufo.module.session_pool import SessionFactory\n\nfactory = SessionFactory()\n\n# Automatically detects platform and creates appropriate session\nsessions = factory.create_session(\n task=\"email_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Send an email to John\"\n)\n```\n\n### Supported Modes\n\n| Mode | Platform | Session Type | Use Case |\n|------|----------|--------------|----------|\n| `normal` | Windows | `Session` | Interactive with HostAgent |\n| `normal` | Linux | `LinuxSession` | Interactive without HostAgent |\n| `normal_operator` | Windows | `Session` | Normal with operator mode |\n| `normal_operator` | Linux | `LinuxSession` | Normal with operator mode |\n| `service` | Windows | `ServiceSession` | WebSocket-controlled |\n| `service` | Linux | `LinuxServiceSession` | WebSocket-controlled |\n| `follower` | Windows | `FollowerSession` | Replay saved plans |\n| `batch_normal` | Windows | `FromFileSession` | Batch execution from files |\n| `operator` | Windows | `OpenAIOperatorSession` | Pure operator mode |\n\n!!!note \"Linux Mode Limitations\"\n Currently, Linux only supports `normal`, `normal_operator`, and `service` modes. Follower and batch modes are planned for future releases.\n\n---\n\n### create_session()\n\nCreates one or more sessions based on platform, mode, and plan configuration.\n\n#### Signature\n\n```python\ndef create_session(\n self,\n task: str,\n mode: str,\n plan: str,\n request: str = \"\",\n platform_override: Optional[str] = None,\n **kwargs,\n) -> List[BaseSession]\n```\n\n#### Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `task` | `str` | Required | Task name for logging/identification |\n| `mode` | `str` | Required | Execution mode (see table above) |\n| `plan` | `str` | Required | Plan file/folder path (for follower/batch modes) |\n| `request` | `str` | `\"\"` | User's natural language request |\n| `platform_override` | `Optional[str]` | `None` | Force platform: `\"windows\"` or `\"linux\"` |\n| `**kwargs` | Various | - | Additional parameters (see below) |\n\n**Additional kwargs:**\n\n| Key | Type | Used By | Description |\n|-----|------|---------|-------------|\n| `id` | `int` | All modes | Session ID for tracking |\n| `task_protocol` | `TaskExecutionProtocol` | Service modes | WebSocket protocol instance |\n| `application_name` | `str` | Linux modes | Target application |\n\n#### Return Value\n\n`List[BaseSession]` - List of created sessions\n\n- **Single session modes** (normal, service, operator): Returns 1-element list\n- **Batch modes** (follower, batch_normal with folder): Returns list of sessions for each plan file\n\n#### Platform Detection\n\n```mermaid\ngraph TB\n START[create_session called]\n CHECK{platform_override?}\n AUTO[platform.system.lower]\n OVERRIDE[Use override value]\n \n WINDOWS{Platform == 'windows'?}\n LINUX{Platform == 'linux'?}\n ERROR[NotImplementedError]\n \n WIN_METHOD[_create_windows_session]\n LINUX_METHOD[_create_linux_session]\n \n RETURN[Return session list]\n \n START --> CHECK\n CHECK -->|None| AUTO\n CHECK -->|Set| OVERRIDE\n \n AUTO --> WINDOWS\n OVERRIDE --> WINDOWS\n \n WINDOWS -->|Yes| WIN_METHOD\n WINDOWS -->|No| LINUX\n \n LINUX -->|Yes| LINUX_METHOD\n LINUX -->|No| ERROR\n \n WIN_METHOD --> RETURN\n LINUX_METHOD --> RETURN\n \n style START fill:#e1f5ff\n style WIN_METHOD fill:#f0ffe1\n style LINUX_METHOD fill:#fff4e1\n style ERROR fill:#ffe1e1\n```\n\n#### Examples\n\n**Example 1: Normal Windows Session**\n\n```python\nfactory = SessionFactory()\n\nsessions = factory.create_session(\n task=\"browse_web\",\n mode=\"normal\",\n plan=\"\",\n request=\"Open Chrome and navigate to google.com\"\n)\n\n# Returns: [Session(task=\"browse_web\", ...)]\nsession = sessions[0]\nawait session.run()\n```\n\n**Example 2: Service Session (Auto-detected Platform)**\n\n```python\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\nprotocol = TaskExecutionProtocol(websocket_connection)\n\nsessions = factory.create_session(\n task=\"remote_control\",\n mode=\"service\",\n plan=\"\",\n request=\"Click the Start button\",\n task_protocol=protocol\n)\n\n# On Windows: Returns [ServiceSession(...)]\n# On Linux: Returns [LinuxServiceSession(...)]\n```\n\n**Example 3: Batch Follower Sessions**\n\n```python\nsessions = factory.create_session(\n task=\"batch_email\",\n mode=\"follower\",\n plan=\"/path/to/plan_folder\", # Folder with multiple .json plan files\n request=\"\"\n)\n\n# Returns: [\n# FollowerSession(task=\"batch_email/plan1\", ...),\n# FollowerSession(task=\"batch_email/plan2\", ...),\n# FollowerSession(task=\"batch_email/plan3\", ...)\n# ]\n\n# Execute with SessionPool\npool = SessionPool(sessions)\nawait pool.run_all()\n```\n\n**Example 4: Linux Session with Application**\n\n```python\nsessions = factory.create_session(\n task=\"edit_document\",\n mode=\"normal\",\n plan=\"\",\n request=\"Type 'Hello World'\",\n platform_override=\"linux\",\n application_name=\"gedit\"\n)\n\n# Returns: [LinuxSession(task=\"edit_document\", application_name=\"gedit\")]\n```\n\n**Example 5: Operator Mode**\n\n```python\nsessions = factory.create_session(\n task=\"complex_workflow\",\n mode=\"operator\",\n plan=\"\",\n request=\"Organize my desktop files by date\"\n)\n\n# Returns: [OpenAIOperatorSession(task=\"complex_workflow\", ...)]\n```\n\n---\n\n### create_service_session()\n\nSimplified method specifically for creating service sessions on any platform.\n\n#### Signature\n\n```python\ndef create_service_session(\n self,\n task: str,\n should_evaluate: bool,\n id: str,\n request: str,\n task_protocol: Optional[\"TaskExecutionProtocol\"] = None,\n platform_override: Optional[str] = None,\n) -> BaseSession\n```\n\n#### Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `task` | `str` | Required | Task name |\n| `should_evaluate` | `bool` | Required | Enable evaluation |\n| `id` | `str` | Required | Session ID |\n| `request` | `str` | Required | User request |\n| `task_protocol` | `TaskExecutionProtocol` | `None` | AIP protocol instance |\n| `platform_override` | `Optional[str]` | `None` | Force platform |\n\n#### Return Value\n\n`BaseSession` - Single service session instance\n\n- **Windows**: Returns `ServiceSession`\n- **Linux**: Returns `LinuxServiceSession`\n\n#### Example\n\n```python\nfactory = SessionFactory()\nprotocol = TaskExecutionProtocol(websocket)\n\nsession = factory.create_service_session(\n task=\"remote_task\",\n should_evaluate=True,\n id=\"session_001\",\n request=\"Open Notepad\",\n task_protocol=protocol\n)\n\n# Type varies by platform\nif isinstance(session, ServiceSession):\n print(\"Windows service session\")\nelif isinstance(session, LinuxServiceSession):\n print(\"Linux service session\")\n\nawait session.run()\n```\n\n---\n\n### _create_windows_session() (Internal)\n\n!!!warning \"Internal Method\"\n Called by `create_session()` when platform is Windows. Not meant for direct use.\n\n#### Mode Routing\n\n```mermaid\ngraph TB\n START[_create_windows_session]\n MODE{mode value}\n \n NORMAL[normal/normal_operator]\n SERVICE[service]\n FOLLOWER[follower]\n BATCH[batch_normal]\n OPERATOR[operator]\n ERROR[ValueError]\n \n S1[Session]\n S2[ServiceSession]\n S3_CHECK{plan is folder?}\n S3_BATCH[create_follower_session_in_batch]\n S3_SINGLE[FollowerSession single]\n S4_CHECK{plan is folder?}\n S4_BATCH[create_sessions_in_batch]\n S4_SINGLE[FromFileSession single]\n S5[OpenAIOperatorSession]\n \n START --> MODE\n \n MODE -->|normal| NORMAL\n MODE -->|normal_operator| NORMAL\n MODE -->|service| SERVICE\n MODE -->|follower| FOLLOWER\n MODE -->|batch_normal| BATCH\n MODE -->|operator| OPERATOR\n MODE -->|other| ERROR\n \n NORMAL --> S1\n SERVICE --> S2\n \n FOLLOWER --> S3_CHECK\n S3_CHECK -->|Yes| S3_BATCH\n S3_CHECK -->|No| S3_SINGLE\n \n BATCH --> S4_CHECK\n S4_CHECK -->|Yes| S4_BATCH\n S4_CHECK -->|No| S4_SINGLE\n \n OPERATOR --> S5\n \n style START fill:#e1f5ff\n style S1 fill:#f0ffe1\n style S2 fill:#fff4e1\n style ERROR fill:#ffe1e1\n```\n\n#### Created Session Types\n\n| Mode | Condition | Session Type | Notes |\n|------|-----------|--------------|-------|\n| `normal` | - | `Session` | Standard interactive |\n| `normal_operator` | - | `Session` | With operator mode flag |\n| `service` | - | `ServiceSession` | Requires `task_protocol` |\n| `follower` | Plan is file | `FollowerSession` | Single plan replay |\n| `follower` | Plan is folder | `List[FollowerSession]` | Batch plan replay |\n| `batch_normal` | Plan is file | `FromFileSession` | Single file execution |\n| `batch_normal` | Plan is folder | `List[FromFileSession]` | Batch file execution |\n| `operator` | - | `OpenAIOperatorSession` | Pure operator mode |\n\n---\n\n### _create_linux_session() (Internal)\n\n!!!warning \"Internal Method\"\n Called by `create_session()` when platform is Linux. Not meant for direct use.\n\n#### Mode Routing\n\n```mermaid\ngraph TB\n START[_create_linux_session]\n MODE{mode value}\n \n NORMAL[normal/normal_operator]\n SERVICE[service]\n ERROR[ValueError]\n \n S1[LinuxSession]\n S2[LinuxServiceSession]\n \n START --> MODE\n \n MODE -->|normal| NORMAL\n MODE -->|normal_operator| NORMAL\n MODE -->|service| SERVICE\n MODE -->|other| ERROR\n \n NORMAL --> S1\n SERVICE --> S2\n \n style START fill:#e1f5ff\n style S1 fill:#f0ffe1\n style S2 fill:#fff4e1\n style ERROR fill:#ffe1e1\n```\n\n#### Supported Modes\n\n| Mode | Session Type | Notes |\n|------|--------------|-------|\n| `normal` | `LinuxSession` | Standard Linux interactive |\n| `normal_operator` | `LinuxSession` | With operator mode flag |\n| `service` | `LinuxServiceSession` | Requires `task_protocol` |\n\n!!!note \"Upcoming Features\"\n Follower and batch_normal modes for Linux are planned for future releases.\n\n---\n\n### Batch Session Creation\n\n#### create_follower_session_in_batch()\n\nCreates multiple follower sessions from a folder of plan files:\n\n```python\ndef create_follower_session_in_batch(\n self, \n task: str, \n plan: str\n) -> List[BaseSession]\n```\n\n**Process:**\n\n1. Scan folder for `.json` files\n2. Extract file names (without extension)\n3. Create `FollowerSession` for each plan file\n4. Assign sequential IDs\n5. Prefix task name with file name: `{task}/{filename}`\n\n**Example:**\n\n```python\n# Folder structure:\n# /plans/\n# ├── email_john.json\n# ├── email_jane.json\n# └── email_bob.json\n\nsessions = factory.create_follower_session_in_batch(\n task=\"send_emails\",\n plan=\"/plans/\"\n)\n\n# Returns:\n# [\n# FollowerSession(task=\"send_emails/email_john\", plan=\"/plans/email_john.json\", id=0),\n# FollowerSession(task=\"send_emails/email_jane\", plan=\"/plans/email_jane.json\", id=1),\n# FollowerSession(task=\"send_emails/email_bob\", plan=\"/plans/email_bob.json\", id=2)\n# ]\n```\n\n#### create_sessions_in_batch()\n\nCreates multiple FromFileSession instances with task status tracking:\n\n```python\ndef create_sessions_in_batch(\n self, \n task: str, \n plan: str\n) -> List[BaseSession]\n```\n\n**Features:**\n\n- Tracks completed tasks in `tasks_status.json`\n- Skips already-completed tasks\n- Resumes from last incomplete task\n\n**Task Status File:**\n\n```json\n{\n \"email_john\": true,\n \"email_jane\": false,\n \"email_bob\": false\n}\n```\n\n**Example:**\n\n```python\n# First run\nsessions = factory.create_sessions_in_batch(\n task=\"batch_emails\",\n plan=\"/requests/\"\n)\n# Returns 3 sessions: email_john, email_jane, email_bob\n\n# email_john completes successfully\n# tasks_status.json updated: {\"email_john\": true, \"email_jane\": false, \"email_bob\": false}\n\n# Second run (after restart)\nsessions = factory.create_sessions_in_batch(\n task=\"batch_emails\",\n plan=\"/requests/\"\n)\n# Returns 2 sessions: email_jane, email_bob (skips completed email_john)\n```\n\n**Configuration:**\n\n```python\n# Enable task status tracking\nufo_config.system.task_status = True\n\n# Custom status file location\nufo_config.system.task_status_file = \"/path/to/status.json\"\n```\n\n---\n\n## SessionPool\n\n`SessionPool` manages multiple sessions and executes them sequentially.\n\n### Class Overview\n\n```python\nfrom ufo.module.session_pool import SessionPool\n\n# Create sessions\nsessions = factory.create_session(\n task=\"batch_task\",\n mode=\"follower\",\n plan=\"/plans_folder/\"\n)\n\n# Create pool\npool = SessionPool(session_list=sessions)\n\n# Execute all\nawait pool.run_all()\n```\n\n### Constructor\n\n```python\ndef __init__(self, session_list: List[BaseSession]) -> None\n```\n\n**Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `session_list` | `List[BaseSession]` | Initial list of sessions |\n\n### Methods\n\n#### run_all()\n\nExecute all sessions in the pool sequentially:\n\n```python\nasync def run_all(self) -> None\n```\n\n**Execution Flow:**\n\n```mermaid\nsequenceDiagram\n participant Pool as SessionPool\n participant S1 as Session 1\n participant S2 as Session 2\n participant S3 as Session 3\n \n Pool->>S1: await session.run()\n S1->>S1: Execute task\n S1-->>Pool: Complete\n \n Pool->>S2: await session.run()\n S2->>S2: Execute task\n S2-->>Pool: Complete\n \n Pool->>S3: await session.run()\n S3->>S3: Execute task\n S3-->>Pool: Complete\n \n Pool-->>Pool: All sessions complete\n```\n\n**Example:**\n\n```python\npool = SessionPool(sessions)\n\n# Execute all sequentially\nawait pool.run_all()\n\n# All sessions have completed\nprint(\"Batch execution complete\")\n```\n\n#### add_session()\n\nAdd a session to the pool:\n\n```python\ndef add_session(self, session: BaseSession) -> None\n```\n\n**Example:**\n\n```python\npool = SessionPool([session1, session2])\n\n# Add another session\npool.add_session(session3)\n\n# Now pool has 3 sessions\n```\n\n#### next_session()\n\nGet and remove the next session from the pool:\n\n```python\ndef next_session(self) -> BaseSession\n```\n\n**Example:**\n\n```python\npool = SessionPool([session1, session2, session3])\n\n# Get next session (FIFO)\nnext_sess = pool.next_session()\n# next_sess == session1\n# Pool now has [session2, session3]\n\nawait next_sess.run()\n```\n\n#### session_list (Property)\n\nGet the current session list:\n\n```python\n@property\ndef session_list(self) -> List[BaseSession]\n```\n\n**Example:**\n\n```python\npool = SessionPool(sessions)\n\nprint(f\"Pool has {len(pool.session_list)} sessions\")\n\nfor session in pool.session_list:\n print(f\"Task: {session.task}\")\n```\n\n---\n\n## Usage Patterns\n\n### Pattern 1: Single Interactive Session\n\n```python\nfactory = SessionFactory()\n\nsessions = factory.create_session(\n task=\"user_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Open Word and create a document\"\n)\n\nsession = sessions[0]\nawait session.run()\n```\n\n### Pattern 2: Service Session with WebSocket\n\n```python\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\n# WebSocket connection established\nprotocol = TaskExecutionProtocol(websocket)\n\nfactory = SessionFactory()\n\nsession = factory.create_service_session(\n task=\"remote_automation\",\n should_evaluate=True,\n id=\"session_123\",\n request=\"Click the Submit button\",\n task_protocol=protocol\n)\n\nawait session.run()\n```\n\n### Pattern 3: Batch Execution\n\n```python\n# Create batch sessions\nfactory = SessionFactory()\n\nsessions = factory.create_session(\n task=\"daily_reports\",\n mode=\"batch_normal\",\n plan=\"/request_files/\", # Folder with .json request files\n request=\"\"\n)\n\n# Execute with pool\npool = SessionPool(sessions)\nawait pool.run_all()\n\nprint(f\"Completed {len(sessions)} tasks\")\n```\n\n### Pattern 4: Cross-Platform Application\n\n```python\nimport platform\n\nfactory = SessionFactory()\n\n# Detect current platform\ncurrent_os = platform.system().lower()\n\nsessions = factory.create_session(\n task=\"cross_platform_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Open text editor\",\n application_name=\"gedit\" if current_os == \"linux\" else None\n)\n\n# Correct session type automatically created\nawait sessions[0].run()\n```\n\n### Pattern 5: Dynamic Session Pool\n\n```python\npool = SessionPool([])\n\n# Add sessions dynamically\nfor user_request in user_requests:\n sessions = factory.create_session(\n task=f\"request_{len(pool.session_list)}\",\n mode=\"normal\",\n plan=\"\",\n request=user_request\n )\n pool.add_session(sessions[0])\n\n# Execute all\nawait pool.run_all()\n```\n\n### Pattern 6: Resumable Batch Processing\n\n```python\n# Enable task status tracking\nufo_config.system.task_status = True\nufo_config.system.task_status_file = \"progress.json\"\n\nfactory = SessionFactory()\n\n# First run\nsessions = factory.create_sessions_in_batch(\n task=\"large_batch\",\n plan=\"/tasks/\"\n)\n\npool = SessionPool(sessions)\n\ntry:\n await pool.run_all()\nexcept KeyboardInterrupt:\n print(\"Interrupted - progress saved\")\n\n# Second run (resumes from last incomplete)\nsessions = factory.create_sessions_in_batch(\n task=\"large_batch\",\n plan=\"/tasks/\"\n)\n# Only uncompleted tasks loaded\n\npool = SessionPool(sessions)\nawait pool.run_all()\n```\n\n---\n\n## Configuration Integration\n\n### UFO Config Settings\n\n| Setting | Type | Purpose |\n|---------|------|---------|\n| `ufo_config.system.eva_session` | `bool` | Enable session evaluation |\n| `ufo_config.system.task_status` | `bool` | Enable task status tracking |\n| `ufo_config.system.task_status_file` | `str` | Custom status file path |\n\n### Example Configuration\n\n```yaml\n# config/ufo/config.yaml\nsystem:\n eva_session: true\n task_status: true\n task_status_file: \"./logs/task_status.json\"\n```\n\n**Usage:**\n\n```python\nfrom config.config_loader import get_ufo_config\n\nufo_config = get_ufo_config()\n\n# These settings affect SessionFactory behavior\nfactory = SessionFactory()\n\n# Uses ufo_config.system.eva_session for should_evaluate\nsessions = factory.create_session(\n task=\"configured_task\",\n mode=\"normal\",\n plan=\"\",\n request=\"Do something\"\n)\n```\n\n---\n\n## Platform Detection\n\n### Automatic Detection\n\n```python\nimport platform\n\ncurrent_platform = platform.system().lower()\n# Returns: \"windows\", \"linux\", \"darwin\" (macOS)\n```\n\n**Supported Platforms:**\n\n- `\"windows\"` → Windows-specific sessions\n- `\"linux\"` → Linux-specific sessions\n- Others → `NotImplementedError`\n\n### Manual Override\n\nForce platform selection:\n\n```python\n# Force Windows session on Linux machine (for testing)\nsessions = factory.create_session(\n task=\"test\",\n mode=\"normal\",\n plan=\"\",\n request=\"Test request\",\n platform_override=\"windows\"\n)\n\n# Creates Session instead of LinuxSession\n```\n\n!!!warning \"Override Use Cases\"\n - **Testing**: Test Windows sessions on Linux\n - **Development**: Test platform-specific code\n - **Cross-compilation**: Generate plans for other platforms\n - **Not for production**: Always use auto-detection in production\n\n---\n\n## Error Handling\n\n### NotImplementedError\n\n**Trigger:** Unsupported platform or mode\n\n```python\ntry:\n sessions = factory.create_session(\n task=\"task\",\n mode=\"follower\",\n plan=\"\",\n request=\"\",\n platform_override=\"darwin\" # macOS not supported\n )\nexcept NotImplementedError as e:\n print(f\"Error: {e}\")\n # Error: Platform darwin is not supported yet.\n```\n\n### ValueError\n\n**Trigger:** Invalid mode for platform\n\n```python\ntry:\n sessions = factory.create_session(\n task=\"task\",\n mode=\"follower\",\n plan=\"\",\n request=\"\",\n platform_override=\"linux\"\n )\nexcept ValueError as e:\n print(f\"Error: {e}\")\n # Error: The follower mode is not supported on Linux yet.\n # Supported modes: normal, normal_operator, service\n```\n\n### Graceful Handling\n\n```python\ndef create_session_safely(task, mode, plan, request):\n \"\"\"Create session with error handling.\"\"\"\n factory = SessionFactory()\n \n try:\n sessions = factory.create_session(\n task=task,\n mode=mode,\n plan=plan,\n request=request\n )\n return sessions\n \n except NotImplementedError as e:\n logger.error(f\"Platform not supported: {e}\")\n return []\n \n except ValueError as e:\n logger.error(f\"Invalid mode: {e}\")\n # Fallback to normal mode\n return factory.create_session(\n task=task,\n mode=\"normal\",\n plan=\"\",\n request=request\n )\n```\n\n---\n\n## Best Practices\n\n### Session Creation\n\n!!!tip \"Efficient Session Management\"\n - ✅ Use `create_service_session()` for service sessions (cleaner API)\n - ✅ Let platform auto-detect unless testing\n - ✅ Use batch modes for multiple similar tasks\n - ✅ Enable task status tracking for long-running batches\n - ❌ Don't create sessions in tight loops (use batch modes)\n - ❌ Don't mix session types in same pool without reason\n\n### Batch Processing\n\n!!!success \"Optimal Batch Execution\"\n 1. **Group similar tasks** in same folder\n 2. **Enable task status** tracking for resumability\n 3. **Use descriptive filenames** for task identification\n 4. **Handle failures** gracefully (don't stop entire batch)\n 5. **Monitor progress** with logging\n\n### Platform Handling\n\n!!!warning \"Cross-Platform Considerations\"\n - Always check platform before platform-specific operations\n - Use `application_name` parameter for Linux sessions\n - Test on both platforms if deploying cross-platform\n - Document platform-specific features clearly\n\n---\n\n## Troubleshooting\n\n### Issue: Wrong Session Type Created\n\n**Symptoms:**\n- Expected `LinuxSession` but got `Session`\n- Mode not working as expected\n\n**Diagnosis:**\n```python\nsession = sessions[0]\nprint(f\"Session type: {type(session).__name__}\")\nprint(f\"Platform: {platform.system().lower()}\")\n```\n\n**Solutions:**\n1. Check platform detection: `platform.system().lower()`\n2. Verify mode spelling and case\n3. Use `platform_override` if needed for testing\n\n### Issue: Batch Sessions Not Found\n\n**Symptoms:**\n- Empty session list from batch creation\n- `create_sessions_in_batch()` returns `[]`\n\n**Diagnosis:**\n```python\nplan_files = factory.get_plan_files(\"/path/to/folder\")\nprint(f\"Found {len(plan_files)} plan files\")\nprint(f\"Files: {plan_files}\")\n```\n\n**Solutions:**\n1. Ensure folder exists: `os.path.isdir(plan_folder)`\n2. Check files have `.json` extension\n3. Verify file permissions\n4. Check task status file hasn't marked all as done\n\n### Issue: Service Session Missing Protocol\n\n**Symptoms:**\n- `ValueError` about missing protocol\n- Service session fails to initialize\n\n**Diagnosis:**\n```python\nprotocol = kwargs.get(\"task_protocol\")\nprint(f\"Protocol: {protocol}\")\nprint(f\"Type: {type(protocol)}\")\n```\n\n**Solution:**\nAlways provide `task_protocol` for service sessions:\n\n```python\nfrom aip.protocol.task_execution import TaskExecutionProtocol\n\nprotocol = TaskExecutionProtocol(websocket)\n\nsession = factory.create_service_session(\n task=\"service_task\",\n should_evaluate=True,\n id=\"sess_001\",\n request=\"Do something\",\n task_protocol=protocol # ← Required!\n)\n```\n\n---\n\n## Reference\n\n### SessionFactory Methods\n\n::: module.session_pool.SessionFactory\n\n### SessionPool Methods\n\n::: module.session_pool.SessionPool\n\n---\n\n## See Also\n\n- [Session](./session.md) - Session lifecycle and execution\n- [Platform Sessions](./platform_sessions.md) - Windows vs Linux differences\n- [Overview](./overview.md) - Module system architecture\n- [AIP Protocol](../../aip/overview.md) - Service session WebSocket protocol\n\n"
},
{
"path": "documents/docs/javascripts/mermaid-init.js",
"content": "// Initialize Mermaid for ReadTheDocs theme\n(function() {\n // Wait for DOM to be ready\n if (document.readyState === 'loading') {\n document.addEventListener('DOMContentLoaded', initMermaid);\n } else {\n initMermaid();\n }\n\n function initMermaid() {\n // Initialize Mermaid\n if (typeof mermaid !== 'undefined') {\n mermaid.initialize({\n startOnLoad: true,\n theme: 'default',\n securityLevel: 'loose',\n flowchart: {\n useMaxWidth: true,\n htmlLabels: true,\n curve: 'basis'\n },\n sequence: {\n useMaxWidth: true,\n wrap: true\n },\n gantt: {\n useMaxWidth: true\n }\n });\n }\n }\n})();\n\n"
},
{
"path": "documents/docs/linux/as_galaxy_device.md",
"content": "# Using Linux Agent as Galaxy Device\n\nConfigure Linux Agent as a sub-agent in UFO's Galaxy framework to enable cross-platform, multi-device task orchestration. Galaxy can coordinate Linux agents alongside Windows devices to execute complex workflows spanning multiple systems.\n\n## Overview\n\nThe **Galaxy framework** provides multi-tier orchestration capabilities, allowing you to manage multiple device agents (Windows, Linux, etc.) from a central ConstellationAgent. When configured as a Galaxy device, LinuxAgent becomes a **sub-agent** that can:\n\n- Execute Linux-specific subtasks assigned by Galaxy\n- Participate in cross-platform workflows (e.g., Windows + Linux collaboration)\n- Report execution status back to the orchestrator\n- Be dynamically selected based on capabilities and metadata\n\nFor detailed information about LinuxAgent's design and capabilities, see [Linux Agent Overview](overview.md).\n\n## Galaxy Architecture with Linux Agent\n\n```mermaid\ngraph TB\n User[User Request]\n Galaxy[Galaxy ConstellationAgent Orchestrator]\n \n subgraph \"Device Pool\"\n Win1[Windows Device 1 HostAgent]\n Win2[Windows Device 2 HostAgent]\n Linux1[Linux Agent 1 CLI Executor]\n Linux2[Linux Agent 2 CLI Executor]\n Linux3[Linux Agent 3 CLI Executor]\n end\n \n User -->|Complex Task| Galaxy\n Galaxy -->|Windows Subtask| Win1\n Galaxy -->|Windows Subtask| Win2\n Galaxy -->|Linux Subtask| Linux1\n Galaxy -->|Linux Subtask| Linux2\n Galaxy -->|Linux Subtask| Linux3\n \n style Galaxy fill:#ffe1e1\n style Linux1 fill:#e1f5ff\n style Linux2 fill:#e1f5ff\n style Linux3 fill:#e1f5ff\n```\n\nGalaxy orchestrates task decomposition, device selection based on capabilities, parallel execution, and result aggregation across all devices.\n\n## Configuration Guide\n\n### Step 1: Configure Device in `devices.yaml`\n\nAdd your Linux agent(s) to the device list in `config/galaxy/devices.yaml`:\n\n**Example Configuration:**\n\n```yaml\ndevices:\n - device_id: \"linux_agent_1\"\n server_url: \"ws://172.23.48.1:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server\"\n - \"log_analysis\"\n - \"file_operations\"\n - \"database_management\"\n metadata:\n os: \"linux\"\n performance: \"high\"\n logs_file_path: \"/var/log/myapp/app.log\"\n dev_path: \"/home/user/development/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n description: \"Production web server\"\n auto_connect: true\n max_retries: 5\n```\n\n### Step 2: Understanding Configuration Fields\n\n| Field | Required | Type | Description |\n|-------|----------|------|-------------|\n| `device_id` | ✅ Yes | string | **Unique identifier** - must match client `--client-id` |\n| `server_url` | ✅ Yes | string | WebSocket URL - must match server endpoint |\n| `os` | ✅ Yes | string | Operating system - set to `\"linux\"` |\n| `capabilities` | ❌ Optional | list | Skills/capabilities for task routing |\n| `metadata` | ❌ Optional | dict | Custom context for LLM-based task execution |\n| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup (default: `true`) |\n| `max_retries` | ❌ Optional | integer | Connection retry attempts (default: `5`) |\n\n### Step 3: Capabilities-Based Task Routing\n\nGalaxy uses the `capabilities` field to intelligently route subtasks to appropriate devices. Define capabilities based on server roles, task types, installed software, or data access requirements.\n\n**Example Capability Configurations:**\n\n**Web Server:**\n```yaml\ncapabilities:\n - \"web_server\"\n - \"nginx\"\n - \"ssl_management\"\n - \"log_analysis\"\n```\n\n**Database Server:**\n```yaml\ncapabilities:\n - \"database_server\"\n - \"postgresql\"\n - \"backup_management\"\n - \"query_optimization\"\n```\n\n**CI/CD Server:**\n```yaml\ncapabilities:\n - \"ci_cd\"\n - \"docker\"\n - \"kubernetes\"\n - \"deployment\"\n```\n\n**Monitoring Server:**\n```yaml\ncapabilities:\n - \"monitoring\"\n - \"prometheus\"\n - \"grafana\"\n - \"alerting\"\n```\n\n### Step 4: Metadata for Contextual Execution\n\nThe `metadata` field provides contextual information that the LLM uses when generating commands for the Linux agent.\n\n**Metadata Examples:**\n\n**Web Server Metadata:**\n```yaml\nmetadata:\n os: \"linux\"\n logs_file_path: \"/var/log/nginx/access.log\"\n error_log_path: \"/var/log/nginx/error.log\"\n web_root: \"/var/www/html\"\n ssl_cert_path: \"/etc/letsencrypt/live/example.com/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n performance: \"high\"\n description: \"Production nginx web server\"\n```\n\n**Database Server Metadata:**\n```yaml\nmetadata:\n os: \"linux\"\n logs_file_path: \"/var/log/postgresql/postgresql.log\"\n data_path: \"/var/lib/postgresql/14/main\"\n backup_path: \"/mnt/backups/postgresql\"\n warning_log_pattern: \"WARNING\"\n error_log_pattern: \"ERROR|FATAL|PANIC\"\n performance: \"high\"\n description: \"Production PostgreSQL 14 database\"\n```\n\n**Development Server Metadata:**\n```yaml\nmetadata:\n os: \"linux\"\n dev_path: \"/home/developer/projects\"\n logs_file_path: \"/var/log/app/dev.log\"\n git_repo_path: \"/home/developer/repos\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR\"\n performance: \"medium\"\n description: \"Development and testing environment\"\n```\n\n**How Metadata is Used:**\n\nThe LLM receives metadata in the system prompt, enabling context-aware command generation. For example, with the web server metadata above, when the user requests \"Find all 500 errors in the last hour\", the LLM can generate the appropriate command using the correct log path.\n\n## Multi-Device Configuration Example\n\n**Complete Galaxy Setup:**\n\n```yaml\ndevices:\n # Windows Desktop Agent\n - device_id: \"windows_desktop_1\"\n server_url: \"ws://192.168.1.100:5000/ws\"\n os: \"windows\"\n capabilities:\n - \"office_applications\"\n - \"email\"\n - \"web_browsing\"\n metadata:\n os: \"windows\"\n description: \"Office productivity workstation\"\n auto_connect: true\n max_retries: 5\n \n # Linux Web Server\n - device_id: \"linux_web_server\"\n server_url: \"ws://192.168.1.101:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"web_server\"\n - \"nginx\"\n - \"log_analysis\"\n metadata:\n os: \"linux\"\n logs_file_path: \"/var/log/nginx/access.log\"\n web_root: \"/var/www/html\"\n description: \"Production web server\"\n auto_connect: true\n max_retries: 5\n \n # Linux Database Server\n - device_id: \"linux_db_server\"\n server_url: \"ws://192.168.1.102:5002/ws\"\n os: \"linux\"\n capabilities:\n - \"database_server\"\n - \"postgresql\"\n - \"backup_management\"\n metadata:\n os: \"linux\"\n logs_file_path: \"/var/log/postgresql/postgresql.log\"\n data_path: \"/var/lib/postgresql/14/main\"\n description: \"Production database server\"\n auto_connect: true\n max_retries: 5\n \n # Linux Monitoring Server\n - device_id: \"linux_monitoring\"\n server_url: \"ws://192.168.1.103:5003/ws\"\n os: \"linux\"\n capabilities:\n - \"monitoring\"\n - \"prometheus\"\n - \"alerting\"\n metadata:\n os: \"linux\"\n logs_file_path: \"/var/log/prometheus/prometheus.log\"\n metrics_path: \"/var/lib/prometheus\"\n description: \"System monitoring server\"\n auto_connect: true\n max_retries: 5\n```\n\n## Starting Galaxy with Linux Agents\n\n### Prerequisites\n\nEnsure all components are running before starting Galaxy:\n\n1. Device Agent Servers running on all machines\n2. Device Agent Clients connected to their respective servers\n3. MCP Services running on all Linux agents\n4. LLM configured in `config/ufo/agents.yaml` (for UFO) or `config/galaxy/agent.yaml` (for Galaxy)\n\n### Launch Sequence\n\n**Step 1: Start all Device Agent Servers**\n\n```bash\n# On web server machine (192.168.1.101)\npython -m ufo.server.app --port 5001\n\n# On database server machine (192.168.1.102)\npython -m ufo.server.app --port 5002\n\n# On monitoring server machine (192.168.1.103)\npython -m ufo.server.app --port 5003\n```\n\n**Step 2: Start all Linux Clients**\n\n```bash\n# On web server\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.101:5001/ws \\\n --client-id linux_web_server \\\n --platform linux\n\n# On database server\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.102:5002/ws \\\n --client-id linux_db_server \\\n --platform linux\n\n# On monitoring server\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.103:5003/ws \\\n --client-id linux_monitoring \\\n --platform linux\n```\n\n**Step 3: Start all MCP Services**\n\n```bash\n# On each Linux machine\npython -m ufo.client.mcp.http_servers.linux_mcp_server\n```\n\n**Step 4: Launch Galaxy**\n\n```bash\n# On your control machine (interactive mode)\npython -m galaxy --interactive\n```\n\n**Or launch with a specific request:**\n\n```bash\npython -m galaxy \"Your task description here\"\n```\n\nGalaxy will automatically connect to all configured devices and display the orchestration interface.\n\n## Example Multi-Device Workflows\n\n### Workflow 1: Cross-Platform Data Processing\n\n**User Request:**\n> \"Generate a sales report in Excel from the database, then email it to the team\"\n\n**Galaxy Orchestration:**\n\n```mermaid\nsequenceDiagram\n participant User\n participant Galaxy\n participant LinuxDB as Linux DB Server\n participant WinDesktop as Windows Desktop\n \n User->>Galaxy: Request sales report\n Galaxy->>Galaxy: Decompose task\n \n Note over Galaxy,LinuxDB: Subtask 1: Extract data\n Galaxy->>LinuxDB: \"Export sales data from PostgreSQL to CSV\"\n LinuxDB->>LinuxDB: Execute SQL query\n LinuxDB->>LinuxDB: Generate CSV file\n LinuxDB-->>Galaxy: CSV file location\n \n Note over Galaxy,WinDesktop: Subtask 2: Create Excel report\n Galaxy->>WinDesktop: \"Create Excel report from CSV\"\n WinDesktop->>WinDesktop: Open Excel\n WinDesktop->>WinDesktop: Import CSV\n WinDesktop->>WinDesktop: Format report\n WinDesktop-->>Galaxy: Excel file created\n \n Note over Galaxy,WinDesktop: Subtask 3: Send email\n Galaxy->>WinDesktop: \"Email report to team\"\n WinDesktop->>WinDesktop: Open Outlook\n WinDesktop->>WinDesktop: Attach file\n WinDesktop->>WinDesktop: Send email\n WinDesktop-->>Galaxy: Email sent\n \n Galaxy-->>User: Task completed\n```\n\n### Workflow 2: Multi-Server Log Analysis\n\n**User Request:**\n> \"Check all servers for error patterns in the last hour and summarize findings\"\n\n**Galaxy Orchestration:**\n\n1. **Linux Web Server**: Analyze nginx logs for HTTP 500 errors\n2. **Linux DB Server**: Check PostgreSQL logs for query failures\n3. **Linux Monitoring**: Review Prometheus alerts\n4. **Galaxy**: Aggregate results and generate summary report\n\n### Workflow 3: Deployment Pipeline\n\n**User Request:**\n> \"Deploy the new application version to production\"\n\n**Galaxy Orchestration:**\n\n1. **Linux CI/CD Server**: Build Docker image from Git repository\n2. **Linux Web Server**: Stop current service, pull new image, restart\n3. **Linux DB Server**: Run database migrations\n4. **Linux Monitoring**: Verify health checks and metrics\n5. **Windows Desktop**: Send deployment notification email\n\n---\n\n## Task Assignment Behavior\n\n### How Galaxy Routes Tasks to Linux Agents\n\nGalaxy's ConstellationAgent uses several factors to select the appropriate device for each subtask:\n\n| Factor | Description | Example |\n|--------|-------------|---------|\n| **Capabilities** | Match subtask requirements to device capabilities | `\"database_server\"` → DB server agent |\n| **OS Requirement** | Platform-specific tasks routed to correct OS | Linux commands → Linux agents |\n| **Metadata Context** | Use device-specific paths and configurations | Log analysis → agent with correct log path |\n| **Device Status** | Only assign to online, healthy devices | Skip offline or failing devices |\n| **Load Balancing** | Distribute tasks across similar devices | Round-robin across web servers |\n\n### Example Task Decomposition\n\n**User Request:**\n> \"Monitor system health across all servers and alert if any issues found\"\n\n**Galaxy Decomposition:**\n\n```yaml\nTask 1:\n Description: \"Check web server health\"\n Target: linux_web_server\n Reason: Has \"web_server\" capability\n \nTask 2:\n Description: \"Check database health\"\n Target: linux_db_server\n Reason: Has \"database_server\" capability\n \nTask 3:\n Description: \"Review monitoring alerts\"\n Target: linux_monitoring\n Reason: Has \"monitoring\" capability\n \nTask 4:\n Description: \"Aggregate results and send alert email\"\n Target: windows_desktop_1\n Reason: Has \"email\" capability\n```\n\n## Critical Configuration Requirements\n\n!!!danger \"Configuration Validation\"\n Ensure these match exactly or Galaxy cannot control the device:\n \n - **Device ID**: `device_id` in `devices.yaml` must match `--client-id` in client command\n - **Server URL**: `server_url` in `devices.yaml` must match `--ws-server` in client command \n - **Platform**: Must include `--platform linux` in client command\n\n## Monitoring & Debugging\n\n### Verify Device Registration\n\n**Check Galaxy device pool:**\n\n```bash\n# List all connected devices\ncurl http://:5000/api/devices\n```\n\n**Expected response:**\n\n```json\n{\n \"devices\": [\n {\n \"device_id\": \"linux_web_server\",\n \"os\": \"linux\",\n \"status\": \"online\",\n \"capabilities\": [\"web_server\", \"nginx\", \"log_analysis\"]\n },\n {\n \"device_id\": \"linux_db_server\",\n \"os\": \"linux\",\n \"status\": \"online\",\n \"capabilities\": [\"database_server\", \"postgresql\"]\n }\n ]\n}\n```\n\n### View Task Assignments\n\nGalaxy logs show task routing decisions:\n\n```log\nINFO - [Galaxy] Task decomposition: 3 subtasks created\nINFO - [Galaxy] Subtask 1 → linux_web_server (capability match: web_server)\nINFO - [Galaxy] Subtask 2 → linux_db_server (capability match: database_server)\nINFO - [Galaxy] Subtask 3 → windows_desktop_1 (capability match: email)\n```\n\n### Troubleshooting Device Connection\n\n**Issue**: Linux agent not appearing in Galaxy device pool\n\n**Diagnosis:**\n\n1. Check if client is connected to server:\n ```bash\n curl http://192.168.1.101:5001/api/clients\n ```\n\n2. Verify `devices.yaml` configuration matches client parameters\n\n3. Check Galaxy logs for connection errors\n\n4. Ensure `auto_connect: true` in `devices.yaml`\n\n## Related Documentation\n\n- [Linux Agent Overview](overview.md) - Architecture and design principles\n- [Quick Start Guide](../getting_started/quick_start_linux.md) - Step-by-step setup\n- [Galaxy Overview](../galaxy/overview.md) - Multi-device orchestration framework\n- [Galaxy Quick Start](../getting_started/quick_start_galaxy.md) - Galaxy deployment guide\n- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Task orchestration\n- [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md) - Complete device configuration reference\n\n## Summary\n\nUsing Linux Agent as a Galaxy device enables multi-device orchestration with capability-based routing, metadata context for LLM-aware command generation, parallel execution, and seamless cross-platform workflows between Linux and Windows agents.\n\n"
},
{
"path": "documents/docs/linux/commands.md",
"content": "# LinuxAgent MCP Commands\n\nLinuxAgent interacts with Linux systems through MCP (Model Context Protocol) tools provided by the Linux MCP Server. These tools provide atomic building blocks for CLI task execution, isolating system-specific operations within the MCP server layer.\n\n## Command Architecture\n\n### MCP Server Integration\n\nLinuxAgent commands are executed through the MCP server infrastructure:\n\n```mermaid\ngraph LR\n A[LinuxAgent] --> B[Command Dispatcher]\n B --> C[MCP Server]\n C --> D[Linux Shell]\n D --> E[stdout/stderr]\n E --> C\n C --> B\n B --> A\n```\n\n### Command Dispatcher\n\nThe command dispatcher routes commands to the appropriate MCP server:\n\n```python\nfrom aip.messages import Command\n\n# Create command\ncommand = Command(\n tool_name=\"execute_command\",\n parameters={\"command\": \"df -h\", \"timeout\": 30},\n tool_type=\"action\"\n)\n\n# Execute command via dispatcher\nresults = await command_dispatcher.execute_commands([command])\nexecution_result = results[0].result\n```\n\n## Primary MCP Tools\n\n### 1. execute_command - Execute Shell Commands\n\n**Purpose**: Execute arbitrary shell commands and capture structured results.\n\n#### Tool Specification\n\n```python\ntool_name = \"execute_command\"\nparameters = {\n \"command\": \"df -h\", # Shell command to execute\n \"timeout\": 30, # Execution timeout (seconds, default: 30)\n \"cwd\": \"/home/user\" # Optional working directory\n}\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant Dispatcher\n participant MCP\n participant Shell\n \n Agent->>Dispatcher: execute_command: df -h\n Dispatcher->>MCP: Forward command\n MCP->>Shell: Execute: df -h\n Shell->>Shell: Run command\n Shell-->>MCP: stdout + stderr + exit_code\n MCP->>MCP: Structure result\n MCP-->>Dispatcher: Execution result\n Dispatcher-->>Agent: Structured result\n```\n\n#### Result Structure\n\n```python\n{\n \"success\": True, # Boolean indicating success\n \"exit_code\": 0, # Process exit code\n \"stdout\": \"Filesystem Size Used Avail Use% Mounted on\\n/dev/sda1 100G 50G 46G 52% /\\n\",\n \"stderr\": \"\" # Standard error output\n}\n```\n\n#### Common Use Cases\n\n| Use Case | Command Example | Description |\n|----------|----------------|-------------|\n| **File Operations** | `ls -la /home/user` | List directory contents |\n| **Text Processing** | `grep \"error\" /var/log/syslog` | Search log files |\n| **System Monitoring** | `top -bn1` | Check system processes |\n| **Disk Management** | `df -h` | Check disk space |\n| **Network Operations** | `ping -c 4 example.com` | Test network connectivity |\n| **Archive Creation** | `tar -czf backup.tar.gz /data` | Create compressed archives |\n| **Package Management** | `apt list --installed` | List installed packages |\n\n#### Error Handling\n\n**Exit Code Interpretation**:\n\n- **0**: Success\n- **1-125**: Command-specific errors\n- **126**: Command not executable\n- **127**: Command not found\n- **128+n**: Terminated by signal n\n\n**Example Error Result**:\n\n```python\n{\n \"success\": False,\n \"error\": \"Command not found: invalid_cmd\"\n}\n```\n\n#### Security Considerations\n\n!!!warning \"Command Safety\"\n The MCP server blocks dangerous commands including:\n \n - `rm -rf /` - Recursive root deletion\n - Fork bombs - `:(){ :|:& };:`\n - `mkfs` - Filesystem formatting\n - `dd if=/dev/zero` - Device overwriting\n - `shutdown`, `reboot` - System shutdown\n \n Commands execute with user permissions, no automatic privilege escalation. Timeout protection prevents hung processes.\n\n### 2. get_system_info - Collect System Information\n\n**Purpose**: Gather basic Linux system information using standard commands.\n\n#### Tool Specification\n\n```python\ntool_name = \"get_system_info\"\nparameters = {} # No parameters required\n```\n\n#### Information Collected\n\nThe tool executes these commands and returns their output:\n\n| Info Type | Command | Data Returned |\n|-----------|---------|---------------|\n| **uname** | `uname -a` | System and kernel information |\n| **uptime** | `uptime` | System uptime and load averages |\n| **memory** | `free -h` | Memory usage statistics (human-readable) |\n| **disk** | `df -h` | Disk space for all mounted filesystems |\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant Dispatcher\n participant MCP\n participant System\n \n Agent->>Dispatcher: get_system_info\n Dispatcher->>MCP: Forward request\n MCP->>System: Execute uname, uptime, free, df\n System-->>MCP: Command outputs\n MCP->>MCP: Aggregate results\n MCP-->>Dispatcher: Structured info\n Dispatcher-->>Agent: System information\n```\n\n#### Result Example\n\n```python\n{\n \"uname\": \"Linux hostname 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux\",\n \"uptime\": \" 14:23:45 up 5 days, 3:12, 2 users, load average: 0.52, 0.58, 0.59\",\n \"memory\": \" total used free shared buff/cache available\\nMem: 15Gi 8.2Gi 1.5Gi 256Mi 5.8Gi 7.0Gi\\nSwap: 8.0Gi 512Mi 7.5Gi\",\n \"disk\": \"Filesystem Size Used Avail Use% Mounted on\\n/dev/sda1 100G 50G 46G 52% /\\n/dev/sdb1 500G 200G 276G 42% /data\"\n}\n```\n\n## Command Execution Pipeline\n\n### Atomic Building Blocks\n\nThe MCP tools `execute_command` and `get_system_info` serve as atomic operations:\n\n```mermaid\ngraph TD\n A[User Request] --> B[LLM Reasoning]\n B --> C{Select Tool}\n C -->|Execute CLI| D[execute_command]\n C -->|Get System Info| E[get_system_info]\n \n D --> F[Capture Result]\n E --> F\n \n F --> G[Update Memory]\n G --> H{Task Complete?}\n H -->|No| B\n H -->|Yes| I[FINISH]\n```\n\n### Isolation of System Operations\n\nBy isolating system operations in the MCP server layer, the architecture achieves clear separation: the Agent layer focuses on LLM reasoning and workflow orchestration, while the MCP layer handles system-specific command execution. This provides testability (commands can be mocked) and portability (MCP servers can be deployed remotely).\n\n## Command Composition\n\n### Sequential Execution\n\nLinuxAgent executes commands sequentially, building on previous results:\n\n```python\n# Round 1: Check disk space\n{\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\"command\": \"df -h /data\"}\n }\n}\n# Result: 276GB available\n\n# Round 2: Create backup (informed by Round 1 result)\n{\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\n \"command\": \"tar -czf /data/backup.tar.gz /home/user\"\n }\n }\n}\n```\n\n### Conditional Execution\n\nLLM can make decisions based on command results:\n\n```python\n# Round 1: Check if file exists\n{\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\"command\": \"ls /data/backup.tar.gz\"}\n }\n}\n# Result: exit_code=2 (file not found)\n\n# Round 2: File doesn't exist, create it\n{\n \"thought\": \"Backup doesn't exist, creating new one\",\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\"command\": \"tar -czf /data/backup.tar.gz /home/user\"}\n }\n}\n```\n\n### Error Recovery\n\nCommands can be retried or alternatives attempted:\n\n```python\n# Round 1: Try privileged command\n{\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\"command\": \"systemctl restart nginx\"}\n }\n}\n# Result: success=False, error=\"Permission denied\"\n\n# Round 2: Switch to user-level alternative\n{\n \"thought\": \"Don't have sudo access, using alternative approach\",\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\"command\": \"kill -HUP $(cat /var/run/nginx.pid)\"}\n }\n}\n```\n\n---\n\n## Command Implementation\n\n### MCP Server Location\n\nThe MCP server implementation for Linux commands can be found in:\n\n```\nufo/client/mcp/http_servers/\n└── linux_mcp_server.py\n```\n\n### Example Implementation Skeleton\n\n```python\nclass LinuxMCPServer:\n \"\"\"MCP server for Linux CLI commands\"\"\"\n \n @mcp.tool()\n async def execute_command(\n self, \n command: str, \n timeout: int = 30,\n cwd: Optional[str] = None\n ) -> Dict:\n \"\"\"Execute a shell command\"\"\"\n # Block dangerous commands\n dangerous = [\"rm -rf /\", \":(){ :|:& };:\", \"mkfs\", ...]\n if any(d in command.lower() for d in dangerous):\n return {\"success\": False, \"error\": \"Blocked dangerous command.\"}\n \n try:\n proc = await asyncio.create_subprocess_shell(\n command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n cwd=cwd\n )\n try:\n stdout, stderr = await asyncio.wait_for(\n proc.communicate(), \n timeout=timeout\n )\n except asyncio.TimeoutError:\n proc.kill()\n await proc.wait()\n return {\"success\": False, \"error\": f\"Timeout after {timeout}s.\"}\n \n return {\n \"success\": proc.returncode == 0,\n \"exit_code\": proc.returncode,\n \"stdout\": stdout.decode(\"utf-8\", errors=\"replace\"),\n \"stderr\": stderr.decode(\"utf-8\", errors=\"replace\")\n }\n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n \n @mcp.tool()\n async def get_system_info(self) -> Dict:\n \"\"\"Collect system information\"\"\"\n info = {}\n cmds = {\n \"uname\": \"uname -a\",\n \"uptime\": \"uptime\",\n \"memory\": \"free -h\",\n \"disk\": \"df -h\"\n }\n \n for k, cmd in cmds.items():\n try:\n proc = await asyncio.create_subprocess_shell(\n cmd, stdout=asyncio.subprocess.PIPE\n )\n out, _ = await proc.communicate()\n info[k] = out.decode(\"utf-8\", errors=\"replace\").strip()\n except Exception as e:\n info[k] = f\"Error: {e}\"\n \n return info\n```\n\n---\n\n## Best Practices\n\n### Tool Usage\n\n- Use `get_system_info` for quick system overview\n- Use `execute_command` for custom or complex operations\n- Check `success` field and `exit_code` to detect errors\n- Parse `stdout` for structured data when possible\n- Set timeouts appropriately to prevent hung processes\n\n### Security\n\n!!!warning \"Security Best Practices\"\n The MCP server has built-in protections, but be cautious:\n \n - Dangerous commands are automatically blocked\n - Commands execute with user permissions only\n - Avoid sudo when possible (requires user interaction)\n - Sanitize outputs before logging (may contain sensitive data)\n\n### Error Handling\n\n- Check `success` field before considering command successful\n- Parse `stderr` for error messages\n- Implement retries for transient errors\n- Provide alternatives when primary approach fails\n\n## Comparison with Other Agent Commands\n\n| Agent | Command Types | Execution Layer | Result Format |\n|-------|--------------|-----------------|---------------|\n| **LinuxAgent** | CLI + SysInfo | MCP server | success/exit_code/stdout/stderr |\n| **AppAgent** | UI + API | Automator + MCP | UI state + API responses |\n| **HostAgent** | Desktop + Shell | Automator + MCP | Desktop state + results |\n\nLinuxAgent's command set is intentionally minimal and focused:\n\n- **execute_command**: General-purpose command execution\n- **get_system_info**: Standardized system information\n\nThis simplicity reflects the CLI environment's text-based, command-driven nature.\n\n## Next Steps\n\n- [State Machine](state.md) - Understand how command execution fits into the FSM\n- [Processing Strategy](strategy.md) - See how commands are integrated into the 3-phase pipeline\n- [Overview](overview.md) - Return to LinuxAgent architecture overview\n- [MCP Overview](../mcp/overview.md) - MCP server implementation details\n"
},
{
"path": "documents/docs/linux/overview.md",
"content": "# LinuxAgent: CLI Task Executor\n\n**LinuxAgent** is a specialized lightweight agent designed for executing command-line instructions on Linux systems. It demonstrates how a standalone device agent can leverage the layered FSM architecture and server-client design to perform intelligent, iterative task execution in a CLI-based environment.\n\n**Quick Links:**\n\n- New to Linux Agent? Start with the [Quick Start Guide](../getting_started/quick_start_linux.md)\n- Using as Sub-Agent in Galaxy? See [Using Linux Agent as Galaxy Device](as_galaxy_device.md)\n\n## Architecture Overview\n\nLinuxAgent operates as a single-agent instance that interacts with Linux systems through command-line interface (CLI) commands. Unlike the two-tier architecture of UFO (HostAgent + AppAgent), LinuxAgent uses a simplified single-agent model optimized for shell-based automation.\n\n## Core Responsibilities\n\nLinuxAgent provides the following capabilities for Linux CLI automation:\n\n### Command-Line Execution\n\nLinuxAgent interprets user requests and translates them into appropriate shell commands for execution on Linux systems.\n\n**Example:** User request \"Check disk space and create a backup\" becomes:\n\n1. Execute `df -h` to check disk space\n2. Execute `tar -czf backup.tar.gz /data` to create backup\n\n### System Information Collection\n\nThe agent can proactively gather system-level information to inform decision-making:\n\n- Memory usage (`free -h`)\n- Disk space (`df -h`)\n- Process status (`ps aux`)\n- Hardware configuration (`lscpu`, `lshw`)\n\n### Iterative Task Execution\n\nLinuxAgent executes tasks iteratively, evaluating execution outcomes at each step and determining the next action based on results and LLM reasoning.\n\n### Error Handling and Recovery\n\nThe agent monitors command execution results (`stdout`, `stderr`, exit codes) and can adapt its strategy when errors occur.\n\n## Key Characteristics\n\n- **Scope**: Single Linux system (CLI-based automation)\n- **Lifecycle**: One instance per task session\n- **Hierarchy**: Standalone agent (no child agents)\n- **Communication**: Direct MCP server integration\n- **Control**: 3-state finite state machine with 3-phase processing pipeline\n\n## Execution Workflow\n\n```mermaid\nsequenceDiagram\n participant User\n participant LinuxAgent\n participant LLM\n participant MCPServer\n participant Linux\n \n User->>LinuxAgent: \"Check disk space and create backup\"\n LinuxAgent->>LinuxAgent: State: CONTINUE\n LinuxAgent->>LLM: Send prompt with request & context\n LLM-->>LinuxAgent: Return command: df -h\n LinuxAgent->>MCPServer: execute_command: df -h\n MCPServer->>Linux: Execute command\n Linux-->>MCPServer: stdout + stderr\n MCPServer-->>LinuxAgent: Execution result\n LinuxAgent->>LinuxAgent: Update memory\n \n LinuxAgent->>LLM: Send prompt with previous result\n LLM-->>LinuxAgent: Return command: tar -czf ...\n LinuxAgent->>MCPServer: execute_command: tar -czf backup.tar.gz /data\n MCPServer->>Linux: Execute command\n Linux-->>MCPServer: stdout + stderr\n MCPServer-->>LinuxAgent: Execution result\n LinuxAgent->>LinuxAgent: State: FINISH\n LinuxAgent-->>User: Task completed\n```\n\n## Comparison with UFO Agents\n\n| Aspect | LinuxAgent | HostAgent | AppAgent |\n|--------|------------|-----------|----------|\n| **Platform** | Linux (CLI) | Windows Desktop | Windows Applications |\n| **States** | 3 (CONTINUE, FINISH, FAIL) | 7 states | 6 states |\n| **Architecture** | Single-agent | Parent orchestrator | Child executor |\n| **Interface** | Command-line | Desktop GUI + Shell | Application GUI + API |\n| **Processing Phases** | 3 phases | 4 phases | 4 phases |\n| **MCP Tools** | execute_command, get_system_info | Desktop commands | UI + API commands |\n\n## Design Principles\n\nLinuxAgent exemplifies a minimal viable design for single-agent systems with a small state set (only 3 states for deterministic control flow), modular strategies (clear separation between LLM interaction, action execution, and memory updates), well-defined commands (atomic CLI operations isolated in MCP server layer), proactive information gathering (on-demand system info collection), and traceable execution (complete logging of commands, results, and state transitions).\n\n## Deep Dive Topics\n\nExplore the detailed architecture and implementation:\n\n- [State Machine](state.md) - 3-state FSM lifecycle and transitions\n- [Processing Strategy](strategy.md) - 3-phase pipeline (LLM, Action, Memory)\n- [MCP Commands](commands.md) - CLI execution and system information commands\n\n## Use Cases\n\nLinuxAgent is ideal for:\n\n- **System Administration**: Automated system maintenance and monitoring\n- **DevOps Tasks**: Deployment scripts, log analysis, configuration management\n- **Data Processing**: File operations, text processing, batch jobs\n- **Monitoring & Alerts**: System health checks and automated responses\n- **Cross-Device Workflows**: As a sub-agent in Galaxy multi-device orchestration\n\n!!!tip \"Galaxy Integration\"\n LinuxAgent can serve as a device agent in Galaxy's multi-device orchestration framework, executing Linux-specific tasks as part of cross-platform workflows alongside Windows and other devices.\n \n See [Using Linux Agent as Galaxy Device](as_galaxy_device.md) for configuration details.\n\n## Implementation Location\n\nThe LinuxAgent implementation can be found in:\n\n```\nufo/\n├── agents/\n│ ├── agent/\n│ │ └── customized_agent.py # LinuxAgent class definition\n│ ├── states/\n│ │ └── linux_agent_state.py # State machine implementation\n│ └── processors/\n│ ├── customized/\n│ │ └── customized_agent_processor.py # LinuxAgentProcessor\n│ └── strategies/\n│ └── linux_agent_strategy.py # Processing strategies\n```\n\n## Next Steps\n\nTo understand LinuxAgent's complete architecture:\n\n1. [State Machine](state.md) - Learn about the 3-state FSM\n2. [Processing Strategy](strategy.md) - Understand the 3-phase pipeline\n3. [MCP Commands](commands.md) - Explore CLI command execution\n\nFor deployment and configuration, see the [Getting Started Guide](../getting_started/quick_start_linux.md).\n"
},
{
"path": "documents/docs/linux/state.md",
"content": "# LinuxAgent State Machine\n\nLinuxAgent uses a **3-state finite state machine (FSM)** to manage CLI task execution flow. The minimal state set captures essential execution progression while maintaining simplicity and predictability. States transition based on LLM decisions and command execution results.\n\n## State Machine Architecture\n\n### State Enumeration\n\n```python\nclass LinuxAgentStatus(Enum):\n \"\"\"Store the status of the linux agent\"\"\"\n CONTINUE = \"CONTINUE\" # Task is ongoing, requires further commands\n FINISH = \"FINISH\" # Task completed successfully\n FAIL = \"FAIL\" # Task cannot proceed, unrecoverable error\n```\n\n### State Management\n\nLinuxAgent states are managed by `LinuxAgentStateManager`, which implements the agent state registry pattern:\n\n```python\nclass LinuxAgentStateManager(AgentStateManager):\n \"\"\"Manages the states of the linux agent\"\"\"\n _state_mapping: Dict[str, Type[LinuxAgentState]] = {}\n \n @property\n def none_state(self) -> AgentState:\n return NoneLinuxAgentState()\n```\n\nAll LinuxAgent states are registered using the `@LinuxAgentStateManager.register` decorator, enabling dynamic state lookup by name.\n\n## State Transition Diagram\n\n\n \n Figure: Lifecycle state transitions of the LinuxAgent. The agent starts in CONTINUE state, executes CLI commands iteratively, and transitions to FINISH upon completion or FAIL upon encountering unrecoverable errors.\n\n\n## State Definitions\n\n### 1. CONTINUE State\n\n**Purpose**: Active execution state where LinuxAgent processes the user request and executes CLI commands.\n\n```python\n@LinuxAgentStateManager.register\nclass ContinueLinuxAgentState(LinuxAgentState):\n \"\"\"The class for the continue linux agent state\"\"\"\n \n async def handle(self, agent: \"LinuxAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Execute the 3-phase processing pipeline\"\"\"\n await agent.process(context)\n \n def is_round_end(self) -> bool:\n return False # Round continues\n \n def is_subtask_end(self) -> bool:\n return False # Subtask continues\n \n @classmethod\n def name(cls) -> str:\n return LinuxAgentStatus.CONTINUE.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Active |\n| **Processor Executed** | ✓ Yes (3 phases) |\n| **Round Ends** | No |\n| **Subtask Ends** | No |\n| **Duration** | Single round |\n| **Next States** | CONTINUE, FINISH, FAIL |\n\n**Behavior**:\n\n1. Constructs prompts with previous execution results\n2. Gets next CLI command from LLM\n3. Executes command via MCP server\n4. Updates memory with execution results\n5. Determines next state based on LLM response\n\n**State Transition Logic**:\n\n- **CONTINUE → CONTINUE**: Task requires more commands to complete\n- **CONTINUE → FINISH**: LLM determines task is complete\n- **CONTINUE → FAIL**: Unrecoverable error encountered (e.g., permission denied, resource unavailable)\n\n### 2. FINISH State\n\n**Purpose**: Terminal state indicating successful task completion.\n\n```python\n@LinuxAgentStateManager.register\nclass FinishLinuxAgentState(LinuxAgentState):\n \"\"\"The class for the finish linux agent state\"\"\"\n \n def next_agent(self, agent: \"LinuxAgent\") -> \"LinuxAgent\":\n return agent\n \n def next_state(self, agent: \"LinuxAgent\") -> LinuxAgentState:\n return FinishLinuxAgentState() # Remains in FINISH\n \n def is_subtask_end(self) -> bool:\n return True # Subtask completed\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n @classmethod\n def name(cls) -> str:\n return LinuxAgentStatus.FINISH.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Subtask Ends** | Yes |\n| **Duration** | Permanent |\n| **Next States** | FINISH (no transition) |\n\n**Behavior**:\n\n- Signals task completion to session manager\n- No further processing occurs\n- Agent instance can be terminated\n\nFINISH state is reached when all required CLI commands have been executed successfully, the LLM determines the user request has been fulfilled, and no errors or exceptions occurred during execution.\n\n### 3. FAIL State\n\n**Purpose**: Terminal state indicating task failure due to unrecoverable errors.\n\n```python\n@LinuxAgentStateManager.register\nclass FailLinuxAgentState(LinuxAgentState):\n \"\"\"The class for the fail linux agent state\"\"\"\n \n def next_agent(self, agent: \"LinuxAgent\") -> \"LinuxAgent\":\n return agent\n \n def next_state(self, agent: \"LinuxAgent\") -> LinuxAgentState:\n return FinishLinuxAgentState() # Transitions to FINISH for cleanup\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n def is_subtask_end(self) -> bool:\n return True # Subtask failed\n \n @classmethod\n def name(cls) -> str:\n return LinuxAgentStatus.FAIL.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal (Error) |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Subtask Ends** | Yes |\n| **Duration** | Transitions to FINISH |\n| **Next States** | FINISH |\n\n**Behavior**:\n\n- Logs failure reason and context\n- Transitions to FINISH state for cleanup\n- Session manager receives failure status\n\n!!!error \"Failure Conditions\"\n FAIL state is reached when insufficient privileges prevent command execution, required system resources are not accessible (disk full, network unreachable), repeated command syntax errors occur, the LLM explicitly indicates task cannot be completed, or task requirements exceed current system capabilities.\n\n**Error Recovery**:\n\nWhile FAIL is a terminal state, the error information is logged for debugging:\n\n```python\n# Example error logging in FAIL state\nagent.logger.error(f\"Task failed: {error_message}\")\nagent.logger.debug(f\"Last command: {last_command}\")\nagent.logger.debug(f\"Command output: {stderr}\")\n```\n\n## State Transition Rules\n\n### Transition Decision Logic\n\nState transitions are determined by the LLM's response in the **CONTINUE** state:\n\n```python\n# LLM returns status in response\nparsed_response = {\n \"action\": {\n \"command\": \"df -h\",\n \"status\": \"CONTINUE\" # or \"FINISH\" or \"FAIL\"\n },\n \"thought\": \"Need to check disk space first\"\n}\n\n# Agent updates its status based on LLM decision\nagent.status = parsed_response[\"action\"][\"status\"]\nnext_state = LinuxAgentStateManager().get_state(agent.status)\n```\n\n### Transition Matrix\n\n| Current State | Condition | Next State | Trigger |\n|---------------|-----------|------------|---------|\n| **CONTINUE** | LLM returns CONTINUE | CONTINUE | More commands needed |\n| **CONTINUE** | LLM returns FINISH | FINISH | Task completed |\n| **CONTINUE** | LLM returns FAIL | FAIL | Unrecoverable error |\n| **CONTINUE** | Exception raised | FAIL | System error |\n| **FINISH** | Any | FINISH | No transition |\n| **FAIL** | Any | FINISH | Cleanup transition |\n\n## State-Specific Processing\n\n### CONTINUE State Processing Pipeline\n\nWhen in CONTINUE state, LinuxAgent executes the full 3-phase pipeline:\n\n```mermaid\ngraph TD\n A[CONTINUE State] --> B[Phase 1: LLM Interaction]\n B --> C[Phase 2: Action Execution]\n C --> D[Phase 3: Memory Update]\n D --> E{Check Status}\n E -->|CONTINUE| A\n E -->|FINISH| F[FINISH State]\n E -->|FAIL| G[FAIL State]\n```\n\n### Terminal States (FINISH / FAIL)\n\nTerminal states perform no processing:\n\n- **FINISH**: Clean termination, results available in memory\n- **FAIL**: Error termination, error details logged\n\n## Deterministic Control Flow\n\nThe 3-state design ensures deterministic, traceable execution with predictable behavior (every execution path is well-defined), debuggability (state transitions are logged and traceable), testability (finite state space simplifies testing), and maintainability (simple state set reduces complexity).\n\n## Comparison with Other Agents\n\n| Agent | States | Complexity | Use Case |\n|-------|--------|------------|----------|\n| **LinuxAgent** | 3 | Minimal | CLI task execution |\n| **AppAgent** | 6 | Moderate | Windows app automation |\n| **HostAgent** | 7 | High | Desktop orchestration |\n\nLinuxAgent's minimal 3-state design reflects its focused scope: execute CLI commands to fulfill user requests. The simplified state machine eliminates unnecessary complexity while maintaining robust error handling and completion detection.\n\n## Implementation Details\n\nThe state machine implementation can be found in:\n\n```\nufo/agents/states/linux_agent_state.py\n```\n\nKey classes:\n\n- `LinuxAgentStatus`: State enumeration\n- `LinuxAgentStateManager`: State registry and lookup\n- `LinuxAgentState`: Abstract base class\n- `ContinueLinuxAgentState`: Active execution state\n- `FinishLinuxAgentState`: Successful completion state\n- `FailLinuxAgentState`: Error termination state\n- `NoneLinuxAgentState`: Initial/undefined state\n\n## Next Steps\n\n- [Processing Strategy](strategy.md) - Understand the 3-phase processing pipeline executed in CONTINUE state\n- [MCP Commands](commands.md) - Explore CLI command execution and system information retrieval\n- [Overview](overview.md) - Return to LinuxAgent architecture overview\n"
},
{
"path": "documents/docs/linux/strategy.md",
"content": "# LinuxAgent Processing Strategy\n\nLinuxAgent executes a **3-phase processing pipeline** in the **CONTINUE** state. Each phase handles a specific aspect of CLI task execution: LLM decision making, action execution, and memory recording. This streamlined design separates prompt construction and LLM reasoning from command execution and state updates, enhancing modularity and traceability.\n\n## Strategy Assembly\n\nProcessing strategies are assembled and orchestrated by the `LinuxAgentProcessor` class defined in `ufo/agents/processors/customized/customized_agent_processor.py`. The processor coordinates the 3-phase pipeline execution.\n\n### LinuxAgentProcessor Overview\n\nThe `LinuxAgentProcessor` extends `CustomizedProcessor` and manages the Linux-specific workflow:\n\n```python\nclass LinuxAgentProcessor(CustomizedProcessor):\n \"\"\"\n Processor for Linux MCP Agent.\n Manages CLI command execution workflow with:\n - LLM-based command generation\n - MCP-based command execution\n - Memory-based result tracking\n \"\"\"\n \n def _setup_strategies(self) -> None:\n \"\"\"Setup the 3-phase processing pipeline\"\"\"\n \n # Phase 1: LLM Interaction (critical - fail_fast=True)\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n LinuxLLMInteractionStrategy(fail_fast=True)\n )\n \n # Phase 2: Action Execution (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n LinuxActionExecutionStrategy(fail_fast=False)\n )\n \n # Phase 3: Memory Update (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(fail_fast=False)\n )\n```\n\n### Strategy Registration\n\n| Phase | Strategy Class | fail_fast | Rationale |\n|-------|---------------|-----------|-----------|\n| **LLM_INTERACTION** | `LinuxLLMInteractionStrategy` | ✓ True | LLM failure requires immediate recovery |\n| **ACTION_EXECUTION** | `LinuxActionExecutionStrategy` | ✗ False | Command failures can be handled gracefully |\n| **MEMORY_UPDATE** | `AppMemoryUpdateStrategy` | ✗ False | Memory failures shouldn't block execution |\n\n**Fail-Fast vs Graceful:**\n\n- **fail_fast=True**: Critical phases where errors should immediately transition to FAIL state\n- **fail_fast=False**: Non-critical phases where errors can be logged and execution continues\n\n## Three-Phase Pipeline\n\n### Pipeline Execution Flow\n\n```mermaid\ngraph LR\n A[CONTINUE State] --> B[Phase 1: LLM Interaction]\n B --> C[Phase 2: Action Execution]\n C --> D[Phase 3: Memory Update]\n D --> E[Determine Next State]\n E --> F{Status?}\n F -->|CONTINUE| A\n F -->|FINISH| G[FINISH State]\n F -->|FAIL| H[FAIL State]\n```\n\n## Phase 1: LLM Interaction Strategy\n\n**Purpose**: Construct prompts with execution context and obtain next CLI command from LLM.\n\n### Strategy Implementation\n\n```python\n@depends_on(\"request\")\n@provides(\"parsed_response\", \"response_text\", \"llm_cost\", \n \"prompt_message\", \"action\", \"thought\", \"comment\")\nclass LinuxLLMInteractionStrategy(AppLLMInteractionStrategy):\n \"\"\"\n Strategy for LLM interaction with Linux Agent specific prompting.\n \n Handles:\n - Context-aware prompt construction with previous results\n - LLM interaction with retry logic\n - Response parsing and validation\n \"\"\"\n \n async def execute(self, agent: \"LinuxAgent\", \n context: ProcessingContext) -> ProcessingResult:\n \"\"\"Execute LLM interaction for Linux Agent\"\"\"\n```\n\n### Phase 1 Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Agent\n participant Prompter\n participant LLM\n \n Strategy->>Agent: Get previous plan\n Strategy->>Agent: Get blackboard context\n Agent-->>Strategy: Previous execution results\n \n Strategy->>Prompter: Construct prompt\n Prompter->>Prompter: Build system message\n Prompter->>Prompter: Build user message with context\n Prompter-->>Strategy: Complete prompt\n \n Strategy->>LLM: Send prompt\n LLM-->>Strategy: CLI command + status\n \n Strategy->>Strategy: Parse response\n Strategy->>Strategy: Validate command\n Strategy-->>Agent: Parsed response + cost\n```\n\n### Prompt Construction\n\nThe strategy constructs comprehensive prompts using:\n\n1. **System Message**: Agent role and capabilities\n2. **User Request**: Original task description\n3. **Previous Results**: Command outputs from prior executions\n4. **Blackboard Context**: Shared state from other agents (if any)\n5. **Last Success Actions**: Previously successful commands\n\n```python\nprompt_message = agent.message_constructor(\n dynamic_examples=[], # Few-shot examples (optional)\n dynamic_knowledge=\"\", # Retrieved knowledge (optional)\n plan=plan, # Previous execution plan\n request=request, # User request\n blackboard_prompt=blackboard_prompt, # Shared context\n last_success_actions=last_success_actions # Successful commands\n)\n```\n\n### LLM Response Format\n\nThe LLM returns a structured response:\n\n```json\n{\n \"thought\": \"Need to check disk space before creating backup\",\n \"action\": {\n \"tool\": \"execute_command\",\n \"arguments\": {\n \"command\": \"df -h\"\n },\n \"status\": \"CONTINUE\"\n },\n \"comment\": \"Checking available disk space\"\n}\n```\n\n### Proactive Information Gathering\n\nLinuxAgent proactively requests system information when needed, eliminating unnecessary overhead and increasing responsiveness.\n\n### Error Handling\n\n```python\ntry:\n response_text, llm_cost = await self._get_llm_response(\n agent, prompt_message\n )\n parsed_response = self._parse_app_response(agent, response_text)\n \n return ProcessingResult(\n success=True,\n data={\n \"parsed_response\": parsed_response,\n \"response_text\": response_text,\n \"llm_cost\": llm_cost,\n ...\n }\n )\nexcept Exception as e:\n self.logger.error(f\"LLM interaction failed: {str(e)}\")\n return self.handle_error(e, ProcessingPhase.LLM_INTERACTION, context)\n```\n\n---\n\n## Phase 2: Action Execution Strategy\n\n**Purpose**: Execute CLI commands returned by LLM and capture structured results.\n\n### Strategy Implementation\n\n```python\nclass LinuxActionExecutionStrategy(AppActionExecutionStrategy):\n \"\"\"\n Strategy for executing actions in Linux Agent.\n \n Handles:\n - CLI command execution via MCP server\n - Result capturing (stdout, stderr, exit code)\n - Error handling and retry logic\n \"\"\"\n \n async def execute(self, agent: \"LinuxAgent\",\n context: ProcessingContext) -> ProcessingResult:\n \"\"\"Execute Linux Agent actions\"\"\"\n```\n\n### Phase 2 Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant MCP\n participant Linux\n \n Strategy->>Strategy: Extract command from LLM response\n Strategy->>MCP: execute_command: df -h\n \n MCP->>Linux: Execute shell command\n Linux-->>MCP: stdout + stderr + exit_code\n \n MCP-->>Strategy: Execution result\n Strategy->>Strategy: Create action info\n Strategy->>Strategy: Format for memory\n Strategy-->>Agent: Execution results\n```\n\n### Command Execution\n\nThe strategy dispatches commands to the MCP server:\n\n```python\n# Extract parsed LLM response\nparsed_response: AppAgentResponse = context.get_local(\"parsed_response\")\ncommand_dispatcher = context.global_context.command_dispatcher\n\n# Execute the command via MCP\nexecution_results = await self._execute_app_action(\n command_dispatcher,\n parsed_response.action\n)\n```\n\n### Result Capture\n\nExecution results are structured for downstream processing:\n\n```python\n{\n \"success\": True,\n \"exit_code\": 0,\n \"stdout\": \"Filesystem Size Used Avail Use% Mounted on\\n/dev/sda1 100G 50G 46G 52% /\",\n \"stderr\": \"\"\n}\n```\n\n### Action Info Creation\n\nResults are formatted into `ActionCommandInfo` objects:\n\n```python\nactions = self._create_action_info(\n parsed_response.action,\n execution_results,\n)\n\naction_info = ListActionCommandInfo(actions)\naction_info.color_print() # Pretty print to console\n```\n\n### Error Handling\n\n```python\ntry:\n execution_results = await self._execute_app_action(...)\n \n return ProcessingResult(\n success=True,\n data={\n \"execution_result\": execution_results,\n \"action_info\": action_info,\n \"control_log\": control_log,\n \"status\": status\n }\n )\nexcept Exception as e:\n self.logger.error(f\"Action execution failed: {traceback.format_exc()}\")\n return self.handle_error(e, ProcessingPhase.ACTION_EXECUTION, context)\n```\n\n---\n\n## Phase 3: Memory Update Strategy\n\n**Purpose**: Persist execution results and commands into agent memory for future reference.\n\n### Strategy Implementation\n\nLinuxAgent reuses the `AppMemoryUpdateStrategy` from the app agent framework:\n\n```python\nself.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy(\n fail_fast=False # Memory failures shouldn't stop process\n)\n```\n\n### Phase 3 Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Memory\n participant Context\n \n Strategy->>Context: Get execution results\n Strategy->>Context: Get LLM response\n \n Strategy->>Memory: Create memory item\n Memory->>Memory: Store command\n Memory->>Memory: Store stdout/stderr\n Memory->>Memory: Store timestamp\n \n Strategy->>Context: Update round result\n Strategy-->>Agent: Memory updated\n```\n\n### Memory Structure\n\nEach execution round is stored as a memory item:\n\n```python\n{\n \"round\": 1,\n \"request\": \"Check disk space and create backup\",\n \"thought\": \"Need to check disk space first\",\n \"action\": {\n \"command\": \"EXEC_CLI\",\n \"parameters\": {\"command\": \"df -h\"}\n },\n \"result\": {\n \"stdout\": \"Filesystem Size Used...\",\n \"stderr\": \"\",\n \"exit_code\": 0\n },\n \"status\": \"CONTINUE\",\n \"timestamp\": \"2025-11-06T10:30:45\"\n}\n```\n\n### Iterative Refinement\n\nMemory enables iterative refinement:\n\n1. **Round 1**: Check disk space → Result: 50G available\n2. **Round 2**: Create backup (knowing 50G is available)\n3. **Round 3**: Verify backup creation\n\nEach round builds on previous results stored in memory.\n\n### Error Recovery\n\nMemory also stores errors for recovery:\n\n```python\n{\n \"round\": 2,\n \"action\": {\"tool\": \"execute_command\", \"arguments\": {\"command\": \"invalid_cmd\"}},\n \"result\": {\n \"success\": False,\n \"error\": \"Command not found: invalid_cmd\"\n },\n \"status\": \"FAIL\"\n}\n```\n\n## Middleware Stack\n\nLinuxAgent uses specialized middleware for logging:\n\n```python\ndef _setup_middleware(self) -> None:\n \"\"\"Setup middleware pipeline for Linux Agent\"\"\"\n self.middleware_chain = [LinuxLoggingMiddleware()]\n```\n\n### LinuxLoggingMiddleware\n\nProvides enhanced logging specific to Linux operations:\n\n```python\nclass LinuxLoggingMiddleware(AppAgentLoggingMiddleware):\n \"\"\"Specialized logging middleware for Linux Agent\"\"\"\n \n def starting_message(self, context: ProcessingContext) -> str:\n request = context.get_local(\"request\")\n return f\"Completing the user request [{request}] on Linux.\"\n```\n\n**Logged Information**:\n\n- User request\n- Each CLI command executed\n- Command outputs (stdout/stderr)\n- Execution timestamps\n- State transitions\n- LLM costs\n\n---\n\n## Context Finalization\n\nAfter processing, the processor updates global context:\n\n```python\ndef _finalize_processing_context(self, processing_context: ProcessingContext):\n \"\"\"Finalize processing context by updating ContextNames fields\"\"\"\n super()._finalize_processing_context(processing_context)\n \n try:\n result = processing_context.get_local(\"result\")\n if result:\n self.global_context.set(ContextNames.ROUND_RESULT, result)\n except Exception as e:\n self.logger.warning(f\"Failed to update context: {e}\")\n```\n\nThis makes execution results available to:\n\n- Subsequent rounds (iterative execution)\n- Other agents (if part of multi-agent workflow)\n- Session manager (for monitoring and logging)\n\n---\n\n## Strategy Dependency Graph\n\nThe three phases have clear dependencies:\n\n```mermaid\ngraph TD\n A[request] --> B[Phase 1: LLM Interaction]\n B --> C[parsed_response]\n B --> D[llm_cost]\n B --> E[prompt_message]\n \n C --> F[Phase 2: Action Execution]\n F --> G[execution_result]\n F --> H[action_info]\n \n C --> I[Phase 3: Memory Update]\n G --> I\n H --> I\n I --> J[Memory Updated]\n \n J --> K[Next Round or Terminal State]\n```\n\nDependencies are declared using decorators:\n\n```python\n@depends_on(\"request\")\n@provides(\"parsed_response\", \"response_text\", \"llm_cost\", ...)\nclass LinuxLLMInteractionStrategy(AppLLMInteractionStrategy):\n ...\n```\n\n---\n\n## Modular Design Benefits\n\nThe 3-phase strategy design provides:\n\n!!!success \"Modularity Benefits\"\n - **Separation of Concerns**: LLM reasoning, command execution, and memory are isolated\n - **Testability**: Each phase can be tested independently\n - **Extensibility**: New strategies can be added without modifying existing code\n - **Reusability**: Memory strategy is shared with AppAgent\n - **Maintainability**: Clear boundaries between decision-making and execution\n - **Traceability**: Each phase logs its operations independently\n\n---\n\n## Comparison with Other Agents\n\n| Agent | Phases | Data Collection | LLM | Action | Memory |\n|-------|--------|----------------|-----|--------|--------|\n| **LinuxAgent** | 3 | ✗ None | ✓ CLI commands | ✓ MCP execute_command | ✓ Results |\n| **AppAgent** | 4 | ✓ Screenshots + UI | ✓ UI actions | ✓ GUI + API | ✓ Results |\n| **HostAgent** | 4 | ✓ Desktop snapshot | ✓ App selection | ✓ Orchestration | ✓ Results |\n\nLinuxAgent omits the **DATA_COLLECTION** phase because there's no GUI to capture (CLI-based), system info is obtained on-demand via MCP tools, and previous execution results provide necessary context. This reflects the proactive information gathering principle.\n\n## Implementation Location\n\nThe strategy implementations can be found in:\n\n```\nufo/agents/processors/\n├── customized/\n│ └── customized_agent_processor.py # LinuxAgentProcessor\n└── strategies/\n └── linux_agent_strategy.py # Linux-specific strategies\n```\n\nKey classes:\n\n- `LinuxAgentProcessor`: Strategy orchestrator\n- `LinuxLLMInteractionStrategy`: Prompt construction and LLM interaction\n- `LinuxActionExecutionStrategy`: CLI command execution\n- `LinuxLoggingMiddleware`: Enhanced logging\n\n## Next Steps\n\n- [MCP Commands](commands.md) - Explore the CLI execution commands used by LinuxAgent\n- [State Machine](state.md) - Understand the 3-state FSM that controls strategy execution\n- [Overview](overview.md) - Return to LinuxAgent architecture overview\n"
},
{
"path": "documents/docs/mcp/action.md",
"content": "# Action Servers\n\n## Overview\n\n**Action Servers** provide tools that modify system state by executing actions. These servers enable agents to interact with the environment, automate tasks, and implement decisions.\n\n**Action servers are the only servers whose tools can be selected by the LLM agent.** At each step, the agent chooses which action tool to execute based on the task and current context.\n\n- **LLM Decision**: Agent actively selects from available action tools\n- **Dynamic Selection**: Different action chosen at each step based on needs\n- **Tool Visibility**: All action tools are presented to the LLM in the prompt\n\n**[Data Collection Servers](./data_collection.md) are NOT LLM-selectable** - they are automatically invoked by the framework.\n\n### How Tool Metadata Becomes LLM Instructions\n\n**Every action tool's implementation directly affects what the LLM sees and understands.** The UFO² framework automatically extracts:\n\n- **`Annotated` type hints**: Parameter types, constraints, and descriptions\n- **Docstrings**: Tool purpose, parameter explanations, return value descriptions\n- **Function signatures**: Parameter names, defaults, required vs. optional\n\nThese are automatically assembled into structured tool instructions that appear in the LLM's prompt. The LLM uses these instructions to understand what each tool does, select the appropriate tool for each step, and call the tool with correct parameters.\n\n**Therefore, developers MUST write clear, comprehensive metadata.** For examples:\n\n- See [AppUIExecutor documentation](servers/app_ui_executor.md) for well-documented UI automation tools\n- See [WordCOMExecutor documentation](servers/word_com_executor.md) for COM API tool examples\n- See [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) for step-by-step guide on writing tool metadata\n\n```mermaid\ngraph TB\n LLM[\"LLM Agent Decision (Selects Action Tool)\"]\n \n Agent[\"Agent Decision 'Click OK Button'\"]\n \n MCP[\"MCP Server Action Server\"]\n \n subgraph Tools[\"Available Action Tools\"]\n Click[\"click()\"]\n Type[\"type_text()\"]\n Insert[\"insert_table()\"]\n Shell[\"run_shell()\"]\n end\n \n System[\"System Modified ✅ Side Effects\"]\n \n LLM --> Agent\n Agent --> MCP\n MCP --> Tools\n Tools --> System\n \n style LLM fill:#e3f2fd,stroke:#1976d2,stroke-width:2px\n style Agent fill:#fff3e0,stroke:#f57c00,stroke-width:2px\n style MCP fill:#e8f5e9,stroke:#388e3c,stroke-width:2px\n style Tools fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px\n style System fill:#ffebee,stroke:#c62828,stroke-width:2px\n```\n\n**Side Effects:**\n\n- **✅ Modifies State**: Can change system, files, UI\n- **⚠️ Not Idempotent**: Same action may have different results\n- **🔒 Use with Caution**: Always verify before executing\n- **📝 Audit Trail**: Log all actions for debugging\n- **🤖 LLM-Controlled**: Agent decides when and which action to execute\n\n## Tool Type Identifier\n\nAll action tools use the tool type:\n\n```python\ntool_type = \"action\"\n```\n\nTool keys follow the format:\n\n```python\ntool_key = \"action::{tool_name}\"\n\n# Examples:\n\"action::click\"\n\"action::type_text\"\n\"action::run_shell\"\n```\n\n## Built-in Action Servers\n\nUFO² provides several built-in action servers for different automation scenarios. Below is a summary - click each server name for detailed documentation including all tools, parameters, and usage examples.\n\n### UI Automation Servers\n\n| Server | Agent | Description | Documentation |\n|--------|-------|-------------|---------------|\n| **[HostUIExecutor](servers/host_ui_executor.md)** | HostAgent | Window selection and desktop-level UI automation | [Full Details →](servers/host_ui_executor.md) |\n| **[AppUIExecutor](servers/app_ui_executor.md)** | AppAgent | Application-level UI automation (clicks, typing, scrolling) | [Full Details →](servers/app_ui_executor.md) |\n\n### Command Execution Servers\n\n| Server | Platform | Description | Documentation |\n|--------|----------|-------------|---------------|\n| **[CommandLineExecutor](servers/command_line_executor.md)** | Windows | Execute shell commands and launch applications | [Full Details →](servers/command_line_executor.md) |\n| **[BashExecutor](servers/bash_executor.md)** | Linux | Execute Linux commands via HTTP server | [Full Details →](servers/bash_executor.md) |\n\n### Office Automation Servers (COM API)\n\n| Server | Application | Description | Documentation |\n|--------|-------------|-------------|---------------|\n| **[WordCOMExecutor](servers/word_com_executor.md)** | Microsoft Word | Word document automation (insert table, format text, etc.) | [Full Details →](servers/word_com_executor.md) |\n| **[ExcelCOMExecutor](servers/excel_com_executor.md)** | Microsoft Excel | Excel automation (insert data, create charts, etc.) | [Full Details →](servers/excel_com_executor.md) |\n| **[PowerPointCOMExecutor](servers/ppt_com_executor.md)** | Microsoft PowerPoint | PowerPoint automation (slides, formatting, etc.) | [Full Details →](servers/ppt_com_executor.md) |\n\n### Specialized Servers\n\n| Server | Purpose | Description | Documentation |\n|--------|---------|-------------|---------------|\n| **[PDFReaderExecutor](servers/pdf_reader_executor.md)** | PDF Processing | Extract text from PDFs with human simulation | [Full Details →](servers/pdf_reader_executor.md) |\n| **[ConstellationEditor](servers/constellation_editor.md)** | Multi-Device | Create and manage multi-device task workflows | [Full Details →](servers/constellation_editor.md) |\n| **[HardwareExecutor](servers/hardware_executor.md)** | Hardware Control | Control Arduino, robot arms, test fixtures, mobile devices | [Full Details →](servers/hardware_executor.md) |\n\n**Quick Reference:** Each server documentation page includes:\n\n- 📋 **Complete tool reference** with all parameters and return values\n- 💡 **Code examples** showing actual usage patterns\n- ⚙️ **Configuration examples** for different scenarios\n- ✅ **Best practices** with do's and don'ts\n- 🎯 **Use cases** with complete workflows\n\n## Configuration Examples\n\nAction servers are configured in `config/ufo/mcp.yaml`. Each server's documentation provides detailed configuration examples.\n\n### Basic Configuration\n\n```yaml\nHostAgent:\n default:\n action:\n - namespace: HostUIExecutor\n type: local\n reset: false\n - namespace: CommandLineExecutor\n type: local\n reset: false\n```\n\n### App-Specific Configuration\n\n```yaml\nAppAgent:\n # Default configuration for all apps\n default:\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n \n # Word-specific configuration\n WINWORD.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n - namespace: WordCOMExecutor\n type: local\n reset: true # Reset when switching documents\n \n # Excel-specific configuration\n EXCEL.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n - namespace: ExcelCOMExecutor\n type: local\n reset: true # Reset when switching workbooks\n```\n\n### Multi-Platform Configuration\n\n```yaml\n# Windows agent\nHostAgent:\n default:\n action:\n - namespace: HostUIExecutor\n type: local\n - namespace: CommandLineExecutor\n type: local\n\n# Linux agent\nLinuxAgent:\n default:\n action:\n - namespace: BashExecutor\n type: http\n host: \"192.168.1.100\"\n port: 8010\n path: \"/mcp\"\n```\n\nFor complete configuration details, see:\n\n- [MCP Configuration Guide](configuration.md) - Complete configuration reference\n- Individual server documentation for server-specific configuration options\n\n## Best Practices\n\n### General Principles\n\n#### 1. Verify Before Acting\n\nAlways observe before executing actions:\n\n```python\n# ✅ Good: Verify target exists\ncontrol_info = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_control_info\", ...)\n])\n\nif control_info[0].data and control_info[0].data[\"is_enabled\"]:\n await computer.run_actions([\n MCPToolCall(tool_key=\"action::click\", ...)\n ])\n```\n\n#### 2. Handle Action Failures\n\nActions can fail for many reasons - always implement error handling and retries.\n\n#### 3. Validate Inputs\n\nNever execute unsanitized commands, especially with `run_shell` and similar tools.\n\n#### 4. Wait for Action Completion\n\nSome actions need time to complete - add appropriate delays after launching applications or triggering UI changes.\n\nFor detailed best practices including code examples, error handling patterns, and common pitfalls, see the individual server documentation:\n\n- [HostUIExecutor Best Practices](servers/host_ui_executor.md)\n- [AppUIExecutor Best Practices](servers/app_ui_executor.md)\n- [CommandLineExecutor Best Practices](servers/command_line_executor.md)\n- [WordCOMExecutor Best Practices](servers/word_com_executor.md)\n- [ExcelCOMExecutor Best Practices](servers/excel_com_executor.md)\n- [PowerPointCOMExecutor Best Practices](servers/ppt_com_executor.md)\n- [PDFReaderExecutor Best Practices](servers/pdf_reader_executor.md)\n- [ConstellationEditor Best Practices](servers/constellation_editor.md)\n- [HardwareExecutor Best Practices](servers/hardware_executor.md)\n- [BashExecutor Best Practices](servers/bash_executor.md)\n\n## Common Use Cases\n\nFor complete use case examples with detailed workflows, see the individual server documentation:\n\n### UI Automation\n\n- **Form Filling**: [AppUIExecutor](servers/app_ui_executor.md)\n- **Window Management**: [HostUIExecutor](servers/host_ui_executor.md)\n\n### Document Automation\n\n- **Word Processing**: [WordCOMExecutor](servers/word_com_executor.md)\n- **Excel Data Processing**: [ExcelCOMExecutor](servers/excel_com_executor.md)\n- **PowerPoint Generation**: [PowerPointCOMExecutor](servers/ppt_com_executor.md)\n- **PDF Extraction**: [PDFReaderExecutor](servers/pdf_reader_executor.md)\n\n### System Automation\n\n- **Application Launching**: [CommandLineExecutor](servers/command_line_executor.md)\n- **Linux Command Execution**: [BashExecutor](servers/bash_executor.md)\n\n### Multi-Device Workflows\n\n- **Task Distribution**: [ConstellationEditor](servers/constellation_editor.md)\n- **Hardware Control**: [HardwareExecutor](servers/hardware_executor.md)\n\n## Error Handling\n\nAction servers implement robust error handling with timeouts and retries. For detailed error handling patterns specific to each server, see:\n\n- [HostUIExecutor](servers/host_ui_executor.md)\n- [AppUIExecutor](servers/app_ui_executor.md)\n- [CommandLineExecutor](servers/command_line_executor.md)\n- [BashExecutor](servers/bash_executor.md)\n- And other server-specific documentation\n\n### General Timeout Handling\n\nActions are executed with a timeout (default: 6000 seconds):\n\n```python\ntry:\n result = await computer.run_actions([\n MCPToolCall(tool_key=\"action::run_shell\", ...)\n ])\nexcept asyncio.TimeoutError:\n logger.error(\"Action timed out after 6000 seconds\")\n # Cleanup or retry logic...\n```\n\n### General Retry Pattern\n\n```python\nasync def retry_action(action: MCPToolCall, max_retries: int = 3):\n \"\"\"Retry an action with exponential backoff.\"\"\"\n for attempt in range(max_retries):\n try:\n result = await computer.run_actions([action])\n if not result[0].is_error:\n return result[0]\n logger.warning(f\"Attempt {attempt + 1} failed: {result[0].content}\")\n if attempt < max_retries - 1:\n await asyncio.sleep(2 ** attempt) # Exponential backoff\n except Exception as e:\n logger.error(f\"Exception on attempt {attempt + 1}: {e}\")\n if attempt < max_retries - 1:\n await asyncio.sleep(2 ** attempt)\n raise ValueError(f\"Action failed after {max_retries} attempts\")\n```\n\n## Integration with Data Collection\n\nActions should be paired with data collection for verification:\n\n```python\n# Pattern: Observe → Act → Verify\n\n# 1. Observe: Capture initial state\nbefore_screenshot = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::take_screenshot\", ...)\n])\n\n# 2. Act: Execute action\naction_result = await computer.run_actions([\n MCPToolCall(tool_key=\"action::click\", ...)\n])\n\n# 3. Verify: Check result\nawait asyncio.sleep(1) # Wait for UI update\nafter_screenshot = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::take_screenshot\", ...)\n])\n```\n\nFor more details on agent execution patterns:\n\n- [HostAgent Commands](../ufo2/host_agent/commands.md) - HostAgent command patterns\n- [AppAgent Commands](../ufo2/app_agent/commands.md) - AppAgent action patterns\n- [Agent Overview](../ufo2/overview.md) - UFO² agent architecture\n\nFor more details on data collection:\n\n- [Data Collection Servers](data_collection.md) - Observation tools\n- [UICollector Documentation](servers/ui_collector.md) - Complete data collection reference\n\n## Related Documentation\n\n- [Data Collection Servers](data_collection.md) - Observation tools\n- [Configuration Guide](configuration.md) - Configure action servers\n- [Local Servers](local_servers.md) - Built-in action servers overview\n- [Remote Servers](remote_servers.md) - HTTP deployment for actions\n- [Computer](../client/computer.md) - Action execution layer\n- [MCP Overview](overview.md) - High-level MCP architecture\n\n**Safety Reminder:** Action servers can **modify system state**. Always:\n\n1. ✅ **Validate inputs** before execution\n2. ✅ **Verify targets** exist and are accessible\n3. ✅ **Log all actions** for audit trail\n4. ✅ **Handle failures** gracefully with retries\n5. ✅ **Test in safe environment** before production use\n"
},
{
"path": "documents/docs/mcp/configuration.md",
"content": "# MCP Configuration Guide\n\n## Overview\n\nMCP configuration in UFO² uses a **hierarchical YAML structure** that maps agents to their MCP servers. The configuration file is located at:\n\n```\nconfig/ufo/mcp.yaml\n```\n\nFor complete field documentation, see [MCP Reference](../configuration/system/mcp_reference.md).\n\n## Configuration Structure\n\n```yaml\nAgentName: # Name of the agent\n SubType: # Sub-type (e.g., \"default\", \"WINWORD.EXE\")\n data_collection: # Data collection servers\n - namespace: ... # Server namespace\n type: ... # Server type (local/http/stdio)\n ... # Additional server config\n action: # Action servers\n - namespace: ...\n type: ...\n ...\n```\n\n### Hierarchy Levels\n\n1. **Agent Name** - Top-level agent identifier (e.g., `HostAgent`, `AppAgent`)\n2. **Sub-Type** - Context-specific configuration (e.g., `default`, `WINWORD.EXE`)\n3. **Tool Type** - `data_collection` or `action`\n4. **Server List** - Array of MCP server configurations\n\n```\nAgentName\n └─ SubType\n ├─ data_collection\n │ ├─ Server 1\n │ ├─ Server 2\n │ └─ ...\n └─ action\n ├─ Server 1\n ├─ Server 2\n └─ ...\n```\n\n**Default Sub-Type:** Always define a `default` sub-type as a fallback configuration. If a specific sub-type is not found, the agent will use `default`.\n\n## Server Configuration Fields\n\n### Common Fields\n\nAll MCP servers share these fields:\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `namespace` | `string` | ✅ Yes | Unique identifier for the server |\n| `type` | `string` | ✅ Yes | Server type: `local`, `http`, or `stdio` |\n| `reset` | `boolean` | ❌ No | Whether to reset server state (default: `false`) |\n| `start_args` | `array` | ❌ No | Arguments passed to server initialization |\n\n### Local Server Fields\n\nFor `type: local`:\n\n```yaml\n- namespace: UICollector\n type: local\n start_args: []\n reset: false\n```\n\n| Field | Description |\n|-------|-------------|\n| `start_args` | Arguments passed to server factory function |\n\nLocal servers are retrieved from the `MCPRegistry` and run in-process.\n\n### HTTP Server Fields\n\nFor `type: http`:\n\n```yaml\n- namespace: HardwareCollector\n type: http\n host: \"localhost\"\n port: 8006\n path: \"/mcp\"\n reset: false\n```\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `host` | `string` | ✅ Yes | Server hostname or IP |\n| `port` | `integer` | ✅ Yes | Server port number |\n| `path` | `string` | ✅ Yes | URL path to MCP endpoint |\n\nHTTP servers run on remote machines and are accessed via REST API.\n\n### Stdio Server Fields\n\nFor `type: stdio`:\n\n```yaml\n- namespace: CustomProcessor\n type: stdio\n command: \"python\"\n start_args: [\"-m\", \"custom_mcp_server\"]\n env: {\"API_KEY\": \"secret\"}\n cwd: \"/path/to/server\"\n reset: false\n```\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `command` | `string` | ✅ Yes | Executable command |\n| `start_args` | `array` | ❌ No | Command-line arguments |\n| `env` | `object` | ❌ No | Environment variables |\n| `cwd` | `string` | ❌ No | Working directory |\n\nStdio servers run as child processes and communicate via stdin/stdout.\n\n## Agent Configurations\n\n### HostAgent\n\nSystem-level agent for OS-wide automation:\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: HostUIExecutor\n type: local\n start_args: []\n reset: false\n - namespace: CommandLineExecutor\n type: local\n start_args: []\n reset: false\n```\n\n**Tools Available**:\n- **Data Collection**: UI detection, screenshots\n- **Actions**: System-wide clicks, window management, CLI execution\n\n### AppAgent\n\nApplication-specific agent:\n\n#### Default Configuration\n\n```yaml\nAppAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: AppUIExecutor\n type: local\n start_args: []\n reset: false\n - namespace: CommandLineExecutor\n type: local\n start_args: []\n reset: false\n```\n\n#### Word-Specific Configuration\n\n```yaml\nAppAgent:\n WINWORD.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: AppUIExecutor\n type: local\n start_args: []\n reset: false\n - namespace: WordCOMExecutor\n type: local\n start_args: []\n reset: true # Reset COM state when switching documents\n```\n\n**Tools Available**:\n- **Data Collection**: Same as default\n- **Actions**: App UI automation + Word COM API (insert_table, select_text, etc.)\n\n**Reset Flag:** Set `reset: true` for stateful tools (like COM executors) to prevent state leakage between contexts (e.g., different documents).\n\n#### Excel-Specific Configuration\n\n```yaml\nAppAgent:\n EXCEL.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n - namespace: ExcelCOMExecutor\n type: local\n reset: true\n```\n\n#### PowerPoint-Specific Configuration\n\n```yaml\nAppAgent:\n POWERPNT.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n - namespace: PowerPointCOMExecutor\n type: local\n reset: true\n```\n\n#### File Explorer Configuration\n\n```yaml\nAppAgent:\n explorer.exe:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n - namespace: PDFReaderExecutor\n type: local\n reset: true\n```\n\n### ConstellationAgent\n\nMulti-device coordination agent:\n\n```yaml\nConstellationAgent:\n default:\n action:\n - namespace: ConstellationEditor\n type: local\n start_args: []\n reset: false\n```\n\n**Tools Available**:\n- **Actions**: Create tasks, assign devices, check task status\n\n### HardwareAgent\n\nRemote hardware monitoring agent:\n\n```yaml\nHardwareAgent:\n default:\n data_collection:\n - namespace: HardwareCollector\n type: http\n host: \"localhost\"\n port: 8006\n path: \"/mcp\"\n reset: false\n action:\n - namespace: HardwareExecutor\n type: http\n host: \"localhost\"\n port: 8006\n path: \"/mcp\"\n reset: false\n```\n\n**Tools Available**:\n- **Data Collection**: CPU info, memory info, disk info\n- **Actions**: Hardware control commands\n\n**Remote Deployment:** For remote servers, ensure the HTTP MCP server is running on the target machine. See [Remote Servers](remote_servers.md) for deployment guide.\n\n### LinuxAgent\n\nLinux system agent:\n\n```yaml\nLinuxAgent:\n default:\n action:\n - namespace: BashExecutor\n type: http\n host: \"localhost\"\n port: 8010\n path: \"/mcp\"\n reset: false\n```\n\n**Tools Available**:\n- **Actions**: Bash command execution\n\n## Configuration Examples\n\n### Example 1: Local-Only Agent\n\n```yaml\nSimpleAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n action:\n - namespace: SimpleExecutor\n type: local\n reset: false\n```\n\n### Example 2: Hybrid Agent (Local + Remote)\n\n```yaml\nHybridAgent:\n default:\n data_collection:\n # Local UI detection\n - namespace: UICollector\n type: local\n reset: false\n \n # Remote hardware monitoring\n - namespace: HardwareCollector\n type: http\n host: \"192.168.1.100\"\n port: 8006\n path: \"/mcp\"\n reset: false\n \n action:\n # Local UI automation\n - namespace: UIExecutor\n type: local\n reset: false\n \n # Remote command execution\n - namespace: RemoteExecutor\n type: http\n host: \"192.168.1.100\"\n port: 8007\n path: \"/mcp\"\n reset: false\n```\n\n### Example 3: Multi-Context Agent\n\n```yaml\nMultiContextAgent:\n # Default configuration\n default:\n data_collection:\n - namespace: BasicCollector\n type: local\n action:\n - namespace: BasicExecutor\n type: local\n \n # Specialized for Chrome\n chrome.exe:\n data_collection:\n - namespace: BasicCollector\n type: local\n - namespace: WebCollector\n type: local\n action:\n - namespace: BasicExecutor\n type: local\n - namespace: BrowserExecutor\n type: local\n reset: true\n \n # Specialized for VS Code\n Code.exe:\n data_collection:\n - namespace: BasicCollector\n type: local\n - namespace: IDECollector\n type: local\n action:\n - namespace: BasicExecutor\n type: local\n - namespace: CodeExecutor\n type: local\n reset: true\n```\n\n## Best Practices\n\n### 1. Use Descriptive Namespaces\n\n```yaml\n# ✅ Good: Clear and descriptive\nnamespace: WindowsUICollector\nnamespace: ExcelCOMExecutor\nnamespace: LinuxBashExecutor\n\n# ❌ Bad: Generic and unclear\nnamespace: Collector1\nnamespace: Server\nnamespace: Tools\n```\n\n### 2. Group Related Servers\n\n```yaml\n# ✅ Good: Logical grouping\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector # All UI-related\n - namespace: ScreenshotTaker\n action:\n - namespace: UIExecutor # All UI actions\n - namespace: WindowManager\n\n# ❌ Bad: Mixed purposes\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n - namespace: HardwareMonitor # Different purpose\n```\n\n### 3. Reset Stateful Servers\n\n```yaml\n# ✅ Good: Reset COM servers\nWordCOMExecutor:\n type: local\n reset: true # Prevents state leakage\n\n# ❌ Bad: Not resetting can cause issues\nWordCOMExecutor:\n type: local\n reset: false # May retain state from previous document\n```\n\n### 4. Validate Remote Server Availability\n\n```yaml\n# When using remote servers, ensure they're accessible\nHardwareCollector:\n type: http\n host: \"192.168.1.100\" # ✅ Verify this host is reachable\n port: 8006 # ✅ Verify port is open\n path: \"/mcp\" # ✅ Verify endpoint exists\n```\n\n### 5. Use Environment Variables for Secrets\n\n```yaml\n# ✅ Good: Use environment variables\n- namespace: SecureAPI\n type: http\n host: \"${API_HOST}\"\n port: \"${API_PORT}\"\n auth:\n token: \"${API_TOKEN}\"\n\n# ❌ Bad: Hardcoded secrets\n- namespace: SecureAPI\n type: http\n host: \"api.example.com\"\n auth:\n token: \"secret123\" # Don't commit this!\n```\n\n## Loading Configuration\n\n### From File\n\n```python\nimport yaml\nfrom pathlib import Path\n\n# Load MCP configuration\nconfig_path = Path(\"config/ufo/mcp.yaml\")\nwith open(config_path) as f:\n mcp_config = yaml.safe_load(f)\n\n# Access agent configuration\nhost_agent_config = mcp_config[\"HostAgent\"][\"default\"]\n```\n\n### Programmatically\n\n```python\nfrom ufo.config import get_config\n\n# Get full configuration\nconfigs = get_config()\n\n# Access MCP section\nmcp_config = configs.get(\"mcp\", {})\n\n# Get specific agent\nhost_agent = mcp_config.get(\"HostAgent\", {}).get(\"default\", {})\n```\n\n## Validation\n\n### Schema Validation\n\nUFO² validates MCP configuration on load:\n\n```python\nfrom ufo.config.config_schemas import MCPConfigSchema\n\n# Validate configuration\ntry:\n MCPConfigSchema.validate(mcp_config)\n print(\"✅ Configuration is valid\")\nexcept ValidationError as e:\n print(f\"❌ Configuration error: {e}\")\n```\n\n### Common Validation Errors\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| `Missing required field: namespace` | Server missing namespace | Add `namespace` field |\n| `Invalid server type: unknown` | Unsupported type | Use `local`, `http`, or `stdio` |\n| `Missing host for http server` | HTTP server without host | Add `host` and `port` |\n| `Duplicate namespace` | Same namespace used twice | Use unique namespaces |\n\n## Debugging Configuration\n\n### Enable Debug Logging\n\n```python\nimport logging\n\nlogging.basicConfig(level=logging.DEBUG)\nlogger = logging.getLogger(\"ufo.client.mcp\")\n\n# Will show server creation and registration\n# DEBUG: Creating MCP server 'UICollector' of type local\n# DEBUG: Registered MCP server 'UICollector' with 15 tools\n```\n\n### Check Loaded Servers\n\n```python\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\n# List all registered servers\nservers = MCPServerManager._servers_mapping\nfor namespace, server in servers.items():\n print(f\"Server: {namespace}, Type: {type(server).__name__}\")\n```\n\n### Test Server Connectivity\n\n```python\nasync def test_server(config):\n \"\"\"Test if MCP server is accessible.\"\"\"\n try:\n server = MCPServerManager.create_mcp_server(config)\n print(f\"✅ Server '{config['namespace']}' is accessible\")\n \n # List tools\n if hasattr(server, 'server'):\n from fastmcp.client import Client\n async with Client(server.server) as client:\n tools = await client.list_tools()\n print(f\" Tools: {[tool.name for tool in tools]}\")\n except Exception as e:\n print(f\"❌ Server '{config['namespace']}' failed: {e}\")\n```\n\n## Migration Guide\n\n### From Old Configuration Format\n\nIf you're migrating from an older UFO configuration:\n\n**Old Format** (config.yaml):\n```yaml\nMCP_SERVERS:\n - name: ui_collector\n module: ufo.mcp.ui_server\n```\n\n**New Format** (mcp.yaml):\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n```\n\nFor detailed migration instructions, see [Configuration Migration Guide](../configuration/system/migration.md).\n\n## Related Documentation\n\n- [MCP Overview](overview.md) - High-level MCP architecture\n- [Data Collection Servers](data_collection.md) - Data collection configuration\n- [Action Servers](action.md) - Action server configuration\n- [Local Servers](local_servers.md) - Built-in local MCP servers\n- [Remote Servers](remote_servers.md) - HTTP and Stdio deployment\n- [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) - Build your own servers\n- [MCP Reference](../configuration/system/mcp_reference.md) - Complete field reference\n- [Configuration Guide](../configuration/system/overview.md) - General configuration guide\n- [HostAgent Overview](../ufo2/host_agent/overview.md) - HostAgent configuration examples\n- [AppAgent Overview](../ufo2/app_agent/overview.md) - AppAgent configuration examples\n\n**Configuration Philosophy:**\n\nMCP configuration follows the **convention over configuration** principle:\n\n- **Sensible defaults** - Minimal configuration required\n- **Explicit when needed** - Full control when customization is necessary\n- **Type-safe** - Validated on load to catch errors early\n- **Hierarchical** - Inherit from defaults, override as needed\n"
},
{
"path": "documents/docs/mcp/data_collection.md",
"content": "# Data Collection Servers\n\n## Overview\n\n**Data Collection Servers** provide read-only tools that observe and retrieve system state without modifying it. These servers are essential for agents to understand the current environment before taking actions.\n\n**Data Collection servers are automatically invoked by the UFO² framework** to gather context and build observation prompts for the LLM. The LLM agent **does not select these tools** - they run in the background to provide system state information.\n\n- **Framework-Driven**: Automatically called to collect screenshots, UI controls, system info\n- **Observation Purpose**: Build the prompt that the LLM uses for decision-making\n- **Not in Tool List**: These tools are NOT presented to the LLM as selectable actions\n\n**Only [Action Servers](./action.md) are LLM-selectable.**\n\n```mermaid\ngraph TB\n Framework[\"UFO² Framework (Automatic Invocation)\"]\n \n AgentStep[\"Agent Step Observation & Prompt Build\"]\n \n MCP[\"MCP Server UICollector\"]\n \n subgraph Tools[\"Data Collection Tools\"]\n Screenshot[\"take_screenshot()\"]\n WindowList[\"get_window_list()\"]\n ControlInfo[\"get_control_info()\"]\n end\n \n SystemState[\"System State → LLM Context\"]\n \n Framework --> AgentStep\n Framework --> MCP\n MCP --> Tools\n Tools --> SystemState\n SystemState --> AgentStep\n \n style Framework fill:#e3f2fd,stroke:#1976d2,stroke-width:2px\n style AgentStep fill:#fff3e0,stroke:#f57c00,stroke-width:2px\n style MCP fill:#e8f5e9,stroke:#388e3c,stroke-width:2px\n style Tools fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px\n style SystemState fill:#fce4ec,stroke:#c2185b,stroke-width:2px\n```\n\n**Characteristics:**\n\n- **❌ No Side Effects**: Cannot modify system state\n- **✅ Safe to Retry**: Can be called multiple times without risk\n- **✅ Idempotent**: Same input always produces same output\n- **📊 Observation Only**: Provides information for decision-making\n- **🤖 Framework-Invoked**: Not selectable by LLM agent\n\n## Tool Type Identifier\n\nAll data collection tools use the tool type:\n\n```python\ntool_type = \"data_collection\"\n```\n\nTool keys follow the format:\n\n```python\ntool_key = \"data_collection::{tool_name}\"\n\n# Examples:\n\"data_collection::take_screenshot\"\n\"data_collection::get_window_list\"\n\"data_collection::get_control_info\"\n```\n\n## Built-in Data Collection Servers\n\n### UICollector\n\n**Purpose**: Collect UI element information and screenshots\n\n**Namespace**: `UICollector`\n\n**Platform**: Windows (using pywinauto)\n\n**Tools**: 8 tools for UI observation including screenshots, window lists, control info, and annotations\n\nFor complete documentation including all tool details, parameters, return types, and usage examples, see:\n\n**[→ UICollector Full Documentation](servers/ui_collector.md)**\n\n#### Quick Example\n\n```python\nfrom aip.messages import Command\n\n# Take a screenshot of the active window\nscreenshot_cmd = Command(\n tool_name=\"take_screenshot\",\n tool_type=\"data_collection\",\n parameters={\n \"region\": \"active_window\",\n \"save_path\": \"screenshots/current.png\"\n }\n)\n\n# Get list of all windows\nwindows_cmd = Command(\n tool_name=\"get_window_list\",\n tool_type=\"data_collection\",\n parameters={}\n)\n```\n\nFor detailed tool specifications, advanced usage patterns, and best practices, see the [UICollector documentation](servers/ui_collector.md).\n\n## Configuration Examples\n\nData collection servers are configured in `config/ufo/mcp.yaml`. For detailed configuration options, see the [UICollector documentation](servers/ui_collector.md#configuration).\n\n### Basic Configuration\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n```\n\n### Multi-Server Configuration\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n```\n\n### App-Specific Configuration\n\n```yaml\nAppAgent:\n WINWORD.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false # Don't reset when switching between documents\n \n EXCEL.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n reset: true # Reset when switching between spreadsheets\n```\n\n## Best Practices\n\nFor detailed best practices with complete code examples, see the [UICollector documentation](servers/ui_collector.md).\n\n### General Guidelines\n\n#### 1. Call Before Action\n\nAlways collect data before executing actions to make informed decisions.\n\n#### 2. Cache Results\n\nData collection results can be cached when state hasn't changed to improve performance.\n\n#### 3. Handle Failures Gracefully\n\nData collection can fail if windows close or controls disappear - implement proper error handling.\n\n#### 4. Minimize Screenshot Calls\n\nScreenshots are expensive operations - take one screenshot and analyze it multiple times rather than taking multiple screenshots.\n\n5. **Use Appropriate Regions**\n\nChoose the smallest region that contains needed information (e.g., active window vs. full screen).\n\nSee the [UICollector documentation](servers/ui_collector.md) for detailed examples and anti-patterns.\n\n## Common Use Cases\n\nFor complete use case examples with detailed code, see the [UICollector documentation](servers/ui_collector.md).\n\n### UI Element Detection\n\nDiscover windows and controls for automation targeting.\n\n### Screen Monitoring\n\nMonitor screen changes for event-driven automation.\n\n### System Health Check\n\nCheck system resources before executing heavy tasks.\n\nSee the [UICollector documentation](servers/ui_collector.md) for complete workflow examples.\n\n## Error Handling\n\nFor detailed error handling patterns, see the [UICollector documentation](servers/ui_collector.md).\n\n### Common Errors\n\n| Error | Cause | Solution |\n|-------|-------|----------|\n| `WindowNotFoundError` | Target window closed | Check window existence first |\n| `ControlNotFoundError` | Control not accessible | Use alternative identification method |\n| `ScreenshotFailedError` | Graphics driver issue | Retry with different region |\n| `TimeoutError` | Operation took too long | Increase timeout or simplify query |\n\nSee the [UICollector documentation](servers/ui_collector.md) for complete error recovery examples.\n\n## Performance Considerations\n\nFor detailed performance optimization techniques, see the [UICollector documentation](servers/ui_collector.md).\n\n### Key Optimizations\n\n- **Screenshot Optimization**: Use region parameters to capture only needed areas\n- **Parallel Data Collection**: Collect independent data in parallel when possible\n- **Caching**: Cache results when state hasn't changed\n\nSee the [UICollector documentation](servers/ui_collector.md) for complete examples.\n\n## Integration with Agents\n\nData collection servers are typically used in the **observation phase** of agent execution. See the [UICollector documentation](servers/ui_collector.md) for complete integration patterns.\n\nFor more details on agent architecture and execution flow:\n\n- [HostAgent Overview](../ufo2/host_agent/overview.md) - HostAgent architecture and workflow\n- [AppAgent Overview](../ufo2/app_agent/overview.md) - AppAgent architecture and workflow\n- [Agent Overview](../ufo2/overview.md) - UFO² agent system architecture\n\n```python\n# Agent execution loop\nwhile not task_complete:\n # 1. Observe: Collect current state\n screenshot = await data_collection_server.take_screenshot()\n \n # 2. Reason: Agent decides next action\n next_action = agent.plan(screenshot)\n \n # 3. Act: Execute action\n result = await action_server.execute(next_action)\n \n # 4. Verify: Check action result\n new_screenshot = await data_collection_server.take_screenshot()\n```\n\n## Related Documentation\n\n- **[UICollector Full Documentation](servers/ui_collector.md)** - Complete tool reference with all parameters and examples\n- [Action Servers](action.md) - State-changing execution tools\n- [Configuration Guide](configuration.md) - How to configure data collection servers\n- [Local Servers](local_servers.md) - Built-in local MCP servers\n- [Remote Servers](remote_servers.md) - HTTP deployment for data collection\n- [Computer](../client/computer.md) - Tool execution layer\n- [MCP Overview](overview.md) - High-level MCP architecture\n\n**Key Takeaways:**\n\n- Data collection servers are **read-only** and **safe to retry**\n- Always **observe before acting** to make informed decisions\n- **Cache results** when state hasn't changed to improve performance\n- Handle **errors gracefully** with retries and fallback logic\n- Use **appropriate regions** and **parallel collection** for performance\n- See the **[UICollector documentation](servers/ui_collector.md)** for complete details\n"
},
{
"path": "documents/docs/mcp/local_servers.md",
"content": "# Local MCP Servers\n\nLocal MCP servers run in-process with the UFO² agent, providing fast and efficient access to tools without network overhead. They are the most common server type for built-in functionality.\n\n**For remote HTTP servers** (BashExecutor, HardwareExecutor, MobileExecutor), see [Remote Servers](./remote_servers.md).\n\n## Overview\n\nUFO² includes several built-in local MCP servers organized by functionality. This page provides a quick reference - click each server name for complete documentation.\n\n| Server | Type | Description | Full Documentation |\n|--------|------|-------------|-------------------|\n| **UICollector** | Data Collection | Windows UI observation | **[→ Full Docs](servers/ui_collector.md)** |\n| **HostUIExecutor** | Action | Desktop-level UI automation | **[→ Full Docs](servers/host_ui_executor.md)** |\n| **AppUIExecutor** | Action | Application-level UI automation | **[→ Full Docs](servers/app_ui_executor.md)** |\n| **CommandLineExecutor** | Action | Shell command execution | **[→ Full Docs](servers/command_line_executor.md)** |\n| **WordCOMExecutor** | Action | Microsoft Word COM API | **[→ Full Docs](servers/word_com_executor.md)** |\n| **ExcelCOMExecutor** | Action | Microsoft Excel COM API | **[→ Full Docs](servers/excel_com_executor.md)** |\n| **PowerPointCOMExecutor** | Action | Microsoft PowerPoint COM API | **[→ Full Docs](servers/ppt_com_executor.md)** |\n| **PDFReaderExecutor** | Action | PDF text extraction | **[→ Full Docs](servers/pdf_reader_executor.md)** |\n| **ConstellationEditor** | Action | Multi-device task orchestration | **[→ Full Docs](servers/constellation_editor.md)** |\n\n---\n\n## Server Summaries\n\n### UICollector\n\n**Type**: Data Collection (read-only, automatically invoked) \n**Platform**: Windows \n**Tools**: 8 tools for screenshots, window lists, control info, and annotations\n\n**[→ See complete UICollector documentation](servers/ui_collector.md)** for all tool details, parameters, return values, and examples.\n\n---\n\n### HostUIExecutor\n\n**Type**: Action (LLM-selectable, state-modifying) \n**Platform**: Windows \n**Agent**: HostAgent \n**Tool**: `select_application_window` - Window selection and focus management\n\n**[→ See complete HostUIExecutor documentation](servers/host_ui_executor.md)** for tool specifications and workflow examples.\n\n---\n\n### AppUIExecutor\n\n**Type**: Action (LLM-selectable, GUI automation) \n**Platform**: Windows \n**Agent**: AppAgent \n**Tools**: 9 tools for clicks, typing, scrolling, and UI interaction\n\n**[→ See complete AppUIExecutor documentation](servers/app_ui_executor.md)** for all automation tools and usage patterns.\n\n---\n\n### CommandLineExecutor\n\n**Type**: Action (LLM-selectable, shell execution) \n**Platform**: Cross-platform \n**Agent**: HostAgent, AppAgent \n**Tool**: `run_shell` - Execute shell commands\n\n**[→ See complete CommandLineExecutor documentation](servers/command_line_executor.md)** for security guidelines and examples.\n\n---\n\n### WordCOMExecutor\n\n**Type**: Action (LLM-selectable, Word COM API) \n**Platform**: Windows \n**Agent**: AppAgent (WINWORD.EXE only) \n**Tools**: 6 tools for Word document automation\n\n**[→ See complete WordCOMExecutor documentation](servers/word_com_executor.md)** for all Word automation tools.\n\n---\n\n### ExcelCOMExecutor\n\n**Type**: Action (LLM-selectable, Excel COM API) \n**Platform**: Windows \n**Agent**: AppAgent (EXCEL.EXE only) \n**Tools**: 6 tools for Excel automation\n\n**[→ See complete ExcelCOMExecutor documentation](servers/excel_com_executor.md)** for all Excel manipulation tools.\n\n---\n\n### PowerPointCOMExecutor\n\n**Type**: Action (LLM-selectable, PowerPoint COM API) \n**Platform**: Windows \n**Agent**: AppAgent (POWERPNT.EXE only) \n**Tools**: 2 tools for PowerPoint automation\n\n**[→ See complete PowerPointCOMExecutor documentation](servers/ppt_com_executor.md)** for PowerPoint tools and examples.\n\n---\n\n### PDFReaderExecutor\n\n**Type**: Action (LLM-selectable, PDF text extraction) \n**Platform**: Windows \n**Agent**: AppAgent (explorer.exe) \n**Tools**: 3 tools for PDF text extraction with human simulation\n\n**[→ See complete PDFReaderExecutor documentation](servers/pdf_reader_executor.md)** for PDF extraction tools and workflows.\n\n---\n\n### ConstellationEditor\n\n**Type**: Action (LLM-selectable, multi-device orchestration) \n**Platform**: Cross-platform \n**Agent**: ConstellationAgent \n**Tools**: 7 tools for task and dependency management\n\n**[→ See complete ConstellationEditor documentation](servers/constellation_editor.md)** for multi-device workflow tools.\n\n---\n\n## Configuration\n\nAll local servers are configured in `config/ufo/mcp.yaml`. For detailed configuration options, see:\n\n- [MCP Configuration Guide](./configuration.md) - Complete configuration reference\n- Individual server documentation for server-specific configuration\n\n**Example configuration:**\n\n```yaml\nAppAgent:\n WINWORD.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n action:\n - namespace: AppUIExecutor # GUI automation\n type: local\n reset: false\n - namespace: WordCOMExecutor # API automation\n type: local\n reset: true # Reset when switching documents\n - namespace: CommandLineExecutor\n type: local\n reset: false\n```\n\n## See Also\n\n- [MCP Overview](./overview.md) - MCP architecture and concepts\n- [Data Collection Servers](./data_collection.md) - Data collection overview\n- [Action Servers](./action.md) - Action server overview\n- [MCP Configuration](./configuration.md) - Configuration guide\n- [Remote Servers](./remote_servers.md) - HTTP/Stdio deployment\n- [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) - Learn to build your own servers\n- [HostAgent Overview](../ufo2/host_agent/overview.md) - HostAgent architecture\n- [AppAgent Overview](../ufo2/app_agent/overview.md) - AppAgent architecture\n- [Hybrid Actions](../ufo2/core_features/hybrid_actions.md) - GUI + API dual-mode automation\n"
},
{
"path": "documents/docs/mcp/overview.md",
"content": "# MCP (Model Context Protocol) - Overview\n\n## What is MCP?\n\n**MCP (Model Context Protocol)** is a standardized protocol that enables UFO² agents to interact with external tools and services in a unified way. It provides a **tool execution framework** where agents can:\n\n- **Collect system state** through data collection servers\n- **Execute actions** through action servers\n- **Extend capabilities** through custom MCP servers\n\n```mermaid\ngraph TB\n subgraph Agent[\"UFO² Agent\"]\n HostAgent[HostAgent]\n AppAgent[AppAgent]\n end\n \n Computer[\"Computer (MCP Tool Manager)\"]\n \n subgraph DataServers[\"Data Collection Servers\"]\n UICollector[\"UICollector • Screenshots • Window Info\"]\n HWInfo[\"Hardware Info • CPU/Memory • System State\"]\n end\n \n subgraph ActionServers[\"Action Servers\"]\n UIExecutor[\"UIExecutor • Click/Type • UI Automation\"]\n CLIExecutor[\"CLI Executor • Shell Commands\"]\n COMExecutor[\"COM Executor • API Calls\"]\n end\n \n HostAgent --> Computer\n AppAgent --> Computer\n Computer --> DataServers\n Computer --> ActionServers\n \n style Agent fill:#e3f2fd,stroke:#1976d2,stroke-width:2px\n style Computer fill:#fff3e0,stroke:#f57c00,stroke-width:2px\n style DataServers fill:#e8f5e9,stroke:#388e3c,stroke-width:2px\n style ActionServers fill:#fce4ec,stroke:#c2185b,stroke-width:2px\n```\n\nMCP serves as the **execution layer** in UFO²'s architecture. While agents make decisions about *what* to do, MCP servers handle *how* to do it by providing concrete tool implementations.\n\n## Key Concepts\n\n### 1. Two Server Types\n\nMCP servers in UFO² are categorized into two types based on their purpose:\n\n| Type | Purpose | Examples | Side Effects | LLM Selectable? |\n|------|---------|----------|--------------|-----------------|\n| **Data Collection** | Retrieve system state Read-only operations | UI detection, Screenshot, System info | ❌ None | ❌ **No** - Auto-invoked |\n| **Action** | Modify system state State-changing operations | Click, Type text, Run command | ✅ Yes | ✅ **Yes** - LLM chooses |\n\n**Server Selection Model:**\n\n- **[Data Collection Servers](./data_collection.md)**: Automatically invoked by the framework to gather context and build observation prompts. Not selectable by LLM.\n- **[Action Servers](./action.md)**: LLM agent actively selects which action tool to execute at each step based on the task. Only action tools are LLM-selectable.\n\n**How Action Tools Reach the LLM**: Each action tool's `Annotated` type hints and docstring are automatically extracted and converted into structured instructions that appear in the LLM's prompt. The LLM uses these instructions to understand what each tool does, what parameters it requires, and when to use it. Therefore, developers should write clear, comprehensive docstrings and type annotations - they directly impact the LLM's ability to use the tool correctly.\n\n### 2. Server Deployment Models\n\nUFO² supports three deployment models for MCP servers:\n\n| Model | Description | Benefits | Trade-offs |\n|-------|-------------|----------|------------|\n| **Local (In-Process)** | Server runs in same process as agent | Fast (no IPC overhead), Simple setup | Shares process resources |\n| **HTTP (Remote)** | Server runs as HTTP service (e.g., Port 8006) | Process isolation, Language-agnostic | Network overhead |\n| **Stdio (Process)** | Server runs as child process using stdin/stdout | Process isolation, Bidirectional streaming | Platform-specific |\n\n### 3. Namespace Isolation\n\nEach MCP server has a **namespace** that groups related tools together:\n\n```yaml\n# Example: HostAgent configuration\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector # Namespace for UI detection tools\n type: local\n action:\n - namespace: HostUIExecutor # Namespace for UI automation tools\n type: local\n - namespace: CommandLineExecutor # Namespace for CLI tools\n type: local\n```\n\n**Tool Key Format**: `{tool_type}::{tool_name}`\n\n- Example: `data_collection::screenshot` - Screenshot tool in data_collection\n- Example: `action::click` - Click tool in action\n- Example: `action::run_shell` - Shell command in action\n\n## Key Features\n\n### 1. GUI + API Dual-Mode Automation\n\n**UFO² supports both GUI automation and API-based automation simultaneously.** Each agent can register multiple action servers, combining:\n\n- **GUI Automation**: Windows UI Automation (UIA) - clicking, typing, scrolling when visual interaction is needed\n- **API Automation**: Direct COM API calls, shell commands, REST APIs for efficient, reliable operations\n\n**The LLM agent dynamically chooses the best action at each step** based on task requirements, reliability, speed, and availability.\n\n**Example: Word Document Automation**\n\n```yaml\nAppAgent:\n WINWORD.EXE:\n action:\n - namespace: WordCOMExecutor # API: Fast, reliable\n - namespace: AppUIExecutor # GUI: Visual navigation fallback \n - namespace: CommandLineExecutor # Shell: File operations\n```\n\n**LLM's Dynamic Selection:**\n\n```\nTask: \"Create a report with a 3x2 table and bold the title\"\n\nStep 1: Insert table\n → LLM selects: WordCOMExecutor::insert_table(rows=3, cols=2)\n → Reason: API is fast, reliable, no GUI navigation needed\n\nStep 2: Navigate to Design tab \n → LLM selects: AppUIExecutor::click_input(id=\"5\", name=\"Design\")\n → Reason: Visual navigation, COM API doesn't expose tab selection\n\nStep 3: Type table header\n → LLM selects: AppUIExecutor::set_edit_text(id=\"cell_1_1\", text=\"Product\")\n → Reason: GUI interaction needed for cell input\n\nStep 4: Bold title text\n → LLM selects: WordCOMExecutor::select_text(text=\"Report Title\")\n → WordCOMExecutor::set_font(font_size=16)\n → Reason: API is more reliable than GUI button clicking\n\nStep 5: Save as PDF\n → LLM selects: WordCOMExecutor::save_as(file_ext=\".pdf\")\n → Reason: One API call vs. multiple GUI clicks (File → Save As → Format → PDF)\n```\n\n**Why Hybrid Automation Matters:**\n\n- **APIs**: ~10x faster, deterministic, no visual dependency\n- **GUI**: Handles visual elements, fallback when API unavailable\n- **LLM Decision**: Chooses optimal approach per step, not locked into one mode\n\n### 2. Multi-Server Per Agent\n\nEach agent can register **multiple action servers**, each providing a different set of tools:\n\n**HostAgent Example:**\n```yaml\nHostAgent:\n default:\n data_collection:\n - UICollector # Automatically invoked\n action:\n - HostUIExecutor # LLM selects: Window selection\n - CommandLineExecutor # LLM selects: Launch apps, shell commands\n```\n\n**AppAgent Example (Word-specific):**\n```yaml\nAppAgent:\n WINWORD.EXE:\n data_collection:\n - UICollector # Automatically invoked\n action:\n - WordCOMExecutor # LLM selects: insert_table, select_text, save_as\n - AppUIExecutor # LLM selects: click_input, set_edit_text\n - CommandLineExecutor # LLM selects: run_shell\n```\n\n**HardwareAgent Example (Cross-platform):**\n```yaml\nHardwareAgent:\n default:\n data_collection:\n - HardwareCollector # Auto-invoked (HTTP remote)\n action:\n - HardwareExecutor # LLM selects: touch_screen, swipe, press_key (HTTP remote)\n```\n\n**At each step, the LLM sees all available action tools and selects the most appropriate one.**\n\n### 3. Process Isolation\n\nMCP servers can run:\n\n- **In-process** (local): Fast, low overhead\n- **HTTP** (remote): Process isolation, cross-platform, distributed\n- **Stdio** (child process): Sandboxed execution, clean resource management\n\n### 4. Namespace Isolation\n\nEach MCP server has a unique namespace that groups related tools together, preventing naming conflicts and enabling modular organization. See [Namespace Isolation](#3-namespace-isolation) section above for details.\n\n## Architecture\n\n### MCP Server Lifecycle\n\n```mermaid\ngraph TB\n Start([MCP Server Lifecycle])\n \n Config[\"1. Configuration Loading (mcp.yaml)\"]\n Manager[\"2. MCPServerManager Creates BaseMCPServer\"]\n ServerStart[\"3. Server.start() • Local: Get from registry • HTTP: Build URL • Stdio: Spawn process\"]\n Register[\"4. Computer Registration • List tools from server • Register in tool registry\"]\n Execute[\"5. Tool Execution • Agent sends Command • Computer routes to tool • MCP server executes\"]\n Reset[\"6. Server.reset() (optional) Reset server state\"]\n \n Start --> Config\n Config --> Manager\n Manager --> ServerStart\n ServerStart --> Register\n Register --> Execute\n Execute --> Reset\n \n style Start fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px\n style Config fill:#e1f5fe,stroke:#0277bd,stroke-width:2px\n style Manager fill:#fff9c4,stroke:#f57f17,stroke-width:2px\n style ServerStart fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px\n style Register fill:#fce4ec,stroke:#c2185b,stroke-width:2px\n style Execute fill:#e0f2f1,stroke:#00695c,stroke-width:2px\n style Reset fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px\n```\n\n### Component Relationships\n\n```mermaid\ngraph TB\n subgraph Architecture[\"MCP Architecture\"]\n Registry[\"MCPRegistry (Singleton) • Stores server factories • Lazy initialization\"]\n \n Manager[\"MCPServerManager (Singleton) • Creates server instances • Maps server types to classes • Manages server lifecycle\"]\n \n subgraph ServerTypes[\"MCP Server Types\"]\n Local[\"Local MCP Server (In-Process)\"]\n HTTP[\"HTTP MCP Server (Remote)\"]\n Stdio[\"Stdio MCP Server (Child Process)\"]\n end\n \n Computer[\"Computer (Per Agent) • Manages multiple MCP servers • Routes commands to tools • Maintains tool registry\"]\n \n Registry --> Manager\n Manager --> ServerTypes\n Manager --> Computer\n end\n \n style Architecture fill:#fafafa,stroke:#424242,stroke-width:2px\n style Registry fill:#e1f5fe,stroke:#01579b,stroke-width:2px\n style Manager fill:#fff9c4,stroke:#f57f17,stroke-width:2px\n style ServerTypes fill:#e8f5e9,stroke:#1b5e20,stroke-width:2px\n style Local fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px\n style HTTP fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px\n style Stdio fill:#c8e6c9,stroke:#2e7d32,stroke-width:1px\n style Computer fill:#fce4ec,stroke:#880e4f,stroke-width:2px\n```\n\n## Built-in MCP Servers\n\nUFO² comes with several **built-in MCP servers** that cover common automation scenarios:\n\n### Data Collection Servers\n\n| Namespace | Purpose | Key Tools | Platform |\n|-----------|---------|-----------|----------|\n| **UICollector** | UI element detection | `get_control_info`, `take_screenshot`, `get_window_list` | Windows |\n| **HardwareCollector** | Hardware information | `get_cpu_info`, `get_memory_info` | Cross-platform |\n| **MobileDataCollector** | Android device observation | `capture_screenshot`, `get_ui_tree`, `get_device_info`, `get_mobile_app_target_info` | Android (ADB) |\n\n### Action Servers\n\n| Namespace | Purpose | Key Tools | Platform |\n|-----------|---------|-----------|----------|\n| **HostUIExecutor** | UI automation (host-level) | `click`, `type_text`, `scroll` | Windows |\n| **AppUIExecutor** | UI automation (app-level) | `click`, `type_text`, `set_edit_text` | Windows |\n| **CommandLineExecutor** | CLI execution | `run_shell` | Cross-platform |\n| **WordCOMExecutor** | Word automation | `insert_table`, `select_text`, `format_text` | Windows |\n| **ExcelCOMExecutor** | Excel automation | `insert_cell`, `select_range`, `format_cell` | Windows |\n| **PowerPointCOMExecutor** | PowerPoint automation | `insert_slide`, `add_text`, `format_shape` | Windows |\n| **ConstellationEditor** | Multi-device coordination | `create_task`, `assign_device` | Cross-platform |\n| **BashExecutor** | Linux commands | `execute_bash` | Linux |\n| **MobileExecutor** | Android device control | `tap`, `swipe`, `type_text`, `launch_app`, `click_control` | Android (ADB) |\n\n!!!example \"Tool Examples\"\n ```python\n # Data Collection: Take a screenshot\n {\n \"tool_type\": \"data_collection\",\n \"tool_name\": \"take_screenshot\",\n \"parameters\": {\"region\": \"active_window\"}\n }\n \n # Action: Click a button\n {\n \"tool_type\": \"action\",\n \"tool_name\": \"click\",\n \"parameters\": {\"control_label\": \"Submit\"}\n }\n \n # Action: Run a shell command\n {\n \"tool_type\": \"action\",\n \"tool_name\": \"run_shell\",\n \"parameters\": {\"bash_command\": \"notepad.exe\"}\n }\n ```\n\n## Agent-Specific Configurations\n\nDifferent agents can have **different MCP configurations** based on their roles:\n\n```yaml\n# HostAgent: System-level operations\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: HostUIExecutor\n type: local\n - namespace: CommandLineExecutor\n type: local\n\n# AppAgent: Application-specific operations\nAppAgent:\n WINWORD.EXE: # Word-specific configuration\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: WordCOMExecutor # Word COM API\n type: local\n reset: true # Reset when switching documents\n\n# HardwareAgent: Remote hardware monitoring\nHardwareAgent:\n default:\n data_collection:\n - namespace: HardwareCollector\n type: http # Remote server\n host: \"localhost\"\n port: 8006\n path: \"/mcp\"\n\n# MobileAgent: Android device automation\nMobileAgent:\n default:\n data_collection:\n - namespace: MobileDataCollector\n type: http # Remote server\n host: \"localhost\"\n port: 8020\n path: \"/mcp\"\n action:\n - namespace: MobileExecutor\n type: http\n host: \"localhost\"\n port: 8021\n path: \"/mcp\"\n```\n\n**Configuration Hierarchy:**\n\nAgent configurations follow this hierarchy:\n\n1. **Agent Name** (e.g., `HostAgent`, `AppAgent`)\n2. **Sub-type** (e.g., `default`, `WINWORD.EXE`)\n3. **Tool Type** (e.g., `data_collection`, `action`)\n4. **Server List** (array of server configurations)\n\n## Key Features\n\n### 1. Process Isolation with Reset\n\nSome MCP servers support **state reset** to ensure clean execution:\n\n```yaml\nAppAgent:\n WINWORD.EXE:\n action:\n - namespace: WordCOMExecutor\n type: local\n reset: true\n\n**When to use reset:**\n\n- Server state is cleared when switching contexts\n- Prevents state leakage between tasks\n- Useful for stateful tools (e.g., COM APIs)\n\n### 2. Thread Isolation\n\nMCP tools execute in **isolated thread pools** to prevent blocking:\n\n```python\n# From Computer class\nself._executor = ThreadPoolExecutor(max_workers=10)\nself._tool_timeout = 6000 # 100 minutes\n```\n\n**Benefits**:\n- Prevents blocking the main event loop\n- Protects WebSocket connections from timeouts\n- Enables concurrent tool execution\n\n**Timeout Protection:** If a tool takes longer than 6000 seconds, it will be cancelled and return a timeout error. Adjust `_tool_timeout` for long-running operations.\n\n### 3. Dynamic Server Management\n\nAdd or remove MCP servers at runtime:\n\n```python\n# Add a custom server\nawait computer.add_server(\n namespace=\"CustomTools\",\n mcp_server=custom_server,\n tool_type=\"action\"\n)\n\n# Remove a server\nawait computer.delete_server(\n namespace=\"CustomTools\",\n tool_type=\"action\"\n)\n```\n\n### 4. Tool Introspection\n\nUse meta tools to discover available tools:\n\n```python\n# List all action tools\ntool_call = MCPToolCall(\n tool_key=\"action::list_tools\",\n tool_name=\"list_tools\",\n parameters={\"tool_type\": \"action\"}\n)\n\nresult = await computer.run_actions([tool_call])\n# Returns: List of all available action tools\n```\n\nFor more details on introspection capabilities, see [Computer - Meta Tools](../client/computer.md#meta-tools).\n\n## Configuration Files\n\nMCP configuration is located at:\n\n```\nconfig/ufo/mcp.yaml\n```\n\nFor detailed configuration options, see:\n\n- [MCP Configuration Guide](configuration.md) - Complete configuration reference\n- [System Configuration](../configuration/system/system_config.md) - MCP-related system settings \n- [MCP Reference](../configuration/system/mcp_reference.md) - MCP-specific settings\n\n## Use Cases\n\n### 1. UI Automation\n\n```yaml\n# Agent that automates UI interactions\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector # Detect UI elements\n action:\n - namespace: HostUIExecutor # Click, type, scroll\n```\n\n### 2. Document Processing\n\n```yaml\n# Agent specialized for Word documents\nAppAgent:\n WINWORD.EXE:\n data_collection:\n - namespace: UICollector # Read document structure\n action:\n - namespace: WordCOMExecutor # Insert tables, format text\n```\n\n### 3. Multi-Device Coordination\n\n```yaml\n# Agent that coordinates tasks across devices\nConstellationAgent:\n default:\n action:\n - namespace: ConstellationEditor # Create and assign tasks\n```\n\n### 4. Remote Hardware Monitoring\n\n```yaml\n# Agent that monitors remote hardware\nHardwareAgent:\n default:\n data_collection:\n - namespace: HardwareCollector\n type: http\n host: \"192.168.1.100\"\n port: 8006\n```\n\n### 5. Android Device Automation\n\n```yaml\n# Agent that automates Android devices via ADB\nMobileAgent:\n default:\n data_collection:\n - namespace: MobileDataCollector\n type: http\n host: \"localhost\" # Or remote Android automation server\n port: 8020\n path: \"/mcp\"\n action:\n - namespace: MobileExecutor\n type: http\n host: \"localhost\"\n port: 8021\n path: \"/mcp\"\n```\n\n## Getting Started\n\nTo start using MCP in UFO²:\n\n1. **Understand the two server types** - Read about [Data Collection](data_collection.md) and [Action](action.md) servers\n2. **Configure your agents** - See [Configuration Guide](configuration.md) for setup details\n3. **Use built-in servers** - Explore available [Local Servers](local_servers.md)\n4. **Create custom servers** - Follow the [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md)\n5. **Deploy remotely** - Learn about [Remote Servers](remote_servers.md) deployment\n\n## Related Documentation\n\n- [Data Collection Servers](data_collection.md) - Read-only observation tools\n- [Action Servers](action.md) - State-changing execution tools\n- [Configuration Guide](configuration.md) - How to configure MCP for agents\n- [Local Servers](local_servers.md) - Built-in MCP servers\n- [Remote Servers](remote_servers.md) - HTTP and Stdio deployment\n- [Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md) - Step-by-step guide to building custom servers\n- [Computer](../client/computer.md) - MCP tool execution layer\n- [Agent Client](../client/overview.md) - Client architecture overview\n- [Agent Overview](../ufo2/overview.md) - UFO² agent system architecture\n\n**Design Philosophy:**\n\nMCP in UFO² follows the **separation of concerns** principle:\n\n- **Agents** decide *what* to do (high-level planning)\n- **MCP servers** implement *how* to do it (low-level execution)\n- **Computer** manages the routing between them (middleware)\n\nThis architecture enables flexibility, extensibility, and maintainability.\n"
},
{
"path": "documents/docs/mcp/remote_servers.md",
"content": "# Remote MCP Servers\n\nRemote MCP servers run as separate processes or on different machines, communicating with UFO² over HTTP or stdio. This enables **cross-platform automation**, process isolation, and distributed workflows.\n\n**Cross-Platform Automation:** Remote servers enable **Windows UFO² agents to control Linux systems, mobile devices, and hardware** through HTTP MCP servers running on those platforms.\n\n## Deployment Models\n\n### HTTP Servers\n\nHTTP MCP servers run as standalone HTTP services, accessible via REST-like endpoints.\n\n**Advantages:**\n- Cross-platform communication (Windows ↔ Linux, Windows ↔ Hardware)\n- Language-agnostic (server can be in Python, Go, Rust, etc.)\n- Network-accessible (local or remote deployment)\n- Stateless design (each request is independent)\n\n**Use Cases:**\n- Linux command execution from Windows\n- Hardware device control (Arduino, robot arms, test fixtures)\n- Mobile device automation (Android, iOS via robot arm)\n- Distributed multi-machine workflows\n\n### Stdio Servers\n\nStdio MCP servers run as child processes, communicating via stdin/stdout.\n\n**Advantages:**\n- Process isolation (sandboxed execution)\n- Clean resource management (process lifetime)\n- Standard protocol (works with any language)\n\n**Use Cases:**\n- Custom Python/Node.js tools running in separate environments\n- Third-party MCP servers\n- Sandboxed execution for security\n\n\n---\n\n## Built-in Remote Servers\n\n### HardwareExecutor\n\n**Type**: Action (HTTP deployment) \n**Purpose**: Control hardware devices (Arduino HID, BB-8 test fixture, robot arm, mobile devices) \n**Deployment**: HTTP server on hardware controller machine \n**Agent**: HardwareAgent \n**Tools**: 30+ hardware control tools\n\n**[→ See complete HardwareExecutor documentation](servers/hardware_executor.md)** for all hardware control tools, deployment instructions, and usage examples.\n\n---\n\n### BashExecutor\n\n**Type**: Action (HTTP deployment) \n**Purpose**: Execute shell commands on Linux systems \n**Deployment**: HTTP server on Linux machine \n**Agent**: LinuxAgent \n**Tools**: 2 tools for command execution and system info\n\n**[→ See complete BashExecutor documentation](servers/bash_executor.md)** for Linux command execution, security guidelines, and systemd setup.\n\n---\n\n### MobileExecutor\n\n**Type**: Action + Data Collection (HTTP deployment, dual-server) \n**Purpose**: Android device automation via ADB \n**Deployment**: HTTP servers on machine with ADB access \n**Agent**: MobileAgent \n**Ports**: 8020 (data collection), 8021 (action) \n**Tools**: 13+ tools for Android automation\n\n**Architecture**: Runs as **two HTTP servers** that share a singleton state manager for coordinated operations:\n- **Mobile Data Collection Server** (port 8020): Screenshots, UI tree, device info, app list, controls\n- **Mobile Action Server** (port 8021): Tap, swipe, type, launch apps, press keys, control clicks\n\n**[→ See complete MobileExecutor documentation](servers/mobile_executor.md)** for all Android automation tools, dual-server architecture, deployment instructions, and usage examples.\n\n---\n\n## Configuration Reference\n\n### HTTP Server Fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `namespace` | String | ✅ Yes | Unique server identifier |\n| `type` | String | ✅ Yes | Must be `\"http\"` |\n| `host` | String | ✅ Yes | Server hostname or IP |\n| `port` | Integer | ✅ Yes | Server port number |\n| `path` | String | ✅ Yes | HTTP endpoint path |\n| `reset` | Boolean | ❌ No | Reset on context switch (default: `false`) |\n\n### Stdio Server Fields\n\n| Field | Type | Required | Description |\n|-------|------|----------|-------------|\n| `namespace` | String | ✅ Yes | Unique server identifier |\n| `type` | String | ✅ Yes | Must be `\"stdio\"` |\n| `command` | String | ✅ Yes | Executable command |\n| `start_args` | List[String] | ❌ No | Command-line arguments |\n| `env` | Dict | ❌ No | Environment variables |\n| `cwd` | String | ❌ No | Working directory |\n| `reset` | Boolean | ❌ No | Reset on context switch (default: `false`) |\n\n---\n\n## Example Configurations\n\n### HTTP: Hardware Control\n\n```yaml\nHardwareAgent:\n default:\n action:\n - namespace: HardwareExecutor\n type: http\n host: \"192.168.1.100\"\n port: 8006\n path: \"/mcp\"\n```\n\n**Server Start:**\n```bash\npython -m ufo.client.mcp.http_servers.hardware_mcp_server --host 0.0.0.0 --port 8006\n```\n\nSee the [HardwareExecutor documentation](servers/hardware_executor.md) for complete deployment instructions.\n\n### HTTP: Linux Command Execution\n\n```yaml\nLinuxAgent:\n default:\n action:\n - namespace: BashExecutor\n type: http\n host: \"192.168.1.50\"\n port: 8010\n path: \"/mcp\"\n```\n\n**Server Start:**\n```bash\npython -m ufo.client.mcp.http_servers.linux_mcp_server --host 0.0.0.0 --port 8010\n```\n\nSee the [BashExecutor documentation](servers/bash_executor.md) for systemd service setup.\n\n### HTTP: Android Device Automation\n\n```yaml\nMobileAgent:\n default:\n data_collection:\n - namespace: MobileDataCollector\n type: http\n host: \"192.168.1.60\" # Android automation server\n port: 8020\n path: \"/mcp\"\n action:\n - namespace: MobileExecutor\n type: http\n host: \"192.168.1.60\"\n port: 8021\n path: \"/mcp\"\n```\n\n**Server Start:**\n```bash\n# Start both servers (recommended - they share state)\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server both --host 0.0.0.0 --data-port 8020 --action-port 8021\n```\n\nSee the [MobileExecutor documentation](servers/mobile_executor.md) for complete deployment instructions and ADB setup.\n\n### Stdio: Custom Python Server\n\n```yaml\nCustomAgent:\n default:\n action:\n - namespace: CustomProcessor\n type: stdio\n command: \"python\"\n start_args: [\"-m\", \"custom_mcp_server\"]\n env:\n API_KEY: \"secret_key\"\n cwd: \"/path/to/server\"\n```\n\n---\n\n## Best Practices\n\n**Recommended Practices:**\n\n- ✅ **Use HTTP for cross-platform automation**\n- ✅ **Use stdio for process isolation**\n- ✅ **Validate remote server connectivity** before deployment\n- ✅ **Set appropriate timeouts** for long-running commands\n- ✅ **Use environment variables** for sensitive credentials\n\n**Anti-Patterns to Avoid:**\n\n- ❌ **Don't expose HTTP servers to public internet** without authentication\n- ❌ **Don't hardcode credentials** in configuration files\n- ❌ **Don't forget to start remote servers** before client connection\n\n---\n\n## See Also\n\n- [MCP Overview](./overview.md) - MCP architecture and deployment models\n- [Local Servers](./local_servers.md) - In-process servers\n- [MCP Configuration](./configuration.md) - Complete configuration reference\n- [Action Servers](./action.md) - Action execution overview\n- **[Creating Custom MCP Servers Tutorial](../tutorials/creating_mcp_servers.md)** - Step-by-step guide for HTTP/Stdio servers\n- [HardwareExecutor](servers/hardware_executor.md) - Complete hardware control reference\n- [BashExecutor](servers/bash_executor.md) - Complete Linux command reference\n"
},
{
"path": "documents/docs/mcp/servers/app_ui_executor.md",
"content": "# AppUIExecutor Server\n\n## Overview\n\n**AppUIExecutor** is an action server that provides application-level UI automation for the AppAgent. It enables precise interaction with UI controls within the currently selected application window.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** AppAgent \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `AppUIExecutor` |\n| **Server Name** | `UFO UI AppAgent Action MCP Server` |\n| **Platform** | Windows |\n| **Tool Type** | `action` |\n| **Tool Key Format** | `action::{tool_name}` |\n\n## Tools Summary\n\n| Tool Name | Description | Parameters |\n|-----------|-------------|------------|\n| `click_input` | Click on a UI control | `id`, `name`, `button`, `double` |\n| `click_on_coordinates` | Click at fractional coordinates | `x`, `y`, `button`, `double` |\n| `drag_on_coordinates` | Drag between two points | `start_x`, `start_y`, `end_x`, `end_y`, `button`, `duration`, `key_hold` |\n| `set_edit_text` | Set text in edit control | `id`, `name`, `text`, `clear_current_text` |\n| `keyboard_input` | Send keyboard keys | `id`, `name`, `keys`, `control_focus` |\n| `wheel_mouse_input` | Scroll with mouse wheel | `id`, `name`, `wheel_dist` |\n| `texts` | Get text from control | `id`, `name` |\n| `wait` | Wait for specified time | `seconds` |\n| `summary` | Provide observation summary | `text` |\n\n## Tool Details\n\n### click_input\n\nClick on a UI control element using the mouse.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `id` | `str` | ✅ Yes | - | Control ID from `get_app_window_controls_info` |\n| `name` | `str` | ✅ Yes | - | Control name matching the ID |\n| `button` | `str` | No | `\"left\"` | Mouse button: `\"left\"`, `\"right\"`, `\"middle\"`, `\"x\"` |\n| `double` | `bool` | No | `False` | Perform double-click |\n\n#### Returns\n\n`str` - Result message or warning if name doesn't match ID\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_input\",\n tool_name=\"click_input\",\n parameters={\n \"id\": \"5\",\n \"name\": \"Submit Button\",\n \"button\": \"left\",\n \"double\": False\n }\n )\n])\n```\n\n---\n\n### click_on_coordinates\n\nClick at specific fractional coordinates within the window (0.0-1.0).\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `x` | `float` | ✅ Yes | - | Relative x-coordinate (0.0-1.0) |\n| `y` | `float` | ✅ Yes | - | Relative y-coordinate (0.0-1.0) |\n| `button` | `str` | No | `\"left\"` | Mouse button |\n| `double` | `bool` | No | `False` | Double-click |\n\n#### Example\n\n```python\n# Click at center of window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_on_coordinates\",\n tool_name=\"click_on_coordinates\",\n parameters={\"x\": 0.5, \"y\": 0.5, \"button\": \"left\"}\n )\n])\n```\n\n---\n\n### drag_on_coordinates\n\nDrag from one fractional coordinate to another.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `start_x` | `float` | ✅ Yes | - | Start x-coordinate (0.0-1.0) |\n| `start_y` | `float` | ✅ Yes | - | Start y-coordinate (0.0-1.0) |\n| `end_x` | `float` | ✅ Yes | - | End x-coordinate (0.0-1.0) |\n| `end_y` | `float` | ✅ Yes | - | End y-coordinate (0.0-1.0) |\n| `button` | `str` | No | `\"left\"` | Mouse button |\n| `duration` | `float` | No | `1.0` | Drag duration in seconds |\n| `key_hold` | `str` | No | `None` | Key to hold (`\"ctrl\"`, `\"shift\"`) |\n\n#### Example\n\n```python\n# Drag from top-left to bottom-right\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::drag_on_coordinates\",\n tool_name=\"drag_on_coordinates\",\n parameters={\n \"start_x\": 0.2, \"start_y\": 0.2,\n \"end_x\": 0.8, \"end_y\": 0.8,\n \"duration\": 1.5\n }\n )\n])\n```\n\n---\n\n### set_edit_text\n\nSet text in an edit control.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `id` | `str` | ✅ Yes | - | Control ID |\n| `name` | `str` | ✅ Yes | - | Control name |\n| `text` | `str` | ✅ Yes | - | Text to set |\n| `clear_current_text` | `bool` | No | `False` | Clear existing text first |\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_edit_text\",\n tool_name=\"set_edit_text\",\n parameters={\n \"id\": \"3\",\n \"name\": \"Search Box\",\n \"text\": \"Hello World\",\n \"clear_current_text\": True\n }\n )\n])\n```\n\n---\n\n### keyboard_input\n\nSend keyboard input to a control or application.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `id` | `str` | ✅ Yes | - | Control ID |\n| `name` | `str` | ✅ Yes | - | Control name |\n| `keys` | `str` | ✅ Yes | - | Key sequence (e.g., `\"{VK_CONTROL}c\"`, `\"{TAB 2}\"`) |\n| `control_focus` | `bool` | No | `True` | Focus control before sending keys |\n\n#### Example\n\n```python\n# Copy selected text (Ctrl+C)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::keyboard_input\",\n tool_name=\"keyboard_input\",\n parameters={\n \"id\": \"1\",\n \"name\": \"Editor\",\n \"keys\": \"{VK_CONTROL}c\",\n \"control_focus\": True\n }\n )\n])\n\n# Press Tab twice\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::keyboard_input\",\n tool_name=\"keyboard_input\",\n parameters={\n \"id\": \"1\",\n \"name\": \"Form\",\n \"keys\": \"{TAB 2}\"\n }\n )\n])\n```\n\n---\n\n### wheel_mouse_input\n\nScroll using mouse wheel on a control.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `id` | `str` | ✅ Yes | - | Control ID |\n| `name` | `str` | ✅ Yes | - | Control name |\n| `wheel_dist` | `int` | No | `0` | Wheel notches (positive=up, negative=down) |\n\n#### Example\n\n```python\n# Scroll down 5 notches\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::wheel_mouse_input\",\n tool_name=\"wheel_mouse_input\",\n parameters={\n \"id\": \"10\",\n \"name\": \"Content Panel\",\n \"wheel_dist\": -5\n }\n )\n])\n```\n\n---\n\n### texts\n\nRetrieve all text content from a control.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `id` | `str` | ✅ Yes | - | Control ID |\n| `name` | `str` | ✅ Yes | - | Control name |\n\n#### Returns\n\n`str` - Text content of the control\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::texts\",\n tool_name=\"texts\",\n parameters={\"id\": \"7\", \"name\": \"Status Label\"}\n )\n])\n# result[0].data = \"Operation completed successfully\"\n```\n\n---\n\n### wait\n\nWait for a specified duration (non-blocking).\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `seconds` | `float` | ✅ Yes | - | Wait duration (max 300s) |\n\n#### Example\n\n```python\n# Wait for 2 seconds\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::wait\",\n tool_name=\"wait\",\n parameters={\"seconds\": 2.0}\n )\n])\n```\n\n---\n\n### summary\n\nProvide a visual summary of observations.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `text` | `str` | ✅ Yes | - | Summary text based on visual observation |\n\n#### Returns\n\n`str` - The summary text (passed through)\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::summary\",\n tool_name=\"summary\",\n parameters={\n \"text\": \"Window shows login form with username and password fields. Submit button is enabled.\"\n }\n )\n])\n```\n\n## Configuration\n\n```yaml\nAppAgent:\n default:\n action:\n - namespace: AppUIExecutor\n type: local\n reset: false\n \n # App-specific configuration\n WINWORD.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: WordCOMExecutor # Additional server for Word\n type: local\n```\n\n## Best Practices\n\n### 1. Always Verify Control ID and Name\n\n```python\n# ✅ Good\ncontrols = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_app_window_controls_info\", ...)\n])\n\ncontrol = controls[0].data[0] # Get first control\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_input\",\n parameters={\n \"id\": control[\"label\"],\n \"name\": control[\"control_text\"]\n }\n )\n])\n\n# ❌ Bad: Hardcode IDs\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_input\",\n parameters={\"id\": \"1\", \"name\": \"Button\"} # May not exist\n )\n])\n```\n\n### 2. Use Coordinates for Unlabeled Elements\n\n```python\n# When control not in control list\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_on_coordinates\",\n parameters={\"x\": 0.75, \"y\": 0.25} # Top-right area\n )\n])\n```\n\n### 3. Wait After Actions\n\n```python\n# Click button\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::click_input\", ...)\n])\n\n# Wait for UI update\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::wait\", parameters={\"seconds\": 1.0})\n])\n\n# Verify result\nscreenshot = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::capture_window_screenshot\", ...)\n])\n```\n\n## Related Documentation\n\n- [HostUIExecutor](./host_ui_executor.md) - Window selection\n- [UICollector](./ui_collector.md) - Control discovery\n- [Action Servers](../action.md) - Action concepts\n- [AppAgent Overview](../../ufo2/app_agent/overview.md) - AppAgent architecture\n"
},
{
"path": "documents/docs/mcp/servers/bash_executor.md",
"content": "# BashExecutor Server\n\n## Overview\n\n**BashExecutor** provides Linux shell command execution with output capture and system information retrieval via HTTP MCP server.\n\n**Server Type:** Action \n**Deployment:** HTTP (remote Linux server) \n**Default Port:** 8010 \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `BashExecutor` |\n| **Server Name** | `Linux Bash MCP Server` |\n| **Platform** | Linux |\n| **Tool Type** | `action` |\n| **Deployment** | HTTP server (stateless) |\n\n## Tools\n\n### execute_command\n\nExecute a shell command on Linux and return stdout/stderr with exit code.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `command` | `str` | ✅ Yes | - | Shell command to execute (valid bash/sh command) |\n| `timeout` | `int` | No | `30` | Maximum execution time in seconds (default: 30, max: any) |\n| `cwd` | `str` | No | `None` | Working directory path (absolute path recommended) |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool, # True if exit code == 0\n \"exit_code\": int, # Process exit code\n \"stdout\": str, # Standard output\n \"stderr\": str, # Standard error output\n # OR\n \"error\": str # Error message if execution failed\n}\n```\n\n#### Example\n\n```python\n# Simple command\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n tool_name=\"execute_command\",\n parameters={\n \"command\": \"ls -la /home\",\n \"timeout\": 30\n }\n )\n])\n\n# Output:\n# {\n# \"success\": True,\n# \"exit_code\": 0,\n# \"stdout\": \"total 12\\ndrwxr-xr-x 3 root root 4096 ...\",\n# \"stderr\": \"\"\n# }\n\n# Command with specific working directory\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n tool_name=\"execute_command\",\n parameters={\n \"command\": \"python script.py --arg value\",\n \"timeout\": 60,\n \"cwd\": \"/home/user/project\"\n }\n )\n])\n\n# Check system info\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n tool_name=\"execute_command\",\n parameters={\"command\": \"cat /etc/os-release\"}\n )\n])\n```\n\n#### Security Blocklist\n\nDangerous commands are automatically blocked:\n\n| Blocked Command | Reason |\n|-----------------|--------|\n| `rm -rf /` | System destruction |\n| `:(){ :\\|:& };:` | Fork bomb |\n| `mkfs` | Filesystem formatting |\n| `dd if=/dev/zero` | Disk overwrite |\n| `shutdown` | System shutdown |\n| `reboot` | System reboot |\n\n**Returns**: `{\"success\": False, \"error\": \"Blocked dangerous command.\"}`\n\n#### Timeout Handling\n\nIf command exceeds timeout:\n\n```python\n{\n \"success\": False,\n \"error\": \"Timeout after {timeout}s.\"\n}\n```\n\n#### Error Handling\n\nIf execution fails:\n\n```python\n{\n \"success\": False,\n \"error\": \"{exception_details}\"\n}\n```\n\n---\n\n### get_system_info\n\nGet basic Linux system information (uname, uptime, memory, disk).\n\n#### Parameters\n\nNone\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"uname\": str, # System and kernel info (uname -a)\n \"uptime\": str, # System uptime and load averages\n \"memory\": str, # Memory usage statistics (free -h)\n \"disk\": str # Disk space usage (df -h)\n}\n```\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::get_system_info\",\n tool_name=\"get_system_info\",\n parameters={}\n )\n])\n\n# Output:\n# {\n# \"uname\": \"Linux server 5.15.0-91-generic #101-Ubuntu SMP x86_64 GNU/Linux\",\n# \"uptime\": \" 10:30:45 up 5 days, 2:15, 3 users, load average: 0.52, 0.58, 0.59\",\n# \"memory\": \" total used free shared buff/cache available\\nMem: 15Gi 4.2Gi 7.8Gi 123Mi 3.0Gi 10Gi\\nSwap: 2.0Gi 0B 2.0Gi\",\n# \"disk\": \"Filesystem Size Used Avail Use% Mounted on\\n/dev/sda1 100G 45G 50G 48% /\"\n# }\n```\n\n#### Error Handling\n\nIf command fails, value is error message:\n\n```python\n{\n \"uname\": \"Linux ubuntu ...\",\n \"uptime\": \"Error: No such file or directory\",\n \"memory\": \"...\",\n \"disk\": \"...\"\n}\n```\n\n## Configuration\n\n### Client Configuration\n\n```yaml\n# Windows client connecting to Linux server\nHostAgent:\n default:\n action:\n - namespace: BashExecutor\n type: http\n host: \"192.168.1.100\" # Linux server IP\n port: 8010\n path: \"/mcp\"\n\n# Linux client (local)\nHostAgent:\n default:\n action:\n - namespace: BashExecutor\n type: http\n host: \"localhost\"\n port: 8010\n path: \"/mcp\"\n```\n\n## Deployment\n\n### Starting the Server\n\n```bash\n# Start Bash MCP server on Linux\npython -m ufo.client.mcp.http_servers.linux_mcp_server --host 0.0.0.0 --port 8010\n\n# Output:\n# ==================================================\n# UFO Linux Bash MCP Server\n# Linux command execution via Model Context Protocol\n# Running on 0.0.0.0:8010\n# ==================================================\n```\n\n### Command-Line Arguments\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--host` | `localhost` | Host to bind server to |\n| `--port` | `8010` | Port to run server on |\n\n### Systemd Service (Optional)\n\n```ini\n# /etc/systemd/system/ufo-bash-mcp.service\n[Unit]\nDescription=UFO Bash MCP Server\nAfter=network.target\n\n[Service]\nType=simple\nUser=ufo\nWorkingDirectory=/home/ufo/UFO2\nExecStart=/usr/bin/python3 -m ufo.client.mcp.http_servers.linux_mcp_server --host 0.0.0.0 --port 8010\nRestart=on-failure\n\n[Install]\nWantedBy=multi-user.target\n```\n\nEnable and start:\n```bash\nsudo systemctl enable ufo-bash-mcp\nsudo systemctl start ufo-bash-mcp\nsudo systemctl status ufo-bash-mcp\n```\n\n## Best Practices\n\n### 1. Use Absolute Paths\n\n```python\n# ✅ Good: Absolute paths\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\n \"command\": \"ls /home/user/project\",\n \"cwd\": \"/home/user\"\n }\n )\n])\n\n# ❌ Bad: Relative paths may fail\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\n \"command\": \"ls project\", # May fail if cwd unclear\n \"cwd\": None\n }\n )\n])\n```\n\n### 2. Set Appropriate Timeouts\n\n```python\n# Quick commands: short timeout\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\"command\": \"ls -la\", \"timeout\": 5}\n )\n])\n\n# Long-running: increase timeout\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\"command\": \"python train_model.py\", \"timeout\": 3600} # 1 hour\n )\n])\n```\n\n### 3. Check Exit Codes\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\"command\": \"grep 'pattern' file.txt\"}\n )\n])\n\nif result[0].data[\"success\"]:\n logger.info(f\"Found: {result[0].data['stdout']}\")\nelse:\n logger.warning(f\"Not found (exit code {result[0].data['exit_code']})\")\n```\n\n### 4. Validate Commands\n\n```python\ndef safe_execute(command: str, allowed_commands: List[str]):\n \"\"\"Whitelist-based command validation\"\"\"\n cmd_base = command.split()[0]\n \n if cmd_base not in allowed_commands:\n raise ValueError(f\"Command not allowed: {cmd_base}\")\n \n return MCPToolCall(\n tool_key=\"action::execute_command\",\n tool_name=\"execute_command\",\n parameters={\"command\": command}\n )\n\n# Usage\nallowed = [\"ls\", \"cat\", \"grep\", \"find\", \"python3\"]\nawait computer.run_actions([safe_execute(\"ls -la /home\", allowed)])\n```\n\n## Use Cases\n\n### 1. System Monitoring\n\n```python\n# Get system info\ninfo = await computer.run_actions([\n MCPToolCall(tool_key=\"action::get_system_info\", parameters={})\n])\n\n# Parse disk usage\ndisk_info = info[0].data[\"disk\"]\nif \"98%\" in disk_info:\n logger.warning(\"Disk almost full!\")\n```\n\n### 2. Log Analysis\n\n```python\n# Search logs\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\n \"command\": \"grep ERROR /var/log/application.log | tail -20\",\n \"timeout\": 10\n }\n )\n])\n\nerrors = result[0].data[\"stdout\"]\n```\n\n### 3. File Operations\n\n```python\n# Create directory\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\"command\": \"mkdir -p /tmp/workspace/data\"}\n )\n])\n\n# Copy files\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\"command\": \"cp source.txt /tmp/workspace/\"}\n )\n])\n```\n\n### 4. Script Execution\n\n```python\n# Run Python script\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::execute_command\",\n parameters={\n \"command\": \"python3 process_data.py --input data.csv --output results.json\",\n \"timeout\": 300,\n \"cwd\": \"/home/user/scripts\"\n }\n )\n])\n\nif result[0].data[\"success\"]:\n logger.info(\"Script completed successfully\")\nelse:\n logger.error(f\"Script failed: {result[0].data['stderr']}\")\n```\n\n## Comparison with CommandLineExecutor\n\n| Feature | CommandLineExecutor | BashExecutor |\n|---------|---------------------|--------------|\n| **Platform** | Windows/Cross-platform | Linux only |\n| **Output Capture** | ❌ No | ✅ Yes (stdout/stderr) |\n| **Exit Code** | ❌ No | ✅ Yes |\n| **Timeout** | Fixed 5s | ✅ Configurable |\n| **Working Directory** | ❌ No | ✅ Yes |\n| **Deployment** | Local | HTTP (remote) |\n| **Security** | ⚠️ No blocklist | ✅ Dangerous commands blocked |\n\n## Security Considerations\n\n!!!danger \"Security Warning\"\n - **Command injection risk**: Always validate/sanitize commands\n - **Privilege escalation**: Server runs with user permissions\n - **Network exposure**: Use firewall rules to limit access\n - **Sensitive data**: Stdout/stderr may contain secrets\n\n### Recommendations\n\n1. **Use firewall**: Restrict access to trusted IPs\n ```bash\n sudo ufw allow from 192.168.1.0/24 to any port 8010\n ```\n\n2. **Run as limited user**: Don't run server as root\n ```bash\n useradd -m -s /bin/bash ufo\n sudo -u ufo python3 -m ufo.client.mcp.http_servers.linux_mcp_server\n ```\n\n3. **Implement command whitelist**: Don't execute arbitrary commands\n\n4. **Use HTTPS**: For production, add TLS encryption\n\n## Related Documentation\n\n- [CommandLineExecutor](./command_line_executor.md) - Windows command execution\n- [HardwareExecutor](./hardware_executor.md) - Hardware control via HTTP\n- [Remote Servers](../remote_servers.md) - HTTP deployment guide\n- [Action Servers](../action.md) - Action server concepts\n"
},
{
"path": "documents/docs/mcp/servers/command_line_executor.md",
"content": "# CommandLineExecutor Server\n\n## Overview\n\n**CommandLineExecutor** provides shell command execution capabilities for launching applications and running system commands.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** HostAgent, AppAgent \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `CommandLineExecutor` |\n| **Server Name** | `UFO CLI MCP Server` |\n| **Platform** | Cross-platform (Windows, Linux, macOS) |\n| **Tool Type** | `action` |\n\n## Tools\n\n### run_shell\n\nExecute a shell command to launch applications or perform system operations.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `bash_command` | `str` | ✅ Yes | Command to execute in shell |\n\n#### Returns\n\n`None` - Command is launched asynchronously (5-second wait after execution)\n\n#### Example\n\n```python\n# Launch Notepad\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n tool_name=\"run_shell\",\n parameters={\"bash_command\": \"notepad.exe\"}\n )\n])\n\n# Launch application with arguments\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n tool_name=\"run_shell\",\n parameters={\"bash_command\": \"python script.py --arg value\"}\n )\n])\n\n# Create directory (Windows)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n tool_name=\"run_shell\",\n parameters={\"bash_command\": \"mkdir C:\\\\temp\\\\newfolder\"}\n )\n])\n```\n\n#### Error Handling\n\nRaises `ToolError` if:\n- Command is empty\n- Execution fails\n\n```python\n# Error: Empty command\nToolError(\"Bash command cannot be empty.\")\n\n# Error: Execution failed\nToolError(\"Failed to launch application: {error_details}\")\n```\n\n#### Implementation Details\n\n- Uses `subprocess.Popen` with `shell=True`\n- Waits 5 seconds after launch for application to start\n- Non-blocking: Returns immediately after launch\n\n!!!danger \"Security Warning\"\n **Arbitrary command execution risk!** Always validate commands before execution.\n \n Dangerous examples:\n - `rm -rf /` (Linux)\n - `del /F /S /Q C:\\*` (Windows)\n - `shutdown /s /t 0`\n \n **Best Practice**: Implement command whitelist or validation.\n\n## Configuration\n\n```yaml\nHostAgent:\n default:\n action:\n - namespace: HostUIExecutor\n type: local\n - namespace: CommandLineExecutor\n type: local # Enable shell execution\n\nAppAgent:\n default:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: CommandLineExecutor\n type: local # Enable if app needs to launch child processes\n```\n\n## Best Practices\n\n### 1. Validate Commands\n\n```python\ndef safe_run_shell(command: str):\n \"\"\"Whitelist-based command validation\"\"\"\n allowed_commands = [\n \"notepad.exe\",\n \"calc.exe\",\n \"mspaint.exe\",\n \"code\", # VS Code\n ]\n \n cmd_base = command.split()[0]\n if cmd_base not in allowed_commands:\n raise ValueError(f\"Command not allowed: {cmd_base}\")\n \n return MCPToolCall(\n tool_key=\"action::run_shell\",\n tool_name=\"run_shell\",\n parameters={\"bash_command\": command}\n )\n\n# Usage\nawait computer.run_actions([safe_run_shell(\"notepad.exe test.txt\")])\n```\n\n### 2. Wait for Application Launch\n\n```python\n# Launch application\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"notepad.exe\"}\n )\n])\n\n# Wait for launch (5 seconds built-in + extra)\nawait asyncio.sleep(2)\n\n# Get window list\nwindows = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_desktop_app_info\", ...)\n])\n\n# Find Notepad window\nnotepad_windows = [w for w in windows[0].data if \"Notepad\" in w[\"name\"]]\n```\n\n### 3. Platform-Specific Commands\n\n```python\nimport platform\n\ndef get_platform_command(app_name: str) -> str:\n \"\"\"Get platform-specific command\"\"\"\n if platform.system() == \"Windows\":\n commands = {\n \"notepad\": \"notepad.exe\",\n \"terminal\": \"cmd.exe\",\n \"browser\": \"start msedge\"\n }\n elif platform.system() == \"Darwin\": # macOS\n commands = {\n \"notepad\": \"open -a TextEdit\",\n \"terminal\": \"open -a Terminal\",\n \"browser\": \"open -a Safari\"\n }\n else: # Linux\n commands = {\n \"notepad\": \"gedit\",\n \"terminal\": \"gnome-terminal\",\n \"browser\": \"firefox\"\n }\n \n return commands.get(app_name, app_name)\n\n# Usage\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": get_platform_command(\"notepad\")}\n )\n])\n```\n\n### 4. Handle Launch Failures\n\n```python\ntry:\n result = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"nonexistent.exe\"}\n )\n ])\n \n if result[0].is_error:\n logger.error(f\"Failed to launch: {result[0].content}\")\n # Retry with alternative command\n \nexcept Exception as e:\n logger.error(f\"Command execution exception: {e}\")\n```\n\n## Use Cases\n\n### 1. Application Launching\n\n```python\n# Launch text editor\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"notepad.exe\"}\n )\n])\n\n# Launch browser with URL\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"start https://www.example.com\"}\n )\n])\n```\n\n### 2. File Operations\n\n```python\n# Create directory\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"mkdir C:\\\\temp\\\\workspace\"}\n )\n])\n\n# Copy file\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"copy source.txt dest.txt\"}\n )\n])\n```\n\n### 3. Script Execution\n\n```python\n# Run Python script\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"python automation_script.py --mode batch\"}\n )\n])\n\n# Run PowerShell script\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::run_shell\",\n parameters={\"bash_command\": \"powershell -File script.ps1\"}\n )\n])\n```\n\n## Limitations\n\n- **No output capture**: Command output (stdout/stderr) is not returned\n- **No exit code**: Cannot determine if command succeeded\n- **Async execution**: No way to know when command completes\n- **Security risk**: Arbitrary command execution\n\n**Tip:** For Linux systems with output capture and better control, use **BashExecutor** server instead.\n\n## Related Documentation\n\n- [BashExecutor](./bash_executor.md) - Linux command execution with output\n- [Action Servers](../action.md) - Action server concepts\n- [HostAgent](../../ufo2/host_agent/overview.md) - HostAgent architecture\n\n"
},
{
"path": "documents/docs/mcp/servers/constellation_editor.md",
"content": "# ConstellationEditor Server\n\n## Overview\n\n**ConstellationEditor** provides multi-device task coordination and dependency management for distributed workflows in UFO².\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** GalaxyAgent \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `ConstellationEditor` |\n| **Server Name** | `UFO Constellation Editor MCP Server` |\n| **Platform** | Cross-platform |\n| **Tool Type** | `action` |\n\n## Tools Summary\n\n| Category | Tool Name | Description |\n|----------|-----------|-------------|\n| **Task Management** | `add_task` | Create new task |\n| | `remove_task` | Delete task |\n| | `update_task` | Modify task properties |\n| **Dependency Management** | `add_dependency` | Create task dependency |\n| | `remove_dependency` | Delete dependency |\n| | `update_dependency` | Modify dependency description |\n| **Bulk Operations** | `build_constellation` | Build complete constellation from config |\n\n## Task Management Tools\n\n### add_task\n\nAdd a new task to the constellation.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `task_id` | `str` | ✅ Yes | - | Unique task identifier (e.g., `\"open_browser\"`, `\"login_system\"`) |\n| `name` | `str` | ✅ Yes | - | Human-readable task name (e.g., `\"Open Browser\"`) |\n| `description` | `str` | ✅ Yes | - | Detailed task description with steps and expected outcomes |\n| `target_device_id` | `str` | No | `None` | Device ID where task should execute (from Device Info List) |\n| `tips` | `List[str]` | No | `None` | List of tips and best practices for task execution |\n\n#### Returns\n\n`str` - JSON representation of complete TaskConstellation after adding task\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::add_task\",\n tool_name=\"add_task\",\n parameters={\n \"task_id\": \"extract_data\",\n \"name\": \"Extract Data from Excel\",\n \"description\": \"Open Excel file, extract data from Sheet1, save to CSV format\",\n \"target_device_id\": \"device_windows_001\",\n \"tips\": [\n \"Ensure Excel is installed\",\n \"Close Excel before running task\",\n \"Verify file path exists\"\n ]\n }\n )\n])\n```\n\n---\n\n### remove_task\n\nRemove a task from the constellation (also removes all dependencies involving this task).\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `task_id` | `str` | ✅ Yes | Unique task identifier to remove |\n\n#### Returns\n\n`str` - JSON representation of constellation after removal\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::remove_task\",\n tool_name=\"remove_task\",\n parameters={\"task_id\": \"extract_data\"}\n )\n])\n```\n\n---\n\n### update_task\n\nUpdate specific fields of an existing task.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `task_id` | `str` | ✅ Yes | - | Task to update |\n| `name` | `str` | No | `None` | New task name (leave empty to keep current) |\n| `description` | `str` | No | `None` | New description |\n| `target_device_id` | `str` | No | `None` | New target device |\n| `tips` | `List[str]` | No | `None` | New tips list |\n\n**Note:** Only provided fields are updated; others remain unchanged.\n\n#### Returns\n\n`str` - JSON representation of constellation after update\n\n#### Example\n\n```python\n# Update only description and tips\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::update_task\",\n tool_name=\"update_task\",\n parameters={\n \"task_id\": \"extract_data\",\n \"description\": \"Extract data from Excel Sheet1 and Sheet2, merge into single CSV\",\n \"tips\": [\n \"Ensure Excel is installed\",\n \"Handle merged cells properly\",\n \"Verify output CSV encoding\"\n ]\n }\n )\n])\n```\n\n## Dependency Management Tools\n\n### add_dependency\n\nCreate a dependency relationship between two tasks (source task must complete before target task can start).\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `dependency_id` | `str` | ✅ Yes | **MUST generate unique ID** (e.g., `\"login->extract_data\"`) |\n| `from_task_id` | `str` | ✅ Yes | Source/prerequisite task ID |\n| `to_task_id` | `str` | ✅ Yes | Target/dependent task ID |\n| `condition_description` | `str` | No | `None` | Human-readable description of dependency condition |\n\n!!!warning \"dependency_id Required\"\n You **MUST** generate and provide a unique `dependency_id`. Do not omit this parameter!\n\n#### Returns\n\n`str` - JSON representation of constellation after adding dependency\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::add_dependency\",\n tool_name=\"add_dependency\",\n parameters={\n \"dependency_id\": \"login_system->extract_data\", # MUST provide\n \"from_task_id\": \"login_system\",\n \"to_task_id\": \"extract_data\",\n \"condition_description\": \"Wait for successful user authentication before accessing user data\"\n }\n )\n])\n```\n\n---\n\n### remove_dependency\n\nRemove a dependency relationship without affecting the tasks themselves.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `dependency_id` | `str` | ✅ Yes | Dependency ID (line_id) to remove |\n\n#### Returns\n\n`str` - JSON representation of constellation after removal\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::remove_dependency\",\n tool_name=\"remove_dependency\",\n parameters={\"dependency_id\": \"login_system->extract_data\"}\n )\n])\n```\n\n---\n\n### update_dependency\n\nUpdate the condition description of an existing dependency.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `dependency_id` | `str` | ✅ Yes | Dependency to update |\n| `condition_description` | `str` | ✅ Yes | New condition description |\n\n#### Returns\n\n`str` - JSON representation of constellation after update\n\n#### Example\n\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::update_dependency\",\n tool_name=\"update_dependency\",\n parameters={\n \"dependency_id\": \"login_system->extract_data\",\n \"condition_description\": \"Wait for successful authentication and database connection before data extraction\"\n }\n )\n])\n```\n\n## Bulk Operations\n\n### build_constellation\n\nBuild a complete constellation from configuration data (batch creation).\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `config` | `TaskConstellationSchema` | ✅ Yes | - | Complete constellation configuration |\n| `clear_existing` | `bool` | No | `True` | Clear existing tasks/dependencies before building |\n\n#### Configuration Schema\n\n```python\n{\n \"tasks\": [\n {\n \"task_id\": \"string (required)\",\n \"name\": \"string (optional)\",\n \"description\": \"string (required)\",\n \"target_device_id\": \"string (optional)\",\n \"priority\": int (1-4, optional),\n \"status\": \"string (optional)\",\n \"tips\": [\"string\"] (optional)\n }\n ],\n \"dependencies\": [\n {\n \"from_task_id\": \"string (required)\",\n \"to_task_id\": \"string (required)\",\n \"dependency_type\": \"string (optional)\",\n \"condition_description\": \"string (optional)\"\n }\n ],\n \"metadata\": dict (optional)\n}\n```\n\n#### Returns\n\n`str` - JSON representation of built constellation\n\n#### Example\n\n```python\nconfig = {\n \"tasks\": [\n {\n \"task_id\": \"open_browser\",\n \"name\": \"Open Browser\",\n \"description\": \"Launch Chrome and navigate to login page\",\n \"target_device_id\": \"device_001\"\n },\n {\n \"task_id\": \"login\",\n \"name\": \"User Login\",\n \"description\": \"Enter credentials and submit login form\",\n \"target_device_id\": \"device_001\"\n },\n {\n \"task_id\": \"extract_data\",\n \"name\": \"Extract Data\",\n \"description\": \"Navigate to data page and extract table\",\n \"target_device_id\": \"device_002\"\n }\n ],\n \"dependencies\": [\n {\n \"from_task_id\": \"open_browser\",\n \"to_task_id\": \"login\",\n \"condition_description\": \"Browser must be open before login\"\n },\n {\n \"from_task_id\": \"login\",\n \"to_task_id\": \"extract_data\",\n \"condition_description\": \"User must be authenticated before data access\"\n }\n ]\n}\n\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::build_constellation\",\n tool_name=\"build_constellation\",\n parameters={\n \"config\": config,\n \"clear_existing\": True\n }\n )\n])\n```\n\n## Configuration\n\n```yaml\nGalaxyAgent:\n default:\n action:\n - namespace: ConstellationEditor\n type: local\n```\n\n## Best Practices\n\n### 1. Use Descriptive Task IDs\n\n```python\n# ✅ Good: Clear task IDs\n\"task_id\": \"extract_sales_data_from_excel\"\n\"task_id\": \"send_email_notification\"\n\"task_id\": \"process_user_input\"\n\n# ❌ Bad: Unclear IDs\n\"task_id\": \"task1\"\n\"task_id\": \"do_stuff\"\n\"task_id\": \"process\"\n```\n\n### 2. Always Provide dependency_id\n\n```python\n# ✅ Good: Generate unique dependency_id\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::add_dependency\",\n parameters={\n \"dependency_id\": f\"{from_task}->{ to_task}\", # Generate ID\n \"from_task_id\": from_task,\n \"to_task_id\": to_task\n }\n )\n])\n\n# ❌ Bad: Omit dependency_id\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::add_dependency\",\n parameters={\n # Missing dependency_id - will fail!\n \"from_task_id\": from_task,\n \"to_task_id\": to_task\n }\n )\n])\n```\n\n### 3. Provide Detailed Descriptions\n\n```python\n# ✅ Good: Detailed description\n{\n \"description\": \"Open Chrome browser, navigate to https://example.com/login, wait for page to fully load, then take a screenshot and save it to C:\\\\screenshots\\\\login_page.png\"\n}\n\n# ❌ Bad: Vague description\n{\n \"description\": \"Open browser\"\n}\n```\n\n## Use Cases\n\n### Multi-Device Workflow\n\n```python\n# 1. Create tasks on different devices\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::add_task\", parameters={\n \"task_id\": \"windows_extract\",\n \"name\": \"Extract Data on Windows\",\n \"description\": \"Extract Excel data\",\n \"target_device_id\": \"device_windows_001\"\n })\n])\n\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::add_task\", parameters={\n \"task_id\": \"linux_process\",\n \"name\": \"Process Data on Linux\",\n \"description\": \"Run Python analysis script\",\n \"target_device_id\": \"device_linux_001\"\n })\n])\n\n# 2. Create dependency\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::add_dependency\", parameters={\n \"dependency_id\": \"windows_extract->linux_process\",\n \"from_task_id\": \"windows_extract\",\n \"to_task_id\": \"linux_process\",\n \"condition_description\": \"Data must be extracted before processing\"\n })\n])\n```\n\n## Related Documentation\n\n- [Action Servers](../action.md) - Action server concepts\n- [MCP Overview](../overview.md) - MCP architecture\n- [Configuration Guide](../configuration.md) - Constellation setup\n- [Local Servers](../local_servers.md) - Local server deployment\n"
},
{
"path": "documents/docs/mcp/servers/excel_com_executor.md",
"content": "# ExcelCOMExecutor Server\n\n## Overview\n\n**ExcelCOMExecutor** provides Microsoft Excel automation via COM API for efficient spreadsheet manipulation.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** AppAgent \n**Target Application:** Microsoft Excel (`EXCEL.EXE`) \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `ExcelCOMExecutor` |\n| **Platform** | Windows |\n| **Requires** | Microsoft Excel (COM interface) |\n| **Tool Type** | `action` |\n\n## Tools Summary\n\n| Tool Name | Description |\n|-----------|-------------|\n| `table2markdown` | Convert Excel sheet to Markdown table |\n| `insert_excel_table` | Insert table data into sheet |\n| `select_table_range` | Select cell range |\n| `save_as` | Save/export workbook |\n| `reorder_columns` | Reorder columns in sheet |\n| `get_range_values` | Get values from cell range |\n\n## Tool Details\n\n### table2markdown\n\nConvert an Excel sheet to Markdown format table.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `sheet_name` | `str` or `int` | ✅ Yes | Sheet name or index (1-based) |\n\n#### Returns\n\n`str` - Markdown-formatted table\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::table2markdown\",\n tool_name=\"table2markdown\",\n parameters={\"sheet_name\": \"Sales Data\"}\n )\n])\n\n# Output:\n# | Product | Q1 | Q2 | Q3 | Q4 |\n# |---------|----|----|----|----|\n# | A | 100| 150| 120| 180|\n# | B | 200| 180| 210| 190|\n```\n\n---\n\n### insert_excel_table\n\nInsert a table (2D list) into an Excel sheet at a specified position.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `table` | `List[List[Any]]` | ✅ Yes | 2D list of values (strings/numbers) |\n| `sheet_name` | `str` | ✅ Yes | Target sheet name |\n| `start_row` | `int` | ✅ Yes | Start row (1-based) |\n| `start_col` | `int` | ✅ Yes | Start column (1-based) |\n\n#### Returns\n\n`str` - Success message\n\n#### Example\n\n```python\n# Define table data\ndata = [\n [\"Name\", \"Age\", \"Gender\"],\n [\"Alice\", 30, \"Female\"],\n [\"Bob\", 25, \"Male\"],\n [\"Charlie\", 35, \"Male\"]\n]\n\n# Insert at A1\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::insert_excel_table\",\n tool_name=\"insert_excel_table\",\n parameters={\n \"table\": data,\n \"sheet_name\": \"Sheet1\",\n \"start_row\": 1,\n \"start_col\": 1\n }\n )\n])\n```\n\n---\n\n### select_table_range\n\nSelect a range of cells in a sheet (faster than dragging).\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `sheet_name` | `str` | ✅ Yes | Sheet name |\n| `start_row` | `int` | ✅ Yes | Start row (1-based) |\n| `start_col` | `int` | ✅ Yes | Start column (1=A, 2=B, etc.) |\n| `end_row` | `int` | ✅ Yes | End row (`-1` = last row with content) |\n| `end_col` | `int` | ✅ Yes | End column (`-1` = last column with content) |\n\n#### Returns\n\n`str` - Selection confirmation message\n\n#### Example\n\n```python\n# Select A1:D10\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_table_range\",\n tool_name=\"select_table_range\",\n parameters={\n \"sheet_name\": \"Sheet1\",\n \"start_row\": 1,\n \"start_col\": 1, # Column A\n \"end_row\": 10,\n \"end_col\": 4 # Column D\n }\n )\n])\n\n# Select all data (A1 to last used cell)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_table_range\",\n tool_name=\"select_table_range\",\n parameters={\n \"sheet_name\": \"Sheet1\",\n \"start_row\": 1,\n \"start_col\": 1,\n \"end_row\": -1, # Last row with data\n \"end_col\": -1 # Last column with data\n }\n )\n])\n```\n\n---\n\n### save_as\n\nSave or export Excel workbook to specified format.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `file_dir` | `str` | No | `\"\"` | Directory path |\n| `file_name` | `str` | No | `\"\"` | Filename without extension |\n| `file_ext` | `str` | No | `\"\"` | Extension (default: `.csv`) |\n\n#### Supported Extensions\n\n- `.csv` - CSV format (default)\n- `.xlsx` - Excel workbook\n- `.xls` - Excel 97-2003 format\n- `.txt` - Tab-delimited text\n- `.pdf` - PDF format\n\n#### Example\n\n```python\n# Save as CSV\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"C:\\\\Data\\\\Exports\",\n \"file_name\": \"sales_report\",\n \"file_ext\": \".csv\"\n }\n )\n])\n```\n\n---\n\n### reorder_columns\n\nReorder columns in a sheet based on desired column name order.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `sheet_name` | `str` | ✅ Yes | Sheet name |\n| `desired_order` | `List[str]` | ✅ Yes | List of column names in new order |\n\n#### Returns\n\n`str` - Success/failure message\n\n#### Example\n\n```python\n# Original columns: [\"Name\", \"Age\", \"Email\", \"Phone\"]\n# Reorder to: [\"Name\", \"Phone\", \"Email\", \"Age\"]\n\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::reorder_columns\",\n tool_name=\"reorder_columns\",\n parameters={\n \"sheet_name\": \"Contacts\",\n \"desired_order\": [\"Name\", \"Phone\", \"Email\", \"Age\"]\n }\n )\n])\n```\n\n---\n\n### get_range_values\n\nGet values from a specified cell range.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `sheet_name` | `str` | ✅ Yes | Sheet name |\n| `start_row` | `int` | ✅ Yes | Start row |\n| `start_col` | `int` | ✅ Yes | Start column |\n| `end_row` | `int` | ✅ Yes | End row |\n| `end_col` | `int` | ✅ Yes | End column |\n\n#### Returns\n\n`List[List[Any]]` - 2D list of cell values\n\n#### Example\n\n```python\n# Get A1:C3\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::get_range_values\",\n tool_name=\"get_range_values\",\n parameters={\n \"sheet_name\": \"Sheet1\",\n \"start_row\": 1,\n \"start_col\": 1,\n \"end_row\": 3,\n \"end_col\": 3\n }\n )\n])\n\n# Output: [[\"A1\", \"B1\", \"C1\"], [\"A2\", \"B2\", \"C2\"], [\"A3\", \"B3\", \"C3\"]]\n```\n\n## Configuration\n\n```yaml\nAppAgent:\n EXCEL.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: ExcelCOMExecutor\n type: local\n reset: true # Recommended: prevent data leakage between workbooks\n```\n\n## Best Practices\n\n### 1. Use Column Numbers for select_table_range\n\n```python\n# Column mapping: A=1, B=2, C=3, D=4, ...\n# Select A1:D10\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_table_range\",\n parameters={\n \"sheet_name\": \"Sheet1\",\n \"start_row\": 1,\n \"start_col\": 1, # A\n \"end_row\": 10,\n \"end_col\": 4 # D\n }\n )\n])\n```\n\n### 2. Insert Data Efficiently\n\n```python\n# ✅ Good: Insert entire table at once\ndata = [[\"Header1\", \"Header2\"], [\"Val1\", \"Val2\"]]\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::insert_excel_table\", parameters={\n \"table\": data, \"sheet_name\": \"Sheet1\", \"start_row\": 1, \"start_col\": 1\n })\n])\n\n# ❌ Bad: Insert cell by cell\nfor row in data:\n for col in row:\n # Multiple calls...\n```\n\n### 3. Save Frequently\n\n```python\n# After data insertion/manipulation\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::save_as\", parameters={\"file_ext\": \".xlsx\"})\n])\n```\n\n## Use Cases\n\n### Data Processing Workflow\n\n```python\n# 1. Get data\ndata = await computer.run_actions([\n MCPToolCall(tool_key=\"action::get_range_values\", parameters={\n \"sheet_name\": \"Raw Data\", \"start_row\": 1, \"start_col\": 1,\n \"end_row\": -1, \"end_col\": -1\n })\n])\n\n# 2. Process data (Python)\nprocessed = process_data(data[0].data)\n\n# 3. Insert into new sheet\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::insert_excel_table\", parameters={\n \"table\": processed, \"sheet_name\": \"Processed\", \"start_row\": 1, \"start_col\": 1\n })\n])\n\n# 4. Export as CSV\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::save_as\", parameters={\"file_ext\": \".csv\"})\n])\n```\n\n## Related Documentation\n\n- [WordCOMExecutor](./word_com_executor.md) - Word COM automation\n- [PowerPointCOMExecutor](./ppt_com_executor.md) - PowerPoint COM automation\n"
},
{
"path": "documents/docs/mcp/servers/hardware_executor.md",
"content": "# HardwareExecutor Server\n\n## Overview\n\n**HardwareExecutor** provides hardware control capabilities including Arduino HID, BB-8 test fixture, robot arm, mouse control, and screenshot capture.\n\n**Server Type:** Action \n**Deployment:** HTTP (remote server) \n**Default Port:** 8006 \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `HardwareExecutor` |\n| **Server Name** | `Echo Base MCP Server` |\n| **Platform** | Cross-platform (requires hardware) |\n| **Tool Type** | `action` |\n| **Deployment** | HTTP server (stateless) |\n\n## Tool Categories\n\n### 1. Arduino HID Tools (Keyboard/Mouse Emulation)\n### 2. Mouse Control Tools\n### 3. BB-8 Test Fixture Tools\n### 4. Robot Arm Tools\n### 5. Screenshot Tool\n\n## Arduino HID Tools\n\n### arduino_hid_status\n\nGet Arduino HID device status.\n\n**Returns**: `Dict[str, Any]` with `connected`, `status`, `device`\n\n---\n\n### arduino_hid_connect\n\nConnect to Arduino HID device.\n\n**Returns**: `Dict[str, Any]` with success message\n\n---\n\n### arduino_hid_disconnect\n\nDisconnect from Arduino HID device.\n\n**Returns**: `Dict[str, Any]` with success message\n\n---\n\n### type_text\n\nType a string of text via Arduino HID.\n\n**Parameters**:\n- `text` (`str`): Text to type\n\n**Returns**: Success message\n\n**Example**:\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::type_text\",\n tool_name=\"type_text\",\n parameters={\"text\": \"Hello, World!\"}\n )\n])\n```\n\n---\n\n### press_key_sequence\n\nPress a sequence of keys.\n\n**Parameters**:\n- `keys` (`List[str]`): List of key names\n- `interval` (`float`): Interval between key presses (default: 0.1)\n\n**Example**:\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::press_key_sequence\",\n tool_name=\"press_key_sequence\",\n parameters={\n \"keys\": [\"a\", \"b\", \"c\"],\n \"interval\": 0.2\n }\n )\n])\n```\n\n---\n\n### press_hotkey\n\nPress multiple keys simultaneously (hotkey combination).\n\n**Parameters**:\n- `keys` (`List[str]`): List of keys to press together\n\n**Example**:\n```python\n# Ctrl+C\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::press_hotkey\",\n tool_name=\"press_hotkey\",\n parameters={\"keys\": [\"ctrl\", \"c\"]}\n )\n])\n```\n\n## Mouse Control Tools\n\n### move_mouse\n\nMove the mouse pointer.\n\n**Parameters**:\n- `x` (`int`): X coordinate\n- `y` (`int`): Y coordinate\n- `absolute` (`bool`): Absolute (True) or relative (False) positioning (default: False)\n\n---\n\n### click_mouse\n\nClick mouse button.\n\n**Parameters**:\n- `button` (`str`): `\"left\"`, `\"right\"`, or `\"middle\"` (default: `\"left\"`)\n- `count` (`int`): Number of clicks (default: 1)\n- `interval` (`float`): Interval between clicks (default: 0.1)\n\n---\n\n### press_mouse_button\n\nPress and hold mouse button.\n\n**Parameters**:\n- `button` (`str`): Mouse button (default: `\"left\"`)\n\n---\n\n### release_mouse_button\n\nRelease mouse button.\n\n**Parameters**:\n- `button` (`str`): Mouse button (default: `\"left\"`)\n\n---\n\n### scroll_mouse\n\nScroll mouse wheel.\n\n**Parameters**:\n- `vertical` (`int`): Vertical scroll amount (default: 0)\n- `horizontal` (`int`): Horizontal scroll amount (default: 0)\n\n**Example**:\n```python\n# Scroll down\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::scroll_mouse\",\n tool_name=\"scroll_mouse\",\n parameters={\"vertical\": -5, \"horizontal\": 0}\n )\n])\n```\n\n---\n\n### drag_mouse\n\nDrag mouse from start to end position.\n\n**Parameters**:\n- `start` (`Tuple[int, int]`): Start (x, y) coordinates\n- `end` (`Tuple[int, int]`): End (x, y) coordinates\n- `button` (`str`): Mouse button (default: `\"left\"`)\n- `duration` (`float`): Drag duration in seconds (default: 0.5)\n\n**Example**:\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::drag_mouse\",\n tool_name=\"drag_mouse\",\n parameters={\n \"start\": [100, 100],\n \"end\": [300, 300],\n \"duration\": 1.0\n }\n )\n])\n```\n\n---\n\n### double_click_mouse\n\nPerform double-click.\n\n**Parameters**:\n- `button` (`str`): Mouse button (default: `\"left\"`)\n\n---\n\n### right_click_mouse\n\nShortcut for right-click.\n\n---\n\n### middle_click_mouse\n\nShortcut for middle-click.\n\n## BB-8 Test Fixture Tools\n\nTest fixture for Surface device testing.\n\n### bb8_status\n\nGet BB-8 test fixture status.\n\n---\n\n### bb8_connect / bb8_disconnect\n\nConnect/disconnect to BB-8.\n\n---\n\n### bb8_usb_port_plug / bb8_usb_port_unplug\n\nPlug/unplug USB device.\n\n**Parameters**:\n- `port_name` (`str`): USB port name\n\n---\n\n### bb8_psu_charger_plug / bb8_psu_charger_unplug\n\nPlug/unplug PSU charger.\n\n---\n\n### bb8_blade_attach / bb8_blade_detach\n\nAttach/detach blade.\n\n---\n\n### bb8_lid_open / bb8_lid_close\n\nOpen/close lid.\n\n---\n\n### bb8_button_press\n\nPress a physical button.\n\n**Parameters**:\n- `button_name` (`str`): Button name\n\n---\n\n### bb8_button_long_press\n\nLong press a button.\n\n**Parameters**:\n- `button_name` (`str`): Button name\n\n## Robot Arm Tools\n\nPhysical robot arm for touchscreen interaction.\n\n### robot_arm_status\n\nGet robot arm status (position, connection).\n\n---\n\n### robot_arm_connect / robot_arm_disconnect\n\nConnect/disconnect robot arm.\n\n---\n\n### touch_screen\n\nSimulate touch at specific screen location.\n\n**Parameters**:\n- `location` (`Tuple[int, int]`): (x, y) coordinates\n\n**Example**:\n```python\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::touch_screen\",\n tool_name=\"touch_screen\",\n parameters={\"location\": [500, 300]}\n )\n])\n```\n\n---\n\n### draw_on_screen\n\nDraw on screen by following coordinate path.\n\n**Parameters**:\n- `path` (`List[Tuple[int, int]]`): List of (x, y) coordinates\n\n---\n\n### tap_screen\n\nSimulate tap(s) on screen.\n\n**Parameters**:\n- `location` (`Tuple[int, int]`): Tap location\n- `count` (`int`): Number of taps (default: 1)\n- `interval` (`float`): Interval between taps (default: 0.1)\n\n---\n\n### swipe_screen\n\nSimulate swipe gesture.\n\n**Parameters**:\n- `start_location` (`Tuple[int, int]`): Start position\n- `end_location` (`Tuple[int, int]`): End position\n- `duration` (`float`): Swipe duration (default: 0.5)\n\n---\n\n### long_press_screen\n\nSimulate long press.\n\n**Parameters**:\n- `location` (`Tuple[int, int]`): Press location\n- `duration` (`float`): Press duration (default: 1.0)\n\n---\n\n### double_tap_screen\n\nSimulate double tap.\n\n**Parameters**:\n- `location` (`Tuple[int, int]`): Tap location\n\n---\n\n### press_key\n\nSimulate keyboard key press via robot arm.\n\n**Parameters**:\n- `key` (`str`): Key to press\n- `modifiers` (`List[str]`): Modifier keys (e.g., `[\"ctrl\", \"shift\"]`)\n- `duration` (`float`): Press duration (default: 0.1)\n\n---\n\n### tap_trackpad / swipe_trackpad\n\nSimulate trackpad interactions.\n\n## Screenshot Tool\n\n### take_screenshot\n\nCapture a screenshot.\n\n**Returns**: `str` - Base64-encoded image data\n\n**Example**:\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::take_screenshot\",\n tool_name=\"take_screenshot\",\n parameters={}\n )\n])\n# result[0].data = \"iVBORw0KGgoAAAANSUhEUgAA...\"\n```\n\n## Configuration\n\n```yaml\n# Client configuration (Windows agent)\nHostAgent:\n default:\n action:\n - namespace: HardwareExecutor\n type: http\n host: \"192.168.1.100\" # Hardware server IP\n port: 8006\n path: \"/mcp\"\n```\n\n## Deployment\n\n### Starting the Server\n\n```bash\n# Start hardware MCP server\npython -m ufo.client.mcp.http_servers.hardware_mcp_server --host 0.0.0.0 --port 8006\n\n# Output:\n# ==================================================\n# UFO Hardware MCP Server\n# Hardware automation via Model Context Protocol\n# Running on 0.0.0.0:8006\n# ==================================================\n```\n\n### Configuration\n\n**Default Values**:\n- Host: `localhost`\n- Port: `8006`\n- Path: `/mcp`\n\n## Best Practices\n\n### 1. Network Configuration\n\n```yaml\n# Use IP address for remote hardware\naction:\n - namespace: HardwareExecutor\n type: http\n host: \"192.168.1.100\" # Hardware server\n port: 8006\n```\n\n### 2. Error Handling\n\nAll tools return dict with `success` key:\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(tool_key=\"action::touch_screen\", parameters={\"location\": [100, 100]})\n])\n\nif not result[0].data.get(\"success\"):\n logger.error(f\"Touch failed: {result[0].data.get('error')}\")\n```\n\n### 3. Physical Hardware Requirements\n\n- Arduino HID: Requires Arduino board with HID firmware\n- BB-8: Microsoft Surface test fixture\n- Robot Arm: Physical robot arm setup\n- Network: Stable network connection for HTTP communication\n\n## Use Cases\n\n### Automated Testing\n\n```python\n# 1. Connect to hardware\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::robot_arm_connect\", parameters={})\n])\n\n# 2. Touch screen at login button\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::touch_screen\", parameters={\"location\": [500, 700]})\n])\n\n# 3. Take screenshot to verify\nscreenshot = await computer.run_actions([\n MCPToolCall(tool_key=\"action::take_screenshot\", parameters={})\n])\n```\n\n## Related Documentation\n\n- [BashExecutor](./bash_executor.md) - Linux command execution\n- [Remote Servers](../remote_servers.md) - HTTP deployment guide\n- [Action Servers](../action.md) - Action server concepts\n"
},
{
"path": "documents/docs/mcp/servers/host_ui_executor.md",
"content": "# HostUIExecutor Server\n\n## Overview\n\n**HostUIExecutor** is an action server that provides system-level UI automation capabilities for the HostAgent. It enables window management, window switching, and cross-application interactions at the desktop level.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** HostAgent \n**LLM-Selectable:** ✅ Yes (LLM chooses when to execute)\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `HostUIExecutor` |\n| **Server Name** | `UFO UI HostAgent Action MCP Server` |\n| **Platform** | Windows |\n| **Backend** | UIAutomation (UIA) or Win32 |\n| **Tool Type** | `action` |\n| **Tool Key Format** | `action::{tool_name}` |\n\n## Tools\n\n### select_application_window\n\nSelect an application window for UI automation and set it as the active window.\n\n#### Description\n\nThis is the primary tool for window selection in HostAgent workflows. It:\n1. Finds the specified window by ID and name\n2. Sets focus on the window\n3. Optionally maximizes the window\n4. Optionally draws a visual outline (for debugging)\n5. Initializes UI state for subsequent AppAgent operations\n\n!!!warning \"Prerequisites\"\n You must call `get_desktop_app_info` (UICollector) first to obtain valid window IDs and names.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `id` | `str` | ✅ Yes | The precise annotated ID of the application window to select. Must match an ID from `get_desktop_app_info` |\n| `name` | `str` | ✅ Yes | The precise name of the application window. Must match the name of the selected ID |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"root_name\": str, # Application root name (e.g., \"WINWORD.EXE\")\n \"window_info\": dict # WindowInfo object with window details\n}\n```\n\n#### WindowInfo Structure\n\n```python\n{\n \"annotation_id\": str, # Window identifier\n \"name\": str, # Window element name\n \"title\": str, # Window title text\n \"handle\": int, # Window handle (HWND)\n \"class_name\": str, # Window class name\n \"process_id\": int, # Process ID\n \"is_visible\": bool, # Visibility status\n \"is_minimized\": bool, # Minimized state\n \"is_maximized\": bool, # Maximized state\n \"is_active\": bool, # Active window status\n \"rectangle\": { # Window bounding rectangle\n \"x\": int,\n \"y\": int,\n \"width\": int,\n \"height\": int\n },\n \"text_content\": str, # Window text\n \"control_type\": str # Control type (usually \"Window\")\n}\n```\n\n#### Example\n\n```python\n# Step 1: Get available windows\nwindows = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_desktop_app_info\",\n tool_name=\"get_desktop_app_info\",\n parameters={\"remove_empty\": True}\n )\n])\n\n# windows[0].data = [\n# {\"id\": \"1\", \"name\": \"Calculator\", \"type\": \"Window\", \"kind\": \"window\"},\n# {\"id\": \"2\", \"name\": \"Notepad\", \"type\": \"Window\", \"kind\": \"window\"}\n# ]\n\n# Step 2: Select Calculator window\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n tool_name=\"select_application_window\",\n parameters={\n \"id\": \"1\",\n \"name\": \"Calculator\"\n }\n )\n])\n\n# Result:\n{\n \"root_name\": \"ApplicationFrameHost.exe\",\n \"window_info\": {\n \"annotation_id\": \"1\",\n \"title\": \"Calculator\",\n \"handle\": 12345678,\n \"class_name\": \"ApplicationFrameWindow\",\n \"process_id\": 9876,\n \"is_visible\": True,\n \"is_minimized\": False,\n \"is_maximized\": False,\n \"is_active\": True,\n \"rectangle\": {\"x\": 100, \"y\": 100, \"width\": 400, \"height\": 600}\n }\n}\n```\n\n#### Error Handling\n\nThe tool raises `ToolError` in the following cases:\n\n```python\n# Error 1: Missing ID\nToolError(\"Window id is required for select_application_window\")\n\n# Error 2: No windows available\nToolError(\"No application windows available. Please call get_desktop_app_info first.\")\n\n# Error 3: Invalid ID\nToolError(\"Control with id '99' not found. Available control ids: ['1', '2', '3']\")\n\n# Error 4: Failed to set focus\nToolError(\"Failed to set focus on window: {error_details}\")\n```\n\n#### Configuration Behavior\n\nThe tool respects these configuration settings:\n\n**MAXIMIZE_WINDOW** (default: `False`)\n```yaml\n# config.yaml\nMAXIMIZE_WINDOW: true # Window is maximized after selection\n```\n\n**SHOW_VISUAL_OUTLINE_ON_SCREEN** (default: `True`)\n```yaml\n# config.yaml\nSHOW_VISUAL_OUTLINE_ON_SCREEN: true # Red outline drawn around window\n```\n\n#### Side Effects\n\n!!!warning \"Side Effects\"\n - ✅ **Changes focus**: Brings target window to foreground\n - ✅ **May maximize**: If `MAXIMIZE_WINDOW` is enabled\n - ✅ **Visual feedback**: Red outline if `SHOW_VISUAL_OUTLINE_ON_SCREEN` is enabled\n - ✅ **State initialization**: Sets up AppPuppeteer for the window\n\n#### Internal State Changes\n\nAfter `select_application_window` executes:\n1. `ui_state.selected_app_window` is set to the window object\n2. `ui_state.puppeteer` is initialized with `AppPuppeteer`\n3. Available commands are logged for debugging\n4. Subsequent UICollector and AppUIExecutor tools can operate on this window\n\n## Configuration\n\n### Basic Configuration\n\n```yaml\nHostAgent:\n default:\n action:\n - namespace: HostUIExecutor\n type: local\n reset: false\n```\n\n### Configuration Options\n\n| Option | Type | Description |\n|--------|------|-------------|\n| `namespace` | `str` | Must be `\"HostUIExecutor\"` |\n| `type` | `str` | Deployment type: `\"local\"` |\n| `reset` | `bool` | Whether to reset server state between tasks (usually `false` for HostUIExecutor) |\n\n## Usage Patterns\n\n### Pattern 1: Basic Window Selection\n\n```python\n# 1. Discover windows\nwindows = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_desktop_app_info\", ...)\n])\n\n# 2. Select target window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"1\", \"name\": \"Calculator\"}\n )\n])\n\n# 3. Now AppAgent can interact with the window\ncontrols = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_app_window_controls_info\", ...)\n])\n```\n\n### Pattern 2: Multi-Window Workflow\n\n```python\n# Work with first window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"1\", \"name\": \"Word\"}\n )\n])\n# ... perform actions on Word ...\n\n# Switch to second window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"2\", \"name\": \"Excel\"}\n )\n])\n# ... perform actions on Excel ...\n```\n\n### Pattern 3: Verify Before Selection\n\n```python\n# Get windows\nwindows = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_desktop_app_info\", ...)\n])\n\n# Verify target window exists\ntarget_windows = [w for w in windows[0].data if \"Calculator\" in w[\"name\"]]\n\nif not target_windows:\n logger.error(\"Calculator not found\")\nelse:\n # Select window\n await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\n \"id\": target_windows[0][\"id\"],\n \"name\": target_windows[0][\"name\"]\n }\n )\n ])\n```\n\n## Best Practices\n\n### 1. Always Validate ID and Name\n\n```python\n# ✅ Good: Use exact ID and name from get_desktop_app_info\nwindows = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_desktop_app_info\", ...)\n])\n\nwindow = windows[0].data[0] # First window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\n \"id\": window[\"id\"], # Exact ID from response\n \"name\": window[\"name\"] # Exact name from response\n }\n )\n])\n\n# ❌ Bad: Hardcode or guess IDs\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"1\", \"name\": \"Some Window\"} # May not exist\n )\n])\n```\n\n### 2. Handle Selection Failures\n\n```python\ntry:\n result = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": window_id, \"name\": window_name}\n )\n ])\n \n if result[0].is_error:\n logger.error(f\"Failed to select window: {result[0].content}\")\n # Retry or select alternative window\n else:\n logger.info(f\"Selected window: {result[0].data['root_name']}\")\n \nexcept Exception as e:\n logger.error(f\"Window selection exception: {e}\")\n```\n\n### 3. Wait After Selection\n\n```python\n# Select window\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::select_application_window\", ...)\n])\n\n# Wait for window to become active\nawait asyncio.sleep(0.5)\n\n# Now interact with window\nawait computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::capture_window_screenshot\", ...)\n])\n```\n\n### 4. Use Visual Outline for Debugging\n\n```yaml\n# config.yaml - Enable during development\nSHOW_VISUAL_OUTLINE_ON_SCREEN: true # See red outline on selected window\n\n# config.yaml - Disable in production\nSHOW_VISUAL_OUTLINE_ON_SCREEN: false\n```\n\n## Integration with AppAgent\n\nAfter `select_application_window` succeeds, the window becomes the target for **AppAgent** operations:\n\n```python\n# HostAgent: Select window\nhost_result = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"1\", \"name\": \"Calculator\"}\n )\n])\n\n# AppAgent: Get controls in selected window\napp_controls = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_app_window_controls_info\", ...)\n])\n\n# AppAgent: Click a button in selected window\napp_click = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_input\",\n tool_name=\"click_input\",\n parameters={\"id\": \"5\", \"name\": \"Seven\", \"button\": \"left\"}\n )\n])\n```\n\n## Troubleshooting\n\n### Window Not Found\n\n**Problem**: `ToolError(\"Control with id 'X' not found\")`\n\n**Solutions**:\n1. Call `get_desktop_app_info` with `refresh_app_windows=True`\n2. Verify window is not minimized or hidden\n3. Check window still exists (hasn't been closed)\n\n### Focus Failed\n\n**Problem**: `ToolError(\"Failed to set focus on window\")`\n\n**Solutions**:\n1. Check window is not disabled or unresponsive\n2. Verify window process is running\n3. Ensure no modal dialogs are blocking focus\n4. Try again after a short delay\n\n### Wrong Window Selected\n\n**Problem**: Selected wrong window with similar name\n\n**Solutions**:\n1. Use more specific name matching\n2. Check `process_id` or `class_name` in window info\n3. Filter windows by additional criteria before selection\n\n## Related Documentation\n\n\n- [UICollector](./ui_collector.md) - Window discovery server\n- [AppUIExecutor](./app_ui_executor.md) - Window interaction server\n- [Action Servers](../action.md) - Action server concepts\n- [HostAgent](../../ufo2/host_agent/overview.md) - HostAgent architecture\n\n"
},
{
"path": "documents/docs/mcp/servers/mobile_executor.md",
"content": "# MobileExecutor Server\n\n## Overview\n\n**MobileExecutor** provides Android mobile device automation via ADB (Android Debug Bridge). It runs as **two separate HTTP servers** that share state for coordinated operations:\n\n- **Mobile Data Collection Server** (port 8020): Screenshots, UI tree, device info, app list, controls\n- **Mobile Action Server** (port 8021): Tap, swipe, type, launch apps, press keys\n\n**Server Type:** Action + Data Collection \n**Deployment:** HTTP (remote server, runs on machine with ADB) \n**Default Ports:** 8020 (data), 8021 (action) \n**LLM-Selectable:** ✅ Yes (action tools only) \n**Platform:** Android devices via ADB\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `MobileDataCollector` (data), `MobileExecutor` (action) |\n| **Server Names** | `Mobile Data Collection MCP Server`, `Mobile Action MCP Server` |\n| **Platform** | Android (via ADB) |\n| **Tool Types** | `data_collection`, `action` |\n| **Deployment** | HTTP server (stateless with shared cache) |\n| **Architecture** | Dual-server with singleton state manager |\n\n## Architecture\n\n### Dual-Server Design\n\nThe mobile MCP server uses a **dual-server architecture** similar to `linux_mcp_server.py`:\n\n```mermaid\ngraph TB\n Agent[\"Windows UFO² Agent\"]\n \n subgraph Process[\"Mobile MCP Servers (Same Process)\"]\n State[\"MobileServerState (Singleton Cache) • Apps cache • Controls cache • UI tree cache • Device info cache\"]\n \n DataServer[\"Data Collection Server Port 8020 • Screenshots • UI tree • Device info • App list • Controls\"]\n \n ActionServer[\"Action Server Port 8021 • Tap/Swipe • Type text • Launch app • Click control\"]\n \n State -.->|Shared Cache| DataServer\n State -.->|Shared Cache| ActionServer\n end\n \n Device[\"Android Device (via ADB)\"]\n \n Agent -->|HTTP| DataServer\n Agent -->|HTTP| ActionServer\n DataServer -->|ADB Commands| Device\n ActionServer -->|ADB Commands| Device\n \n style Agent fill:#e3f2fd,stroke:#1976d2,stroke-width:2px\n style Process fill:#fafafa,stroke:#424242,stroke-width:2px\n style State fill:#fff3e0,stroke:#f57c00,stroke-width:2px\n style DataServer fill:#e8f5e9,stroke:#388e3c,stroke-width:2px\n style ActionServer fill:#fce4ec,stroke:#c2185b,stroke-width:2px\n style Device fill:#f3e5f5,stroke:#7b1fa2,stroke-width:2px\n```\n\n**Shared State Benefits:**\n\n- **Cache Coordination**: Action server can access controls cached by data server\n- **Performance**: Avoid duplicate ADB queries (UI tree, app list, etc.)\n- **State Consistency**: Both servers see same device state\n- **Resource Efficiency**: Single process, shared memory\n\n### State Management\n\n**MobileServerState** is a singleton that caches:\n\n| Cache | Duration | Purpose |\n|-------|----------|---------|\n| **Installed Apps** | 5 minutes | Package list for `get_mobile_app_target_info` |\n| **UI Controls** | 5 seconds | Control list for `get_app_window_controls_target_info` |\n| **UI Tree XML** | 5 seconds | Raw XML for `get_ui_tree` |\n| **Device Info** | 1 minute | Hardware specs for `get_device_info` |\n\n**Cache Invalidation:**\n\n- Automatically invalidated after interactions (tap, swipe, type)\n- Manually invalidated via `invalidate_cache` tool\n- Expired caches refreshed on next query\n\n## Data Collection Tools\n\nData collection tools are automatically invoked by the framework, not selectable by LLM.\n\n### capture_screenshot\n\nCapture screenshot from Android device.\n\n#### Parameters\n\nNone\n\n#### Returns\n\n**Type**: `str`\n\nBase64-encoded image data URI directly (format: `data:image/png;base64,...`)\n\n#### Example\n\n```python\nresult = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::capture_screenshot\",\n tool_name=\"capture_screenshot\",\n parameters={}\n )\n])\n\n# result[0].data = \"data:image/png;base64,iVBORw0KGgo...\"\n```\n\n#### Implementation Details\n\n1. Captures screenshot on device (`screencap -p /sdcard/screen_temp.png`)\n2. Pulls image from device via ADB (`adb pull`)\n3. Encodes as base64\n4. Cleans up temporary files\n5. Returns data URI directly (matches `ui_mcp_server` format)\n\n---\n\n### get_ui_tree\n\nGet the UI hierarchy tree in XML format.\n\n#### Parameters\n\nNone\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"ui_tree\": str, # XML content\n \"format\": \"xml\",\n # OR\n \"error\": str # Error message if failed\n}\n```\n\n#### Example\n\n```python\nresult = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_ui_tree\",\n tool_name=\"get_ui_tree\",\n parameters={}\n )\n])\n\n# Parse XML to find elements\nimport xml.etree.ElementTree as ET\ntree = ET.fromstring(result[0].data[\"ui_tree\"])\n```\n\n#### Cache Behavior\n\n- Cached for 5 seconds\n- Automatically invalidated after interactions\n- Shared with `get_app_window_controls_target_info`\n\n---\n\n### get_device_info\n\nGet comprehensive Android device information.\n\n#### Parameters\n\nNone\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"device_info\": {\n \"model\": str, # Device model\n \"android_version\": str, # Android version (e.g., \"13\")\n \"sdk_version\": str, # SDK version (e.g., \"33\")\n \"screen_size\": str, # Screen resolution (e.g., \"Physical size: 1080x2400\")\n \"screen_density\": str, # Screen density (e.g., \"Physical density: 440\")\n \"battery_level\": str, # Battery percentage\n \"battery_status\": str # Charging status\n },\n \"from_cache\": bool, # True if returned from cache\n # OR\n \"error\": str # Error message if failed\n}\n```\n\n#### Example\n\n```python\nresult = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_device_info\",\n tool_name=\"get_device_info\",\n parameters={}\n )\n])\n\ndevice = result[0].data[\"device_info\"]\nprint(f\"Device: {device['model']}\")\nprint(f\"Android: {device['android_version']}\")\nprint(f\"Battery: {device['battery_level']}%\")\n```\n\n#### Cache Behavior\n\n- Cached for 1 minute\n- Returns `from_cache: true` when using cached data\n\n---\n\n### get_mobile_app_target_info\n\nGet information about installed application packages as `TargetInfo` list.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `filter` | `str` | No | `\"\"` | Filter pattern for package names (e.g., `\"com.android\"`) |\n| `include_system_apps` | `bool` | No | `False` | Whether to include system apps (default: only user apps) |\n| `force_refresh` | `bool` | No | `False` | Force refresh from device, ignoring cache |\n\n#### Returns\n\n**Type**: `List[TargetInfo]`\n\n```python\n[\n TargetInfo(\n kind=TargetKind.THIRD_PARTY_AGENT,\n id=\"1\", # Sequential ID\n name=\"com.example.app\", # Package name (displayed)\n type=\"com.example.app\" # Package name (stored)\n ),\n ...\n]\n```\n\n#### Example\n\n```python\n# Get all user-installed apps\nresult = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_mobile_app_target_info\",\n tool_name=\"get_mobile_app_target_info\",\n parameters={\"include_system_apps\": False}\n )\n])\n\napps = result[0].data\nfor app in apps:\n print(f\"ID: {app.id}, Package: {app.name}\")\n\n# Filter by package name\nresult = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_mobile_app_target_info\",\n tool_name=\"get_mobile_app_target_info\",\n parameters={\"filter\": \"com.android\", \"include_system_apps\": True}\n )\n])\n```\n\n#### Cache Behavior\n\n- Cached for 5 minutes (only when no filter and `include_system_apps=False`)\n- Use `force_refresh=True` to bypass cache\n\n---\n\n### get_app_window_controls_target_info\n\nGet UI controls information as `TargetInfo` list.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `force_refresh` | `bool` | No | `False` | Force refresh from device, ignoring cache |\n\n#### Returns\n\n**Type**: `List[TargetInfo]`\n\n```python\n[\n TargetInfo(\n kind=TargetKind.CONTROL,\n id=\"1\", # Sequential ID\n name=\"Button Name\", # Control text or content-desc\n type=\"Button\", # Control class (short name)\n rect=[x1, y1, x2, y2] # Bounding box [left, top, right, bottom]\n ),\n ...\n]\n```\n\n#### Example\n\n```python\nresult = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n tool_name=\"get_app_window_controls_target_info\",\n parameters={}\n )\n])\n\ncontrols = result[0].data\nfor ctrl in controls:\n print(f\"ID: {ctrl.id}, Name: {ctrl.name}, Type: {ctrl.type}\")\n print(f\" Rect: {ctrl.rect}\")\n```\n\n#### Control Selection Criteria\n\nOnly **meaningful controls** are included:\n\n- Clickable controls (`clickable=\"true\"`)\n- Long-clickable controls (`long-clickable=\"true\"`)\n- Checkable controls (`checkable=\"true\"`)\n- Scrollable controls (`scrollable=\"true\"`)\n- Controls with text or content-desc\n- EditText and Button controls\n\n**Rect format**: `[left, top, right, bottom]` in pixels (matches `ui_mcp_server.py` bbox format)\n\n#### Cache Behavior\n\n- Cached for 5 seconds\n- Automatically invalidated after interactions (tap, swipe, type)\n- Shared with action server for `click_control` and `type_text`\n\n---\n\n## Action Tools\n\nAction tools are LLM-selectable, state-modifying operations.\n\n### tap\n\nTap/click at specified coordinates on the screen.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `x` | `int` | ✅ Yes | X coordinate in pixels (from left) |\n| `y` | `int` | ✅ Yes | Y coordinate in pixels (from top) |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"action\": str, # \"tap(x, y)\"\n \"output\": str, # Command output\n \"error\": str # Error message if failed\n}\n```\n\n#### Example\n\n```python\n# Tap at specific coordinates\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::tap\",\n tool_name=\"tap\",\n parameters={\"x\": 500, \"y\": 1200}\n )\n])\n```\n\n#### Side Effects\n\n- Invalidates controls cache (UI likely changed)\n\n---\n\n### swipe\n\nPerform swipe gesture from start to end coordinates.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `start_x` | `int` | ✅ Yes | - | Starting X coordinate |\n| `start_y` | `int` | ✅ Yes | - | Starting Y coordinate |\n| `end_x` | `int` | ✅ Yes | - | Ending X coordinate |\n| `end_y` | `int` | ✅ Yes | - | Ending Y coordinate |\n| `duration` | `int` | No | `300` | Duration in milliseconds |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"action\": str, # \"swipe(x1,y1)->(x2,y2) in Nms\"\n \"output\": str,\n \"error\": str\n}\n```\n\n#### Example\n\n```python\n# Swipe up (scroll down content)\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::swipe\",\n tool_name=\"swipe\",\n parameters={\n \"start_x\": 500,\n \"start_y\": 1500,\n \"end_x\": 500,\n \"end_y\": 500,\n \"duration\": 300\n }\n )\n])\n\n# Swipe left (next page)\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::swipe\",\n tool_name=\"swipe\",\n parameters={\n \"start_x\": 800,\n \"start_y\": 1000,\n \"end_x\": 200,\n \"end_y\": 1000,\n \"duration\": 200\n }\n )\n])\n```\n\n#### Side Effects\n\n- Invalidates controls cache (UI changed)\n\n---\n\n### type_text\n\nType text into a specific input field control.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `text` | `str` | ✅ Yes | - | Text to input (spaces/special chars auto-escaped) |\n| `control_id` | `str` | ✅ Yes | - | Precise annotated ID from `get_app_window_controls_target_info` |\n| `control_name` | `str` | ✅ Yes | - | Precise name of control (must match `control_id`) |\n| `clear_current_text` | `bool` | No | `False` | Clear existing text before typing |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"action\": str, # Full action description\n \"message\": str, # Step-by-step messages\n \"control_info\": {\n \"id\": str,\n \"name\": str,\n \"type\": str\n },\n # OR\n \"error\": str # Error message\n}\n```\n\n#### Example\n\n```python\n# 1. Get controls first\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n tool_name=\"get_app_window_controls_target_info\",\n parameters={}\n )\n])\n\n# 2. Find search input field\nsearch_field = next(c for c in controls[0].data if \"Search\" in c.name)\n\n# 3. Type text\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::type_text\",\n tool_name=\"type_text\",\n parameters={\n \"text\": \"hello world\",\n \"control_id\": search_field.id,\n \"control_name\": search_field.name,\n \"clear_current_text\": True\n }\n )\n])\n```\n\n#### Workflow\n\n1. Verifies control exists in cache (requires prior `get_app_window_controls_target_info` call)\n2. Clicks control to focus it\n3. Optionally clears existing text (deletes up to 50 characters)\n4. Types text (spaces replaced with `%s`, `&` escaped)\n5. Invalidates controls cache\n\n#### Side Effects\n\n- Clicks the control (may trigger navigation)\n- Modifies input field content\n- Invalidates controls cache\n\n---\n\n### launch_app\n\nLaunch an application by package name or app ID.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `package_name` | `str` | ✅ Yes | - | Package name (e.g., `\"com.android.settings\"`) or app name |\n| `id` | `str` | No | `None` | Optional: Precise annotated ID from `get_mobile_app_target_info` |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"message\": str,\n \"package_name\": str, # Actual package launched\n \"output\": str, # ADB monkey output\n \"error\": str,\n \"warning\": str, # Optional: name resolution warning\n \"app_info\": { # Optional: if id provided\n \"id\": str,\n \"name\": str,\n \"package\": str\n }\n}\n```\n\n#### Example\n\n```python\n# Launch by package name\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::launch_app\",\n tool_name=\"launch_app\",\n parameters={\"package_name\": \"com.android.settings\"}\n )\n])\n\n# Launch by app ID (from cache)\napps = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_mobile_app_target_info\",\n tool_name=\"get_mobile_app_target_info\",\n parameters={}\n )\n])\n\nsettings_app = next(a for a in apps[0].data if \"settings\" in a.name.lower())\n\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::launch_app\",\n tool_name=\"launch_app\",\n parameters={\n \"package_name\": settings_app.type, # Package from cache\n \"id\": settings_app.id\n }\n )\n])\n\n# Launch by app name (auto-resolves package)\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::launch_app\",\n tool_name=\"launch_app\",\n parameters={\"package_name\": \"Settings\"} # Resolves to com.android.settings\n )\n])\n```\n\n#### Name Resolution\n\nIf `package_name` doesn't contain `.` (not a package format):\n\n1. Searches installed packages for matching display name\n2. Returns resolved package with warning\n3. Fails if no match found\n\n#### Implementation\n\nUses `adb shell monkey -p -c android.intent.category.LAUNCHER 1`\n\n---\n\n### press_key\n\nPress a hardware or software key.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `key_code` | `str` | ✅ Yes | Key code (e.g., `\"KEYCODE_HOME\"`, `\"KEYCODE_BACK\"`) |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"action\": str, # \"press_key(KEYCODE_X)\"\n \"output\": str,\n \"error\": str\n}\n```\n\n#### Example\n\n```python\n# Press back button\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::press_key\",\n tool_name=\"press_key\",\n parameters={\"key_code\": \"KEYCODE_BACK\"}\n )\n])\n\n# Press home button\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::press_key\",\n tool_name=\"press_key\",\n parameters={\"key_code\": \"KEYCODE_HOME\"}\n )\n])\n\n# Press enter\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::press_key\",\n tool_name=\"press_key\",\n parameters={\"key_code\": \"KEYCODE_ENTER\"}\n )\n])\n```\n\n#### Common Key Codes\n\n| Key Code | Description |\n|----------|-------------|\n| `KEYCODE_HOME` | Home button |\n| `KEYCODE_BACK` | Back button |\n| `KEYCODE_ENTER` | Enter/Return |\n| `KEYCODE_MENU` | Menu button |\n| `KEYCODE_POWER` | Power button |\n| `KEYCODE_VOLUME_UP` | Volume up |\n| `KEYCODE_VOLUME_DOWN` | Volume down |\n\nFull list: [Android KeyEvent](https://developer.android.com/reference/android/view/KeyEvent)\n\n---\n\n### click_control\n\nClick a UI control by its ID and name.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `control_id` | `str` | ✅ Yes | Precise annotated ID from `get_app_window_controls_target_info` |\n| `control_name` | `str` | ✅ Yes | Precise name of control (must match `control_id`) |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"action\": str, # Full action description\n \"message\": str, # Success message with coordinates\n \"control_info\": {\n \"id\": str,\n \"name\": str,\n \"type\": str,\n \"rect\": [int, int, int, int]\n },\n \"warning\": str, # Optional: name mismatch warning\n # OR\n \"error\": str # Error message\n}\n```\n\n#### Example\n\n```python\n# 1. Get controls\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n tool_name=\"get_app_window_controls_target_info\",\n parameters={}\n )\n])\n\n# 2. Find OK button\nok_button = next(c for c in controls[0].data if c.name == \"OK\")\n\n# 3. Click it\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_control\",\n tool_name=\"click_control\",\n parameters={\n \"control_id\": ok_button.id,\n \"control_name\": ok_button.name\n }\n )\n])\n```\n\n#### Workflow\n\n1. Retrieves control from cache by `control_id`\n2. Verifies name matches (warns if different)\n3. Calculates center position from bounding box\n4. Taps at center coordinates\n5. Invalidates controls cache\n\n#### Side Effects\n\n- Taps the control (may trigger navigation)\n- Invalidates controls cache\n\n---\n\n### wait\n\nWait for a specified number of seconds.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `seconds` | `float` | No | `1.0` | Number of seconds to wait (0-60 range) |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"action\": str, # \"wait(Ns)\"\n \"message\": str, # \"Waited for N seconds\"\n # OR\n \"error\": str # Error if invalid seconds\n}\n```\n\n#### Example\n\n```python\n# Wait 1 second\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::wait\",\n tool_name=\"wait\",\n parameters={\"seconds\": 1.0}\n )\n])\n\n# Wait 500ms\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::wait\",\n tool_name=\"wait\",\n parameters={\"seconds\": 0.5}\n )\n])\n\n# Wait 2.5 seconds\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::wait\",\n tool_name=\"wait\",\n parameters={\"seconds\": 2.5}\n )\n])\n```\n\n#### Constraints\n\n- Minimum: 0 seconds\n- Maximum: 60 seconds\n- Use for UI transitions, animations, app loading\n\n---\n\n### invalidate_cache\n\nManually invalidate cached data to force refresh on next query.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `cache_type` | `str` | No | `\"all\"` | Type of cache: `\"controls\"`, `\"apps\"`, `\"ui_tree\"`, `\"device_info\"`, `\"all\"` |\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\n```python\n{\n \"success\": bool,\n \"message\": str, # Confirmation message\n # OR\n \"error\": str # Invalid cache_type\n}\n```\n\n#### Example\n\n```python\n# Invalidate all caches\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::invalidate_cache\",\n tool_name=\"invalidate_cache\",\n parameters={\"cache_type\": \"all\"}\n )\n])\n\n# Invalidate only controls cache\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::invalidate_cache\",\n tool_name=\"invalidate_cache\",\n parameters={\"cache_type\": \"controls\"}\n )\n])\n```\n\n#### Cache Types\n\n| Type | Description |\n|------|-------------|\n| `\"controls\"` | UI controls list |\n| `\"apps\"` | Installed apps list |\n| `\"ui_tree\"` | UI hierarchy XML |\n| `\"device_info\"` | Device information |\n| `\"all\"` | All caches |\n\n#### Use Cases\n\n- After manual device interaction (outside automation)\n- After app installation/uninstallation\n- When device state significantly changed\n- Before critical operations requiring fresh data\n\n---\n\n## Configuration\n\n### Client Configuration (UFO² Agent)\n\n```yaml\n# Windows agent controlling Android device\nMobileAgent:\n default:\n data_collection:\n - namespace: MobileDataCollector\n type: http\n host: \"localhost\" # Or remote machine IP\n port: 8020\n path: \"/mcp\"\n action:\n - namespace: MobileExecutor\n type: http\n host: \"localhost\"\n port: 8021\n path: \"/mcp\"\n\n# Remote Android device\nMobileAgent:\n default:\n data_collection:\n - namespace: MobileDataCollector\n type: http\n host: \"192.168.1.150\" # Android automation server\n port: 8020\n path: \"/mcp\"\n action:\n - namespace: MobileExecutor\n type: http\n host: \"192.168.1.150\"\n port: 8021\n path: \"/mcp\"\n```\n\n## Deployment\n\n### Prerequisites\n\n1. **ADB Installation**\n\n```bash\n# Windows (via Android SDK or standalone)\n# Download from: https://developer.android.com/studio/releases/platform-tools\n\n# Linux\nsudo apt-get install android-tools-adb\n\n# macOS\nbrew install android-platform-tools\n```\n\n2. **Android Device Setup**\n\n- Enable USB debugging in Developer Options\n- Connect device via USB or Wi-Fi\n- Verify connection: `adb devices`\n\n```bash\n# Check connected devices\nadb devices\n\n# Output:\n# List of devices attached\n# R5CR20XXXXX device\n```\n\n### Starting the Servers\n\n```bash\n# Start both servers (recommended)\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server both --host 0.0.0.0 --data-port 8020 --action-port 8021\n\n# Output:\n# ==================================================\n# UFO Mobile MCP Servers (Android)\n# Android device control via ADB and Model Context Protocol\n# ==================================================\n# Using ADB: C:\\...\\adb.exe\n# Found 1 connected device(s)\n# ✅ Starting both servers in same process (shared MobileServerState)\n# - Data Collection Server: 0.0.0.0:8020\n# - Action Server: 0.0.0.0:8021\n# Both servers share MobileServerState cache. Press Ctrl+C to stop.\n\n# Start only data collection server\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server data --host 0.0.0.0 --data-port 8020\n\n# Start only action server\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server action --host 0.0.0.0 --action-port 8021\n```\n\n### Command-Line Arguments\n\n| Argument | Default | Description |\n|----------|---------|-------------|\n| `--server` | `both` | Which server(s): `data`, `action`, or `both` |\n| `--host` | `localhost` | Host to bind servers to |\n| `--data-port` | `8020` | Port for Data Collection Server |\n| `--action-port` | `8021` | Port for Action Server |\n| `--adb-path` | Auto-detect | Path to ADB executable |\n\n### ADB Path Detection\n\nThe server auto-detects ADB from:\n\n1. Common installation paths:\n - Windows: `C:\\Users\\{USER}\\AppData\\Local\\Android\\Sdk\\platform-tools\\adb.exe`\n - Linux: `/usr/bin/adb`, `/usr/local/bin/adb`\n2. System PATH environment variable\n3. Fallback to `adb` command\n\nOverride with `--adb-path`:\n\n```bash\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --adb-path \"C:\\custom\\path\\adb.exe\"\n```\n\n### Network Configuration\n\n**Local Development:**\n```bash\n# Servers on same machine as client\n--host localhost\n```\n\n**Remote Access:**\n```bash\n# Servers accessible from network\n--host 0.0.0.0\n```\n\n**Security:** Use firewall rules to restrict access to trusted IPs.\n\n---\n\n## Best Practices\n\n### 1. Always Run Both Servers Together\n\n```bash\n# ✅ Good: Both servers in same process (shared state)\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server both\n\n# ❌ Bad: Separate processes (no shared state)\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server data &\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --server action &\n```\n\n**Why:** Shared `MobileServerState` enables action server to access controls cached by data server.\n\n### 2. Get Controls Before Interaction\n\n```python\n# ✅ Good: Get controls first\ncontrols = await computer.run_data_collection([\n MCPToolCall(tool_key=\"data_collection::get_app_window_controls_target_info\", ...)\n])\n\n# Then click/type\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::click_control\", parameters={\"control_id\": \"5\", ...})\n])\n\n# ❌ Bad: Click without getting controls\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::click_control\", parameters={\"control_id\": \"5\", ...})\n])\n# Error: Control not found in cache\n```\n\n### 3. Use Control IDs, Not Coordinates\n\n```python\n# ✅ Good: Use click_control (reliable)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_control\",\n parameters={\"control_id\": \"3\", \"control_name\": \"Submit\"}\n )\n])\n\n# ⚠️ OK: Use tap only when control not available\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::tap\",\n parameters={\"x\": 500, \"y\": 1200}\n )\n])\n```\n\n### 4. Handle Cache Expiration\n\n```python\n# Check if controls are stale\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={\"force_refresh\": False} # Use cache if available\n )\n])\n\n# For critical operations, force refresh\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={\"force_refresh\": True} # Always query device\n )\n])\n```\n\n### 5. Wait After Actions\n\n```python\n# ✅ Good: Wait for UI to settle\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::tap\", parameters={\"x\": 500, \"y\": 1200})\n])\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::wait\", parameters={\"seconds\": 1.0})\n])\n\n# Get updated controls\ncontrols = await computer.run_data_collection([\n MCPToolCall(tool_key=\"data_collection::get_app_window_controls_target_info\", ...)\n])\n```\n\n### 6. Validate ADB Connection\n\n```python\n# Check device info before operations\ndevice_info = await computer.run_data_collection([\n MCPToolCall(tool_key=\"data_collection::get_device_info\", parameters={})\n])\n\nif device_info[0].is_error:\n raise RuntimeError(\"No Android device connected\")\n```\n\n---\n\n## Use Cases\n\n### 1. App Automation\n\n```python\n# Launch app\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::launch_app\",\n tool_name=\"launch_app\",\n parameters={\"package_name\": \"com.example.app\"}\n )\n])\n\n# Wait for app to load\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::wait\", parameters={\"seconds\": 2.0})\n])\n\n# Get controls\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={}\n )\n])\n\n# Find and click button\nlogin_btn = next(c for c in controls[0].data if \"Login\" in c.name)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_control\",\n parameters={\n \"control_id\": login_btn.id,\n \"control_name\": login_btn.name\n }\n )\n])\n```\n\n### 2. Form Filling\n\n```python\n# Get controls\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={}\n )\n])\n\n# Type username\nusername_field = next(c for c in controls[0].data if \"username\" in c.name.lower())\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::type_text\",\n tool_name=\"type_text\",\n parameters={\n \"text\": \"john.doe@example.com\",\n \"control_id\": username_field.id,\n \"control_name\": username_field.name,\n \"clear_current_text\": True\n }\n )\n])\n\n# Get updated controls (after typing)\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::wait\", parameters={\"seconds\": 0.5})\n])\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={\"force_refresh\": True}\n )\n])\n\n# Type password\npassword_field = next(c for c in controls[0].data if \"password\" in c.name.lower())\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::type_text\",\n parameters={\n \"text\": \"SecureP@ssw0rd\",\n \"control_id\": password_field.id,\n \"control_name\": password_field.name\n }\n )\n])\n\n# Submit\nsubmit_btn = next(c for c in controls[0].data if \"Submit\" in c.name)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::click_control\",\n parameters={\n \"control_id\": submit_btn.id,\n \"control_name\": submit_btn.name\n }\n )\n])\n```\n\n### 3. Scrolling and Navigation\n\n```python\n# Swipe up to scroll down content\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::swipe\",\n tool_name=\"swipe\",\n parameters={\n \"start_x\": 500,\n \"start_y\": 1500,\n \"end_x\": 500,\n \"end_y\": 500,\n \"duration\": 300\n }\n )\n])\n\n# Wait for scrolling to complete\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::wait\", parameters={\"seconds\": 0.5})\n])\n\n# Get updated controls\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={\"force_refresh\": True}\n )\n])\n```\n\n### 4. Device Testing\n\n```python\n# Get device info\ndevice_info = await computer.run_data_collection([\n MCPToolCall(tool_key=\"data_collection::get_device_info\", parameters={})\n])\n\nprint(f\"Testing on: {device_info[0].data['device_info']['model']}\")\nprint(f\"Android: {device_info[0].data['device_info']['android_version']}\")\n\n# Take screenshot before test\nscreenshot_before = await computer.run_data_collection([\n MCPToolCall(tool_key=\"data_collection::capture_screenshot\", parameters={})\n])\n\n# Perform test actions\n# ...\n\n# Take screenshot after test\nscreenshot_after = await computer.run_data_collection([\n MCPToolCall(tool_key=\"data_collection::capture_screenshot\", parameters={})\n])\n\n# Compare screenshots (external comparison logic)\n```\n\n---\n\n## Comparison with Other Servers\n\n| Feature | MobileExecutor | HardwareExecutor (Robot Arm) | AppUIExecutor (Windows) |\n|---------|----------------|------------------------------|-------------------------|\n| **Platform** | Android (ADB) | Cross-platform (Hardware) | Windows (UIA) |\n| **Controls** | ✅ XML-based | ❌ Coordinate-based | ✅ UIA-based |\n| **Screenshots** | ✅ ADB screencap | ✅ Hardware camera | ✅ Windows API |\n| **Deployment** | HTTP (dual-server) | HTTP (single-server) | Local (in-process) |\n| **State Management** | ✅ Shared singleton | ❌ Stateless | ❌ No caching |\n| **App Launch** | ✅ Package manager | ❌ Manual | ✅ Process spawn |\n| **Text Input** | ✅ ADB input | ✅ HID keyboard | ✅ UIA SetValue |\n| **Cache** | ✅ 5s-5min TTL | ❌ No cache | ❌ No cache |\n\n---\n\n## Troubleshooting\n\n### ADB Connection Issues\n\n```bash\n# Restart ADB server\nadb kill-server\nadb start-server\n\n# Check device connection\nadb devices\n\n# If no devices shown:\n# 1. Check USB cable\n# 2. Verify USB debugging enabled on device\n# 3. Accept \"Allow USB debugging\" prompt on device\n```\n\n### Server Not Starting\n\n```bash\n# Check if ports are in use\nnetstat -an | findstr \"8020\"\nnetstat -an | findstr \"8021\"\n\n# Change ports if needed\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --data-port 8030 --action-port 8031\n```\n\n### Controls Not Found\n\n```python\n# Force refresh cache\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={\"force_refresh\": True}\n )\n])\n\n# Or invalidate cache manually\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::invalidate_cache\",\n parameters={\"cache_type\": \"controls\"}\n )\n])\n```\n\n### Text Input Fails\n\n```python\n# Ensure control is in cache\ncontrols = await computer.run_data_collection([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_target_info\",\n parameters={}\n )\n])\n\n# Verify control ID and name match\nfield = next(c for c in controls[0].data if c.id == \"5\")\nprint(f\"Control name: {field.name}\")\n\n# Use exact ID and name\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::type_text\",\n parameters={\n \"text\": \"test\",\n \"control_id\": field.id,\n \"control_name\": field.name\n }\n )\n])\n```\n\n---\n\n## Related Documentation\n\n- [HardwareExecutor](./hardware_executor.md) - Hardware control (robot arm, mobile devices)\n- [BashExecutor](./bash_executor.md) - Linux command execution\n- [AppUIExecutor](./app_ui_executor.md) - Windows UI automation\n- [Remote Servers](../remote_servers.md) - HTTP deployment guide\n- [Action Servers](../action.md) - Action server concepts\n- [Data Collection Servers](../data_collection.md) - Data collection overview\n"
},
{
"path": "documents/docs/mcp/servers/pdf_reader_executor.md",
"content": "# PDFReaderExecutor Server\n\n## Overview\n\n**PDFReaderExecutor** provides PDF text extraction with optional human simulation capabilities.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** AppAgent, HostAgent \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `PDFReaderExecutor` |\n| **Server Name** | `UFO PDF Reader MCP Server` |\n| **Platform** | Cross-platform (Windows, Linux, macOS) |\n| **Dependencies** | PyPDF2 |\n| **Tool Type** | `action` |\n\n## Tools\n\n### extract_pdf_text\n\nExtract text content from a single PDF file with optional human simulation.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `pdf_path` | `str` | ✅ Yes | - | Full path to PDF file |\n| `simulate_human` | `bool` | No | `True` | Simulate human-like document review |\n\n#### Returns\n\n`str` - Extracted text content with page markers\n\n#### Human Simulation Behavior\n\nWhen `simulate_human=True`:\n1. Opens PDF with default application\n2. Waits 2-5 seconds (random) to simulate reading\n3. Extracts text with page-by-page delays (0.5-1.5 seconds)\n4. Closes PDF file\n\nWhen `simulate_human=False`:\n- Direct text extraction (no delays)\n- No application launching\n\n#### Example\n\n```python\n# With human simulation (default)\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_pdf_text\",\n tool_name=\"extract_pdf_text\",\n parameters={\n \"pdf_path\": \"C:\\\\Documents\\\\report.pdf\",\n \"simulate_human\": True\n }\n )\n])\n\n# Fast extraction (no simulation)\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_pdf_text\",\n tool_name=\"extract_pdf_text\",\n parameters={\n \"pdf_path\": \"C:\\\\Documents\\\\report.pdf\",\n \"simulate_human\": False\n }\n )\n])\n```\n\n#### Output Format\n\n```\n--- Page 1 ---\nThis is the content of page 1.\n\n--- Page 2 ---\nThis is the content of page 2.\n\n--- Page 3 ---\nThis is the content of page 3.\n```\n\n#### Error Handling\n\nReturns error message string if:\n- File not found: `\"Error: PDF file not found at {path}\"`\n- Not a PDF: `\"Error: File {path} is not a PDF file\"`\n- Read error: `\"Error reading PDF {path}: {details}\"`\n\n---\n\n### list_pdfs_in_directory\n\nList all PDF files in a specified directory.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `directory_path` | `str` | ✅ Yes | Directory path to scan |\n\n#### Returns\n\n`List[str]` - List of PDF file paths (sorted)\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::list_pdfs_in_directory\",\n tool_name=\"list_pdfs_in_directory\",\n parameters={\"directory_path\": \"C:\\\\Documents\\\\Reports\"}\n )\n])\n\n# Output: [\n# \"C:\\\\Documents\\\\Reports\\\\Q1_Report.pdf\",\n# \"C:\\\\Documents\\\\Reports\\\\Q2_Report.pdf\",\n# \"C:\\\\Documents\\\\Reports\\\\Q3_Report.pdf\"\n# ]\n```\n\n#### Error Handling\n\nReturns empty list `[]` if:\n- Directory doesn't exist\n- Path is not a directory\n- No PDF files found\n\n---\n\n### extract_all_pdfs_text\n\nExtract text from all PDF files in a directory with human simulation.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `directory_path` | `str` | ✅ Yes | - | Directory containing PDFs |\n| `simulate_human` | `bool` | No | `True` | Simulate human review for each PDF |\n\n#### Returns\n\n`Dict[str, str]` - Dictionary mapping filenames to extracted text\n\n#### Human Simulation Behavior\n\nWhen `simulate_human=True`:\n- Brief pause between files (1-3 seconds random)\n- Each PDF processed with human simulation\n- Progress messages logged\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_all_pdfs_text\",\n tool_name=\"extract_all_pdfs_text\",\n parameters={\n \"directory_path\": \"C:\\\\Documents\\\\Reports\",\n \"simulate_human\": True\n }\n )\n])\n\n# Output: {\n# \"Q1_Report.pdf\": \"--- Page 1 ---\\nQ1 Sales Report\\n...\",\n# \"Q2_Report.pdf\": \"--- Page 1 ---\\nQ2 Sales Report\\n...\",\n# \"Q3_Report.pdf\": \"--- Page 1 ---\\nQ3 Sales Report\\n...\"\n# }\n```\n\n#### Error Handling\n\nReturns dictionary with error key if:\n- Directory not found: `{\"error\": \"Directory not found: {path}\"}`\n- Not a directory: `{\"error\": \"Path is not a directory: {path}\"}`\n- No PDFs found: `{\"message\": \"No PDF files found in directory: {path}\"}`\n\n## Configuration\n\n```yaml\nAppAgent:\n default:\n action:\n - namespace: PDFReaderExecutor\n type: local\n\nHostAgent:\n default:\n action:\n - namespace: PDFReaderExecutor\n type: local\n```\n\n## Best Practices\n\n### 1. Disable Simulation for Batch Processing\n\n```python\n# ✅ Good: Fast batch processing\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_all_pdfs_text\",\n tool_name=\"extract_all_pdfs_text\",\n parameters={\n \"directory_path\": \"C:\\\\Documents\",\n \"simulate_human\": False # Faster\n }\n )\n])\n\n# ❌ Bad: Slow with simulation\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_all_pdfs_text\",\n tool_name=\"extract_all_pdfs_text\",\n parameters={\n \"directory_path\": \"C:\\\\Documents\",\n \"simulate_human\": True # 2-5 seconds per file\n }\n )\n])\n```\n\n### 2. Verify Files Exist\n\n```python\n# List PDFs first\npdf_list = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::list_pdfs_in_directory\",\n parameters={\"directory_path\": \"C:\\\\Documents\"}\n )\n])\n\nif pdf_list[0].data:\n # Extract from first PDF\n text = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_pdf_text\",\n parameters={\n \"pdf_path\": pdf_list[0].data[0],\n \"simulate_human\": False\n }\n )\n ])\nelse:\n logger.warning(\"No PDF files found\")\n```\n\n### 3. Handle Large Documents\n\n```python\n# Extract text\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_pdf_text\",\n parameters={\"pdf_path\": \"large_document.pdf\", \"simulate_human\": False}\n )\n])\n\ntext = result[0].data\n\n# Process in chunks if needed\nif len(text) > 100000: # Large document\n chunks = [text[i:i+50000] for i in range(0, len(text), 50000)]\n for chunk in chunks:\n process_chunk(chunk)\n```\n\n## Use Cases\n\n### Document Analysis Pipeline\n\n```python\n# 1. List all PDFs\npdfs = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::list_pdfs_in_directory\",\n parameters={\"directory_path\": \"C:\\\\Contracts\"}\n )\n])\n\n# 2. Extract text from each\nfor pdf_path in pdfs[0].data:\n text = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_pdf_text\",\n parameters={\"pdf_path\": pdf_path, \"simulate_human\": False}\n )\n ])\n \n # 3. Analyze text\n analyze_contract(text[0].data)\n```\n\n### Batch Report Processing\n\n```python\n# Extract all reports at once\nreports = await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::extract_all_pdfs_text\",\n tool_name=\"extract_all_pdfs_text\",\n parameters={\n \"directory_path\": \"C:\\\\Reports\\\\2024\",\n \"simulate_human\": False\n }\n )\n])\n\n# Process all reports\nfor filename, content in reports[0].data.items():\n logger.info(f\"Processing {filename}\")\n # Extract data from content\n data = extract_report_data(content)\n```\n\n## Limitations\n\n- **Text-only**: Cannot extract images or formatting\n- **OCR not supported**: Scanned PDFs with no text layer will return empty\n- **Table parsing**: Complex tables may not preserve structure\n- **No modification**: Read-only operations (cannot edit PDFs)\n\n## Performance\n\n| Operation | simulate_human=True | simulate_human=False |\n|-----------|---------------------|----------------------|\n| Single PDF (10 pages) | ~10-20 seconds | ~1 second |\n| Batch 10 PDFs | ~2-3 minutes | ~10 seconds |\n| Large PDF (100 pages) | ~2-5 minutes | ~5-10 seconds |\n\n## Related Documentation\n\n- [Action Servers](../action.md) - Action server concepts\n- [Local Servers](../local_servers.md) - Local deployment\n"
},
{
"path": "documents/docs/mcp/servers/ppt_com_executor.md",
"content": "# PowerPointCOMExecutor Server\n\n## Overview\n\n**PowerPointCOMExecutor** provides Microsoft PowerPoint automation via COM API for efficient presentation manipulation.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** AppAgent \n**Target Application:** Microsoft PowerPoint (`POWERPNT.EXE`) \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `PowerPointCOMExecutor` |\n| **Platform** | Windows |\n| **Requires** | Microsoft PowerPoint (COM interface) |\n| **Tool Type** | `action` |\n\n## Tools\n\n### set_background_color\n\nSet the background color for one or more slides in the presentation.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `color` | `str` | ✅ Yes | - | Hex color code (RGB format, e.g., `\"FFFFFF\"`) |\n| `slide_index` | `List[int]` | No | `None` | List of slide indices (1-based). `None` = all slides |\n\n#### Returns\n\n`str` - Success/failure message\n\n#### Example\n\n```python\n# Set white background for slide 1\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n tool_name=\"set_background_color\",\n parameters={\n \"color\": \"FFFFFF\",\n \"slide_index\": [1]\n }\n )\n])\n\n# Set blue background for slides 1, 3, 5\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n tool_name=\"set_background_color\",\n parameters={\n \"color\": \"0000FF\",\n \"slide_index\": [1, 3, 5]\n }\n )\n])\n\n# Set red background for ALL slides\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n tool_name=\"set_background_color\",\n parameters={\n \"color\": \"FF0000\",\n \"slide_index\": None # All slides\n }\n )\n])\n```\n\n#### Color Format\n\nUse 6-character hex RGB codes (without `#`):\n\n| Color | Hex Code |\n|-------|----------|\n| White | `FFFFFF` |\n| Black | `000000` |\n| Red | `FF0000` |\n| Green | `00FF00` |\n| Blue | `0000FF` |\n| Yellow | `FFFF00` |\n| Gray | `808080` |\n\n---\n\n### save_as\n\nSave or export PowerPoint presentation to specified format.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `file_dir` | `str` | No | `\"\"` | Directory path |\n| `file_name` | `str` | No | `\"\"` | Filename without extension |\n| `file_ext` | `str` | No | `\"\"` | Extension (default: `.pptx`) |\n| `current_slide_only` | `bool` | No | `False` | For image formats: save only current slide or all slides |\n\n#### Supported Extensions\n\n**Presentation Formats**:\n- `.pptx` - PowerPoint presentation (default)\n- `.ppt` - PowerPoint 97-2003\n- `.pdf` - PDF format\n\n**Image Formats** (controlled by `current_slide_only`):\n- `.jpg`, `.jpeg` - JPEG image\n- `.png` - PNG image\n- `.gif` - GIF image\n- `.bmp` - Bitmap image\n- `.tiff` - TIFF image\n\n#### Returns\n\n`str` - Success/failure message\n\n#### Example\n\n```python\n# Save as PPTX\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"C:\\\\Presentations\",\n \"file_name\": \"Q4_Report\",\n \"file_ext\": \".pptx\"\n }\n )\n])\n\n# Export as PDF\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_ext\": \".pdf\"\n }\n )\n])\n\n# Save current slide as PNG\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_name\": \"slide_1\",\n \"file_ext\": \".png\",\n \"current_slide_only\": True\n }\n )\n])\n\n# Export all slides as PNG images (creates directory)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"C:\\\\Exports\\\\Slides\",\n \"file_ext\": \".png\",\n \"current_slide_only\": False # Saves all slides\n }\n )\n])\n```\n\n#### Image Export Behavior\n\n| `current_slide_only` | Behavior |\n|----------------------|----------|\n| `True` | Single image file of current slide |\n| `False` | Directory containing multiple image files (one per slide) |\n\n## Configuration\n\n```yaml\nAppAgent:\n POWERPNT.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: PowerPointCOMExecutor\n type: local\n reset: true # Recommended\n```\n\n## Best Practices\n\n### 1. Bulk Background Setting\n\n```python\n# ✅ Good: Set multiple slides at once\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n parameters={\"color\": \"FFFFFF\", \"slide_index\": [1, 2, 3, 4, 5]}\n )\n])\n\n# ❌ Bad: One call per slide\nfor i in range(1, 6):\n await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n parameters={\"color\": \"FFFFFF\", \"slide_index\": [i]}\n )\n ])\n```\n\n### 2. Use save_as for Exports\n\n```python\n# ✅ Good: Fast one-command export\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n parameters={\"file_ext\": \".pdf\"}\n )\n])\n\n# ❌ Bad: Manual UI navigation\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::keyboard_input\", parameters={\"keys\": \"{VK_MENU}f\"}) # Alt+F\n])\n# ... navigate File menu ...\n```\n\n### 3. Verify Hex Colors\n\n```python\ndef validate_hex_color(color: str) -> bool:\n \"\"\"Validate hex color format\"\"\"\n return bool(re.match(r'^[0-9A-Fa-f]{6}$', color))\n\ncolor = \"FFFFFF\"\nif validate_hex_color(color):\n await computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n parameters={\"color\": color, \"slide_index\": [1]}\n )\n ])\n```\n\n## Use Cases\n\n### Presentation Branding\n\n```python\n# Apply company color scheme\nbrand_color = \"003366\" # Company blue\n\n# Set all slides to brand background\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_background_color\",\n tool_name=\"set_background_color\",\n parameters={\n \"color\": brand_color,\n \"slide_index\": None # All slides\n }\n )\n])\n\n# Save as PDF for distribution\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"C:\\\\Distribution\",\n \"file_name\": \"Company_Presentation\",\n \"file_ext\": \".pdf\"\n }\n )\n])\n```\n\n### Slide Export for Documentation\n\n```python\n# Export each slide as PNG for documentation\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"C:\\\\Docs\\\\Images\",\n \"file_name\": \"presentation_slides\",\n \"file_ext\": \".png\",\n \"current_slide_only\": False # Export all\n }\n )\n])\n```\n\n## Limitations\n\n- **Limited tool set**: Only 2 tools (background color and save)\n- **No content creation**: Cannot add text, shapes, or images via COM (use UI automation)\n- **No slide management**: Cannot add/delete/reorder slides (use UI automation)\n\n**Tip:** Combine with **AppUIExecutor** for full PowerPoint automation:\n- **PowerPointCOMExecutor**: Background colors, export\n- **AppUIExecutor**: Add slides, insert text, shapes, animations\n\n## Related Documentation\n\n- [WordCOMExecutor](./word_com_executor.md) - Word COM automation\n- [ExcelCOMExecutor](./excel_com_executor.md) - Excel COM automation\n- [AppUIExecutor](./app_ui_executor.md) - UI-based PowerPoint automation\n"
},
{
"path": "documents/docs/mcp/servers/ui_collector.md",
"content": "# UICollector Server\n\n## Overview\n\n**UICollector** is a data collection MCP server that provides comprehensive UI observation and information retrieval capabilities for the UFO² framework. It automatically gathers screenshots, window lists, control information, and UI trees to build the observation context for LLM decision-making.\n\n**Server Type:** Data Collection \n**Deployment:** Local (in-process) \n**Agent:** HostAgent, AppAgent \n**LLM-Selectable:** ❌ No (automatically invoked by framework)\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `UICollector` |\n| **Server Name** | `UFO UI Data MCP Server` |\n| **Platform** | Windows |\n| **Backend** | UIAutomation (UIA) or Win32 |\n| **Tool Type** | `data_collection` |\n| **Tool Key Format** | `data_collection::{tool_name}` |\n\n## Tools\n\n### 1. get_desktop_app_info\n\nGet information about all application windows currently open on the desktop.\n\n#### Description\n\nRetrieves a list of all visible application windows on the Windows desktop, including window names, types, and identifiers. This is typically the first step in UI automation workflows to discover available applications.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `remove_empty` | `bool` | No | `True` | Whether to remove windows with no visible content |\n| `refresh_app_windows` | `bool` | No | `True` | Whether to refresh the list of application windows |\n\n#### Returns\n\n**Type**: `List[Dict[str, Any]]`\n\nList of window information dictionaries, each containing:\n\n```python\n{\n \"id\": str, # Unique window identifier (e.g., \"1\", \"2\", \"3\")\n \"name\": str, # Window title/text\n \"type\": str, # Control type (e.g., \"Window\", \"Pane\")\n \"kind\": str # Target kind: \"window\"\n}\n```\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_desktop_app_info\",\n tool_name=\"get_desktop_app_info\",\n parameters={\n \"remove_empty\": True,\n \"refresh_app_windows\": True\n }\n )\n])\n\n# Example output:\n[\n {\n \"id\": \"1\",\n \"name\": \"Visual Studio Code\",\n \"type\": \"Window\",\n \"kind\": \"window\"\n },\n {\n \"id\": \"2\",\n \"name\": \"Microsoft Edge\",\n \"type\": \"Window\",\n \"kind\": \"window\"\n }\n]\n```\n\n---\n\n### 2. get_desktop_app_target_info\n\nGet comprehensive target information for all desktop application windows.\n\n#### Description\n\nSimilar to `get_desktop_app_info`, but returns `TargetInfo` objects instead of plain dictionaries. This provides a more structured representation of window information for internal framework use.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `remove_empty` | `bool` | No | `True` | Whether to remove windows with no visible content |\n| `refresh_app_windows` | `bool` | No | `True` | Whether to refresh the list of application windows |\n\n#### Returns\n\n**Type**: `List[TargetInfo]`\n\nList of `TargetInfo` objects with properties:\n- `id`: Unique identifier\n- `name`: Window title\n- `type`: Control type\n- `kind`: TargetKind.WINDOW\n\n---\n\n### 3. get_app_window_info\n\nGet detailed information about the currently selected application window.\n\n#### Description\n\nRetrieves specific fields of information for the active/selected window. You must select a window using `select_application_window` (HostUIExecutor) before calling this tool.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `field_list` | `List[str]` | Yes | - | List of field names to retrieve |\n\n#### Supported Fields\n\nCommon fields include:\n- `\"control_text\"`: Window title/text\n- `\"control_type\"`: Control type (e.g., \"Window\")\n- `\"control_rect\"`: Bounding rectangle coordinates\n- `\"process_id\"`: Process ID\n- `\"class_name\"`: Window class name\n- `\"is_visible\"`: Visibility status\n- `\"is_enabled\"`: Enabled status\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\nDictionary mapping field names to their values.\n\n#### Example\n\n```python\n# First select a window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"1\", \"name\": \"Calculator\"}\n )\n])\n\n# Then get window info\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_info\",\n tool_name=\"get_app_window_info\",\n parameters={\n \"field_list\": [\"control_text\", \"control_type\", \"control_rect\"]\n }\n )\n])\n\n# Example output:\n{\n \"control_text\": \"Calculator\",\n \"control_type\": \"Window\",\n \"control_rect\": {\"x\": 100, \"y\": 100, \"width\": 400, \"height\": 600}\n}\n```\n\n---\n\n### 4. get_app_window_controls_info\n\nGet information about all UI controls in the selected application window.\n\n#### Description\n\nScans the currently selected window and retrieves information about all interactive controls (buttons, text boxes, etc.). This is essential for understanding what actions can be performed on the window.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `field_list` | `List[str]` | Yes | - | List of field names to retrieve for each control |\n\n#### Supported Fields\n\n- `\"label\"`: Control identifier/label\n- `\"control_text\"`: Text content of the control\n- `\"control_type\"`: Type of control (Button, Edit, etc.)\n- `\"control_rect\"`: Bounding rectangle\n- `\"is_enabled\"`: Whether control is enabled\n- `\"is_visible\"`: Whether control is visible\n\n#### Returns\n\n**Type**: `List[Dict[str, Any]]`\n\nList of dictionaries, each representing one UI control.\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_info\",\n tool_name=\"get_app_window_controls_info\",\n parameters={\n \"field_list\": [\"label\", \"control_text\", \"control_type\"]\n }\n )\n])\n\n# Example output:\n[\n {\n \"label\": \"1\",\n \"control_text\": \"Submit\",\n \"control_type\": \"Button\"\n },\n {\n \"label\": \"2\",\n \"control_text\": \"\",\n \"control_type\": \"Edit\"\n }\n]\n```\n\n---\n\n### 5. get_app_window_controls_target_info\n\nGet `TargetInfo` objects for all controls in the selected window.\n\n#### Description\n\nSimilar to `get_app_window_controls_info`, but returns structured `TargetInfo` objects for internal framework use.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `field_list` | `List[str]` | Yes | - | List of field names to retrieve |\n\n#### Returns\n\n**Type**: `List[TargetInfo]`\n\nList of `TargetInfo` objects, each with:\n- `kind`: TargetKind.CONTROL\n- `id`: Control identifier\n- `name`: Control text\n- `type`: Control type\n- `rect`: Bounding rectangle\n- `source`: \"uia\"\n\n---\n\n### 6. capture_window_screenshot\n\nCapture a screenshot of the currently selected application window.\n\n#### Description\n\nTakes a screenshot of the active window and returns it as base64-encoded image data. This is crucial for visual observation and LLM vision capabilities.\n\n#### Parameters\n\nNone\n\n#### Returns\n\n**Type**: `str`\n\nBase64-encoded PNG image data.\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::capture_window_screenshot\",\n tool_name=\"capture_window_screenshot\",\n parameters={}\n )\n])\n\n# Result is base64 string: \"iVBORw0KGgoAAAANSUhEUgAA...\"\n```\n\n#### Error Handling\n\nReturns error message string if screenshot capture fails:\n```\n\"Error: No window selected\"\n\"Error capturing screenshot: {error_details}\"\n```\n\n---\n\n### 7. capture_desktop_screenshot\n\nCapture a screenshot of the entire desktop or primary screen.\n\n#### Description\n\nTakes a screenshot of the desktop environment, either all monitors or just the primary screen.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `all_screens` | `bool` | No | `True` | Capture all screens (True) or primary screen only (False) |\n\n#### Returns\n\n**Type**: `str`\n\nBase64-encoded PNG image data of the desktop screenshot.\n\n#### Example\n\n```python\n# Capture all screens\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::capture_desktop_screenshot\",\n tool_name=\"capture_desktop_screenshot\",\n parameters={\"all_screens\": True}\n )\n])\n\n# Capture primary screen only\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::capture_desktop_screenshot\",\n tool_name=\"capture_desktop_screenshot\",\n parameters={\"all_screens\": False}\n )\n])\n```\n\n---\n\n### 8. get_ui_tree\n\nGet the complete UI tree structure for the selected window.\n\n#### Description\n\nRetrieves the hierarchical structure of all UI elements in the window as a tree. This provides deep insight into the window's layout and control relationships.\n\n#### Parameters\n\nNone\n\n#### Returns\n\n**Type**: `Dict[str, Any]`\n\nUI tree structure as a nested dictionary representing the control hierarchy.\n\n#### Example\n\n```python\nresult = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_ui_tree\",\n tool_name=\"get_ui_tree\",\n parameters={}\n )\n])\n\n# Example output (simplified):\n{\n \"control_type\": \"Window\",\n \"name\": \"Calculator\",\n \"children\": [\n {\n \"control_type\": \"Pane\",\n \"name\": \"Display\",\n \"children\": [...]\n },\n {\n \"control_type\": \"Button\",\n \"name\": \"1\"\n }\n ]\n}\n```\n\n#### Error Handling\n\nReturns error dictionary if UI tree extraction fails:\n```python\n{\"error\": \"No window selected\"}\n{\"error\": \"Error getting UI tree: {details}\"}\n```\n\n## Configuration\n\n### Basic Configuration\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n\nAppAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n```\n\n### Configuration Options\n\n| Option | Type | Description |\n|--------|------|-------------|\n| `namespace` | `str` | Must be `\"UICollector\"` |\n| `type` | `str` | Deployment type: `\"local\"` |\n| `reset` | `bool` | Whether to reset server state between tasks |\n\n## Internal State\n\nThe UICollector maintains shared state across operations:\n\n- **photographer**: Screenshot capture facade\n- **control_inspector**: UI control inspection facade\n- **selected_app_window**: Currently selected window (set by HostUIExecutor)\n- **last_app_windows**: Cached list of desktop windows\n- **control_dict**: Dictionary mapping control IDs to control objects\n\n## Usage Patterns\n\n### Pattern 1: Complete Desktop Observation\n\n```python\n# 1. Get all windows\nwindows = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_desktop_app_info\", ...)\n])\n\n# 2. Capture desktop screenshot\nscreenshot = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::capture_desktop_screenshot\", ...)\n])\n\n# 3. Select target window\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_application_window\",\n parameters={\"id\": \"1\", \"name\": \"Calculator\"}\n )\n])\n\n# 4. Get window controls\ncontrols = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_info\",\n parameters={\"field_list\": [\"label\", \"control_text\", \"control_type\"]}\n )\n])\n```\n\n### Pattern 2: Window-Specific Observation\n\n```python\n# After window is selected by HostUIExecutor...\n\n# Get window info\nwindow_info = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_info\",\n parameters={\"field_list\": [\"control_text\", \"control_rect\"]}\n )\n])\n\n# Get window screenshot\nscreenshot = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::capture_window_screenshot\", ...)\n])\n\n# Get UI controls\ncontrols = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_info\",\n parameters={\"field_list\": [\"label\", \"control_text\"]}\n )\n])\n```\n\n## Best Practices\n\n### 1. Caching Window Lists\n\n```python\n# First call: refresh windows\nwindows = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_desktop_app_info\",\n parameters={\"refresh_app_windows\": True}\n )\n])\n\n# Subsequent calls: use cached data\nwindows = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_desktop_app_info\",\n parameters={\"refresh_app_windows\": False} # Faster\n )\n])\n```\n\n### 2. Selective Field Retrieval\n\n```python\n# ✅ Good: Only request needed fields\ncontrols = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_info\",\n parameters={\"field_list\": [\"label\", \"control_text\"]}\n )\n])\n\n# ❌ Bad: Don't request unnecessary fields\ncontrols = await computer.run_actions([\n MCPToolCall(\n tool_key=\"data_collection::get_app_window_controls_info\",\n parameters={\"field_list\": [\n \"label\", \"control_text\", \"control_type\", \"control_rect\",\n \"is_visible\", \"is_enabled\", \"automation_id\", \"class_name\"\n ]} # Too many fields slow down processing\n )\n])\n```\n\n### 3. Error Handling\n\n```python\n# Always check for window selection\nwindow_info = await computer.run_actions([\n MCPToolCall(tool_key=\"data_collection::get_app_window_info\", ...)\n])\n\nif \"error\" in window_info[0].content[0].text:\n # No window selected\n # Select window first...\n```\n\n## Related Documentation\n\n- [Data Collection Overview](../data_collection.md) - Data collection concepts\n- [HostUIExecutor](./host_ui_executor.md) - Window selection server\n- [AppUIExecutor](./app_ui_executor.md) - UI action execution\n- [Local Servers](../local_servers.md) - Local server deployment\n"
},
{
"path": "documents/docs/mcp/servers/word_com_executor.md",
"content": "# WordCOMExecutor Server\n\n## Overview\n\n**WordCOMExecutor** provides Microsoft Word automation via COM API for efficient document manipulation beyond UI automation.\n\n**Server Type:** Action \n**Deployment:** Local (in-process) \n**Agent:** AppAgent \n**Target Application:** Microsoft Word (`WINWORD.EXE`) \n**LLM-Selectable:** ✅ Yes\n\n## Server Information\n\n| Property | Value |\n|----------|-------|\n| **Namespace** | `WordCOMExecutor` |\n| **Server Name** | `UFO UI AppAgent Action MCP Server` |\n| **Platform** | Windows |\n| **Requires** | Microsoft Word (COM interface) |\n| **Tool Type** | `action` |\n\n## Tools Summary\n\n| Tool Name | Description |\n|-----------|-------------|\n| `insert_table` | Insert table into document |\n| `select_text` | Select specific text |\n| `select_table` | Select table by index |\n| `select_paragraph` | Select paragraph range |\n| `save_as` | Save/export document |\n| `set_font` | Set font properties for selected text |\n\n## Tool Details\n\n### insert_table\n\nInsert a table into the Word document at the current cursor position.\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `rows` | `int` | ✅ Yes | Number of rows in the table |\n| `columns` | `int` | ✅ Yes | Number of columns in the table |\n\n#### Returns\n\n`str` - Result message\n\n#### Example\n\n```python\n# Insert 3x4 table\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::insert_table\",\n tool_name=\"insert_table\",\n parameters={\"rows\": 3, \"columns\": 4}\n )\n])\n```\n\n---\n\n### select_text\n\nSelect exact text in the document for further operations (formatting, deletion, etc.).\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `text` | `str` | ✅ Yes | Exact text to select |\n\n#### Returns\n\n`str` - Selected text if successful, or \"text not found\" message\n\n#### Example\n\n```python\n# Select specific text\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_text\",\n tool_name=\"select_text\",\n parameters={\"text\": \"Annual Report 2024\"}\n )\n])\n\n# Then format it\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_font\",\n tool_name=\"set_font\",\n parameters={\"font_name\": \"Arial\", \"font_size\": 18}\n )\n])\n```\n\n---\n\n### select_table\n\nSelect a table in the document by its index (1-based).\n\n#### Parameters\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `number` | `int` | ✅ Yes | Table index (1-based) |\n\n#### Returns\n\n`str` - Success message or \"out of range\" message\n\n#### Example\n\n```python\n# Select first table\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_table\",\n tool_name=\"select_table\",\n parameters={\"number\": 1}\n )\n])\n```\n\n---\n\n### select_paragraph\n\nSelect a range of paragraphs in the document.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `start_index` | `int` | ✅ Yes | - | Start paragraph index |\n| `end_index` | `int` | ✅ Yes | - | End paragraph index (`-1` = end of document) |\n| `non_empty` | `bool` | No | `True` | Select only non-empty paragraphs |\n\n#### Returns\n\n`str` - Result message\n\n#### Example\n\n```python\n# Select paragraphs 1-5 (non-empty only)\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_paragraph\",\n tool_name=\"select_paragraph\",\n parameters={\n \"start_index\": 1,\n \"end_index\": 5,\n \"non_empty\": True\n }\n )\n])\n\n# Select from paragraph 10 to end\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_paragraph\",\n tool_name=\"select_paragraph\",\n parameters={\"start_index\": 10, \"end_index\": -1}\n )\n])\n```\n\n---\n\n### save_as\n\nSave or export Word document to specified format. **Fastest way to save documents.**\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `file_dir` | `str` | No | `\"\"` | Directory path (empty = current directory) |\n| `file_name` | `str` | No | `\"\"` | Filename without extension (empty = current name) |\n| `file_ext` | `str` | No | `\"\"` | File extension (empty = `.pdf`) |\n\n#### Supported Extensions\n\n- `.pdf` - PDF format (default)\n- `.docx` - Word document\n- `.txt` - Plain text\n- `.html` - HTML format\n- `.rtf` - Rich Text Format\n\n#### Returns\n\n`str` - Success/failure message\n\n#### Example\n\n```python\n# Save as PDF in current directory\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"\",\n \"file_name\": \"\",\n \"file_ext\": \"\" # Defaults to .pdf\n }\n )\n])\n\n# Save as DOCX with specific name and path\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n tool_name=\"save_as\",\n parameters={\n \"file_dir\": \"C:\\\\Documents\\\\Reports\",\n \"file_name\": \"Q4_Report_2024\",\n \"file_ext\": \".docx\"\n }\n )\n])\n```\n\n---\n\n### set_font\n\nSet font properties for currently selected text.\n\n!!!warning \"Selection Required\"\n Text must be selected first using `select_text`, `select_paragraph`, or manual selection.\n\n#### Parameters\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `font_name` | `str` | No | `None` | Font name (e.g., \"Arial\", \"Times New Roman\", \"宋体\") |\n| `font_size` | `int` | No | `None` | Font size in points |\n\n#### Returns\n\n`str` - Font change confirmation or \"no text selected\" message\n\n#### Example\n\n```python\n# Select text first\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::select_text\",\n parameters={\"text\": \"Important Notice\"}\n )\n])\n\n# Set font to Arial 16pt\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_font\",\n tool_name=\"set_font\",\n parameters={\"font_name\": \"Arial\", \"font_size\": 16}\n )\n])\n\n# Change only size\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::set_font\",\n tool_name=\"set_font\",\n parameters={\"font_size\": 20} # Keep current font name\n )\n])\n```\n\n## Configuration\n\n```yaml\nAppAgent:\n # Word-specific configuration\n WINWORD.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: WordCOMExecutor # Add COM automation\n type: local\n reset: true # Reset COM state when switching documents\n```\n\n### Configuration Options\n\n| Option | Value | Description |\n|--------|-------|-------------|\n| `reset` | `true` | **Recommended**: Reset COM state between documents to prevent data leakage |\n| `reset` | `false` | Keep COM state across documents (faster but risky) |\n\n## Best Practices\n\n### 1. Use COM for Bulk Operations\n\n```python\n# ✅ Good: Fast COM API\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::insert_table\", parameters={\"rows\": 10, \"columns\": 5})\n])\n\n# ❌ Bad: Slow UI automation\nfor i in range(10):\n await computer.run_actions([\n MCPToolCall(tool_key=\"action::click_input\", ...) # Click Insert Table\n ])\n```\n\n### 2. Prefer save_as Over Manual Saving\n\n```python\n# ✅ Good: One command\nawait computer.run_actions([\n MCPToolCall(\n tool_key=\"action::save_as\",\n parameters={\"file_dir\": \"C:\\\\Reports\", \"file_name\": \"report\", \"file_ext\": \".pdf\"}\n )\n])\n\n# ❌ Bad: Multiple UI steps\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::keyboard_input\", parameters={\"keys\": \"{VK_CONTROL}s\"})\n])\n# ... then navigate save dialog ...\n```\n\n### 3. Select Before Formatting\n\n```python\n# ✅ Good: Select then format\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::select_text\", parameters={\"text\": \"Title\"})\n])\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::set_font\", parameters={\"font_size\": 24})\n])\n\n# ❌ Bad: Format without selection\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::set_font\", parameters={\"font_size\": 24})\n]) # Fails: \"no text selected\"\n```\n\n## Use Cases\n\n### Document Report Generation\n\n```python\n# 1. Select title\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::select_paragraph\", parameters={\"start_index\": 1, \"end_index\": 1})\n])\n\n# 2. Format title\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::set_font\", parameters={\"font_name\": \"Arial\", \"font_size\": 20})\n])\n\n# 3. Insert data table\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::insert_table\", parameters={\"rows\": 5, \"columns\": 3})\n])\n\n# 4. Save as PDF\nawait computer.run_actions([\n MCPToolCall(tool_key=\"action::save_as\", parameters={\"file_ext\": \".pdf\"})\n])\n```\n\n## Related Documentation\n\n- [ExcelCOMExecutor](./excel_com_executor.md) - Excel COM automation\n- [PowerPointCOMExecutor](./ppt_com_executor.md) - PowerPoint COM automation\n- [AppUIExecutor](./app_ui_executor.md) - UI-based Word automation\n- [Action Servers](../action.md) - Action server concepts\n"
},
{
"path": "documents/docs/mobile/as_galaxy_device.md",
"content": "# Using Mobile Agent as Galaxy Device\n\nConfigure Mobile Agent as a sub-agent in UFO's Galaxy framework to enable cross-platform, multi-device task orchestration. Galaxy can coordinate Mobile agents alongside Windows and Linux devices to execute complex workflows spanning multiple systems and platforms.\n\n> **📖 Prerequisites:**\n> \n> Before configuring Mobile Agent in Galaxy, ensure you have:\n> \n> - Completed the [Mobile Agent Quick Start Guide](../getting_started/quick_start_mobile.md) - Learn how to set up server, MCP services, and client\n> - Read the [Mobile Agent Overview](overview.md) - Understand Mobile Agent's design and capabilities\n> - Reviewed the [Galaxy Overview](../galaxy/overview.md) - Understand multi-device orchestration\n\n## Overview\n\nThe **Galaxy framework** provides multi-tier orchestration capabilities, allowing you to manage multiple device agents (Windows, Linux, Android, etc.) from a central ConstellationAgent. When configured as a Galaxy device, MobileAgent becomes a **sub-agent** that can:\n\n- Execute Android-specific subtasks assigned by Galaxy\n- Participate in cross-platform workflows (e.g., Windows + Android + Linux collaboration)\n- Report execution status back to the orchestrator\n- Be dynamically selected based on capabilities and metadata\n\nFor detailed information about MobileAgent's design and capabilities, see [Mobile Agent Overview](overview.md).\n\n## Galaxy Architecture with Mobile Agent\n\n```mermaid\ngraph TB\n User[User Request]\n Galaxy[Galaxy ConstellationAgent Orchestrator]\n \n subgraph \"Device Pool\"\n Win1[Windows Device 1 HostAgent]\n Linux1[Linux Agent 1 CLI Executor]\n Mobile1[Mobile Agent 1 Android Phone]\n Mobile2[Mobile Agent 2 Android Tablet]\n Mobile3[Mobile Agent 3 Android Emulator]\n end\n \n User -->|Complex Task| Galaxy\n Galaxy -->|Windows Subtask| Win1\n Galaxy -->|Linux Subtask| Linux1\n Galaxy -->|Mobile Subtask| Mobile1\n Galaxy -->|Mobile Subtask| Mobile2\n Galaxy -->|Mobile Subtask| Mobile3\n \n style Galaxy fill:#ffe1e1\n style Mobile1 fill:#c8e6c9\n style Mobile2 fill:#c8e6c9\n style Mobile3 fill:#c8e6c9\n```\n\nGalaxy orchestrates:\n\n- **Task decomposition** - Break complex requests into platform-specific subtasks\n- **Device selection** - Choose appropriate devices based on capabilities\n- **Parallel execution** - Execute subtasks concurrently across devices\n- **Result aggregation** - Combine results from all devices\n\n---\n\n## Configuration Guide\n\n### Step 1: Configure Device in `devices.yaml`\n\nAdd your Mobile agent(s) to the device list in `config/galaxy/devices.yaml`:\n\n#### Example Configuration\n\n```yaml\ndevices:\n - device_id: \"mobile_phone_1\"\n server_url: \"ws://192.168.1.100:5001/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"messaging\"\n - \"camera\"\n - \"location\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"13\"\n screen_size: \"1080x2400\"\n installed_apps:\n - \"com.google.android.apps.maps\"\n - \"com.whatsapp\"\n - \"com.android.chrome\"\n description: \"Personal Android phone for mobile tasks\"\n auto_connect: true\n max_retries: 5\n```\n\n### Step 2: Understanding Configuration Fields\n\n| Field | Required | Type | Description |\n|-------|----------|------|-------------|\n| `device_id` | ✅ Yes | string | **Unique identifier** - must match client `--client-id` |\n| `server_url` | ✅ Yes | string | WebSocket URL - must match server endpoint |\n| `os` | ✅ Yes | string | Operating system - set to `\"mobile\"` |\n| `capabilities` | ❌ Optional | list | Skills/capabilities for task routing |\n| `metadata` | ❌ Optional | dict | Custom context for LLM-based task execution |\n| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup (default: `true`) |\n| `max_retries` | ❌ Optional | integer | Connection retry attempts (default: `5`) |\n\n### Step 3: Capabilities-Based Task Routing\n\nGalaxy uses the `capabilities` field to intelligently route subtasks to appropriate devices. Define capabilities based on device features, installed apps, or task types.\n\n#### Example Capability Configurations\n\n**Personal Phone:**\n```yaml\ncapabilities:\n - \"mobile\"\n - \"android\"\n - \"messaging\"\n - \"whatsapp\"\n - \"maps\"\n - \"camera\"\n - \"location\"\n```\n\n**Work Phone:**\n```yaml\ncapabilities:\n - \"mobile\"\n - \"android\"\n - \"email\"\n - \"calendar\"\n - \"office_apps\"\n - \"vpn\"\n```\n\n**Testing Emulator:**\n```yaml\ncapabilities:\n - \"mobile\"\n - \"android\"\n - \"testing\"\n - \"automation\"\n - \"screenshots\"\n```\n\n**Tablet:**\n```yaml\ncapabilities:\n - \"mobile\"\n - \"android\"\n - \"tablet\"\n - \"large_screen\"\n - \"media\"\n - \"reading\"\n```\n\n### Step 4: Metadata for Contextual Execution\n\nThe `metadata` field provides contextual information that the LLM uses when generating actions for the Mobile agent.\n\n#### Metadata Examples\n\n**Personal Phone Metadata:**\n```yaml\nmetadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"13\"\n sdk_version: \"33\"\n screen_size: \"1080x2400\"\n screen_density: \"420\"\n installed_apps:\n - \"com.google.android.apps.maps\"\n - \"com.whatsapp\"\n - \"com.android.chrome\"\n - \"com.spotify.music\"\n contacts:\n - \"John Doe\"\n - \"Jane Smith\"\n description: \"Personal Android phone with social and navigation apps\"\n```\n\n**Work Device Metadata:**\n```yaml\nmetadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"12\"\n screen_size: \"1080x2340\"\n installed_apps:\n - \"com.microsoft.office.outlook\"\n - \"com.microsoft.teams\"\n - \"com.slack\"\n vpn_configured: true\n email_accounts:\n - \"work@company.com\"\n description: \"Work phone with corporate apps and VPN\"\n```\n\n**Testing Emulator Metadata:**\n```yaml\nmetadata:\n os: \"mobile\"\n device_type: \"emulator\"\n android_version: \"14\"\n sdk_version: \"34\"\n screen_size: \"1080x1920\"\n installed_apps:\n - \"com.example.testapp\"\n adb_over_network: true\n description: \"Android emulator for app testing\"\n```\n\n#### How Metadata is Used\n\nThe LLM receives metadata in the system prompt, enabling context-aware action generation:\n\n- **App Availability**: LLM knows which apps can be launched\n- **Screen Size**: Informs swipe distances and touch coordinates\n- **Android Version**: Affects available features and UI patterns\n- **Device Type**: Phone vs tablet affects UI layout\n- **Custom Fields**: Any additional context you provide\n\n**Example**: With the personal phone metadata above, when the user requests \"Navigate to restaurant\", the LLM knows Maps is installed and can generate `launch_app(package_name=\"com.google.android.apps.maps\")`.\n\n---\n\n## Multi-Device Configuration Example\n\n### Complete Galaxy Setup\n\n```yaml\ndevices:\n # Windows Desktop Agent\n - device_id: \"windows_desktop_1\"\n server_url: \"ws://192.168.1.101:5000/ws\"\n os: \"windows\"\n capabilities:\n - \"office_applications\"\n - \"email\"\n - \"web_browsing\"\n metadata:\n os: \"windows\"\n description: \"Office productivity workstation\"\n auto_connect: true\n max_retries: 5\n \n # Linux Server Agent\n - device_id: \"linux_server_1\"\n server_url: \"ws://192.168.1.102:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"server\"\n - \"database\"\n - \"api\"\n metadata:\n os: \"linux\"\n description: \"Backend server\"\n auto_connect: true\n max_retries: 5\n \n # Personal Android Phone\n - device_id: \"mobile_phone_personal\"\n server_url: \"ws://192.168.1.103:5002/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"messaging\"\n - \"whatsapp\"\n - \"maps\"\n - \"camera\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"13\"\n screen_size: \"1080x2400\"\n installed_apps:\n - \"com.google.android.apps.maps\"\n - \"com.whatsapp\"\n - \"com.android.chrome\"\n description: \"Personal phone with social apps\"\n auto_connect: true\n max_retries: 5\n \n # Work Android Phone\n - device_id: \"mobile_phone_work\"\n server_url: \"ws://192.168.1.104:5003/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"email\"\n - \"calendar\"\n - \"teams\"\n metadata:\n os: \"mobile\"\n device_type: \"phone\"\n android_version: \"12\"\n screen_size: \"1080x2340\"\n installed_apps:\n - \"com.microsoft.office.outlook\"\n - \"com.microsoft.teams\"\n description: \"Work phone with corporate apps\"\n auto_connect: true\n max_retries: 5\n \n # Android Tablet\n - device_id: \"mobile_tablet_home\"\n server_url: \"ws://192.168.1.105:5004/ws\"\n os: \"mobile\"\n capabilities:\n - \"mobile\"\n - \"android\"\n - \"tablet\"\n - \"media\"\n - \"reading\"\n metadata:\n os: \"mobile\"\n device_type: \"tablet\"\n android_version: \"13\"\n screen_size: \"2560x1600\"\n installed_apps:\n - \"com.netflix.mediaclient\"\n - \"com.google.android.youtube\"\n description: \"Tablet for media and entertainment\"\n auto_connect: true\n max_retries: 5\n```\n\n---\n\n## Starting Galaxy with Mobile Agents\n\n### Prerequisites\n\nEnsure all components are running before starting Galaxy:\n\n1. ✅ Device Agent Servers running on all machines\n2. ✅ Device Agent Clients connected to their respective servers\n3. ✅ MCP Services running (both data collection and action servers)\n4. ✅ ADB accessible and Android devices connected\n5. ✅ USB debugging enabled on all Android devices\n6. ✅ LLM configured in `config/ufo/agents.yaml` or `config/galaxy/agent.yaml`\n\n### Launch Sequence\n\n#### Step 1: Prepare Android Devices\n\n```bash\n# Check ADB connection to all devices\nadb devices\n\n# Expected output:\n# List of devices attached\n# 192.168.1.103:5555 device\n# 192.168.1.104:5555 device\n# emulator-5554 device\n```\n\n**For Physical Devices:**\n1. Enable USB debugging in Developer Options\n2. Connect via USB or wireless ADB\n3. Accept ADB debugging prompt on device\n\n**For Emulators:**\n1. Start Android emulator\n2. ADB connects automatically\n\n#### Step 2: Start Device Agent Servers\n\n```bash\n# On machine hosting personal phone agent (192.168.1.103)\npython -m ufo.server.app --port 5002 --platform mobile\n\n# On machine hosting work phone agent (192.168.1.104)\npython -m ufo.server.app --port 5003 --platform mobile\n\n# On machine hosting tablet agent (192.168.1.105)\npython -m ufo.server.app --port 5004 --platform mobile\n```\n\n#### Step 3: Start MCP Servers for Each Device\n\n```bash\n# On machine hosting personal phone\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --data-port 8020 \\\n --action-port 8021 \\\n --server both\n\n# On machine hosting work phone\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --data-port 8022 \\\n --action-port 8023 \\\n --server both\n\n# On machine hosting tablet\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --data-port 8024 \\\n --action-port 8025 \\\n --server both\n```\n\n#### Step 4: Start Mobile Clients\n\n```bash\n# Personal phone client\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.103:5002/ws \\\n --client-id mobile_phone_personal \\\n --platform mobile\n\n# Work phone client\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.104:5003/ws \\\n --client-id mobile_phone_work \\\n --platform mobile\n\n# Tablet client\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.105:5004/ws \\\n --client-id mobile_tablet_home \\\n --platform mobile\n```\n\n#### Step 5: Launch Galaxy\n\n```bash\n# On your control machine (interactive mode)\npython -m galaxy --interactive\n```\n\n**Or launch with a specific request:**\n\n```bash\npython -m galaxy \"Your cross-device task description here\"\n```\n\nGalaxy will automatically connect to all configured devices and display the orchestration interface.\n\n---\n\n## Example Multi-Device Workflows\n\n### Workflow 1: Cross-Platform Productivity\n\n**User Request:**\n> \"Get my meeting notes from email on work phone, summarize them on desktop, and send summary to team via WhatsApp on personal phone\"\n\n**Galaxy Orchestration:**\n\n```mermaid\nsequenceDiagram\n participant User\n participant Galaxy\n participant WorkPhone as Work Phone (Android)\n participant Desktop as Windows Desktop\n participant PersonalPhone as Personal Phone (Android)\n \n User->>Galaxy: Request meeting workflow\n Galaxy->>Galaxy: Decompose task\n \n Note over Galaxy,WorkPhone: Subtask 1: Get notes from email\n Galaxy->>WorkPhone: \"Open Outlook and find meeting notes\"\n WorkPhone->>WorkPhone: Launch Outlook app\n WorkPhone->>WorkPhone: Navigate to inbox\n WorkPhone->>WorkPhone: Find meeting email\n WorkPhone->>WorkPhone: Extract notes text\n WorkPhone-->>Galaxy: Notes content\n \n Note over Galaxy,Desktop: Subtask 2: Summarize on desktop\n Galaxy->>Desktop: \"Summarize meeting notes\"\n Desktop->>Desktop: Open Word\n Desktop->>Desktop: Paste notes\n Desktop->>Desktop: Generate summary\n Desktop-->>Galaxy: Summary document\n \n Note over Galaxy,PersonalPhone: Subtask 3: Send via WhatsApp\n Galaxy->>PersonalPhone: \"Send summary to team on WhatsApp\"\n PersonalPhone->>PersonalPhone: Launch WhatsApp\n PersonalPhone->>PersonalPhone: Select team group\n PersonalPhone->>PersonalPhone: Type summary message\n PersonalPhone->>PersonalPhone: Send message\n PersonalPhone-->>Galaxy: Message sent\n \n Galaxy-->>User: Workflow completed\n```\n\n### Workflow 2: Mobile Testing Across Devices\n\n**User Request:**\n> \"Test the new app on phone, tablet, and emulator, capture screenshots of each screen\"\n\n**Galaxy Orchestration:**\n\n1. **Mobile Phone**: Install app, navigate through screens, capture screenshots\n2. **Mobile Tablet**: Install app (tablet layout), navigate screens, capture screenshots\n3. **Mobile Emulator**: Install app, run automated test suite, capture screenshots\n4. **Windows Desktop**: Aggregate screenshots, generate test report\n\n### Workflow 3: Location-Based Multi-Device Task\n\n**User Request:**\n> \"Find nearest coffee shops on phone, book table using tablet, add calendar event on work phone\"\n\n**Galaxy Orchestration:**\n\n1. **Personal Phone**: Launch Maps, search \"coffee shops near me\", get results\n2. **Tablet**: Open booking app, select coffee shop, book table\n3. **Work Phone**: Open Calendar, create event with location and time\n4. **Galaxy**: Aggregate confirmations and notify user\n\n---\n\n## Task Assignment Behavior\n\n### How Galaxy Routes Tasks to Mobile Agents\n\nGalaxy's ConstellationAgent uses several factors to select the appropriate mobile device for each subtask:\n\n| Factor | Description | Example |\n|--------|-------------|---------|\n| **Capabilities** | Match subtask requirements to device capabilities | `\"messaging\"` → Personal phone |\n| **OS Requirement** | Platform-specific tasks routed to correct OS | Mobile tasks → Mobile agents |\n| **Metadata Context** | Use device-specific apps and configurations | WhatsApp task → Device with WhatsApp |\n| **Device Type** | Phone vs tablet for different UI requirements | Media viewing → Tablet |\n| **Device Status** | Only assign to online, healthy devices | Skip offline or failing devices |\n| **Load Balancing** | Distribute tasks across similar devices | Round-robin across phones |\n\n### Example Task Decomposition\n\n**User Request:**\n> \"Check messages on personal phone, review calendar on work phone, and play video on tablet\"\n\n**Galaxy Decomposition:**\n\n```yaml\nTask 1:\n Description: \"Check messages on WhatsApp\"\n Target: mobile_phone_personal\n Reason: Has \"whatsapp\" capability and personal messaging apps\n \nTask 2:\n Description: \"Review today's calendar events\"\n Target: mobile_phone_work\n Reason: Has \"calendar\" capability and work email/calendar\n \nTask 3:\n Description: \"Play video on YouTube\"\n Target: mobile_tablet_home\n Reason: Has \"media\" capability and larger screen suitable for video\n```\n\n---\n\n## Critical Configuration Requirements\n\n!!!danger \"Configuration Validation\"\n Ensure these match exactly or Galaxy cannot control the device:\n \n - **Device ID**: `device_id` in `devices.yaml` must match `--client-id` in client command\n - **Server URL**: `server_url` in `devices.yaml` must match `--ws-server` in client command\n - **Platform**: Must include `--platform mobile` in client command\n - **ADB Access**: Android device must be accessible via ADB\n - **MCP Servers**: Both data collection and action servers must be running\n\n---\n\n## Monitoring & Debugging\n\n### Verify Device Registration\n\n**Check Galaxy device pool:**\n\n```bash\ncurl http://:5000/api/devices\n```\n\n**Expected response:**\n\n```json\n{\n \"devices\": [\n {\n \"device_id\": \"mobile_phone_personal\",\n \"os\": \"mobile\",\n \"status\": \"online\",\n \"capabilities\": [\"mobile\", \"android\", \"messaging\", \"whatsapp\", \"maps\"]\n },\n {\n \"device_id\": \"mobile_phone_work\",\n \"os\": \"mobile\",\n \"status\": \"online\",\n \"capabilities\": [\"mobile\", \"android\", \"email\", \"calendar\", \"teams\"]\n }\n ]\n}\n```\n\n### View Task Assignments\n\nGalaxy logs show task routing decisions:\n\n```log\nINFO - [Galaxy] Task decomposition: 3 subtasks created\nINFO - [Galaxy] Subtask 1 → mobile_phone_personal (capability match: messaging)\nINFO - [Galaxy] Subtask 2 → mobile_phone_work (capability match: calendar)\nINFO - [Galaxy] Subtask 3 → mobile_tablet_home (capability match: media)\n```\n\n### Troubleshooting Device Connection\n\n**Issue**: Mobile agent not appearing in Galaxy device pool\n\n**Diagnosis:**\n\n1. **Check ADB connection:**\n ```bash\n adb devices\n ```\n\n2. **Verify client connection:**\n ```bash\n curl http://192.168.1.103:5002/api/clients\n ```\n\n3. **Check `devices.yaml` configuration** matches client parameters\n\n4. **Review Galaxy logs** for connection errors\n\n5. **Ensure `auto_connect: true`** in `devices.yaml`\n\n6. **Check MCP servers** are running:\n ```bash\n curl http://localhost:8020/health # Data collection server\n curl http://localhost:8021/health # Action server\n ```\n\n---\n\n## Mobile-Specific Considerations\n\n### Screenshot Capture for Galaxy\n\nMobile agents automatically capture screenshots during execution, which Galaxy can:\n\n- Display in orchestration UI\n- Include in execution reports\n- Use for debugging failed tasks\n- Share with other agents for context\n\n### Touch Coordinates Across Devices\n\nDifferent Android devices have different screen sizes and densities. Galaxy handles this by:\n\n- Using control IDs instead of absolute coordinates\n- Having each mobile agent handle device-specific coordinate calculations\n- Storing device resolution in metadata for reference\n\n### App Availability\n\nGalaxy can query `installed_apps` from metadata to:\n\n- Route tasks to devices with required apps\n- Skip devices missing necessary apps\n- Suggest app installation when needed\n\n---\n\n## Related Documentation\n\n- [Mobile Agent Overview](overview.md) - Architecture and design principles\n- [Mobile Agent Commands](commands.md) - MCP tools for device interaction\n- [Galaxy Overview](../galaxy/overview.md) - Multi-device orchestration framework\n- [Galaxy Quick Start](../getting_started/quick_start_galaxy.md) - Galaxy deployment guide\n- [Constellation Orchestrator](../galaxy/constellation_orchestrator/overview.md) - Task orchestration\n- [Galaxy Devices Configuration](../configuration/system/galaxy_devices.md) - Complete device configuration reference\n\n---\n\n## Summary\n\nUsing Mobile Agent as a Galaxy device enables sophisticated multi-device orchestration:\n\n- **Cross-Platform Workflows**: Seamlessly combine Android, Windows, and Linux tasks\n- **Capability-Based Routing**: Galaxy selects the right device for each subtask\n- **Visual Context**: Screenshots provide rich execution tracing\n- **Parallel Execution**: Multiple mobile devices work concurrently\n- **Metadata-Aware**: LLM uses device-specific context (installed apps, screen size, etc.)\n- **Robust Caching**: Efficient ADB usage through smart caching strategies\n\nWith Mobile Agent in Galaxy, you can orchestrate complex workflows spanning mobile apps, desktop applications, and server systems from a single unified interface.\n"
},
{
"path": "documents/docs/mobile/commands.md",
"content": "# MobileAgent MCP Commands\n\nMobileAgent interacts with Android devices through MCP (Model Context Protocol) tools provided by two specialized MCP servers. These tools provide atomic building blocks for mobile task execution, isolating device-specific operations within the MCP server layer.\n\n> **📖 Related Documentation:**\n> \n> - [Mobile Agent Overview](overview.md) - Architecture and core responsibilities\n> - [State Machine](state.md) - FSM states and transitions\n> - [Processing Strategy](strategy.md) - How commands are orchestrated in the 4-phase pipeline\n> - [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up MCP servers for your device\n\n## Command Architecture\n\n### Dual MCP Server Design\n\nMobileAgent uses two separate MCP servers for different responsibilities:\n\n```mermaid\ngraph LR\n A[MobileAgent] --> B[Command Dispatcher]\n B --> C[Data Collection Server Port 8020]\n B --> D[Action Server Port 8021]\n \n C --> E[ADB Commands screencap, uiautomator, pm list]\n D --> F[ADB Commands input tap/swipe/text, monkey]\n \n E --> G[Android Device]\n F --> G\n \n C -.Shared State.-> H[MobileServerState Singleton]\n D -.Shared State.-> H\n```\n\n**Why Two Servers?**\n\n- **Separation of Concerns**: Data retrieval vs. device control\n- **Performance**: Data collection can cache aggressively, actions invalidate caches\n- **Security**: Different tools can have different permission levels\n- **Scalability**: Servers can run on different hosts if needed\n\n**Shared State**: Both servers share a singleton `MobileServerState` for:\n- Caching control information (5 seconds TTL)\n- Caching installed apps (5 minutes TTL)\n- Caching UI tree (5 seconds TTL)\n- Coordinating cache invalidation after actions\n\n### Command Dispatcher\n\nThe command dispatcher routes commands to the appropriate MCP server:\n\n```python\nfrom aip.messages import Command\n\n# Create data collection command\ncommand = Command(\n tool_name=\"capture_screenshot\",\n parameters={},\n tool_type=\"data_collection\"\n)\n\n# Execute command via dispatcher\nresults = await command_dispatcher.execute_commands([command])\nscreenshot_url = results[0].result\n```\n\n---\n\n## Data Collection Server Tools (Port 8020)\n\nThe Data Collection Server provides read-only tools for gathering device information.\n\n### 1. capture_screenshot - Capture Device Screenshot\n\n**Purpose**: Take screenshot from Android device and return as base64-encoded image.\n\n#### Tool Specification\n\n```python\ntool_name = \"capture_screenshot\"\nparameters = {} # No parameters required\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant ADB\n participant Device\n \n Agent->>MCP: capture_screenshot()\n MCP->>ADB: screencap -p /sdcard/screen_temp.png\n ADB->>Device: Execute screenshot\n Device-->>ADB: Screenshot saved\n \n ADB->>Device: pull /sdcard/screen_temp.png\n Device-->>ADB: PNG file\n \n MCP->>MCP: Encode to base64\n MCP->>ADB: rm /sdcard/screen_temp.png\n MCP-->>Agent: data:image/png;base64,...\n```\n\n#### Result Format\n\n```python\n# Direct base64 data URI string (not a dict)\n\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA...\"\n```\n\n#### Use Cases\n\n| Use Case | Description |\n|----------|-------------|\n| **UI Analysis** | Understand current screen state |\n| **Visual Context** | Provide screenshots to LLM for decision making |\n| **Debugging** | Capture UI state at each step |\n| **Annotation Base** | Base image for control labeling |\n\n#### Error Handling\n\n```python\n# Failures return as exceptions\ntry:\n screenshot_url = await capture_screenshot()\nexcept Exception as e:\n # \"Failed to capture screenshot on device\"\n # \"Failed to pull screenshot from device\"\n pass\n```\n\n---\n\n### 2. get_ui_tree - Get UI Hierarchy XML\n\n**Purpose**: Retrieve the complete UI hierarchy in XML format for detailed UI structure analysis.\n\n#### Tool Specification\n\n```python\ntool_name = \"get_ui_tree\"\nparameters = {} # No parameters required\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant ADB\n participant Device\n \n Agent->>MCP: get_ui_tree()\n MCP->>ADB: uiautomator dump /sdcard/window_dump.xml\n ADB->>Device: Dump UI hierarchy\n Device-->>ADB: XML created\n \n ADB->>Device: cat /sdcard/window_dump.xml\n Device-->>ADB: XML content\n ADB-->>MCP: XML string\n \n MCP->>MCP: Cache UI tree (5s TTL)\n MCP-->>Agent: UI tree dictionary\n```\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"ui_tree\": \"\"\"\n \n \n \n ...\n \n \"\"\",\n \"format\": \"xml\"\n}\n```\n\n#### Use Cases\n\n- Advanced UI analysis requiring full hierarchy\n- Custom control parsing logic\n- Debugging UI structure\n- Extracting accessibility information\n\n---\n\n### 3. get_device_info - Get Device Information\n\n**Purpose**: Gather comprehensive device information including model, Android version, screen size, and battery status.\n\n#### Tool Specification\n\n```python\ntool_name = \"get_device_info\"\nparameters = {} # No parameters required\n```\n\n#### Information Collected\n\n| Info Type | ADB Command | Data Returned |\n|-----------|-------------|---------------|\n| **Model** | `getprop ro.product.model` | Device model name |\n| **Android Version** | `getprop ro.build.version.release` | Android version (e.g., \"13\") |\n| **SDK Version** | `getprop ro.build.version.sdk` | API level (e.g., \"33\") |\n| **Screen Size** | `wm size` | Resolution (e.g., \"Physical size: 1080x2400\") |\n| **Screen Density** | `wm density` | DPI (e.g., \"Physical density: 420\") |\n| **Battery Level** | `dumpsys battery` | Battery percentage |\n| **Battery Status** | `dumpsys battery` | Charging status |\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"device_info\": {\n \"model\": \"Pixel 6\",\n \"android_version\": \"13\",\n \"sdk_version\": \"33\",\n \"screen_size\": \"Physical size: 1080x2400\",\n \"screen_density\": \"Physical density: 420\",\n \"battery_level\": \"85\",\n \"battery_status\": \"2\" # 2 = Charging, 3 = Discharging\n },\n \"from_cache\": False # True if returned from cache\n}\n```\n\n**Caching**: Device info is cached for 60 seconds as it changes infrequently.\n\n---\n\n### 4. get_mobile_app_target_info - List Installed Apps\n\n**Purpose**: Retrieve list of installed applications as TargetInfo objects.\n\n#### Tool Specification\n\n```python\ntool_name = \"get_mobile_app_target_info\"\nparameters = {\n \"filter\": \"\", # Filter pattern (optional)\n \"include_system_apps\": False, # Include system apps (default: False)\n \"force_refresh\": False # Bypass cache (default: False)\n}\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant Cache\n participant ADB\n participant Device\n \n Agent->>MCP: get_mobile_app_target_info(include_system_apps=False)\n \n alt Cache Hit (not forced refresh)\n MCP->>Cache: Check cache (5min TTL)\n Cache-->>MCP: Cached app list\n MCP-->>Agent: Apps from cache\n else Cache Miss\n MCP->>ADB: pm list packages -3\n ADB->>Device: List user-installed packages\n Device-->>ADB: Package list\n ADB-->>MCP: Packages\n \n MCP->>MCP: Parse to TargetInfo objects\n MCP->>Cache: Update cache\n MCP-->>Agent: App list\n end\n```\n\n#### Result Format\n\n```python\n[\n {\n \"id\": \"1\",\n \"name\": \"com.android.chrome\",\n \"package\": \"com.android.chrome\"\n },\n {\n \"id\": \"2\",\n \"name\": \"com.google.android.apps.maps\",\n \"package\": \"com.google.android.apps.maps\"\n },\n {\n \"id\": \"3\",\n \"name\": \"com.whatsapp\",\n \"package\": \"com.whatsapp\"\n }\n]\n```\n\n**Notes**:\n- `id`: Sequential number for LLM reference\n- `name`: Package name (display name not available via simple ADB)\n- `package`: Full package identifier\n\n**Caching**: Apps list is cached for 5 minutes to reduce overhead.\n\n---\n\n### 5. get_app_window_controls_target_info - Get UI Controls\n\n**Purpose**: Extract UI controls from current screen with IDs for precise interaction.\n\n#### Tool Specification\n\n```python\ntool_name = \"get_app_window_controls_target_info\"\nparameters = {\n \"force_refresh\": False # Bypass cache (default: False)\n}\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant Cache\n participant ADB\n participant Device\n \n Agent->>MCP: get_app_window_controls_target_info()\n \n alt Cache Hit (not forced refresh)\n MCP->>Cache: Check cache (5s TTL)\n Cache-->>MCP: Cached controls\n MCP-->>Agent: Controls from cache\n else Cache Miss\n MCP->>ADB: uiautomator dump /sdcard/window_dump.xml\n ADB->>Device: Dump UI\n Device-->>ADB: XML file\n \n ADB->>Device: cat /sdcard/window_dump.xml\n Device-->>ADB: XML content\n ADB-->>MCP: UI hierarchy\n \n MCP->>MCP: Parse XML\n MCP->>MCP: Filter meaningful controls\n MCP->>MCP: Validate rectangles\n MCP->>MCP: Assign sequential IDs\n MCP->>Cache: Update cache\n MCP-->>Agent: Controls list\n end\n```\n\n#### Control Selection Criteria\n\nControls are included if they meet any of these criteria:\n\n- `clickable=\"true\"` - Can be tapped\n- `long-clickable=\"true\"` - Supports long-press\n- `scrollable=\"true\"` - Can be scrolled\n- `checkable=\"true\"` - Checkbox or toggle\n- Has `text` or `content-desc` - Has label\n- Type includes \"Edit\", \"Button\" - Input or action element\n\n#### Rectangle Validation\n\nControls with invalid rectangles are filtered out:\n\n```python\n# Bounds format: [left, top, right, bottom]\n# Valid rectangle must have:\n# - right > left (positive width)\n# - bottom > top (positive height)\n# - All coordinates > 0\nif right <= left or bottom <= top or right == 0 or bottom == 0:\n skip_control() # Invalid rectangle\n```\n\n#### Result Format\n\n```python\n[\n {\n \"id\": \"1\",\n \"name\": \"Search\",\n \"type\": \"EditText\",\n \"rect\": [48, 96, 912, 192] # [left, top, right, bottom] in pixels\n },\n {\n \"id\": \"2\",\n \"name\": \"Search\",\n \"type\": \"ImageButton\",\n \"rect\": [912, 96, 1032, 192]\n },\n {\n \"id\": \"3\",\n \"name\": \"Maps\",\n \"type\": \"TextView\",\n \"rect\": [0, 216, 1080, 360]\n }\n]\n```\n\n**Caching**: Controls are cached for 5 seconds but **automatically invalidated** after any action (UI likely changed).\n\n---\n\n## Action Server Tools (Port 8021)\n\nThe Action Server provides tools for device control and manipulation.\n\n### 6. tap - Tap at Coordinates\n\n**Purpose**: Perform tap/click action at specified screen coordinates.\n\n#### Tool Specification\n\n```python\ntool_name = \"tap\"\nparameters = {\n \"x\": 480, # X coordinate (pixels from left)\n \"y\": 240 # Y coordinate (pixels from top)\n}\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant ADB\n participant Device\n \n Agent->>MCP: tap(x=480, y=240)\n MCP->>ADB: input tap 480 240\n ADB->>Device: Inject tap event\n Device-->>ADB: Success\n ADB-->>MCP: Success\n \n MCP->>MCP: Invalidate controls cache\n MCP-->>Agent: Result\n```\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"action\": \"tap(480, 240)\",\n \"output\": \"\",\n \"error\": \"\"\n}\n```\n\n**Cache Invalidation**: Automatically invalidates control cache after tap (UI likely changed).\n\n---\n\n### 7. swipe - Swipe Gesture\n\n**Purpose**: Perform swipe gesture from start to end coordinates.\n\n#### Tool Specification\n\n```python\ntool_name = \"swipe\"\nparameters = {\n \"start_x\": 500,\n \"start_y\": 1500,\n \"end_x\": 500,\n \"end_y\": 500,\n \"duration\": 300 # milliseconds (default: 300)\n}\n```\n\n#### Common Use Cases\n\n| Use Case | Start | End | Description |\n|----------|-------|-----|-------------|\n| **Scroll Up** | (500, 1500) | (500, 500) | Swipe from bottom to top |\n| **Scroll Down** | (500, 500) | (500, 1500) | Swipe from top to bottom |\n| **Scroll Left** | (900, 600) | (100, 600) | Swipe from right to left |\n| **Scroll Right** | (100, 600) | (900, 600) | Swipe from left to right |\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"action\": \"swipe(500,1500)->(500,500) in 300ms\",\n \"output\": \"\",\n \"error\": \"\"\n}\n```\n\n**Cache Invalidation**: Automatically invalidates control cache after swipe.\n\n---\n\n### 8. type_text - Type Text into Control\n\n**Purpose**: Type text into a specific input field control.\n\n#### Tool Specification\n\n```python\ntool_name = \"type_text\"\nparameters = {\n \"text\": \"hello world\",\n \"control_id\": \"5\", # REQUIRED: Control ID from get_app_window_controls_target_info\n \"control_name\": \"Search\", # REQUIRED: Control name (must match)\n \"clear_current_text\": False # Clear existing text first (default: False)\n}\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant Cache\n participant ADB\n participant Device\n \n Agent->>MCP: type_text(text=\"hello\", control_id=\"5\", control_name=\"Search\")\n \n MCP->>Cache: Get control by ID\n Cache-->>MCP: Control with rect\n \n MCP->>MCP: Calculate center position\n MCP->>ADB: input tap x y (focus control)\n ADB->>Device: Tap input field\n \n alt clear_current_text=True\n MCP->>ADB: input keyevent KEYCODE_DEL (x50)\n ADB->>Device: Delete existing text\n end\n \n MCP->>MCP: Escape text (spaces -> %s)\n MCP->>ADB: input text hello%sworld\n ADB->>Device: Type text\n Device-->>ADB: Success\n \n MCP->>MCP: Invalidate controls cache\n MCP-->>Agent: Result\n```\n\n#### Important Notes\n\n!!!warning \"Control ID Requirement\"\n The `control_id` parameter is **REQUIRED**. You must:\n \n 1. Call `get_app_window_controls_target_info` first\n 2. Identify the input field control\n 3. Use its `id` and `name` in `type_text`\n \n The tool will:\n - Verify the control exists in cache\n - Click the control to focus it\n - Then type the text\n\n**Text Escaping**: Spaces are automatically converted to `%s` for Android input shell compatibility.\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"action\": \"type_text(text='hello world', control_id='5', control_name='Search')\",\n \"message\": \"Clicked control 'Search' at (480, 144) | Typed text: 'hello world'\",\n \"control_info\": {\n \"id\": \"5\",\n \"name\": \"Search\",\n \"type\": \"EditText\"\n }\n}\n```\n\n---\n\n### 9. launch_app - Launch Application\n\n**Purpose**: Launch an application by package name or app ID.\n\n#### Tool Specification\n\n```python\ntool_name = \"launch_app\"\nparameters = {\n \"package_name\": \"com.google.android.apps.maps\", # Package name\n \"id\": \"2\" # Optional: App ID from get_mobile_app_target_info\n}\n```\n\n#### Usage Modes\n\n**Mode 1: Launch by package name**\n\n```python\nlaunch_app(package_name=\"com.android.settings\")\n```\n\n**Mode 2: Launch from cached app list**\n\n```python\n# First call get_mobile_app_target_info to cache apps\n# Then use app ID from the list\nlaunch_app(package_name=\"com.android.settings\", id=\"5\")\n```\n\n**Mode 3: Launch by app name (fuzzy search)**\n\n```python\n# If package_name doesn't contain \".\", search by name\nlaunch_app(package_name=\"Maps\") # Finds \"com.google.android.apps.maps\"\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant ADB\n participant Device\n \n Agent->>MCP: launch_app(package_name=\"com.google.android.apps.maps\")\n \n alt ID provided\n MCP->>MCP: Verify ID in cache\n MCP->>MCP: Get package from cache\n else Name only (no dots)\n MCP->>ADB: pm list packages\n MCP->>MCP: Search for matching package\n end\n \n MCP->>ADB: monkey -p com.google.android.apps.maps -c android.intent.category.LAUNCHER 1\n ADB->>Device: Launch app\n Device-->>ADB: App started\n ADB-->>MCP: Success\n MCP-->>Agent: Result\n```\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"message\": \"Launched com.google.android.apps.maps\",\n \"package_name\": \"com.google.android.apps.maps\",\n \"output\": \"Events injected: 1\",\n \"error\": \"\",\n \"app_info\": { # If ID was provided\n \"id\": \"2\",\n \"name\": \"com.google.android.apps.maps\",\n \"package\": \"com.google.android.apps.maps\"\n }\n}\n```\n\n---\n\n### 10. press_key - Press Hardware/Software Key\n\n**Purpose**: Press a hardware or software key for navigation and system actions.\n\n#### Tool Specification\n\n```python\ntool_name = \"press_key\"\nparameters = {\n \"key_code\": \"KEYCODE_BACK\" # Key code name\n}\n```\n\n#### Common Key Codes\n\n| Key Code | Description | Use Case |\n|----------|-------------|----------|\n| `KEYCODE_HOME` | Home button | Return to home screen |\n| `KEYCODE_BACK` | Back button | Navigate back |\n| `KEYCODE_MENU` | Menu button | Open options menu |\n| `KEYCODE_ENTER` | Enter key | Submit form |\n| `KEYCODE_DEL` | Delete key | Delete character |\n| `KEYCODE_APP_SWITCH` | Recent apps | Switch between apps |\n| `KEYCODE_POWER` | Power button | Lock screen |\n| `KEYCODE_VOLUME_UP` | Volume up | Increase volume |\n| `KEYCODE_VOLUME_DOWN` | Volume down | Decrease volume |\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"action\": \"press_key(KEYCODE_BACK)\",\n \"output\": \"\",\n \"error\": \"\"\n}\n```\n\n---\n\n### 11. click_control - Click Control by ID\n\n**Purpose**: Click a UI control by its ID from the cached control list.\n\n#### Tool Specification\n\n```python\ntool_name = \"click_control\"\nparameters = {\n \"control_id\": \"5\", # REQUIRED: Control ID from get_app_window_controls_target_info\n \"control_name\": \"Search Button\" # REQUIRED: Control name (must match)\n}\n```\n\n#### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Agent\n participant MCP\n participant Cache\n participant ADB\n participant Device\n \n Agent->>MCP: click_control(control_id=\"5\", control_name=\"Search\")\n \n MCP->>Cache: Get control by ID \"5\"\n Cache-->>MCP: Control with rect [48,96,912,192]\n \n MCP->>MCP: Verify name matches\n MCP->>MCP: Calculate center: x=(48+912)/2, y=(96+192)/2\n \n MCP->>ADB: input tap 480 144\n ADB->>Device: Tap at (480, 144)\n Device-->>ADB: Success\n \n MCP->>MCP: Invalidate controls cache\n MCP-->>Agent: Result\n```\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"action\": \"click_control(id=5, name=Search)\",\n \"message\": \"Clicked control 'Search' at (480, 144)\",\n \"control_info\": {\n \"id\": \"5\",\n \"name\": \"Search\",\n \"type\": \"EditText\",\n \"rect\": [48, 96, 912, 192]\n }\n}\n```\n\n**Name Verification**: If the provided `control_name` doesn't match the cached control's name, a warning is returned but the action still executes using the ID.\n\n---\n\n### 12. wait - Wait/Sleep\n\n**Purpose**: Wait for a specified duration.\n\n#### Tool Specification\n\n```python\ntool_name = \"wait\"\nparameters = {\n \"seconds\": 1.0 # Duration in seconds (can be decimal)\n}\n```\n\n#### Use Cases\n\n- Wait for app to load\n- Wait for animation to complete\n- Wait for UI transition\n- Pace actions for stability\n\n#### Examples\n\n```python\nwait(seconds=1.0) # Wait 1 second\nwait(seconds=0.5) # Wait 500ms\nwait(seconds=2.5) # Wait 2.5 seconds\n```\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"action\": \"wait(1.0s)\",\n \"message\": \"Waited for 1.0 seconds\"\n}\n```\n\n**Limits**: \n- Minimum: 0 seconds\n- Maximum: 60 seconds\n\n---\n\n### 13. invalidate_cache - Manual Cache Invalidation\n\n**Purpose**: Manually invalidate cached data to force refresh on next query.\n\n#### Tool Specification\n\n```python\ntool_name = \"invalidate_cache\"\nparameters = {\n \"cache_type\": \"all\" # \"controls\", \"apps\", \"ui_tree\", \"device_info\", or \"all\"\n}\n```\n\n#### Cache Types\n\n| Cache Type | Description | Auto-Invalidated |\n|------------|-------------|------------------|\n| `controls` | UI controls list | ✓ After actions |\n| `apps` | Installed apps list | ✗ Never |\n| `ui_tree` | UI hierarchy XML | ✗ Never |\n| `device_info` | Device information | ✗ Never |\n| `all` | All caches | Varies |\n\n#### Result Format\n\n```python\n{\n \"success\": True,\n \"message\": \"Controls cache invalidated\"\n}\n```\n\n**Use Cases**:\n- Manually refresh apps list after installing/uninstalling\n- Force UI tree refresh after significant screen change\n- Debug caching issues\n\n---\n\n## Command Execution Pipeline\n\n### Atomic Building Blocks\n\nThe MCP tools serve as atomic operations for mobile task execution:\n\n```mermaid\ngraph TD\n A[User Request] --> B[Data Collection Phase]\n B --> B1[capture_screenshot]\n B --> B2[get_mobile_app_target_info]\n B --> B3[get_app_window_controls_target_info]\n \n B1 --> C[LLM Reasoning]\n B2 --> C\n B3 --> C\n \n C --> D{Select Action}\n D -->|Launch| E[launch_app]\n D -->|Type| F[type_text]\n D -->|Click| G[click_control]\n D -->|Swipe| H[swipe]\n D -->|Navigate| I[press_key]\n D -->|Wait| J[wait]\n \n E --> K[Capture Result]\n F --> K\n G --> K\n H --> K\n I --> K\n J --> K\n \n K --> L[Update Memory]\n L --> M{Task Complete?}\n M -->|No| B\n M -->|Yes| N[FINISH]\n```\n\n### Command Composition\n\nMobileAgent executes commands sequentially, building on previous results:\n\n```python\n# Round 1: Capture UI and launch app\n{\n \"action\": {\n \"function\": \"launch_app\",\n \"arguments\": {\"package_name\": \"com.google.android.apps.maps\", \"id\": \"2\"}\n }\n}\n# Result: Maps launched\n\n# Round 2: Capture new UI, identify search field\n{\n \"action\": {\n \"function\": \"click_control\",\n \"arguments\": {\"control_id\": \"5\", \"control_name\": \"Search\"}\n }\n}\n# Result: Search field focused\n\n# Round 3: Type query\n{\n \"action\": {\n \"function\": \"type_text\",\n \"arguments\": {\n \"text\": \"restaurants\",\n \"control_id\": \"5\",\n \"control_name\": \"Search\"\n }\n }\n}\n# Result: Text entered\n```\n\n---\n\n## Best Practices\n\n### Data Collection Tools\n\n- Use `get_app_window_controls_target_info` before every action to get fresh control IDs\n- Cache is your friend: don't force refresh unless necessary\n- Annotated screenshots help LLM identify controls precisely\n\n### Action Tools\n\n!!!success \"Action Best Practices\"\n - **Always** call `get_app_window_controls_target_info` before `click_control` or `type_text`\n - Use control IDs instead of coordinates for robustness\n - Add `wait` after actions that trigger UI changes (app launch, navigation)\n - Check `success` field in results before considering action successful\n - Use `press_key(KEYCODE_BACK)` for navigation instead of screen taps when possible\n\n### Caching\n\n- Controls cache: 5 seconds TTL, invalidated after actions\n- Apps cache: 5 minutes TTL, manually invalidate if apps change\n- Device info cache: 60 seconds TTL, useful for metadata\n\n### Error Handling\n\n```python\n# Always check success field\nresult = await click_control(control_id=\"5\", control_name=\"Search\")\nif not result[\"success\"]:\n # Handle error: control not found, device disconnected, etc.\n pass\n```\n\n---\n\n## Implementation Location\n\nThe MCP server implementation can be found in:\n\n```\nufo/client/mcp/http_servers/\n└── mobile_mcp_server.py\n```\n\nKey components:\n\n- `MobileServerState`: Singleton state manager for caching\n- `create_mobile_data_collection_server()`: Data collection server (port 8020)\n- `create_mobile_action_server()`: Action server (port 8021)\n\n---\n\n## Comparison with Other Agent Commands\n\n| Agent | Command Types | Execution Layer | Visual Context | Result Format |\n|-------|--------------|-----------------|----------------|---------------|\n| **MobileAgent** | UI + Apps + Touch | MCP (ADB) | ✓ Screenshots + Controls | success/message/control_info |\n| **LinuxAgent** | CLI + SysInfo | MCP (SSH) | ✗ Text-only | success/exit_code/stdout/stderr |\n| **AppAgent** | UI + API | Automator + MCP | ✓ Screenshots + Controls | UI state + API responses |\n\nMobileAgent's command set reflects the mobile environment:\n\n- **Touch-based**: tap, swipe instead of click, drag\n- **Visual**: Screenshots are essential for UI understanding\n- **App-centric**: Focus on app launching and switching\n- **Control-based**: Precise control IDs instead of coordinates\n- **Cached**: Aggressive caching to reduce ADB overhead\n\n---\n\n## Next Steps\n\n- [State Machine](state.md) - Understand how command execution fits into the FSM\n- [Processing Strategy](strategy.md) - See how commands are integrated into the 4-phase pipeline\n- [Overview](overview.md) - Return to MobileAgent architecture overview\n- [As Galaxy Device](as_galaxy_device.md) - Configure MobileAgent for multi-device workflows\n"
},
{
"path": "documents/docs/mobile/overview.md",
"content": "# MobileAgent: Android Task Executor\n\n**MobileAgent** is a specialized agent designed for executing tasks on Android mobile devices. It leverages the layered FSM architecture and server-client design to perform intelligent, iterative task execution in mobile environments through UI interaction and app control.\n\n**Quick Links:**\n\n- **New to Mobile Agent?** Start with the [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up your first Android device agent in 10 minutes\n- **Using as Sub-Agent in Galaxy?** See [Using Mobile Agent as Galaxy Device](as_galaxy_device.md)\n- **Deep Dive:** [State Machine](state.md) • [Processing Strategy](strategy.md) • [MCP Commands](commands.md)\n\n## Architecture Overview\n\nMobileAgent operates as a single-agent instance that interacts with Android devices through UI controls and app management. Unlike the two-tier architecture of UFO (HostAgent + AppAgent), MobileAgent uses a simplified single-agent model optimized for mobile device automation, similar to LinuxAgent but with visual interface support.\n\n## Core Responsibilities\n\nMobileAgent provides the following capabilities for Android device automation:\n\n### UI Interaction\n\nMobileAgent interprets user requests and translates them into appropriate UI interactions on Android devices through screenshots analysis and control manipulation.\n\n**Example:** User request \"Search for restaurants on Maps\" becomes:\n\n1. Take screenshot and identify app icons\n2. Launch Google Maps app\n3. Identify search field control\n4. Type \"restaurants\" into search field\n5. Tap search button\n\n### Visual Context Understanding\n\nThe agent captures and analyzes device screenshots to understand the current UI state:\n\n- Screenshot capture (clean and annotated)\n- Control identification and labeling\n- UI hierarchy parsing\n- App detection and recognition\n\n### App Management\n\nMobileAgent can manage installed applications:\n\n- List installed apps (user apps and system apps)\n- Launch apps by package name or app name\n- Switch between apps\n- Monitor current app state\n\n### Iterative Task Execution\n\nMobileAgent executes tasks iteratively, evaluating execution outcomes at each step and determining the next action based on results and LLM reasoning.\n\n### Error Handling and Recovery\n\nThe agent monitors action execution results and can adapt its strategy when errors occur, such as controls not found or apps not responding.\n\n## Key Characteristics\n\n- **Scope**: Single Android device (UI-based automation)\n- **Lifecycle**: One instance per task session\n- **Hierarchy**: Standalone agent (no child agents)\n- **Communication**: MCP server integration via ADB\n- **Control**: 3-state finite state machine with 4-phase processing pipeline\n- **Visual**: Screenshot-based UI understanding with control annotation\n\n## Execution Workflow\n\n```mermaid\nsequenceDiagram\n participant User\n participant MobileAgent\n participant LLM\n participant MCPServer\n participant Android\n \n User->>MobileAgent: \"Search for restaurants on Maps\"\n MobileAgent->>MobileAgent: State: CONTINUE\n \n MobileAgent->>MCPServer: Capture screenshot\n MCPServer->>Android: Take screenshot via ADB\n Android-->>MCPServer: Screenshot PNG\n MCPServer-->>MobileAgent: Base64 screenshot\n \n MobileAgent->>MCPServer: Get installed apps\n MCPServer->>Android: List packages via ADB\n Android-->>MCPServer: App list\n MCPServer-->>MobileAgent: Installed apps\n \n MobileAgent->>MCPServer: Get current controls\n MCPServer->>Android: UI dump via ADB\n Android-->>MCPServer: UI hierarchy XML\n MCPServer-->>MobileAgent: Control list with IDs\n \n MobileAgent->>LLM: Send prompt with screenshot + apps + controls\n LLM-->>MobileAgent: Action: launch_app(package=\"com.google.android.apps.maps\")\n \n MobileAgent->>MCPServer: launch_app\n MCPServer->>Android: Start app via ADB\n Android-->>MCPServer: App launched\n MCPServer-->>MobileAgent: Success\n \n MobileAgent->>MobileAgent: Update memory\n MobileAgent->>MobileAgent: State: CONTINUE\n \n Note over MobileAgent: Next round with new screenshot\n \n MobileAgent->>MCPServer: Capture new screenshot + controls\n MobileAgent->>LLM: Prompt with new UI state\n LLM-->>MobileAgent: Action: type_text(control_id=\"5\", text=\"restaurants\")\n \n MobileAgent->>MCPServer: click_control + type_text\n MCPServer->>Android: Execute actions via ADB\n Android-->>MCPServer: Actions completed\n MCPServer-->>MobileAgent: Success\n \n MobileAgent->>MobileAgent: State: FINISH\n MobileAgent-->>User: Task completed\n```\n\n## Comparison with Other Agents\n\n| Aspect | MobileAgent | LinuxAgent | AppAgent |\n|--------|-------------|------------|----------|\n| **Platform** | Android Mobile | Linux (CLI) | Windows Applications |\n| **States** | 3 (CONTINUE, FINISH, FAIL) | 3 states | 6 states |\n| **Architecture** | Single-agent | Single-agent | Child executor |\n| **Interface** | Mobile UI (touch-based) | Command-line | Desktop GUI |\n| **Processing Phases** | 4 phases (with DATA_COLLECTION) | 3 phases | 4 phases |\n| **Visual** | ✓ Screenshots + Annotations | ✗ Text-only | ✓ Screenshots + Annotations |\n| **MCP Tools** | UI controls + App management | CLI commands | UI controls + API |\n| **Input Method** | Touch (tap, swipe, type) | Keyboard commands | Mouse + Keyboard |\n| **Control Identification** | UI hierarchy + bounds | N/A | UI Automation API |\n\n## Design Principles\n\nMobileAgent exemplifies mobile-specific design considerations:\n\n- **Visual Context**: Screenshot-based UI understanding with control annotation for precise interaction\n- **Control Caching**: Efficient control information caching to reduce ADB overhead\n- **Touch-based Interaction**: Specialized actions for mobile gestures (tap, swipe, long-press)\n- **App-centric Navigation**: Focus on app launching and switching rather than window management\n- **Minimal State Set**: 3-state FSM for deterministic control flow\n- **Modular Strategies**: Clear separation between data collection, LLM interaction, action execution, and memory updates\n- **Traceable Execution**: Complete logging of screenshots, actions, and state transitions\n\n## Deep Dive Topics\n\nExplore the detailed architecture and implementation:\n\n- [State Machine](state.md) - 3-state FSM lifecycle and transitions\n- [Processing Strategy](strategy.md) - 4-phase pipeline (Data Collection, LLM, Action, Memory)\n- [MCP Commands](commands.md) - Mobile UI interaction and app management commands\n- [As Galaxy Device](as_galaxy_device.md) - Using Mobile Agent in multi-device workflows\n\n## Technology Stack\n\n### ADB (Android Debug Bridge)\n\nMobileAgent relies on ADB for all device interactions:\n\n- **Screenshot Capture**: `adb shell screencap` for visual context\n- **UI Hierarchy**: `adb shell uiautomator dump` for control information\n- **Touch Input**: `adb shell input tap/swipe` for user interaction\n- **Text Input**: `adb shell input text` for typing\n- **App Control**: `adb shell monkey` for app launching\n- **Device Info**: `adb shell getprop` for device properties\n\n### MCP Server Architecture\n\nTwo separate MCP servers handle different responsibilities:\n\n1. **Data Collection Server** (Port 8020):\n - Screenshot capture\n - UI tree retrieval\n - App list collection\n - Control information gathering\n - Device information\n\n2. **Action Server** (Port 8021):\n - Touch actions (tap, swipe)\n - Text input\n - App launching\n - Key press events\n - Control clicking\n\nBoth servers share a singleton `MobileServerState` for efficient caching and coordination.\n\n## Use Cases\n\nMobileAgent is ideal for:\n\n- **Mobile App Testing**: Automated UI testing across different apps\n- **Cross-App Workflows**: Tasks spanning multiple mobile applications\n- **Data Entry**: Automated form filling and text input\n- **App Navigation**: Exploring and interacting with mobile UIs\n- **Mobile Productivity**: Automating repetitive mobile tasks\n- **Cross-Device Workflows**: As a sub-agent in Galaxy multi-device orchestration\n\n!!!tip \"Galaxy Integration\"\n MobileAgent can serve as a device agent in Galaxy's multi-device orchestration framework, executing Android-specific tasks as part of cross-platform workflows alongside Windows and Linux devices.\n \n See [Using Mobile Agent as Galaxy Device](as_galaxy_device.md) for configuration details.\n\n## Requirements\n\n### Hardware\n\n- Android device or emulator\n- USB connection (for physical devices) or network connection (for emulators)\n- USB debugging enabled on the device\n\n### Software\n\n- ADB (Android Debug Bridge) installed and accessible\n- Android device with API level 21+ (Android 5.0+)\n- Python 3.8+\n- Required Python packages (see requirements.txt)\n\n## Implementation Location\n\nThe MobileAgent implementation can be found in:\n\n```\nufo/\n├── agents/\n│ ├── agent/\n│ │ └── customized_agent.py # MobileAgent class definition\n│ ├── states/\n│ │ └── mobile_agent_state.py # State machine implementation\n│ └── processors/\n│ ├── customized/\n│ │ └── customized_agent_processor.py # MobileAgentProcessor\n│ └── strategies/\n│ └── mobile_agent_strategy.py # Processing strategies\n├── prompter/\n│ └── customized/\n│ └── mobile_agent_prompter.py # Prompt construction\n├── module/\n│ └── sessions/\n│ └── mobile_session.py # Session management\n└── client/\n └── mcp/\n └── http_servers/\n └── mobile_mcp_server.py # MCP server implementation\n```\n\n## Next Steps\n\nTo understand MobileAgent's complete architecture:\n\n1. [State Machine](state.md) - Learn about the 3-state FSM\n2. [Processing Strategy](strategy.md) - Understand the 4-phase pipeline\n3. [MCP Commands](commands.md) - Explore mobile UI interaction commands\n4. [As Galaxy Device](as_galaxy_device.md) - Configure for multi-device workflows\n\nFor deployment and configuration, see the Quick Start Guide (coming soon).\n"
},
{
"path": "documents/docs/mobile/state.md",
"content": "# MobileAgent State Machine\n\nMobileAgent uses a **3-state finite state machine (FSM)** to manage Android device task execution flow. The minimal state set captures essential execution progression while maintaining simplicity and predictability. States transition based on LLM decisions and action execution results.\n\n> **📖 Related Documentation:**\n> \n> - [Mobile Agent Overview](overview.md) - Architecture and core responsibilities\n> - [Processing Strategy](strategy.md) - 4-phase pipeline execution in CONTINUE state\n> - [MCP Commands](commands.md) - Available mobile interaction commands\n> - [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up your first Mobile Agent\n\n## State Machine Architecture\n\n### State Enumeration\n\n```python\nclass MobileAgentStatus(Enum):\n \"\"\"Store the status of the mobile agent\"\"\"\n CONTINUE = \"CONTINUE\" # Task is ongoing, requires further actions\n FINISH = \"FINISH\" # Task completed successfully\n FAIL = \"FAIL\" # Task cannot proceed, unrecoverable error\n```\n\n### State Management\n\nMobileAgent states are managed by `MobileAgentStateManager`, which implements the agent state registry pattern:\n\n```python\nclass MobileAgentStateManager(AgentStateManager):\n \"\"\"Manages the states of the mobile agent\"\"\"\n _state_mapping: Dict[str, Type[MobileAgentState]] = {}\n \n @property\n def none_state(self) -> AgentState:\n return NoneMobileAgentState()\n```\n\nAll MobileAgent states are registered using the `@MobileAgentStateManager.register` decorator, enabling dynamic state lookup by name.\n\n## State Transition Diagram\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE: Start Task\n \n CONTINUE --> CONTINUE: More Actions Needed (LLM returns CONTINUE)\n CONTINUE --> FINISH: Task Complete (LLM returns FINISH)\n CONTINUE --> FAIL: Unrecoverable Error (LLM returns FAIL or Exception)\n \n FINISH --> [*]: Session Ends\n FAIL --> FINISH: Cleanup\n FINISH --> [*]: Session Ends\n \n note right of CONTINUE\n Active execution state:\n - Capture screenshots\n - Collect UI controls\n - Get LLM decision\n - Execute actions\n - Update memory\n end note\n \n note right of FINISH\n Terminal state:\n - Task completed successfully\n - Results available in memory\n - Agent can be terminated\n end note\n \n note right of FAIL\n Error terminal state:\n - Unrecoverable error occurred\n - Error details logged\n - Transitions to FINISH for cleanup\n end note\n```\n\n## State Definitions\n\n### 1. CONTINUE State\n\n**Purpose**: Active execution state where MobileAgent processes the user request and executes mobile actions.\n\n```python\n@MobileAgentStateManager.register\nclass ContinueMobileAgentState(MobileAgentState):\n \"\"\"The class for the continue mobile agent state\"\"\"\n \n async def handle(self, agent: \"MobileAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Execute the 4-phase processing pipeline\"\"\"\n await agent.process(context)\n \n def is_round_end(self) -> bool:\n return False # Round continues\n \n def is_subtask_end(self) -> bool:\n return False # Subtask continues\n \n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.CONTINUE.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Active |\n| **Processor Executed** | ✓ Yes (4 phases) |\n| **Round Ends** | No |\n| **Subtask Ends** | No |\n| **Duration** | Single round |\n| **Next States** | CONTINUE, FINISH, FAIL |\n\n**Behavior**:\n\n1. **Data Collection Phase**:\n - Captures device screenshot\n - Retrieves installed apps list\n - Collects current screen UI controls\n - Creates annotated screenshot with control IDs\n\n2. **LLM Interaction Phase**:\n - Constructs prompts with screenshots and control information\n - Gets next action from LLM\n - Parses and validates response\n\n3. **Action Execution Phase**:\n - Executes mobile actions (tap, swipe, type, launch app, etc.)\n - Captures execution results\n\n4. **Memory Update Phase**:\n - Updates memory with screenshots and action results\n - Stores control information for next round\n\n5. **State Determination**:\n - Analyzes LLM response for next state\n\n**State Transition Logic**:\n\n- **CONTINUE → CONTINUE**: Task requires more actions to complete (e.g., need to navigate through multiple screens)\n- **CONTINUE → FINISH**: LLM determines task is complete (e.g., successfully filled form and submitted)\n- **CONTINUE → FAIL**: Unrecoverable error encountered (e.g., required app not installed, control not found after multiple attempts)\n\n### 2. FINISH State\n\n**Purpose**: Terminal state indicating successful task completion.\n\n```python\n@MobileAgentStateManager.register\nclass FinishMobileAgentState(MobileAgentState):\n \"\"\"The class for the finish mobile agent state\"\"\"\n \n def next_agent(self, agent: \"MobileAgent\") -> \"MobileAgent\":\n return agent\n \n def next_state(self, agent: \"MobileAgent\") -> MobileAgentState:\n return FinishMobileAgentState() # Remains in FINISH\n \n def is_subtask_end(self) -> bool:\n return True # Subtask completed\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.FINISH.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Subtask Ends** | Yes |\n| **Duration** | Permanent |\n| **Next States** | FINISH (no transition) |\n\n**Behavior**:\n\n- Signals task completion to session manager\n- No further processing occurs\n- Agent instance can be terminated\n- Screenshots and action history available in memory\n\n**FINISH state is reached when**:\n\n- All required mobile actions have been executed successfully\n- The LLM determines the user request has been fulfilled\n- Target UI state has been achieved (e.g., form submitted, information displayed)\n- No errors or exceptions occurred during execution\n\n### 3. FAIL State\n\n**Purpose**: Terminal state indicating task failure due to unrecoverable errors.\n\n```python\n@MobileAgentStateManager.register\nclass FailMobileAgentState(MobileAgentState):\n \"\"\"The class for the fail mobile agent state\"\"\"\n \n def next_agent(self, agent: \"MobileAgent\") -> \"MobileAgent\":\n return agent\n \n def next_state(self, agent: \"MobileAgent\") -> MobileAgentState:\n return FinishMobileAgentState() # Transitions to FINISH for cleanup\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n def is_subtask_end(self) -> bool:\n return True # Subtask failed\n \n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.FAIL.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal (Error) |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Subtask Ends** | Yes |\n| **Duration** | Transitions to FINISH |\n| **Next States** | FINISH |\n\n**Behavior**:\n\n- Logs failure reason and context\n- Captures final screenshot for debugging\n- Transitions to FINISH state for cleanup\n- Session manager receives failure status\n\n!!!error \"Failure Conditions\"\n FAIL state is reached when:\n \n - **App Unavailable**: Required app is not installed or cannot be launched\n - **Control Not Found**: Target UI control cannot be located after multiple attempts\n - **Device Disconnected**: ADB connection lost during execution\n - **Permission Denied**: Required permissions not granted on device\n - **Timeout**: Actions take too long to complete\n - **LLM Explicit Failure**: LLM explicitly indicates task cannot be completed\n - **Repeated Action Failures**: Multiple consecutive actions fail\n\n**Error Recovery**:\n\nWhile FAIL is a terminal state, the error information is logged for debugging:\n\n```python\n# Example error logging in FAIL state\nagent.logger.error(f\"Mobile task failed: {error_message}\")\nagent.logger.debug(f\"Last action: {last_action}\")\nagent.logger.debug(f\"Current screenshot saved to: {screenshot_path}\")\nagent.logger.debug(f\"UI controls at failure: {current_controls}\")\n```\n\n## State Transition Rules\n\n### Transition Decision Logic\n\nState transitions are determined by the LLM's response in the **CONTINUE** state:\n\n```python\n# LLM returns status in response\nparsed_response = {\n \"action\": {\n \"function\": \"click_control\",\n \"arguments\": {\"control_id\": \"5\", \"control_name\": \"Search\"},\n \"status\": \"CONTINUE\" # or \"FINISH\" or \"FAIL\"\n },\n \"thought\": \"Need to click the search button to proceed\"\n}\n\n# Agent updates its status based on LLM decision\nagent.status = parsed_response[\"action\"][\"status\"]\nnext_state = MobileAgentStateManager().get_state(agent.status)\n```\n\n### Transition Matrix\n\n| Current State | Condition | Next State | Trigger |\n|---------------|-----------|------------|---------|\n| **CONTINUE** | LLM returns CONTINUE | CONTINUE | More actions needed (e.g., navigating multiple screens) |\n| **CONTINUE** | LLM returns FINISH | FINISH | Task completed (e.g., information found and displayed) |\n| **CONTINUE** | LLM returns FAIL | FAIL | Unrecoverable error (e.g., required control not available) |\n| **CONTINUE** | Exception raised | FAIL | System error (e.g., ADB disconnected) |\n| **FINISH** | Any | FINISH | No transition |\n| **FAIL** | Any | FINISH | Cleanup transition |\n\n## State-Specific Processing\n\n### CONTINUE State Processing Pipeline\n\nWhen in CONTINUE state, MobileAgent executes the full 4-phase pipeline:\n\n```mermaid\ngraph TD\n A[CONTINUE State] --> B[Phase 1: Data Collection]\n B --> B1[Capture Screenshot]\n B1 --> B2[Get Installed Apps]\n B2 --> B3[Get Current Controls]\n B3 --> B4[Create Annotated Screenshot]\n \n B4 --> C[Phase 2: LLM Interaction]\n C --> C1[Construct Prompt with Visual Context]\n C1 --> C2[Send to LLM]\n C2 --> C3[Parse Response]\n \n C3 --> D[Phase 3: Action Execution]\n D --> D1[Execute Mobile Action]\n D1 --> D2[Capture Result]\n \n D2 --> E[Phase 4: Memory Update]\n E --> E1[Store Screenshot]\n E1 --> E2[Store Action Result]\n E2 --> E3[Update Control Cache]\n \n E3 --> F{Check Status}\n F -->|CONTINUE| A\n F -->|FINISH| G[FINISH State]\n F -->|FAIL| H[FAIL State]\n```\n\n### Terminal States (FINISH / FAIL)\n\nTerminal states perform no processing:\n\n- **FINISH**: Clean termination, results and screenshots available in memory\n- **FAIL**: Error termination, error details and final screenshot logged\n\n## Deterministic Control Flow\n\nThe 3-state design ensures deterministic, traceable execution:\n\n- **Predictable Behavior**: Every execution path is well-defined\n- **Debuggability**: State transitions are logged with screenshots for visual debugging\n- **Testability**: Finite state space simplifies testing\n- **Maintainability**: Simple state set reduces complexity\n- **Visual Traceability**: Screenshots at each state provide visual execution history\n\n## Comparison with Other Agents\n\n| Agent | States | Complexity | Visual | Use Case |\n|-------|--------|------------|--------|----------|\n| **MobileAgent** | 3 | Minimal | ✓ Screenshots | Android mobile automation |\n| **LinuxAgent** | 3 | Minimal | ✗ Text-only | Linux CLI task execution |\n| **AppAgent** | 6 | Moderate | ✓ Screenshots | Windows app automation |\n| **HostAgent** | 7 | High | ✓ Screenshots | Desktop orchestration |\n\nMobileAgent's minimal 3-state design reflects its focused scope: execute mobile UI actions to fulfill user requests. The simplified state machine eliminates unnecessary complexity while maintaining robust error handling and completion detection, similar to LinuxAgent but with visual context support.\n\n## Mobile-Specific Considerations\n\n### Screenshot-Based State Tracking\n\nUnlike LinuxAgent (text-based) or AppAgent (Windows UI API), MobileAgent relies heavily on screenshots for state understanding:\n\n- Each CONTINUE round starts with a fresh screenshot\n- Annotated screenshots show control IDs for precise interaction\n- Screenshots are saved to memory for debugging and analysis\n- Visual context helps LLM understand current UI state\n\n### Control Caching\n\nMobileAgent caches control information to minimize ADB overhead:\n\n- Controls are cached for 5 seconds\n- Cache is invalidated after each action (UI likely changed)\n- Control dictionary enables quick lookup by ID\n- Reduces repeated UI tree parsing\n\n### Touch-Based Interaction\n\nState transitions in MobileAgent are triggered by touch actions rather than keyboard commands:\n\n- **Tap**: Primary interaction method\n- **Swipe**: For scrolling and gestures\n- **Type**: Text input (requires focused control)\n- **Long-press**: For context menus (planned)\n\n## Implementation Details\n\nThe state machine implementation can be found in:\n\n```\nufo/agents/states/mobile_agent_state.py\n```\n\nKey classes:\n\n- `MobileAgentStatus`: State enumeration (CONTINUE, FINISH, FAIL)\n- `MobileAgentStateManager`: State registry and lookup\n- `MobileAgentState`: Abstract base class\n- `ContinueMobileAgentState`: Active execution state with 4-phase pipeline\n- `FinishMobileAgentState`: Successful completion state\n- `FailMobileAgentState`: Error termination state\n- `NoneMobileAgentState`: Initial/undefined state\n\n## Next Steps\n\n- [Processing Strategy](strategy.md) - Understand the 4-phase processing pipeline executed in CONTINUE state\n- [MCP Commands](commands.md) - Explore mobile UI interaction and app management commands\n- [Overview](overview.md) - Return to MobileAgent architecture overview\n"
},
{
"path": "documents/docs/mobile/strategy.md",
"content": "# MobileAgent Processing Strategy\n\nMobileAgent executes a **4-phase processing pipeline** in the **CONTINUE** state. Each phase handles a specific aspect of mobile task execution: data collection (screenshots and controls), LLM decision making, action execution, and memory recording. This design separates visual context gathering from prompt construction, LLM reasoning, mobile action execution, and state updates, enhancing modularity and traceability.\n\n> **📖 Related Documentation:**\n> \n> - [Mobile Agent Overview](overview.md) - Architecture and core responsibilities\n> - [State Machine](state.md) - FSM states (this strategy runs in CONTINUE state)\n> - [MCP Commands](commands.md) - Available commands used in each phase\n> - [Quick Start Guide](../getting_started/quick_start_mobile.md) - Set up your first Mobile Agent\n\n## Strategy Assembly\n\nProcessing strategies are assembled and orchestrated by the `MobileAgentProcessor` class defined in `ufo/agents/processors/customized/customized_agent_processor.py`. The processor coordinates the 4-phase pipeline execution.\n\n### MobileAgentProcessor Overview\n\nThe `MobileAgentProcessor` extends `CustomizedProcessor` and manages the Mobile-specific workflow:\n\n```python\nclass MobileAgentProcessor(CustomizedProcessor):\n \"\"\"\n Processor for Mobile Android MCP Agent.\n Handles data collection, LLM interaction, and action execution for Android devices.\n \"\"\"\n \n def _setup_strategies(self) -> None:\n \"\"\"Setup processing strategies for Mobile Agent.\"\"\"\n \n # Phase 1: Data Collection (composed strategy - fail_fast=True)\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(\n strategies=[\n MobileScreenshotCaptureStrategy(fail_fast=True),\n MobileAppsCollectionStrategy(fail_fast=False),\n MobileControlsCollectionStrategy(fail_fast=False),\n ],\n name=\"MobileDataCollectionStrategy\",\n fail_fast=True,\n )\n \n # Phase 2: LLM Interaction (critical - fail_fast=True)\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n MobileLLMInteractionStrategy(fail_fast=True)\n )\n \n # Phase 3: Action Execution (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n MobileActionExecutionStrategy(fail_fast=False)\n )\n \n # Phase 4: Memory Update (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(fail_fast=False)\n )\n```\n\n### Strategy Registration\n\n| Phase | Strategy Class | fail_fast | Rationale |\n|-------|---------------|-----------|-----------|\n| **DATA_COLLECTION** | `ComposedStrategy` (3 sub-strategies) | ✓ True | Visual context is critical for mobile interaction |\n| **LLM_INTERACTION** | `MobileLLMInteractionStrategy` | ✓ True | LLM failure requires immediate recovery |\n| **ACTION_EXECUTION** | `MobileActionExecutionStrategy` | ✗ False | Action failures can be handled gracefully |\n| **MEMORY_UPDATE** | `AppMemoryUpdateStrategy` | ✗ False | Memory failures shouldn't block execution |\n\n**Fail-Fast vs Graceful:**\n\n- **fail_fast=True**: Critical phases where errors should immediately transition to FAIL state\n- **fail_fast=False**: Non-critical phases where errors can be logged and execution continues\n\n## Four-Phase Pipeline\n\n### Pipeline Execution Flow\n\n```mermaid\ngraph LR\n A[CONTINUE State] --> B[Phase 1: Data Collection]\n B --> C[Phase 2: LLM Interaction]\n C --> D[Phase 3: Action Execution]\n D --> E[Phase 4: Memory Update]\n E --> F[Determine Next State]\n F --> G{Status?}\n G -->|CONTINUE| A\n G -->|FINISH| H[FINISH State]\n G -->|FAIL| I[FAIL State]\n```\n\n## Phase 1: Data Collection Strategy (Composed)\n\n**Purpose**: Gather comprehensive visual and structural information about the current mobile UI state.\n\nPhase 1 is a **composed strategy** consisting of three sub-strategies executed sequentially:\n\n1. **Screenshot Capture**: Take device screenshot\n2. **Apps Collection**: List installed applications\n3. **Controls Collection**: Extract UI hierarchy and annotate controls\n\n### Sub-Strategy 1.1: Screenshot Capture\n\n```python\n@depends_on(\"log_path\", \"session_step\")\n@provides(\n \"clean_screenshot_path\",\n \"clean_screenshot_url\",\n \"annotated_screenshot_url\", # Initially None, set by Controls Collection\n \"screenshot_saved_time\",\n)\nclass MobileScreenshotCaptureStrategy(BaseProcessingStrategy):\n \"\"\"\n Strategy for capturing Android device screenshots.\n \"\"\"\n```\n\n#### Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant MCP\n participant ADB\n participant Device\n \n Strategy->>MCP: capture_screenshot command\n MCP->>ADB: screencap -p /sdcard/screen_temp.png\n ADB->>Device: Execute screenshot\n Device-->>ADB: Screenshot saved\n \n ADB->>Device: Pull screenshot\n Device-->>ADB: PNG file\n ADB-->>MCP: PNG data\n \n MCP->>MCP: Encode to base64\n MCP-->>Strategy: data:image/png;base64,...\n \n Strategy->>Strategy: Save to log_path\n Strategy-->>Agent: Screenshot URL + path\n```\n\n#### Output\n\n```python\n{\n \"clean_screenshot_path\": \"logs/.../action_step1.png\",\n \"clean_screenshot_url\": \"data:image/png;base64,iVBORw0KGgoAAAANS...\",\n \"annotated_screenshot_url\": None, # Set by Controls Collection\n \"screenshot_saved_time\": 0.234 # seconds\n}\n```\n\n### Sub-Strategy 1.2: Apps Collection\n\n```python\n@depends_on(\"clean_screenshot_url\")\n@provides(\"installed_apps\", \"apps_collection_time\")\nclass MobileAppsCollectionStrategy(BaseProcessingStrategy):\n \"\"\"\n Strategy for collecting installed apps information from Android device.\n \"\"\"\n```\n\n#### Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant MCP\n participant ADB\n participant Device\n \n Strategy->>MCP: get_mobile_app_target_info\n MCP->>MCP: Check cache (5min TTL)\n \n alt Cache Hit\n MCP-->>Strategy: Cached app list\n else Cache Miss\n MCP->>ADB: pm list packages -3\n ADB->>Device: List user apps\n Device-->>ADB: Package list\n ADB-->>MCP: Packages\n \n MCP->>MCP: Parse to TargetInfo\n MCP->>MCP: Update cache\n MCP-->>Strategy: App list\n end\n \n Strategy-->>Agent: Installed apps\n```\n\n#### Output Format\n\n```python\n{\n \"installed_apps\": [\n {\n \"id\": \"1\",\n \"name\": \"com.android.chrome\",\n \"package\": \"com.android.chrome\"\n },\n {\n \"id\": \"2\",\n \"name\": \"com.google.android.apps.maps\",\n \"package\": \"com.google.android.apps.maps\"\n },\n ...\n ],\n \"apps_collection_time\": 0.156 # seconds\n}\n```\n\n**Caching**: Apps list is cached for 5 minutes to reduce ADB overhead, as installed apps rarely change during a session.\n\n### Sub-Strategy 1.3: Controls Collection\n\n```python\n@depends_on(\"clean_screenshot_url\")\n@provides(\n \"current_controls\",\n \"controls_collection_time\",\n \"annotated_screenshot_url\",\n \"annotated_screenshot_path\",\n \"annotation_dict\",\n)\nclass MobileControlsCollectionStrategy(BaseProcessingStrategy):\n \"\"\"\n Strategy for collecting current screen controls information from Android device.\n Creates annotated screenshots with control labels.\n \"\"\"\n```\n\n#### Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant MCP\n participant ADB\n participant Device\n participant Photographer\n \n Strategy->>MCP: get_app_window_controls_target_info\n MCP->>MCP: Check cache (5s TTL)\n \n alt Cache Hit\n MCP-->>Strategy: Cached controls\n else Cache Miss\n MCP->>ADB: uiautomator dump /sdcard/window_dump.xml\n ADB->>Device: Dump UI hierarchy\n Device-->>ADB: XML file\n \n ADB->>Device: cat /sdcard/window_dump.xml\n Device-->>ADB: XML content\n ADB-->>MCP: UI hierarchy XML\n \n MCP->>MCP: Parse XML\n MCP->>MCP: Extract clickable controls\n MCP->>MCP: Validate rectangles\n MCP->>MCP: Assign IDs\n MCP->>MCP: Update cache\n MCP-->>Strategy: Controls list\n end\n \n Strategy->>Strategy: Convert to TargetInfo\n Strategy->>Photographer: Create annotated screenshot\n Photographer->>Photographer: Draw control IDs on screenshot\n Photographer-->>Strategy: Annotated image\n \n Strategy-->>Agent: Controls + Annotated screenshot\n```\n\n#### UI Hierarchy Parsing\n\nThe strategy parses Android UI XML to extract meaningful controls:\n\n```xml\n\n\n \n \n \n \n \n \n\n```\n\n**Control Selection Criteria**:\n\n- `clickable=\"true\"` - Can be tapped\n- `long-clickable=\"true\"` - Supports long-press\n- `scrollable=\"true\"` - Can be scrolled\n- `checkable=\"true\"` - Checkbox or toggle\n- Has `text` or `content-desc` - Has label\n- Type includes \"Edit\", \"Button\" - Input or action element\n\n**Rectangle Validation**:\n\nControls with invalid rectangles are filtered out:\n\n```python\n# Bounds format: [left, top, right, bottom]\nif right <= left or bottom <= top:\n # Invalid: width or height is zero/negative\n skip_control()\n```\n\n#### Output Format\n\n```python\n{\n \"current_controls\": [\n {\n \"id\": \"1\",\n \"name\": \"Search\",\n \"type\": \"EditText\",\n \"rect\": [48, 96, 912, 192] # [left, top, right, bottom]\n },\n {\n \"id\": \"2\",\n \"name\": \"Search\",\n \"type\": \"ImageButton\",\n \"rect\": [912, 96, 1032, 192]\n },\n ...\n ],\n \"annotated_screenshot_url\": \"data:image/png;base64,...\",\n \"annotated_screenshot_path\": \"logs/.../action_step1_annotated.png\",\n \"annotation_dict\": {\n \"1\": {\"id\": \"1\", \"name\": \"Search\", \"type\": \"EditText\", ...},\n \"2\": {\"id\": \"2\", \"name\": \"Search\", \"type\": \"ImageButton\", ...},\n ...\n },\n \"controls_collection_time\": 0.345 # seconds\n}\n```\n\n**Caching**: Controls are cached for 5 seconds, but the cache is invalidated after every action (UI likely changed).\n\n### Composed Strategy Execution\n\nThe three sub-strategies are executed sequentially in a single composed strategy:\n\n```python\nComposedStrategy(\n strategies=[\n MobileScreenshotCaptureStrategy(fail_fast=True),\n MobileAppsCollectionStrategy(fail_fast=False),\n MobileControlsCollectionStrategy(fail_fast=False),\n ],\n name=\"MobileDataCollectionStrategy\",\n fail_fast=True, # Overall failure if screenshot capture fails\n)\n```\n\n**Execution Order**:\n\n1. Screenshot Capture (critical)\n2. Apps Collection (optional, continues on failure)\n3. Controls Collection (optional, continues on failure)\n\n---\n\n## Phase 2: LLM Interaction Strategy\n\n**Purpose**: Construct mobile-specific prompts with visual context and obtain next action from LLM.\n\n### Strategy Implementation\n\n```python\n@depends_on(\"installed_apps\", \"current_controls\", \"clean_screenshot_url\")\n@provides(\n \"parsed_response\",\n \"response_text\",\n \"llm_cost\",\n \"prompt_message\",\n \"action\",\n \"thought\",\n \"comment\",\n)\nclass MobileLLMInteractionStrategy(AppLLMInteractionStrategy):\n \"\"\"\n Strategy for LLM interaction with Mobile Agent specific prompting.\n \"\"\"\n```\n\n### Phase 2 Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Agent\n participant Prompter\n participant LLM\n \n Strategy->>Agent: Get previous plan\n Strategy->>Agent: Get blackboard context\n Agent-->>Strategy: Previous execution results\n \n Strategy->>Prompter: Construct mobile prompt\n Prompter->>Prompter: Build system message (APIs + examples)\n Prompter->>Prompter: Add screenshot images\n Prompter->>Prompter: Add annotated screenshot\n Prompter->>Prompter: Add text prompt with context\n Prompter-->>Strategy: Complete multimodal prompt\n \n Strategy->>LLM: Send prompt\n LLM-->>Strategy: Mobile action + status\n \n Strategy->>Strategy: Parse response\n Strategy->>Strategy: Validate action\n Strategy-->>Agent: Parsed response + cost\n```\n\n### Prompt Construction\n\nThe strategy constructs comprehensive multimodal prompts:\n\n```python\nprompt_message = agent.message_constructor(\n dynamic_examples=[], # Few-shot examples (optional)\n dynamic_knowledge=\"\", # Retrieved knowledge (optional)\n plan=plan, # Previous execution plan\n request=request, # User request\n installed_apps=installed_apps, # Available apps\n current_controls=current_controls, # UI controls with IDs\n screenshot_url=clean_screenshot_url, # Clean screenshot\n annotated_screenshot_url=annotated_screenshot_url, # With control IDs\n blackboard_prompt=blackboard_prompt, # Shared context\n last_success_actions=last_success_actions # Successful actions\n)\n```\n\n### Multimodal Content Structure\n\nThe prompt includes both visual and textual elements:\n\n```python\nuser_content = [\n # 1. Clean screenshot (for visual understanding)\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": \"data:image/png;base64,iVBORw0KGgo...\"}\n },\n \n # 2. Annotated screenshot (for control identification)\n {\n \"type\": \"image_url\",\n \"image_url\": {\"url\": \"data:image/png;base64,iVBORw0KGgo...\"}\n },\n \n # 3. Text prompt with context\n {\n \"type\": \"text\",\n \"text\": \"\"\"\n [Previous Plan]: [...]\n [User Request]: Search for restaurants on Maps\n [Installed Apps]: [\n {\"id\": \"1\", \"name\": \"com.google.android.apps.maps\", ...},\n ...\n ]\n [Current Screen Controls]: [\n {\"id\": \"1\", \"name\": \"Search\", \"type\": \"EditText\", ...},\n {\"id\": \"2\", \"name\": \"Search\", \"type\": \"ImageButton\", ...},\n ...\n ]\n [Last Success Actions]: [...]\n \"\"\"\n }\n]\n```\n\n### LLM Response Format\n\nThe LLM returns a structured mobile action:\n\n```json\n{\n \"thought\": \"I need to launch Google Maps app first\",\n \"action\": {\n \"function\": \"launch_app\",\n \"arguments\": {\n \"package_name\": \"com.google.android.apps.maps\",\n \"id\": \"1\"\n },\n \"status\": \"CONTINUE\"\n },\n \"comment\": \"Launching Maps to search for restaurants\"\n}\n```\n\n### Mobile-Specific Features\n\n**Visual Context Priority**: LLM sees both clean and annotated screenshots, enabling better UI understanding than text-only descriptions.\n\n**Control ID References**: Annotated screenshot shows control IDs, allowing LLM to precisely reference UI elements in actions.\n\n**App Awareness**: LLM knows which apps are installed, enabling intelligent app selection and launching.\n\n**Touch-Based Actions**: LLM generates mobile-specific actions (tap, swipe, type) instead of desktop actions (click, drag, keyboard).\n\n---\n\n## Phase 3: Action Execution Strategy\n\n**Purpose**: Execute mobile actions returned by LLM and capture structured results.\n\n### Strategy Implementation\n\n```python\nclass MobileActionExecutionStrategy(AppActionExecutionStrategy):\n \"\"\"\n Strategy for executing actions in Mobile Agent.\n \"\"\"\n```\n\n### Phase 3 Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant MCP\n participant ADB\n participant Device\n \n Strategy->>Strategy: Extract action from LLM response\n \n alt launch_app\n Strategy->>MCP: launch_app(package_name)\n MCP->>ADB: monkey -p package_name\n else click_control\n Strategy->>MCP: click_control(control_id, control_name)\n MCP->>MCP: Get control from cache\n MCP->>MCP: Calculate center position\n MCP->>ADB: input tap x y\n else type_text\n Strategy->>MCP: type_text(text, control_id, ...)\n MCP->>ADB: input tap (focus control)\n MCP->>ADB: input text (type)\n else swipe\n Strategy->>MCP: swipe(start_x, start_y, end_x, end_y)\n MCP->>ADB: input swipe ...\n else tap\n Strategy->>MCP: tap(x, y)\n MCP->>ADB: input tap x y\n else press_key\n Strategy->>MCP: press_key(key_code)\n MCP->>ADB: input keyevent KEY_CODE\n else wait\n Strategy->>Strategy: asyncio.sleep(seconds)\n end\n \n ADB->>Device: Execute command\n Device-->>ADB: Result\n ADB-->>MCP: Success/Failure\n \n MCP->>MCP: Invalidate controls cache\n MCP-->>Strategy: Execution result\n \n Strategy->>Strategy: Create action info\n Strategy->>Strategy: Format for memory\n Strategy-->>Agent: Execution results\n```\n\n### Action Execution Flow\n\n```python\n# Extract parsed LLM response\nparsed_response: AppAgentResponse = context.get_local(\"parsed_response\")\ncommand_dispatcher = context.global_context.command_dispatcher\n\n# Execute the action via MCP\nexecution_results = await self._execute_app_action(\n command_dispatcher,\n parsed_response.action\n)\n```\n\n### Result Capture\n\nExecution results are structured for downstream processing:\n\n```python\n{\n \"success\": True,\n \"action\": \"click_control(id=5, name=Search)\",\n \"message\": \"Clicked control 'Search' at (480, 144)\",\n \"control_info\": {\n \"id\": \"5\",\n \"name\": \"Search\",\n \"type\": \"EditText\",\n \"rect\": [48, 96, 912, 192]\n }\n}\n```\n\n### Action Info Creation\n\nResults are formatted into `ActionCommandInfo` objects:\n\n```python\nactions = self._create_action_info(\n parsed_response.action,\n execution_results,\n)\n\naction_info = ListActionCommandInfo(actions)\naction_info.color_print() # Pretty print to console\n```\n\n### Cache Invalidation\n\nAfter each action, control caches are invalidated:\n\n```python\n# Mobile MCP server automatically invalidates caches after actions\n# This ensures next round gets fresh UI state\nmobile_state.invalidate_controls()\n```\n\n---\n\n## Phase 4: Memory Update Strategy\n\n**Purpose**: Persist execution results, screenshots, and control information into agent memory for future reference.\n\n### Strategy Implementation\n\nMobileAgent reuses the `AppMemoryUpdateStrategy` from the app agent framework:\n\n```python\nself.strategies[ProcessingPhase.MEMORY_UPDATE] = AppMemoryUpdateStrategy(\n fail_fast=False # Memory failures shouldn't stop process\n)\n```\n\n### Phase 4 Workflow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Memory\n participant Context\n \n Strategy->>Context: Get execution results\n Strategy->>Context: Get LLM response\n Strategy->>Context: Get screenshots\n \n Strategy->>Memory: Create memory item\n Memory->>Memory: Store screenshots (clean + annotated)\n Memory->>Memory: Store action details\n Memory->>Memory: Store control information\n Memory->>Memory: Store timestamp\n \n Strategy->>Context: Update round result\n Strategy-->>Agent: Memory updated\n```\n\n### Memory Structure\n\nEach execution round is stored as a memory item:\n\n```python\n{\n \"round\": 1,\n \"request\": \"Search for restaurants on Maps\",\n \"thought\": \"I need to launch Google Maps app first\",\n \"action\": {\n \"function\": \"launch_app\",\n \"arguments\": {\n \"package_name\": \"com.google.android.apps.maps\",\n \"id\": \"1\"\n }\n },\n \"result\": {\n \"success\": True,\n \"message\": \"Launched com.google.android.apps.maps\"\n },\n \"screenshots\": {\n \"clean\": \"logs/.../action_step1.png\",\n \"annotated\": \"logs/.../action_step1_annotated.png\"\n },\n \"controls\": [\n {\"id\": \"1\", \"name\": \"Search\", \"type\": \"EditText\", ...},\n ...\n ],\n \"status\": \"CONTINUE\",\n \"timestamp\": \"2025-11-14T10:30:45\"\n}\n```\n\n### Iterative Refinement\n\nMemory enables iterative refinement across rounds:\n\n1. **Round 1**: Launch Maps app → Maps opened\n2. **Round 2**: Click search field (using control ID from Round 1 screenshot)\n3. **Round 3**: Type \"restaurants\" → Text entered\n4. **Round 4**: Click search button → Results displayed\n\nEach round builds on previous results and screenshots stored in memory.\n\n### Visual Debugging\n\nMemory stores screenshots for each round, enabling visual debugging:\n\n- **Clean Screenshots**: Show actual device UI\n- **Annotated Screenshots**: Show control IDs used by LLM\n- **Action Sequence**: Visual trace of entire task execution\n\n---\n\n## Middleware Stack\n\nMobileAgent uses specialized middleware for logging:\n\n```python\ndef _setup_middleware(self) -> None:\n \"\"\"Setup middleware pipeline for Mobile Agent\"\"\"\n self.middleware_chain = [MobileLoggingMiddleware()]\n```\n\n### MobileLoggingMiddleware\n\nProvides enhanced logging specific to Mobile operations:\n\n```python\nclass MobileLoggingMiddleware(AppAgentLoggingMiddleware):\n \"\"\"Specialized logging middleware for Mobile Agent\"\"\"\n \n def starting_message(self, context: ProcessingContext) -> str:\n request = context.get(\"request\") or \"Unknown Request\"\n return f\"Completing the user request: [bold cyan]{request}[/bold cyan] on Mobile.\"\n```\n\n**Logged Information**:\n\n- User request\n- Screenshots captured (with paths)\n- Apps collected\n- Controls identified (with IDs)\n- Each mobile action executed\n- Action results\n- State transitions\n- LLM costs\n- Timing information\n\n---\n\n## Context Finalization\n\nAfter processing, the processor updates global context:\n\n```python\ndef _finalize_processing_context(self, processing_context: ProcessingContext):\n \"\"\"Finalize processing context by updating ContextNames fields\"\"\"\n super()._finalize_processing_context(processing_context)\n \n try:\n result = processing_context.get_local(\"result\")\n if result:\n self.global_context.set(ContextNames.ROUND_RESULT, result)\n except Exception as e:\n self.logger.warning(f\"Failed to update context: {e}\")\n```\n\nThis makes execution results available to:\n\n- Subsequent rounds (iterative execution)\n- Other agents (if part of multi-agent workflow)\n- Session manager (for monitoring and logging)\n\n---\n\n## Strategy Dependency Graph\n\nThe four phases have clear dependencies:\n\n```mermaid\ngraph TD\n A[log_path + session_step] --> B[Phase 1.1: Screenshot Capture]\n B --> C[clean_screenshot_url]\n \n C --> D[Phase 1.2: Apps Collection]\n D --> E[installed_apps]\n \n C --> F[Phase 1.3: Controls Collection]\n F --> G[current_controls]\n F --> H[annotated_screenshot_url]\n F --> I[annotation_dict]\n \n E --> J[Phase 2: LLM Interaction]\n G --> J\n C --> J\n H --> J\n J --> K[parsed_response]\n J --> L[llm_cost]\n \n K --> M[Phase 3: Action Execution]\n I --> M\n M --> N[execution_result]\n M --> O[action_info]\n \n K --> P[Phase 4: Memory Update]\n N --> P\n O --> P\n C --> P\n H --> P\n P --> Q[Memory Updated]\n \n Q --> R[Next Round or Terminal State]\n```\n\n---\n\n## Modular Design Benefits\n\nThe 4-phase strategy design provides:\n\n!!!success \"Modularity Benefits\"\n - **Separation of Concerns**: Data collection, LLM reasoning, action execution, and memory are isolated\n - **Visual Context**: Screenshots provide rich UI understanding beyond text descriptions\n - **Testability**: Each phase can be tested independently with mocked data\n - **Extensibility**: New data collection strategies can be added (e.g., accessibility info)\n - **Reusability**: Memory strategy is shared with AppAgent\n - **Maintainability**: Clear boundaries between perception, decision, and action\n - **Traceability**: Each phase logs its operations independently with visual artifacts\n - **Performance**: Caching strategies reduce ADB overhead\n\n---\n\n## Comparison with Other Agents\n\n| Agent | Phases | Data Collection | Visual | LLM | Action | Memory |\n|-------|--------|----------------|--------|-----|--------|--------|\n| **MobileAgent** | 4 | ✓ Screenshots + Controls + Apps | ✓ Multimodal | ✓ Mobile actions | ✓ Touch/swipe | ✓ Results + Screenshots |\n| **LinuxAgent** | 3 | ✗ On-demand | ✗ Text-only | ✓ CLI commands | ✓ Shell | ✓ Results |\n| **AppAgent** | 4 | ✓ Screenshots + UI | ✓ Multimodal | ✓ UI actions | ✓ GUI + API | ✓ Results + Screenshots |\n| **HostAgent** | 4 | ✓ Desktop snapshot | ✓ Multimodal | ✓ App selection | ✓ Orchestration | ✓ Results |\n\nMobileAgent's 4-phase pipeline includes **DATA_COLLECTION** phase because:\n\n- Mobile UI requires visual context (screenshots)\n- Control identification needs UI hierarchy parsing\n- Touch targets need precise coordinates\n- Apps list informs available actions\n- Annotation creates visual correspondence between LLM and execution\n\nThis reflects the visual, touch-based nature of mobile interaction.\n\n---\n\n## Implementation Location\n\nThe strategy implementations can be found in:\n\n```\nufo/agents/processors/\n├── customized/\n│ └── customized_agent_processor.py # MobileAgentProcessor\n└── strategies/\n └── mobile_agent_strategy.py # Mobile-specific strategies\n```\n\nKey classes:\n\n- `MobileAgentProcessor`: Strategy orchestrator\n- `MobileScreenshotCaptureStrategy`: Screenshot capture via ADB\n- `MobileAppsCollectionStrategy`: Installed apps collection\n- `MobileControlsCollectionStrategy`: UI controls extraction and annotation\n- `MobileLLMInteractionStrategy`: Multimodal prompt construction and LLM interaction\n- `MobileActionExecutionStrategy`: Mobile action execution\n- `MobileLoggingMiddleware`: Enhanced logging\n\n---\n\n## Next Steps\n\n- [MCP Commands](commands.md) - Explore the mobile UI interaction and app management commands\n- [State Machine](state.md) - Understand the 3-state FSM that controls strategy execution\n- [Overview](overview.md) - Return to MobileAgent architecture overview\n"
},
{
"path": "documents/docs/project_directory_structure.md",
"content": "# Project Directory Structure\n\nThis repository implements **UFO³**, a multi-tier AgentOS architecture spanning from single-device automation (UFO²) to cross-device orchestration (Galaxy). This document provides an overview of the directory structure to help you understand the codebase organization.\n\n> **New to UFO³?** Start with the [Documentation Home](index.md) for an introduction and [Quick Start Guide](getting_started/quick_start_galaxy.md) to get up and running.\n\n**Architecture Overview:**\n\n- **🌌 Galaxy**: Multi-device DAG-based orchestration framework that coordinates agents across different platforms\n- **🎯 UFO²**: Single-device Windows desktop agent system that can serve as Galaxy's sub-agent\n- **🔌 AIP**: Agent Integration Protocol for cross-device communication\n- **⚙️ Modular Configuration**: Type-safe configs in `config/galaxy/` and `config/ufo/`\n\n---\n\n## 📦 Root Directory Structure\n\n```\nUFO/\n├── galaxy/ # 🌌 Multi-device orchestration framework\n├── ufo/ # 🎯 Desktop AgentOS (can be Galaxy sub-agent)\n├── config/ # ⚙️ Modular configuration system\n├── aip/ # 🔌 Agent Integration Protocol\n├── documents/ # 📖 MkDocs documentation site\n├── vectordb/ # 🗄️ Vector database for RAG\n├── learner/ # 📚 Help document indexing tools\n├── record_processor/ # 🎥 Human demonstration parser\n├── dataflow/ # 📊 Data collection pipeline\n├── model_worker/ # 🤖 Custom LLM deployment tools\n├── logs/ # 📝 Execution logs (auto-generated)\n├── scripts/ # 🛠️ Utility scripts\n├── tests/ # 🧪 Unit and integration tests\n└── requirements.txt # 📦 Python dependencies\n```\n\n---\n\n## 🌌 Galaxy Framework (`galaxy/`)\n\nThe cross-device orchestration framework that transforms natural language requests into executable DAG workflows distributed across heterogeneous devices.\n\n### Directory Structure\n\n```\ngalaxy/\n├── agents/ # 🤖 Constellation orchestration agents\n│ ├── agent/ # ConstellationAgent and basic agent classes\n│ ├── states/ # Agent state machines\n│ ├── processors/ # Request/result processing\n│ └── presenters/ # Response formatting\n│\n├── constellation/ # 🌟 Core DAG management system\n│ ├── task_constellation.py # TaskConstellation - DAG container\n│ ├── task_star.py # TaskStar - Task nodes\n│ ├── task_star_line.py # TaskStarLine - Dependency edges\n│ ├── enums.py # Enums for constellation components\n│ ├── editor/ # Interactive DAG editing with undo/redo\n│ └── orchestrator/ # Event-driven execution coordination\n│\n├── session/ # 📊 Session lifecycle management\n│ ├── galaxy_session.py # GalaxySession implementation\n│ └── observers/ # Event-driven observers\n│\n├── client/ # 📡 Device management\n│ ├── constellation_client.py # Device registration interface\n│ ├── device_manager.py # Device management coordinator\n│ ├── config_loader.py # Configuration loading\n│ ├── components/ # Device registry, connection manager, etc.\n│ └── support/ # Client support utilities\n│\n├── core/ # ⚡ Foundational components\n│ ├── types.py # Type system (protocols, dataclasses, enums)\n│ ├── interfaces.py # Interface definitions\n│ ├── di_container.py # Dependency injection container\n│ └── events.py # Event system\n│\n├── visualization/ # 🎨 Rich console visualization\n│ ├── dag_visualizer.py # DAG topology visualization\n│ ├── task_display.py # Task status displays\n│ └── components/ # Visualization components\n│\n├── prompts/ # 💬 Prompt templates\n│ ├── constellation_agent/ # ConstellationAgent prompts\n│ └── share/ # Shared examples\n│\n├── trajectory/ # 📈 Execution trajectory parsing\n│\n├── __main__.py # 🚀 Entry point: python -m galaxy\n├── galaxy.py # Main Galaxy orchestrator\n├── galaxy_client.py # Galaxy client interface\n├── README.md # Galaxy overview\n└── README_ZH.md # Galaxy overview (Chinese)\n```\n\n### Key Components\n\n| Component | Description | Documentation |\n|-----------|-------------|---------------|\n| **ConstellationAgent** | AI-powered agent that generates and modifies task DAGs | [Galaxy Overview](galaxy/overview.md) |\n| **TaskConstellation** | DAG container with validation and state management | [Constellation](galaxy/constellation/overview.md) |\n| **TaskOrchestrator** | Event-driven execution coordinator | [Constellation Orchestrator](galaxy/constellation_orchestrator/overview.md) |\n| **DeviceManager** | Multi-device coordination and assignment | [Device Manager](galaxy/client/device_manager.md) |\n| **Visualization** | Rich console DAG monitoring | [Galaxy Overview](galaxy/overview.md) |\n\n**Galaxy Documentation:**\n\n- [Galaxy Overview](galaxy/overview.md) - Architecture and concepts\n- [Quick Start](getting_started/quick_start_galaxy.md) - Get started with Galaxy\n- [Constellation Agent](galaxy/constellation_agent/overview.md) - AI-powered task planning\n- [Constellation Orchestrator](galaxy/constellation_orchestrator/overview.md) - Event-driven coordination\n- [Device Manager](galaxy/client/device_manager.md) - Multi-device management\n\n---\n\n## 🎯 UFO² Desktop AgentOS (`ufo/`)\n\nSingle-device desktop automation system implementing a two-tier agent architecture (HostAgent + AppAgent) with hybrid GUI-API automation.\n\n### Directory Structure\n\n```\nufo/\n├── agents/ # Two-tier agent implementation\n│ ├── agent/ # Base agent classes (HostAgent, AppAgent)\n│ ├── states/ # State machine implementations\n│ ├── processors/ # Processing strategy pipelines\n│ ├── memory/ # Agent memory and blackboard\n│ └── presenters/ # Response presentation logic\n│\n├── server/ # Server-client architecture components\n│ ├── websocket_server.py # WebSocket server for remote agent control\n│ └── handlers/ # Request handlers\n│\n├── client/ # MCP client and device management\n│ ├── mcp/ # MCP server manager\n│ │ ├── local_servers/ # Built-in MCP servers (UI, CLI, Office COM)\n│ │ └── http_servers/ # Remote MCP servers (hardware, Linux)\n│ ├── ufo_client.py # UFO² client implementation\n│ └── computer.py # Computer/device abstraction\n│\n├── automator/ # GUI and API automation layer\n│ ├── ui_control/ # GUI automation (inspector, controller)\n│ ├── puppeteer/ # Execution orchestration\n│ └── *_automator.py # App-specific automators (Excel, Word, etc.)\n│\n├── prompter/ # Prompt construction engines\n├── prompts/ # Jinja2 prompt templates\n│ ├── host_agent/ # HostAgent prompts\n│ ├── app_agent/ # AppAgent prompts\n│ └── share/ # Shared components\n│\n├── llm/ # LLM provider integrations\n├── rag/ # Retrieval-Augmented Generation\n├── trajectory/ # Task trajectory parsing\n├── experience/ # Self-experience learning\n├── module/ # Core modules (session, round, context)\n├── config/ # Legacy config support\n├── logging/ # Logging utilities\n├── utils/ # Utility functions\n├── tools/ # CLI tools (config conversion, etc.)\n│\n├── __main__.py # Entry point: python -m ufo\n└── ufo.py # Main UFO² orchestrator\n```\n\n### Key Components\n\n| Component | Description | Documentation |\n|-----------|-------------|---------------|\n| **HostAgent** | Desktop-level orchestration with 7-state FSM | [HostAgent Overview](ufo2/host_agent/overview.md) |\n| **AppAgent** | Application-level execution with 6-state FSM | [AppAgent Overview](ufo2/app_agent/overview.md) |\n| **MCP System** | Extensible command execution framework | [MCP Overview](mcp/overview.md) |\n| **Automator** | Hybrid GUI-API automation with fallback | [Core Features](ufo2/core_features/hybrid_actions.md) |\n| **RAG** | Knowledge retrieval from multiple sources | [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) |\n\n**UFO² Documentation:**\n\n- [UFO² Overview](ufo2/overview.md) - Architecture and concepts\n- [Quick Start](getting_started/quick_start_ufo2.md) - Get started with UFO²\n- [HostAgent States](ufo2/host_agent/state.md) - Desktop orchestration states\n- [AppAgent States](ufo2/app_agent/state.md) - Application execution states\n- [As Galaxy Device](ufo2/as_galaxy_device.md) - Using UFO² as Galaxy sub-agent\n- [Creating Custom Agents](tutorials/creating_app_agent/overview.md) - Build your own application agents\n\n---\n\n## 🔌 Agent Integration Protocol (`aip/`)\n\nStandardized message passing protocol for cross-device communication between Galaxy and UFO² agents.\n\n```\naip/\n├── messages.py # Message types (Command, Result, Event, Error)\n├── protocol/ # Protocol definitions\n├── transport/ # Transport layers (HTTP, WebSocket, MQTT)\n├── endpoints/ # API endpoints\n├── extensions/ # Protocol extensions\n└── resilience/ # Retry and error handling\n```\n\n**Purpose**: Enables Galaxy to coordinate UFO² agents running on different devices and platforms through standardized messaging over HTTP/WebSocket.\n\n**Documentation**: See [AIP Overview](aip/overview.md) for protocol details and [Message Types](aip/messages.md) for message specifications.\n\n---\n\n## 🐧 Linux Agent\n\nLightweight CLI-based agent for Linux devices that integrates with Galaxy as a third-party device agent.\n\n**Key Features**:\n- **CLI Execution**: Execute shell commands on Linux systems\n- **Galaxy Integration**: Register as device in Galaxy's multi-device orchestration\n- **Simple Architecture**: Minimal dependencies, easy deployment\n- **Cross-Platform Tasks**: Enable Windows + Linux workflows in Galaxy\n\n**Configuration**: Configured in `config/ufo/third_party.yaml` under `THIRD_PARTY_AGENT_CONFIG.LinuxAgent`\n\n**Linux Agent Documentation:**\n\n- [Linux Agent Overview](linux/overview.md) - Architecture and capabilities\n- [Quick Start](getting_started/quick_start_linux.md) - Setup and deployment\n- [As Galaxy Device](linux/as_galaxy_device.md) - Integration with Galaxy\n\n---\n\n## 📱 Mobile Agent\n\nAndroid device automation agent that enables UI automation, app control, and mobile-specific operations through ADB integration.\n\n**Key Features**:\n- **UI Automation**: Touch, swipe, and text input via ADB\n- **Visual Context**: Screenshot capture and UI hierarchy analysis\n- **App Management**: Launch apps, navigate between applications\n- **Galaxy Integration**: Serve as mobile device in cross-platform workflows\n- **Platform Support**: Android devices (physical and emulators)\n\n**Configuration**: Configured in `config/ufo/third_party.yaml` under `THIRD_PARTY_AGENT_CONFIG.MobileAgent`\n\n**Mobile Agent Documentation:**\n\n- [Mobile Agent Overview](mobile/overview.md) - Architecture and capabilities\n- [Quick Start](getting_started/quick_start_mobile.md) - Setup and deployment\n- [As Galaxy Device](mobile/as_galaxy_device.md) - Integration with Galaxy\n\n---\n\n## ⚙️ Configuration (`config/`)\n\nModular configuration system with type-safe schemas and auto-discovery.\n\n```\nconfig/\n├── galaxy/ # Galaxy configuration\n│ ├── agent.yaml.template # ConstellationAgent LLM settings template\n│ ├── agent.yaml # ConstellationAgent LLM settings (active)\n│ ├── constellation.yaml # Constellation orchestration settings\n│ ├── devices.yaml # Multi-device registry\n│ └── dag_templates/ # Pre-built DAG templates (future)\n│\n├── ufo/ # UFO² configuration\n│ ├── agents.yaml.template # Agent LLM configs template\n│ ├── agents.yaml # Agent LLM configs (active)\n│ ├── system.yaml # System settings\n│ ├── rag.yaml # RAG settings\n│ ├── mcp.yaml # MCP server configs\n│ ├── third_party.yaml # Third-party agent configs (LinuxAgent, etc.)\n│ └── prices.yaml # API pricing data\n│\n├── config_loader.py # Auto-discovery config loader\n└── config_schemas.py # Pydantic validation schemas\n```\n\n**Configuration Files:**\n\n- Template files (`.yaml.template`) should be copied to `.yaml` and edited\n- Active config files (`.yaml`) contain API keys and should NOT be committed\n- **Galaxy**: Uses `config/galaxy/agent.yaml` for ConstellationAgent LLM settings\n- **UFO²**: Uses `config/ufo/agents.yaml` for HostAgent/AppAgent LLM settings\n- **Third-Party**: Configure LinuxAgent and HardwareAgent in `config/ufo/third_party.yaml`\n- Use `python -m ufo.tools.convert_config` to migrate from legacy configs\n\n**Configuration Documentation:**\n\n- [Configuration Overview](configuration/system/overview.md) - System architecture\n- [Agents Configuration](configuration/system/agents_config.md) - LLM and agent settings\n- [System Configuration](configuration/system/system_config.md) - Runtime and execution settings\n- [RAG Configuration](configuration/system/rag_config.md) - Knowledge retrieval\n- [Third-Party Configuration](configuration/system/third_party_config.md) - LinuxAgent and external agents\n- [MCP Configuration](configuration/system/mcp_reference.md) - MCP server setup\n- [Model Configuration](configuration/models/overview.md) - LLM provider setup\n\n---\n\n## 📖 Documentation (`documents/`)\n\nMkDocs documentation site with comprehensive guides and API references.\n\n```\ndocuments/\n├── docs/ # Markdown documentation source\n│ ├── getting_started/ # Installation and quick starts\n│ ├── galaxy/ # Galaxy framework docs\n│ ├── ufo2/ # UFO² architecture docs\n│ ├── linux/ # Linux agent documentation\n│ ├── mcp/ # MCP server documentation\n│ ├── aip/ # Agent Interaction Protocol docs\n│ ├── configuration/ # Configuration guides\n│ ├── infrastructure/ # Core infrastructure (agents, modules)\n│ ├── server/ # Server-client architecture docs\n│ ├── client/ # Client components docs\n│ ├── tutorials/ # Step-by-step tutorials\n│ ├── modules/ # Module-specific docs\n│ └── about/ # Project information\n│\n├── mkdocs.yml # MkDocs configuration\n└── site/ # Generated static site\n```\n\n**Documentation Sections**:\n\n| Section | Description |\n|---------|-------------|\n| **Getting Started** | Installation, quick starts, migration guides |\n| **Galaxy** | Multi-device orchestration, DAG workflows, device management |\n| **UFO²** | Desktop agents, automation features, benchmarks |\n| **Linux** | Linux agent integration, CLI executor for Galaxy |\n| **MCP** | Server documentation, custom server development |\n| **AIP** | Agent Interaction Protocol, message types, transport layers |\n| **Configuration** | System settings, model configs, deployment |\n| **Infrastructure** | Core components, agent design, server-client architecture |\n| **Tutorials** | Creating agents, custom automators, advanced usage |\n\n---\n\n## 🗄️ Supporting Modules\n\n### VectorDB (`vectordb/`)\nVector database storage for RAG knowledge sources (help documents, execution traces, user demonstrations). See [RAG Configuration](configuration/system/rag_config.md) for setup details.\n\n### Learner (`learner/`)\nTools for indexing help documents into vector database for RAG retrieval. Integrates with the [Knowledge Substrate](ufo2/core_features/knowledge_substrate/overview.md) feature.\n\n### Record Processor (`record_processor/`)\nParses human demonstrations from Windows Step Recorder for learning from user actions.\n\n### Dataflow (`dataflow/`)\nData collection pipeline for Large Action Model (LAM) training. See the [Dataflow](ufo2/dataflow/overview.md) documentation for workflow details.\n\n### Model Worker (`model_worker/`)\nCustom LLM deployment tools for running local models. See [Model Configuration](configuration/models/overview.md) for supported providers.\n\n### Logs (`logs/`)\nAuto-generated execution logs organized by task and timestamp, including screenshots, UI trees, and agent actions.\n\n---\n\n## 🎯 Galaxy vs UFO² vs Linux Agent vs Mobile Agent: When to Use What?\n\n| Aspect | Galaxy | UFO² | Linux Agent | Mobile Agent |\n|--------|--------|------|-------------|--------------|\n| **Scope** | Multi-device orchestration | Single-device Windows automation | Single-device Linux CLI | Single-device Android automation |\n| **Use Cases** | Cross-platform workflows, distributed tasks | Desktop automation, Office tasks | Server management, CLI operations | Mobile app testing, UI automation |\n| **Architecture** | DAG-based task workflows | Two-tier state machines | Simple CLI executor | UI automation via ADB |\n| **Platform** | Orchestrator (platform-agnostic) | Windows | Linux | Android |\n| **Complexity** | Complex multi-step workflows | Simple to moderate tasks | Simple command execution | UI interaction and app control |\n| **Best For** | Cross-device collaboration | Windows desktop tasks | Linux server operations | Mobile app automation |\n| **Integration** | Orchestrates all agents | Can be Galaxy device | Can be Galaxy device | Can be Galaxy device |\n\n**Choosing the Right Framework:**\n\n- **Use Galaxy** when: Tasks span multiple devices/platforms, complex workflows with dependencies\n- **Use UFO² Standalone** when: Single-device Windows automation, rapid prototyping\n- **Use Linux Agent** when: Linux server/CLI operations needed in Galaxy workflows\n- **Use Mobile Agent** when: Android device automation, mobile app testing, UI interactions\n- **Best Practice**: Galaxy orchestrates UFO² (Windows) + Linux Agent (Linux) + Mobile Agent (Android) for comprehensive cross-platform tasks\n\n---\n\n## 🚀 Quick Start\n\n### Galaxy Multi-Device Orchestration\n\n```bash\n# Interactive mode\npython -m galaxy --interactive\n\n# Single request\npython -m galaxy --request \"Your cross-device task\"\n```\n\n**Documentation**: [Galaxy Quick Start](getting_started/quick_start_galaxy.md)\n\n### UFO² Desktop Automation\n\n```bash\n# Interactive mode\npython -m ufo --task \n\n# With custom config\npython -m ufo --task --config_path config/ufo/\n```\n\n**Documentation**: [UFO² Quick Start](getting_started/quick_start_ufo2.md)\n\n---\n\n## 📚 Key Documentation Links\n\n### Getting Started\n- [Installation & Setup](getting_started/quick_start_galaxy.md)\n- [Galaxy Quick Start](getting_started/quick_start_galaxy.md)\n- [UFO² Quick Start](getting_started/quick_start_ufo2.md)\n- [Linux Agent Quick Start](getting_started/quick_start_linux.md)\n- [Mobile Agent Quick Start](getting_started/quick_start_mobile.md)\n- [Migration Guide](getting_started/migration_ufo2_to_galaxy.md)\n\n### Galaxy Framework\n- [Galaxy Overview](galaxy/overview.md)\n- [Constellation Agent](galaxy/constellation_agent/overview.md)\n- [Constellation Orchestrator](galaxy/constellation_orchestrator/overview.md)\n- [Task Constellation](galaxy/constellation/overview.md)\n- [Device Manager](galaxy/client/device_manager.md)\n\n### UFO² Desktop AgentOS\n- [UFO² Overview](ufo2/overview.md)\n- [HostAgent](ufo2/host_agent/overview.md)\n- [AppAgent](ufo2/app_agent/overview.md)\n- [Core Features](ufo2/core_features/hybrid_actions.md)\n- [As Galaxy Device](ufo2/as_galaxy_device.md)\n\n### Linux Agent\n- [Linux Agent Overview](linux/overview.md)\n- [As Galaxy Device](linux/as_galaxy_device.md)\n\n### Mobile Agent\n- [Mobile Agent Overview](mobile/overview.md)\n- [As Galaxy Device](mobile/as_galaxy_device.md)\n\n### MCP System\n- [MCP Overview](mcp/overview.md)\n- [Local Servers](mcp/local_servers.md)\n- [Creating MCP Servers](tutorials/creating_mcp_servers.md)\n\n### Agent Integration Protocol\n- [AIP Overview](aip/overview.md)\n- [Message Types](aip/messages.md)\n- [Transport Layers](aip/transport.md)\n\n### Configuration\n- [Configuration Overview](configuration/system/overview.md)\n- [Agents Configuration](configuration/system/agents_config.md)\n- [System Configuration](configuration/system/system_config.md)\n- [Model Configuration](configuration/models/overview.md)\n- [MCP Configuration](configuration/system/mcp_reference.md)\n\n---\n\n## 🏗️ Architecture Principles\n\nUFO³ follows **SOLID principles** and established software engineering patterns:\n\n- **Single Responsibility**: Each component has a focused purpose\n- **Open/Closed**: Extensible through interfaces and plugins\n- **Interface Segregation**: Focused interfaces for different capabilities\n- **Dependency Inversion**: Dependency injection for loose coupling\n- **Event-Driven**: Observer pattern for real-time monitoring\n- **State Machines**: Well-defined states and transitions for agents\n- **Command Pattern**: Encapsulated DAG editing with undo/redo\n\n---\n\n## 📝 Additional Resources\n\n- **[GitHub Repository](https://github.com/microsoft/UFO)** - Source code and issues\n- **[Research Paper](https://arxiv.org/abs/2504.14603)** - UFO³ technical details\n- **[Documentation Site](https://microsoft.github.io/UFO/)** - Full documentation\n- **[Video Demo](https://www.youtube.com/watch?v=QT_OhygMVXU)** - YouTube demonstration\n\n---\n\n**Next Steps:**\n\n1. Start with [Galaxy Quick Start](getting_started/quick_start_galaxy.md) for multi-device orchestration\n2. Or explore [UFO² Quick Start](getting_started/quick_start_ufo2.md) for single-device automation\n3. Check [FAQ](faq.md) for common questions\n4. Join our community and contribute!\n"
},
{
"path": "documents/docs/server/api.md",
"content": "# HTTP API Reference\n\nThe UFO Server provides a RESTful HTTP API for external systems to dispatch tasks, monitor client connections, retrieve results, and perform health checks. All endpoints are prefixed with `/api`.\n\n## 🎯 Overview\n\n```mermaid\ngraph LR\n subgraph \"External Systems\"\n Web[Web App]\n Script[Python Script]\n Tool[Automation Tool]\n end\n \n subgraph \"UFO Server HTTP API\"\n Dispatch[POST /api/dispatch]\n Clients[GET /api/clients]\n Result[GET /api/task_result]\n Health[GET /api/health]\n end\n \n subgraph \"Server Core\"\n WSM[Client Connection Manager]\n SM[Session Manager]\n WH[WebSocket Handler]\n end\n \n Web --> Dispatch\n Script --> Clients\n Tool --> Result\n Tool --> Health\n \n Dispatch --> WSM\n Dispatch --> SM\n Clients --> WSM\n Result --> SM\n Health --> WSM\n Health --> SM\n \n WSM --> WH\n SM --> WH\n \n style Dispatch fill:#bbdefb\n style Clients fill:#c8e6c9\n style Result fill:#fff9c4\n style Health fill:#ffcdd2\n```\n\n**Core Capabilities:**\n\n| Capability | Endpoint | Description |\n|------------|----------|-------------|\n| **Task Dispatch** | `POST /api/dispatch` | Send tasks to connected devices via HTTP |\n| **Client Monitoring** | `GET /api/clients` | Query connected devices and constellations |\n| **Result Retrieval** | `GET /api/task_result/{task_name}` | Fetch task execution results |\n| **Health Checks** | `GET /api/health` | Monitor server status and uptime |\n\n**Why Use the HTTP API?**\n\n- **External Integration**: Trigger UFO tasks from web apps, scripts, or CI/CD pipelines\n- **Stateless**: No WebSocket connection required\n- **RESTful**: Standard HTTP methods and JSON payloads\n- **Monitoring**: Health checks for load balancers and monitoring systems\n\n---\n\n## 📡 Endpoints\n\n### POST /api/dispatch\n\nSend a task to a connected device without establishing a WebSocket connection. Ideal for external systems, web apps, and automation scripts.\n\n#### Request Format\n\n**Corrected Request Body** (based on actual source code):\n\n```json\n{\n \"client_id\": \"device_windows_001\",\n \"request\": \"Open Chrome and navigate to github.com\",\n \"task_name\": \"github_navigation_task\"\n}\n```\n\n**Request Schema:**\n\n| Field | Type | Required | Default | Description |\n|-------|------|----------|---------|-------------|\n| `client_id` | `string` | ✅ **Yes** | - | Target client identifier (device or constellation) |\n| `request` | `string` | ✅ **Yes** | - | Natural language task description (user request) |\n| `task_name` | `string` | ⚠️ No | Auto-generated UUID | Human-readable task identifier |\n\n**Important:** The correct parameter names (verified from source code) are:\n- `client_id` (not `device_id`)\n- `request` (not `task`)\n- `task_name` (optional identifier)\n\n#### Success Response (200)\n\n```json\n{\n \"status\": \"dispatched\",\n \"task_name\": \"github_navigation_task\",\n \"client_id\": \"device_windows_001\",\n \"session_id\": \"d4e5f6a7-b8c9-1234-5678-9abcdef01234\"\n}\n```\n\n**Response Schema:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `status` | `string` | Always `\"dispatched\"` on success |\n| `task_name` | `string` | Task identifier (from request or auto-generated) |\n| `client_id` | `string` | Target client ID |\n| `session_id` | `string` | UUID for tracking task execution (use with `/api/task_result`) |\n\n#### Error Responses\n\n**Client Not Online (404):**\n```json\n{\n \"detail\": \"Client not online\"\n}\n```\n \n**Cause:** Target client is not connected to the server.\n \n**Solution:** Check `/api/clients` to see available clients.\n\n**Empty Client ID (400):**\n```json\n{\n \"detail\": \"Empty client ID\"\n}\n```\n \n**Cause:** `client_id` field is missing or empty.\n \n**Solution:** Provide a valid `client_id` in the request body.\n\n**Empty Task Content (400):**\n```json\n{\n \"detail\": \"Empty task content\"\n}\n```\n \n**Cause:** `request` field is missing or empty.\n \n**Solution:** Provide a non-empty task description in the `request` field.\n\n#### Implementation Details\n\n**Source Code** (verified from `ufo/server/services/api.py`):\n\n```python\n@router.post(\"/api/dispatch\")\nasync def dispatch_task_api(data: Dict[str, Any]):\n # Extract parameters\n client_id = data.get(\"client_id\")\n user_request = data.get(\"request\", \"\")\n task_name = data.get(\"task_name\", str(uuid4())) # Auto-generate if not provided\n \n # Validation: Empty request\n if not user_request:\n logger.error(f\"Got empty task content for client {client_id}.\")\n raise HTTPException(status_code=400, detail=\"Empty task content\")\n \n # Validation: Empty client ID\n if not client_id:\n logger.error(\"Client ID must be provided.\")\n raise HTTPException(status_code=400, detail=\"Empty client ID\")\n \n # Logging\n if not task_name:\n logger.warning(f\"Task name not provided, using {task_name}.\")\n else:\n logger.info(f\"Task name: {task_name}.\")\n \n logger.info(f\"Dispatching task '{user_request}' to client '{client_id}'\")\n \n # Get client WebSocket\n ws = client_manager.get_client(client_id)\n if not ws:\n logger.error(f\"Client {client_id} not online.\")\n raise HTTPException(status_code=404, detail=\"Client not online\")\n \n # Use AIP TaskExecutionProtocol to send task\n transport = WebSocketTransport(ws)\n task_protocol = TaskExecutionProtocol(transport)\n \n session_id = str(uuid4())\n response_id = str(uuid4())\n \n logger.info(\n f\"[AIP] Sending task assignment via API: task_name={task_name}, \"\n f\"session_id={session_id}, client_id={client_id}\"\n )\n \n # Send via AIP protocol\n await task_protocol.send_task_assignment(\n user_request=user_request,\n task_name=task_name,\n session_id=session_id,\n response_id=response_id,\n )\n \n return {\n \"status\": \"dispatched\",\n \"task_name\": task_name,\n \"client_id\": client_id,\n \"session_id\": session_id,\n }\n ```\n\n**Tip:** Use the returned `session_id` to track results via `GET /api/task_result/{task_name}`.\n\n#### Sequence Diagram\n\n```mermaid\nsequenceDiagram\n participant Client as External Client\n participant API as HTTP API\n participant WSM as Client Connection Manager\n participant WS as Client WebSocket\n \n Client->>API: POST /api/dispatch {client_id, request, task_name}\n \n Note over API: Validate request (client_id, request not empty)\n \n API->>WSM: get_client(client_id)\n WSM-->>API: WebSocket connection\n \n alt Client Not Online\n WSM-->>API: None\n API-->>Client: 404: Client not online\n end\n \n Note over API: Generate session_id Generate response_id\n \n API->>WS: send_task_assignment() (via AIP TaskExecutionProtocol)\n \n Note over WS: Task queued for execution\n \n API-->>Client: 200: {status: \"dispatched\", session_id, task_name}\n \n Note over Client: Poll /api/task_result/{task_name} to get result\n```\n\n---\n\n### GET /api/clients\n\nQuery all currently connected clients (devices and constellations) to determine which targets are available for task dispatch.\n\n#### Request\n\n```http\nGET /api/clients\n```\n\n**No parameters required.**\n\n#### Success Response (200)\n\n```json\n{\n \"online_clients\": [\n \"device_windows_001\",\n \"device_linux_002\",\n \"constellation_orchestrator_001\"\n ]\n}\n```\n\n**Response Schema:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `online_clients` | `array` | List of all connected client IDs |\n\n**Source Code:**\n\n```python\n@router.get(\"/api/clients\")\nasync def list_clients():\n return {\"online_clients\": client_manager.list_clients()}\n```\n\n#### Usage Patterns\n\n**Source Code:**\n\n```python\n@router.get(\"/api/clients\")\nasync def list_clients():\n return {\"online_clients\": client_manager.list_clients()}\n```\n\n#### Usage Patterns\n\n**Check Device Availability:**\n```python\nimport requests\n \nresponse = requests.get(\"http://localhost:5000/api/clients\")\nclients = response.json()[\"online_clients\"]\n \ntarget_device = \"device_windows_001\"\n \nif target_device in clients:\n print(f\"✅ {target_device} is online\")\n # Dispatch task\nelse:\n print(f\"❌ {target_device} is offline\")\n```\n\n**Filter by Client Type:**\n```python\n# Note: Current API doesn't return client types\n# You must know your client naming convention\n# Example: devices start with \"device_\", constellations with \"constellation_\"\n \nclients = response.json()[\"online_clients\"]\n \ndevices = [c for c in clients if c.startswith(\"device_\")]\nconstellations = [c for c in clients if c.startswith(\"constellation_\")]\n \nprint(f\"Devices online: {len(devices)}\")\nprint(f\"Constellations online: {len(constellations)}\")\n```\n\n**Monitor Client Count:**\n```python\nimport time\n \nwhile True:\n response = requests.get(\"http://localhost:5000/api/clients\")\n clients = response.json()[\"online_clients\"]\n \n print(f\"[{time.strftime('%H:%M:%S')}] Clients online: {len(clients)}\")\n \n time.sleep(10) # Check every 10 seconds\n```\n\n---\n\n### GET /api/task_result/{task_name}\n\nPoll this endpoint to get the result of a dispatched task. Use the `task_name` returned from `/api/dispatch`.\n\n#### Request\n\n```http\nGET /api/task_result/github_navigation_task\n```\n\n**Path Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `task_name` | `string` | Task identifier (from `/api/dispatch` response) |\n\n#### Response States\n\n**Pending (200):**\nTask is still running:\n \n```json\n{\n \"status\": \"pending\"\n}\n```\n \n**Action:** Continue polling until status changes to `\"done\"`.\n\n**Completed (200):**\nTask has finished:\n \n```json\n{\n \"status\": \"done\",\n \"result\": {\n \"action\": \"Opened Chrome and navigated to github.com\",\n \"screenshot\": \"base64_encoded_image_data\",\n \"control_label\": \"Address bar\",\n \"control_text\": \"github.com\"\n }\n}\n```\n \n**Action:** Process the result. The `result` structure depends on the task type and device implementation.\n\n**Not Found (Implicit):**\nIf `task_name` doesn't exist in session manager:\n \n```json\n{\n \"status\": \"pending\"\n}\n```\n \n**Note:** Current implementation returns `{\"status\": \"pending\"}` for non-existent tasks (not a 404 error).\n\n#### Implementation Details\n\n**Source Code:**\n\n```python\n@router.get(\"/api/task_result/{task_name}\")\nasync def get_task_result(task_name: str):\n # Query session manager for result\n result = session_manager.get_result_by_task(task_name)\n \n if not result:\n return {\"status\": \"pending\"}\n \n return {\"status\": \"done\", \"result\": result}\n```\n\n**Note on Result Retention:**\n\nResults are stored in memory and may be cleared after:\n\n- Server restart\n- Session cleanup (if implemented)\n- Memory limits reached\n\n**Recommendation:** Poll frequently and persist results on the client side.\n\n#### Polling Pattern\n\n**Recommended Polling Implementation:**\n\n```python\n import requests\n import time\n \n def wait_for_result(task_name: str, timeout: int = 300, interval: int = 2) -> dict:\n \"\"\"\n Poll for task result with timeout.\n \n Args:\n task_name: Task identifier\n timeout: Maximum wait time in seconds (default: 5 minutes)\n interval: Poll interval in seconds (default: 2 seconds)\n \n Returns:\n Task result dictionary\n \n Raises:\n TimeoutError: If task doesn't complete within timeout\n \"\"\"\n start_time = time.time()\n \n while True:\n elapsed = time.time() - start_time\n \n if elapsed > timeout:\n raise TimeoutError(\n f\"Task '{task_name}' did not complete within {timeout}s\"\n )\n \n response = requests.get(\n f\"http://localhost:5000/api/task_result/{task_name}\"\n )\n data = response.json()\n \n if data[\"status\"] == \"done\":\n print(f\"✅ Task completed in {elapsed:.1f}s\")\n return data[\"result\"]\n \n print(f\"⏳ Waiting for task... ({elapsed:.0f}s)\")\n time.sleep(interval)\n \n # Usage\n try:\n result = wait_for_result(\"github_navigation_task\", timeout=60)\n print(\"Result:\", result)\n except TimeoutError as e:\n print(f\"❌ {e}\")\n ```\n\n---\n\n### GET /api/health\n\nUse this endpoint for monitoring systems, load balancers, and Kubernetes liveness/readiness probes.\n\n#### Request\n\n```http\nGET /api/health\n```\n\n**No parameters required.**\n\n#### Success Response (200)\n\n```json\n{\n \"status\": \"healthy\",\n \"online_clients\": [\n \"device_windows_001\",\n \"device_linux_002\",\n \"constellation_orchestrator_001\"\n ]\n}\n```\n\n**Response Schema:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `status` | `string` | Always `\"healthy\"` if server is responding |\n| `online_clients` | `array` | List of connected client IDs |\n\n#### Implementation Details\n\n**Source Code:**\n\n```python\n@router.get(\"/api/health\")\nasync def health_check():\n return {\n \"status\": \"healthy\",\n \"online_clients\": client_manager.list_clients()\n }\n```\n\n#### Integration Examples\n\n**Kubernetes Liveness Probe:**\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n name: ufo-server\nspec:\n containers:\n - name: ufo-server\n image: ufo-server:latest\n ports:\n - containerPort: 5000\n livenessProbe:\n httpGet:\n path: /api/health\n port: 5000\n initialDelaySeconds: 10\n periodSeconds: 30\n timeoutSeconds: 5\n failureThreshold: 3\n readinessProbe:\n httpGet:\n path: /api/health\n port: 5000\n initialDelaySeconds: 5\n periodSeconds: 10\n```\n\n**Monitoring Script:**\n```python\nimport requests\nimport time\n \ndef monitor_server_health(url=\"http://localhost:5000/api/health\"):\n \"\"\"Continuous health monitoring.\"\"\"\n consecutive_failures = 0\n \n while True:\n try:\n response = requests.get(url, timeout=5)\n \n if response.status_code == 200:\n data = response.json()\n client_count = len(data.get(\"online_clients\", []))\n \n print(\n f\"✅ Server healthy - {client_count} clients connected\"\n )\n consecutive_failures = 0\n else:\n consecutive_failures += 1\n print(\n f\"⚠️ Server returned {response.status_code} \"\n f\"(failures: {consecutive_failures})\"\n )\n except requests.RequestException as e:\n consecutive_failures += 1\n print(\n f\"❌ Server unreachable: {e} \"\n f\"(failures: {consecutive_failures})\"\n )\n \n if consecutive_failures >= 3:\n # Trigger alert (email, Slack, PagerDuty, etc.)\n send_alert(f\"Server down for {consecutive_failures} checks\")\n \n time.sleep(30)\n```\n\n**nginx Health Check:**\n```nginx\nupstream ufo_backend {\n server localhost:5000;\n \n # Health check (requires nginx plus or third-party module)\n check interval=10000 rise=2 fall=3 timeout=5000 type=http;\n check_http_send \"GET /api/health HTTP/1.0\\r\\n\\r\\n\";\n check_http_expect_alive http_2xx http_3xx;\n}\n```\n\n---\n\n## 💻 Usage Examples\n\n### Python (requests)\n\n**Complete Task Dispatch Workflow:**\n\n```python\n import requests\n import time\n \n BASE_URL = \"http://localhost:5000\"\n \n # Step 1: Check if target device is online\n response = requests.get(f\"{BASE_URL}/api/clients\")\n clients = response.json()[\"online_clients\"]\n \n target_client = \"device_windows_001\"\n \n if target_client not in clients:\n print(f\"❌ {target_client} is not online\")\n exit(1)\n \n print(f\"✅ {target_client} is online\")\n \n # Step 2: Dispatch task\n dispatch_response = requests.post(\n f\"{BASE_URL}/api/dispatch\",\n json={\n \"client_id\": target_client,\n \"request\": \"Open Notepad and type 'Hello from UFO API'\",\n \"task_name\": \"notepad_hello_world\"\n }\n )\n \n if dispatch_response.status_code != 200:\n print(f\"❌ Dispatch failed: {dispatch_response.json()}\")\n exit(1)\n \n dispatch_data = dispatch_response.json()\n task_name = dispatch_data[\"task_name\"]\n session_id = dispatch_data[\"session_id\"]\n \n print(f\"Task dispatched: {task_name} (session: {session_id})\")\n \n # Step 3: Poll for result\n print(\"⏳ Waiting for result...\")\n \n max_wait = 120 # 2 minutes\n poll_interval = 2\n waited = 0\n \n while waited < max_wait:\n result_response = requests.get(\n f\"{BASE_URL}/api/task_result/{task_name}\"\n )\n result_data = result_response.json()\n \n if result_data[\"status\"] == \"done\":\n print(f\"✅ Task completed!\")\n print(f\"Result: {result_data['result']}\")\n break\n \n time.sleep(poll_interval)\n waited += poll_interval\n print(f\"⏳ Still waiting... ({waited}s)\")\n else:\n print(f\"⚠️ Timeout: Task did not complete in {max_wait}s\")\n ```\n\n### cURL\n\n**Command-Line HTTP Requests:**\n\n**Dispatch Task:**\n```bash\n curl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"device_windows_001\",\n \"request\": \"Open Calculator\",\n \"task_name\": \"open_calculator\"\n }'\n \n # Response:\n # {\n # \"status\": \"dispatched\",\n # \"task_name\": \"open_calculator\",\n # \"client_id\": \"device_windows_001\",\n # \"session_id\": \"a1b2c3d4-...\"\n # }\n ```\n \n **Get Clients:**\n ```bash\n curl http://localhost:5000/api/clients\n \n # Response:\n # {\n # \"online_clients\": [\n # \"device_windows_001\",\n # \"device_linux_002\"\n # ]\n # }\n ```\n \n **Get Task Result:**\n ```bash\n curl http://localhost:5000/api/task_result/open_calculator\n \n # Response (pending):\n # {\"status\": \"pending\"}\n \n # Response (complete):\n # {\n # \"status\": \"done\",\n # \"result\": {\"action\": \"Opened Calculator\", ...}\n # }\n ```\n \n **Health Check:**\n ```bash\n curl http://localhost:5000/api/health\n \n # Response:\n # {\n # \"status\": \"healthy\",\n # \"online_clients\": [\"device_windows_001\", ...]\n # }\n ```\n\n### JavaScript (fetch)\n\n**Browser/Node.js Integration:**\n\n```javascript\n // Dispatch task and wait for result\n async function dispatchAndWait(clientId, request, taskName) {\n const BASE_URL = 'http://localhost:5000';\n \n // Step 1: Dispatch\n console.log(`📤 Dispatching task to ${clientId}...`);\n \n const dispatchResponse = await fetch(`${BASE_URL}/api/dispatch`, {\n method: 'POST',\n headers: {'Content-Type': 'application/json'},\n body: JSON.stringify({\n client_id: clientId,\n request: request,\n task_name: taskName\n })\n });\n \n if (!dispatchResponse.ok) {\n const error = await dispatchResponse.json();\n throw new Error(`Dispatch failed: ${error.detail}`);\n }\n \n const {session_id, task_name} = await dispatchResponse.json();\n console.log(`✅ Dispatched: ${task_name} (session: ${session_id})`);\n \n // Step 2: Poll for result\n console.log('⏳ Waiting for result...');\n \n const maxWait = 120000; // 2 minutes in ms\n const pollInterval = 2000; // 2 seconds\n const startTime = Date.now();\n \n while (true) {\n const elapsed = Date.now() - startTime;\n \n if (elapsed > maxWait) {\n throw new Error(`Timeout: Task did not complete in ${maxWait / 1000}s`);\n }\n \n const resultResponse = await fetch(\n `${BASE_URL}/api/task_result/${task_name}`\n );\n const resultData = await resultResponse.json();\n \n if (resultData.status === 'done') {\n console.log('✅ Task completed!');\n return resultData.result;\n }\n \n console.log(`⏳ Still waiting... (${Math.floor(elapsed / 1000)}s)`);\n await new Promise(resolve => setTimeout(resolve, pollInterval));\n }\n }\n \n // Usage\n try {\n const result = await dispatchAndWait(\n 'device_windows_001',\n 'Open Chrome and go to google.com',\n 'chrome_google'\n );\n console.log('Result:', result);\n } catch (error) {\n console.error(', error.message);\n }\n ```\n\n---\n\n## ⚠️ Error Handling\n\n### Standard Error Format\n\nAll API errors follow FastAPI's standard format:\n\n```json\n{\n \"detail\": \"Error message description\"\n}\n```\n\n### HTTP Status Codes\n\n| Code | Meaning | When It Occurs | How to Handle |\n|------|---------|----------------|---------------|\n| **200** | OK | Request succeeded | Process response data |\n| **400** | Bad Request | Missing/empty `client_id` or `request` | Check request parameters |\n| **404** | Not Found | Client not online | Check `/api/clients` first |\n| **422** | Unprocessable Entity | Invalid JSON schema | Validate request body |\n| **500** | Internal Server Error | Unexpected server error | Retry or contact admin |\n\n### Error Handling Patterns\n\n**Robust Error Handling:**\n\n```python\n import requests\n from requests.exceptions import RequestException\n \n def dispatch_task_safe(client_id: str, request: str, task_name: str = None):\n \"\"\"\n Dispatch task with comprehensive error handling.\n \n Returns:\n dict: Response data if successful\n None: If dispatch failed\n \"\"\"\n try:\n response = requests.post(\n \"http://localhost:5000/api/dispatch\",\n json={\n \"client_id\": client_id,\n \"request\": request,\n \"task_name\": task_name\n },\n timeout=10\n )\n \n # Raise exception for 4xx/5xx status codes\n response.raise_for_status()\n \n return response.json()\n \n except requests.HTTPError as e:\n if e.response.status_code == 400:\n detail = e.response.json().get(\"detail\", \"Unknown error\")\n print(f\"Bad request: {detail}\")\n \n if \"Empty client ID\" in detail:\n print(\" Ensure 'client_id' is provided and not empty\")\n elif \"Empty task content\" in detail:\n print(\" Ensure 'request' is provided and not empty\")\n \n elif e.response.status_code == 404:\n print(f\"Client '{client_id}' is not online\")\n print(\" Check /api/clients for available devices\")\n \n elif e.response.status_code == 422:\n print(f\"Invalid request format\")\n print(\" Verify JSON structure matches API schema\")\n \n else:\n print(f\"HTTP {e.response.status_code}: {e.response.text}\")\n \n return None\n \n except requests.Timeout:\n print(\"Request timeout (server not responding)\")\n return None\n \n except RequestException as e:\n print(f\"Network error: {e}\")\n return None\n \n # Usage\n result = dispatch_task_safe(\n \"device_windows_001\",\n \"Open Notepad\",\n \"notepad_task\"\n )\n \n if result:\n print(f\"✅ Dispatched successfully: {result['session_id']}\")\n else:\n print(\"❌ Dispatch failed, check errors above\")\n ```\n\n---\n\n## 💡 Best Practices\n\n### 1. Validate Client Availability\n\nAlways verify the target client is online before dispatching tasks.\n\n```python\ndef is_client_online(client_id: str) -> bool:\n \"\"\"Check if a client is currently connected.\"\"\"\n response = requests.get(\"http://localhost:5000/api/clients\")\n clients = response.json()[\"online_clients\"]\n return client_id in clients\n\n# Usage\nif is_client_online(\"device_windows_001\"):\n # Dispatch task\n pass\nelse:\n print(\"Device is offline\")\n```\n\n### 2. Implement Exponential Backoff\n\nUse exponential backoff to reduce server load when polling for results.\n\n```python\nimport time\n\ndef poll_with_backoff(task_name: str, max_wait: int = 300):\n \"\"\"Poll for result with exponential backoff.\"\"\"\n interval = 1 # Start with 1 second\n max_interval = 30 # Cap at 30 seconds\n waited = 0\n \n while waited < max_wait:\n response = requests.get(\n f\"http://localhost:5000/api/task_result/{task_name}\"\n )\n data = response.json()\n \n if data[\"status\"] == \"done\":\n return data[\"result\"]\n \n time.sleep(interval)\n waited += interval\n \n # Exponential backoff: 1s 2s 4s 8s 16s 30s (capped)\n interval = min(interval * 2, max_interval)\n \n raise TimeoutError(f\"Task did not complete in {max_wait}s\")\n```\n\n### 3. Use Health Checks for Monitoring\n\nIntegrate health checks into your monitoring infrastructure.\n\n```python\nimport requests\nimport logging\n\ndef check_server_health() -> bool:\n \"\"\"\n Check server health for monitoring.\n \n Returns:\n True if healthy, False otherwise\n \"\"\"\n try:\n response = requests.get(\n \"http://localhost:5000/api/health\",\n timeout=5\n )\n \n if response.status_code == 200:\n data = response.json()\n logging.info(\n f\"Server healthy - {len(data.get('online_clients', []))} clients\"\n )\n return True\n else:\n logging.warning(f\"Server returned {response.status_code}\")\n return False\n \n except requests.RequestException as e:\n logging.error(f\"Health check failed: {e}\")\n return False\n```\n\n### 4. Handle Timeouts Gracefully\n\nSet appropriate timeouts - different tasks have different execution times.\n\n```python\ndef dispatch_with_timeout(\n client_id: str,\n request: str,\n task_name: str,\n result_timeout: int = 60\n):\n \"\"\"Dispatch task and wait for result with custom timeout.\"\"\"\n \n # Dispatch (short timeout for HTTP request)\n dispatch_response = requests.post(\n \"http://localhost:5000/api/dispatch\",\n json={\"client_id\": client_id, \"request\": request, \"task_name\": task_name},\n timeout=10 # 10 seconds for dispatch\n )\n \n task_name = dispatch_response.json()[\"task_name\"]\n \n # Wait for result (longer timeout for task execution)\n start_time = time.time()\n \n while time.time() - start_time < result_timeout:\n result_response = requests.get(\n f\"http://localhost:5000/api/task_result/{task_name}\",\n timeout=5 # 5 seconds per poll\n )\n \n data = result_response.json()\n if data[\"status\"] == \"done\":\n return data[\"result\"]\n \n time.sleep(2)\n \n raise TimeoutError(\n f\"Task '{task_name}' did not complete within {result_timeout}s\"\n )\n```\n\n### 5. Log All API Interactions\n\n**Production Logging:**\n\n```python\n import logging\n import requests\n \n logging.basicConfig(\n level=logging.INFO,\n format='%(asctime)s - %(levelname)s - %(message)s'\n )\n \n def dispatch_with_logging(client_id: str, request: str, task_name: str):\n \"\"\"Dispatch task with detailed logging.\"\"\"\n \n logging.info(\n f\"Dispatching task: client_id={client_id}, \"\n f\"task_name={task_name}, request='{request}'\"\n )\n \n try:\n response = requests.post(\n \"http://localhost:5000/api/dispatch\",\n json={\n \"client_id\": client_id,\n \"request\": request,\n \"task_name\": task_name\n }\n )\n \n response.raise_for_status()\n data = response.json()\n \n logging.info(\n f\"Task dispatched successfully: session_id={data['session_id']}\"\n )\n \n return data\n \n except requests.HTTPError as e:\n logging.error(\n f\"Dispatch failed: {e.response.status_code} - \"\n f\"{e.response.json().get('detail')}\"\n )\n raise\n except Exception as e:\n logging.error(f\"Unexpected error during dispatch: {e}\")\n raise\n ```\n\n### 6. Cache Client List\n\nReduce API calls by caching the client list if you're dispatching multiple tasks.\n\n```python\nfrom datetime import datetime, timedelta\n\nclass ClientCache:\n def __init__(self, ttl_seconds=10):\n self.ttl = timedelta(seconds=ttl_seconds)\n self.cache = None\n self.last_fetch = None\n \n def get_clients(self):\n \"\"\"Get clients with caching.\"\"\"\n now = datetime.now()\n \n # Return cache if still valid\n if self.cache and self.last_fetch and (now - self.last_fetch) < self.ttl:\n return self.cache\n \n # Fetch new data\n response = requests.get(\"http://localhost:5000/api/clients\")\n self.cache = response.json()[\"online_clients\"]\n self.last_fetch = now\n \n return self.cache\n\n# Usage\ncache = ClientCache(ttl_seconds=30)\n\nfor task in tasks:\n clients = cache.get_clients() # Uses cache if within TTL\n if task[\"client_id\"] in clients:\n dispatch_task(task)\n```\n\n---\n\n## 🔗 Integration Points\n\n### API Router Architecture\n\n```mermaid\ngraph TB\n subgraph \"HTTP API Layer\"\n Router[FastAPI Router]\n Dispatch[POST /api/dispatch]\n Clients[GET /api/clients]\n Result[GET /api/task_result]\n Health[GET /api/health]\n end\n \n subgraph \"Service Layer\"\n WSM[Client Connection Manager]\n SM[Session Manager]\n end\n \n subgraph \"Protocol Layer\"\n AIP[AIP TaskExecutionProtocol]\n WS[WebSocket Transport]\n end\n \n Router --> Dispatch\n Router --> Clients\n Router --> Result\n Router --> Health\n \n Dispatch --> WSM\n Dispatch --> AIP\n Clients --> WSM\n Result --> SM\n Health --> WSM\n \n AIP --> WS\n WSM --> WS\n \n style Dispatch fill:#bbdefb\n style Clients fill:#c8e6c9\n style Result fill:#fff9c4\n style Health fill:#ffcdd2\n```\n\n### With Client Connection Manager\n\n**API ClientConnectionManager:**\n\n- `get_client(client_id)`: Get WebSocket connection for task dispatch\n- `list_clients()`: List all online clients\n\n**Example:**\n\n```python\n# In POST /api/dispatch\nws = client_manager.get_client(client_id)\nif not ws:\n raise HTTPException(status_code=404, detail=\"Client not online\")\n\n# In GET /api/clients\nclients = client_manager.list_clients()\nreturn {\"online_clients\": clients}\n```\n\n### With Session Manager\n\n**API SessionManager:**\n\n- `get_result_by_task(task_name)`: Retrieve task result by task name\n\n**Example:**\n\n```python\n# In GET /api/task_result/{task_name}\nresult = session_manager.get_result_by_task(task_name)\n\nif not result:\n return {\"status\": \"pending\"}\n\nreturn {\"status\": \"done\", \"result\": result}\n```\n\n### With AIP Protocol\n\n**API uses AIP for task dispatch:**\n\n```python\n# Create AIP protocol instance\ntransport = WebSocketTransport(ws)\ntask_protocol = TaskExecutionProtocol(transport)\n\n# Send task via AIP\nawait task_protocol.send_task_assignment(\n user_request=user_request,\n task_name=task_name,\n session_id=session_id,\n response_id=response_id,\n)\n```\n\n---\n\n## 📚 Complete API Reference\n\n### Endpoints Summary\n\n| Method | Endpoint | Description | Auth Required |\n|--------|----------|-------------|---------------|\n| `POST` | `/api/dispatch` | Dispatch task to client | No |\n| `GET` | `/api/clients` | List online clients | No |\n| `GET` | `/api/task_result/{task_name}` | Get task result | No |\n| `GET` | `/api/health` | Health check | No |\n\n**Note on Authentication:**\n\nThe current API implementation does **not** include authentication. For production deployments, consider adding:\n\n- API keys\n- OAuth2/JWT tokens\n- Rate limiting\n- IP whitelisting\n\n### Request/Response Models\n\n#### Dispatch Request\n\n```python\n{\n \"client_id\": str, # Required\n \"request\": str, # Required\n \"task_name\": str # Optional (auto-generated if not provided)\n}\n```\n\n#### Dispatch Response\n\n```python\n{\n \"status\": \"dispatched\",\n \"task_name\": str,\n \"client_id\": str,\n \"session_id\": str # UUID\n}\n```\n\n#### Clients Response\n\n```python\n{\n \"online_clients\": List[str]\n}\n```\n\n#### Task Result Response\n\n```python\n# Pending\n{\n \"status\": \"pending\"\n}\n\n# Complete\n{\n \"status\": \"done\",\n \"result\": Dict[str, Any] # Structure depends on task type\n}\n```\n\n#### Health Response\n\n```python\n{\n \"status\": \"healthy\",\n \"online_clients\": List[str]\n}\n```\n\n---\n\n## 🎓 Summary\n\nThe HTTP API provides a **stateless, RESTful interface** for external systems to interact with the UFO server without maintaining WebSocket connections.\n\n**Key Characteristics:**\n\n| Aspect | Details |\n|--------|---------|\n| **Protocol** | HTTP/1.1, REST, JSON |\n| **Port** | 5000 (default, configurable) |\n| **Authentication** | None (add for production) |\n| **State** | Stateless (uses Client Connection Manager for client state) |\n| **Task Dispatch** | Via AIP TaskExecutionProtocol |\n| **Result Retrieval** | Polling-based (no push notifications) |\n\n**Use Cases:**\n\n1. **Web Applications**: Trigger UFO tasks from web frontends\n2. **Automation Scripts**: Integrate UFO into CI/CD pipelines\n3. **External Tools**: Connect third-party systems to UFO\n4. **Monitoring**: Health checks for infrastructure monitoring\n\n**Architecture Position:**\n\n```mermaid\ngraph TD\n subgraph \"External World\"\n E1[Web App]\n E2[Python Script]\n E3[Automation Tool]\n end\n \n subgraph \"UFO Server\"\n API[HTTP API]\n WSM[Client Connection Manager]\n SM[Session Manager]\n WH[WebSocket Handler]\n end\n \n subgraph \"Clients\"\n D1[Device 1]\n D2[Device 2]\n C1[Constellation]\n end\n \n E1 -->|HTTP POST/GET| API\n E2 -->|HTTP POST/GET| API\n E3 -->|HTTP POST/GET| API\n \n API --> WSM\n API --> SM\n \n WSM --> WH\n SM --> WH\n \n WH <-->|WebSocket| D1\n WH <-->|WebSocket| D2\n WH <-->|WebSocket| C1\n \n style API fill:#bbdefb\n style WSM fill:#c8e6c9\n style SM fill:#fff9c4\n```\n\n**For More Information:**\n\n- [Server Overview](./overview.md) - UFO server architecture and components\n- [Client Connection Manager](./client_connection_manager.md) - Client registry and connection management\n- [Session Manager](./session_manager.md) - Task execution and result tracking\n- [Quick Start](./quick_start.md) - Get started with UFO server\n\n"
},
{
"path": "documents/docs/server/client_connection_manager.md",
"content": "# Client Connection Manager\n\nThe **ClientConnectionManager** is the central registry for all connected clients, maintaining connection state, session mappings, device information, and providing efficient lookup mechanisms for client routing and management.\n\nFor more context on how this component fits into the server architecture, see the [Server Overview](overview.md).\n\n---\n\n## 🎯 Overview\n\nThe Client Connection Manager serves as the \"address book\" and \"session tracker\" for the entire server:\n\n| Responsibility | Description | Benefit |\n|----------------|-------------|---------|\n| **Client Registry** | Store all connected device and constellation clients | Fast O(1) client lookup by ID |\n| **Session Tracking** | Map sessions to their constellation orchestrators | Enable proper cleanup on disconnection |\n| **Device Mapping** | Track which device is executing which session | Route task results correctly |\n| **Connection State** | Monitor which clients are online | Validate before dispatching tasks |\n| **System Info Caching** | Store device capabilities and configuration | Optimize constellation decision-making |\n| **Statistics** | Provide connection metrics | Monitoring and capacity planning |\n\n### Architecture Position\n\n```mermaid\ngraph TB\n subgraph \"Clients\"\n D1[Device 1]\n D2[Device 2]\n C1[Constellation 1]\n end\n \n subgraph \"Server - ClientConnectionManager\"\n WSM[Client Connection Manager]\n \n subgraph \"Storage\"\n CR[Client Registry online_clients]\n CS[Constellation Sessions _constellation_sessions]\n DS[Device Sessions _device_sessions]\n SI[System Info Cache system_info]\n end\n end\n \n subgraph \"Server Components\"\n WH[WebSocket Handler]\n SM[Session Manager]\n API[API Router]\n end\n \n D1 -->|\"add_client()\"| WSM\n D2 -->|\"add_client()\"| WSM\n C1 -->|\"add_client()\"| WSM\n \n WSM --> CR\n WSM --> CS\n WSM --> DS\n WSM --> SI\n \n WH -->|\"get_client()\"| WSM\n WH -->|\"is_device_connected()\"| WSM\n SM -->|\"get_device_sessions()\"| WSM\n API -->|\"list_clients()\"| WSM\n \n style WSM fill:#ffecb3\n style CR fill:#c8e6c9\n style CS fill:#bbdefb\n style DS fill:#f8bbd0\n```\n\n---\n\n## 📦 Core Data Structures\n\n### ClientInfo Dataclass\n\nEach connected client is represented by a `ClientInfo` dataclass that stores all relevant connection details:\n\n```python\n@dataclass\nclass ClientInfo:\n \"\"\"Information about a connected client.\"\"\"\n websocket: WebSocket # Active WebSocket connection\n client_type: ClientType # DEVICE or CONSTELLATION\n connected_at: datetime # Connection timestamp\n metadata: Dict = None # Additional client metadata\n platform: str = \"windows\" # OS platform (windows/linux)\n system_info: Dict = None # Device system information (for devices only)\n \n # AIP protocol instances for this client\n transport: Optional[WebSocketTransport] = None # AIP WebSocket transport\n task_protocol: Optional[TaskExecutionProtocol] = None # AIP task protocol\n```\n\n**Field Descriptions:**\n\n| Field | Type | Purpose | Example |\n|-------|------|---------|---------|\n| `websocket` | `WebSocket` | FastAPI WebSocket connection object | `` |\n| `client_type` | `ClientType` | Whether DEVICE or CONSTELLATION | `ClientType.DEVICE` |\n| `connected_at` | `datetime` | When client registered | `2024-11-04 14:30:22` |\n| `metadata` | `Dict` | Custom metadata from registration message | `{\"hostname\": \"WIN-001\"}` |\n| `platform` | `str` | Operating system | `\"windows\"`, `\"linux\"` |\n| `system_info` | `Dict` | Device capabilities and system specs | See System Info Structure below |\n| `transport` | `Optional[WebSocketTransport]` | AIP WebSocket transport layer | `` |\n| `task_protocol` | `Optional[TaskExecutionProtocol]` | AIP task execution protocol handler | `` |\n\n**System Info Structure Example:**\n\n```json\n{\n \"os\": \"Windows\",\n \"os_version\": \"11 Pro 22H2\",\n \"processor\": \"Intel Core i7-1185G7\",\n \"memory_total\": 17014632448,\n \"memory_available\": 8459743232,\n \"screen_resolution\": \"1920x1080\",\n \"installed_applications\": [\"Chrome\", \"Excel\", \"Notepad++\"],\n \"supported_features\": [\"ui_automation\", \"web_browsing\", \"file_ops\"],\n \"custom_metadata\": {\n \"tags\": [\"production\", \"office\"],\n \"tier\": \"high_performance\"\n }\n}\n```\n\n---\n\n## 👥 Client Registry Management\n\nThe client registry (`online_clients`) is the authoritative source of truth for all connected clients.\n\n### Adding Clients\n\n```python\ndef add_client(\n self,\n client_id: str,\n platform: str,\n ws: WebSocket,\n client_type: ClientType = ClientType.DEVICE,\n metadata: Dict = None,\n transport: Optional[WebSocketTransport] = None,\n task_protocol: Optional[TaskExecutionProtocol] = None\n):\n \"\"\"Register a new client connection.\"\"\"\n \n with self.lock: # Thread-safe access\n # Extract system info if provided (device clients only)\n system_info = None\n if metadata and \"system_info\" in metadata and client_type == ClientType.DEVICE:\n system_info = metadata.get(\"system_info\")\n \n # Merge with server-configured metadata if available\n server_config = self._device_configs.get(client_id, {})\n if server_config:\n system_info = self._merge_device_info(system_info, server_config)\n logger.info(f\"Merged server config for device {client_id}\")\n \n # Create ClientInfo and add to registry\n self.online_clients[client_id] = ClientInfo(\n websocket=ws,\n platform=platform,\n client_type=client_type,\n connected_at=datetime.now(),\n metadata=metadata or {},\n system_info=system_info,\n transport=transport,\n task_protocol=task_protocol\n )\n```\n\n**Example - Adding a Device Client:**\n\n```python\nclient_manager.add_client(\n client_id=\"device_windows_001\",\n platform=\"windows\",\n ws=websocket,\n client_type=ClientType.DEVICE,\n metadata={\n \"hostname\": \"WIN-OFFICE-01\",\n \"system_info\": {\n \"os\": \"Windows\",\n \"screen_resolution\": \"1920x1080\",\n \"installed_applications\": [\"Chrome\", \"Excel\"]\n }\n },\n transport=websocket_transport,\n task_protocol=task_execution_protocol\n)\n```\n\n**Example - Adding a Constellation Client:**\n\n```python\nclient_manager.add_client(\n client_id=\"constellation_orchestrator_001\",\n platform=\"linux\", # Platform of the constellation server\n ws=websocket,\n client_type=ClientType.CONSTELLATION,\n metadata={\n \"orchestrator_version\": \"2.0.0\",\n \"max_concurrent_tasks\": 10\n },\n transport=websocket_transport,\n task_protocol=task_execution_protocol\n)\n```\n\n**Thread Safety:**\n\n```python\nwith self.lock: # threading.Lock ensures atomic operations\n self.online_clients[client_id] = client_info\n```\n\n!!! warning \"Client ID Uniqueness\"\n If a client reconnects with the same `client_id`, the new connection **overwrites** the old entry. This effectively disconnects the old WebSocket. Use unique IDs to prevent collisions.\n\n### Retrieving Clients\n\nThe ClientConnectionManager provides several methods to lookup clients based on different criteria:\n\n**Get WebSocket Connection:**\n```python\ndef get_client(self, client_id: str) -> WebSocket | None:\n \"\"\"Get WebSocket connection for a client.\"\"\"\n with self.lock:\n client_info = self.online_clients.get(client_id)\n return client_info.websocket if client_info else None\n```\n \n**Usage:**\n```python\ntarget_ws = client_manager.get_client(\"device_windows_001\")\nif target_ws:\n await target_ws.send_text(message)\n```\n\n**Get Full Client Info:**\n```python\ndef get_client_info(self, client_id: str) -> ClientInfo | None:\n \"\"\"Get complete information about a client.\"\"\"\n with self.lock:\n return self.online_clients.get(client_id)\n```\n \n**Usage:**\n```python\nclient_info = client_manager.get_client_info(\"device_windows_001\")\nif client_info:\n print(f\"Platform: {client_info.platform}\")\n print(f\"Connected at: {client_info.connected_at}\")\n print(f\"Type: {client_info.client_type}\")\n```\n\n**Get Client Type:**\n```python\ndef get_client_type(self, client_id: str) -> ClientType | None:\n \"\"\"Get the type of a client.\"\"\"\n with self.lock:\n client_info = self.online_clients.get(client_id)\n return client_info.client_type if client_info else None\n```\n \n**Usage:**\n```python\nclient_type = client_manager.get_client_type(\"client_001\")\nif client_type == ClientType.DEVICE:\n # Handle device-specific logic\nelif client_type == ClientType.CONSTELLATION:\n # Handle constellation-specific logic\n```\n\n**List All Clients:**\n```python\ndef list_clients(self) -> List[str]:\n \"\"\"List all online client IDs.\"\"\"\n with self.lock:\n return list(self.online_clients.keys())\n```\n \n**Usage:**\n```python\nonline_ids = client_manager.list_clients()\nprint(f\"Currently online: {len(online_ids)} clients\")\n```\n\n**List by Type:**\n```python\ndef list_clients_by_type(self, client_type: ClientType) -> List[str]:\n \"\"\"List all online clients of a specific type.\"\"\"\n with self.lock:\n return [\n client_id\n for client_id, client_info in self.online_clients.items()\n if client_info.client_type == client_type\n ]\n```\n \n**Usage:**\n```python\ndevices = client_manager.list_clients_by_type(ClientType.DEVICE)\nconstellations = client_manager.list_clients_by_type(ClientType.CONSTELLATION)\n \nprint(f\"Devices online: {len(devices)}\")\nprint(f\"Constellations online: {len(constellations)}\")\n```\n\n### Removing Clients\n\n```python\ndef remove_client(self, client_id: str):\n \"\"\"Remove a client from the registry.\"\"\"\n with self.lock:\n self.online_clients.pop(client_id, None)\n logger.info(f\"[ClientConnectionManager] Removed client: {client_id}\")\n```\n\n!!!danger \"Cleanup Required\"\n When removing a client, you should **also** clean up:\n \n - Session mappings (`_constellation_sessions`, `_device_sessions`)\n - Cached system info (automatically removed via ClientInfo deletion)\n - Active sessions (via SessionManager.cancel_task())\n \n See client disconnect cleanup pattern below.\n```\n\n---\n\n## 🔍 Connection State Checking\n\nAlways check if the target device is connected before attempting to dispatch tasks. This prevents errors and improves user experience.\n\n### Device Connection Validation\n\n```python\ndef is_device_connected(self, device_id: str) -> bool:\n \"\"\"Check if a device client is currently connected.\"\"\"\n \n with self.lock:\n client_info = self.online_clients.get(device_id)\n \n if not client_info:\n return False\n \n # Verify it's a DEVICE client (not constellation)\n return client_info.client_type == ClientType.DEVICE\n```\n\n**Example - Validate Before Task Dispatch:**\n\n```python\n# In WebSocket Handler - constellation requesting task on device\ntarget_device_id = data.target_id\n\nif not client_manager.is_device_connected(target_device_id):\n error_msg = f\"Target device '{target_device_id}' is not connected\"\n await send_error(error_msg)\n raise ValueError(error_msg)\n\n# Safe to dispatch\ntarget_ws = client_manager.get_client(target_device_id)\nawait dispatch_task(target_ws, task_request)\n```\n\n!!! warning \"Type Check is Critical\"\n The method returns `False` if the client exists but is **not a device** (e.g., it's a constellation). This prevents accidentally dispatching device tasks to constellation clients.\n\n### Generic Online Status Check\n\n```python\n# Not shown in source but implied\ndef is_online(self, client_id: str) -> bool:\n \"\"\"Check if any client (device or constellation) is currently online.\"\"\"\n with self.lock:\n return client_id in self.online_clients\n```\n\n**Comparison:**\n\n| Method | Checks | Returns True When |\n|--------|--------|-------------------|\n| `is_device_connected(device_id)` | Client exists **AND** is DEVICE type | Device client is online |\n| `is_online(client_id)` | Client exists (any type) | Any client is online |\n\n---\n\n## 📋 Session Mapping\n\nThe ClientConnectionManager tracks sessions from **two perspectives**:\n\n1. **Constellation → Sessions**: Which sessions did a constellation initiate?\n2. **Device → Sessions**: Which sessions is a device currently executing?\n\nThis dual tracking enables proper cleanup when either constellation or device disconnects.\n\n```mermaid\ngraph TB\n subgraph \"Constellation Perspective\"\n C[Constellation_001]\n CS[_constellation_sessions]\n CS --> S1[session_abc]\n CS --> S2[session_def]\n CS --> S3[session_ghi]\n end\n \n subgraph \"Device Perspective\"\n D[Device_windows_001]\n DS[_device_sessions]\n DS --> S1\n DS --> S4[session_jkl]\n end\n \n subgraph \"Disconnection Cleanup\"\n DC{Constellation Disconnects}\n DD{Device Disconnects}\n end\n \n DC -->|Cancel| S1\n DC -->|Cancel| S2\n DC -->|Cancel| S3\n \n DD -->|Cancel| S1\n DD -->|Cancel| S4\n \n style C fill:#bbdefb\n style D fill:#c8e6c9\n style S1 fill:#ffcdd2\n```\n\n### Constellation Session Mapping\n\nConstellation clients initiate tasks on remote devices. Track these sessions to enable cleanup when the orchestrator disconnects.\n\n**Add Constellation Session:**\n\n```python\ndef add_constellation_session(self, client_id: str, session_id: str):\n \"\"\"Map a session to its constellation orchestrator.\"\"\"\n \n with self.lock:\n if client_id not in self._constellation_sessions:\n self._constellation_sessions[client_id] = []\n self._constellation_sessions[client_id].append(session_id)\n```\n\n**Get Constellation Sessions:**\n\n```python\ndef get_constellation_sessions(self, client_id: str) -> List[str]:\n \"\"\"Get all sessions initiated by a constellation client.\"\"\"\n \n with self.lock:\n return self._constellation_sessions.get(client_id, []).copy()\n # .copy() prevents external modification of internal list\n```\n\n**Remove Constellation Sessions:**\n\n```python\ndef remove_constellation_sessions(self, client_id: str) -> List[str]:\n \"\"\"Remove and return all sessions for a constellation.\"\"\"\n \n with self.lock:\n return self._constellation_sessions.pop(client_id, [])\n # Returns removed sessions for cleanup\n```\n\n**Example - Constellation Disconnect Cleanup:**\n\n```python\n# In WebSocket Handler - when constellation disconnects\nconstellation_id = \"constellation_001\"\n\n# Get all sessions this constellation initiated\nsession_ids = client_manager.get_constellation_sessions(constellation_id)\n\nlogger.info(\n f\"Constellation {constellation_id} disconnected, \"\n f\"cancelling {len(session_ids)} sessions\"\n)\n\n# Cancel each session\nfor session_id in session_ids:\n await session_manager.cancel_task(\n session_id,\n reason=\"constellation_disconnected\" # Don't send callback\n )\n\n# Remove mappings\nclient_manager.remove_constellation_sessions(constellation_id)\n```\n\n### Device Session Mapping\n\nDevice clients execute tasks sent by constellations (or themselves). Track these sessions to enable cleanup when the device disconnects.\n\n**Add Device Session:**\n\n```python\ndef add_device_session(self, device_id: str, session_id: str):\n \"\"\"Map a session to the device executing it.\"\"\"\n \n with self.lock:\n if device_id not in self._device_sessions:\n self._device_sessions[device_id] = []\n self._device_sessions[device_id].append(session_id)\n```\n\n**Get Device Sessions:**\n\n```python\ndef get_device_sessions(self, device_id: str) -> List[str]:\n \"\"\"Get all sessions running on a specific device.\"\"\"\n \n with self.lock:\n return self._device_sessions.get(device_id, []).copy()\n```\n\n**Remove Device Sessions:**\n\n```python\ndef remove_device_sessions(self, device_id: str) -> List[str]:\n \"\"\"Remove and return all sessions for a device.\"\"\"\n \n with self.lock:\n return self._device_sessions.pop(device_id, [])\n```\n\n!!!example \"Device Disconnect Cleanup\"\n ```python\n # In WebSocket Handler - when device disconnects\n device_id = \"device_windows_001\"\n \n # Get all sessions running on this device\n session_ids = client_manager.get_device_sessions(device_id)\n \n logger.info(\n f\"Device {device_id} disconnected, \"\n f\"cancelling {len(session_ids)} sessions\"\n )\n \n # Cancel each session\n for session_id in session_ids:\n await session_manager.cancel_task(\n session_id,\n reason=\"device_disconnected\" # Send callback to constellation\n )\n \n # Remove mappings\n client_manager.remove_device_sessions(device_id)\n ```\n\n### Session Mapping Lifecycle\n\n```mermaid\nsequenceDiagram\n participant C as Constellation\n participant WH as WebSocket Handler\n participant WSM as ClientConnectionManager\n participant D as Device\n \n Note over C,D: Task Dispatch\n C->>WH: TASK request (target_id=device_001)\n WH->>WH: Generate session_id=\"session_abc\"\n \n Note over WH,WSM: Map Session to Both Clients\n WH->>WSM: add_constellation_session(\"constellation_001\", \"session_abc\")\n WH->>WSM: add_device_session(\"device_001\", \"session_abc\")\n \n Note over WSM: Session Mappings\n WSM->>WSM: _constellation_sessions[\"constellation_001\"] = [\"session_abc\"]\n WSM->>WSM: _device_sessions[\"device_001\"] = [\"session_abc\"]\n \n Note over WH,D: Task Execution\n WH->>D: TASK_ASSIGNMENT (session_abc)\n D->>D: Execute task\n \n Note over D,WH: Result Delivery\n D->>WH: TASK_END (session_abc)\n WH->>C: TASK_END (session_abc)\n \n Note over WH,WSM: Cleanup (not shown in actual code)\n Note right of WH: Sessions remain in mappings until client disconnects!\n```\n\n!!!warning \"Sessions Persist Until Cleanup\"\n Session mappings are **not automatically removed** when tasks complete. They persist until:\n \n 1. The constellation disconnects (removes all its sessions)\n 2. The device disconnects (removes all its sessions)\n 3. Manual cleanup (future feature)\n \n **Implication:** Over time, `_constellation_sessions` and `_device_sessions` can grow large. Consider implementing periodic cleanup for completed sessions.\n\n### Dual Mapping Example\n\n!!!example \"Single Session, Dual Mapping\"\n When a constellation dispatches a task to a device:\n \n ```python\n constellation_id = \"constellation_orchestrator_001\"\n device_id = \"device_windows_001\"\n session_id = \"session_abc123\"\n \n # Session is mapped to BOTH the constellation and the device\n client_manager.add_constellation_session(constellation_id, session_id)\n client_manager.add_device_session(device_id, session_id)\n \n # Later retrieval\n constellation_sessions = client_manager.get_constellation_sessions(constellation_id)\n # Returns: [\"session_abc123\", ...]\n \n device_sessions = client_manager.get_device_sessions(device_id)\n # Returns: [\"session_abc123\", ...]\n ```\n \n **Why dual mapping?**\n \n - If **constellation disconnects**: Cancel all its sessions (notify devices)\n - If **device disconnects**: Cancel all sessions on that device (notify constellations)\n\n---\n\n## 💻 System Information Management\n\nThe ClientConnectionManager caches device system information to enable intelligent task routing by constellations without repeatedly querying devices.\n\n### System Info Storage\n\n**Stored Automatically During Registration:**\n\n```python\ndef add_client(self, client_id, platform, ws, client_type, metadata):\n \"\"\"Add client and extract system info if provided.\"\"\"\n \n system_info = None\n if metadata and \"system_info\" in metadata and client_type == ClientType.DEVICE:\n system_info = metadata.get(\"system_info\")\n \n # Merge with server configuration if available\n server_config = self._device_configs.get(client_id, {})\n if server_config:\n system_info = self._merge_device_info(system_info, server_config)\n \n self.online_clients[client_id] = ClientInfo(\n websocket=ws,\n platform=platform,\n client_type=client_type,\n system_info=system_info, # Cached here\n ...\n )\n```\n\n### Retrieving System Information\n\n**Get Single Device Info:**\n```python\ndef get_device_system_info(self, device_id: str) -> Optional[Dict[str, Any]]:\n \"\"\"Get device system information by device ID.\"\"\"\n \n with self.lock:\n client_info = self.online_clients.get(device_id)\n if client_info and client_info.client_type == ClientType.DEVICE:\n return client_info.system_info\n return None\n```\n \n**Usage:**\n```python\ndevice_info = client_manager.get_device_system_info(\"device_windows_001\")\n \nif device_info:\n screen_res = device_info.get(\"screen_resolution\")\n apps = device_info.get(\"installed_applications\", [])\n \n print(f\"Screen: {screen_res}\")\n print(f\"Apps: {len(apps)} installed\")\n```\n\n**Get All Devices Info:**\n```python\ndef get_all_devices_info(self) -> Dict[str, Dict[str, Any]]:\n \"\"\"Get system information for all connected devices.\"\"\"\n \n with self.lock:\n return {\n device_id: client_info.system_info\n for device_id, client_info in self.online_clients.items()\n if client_info.client_type == ClientType.DEVICE\n and client_info.system_info\n }\n```\n \n**Usage:**\n```python\nall_devices = client_manager.get_all_devices_info()\n \nfor device_id, info in all_devices.items():\n print(f\"{device_id}: {info.get('os')} - {info.get('screen_resolution')}\")\n \n# Example output:\n# device_windows_001: Windows - 1920x1080\n# device_linux_001: Linux - 2560x1440\n```\n\n### Server Configuration Merging\n\nThe ClientConnectionManager supports loading device-specific configuration from YAML/JSON files and **merging** them with auto-detected system info.\n\n**Device Configuration File (`device_config.yaml`):**\n\n```yaml\ndevices:\n device_windows_001:\n tags: [\"production\", \"office\", \"high_priority\"]\n tier: \"high_performance\"\n additional_features: [\"excel_automation\", \"pdf_generation\"]\n max_concurrent_tasks: 5\n \n device_linux_001:\n tags: [\"development\", \"testing\"]\n tier: \"standard\"\n additional_features: [\"docker_support\"]\n```\n\n**Loading Configuration:**\n\n```python\n# Initialize ClientConnectionManager with config file\nclient_manager = ClientConnectionManager(device_config_path=\"config/device_config.yaml\")\n\n# Configuration is automatically loaded during __init__\n```\n\n**Merge Process:**\n\n```python\ndef _merge_device_info(\n self,\n system_info: Dict[str, Any],\n server_config: Dict[str, Any]\n) -> Dict[str, Any]:\n \"\"\"Merge auto-detected system info with server configuration.\"\"\"\n \n merged = {**system_info} # Start with auto-detected info\n \n # Add all server config to custom_metadata\n if \"custom_metadata\" not in merged:\n merged[\"custom_metadata\"] = {}\n merged[\"custom_metadata\"].update(server_config)\n \n # Special handling: merge capabilities\n if \"supported_features\" in system_info and \"additional_features\" in server_config:\n merged[\"supported_features\"] = list(\n set(system_info[\"supported_features\"] + server_config[\"additional_features\"])\n )\n \n # Add server tags\n if \"tags\" in server_config:\n merged[\"tags\"] = server_config[\"tags\"]\n \n return merged\n```\n\n**Result:**\n\n```json\n{\n \"os\": \"Windows\",\n \"screen_resolution\": \"1920x1080\",\n \"supported_features\": [\n \"ui_automation\",\n \"web_browsing\",\n \"file_ops\",\n \"excel_automation\",\n \"pdf_generation\"\n ],\n \"tags\": [\"production\", \"office\", \"high_priority\"],\n \"custom_metadata\": {\n \"tier\": \"high_performance\",\n \"max_concurrent_tasks\": 5,\n \"tags\": [\"production\", \"office\", \"high_priority\"],\n \"additional_features\": [\"excel_automation\", \"pdf_generation\"]\n }\n}\n```\n\n**Why Merge Configuration?**\n\n- **Auto-detected info**: Always accurate (OS, memory, screen resolution)\n- **Server config**: Administrative metadata (tags, tier, priorities)\n- **Combined**: Rich device profile for intelligent task routing\n\n---\n\n## 📊 Client Statistics and Monitoring\n\nThe `get_stats()` method provides basic metrics for monitoring connected clients.\n\n### Get Statistics\n\n```python\ndef get_stats(self) -> Dict[str, int]:\n \"\"\"Get statistics about connected clients.\"\"\"\n \n with self.lock:\n device_count = sum(\n 1\n for info in self.online_clients.values()\n if info.client_type == ClientType.DEVICE\n )\n constellation_count = sum(\n 1\n for info in self.online_clients.values()\n if info.client_type == ClientType.CONSTELLATION\n )\n return {\n \"total\": len(self.online_clients),\n \"device_clients\": device_count,\n \"constellation_clients\": constellation_count\n }\n```\n\n**Example Usage:**\n\n```python\n# Get current statistics\nstats = client_manager.get_stats()\n\nprint(f\"📊 Server Statistics:\")\nprint(f\" Total Clients: {stats['total']}\")\nprint(f\" Devices: {stats['device_clients']}\")\nprint(f\" Constellations: {stats['constellation_clients']}\")\n\n# Output:\n# 📊 Server Statistics:\n# Total Clients: 5\n# Devices: 3\n# Constellations: 2\n```\n\n### Filtering and Querying\n\n**Filter by Platform:**\n```python\ndef get_devices_by_platform(self, platform: str) -> List[str]:\n \"\"\"Get all device IDs for a specific platform.\"\"\"\n \n with self.lock:\n return [\n device_id\n for device_id, client_info in self.online_clients.items()\n if client_info.client_type == ClientType.DEVICE\n and client_info.platform == platform\n ]\n \n# Usage\nwindows_devices = client_manager.get_devices_by_platform(\"Windows\")\nlinux_devices = client_manager.get_devices_by_platform(\"Linux\")\n```\n\n**Filter by Connection Time:**\n```python\nfrom datetime import datetime, timedelta\n \ndef get_recently_connected(self, minutes: int = 5) -> List[str]:\n \"\"\"Get clients connected in the last N minutes.\"\"\"\n \n cutoff_time = datetime.now() - timedelta(minutes=minutes)\n \n with self.lock:\n return [\n client_id\n for client_id, client_info in self.online_clients.items()\n if client_info.connected_at >= cutoff_time\n ]\n \n# Usage\nrecent_clients = client_manager.get_recently_connected(minutes=10)\n```\n\n**Filter by Capability:**\n```python\ndef find_devices_with_capability(self, capability: str) -> List[str]:\n \"\"\"Find devices that support a specific capability.\"\"\"\n \n with self.lock:\n matches = []\n for device_id, client_info in self.online_clients.items():\n if client_info.client_type != ClientType.DEVICE:\n continue\n \n if not client_info.system_info:\n continue\n \n features = client_info.system_info.get(\"supported_features\", [])\n if capability in features:\n matches.append(device_id)\n \n return matches\n \n# Usage\nexcel_devices = client_manager.find_devices_with_capability(\"excel_automation\")\ndocker_devices = client_manager.find_devices_with_capability(\"docker_support\")\n```\n\n---\n\n## 🎯 Usage Patterns\n\n### Safe Task Dispatch\n\n```python\nasync def dispatch_task_to_device(\n client_manager: ClientConnectionManager,\n constellation_id: str,\n target_device_id: str,\n task_request: dict,\n session_id: str\n):\n \"\"\"Dispatch task with comprehensive validation.\"\"\"\n \n # Step 1: Validate constellation is connected\n if not client_manager.is_online(constellation_id):\n raise ValueError(f\"Constellation {constellation_id} not connected\")\n \n # Step 2: Validate target device is connected\n if not client_manager.is_device_connected(target_device_id):\n raise ValueError(f\"Device {target_device_id} not connected\")\n \n # Step 3: Get device WebSocket\n device_ws = client_manager.get_client(target_device_id)\n if not device_ws:\n raise ValueError(f\"Could not get WebSocket for device {target_device_id}\")\n \n # Step 4: Track session mappings\n client_manager.add_constellation_session(constellation_id, session_id)\n client_manager.add_device_session(target_device_id, session_id)\n \n # Step 5: Send task\n await device_ws.send_json({\n \"type\": \"TASK_ASSIGNMENT\",\n \"session_id\": session_id,\n \"request\": task_request\n })\n \n logger.info(\n f\"Task {session_id} dispatched: \"\n f\"{constellation_id} → {target_device_id}\"\n )\n```\n\n### Graceful Client Disconnect Handling\n\n```python\nasync def handle_client_disconnect(\n client_manager: ClientConnectionManager,\n session_manager: SessionManager,\n client_id: str,\n client_type: ClientType\n):\n \"\"\"Handle client disconnect with full cleanup.\"\"\"\n \n logger.info(f\"Client disconnected: {client_id} ({client_type})\")\n \n # Step 1: Get all related sessions\n if client_type == ClientType.CONSTELLATION:\n session_ids = client_manager.get_constellation_sessions(client_id)\n cancel_reason = \"constellation_disconnected\"\n else: # DEVICE\n session_ids = client_manager.get_device_sessions(client_id)\n cancel_reason = \"device_disconnected\"\n \n # Step 2: Cancel all sessions\n for session_id in session_ids:\n try:\n await session_manager.cancel_task(session_id, reason=cancel_reason)\n logger.info(f\"Cancelled session {session_id}\")\n except Exception as e:\n logger.error(f\"Failed to cancel {session_id}: {e}\")\n \n # Step 3: Remove session mappings\n if client_type == ClientType.CONSTELLATION:\n client_manager.remove_constellation_sessions(client_id)\n else:\n client_manager.remove_device_sessions(client_id)\n \n # Step 4: Remove client from registry\n client_manager.remove_client(client_id)\n \n logger.info(\n f\"Cleanup complete: {client_id}, \"\n f\"cancelled {len(session_ids)} sessions\"\n )\n```\n\n### Intelligent Device Selection\n\n```python\ndef select_optimal_device(\n client_manager: ClientConnectionManager,\n required_platform: str = None,\n required_capabilities: List[str] = None,\n preferred_tags: List[str] = None\n) -> Optional[str]:\n \"\"\"Select the best available device for a task.\"\"\"\n \n with client_manager.lock:\n candidates = []\n \n for device_id, client_info in client_manager.online_clients.items():\n # Filter by type\n if client_info.client_type != ClientType.DEVICE:\n continue\n \n # Filter by platform\n if required_platform and client_info.platform != required_platform:\n continue\n \n # Filter by capabilities\n if required_capabilities and client_info.system_info:\n features = client_info.system_info.get(\"supported_features\", [])\n if not all(cap in features for cap in required_capabilities):\n continue\n \n # Calculate score based on preferred tags\n score = 0\n if preferred_tags and client_info.system_info:\n tags = client_info.system_info.get(\"tags\", [])\n score = len(set(tags) & set(preferred_tags))\n \n candidates.append((device_id, score))\n \n if not candidates:\n return None\n \n # Return device with highest score (or first if all score 0)\n candidates.sort(key=lambda x: x[1], reverse=True)\n return candidates[0][0]\n\n# Usage\ndevice_id = select_optimal_device(\n client_manager,\n required_platform=\"Windows\",\n required_capabilities=[\"excel_automation\"],\n preferred_tags=[\"production\", \"high_priority\"]\n)\n\nif device_id:\n print(f\"Selected device: {device_id}\")\nelse:\n print(\"No suitable device available\")\n```\n\n### Session Cleanup After Task Completion\n\n**Note:** Current implementation does **not automatically remove** session mappings when tasks complete. Consider implementing this pattern:\n\n```python\nasync def handle_task_completion(\n client_manager: ClientConnectionManager,\n session_id: str,\n constellation_id: str,\n device_id: str\n):\n \"\"\"Clean up session mappings after task completes.\"\"\"\n \n # Task has completed (or failed)\n \n # Option 1: Remove from both mappings\n # (Requires adding remove_session method to ClientConnectionManager)\n # client_manager.remove_session(session_id)\n \n # Option 2: Leave mappings until disconnect\n # (Current behavior - sessions accumulate)\n \n logger.info(f\"Task {session_id} completed, mappings retained\")\n```\n\n---\n\n## 💡 Best Practices\n\n### Thread Safety\n\nThe ClientConnectionManager is accessed by multiple WebSocket handlers concurrently. **Always** acquire the lock before modifying shared state.\n\n```python\n# WRONG - No thread safety\ndef bad_example(self):\n if \"device_001\" in self.online_clients:\n client = self.online_clients[\"device_001\"]\n # Another thread might remove the client here!\n return client.websocket\n\n# CORRECT - Thread-safe\ndef good_example(self):\n with self.lock:\n if \"device_001\" in self.online_clients:\n client = self.online_clients[\"device_001\"]\n return client.websocket\n return None\n```\n\n### Validate Before Dispatch\n\nAlways check if the target device is connected before attempting to send messages.\n\n```python\n# CORRECT - Validation first\nif client_manager.is_device_connected(target_device_id):\n device_ws = client_manager.get_client(target_device_id)\n await device_ws.send_json(task_data)\nelse:\n logger.error(f\"Device {target_device_id} not connected\")\n # Handle error appropriately\n```\n\n### Cleanup on Disconnect\n\nWhen a client disconnects, clean up **all** related resources:\n\n**Checklist:**\n\n- [x] Cancel all related sessions\n- [x] Remove session mappings (constellation/device)\n- [x] Remove client from online registry\n- [x] Remove device info cache (if applicable)\n- [x] Notify affected parties\n\n### Cache Device Information\n\nBalance freshness and performance:\n\n- **Cache during registration**: Fast lookups for task routing\n- **Update on REQUEST_DEVICE_LIST**: Keep cache fresh\n- **Don't cache sensitive data**: Only cache non-sensitive system info\n\n```python\n# During registration - cache system info\nclient_manager.add_client(\n device_id,\n platform=\"Windows\",\n ws=websocket,\n client_type=ClientType.DEVICE,\n metadata={\"system_info\": system_info} # Cached automatically\n)\n\n# Later - fast lookup without querying device\ndevice_info = client_manager.get_device_system_info(device_id)\n```\n\n### Handle Edge Cases\n\n**Case 1: Client re-connects with same ID**\n ```python\n # Old connection still in registry\n if client_manager.is_online(client_id):\n logger.warning(f\"Client {client_id} already connected, removing old connection\")\n client_manager.remove_client(client_id)\n \n # Now add new connection\n client_manager.add_client(client_id, platform, ws, client_type, metadata)\n```\n\n**Case 2: Session mapped to disconnected clients**\n\n```python\n # Before dispatching\n if not client_manager.is_device_connected(device_id):\n # Device disconnected, session mapping might still exist\n # This is expected - cleanup happens on disconnect\n raise ValueError(f\"Device {device_id} no longer connected\")\n```\n\n**Case 3: Constellation and device both disconnect**\n\n```python\n # Session will be cancelled twice (once for each disconnect)\n # Ensure cancel_task is idempotent:\n async def cancel_task(self, session_id, reason):\n if session_id not in self.sessions:\n logger.debug(f\"Session {session_id} already cancelled\")\n return # Idempotent\n \n # Proceed with cancellation\n```\n\n### Monitor Session Accumulation\n\n**Note:** Session mappings are **not automatically removed** after task completion. Over time, this can cause memory growth.\n\n**Mitigation strategies:**\n\n**Periodic Cleanup:**\n```python\nasync def cleanup_completed_sessions(client_manager, session_manager):\n \"\"\"Remove mappings for completed sessions.\"\"\"\n \n all_sessions = set()\n all_sessions.update(\n session_id\n for sessions in client_manager._constellation_sessions.values()\n for session_id in sessions\n )\n all_sessions.update(\n session_id\n for sessions in client_manager._device_sessions.values()\n for session_id in sessions\n )\n \n for session_id in all_sessions:\n session = session_manager.get_session(session_id)\n if session and session.state in [SessionState.COMPLETED, SessionState.FAILED]:\n # Remove from ClientConnectionManager\n # (Requires implementing remove_session method)\n pass\n```\n\n**Cleanup on Completion:**\n```python\n# In task completion handler\nasync def on_task_complete(session_id, constellation_id, device_id):\n # Remove specific session from mappings\n client_manager.remove_session_from_constellation(constellation_id, session_id)\n client_manager.remove_session_from_device(device_id, session_id)\n```\n\n---\n\n## 🔗 Integration Points\n\n### With WebSocket Handler\n\n```mermaid\nsequenceDiagram\n participant WH as WebSocket Handler\n participant WSM as ClientConnectionManager\n participant SM as Session Manager\n \n Note over WH,WSM: Client Registration\n WH->>WSM: add_client(id, platform, ws, type, metadata)\n WSM-->>WH: Client added\n \n Note over WH,WSM: Task Dispatch\n WH->>WSM: is_device_connected(device_id)\n WSM-->>WH: True\n WH->>WSM: get_client(device_id)\n WSM-->>WH: WebSocket\n WH->>WSM: add_constellation_session(const_id, session_id)\n WH->>WSM: add_device_session(device_id, session_id)\n \n Note over WH,SM: Task Execution\n WH->>SM: execute_task_async(...)\n \n Note over WH,WSM: Client Disconnect\n WH->>WSM: get_constellation_sessions(client_id)\n WSM-->>WH: [session_ids...]\n WH->>SM: cancel_task(session_id, reason)\n WH->>WSM: remove_constellation_sessions(client_id)\n WH->>WSM: remove_client(client_id)\n```\n\n**ClientConnectionManager provides:**\n\n- Client registration and lookup\n- Connection state validation\n- Session tracking for cleanup\n\n**WebSocket Handler provides:**\n\n- WebSocket lifecycle management\n- Protocol message handling\n- Disconnect notifications\n\n### With Session Manager\n\n**ClientConnectionManager provides:**\n\n- Client connectivity status Session Manager checks before execution\n- Session mappings Session Manager uses for cleanup\n\n**Session Manager provides:**\n\n- Session state ClientConnectionManager can query to cleanup completed sessions (future)\n- Cancellation results ClientConnectionManager uses to notify clients\n\n### With API Router\n\n```python\n# In HTTP API endpoints\nfrom fastapi import APIRouter, HTTPException\n\n@router.get(\"/devices\")\nasync def list_devices():\n \"\"\"List all connected devices.\"\"\"\n stats = client_manager.get_stats()\n return {\n \"devices\": stats[\"devices\"][\"ids\"],\n \"count\": stats[\"devices\"][\"count\"],\n \"by_platform\": stats[\"devices\"][\"platforms\"]\n }\n\n@router.get(\"/devices/{device_id}\")\nasync def get_device_info(device_id: str):\n \"\"\"Get device system information.\"\"\"\n if not client_manager.is_device_connected(device_id):\n raise HTTPException(status_code=404, detail=\"Device not connected\")\n \n system_info = client_manager.get_device_system_info(device_id)\n return {\"device_id\": device_id, \"system_info\": system_info}\n\n@router.get(\"/stats\")\nasync def get_server_stats():\n \"\"\"Get server statistics.\"\"\"\n return client_manager.get_stats()\n```\n\n---\n\n## 📚 Complete API Reference\n\n### Client Management\n\n| Method | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `add_client()` | `client_id: str` `platform: str` `ws: WebSocket` `client_type: ClientType` `metadata: Optional[Dict]` | `None` | Register a new client connection |\n| `remove_client()` | `client_id: str` | `None` | Remove client from registry |\n| `get_client()` | `client_id: str` | `Optional[WebSocket]` | Get client WebSocket connection |\n| `get_client_info()` | `client_id: str` | `Optional[ClientInfo]` | Get full client information |\n| `get_client_type()` | `client_id: str` | `Optional[ClientType]` | Get client type (DEVICE/CONSTELLATION) |\n| `list_clients()` | `client_type: Optional[ClientType]` | `List[str]` | List all or filtered client IDs |\n\n### Connection State\n\n| Method | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `is_device_connected()` | `device_id: str` | `bool` | Check if device is connected |\n| `is_online()` | `client_id: str` | `bool` | Check if any client is online |\n\n### Session Mapping\n\n| Method | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `add_constellation_session()` | `client_id: str` `session_id: str` | `None` | Map session to constellation |\n| `get_constellation_sessions()` | `client_id: str` | `List[str]` | Get constellation's sessions |\n| `remove_constellation_sessions()` | `client_id: str` | `List[str]` | Remove and return sessions |\n| `add_device_session()` | `device_id: str` `session_id: str` | `None` | Map session to device |\n| `get_device_sessions()` | `device_id: str` | `List[str]` | Get device's sessions |\n| `remove_device_sessions()` | `device_id: str` | `List[str]` | Remove and return sessions |\n\n### Device Information\n\n| Method | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `get_device_system_info()` | `device_id: str` | `Optional[Dict]` | Get device system information |\n| `get_all_devices_info()` | None | `Dict[str, Dict]` | Get all devices' system info |\n\n### Statistics and Monitoring\n\n| Method | Parameters | Returns | Description |\n|--------|------------|---------|-------------|\n| `get_stats()` | None | `Dict[str, Any]` | Get comprehensive server statistics |\n\n### Data Structures\n\n**ClientInfo:**\n\n```python\n@dataclass\nclass ClientInfo:\n websocket: WebSocket # WebSocket connection\n client_type: ClientType # DEVICE or CONSTELLATION\n connected_at: datetime # Connection timestamp\n metadata: Optional[Dict] # Registration metadata\n platform: Optional[str] # \"Windows\", \"Linux\", \"Darwin\"\n system_info: Optional[Dict] # Device capabilities and configuration\n```\n\n**ClientType:**\n\n```python\nclass ClientType(Enum):\n DEVICE = \"device\" # Execution client\n CONSTELLATION = \"constellation\" # Orchestrator client\n```\n\n---\n\n## 🎓 Summary\n\nThe ClientConnectionManager is the **central registry** for all client connections and session mappings in the UFO server. It provides thread-safe operations for tracking clients, validating connectivity, mapping sessions, and caching device information.\n\n**Core Capabilities:**\n\n| Capability | Description | Key Benefit |\n|------------|-------------|-------------|\n| **Client Registry** | Track connected devices and constellation clients | Single source of truth for client state |\n| **Connection State** | Query client availability and type | Prevent dispatch to disconnected clients |\n| **Session Mapping** | Associate sessions with orchestrators and executors | Enable proper cleanup on disconnect |\n| **Device Info** | Cache device capabilities for routing decisions | Fast task routing without repeated queries |\n| **Thread Safety** | Lock-protected concurrent access | Safe operation in async/multi-threaded environment |\n| **Statistics** | Real-time metrics on clients and sessions | Monitoring and observability |\n\n**Key Design Patterns:**\n\n1. **Dual Session Mapping**: Track sessions from both constellation and device perspectives for comprehensive cleanup\n2. **Lazy Cleanup**: Session mappings persist until disconnect (consider periodic cleanup for production)\n3. **Configuration Merging**: Combine auto-detected device info with server-configured metadata\n4. **Type-Safe Validation**: Always verify client type (DEVICE vs CONSTELLATION) before operations\n\n**Integration with UFO Server:**\n\n```mermaid\ngraph TD\n subgraph \"ClientConnectionManager Core\"\n R[Client Registry]\n S[Session Mapping]\n D[Device Info Cache]\n T[Thread Safety]\n end\n \n WH[WebSocket Handler] -->|Register/Lookup| R\n WH -->|Track Sessions| S\n WH -->|Cache System Info| D\n \n SM[Session Manager] -->|Validate Connection| R\n SM -->|Query Sessions| S\n \n API[API Router] -->|List Devices| R\n API -->|Get Stats| R\n API -->|Device Info| D\n \n R -.->|Thread Lock| T\n S -.->|Thread Lock| T\n D -.->|Thread Lock| T\n \n style R fill:#bbdefb\n style S fill:#c8e6c9\n style D fill:#fff9c4\n style T fill:#ffcdd2\n```\n\n**For More Information:**\n\n- [Server Overview](./overview.md) - UFO server architecture and components\n- [WebSocket Handler](./websocket_handler.md) - AIP protocol message handling\n- [Session Manager](./session_manager.md) - Session lifecycle and background execution\n- [Quick Start](./quick_start.md) - Get started with UFO server\n\n"
},
{
"path": "documents/docs/server/monitoring.md",
"content": "# Monitoring and Observability\n\nMonitor the health, performance, and reliability of your UFO Server deployment with comprehensive observability tools, metrics, and alerting strategies.\n\n!!! tip \"Before You Begin\"\n Make sure you have the UFO Server running. See the [Quick Start Guide](./quick_start.md) for setup instructions.\n\n## 🎯 Overview\n\n```mermaid\ngraph TB\n subgraph \"Monitoring Layers\"\n Health[Health Checks]\n Metrics[Performance Metrics]\n Logs[Logs & Analysis]\n Alerts[Alerting]\n end\n \n subgraph \"UFO Server\"\n API[HTTP API]\n WS[WebSocket]\n end\n \n subgraph \"Tools\"\n K8s[Kubernetes]\n Prom[Prometheus]\n Slack[Notifications]\n end\n \n Health --> API\n Metrics --> WS\n Logs --> WS\n \n Health --> K8s\n Metrics --> Prom\n Alerts --> Slack\n \n style Health fill:#bbdefb\n style Metrics fill:#c8e6c9\n style Logs fill:#fff9c4\n style Alerts fill:#ffcdd2\n```\n\n**Monitoring Capabilities:**\n\n| Layer | Purpose | Tools |\n|-------|---------|-------|\n| **Health Checks** | Service availability and uptime | `/api/health`, Kubernetes probes |\n| **Performance Metrics** | Latency, throughput, resource usage | Prometheus, custom dashboards |\n| **Logging** | Event tracking, debugging, auditing | Python logging, log aggregation |\n| **Alerting** | Proactive issue detection | Slack, Email, PagerDuty |\n\n**Why Monitor?**\n\n- **Detect Issues Early**: Catch problems before users notice\n- **Performance Optimization**: Identify bottlenecks and inefficiencies\n- **Capacity Planning**: Track growth and resource utilization\n- **Debugging**: Trace errors and understand system behavior\n- **SLA Compliance**: Ensure service level objectives are met\n\n---\n\n## 🏥 Health Checks\n\n### HTTP Health Endpoint\n\nThe `/api/health` endpoint provides real-time server status without authentication. For detailed API specifications, see the [HTTP API Reference](./api.md).\n\n#### Endpoint Details\n\n```http\nGET /api/health\n```\n\n**Response (200 OK):**\n\n```json\n{\n \"status\": \"healthy\",\n \"online_clients\": [\n \"device_windows_001\",\n \"device_linux_002\",\n \"constellation_orchestrator_001\"\n ]\n}\n```\n\n**Response Schema:**\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `status` | `string` | Always `\"healthy\"` if server is responding |\n| `online_clients` | `array` | List of connected client IDs |\n\n**Quick Test:**\n\n```bash\n# Test health endpoint\ncurl http://localhost:5000/api/health\n\n# With jq for formatted output\ncurl -s http://localhost:5000/api/health | jq .\n```\n\n### Automated Health Monitoring\n\n#### Kubernetes Liveness and Readiness Probes\n\nExample production Kubernetes configuration:\n\n```yaml\napiVersion: v1\nkind: Pod\nmetadata:\n name: ufo-server\n labels:\n app: ufo-server\nspec:\n containers:\n - name: ufo-server\n image: ufo-server:latest\n ports:\n - containerPort: 5000\n name: http\n protocol: TCP\n \n # Liveness probe - restart container if failing\n livenessProbe:\n httpGet:\n path: /api/health\n port: 5000\n scheme: HTTP\n initialDelaySeconds: 30 # Wait 30s after startup\n periodSeconds: 10 # Check every 10s\n timeoutSeconds: 5 # 5s timeout per check\n successThreshold: 1 # 1 success = healthy\n failureThreshold: 3 # 3 failures = restart\n \n # Readiness probe - remove from service if failing\n readinessProbe:\n httpGet:\n path: /api/health\n port: 5000\n scheme: HTTP\n initialDelaySeconds: 10 # Wait 10s after startup\n periodSeconds: 5 # Check every 5s\n timeoutSeconds: 3 # 3s timeout\n successThreshold: 1 # 1 success = ready\n failureThreshold: 2 # 2 failures = not ready\n \n # Resource limits\n resources:\n requests:\n memory: \"256Mi\"\n cpu: \"250m\"\n limits:\n memory: \"512Mi\"\n cpu: \"500m\"\n```\n\n**Probe Configuration Guide:**\n\n| Parameter | Liveness | Readiness | Purpose |\n|-----------|----------|-----------|---------|\n| `initialDelaySeconds` | 30 | 10 | Time before first check (allow startup) |\n| `periodSeconds` | 10 | 5 | How often to check |\n| `timeoutSeconds` | 5 | 3 | Max time for response |\n| `failureThreshold` | 3 | 2 | Failures before action (restart/unready) |\n\n#### Uptime Monitoring Script\n\nExample continuous health monitoring script:\n\n```python\nimport requests\nimport time\nfrom datetime import datetime\nimport logging\n\nlogging.basicConfig(\n level=logging.INFO,\n format='%(asctime)s - %(levelname)s - %(message)s'\n)\nlogging.basicConfig(\n level=logging.INFO,\n format='%(asctime)s - %(levelname)s - %(message)s'\n)\n\nclass HealthMonitor:\n def __init__(self, url=\"http://localhost:5000/api/health\", interval=30):\n self.url = url\n self.interval = interval\n self.consecutive_failures = 0\n self.uptime_start = None\n self.total_checks = 0\n self.failed_checks = 0\n \n def check_health(self):\n \"\"\"Perform single health check.\"\"\"\n self.total_checks += 1\n \n try:\n response = requests.get(self.url, timeout=5)\n \n if response.status_code == 200:\n data = response.json()\n client_count = len(data.get(\"online_clients\", []))\n \n if self.uptime_start is None:\n self.uptime_start = datetime.now()\n \n uptime = datetime.now() - self.uptime_start\n availability = ((self.total_checks - self.failed_checks) / \n self.total_checks * 100)\n \n logging.info(\n f\"✅ Server healthy - {client_count} clients connected | \"\n f\"Uptime: {uptime} | Availability: {availability:.2f}%\"\n )\n \n self.consecutive_failures = 0\n return True\n else:\n raise Exception(f\"HTTP {response.status_code}\")\n \n except Exception as e:\n self.consecutive_failures += 1\n self.failed_checks += 1\n self.uptime_start = None # Reset uptime on failure\n \n logging.error(\n f\"❌ Health check failed: {e} \"\n f\"(consecutive failures: {self.consecutive_failures})\"\n )\n \n # Alert after 3 consecutive failures\n if self.consecutive_failures == 3:\n self.send_alert(\n f\"Server down for {self.consecutive_failures} checks! \"\n f\"Last error: {e}\"\n )\n \n return False\n \n def send_alert(self, message):\n \"\"\"Send alert (implement your alerting mechanism).\"\"\"\n logging.critical(f\"🚨 ALERT: {message}\")\n # TODO: Implement Slack/Email/PagerDuty notification\n \n def run(self):\n \"\"\"Run continuous monitoring.\"\"\"\n logging.info(f\"Starting health monitor (interval: {self.interval}s)\")\n \n while True:\n self.check_health()\n time.sleep(self.interval)\n\n# Run monitor\nif __name__ == \"__main__\":\n monitor = HealthMonitor(interval=30)\n monitor.run()\n```\n\n#### Docker Healthcheck\n\nDocker Compose health configuration:\n\n```yaml\nversion: '3.8'\n\nservices:\n ufo-server:\n image: ufo-server:latest\n ports:\n - \"5000:5000\"\n \n # Docker health check\n healthcheck:\n test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:5000/api/health\"]\n interval: 30s\n timeout: 5s\n retries: 3\n start_period: 40s\n \n restart: unless-stopped\n \n environment:\n - LOG_LEVEL=INFO\n \n volumes:\n - ./logs:/app/logs\n - ./config:/app/config\n```\n\n---\n\n## 📊 Performance Metrics\n\n### Request Latency Monitoring\n\nTrack API response times to detect performance degradation.\n\n#### Latency Measurement\n\n```python\nimport requests\nimport time\nimport statistics\nfrom typing import List, Dict\n\nclass LatencyMonitor:\n def __init__(self):\n self.measurements: Dict[str, List[float]] = {}\n \n def measure_endpoint(self, url: str, name: str = None) -> float:\n \"\"\"Measure endpoint latency in milliseconds.\"\"\"\n if name is None:\n name = url\n \n start = time.time()\n try:\n response = requests.get(url, timeout=10)\n latency_ms = (time.time() - start) * 1000\n \n # Store measurement\n if name not in self.measurements:\n self.measurements[name] = []\n self.measurements[name].append(latency_ms)\n \n return latency_ms\n except Exception as e:\n logging.error(f\"Failed to measure {name}: {e}\")\n return -1\n \n def get_stats(self, name: str) -> Dict[str, float]:\n \"\"\"Get statistics for an endpoint.\"\"\"\n if name not in self.measurements or not self.measurements[name]:\n return {}\n \n data = self.measurements[name]\n return {\n \"count\": len(data),\n \"min\": min(data),\n \"max\": max(data),\n \"mean\": statistics.mean(data),\n \"median\": statistics.median(data),\n \"p95\": statistics.quantiles(data, n=20)[18] if len(data) >= 20 else max(data),\n \"p99\": statistics.quantiles(data, n=100)[98] if len(data) >= 100 else max(data)\n }\n \n def print_report(self):\n \"\"\"Print latency report.\"\"\"\n print(\"\\n📊 Latency Report:\")\n print(\"=\" * 80)\n \n for name, measurements in self.measurements.items():\n stats = self.get_stats(name)\n \n print(f\"\\n{name}:\")\n print(f\" Count: {stats['count']}\")\n print(f\" Min: {stats['min']:.2f} ms\")\n print(f\" Max: {stats['max']:.2f} ms\")\n print(f\" Mean: {stats['mean']:.2f} ms\")\n print(f\" Median: {stats['median']:.2f} ms\")\n print(f\" P95: {stats['p95']:.2f} ms\")\n print(f\" P99: {stats['p99']:.2f} ms\")\n\n# Usage\nmonitor = LatencyMonitor()\n\nfor _ in range(100):\n monitor.measure_endpoint(\"http://localhost:5000/api/health\", \"health\")\n monitor.measure_endpoint(\"http://localhost:5000/api/clients\", \"clients\")\n time.sleep(1)\n\nmonitor.print_report()\n```\n\n**Sample Output:**\n\n```\n📊 Latency Report:\n================================================================================\n\nhealth:\n Count: 100\n Min: 2.34 ms\n Max: 45.67 ms\n Mean: 8.12 ms\n Median: 6.89 ms\n P95: 15.23 ms\n P99: 32.45 ms\n\nclients:\n Count: 100\n Min: 3.12 ms\n Max: 52.34 ms\n Mean: 10.45 ms\n Median: 9.12 ms\n P95: 18.90 ms\n P99: 38.67 ms\n```\n\n### Task Throughput Monitoring\n\nTrack task completion rate to detect bottlenecks.\n\n```python\nfrom collections import deque\nimport time\n\nclass ThroughputMonitor:\n def __init__(self, window_seconds=60):\n self.window = window_seconds\n self.completions = deque()\n self.total_completions = 0\n \n def record_completion(self):\n \"\"\"Record a task completion.\"\"\"\n now = time.time()\n self.completions.append(now)\n self.total_completions += 1\n \n # Remove completions outside the time window\n cutoff = now - self.window\n while self.completions and self.completions[0] < cutoff:\n self.completions.popleft()\n \n def get_rate_per_minute(self) -> float:\n \"\"\"Get current completion rate (tasks per minute).\"\"\"\n return len(self.completions) * (60.0 / self.window)\n \n def get_rate_per_second(self) -> float:\n \"\"\"Get current completion rate (tasks per second).\"\"\"\n return len(self.completions) / self.window\n \n def get_stats(self) -> dict:\n \"\"\"Get comprehensive statistics.\"\"\"\n return {\n \"window_seconds\": self.window,\n \"completions_in_window\": len(self.completions),\n \"rate_per_second\": self.get_rate_per_second(),\n \"rate_per_minute\": self.get_rate_per_minute(),\n \"total_completions\": self.total_completions\n }\n\n# Usage\nmonitor = ThroughputMonitor(window_seconds=60)\n\n# Record completions as they happen\nfor task in completed_tasks:\n monitor.record_completion()\n\n# Get current rate\nstats = monitor.get_stats()\nprint(f\"Current throughput: {stats['rate_per_minute']:.2f} tasks/min\")\nprint(f\"Tasks in last {stats['window_seconds']}s: {stats['completions_in_window']}\")\n```\n\n### Connection Stability Metrics\n\n!!! warning \"Monitor Client Connection Reliability\"\n Track disconnection rates to identify network or client issues. For more on client management, see the [Client Connection Manager](./client_connection_manager.md) documentation.\n\n```python\nfrom datetime import datetime, timedelta\n\nclass ConnectionStabilityMonitor:\n def __init__(self):\n self.connections = []\n self.disconnections = []\n self.reconnections = {} # client_id -> count\n \n def on_connect(self, client_id: str):\n \"\"\"Record client connection.\"\"\"\n now = datetime.now()\n self.connections.append({\n \"client_id\": client_id,\n \"timestamp\": now\n })\n \n # Track reconnections\n if client_id in self.reconnections:\n self.reconnections[client_id] += 1\n else:\n self.reconnections[client_id] = 0\n \n def on_disconnect(self, client_id: str, reason: str = \"unknown\"):\n \"\"\"Record client disconnection.\"\"\"\n now = datetime.now()\n self.disconnections.append({\n \"client_id\": client_id,\n \"timestamp\": now,\n \"reason\": reason\n })\n \n def get_stability_rate(self) -> float:\n \"\"\"\n Calculate connection stability (0.0 to 1.0).\n Returns: 1.0 - (disconnections / connections)\n \"\"\"\n if not self.connections:\n return 1.0\n \n return 1.0 - (len(self.disconnections) / len(self.connections))\n \n def get_disconnection_rate_per_hour(self) -> float:\n \"\"\"Get average disconnections per hour.\"\"\"\n if not self.disconnections:\n return 0.0\n \n first = self.disconnections[0][\"timestamp\"]\n last = self.disconnections[-1][\"timestamp\"]\n duration_hours = (last - first).total_seconds() / 3600\n \n if duration_hours == 0:\n return 0.0\n \n return len(self.disconnections) / duration_hours\n \n def get_flaky_clients(self, threshold=3) -> list:\n \"\"\"Get clients with excessive reconnections.\"\"\"\n return [\n (client_id, count)\n for client_id, count in self.reconnections.items()\n if count >= threshold\n ]\n \n def get_stats(self) -> dict:\n \"\"\"Get comprehensive stability statistics.\"\"\"\n return {\n \"total_connections\": len(self.connections),\n \"total_disconnections\": len(self.disconnections),\n \"stability_rate\": self.get_stability_rate(),\n \"disconnections_per_hour\": self.get_disconnection_rate_per_hour(),\n \"flaky_clients\": self.get_flaky_clients()\n }\n\n# Usage\nmonitor = ConnectionStabilityMonitor()\n\n# Record events\nmonitor.on_connect(\"device_windows_001\")\nmonitor.on_disconnect(\"device_windows_001\", reason=\"network_error\")\nmonitor.on_connect(\"device_windows_001\") # Reconnection\n\n# Get statistics\nstats = monitor.get_stats()\nprint(f\"Stability: {stats['stability_rate'] * 100:.1f}%\")\nprint(f\"Flaky clients: {stats['flaky_clients']}\")\n```\n\n---\n\n## 📝 Logging and Analysis\n\n### Log Configuration\n\nProduction logging setup:\n\n```python\nimport logging\n import sys\n from logging.handlers import RotatingFileHandler, TimedRotatingFileHandler\n import json\n from datetime import datetime\n \n # Custom JSON formatter for structured logging\n class JsonFormatter(logging.Formatter):\n def format(self, record):\n log_data = {\n \"timestamp\": datetime.utcnow().isoformat(),\n \"level\": record.levelname,\n \"logger\": record.name,\n \"message\": record.getMessage(),\n \"module\": record.module,\n \"function\": record.funcName,\n \"line\": record.lineno\n }\n \n # Add exception info if present\n if record.exc_info:\n log_data[\"exception\"] = self.formatException(record.exc_info)\n \n # Add custom fields\n if hasattr(record, 'client_id'):\n log_data[\"client_id\"] = record.client_id\n if hasattr(record, 'session_id'):\n log_data[\"session_id\"] = record.session_id\n \n return json.dumps(log_data)\n \n # Configure root logger\n def setup_logging(log_level=logging.INFO, log_dir=\"logs\"):\n \"\"\"Set up production logging configuration.\"\"\"\n \n # Create logger\n logger = logging.getLogger()\n logger.setLevel(log_level)\n \n # Remove default handlers\n logger.handlers = []\n \n # Console handler (human-readable)\n console_handler = logging.StreamHandler(sys.stdout)\n console_handler.setLevel(logging.INFO)\n console_formatter = logging.Formatter(\n '%(asctime)s - %(name)s - %(levelname)s - %(message)s'\n )\n console_handler.setFormatter(console_formatter)\n logger.addHandler(console_handler)\n \n # File handler (JSON, rotating by size)\n file_handler = RotatingFileHandler(\n filename=f\"{log_dir}/ufo_server.log\",\n maxBytes=10 * 1024 * 1024, # 10 MB\n backupCount=5, # Keep 5 backup files\n encoding='utf-8'\n )\n file_handler.setLevel(logging.DEBUG)\n file_handler.setFormatter(JsonFormatter())\n logger.addHandler(file_handler)\n \n # Daily rotation handler (for long-term storage)\n daily_handler = TimedRotatingFileHandler(\n filename=f\"{log_dir}/ufo_server_daily.log\",\n when='midnight',\n interval=1,\n backupCount=30, # Keep 30 days\n encoding='utf-8'\n )\n daily_handler.setLevel(logging.INFO)\n daily_handler.setFormatter(JsonFormatter())\n logger.addHandler(daily_handler)\n \n # Error-only handler (separate file for errors)\n error_handler = RotatingFileHandler(\n filename=f\"{log_dir}/ufo_server_errors.log\",\n maxBytes=5 * 1024 * 1024, # 5 MB\n backupCount=10,\n encoding='utf-8'\n )\n error_handler.setLevel(logging.ERROR)\n error_handler.setFormatter(JsonFormatter())\n logger.addHandler(error_handler)\n \n logger.info(\"Logging configured successfully\")\n \n # Usage\n setup_logging(log_level=logging.INFO, log_dir=\"./logs\")\n```\n\n### Log Event Categories\n\n**Key Events to Log:**\n\n**Connection Events:**\n\n```python\n# These log messages are generated by the WebSocket Handler\n# See: WebSocket Handler documentation for connection lifecycle details\nlogger.info(f\"[WS] ✅ Registered {client_type} client: {client_id}\", \n extra={\"client_id\": client_id, \"client_type\": client_type})\n\nlogger.info(f\"[WS] 🔌 Client disconnected: {client_id}\",\n extra={\"client_id\": client_id})\n```\n\n**Task Events:**\n\n```python\n# These log messages are generated by the Session Manager\n# See: Session Manager documentation for task lifecycle details\nlogger.info(f\"[Session] Created session {session_id} for task: {task_name}\",\n extra={\"session_id\": session_id, \"task_name\": task_name})\n \n logger.info(f\"[Session] Task completed: {session_id}\",\n extra={\"session_id\": session_id, \"duration_seconds\": duration})\n \nlogger.warning(f\"[Session] Task cancelled: {session_id} (reason: {reason})\",\n extra={\"session_id\": session_id, \"cancel_reason\": reason})\n```\n\n**Error Events:**\n\n```python\nlogger.error(f\"[WS] ❌ Failed to send result for session {session_id}: {error}\",\n extra={\"session_id\": session_id}, exc_info=True)\n\nlogger.error(f\"[Session] Task execution failed: {session_id}\",\n extra={\"session_id\": session_id}, exc_info=True)\n```\n\n### Log Analysis Scripts\n\nParse and analyze JSON logs:\n\n```python\n import json\n from collections import Counter, defaultdict\n from datetime import datetime\n \n def analyze_logs(log_file=\"logs/ufo_server.log\"):\n \"\"\"Analyze JSON logs and generate statistics.\"\"\"\n \n # Counters\n level_counts = Counter()\n module_counts = Counter()\n error_types = Counter()\n client_activity = defaultdict(int)\n hourly_activity = defaultdict(int)\n \n with open(log_file, 'r') as f:\n for line in f:\n try:\n log = json.loads(line)\n \n # Count by level\n level_counts[log.get(\"level\")] += 1\n \n # Count by module\n module_counts[log.get(\"module\")] += 1\n \n # Track errors\n if log.get(\"level\") in [\"ERROR\", \"WARNING\"]:\n error_types[log.get(\"message\")[:50]] += 1\n \n # Track client activity\n if \"client_id\" in log:\n client_activity[log[\"client_id\"]] += 1\n \n # Hourly activity\n timestamp = datetime.fromisoformat(log.get(\"timestamp\"))\n hour = timestamp.hour\n hourly_activity[hour] += 1\n \n except json.JSONDecodeError:\n continue\n \n # Print report\n print(\"\\n📊 Log Analysis Report\")\n print(\"=\" * 80)\n \n print(\"\\n📈 Events by Level:\")\n for level, count in level_counts.most_common():\n print(f\" {level:10s}: {count:6d}\")\n \n print(\"\\n📦 Events by Module:\")\n for module, count in module_counts.most_common(10):\n print(f\" {module:20s}: {count:6d}\")\n \n print(\"\\n⚠️ Top Errors/Warnings:\")\n for message, count in error_types.most_common(5):\n print(f\" [{count:3d}] {message}\")\n \n print(\"\\n👥 Top Active Clients:\")\n for client_id, count in sorted(client_activity.items(), \n key=lambda x: x[1], reverse=True)[:10]:\n print(f\" {client_id:30s}: {count:6d} events\")\n \n print(\"\\n🕐 Activity by Hour:\")\n for hour in sorted(hourly_activity.keys()):\n bar = \"█\" * (hourly_activity[hour] // 10)\n print(f\" {hour:02d}:00 - {bar} ({hourly_activity[hour]} events)\")\n \n # Run analysis\n analyze_logs(\"logs/ufo_server.log\")\n```\n\n---\n\n## 🚨 Alerting Systems\n\n### Alert Conditions\n\n!!! danger \"Critical Conditions to Monitor\"\n \n Track these critical conditions to maintain server reliability.\n \n **1. No Connected Devices**\n \n ```python\n def check_device_availability():\n \"\"\"Alert if no devices are connected.\"\"\"\n response = requests.get(\"http://localhost:5000/api/clients\")\n clients = response.json()[\"online_clients\"]\n \n devices = [c for c in clients if c.startswith(\"device_\")]\n \n if len(devices) == 0:\n send_alert(\n severity=\"critical\",\n title=\"No Devices Connected\",\n message=\"UFO server has no connected devices. Task dispatch unavailable.\"\n )\n return False\n elif len(devices) < 3:\n send_alert(\n severity=\"warning\",\n title=\"Low Device Count\",\n message=f\"Only {len(devices)} devices connected (expected 3+).\"\n )\n \n return True\n ```\n \n **2. High Error Rate**\n \n ```python\n def check_error_rate(log_file=\"logs/ufo_server.log\", threshold=0.1):\n \"\"\"Alert if error rate exceeds threshold.\"\"\"\n import json\n \n total = 0\n errors = 0\n \n with open(log_file, 'r') as f:\n for line in f:\n try:\n log = json.loads(line)\n total += 1\n if log.get(\"level\") in [\"ERROR\", \"CRITICAL\"]:\n errors += 1\n except:\n continue\n \n error_rate = errors / total if total > 0 else 0\n \n if error_rate > threshold:\n send_alert(\n severity=\"warning\",\n title=f\"High Error Rate: {error_rate * 100:.1f}%\",\n message=f\"{errors} errors out of {total} log entries\"\n )\n return False\n \n return True\n ```\n \n **3. Slow Response Times**\n \n ```python\n def check_latency(threshold_ms=1000):\n \"\"\"Alert if health endpoint is slow.\"\"\"\n start = time.time()\n \n try:\n response = requests.get(\"http://localhost:5000/api/health\", timeout=5)\n latency_ms = (time.time() - start) * 1000\n \n if latency_ms > threshold_ms:\n send_alert(\n severity=\"warning\",\n title=f\"Slow Response Time: {latency_ms:.0f}ms\",\n message=f\"/api/health responded in {latency_ms:.0f}ms (threshold: {threshold_ms}ms)\"\n )\n return False\n \n return True\n except Exception as e:\n send_alert(\n severity=\"critical\",\n title=\"Health Check Failed\",\n message=f\"Cannot reach health endpoint: {e}\"\n )\n return False\n ```\n \n **4. Session Failure Rate**\n \n ```python\n def check_session_failure_rate(threshold=0.2):\n \"\"\"Alert if too many sessions are failing.\"\"\"\n # Requires session tracking in logs\n import json\n \n completed = 0\n failed = 0\n \n with open(\"logs/ufo_server.log\", 'r') as f:\n for line in f:\n try:\n log = json.loads(line)\n message = log.get(\"message\", \"\")\n \n if \"Task completed\" in message:\n completed += 1\n elif \"Task failed\" in message or \"Task cancelled\" in message:\n failed += 1\n except:\n continue\n \n total = completed + failed\n failure_rate = failed / total if total > 0 else 0\n \n if failure_rate > threshold:\n send_alert(\n severity=\"warning\",\n title=f\"High Task Failure Rate: {failure_rate * 100:.1f}%\",\n message=f\"{failed} failed out of {total} tasks\"\n )\n return False\n \n return True\n ```\n\n### Alert Delivery Methods\n\n**Email Alerts:**\n```python\nimport smtplib\nfrom email.message import EmailMessage\n \ndef send_email_alert(title, message, severity=\"info\"):\n \"\"\"Send email alert via SMTP.\"\"\"\n \n # Email configuration\n smtp_host = \"smtp.gmail.com\"\n smtp_port = 587\n sender = \"alerts@example.com\"\n receiver = \"admin@example.com\"\n password = \"your_app_password\"\n \n # Create message\n msg = EmailMessage()\n msg['Subject'] = f\"[{severity.upper()}] UFO Server - {title}\"\n msg['From'] = sender\n msg['To'] = receiver\n \n # Email body\n body = f\"\"\"\n UFO Server Alert\n \n Severity: {severity.upper()}\n Title: {title}\n \n Message:\n {message}\n \n Timestamp: {datetime.now().isoformat()}\n \n --\n UFO Server Monitoring System\n \"\"\"\n msg.set_content(body)\n \n try:\n with smtplib.SMTP(smtp_host, smtp_port) as server:\n server.starttls()\n server.login(sender, password)\n server.send_message(msg)\n \n logging.info(f\"Email alert sent: {title}\")\n except Exception as e:\n logging.error(f\"Failed to send email alert: {e}\")\n```\n\n**Slack Alerts:**\n```python\nimport requests\n \ndef send_slack_alert(title, message, severity=\"info\"):\n \"\"\"Send alert to Slack via webhook.\"\"\"\n \n webhook_url = \"https://hooks.slack.com/services/YOUR/WEBHOOK/URL\"\n \n # Color coding by severity\n colors = {\n \"critical\": \"#ff0000\",\n \"error\": \"#ff6600\",\n \"warning\": \"#ffcc00\",\n \"info\": \"#00ccff\"\n }\n \n # Slack message payload\n payload = {\n \"attachments\": [{\n \"color\": colors.get(severity, \"#cccccc\"),\n \"title\": f\"🚨 UFO Server Alert - {title}\",\n \"text\": message,\n \"fields\": [\n {\n \"title\": \"Severity\",\n \"value\": severity.upper(),\n \"short\": True\n },\n {\n \"title\": \"Timestamp\",\n \"value\": datetime.now().strftime(\"%Y-%m-%d %H:%M:%S\"),\n \"short\": True\n }\n ],\n \"footer\": \"UFO Server Monitoring\"\n }]\n }\n \n try:\n response = requests.post(webhook_url, json=payload, timeout=5)\n response.raise_for_status()\n logging.info(f\"Slack alert sent: {title}\")\n except Exception as e:\n logging.error(f\"Failed to send Slack alert: {e}\")\n```\n\n**PagerDuty Integration:**\n```python\nimport requests\n \ndef send_pagerduty_alert(title, message, severity=\"error\"):\n \"\"\"Send alert to PagerDuty.\"\"\"\n \n routing_key = \"YOUR_PAGERDUTY_ROUTING_KEY\"\n \n # Map severity to PagerDuty severity\n pd_severity_map = {\n \"critical\": \"critical\",\n \"error\": \"error\",\n \"warning\": \"warning\",\n \"info\": \"info\"\n }\n \n payload = {\n \"routing_key\": routing_key,\n \"event_action\": \"trigger\",\n \"payload\": {\n \"summary\": title,\n \"source\": \"ufo-server\",\n \"severity\": pd_severity_map.get(severity, \"error\"),\n \"custom_details\": {\n \"message\": message,\n \"timestamp\": datetime.now().isoformat()\n }\n }\n }\n \n try:\n response = requests.post(\n \"https://events.pagerduty.com/v2/enqueue\",\n json=payload,\n timeout=5\n )\n response.raise_for_status()\n logging.info(f\"PagerDuty alert sent: {title}\")\n except Exception as e:\n logging.error(f\"Failed to send PagerDuty alert: {e}\")\n```\n\n### Unified Alert Function\n\n```python\ndef send_alert(title: str, message: str, severity: str = \"info\", \n channels: list = [\"slack\", \"email\"]):\n \"\"\"\n Send alert to multiple channels.\n \n Args:\n title: Alert title\n message: Alert message\n severity: One of \"critical\", \"error\", \"warning\", \"info\"\n channels: List of channels to send to\n \"\"\"\n for channel in channels:\n try:\n if channel == \"slack\":\n send_slack_alert(title, message, severity)\n elif channel == \"email\":\n send_email_alert(title, message, severity)\n elif channel == \"pagerduty\":\n send_pagerduty_alert(title, message, severity)\n else:\n logging.warning(f\"Unknown alert channel: {channel}\")\n except Exception as e:\n logging.error(f\"Failed to send alert via {channel}: {e}\")\n\n# Usage\nsend_alert(\n title=\"Server Healthy\",\n message=\"All systems operational\",\n severity=\"info\",\n channels=[\"slack\"]\n)\n\nsend_alert(\n title=\"No Devices Connected\",\n message=\"Critical: UFO server has no connected devices\",\n severity=\"critical\",\n channels=[\"slack\", \"email\", \"pagerduty\"]\n)\n```\n\n---\n\n## Best Practices\n\n### 1. Monitoring Strategy\n\n**Layered Monitoring Approach:**\n\n| Layer | Purpose | Frequency |\n|-------|---------|-----------|\n| **Health Checks** | Service availability | Every 30-60 seconds |\n| **Performance Metrics** | Response times, throughput | Continuous |\n| **Error Logs** | Debugging and diagnostics | Real-time |\n| **Alerts** | Critical issue notification | Event-driven |\n\n### 2. Alert Thresholds\n\n!!! warning \"Avoid Alert Fatigue\"\n Set reasonable thresholds to prevent excessive alerting:\n \n - **No devices for > 5 minutes**: Critical\n - **Error rate > 10%**: Warning\n - **Response time > 2 seconds**: Warning\n - **Session failure rate > 20%**: Warning\n - **3 consecutive health check failures**: Critical\n\n### 3. Log Retention\n\n**Log Retention Policy:**\n\n| Log Type | Retention | Storage |\n|----------|-----------|---------|\n| **Detailed logs** | 7 days | Local SSD |\n| **Summary logs** | 30 days | Local disk |\n| **Monthly summaries** | 1 year | Archive storage |\n| **Error logs** | 90 days | Separate file |\n\n### 4. Performance Baselines\n\n**Establish Baselines:**\n\nTrack normal operating metrics to detect anomalies:\n \n ```python\n BASELINE_METRICS = {\n \"health_latency_ms\": 10, # Typical: 5-15ms\n \"clients_latency_ms\": 15, # Typical: 10-20ms\n \"active_clients\": 5, # Expected: 3-10\n \"tasks_per_minute\": 2, # Expected: 1-5\n \"error_rate\": 0.02, # Expected: < 5%\n }\n \n # Alert if deviation > 50%\n if actual_latency > BASELINE_METRICS[\"health_latency_ms\"] * 1.5:\n send_alert(\"Performance degradation detected\")\n ```\n\n### 5. Multi-Channel Alerting\n\n!!!example \"Route Alerts by Severity\"\n \n ```python\n ALERT_ROUTING = {\n \"critical\": [\"slack\", \"email\", \"pagerduty\"],\n \"error\": [\"slack\", \"email\"],\n \"warning\": [\"slack\"],\n \"info\": [\"slack\"]\n }\n \n def send_alert(title, message, severity=\"info\"):\n channels = ALERT_ROUTING.get(severity, [\"slack\"])\n # Send to appropriate channels...\n```\n\n---\n\n## 🎓 Summary\n\nProduction monitoring requires a **layered approach** combining health checks, performance metrics, structured logging, and proactive alerting.\n\n**Monitoring Stack:**\n\n```mermaid\ngraph LR\n subgraph \"Collect\"\n HC[Health Checks]\n PM[Metrics]\n LOG[Logs]\n end\n \n subgraph \"Store & Analyze\"\n Files[Log Files]\n Dash[Dashboard]\n end\n \n subgraph \"Alert\"\n Rules[Alert Rules]\n Notify[Notifications]\n end\n \n HC --> Dash\n PM --> Dash\n LOG --> Files\n \n Files --> Rules\n Dash --> Rules\n Rules --> Notify\n \n style HC fill:#bbdefb\n style PM fill:#c8e6c9\n style LOG fill:#fff9c4\n style Rules fill:#ffcdd2\n```\n\n**Key Takeaways:**\n\n1. **Health Checks**: Use `/api/health` for liveness/readiness probes\n2. **Metrics**: Track latency, throughput, and stability continuously\n3. **Logging**: Use structured JSON logs for machine-readable analysis\n4. **Alerting**: Set up multi-channel alerts with appropriate thresholds\n5. **Dashboards**: Build real-time dashboards for visibility\n\n**For More Information:**\n\n- [HTTP API](./api.md) - Health endpoint details\n- [Client Connection Manager](./client_connection_manager.md) - Client statistics\n- [Session Manager](./session_manager.md) - Task tracking\n- [Quick Start](./quick_start.md) - Get started with UFO server\n\n## Next Steps\n\n- [Quick Start](./quick_start.md) - Get the server running\n- [HTTP API](./api.md) - API endpoint reference\n- [WebSocket Handler](./websocket_handler.md) - Connection management\n- [Session Manager](./session_manager.md) - Task execution tracking\n\n"
},
{
"path": "documents/docs/server/overview.md",
"content": "# Agent Server Overview\n\nThe **Agent Server** is the central orchestration engine that transforms UFO into a distributed multi-agent system, enabling seamless task coordination across heterogeneous devices through persistent WebSocket connections and robust state management.\n\nNew to the Agent Server? Start with the [Quick Start Guide](./quick_start.md) to get up and running in minutes.\n\n## What is the Agent Server?\n\nThe Agent Server is a **FastAPI-based asynchronous WebSocket server** that serves as the communication hub for UFO's distributed architecture. It bridges constellation orchestrators, device agents, and external systems through a unified protocol interface.\n\n### Core Responsibilities\n\n| Capability | Description | Key Benefit |\n|------------|-------------|-------------|\n| **🔌 Connection Management** | Tracks device & constellation client lifecycles | Real-time device availability awareness |\n| **🎯 Task Orchestration** | Coordinates execution across distributed devices | Centralized workflow control |\n| **💾 State Management** | Maintains session lifecycles & execution contexts | Stateful multi-turn task execution |\n| **🌐 Dual API Interface** | WebSocket (AIP) + HTTP (REST) endpoints | Flexible integration options |\n| **🛡️ Resilience** | Handles disconnections, timeouts, failures gracefully | Production-grade reliability |\n\n**Why Use the Agent Server?**\n\n- **Centralized Control**: Single point of orchestration for multi-device workflows\n- **Protocol Abstraction**: Clients communicate via [AIP](../aip/overview.md), hiding network complexity\n- **Async by Design**: Non-blocking execution enables high concurrency\n- **Platform Agnostic**: Supports Windows, Linux, macOS (in development)\n\nThe Agent Server is part of UFO's distributed **server-client architecture**, where it handles orchestration and state management while [Agent Clients](../client/overview.md) handle command execution. See [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md) for the complete design rationale and communication patterns.\n\n---\n\n## Architecture\n\nThe server follows a clean separation of concerns with distinct layers for web service, connection management, and protocol handling.\n\n### Architectural Overview\n\n**Component Interaction Diagram:**\n\n```mermaid\ngraph TB\n subgraph \"Web Layer\"\n FastAPI[FastAPI App]\n HTTP[HTTP API]\n WS[WebSocket /ws]\n end\n \n subgraph \"Service Layer\"\n WSM[Client Manager]\n SM[Session Manager]\n WSH[WebSocket Handler]\n end\n \n subgraph \"Clients\"\n DC[Device Clients]\n CC[Constellation Clients]\n end\n \n FastAPI --> HTTP\n FastAPI --> WS\n \n HTTP --> SM\n HTTP --> WSM\n WS --> WSH\n \n WSH --> WSM\n WSH --> SM\n \n DC -->|WebSocket| WS\n CC -->|WebSocket| WS\n \n style FastAPI fill:#e1f5ff\n style WSM fill:#fff4e1\n style SM fill:#f0ffe1\n style WSH fill:#ffe1f5\n```\n\nThis layered design ensures each component has a single, well-defined responsibility. The managers maintain state while the handler implements protocol logic.\n\n### Core Components\n\n| Component | Responsibility | Key Operations |\n|-----------|---------------|----------------|\n| **FastAPI Application** | Web service layer | ✅ HTTP endpoint routing ✅ WebSocket connection acceptance ✅ Request/response handling ✅ CORS and middleware |\n| **Client Connection Manager** | Connection registry | ✅ Client identity tracking ✅ Session ↔ client mapping ✅ Device info caching ✅ Connection lifecycle hooks |\n| **Session Manager** | Execution lifecycle | ✅ Platform-specific session creation ✅ Background async task execution ✅ Result callback delivery ✅ Session cancellation |\n| **WebSocket Handler** | Protocol implementation | ✅ AIP message parsing/routing ✅ Client registration ✅ Heartbeat monitoring ✅ Task/command dispatch |\n\n**Component Documentation:**\n- [Session Manager](./session_manager.md) - Session lifecycle and background execution\n- [Client Connection Manager](./client_connection_manager.md) - Connection registry and client tracking\n- [WebSocket Handler](./websocket_handler.md) - AIP protocol message handling\n- [HTTP API](./api.md) - REST endpoint specifications\n\n---\n\n## Key Capabilities\n\n### 1. Multi-Client Coordination\n\nThe server supports two distinct client types with different roles in the distributed architecture.\n\n**Client Type Comparison:**\n\n| Aspect | Device Client | Constellation Client |\n|--------|---------------|---------------------|\n| **Role** | Task executor | Task orchestrator |\n| **Connection** | Long-lived WebSocket | Long-lived WebSocket |\n| **Registration** | `ClientType.DEVICE` | `ClientType.CONSTELLATION` |\n| **Capabilities** | Local execution, telemetry | Multi-device coordination |\n| **Target Field** | Not required | Required for routing |\n| **Example** | Windows agent, Linux agent | ConstellationClient orchestrator |\n\n**Device Clients**\n- Execute tasks locally on Windows/Linux machines\n- Report hardware specs and real-time status\n- Respond to commands via MCP tool servers\n- Stream execution logs back to server\n\nSee [Agent Client Overview](../client/overview.md) for detailed client architecture.\n\n**Constellation Clients** \n- Orchestrate multi-device workflows from a central point\n- Dispatch tasks to specific target devices via `target_id`\n- Coordinate complex cross-device DAG execution\n- Aggregate results from multiple devices\n\nBoth client types connect to `/ws` and register using the `REGISTER` message. The server differentiates behavior based on `client_type` field. For the complete server-client architecture and design rationale, see [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md).\n\nSee [Quick Start](./quick_start.md) for registration examples.\n\n---\n\n### 2. Session Lifecycle Management\n\nUnlike stateless HTTP servers, the Agent Server maintains **session state** throughout task execution, enabling multi-turn interactions and result callbacks.\n\n**Session Lifecycle State Machine:**\n\n```mermaid\nstateDiagram-v2\n [*] --> Created: create_session()\n Created --> Running: Start execution\n Running --> Completed: Success\n Running --> Failed: Error\n Running --> Cancelled: Disconnect\n Completed --> [*]\n Failed --> [*]\n Cancelled --> [*]\n \n note right of Running\n Async background execution\n Non-blocking server\n end note\n```\n\n**Lifecycle Stages:**\n\n| Stage | Trigger | Session Manager Action | Server State |\n|-------|---------|----------------------|--------------|\n| **Created** | HTTP dispatch or AIP `TASK` | Platform-specific session instantiation | Session ID generated |\n| **Running** | Background task start | Async execution without blocking | Awaiting results |\n| **Completed** | `TASK_END` (success) | Callback delivery to client | Results cached |\n| **Failed** | `TASK_END` (error) | Error callback delivery | Error logged |\n| **Cancelled** | Client disconnect | Cancel async task, cleanup | Session removed |\n\n!!!warning \"Platform-Specific Sessions\"\n The SessionManager creates different session types based on the target platform:\n - **Windows**: `WindowsSession` with UI automation support\n - **Linux**: `LinuxSession` with bash automation\n - Auto-detected or overridden via `--platform` flag\n\n**Session Manager Responsibilities:**\n\n- ✅ **Platform abstraction**: Hides Windows/Linux differences\n- ✅ **Background execution**: Non-blocking async task execution\n- ✅ **Callback routing**: Delivers results via WebSocket\n- ✅ **Resource cleanup**: Cancels tasks on disconnect\n- ✅ **Result caching**: Stores results for HTTP retrieval\n\n---\n\n### 3. Resilient Communication\n\nThe server implements the [Agent Interaction Protocol (AIP)](../aip/overview.md), providing structured, type-safe communication with automatic failure handling.\n\n**Protocol Features:**\n\n| Feature | Implementation | Benefit |\n|---------|----------------|---------|\n| **Structured Messages** | Pydantic models with validation | Type safety, automatic serialization |\n| **Connection Health** | Heartbeat every 20-30s | Early failure detection |\n| **Error Recovery** | Exponential backoff reconnection | Transient fault tolerance |\n| **State Tracking** | Session client mapping | Proper cleanup on disconnect |\n| **Message Correlation** | `request_id`, `prev_response_id` chains | Request-response tracing |\n\n**Disconnection Handling Flow:**\n\n```mermaid\nsequenceDiagram\n participant Client\n participant Server\n participant SM as Session Manager\n \n Client-xServer: Connection lost\n Server->>SM: Cancel sessions\n SM->>SM: Cleanup resources\n Server->>Server: Remove from registry\n \n Note over Server: Client can reconnect with same client_id\n```\n\n!!!danger \"Important: Session Cancellation on Disconnect\"\n When a client disconnects (device or constellation), **all associated sessions are immediately cancelled** to prevent orphaned tasks and resource leaks.\n\n---\n\n### 4. Dual API Interface\n\nThe server provides two API styles to support different integration patterns: real-time WebSocket for agents and simple HTTP for external systems.\n\n**WebSocket API (AIP-based)**\n\nPurpose: Real-time bidirectional communication with agent clients\n\n| Message Type | Direction | Purpose |\n|--------------|-----------|---------|\n| `REGISTER` | Client Server | Initial capability advertisement |\n| `TASK` | Server Client | Task assignment with commands |\n| `COMMAND` | Server Client | Individual command execution |\n| `COMMAND_RESULTS` | Client Server | Execution results |\n| `TASK_END` | Bidirectional | Task completion notification |\n| `HEARTBEAT` | Bidirectional | Connection keepalive |\n| `DEVICE_INFO_REQUEST/RESPONSE` | Bidirectional | Telemetry exchange |\n| `ERROR` | Bidirectional | Error condition reporting |\n\n!!!example \"WebSocket Connection\"\n ```python\n import websockets\n \n async with websockets.connect(\"ws://localhost:5000/ws\") as ws:\n # Register as device client\n await ws.send(json.dumps({\n \"message_type\": \"REGISTER\",\n \"client_id\": \"windows_agent_001\",\n \"client_type\": \"device\",\n \"metadata\": {\"platform\": \"windows\", \"gpu\": \"NVIDIA RTX 3080\"}\n }))\n ```\n\n**HTTP REST API**\n\nPurpose: Task dispatch and monitoring from external systems (HTTP clients, CI/CD, etc.)\n\n| Endpoint | Method | Purpose | Authentication |\n|----------|--------|---------|----------------|\n| `/api/dispatch` | POST | Dispatch task to device | Optional (if configured) |\n| `/api/task_result/{task_name}` | GET | Retrieve task results | Optional |\n| `/api/clients` | GET | List connected clients | Optional |\n| `/api/health` | GET | Server health check | None |\n\n!!!example \"HTTP Task Dispatch\"\n ```bash\n # Dispatch task to device\n curl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"my_windows_device\",\n \"request\": \"Open Notepad and type Hello World\",\n \"task_name\": \"test_task_001\"\n }'\n \n # Response: {\"status\": \"dispatched\", \"session_id\": \"session_abc123\", \"task_name\": \"test_task_001\"}\n \n # Retrieve results\n curl http://localhost:5000/api/task_result/test_task_001\n ```\n\nSee [HTTP API Reference](./api.md) for complete endpoint documentation.\n\n---\n\n## Workflow Examples\n\n### Complete Task Dispatch Flow\n\n**End-to-End HTTP WebSocket Device Execution:**\n\n```mermaid\nsequenceDiagram\n participant EXT as External System\n participant HTTP as HTTP API\n participant SM as Session Manager\n participant WSH as WebSocket Handler\n participant DC as Device Client\n \n EXT->>HTTP: POST /api/dispatch {client_id, request, task_name}\n HTTP->>SM: create_session()\n SM->>SM: Create platform session\n SM-->>HTTP: session_id\n HTTP-->>EXT: 200 {session_id, task_name}\n \n SM->>WSH: send_task(session_id, task)\n WSH->>DC: TASK message (AIP)\n DC-->>WSH: ACK\n \n rect rgb(240, 255, 240)\n Note over DC: Background Execution\n DC->>DC: Execute via MCP tools\n DC->>DC: Generate results\n end\n \n DC->>WSH: COMMAND_RESULTS\n WSH->>SM: on_result_callback()\n SM->>SM: Cache results\n \n DC->>WSH: TASK_END (COMPLETED)\n WSH->>SM: on_task_end()\n \n EXT->>HTTP: GET /task_result/{session_id}\n HTTP->>SM: get_results()\n SM-->>HTTP: results\n HTTP-->>EXT: 200 {results}\n```\n\nThe green box highlights async execution on the device side, which doesn't block the server.\n\n### Multi-Device Constellation Workflow\n\n**Constellation Client Coordinating Multiple Devices:**\n\n```mermaid\nsequenceDiagram\n participant CC as Constellation Client\n participant Server as Agent Server\n participant D1 as Device 1 (GPU)\n participant D2 as Device 2 (CPU)\n \n CC->>Server: REGISTER (constellation)\n Server-->>CC: HEARTBEAT (OK)\n \n Note over CC: Plan multi-device DAG\n \n CC->>Server: TASK (target: device_1) Subtask 1: Image processing\n Server->>D1: TASK (forward)\n \n CC->>Server: TASK (target: device_2) Subtask 2: Data extraction\n Server->>D2: TASK (forward)\n \n par Parallel Execution\n D1->>D1: Process image on GPU\n D2->>D2: Extract data from DB\n end\n \n D1->>Server: COMMAND_RESULTS\n Server->>CC: COMMAND_RESULTS (from device_1)\n \n D2->>Server: COMMAND_RESULTS\n Server->>CC: COMMAND_RESULTS (from device_2)\n \n Note over CC: Combine results, Update DAG\n \n D1->>Server: TASK_END\n D2->>Server: TASK_END\n Server->>CC: TASK_END (both tasks)\n```\n\nThe server acts as a message router, forwarding tasks to target devices and routing results back to the constellation orchestrator. See [Constellation Documentation](../galaxy/overview.md) for more details on multi-device orchestration.\n\n---\n\n## Platform Support\n\nThe server automatically detects client platforms and creates appropriate session implementations.\n\n**Supported Platforms:**\n\n| Platform | Session Type | Capabilities | Status |\n|----------|--------------|--------------|--------|\n| **Windows** | `WindowsSession` | UI automation (UIA) COM API integration Native app control Screenshot capture | Full support |\n| **Linux** | `LinuxSession` | Bash automation GUI tools (xdotool) Package management Process control | Full support |\n| **macOS** | (Planned) | AppleScript UI automation Native app control | 🚧 In development |\n\n**Platform Auto-Detection:**\n\nThe server automatically detects the client's platform during registration. You can override this globally with the `--platform` flag when needed for testing or specific deployment scenarios.\n\n```bash\npython -m ufo.server.app --platform windows # Force Windows sessions\npython -m ufo.server.app --platform linux # Force Linux sessions\npython -m ufo.server.app # Auto-detect (default)\n```\n\n!!!warning \"When to Use Platform Override\"\n Use `--platform` override when:\n - Testing cross-platform sessions without actual devices\n - Running server in container different from target platform\n - Debugging platform-specific session behavior\n\nFor more details on platform-specific implementations, see [Windows Agent](../linux/overview.md) and [Linux Agent](../linux/overview.md).\n\n---\n\n## Configuration\n\nThe server runs out-of-the-box with sensible defaults. Advanced configuration inherits from UFO's central config system.\n\n### Command-Line Arguments\n\n```bash\npython -m ufo.server.app [OPTIONS]\n```\n\n**Available Options:**\n\n| Option | Type | Default | Description |\n|--------|------|---------|-------------|\n| `--port` | int | 5000 | Server listening port |\n| `--host` | str | `0.0.0.0` | Bind address (use `127.0.0.1` for localhost only) |\n| `--platform` | str | auto | Force platform (`windows`, `linux`) |\n| `--log-level` | str | `INFO` | Logging level (`DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`, `OFF`) |\n| `--local` | flag | False | Restrict to local connections only |\n\n!!!example \"Configuration Examples\"\n ```bash\n # Development: Local-only with debug logging\n python -m ufo.server.app --local --log-level DEBUG --port 8000\n \n # Production: External access, info logging\n python -m ufo.server.app --host 0.0.0.0 --port 5000 --log-level INFO\n \n # Testing: Force Linux sessions\n python -m ufo.server.app --platform linux --port 9000\n ```\n\n### UFO Configuration Inheritance\n\nThe server uses UFO's central configuration from `config_dev.yaml`:\n\n| Config Section | Inherited Settings |\n|----------------|-------------------|\n| **Agent Strategies** | HostAgent, AppAgent, EvaluationAgent configurations |\n| **LLM Models** | Model endpoints, API keys, temperature settings |\n| **Automators** | UI automation, COM API, web automation configs |\n| **Logging** | Log file paths, rotation, format |\n| **Prompts** | Agent system prompts, example templates |\n\nSee [Configuration Guide](../configuration/system/overview.md) for comprehensive config documentation.\n\n---\n\n## Monitoring & Operations\n\n### Health Monitoring\n\nMonitor server status and performance using HTTP endpoints.\n\n**Health Check Endpoints:**\n\n```bash\n# Server health and uptime\ncurl http://localhost:5000/api/health\n\n# Response:\n# {\n# \"status\": \"healthy\",\n# \"online_clients\": [...]\n# }\n\n# Connected clients list\ncurl http://localhost:5000/api/clients\n\n# Response:\n# {\n# \"online_clients\": [\"windows_001\", \"linux_002\", ...]\n# }\n```\n\nFor comprehensive monitoring strategies including performance metrics collection, log aggregation patterns, alert configuration, and dashboard setup, see [Monitoring Guide](./monitoring.md).\n\n### Error Handling\n\nThe server handles common failure scenarios gracefully to maintain system stability.\n\n**Disconnection Handling Matrix:**\n\n| Scenario | Server Detection | Automatic Action | Client Impact |\n|----------|-----------------|------------------|---------------|\n| **Device Disconnect** | Heartbeat timeout / WebSocket close | Cancel device sessions, notify constellation | Task fails, constellation retries |\n| **Constellation Disconnect** | Heartbeat timeout / WebSocket close | Continue device execution, skip callbacks | Device completes but results not delivered |\n| **Task Execution Failure** | `TASK_END` with error status | Log error, store in results | Client receives error via callback/HTTP |\n| **Network Partition** | Heartbeat timeout | Mark disconnected, enable reconnection | Client reconnects with same ID |\n| **Server Crash** | N/A | Clients detect via heartbeat | Clients reconnect to new instance |\n\n!!!note \"Reconnection Support\"\n Clients can reconnect with the same `client_id`. The server will re-register the client and restore heartbeat monitoring, but **will not restore previous sessions** (sessions are ephemeral).\n\n---\n\n## Best Practices\n\n### Development Environment\n\nOptimize your development workflow with these recommended practices.\n\n**Recommended Development Configuration:**\n\n```bash\n# Isolate to localhost, enable detailed logging\npython -m ufo.server.app \\\n --host 127.0.0.1 \\\n --port 5000 \\\n --local \\\n --log-level DEBUG\n```\n\n**Development Checklist:**\n\n- Use `--local` flag to prevent external access\n- Enable `DEBUG` logging for detailed traces\n- Monitor logs in separate terminal: `tail -f logs/ufo_server.log`\n- Test with single device before adding multiple clients\n- Use HTTP API for quick task dispatch testing\n- Verify heartbeat monitoring with client disconnection\n\n!!!example \"Development Testing Pattern\"\n ```bash\n # Terminal 1: Start server with debug logging\n python -m ufo.server.app --local --log-level DEBUG\n \n # Terminal 2: Connect device client\n python -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws\n \n # Terminal 3: Dispatch test task\n curl -X POST http://127.0.0.1:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\"client_id\": \"windowsagent\", \"request\": \"Open Notepad\", \"task_name\": \"test_001\"}'\n ```\n\n### Production Deployment\n\nThe default configuration is **not production-ready**. Implement these security and reliability measures.\n\n**Production Architecture:**\n\n```mermaid\ngraph LR\n Internet[Internet]\n LB[Load Balancer nginx/HAProxy]\n SSL[SSL/TLS Termination]\n \n subgraph \"UFO Server Cluster\"\n S1[Server Instance 1 :5000]\n S2[Server Instance 2 :5001]\n S3[Server Instance 3 :5002]\n end\n \n Monitor[Monitoring Prometheus/Grafana]\n PM[Process Manager systemd/PM2]\n \n Internet --> LB\n LB --> SSL\n SSL --> S1\n SSL --> S2\n SSL --> S3\n \n PM -.Manages.-> S1\n PM -.Manages.-> S2\n PM -.Manages.-> S3\n \n S1 -.Metrics.-> Monitor\n S2 -.Metrics.-> Monitor\n S3 -.Metrics.-> Monitor\n \n style LB fill:#ffe1f5\n style SSL fill:#fff4e1\n style Monitor fill:#f0ffe1\n```\n\n**Production Checklist:**\n\n| Category | Recommendation | Rationale |\n|----------|---------------|-----------|\n| **Reverse Proxy** | nginx, Apache, or cloud load balancer | SSL termination, rate limiting, DDoS protection |\n| **SSL/TLS** | Enable WSS (WebSocket Secure) | Encrypt client-server communication |\n| **Authentication** | Add auth middleware to FastAPI | Prevent unauthorized access |\n| **Process Management** | systemd (Linux), PM2 (Node.js), Docker | Auto-restart on crash, resource limits |\n| **Monitoring** | `/api/health` polling, metrics export | Detect issues proactively |\n| **Logging** | Structured logging, log aggregation (ELK) | Centralized debugging and audit trails |\n| **Resource Limits** | Set max connections, memory limits | Prevent resource exhaustion |\n\n**Example Nginx Configuration:**\n\n```nginx\nupstream ufo_server {\n server localhost:5000;\n}\n\nserver {\n listen 443 ssl;\n server_name ufo-server.example.com;\n \n ssl_certificate /path/to/cert.pem;\n ssl_certificate_key /path/to/key.pem;\n \n # WebSocket endpoint\n location /ws {\n proxy_pass http://ufo_server;\n proxy_http_version 1.1;\n proxy_set_header Upgrade $http_upgrade;\n proxy_set_header Connection \"upgrade\";\n proxy_set_header Host $host;\n proxy_read_timeout 3600s;\n }\n \n # HTTP API\n location /api/ {\n proxy_pass http://ufo_server;\n proxy_set_header Host $host;\n }\n}\n```\n\n\n\n### Scaling Strategies\n\nThe server can scale horizontally for high-load deployments, but requires careful session management.\n\n**Scaling Patterns:**\n\n| Pattern | Description | Use Case | Considerations |\n|---------|-------------|----------|----------------|\n| **Vertical** | Increase CPU/RAM on single instance | < 100 concurrent clients | Simplest, no session distribution |\n| **Horizontal (Sticky Sessions)** | Multiple instances with session affinity | 100-1000 clients | Load balancer routes same client to same instance |\n| **Horizontal (Shared State)** | Multiple instances with Redis | > 1000 clients | Requires session state externalization |\n\n!!!warning \"Current Limitation\"\n The current implementation stores session state in-memory. For horizontal scaling, use **sticky sessions** (client affinity) in your load balancer to route clients to consistent server instances. **Future**: Shared state backend (Redis) for true stateless horizontal scaling.\n\n\n\n---\n\n## Troubleshooting\n\n### Common Issues\n\n**Issue: Clients Can't Connect**\n\n```bash\n# Symptom: Connection refused\nError: WebSocket connection to 'ws://localhost:5000/ws' failed\n\n# Diagnosis:\n1. Check server is running: curl http://localhost:5000/api/health\n2. Verify port: netstat -an | grep 5000\n3. Check firewall: sudo ufw status\n\n# Solution:\n# Start server with correct host binding\npython -m ufo.server.app --host 0.0.0.0 --port 5000\n```\n\n**Issue: Sessions Not Executing**\n\n```bash\n# Symptom: Task dispatched but no results\n\n# Diagnosis:\n1. Check server logs for errors\n2. Verify client is connected: curl http://localhost:5000/api/clients\n3. Check target_id matches client_id\n\n# Solution:\n# Ensure client_id in request matches registered client\ncurl -X POST http://localhost:5000/api/dispatch \\\n -d '{\"client_id\": \"correct_client_id\", \"request\": \"test\", \"task_name\": \"test_001\"}'\n```\n\n**Issue: Memory Leak / High Memory Usage**\n\n```bash\n# Symptom: Server memory grows over time\n\n# Diagnosis:\n1. Check session cleanup in logs\n2. Monitor /api/health for session count\n3. Profile with memory_profiler\n\n# Solution:\n# Ensure clients send TASK_END to complete sessions\n# Restart server periodically (systemd handles this)\n# Implement session timeout (future feature)\n```\n\n### Debug Mode\n\n!!!example \"Enable Maximum Verbosity\"\n ```bash\n # Ultra-verbose debugging\n python -m ufo.server.app \\\n --log-level DEBUG \\\n --local \\\n --port 5000 2>&1 | tee debug.log\n \n # Watch logs in real-time\n tail -f debug.log | grep -E \"(ERROR|WARNING|Session|WebSocket)\"\n ```\n\n---\n\n## Documentation Map\n\nExplore related documentation to deepen your understanding of the Agent Server ecosystem.\n\n### Getting Started\n\n| Document | Purpose |\n|----------|---------|\n| [Quick Start](./quick_start.md) | Get server running in < 5 minutes |\n| [Client Registration](./quick_start.md) | How clients connect to server |\n\n### Architecture & Components\n\n| Document | Purpose |\n|----------|---------|\n| [Session Manager](./session_manager.md) | Task execution lifecycle deep-dive |\n| [Client Connection Manager](./client_connection_manager.md) | Connection registry internals |\n| [WebSocket Handler](./websocket_handler.md) | AIP protocol message handling |\n| [HTTP API](./api.md) | REST endpoint specifications |\n\n### Operations\n\n| Document | Purpose |\n|----------|---------|\n| [Monitoring](./monitoring.md) | Health checks, metrics, alerting |\n\n### Related Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [AIP Protocol](../aip/overview.md) | Communication protocol specification |\n| [Agent Architecture](../infrastructure/agents/overview.md) | Agent design and FSM framework |\n| [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md) | Distributed architecture rationale |\n| [Client Overview](../client/overview.md) | Device client architecture |\n| [MCP Integration](../mcp/overview.md) | Model Context Protocol tool servers |\n\n---\n\n## Next Steps\n\nFollow this recommended sequence to master the Agent Server:\n\n**1. Run the Server** (5 minutes)\n- Follow the [Quick Start Guide](./quick_start.md)\n- Verify server responds to `/api/health`\n\n**2. Connect a Client** (10 minutes)\n- Use [Device Client](../client/quick_start.md)\n- Verify registration in server logs\n- Check `/api/clients` endpoint\n\n**3. Dispatch Tasks** (15 minutes)\n- Use [HTTP API](./api.md) to send tasks\n- Retrieve results via `/api/task_result`\n- Observe WebSocket message flow in logs\n\n**4. Understand Architecture** (30 minutes)\n- Read [Session Manager](./session_manager.md) internals\n- Study [WebSocket Handler](./websocket_handler.md) protocol implementation\n- Review [AIP Protocol](../aip/overview.md) message types\n\n**5. Deploy to Production** (varies)\n- Set up reverse proxy (nginx)\n- Configure SSL/TLS\n- Implement monitoring\n- Test failover scenarios\n\n"
},
{
"path": "documents/docs/server/quick_start.md",
"content": "# Quick Start\n\nThis hands-on guide walks you through starting the UFO Agent Server, connecting clients, and dispatching your first task. Perfect for first-time users.\n\n---\n\n## 📋 Prerequisites\n\nBefore you begin, ensure you have:\n\n- **Python 3.10+** installed\n- **UFO dependencies** installed (`pip install -r requirements.txt`)\n- **Network connectivity** for WebSocket connections\n- **Terminal access** (PowerShell, bash, or equivalent)\n\n| Component | Minimum Version | Recommended |\n|-----------|----------------|-------------|\n| Python | 3.10 | 3.11+ |\n| FastAPI | 0.104+ | Latest |\n| Uvicorn | 0.24+ | Latest |\n| UFO | - | Latest commit |\n\n---\n\n## 🚀 Starting the Server\n\n### Basic Startup\n\nStart the server with default settings (port **5000**):\n\n```bash\npython -m ufo.server.app\n```\n\n**Expected Output:**\n\n```console\n2024-11-04 14:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5000\n2024-11-04 14:30:22 - ufo.server.app - INFO - Platform: auto-detected\n2024-11-04 14:30:22 - ufo.server.app - INFO - Log level: WARNING\nINFO: Started server process [12345]\nINFO: Waiting for application startup.\nINFO: Application startup complete.\nINFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)\n```\n\nOnce you see \"Uvicorn running\", the server is ready to accept WebSocket connections at `ws://0.0.0.0:5000/ws`.\n\n### Configuration Options\n\n| Argument | Type | Default | Description | Example |\n|----------|------|---------|-------------|---------|\n| `--port` | int | `5000` | Server listening port | `--port 8080` |\n| `--host` | str | `0.0.0.0` | Bind address (0.0.0.0 = all interfaces) | `--host 127.0.0.1` |\n| `--platform` | str | `auto` | Platform override (`windows`, `linux`) | `--platform windows` |\n| `--log-level` | str | `WARNING` | Logging verbosity | `--log-level DEBUG` |\n| `--local` | flag | `False` | Restrict to localhost connections only | `--local` |\n\n**Common Startup Configurations:**\n\n**Development (Local Only):**\n```bash\npython -m ufo.server.app --local --log-level DEBUG\n```\n- Accepts connections only from `localhost`\n- Verbose debug logging\n- Default port 5000\n\n**Custom Port:**\n```bash\npython -m ufo.server.app --port 8080\n```\n- Useful if port 5000 is already in use\n- Accessible from network\n\n**Production (Linux):**\n```bash\npython -m ufo.server.app --port 5000 --platform linux --log-level WARNING\n```\n- Explicit platform specification\n- Reduced logging for performance\n- Production-ready configuration\n\n**Multi-Interface Binding:**\n```bash\npython -m ufo.server.app --host 192.168.1.100 --port 5000\n```\n- Binds to specific network interface\n- Useful for multi-homed servers\n\n---\n\n## 🖥️ Connecting Device Clients\n\nA Device Client is an agent running on a physical or virtual machine that can execute tasks. Each device connects via WebSocket and registers with a unique `client_id`.\n\nOnce the server is running, connect device agents using the command line:\n\n### Platform-Specific Commands\n\n**Windows Device:**\n```bash\npython -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws --client-id my_windows_device\n```\n\n**Linux Device:**\n```bash\npython -m ufo.client.client --ws --ws-server ws://127.0.0.1:5000/ws --client-id my_linux_device --platform linux\n```\n\nWhen a client connects successfully, the server logs will display:\n```console\nINFO: [WS] 📱 Device client my_windows_device connected\n```\n\n### Client Connection Parameters\n\n| Parameter | Required | Type | Description | Example |\n|-----------|----------|------|-------------|---------|\n| `--ws` | Yes | flag | Enable WebSocket mode (vs. local mode) | `--ws` |\n| `--ws-server` | Yes | URL | Server WebSocket endpoint | `ws://127.0.0.1:5000/ws` |\n| `--client-id` | Yes | string | Unique device identifier (must be unique across all clients) | `device_win_001` |\n| `--platform` | ⚠️ Optional | string | Platform type: `windows`, `linux` | `--platform windows` |\n\n!!!warning \"Important: Client ID Uniqueness\"\n Each `client_id` must be globally unique. If a client connects with an existing ID, the old connection will be terminated.\n\n!!!tip \"Platform Auto-Detection\"\n If you don't specify `--platform`, the client will auto-detect the operating system. However, **explicit specification is recommended** for clarity.\n\n### Registration Protocol Flow\n\n```mermaid\nsequenceDiagram\n participant C as Device Client\n participant S as Agent Server\n \n C->>S: WebSocket CONNECT /ws\n S-->>C: Connection accepted\n \n C->>S: REGISTER {client_id, platform}\n S->>S: Validate & register\n S-->>C: REGISTER_CONFIRM\n \n Note over C: Client Ready\n```\n\nThe registration process uses the **Agent Interaction Protocol (AIP)** for structured communication. See [AIP Documentation](../aip/overview.md) for details.\n\n---\n\n## 🌌 Connecting Constellation Clients\n\nA Constellation Client is an orchestrator that coordinates multi-device tasks. It connects to the server and can dispatch work across multiple registered device clients.\n\n### Basic Constellation Connection\n\n```bash\npython -m galaxy.constellation.constellation --ws --ws-server ws://127.0.0.1:5000/ws --target-id my_windows_device\n```\n\n### Constellation Parameters\n\n| Parameter | Required | Description | Example |\n|-----------|----------|-------------|---------|\n| `--ws` | Yes | Enable WebSocket mode | `--ws` |\n| `--ws-server` | Yes | Server WebSocket URL | `ws://127.0.0.1:5000/ws` |\n| `--target-id` | ⚠️ Optional | Initial target device ID for tasks | `my_windows_device` |\n\n!!!danger \"Important: Target Device Must Be Online\"\n If you specify `--target-id`, that device **must already be connected** to the server. Otherwise, registration will fail with: `Target device 'my_windows_device' is not connected`\n\nA constellation can dynamically dispatch tasks to different devices, not just the `target-id`. For more on multi-device orchestration, see [Constellation Documentation](../galaxy/overview.md).\n\n---\n\n## Verifying the Setup\n\n### Method 1: Check Connected Clients\n\nUse the HTTP API to verify connections:\n\n```bash\ncurl http://localhost:5000/api/clients\n```\n\n**Expected Response:**\n\n```json\n{\n \"online_clients\": [\"my_windows_device\", \"my_linux_device\"]\n}\n```\n\nIf you see your `client_id` in the list, the device is successfully connected and ready to receive tasks.\n\n### Method 2: Health Check\n\n```bash\ncurl http://localhost:5000/api/health\n```\n\n**Expected Response:**\n\n```json\n{\n \"status\": \"healthy\",\n \"online_clients\": [\"my_windows_device\"]\n}\n```\n\nThe `/api/health` endpoint is useful for health checks in production monitoring systems.\n\n---\n\n## 🎯 Dispatching Your First Task\n\nThe easiest way to send a task to a connected device is through the HTTP `/api/dispatch` endpoint.\n\n### Basic Task Dispatch\n\nUse the HTTP API to dispatch a task to a connected device:\n\n```bash\ncurl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"my_windows_device\",\n \"request\": \"Open Notepad and type Hello World\",\n \"task_name\": \"test_task_001\"\n }'\n```\n\n**Request Body Parameters:**\n\n| Field | Required | Type | Description | Example |\n|-------|----------|------|-------------|---------|\n| `client_id` | Yes | string | Target device identifier | `\"my_windows_device\"` |\n| `request` | Yes | string | Natural language task description | `\"Open Notepad\"` |\n| `task_name` | ⚠️ Optional | string | Unique task identifier (auto-generated if omitted) | `\"task_001\"` |\n\n**Successful Response:**\n\n```json\n{\n \"status\": \"dispatched\",\n \"task_name\": \"test_task_001\",\n \"client_id\": \"my_windows_device\",\n \"session_id\": \"3f4a2b1c-9d8e-4f3a-b2c1-9a8b7c6d5e4f\"\n}\n```\n\nThe `status: \"dispatched\"` indicates the task was successfully sent to the device. The device will begin executing immediately.\n\n!!!warning \"Client Must Be Online\"\n If the target `client_id` is not connected, you'll receive `{\"detail\": \"Client not online\"}`. Use `/api/clients` to verify the device is connected first.\n\n### Task Execution Flow\n\n```mermaid\nsequenceDiagram\n participant API as HTTP Client\n participant S as Server\n participant D as Device\n \n API->>S: POST /api/dispatch\n S->>D: TASK (AIP)\n D->>D: Execute task\n D->>S: TASK_RESULT\n API->>S: GET /task_result\n S->>API: Results\n```\n\nFor detailed API specifications, see [HTTP API Reference](./api.md).\n\n### Checking Task Results\n\nUse the task name to retrieve results:\n\n```bash\ncurl http://localhost:5000/api/task_result/test_task_001\n```\n\n**While Task is Running:**\n\n```json\n{\n \"status\": \"pending\"\n}\n```\n\n**When Task Completes:**\n\n```json\n{\n \"status\": \"done\",\n \"result\": {\n \"action_taken\": \"Opened Notepad and typed 'Hello World'\",\n \"screenshot\": \"base64_encoded_image...\",\n \"observation\": \"Task completed successfully\"\n }\n}\n```\n\n!!!tip \"Polling Best Practice\"\n For long-running tasks, poll every 2-5 seconds. Most simple tasks complete within 10-30 seconds.\n\n### Advanced Task Dispatch\n\n**Complex Multi-Step Task:**\n```bash\ncurl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"my_windows_device\",\n \"request\": \"Open Excel, create a new worksheet, and enter sales data for Q4 2024\",\n \"task_name\": \"excel_q4_report\"\n }'\n```\n\n**Web Automation Task:**\n```bash\ncurl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"my_windows_device\",\n \"request\": \"Open Chrome, navigate to GitHub.com, and search for UFO framework\",\n \"task_name\": \"github_search\"\n }'\n```\n\n**File Management Task:**\n```bash\ncurl -X POST http://localhost:5000/api/dispatch \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"client_id\": \"my_linux_device\",\n \"request\": \"Create a folder named test_data and copy all .txt files from Documents\",\n \"task_name\": \"file_organization\"\n }'\n```\n\n---\n\n## 🐛 Common Issues & Troubleshooting\n\n### Issue 1: Port Already in Use\n\n**Symptoms:**\n```console\nERROR: [Errno 98] Address already in use\n```\n\n**Cause:** Another process is already using port 5000.\n\n**Solutions:**\n\n**Use Different Port:**\n```bash\npython -m ufo.server.app --port 8080\n```\n\n**Find & Kill Process (Linux/Mac):**\n```bash\n# Find process using port 5000\nlsof -i :5000\n \n# Kill the process\nkill -9 \n```\n\n**Find & Kill Process (Windows):**\n```powershell\n# Find process using port 5000\nnetstat -ano | findstr :5000\n \n# Kill the process\ntaskkill /PID /F\n```\n\n### Issue 2: Connection Refused\n\n**Symptoms:**\n```console\n[WS] Failed to connect to ws://127.0.0.1:5000/ws\nConnection refused\n```\n\n**Diagnosis Checklist:**\n\n- Is the server actually running? Check for \"Uvicorn running\" message\n- Does the port match in both server and client commands?\n- Are you using `--local` mode? If yes, clients must connect from `localhost`\n- Is there a firewall blocking the connection?\n\n**Solutions:**\n\n1. Verify server is running:\n ```bash\n curl http://localhost:5000/api/health\n ```\n\n2. Check server logs for startup errors\n\n3. If using `--local` mode, ensure client uses `127.0.0.1`\n\n4. If connecting from another machine, remove `--local` flag\n\n### Issue 3: Device Not Connected Error\n\n**Symptoms:**\nWhen dispatching a task:\n```json\n{\n \"detail\": \"Client not online\"\n}\n```\n\n**Diagnosis:**\n\n1. List all connected clients:\n ```bash\n curl http://localhost:5000/api/clients\n ```\n\n2. Check the `client_id` matches exactly (case-sensitive!)\n\n**Solutions:**\n\n- Verify the device client is running and successfully registered\n- Check server logs for `📱 Device client connected`\n- Ensure no typos in `client_id` when dispatching\n- If the device disconnected, restart the client connection\n\n### Issue 4: Empty Task Content Error\n\n**Symptoms:**\n```json\n{\n \"detail\": \"Empty task content\"\n}\n```\n\n**Cause:** The `request` field in `/api/dispatch` is missing or empty.\n\n**Solution:** Always include the `request` field with a task description.\n\n### Issue 5: Firewall Blocking Connections\n **Symptoms:** Clients on other machines cannot connect, but `curl localhost:5000/api/health` works on server machine.\n \n **Diagnosis:**\n \n 1. **Check server is listening on all interfaces:**\n ```bash\n # Should show 0.0.0.0:5000 (not 127.0.0.1:5000)\n netstat -tuln | grep 5000\n ```\n \n 2. **Test from remote machine:**\n ```bash\n curl http://:5000/api/health\n ```\n \n **Solutions:**\n \n **Windows Firewall:**\n ```powershell\n # Allow incoming connections on port 5000\n New-NetFirewallRule -DisplayName \"UFO Server\" `\n -Direction Inbound `\n -Protocol TCP `\n -LocalPort 5000 `\n -Action Allow\n ```\n \n **Linux (ufw):**\n ```bash\n sudo ufw allow 5000/tcp\n sudo ufw reload\n ```\n \n **Linux (firewalld):**\n ```bash\n sudo firewall-cmd --zone=public --add-port=5000/tcp --permanent\n sudo firewall-cmd --reload\n ```\n\n### Issue 6: Target Device Not Connected (Constellation)\n\n**Symptoms:**\n```console\nTarget device 'my_windows_device' is not connected\n```\n\n**Solution:**\n\n1. Connect the device client first\n2. Wait for registration confirmation (check server logs)\n3. Then connect constellation\n\n!!!tip \"Debug Mode\"\n For maximum verbosity, start the server with: `python -m ufo.server.app --log-level DEBUG`\n\n---\n\n## 📚 Next Steps\n\nNow that you have the server running and can dispatch tasks, explore these topics:\n\n### Immediate Next Steps\n\n| Step | Topic | Time | Description |\n|------|-------|------|-------------|\n| 1️⃣ | [Server Architecture](./overview.md) | 10 min | Understand the three-tier architecture and component interactions |\n| 2️⃣ | [HTTP API Reference](./api.md) | 15 min | Explore all available API endpoints for integration |\n| 3️⃣ | [Client Setup Guide](../client/quick_start.md) | 10 min | Learn advanced client configuration options |\n| 4️⃣ | [AIP Protocol](../aip/overview.md) | 20 min | Deep dive into the Agent Interaction Protocol |\n\n### Advanced Topics\n\n| Topic | Relevance | Link |\n|-------|-----------|------|\n| **Session Management** | Understanding task lifecycle and state | [Session Manager](./session_manager.md) |\n| **WebSocket Handler** | Low-level connection handling | [WebSocket Handler](./websocket_handler.md) |\n| **Monitoring & Operations** | Production deployment best practices | [Monitoring](./monitoring.md) |\n| **Constellation Mode** | Multi-device orchestration | Coming Soon |\n\n---\n\n## 🚀 Production Deployment\n\n!!!warning \"Production Readiness Checklist\"\n Before deploying to production, ensure you address these critical areas:\n\n### 1. Process Management\n\n!!!example \"Systemd Service (Linux)\"\n Create `/etc/systemd/system/ufo-server.service`:\n \n ```ini\n [Unit]\n Description=UFO Agent Server\n After=network.target\n \n [Service]\n Type=simple\n User=ufo\n WorkingDirectory=/opt/ufo\n Environment=\"PATH=/opt/ufo/venv/bin\"\n ExecStart=/opt/ufo/venv/bin/python -m ufo.server.app --port 5000 --log-level INFO\n Restart=always\n RestartSec=10\n StandardOutput=journal\n StandardError=journal\n \n [Install]\n WantedBy=multi-user.target\n ```\n \n **Enable and start:**\n ```bash\n sudo systemctl daemon-reload\n sudo systemctl enable ufo-server\n sudo systemctl start ufo-server\n sudo systemctl status ufo-server\n ```\n\n**PM2 Process Manager (Cross-Platform):**\n```bash\n# Install PM2\nnpm install -g pm2\n\n# Start server with PM2\npm2 start \"python -m ufo.server.app --port 5000\" --name ufo-server\n\n# Setup auto-restart on system boot\npm2 startup\npm2 save\n\n# Monitor\npm2 logs ufo-server\npm2 monit\n```\n\nFor complete production deployment guidance including SSL/TLS, security hardening, and scaling strategies, see [Server Overview - Production Deployment](./overview.md#production-deployment).\n\n---\n\n## 🎓 What You Learned\n\nYou've successfully:\n\n- Started the UFO Agent Server with custom configurations\n- Connected device and constellation clients via WebSocket\n- Dispatched tasks using the HTTP API\n- Verified connections and monitored health\n- Troubleshot common issues\n- Learned production deployment best practices\n\nContinue your journey with:\n\n- **Architecture Deep Dive**: [Server Overview](./overview.md)\n- **API Exploration**: [HTTP API Reference](./api.md)\n- **Client Development**: [Client Documentation](../client/overview.md)\n- **Multi-Device Coordination**: [Constellation Overview](../galaxy/overview.md)\n"
},
{
"path": "documents/docs/server/session_manager.md",
"content": "# Session Manager\n\nThe **SessionManager** orchestrates agent session lifecycles, coordinates background task execution, and maintains execution state across the server. It serves as the \"execution engine\" that powers UFO's autonomous task capabilities.\n\nFor context on how this component fits into the server architecture, see the [Server Overview](overview.md).\n\n---\n\n## 🎯 Overview\n\nThe SessionManager is a critical server component that bridges task dispatch and actual execution:\n\n| Capability | Description | Benefit |\n|------------|-------------|---------|\n| **Platform-Agnostic Creation** | Automatically creates Windows/Linux sessions | No manual platform handling needed |\n| **Background Execution** | Tasks run without blocking WebSocket event loop | Maintains connection health during long tasks |\n| **State Tracking** | Monitors session lifecycle (created → running → completed/failed) | Enables task monitoring & result retrieval |\n| **Graceful Cancellation** | Handles disconnections with context-aware cleanup | Prevents orphaned tasks & resource leaks |\n| **Concurrent Management** | Multiple sessions can run simultaneously | Supports multi-device orchestration |\n\n### Architecture Position\n\n```mermaid\ngraph TB\n subgraph \"Agent Server\"\n WH[WebSocket Handler]\n SM[Session Manager]\n SF[Session Factory]\n \n subgraph \"Sessions\"\n WS[Windows Service Session]\n LS[Linux Service Session]\n LOC[Local Session]\n end\n end\n \n Client[Device Client] -->|WebSocket| WH\n WH -->|\"execute_task_async()\"| SM\n SM -->|\"create session\"| SF\n SF -->|\"platform=windows\"| WS\n SF -->|\"platform=linux\"| LS\n SF -->|\"local=true\"| LOC\n \n WS -->|\"execute commands\"| Client\n LS -->|\"execute commands\"| Client\n \n SM -->|\"callback(result)\"| WH\n WH -->|\"TASK_END message\"| Client\n \n style SM fill:#ffecb3\n style SF fill:#c8e6c9\n style WH fill:#bbdefb\n```\n\n**Why Background Execution Matters:**\n\nWithout background execution, a long-running task (e.g., 5-minute workflow) would **block the WebSocket event loop**, preventing:\n\n- Heartbeat messages from being sent/received\n- Ping/pong frames from maintaining the connection\n- Other clients' tasks from being dispatched\n\nBackground execution solves this by using Python's `asyncio.create_task()` to run sessions concurrently.\n\n---\n\n## 🏗 Core Functionality\n\n### Session Creation\n\nThe SessionManager uses the **SessionFactory** pattern to create platform-specific session implementations. This abstraction layer automatically selects the correct session type based on platform and mode.\n\n**Creating a Session:**\n\n```python\nsession = session_manager.get_or_create_session(\n session_id=\"session_abc123\",\n task_name=\"create_file\",\n request=\"Open Notepad and create a file\",\n task_protocol=task_protocol, # AIP TaskExecutionProtocol instance\n platform_override=\"windows\" # or \"linux\" or None (auto-detect)\n)\n```\n\n**Session Types:**\n\n| Session Type | Use Case | Platform | Dispatcher | MCP Tools |\n|--------------|----------|----------|------------|-----------|\n| **ServiceSession (Windows)** | Remote Windows device | Windows | AIP protocol-based | Windows MCP servers |\n| **LinuxServiceSession** | Remote Linux device | Linux | AIP protocol-based | Linux MCP servers |\n| **Local Session** | Local testing/debugging | Any | Direct execution | Local MCP servers |\n\n**Platform Detection:**\n\nIf `platform_override=None`, the SessionManager uses Python's `platform.system()` to auto-detect:\n\n- `\"Windows\"` → ServiceSession (Windows)\n- `\"Linux\"` → LinuxServiceSession\n- `\"Darwin\"` (macOS) → Currently uses LinuxServiceSession\n\n**Session Factory Logic Flow:**\n\n```mermaid\ngraph LR\n A[get_or_create_session] --> B{Session exists?}\n B -->|Yes| C[Return existing]\n B -->|No| D{local mode?}\n D -->|Yes| E[Create Local Session]\n D -->|No| F{Platform?}\n F -->|windows| G[ServiceSession]\n F -->|linux| H[LinuxServiceSession]\n E --> I[Store in sessions dict]\n G --> I\n H --> I\n I --> J[Return session]\n \n style D fill:#ffe0b2\n style F fill:#ffe0b2\n style I fill:#c8e6c9\n```\n\n### Background Execution\n\nThe **critical innovation** of the SessionManager is background task execution using `asyncio.create_task()`. This prevents long-running sessions from blocking the WebSocket event loop.\n\n**Execute Task Asynchronously:**\n\n```python\nawait session_manager.execute_task_async(\n session_id=session_id,\n task_name=task_name,\n request=user_request,\n task_protocol=task_protocol, # AIP TaskExecutionProtocol instance\n platform_override=\"windows\",\n callback=result_callback # Called when task completes\n)\n```\n\n**Benefits of Background Execution:**\n\n| Benefit | Description | Impact |\n|---------|-------------|--------|\n| **WebSocket Health** | Ping/pong continues uninterrupted | Prevents connection timeouts (30-60s) |\n| **Heartbeat Flow** | Heartbeat messages can be sent/received | Maintains connection liveness |\n| **Concurrent Sessions** | Multiple sessions run simultaneously | Supports multi-device orchestration |\n| **Event Loop Availability** | Handler can process other messages | Responsive to new connections/dispatches |\n| **Graceful Cancellation** | Tasks can be cancelled mid-execution | Clean disconnection handling |\n\n**Background Execution Flow:**\n\n```mermaid\nsequenceDiagram\n participant WH as WebSocket Handler\n participant SM as Session Manager\n participant BT as Background Task\n participant S as Session\n participant CB as Callback\n \n Note over WH,SM: 1️⃣ Task Dispatch\n WH->>SM: execute_task_async(session_id, request, callback)\n SM->>SM: get_or_create_session()\n SM->>BT: asyncio.create_task(_run_session_background)\n SM-->>WH: Return immediately (non-blocking!)\n \n Note over WH: Event loop free for other tasks\n WH->>WH: Can process heartbeats, ping/pong, new tasks\n \n Note over BT,S: 2️⃣ Background Execution\n BT->>S: await session.run()\n S->>S: LLM reasoning Action selection Execution\n Note over S: Long-running task (30s - 5min)\n S-->>BT: Execution complete\n \n Note over BT,CB: 3️⃣ Result Callback\n BT->>BT: Build ServerMessage (TASK_END)\n BT->>SM: set_results(session_id)\n BT->>CB: await callback(session_id, result_message)\n CB->>WH: Send result via WebSocket\n \n Note over BT: 4️⃣ Cleanup\n BT->>SM: Remove from _running_tasks dict\n```\n\n**Thread Safety:**\n\nThe SessionManager uses `threading.Lock` for thread-safe access to shared dictionaries:\n\n```python\nwith self.lock:\n self.sessions[session_id] = session\n```\n\nThis prevents race conditions in multi-threaded environments (though FastAPI primarily uses async/await).\n\n### Callback Mechanism\n\nWhen a task completes (successfully, with errors, or via cancellation), the SessionManager invokes a registered callback function. This decouples task execution from result delivery.\n\n**Registering a Callback:**\n\n```python\nasync def send_result_to_client(session_id: str, result_msg: ServerMessage):\n \"\"\"Called when task completes.\"\"\"\n await websocket.send_text(result_msg.model_dump_json())\n logger.info(f\"Sent TASK_END for {session_id}\")\n\nawait session_manager.execute_task_async(\n session_id=\"abc123\",\n task_name=\"open_notepad\",\n request=\"Open Notepad\",\n task_protocol=task_protocol,\n callback=send_result_to_client # Register callback\n)\n```\n\n**Callback Execution Flow:**\n\n```mermaid\nstateDiagram-v2\n [*] --> TaskRunning: Background task starts\n TaskRunning --> ResultsCollected: session.run() completes\n ResultsCollected --> StatusDetermined: Check session.is_finished() / is_error()\n StatusDetermined --> MessageBuilt: Create ServerMessage(TASK_END)\n MessageBuilt --> ResultsPersisted: set_results(session_id)\n ResultsPersisted --> CallbackInvoked: await callback(session_id, message)\n CallbackInvoked --> [*]: Cleanup _running_tasks\n \n TaskRunning --> TaskCancelled: asyncio.CancelledError\n TaskCancelled --> CancellationHandled: Check cancellation_reason\n CancellationHandled --> MessageBuilt: Create failure message\n \n TaskRunning --> ErrorOccurred: Exception raised\n ErrorOccurred --> ErrorLogged: Log traceback\n ErrorLogged --> MessageBuilt: Create error message\n```\n\n**ServerMessage Structure:**\n\n```python\nresult_message = ServerMessage(\n type=ServerMessageType.TASK_END,\n status=TaskStatus.COMPLETED, # or FAILED\n session_id=\"abc123\",\n error=None, # or error message if failed\n result=session.results, # Dict[str, Any]\n timestamp=\"2024-11-04T14:30:22.123456+00:00\",\n response_id=\"uuid-v4\"\n)\n```\n\n| Field | Type | Description | Example |\n|-------|------|-------------|---------|\n| `type` | ServerMessageType | Always `TASK_END` for completion | `ServerMessageType.TASK_END` |\n| `status` | TaskStatus | `COMPLETED`, `FAILED`, or `CANCELLED` | `TaskStatus.COMPLETED` |\n| `session_id` | str | Session identifier | `\"abc123\"` |\n| `error` | Optional[str] | Error message if task failed | `\"Device disconnected\"` |\n| `result` | Dict[str, Any] | Task execution results | `{\"action\": \"opened notepad\", \"screenshot\": \"...\"}` |\n| `timestamp` | str | ISO 8601 timestamp (UTC) | `\"2024-11-04T14:30:22Z\"` |\n| `response_id` | str | Unique response UUID | `\"3f4a2b1c-9d8e-4f3a-b2c1-...\"` |\n\n**Callback Error Handling:**\n\nIf the callback raises an exception, the SessionManager **logs the error but doesn't fail the session**:\n\n```python\ntry:\n await callback(session_id, result_message)\nexcept Exception as e:\n self.logger.error(f\"Callback error: {e}\")\n # Session results are still persisted!\n```\n\nThis prevents callback bugs from breaking task execution.\n\n### Task Cancellation\n\nThe SessionManager supports **graceful task cancellation** with different behaviors based on **why** the cancellation occurred. This is critical for handling client disconnections properly.\n\n**Cancel a Running Task:**\n\n```python\nawait session_manager.cancel_task(\n session_id=\"session_abc123\",\n reason=\"device_disconnected\" # or \"constellation_disconnected\"\n)\n```\n\n**Cancellation Reasons:**\n\n| Reason | Scenario | Callback Behavior | Use Case |\n|--------|----------|-------------------|----------|\n| `constellation_disconnected` | Constellation client lost connection | **No callback** (client is gone) | Task requester disconnected, no one to notify |\n| `device_disconnected` | Target device lost connection | **Send callback** to constellation | Notify orchestrator to reassign task |\n| `user_requested` | Manual cancellation via API | **Send callback** to requester | Explicit cancellation command |\n\n**Cancellation Flow:**\n\n```mermaid\nsequenceDiagram\n participant C as Client (Constellation)\n participant WH as WebSocket Handler\n participant SM as Session Manager\n participant BT as Background Task\n participant D as Device\n \n Note over C,BT: Scenario 1: Device Disconnects During Task\n C->>WH: Task dispatched to device\n WH->>SM: execute_task_async(session_id, callback)\n SM->>BT: Background task running\n BT->>D: Executing actions\n \n Note over D: Device disconnects\n D--xWH: WebSocket closed\n WH->>SM: cancel_task(session_id, reason=\"device_disconnected\")\n SM->>BT: task.cancel()\n BT->>BT: Catch asyncio.CancelledError\n BT->>BT: Build failure message\n BT->>WH: callback(session_id, failure_msg)\n WH->>C: TASK_END (status=FAILED, error=\"Device disconnected\")\n \n Note over C,SM: Scenario 2: Constellation Disconnects During Task\n C->>WH: Task dispatched\n WH->>SM: execute_task_async()\n SM->>BT: Background task running\n \n Note over C: Constellation disconnects\n C--xWH: WebSocket closed\n WH->>SM: cancel_task(session_id, reason=\"constellation_disconnected\")\n SM->>BT: task.cancel()\n BT->>BT: Catch asyncio.CancelledError\n BT->>BT: Skip callback (client gone)\n BT->>SM: Remove session\n```\n\n**Cancellation Implementation Details:**\n\n```python\nasync def cancel_task(self, session_id: str, reason: str) -> bool:\n \"\"\"Cancel a running background task.\"\"\"\n task = self._running_tasks.get(session_id)\n \n if task and not task.done():\n # Store reason for use in _run_session_background\n self._cancellation_reasons[session_id] = reason\n \n # Request cancellation\n task.cancel()\n \n # Wait for graceful shutdown (max 2 seconds)\n try:\n await asyncio.wait_for(task, timeout=2.0)\n except (asyncio.CancelledError, asyncio.TimeoutError):\n pass # Expected\n \n # Cleanup\n self._running_tasks.pop(session_id, None)\n self._cancellation_reasons.pop(session_id, None)\n self.remove_session(session_id)\n \n return True\nreturn False\n```\n\n**Important Notes:**\n\n- **Cancellation is asynchronous**: The background task receives an `asyncio.CancelledError` at the next `await` point. If the session is executing synchronous code (e.g., LLM inference), cancellation won't take effect until that operation completes.\n- **Grace Period**: The SessionManager waits up to **2 seconds** for graceful cancellation before giving up.\n\n**Best Practice:**\n\nWhen a client disconnects, the WebSocket Handler should:\n\n1. Identify all active sessions for that client\n2. Call `cancel_task()` with the appropriate `reason`\n3. Clean up client registration in ClientConnectionManager\n\nThis prevents orphaned sessions from consuming resources.\n\n---\n\n## 🔄 Session Lifecycle\n\nSessions follow a predictable lifecycle from initial dispatch through execution to final cleanup. Understanding this flow is essential for debugging and monitoring.```mermaid\nstateDiagram-v2\n [*] --> Created: get_or_create_session()\n Created --> Stored: Add to sessions dict\n Stored --> BackgroundTask: execute_task_async()\n BackgroundTask --> Running: await session.run()\n \n Running --> Completed: session.is_finished() == True\n Running --> Failed: session.is_error() == True\n Running --> Cancelled: asyncio.CancelledError\n Running --> Exception: Exception raised\n \n Completed --> ResultsCollected: Gather session.results\n Failed --> ResultsCollected: Include error details\n Cancelled --> ResultsCollected: Include cancellation reason\n Exception --> ResultsCollected: Include exception message\n \n ResultsCollected --> ResultsPersisted: set_results(session_id)\n ResultsPersisted --> CallbackInvoked: await callback(session_id, message)\n CallbackInvoked --> Cleanup: remove_session(session_id)\n Cleanup --> [*]\n```\n\n### Lifecycle Stages\n\n| Stage | Description | Key Operations | Duration |\n|-------|-------------|----------------|----------|\n| **1. Creation** | Session object instantiated | `get_or_create_session()` | < 100ms |\n| **2. Registration** | Stored in sessions dict with ID | `sessions[session_id] = session` | < 10ms |\n| **3. Background Dispatch** | Task created with `asyncio.create_task()` | `_running_tasks[session_id] = task` | < 50ms |\n| **4. Execution** | Session runs (LLM + actions) | `await session.run()` | 10s - 5min |\n| **5. Result Collection** | Gather results and determine status | `session.results`, `session.is_finished()` | < 100ms |\n| **6. Persistence** | Save results to results dict | `set_results(session_id)` | < 10ms |\n| **7. Callback** | Notify registered callback | `await callback(session_id, msg)` | 50-500ms |\n| **8. Cleanup** | Remove from active sessions | `remove_session(session_id)` | < 10ms |\n\n**Complete Lifecycle Example:**\n\n```python\n# Stage 1-2: Creation\nsession = session_manager.get_or_create_session(\n session_id=\"abc123\",\n task_name=\"demo_task\",\n request=\"Open Notepad\",\n task_protocol=task_protocol,\n platform_override=\"windows\"\n)\n\n# Stage 3: Background Dispatch\nawait session_manager.execute_task_async(\n session_id=\"abc123\",\n task_name=\"demo_task\",\n request=\"Open Notepad\",\n task_protocol=task_protocol,\n platform_override=\"windows\",\n callback=send_result_callback\n)\n# Returns immediately! Task runs in background\n\n# Stage 4: Execution (happens in background)\n# session.run() executes:\n# - LLM reasoning\n# - Action selection\n# - Command execution via device\n# - Result observation\n\n# Stage 5-6: Results (automatic)\n# Session completes, results collected and persisted\n\n# Stage 7: Callback (automatic)\n# await callback(\"abc123\", ServerMessage(...))\n\n# Stage 8: Cleanup (manual or automatic)\nsession_manager.remove_session(\"abc123\")\n```\n\n**Session Persistence:**\n\nSessions remain in the `sessions` dict until explicitly removed via `remove_session()`. This allows:\n\n- **Result retrieval** via `/api/task_result/{task_name}`\n- **Session inspection** for debugging\n- **Reconnection scenarios** (future feature)\n\nHowever, this means **sessions consume memory** until cleaned up. Implement periodic cleanup for production deployments.\n\n---\n\n## 💾 State Management\n\nThe SessionManager maintains three separate dictionaries for different aspects of session state:\n\n### 1. Active Sessions Storage\n\n```python\nself.sessions: Dict[str, BaseSession] = {}\n```\n\n| Purpose | Structure | Lifecycle | Thread Safety |\n|---------|-----------|-----------|---------------|\n| Store active session objects | `{session_id: BaseSession}` | Until `remove_session()` called | `threading.Lock` |\n\n**Session Storage Operations:**\n\n```python\n# Store session\nwith self.lock:\n self.sessions[session_id] = session\n\n# Retrieve session\nwith self.lock:\n session = self.sessions.get(session_id)\n\n# Remove session\nwith self.lock:\n self.sessions.pop(session_id, None)\n```\n\n**Benefits:**\n\n- Fast O(1) lookup by session ID\n- Thread-safe with lock\n- Supports session reuse (future reconnections)\n\n**Considerations:**\n\n- ⚠️ Memory grows with active sessions\n- ⚠️ Manual cleanup required (`remove_session()`)\n- ⚠️ No automatic expiration\n\n### 2. Result Caching\n\n```python\nself.results: Dict[str, Dict[str, Any]] = {}\n```\n\n| Purpose | Structure | When Populated | Retrieval Methods |\n|---------|-----------|----------------|-------------------|\n| Cache completed task results | `{session_id: results_dict}` | After task completion via `set_results()` | `get_result()`, `get_result_by_task()` |\n\n**Result Storage & Retrieval:**\n\n```python\n# Persist results after completion\ndef set_results(self, session_id: str):\n with self.lock:\n if session_id in self.sessions:\n self.results[session_id] = self.sessions[session_id].results\n\n# Retrieve by session ID\nresult = session_manager.get_result(\"abc123\")\n# Returns: {\"action\": \"opened notepad\", \"screenshot\": \"base64...\"}\n\n# Retrieve by task name\nresult = session_manager.get_result_by_task(\"demo_task\")\n```\n\n**Result Structure Example:**\n\n```json\n{\n \"action_taken\": \"Opened Notepad and typed 'Hello World'\",\n \"screenshot\": \"base64_encoded_screenshot_data...\",\n \"observation\": \"Notepad window is visible with text 'Hello World'\",\n \"success\": true,\n \"metadata\": {\n \"steps_taken\": 3,\n \"execution_time_seconds\": 12.5\n }\n}\n```\n\n### 3. Task Name Mapping\n\n```python\nself.session_id_dict: Dict[str, str] = {}\n```\n\n| Purpose | Structure | Use Case |\n|---------|-----------|----------|\n| Map task names to session IDs | `{task_name: session_id}` | Allow result retrieval by task name instead of session ID |\n\n**Task Name Mapping:**\n\n```python\n# Created during session creation\nself.session_id_dict[task_name] = session_id\n\n# Usage: Get result by task name\ndef get_result_by_task(self, task_name: str):\n with self.lock:\n session_id = self.session_id_dict.get(task_name)\n if session_id:\n return self.get_result(session_id)\n```\n\n**Why This Matters:**\n\nThe HTTP API endpoint `/api/task_result/{task_name}` allows clients to check results using the **task name** they provided, without needing to track session IDs:\n\n```bash\n# Client only needs to remember task name\ncurl http://localhost:5000/api/task_result/demo_task\n\n# Instead of tracking session ID\ncurl http://localhost:5000/api/task_result/abc123\n```\n\n### 4. Running Tasks Tracking\n\n```python\nself._running_tasks: Dict[str, asyncio.Task] = {}\n```\n\n| Purpose | Structure | Use Case |\n|---------|-----------|----------|\n| Track active background tasks for cancellation | `{session_id: asyncio.Task}` | Enable graceful task cancellation when clients disconnect |\n\n**Running Task Management:**\n\n```python\n# Register background task\ntask = asyncio.create_task(self._run_session_background(...))\nself._running_tasks[session_id] = task\n\n# Cancel running task\ntask = self._running_tasks.get(session_id)\nif task and not task.done():\n task.cancel()\n await asyncio.wait_for(task, timeout=2.0)\n\n# Cleanup after completion\nself._running_tasks.pop(session_id, None)\n```\n\n### 5. Cancellation Reasons Tracking\n\n```python\nself._cancellation_reasons: Dict[str, str] = {}\n```\n\n| Purpose | Structure | Lifecycle |\n|---------|-----------|-----------|\n| Store why each task was cancelled | `{session_id: reason}` | From `cancel_task()` to `_run_session_background()` cleanup |\n\n**Cancellation Reason Flow:**\n\n```python\n# Store reason when cancelling\nasync def cancel_task(self, session_id: str, reason: str):\n self._cancellation_reasons[session_id] = reason\n task.cancel()\n\n# Retrieve reason during cancellation handling\nasync def _run_session_background(...):\n try:\n await session.run()\n except asyncio.CancelledError:\n reason = self._cancellation_reasons.get(session_id, \"unknown\")\n if reason == \"device_disconnected\":\n # Send callback to constellation\n elif reason == \"constellation_disconnected\":\n # Skip callback\n```\n\n---\n\n### Thread Safety\n\nThe SessionManager uses `threading.Lock` for thread-safe access to shared dictionaries:\n\n```python\ndef __init__(self):\n self.lock = threading.Lock()\n\ndef get_or_create_session(self, ...):\n with self.lock:\n if session_id not in self.sessions:\n self.sessions[session_id] = session\n return self.sessions[session_id]\n```\n\n**Why this matters:** Although FastAPI primarily uses async/await (single-threaded event loop), the lock protects against:\n\n- **Thread pool executors** for sync operations\n- **Background tasks** accessing shared state\n- **Future multi-threading** in FastAPI/Uvicorn\n\n**Performance Consideration:**\n\nLock contention is minimal because:\n\n- Lock is held only for **dictionary operations** (O(1) operations)\n- Session execution happens **outside the lock** (async background tasks)\n- Most operations are **read-heavy** (get_result) which are fast\n\n---\n\n## 🖥 Platform Support\n\nThe SessionManager supports both Windows and Linux platforms through the **SessionFactory** abstraction layer. Platform-specific implementations handle OS-specific UI automation and tool execution.\n\n### Platform Detection\n\n```mermaid\ngraph TD\n A[get_or_create_session] --> B{platform_override specified?}\n B -->|Yes| C[Use specified platform]\n B -->|No| D[Auto-detect via platform.system]\n D --> E{OS Detected}\n E -->|\"Windows\"| F[platform = 'windows']\n E -->|\"Linux\"| G[platform = 'linux']\n E -->|\"Darwin\" macOS| H[platform = 'linux' ⚠️ Treated as Linux]\n \n C --> I[SessionFactory.create_service_session]\n F --> I\n G --> I\n H --> I\n \n I --> J{Platform?}\n J -->|windows| K[ServiceSession]\n J -->|linux| L[LinuxServiceSession]\n \n style H fill:#ffe0b2\n style K fill:#c8e6c9\n style L fill:#bbdefb\n```\n\n**Platform Detection Code:**\n\n```python\ndef __init__(self, platform_override: Optional[str] = None):\n self.platform = platform_override or platform.system().lower()\n # platform.system() returns: \"Windows\", \"Linux\", or \"Darwin\"\n self.logger.info(f\"SessionManager initialized for platform: {self.platform}\")\n```\n\n### Platform-Specific Sessions\n\n| Platform | Session Class | UI Automation | MCP Tools | Status |\n|----------|---------------|---------------|-----------|--------|\n| **Windows** | `ServiceSession` | Win32 API, UI Automation | Windows MCP servers (filesystem, browser, etc.) | Fully Supported |\n| **Linux** | `LinuxServiceSession` | X11/Wayland, AT-SPI | Linux MCP servers | Fully Supported |\n| **macOS (Darwin)** | `LinuxServiceSession` | Currently treated as Linux | Linux MCP servers | ⚠️ Experimental |\n\n**Windows Session Creation:**\n\n```python\n# Explicit Windows platform\nsession = session_manager.get_or_create_session(\n session_id=\"win_session_001\",\n task_name=\"windows_task\",\n request=\"Open File Explorer and navigate to Downloads\",\n task_protocol=task_protocol,\n platform_override=\"windows\"\n)\n# Creates ServiceSession\n```\n\n**Linux Session Creation:**\n\n```python\n# Explicit Linux platform\nsession = session_manager.get_or_create_session(\n session_id=\"linux_session_001\",\n task_name=\"linux_task\",\n request=\"Open Nautilus and create a new folder\",\n task_protocol=task_protocol,\n platform_override=\"linux\"\n)\n# Creates LinuxServiceSession\n```\n\n**Auto-Detection:**\n\n```python\n# Let SessionManager detect platform\nsession = session_manager.get_or_create_session(\n session_id=\"auto_session_001\",\n task_name=\"auto_task\",\n request=\"Open text editor\",\n task_protocol=task_protocol,\n platform_override=None # Auto-detect\n)\n# Uses platform.system() to determine session type\n```\n\n**macOS Limitations:**\n\nmacOS (Darwin) is currently treated as Linux, which may result in:\n\n- Incorrect UI automation commands\n- Missing macOS-specific tool integrations\n- ⚠️ Limited functionality\n\n**Recommendation:** Use explicit `platform_override=\"linux\"` for Linux-like behavior, or wait for dedicated macOS session implementation.\n\n---\n\n## 🐛 Error Handling\n\nThe SessionManager implements comprehensive error handling to prevent task failures from breaking the server.\n\n### Error Categories\n\n| Error Type | Handler | Behavior | Example |\n|------------|---------|----------|---------|\n| **Session Execution Errors** | `try/except in _run_session_background` | Status = FAILED, error message in results | LLM API timeout, invalid action |\n| **Callback Errors** | `try/except around callback invocation` | Log error, continue execution | WebSocket closed before callback |\n| **Cancellation** | `asyncio.CancelledError handler` | Check reason, conditional callback | Client disconnected mid-task |\n| **Unknown State** | Status check after `session.run()` | Status = FAILED, error = \"unknown state\" | Session neither finished nor errored |\n\n### Session Execution Error Handling\n\n```python\nasync def _run_session_background(...):\n try:\n await session.run() # May raise exceptions\n \n # Determine status\n if session.is_error():\n status = TaskStatus.FAILED\n session.results = session.results or {\"failure\": \"session ended with an error\"}\n elif session.is_finished():\n status = TaskStatus.COMPLETED\n else:\n status = TaskStatus.FAILED\n error = \"Session ended in unknown state\"\n \n except asyncio.CancelledError:\n # Handle cancellation (see Cancellation section)\n ...\n \n except Exception as e:\n # Catch all other exceptions\n import traceback\n traceback.print_exc()\n self.logger.error(f\"Error in session {session_id}: {e}\")\n status = TaskStatus.FAILED\n error = str(e)\n```\n\n**Error Result Structure:**\n\nWhen a session fails, the result includes error details:\n\n```json\n{\n \"status\": \"FAILED\",\n \"error\": \"LLM API timeout after 60 seconds\",\n \"session_id\": \"abc123\",\n \"result\": {\n \"failure\": \"session ended with an error\",\n \"last_action\": \"open_notepad\",\n \"traceback\": \"Traceback (most recent call last)...\"\n }\n}\n```\n\n### Callback Error Handling\n\n```python\ntry:\n await callback(session_id, result_message)\nexcept Exception as e:\n import traceback\n self.logger.error(\n f\"Callback error for session {session_id}: {e}\\n{traceback.format_exc()}\"\n )\n # Session results are STILL persisted!\n # Client may not receive notification\n```\n\n**Callback Failures Don't Fail Sessions:**\n\nIf the callback raises an exception (e.g., WebSocket already closed), the SessionManager:\n\n- **Logs the error** for debugging\n- **Persists the results** in `self.results`\n- **Completes cleanup** (removes from `_running_tasks`)\n- **Does NOT re-raise** the exception\n\n**Implication:** Results can be retrieved via `/api/task_result/{task_name}` even if WebSocket notification failed.\n\n### Unknown State Handling\n\n```python\nif session.is_error():\n status = TaskStatus.FAILED\nelif session.is_finished():\n status = TaskStatus.COMPLETED\nelse:\n # Unknown state - neither finished nor errored\n status = TaskStatus.FAILED\n error = \"Session ended in unknown state\"\n self.logger.warning(f\"Session {session_id} ended in unknown state\")\n```\n\n**Edge Case - Session Hangs:**\n\nIf `session.run()` completes but the session is neither `is_finished()` nor `is_error()`, this indicates:\n\n- Possible bug in session state management\n- Incomplete session implementation\n- Unexpected session interruption\n\nThe SessionManager marks this as **FAILED** to prevent silent failures.\n\n---\n\n## 💡 Best Practices\n\nFollow these best practices to ensure reliable, scalable session management:\n\n### 1. Configure Appropriate Timeouts\n\nSession timeouts should match task complexity:\n \n | Task Type | Timeout | Reason |\n |-----------|---------|--------|\n | **Simple UI Actions** | 60-120s | Open app, click button, type text |\n | **Medium Workflows** | 120-300s | Multi-step automation (3-5 steps) |\n | **Complex Tasks** | 300-600s | Complex workflows requiring LLM reasoning |\n | **Batch Operations** | 600-1800s | Processing multiple files, data entry |\n \n ```python\n # Configure in UFO config\n ufo_config.system.timeout = 300 # 5 minutes for medium tasks\n ```\n\n### 2. Monitor Session Count\n\nSessions consume memory. Implement limits to prevent resource exhaustion:\n\n```python\nMAX_CONCURRENT_SESSIONS = 100 # Adjust based on server resources\n\nasync def execute_task_safe(session_manager, ...):\n active_count = len(session_manager.sessions)\n \n if active_count >= MAX_CONCURRENT_SESSIONS:\n # Option 1: Reject new sessions\n raise HTTPException(\n status_code=503,\n detail=f\"Server at capacity ({active_count} active sessions)\"\n )\n \n # Option 2: Cancel oldest sessions\n oldest_session_id = min(\n session_manager.sessions.keys(),\n key=lambda s: session_manager.sessions[s].created_at\n )\n await session_manager.cancel_task(\n oldest_session_id,\n reason=\"capacity_limit\"\n )\n \n # Proceed with new session\n await session_manager.execute_task_async(...)\n```\n\n### 3. Clean Up Completed Sessions\n\n⚠️ **Memory Leak Prevention:**\n\nSessions persist in `sessions` dict until explicitly removed. Implement cleanup:\n \n ```python\n # Option 1: Cleanup immediately after result retrieval\n result = session_manager.get_result(session_id)\n if result:\n session_manager.remove_session(session_id)\n \n # Option 2: Periodic cleanup task\n import asyncio\n \n async def cleanup_old_sessions(session_manager, max_age_seconds=3600):\n \"\"\"Remove sessions older than max_age_seconds.\"\"\"\n while True:\n await asyncio.sleep(300) # Check every 5 minutes\n \n current_time = time.time()\n with session_manager.lock:\n to_remove = []\n for session_id, session in session_manager.sessions.items():\n age = current_time - session.created_at\n if age > max_age_seconds and session_id not in session_manager._running_tasks:\n to_remove.append(session_id)\n \n for session_id in to_remove:\n session_manager.remove_session(session_id)\n logger.info(f\"Cleaned up old session: {session_id}\")\n \n # Start cleanup task on server startup\nasyncio.create_task(cleanup_old_sessions(session_manager))\n```\n\n### 4. Handle Cancellation Gracefully\n\nDifferent cancellation reasons require different responses:\n\n```python\nasync def handle_client_disconnect(client_id, client_type, session_manager, client_manager):\n \"\"\"Handle disconnection based on client type.\"\"\"\n \n if client_type == ClientType.CONSTELLATION:\n # Constellation disconnected - cancel all its tasks\n session_ids = client_manager.get_constellation_sessions(client_id)\n for session_id in session_ids:\n await session_manager.cancel_task(\n session_id,\n reason=\"constellation_disconnected\" # Don't send callback\n )\n \n elif client_type == ClientType.DEVICE:\n # Device disconnected - notify constellations to reassign\n session_ids = client_manager.get_device_sessions(client_id)\n for session_id in session_ids:\n await session_manager.cancel_task(\n session_id,\n reason=\"device_disconnected\" # Send callback to constellation\n )\n \n # Clean up client registration\n client_manager.remove_client(client_id)\n```\n\n### 5. Log Session Lifecycle Events\n\nLog key lifecycle events for debugging and monitoring: ```python\n # Session creation\n self.logger.info(f\"Created {platform} session: {session_id} (type: {session_type})\")\n \n # Background task start\n self.logger.info(f\"🚀 Started background task {session_id}\")\n \n # Execution timing\n elapsed = loop.time() - start_time\n self.logger.info(f\"⏱️ Session {session_id} execution took {elapsed:.2f}s\")\n \n # Status determination\n self.logger.info(f\"Session {session_id} finished successfully\")\n self.logger.warning(f\"⚠️ Session {session_id} ended with error\")\n \n # Cancellation\n self.logger.warning(f\"🛑 Session {session_id} was cancelled (reason: {reason})\")\n \n # Cleanup\n self.logger.info(f\"Session {session_id} completed with status {status}\")\n```\n\n### 6. Implement Result Expiration\n\nPrevent `results` dict from growing indefinitely:\n\n```python\nfrom collections import OrderedDict\nimport time\n\nclass SessionManagerWithExpiration(SessionManager):\n def __init__(self, *args, **kwargs):\n super().__init__(*args, **kwargs)\n # Store (result, timestamp) tuples\n self.results: Dict[str, Tuple[Dict, float]] = {}\n self.result_ttl = 3600 # 1 hour\n \n def set_results(self, session_id: str):\n with self.lock:\n if session_id in self.sessions:\n self.results[session_id] = (\n self.sessions[session_id].results,\n time.time()\n )\n \n def get_result(self, session_id: str):\n with self.lock:\n if session_id in self.results:\n result, timestamp = self.results[session_id]\n # Check expiration\n if time.time() - timestamp > self.result_ttl:\n self.results.pop(session_id)\n return None\n return result\n return None\n```\n\n### 7. Monitor Background Tasks\n\nMonitor background tasks for unexpectedly long execution:\n \n ```python\n import asyncio\n \n async def monitor_long_running_tasks(session_manager, threshold=600):\n \"\"\"Alert on tasks running longer than threshold seconds.\"\"\"\n while True:\n await asyncio.sleep(60) # Check every minute\n \n current_time = asyncio.get_event_loop().time()\n for session_id, task in session_manager._running_tasks.items():\n # Calculate task age (approximation)\n session = session_manager.sessions.get(session_id)\n if session and hasattr(session, 'start_time'):\n age = current_time - session.start_time\n if age > threshold:\n logger.warning(\n f\"⚠️ Long-running task detected: {session_id} \"\n f\"(running for {age:.1f}s)\"\n )\n ```\n\n---\n\n## 🔗 Integration with Server Components\n\nThe SessionManager doesn't operate in isolation—it's deeply integrated with other server components.\n\n### Integration Architecture\n\n```mermaid\ngraph TB\n subgraph \"External\"\n HTTP[HTTP API Client]\n WS_C[WebSocket Client]\n end\n \n subgraph \"Server Components\"\n API[API Router /api/dispatch]\n WH[WebSocket Handler]\n WSM[Client Connection Manager]\n SM[Session Manager]\n SF[Session Factory]\n end\n \n subgraph \"Sessions\"\n WIN[Windows Session]\n LIN[Linux Session]\n end\n \n HTTP -->|POST /api/dispatch| API\n WS_C -->|WebSocket /ws| WH\n \n API -->|execute_task_async| SM\n WH -->|execute_task_async| SM\n \n SM -->|create session| SF\n SF -->|windows| WIN\n SF -->|linux| LIN\n \n SM -->|add_constellation_session| WSM\n SM -->|add_device_session| WSM\n \n SM -->|callback| WH\n WH -->|TASK_END message| WS_C\n \n style SM fill:#ffecb3\n style SF fill:#c8e6c9\n style WSM fill:#bbdefb\n```\n\n### 1. WebSocket Handler Integration\n\nThe WebSocket Handler creates sessions with callbacks to send results back to clients:\n\n```python\n# In WebSocket Handler\nasync def handle_task_dispatch(self, session_id, request, client_id):\n \"\"\"Handle incoming task from constellation.\"\"\"\n \n # Define callback to send results back\n async def send_result(sid: str, msg: ServerMessage):\n await self.websocket.send_text(msg.model_dump_json())\n logger.info(f\"Sent TASK_END for {sid}\")\n \n # Execute task with callback\n await self.session_manager.execute_task_async(\n session_id=session_id,\n task_name=f\"task_{session_id[:8]}\",\n request=request,\n task_protocol=self.task_protocol, # AIP protocol instance\n platform_override=None, # Auto-detect\n callback=send_result # Register callback\n )\n```\n\nFor more details, see the [WebSocket Handler Documentation](websocket_handler.md).\n\n### 2. Client Connection Manager Integration\n\nThe Client Connection Manager tracks which clients own which sessions:\n \n ```python\n # Track constellation sessions\n client_manager.add_constellation_session(\n constellation_id=\"constellation_001\",\n session_id=\"session_abc123\"\n )\n \n # Track device sessions\n client_manager.add_device_session(\n device_id=\"device_windows_001\",\n session_id=\"session_abc123\"\n )\n \n # Retrieve all sessions for a client\n session_ids = client_manager.get_constellation_sessions(\"constellation_001\")\n \n# On disconnect, cancel all client sessions\nfor session_id in session_ids:\n await session_manager.cancel_task(session_id, reason=\"client_disconnected\")\n```\n\nFor more details, see the [Client Connection Manager Documentation](client_connection_manager.md).\n\n### 3. HTTP API Integration\n\nThe API router uses SessionManager to retrieve results:\n\n```python\n# In API router (ufo/server/services/api.py)\n@router.post(\"/api/dispatch\")\nasync def dispatch_task_api(data: Dict[str, Any]):\n client_id = data.get(\"client_id\")\n user_request = data.get(\"request\")\n task_name = data.get(\"task_name\", str(uuid4()))\n \n # Get client protocol\n task_protocol = client_manager.get_task_protocol(client_id)\n if not task_protocol:\n raise HTTPException(status_code=404, detail=\"Client not online\")\n \n session_id = str(uuid4())\n \n # Use AIP protocol to send task\n # ... send TASK_ASSIGNMENT via protocol ...\n \n return {\n \"status\": \"dispatched\",\n \"task_name\": task_name,\n \"client_id\": client_id,\n \"session_id\": session_id\n }\n\n@router.get(\"/api/task_result/{task_name}\")\nasync def get_task_result(task_name: str):\n # Use SessionManager to retrieve results\n result = session_manager.get_result_by_task(task_name)\n if not result:\n return {\"status\": \"pending\"}\n return {\"status\": \"done\", \"result\": result}\n```\n\n---\n\n## 📖 API Reference\n\nComplete SessionManager API reference:### Initialization\n\n```python\nfrom ufo.server.services.session_manager import SessionManager\n\n# Initialize with platform override\nmanager = SessionManager(platform_override=\"windows\")\n\n# Initialize with auto-detection\nmanager = SessionManager(platform_override=None)\n```\n\n**Parameters:**\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `platform_override` | `Optional[str]` | `None` | Platform type (`\"windows\"`, `\"linux\"`, or `None` for auto-detect) |\n\n---\n\n### get_or_create_session()\n\n```python\nsession = manager.get_or_create_session(\n session_id=\"abc123\",\n task_name=\"demo_task\",\n request=\"Open Notepad\",\n task_protocol=task_protocol,\n platform_override=\"windows\",\n local=False\n)\n```\n\n**Parameters:**\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `session_id` | `str` | Yes | - | Unique session identifier |\n| `task_name` | `Optional[str]` | No | `\"test_task\"` | Human-readable task name |\n| `request` | `Optional[str]` | No | `None` | User request text |\n| `task_protocol` | `Optional[TaskExecutionProtocol]` | No | `None` | AIP TaskExecutionProtocol instance |\n| `platform_override` | `Optional[str]` | No | `None` | Platform type override |\n| `local` | `bool` | No | `False` | Whether to create local session (for testing) |\n\n**Returns:** `BaseSession` - Platform-specific session instance\n\n---\n\n### execute_task_async()\n\n```python\nsession_id = await manager.execute_task_async(\n session_id=\"abc123\",\n task_name=\"demo_task\",\n request=\"Open Notepad\",\n task_protocol=task_protocol,\n platform_override=\"windows\",\n callback=my_callback\n)\n```\n\n**Parameters:**\n\n| Parameter | Type | Required | Description |\n|-----------|------|----------|-------------|\n| `session_id` | `str` | Yes | Session identifier |\n| `task_name` | `str` | Yes | Task name |\n| `request` | `str` | Yes | User request text |\n| `task_protocol` | `Optional[TaskExecutionProtocol]` | No | AIP TaskExecutionProtocol instance |\n| `platform_override` | `str` | Yes | Platform type |\n| `callback` | `Optional[Callable]` | No | Async function called on completion |\n\n**Callback Signature:**\n\n```python\nasync def callback(session_id: str, result_message: ServerMessage) -> None:\n ...\n```\n\n**Returns:** `str` - The session ID (same as input)\n\n---\n\n### cancel_task()\n\n```python\nsuccess = await manager.cancel_task(\n session_id=\"abc123\",\n reason=\"device_disconnected\"\n)\n```\n\n**Parameters:**\n\n| Parameter | Type | Required | Default | Description |\n|-----------|------|----------|---------|-------------|\n| `session_id` | `str` | Yes | - | Session to cancel |\n| `reason` | `str` | No | `\"constellation_disconnected\"` | Cancellation reason |\n\n**Valid Reasons:**\n\n- `\"constellation_disconnected\"` - Don't send callback\n- `\"device_disconnected\"` - Send callback to constellation\n- `\"user_requested\"` - Manual cancellation\n\n**Returns:** `bool` - `True` if task was found and cancelled, `False` otherwise\n\n---\n\n### get_result()\n\n```python\nresult = manager.get_result(\"abc123\")\n```\n\n**Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `session_id` | `str` | Session identifier |\n\n**Returns:** `Optional[Dict[str, Any]]` - Session results dict, or `None` if not found\n\n---\n\n### get_result_by_task()\n\n```python\nresult = manager.get_result_by_task(\"demo_task\")\n```\n\n**Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `task_name` | `str` | Task name |\n\n**Returns:** `Optional[Dict[str, Any]]` - Session results dict, or `None` if not found\n\n---\n\n### set_results()\n\n```python\nmanager.set_results(\"abc123\")\n```\n\n**Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `session_id` | `str` | Session identifier |\n\n**Returns:** `None`\n\n**Purpose:** Persist session results to `results` dict for later retrieval\n\n---\n\n### remove_session()\n\n```python\nmanager.remove_session(\"abc123\")\n```\n\n**Parameters:**\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `session_id` | `str` | Session to remove |\n\n**Returns:** `None`\n\n**Purpose:** Remove session from active sessions dict (cleanup)\n\n---\n\n## 📚 Related Documentation\n\nExplore related components to understand the full server architecture:\n\n| Component | Purpose | Link |\n|-----------|---------|------|\n| **Server Overview** | High-level architecture and capabilities | [Overview](./overview.md) |\n| **Quick Start** | Start server and dispatch first task | [Quick Start](./quick_start.md) |\n| **WebSocket Handler** | Message handling and protocol implementation | [WebSocket Handler](./websocket_handler.md) |\n| **Client Connection Manager** | Connection management and client tracking | [Client Connection Manager](./client_connection_manager.md) |\n| **HTTP API** | RESTful API endpoints | [API Reference](./api.md) |\n| **Session Factory** | Session creation patterns | [Session Pool](../infrastructure/modules/session_pool.md) |\n| **AIP Protocol** | Agent Interaction Protocol details | [AIP Overview](../aip/overview.md) |\n\n---\n\n## 🎓 Key Takeaways\n\nAfter reading this guide, you should understand:\n\n- **Background execution** prevents WebSocket blocking during long tasks\n- **SessionFactory** creates platform-specific sessions (Windows/Linux)\n- **Callbacks** decouple task execution from result delivery\n- **Cancellation reasons** enable context-aware disconnection handling\n- **Thread safety** protects shared state in concurrent environments\n- **State management** uses five separate dicts (sessions, results, task_names, running_tasks, cancellation_reasons)\n- **Best practices** prevent resource exhaustion and memory leaks\n\n**Next Steps:**\n\n- Explore [WebSocket Handler](./websocket_handler.md) to see how sessions are triggered\n- Learn about [AIP Protocol](../aip/overview.md) for task assignment message format\n- Review [Client Connection Manager](./client_connection_manager.md) for session-to-client mapping\n\n"
},
{
"path": "documents/docs/server/websocket_handler.md",
"content": "# WebSocket Handler\n\nThe **UFOWebSocketHandler** is the central nervous system of the server, implementing the Agent Interaction Protocol (AIP) to manage structured, reliable communication between the server and all connected clients.\n\nFor context on how this component fits into the server architecture, see the [Server Overview](overview.md).\n\n---\n\n## 🎯 Overview\n\nThe WebSocket Handler acts as the protocol orchestrator, managing all aspects of client communication:\n\n| Responsibility | Description | Protocol Used |\n|----------------|-------------|---------------|\n| **Client Registration** | Validate and register new device/constellation connections | AIP Registration Protocol |\n| **Task Dispatch** | Route task requests to appropriate devices | AIP Task Execution Protocol |\n| **Heartbeat Monitoring** | Maintain connection health via periodic pings | AIP Heartbeat Protocol |\n| **Device Info Exchange** | Query and share device capabilities | AIP Device Info Protocol |\n| **Command Results** | Relay execution results from devices to requesters | AIP Message Transport |\n| **Error Handling** | Gracefully handle communication failures | AIP Error Protocol |\n| **Connection Lifecycle** | Manage registration → active → cleanup flow | WebSocket + AIP |\n\n### Architecture Position\n\n```mermaid\ngraph TB\n subgraph \"Clients\"\n DC[Device Clients]\n CC[Constellation Clients]\n end\n \n subgraph \"Server - WebSocket Handler\"\n WH[UFOWebSocketHandler]\n \n subgraph \"AIP Protocols\"\n REG[Registration Protocol]\n HB[Heartbeat Protocol]\n DI[Device Info Protocol]\n TE[Task Execution Protocol]\n end\n \n subgraph \"Message Router\"\n MH[handle_message]\n end\n end\n \n subgraph \"Server Components\"\n WSM[Client Connection Manager]\n SM[Session Manager]\n end\n \n DC -->|WebSocket /ws| WH\n CC -->|WebSocket /ws| WH\n \n WH --> REG\n WH --> HB\n WH --> DI\n WH --> TE\n \n WH --> MH\n MH -->|\"handle_task_request\"| TE\n MH -->|\"handle_heartbeat\"| HB\n MH -->|\"handle_device_info_request\"| DI\n MH -->|\"handle_command_result\"| SM\n \n WH -->|\"add_client / get_client\"| WSM\n WH -->|\"execute_task_async\"| SM\n \n style WH fill:#ffecb3\n style MH fill:#bbdefb\n style SM fill:#c8e6c9\n style WSM fill:#f8bbd0\n```\n\n---\n\n## 🔌 AIP Protocol Integration\n\nThe handler uses **four specialized AIP protocols**, each handling a specific aspect of communication. This separation of concerns makes the code maintainable and testable. For detailed protocol specifications, see the [AIP Protocol Documentation](../aip/overview.md).\n\n```python\ndef __init__(self, client_manager, session_manager, local=False):\n # Initialize per-connection protocols\n self.transport = None\n self.registration_protocol = None\n self.heartbeat_protocol = None\n self.device_info_protocol = None\n self.task_protocol = None\n```\n\n| Protocol | Purpose | Key Methods | Message Types |\n|----------|---------|-------------|---------------|\n| **Registration Protocol** | Client identity and validation | `send_registration_confirmation()`, `send_registration_error()` | `REGISTER`, `REGISTER_CONFIRM` |\n| **Heartbeat Protocol** | Connection health monitoring | `send_heartbeat_ack()` | `HEARTBEAT`, `HEARTBEAT_ACK` |\n| **Device Info Protocol** | Capability exchange | `send_device_info_response()`, `send_device_info_request()` | `DEVICE_INFO_REQUEST`, `DEVICE_INFO_RESPONSE` |\n| **Task Execution Protocol** | Task lifecycle management | `send_task_assignment()`, `send_ack()`, `send_error()` | `TASK`, `TASK_ASSIGNMENT`, `TASK_END` |\n\n**Protocol Initialization per Connection:**\n\n```python\nasync def connect(self, websocket: WebSocket) -> str:\n await websocket.accept()\n \n # Initialize AIP protocols for this connection\n self.transport = WebSocketTransport(websocket)\n self.registration_protocol = RegistrationProtocol(self.transport)\n self.heartbeat_protocol = HeartbeatProtocol(self.transport)\n self.device_info_protocol = DeviceInfoProtocol(self.transport)\n self.task_protocol = TaskExecutionProtocol(self.transport)\n \n # ... registration flow ...\n```\n\n**Per-Connection Protocol Instances:**\n\nEach WebSocket connection gets its **own set of protocol instances**, ensuring message routing and state management are isolated between clients.\n\n---\n\n## 📝 Client Registration\n\nRegistration is the critical first step when a client connects. The handler validates client identity, checks permissions, and establishes the communication session.\n\n### Registration Flow\n\n```mermaid\nsequenceDiagram\n participant C as Client (Device/Constellation)\n participant WS as WebSocket Handler\n participant RP as Registration Protocol\n participant WSM as Client Connection Manager\n \n Note over C,WS: 1️⃣ Connection Establishment\n C->>WS: WebSocket CONNECT /ws\n WS->>WS: await websocket.accept()\n WS->>WS: Initialize AIP protocols\n \n Note over WS,RP: 2️⃣ Registration Message\n WS->>C: (AIP Transport ready)\n C->>RP: REGISTER {client_id, client_type, platform, metadata}\n RP->>RP: Parse & validate JSON\n \n Note over RP,WS: 3️⃣ Validation\n RP->>WS: ClientMessage object\n WS->>WS: Validate client_id exists\n WS->>WS: Validate message type = REGISTER\n \n alt Client Type = Constellation\n WS->>WSM: is_device_connected(target_id)?\n alt Target device offline\n WSM-->>WS: False\n WS->>RP: send_registration_error()\n RP->>C: ERROR: \"Target device not connected\"\n WS->>C: WebSocket close()\n else Target device online\n WSM-->>WS: True\n WS->>WSM: add_client(client_id, ...)\n end\n else Client Type = Device\n WS->>WSM: add_client(client_id, platform, ...)\n end\n \n Note over WS,C: 4️⃣ Confirmation\n WS->>RP: send_registration_confirmation()\n RP->>C: REGISTER_CONFIRM {status: \"success\"}\n \n Note over C: Client registered, ready for tasks\n C->>C: Start message listening loop\n```\n\n### Registration Steps (Code Walkthrough)\n\n**Step 1: WebSocket Connection Accepted**\n\n```python\nasync def connect(self, websocket: WebSocket) -> str:\n # Accept WebSocket connection\n await websocket.accept()\n \n # Initialize AIP protocols for this connection\n self.transport = WebSocketTransport(websocket)\n self.registration_protocol = RegistrationProtocol(self.transport)\n self.heartbeat_protocol = HeartbeatProtocol(self.transport)\n self.device_info_protocol = DeviceInfoProtocol(self.transport)\n self.task_protocol = TaskExecutionProtocol(self.transport)\n```\n\n**Step 2: Receive Registration Message**\n\n```python\nasync def _parse_registration_message(self) -> ClientMessage:\n \"\"\"Parse and validate registration message using AIP Transport.\"\"\"\n self.logger.info(\"[WS] [AIP] Waiting for registration message...\")\n \n # Receive via AIP Transport\n reg_data = await self.transport.receive()\n if isinstance(reg_data, bytes):\n reg_data = reg_data.decode(\"utf-8\")\n \n # Parse using Pydantic model\n reg_info = ClientMessage.model_validate_json(reg_data)\n \n self.logger.info(\n f\"[WS] [AIP] Received registration from {reg_info.client_id}, \"\n f\"type={reg_info.client_type}\"\n )\n \n return reg_info\n```\n\n**Expected Registration Message:**\n\n```json\n{\n \"type\": \"REGISTER\",\n \"client_id\": \"device_windows_001\",\n \"client_type\": \"DEVICE\",\n \"platform\": \"windows\",\n \"metadata\": {\n \"hostname\": \"DESKTOP-ABC123\",\n \"os_version\": \"Windows 11 Pro\",\n \"screen_resolution\": \"1920x1080\"\n }\n}\n```\n\n**Step 3: Validation**\n\n```python\n# Basic validation\nif not reg_info.client_id:\n raise ValueError(\"Client ID is required for WebSocket registration\")\nif reg_info.type != ClientMessageType.REGISTER:\n raise ValueError(\"First message must be a registration message\")\n\n# Constellation-specific validation\nif client_type == ClientType.CONSTELLATION:\n await self._validate_constellation_client(reg_info)\n```\n\n**Constellation Validation:**\n\n```python\nasync def _validate_constellation_client(self, reg_info: ClientMessage) -> None:\n \"\"\"Validate constellation's claimed target_id.\"\"\"\n claimed_device_id = reg_info.target_id\n \n if not claimed_device_id:\n return # No device_id to validate\n \n # Check if target device is connected\n if not self.client_manager.is_device_connected(claimed_device_id):\n error_msg = f\"Target device '{claimed_device_id}' is not connected\"\n self.logger.warning(f\"[WS] Constellation registration failed: {error_msg}\")\n \n # Send error via AIP protocol\n await self._send_error_response(error_msg)\n await self.transport.close()\n raise ValueError(error_msg)\n```\n\n**Step 4: Register Client in [ClientConnectionManager](./client_connection_manager.md)**\n\n```python\nclient_type = reg_info.client_type\nplatform = reg_info.metadata.get(\"platform\", \"windows\") if reg_info.metadata else \"windows\"\n\n# Register in Client Connection Manager\nself.client_manager.add_client(\n client_id,\n platform,\n websocket,\n client_type,\n reg_info.metadata,\n transport=self.transport,\n task_protocol=self.task_protocol,\n)\n```\n\n**Step 5: Send Confirmation**\n\n```python\nasync def _send_registration_confirmation(self) -> None:\n \"\"\"Send successful registration confirmation using AIP RegistrationProtocol.\"\"\"\n self.logger.info(\"[WS] [AIP] Sending registration confirmation...\")\n await self.registration_protocol.send_registration_confirmation()\n self.logger.info(\"[WS] [AIP] Registration confirmation sent\")\n```\n\n**Confirmation Message:**\n\n```json\n{\n \"type\": \"REGISTER_CONFIRM\",\n \"status\": \"success\",\n \"timestamp\": \"2024-11-04T14:30:22.123456+00:00\",\n \"response_id\": \"uuid-v4\"\n}\n```\n\n**Step 6: Log Success**\n\n```python\ndef _log_client_connection(self, client_id: str, client_type: ClientType) -> None:\n \"\"\"Log successful client connection with appropriate emoji.\"\"\"\n if client_type == ClientType.DEVICE:\n self.logger.info(f\"[WS] Registered device client: {client_id}\")\n elif client_type == ClientType.CONSTELLATION:\n self.logger.info(f\"[WS] 🌟 Registered constellation client: {client_id}\")\n```\n\n### Validation Rules\n\n| Validation | Check | Error Message | Action |\n|------------|-------|---------------|--------|\n| **Client ID Presence** | `client_id` field exists and not empty | `\"Client ID is required\"` | Reject connection |\n| **Message Type** | First message type == `REGISTER` | `\"First message must be a registration message\"` | Reject connection |\n| **Target Device (Constellation)** | If `target_id` specified, device must be online | `\"Target device '' is not connected\"` | Send error + close |\n| **Client ID Uniqueness** | No existing client with same ID | Handled by ClientConnectionManager | Disconnect old connection |\n\n**Constellation Dependency:**\n\nConstellations **must** specify a valid `target_id` that refers to an already-connected device. If the device is offline or doesn't exist, registration fails immediately.\n\n**Workaround:** Connect devices first, then constellations.\n\n**Security Consideration:**\n The current implementation does **not** authenticate clients. Any client can register with any `client_id`. For production deployments:\n \n - Implement authentication tokens in `metadata`\n - Validate client certificates (TLS client auth)\n - Use API keys or OAuth tokens\n - Whitelist allowed `client_id` patterns\n\n---\n\n## 📨 Message Handling\n\nAfter registration, the handler enters a message loop, routing incoming client messages to specialized handlers based on message type.\n\n### Message Dispatcher\n\n```mermaid\ngraph TB\n WS[WebSocket receive_text] --> Parse[Parse ClientMessage JSON]\n Parse --> Router{Message Type?}\n \n Router -->|TASK| HT[handle_task_request]\n Router -->|COMMAND_RESULTS| HC[handle_command_result]\n Router -->|HEARTBEAT| HH[handle_heartbeat]\n Router -->|ERROR| HE[handle_error]\n Router -->|DEVICE_INFO_REQUEST| HD[handle_device_info_request]\n Router -->|DEVICE_INFO_RESPONSE| HDR[handle_device_info_response]\n Router -->|Unknown| HU[handle_unknown]\n \n HT --> SM[Session Manager]\n HC --> CD[Command Dispatcher]\n HH --> HP[Heartbeat Protocol]\n HE --> Log[Error Logging]\n HD --> DIP[Device Info Protocol]\n \n style Router fill:#ffe0b2\n style SM fill:#c8e6c9\n style HP fill:#bbdefb\n```\n\n**Dispatcher Implementation:**\n\n```python\nasync def handle_message(self, msg: str, websocket: WebSocket) -> None:\n \"\"\"Dispatch incoming messages to specific handlers.\"\"\"\n try:\n # Parse message using Pydantic model\n data = ClientMessage.model_validate_json(msg)\n \n client_id = data.client_id\n client_type = data.client_type\n msg_type = data.type\n \n # Route to appropriate handler\n if msg_type == ClientMessageType.TASK:\n await self.handle_task_request(data, websocket)\n elif msg_type == ClientMessageType.COMMAND_RESULTS:\n await self.handle_command_result(data)\n elif msg_type == ClientMessageType.HEARTBEAT:\n await self.handle_heartbeat(data, websocket)\n elif msg_type == ClientMessageType.ERROR:\n await self.handle_error(data, websocket)\n elif msg_type == ClientMessageType.DEVICE_INFO_REQUEST:\n await self.handle_device_info_request(data, websocket)\n elif msg_type == ClientMessageType.DEVICE_INFO_RESPONSE:\n await self.handle_device_info_response(data, websocket)\n else:\n await self.handle_unknown(data, websocket)\n \n except Exception as e:\n self.logger.error(f\"Error handling message: {e}\")\n try:\n await self.task_protocol.send_error(str(e))\n except (ConnectionError, IOError):\n pass # Connection already closed\n```\n\n**Message Type Handlers:**\n\n| Handler | Triggered By | Purpose | Response |\n|---------|-------------|---------|----------|\n| `handle_task_request` | `TASK` | Client requests task execution | `TASK_ASSIGNMENT` device |\n| `handle_command_result` | `COMMAND_RESULTS` | Device reports command execution result | Unblock command dispatcher |\n| `handle_heartbeat` | `HEARTBEAT` | Connection health ping | `HEARTBEAT_ACK` |\n| `handle_error` | `ERROR` | Client reports error | Log + send error acknowledgment |\n| `handle_device_info_request` | `DEVICE_INFO_REQUEST` | Constellation queries device capabilities | `DEVICE_INFO_RESPONSE` |\n| `handle_device_info_response` | `DEVICE_INFO_RESPONSE` | Device provides info (pull model) | Store in ClientConnectionManager |\n| `handle_unknown` | Any other type | Unknown/unsupported message | Log warning + send error |\n\n---\n\n### Task Request Handling\n\nThe handler supports task requests from **both device clients** (self-execution) and **constellation clients** (orchestrated execution on target devices).\n\n**Task Request Flow:**\n\n```mermaid\nsequenceDiagram\n participant C as Constellation\n participant WH as WebSocket Handler\n participant WSM as Client Connection Manager\n participant SM as Session Manager\n participant D as Device\n \n Note over C,WH: 1️⃣ Task Request\n C->>WH: TASK {request, target_id, session_id}\n WH->>WH: Validate target_id\n \n Note over WH,WSM: 2️⃣ Resolve Target Device\n WH->>WSM: get_client(target_id)\n WSM-->>WH: Device WebSocket\n WH->>WSM: get_client_info(target_id)\n WSM-->>WH: {platform: \"windows\"}\n \n Note over WH,SM: 3️⃣ Create Session\n WH->>SM: execute_task_async( session_id, request, target_ws, platform, callback )\n SM-->>WH: session_id (non-blocking!)\n \n Note over WH,C: 4️⃣ Immediate Acknowledgment\n WH->>C: ACK {session_id, status: \"dispatched\"}\n \n Note over SM,D: 5️⃣ Background Execution\n SM->>D: TASK_ASSIGNMENT {request, session_id}\n D->>D: Execute task (LLM + actions)\n \n Note over D,WH: 6️⃣ Result Callback\n D-->>SM: Task complete\n SM->>WH: callback(session_id, result_msg)\n WH->>C: TASK_END {status, result}\n```\n\n**Device Client Self-Execution:**\n\nWhen a **device** requests a task for itself:\n\n```python\nasync def handle_task_request(self, data: ClientMessage, websocket: WebSocket):\n client_id = data.client_id\n client_type = data.client_type\n \n if client_type == ClientType.DEVICE:\n # Device executing task on itself\n target_ws = websocket # Use requesting client's WebSocket\n platform = self.client_manager.get_client_info(client_id).platform\n target_device_id = client_id\n # ...\n```\n\n**Constellation Orchestrated Execution:**\n\nWhen a **constellation** dispatches a task to a target device:\n\n```python\nasync def handle_task_request(self, data: ClientMessage, websocket: WebSocket):\n client_id = data.client_id\n client_type = data.client_type\n \n if client_type == ClientType.CONSTELLATION:\n # Constellation dispatching to target device\n target_device_id = data.target_id\n target_ws = self.client_manager.get_client(target_device_id)\n platform = self.client_manager.get_client_info(target_device_id).platform\n \n # Validate target device exists\n if not target_ws:\n raise ValueError(f\"Target device '{target_device_id}' not connected\")\n \n # Track session mappings\n session_id = data.session_id or str(uuid.uuid4())\n self.client_manager.add_constellation_session(client_id, session_id)\n self.client_manager.add_device_session(target_device_id, session_id)\n # ...\n ```\n\n**Background Task Execution:**\n\n```python\n# Define callback for result delivery\nasync def send_result(sid: str, result_msg: ServerMessage):\n \"\"\"Send result back to requester when task completes.\"\"\"\n # Send to constellation client\n if client_type == ClientType.CONSTELLATION:\n if websocket.client_state == WebSocketState.CONNECTED:\n await websocket.send_text(result_msg.model_dump_json())\n \n # Also send to target device (optional)\n if target_ws and target_ws.client_state == WebSocketState.CONNECTED:\n await target_ws.send_text(result_msg.model_dump_json())\n else:\n # Send to device client\n if websocket.client_state == WebSocketState.CONNECTED:\n await websocket.send_text(result_msg.model_dump_json())\n\n# Execute in background via SessionManager\nawait self.session_manager.execute_task_async(\n session_id=session_id,\n task_name=task_name,\n request=data.request,\n websocket=target_ws, # Device WebSocket for command dispatcher\n platform_override=platform,\n callback=send_result # Called when task completes\n)\n\n# Send immediate acknowledgment (non-blocking)\nawait self.task_protocol.send_ack(session_id=session_id)\n```\n\n**Why Immediate ACK?**\n\nThe handler sends an **immediate ACK** after dispatching the task to the [SessionManager](./session_manager.md). This confirms:\n\n- Task was received and validated\n- Session was created successfully\n- Task is now executing in background\n\nThe actual task result is delivered later via the `send_result` callback.\n\n**Session Tracking:**\n\n| Client Type | Session Tracking | Purpose |\n|-------------|------------------|---------|\n| **Device** | `client_manager.add_device_session(device_id, session_id)` | Track which device is executing the session |\n| **Constellation** | `client_manager.add_constellation_session(constellation_id, session_id)` | Track which constellation requested the session |\n| Both | Session Manager stores session `BaseSession` object | Execute and manage task lifecycle |\n\n---\n\n### Command Result Handling\n\nWhen a device executes a command (e.g., \"click button\", \"type text\"), it sends results back to the server for processing by the session's command dispatcher.\n\n**Command Result Flow:**\n\n```mermaid\nsequenceDiagram\n participant S as Session (on server)\n participant CD as Command Dispatcher\n participant D as Device\n participant WH as WebSocket Handler\n \n Note over S,D: Session is running on server\n S->>CD: execute_command(\"open_notepad\")\n CD->>D: Send command via WebSocket (response_id=\"cmd_123\")\n CD->>CD: await response (blocking)\n \n Note over D: Device executes command\n D->>D: Open Notepad application\n D->>D: Take screenshot\n \n Note over D,WH: Send result back\n D->>WH: COMMAND_RESULTS {response_id=\"cmd_123\", result, screenshot}\n WH->>WH: handle_command_result()\n WH->>CD: set_result(response_id, data)\n \n Note over CD: Unblocks await!\n CD-->>S: Return command result\n S->>S: Continue session execution\n```\n\n**Handler Implementation:**\n\n```python\nasync def handle_command_result(self, data: ClientMessage):\n \"\"\"\n Handle command execution results from devices.\n Unblocks the command dispatcher waiting for this response.\n \"\"\"\n response_id = data.prev_response_id # ID of the command request\n session_id = data.session_id\n \n self.logger.debug(\n f\"[WS] Received command result for response_id={response_id}, \"\n f\"session_id={session_id}\"\n )\n \n # Get session's command dispatcher\n session = self.session_manager.get_or_create_session(session_id)\n command_dispatcher = session.context.command_dispatcher\n \n # Set result (unblocks waiting dispatcher)\n await command_dispatcher.set_result(response_id, data)\n \n self.logger.debug(\n f\"[WS] Command result set for response_id={response_id}\"\n )\n```\n\n**Critical for Session Execution:**\n\nWithout proper command result handling, sessions would **hang indefinitely** waiting for device responses. The `set_result()` call is what unblocks the `await` in the command dispatcher.\n\n---\n\n### Heartbeat Handling\n\nHeartbeats are lightweight ping/pong messages that ensure the WebSocket connection is alive and healthy.\n\n```python\nasync def handle_heartbeat(self, data: ClientMessage, websocket: WebSocket) -> None:\n \"\"\"Handle heartbeat messages using AIP HeartbeatProtocol.\"\"\"\n self.logger.debug(f\"[WS] [AIP] Heartbeat from {data.client_id}\")\n \n try:\n # Send acknowledgment via AIP protocol\n await self.heartbeat_protocol.send_heartbeat_ack()\n self.logger.debug(f\"[WS] [AIP] Heartbeat response sent to {data.client_id}\")\n except (ConnectionError, IOError) as e:\n # Connection closed - log but don't fail\n self.logger.debug(f\"[WS] [AIP] Could not send heartbeat ack: {e}\")\n```\n\n**Heartbeat Message:**\n\n```json\n{\n \"type\": \"HEARTBEAT\",\n \"client_id\": \"device_windows_001\",\n \"timestamp\": \"2024-11-04T14:30:22.123456+00:00\"\n}\n```\n\n**Heartbeat ACK:**\n\n```json\n{\n \"type\": \"HEARTBEAT_ACK\",\n \"timestamp\": \"2024-11-04T14:30:22.234567+00:00\",\n \"response_id\": \"uuid-v4\"\n}\n```\n\n**Heartbeat Best Practices:**\n\n- **Frequency:** Clients should send heartbeats every **30-60 seconds**\n- **Timeout:** Server should consider connection dead after **2-3 missed heartbeats**\n- **Lightweight:** Heartbeat messages are small and processed quickly\n- **Non-blocking:** Heartbeat handling doesn't block task execution\n\n---\n\n### Device Info Handling\n\nConstellations can query device capabilities (screen resolution, installed apps, OS version) to make intelligent task routing decisions.\n\n**Device Info Request Flow:**\n\n```mermaid\nsequenceDiagram\n participant C as Constellation\n participant WH as WebSocket Handler\n participant WSM as Client Connection Manager\n participant DIP as Device Info Protocol\n participant D as Device\n \n Note over C,WH: 1️⃣ Request Device Info\n C->>WH: DEVICE_INFO_REQUEST {target_id, request_id}\n \n Note over WH,WSM: 2️⃣ Resolve Device\n WH->>WSM: get_client(target_id)\n WSM-->>WH: Device WebSocket\n \n Note over WH,D: 3️⃣ Forward Request\n WH->>DIP: send_device_info_request()\n DIP->>D: DEVICE_INFO_REQUEST\n \n Note over D: 4️⃣ Collect Info\n D->>D: Gather system info (screen, OS, apps)\n \n Note over D,WH: 5️⃣ Response\n D->>DIP: DEVICE_INFO_RESPONSE {screen_res, os_version, ...}\n DIP->>WH: Parse response\n \n Note over WH,C: 6️⃣ Forward to Constellation\n WH->>C: DEVICE_INFO_RESPONSE {device_info, request_id}\n```\n\n```python\nasync def handle_device_info_request(\n self, data: ClientMessage, websocket: WebSocket\n) -> None:\n \"\"\"Handle device info requests from constellations.\"\"\"\n device_id = data.target_id\n request_id = data.request_id\n \n self.logger.info(\n f\"[WS] Constellation {data.client_id} requesting info for device {device_id}\"\n )\n \n # Get device info (may involve querying the device)\n device_info = await self.get_device_info(device_id)\n \n # Send via AIP protocol\n await self.device_info_protocol.send_device_info_response(\n device_info=device_info,\n request_id=request_id\n )\n```\n\n**Device Info Structure:**\n\n```json\n{\n \"device_id\": \"device_windows_001\",\n \"platform\": \"windows\",\n \"os_version\": \"Windows 11 Pro 22H2\",\n \"screen_resolution\": \"1920x1080\",\n \"installed_applications\": [\"Chrome\", \"Excel\", \"Notepad\", \"...\"],\n \"capabilities\": [\"ui_automation\", \"file_operations\", \"web_browsing\"],\n \"cpu_cores\": 8,\n \"memory_gb\": 16\n}\n```\n\n---\n\n## 🔌 Client Disconnection\n\n**Critical Cleanup Process:**\n\nWhen a client disconnects (gracefully or abruptly), the handler must clean up sessions, remove registry entries, and prevent resource leaks.\n\n### Disconnection Detection\n\n```python\nasync def handler(self, websocket: WebSocket) -> None:\n \"\"\"FastAPI WebSocket entry point.\"\"\"\n client_id = None\n \n try:\n # Registration\n client_id = await self.connect(websocket)\n \n # Message loop\n while True:\n msg = await websocket.receive_text()\n asyncio.create_task(self.handle_message(msg, websocket))\n \n except WebSocketDisconnect as e:\n # Normal disconnection\n self.logger.warning(\n f\"[WS] {client_id} disconnected code={e.code}, reason={e.reason}\"\n )\n if client_id:\n await self.disconnect(client_id)\n \n except Exception as e:\n # Unexpected error\n self.logger.error(f\"[WS] Error with client {client_id}: {e}\")\n if client_id:\n await self.disconnect(client_id)\n```\n\n### Cleanup Process\n\n```mermaid\ngraph TD\n A[Client Disconnects] --> B{Get Client Info}\n B --> C{Client Type?}\n \n C -->|Device| D[Get Device Sessions]\n C -->|Constellation| E[Get Constellation Sessions]\n \n D --> F[Cancel Each Session reason='device_disconnected']\n E --> G[Cancel Each Session reason='constellation_disconnected']\n \n F --> H[Remove Device Session Mappings]\n G --> I[Remove Constellation Session Mappings]\n \n H --> J[Remove Client from ClientConnectionManager]\n I --> J\n \n J --> K[Log Disconnection]\n K --> L[Cleanup Complete]\n \n style F fill:#ffcdd2\n style G fill:#ffcdd2\n style J fill:#c8e6c9\n```\n\n**Device Client Cleanup:**\n\n```python\nasync def disconnect(self, client_id: str) -> None:\n \"\"\"Handle client disconnection and cleanup.\"\"\"\n client_info = self.client_manager.get_client_info(client_id)\n \n if client_info and client_info.client_type == ClientType.DEVICE:\n # Get all sessions running on this device\n session_ids = self.client_manager.get_device_sessions(client_id)\n \n if session_ids:\n self.logger.info(\n f\"[WS] 📱 Device {client_id} disconnected, \"\n f\"cancelling {len(session_ids)} active session(s)\"\n )\n \n # Cancel all sessions\n for session_id in session_ids:\n try:\n await self.session_manager.cancel_task(\n session_id,\n reason=\"device_disconnected\" # Send callback to constellation\n )\n except Exception as e:\n self.logger.error(f\"Error cancelling session {session_id}: {e}\")\n \n # Clean up mappings\n self.client_manager.remove_device_sessions(client_id)\n```\n\n**Constellation Client Cleanup:**\n\n```python\nif client_info and client_info.client_type == ClientType.CONSTELLATION:\n # Get all sessions initiated by constellation\n session_ids = self.client_manager.get_constellation_sessions(client_id)\n \n if session_ids:\n self.logger.info(\n f\"[WS] 🌟 Constellation {client_id} disconnected, \"\n f\"cancelling {len(session_ids)} active session(s)\"\n )\n \n # Cancel all associated sessions\n for session_id in session_ids:\n try:\n await self.session_manager.cancel_task(\n session_id,\n reason=\"constellation_disconnected\" # Don't send callback\n )\n except Exception as e:\n self.logger.error(f\"Error cancelling session {session_id}: {e}\")\n \n # Clean up mappings\n self.client_manager.remove_constellation_sessions(client_id)\n```\n\n**Final Registry Cleanup:**\n\n```python\n# Remove client from registry\nself.client_manager.remove_client(client_id)\nself.logger.info(f\"[WS] {client_id} disconnected\")\n```\n\n### Cancellation Behavior Comparison\n\n| Scenario | Cancellation Reason | Callback Sent? | Why? |\n|----------|---------------------|----------------|------|\n| **Device Disconnects** | `device_disconnected` | Yes Constellation | Notify orchestrator to reassign task |\n| **Constellation Disconnects** | `constellation_disconnected` | No | Requester is gone, no one to notify |\n\n**Proper Cleanup is Critical:**\n\nFailing to clean up disconnected clients leads to:\n\n- **Orphaned sessions** consuming server memory\n- **Stale WebSocket references** causing errors\n- **Registry pollution** with non-existent clients\n- **Resource leaks** (file handles, memory)\n\n---\n\n## 🚨 Error Handling\n\nThe handler implements comprehensive error handling to prevent failures from cascading and breaking the entire server.\n\n### Error Categories\n\n| Error Type | Handler Location | Recovery Strategy |\n|------------|------------------|-------------------|\n| **Connection Errors** | `send_*` methods | Log and skip (connection already closed) |\n| **Message Parsing Errors** | `handle_message` | Send error response via AIP |\n| **Task Execution Errors** | `handle_task_request` | Log + send error via task protocol |\n| **Validation Errors** | `_validate_*` methods | Send error + close connection |\n| **Callback Errors** | Session Manager | Log but don't fail session |\n\n### Connection Error Handling\n\n```python\nasync def handle_heartbeat(self, data: ClientMessage, websocket: WebSocket):\n try:\n await self.heartbeat_protocol.send_heartbeat_ack()\n except (ConnectionError, IOError) as e:\n # Connection closed - log but don't fail\n self.logger.debug(f\"Could not send heartbeat ack: {e}\")\n # Don't raise - connection is already closed\n```\n\n**Why Catch and Ignore?**\n\nWhen a connection is abruptly closed, attempts to send messages will raise `ConnectionError`. Since the client is already gone, there's no point in propagating the error—just log it and continue cleanup.\n\n### Message Parsing Errors\n\n```python\nasync def handle_message(self, msg: str, websocket: WebSocket):\n try:\n data = ClientMessage.model_validate_json(msg)\n # ... route to handlers ...\n \n except Exception as e:\n import traceback\n traceback.print_exc()\n self.logger.error(f\"Error handling message: {e}\")\n \n # Try to send error response\n try:\n await self.task_protocol.send_error(str(e))\n except (ConnectionError, IOError) as send_error:\n self.logger.debug(f\"Could not send error response: {send_error}\")\n```\n\n**Error Message Format:**\n\n```json\n{\n \"type\": \"ERROR\",\n \"error\": \"Invalid message format: missing required field 'client_id'\",\n \"timestamp\": \"2024-11-04T14:30:22.123456+00:00\",\n \"response_id\": \"uuid-v4\"\n}\n```\n\n### Task Execution Errors\n\n```python\nasync def handle_task_request(self, data: ClientMessage, websocket: WebSocket):\n try:\n # Validate target device\n if client_type == ClientType.CONSTELLATION:\n target_ws = self.client_manager.get_client(target_device_id)\n if not target_ws:\n raise ValueError(f\"Target device '{target_device_id}' not connected\")\n \n # Execute task\n await self.session_manager.execute_task_async(...)\n \n except Exception as e:\n self.logger.error(f\"Error handling task: {e}\")\n await self.task_protocol.send_error(str(e))\n```\n\n### Validation Errors with Connection Closure\n\n```python\nasync def _validate_constellation_client(self, reg_info: ClientMessage) -> None:\n \"\"\"Validate constellation's target device.\"\"\"\n claimed_device_id = reg_info.target_id\n \n if not self.client_manager.is_device_connected(claimed_device_id):\n error_msg = f\"Target device '{claimed_device_id}' is not connected\"\n self.logger.warning(f\"Constellation registration failed: {error_msg}\")\n \n # Send error via AIP protocol\n await self._send_error_response(error_msg)\n \n # Close connection immediately\n await self.transport.close()\n \n # Raise to prevent further processing\n raise ValueError(error_msg)\n```\n\n**When to Close Connections:**\n\nClose connections immediately for:\n\n- **Invalid registration** (missing client_id, wrong message type)\n- **Authorization failures** (target device not connected for constellations)\n- **Protocol violations** (sending TASK before REGISTER)\n\nFor other errors, log and send error messages but **keep connection alive**.\n\n---\n\n## Best Practices\n\n### 1. Validate Early and Thoroughly\n\n```python\n# Good: Validate immediately after parsing\nasync def handle_task_request(self, data: ClientMessage, websocket: WebSocket):\n if not data.request:\n raise ValueError(\"Task request cannot be empty\")\n if not data.client_id:\n raise ValueError(\"Client ID required\")\n if data.client_type == ClientType.CONSTELLATION and not data.target_id:\n raise ValueError(\"Constellation must specify target_id\")\n # ... proceed with validated data ...\n```\n\n### 2. Always Check Connection State Before Sending\n\n```python\nfrom starlette.websockets import WebSocketState\n\n# Good: Check state before sending\nasync def send_result(sid: str, result_msg: ServerMessage):\n if websocket.client_state == WebSocketState.CONNECTED:\n await websocket.send_text(result_msg.model_dump_json())\n else:\n self.logger.debug(f\"Cannot send result, connection closed for {sid}\")\n```\n\n**WebSocket States:**\n\n| State | Description | Can Send? |\n|-------|-------------|-----------|\n| `CONNECTING` | Handshake in progress | No |\n| `CONNECTED` | Active connection | Yes |\n| `DISCONNECTED` | Connection closed | No |\n\n### 3. Handle Cancellation Gracefully with Context\n\n```python\n# Good: Different reasons need different handling\nasync def disconnect(self, client_id: str):\n client_info = self.client_manager.get_client_info(client_id)\n \n if client_info.client_type == ClientType.CONSTELLATION:\n reason = \"constellation_disconnected\" # Don't send callback\n else:\n reason = \"device_disconnected\" # Send callback to constellation\n \n for session_id in session_ids:\n await self.session_manager.cancel_task(session_id, reason=reason)\n```\n\n### 4. Use Structured Logging with Context\n\n```python\n# Good: Include client type and context\nif client_type == ClientType.CONSTELLATION:\n self.logger.info(\n f\"[WS] 🌟 Constellation {client_id} requesting task on {target_id}\"\n )\nelse:\n self.logger.debug(\n f\"[WS] 📱 Received device message from {client_id}, type: {data.type}\"\n )\n```\n\n**Logging Levels:**\n\n- `DEBUG`: Heartbeats, message routing, low-level protocol details\n- `INFO`: Registration, disconnection, task dispatch, major lifecycle events\n- `WARNING`: Validation failures, connection issues, recoverable errors\n- `ERROR`: Unexpected exceptions, critical failures\n\n### 5. Implement Async Message Handling\n\n```python\n# Good: Process messages in background tasks\nasync def handler(self, websocket: WebSocket):\n while True:\n msg = await websocket.receive_text()\n asyncio.create_task(self.handle_message(msg, websocket))\n # Loop continues immediately, doesn't wait for handler to finish\n```\n\n**Why `asyncio.create_task`?**\n\nWithout `create_task`, the handler would process messages **sequentially**, blocking new messages while handling the current one. This is problematic for:\n\n- Long-running task dispatches\n- Command result processing\n- Device info queries\n\nBackground tasks allow **concurrent message processing** while keeping the receive loop responsive.\n\n---\n\n## 📚 Related Documentation\n\nExplore related components to understand the full server architecture:\n\n| Component | Purpose | Link |\n|-----------|---------|------|\n| **Server Overview** | High-level architecture and capabilities | [Overview](./overview.md) |\n| **Quick Start** | Start server and dispatch first task | [Quick Start](./quick_start.md) |\n| **Session Manager** | Session lifecycle and background execution | [Session Manager](./session_manager.md) |\n| **Client Connection Manager** | Connection registry and session tracking | [Client Connection Manager](./client_connection_manager.md) |\n| **HTTP API** | RESTful API endpoints | [API Reference](./api.md) |\n| **AIP Protocol** | Agent Interaction Protocol details | [AIP Overview](../aip/overview.md) |\n\n---\n\n## 🎓 What You Learned\n\nAfter reading this guide, you should understand:\n\n- **AIP Protocol Integration** - Four specialized protocols handle different communication aspects\n- **Registration Flow** - Validation → Registration → Confirmation\n- **Message Routing** - Central dispatcher routes messages to specialized handlers\n- **Dual Client Support** - Devices (self-execution) vs. Constellations (orchestration)\n- **Background Task Dispatch** - Immediate ACK + async execution\n- **Command Result Handling** - Unblocks command dispatcher waiting for device responses\n- **Heartbeat Monitoring** - Lightweight connection health checks\n- **Disconnection Cleanup** - Context-aware session cancellation and registry cleanup\n- **Error Handling** - Graceful degradation without cascading failures\n\n**Next Steps:**\n\n- Explore [Session Manager](./session_manager.md) to understand background execution internals\n- Learn about [Client Connection Manager](./client_connection_manager.md) for client registry management\n- Review [AIP Protocol Documentation](../aip/overview.md) for message format specifications\n\n"
},
{
"path": "documents/docs/tutorials/creating_app_agent/demonstration_provision.md",
"content": "# Provide Human Demonstrations to the AppAgent\n\nUsers or application developers can provide human demonstrations to the `AppAgent` to guide it in executing similar tasks in the future. The `AppAgent` uses these demonstrations to understand the context of the task and the steps required to execute it, effectively becoming an expert in the application.\n\n## How to Prepare Human Demonstrations for the AppAgent?\n\nCurrently, UFO supports learning from user trajectories recorded by [Steps Recorder](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47) integrated within Windows. More tools will be supported in the future.\n\n### Step 1: Recording User Demonstrations\n\nFollow the [official guidance](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47) to use Steps Recorder to record user demonstrations.\n\n### Step 2: Add Additional Information or Comments as Needed\n\nInclude any specific details or instructions for UFO to notice by adding comments. Since Steps Recorder doesn't capture typed text, include any necessary typed content in the comments as well.\n\n\n \n Adding comments in Steps Recorder for additional context\n\n\n\n### Step 3: Review and Save the Recorded Demonstrations\n\nReview the recorded steps and save them to a ZIP file. Refer to the [sample_record.zip](https://github.com/microsoft/UFO/blob/main/record_processor/example/sample_record.zip) for an example of recorded steps for a specific request, such as \"sending an email to example@gmail.com to say hi.\"\n\n### Step 4: Create an Action Trajectory Indexer\n\nOnce you have your demonstration record ZIP file ready, you can parse it as an example to support RAG for UFO. Follow these steps:\n\n```bash\n# Assume you are in the cloned UFO folder\npython -m record_processor -r \"\" -p \"\"\n```\n\n- Replace `` with the specific request, such as \"sending an email to example@gmail.com to say hi.\"\n- Replace `` with the full path to the ZIP file you just created.\n\nThis command will parse the record and summarize it into an execution plan. You'll see a confirmation message similar to the following:\n\n```bash\nHere are the plans summarized from your demonstration:\nPlan [1]\n(1) Input the email address 'example@gmail.com' in the 'To' field.\n(2) Input the subject of the email. I need to input 'Greetings'.\n(3) Input the content of the email. I need to input 'Hello,\\nI hope this message finds you well. I am writing to send you a warm greeting and to wish you a great day.\\nBest regards.'\n(4) Click the Send button to send the email.\nPlan [2]\n(1) ***\n(2) ***\n(3) ***\nPlan [3]\n(1) ***\n(2) ***\n(3) ***\nWould you like to save any one of them as a future reference for the agent? Press [1] [2] [3] to save the corresponding plan, or press any other key to skip.\n```\n\nPress `1` to save the plan into its memory for future reference. A sample can be found [here](https://github.com/microsoft/UFO/blob/main/vectordb/demonstration/example.yaml).\n\nYou can view a demonstration video [here](https://github.com/yunhao0204/UFO/assets/59384816/0146f83e-1b5e-4933-8985-fe3f24ec4777).\n\n## How to Use Human Demonstrations to Enhance the AppAgent?\n\nAfter creating the offline indexer, refer to the [Learning from User Demonstrations](../../ufo2/core_features/knowledge_substrate/learning_from_demonstration.md) section for guidance on how to use human demonstrations to enhance the AppAgent.\n\n## Related Documentation\n\n- [Overview: Enhancing AppAgent Capabilities](./overview.md) - Learn about all enhancement approaches\n- [Help Document Provision](./help_document_provision.md) - Provide knowledge through documentation\n- [Wrapping App-Native API](./warpping_app_native_api.md) - Create efficient MCP action servers\n- [Knowledge Substrate Overview](../../ufo2/core_features/knowledge_substrate/overview.md) - Understanding the RAG architecture"
},
{
"path": "documents/docs/tutorials/creating_app_agent/help_document_provision.md",
"content": "# Providing Help Documents to the AppAgent\n\nHelp documents provide guidance to the `AppAgent` in executing specific tasks. The `AppAgent` uses these documents to understand the context of the task and the steps required to execute it, effectively becoming an expert in the application.\n\n## Step 1: Prepare Help Documents and Metadata\n\nUFO currently supports processing help documents in `json` format. More formats will be supported in the future.\n\nAn example of a help document in `json` format is as follows:\n\n```json\n{\n \"application\": \"chrome\",\n \"request\": \"How to change the username in chrome profiles?\",\n \"guidance\": [\n \"Click the profile icon in the upper-right corner of the Chrome window.\",\n \"Click the gear icon labeled 'Manage Chrome Profiles' in the profile menu.\",\n \"In the list of profiles, locate the profile whose name you want to change.\",\n \"Hover over the desired profile and click the three-dot menu icon on that profile card.\",\n \"Select 'Edit' from the dropdown menu.\",\n \"In the Edit Profile dialog, click inside the name field.\",\n \"Delete the current name and type your new desired username.\",\n \"Click 'Save' to confirm the changes.\",\n \"Verify that the profile name is updated in the profile list and in the top-right corner of Chrome.\"\n ]\n}\n```\n\nSave each help document in a `json` file of your target folder.\n\n## Step 2: Place Help Documents in the AppAgent Directory\n\nOnce you have prepared all help documents and their metadata, place them into a folder. Sub-folders for the help documents are allowed, but ensure that each help document and its corresponding metadata are placed in the same directory.\n\n## Step 3: Create a Help Document Indexer\n\nAfter organizing your documents in a folder named `path_of_the_docs`, you can create an offline indexer to support RAG for UFO. Follow these steps:\n\n```bash\n# Assume you are in the cloned UFO folder\npython -m learner --app --docs \n```\n\n- Replace `` with the **Exact Process Name** of the application, such as `WINWORD.EXE` for Microsoft Word or `POWERPNT.EXE` for PowerPoint. \n- Replace `` with the full path to the folder containing all your documents.\n\nThis command will create an offline indexer for all documents in the `path_of_the_docs` folder using Faiss and embedding with sentence transformer (additional embeddings will be supported soon). By default, the created index will be placed [here](https://github.com/microsoft/UFO/tree/main/vectordb/docs).\n\n!!! note \"Application Name Requirement\"\n Ensure the `app_name` is accurately defined, as it is used to match the offline indexer in online RAG.\n\n## How to Use Help Documents to Enhance the AppAgent?\n\nAfter creating the offline indexer, refer to the [Learning from Help Documents](../../ufo2/core_features/knowledge_substrate/learning_from_help_document.md) section for guidance on how to use the help documents to enhance the `AppAgent`.\n\n## Related Documentation\n\n- [Overview: Enhancing AppAgent Capabilities](./overview.md) - Learn about all enhancement approaches\n- [User Demonstrations Provision](./demonstration_provision.md) - Teach through examples\n- [Wrapping App-Native API](./warpping_app_native_api.md) - Create efficient MCP action servers\n- [Knowledge Substrate Overview](../../ufo2/core_features/knowledge_substrate/overview.md) - Understanding the RAG architecture"
},
{
"path": "documents/docs/tutorials/creating_app_agent/overview.md",
"content": "# Enhancing AppAgent Capabilities\n\nUFO² provides a flexible framework for application developers and users to enhance `AppAgent` capabilities for specific applications. AppAgent enhancement is about **augmenting** the existing AppAgent's capabilities through:\n\n- **Knowledge** (help documents, demonstrations) to guide decision-making\n- **Native API tools** (via MCP servers) for efficient automation\n- **Application-specific context** for better understanding\n\n## Enhancement Components\n\nThe `AppAgent` can be enhanced through three complementary approaches:\n\n| Component | Description | Tutorial | Implementation Guide |\n| --- | --- | --- | --- |\n| **[Help Documents](./help_document_provision.md)** | Provide application-specific guidance and instructions to help the agent understand tasks and workflows | [Provision Guide](./help_document_provision.md) | [Learning from Help Documents](../../ufo2/core_features/knowledge_substrate/learning_from_help_document.md) |\n| **[User Demonstrations](./demonstration_provision.md)** | Supply recorded user interactions to teach the agent how to perform specific tasks through examples | [Provision Guide](./demonstration_provision.md) | [Learning from Demonstrations](../../ufo2/core_features/knowledge_substrate/learning_from_demonstration.md) |\n| **[Native API Tools](./warpping_app_native_api.md)** | Create custom MCP action servers that wrap application COM APIs or other native interfaces for efficient automation | [Wrapping Guide](./warpping_app_native_api.md) | [Creating MCP Servers](../creating_mcp_servers.md) |\n\n## Enhancement Workflow\n\n```mermaid\ngraph TB\n Enhancement[AppAgent Enhancement Workflow]\n \n Enhancement --> KnowledgeLayer[Knowledge Layer RAG-based]\n Enhancement --> ToolLayer[Tool Layer MCP Servers]\n \n KnowledgeLayer --> HelpDocs[Help Documents]\n KnowledgeLayer --> DemoTraj[User Demonstrations]\n \n ToolLayer --> UITools[UI Automation Tools]\n ToolLayer --> APITools[Native API Tools]\n \n HelpDocs --> EnhancedAgent[Enhanced AppAgent]\n DemoTraj --> EnhancedAgent\n UITools --> EnhancedAgent\n APITools --> EnhancedAgent\n \n style Enhancement fill:#e1f5ff,stroke:#01579b,stroke-width:3px\n style KnowledgeLayer fill:#fff3e0,stroke:#e65100,stroke-width:2px\n style ToolLayer fill:#f3e5f5,stroke:#4a148c,stroke-width:2px\n style HelpDocs fill:#fffde7,stroke:#f57f17,stroke-width:2px\n style DemoTraj fill:#fffde7,stroke:#f57f17,stroke-width:2px\n style UITools fill:#fce4ec,stroke:#880e4f,stroke-width:2px\n style APITools fill:#fce4ec,stroke:#880e4f,stroke-width:2px\n style EnhancedAgent fill:#e8f5e9,stroke:#1b5e20,stroke-width:3px\n```\n\n## When to Use Each Component?\n\n### Help Documents\n**Use when:**\n- You have official documentation, tutorials, or guides for your application\n- Tasks require domain-specific knowledge or procedures\n- You want the agent to understand application concepts and terminology\n\n**Example:** Providing Excel formula documentation to help the agent use advanced Excel functions correctly.\n\n### User Demonstrations\n**Use when:**\n- You can demonstrate the task yourself\n- The task involves a specific sequence of UI interactions\n- Visual/procedural knowledge is easier to show than describe\n\n**Example:** Recording how to create a pivot table in Excel to teach the agent the exact steps.\n\n### Native API Tools\n**Use when:**\n- Your application exposes COM APIs, REST APIs, or other programmable interfaces\n- GUI automation is slow or unreliable for certain operations\n- You need deterministic, high-performance automation\n\n**Example:** Creating an MCP server that wraps Excel's COM API for inserting tables, formatting cells, etc.\n\n## Enhancement Strategy\n\n!!!tip \"Hybrid Approach for Best Results\"\n Combine all three components for maximum effectiveness:\n \n 1. **Knowledge Foundation**: Provide help documents for conceptual understanding\n 2. **Procedural Learning**: Add demonstrations for complex workflows\n 3. **Efficient Execution**: Implement native API tools for performance-critical operations\n \n The AppAgent will:\n - Use knowledge to **understand** what to do\n - Reference demonstrations to **learn** how to do it\n - Leverage API tools when available for **efficient** execution\n - Fall back to UI automation when needed\n\n## Getting Started\n\nFollow the tutorials in order to enhance your AppAgent:\n\n1. **[Provide Help Documents](./help_document_provision.md)** - Start with knowledge\n2. **[Add User Demonstrations](./demonstration_provision.md)** - Teach by example\n3. **[Wrap Native APIs](./warpping_app_native_api.md)** - Enable efficient automation\n\n## Related Documentation\n\n- [AppAgent Overview](../../ufo2/app_agent/overview.md) - Understanding AppAgent architecture\n- [Knowledge Substrate](../../ufo2/core_features/knowledge_substrate/overview.md) - How knowledge enhancement works\n- [Creating MCP Servers](../creating_mcp_servers.md) - Building custom automation tools\n- [MCP Configuration](../../mcp/configuration.md) - Registering MCP servers with AppAgent\n- [Hybrid GUI–API Actions](../../ufo2/core_features/hybrid_actions.md) - Understanding dual-mode automation\n"
},
{
"path": "documents/docs/tutorials/creating_app_agent/warpping_app_native_api.md",
"content": "# Wrapping Application Native APIs as MCP Action Servers\n\nUFO² uses **MCP (Model Context Protocol) servers** to expose application native APIs to the AppAgent. This document shows you how to create custom MCP action servers that wrap your application's COM APIs, REST APIs, or other programmable interfaces.\n\n## Overview\n\nWhile AppAgent can automate applications through UI controls, providing **native API tools** via MCP servers offers significant advantages:\n\n| Automation Method | Speed | Reliability | Use Case |\n|-------------------|-------|-------------|----------|\n| **UI Automation** | Slower | Prone to UI changes | Visual elements, dialogs, menus |\n| **Native API** | ~10x faster | Deterministic | Data manipulation, batch operations |\n\n!!! tip \"Hybrid Automation\"\n AppAgent combines both approaches - the LLM intelligently selects **GUI tools** (from UIExecutor) or **API tools** (from your custom MCP server) based on the task requirements.\n\n## Prerequisites\n\nBefore creating a native API MCP server:\n\n1. **Understand MCP Servers**: Read [Creating MCP Servers Tutorial](../creating_mcp_servers.md)\n2. **Know Your API**: Familiarize yourself with your application's COM API, REST API, or SDK\n3. **Review Examples**: Study existing servers in `ufo/client/mcp/local_servers/`\n\n## Step-by-Step Guide\n\n### Step 1: Create Your MCP Server File\n\nCreate a new Python file in `ufo/client/mcp/local_servers/` for your application's MCP server:\n\n```python\n# File: ufo/client/mcp/local_servers/your_app_executor.py\n\nfrom typing import Annotated, Optional\nfrom fastmcp import FastMCP\nfrom pydantic import Field\nfrom ufo.client.mcp.mcp_registry import MCPRegistry\nfrom ufo.automator.puppeteer import AppPuppeteer\nfrom ufo.automator.action_execution import ActionExecutor\nfrom ufo.agents.processors.schemas.actions import ActionCommandInfo\n\n\n@MCPRegistry.register_factory_decorator(\"YourAppExecutor\")\ndef create_your_app_executor(process_name: str, *args, **kwargs) -> FastMCP:\n \"\"\"\n Create MCP server for YourApp COM API automation.\n \n :param process_name: Process name for UI automation context.\n :return: FastMCP instance with YourApp tools.\n \"\"\"\n \n # Initialize puppeteer for UI context\n puppeteer = AppPuppeteer(\n process_name=process_name,\n app_root_name=\"YOURAPP.EXE\", # Your app's executable name\n )\n \n # Create COM API receiver\n puppeteer.receiver_manager.create_api_receiver(\n app_root_name=\"YOURAPP.EXE\",\n process_name=process_name,\n )\n \n executor = ActionExecutor()\n \n def _execute_action(action: ActionCommandInfo) -> dict:\n \"\"\"Execute action via puppeteer.\"\"\"\n return executor.execute(action, puppeteer, control_dict={})\n \n # Create FastMCP instance\n mcp = FastMCP(\"YourApp COM Executor MCP Server\")\n \n # Define tools below...\n \n return mcp\n```\n\n### Step 2: Define Tool Methods with @mcp.tool()\n\nAdd tool methods to your MCP server using the `@mcp.tool()` decorator. Each tool wraps a native API call:\n\n```python\n @mcp.tool()\n def insert_data_table(\n data: Annotated[\n list[list[str]], \n Field(description=\"2D array of table data. Example: [['Name', 'Age'], ['Alice', '25']]\")\n ],\n start_row: Annotated[\n int, \n Field(description=\"Starting row index (1-based).\")\n ] = 1,\n start_col: Annotated[\n int, \n Field(description=\"Starting column index (1-based).\")\n ] = 1,\n ) -> Annotated[str, Field(description=\"Result message.\")]:\n \"\"\"\n Insert a data table into the application at the specified position.\n Use this for bulk data insertion instead of manual cell-by-cell input.\n \n Example usage:\n - Insert CSV data: insert_data_table(data=csv_data, start_row=1, start_col=1)\n - Add header and rows: insert_data_table(data=[['ID', 'Name'], ['1', 'Alice'], ['2', 'Bob']])\n \"\"\"\n action = ActionCommandInfo(\n function=\"insert_table\",\n arguments={\n \"data\": data,\n \"start_row\": start_row,\n \"start_col\": start_col,\n },\n )\n return _execute_action(action)\n \n @mcp.tool()\n def format_range(\n start_cell: Annotated[\n str, \n Field(description=\"Starting cell address (e.g., 'A1').\")\n ],\n end_cell: Annotated[\n str, \n Field(description=\"Ending cell address (e.g., 'B10').\")\n ],\n font_bold: Annotated[\n Optional[bool], \n Field(description=\"Make font bold?\")\n ] = None,\n font_size: Annotated[\n Optional[int], \n Field(description=\"Font size in points.\")\n ] = None,\n background_color: Annotated[\n Optional[str], \n Field(description=\"Background color (hex code like '#FF0000' for red).\")\n ] = None,\n ) -> Annotated[str, Field(description=\"Formatting result.\")]:\n \"\"\"\n Apply formatting to a cell range in the application.\n Much faster than clicking format buttons multiple times.\n \n Example:\n - Bold header: format_range(start_cell='A1', end_cell='E1', font_bold=True)\n - Highlight cells: format_range(start_cell='A2', end_cell='A10', background_color='#FFFF00')\n \"\"\"\n action = ActionCommandInfo(\n function=\"format_cells\",\n arguments={\n \"start_cell\": start_cell,\n \"end_cell\": end_cell,\n \"font_bold\": font_bold,\n \"font_size\": font_size,\n \"background_color\": background_color,\n },\n )\n return _execute_action(action)\n \n @mcp.tool()\n def save_as_pdf(\n output_path: Annotated[\n str, \n Field(description=\"Full path for the PDF file (e.g., 'C:/Users/Documents/report.pdf').\")\n ],\n ) -> Annotated[str, Field(description=\"Save result message.\")]:\n \"\"\"\n Export the current document as a PDF file.\n One-click operation - much faster than File > Save As > PDF > Navigate > Save.\n \n Example: save_as_pdf(output_path='C:/Reports/monthly_report.pdf')\n \"\"\"\n action = ActionCommandInfo(\n function=\"save_as\",\n arguments={\n \"file_path\": output_path,\n \"file_format\": \"pdf\",\n },\n )\n return _execute_action(action)\n```\n\n!!!tip \"Tool Design Best Practices\"\n - **Clear docstrings**: Explain what the tool does, when to use it, and provide examples\n - **Descriptive parameters**: Use `Annotated` with `Field(description=...)`for all parameters\n - **Error handling**: Return descriptive error messages when operations fail\n - **Comprehensive coverage**: Wrap common operations that benefit from API speed\n\n### Step 3: Implement the Underlying API Receiver\n\nThe receiver class executes the actual COM API calls. Create it in `ufo/automator/app_apis/`:\n\n```python\n# File: ufo/automator/app_apis/your_app/your_app_client.py\n\nimport win32com.client\nfrom typing import Dict, Any, List, Optional\nfrom ufo.automator.app_apis.basic import WinCOMReceiverBasic\nfrom ufo.automator.basic import CommandBasic\n\n\nclass YourAppCOMReceiver(WinCOMReceiverBasic):\n \"\"\"\n COM API receiver for YourApp automation.\n \"\"\"\n \n _command_registry: Dict[str, type[CommandBasic]] = {}\n \n def __init__(self, app_root_name: str, process_name: str, clsid: str) -> None:\n \"\"\"\n Initialize the YourApp COM client.\n :param app_root_name: Application root name.\n :param process_name: Process name.\n :param clsid: COM object CLSID.\n \"\"\"\n super().__init__(app_root_name, process_name, clsid)\n \n def insert_table_data(\n self, \n data: List[List[str]], \n start_row: int = 1, \n start_col: int = 1\n ) -> str:\n \"\"\"\n Insert table data using COM API.\n :param data: 2D array of table data.\n :param start_row: Starting row (1-based).\n :param start_col: Starting column (1-based).\n :return: Result message.\n \"\"\"\n try:\n # Access the active document/workbook via COM\n doc = self.com_object.ActiveDocument # Or ActiveWorkbook for Excel\n \n # Insert data row by row\n for i, row in enumerate(data):\n for j, cell_value in enumerate(row):\n # Example: Set cell value\n cell = doc.Tables(1).Cell(start_row + i, start_col + j)\n cell.Range.Text = str(cell_value)\n \n return f\"Successfully inserted {len(data)} rows of data\"\n except Exception as e:\n return f\"Error inserting table: {str(e)}\"\n \n def format_cells(\n self,\n start_cell: str,\n end_cell: str,\n font_bold: Optional[bool] = None,\n font_size: Optional[int] = None,\n background_color: Optional[str] = None,\n ) -> str:\n \"\"\"\n Format cell range using COM API.\n \"\"\"\n try:\n doc = self.com_object.ActiveDocument\n range_obj = doc.Range(start_cell, end_cell)\n \n if font_bold is not None:\n range_obj.Font.Bold = font_bold\n if font_size is not None:\n range_obj.Font.Size = font_size\n if background_color is not None:\n # Convert hex to RGB and apply\n range_obj.Shading.BackgroundPatternColor = self._hex_to_rgb(background_color)\n \n return f\"Successfully formatted range {start_cell}:{end_cell}\"\n except Exception as e:\n return f\"Error formatting cells: {str(e)}\"\n \n def save_document_as(self, file_path: str, file_format: str) -> str:\n \"\"\"\n Save document in specified format.\n \"\"\"\n try:\n doc = self.com_object.ActiveDocument\n \n # Map format string to COM constant\n format_map = {\n \"pdf\": 17, # wdFormatPDF\n \"docx\": 16, # wdFormatXMLDocument\n # Add more formats as needed\n }\n \n format_code = format_map.get(file_format.lower(), 16)\n doc.SaveAs2(file_path, FileFormat=format_code)\n \n return f\"Successfully saved document to {file_path}\"\n except Exception as e:\n return f\"Error saving document: {str(e)}\"\n \n @staticmethod\n def _hex_to_rgb(hex_color: str) -> int:\n \"\"\"Convert hex color to RGB integer for COM.\"\"\"\n hex_color = hex_color.lstrip('#')\n r, g, b = tuple(int(hex_color[i:i+2], 16) for i in (0, 2, 4))\n return r + (g << 8) + (b << 16)\n```\n\n### Step 4: Create Command Classes\n\nDefine command classes that bridge the MCP tools to the receiver methods:\n\n```python\n# In the same file: ufo/automator/app_apis/your_app/your_app_client.py\n\n@YourAppCOMReceiver.register\nclass InsertTableCommand(CommandBasic):\n \"\"\"Command to insert table data.\"\"\"\n \n def execute(self) -> Dict[str, Any]:\n \"\"\"Execute table insertion.\"\"\"\n return self.receiver.insert_table_data(\n data=self.params.get(\"data\", []),\n start_row=self.params.get(\"start_row\", 1),\n start_col=self.params.get(\"start_col\", 1),\n )\n\n\n@YourAppCOMReceiver.register\nclass FormatCellsCommand(CommandBasic):\n \"\"\"Command to format cell range.\"\"\"\n \n def execute(self) -> Dict[str, Any]:\n \"\"\"Execute cell formatting.\"\"\"\n return self.receiver.format_cells(\n start_cell=self.params.get(\"start_cell\"),\n end_cell=self.params.get(\"end_cell\"),\n font_bold=self.params.get(\"font_bold\"),\n font_size=self.params.get(\"font_size\"),\n background_color=self.params.get(\"background_color\"),\n )\n\n\n@YourAppCOMReceiver.register\nclass SaveAsCommand(CommandBasic):\n \"\"\"Command to save document.\"\"\"\n \n def execute(self) -> Dict[str, Any]:\n \"\"\"Execute document save.\"\"\"\n return self.receiver.save_document_as(\n file_path=self.params.get(\"file_path\"),\n file_format=self.params.get(\"file_format\", \"pdf\"),\n )\n```\n\n!!!note \"Command Registration\"\n Use `@YourAppCOMReceiver.register` decorator to register each command class with the receiver.\n\n### Step 5: Register Your Receiver in the Factory\n\nAdd your receiver to the COM receiver factory in `ufo/automator/app_apis/factory.py`:\n\n```python\ndef __com_client_mapper(self, app_root_name: str) -> Type[WinCOMReceiverBasic]:\n \"\"\"Map application to its COM receiver class.\"\"\"\n mapping = {\n \"WINWORD.EXE\": WordWinCOMReceiver,\n \"EXCEL.EXE\": ExcelWinCOMReceiver,\n \"POWERPNT.EXE\": PowerPointWinCOMReceiver,\n \"YOURAPP.EXE\": YourAppCOMReceiver, # Add your app here\n }\n return mapping.get(app_root_name)\n\ndef __app_root_mappping(self, app_root_name: str) -> Optional[str]:\n \"\"\"Map application to its COM CLSID.\"\"\"\n mapping = {\n \"WINWORD.EXE\": \"Word.Application\",\n \"EXCEL.EXE\": \"Excel.Application\",\n \"POWERPNT.EXE\": \"PowerPoint.Application\",\n \"YOURAPP.EXE\": \"YourApp.Application\", # Add your CLSID here\n }\n return mapping.get(app_root_name)\n```\n\n### Step 6: Register the MCP Server in mcp.yaml\n\nConfigure the MCP server for your application in `config/ufo/mcp.yaml`:\n\n```yaml\nAppAgent:\n YOURAPP.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n reset: false\n action:\n - namespace: AppUIExecutor # Generic UI automation\n type: local\n reset: false\n - namespace: YourAppExecutor # Your custom COM API tools\n type: local\n reset: true # Reset COM state when switching documents\n - namespace: CommandLineExecutor # Shell commands\n type: local\n reset: false\n```\n\n!!!tip \"Why `reset: true`?\"\n Set `reset: true` for COM-based MCP servers to prevent state leakage when switching between documents or application instances.\n\n### Step 7: Test Your MCP Server\n\nTest your server in isolation before integration:\n\n```python\n# File: test_your_app_server.py\n\nimport asyncio\nfrom fastmcp.client import Client\nfrom ufo.client.mcp.local_servers.your_app_executor import create_your_app_executor\n\n\nasync def test_server():\n \"\"\"Test YourApp MCP server.\"\"\"\n process_name = \"your_app_process\"\n server = create_your_app_executor(process_name)\n \n async with Client(server) as client:\n # List available tools\n tools = await client.list_tools()\n print(f\"Available tools: {[t.name for t in tools]}\")\n \n # Test insert_data_table\n result = await client.call_tool(\n \"insert_data_table\",\n arguments={\n \"data\": [[\"Name\", \"Age\"], [\"Alice\", \"25\"], [\"Bob\", \"30\"]],\n \"start_row\": 1,\n \"start_col\": 1,\n }\n )\n print(f\"Insert result: {result.data}\")\n \n # Test format_range\n result = await client.call_tool(\n \"format_range\",\n arguments={\n \"start_cell\": \"A1\",\n \"end_cell\": \"B1\",\n \"font_bold\": True,\n \"font_size\": 14,\n }\n )\n print(f\"Format result: {result.data}\")\n\n\nif __name__ == \"__main__\":\n asyncio.run(test_server())\n```\n\n## Complete Example: Excel COM Executor\n\nSee the complete implementation in UFO²'s codebase:\n\n- **MCP Server**: `ufo/client/mcp/local_servers/excel_wincom_mcp_server.py`\n- **COM Receiver**: `ufo/automator/app_apis/excel/excel_client.py`\n- **Configuration**: `config/ufo/mcp.yaml` (under `AppAgent.EXCEL.EXE`)\n\nKey features:\n- `insert_table`: Bulk data insertion\n- `format_cells`: Cell formatting (fonts, colors, borders)\n- `create_chart`: Chart generation\n- `apply_formula`: Formula application\n- `save_as`: Export to PDF/CSV\n\n## Legacy Approach: API Prompt Files (Deprecated)\n\n!!!warning \"Deprecated: API Prompt Files\"\n The old approach of creating `api.yaml` prompt files and configuring `APP_API_PROMPT_ADDRESS` is **deprecated**. The new MCP architecture provides:\n \n - ✅ **Better tool discovery**: Tools are automatically introspected from MCP servers\n - ✅ **Type safety**: Pydantic models ensure parameter validation\n - ✅ **Cleaner code**: No manual prompt file maintenance\n - ✅ **Better testing**: Direct server testing with FastMCP Client\n \n If you're migrating from the old system, see [Creating MCP Servers Tutorial](../creating_mcp_servers.md).\n\n## Best Practices\n\n### 1. Comprehensive Docstrings\n\n```python\n@mcp.tool()\ndef insert_data_table(...) -> ...:\n \"\"\"\n Insert a data table into the application at the specified position.\n Use this for bulk data insertion instead of manual cell-by-cell input.\n \n When to use:\n - Inserting CSV/Excel data\n - Creating tables from lists\n - Bulk data population\n \n Example usage:\n - Insert CSV data: insert_data_table(data=csv_data, start_row=1, start_col=1)\n - Add header and rows: insert_data_table(data=[['ID', 'Name'], ['1', 'Alice']])\n \"\"\"\n```\n\n### 2. Error Handling\n\n```python\ndef insert_table_data(self, data: List[List[str]], ...) -> str:\n \"\"\"Insert table data using COM API.\"\"\"\n try:\n # Validate input\n if not data or not data[0]:\n return \"Error: Empty data table provided\"\n \n # Execute COM operation\n doc = self.com_object.ActiveDocument\n # ... insert logic ...\n \n return f\"Successfully inserted {len(data)} rows\"\n except Exception as e:\n return f\"Error inserting table: {str(e)}\"\n```\n\n### 3. Parameter Validation\n\n```python\n@mcp.tool()\ndef format_range(\n start_cell: Annotated[\n str, \n Field(\n description=\"Starting cell address (e.g., 'A1'). Must be valid Excel notation.\",\n pattern=r\"^[A-Z]+[0-9]+$\" # Regex validation\n )\n ],\n ...\n) -> ...:\n \"\"\"Format cell range.\"\"\"\n```\n\n### 4. Fallback to UI Automation\n\nDesign your API tools to complement (not replace) UI automation:\n\n```python\n@mcp.tool()\ndef apply_table_style(style_name: str) -> str:\n \"\"\"\n Apply a predefined table style.\n \n Note: For custom styling, use format_range() or UI automation\n via AppUIExecutor::click_input() on the Design tab.\n \"\"\"\n```\n\n## Troubleshooting\n\n### Issue: COM Object Not Found\n\n**Symptom**: `pywintypes.com_error: (-2147221005, 'Invalid class string', None, None)`\n\n**Solution**:\n1. Verify the CLSID is correct for your application\n2. Ensure the application is installed and registered\n3. Check if the application supports COM automation\n\n### Issue: Permission Denied\n\n**Symptom**: `com_error: (-2147352567, 'Exception occurred.', ...)`\n\n**Solution**:\n- Run UFO² with administrator privileges\n- Check application security settings\n- Verify COM permissions in `dcomcnfg`\n\n### Issue: Tools Not Appearing in LLM Prompt\n\n**Symptom**: AppAgent doesn't use your API tools\n\n**Solution**:\n1. Verify MCP server is registered in `mcp.yaml`\n2. Check namespace matches: `@MCPRegistry.register_factory_decorator(\"YourAppExecutor\")`\n3. Ensure server is under `action:` (not `data_collection:`)\n4. Test server independently with FastMCP Client\n\n## Related Documentation\n\n**Core Tutorials:**\n\n- **[Creating MCP Servers Tutorial](../creating_mcp_servers.md)** - Complete MCP server development guide\n- [Overview: Enhancing AppAgent Capabilities](./overview.md) - Learn about all enhancement approaches\n- [Help Document Provision](./help_document_provision.md) - Provide knowledge through documentation\n- [User Demonstrations Provision](./demonstration_provision.md) - Teach through examples\n\n**MCP Documentation:**\n\n- [MCP Configuration](../../mcp/configuration.md) - Registering MCP servers\n- [MCP Overview](../../mcp/overview.md) - Understanding MCP architecture\n- [WordCOMExecutor](../../mcp/servers/word_com_executor.md) - Reference implementation\n- [ExcelCOMExecutor](../../mcp/servers/excel_com_executor.md) - Reference implementation\n\n**Advanced Features:**\n\n- [Hybrid GUI–API Actions](../../ufo2/core_features/hybrid_actions.md) - How AppAgent chooses tools\n- [Knowledge Substrate Overview](../../ufo2/core_features/knowledge_substrate/overview.md) - Understanding the RAG architecture\n\n---\n\nBy following this guide, you've successfully wrapped your application's native API as an MCP action server, enabling the AppAgent to perform fast, reliable automation through direct API calls!\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/client_setup.md",
"content": "# Part 3: Client Setup\n\nThis tutorial teaches you how to set up the **UFO device client** that runs on the target device, manages MCP servers, and communicates with the agent server via WebSocket. We'll use the existing client implementation as reference.\n\n---\n\n## Table of Contents\n\n1. [Client Architecture Overview](#client-architecture-overview)\n2. [Client Components](#client-components)\n3. [UFO Client Implementation](#ufo-client-implementation)\n4. [WebSocket Client](#websocket-client)\n5. [MCP Server Manager](#mcp-server-manager)\n6. [Platform Detection](#platform-detection)\n7. [Configuration and Deployment](#configuration-and-deployment)\n8. [Testing Your Client](#testing-your-client)\n\n---\n\n## Client Architecture Overview\n\n### Client Role in Device Agent System\n\n```mermaid\ngraph TB\n subgraph \"Agent Server (Orchestrator)\"\n Agent[Device Agent]\n Dispatcher[Command Dispatcher]\n end\n \n subgraph \"Network Layer\"\n WS[WebSocket AIP Protocol]\n end\n \n subgraph \"Device Client (Your Implementation)\"\n Main[client.py Entry Point]\n UFOClient[UFOClient Core Logic]\n WSClient[WebSocketClient Communication]\n \n subgraph \"Managers\"\n MCPMgr[MCP Server Manager]\n CompMgr[Computer Manager]\n CmdRouter[Command Router]\n end\n \n Main --> UFOClient\n Main --> WSClient\n UFOClient --> MCPMgr\n UFOClient --> CompMgr\n UFOClient --> CmdRouter\n WSClient --> UFOClient\n end\n \n subgraph \"MCP Servers\"\n MCP1[Mobile MCP Server]\n MCP2[Linux MCP Server]\n MCPN[...]\n end\n \n Agent --> Dispatcher\n Dispatcher -->|Commands| WS\n WS -->|Commands| WSClient\n WSClient -->|Results| WS\n WS -->|Results| Agent\n \n UFOClient --> MCPMgr\n MCPMgr --> MCP1 & MCP2 & MCPN\n \n style Main fill:#c8e6c9\n style UFOClient fill:#e1f5ff\n style WSClient fill:#fff3e0\n style MCPMgr fill:#f3e5f5\n```\n\n**Client Responsibilities**:\n\n| Component | Responsibility | Example |\n|-----------|----------------|---------|\n| **Entry Point** | Parse args, initialize services | `client.py main()` |\n| **UFO Client** | Execute commands, route actions | `UFOClient.execute_actions()` |\n| **WebSocket Client** | Bidirectional communication | `UFOWebSocketClient.handle_messages()` |\n| **MCP Server Manager** | Start/stop MCP servers | `MCPServerManager.start()` |\n| **Computer Manager** | Manage device computers | `ComputerManager.get_computer()` |\n| **Command Router** | Route commands to MCP tools | `CommandRouter.execute()` |\n\n---\n\n## Client Components\n\n### Component Hierarchy\n\n```mermaid\ngraph TB\n subgraph \"Client Entry Point\"\n Main[client.py main function]\n end\n \n subgraph \"Core Components\"\n UFO[UFOClient]\n WS[UFOWebSocketClient]\n end\n \n subgraph \"Management Layer\"\n MCP[MCPServerManager]\n Comp[ComputerManager]\n Router[CommandRouter]\n end\n \n subgraph \"Protocol Layer\"\n AIP[AIP Protocol]\n Reg[RegistrationProtocol]\n Heart[HeartbeatProtocol]\n Task[TaskExecutionProtocol]\n end\n \n subgraph \"MCP Integration\"\n HTTP[HTTPMCPServer]\n Local[LocalMCPServer]\n Stdio[StdioMCPServer]\n end\n \n Main --> UFO\n Main --> WS\n UFO --> MCP\n UFO --> Comp\n UFO --> Router\n WS --> Reg & Heart & Task\n MCP --> HTTP & Local & Stdio\n \n style Main fill:#c8e6c9\n style UFO fill:#e1f5ff\n style WS fill:#fff3e0\n style MCP fill:#f3e5f5\n```\n\n---\n\n## UFO Client Implementation\n\n### File Location\n\n**Path**: `ufo/client/ufo_client.py`\n\n### Core UFO Client Class\n\n```python\n# ufo/client/ufo_client.py\n\nimport asyncio\nimport logging\nfrom typing import List, Optional\n\nfrom ufo.client.computer import CommandRouter, ComputerManager\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\nfrom aip.messages import Command, Result, ServerMessage\n\n\nclass UFOClient:\n \"\"\"\n Client for interacting with the UFO web service.\n Executes commands from agent server and returns results.\n \"\"\"\n\n def __init__(\n self,\n mcp_server_manager: MCPServerManager,\n computer_manager: ComputerManager,\n client_id: Optional[str] = None,\n platform: Optional[str] = None,\n ):\n \"\"\"\n Initialize the UFO client.\n \n :param mcp_server_manager: Manages MCP servers\n :param computer_manager: Manages computer instances\n :param client_id: Unique client identifier\n :param platform: Platform type ('windows', 'linux', 'android', 'ios')\n \"\"\"\n self.mcp_server_manager = mcp_server_manager\n self.computer_manager = computer_manager\n self.command_router = CommandRouter(\n computer_manager=self.computer_manager,\n )\n self.logger = logging.getLogger(__name__)\n self.task_lock = asyncio.Lock() # Thread safety\n\n self.client_id = client_id or \"client_001\"\n self.platform = platform\n\n # Session state\n self._agent_name: Optional[str] = None\n self._process_name: Optional[str] = None\n self._root_name: Optional[str] = None\n self._session_id: Optional[str] = None\n\n async def execute_step(self, response: ServerMessage) -> List[Result]:\n \"\"\"\n Execute a single step from the agent server.\n \n :param response: ServerMessage with commands to execute\n :return: List of execution results\n \"\"\"\n # Update agent context\n self.agent_name = response.agent_name\n self.process_name = response.process_name\n self.root_name = response.root_name\n\n # Execute actions and collect results\n action_results = await self.execute_actions(response.actions)\n return action_results\n\n async def execute_actions(\n self, commands: Optional[List[Command]]\n ) -> List[Result]:\n \"\"\"\n Execute commands via MCP servers.\n \n :param commands: List of commands to execute\n :return: List of execution results\n \"\"\"\n action_results = []\n\n if commands:\n self.logger.info(f\"Executing {len(commands)} commands\")\n \n # Route commands to appropriate MCP servers\n action_results = await self.command_router.execute(\n agent_name=self.agent_name,\n process_name=self.process_name,\n root_name=self.root_name,\n commands=commands,\n )\n\n return action_results\n\n # Property setters/getters for agent context\n @property\n def session_id(self) -> Optional[str]:\n \"\"\"Get current session ID.\"\"\"\n return self._session_id\n\n @session_id.setter\n def session_id(self, value: Optional[str]):\n \"\"\"Set session ID.\"\"\"\n if value is not None and not isinstance(value, str):\n raise ValueError(\"Session ID must be a string or None.\")\n self._session_id = value\n self.logger.info(f\"Session ID set to: {value}\")\n\n @property\n def agent_name(self) -> Optional[str]:\n \"\"\"Get agent name.\"\"\"\n return self._agent_name\n\n @agent_name.setter\n def agent_name(self, value: Optional[str]):\n \"\"\"Set agent name.\"\"\"\n self._agent_name = value\n self.logger.info(f\"Agent name: {value}\")\n\n @property\n def process_name(self) -> Optional[str]:\n \"\"\"Get process name.\"\"\"\n return self._process_name\n\n @process_name.setter\n def process_name(self, value: Optional[str]):\n \"\"\"Set process name.\"\"\"\n self._process_name = value\n\n @property\n def root_name(self) -> Optional[str]:\n \"\"\"Get root name.\"\"\"\n return self._root_name\n\n @root_name.setter\n def root_name(self, value: Optional[str]):\n \"\"\"Set root name.\"\"\"\n self._root_name = value\n```\n\n### Key Client Methods\n\n| Method | Purpose | Called By |\n|--------|---------|-----------|\n| `execute_step()` | Process one agent step | WebSocket client |\n| `execute_actions()` | Execute command list | `execute_step()` |\n| Property setters | Update agent context | WebSocket client |\n\n---\n\n## WebSocket Client\n\n### File Location\n\n**Path**: `ufo/client/websocket.py`\n\n### WebSocket Client Implementation\n\n```python\n# ufo/client/websocket.py (simplified)\n\nimport asyncio\nimport logging\nimport websockets\nfrom typing import TYPE_CHECKING, Optional\n\nfrom aip.protocol.registration import RegistrationProtocol\nfrom aip.protocol.heartbeat import HeartbeatProtocol\nfrom aip.protocol.task_execution import TaskExecutionProtocol\nfrom aip.transport.websocket import WebSocketTransport\nfrom aip.messages import ServerMessage, ServerMessageType\n\nif TYPE_CHECKING:\n from ufo.client.ufo_client import UFOClient\n\n\nclass UFOWebSocketClient:\n \"\"\"\n WebSocket client for UFO device agents.\n Uses AIP (Agent Interaction Protocol) for structured communication.\n \"\"\"\n\n def __init__(\n self,\n ws_url: str,\n ufo_client: \"UFOClient\",\n max_retries: int = 3,\n timeout: float = 120,\n ):\n \"\"\"\n Initialize WebSocket client.\n \n :param ws_url: WebSocket server URL (e.g., ws://localhost:5010/ws)\n :param ufo_client: UFOClient instance\n :param max_retries: Maximum connection retries\n :param timeout: Connection timeout in seconds\n \"\"\"\n self.ws_url = ws_url\n self.ufo_client = ufo_client\n self.max_retries = max_retries\n self.retry_count = 0\n self.timeout = timeout\n self.logger = logging.getLogger(__name__)\n \n self.connected_event = asyncio.Event()\n self._ws: Optional[websockets.WebSocketClientProtocol] = None\n\n # AIP protocol instances\n self.transport: Optional[WebSocketTransport] = None\n self.registration_protocol: Optional[RegistrationProtocol] = None\n self.heartbeat_protocol: Optional[HeartbeatProtocol] = None\n self.task_protocol: Optional[TaskExecutionProtocol] = None\n\n async def connect_and_listen(self):\n \"\"\"\n Connect to server and listen for messages.\n Automatically retries on failure.\n \"\"\"\n while True:\n try:\n # Check retry limit\n if self.retry_count >= self.max_retries:\n self.logger.error(f\"Max retries ({self.max_retries}) reached\")\n break\n\n self.logger.info(\n f\"Connecting to {self.ws_url} \"\n f\"(attempt {self.retry_count + 1}/{self.max_retries})\"\n )\n\n # Reset connection state\n self.connected_event.clear()\n self._ws = None\n\n # Establish WebSocket connection\n async with websockets.connect(\n self.ws_url,\n ping_interval=20,\n ping_timeout=180,\n close_timeout=10,\n max_size=100 * 1024 * 1024, # 100MB max message size\n ) as ws:\n self._ws = ws\n\n # Initialize AIP protocols\n self.transport = WebSocketTransport(ws)\n self.registration_protocol = RegistrationProtocol(self.transport)\n self.heartbeat_protocol = HeartbeatProtocol(self.transport)\n self.task_protocol = TaskExecutionProtocol(self.transport)\n\n # Register with server\n await self.register_client()\n \n # Reset retry count on success\n self.retry_count = 0\n \n # Start message handling loop\n await self.handle_messages()\n\n except (\n websockets.ConnectionClosed,\n websockets.ConnectionClosedError,\n asyncio.TimeoutError,\n ) as e:\n self.logger.warning(f\"Connection closed: {e}. Retrying...\")\n self.connected_event.clear()\n self.retry_count += 1\n await self._maybe_retry()\n\n except Exception as e:\n self.logger.error(f\"Unexpected error: {e}\", exc_info=True)\n self.connected_event.clear()\n self.retry_count += 1\n await self._maybe_retry()\n\n async def register_client(self):\n \"\"\"\n Register client with server.\n Sends client ID and device system information.\n \"\"\"\n from ufo.client.device_info_provider import DeviceInfoProvider\n\n # Collect device system information\n system_info = DeviceInfoProvider.collect_system_info(\n self.ufo_client.client_id,\n custom_metadata=None,\n )\n\n # Prepare metadata\n metadata = {\n \"system_info\": system_info,\n \"platform\": self.ufo_client.platform,\n \"client_version\": \"3.0\",\n }\n\n # Send registration via AIP\n response = await self.registration_protocol.register(\n client_id=self.ufo_client.client_id,\n metadata=metadata,\n )\n\n if response.status == \"success\":\n self.logger.info(f\"✅ Client registered: {self.ufo_client.client_id}\")\n self.connected_event.set() # Signal connection ready\n else:\n raise ConnectionError(f\"Registration failed: {response.message}\")\n\n async def handle_messages(self):\n \"\"\"\n Handle incoming messages from server.\n Dispatches to appropriate protocol handlers.\n \"\"\"\n self.logger.info(\"Starting message handling loop\")\n\n while True:\n try:\n # Receive message via transport\n message = await self.transport.receive()\n\n if message is None:\n self.logger.warning(\"Received None message, closing\")\n break\n\n # Dispatch based on message type\n if message.type == ServerMessageType.TASK_REQUEST:\n await self._handle_task_request(message)\n \n elif message.type == ServerMessageType.HEARTBEAT:\n await self._handle_heartbeat(message)\n \n elif message.type == ServerMessageType.RESULT_ACK:\n await self._handle_result_ack(message)\n \n else:\n self.logger.warning(f\"Unknown message type: {message.type}\")\n\n except Exception as e:\n self.logger.error(f\"Error handling message: {e}\", exc_info=True)\n break\n\n async def _handle_task_request(self, message: ServerMessage):\n \"\"\"Handle task request from server.\"\"\"\n self.logger.info(f\"📨 Task request received: {message.task_id}\")\n\n # Execute task via UFO client\n results = await self.ufo_client.execute_step(message)\n\n # Send results back via AIP\n await self.task_protocol.send_result(\n task_id=message.task_id,\n results=results,\n )\n\n self.logger.info(f\"✅ Task completed: {message.task_id}\")\n\n async def _handle_heartbeat(self, message: ServerMessage):\n \"\"\"Handle heartbeat from server.\"\"\"\n await self.heartbeat_protocol.send_heartbeat_ack(\n timestamp=message.timestamp\n )\n\n async def _handle_result_ack(self, message: ServerMessage):\n \"\"\"Handle result acknowledgment from server.\"\"\"\n self.logger.info(f\"✅ Result acknowledged: {message.task_id}\")\n\n async def _maybe_retry(self):\n \"\"\"Wait before retrying connection.\"\"\"\n if self.retry_count < self.max_retries:\n wait_time = 2 ** self.retry_count # Exponential backoff\n self.logger.info(f\"Retrying in {wait_time}s...\")\n await asyncio.sleep(wait_time)\n\n async def start_task(self, request_text: str, task_name: Optional[str] = None):\n \"\"\"\n Initiate a task from client side (optional feature).\n \n :param request_text: Task description\n :param task_name: Optional task name\n \"\"\"\n await self.task_protocol.request_task(\n request_text=request_text,\n task_name=task_name or \"client_task\",\n )\n```\n\n### WebSocket Communication Flow\n\n```mermaid\nsequenceDiagram\n participant Client as UFOWebSocketClient\n participant Server as Agent Server\n participant UFO as UFOClient\n participant MCP as MCP Server\n\n Client->>Server: REGISTER (client_id, metadata)\n Server->>Client: REGISTER_ACK (success)\n \n Note over Client,Server: Connection Established\n \n Server->>Client: HEARTBEAT\n Client->>Server: HEARTBEAT_ACK\n \n Server->>Client: TASK_REQUEST (commands)\n Client->>UFO: execute_step(message)\n UFO->>MCP: execute(commands)\n MCP->>UFO: results\n UFO->>Client: results\n Client->>Server: TASK_RESULT (results)\n Server->>Client: RESULT_ACK\n \n Note over Client,Server: Continuous Loop\n```\n\n---\n\n## MCP Server Manager\n\n### Manager Architecture\n\n```mermaid\ngraph TB\n subgraph \"MCP Server Manager\"\n Mgr[MCPServerManager]\n \n subgraph \"Server Types\"\n HTTP[HTTPMCPServer Remote HTTP]\n Local[LocalMCPServer In-Memory]\n Stdio[StdioMCPServer Process]\n end\n \n Mgr --> HTTP & Local & Stdio\n end\n \n subgraph \"MCP Servers\"\n MCP1[Mobile MCP port 8020]\n MCP2[Linux MCP port 8010]\n MCP3[Custom MCP port 8030]\n end\n \n HTTP --> MCP1 & MCP2\n Local --> MCP3\n \n style Mgr fill:#c8e6c9\n style HTTP fill:#e1f5ff\n style MCP1 fill:#fff3e0\n```\n\n### MCP Server Manager Implementation\n\n```python\n# ufo/client/mcp/mcp_server_manager.py (simplified)\n\nfrom typing import Dict, Any, Optional\nfrom abc import ABC, abstractmethod\n\n\nclass BaseMCPServer(ABC):\n \"\"\"Base class for MCP servers.\"\"\"\n\n def __init__(self, config: Dict[str, Any]):\n self._config = config\n self._server = None\n self._namespace = config.get(\"namespace\", \"default\")\n\n @abstractmethod\n def start(self, *args, **kwargs) -> None:\n \"\"\"Start the MCP server.\"\"\"\n pass\n\n @abstractmethod\n def stop(self) -> None:\n \"\"\"Stop the MCP server.\"\"\"\n pass\n\n\nclass HTTPMCPServer(BaseMCPServer):\n \"\"\"HTTP-based MCP server (most common for device agents).\"\"\"\n\n def start(self, *args, **kwargs) -> None:\n \"\"\"Construct HTTP URL for MCP server.\"\"\"\n host = self._config.get(\"host\", \"localhost\")\n port = self._config.get(\"port\", 8000)\n path = self._config.get(\"path\", \"/mcp\")\n self._server = f\"http://{host}:{port}{path}\"\n\n def stop(self) -> None:\n \"\"\"HTTP servers are typically managed externally.\"\"\"\n pass\n\n\nclass LocalMCPServer(BaseMCPServer):\n \"\"\"Local in-memory MCP server.\"\"\"\n\n def start(self, *args, **kwargs) -> None:\n \"\"\"Get server from registry.\"\"\"\n from ufo.client.mcp.mcp_registry import MCPRegistry\n \n server_namespace = self._config.get(\"namespace\")\n self._server = MCPRegistry.get(server_namespace, *args, **kwargs)\n\n\nclass StdioMCPServer(BaseMCPServer):\n \"\"\"Standard I/O MCP server (for subprocess-based tools).\"\"\"\n\n def start(self, *args, **kwargs) -> None:\n \"\"\"Create StdioTransport.\"\"\"\n from fastmcp.client.transports import StdioTransport\n \n command = self._config.get(\"command\", \"python\")\n start_args = self._config.get(\"start_args\", [])\n self._server = StdioTransport(command, start_args)\n\n\nclass MCPServerManager:\n \"\"\"Manages multiple MCP servers.\"\"\"\n\n def __init__(self):\n self.servers: Dict[str, BaseMCPServer] = {}\n\n def register_server(self, name: str, server_type: str, config: Dict):\n \"\"\"\n Register an MCP server.\n \n :param name: Server name\n :param server_type: Type ('http', 'local', 'stdio')\n :param config: Server configuration\n \"\"\"\n if server_type == \"http\":\n server = HTTPMCPServer(config)\n elif server_type == \"local\":\n server = LocalMCPServer(config)\n elif server_type == \"stdio\":\n server = StdioMCPServer(config)\n else:\n raise ValueError(f\"Unknown server type: {server_type}\")\n \n self.servers[name] = server\n\n def start_server(self, name: str):\n \"\"\"Start a registered MCP server.\"\"\"\n if name not in self.servers:\n raise KeyError(f\"Server '{name}' not registered\")\n \n self.servers[name].start()\n\n def get_server(self, name: str) -> BaseMCPServer:\n \"\"\"Get a registered server.\"\"\"\n return self.servers.get(name)\n```\n\n---\n\n## Platform Detection\n\n### Auto-Detection Logic\n\n```python\n# ufo/client/client.py (platform detection)\n\nimport platform as platform_module\n\n# Auto-detect platform if not specified\nif args.platform is None:\n detected_platform = platform_module.system().lower()\n \n if detected_platform in [\"windows\", \"linux\"]:\n args.platform = detected_platform\n \n elif detected_platform == \"darwin\":\n # macOS detection\n args.platform = \"macos\"\n \n else:\n # Fallback for unknown platforms\n args.platform = \"windows\"\n\nlogger.info(f\"Platform: {args.platform}\")\n```\n\n### Platform-Specific Configuration\n\n```python\n# Platform-specific MCP server registration\n\ndef setup_mcp_servers(platform: str, mcp_manager: MCPServerManager):\n \"\"\"Setup MCP servers based on platform.\"\"\"\n \n if platform == \"android\":\n # Register Android MCP server\n mcp_manager.register_server(\n name=\"mobile_mcp\",\n server_type=\"http\",\n config={\n \"host\": \"localhost\",\n \"port\": 8020,\n \"path\": \"/mcp\",\n \"namespace\": \"mobile\",\n }\n )\n mcp_manager.start_server(\"mobile_mcp\")\n \n elif platform == \"linux\":\n # Register Linux MCP server\n mcp_manager.register_server(\n name=\"linux_mcp\",\n server_type=\"http\",\n config={\n \"host\": \"localhost\",\n \"port\": 8010,\n \"path\": \"/mcp\",\n \"namespace\": \"linux\",\n }\n )\n mcp_manager.start_server(\"linux_mcp\")\n \n elif platform == \"windows\":\n # Windows uses local MCP servers\n mcp_manager.register_server(\n name=\"windows_mcp\",\n server_type=\"local\",\n config={\"namespace\": \"windows\"}\n )\n mcp_manager.start_server(\"windows_mcp\")\n```\n\n---\n\n## Configuration and Deployment\n\n### Client Entry Point\n\n**File**: `ufo/client/client.py`\n\n```python\n#!/usr/bin/env python\n# ufo/client/client.py\n\nimport argparse\nimport asyncio\nimport logging\nimport platform as platform_module\n\nfrom ufo.client.computer import ComputerManager\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\nfrom ufo.client.ufo_client import UFOClient\nfrom ufo.client.websocket import UFOWebSocketClient\nfrom config.config_loader import get_ufo_config\nfrom ufo.logging.setup import setup_logger\n\n\ndef parse_arguments():\n \"\"\"Parse command line arguments.\"\"\"\n parser = argparse.ArgumentParser(description=\"UFO Device Client\")\n \n parser.add_argument(\n \"--client-id\",\n default=\"client_001\",\n help=\"Unique client ID (default: client_001)\"\n )\n \n parser.add_argument(\n \"--ws-server\",\n default=\"ws://localhost:5000/ws\",\n help=\"WebSocket server URL (default: ws://localhost:5000/ws)\"\n )\n \n parser.add_argument(\n \"--ws\",\n action=\"store_true\",\n help=\"Enable WebSocket mode (required)\"\n )\n \n parser.add_argument(\n \"--max-retries\",\n type=int,\n default=5,\n help=\"Maximum connection retries (default: 5)\"\n )\n \n parser.add_argument(\n \"--platform\",\n choices=[\"windows\", \"linux\", \"android\", \"ios\"],\n default=None,\n help=\"Platform type (auto-detected if not specified)\"\n )\n \n parser.add_argument(\n \"--log-level\",\n default=\"WARNING\",\n choices=[\"DEBUG\", \"INFO\", \"WARNING\", \"ERROR\", \"CRITICAL\", \"OFF\"],\n help=\"Logging level (default: WARNING)\"\n )\n \n return parser.parse_args()\n\n\nasync def main():\n \"\"\"Main client entry point.\"\"\"\n \n # Parse arguments\n args = parse_arguments()\n \n # Auto-detect platform if not specified\n if args.platform is None:\n detected = platform_module.system().lower()\n args.platform = detected if detected in [\"windows\", \"linux\"] else \"windows\"\n \n # Setup logging\n setup_logger(args.log_level)\n logger = logging.getLogger(__name__)\n logger.info(f\"Platform: {args.platform}\")\n \n # Load configuration\n ufo_config = get_ufo_config()\n \n # Initialize managers\n mcp_server_manager = MCPServerManager()\n computer_manager = ComputerManager(ufo_config.to_dict(), mcp_server_manager)\n \n # Setup platform-specific MCP servers\n setup_mcp_servers(args.platform, mcp_server_manager)\n \n # Create UFO client\n client = UFOClient(\n mcp_server_manager=mcp_server_manager,\n computer_manager=computer_manager,\n client_id=args.client_id,\n platform=args.platform,\n )\n \n logger.info(f\"UFO Client initialized: {args.client_id}\")\n \n # Create WebSocket client\n ws_client = UFOWebSocketClient(\n args.ws_server,\n client,\n max_retries=args.max_retries,\n )\n \n # Start connection\n try:\n await ws_client.connect_and_listen()\n except Exception as e:\n logger.error(f\"Client error: {e}\", exc_info=True)\n return 1\n \n return 0\n\n\nif __name__ == \"__main__\":\n exit_code = asyncio.run(main())\n exit(exit_code)\n```\n\n### Deployment Commands\n\n```bash\n# ========================================\n# Mobile Agent Client (Android)\n# ========================================\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5010/ws \\\n --client-id mobile_agent_1 \\\n --platform android \\\n --log-level INFO\n\n# ========================================\n# Linux Agent Client\n# ========================================\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id linux_agent_1 \\\n --platform linux \\\n --max-retries 10\n\n# ========================================\n# iOS Agent Client\n# ========================================\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://192.168.1.100:5020/ws \\\n --client-id ios_agent_1 \\\n --platform ios \\\n --log-level DEBUG\n```\n\n### Configuration File (Optional)\n\n**File**: `config/client_config.yaml`\n\n```yaml\n# Client Configuration\n\nclient:\n client_id: \"mobile_agent_1\"\n platform: \"android\"\n \nwebsocket:\n server_url: \"ws://192.168.1.100:5010/ws\"\n max_retries: 5\n timeout: 120\n \nlogging:\n level: \"INFO\"\n file: \"logs/client.log\"\n\nmcp_servers:\n - name: \"mobile_mcp\"\n type: \"http\"\n config:\n host: \"localhost\"\n port: 8020\n path: \"/mcp\"\n```\n\n---\n\n## Testing Your Client\n\n### Unit Testing\n\n```python\n# tests/unit/test_ufo_client.py\n\nimport pytest\nfrom unittest.mock import MagicMock, AsyncMock\nfrom ufo.client.ufo_client import UFOClient\nfrom aip.messages import ServerMessage, Command\n\n\nclass TestUFOClient:\n \"\"\"Unit tests for UFO Client.\"\"\"\n\n @pytest.fixture\n def client(self):\n \"\"\"Create test client.\"\"\"\n mcp_manager = MagicMock()\n comp_manager = MagicMock()\n \n return UFOClient(\n mcp_server_manager=mcp_manager,\n computer_manager=comp_manager,\n client_id=\"test_client\",\n platform=\"android\",\n )\n\n @pytest.mark.asyncio\n async def test_execute_actions(self, client):\n \"\"\"Test command execution.\"\"\"\n commands = [\n Command(function=\"tap_screen\", arguments={\"x\": 100, \"y\": 200})\n ]\n \n # Mock command router\n client.command_router.execute = AsyncMock(return_value=[\n {\"success\": True, \"message\": \"Tapped\"}\n ])\n \n results = await client.execute_actions(commands)\n \n assert len(results) == 1\n assert results[0][\"success\"] == True\n\n def test_session_id_setter(self, client):\n \"\"\"Test session ID property.\"\"\"\n client.session_id = \"session_123\"\n assert client.session_id == \"session_123\"\n```\n\n### Integration Testing\n\n```python\n# tests/integration/test_client_integration.py\n\nimport pytest\nimport asyncio\nfrom ufo.client.client import main\n\n\nclass TestClientIntegration:\n \"\"\"Integration tests for client.\"\"\"\n\n @pytest.mark.asyncio\n async def test_client_startup(self):\n \"\"\"Test client starts successfully.\"\"\"\n # Mock arguments\n import sys\n sys.argv = [\n \"client.py\",\n \"--ws\",\n \"--ws-server\", \"ws://localhost:5010/ws\",\n \"--client-id\", \"test_client\",\n \"--platform\", \"android\",\n ]\n \n # Should not raise exceptions\n # (Note: Will timeout waiting for server)\n task = asyncio.create_task(main())\n await asyncio.sleep(2)\n task.cancel()\n```\n\n### Manual Testing\n\n```bash\n# 1. Start MCP server\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --port 8020\n\n# 2. Start agent server\npython -m ufo.server.app --port 5010\n\n# 3. Start client (in another terminal)\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5010/ws \\\n --client-id test_client \\\n --platform android \\\n --log-level DEBUG\n\n# 4. Check logs\ntail -f logs/client.log\n```\n\n---\n\n## Summary\n\n**What You've Built**:\n\n- ✅ UFO Client for command execution\n- ✅ WebSocket client for server communication\n- ✅ MCP Server Manager for MCP integration\n- ✅ Platform detection and configuration\n- ✅ Complete deployment setup\n\n**Key Takeaways**:\n\n| Component | Purpose | Key Methods |\n|-----------|---------|-------------|\n| **UFOClient** | Execute commands | `execute_step()`, `execute_actions()` |\n| **UFOWebSocketClient** | Server communication | `connect_and_listen()`, `handle_messages()` |\n| **MCPServerManager** | Manage MCP servers | `register_server()`, `start_server()` |\n| **client.py** | Entry point | `main()`, argument parsing |\n\n---\n\n## Next Steps\n\n**Continue to**: [Part 4: Configuration & Deployment →](configuration.md)\n\nLearn how to configure your device agent in `third_party.yaml`, register devices in `devices.yaml`, and deploy the complete system.\n\n---\n\n## Related Documentation\n\n- **[Client Overview](../../client/overview.md)** - Client architecture deep dive\n- **[AIP Protocol](../../aip/overview.md)** - Agent Interaction Protocol\n- **[MCP Integration](../../mcp/overview.md)** - MCP fundamentals\n\n---\n\n**Previous**: [← Part 2: MCP Server](mcp_server.md) \n**Next**: [Part 4: Configuration & Deployment →](configuration.md)\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/configuration.md",
"content": "# Part 4: Configuration & Deployment\n\nThis tutorial covers the **configuration files and deployment procedures** needed to integrate your device agent into UFO³. You'll learn to configure `third_party.yaml`, register devices in `devices.yaml`, create prompt templates, and deploy the complete system.\n\n---\n\n## Table of Contents\n\n1. [Configuration Overview](#configuration-overview)\n2. [Third-Party Agent Configuration](#third-party-agent-configuration)\n3. [Device Registration](#device-registration)\n4. [Prompt Template Creation](#prompt-template-creation)\n5. [Step-by-Step Deployment](#step-by-step-deployment)\n6. [Galaxy Multi-Device Integration](#galaxy-multi-device-integration)\n7. [Common Configuration Patterns](#common-configuration-patterns)\n\n---\n\n## Configuration Overview\n\n### Configuration File Hierarchy\n\n```mermaid\ngraph TB\n subgraph \"UFO Configuration\"\n UFOConfig[config/ufo/ UFO Framework Config]\n ThirdParty[third_party.yaml Agent Registration]\n \n UFOConfig --> ThirdParty\n end\n \n subgraph \"Galaxy Configuration\"\n GalaxyConfig[config/galaxy/ Multi-Device Config]\n Devices[devices.yaml Device Registry]\n Constellation[constellation.yaml Orchestration]\n \n GalaxyConfig --> Devices\n GalaxyConfig --> Constellation\n end\n \n subgraph \"Prompt Templates\"\n MainPrompt[ufo/prompts/third_party/ agent_name.yaml]\n ExamplePrompt[ufo/prompts/third_party/ agent_name_example.yaml]\n end\n \n ThirdParty -.references.-> MainPrompt\n ThirdParty -.references.-> ExamplePrompt\n Devices -.references.-> ThirdParty\n \n style ThirdParty fill:#c8e6c9\n style Devices fill:#e1f5ff\n style MainPrompt fill:#fff3e0\n```\n\n**Configuration Files**:\n\n| File | Purpose | Required |\n|------|---------|----------|\n| `config/ufo/third_party.yaml` | Register agent with UFO | ✅ Yes |\n| `config/galaxy/devices.yaml` | Register device instances | ✅ Yes (for Galaxy) |\n| `config/galaxy/constellation.yaml` | Multi-device orchestration | Optional |\n| `ufo/prompts/third_party/.yaml` | Main prompt template | ✅ Yes |\n| `ufo/prompts/third_party/_example.yaml` | Few-shot examples | ✅ Yes |\n\n---\n\n## Third-Party Agent Configuration\n\n### File Location\n\n**Path**: `config/ufo/third_party.yaml`\n\n### Configuration Structure\n\n```yaml\n# Third-Party Agent Integration Configuration\n# This file configures external/third-party agents that extend UFO's capabilities\n\n# ========================================\n# Enabled Agents\n# ========================================\n# List of third-party agents to enable\nENABLED_THIRD_PARTY_AGENTS: [\"MobileAgent\", \"LinuxAgent\"]\n\n\n# ========================================\n# Agent Configurations\n# ========================================\nTHIRD_PARTY_AGENT_CONFIG:\n\n # ----------------------------------\n # MobileAgent Configuration\n # ----------------------------------\n MobileAgent:\n # Visual mode enables screenshot capture\n VISUAL_MODE: True\n \n # Agent name (must match @AgentRegistry.register)\n AGENT_NAME: \"MobileAgent\"\n \n # Prompt template paths (relative to project root)\n APPAGENT_PROMPT: \"ufo/prompts/third_party/mobile_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/mobile_agent_example.yaml\"\n \n # Optional: API prompt template (for custom tool descriptions)\n # API_PROMPT: \"ufo/prompts/third_party/mobile_agent_api.yaml\"\n \n # Agent introduction (shown to HostAgent for delegation)\n INTRODUCTION: >\n The MobileAgent controls Android and iOS mobile devices.\n It can perform UI automation, tap/swipe gestures, type text,\n launch apps, and capture screenshots. Use it for mobile\n app testing, automation, and device control tasks.\n\n # ----------------------------------\n # LinuxAgent Configuration (Reference)\n # ----------------------------------\n LinuxAgent:\n # Visual mode disabled for CLI-based agent\n VISUAL_MODE: False\n \n AGENT_NAME: \"LinuxAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/linux_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/linux_agent_example.yaml\"\n \n INTRODUCTION: >\n The LinuxAgent executes commands on Linux systems.\n It can run bash commands, manage files, inspect processes,\n configure services, and perform system administration tasks.\n Use it for Linux server management and automation.\n```\n\n### Configuration Field Reference\n\n| Field | Type | Required | Description | Example |\n|-------|------|----------|-------------|---------|\n| `VISUAL_MODE` | boolean | ✅ Yes | Enable screenshot capture | `True` for mobile/GUI, `False` for CLI |\n| `AGENT_NAME` | string | ✅ Yes | Must match `@AgentRegistry.register` | `\"MobileAgent\"` |\n| `APPAGENT_PROMPT` | string | ✅ Yes | Path to main prompt template | `\"ufo/prompts/third_party/mobile_agent.yaml\"` |\n| `APPAGENT_EXAMPLE_PROMPT` | string | ✅ Yes | Path to example prompt template | `\"ufo/prompts/third_party/mobile_agent_example.yaml\"` |\n| `API_PROMPT` | string | Optional | Custom API descriptions | `\"ufo/prompts/third_party/mobile_agent_api.yaml\"` |\n| `INTRODUCTION` | string | ✅ Yes | Agent description for HostAgent | Multi-line string describing capabilities |\n\n!!! warning \"Configuration Checklist\"\n - ✅ Add your agent to `ENABLED_THIRD_PARTY_AGENTS` list\n - ✅ Create a config section with agent name as key\n - ✅ Set `AGENT_NAME` to match `@AgentRegistry.register(agent_name=\"...\")`\n - ✅ Set `VISUAL_MODE` based on whether agent uses screenshots\n - ✅ Create prompt template files before starting UFO\n - ✅ Write descriptive `INTRODUCTION` for Galaxy orchestration\n\n---\n\n## Device Registration\n\n### File Location\n\n**Path**: `config/galaxy/devices.yaml`\n\n### Device Configuration Structure\n\n```yaml\n# Device Configuration - YAML Format\n# This configuration defines device instances for Galaxy constellation\n\ndevices:\n # ----------------------------------\n # Mobile Agent Device 1 (Android)\n # ----------------------------------\n - device_id: \"mobile_agent_1\"\n \n # WebSocket server URL for this device\n server_url: \"ws://192.168.1.100:5010/ws\"\n \n # Operating system\n os: \"android\"\n \n # Device capabilities (used by Galaxy for task routing)\n capabilities:\n - \"ui_automation\"\n - \"mobile_app_testing\"\n - \"touch_gestures\"\n - \"screenshot_capture\"\n - \"android_apps\"\n \n # Custom metadata (accessible in prompts via {tips})\n metadata:\n device_model: \"Google Pixel 6\"\n android_version: \"14\"\n screen_resolution: \"1080x2400\"\n device_location: \"Test Lab A\"\n performance: \"high\"\n description: \"Primary Android test device\"\n \n # Custom instructions for the agent\n tips: >\n This device runs Android 14 on Google Pixel 6.\n Screen resolution is 1080x2400 pixels.\n All standard Android apps are installed.\n For app testing, use package name format: com.example.app\n \n # Auto-connect on startup\n auto_connect: true\n \n # Maximum connection retries\n max_retries: 5\n\n # ----------------------------------\n # Mobile Agent Device 2 (iOS)\n # ----------------------------------\n - device_id: \"mobile_agent_2\"\n server_url: \"ws://192.168.1.101:5020/ws\"\n os: \"ios\"\n capabilities:\n - \"ui_automation\"\n - \"ios_app_testing\"\n - \"xcuitest\"\n - \"screenshot_capture\"\n metadata:\n device_model: \"iPhone 14 Pro\"\n ios_version: \"17.2\"\n screen_resolution: \"1179x2556\"\n device_location: \"Test Lab B\"\n tips: >\n iOS device using XCUITest for automation.\n Use bundle ID format: com.company.AppName\n auto_connect: true\n max_retries: 5\n\n # ----------------------------------\n # Linux Agent (Server)\n # ----------------------------------\n - device_id: \"linux_agent_1\"\n server_url: \"ws://192.168.1.50:5001/ws\"\n os: \"linux\"\n capabilities:\n - \"bash_commands\"\n - \"server_management\"\n - \"file_operations\"\n - \"process_management\"\n metadata:\n os_version: \"Ubuntu 22.04\"\n hostname: \"server-01\"\n logs_file_path: \"/var/log/app/app.log\"\n dev_path: \"/home/developer/projects/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n tips: >\n Ubuntu 22.04 server.\n Application logs: /var/log/app/app.log\n Development path: /home/developer/projects/\n Use 'sudo' for privileged operations.\n auto_connect: true\n max_retries: 10\n\n # ----------------------------------\n # Additional Device Template\n # ----------------------------------\n # - device_id: \"your_device_id\"\n # server_url: \"ws://HOST:PORT/ws\"\n # os: \"android|ios|linux|windows\"\n # capabilities: [\"capability1\", \"capability2\"]\n # metadata:\n # key: \"value\"\n # tips: \"Custom instructions\"\n # auto_connect: true\n # max_retries: 5\n```\n\n### Device Configuration Field Reference\n\n| Field | Type | Required | Description | Example |\n|-------|------|----------|-------------|---------|\n| `device_id` | string | ✅ Yes | Unique device identifier | `\"mobile_agent_1\"` |\n| `server_url` | string | ✅ Yes | WebSocket server URL | `\"ws://192.168.1.100:5010/ws\"` |\n| `os` | string | ✅ Yes | Operating system | `\"android\"`, `\"ios\"`, `\"linux\"`, `\"windows\"` |\n| `capabilities` | list[string] | ✅ Yes | Device capabilities | `[\"ui_automation\", \"app_testing\"]` |\n| `metadata` | dict | Optional | Custom device metadata | `{device_model: \"Pixel 6\", ...}` |\n| `metadata.tips` | string | Recommended | Agent-specific instructions | Multi-line instructions |\n| `auto_connect` | boolean | Optional | Auto-connect on startup | `true` (default: `false`) |\n| `max_retries` | integer | Optional | Connection retry limit | `5` (default: `3`) |\n\n!!! tip \"Device Configuration Best Practices\"\n - ✅ Use descriptive `device_id` (e.g., `mobile_android_pixel6_lab1`)\n - ✅ Add comprehensive `capabilities` for Galaxy task routing\n - ✅ Include device-specific details in `metadata.tips`\n - ✅ Set `auto_connect: true` for production devices\n - ✅ Use higher `max_retries` for unstable networks\n - ✅ Include log paths, dev paths, and patterns in `metadata`\n\n---\n\n## Prompt Template Creation\n\n### Main Prompt Template\n\n**File**: `ufo/prompts/third_party/mobile_agent.yaml`\n\n```yaml\nversion: 1.0\n\nsystem: |-\n You are **MobileAgent**, the UFO framework's intelligent agent for mobile device automation.\n Your goal is to **complete the entire User Request** by interacting with mobile devices using touch gestures, UI automation, and available APIs.\n\n ## Capabilities\n - **Tap** elements by coordinates or UI element properties\n - **Swipe** gestures (up, down, left, right) for scrolling and navigation\n - **Type** text into input fields\n - **Launch** applications by package/bundle ID\n - **Capture** screenshots for visual inspection\n - **Extract** UI hierarchy (XML tree on Android, Accessibility tree on iOS)\n\n ## Platform Support\n - **Android**: Via ADB (Android Debug Bridge) and UI Automator\n - **iOS**: Via XCTest framework and accessibility APIs\n\n ## Task Status\n After each step, decide the overall status of the **User Request**:\n - `CONTINUE` — the request is partially complete; further actions are required.\n - `FINISH` — the request has been successfully fulfilled; no further actions are needed.\n - `FAIL` — the request cannot be completed due to errors, invalid UI state, or repeated ineffective attempts.\n\n ## Response Format\n Always respond **only** with valid JSON that strictly follows the structure below.\n Your output must be directly parseable by `json.loads()` — no markdown, comments, or extra text.\n\n Required JSON keys:\n\n {{{{\n \"observation\": str, \"\",\n \"thought\": str, \"\",\n \"action\": {{{{\n \"function\": str, \"\",\n \"arguments\": Dict[str, Any], \"': ''}}, for the function. Use an empty dictionary if no arguments are needed.>\",\n \"status\": str, \"\"\n }}}},\n \"plan\": List[str], \"\",\n \"result\": str, \"\"\n }}}}\n\n ## Operational Rules\n - **Use screenshots and UI tree** to understand the current mobile UI state\n - **Identify UI elements** by text, content-desc, resource-id, or coordinates\n - **Plan actions carefully** - mobile UIs may have animations, loading states, or modal dialogs\n - **Verify actions** - after tapping a button, check if the expected screen transition occurred\n - **Handle edge cases** - pop-ups, permissions dialogs, network errors, app crashes\n - Do **not** ask for user confirmation\n - Avoid **destructive actions** (uninstall apps, factory reset) unless explicitly instructed\n - Review previous actions to avoid repeating ineffective steps\n\n ## Actions\n - You are able to use the following APIs to interact with the mobile device.\n {{apis}}\n\n ## Examples\n - Below are some examples for your reference. Only use them as guidance and do not copy them directly.\n {{examples}}\n\n ## Final Reminder\n Please observe the previous steps, current screenshot, and UI tree carefully to decide your next action.\n Think step-by-step, act carefully, and output only the required JSON structure.\n Any invalid JSON or extra text will crash the system.\n\n\nuser: |-\n {{user_request}}\n [See attached image]\n {{installed_apps}}\n {{current_controls}}\n {{last_success_actions}}\n {{prev_plan}}\n \n```\n\n### Example Prompt Template\n\n**File**: `ufo/prompts/third_party/mobile_agent_example.yaml`\n\n```yaml\nversion: 1.0\n\nexample1:\n Request: |-\n Open the Settings app and enable Wi-Fi.\n Response:\n observation: |-\n The device is on the home screen. I can see app icons for various applications. The Settings app icon is visible in the UI tree with package name \"com.android.settings\".\n thought: |-\n To enable Wi-Fi, I need to first launch the Settings app. I'll use the launch_app function with the Settings package name.\n action:\n function: |-\n launch_app\n arguments:\n {\"package_name\": \"com.android.settings\"}\n status: |-\n CONTINUE\n plan:\n - (1) Launch Settings app.\n - (2) Navigate to Wi-Fi settings (may require scrolling).\n - (3) Tap the Wi-Fi toggle to enable it.\n - (4) Verify Wi-Fi is enabled and mark FINISH.\n result: |-\n Request in CONTINUE state. Successfully launched Settings app. Next steps: navigate to Wi-Fi settings and enable the toggle.\n\nexample2:\n Request: |-\n Find and tap the \"Login\" button on the current screen.\n Response:\n observation: |-\n The current screenshot shows a login screen with email and password input fields. There is a button with text \"Login\" visible near the bottom of the screen. According to the current screen controls list, the button is located at coordinates (540, 1650) with resource-id \"com.example.app:id/login_button\".\n thought: |-\n I can see the Login button in the controls list. I'll tap it using the coordinates provided.\n action:\n function: |-\n tap_screen\n arguments:\n {\"x\": 540, \"y\": 1650}\n status: |-\n CONTINUE\n plan:\n - (1) Tap the Login button at (540, 1650).\n - (2) Wait for navigation or error message.\n - (3) Verify successful login or handle errors.\n - (4) Mark FINISH if logged in successfully.\n result: |-\n Request in CONTINUE state. Tapped Login button at coordinates (540, 1650). Awaiting app response (loading indicator or next screen).\n\nexample3:\n Request: |-\n Scroll down to view more content in the current list.\n Response:\n observation: |-\n The screenshot shows a scrollable list view with multiple items. The UI tree indicates this is a RecyclerView with id \"com.example.app:id/content_list\". The list appears to have more content below the visible area based on the scroll indicators.\n thought: |-\n To scroll down, I'll perform a swipe gesture from bottom to top. I'll use coordinates in the middle of the screen, starting near the bottom and ending near the top.\n action:\n function: |-\n swipe\n arguments:\n {\"start_x\": 540, \"start_y\": 1800, \"end_x\": 540, \"end_y\": 600, \"duration_ms\": 300}\n status: |-\n CONTINUE\n plan:\n - (1) Swipe down to scroll the list.\n - (2) Capture new screenshot to see newly visible content.\n - (3) Check if target content is now visible.\n - (4) Repeat scroll if needed or mark FINISH.\n result: |-\n Request in CONTINUE state. Performed swipe gesture to scroll down. The list should now display content that was previously below the visible area.\n\nexample4:\n Request: |-\n Type \"testuser@example.com\" into the email field.\n Response:\n observation: |-\n The screenshot shows a login form with two input fields. The UI tree shows an EditText with hint \"Email\" at coordinates (540, 800) with resource-id \"com.example.app:id/email_input\". The field appears to be empty and ready for input.\n thought: |-\n I need to first tap the email field to focus it, then type the email address. I'll tap the field coordinates first.\n action:\n function: |-\n tap_screen\n arguments:\n {\"x\": 540, \"y\": 800}\n status: |-\n CONTINUE\n plan:\n - (1) Tap the email input field to focus it.\n - (2) Type the email address \"testuser@example.com\".\n - (3) Verify text is entered correctly.\n - (4) Mark FINISH.\n result: |-\n Request in CONTINUE state. Tapped email field at (540, 800) to focus it. Next step: type the email address into the focused field.\n```\n\n### Prompt Template Best Practices\n\n| Component | Best Practice | Example |\n|-----------|---------------|---------|\n| **System Prompt** | Comprehensive instructions | Capabilities, rules, response format |\n| **Response Format** | JSON schema with examples | `{\"observation\": ..., \"thought\": ..., \"action\": ...}` |\n| **API Placeholder** | Use `{apis}` for tool injection | Populated by prompter |\n| **Examples Placeholder** | Use `{examples}` for few-shot | Populated from example template |\n| **User Prompt** | Include all context | Request, screenshot, UI tree, history |\n| **Examples** | Cover common scenarios | Launch app, tap, swipe, type, scroll |\n\n!!! tip \"Prompt Template Tips\"\n - ✅ Use `{{variable}}` for template variables (double braces)\n - ✅ Provide clear JSON structure with type annotations\n - ✅ Include platform-specific guidance (Android vs iOS)\n - ✅ Add examples covering success and failure cases\n - ✅ Reference screenshots and UI trees in prompts\n - ✅ Emphasize JSON-only output (no markdown)\n - ❌ Don't hardcode API descriptions (use `{apis}` placeholder)\n\n---\n\n## Step-by-Step Deployment\n\n### Deployment Checklist\n\n```mermaid\ngraph TB\n Start([Start Deployment]) --> Config[1. Configure Files]\n Config --> Code[2. Implement Agent Code]\n Code --> MCP[3. Create MCP Server]\n MCP --> Test[4. Test Components]\n Test --> Server[5. Start Agent Server]\n Server --> MCPStart[6. Start MCP Server]\n MCPStart --> Client[7. Start Device Client]\n Client --> Verify[8. Verify Connection]\n Verify --> Ready[9. Ready for Tasks]\n \n style Start fill:#c8e6c9\n style Ready fill:#c8e6c9\n style Test fill:#fff3e0\n style Verify fill:#fff3e0\n```\n\n### Step 1: Configure third_party.yaml\n\n```bash\n# Edit config/ufo/third_party.yaml\nnano config/ufo/third_party.yaml\n```\n\nAdd your agent configuration:\n\n```yaml\nENABLED_THIRD_PARTY_AGENTS: [\"MobileAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n MobileAgent:\n VISUAL_MODE: True\n AGENT_NAME: \"MobileAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/mobile_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/mobile_agent_example.yaml\"\n INTRODUCTION: \"MobileAgent controls Android/iOS devices...\"\n```\n\n### Step 2: Register Device in devices.yaml\n\n```bash\n# Edit config/galaxy/devices.yaml\nnano config/galaxy/devices.yaml\n```\n\nAdd device registration:\n\n```yaml\ndevices:\n - device_id: \"mobile_agent_1\"\n server_url: \"ws://192.168.1.100:5010/ws\"\n os: \"android\"\n capabilities: [\"ui_automation\", \"app_testing\"]\n metadata:\n device_model: \"Pixel 6\"\n tips: \"Android device for testing\"\n auto_connect: true\n max_retries: 5\n```\n\n### Step 3: Create Prompt Templates\n\n```bash\n# Create main prompt\ntouch ufo/prompts/third_party/mobile_agent.yaml\n\n# Create example prompt\ntouch ufo/prompts/third_party/mobile_agent_example.yaml\n```\n\nCopy content from [Prompt Template Creation](#prompt-template-creation) section.\n\n### Step 4: Implement Agent Components\n\n```bash\n# Agent class\n# Edit: ufo/agents/agent/customized_agent.py\n\n# Processor\n# Edit: ufo/agents/processors/customized/customized_agent_processor.py\n\n# States\n# Create: ufo/agents/states/mobile_agent_state.py\n\n# Strategies\n# Create: ufo/agents/processors/strategies/mobile_agent_strategy.py\n\n# Prompter\n# Create: ufo/prompter/customized/mobile_agent_prompter.py\n```\n\n### Step 5: Create MCP Server\n\n```bash\n# Create MCP server\ntouch ufo/client/mcp/http_servers/mobile_mcp_server.py\n```\n\nImplement MCP server from [Part 2: MCP Server Development](mcp_server.md).\n\n### Step 6: Test Components\n\n```bash\n# Run unit tests\npytest tests/unit/test_mobile_agent.py\n\n# Run integration tests\npytest tests/integration/test_mobile_agent_integration.py\n```\n\n### Step 7: Start Agent Server\n\n```bash\n# Terminal 1: Start UFO agent server\npython -m ufo.server.app --port 5010\n```\n\nExpected output:\n```\n========================================\nUFO Agent Server\n========================================\nINFO: Server starting on 0.0.0.0:5010\nINFO: Registered agents: MobileAgent, LinuxAgent\nINFO: WebSocket endpoint: ws://localhost:5010/ws\n========================================\n```\n\n### Step 8: Start MCP Server\n\n```bash\n# Terminal 2: Start MCP server\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --port 8020 \\\n --platform android\n```\n\nExpected output:\n```\n==================================================\nUFO Mobile MCP Server (Android)\nMobile device automation via Model Context Protocol\nRunning on localhost:8020\n==================================================\nINFO: Server started successfully\nINFO: Registered tools: tap_screen, swipe, type_text, ...\n```\n\n### Step 9: Start Device Client\n\n```bash\n# Terminal 3: Start device client\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5010/ws \\\n --client-id mobile_agent_1 \\\n --platform android \\\n --log-level INFO\n```\n\nExpected output:\n```\nINFO: Platform: android\nINFO: UFO Client initialized: mobile_agent_1\nINFO: Connecting to ws://localhost:5010/ws (attempt 1/5)\nINFO: ✅ Client registered: mobile_agent_1\nINFO: Starting message handling loop\n```\n\n### Step 10: Verify Connection\n\n```bash\n# Check agent server logs\n# Should show: \"Client mobile_agent_1 registered\"\n\n# Check client logs\n# Should show: \"✅ Client registered: mobile_agent_1\"\n\n# Test basic command (optional)\ncurl -X POST http://localhost:5010/api/v1/task \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"request\": \"Tap at coordinates (500, 1000)\",\n \"device_id\": \"mobile_agent_1\"\n }'\n```\n\n---\n\n## Galaxy Multi-Device Integration\n\n### Constellation Configuration\n\n**File**: `config/galaxy/constellation.yaml`\n\n```yaml\n# Galaxy Constellation Configuration\n# Multi-device orchestration settings\n\nconstellation:\n # Constellation ID (unique identifier)\n constellation_id: \"mobile_test_constellation\"\n \n # Heartbeat interval (seconds)\n heartbeat_interval: 30\n \n # Task timeout (seconds)\n task_timeout: 300\n \n # Retry strategy\n max_task_retries: 3\n retry_delay: 5\n \n # Load balancing\n load_balancing_strategy: \"round_robin\" # Options: round_robin, least_loaded, capability_based\n \n # Device selection\n device_selection_strategy: \"capability_match\" # Options: capability_match, explicit, random\n\n# Task routing rules\nrouting_rules:\n - task_type: \"mobile_app_testing\"\n preferred_devices: [\"mobile_agent_1\", \"mobile_agent_2\"]\n required_capabilities: [\"ui_automation\"]\n \n - task_type: \"server_management\"\n preferred_devices: [\"linux_agent_1\", \"linux_agent_2\"]\n required_capabilities: [\"bash_commands\"]\n```\n\n### Galaxy Deployment Example\n\n```bash\n# ========================================\n# Start Galaxy with Multiple Devices\n# ========================================\n\n# Terminal 1: Galaxy orchestrator\npython -m galaxy \\\n --constellation-id mobile_test_constellation \\\n --config config/galaxy/constellation.yaml\n\n# Terminal 2-4: Device clients\npython -m ufo.client.client --ws --ws-server ws://localhost:5010/ws --client-id mobile_agent_1 --platform android &\npython -m ufo.client.client --ws --ws-server ws://localhost:5011/ws --client-id mobile_agent_2 --platform ios &\npython -m ufo.client.client --ws --ws-server ws://localhost:5001/ws --client-id linux_agent_1 --platform linux &\n\n# Terminal 5: Submit multi-device task\npython -m galaxy.client.submit_task \\\n --constellation mobile_test_constellation \\\n --request \"Test app on both Android and iOS devices\" \\\n --devices mobile_agent_1,mobile_agent_2\n```\n\n---\n\n## Common Configuration Patterns\n\n### Pattern 1: Development vs Production\n\n```yaml\n# Development configuration\nENABLED_THIRD_PARTY_AGENTS: [\"MobileAgent\"]\nTHIRD_PARTY_AGENT_CONFIG:\n MobileAgent:\n VISUAL_MODE: True\n # Use local test device\n \n# config/galaxy/devices.yaml (dev)\ndevices:\n - device_id: \"mobile_dev\"\n server_url: \"ws://localhost:5010/ws\"\n auto_connect: false # Manual connection for debugging\n\n---\n\n# Production configuration\nENABLED_THIRD_PARTY_AGENTS: [\"MobileAgent\", \"LinuxAgent\"]\nTHIRD_PARTY_AGENT_CONFIG:\n MobileAgent:\n VISUAL_MODE: True\n # Use production device farm\n\n# config/galaxy/devices.yaml (prod)\ndevices:\n - device_id: \"mobile_prod_01\"\n server_url: \"ws://192.168.1.100:5010/ws\"\n auto_connect: true # Auto-connect for reliability\n max_retries: 10\n```\n\n### Pattern 2: Multi-Platform Support\n\n```yaml\n# Support both Android and iOS with same agent\nTHIRD_PARTY_AGENT_CONFIG:\n MobileAgent:\n VISUAL_MODE: True\n AGENT_NAME: \"MobileAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/mobile_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/mobile_agent_example.yaml\"\n \n# Separate device registrations\ndevices:\n - device_id: \"android_device\"\n os: \"android\"\n capabilities: [\"ui_automation\", \"android_apps\"]\n \n - device_id: \"ios_device\"\n os: \"ios\"\n capabilities: [\"ui_automation\", \"ios_apps\", \"xcuitest\"]\n```\n\n### Pattern 3: Device Pool Management\n\n```yaml\n# Multiple devices of same type for load balancing\ndevices:\n - device_id: \"android_pool_1\"\n server_url: \"ws://192.168.1.101:5010/ws\"\n os: \"android\"\n capabilities: [\"ui_automation\"]\n metadata:\n pool: \"android_test_farm\"\n device_index: 1\n \n - device_id: \"android_pool_2\"\n server_url: \"ws://192.168.1.102:5010/ws\"\n os: \"android\"\n capabilities: [\"ui_automation\"]\n metadata:\n pool: \"android_test_farm\"\n device_index: 2\n```\n\n---\n\n## Summary\n\n**What You've Configured**:\n\n- ✅ Third-party agent registration in `third_party.yaml`\n- ✅ Device registration in `devices.yaml`\n- ✅ Main and example prompt templates\n- ✅ Step-by-step deployment procedure\n- ✅ Galaxy multi-device integration (optional)\n\n**Key Takeaways**:\n\n| Configuration | Purpose | File |\n|---------------|---------|------|\n| **Agent Registration** | Enable agent in UFO | `config/ufo/third_party.yaml` |\n| **Device Registry** | Register device instances | `config/galaxy/devices.yaml` |\n| **Prompt Templates** | Define LLM prompts | `ufo/prompts/third_party/*.yaml` |\n| **Deployment** | Start servers and clients | Terminal commands |\n\n---\n\n## Next Steps\n\n**Continue to**: [Part 5: Testing & Debugging →](testing.md)\n\nLearn comprehensive testing strategies, debugging techniques, and common issue resolution.\n\n---\n\n## Related Documentation\n\n- **[Galaxy Overview](../../galaxy/overview.md)** - Multi-device orchestration\n- **[Third-Party Agents](../creating_third_party_agents.md)** - Related tutorial\n- **[Agent Architecture](../../infrastructure/agents/overview.md)** - Agent design patterns\n\n---\n\n**Previous**: [← Part 3: Client Setup](client_setup.md) \n**Next**: [Part 5: Testing & Debugging →](testing.md)\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/core_components.md",
"content": "# Part 1: Core Components - Server-Side Implementation\n\nThis tutorial covers the **server-side components** of your device agent. You'll learn to implement the Agent Class, Processor, State Manager, Strategies, and Prompter using **LinuxAgent** as reference.\n\n---\n\n## Table of Contents\n\n1. [Component Overview](#component-overview)\n2. [Step 1: Agent Class](#step-1-agent-class)\n3. [Step 2: Processor](#step-2-processor)\n4. [Step 3: State Manager](#step-3-state-manager)\n5. [Step 4: Processing Strategies](#step-4-processing-strategies)\n6. [Step 5: Prompter](#step-5-prompter)\n7. [Testing Your Implementation](#testing-your-implementation)\n\n---\n\n## Component Overview\n\n### What You'll Build\n\n```mermaid\ngraph TB\n subgraph \"Server-Side Components\"\n A[MobileAgent Class Agent Definition]\n B[MobileAgentProcessor Strategy Orchestration]\n C[MobileAgentStateManager FSM Control]\n D[Strategies LLM & Action Logic]\n E[MobileAgentPrompter Prompt Construction]\n \n A --> B\n A --> C\n B --> D\n A --> E\n end\n \n style A fill:#c8e6c9\n style B fill:#fff3e0\n style C fill:#e1f5ff\n style D fill:#f3e5f5\n style E fill:#ffe1e1\n```\n\n**Component Responsibilities**:\n\n| Component | File | Purpose | Example (LinuxAgent) |\n|-----------|------|---------|---------------------|\n| **Agent Class** | `customized_agent.py` | Agent definition, initialization | `LinuxAgent` class |\n| **Processor** | `customized_agent_processor.py` | Strategy orchestration | `LinuxAgentProcessor` |\n| **State Manager** | `linux_agent_state.py` | FSM states and transitions | `LinuxAgentStateManager` |\n| **Strategies** | `linux_agent_strategy.py` | LLM and action execution logic | `LinuxLLMInteractionStrategy` |\n| **Prompter** | `linux_agent_prompter.py` | Prompt construction for LLM | `LinuxAgentPrompter` |\n\n---\n\n## Step 1: Agent Class\n\n### Understanding the Agent Class\n\nThe **Agent Class** is the entry point for your device agent. It:\n\n- Inherits from `CustomizedAgent` (which extends `AppAgent`)\n- Registers with `AgentRegistry` for automatic discovery\n- Initializes prompter and default state\n- Maintains blackboard for multi-agent coordination\n\n### LinuxAgent Implementation\n\n```python\n# File: ufo/agents/agent/customized_agent.py\n\nfrom ufo.agents.agent.app_agent import AppAgent\nfrom ufo.agents.agent.basic import AgentRegistry\nfrom ufo.agents.memory.blackboard import Blackboard\nfrom ufo.agents.processors.customized.customized_agent_processor import (\n LinuxAgentProcessor,\n)\nfrom ufo.agents.states.linux_agent_state import ContinueLinuxAgentState\nfrom ufo.prompter.customized.linux_agent_prompter import LinuxAgentPrompter\n\n\n@AgentRegistry.register(\n agent_name=\"LinuxAgent\", # Unique identifier\n third_party=True, # Mark as third-party/device agent\n processor_cls=LinuxAgentProcessor # Link to processor class\n)\nclass LinuxAgent(CustomizedAgent):\n \"\"\"\n LinuxAgent is a specialized agent that interacts with Linux systems.\n Executes shell commands via MCP and manages Linux device tasks.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n main_prompt: str,\n example_prompt: str,\n ) -> None:\n \"\"\"\n Initialize the LinuxAgent.\n \n :param name: The name of the agent instance\n :param main_prompt: Path to main prompt template YAML\n :param example_prompt: Path to example prompt template YAML\n \"\"\"\n # Call parent constructor with None for process/app (not GUI-based)\n super().__init__(\n name=name,\n main_prompt=main_prompt,\n example_prompt=example_prompt,\n process_name=None, # No Windows process for Linux\n app_root_name=None, # No Windows app for Linux\n is_visual=None, # Typically False for CLI-based agents\n )\n \n # Initialize blackboard for multi-agent coordination\n self._blackboard = Blackboard()\n \n # Set default state (ContinueLinuxAgentState)\n self.set_state(self.default_state)\n \n # Flag to track context provision\n self._context_provision_executed = False\n \n # Logger for debugging\n self.logger = logging.getLogger(__name__)\n self.logger.info(\n f\"LinuxAgent initialized with prompts: {main_prompt}, {example_prompt}\"\n )\n\n def get_prompter(\n self, is_visual: bool, main_prompt: str, example_prompt: str\n ) -> LinuxAgentPrompter:\n \"\"\"\n Get the prompter for the agent.\n \n :param is_visual: Whether the agent uses visual mode (screenshots)\n :param main_prompt: Path to main prompt template\n :param example_prompt: Path to example prompt template\n :return: LinuxAgentPrompter instance\n \"\"\"\n return LinuxAgentPrompter(main_prompt, example_prompt)\n\n @property\n def default_state(self) -> ContinueLinuxAgentState:\n \"\"\"\n Get the default state for LinuxAgent.\n \n :return: ContinueLinuxAgentState instance\n \"\"\"\n return ContinueLinuxAgentState()\n\n @property\n def blackboard(self) -> Blackboard:\n \"\"\"\n Get the blackboard for multi-agent coordination.\n \n :return: Blackboard instance\n \"\"\"\n return self._blackboard\n```\n\n### Creating Your MobileAgent Class\n\nNow let's create `MobileAgent` following the same pattern:\n\n```python\n# File: ufo/agents/agent/customized_agent.py\n\nimport logging\nfrom ufo.agents.agent.app_agent import AppAgent\nfrom ufo.agents.agent.basic import AgentRegistry\nfrom ufo.agents.memory.blackboard import Blackboard\nfrom ufo.agents.processors.customized.customized_agent_processor import (\n MobileAgentProcessor, # We'll create this in Step 2\n)\nfrom ufo.agents.states.mobile_agent_state import ContinueMobileAgentState\nfrom ufo.prompter.customized.mobile_agent_prompter import MobileAgentPrompter\n\n\n@AgentRegistry.register(\n agent_name=\"MobileAgent\",\n third_party=True,\n processor_cls=MobileAgentProcessor\n)\nclass MobileAgent(CustomizedAgent):\n \"\"\"\n MobileAgent controls Android/iOS mobile devices.\n Supports UI automation, app testing, and mobile-specific operations.\n \"\"\"\n\n def __init__(\n self,\n name: str,\n main_prompt: str,\n example_prompt: str,\n platform: str = \"android\", # Platform: \"android\" or \"ios\"\n ) -> None:\n \"\"\"\n Initialize the MobileAgent.\n \n :param name: Agent instance name\n :param main_prompt: Main prompt template path\n :param example_prompt: Example prompt template path\n :param platform: Mobile platform (\"android\" or \"ios\")\n \"\"\"\n super().__init__(\n name=name,\n main_prompt=main_prompt,\n example_prompt=example_prompt,\n process_name=None,\n app_root_name=None,\n is_visual=True, # Mobile agents typically use screenshots\n )\n \n # Store platform information\n self._platform = platform\n \n # Initialize blackboard\n self._blackboard = Blackboard()\n \n # Set default state\n self.set_state(self.default_state)\n \n # Logger\n self.logger = logging.getLogger(__name__)\n self.logger.info(\n f\"MobileAgent initialized for platform: {platform}\"\n )\n\n def get_prompter(\n self, is_visual: bool, main_prompt: str, example_prompt: str\n ) -> MobileAgentPrompter:\n \"\"\"Get the prompter for MobileAgent.\"\"\"\n return MobileAgentPrompter(main_prompt, example_prompt)\n\n @property\n def default_state(self) -> ContinueMobileAgentState:\n \"\"\"Get the default state.\"\"\"\n return ContinueMobileAgentState()\n\n @property\n def blackboard(self) -> Blackboard:\n \"\"\"Get the blackboard.\"\"\"\n return self._blackboard\n \n @property\n def platform(self) -> str:\n \"\"\"Get the mobile platform (android/ios).\"\"\"\n return self._platform\n```\n\n### Key Differences from LinuxAgent\n\n| Aspect | LinuxAgent | MobileAgent |\n|--------|-----------|-------------|\n| **is_visual** | `None` (no screenshots) | `True` (UI screenshots needed) |\n| **Platform Tracking** | Not needed | `self._platform` stores \"android\"/\"ios\" |\n| **Processor** | `LinuxAgentProcessor` | `MobileAgentProcessor` |\n| **Prompter** | `LinuxAgentPrompter` | `MobileAgentPrompter` |\n| **Default State** | `ContinueLinuxAgentState` | `ContinueMobileAgentState` |\n\n!!! tip \"Agent Class Best Practices\"\n - ✅ Always call `super().__init__()` first\n - ✅ Initialize blackboard for multi-agent coordination\n - ✅ Set `is_visual=True` if your agent uses screenshots\n - ✅ Use meaningful logger messages for debugging\n - ✅ Store platform-specific metadata as properties\n - ✅ Keep initialization logic minimal (delegate to processor)\n\n---\n\n## Step 2: Processor\n\n### Understanding the Processor\n\nThe **Processor** orchestrates the execution pipeline through modular strategies. It:\n\n- Manages strategy execution across 4 phases\n- Configures middleware (logging, error handling, metrics)\n- Validates strategy dependencies\n- Finalizes processing context\n\n### Four Processing Phases\n\n```mermaid\ngraph LR\n A[DATA_COLLECTION Screenshots, UI Tree] --> B[LLM_INTERACTION Prompt → LLM → Response]\n B --> C[ACTION_EXECUTION Execute Commands]\n C --> D[MEMORY_UPDATE Update Context]\n \n style A fill:#e3f2fd\n style B fill:#fff3e0\n style C fill:#f3e5f5\n style D fill:#e8f5e9\n```\n\n### LinuxAgentProcessor Implementation\n\n```python\n# File: ufo/agents/processors/customized/customized_agent_processor.py\n\nfrom typing import TYPE_CHECKING\nfrom ufo.agents.processors.app_agent_processor import AppAgentProcessor\nfrom ufo.agents.processors.context.processing_context import ProcessingPhase\nfrom ufo.agents.processors.strategies.app_agent_processing_strategy import (\n AppMemoryUpdateStrategy,\n)\nfrom ufo.agents.processors.strategies.linux_agent_strategy import (\n LinuxActionExecutionStrategy,\n LinuxLLMInteractionStrategy,\n LinuxLoggingMiddleware,\n)\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import LinuxAgent\n\n\nclass LinuxAgentProcessor(CustomizedProcessor):\n \"\"\"\n Processor for Linux MCP Agent.\n \n Manages the execution pipeline with strategies for:\n - LLM Interaction: Generate shell commands\n - Action Execution: Execute commands via Linux MCP\n - Memory Update: Update agent memory and blackboard\n \"\"\"\n\n def _setup_strategies(self) -> None:\n \"\"\"\n Setup processing strategies for LinuxAgent.\n \n Note: No DATA_COLLECTION strategy since LinuxAgent doesn't\n use screenshots (relies on shell command output).\n \"\"\"\n \n # Phase 2: LLM Interaction\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n LinuxLLMInteractionStrategy(\n fail_fast=True # LLM failures should halt processing\n )\n )\n\n # Phase 3: Action Execution\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n LinuxActionExecutionStrategy(\n fail_fast=False # Continue on action failures\n )\n )\n\n # Phase 4: Memory Update\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(\n fail_fast=False # Memory failures shouldn't stop agent\n )\n )\n\n def _setup_middleware(self) -> None:\n \"\"\"\n Setup middleware pipeline for LinuxAgent.\n \n Uses custom logging middleware for Linux-specific context.\n \"\"\"\n self.middleware_chain = [LinuxLoggingMiddleware()]\n\n def _finalize_processing_context(\n self, processing_context: ProcessingContext\n ) -> None:\n \"\"\"\n Finalize processing context by updating global context.\n \n :param processing_context: The processing context to finalize\n \"\"\"\n super()._finalize_processing_context(processing_context)\n \n try:\n # Extract result from local context\n result = processing_context.get_local(\"result\")\n if result:\n # Update global context with result\n self.global_context.set(ContextNames.ROUND_RESULT, result)\n except Exception as e:\n self.logger.warning(\n f\"Failed to update ContextNames from results: {e}\"\n )\n```\n\n### Creating MobileAgentProcessor\n\n```python\n# File: ufo/agents/processors/customized/customized_agent_processor.py\n\nfrom ufo.agents.processors.strategies.customized_agent_processing_strategy import (\n CustomizedScreenshotCaptureStrategy,\n)\nfrom ufo.agents.processors.strategies.mobile_agent_strategy import (\n MobileActionExecutionStrategy,\n MobileLLMInteractionStrategy,\n MobileLoggingMiddleware,\n)\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import MobileAgent\n\n\nclass MobileAgentProcessor(CustomizedProcessor):\n \"\"\"\n Processor for MobileAgent.\n \n Manages execution pipeline with mobile-specific strategies:\n - Data Collection: Screenshots and UI hierarchy\n - LLM Interaction: Mobile UI understanding\n - Action Execution: Touch gestures, swipes, taps\n - Memory Update: Context tracking\n \"\"\"\n\n def _setup_strategies(self) -> None:\n \"\"\"Setup processing strategies for MobileAgent.\"\"\"\n \n # Phase 1: Data Collection (screenshots + UI tree)\n self.strategies[ProcessingPhase.DATA_COLLECTION] = (\n CustomizedScreenshotCaptureStrategy(\n fail_fast=True # Stop if screenshot capture fails\n )\n )\n\n # Phase 2: LLM Interaction (mobile UI understanding)\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n MobileLLMInteractionStrategy(\n fail_fast=True # LLM failures should halt\n )\n )\n\n # Phase 3: Action Execution (touch gestures)\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n MobileActionExecutionStrategy(\n fail_fast=False # Retry on action failures\n )\n )\n\n # Phase 4: Memory Update\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(\n fail_fast=False # Don't stop on memory errors\n )\n )\n\n def _setup_middleware(self) -> None:\n \"\"\"Setup middleware for mobile-specific logging.\"\"\"\n self.middleware_chain = [MobileLoggingMiddleware()]\n\n def _finalize_processing_context(\n self, processing_context: ProcessingContext\n ) -> None:\n \"\"\"Finalize context with mobile-specific results.\"\"\"\n super()._finalize_processing_context(processing_context)\n \n try:\n # Extract mobile-specific results\n result = processing_context.get_local(\"result\")\n ui_state = processing_context.get_local(\"ui_state\")\n \n if result:\n self.global_context.set(ContextNames.ROUND_RESULT, result)\n if ui_state:\n # Store UI state for next round\n self.global_context.set(\"MOBILE_UI_STATE\", ui_state)\n \n except Exception as e:\n self.logger.warning(f\"Failed to finalize context: {e}\")\n```\n\n### Processor Configuration Guide\n\n| Phase | Required? | When to Use | Example Strategies |\n|-------|-----------|-------------|-------------------|\n| **DATA_COLLECTION** | Optional | Agent needs observations (screenshots, sensor data) | `CustomizedScreenshotCaptureStrategy` |\n| **LLM_INTERACTION** | **Required** | All agents need LLM reasoning | `LinuxLLMInteractionStrategy`, `MobileLLMInteractionStrategy` |\n| **ACTION_EXECUTION** | **Required** | All agents need command execution | `LinuxActionExecutionStrategy`, `MobileActionExecutionStrategy` |\n| **MEMORY_UPDATE** | Recommended | Track agent history and context | `AppMemoryUpdateStrategy` |\n\n!!! warning \"Common Processor Mistakes\"\n ❌ **Don't** skip `_setup_strategies()` - processor won't execute \n ❌ **Don't** use `fail_fast=True` for all strategies - agent becomes brittle \n ❌ **Don't** forget to call `super()._finalize_processing_context()` - context won't propagate \n ✅ **Do** use `fail_fast=True` for LLM_INTERACTION - ensures valid responses \n ✅ **Do** use `fail_fast=False` for ACTION_EXECUTION - allows retry logic \n ✅ **Do** add custom middleware for debugging and logging\n\n---\n\n## Step 3: State Manager\n\n### Understanding State Manager\n\nThe **State Manager** implements the Finite State Machine (FSM) that controls agent lifecycle. It defines:\n\n- **States**: `CONTINUE`, `FINISH`, `FAIL`, etc.\n- **Transitions**: Rules for moving between states\n- **State Handlers**: Logic executed in each state\n\n### LinuxAgent States\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE: Agent Started\n \n CONTINUE --> CONTINUE: Processing\n CONTINUE --> FINISH: Task Complete\n CONTINUE --> FAIL: Error Occurred\n \n FAIL --> FINISH: Terminal State\n \n FINISH --> [*]\n \n note right of CONTINUE\n Calls agent.process(context)\n Executes processor strategies\n end note\n \n note right of FINISH\n Task completed successfully\n Returns control to orchestrator\n end note\n \n note right of FAIL\n Error occurred\n Transitions to FINISH\n end note\n```\n\n### LinuxAgent State Implementation\n\n```python\n# File: ufo/agents/states/linux_agent_state.py\n\nfrom enum import Enum\nfrom typing import TYPE_CHECKING, Dict, Optional, Type\nfrom ufo.agents.states.basic import AgentState, AgentStateManager\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import LinuxAgent\n\n\nclass LinuxAgentStatus(Enum):\n \"\"\"Status enum for LinuxAgent states.\"\"\"\n FINISH = \"FINISH\"\n CONTINUE = \"CONTINUE\"\n FAIL = \"FAIL\"\n\n\nclass LinuxAgentStateManager(AgentStateManager):\n \"\"\"\n State manager for LinuxAgent.\n Manages state registration and retrieval.\n \"\"\"\n \n _state_mapping: Dict[str, Type[LinuxAgentState]] = {}\n\n @property\n def none_state(self) -> AgentState:\n \"\"\"Return the none state.\"\"\"\n return NoneLinuxAgentState()\n\n\nclass LinuxAgentState(AgentState):\n \"\"\"\n Abstract base class for LinuxAgent states.\n All LinuxAgent states inherit from this class.\n \"\"\"\n\n async def handle(\n self, agent: \"LinuxAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"\n Handle the agent for the current step.\n \n :param agent: The LinuxAgent instance\n :param context: The global context\n \"\"\"\n pass\n\n @classmethod\n def agent_class(cls) -> Type[LinuxAgent]:\n \"\"\"Return the agent class this state belongs to.\"\"\"\n from ufo.agents.agent.customized_agent import LinuxAgent\n return LinuxAgent\n\n def next_agent(self, agent: \"LinuxAgent\") -> \"LinuxAgent\":\n \"\"\"\n Get the agent for the next step.\n Default: return same agent (no delegation).\n \n :param agent: Current agent\n :return: Next agent (typically same agent for device agents)\n \"\"\"\n return agent\n\n def next_state(self, agent: \"LinuxAgent\") -> LinuxAgentState:\n \"\"\"\n Determine next state based on agent status.\n \n :param agent: Current agent\n :return: Next state instance\n \"\"\"\n status = agent.status\n state = LinuxAgentStateManager().get_state(status)\n return state\n\n def is_round_end(self) -> bool:\n \"\"\"Check if the round ends.\"\"\"\n return False\n\n\n@LinuxAgentStateManager.register\nclass ContinueLinuxAgentState(LinuxAgentState):\n \"\"\"\n CONTINUE state: Normal execution state.\n Calls agent.process() to execute processor strategies.\n \"\"\"\n\n async def handle(\n self, agent: \"LinuxAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"\n Handle CONTINUE state by executing processor.\n \n :param agent: LinuxAgent instance\n :param context: Global context\n \"\"\"\n await agent.process(context)\n\n def is_subtask_end(self) -> bool:\n \"\"\"Subtask does not end in CONTINUE state.\"\"\"\n return False\n\n @classmethod\n def name(cls) -> str:\n \"\"\"State name matching LinuxAgentStatus enum.\"\"\"\n return LinuxAgentStatus.CONTINUE.value\n\n\n@LinuxAgentStateManager.register\nclass FinishLinuxAgentState(LinuxAgentState):\n \"\"\"\n FINISH state: Terminal state indicating task completion.\n \"\"\"\n\n def next_agent(self, agent: \"LinuxAgent\") -> \"LinuxAgent\":\n \"\"\"Return same agent (no further delegation).\"\"\"\n return agent\n\n def next_state(self, agent: \"LinuxAgent\") -> LinuxAgentState:\n \"\"\"Stay in FINISH state.\"\"\"\n return FinishLinuxAgentState()\n\n def is_subtask_end(self) -> bool:\n \"\"\"Subtask ends in FINISH state.\"\"\"\n return True\n\n def is_round_end(self) -> bool:\n \"\"\"Round ends in FINISH state.\"\"\"\n return True\n\n @classmethod\n def name(cls) -> str:\n \"\"\"State name.\"\"\"\n return LinuxAgentStatus.FINISH.value\n\n\n@LinuxAgentStateManager.register\nclass FailLinuxAgentState(LinuxAgentState):\n \"\"\"\n FAIL state: Error occurred, transition to FINISH.\n \"\"\"\n\n def next_agent(self, agent: \"LinuxAgent\") -> \"LinuxAgent\":\n \"\"\"Return same agent.\"\"\"\n return agent\n\n def next_state(self, agent: \"LinuxAgent\") -> LinuxAgentState:\n \"\"\"Transition to FINISH after failure.\"\"\"\n return FinishLinuxAgentState()\n\n def is_round_end(self) -> bool:\n \"\"\"Round ends after failure.\"\"\"\n return True\n\n def is_subtask_end(self) -> bool:\n \"\"\"Subtask ends after failure.\"\"\"\n return True\n\n @classmethod\n def name(cls) -> str:\n \"\"\"State name.\"\"\"\n return LinuxAgentStatus.FAIL.value\n\n\n@LinuxAgentStateManager.register\nclass NoneLinuxAgentState(LinuxAgentState):\n \"\"\"\n NONE state: Initial/default state, transitions to FINISH.\n \"\"\"\n\n def next_agent(self, agent: \"LinuxAgent\") -> \"LinuxAgent\":\n \"\"\"Return same agent.\"\"\"\n return agent\n\n def next_state(self, agent: \"LinuxAgent\") -> LinuxAgentState:\n \"\"\"Transition to FINISH.\"\"\"\n return FinishLinuxAgentState()\n\n def is_subtask_end(self) -> bool:\n \"\"\"Subtask ends in NONE state.\"\"\"\n return True\n\n def is_round_end(self) -> bool:\n \"\"\"Round ends in NONE state.\"\"\"\n return True\n\n @classmethod\n def name(cls) -> str:\n \"\"\"Empty name for NONE state.\"\"\"\n return \"\"\n```\n\n### Creating MobileAgent States\n\n```python\n# File: ufo/agents/states/mobile_agent_state.py\n\nfrom enum import Enum\nfrom typing import TYPE_CHECKING, Dict, Optional, Type\nfrom ufo.agents.states.basic import AgentState, AgentStateManager\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import MobileAgent\n\n\nclass MobileAgentStatus(Enum):\n \"\"\"Status enum for MobileAgent states.\"\"\"\n FINISH = \"FINISH\"\n CONTINUE = \"CONTINUE\"\n FAIL = \"FAIL\"\n WAITING = \"WAITING\" # Waiting for app to load\n\n\nclass MobileAgentStateManager(AgentStateManager):\n \"\"\"State manager for MobileAgent.\"\"\"\n \n _state_mapping: Dict[str, Type[MobileAgentState]] = {}\n\n @property\n def none_state(self) -> AgentState:\n \"\"\"Return the none state.\"\"\"\n return NoneMobileAgentState()\n\n\nclass MobileAgentState(AgentState):\n \"\"\"Abstract base class for MobileAgent states.\"\"\"\n\n async def handle(\n self, agent: \"MobileAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Handle the agent for the current step.\"\"\"\n pass\n\n @classmethod\n def agent_class(cls) -> Type[MobileAgent]:\n \"\"\"Return the agent class.\"\"\"\n from ufo.agents.agent.customized_agent import MobileAgent\n return MobileAgent\n\n def next_agent(self, agent: \"MobileAgent\") -> \"MobileAgent\":\n \"\"\"Get next agent (same agent for device agents).\"\"\"\n return agent\n\n def next_state(self, agent: \"MobileAgent\") -> MobileAgentState:\n \"\"\"Determine next state based on status.\"\"\"\n status = agent.status\n state = MobileAgentStateManager().get_state(status)\n return state\n\n def is_round_end(self) -> bool:\n \"\"\"Check if round ends.\"\"\"\n return False\n\n\n@MobileAgentStateManager.register\nclass ContinueMobileAgentState(MobileAgentState):\n \"\"\"CONTINUE state for MobileAgent.\"\"\"\n\n async def handle(\n self, agent: \"MobileAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Execute processor strategies.\"\"\"\n await agent.process(context)\n\n def is_subtask_end(self) -> bool:\n return False\n\n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.CONTINUE.value\n\n\n@MobileAgentStateManager.register\nclass FinishMobileAgentState(MobileAgentState):\n \"\"\"FINISH state for MobileAgent.\"\"\"\n\n def next_state(self, agent: \"MobileAgent\") -> MobileAgentState:\n return FinishMobileAgentState()\n\n def is_subtask_end(self) -> bool:\n return True\n\n def is_round_end(self) -> bool:\n return True\n\n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.FINISH.value\n\n\n@MobileAgentStateManager.register\nclass FailMobileAgentState(MobileAgentState):\n \"\"\"FAIL state for MobileAgent.\"\"\"\n\n def next_state(self, agent: \"MobileAgent\") -> MobileAgentState:\n return FinishMobileAgentState()\n\n def is_round_end(self) -> bool:\n return True\n\n def is_subtask_end(self) -> bool:\n return True\n\n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.FAIL.value\n\n\n@MobileAgentStateManager.register\nclass WaitingMobileAgentState(MobileAgentState):\n \"\"\"\n WAITING state: Wait for app to load or animation to complete.\n \"\"\"\n\n async def handle(\n self, agent: \"MobileAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Wait and then transition to CONTINUE.\"\"\"\n import asyncio\n await asyncio.sleep(2) # Wait 2 seconds\n agent.status = MobileAgentStatus.CONTINUE.value\n\n def is_subtask_end(self) -> bool:\n return False\n\n @classmethod\n def name(cls) -> str:\n return MobileAgentStatus.WAITING.value\n\n\n@MobileAgentStateManager.register\nclass NoneMobileAgentState(MobileAgentState):\n \"\"\"NONE state for MobileAgent.\"\"\"\n\n def next_state(self, agent: \"MobileAgent\") -> MobileAgentState:\n return FinishMobileAgentState()\n\n def is_subtask_end(self) -> bool:\n return True\n\n def is_round_end(self) -> bool:\n return True\n\n @classmethod\n def name(cls) -> str:\n return \"\"\n```\n\n### State Design Guidelines\n\n| State | When to Use | Required Methods | Terminal? |\n|-------|-------------|------------------|-----------|\n| **CONTINUE** | Normal execution | `handle()` calls `agent.process()` | No |\n| **FINISH** | Task complete | `is_subtask_end()` → `True` | Yes |\n| **FAIL** | Error occurred | `next_state()` → `FINISH` | Yes |\n| **WAITING** | Async delays | `handle()` with `await asyncio.sleep()` | No |\n| **NONE** | Default/initial | `next_state()` → `FINISH` | Yes |\n\n!!! tip \"State Design Best Practices\"\n - ✅ Always register states with `@StateManager.register`\n - ✅ Implement `name()` to match status enum value\n - ✅ Call `agent.process()` in `CONTINUE.handle()`\n - ✅ Set `is_round_end()` = `True` for terminal states\n - ✅ Transition `FAIL` → `FINISH` for graceful termination\n - ❌ Don't create too many states - keep it simple\n - ❌ Don't call processor directly - use `agent.process()`\n\n---\n\n## Step 4: Processing Strategies\n\n### Understanding Strategies\n\n**Strategies** are modular execution units that implement specific phases of the processing pipeline. Each strategy:\n\n- Executes independently within its phase\n- Declares dependencies using `@depends_on` decorator\n- Provides results using `@provides` decorator\n- Returns `ProcessingResult` with success/failure status\n\n### LinuxAgent Strategies\n\n#### Strategy 1: LLM Interaction\n\n```python\n# File: ufo/agents/processors/strategies/linux_agent_strategy.py\n\nfrom typing import TYPE_CHECKING\nfrom ufo.agents.processors.strategies.app_agent_processing_strategy import (\n AppLLMInteractionStrategy,\n)\nfrom ufo.agents.processors.context.processing_context import (\n ProcessingContext,\n ProcessingResult,\n ProcessingPhase,\n)\nfrom ufo.agents.processors.core.strategy_dependency import depends_on, provides\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import LinuxAgent\n\n\n@depends_on(\"request\") # Requires \"request\" in context\n@provides(\n \"parsed_response\", # Provides LLM parsed response\n \"response_text\", # Raw LLM response text\n \"llm_cost\", # LLM API cost\n \"prompt_message\", # Prompt sent to LLM\n \"action\", # Action to execute\n \"thought\", # LLM reasoning\n \"comment\", # LLM comment\n)\nclass LinuxLLMInteractionStrategy(AppLLMInteractionStrategy):\n \"\"\"\n Strategy for LLM interaction with Linux Agent.\n \n Constructs prompts with Linux context, calls LLM,\n parses response into structured action.\n \"\"\"\n\n def __init__(self, fail_fast: bool = True) -> None:\n \"\"\"\n Initialize Linux LLM interaction strategy.\n \n :param fail_fast: Raise exceptions immediately on errors\n \"\"\"\n super().__init__(fail_fast=fail_fast)\n\n async def execute(\n self, agent: \"LinuxAgent\", context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"\n Execute LLM interaction for LinuxAgent.\n \n :param agent: LinuxAgent instance\n :param context: Processing context with request data\n :return: ProcessingResult with parsed LLM response\n \"\"\"\n try:\n # Step 1: Extract request from context\n request = context.get(\"request\")\n plan = self._get_prev_plan(agent)\n\n # Step 2: Build comprehensive prompt\n self.logger.info(\"Building Linux Agent prompt\")\n \n # Get blackboard context (if multi-agent)\n blackboard_prompt = []\n if not agent.blackboard.is_empty():\n blackboard_prompt = agent.blackboard.blackboard_to_prompt()\n\n # Construct prompt message\n prompt_message = agent.message_constructor(\n dynamic_examples=[],\n dynamic_knowledge=\"\",\n plan=plan,\n request=request,\n blackboard_prompt=blackboard_prompt,\n last_success_actions=self._get_last_success_actions(agent),\n )\n\n # Step 3: Get LLM response\n self.logger.info(\"Getting LLM response for Linux Agent\")\n response_text, llm_cost = await self._get_llm_response(\n agent, prompt_message\n )\n\n # Step 4: Parse and validate response\n self.logger.info(\"Parsing Linux Agent response\")\n parsed_response = self._parse_app_response(agent, response_text)\n\n # Step 5: Extract structured data\n structured_data = parsed_response.model_dump()\n\n return ProcessingResult(\n success=True,\n data={\n \"parsed_response\": parsed_response,\n \"response_text\": response_text,\n \"llm_cost\": llm_cost,\n \"prompt_message\": prompt_message,\n **structured_data, # action, thought, comment, etc.\n },\n phase=ProcessingPhase.LLM_INTERACTION,\n )\n\n except Exception as e:\n error_msg = f\"Linux LLM interaction failed: {str(e)}\"\n self.logger.error(error_msg)\n return self.handle_error(e, ProcessingPhase.LLM_INTERACTION, context)\n```\n\n#### Strategy 2: Action Execution\n\n```python\n# File: ufo/agents/processors/strategies/linux_agent_strategy.py\n\n@depends_on(\"parsed_response\", \"command_dispatcher\")\n@provides(\"execution_result\", \"action_info\", \"control_log\", \"status\")\nclass LinuxActionExecutionStrategy(AppActionExecutionStrategy):\n \"\"\"\n Strategy for executing actions in LinuxAgent.\n \n Dispatches shell commands to Linux MCP server,\n captures results, and creates action logs.\n \"\"\"\n\n def __init__(self, fail_fast: bool = False) -> None:\n \"\"\"\n Initialize Linux action execution strategy.\n \n :param fail_fast: Raise exceptions immediately (typically False)\n \"\"\"\n super().__init__(fail_fast=fail_fast)\n\n async def execute(\n self, agent: \"LinuxAgent\", context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"\n Execute Linux Agent actions.\n \n :param agent: LinuxAgent instance\n :param context: Processing context with parsed response\n :return: ProcessingResult with execution results\n \"\"\"\n try:\n # Step 1: Extract context variables\n parsed_response = context.get_local(\"parsed_response\")\n command_dispatcher = context.global_context.command_dispatcher\n\n if not parsed_response:\n return ProcessingResult(\n success=True,\n data={\"message\": \"No response for action execution\"},\n phase=ProcessingPhase.ACTION_EXECUTION,\n )\n\n # Step 2: Execute the action via command dispatcher\n execution_results = await self._execute_app_action(\n command_dispatcher, parsed_response.action\n )\n\n # Step 3: Create action info for memory tracking\n actions = self._create_action_info(\n parsed_response.action,\n execution_results,\n )\n\n # Step 4: Print action info (for debugging)\n action_info = ListActionCommandInfo(actions)\n action_info.color_print()\n\n # Step 5: Create control log\n control_log = action_info.get_target_info()\n\n status = (\n parsed_response.action.status\n if isinstance(parsed_response.action, ActionCommandInfo)\n else action_info.status\n )\n\n return ProcessingResult(\n success=True,\n data={\n \"execution_result\": execution_results,\n \"action_info\": action_info,\n \"control_log\": control_log,\n \"status\": status,\n },\n phase=ProcessingPhase.ACTION_EXECUTION,\n )\n\n except Exception as e:\n error_msg = f\"Linux action execution failed: {str(e)}\"\n self.logger.error(error_msg)\n return self.handle_error(e, ProcessingPhase.ACTION_EXECUTION, context)\n```\n\n### Creating MobileAgent Strategies\n\n```python\n# File: ufo/agents/processors/strategies/mobile_agent_strategy.py\n\nfrom typing import TYPE_CHECKING\nfrom ufo.agents.processors.strategies.app_agent_processing_strategy import (\n AppLLMInteractionStrategy,\n AppActionExecutionStrategy,\n)\nfrom ufo.agents.processors.context.processing_context import (\n ProcessingContext,\n ProcessingResult,\n ProcessingPhase,\n)\nfrom ufo.agents.processors.core.strategy_dependency import depends_on, provides\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import MobileAgent\n\n\n@depends_on(\"request\", \"screenshot\", \"ui_tree\")\n@provides(\n \"parsed_response\",\n \"response_text\",\n \"llm_cost\",\n \"prompt_message\",\n \"action\",\n \"thought\",\n \"comment\",\n)\nclass MobileLLMInteractionStrategy(AppLLMInteractionStrategy):\n \"\"\"\n LLM interaction strategy for MobileAgent.\n \n Handles mobile UI screenshots and hierarchy for LLM understanding.\n \"\"\"\n\n def __init__(self, fail_fast: bool = True) -> None:\n super().__init__(fail_fast=fail_fast)\n\n async def execute(\n self, agent: \"MobileAgent\", context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Execute LLM interaction for mobile UI.\"\"\"\n try:\n # Extract mobile-specific context\n request = context.get(\"request\")\n screenshot = context.get_local(\"screenshot\")\n ui_tree = context.get_local(\"ui_tree\")\n \n self.logger.info(f\"Building Mobile Agent prompt for {agent.platform}\")\n \n # Build prompt with mobile context\n prompt_message = agent.message_constructor(\n dynamic_examples=[],\n dynamic_knowledge=\"\",\n plan=self._get_prev_plan(agent),\n request=request,\n screenshot=screenshot,\n ui_tree=ui_tree,\n blackboard_prompt=(\n agent.blackboard.blackboard_to_prompt()\n if not agent.blackboard.is_empty() else []\n ),\n last_success_actions=self._get_last_success_actions(agent),\n )\n\n # Get LLM response\n response_text, llm_cost = await self._get_llm_response(\n agent, prompt_message\n )\n\n # Parse response\n parsed_response = self._parse_app_response(agent, response_text)\n\n return ProcessingResult(\n success=True,\n data={\n \"parsed_response\": parsed_response,\n \"response_text\": response_text,\n \"llm_cost\": llm_cost,\n \"prompt_message\": prompt_message,\n **parsed_response.model_dump(),\n },\n phase=ProcessingPhase.LLM_INTERACTION,\n )\n\n except Exception as e:\n self.logger.error(f\"Mobile LLM interaction failed: {e}\")\n return self.handle_error(e, ProcessingPhase.LLM_INTERACTION, context)\n\n\n@depends_on(\"parsed_response\", \"command_dispatcher\")\n@provides(\"execution_result\", \"action_info\", \"control_log\", \"status\")\nclass MobileActionExecutionStrategy(AppActionExecutionStrategy):\n \"\"\"\n Action execution strategy for MobileAgent.\n \n Executes mobile-specific actions (tap, swipe, type, etc.)\n via Mobile MCP server.\n \"\"\"\n\n def __init__(self, fail_fast: bool = False) -> None:\n super().__init__(fail_fast=fail_fast)\n\n async def execute(\n self, agent: \"MobileAgent\", context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"Execute mobile actions.\"\"\"\n try:\n parsed_response = context.get_local(\"parsed_response\")\n command_dispatcher = context.global_context.command_dispatcher\n\n if not parsed_response:\n return ProcessingResult(\n success=True,\n data={\"message\": \"No action to execute\"},\n phase=ProcessingPhase.ACTION_EXECUTION,\n )\n\n # Execute mobile action\n execution_results = await self._execute_app_action(\n command_dispatcher, parsed_response.action\n )\n\n # Create action info\n actions = self._create_action_info(\n parsed_response.action,\n execution_results,\n )\n\n action_info = ListActionCommandInfo(actions)\n action_info.color_print()\n\n control_log = action_info.get_target_info()\n status = action_info.status\n\n return ProcessingResult(\n success=True,\n data={\n \"execution_result\": execution_results,\n \"action_info\": action_info,\n \"control_log\": control_log,\n \"status\": status,\n },\n phase=ProcessingPhase.ACTION_EXECUTION,\n )\n\n except Exception as e:\n self.logger.error(f\"Mobile action execution failed: {e}\")\n return self.handle_error(e, ProcessingPhase.ACTION_EXECUTION, context)\n\n\n# Middleware for mobile-specific logging\nclass MobileLoggingMiddleware(AppAgentLoggingMiddleware):\n \"\"\"Logging middleware for MobileAgent.\"\"\"\n\n def starting_message(self, context: ProcessingContext) -> str:\n \"\"\"Return starting message.\"\"\"\n request = context.get_local(\"request\")\n return f\"Executing mobile task: [{request}]\"\n```\n\n### Strategy Design Checklist\n\n- [ ] Use `@depends_on()` decorator to declare dependencies\n- [ ] Use `@provides()` decorator to declare outputs\n- [ ] Return `ProcessingResult` with success status\n- [ ] Handle exceptions gracefully (log, return error result)\n- [ ] Respect `fail_fast` setting\n- [ ] Use `self.logger` for debugging\n- [ ] Call `self.handle_error()` in except blocks\n\n---\n\n## Step 5: Prompter\n\n### Understanding the Prompter\n\nThe **Prompter** constructs prompts for LLM interaction. It:\n\n- Loads prompt templates from YAML files\n- Constructs system and user messages\n- Inserts dynamic context (request, plan, examples)\n- Formats API/tool descriptions\n\n### LinuxAgent Prompter\n\n```python\n# File: ufo/prompter/customized/linux_agent_prompter.py\n\nimport json\nfrom typing import Any, Dict, List\nfrom ufo.prompter.agent_prompter import AppAgentPrompter\n\n\nclass LinuxAgentPrompter(AppAgentPrompter):\n \"\"\"\n Prompter for LinuxAgent.\n \n Constructs prompts for shell command generation.\n \"\"\"\n\n def __init__(\n self,\n prompt_template: str,\n example_prompt_template: str,\n ):\n \"\"\"\n Initialize LinuxAgentPrompter.\n \n :param prompt_template: Path to main prompt YAML\n :param example_prompt_template: Path to example prompt YAML\n \"\"\"\n super().__init__(None, prompt_template, example_prompt_template)\n self.api_prompt_template = None\n\n def system_prompt_construction(\n self, additional_examples: List[str] = []\n ) -> str:\n \"\"\"\n Construct system prompt for LinuxAgent.\n \n :param additional_examples: Additional examples to include\n :return: System prompt string\n \"\"\"\n # Format API descriptions\n apis = self.api_prompt_helper(verbose=1)\n \n # Format examples\n examples = self.examples_prompt_helper(\n additional_examples=additional_examples\n )\n\n # Fill template\n return self.prompt_template[\"system\"].format(\n apis=apis, examples=examples\n )\n\n def user_prompt_construction(\n self,\n prev_plan: List[str],\n user_request: str,\n retrieved_docs: str = \"\",\n last_success_actions: List[Dict[str, Any]] = [],\n ) -> str:\n \"\"\"\n Construct user prompt for LinuxAgent.\n \n :param prev_plan: Previous execution plan\n :param user_request: User's request\n :param retrieved_docs: Retrieved documentation (optional)\n :param last_success_actions: Last successful actions\n :return: User prompt string\n \"\"\"\n prompt = self.prompt_template[\"user\"].format(\n prev_plan=json.dumps(prev_plan),\n user_request=user_request,\n retrieved_docs=retrieved_docs,\n last_success_actions=json.dumps(last_success_actions),\n )\n\n return prompt\n\n def user_content_construction(\n self,\n prev_plan: List[str],\n user_request: str,\n retrieved_docs: str = \"\",\n last_success_actions: List[Dict[str, Any]] = [],\n ) -> List[Dict[str, str]]:\n \"\"\"\n Construct user content for LLM (supports multi-modal).\n \n :param prev_plan: Previous plan\n :param user_request: User request\n :param retrieved_docs: Retrieved docs\n :param last_success_actions: Last actions\n :return: List of content dicts\n \"\"\"\n user_content = []\n\n user_content.append({\n \"type\": \"text\",\n \"text\": self.user_prompt_construction(\n prev_plan=prev_plan,\n user_request=user_request,\n retrieved_docs=retrieved_docs,\n last_success_actions=last_success_actions,\n ),\n })\n\n return user_content\n```\n\n### Creating MobileAgent Prompter\n\n```python\n# File: ufo/prompter/customized/mobile_agent_prompter.py\n\nimport json\nfrom typing import Any, Dict, List\nfrom ufo.prompter.agent_prompter import AppAgentPrompter\n\n\nclass MobileAgentPrompter(AppAgentPrompter):\n \"\"\"\n Prompter for MobileAgent.\n \n Handles mobile UI screenshots and hierarchy for LLM prompts.\n \"\"\"\n\n def __init__(\n self,\n prompt_template: str,\n example_prompt_template: str,\n ):\n \"\"\"\n Initialize MobileAgentPrompter.\n \n :param prompt_template: Path to main prompt YAML\n :param example_prompt_template: Path to example prompt YAML\n \"\"\"\n super().__init__(None, prompt_template, example_prompt_template)\n self.api_prompt_template = None\n\n def system_prompt_construction(\n self, additional_examples: List[str] = []\n ) -> str:\n \"\"\"Construct system prompt for MobileAgent.\"\"\"\n apis = self.api_prompt_helper(verbose=1)\n examples = self.examples_prompt_helper(\n additional_examples=additional_examples\n )\n\n return self.prompt_template[\"system\"].format(\n apis=apis, examples=examples\n )\n\n def user_prompt_construction(\n self,\n prev_plan: List[str],\n user_request: str,\n ui_tree: str = \"\",\n retrieved_docs: str = \"\",\n last_success_actions: List[Dict[str, Any]] = [],\n ) -> str:\n \"\"\"\n Construct user prompt with mobile UI context.\n \n :param prev_plan: Previous plan\n :param user_request: User request\n :param ui_tree: Mobile UI hierarchy (XML/JSON)\n :param retrieved_docs: Retrieved docs\n :param last_success_actions: Last actions\n :return: User prompt string\n \"\"\"\n prompt = self.prompt_template[\"user\"].format(\n prev_plan=json.dumps(prev_plan),\n user_request=user_request,\n ui_tree=ui_tree, # Mobile-specific\n retrieved_docs=retrieved_docs,\n last_success_actions=json.dumps(last_success_actions),\n )\n\n return prompt\n\n def user_content_construction(\n self,\n prev_plan: List[str],\n user_request: str,\n screenshot: Any = None, # Mobile screenshot\n ui_tree: str = \"\",\n retrieved_docs: str = \"\",\n last_success_actions: List[Dict[str, Any]] = [],\n ) -> List[Dict[str, str]]:\n \"\"\"\n Construct user content with screenshot for vision LLMs.\n \n :param prev_plan: Previous plan\n :param user_request: User request\n :param screenshot: Screenshot image (base64 or path)\n :param ui_tree: UI hierarchy\n :param retrieved_docs: Retrieved docs\n :param last_success_actions: Last actions\n :return: List of content dicts (text + image)\n \"\"\"\n user_content = []\n\n # Add text prompt\n user_content.append({\n \"type\": \"text\",\n \"text\": self.user_prompt_construction(\n prev_plan=prev_plan,\n user_request=user_request,\n ui_tree=ui_tree,\n retrieved_docs=retrieved_docs,\n last_success_actions=last_success_actions,\n ),\n })\n\n # Add screenshot if available (for vision LLMs)\n if screenshot:\n user_content.append({\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": f\"data:image/png;base64,{screenshot}\"\n },\n })\n\n return user_content\n```\n\n### Prompter Best Practices\n\n- ✅ Inherit from `AppAgentPrompter` for standard structure\n- ✅ Use `self.prompt_template` and `self.example_prompt_template`\n- ✅ Implement `system_prompt_construction()` and `user_prompt_construction()`\n- ✅ Use `user_content_construction()` for multi-modal content\n- ✅ Format examples with `examples_prompt_helper()`\n- ✅ Format APIs with `api_prompt_helper()`\n- ❌ Don't hardcode prompts - use YAML templates\n\n---\n\n## Testing Your Implementation\n\n### Unit Test: Agent Class\n\n```python\n# File: tests/unit/test_mobile_agent.py\n\nimport pytest\nfrom ufo.agents.agent.customized_agent import MobileAgent\nfrom ufo.agents.processors.customized.customized_agent_processor import (\n MobileAgentProcessor\n)\n\n\nclass TestMobileAgent:\n \"\"\"Unit tests for MobileAgent.\"\"\"\n\n @pytest.fixture\n def agent(self):\n \"\"\"Create test MobileAgent instance.\"\"\"\n return MobileAgent(\n name=\"test_mobile_agent\",\n main_prompt=\"ufo/prompts/third_party/mobile_agent.yaml\",\n example_prompt=\"ufo/prompts/third_party/mobile_agent_example.yaml\",\n platform=\"android\",\n )\n\n def test_agent_initialization(self, agent):\n \"\"\"Test agent initializes correctly.\"\"\"\n assert agent.name == \"test_mobile_agent\"\n assert agent.platform == \"android\"\n assert agent.prompter is not None\n assert agent.blackboard is not None\n\n def test_processor_registration(self, agent):\n \"\"\"Test processor is registered correctly.\"\"\"\n assert hasattr(agent, \"_processor_cls\")\n assert agent._processor_cls == MobileAgentProcessor\n\n def test_default_state(self, agent):\n \"\"\"Test default state is set.\"\"\"\n from ufo.agents.states.mobile_agent_state import ContinueMobileAgentState\n assert isinstance(agent.default_state, ContinueMobileAgentState)\n```\n\n### Integration Test: Full Pipeline\n\n```python\n# File: tests/integration/test_mobile_agent_pipeline.py\n\nimport pytest\nfrom ufo.agents.agent.customized_agent import MobileAgent\nfrom ufo.module.context import Context\n\n\nclass TestMobileAgentPipeline:\n \"\"\"Integration tests for MobileAgent pipeline.\"\"\"\n\n @pytest.fixture\n async def agent_with_context(self):\n \"\"\"Create agent with context.\"\"\"\n agent = MobileAgent(\n name=\"test_agent\",\n main_prompt=\"ufo/prompts/third_party/mobile_agent.yaml\",\n example_prompt=\"ufo/prompts/third_party/mobile_agent_example.yaml\",\n platform=\"android\",\n )\n \n context = Context()\n context.set(\"request\", \"Tap the login button\")\n \n return agent, context\n\n @pytest.mark.asyncio\n async def test_processor_execution(self, agent_with_context):\n \"\"\"Test processor executes all strategies.\"\"\"\n agent, context = agent_with_context\n \n # Execute processor\n processor = agent._processor_cls(agent, context)\n result = await processor.process()\n \n # Verify strategies executed\n assert result is not None\n assert \"parsed_response\" in context.get_all_local()\n```\n\n---\n\n## Summary\n\n**What You've Built**:\n\n- **Agent Class** - MobileAgent with registration and initialization\n- **Processor** - MobileAgentProcessor with strategy orchestration\n- **State Manager** - MobileAgentStateManager with FSM states\n- **Strategies** - LLM and action execution strategies\n- **Prompter** - MobileAgentPrompter for prompt construction\n\n**Next Step**: [Part 2: MCP Server Development →](mcp_server.md)\n\n---\n\n## Related Documentation\n\n- **[Agent Architecture](../../infrastructure/agents/overview.md)** - Three-layer architecture\n- **[Processor Design](../../infrastructure/agents/design/processor.md)** - Processor deep dive\n- **[Strategy Pattern](../../infrastructure/agents/design/strategy.md)** - Strategy implementation\n- **[State Machine](../../infrastructure/agents/design/state.md)** - State management\n\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/example_mobile_agent.md",
"content": "# Part 6: Complete Example - MobileAgent\n\n**Note**: This comprehensive hands-on tutorial is currently under development. Check back soon for a complete MobileAgent implementation walkthrough.\n\n## What You'll Build\n\nA fully functional **MobileAgent** that can:\n\n- Control Android/iOS devices\n- Perform UI automation\n- Execute touch gestures (tap, swipe, type)\n- Capture screenshots and UI hierarchy\n- Integrate with Galaxy orchestration\n\n## Planned Content\n\n### 1. Platform-Specific Setup\n\n#### Android\n- ADB (Android Debug Bridge) integration\n- UI Automator framework\n- Accessibility services\n\n#### iOS\n- XCTest framework\n- Accessibility API\n- Instrument tools\n\n### 2. Complete Implementation\n\n- Agent class\n- Processor and strategies\n- State manager\n- MCP server with mobile tools\n- Prompter for mobile UI\n\n### 3. Advanced Features\n\n- Multi-device coordination\n- App-specific automation\n- Error recovery strategies\n- Performance optimization\n\n## Temporary Reference\n\nFor now, study the **LinuxAgent** implementation as a complete reference:\n\n### Key Files\n\n| Component | File Path |\n|-----------|-----------|\n| Agent Class | `ufo/agents/agent/customized_agent.py` |\n| Processor | `ufo/agents/processors/customized/customized_agent_processor.py` |\n| Strategies | `ufo/agents/processors/strategies/linux_agent_strategy.py` |\n| States | `ufo/agents/states/linux_agent_state.py` |\n| Prompter | `ufo/prompter/customized/linux_agent_prompter.py` |\n| MCP Server | `ufo/client/mcp/http_servers/linux_mcp_server.py` |\n\n### Quick Start Template\n\n```python\n# Minimal MobileAgent structure (to be expanded)\n\n@AgentRegistry.register(\n agent_name=\"MobileAgent\",\n third_party=True,\n processor_cls=MobileAgentProcessor\n)\nclass MobileAgent(CustomizedAgent):\n def __init__(self, name, main_prompt, example_prompt):\n super().__init__(name, main_prompt, example_prompt,\n process_name=None, app_root_name=None, is_visual=None)\n self._blackboard = Blackboard()\n self.set_state(self.default_state)\n self._context_provision_executed = False\n \n @property\n def default_state(self):\n return ContinueMobileAgentState()\n \n def message_constructor(\n self,\n dynamic_examples,\n dynamic_knowledge,\n plan,\n request,\n installed_apps,\n current_controls,\n screenshot_url=None,\n annotated_screenshot_url=None,\n blackboard_prompt=None,\n last_success_actions=None,\n ):\n # Construct prompt for LLM with mobile-specific context\n return self.prompter.prompt_construction(...)\n```\n\n## Related Documentation\n\n- **[Agent Architecture](../../infrastructure/agents/overview.md)** - Architecture overview\n- **[Agent Types](../../infrastructure/agents/agent_types.md)** - Platform implementations\n- **[Linux Quick Start](../../getting_started/quick_start_linux.md)** - LinuxAgent deployment\n\n---\n\n**Previous**: [← Part 5: Testing & Debugging](testing.md) \n**Back to Index**: [Tutorial Series](index.md)\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/index.md",
"content": "# Creating Device Agents - Tutorial Series\n\nThis tutorial series teaches you how to create new device agents for UFO³, using **LinuxAgent** as a reference implementation.\n\n## 📚 Tutorial Structure\n\n### [Part 0: Overview](overview.md)\n**Introduction to device agents and architecture overview**\n\n- Understanding device agents vs third-party agents\n- Server-client architecture\n- LinuxAgent as reference implementation\n- Tutorial roadmap\n\n**Time**: 15 minutes | **Difficulty**: ⭐\n\n---\n\n### [Part 1: Core Components](core_components.md)\n**Building server-side components**\n\n- Agent Class implementation\n- Processor and strategy orchestration\n- State Manager and FSM\n- Processing Strategies (LLM, Action)\n- Prompter for LLM interaction\n\n**Time**: 45 minutes | **Difficulty**: ⭐⭐⭐\n\n---\n\n### [Part 2: MCP Server Development](mcp_server.md)\n**Creating platform-specific MCP servers** *(Placeholder - Under Development)*\n\n- MCP server architecture\n- Defining MCP tools\n- Command execution logic\n- Error handling and validation\n\n**Time**: 30 minutes | **Difficulty**: ⭐⭐\n\n---\n\n### [Part 3: Client Setup](client_setup.md)\n**Setting up the device client** *(Placeholder - Under Development)*\n\n- Client initialization and configuration\n- MCP server manager integration\n- WebSocket connection setup\n- Platform detection\n\n**Time**: 20 minutes | **Difficulty**: ⭐⭐\n\n---\n\n### [Part 4: Configuration & Deployment](configuration.md)\n**Configuring and deploying your agent** *(Placeholder - Under Development)*\n\n- `third_party.yaml` configuration\n- `devices.yaml` device registration\n- Prompt template creation\n- Deployment steps\n- Galaxy integration\n\n**Time**: 25 minutes | **Difficulty**: ⭐⭐\n\n---\n\n### [Part 5: Testing & Debugging](testing.md)\n**Testing and debugging your implementation** *(Placeholder - Under Development)*\n\n- Unit testing strategies\n- Integration testing\n- Debugging techniques\n- Common issues and solutions\n\n**Time**: 30 minutes | **Difficulty**: ⭐⭐⭐\n\n---\n\n### [Part 6: Complete Example: MobileAgent](example_mobile_agent.md)\n**Hands-on walkthrough creating MobileAgent** *(Placeholder - Under Development)*\n\n- Step-by-step implementation\n- Android/iOS platform specifics\n- UI Automator integration\n- Complete working example\n\n**Time**: 60 minutes | **Difficulty**: ⭐⭐⭐⭐\n\n---\n\n## Quick Navigation\n\n| I Want To... | Go To |\n|--------------|-------|\n| Understand device agent architecture | [Overview](overview.md#understanding-device-agents) |\n| Study LinuxAgent implementation | [Overview](overview.md#linuxagent-reference-implementation) |\n| Create Agent Class | [Core Components - Step 1](core_components.md#step-1-agent-class) |\n| Build Processor | [Core Components - Step 2](core_components.md#step-2-processor) |\n| Implement State Machine | [Core Components - Step 3](core_components.md#step-3-state-manager) |\n| Write Processing Strategies | [Core Components - Step 4](core_components.md#step-4-processing-strategies) |\n| Create Prompter | [Core Components - Step 5](core_components.md#step-5-prompter) |\n| Build MCP Server | [MCP Server](mcp_server.md) *(placeholder)* |\n| Setup Client | [Client Setup](client_setup.md) *(placeholder)* |\n| Configure & Deploy | [Configuration](configuration.md) *(placeholder)* |\n| Test & Debug | [Testing](testing.md) *(placeholder)* |\n| Complete Example | [MobileAgent Example](example_mobile_agent.md) *(placeholder)* |\n\n---\n\n## Prerequisites\n\nBefore starting, ensure you have:\n\n- ✅ Python 3.10+\n- ✅ UFO³ repository cloned\n- ✅ Basic understanding of async programming\n- ✅ Familiarity with [Agent Architecture](../../infrastructure/agents/overview.md)\n\n---\n\n## Learning Path\n\n```mermaid\ngraph LR\n A[Overview ✅ Complete] --> B[Core Components ✅ Complete]\n B --> C[MCP Server 📝 Placeholder]\n C --> D[Client Setup 📝 Placeholder]\n D --> E[Configuration 📝 Placeholder]\n E --> F[Testing 📝 Placeholder]\n F --> G[Complete Example 📝 Placeholder]\n \n style A fill:#c8e6c9\n style B fill:#c8e6c9\n style C fill:#fff3e0\n style D fill:#fff3e0\n style E fill:#fff3e0\n style F fill:#fff3e0\n style G fill:#fff3e0\n```\n\n**Recommended Path**:\n1. ✅ **Completed**: [Overview](overview.md) - Understand architecture\n2. ✅ **Completed**: [Core Components](core_components.md) - Build server-side components\n3. 📝 **Placeholder**: [MCP Server](mcp_server.md) - Create device commands\n4. 📝 **Placeholder**: [Client Setup](client_setup.md) - Setup device client\n5. 📝 **Placeholder**: [Configuration](configuration.md) - Configure and deploy\n6. 📝 **Placeholder**: [Testing](testing.md) - Test and debug\n7. 📝 **Placeholder**: [Complete Example](example_mobile_agent.md) - Full MobileAgent implementation\n\n---\n\n## Additional Resources\n\n- **[Agent Architecture Overview](../../infrastructure/agents/overview.md)** - Three-layer architecture\n- **[Agent Types](../../infrastructure/agents/agent_types.md)** - Platform-specific implementations\n- **[Linux Quick Start](../../getting_started/quick_start_linux.md)** - Deploy LinuxAgent\n- **[Creating Third-Party Agents](../creating_third_party_agents.md)** - Related tutorial\n- **[MCP Overview](../../mcp/overview.md)** - Model Context Protocol\n- **[Server Overview](../../server/overview.md)** - Server architecture\n- **[Client Overview](../../client/overview.md)** - Client architecture\n\n---\n\n## Getting Help\n\nIf you encounter issues:\n\n1. 📖 Review the [FAQ](../../faq.md)\n2. 🐛 Check [troubleshooting guides](core_components.md#testing-your-implementation)\n3. 💬 Ask in GitHub Discussions\n4. 🐞 Report bugs on GitHub Issues\n\n---\n\n## Contributing\n\nFound an issue or want to improve these tutorials?\n\n- 📝 Submit a PR with improvements\n- 💡 Suggest new topics\n- 🔍 Report errors or unclear sections\n\n---\n\n**Ready to start?** → [Begin with Overview](overview.md)\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/mcp_server.md",
"content": "# Part 2: MCP Server Development\n\nThis tutorial teaches you how to create a **platform-specific MCP (Model Context Protocol) server** that enables your device agent to execute commands on the target device. We'll use **LinuxAgent's MCP server** as reference implementation.\n\n---\n\n## Table of Contents\n\n1. [MCP Server Overview](#mcp-server-overview)\n2. [Architecture and Design](#architecture-and-design)\n3. [LinuxAgent MCP Server Analysis](#linuxagent-mcp-server-analysis)\n4. [Creating Your MCP Server](#creating-your-mcp-server)\n5. [Tool Definition Best Practices](#tool-definition-best-practices)\n6. [Error Handling and Validation](#error-handling-and-validation)\n7. [Testing Your MCP Server](#testing-your-mcp-server)\n\n---\n\n## MCP Server Overview\n\n### What is an MCP Server?\n\nAn **MCP Server** is a service that exposes **platform-specific tools** (commands) to LLM agents via the Model Context Protocol. For device agents, the MCP server:\n\n- Runs on or near the target device\n- Exposes tools as callable functions\n- Executes system-level commands safely\n- Returns structured results to the agent\n\n### MCP Server in Device Agent Architecture\n\n```mermaid\ngraph TB\n subgraph \"Agent Server (Orchestrator)\"\n Agent[Device Agent]\n Strategy[Action Execution Strategy]\n Dispatcher[Command Dispatcher]\n end\n \n subgraph \"Device Client\"\n Client[UFO Client]\n Manager[MCP Server Manager]\n end\n \n subgraph \"MCP Server (Device/Remote)\"\n MCP[MCP Server FastMCP]\n Tool1[Tool: execute_command]\n Tool2[Tool: get_system_info]\n ToolN[Tool: ...]\n \n MCP --> Tool1\n MCP --> Tool2\n MCP --> ToolN\n end\n \n subgraph \"Target Device/System\"\n OS[Operating System Linux/Android/iOS]\n Shell[Shell/API]\n \n Tool1 --> Shell\n Tool2 --> Shell\n ToolN --> Shell\n Shell --> OS\n end\n \n Agent --> Strategy\n Strategy --> Dispatcher\n Dispatcher -->|AIP Protocol| Client\n Client --> Manager\n Manager -->|HTTP/Stdio| MCP\n \n style Agent fill:#c8e6c9\n style MCP fill:#e1f5ff\n style OS fill:#fff3e0\n```\n\n**Key Points**:\n\n- **MCP Server** runs separately from agent server (security isolation)\n- **Tools** are atomic operations exposed to LLM\n- **Command Dispatcher** translates LLM actions to MCP tool calls\n- **Results** flow back through the same path\n\n---\n\n## Architecture and Design\n\n### MCP Server Components\n\n```mermaid\ngraph TB\n subgraph \"MCP Server Structure\"\n Server[FastMCP Server HTTP/Stdio Transport]\n \n subgraph \"Tools Layer\"\n T1[Tool 1 @mcp.tool]\n T2[Tool 2 @mcp.tool]\n T3[Tool N @mcp.tool]\n end\n \n subgraph \"Execution Layer\"\n Executor[Command Executor asyncio subprocess]\n Validator[Input Validator Security checks]\n ErrorHandler[Error Handler Exception handling]\n end\n \n subgraph \"Platform Interface\"\n API[Platform API Shell/SDK/ADB]\n end\n \n Server --> T1 & T2 & T3\n T1 & T2 & T3 --> Validator\n Validator --> Executor\n Executor --> ErrorHandler\n ErrorHandler --> API\n end\n \n style Server fill:#e1f5ff\n style T1 fill:#c8e6c9\n style Executor fill:#fff3e0\n style API fill:#f3e5f5\n```\n\n### MCP Server Design Principles\n\n| Principle | Description | Example |\n|-----------|-------------|---------|\n| **Atomic Tools** | Each tool performs one specific operation | `execute_command` vs `execute_and_parse_command` |\n| **Type Safety** | Use Pydantic `Field` for type annotations | `Annotated[str, Field(description=\"...\")]` |\n| **Error Resilience** | Handle all exceptions gracefully | Try/except with structured error responses |\n| **Security First** | Validate and sanitize all inputs | Block dangerous commands, validate paths |\n| **Platform Agnostic** | Abstract platform differences | Use subprocess for shell, ADB for Android |\n| **Async Execution** | Use asyncio for non-blocking operations | `async def`, `await subprocess` |\n\n---\n\n## LinuxAgent MCP Server Analysis\n\n### File Location\n\n**Path**: `ufo/client/mcp/http_servers/linux_mcp_server.py`\n\n### Complete Implementation\n\n```python\n#!/usr/bin/env python3\n# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nLinux MCP Server\nProvides MCP interface for executing shell commands on Linux systems.\n\"\"\"\n\nimport argparse\nimport asyncio\nfrom typing import Annotated, Any, Dict, Optional\nfrom fastmcp import FastMCP\nfrom pydantic import Field\n\n\ndef create_bash_mcp_server(host: str = \"\", port: int = 8010) -> None:\n \"\"\"Create an MCP server for Linux command execution.\"\"\"\n \n # Initialize FastMCP server with configuration\n mcp = FastMCP(\n \"Linux Bash MCP Server\", # Server name\n instructions=\"MCP server for executing shell commands on Linux.\",\n stateless_http=False, # Maintain state across requests\n json_response=True, # Return JSON responses\n host=host,\n port=port,\n )\n\n # ========================================\n # Tool 1: Execute Shell Command\n # ========================================\n @mcp.tool()\n async def execute_command(\n command: Annotated[\n str,\n Field(\n description=\"Shell command to execute on the Linux system. \"\n \"This should be a valid bash/sh command that will be executed \"\n \"in a shell environment. Examples: 'ls -la /home', \"\n \"'cat /etc/os-release', 'python3 --version', \"\n \"'grep -r \\\"pattern\\\" /path/to/search'. Be cautious with \"\n \"destructive commands as some dangerous operations are blocked.\"\n ),\n ],\n timeout: Annotated[\n int,\n Field(\n description=\"Maximum execution time in seconds before the \"\n \"command is forcefully terminated. Default is 30 seconds. \"\n \"Use higher values for long-running operations.\"\n ),\n ] = 30,\n cwd: Annotated[\n Optional[str],\n Field(\n description=\"Working directory path where the command should \"\n \"be executed. If not specified, uses server's current directory. \"\n \"Use absolute paths for reliability.\"\n ),\n ] = None,\n ) -> Annotated[\n Dict[str, Any],\n Field(\n description=\"Dictionary containing execution results with keys: \"\n \"'success' (bool), 'exit_code' (int), 'stdout' (str), \"\n \"'stderr' (str), or 'error' (str error message if execution failed)\"\n ),\n ]:\n \"\"\"\n Execute a shell command on Linux and return stdout/stderr.\n \n Security: Blocks known dangerous commands.\n \"\"\"\n # Security: Block dangerous commands\n dangerous = [\n \"rm -rf /\",\n \":(){ :|:& };:\", # Fork bomb\n \"mkfs\",\n \"dd if=/dev/zero\",\n \"shutdown\",\n \"reboot\",\n ]\n if any(d in command.lower() for d in dangerous):\n return {\"success\": False, \"error\": \"Blocked dangerous command.\"}\n \n try:\n # Create async subprocess\n proc = await asyncio.create_subprocess_shell(\n command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n cwd=cwd,\n )\n \n try:\n # Wait for completion with timeout\n stdout, stderr = await asyncio.wait_for(\n proc.communicate(), timeout=timeout\n )\n except asyncio.TimeoutError:\n # Kill process on timeout\n proc.kill()\n await proc.wait()\n return {\"success\": False, \"error\": f\"Timeout after {timeout}s.\"}\n \n # Return structured result\n return {\n \"success\": proc.returncode == 0,\n \"exit_code\": proc.returncode,\n \"stdout\": stdout.decode(\"utf-8\", errors=\"replace\"),\n \"stderr\": stderr.decode(\"utf-8\", errors=\"replace\"),\n }\n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # ========================================\n # Tool 2: Get System Information\n # ========================================\n @mcp.tool()\n async def get_system_info() -> Annotated[\n Dict[str, Any],\n Field(\n description=\"Dictionary containing basic Linux system information \"\n \"with keys: 'uname', 'uptime', 'memory', 'disk'\"\n ),\n ]:\n \"\"\"\n Get basic system info (uname, uptime, memory, disk).\n \"\"\"\n info = {}\n cmds = {\n \"uname\": \"uname -a\",\n \"uptime\": \"uptime\",\n \"memory\": \"free -h\",\n \"disk\": \"df -h\",\n }\n \n for k, cmd in cmds.items():\n try:\n proc = await asyncio.create_subprocess_shell(\n cmd, stdout=asyncio.subprocess.PIPE\n )\n out, _ = await proc.communicate()\n info[k] = out.decode(\"utf-8\", errors=\"replace\").strip()\n except Exception as e:\n info[k] = f\"Error: {e}\"\n \n return info\n\n # Start the server\n mcp.run(transport=\"streamable-http\")\n\n\ndef main():\n \"\"\"CLI entry point for Linux MCP server.\"\"\"\n parser = argparse.ArgumentParser(description=\"Linux Bash MCP Server\")\n parser.add_argument(\n \"--port\", type=int, default=8010, help=\"Port to run the server on\"\n )\n parser.add_argument(\n \"--host\", default=\"localhost\", help=\"Host to bind the server to\"\n )\n args = parser.parse_args()\n\n print(\"=\" * 50)\n print(\"UFO Linux Bash MCP Server\")\n print(\"Linux command execution via Model Context Protocol\")\n print(f\"Running on {args.host}:{args.port}\")\n print(\"=\" * 50)\n\n create_bash_mcp_server(host=args.host, port=args.port)\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n### Key Design Patterns\n\n#### 1. Type-Safe Tool Definitions\n\n```python\n@mcp.tool()\nasync def execute_command(\n command: Annotated[str, Field(description=\"...\")], # Required parameter\n timeout: Annotated[int, Field(description=\"...\")] = 30, # Optional with default\n cwd: Annotated[Optional[str], Field(description=\"...\")] = None, # Optional\n) -> Annotated[Dict[str, Any], Field(description=\"...\")]: # Return type\n```\n\n**Benefits**:\n- ✅ LLM understands parameter types and descriptions\n- ✅ Runtime validation via Pydantic\n- ✅ Auto-generated API documentation\n- ✅ Clear contracts for consumers\n\n#### 2. Security-First Validation\n\n```python\n# Block dangerous commands\ndangerous = [\"rm -rf /\", \":(){ :|:& };:\", \"mkfs\", ...]\nif any(d in command.lower() for d in dangerous):\n return {\"success\": False, \"error\": \"Blocked dangerous command.\"}\n```\n\n**Best Practices**:\n- ✅ Whitelist safe operations when possible\n- ✅ Blacklist known dangerous patterns\n- ✅ Validate paths (prevent directory traversal)\n- ✅ Limit command complexity\n- ❌ Don't rely on sanitization alone\n\n#### 3. Async Execution with Timeout\n\n```python\nproc = await asyncio.create_subprocess_shell(...)\ntry:\n stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=timeout)\nexcept asyncio.TimeoutError:\n proc.kill()\n await proc.wait()\n return {\"success\": False, \"error\": f\"Timeout after {timeout}s.\"}\n```\n\n**Why Async?**:\n- Non-blocking execution (server remains responsive)\n- Timeout enforcement (prevent hanging)\n- Concurrent tool execution support\n- Better resource utilization\n\n#### 4. Structured Error Handling\n\n```python\nreturn {\n \"success\": proc.returncode == 0, # Boolean success flag\n \"exit_code\": proc.returncode, # Numeric exit code\n \"stdout\": stdout.decode(\"utf-8\", errors=\"replace\"), # Output\n \"stderr\": stderr.decode(\"utf-8\", errors=\"replace\"), # Errors\n}\n```\n\n**Error Response Contract**:\n- Always return dict (never raise exceptions to LLM)\n- Include `success` boolean field\n- Provide detailed error messages\n- Preserve stdout/stderr for debugging\n\n---\n\n## Creating Your MCP Server\n\n### Step-by-Step Guide: MobileAgent MCP Server\n\nLet's create a complete MCP server for mobile automation (Android/iOS):\n\n**File**: `ufo/client/mcp/http_servers/mobile_mcp_server.py`\n\n```python\n#!/usr/bin/env python3\n# Copyright (c) Microsoft Corporation.\n# Licensed under the MIT License.\n\n\"\"\"\nMobile MCP Server\nProvides MCP interface for mobile device automation (Android/iOS).\n\"\"\"\n\nimport argparse\nimport asyncio\nimport subprocess\nfrom typing import Annotated, Any, Dict, Optional, Literal\nfrom fastmcp import FastMCP\nfrom pydantic import Field\n\n\ndef create_mobile_mcp_server(\n host: str = \"localhost\", \n port: int = 8020,\n platform: str = \"android\"\n) -> None:\n \"\"\"Create an MCP server for mobile device automation.\"\"\"\n \n mcp = FastMCP(\n f\"Mobile MCP Server ({platform.capitalize()})\",\n instructions=f\"MCP server for {platform} mobile device automation\",\n stateless_http=False,\n json_response=True,\n host=host,\n port=port,\n )\n\n # ========================================\n # Tool 1: Tap Element by Coordinates\n # ========================================\n @mcp.tool()\n async def tap_screen(\n x: Annotated[int, Field(description=\"X coordinate (pixels from left)\")],\n y: Annotated[int, Field(description=\"Y coordinate (pixels from top)\")],\n duration_ms: Annotated[\n int, \n Field(description=\"Tap duration in milliseconds (default: 100)\")\n ] = 100,\n ) -> Annotated[\n Dict[str, Any],\n Field(description=\"Result with 'success', 'message', and optional 'error'\")\n ]:\n \"\"\"\n Tap the screen at specified coordinates.\n \n Platform support:\n - Android: Uses ADB input tap\n - iOS: Uses xcrun simctl (simulator) or ios-deploy (device)\n \"\"\"\n try:\n if platform == \"android\":\n # Android: adb shell input tap x y\n result = subprocess.run(\n [\"adb\", \"shell\", \"input\", \"tap\", str(x), str(y)],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n if result.returncode == 0:\n return {\n \"success\": True,\n \"message\": f\"Tapped at ({x}, {y})\",\n \"platform\": \"android\"\n }\n else:\n return {\n \"success\": False,\n \"error\": f\"ADB error: {result.stderr}\",\n \"platform\": \"android\"\n }\n \n elif platform == \"ios\":\n # iOS: xcrun simctl (for simulator)\n # Note: Real device requires more complex setup\n result = subprocess.run(\n [\"xcrun\", \"simctl\", \"io\", \"booted\", \"tap\", str(x), str(y)],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n if result.returncode == 0:\n return {\n \"success\": True,\n \"message\": f\"Tapped at ({x}, {y})\",\n \"platform\": \"ios\"\n }\n else:\n return {\n \"success\": False,\n \"error\": f\"iOS error: {result.stderr}\",\n \"platform\": \"ios\"\n }\n \n except subprocess.TimeoutExpired:\n return {\"success\": False, \"error\": \"Command timeout\"}\n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # ========================================\n # Tool 2: Swipe Gesture\n # ========================================\n @mcp.tool()\n async def swipe(\n start_x: Annotated[int, Field(description=\"Start X coordinate\")],\n start_y: Annotated[int, Field(description=\"Start Y coordinate\")],\n end_x: Annotated[int, Field(description=\"End X coordinate\")],\n end_y: Annotated[int, Field(description=\"End Y coordinate\")],\n duration_ms: Annotated[\n int, \n Field(description=\"Swipe duration in milliseconds (default: 300)\")\n ] = 300,\n ) -> Dict[str, Any]:\n \"\"\"\n Perform a swipe gesture from start to end coordinates.\n \"\"\"\n try:\n if platform == \"android\":\n # Android: adb shell input swipe x1 y1 x2 y2 duration\n result = subprocess.run(\n [\n \"adb\", \"shell\", \"input\", \"swipe\",\n str(start_x), str(start_y),\n str(end_x), str(end_y),\n str(duration_ms)\n ],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n return {\n \"success\": result.returncode == 0,\n \"message\": f\"Swiped from ({start_x},{start_y}) to ({end_x},{end_y})\",\n \"error\": result.stderr if result.returncode != 0 else None\n }\n \n elif platform == \"ios\":\n # iOS simulator: multiple taps with delay\n # (Approximates swipe - real swipe requires XCUITest)\n await asyncio.sleep(0.1) # Placeholder\n return {\n \"success\": True,\n \"message\": f\"Swipe gesture simulated (iOS)\",\n \"note\": \"Real device requires XCUITest integration\"\n }\n \n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # ========================================\n # Tool 3: Type Text\n # ========================================\n @mcp.tool()\n async def type_text(\n text: Annotated[str, Field(description=\"Text to type\")],\n clear_first: Annotated[\n bool, \n Field(description=\"Clear existing text before typing\")\n ] = False,\n ) -> Dict[str, Any]:\n \"\"\"\n Type text into the currently focused input field.\n \"\"\"\n try:\n if platform == \"android\":\n # Escape special characters for ADB\n escaped_text = text.replace(\" \", \"%s\").replace(\"'\", \"\\\\'\")\n \n if clear_first:\n # Clear existing text (Ctrl+A + Delete)\n subprocess.run(\n [\"adb\", \"shell\", \"input\", \"keyevent\", \"KEYCODE_CTRL_A\"],\n timeout=2\n )\n subprocess.run(\n [\"adb\", \"shell\", \"input\", \"keyevent\", \"KEYCODE_DEL\"],\n timeout=2\n )\n \n # Type new text\n result = subprocess.run(\n [\"adb\", \"shell\", \"input\", \"text\", escaped_text],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n return {\n \"success\": result.returncode == 0,\n \"message\": f\"Typed: {text}\",\n \"error\": result.stderr if result.returncode != 0 else None\n }\n \n elif platform == \"ios\":\n # iOS: xcrun simctl io booted text\n result = subprocess.run(\n [\"xcrun\", \"simctl\", \"io\", \"booted\", \"text\", text],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n return {\n \"success\": result.returncode == 0,\n \"message\": f\"Typed: {text}\",\n \"error\": result.stderr if result.returncode != 0 else None\n }\n \n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # ========================================\n # Tool 4: Capture Screenshot\n # ========================================\n @mcp.tool()\n async def capture_screenshot(\n save_path: Annotated[\n str, \n Field(description=\"Local path to save screenshot (e.g., '/tmp/screen.png')\")\n ],\n ) -> Dict[str, Any]:\n \"\"\"\n Capture a screenshot from the mobile device.\n \"\"\"\n try:\n if platform == \"android\":\n # Android: adb exec-out screencap -p > file\n result = subprocess.run(\n [\"adb\", \"exec-out\", \"screencap\", \"-p\"],\n capture_output=True,\n timeout=10\n )\n \n if result.returncode == 0:\n with open(save_path, \"wb\") as f:\n f.write(result.stdout)\n return {\n \"success\": True,\n \"message\": f\"Screenshot saved to {save_path}\",\n \"path\": save_path\n }\n else:\n return {\"success\": False, \"error\": result.stderr.decode()}\n \n elif platform == \"ios\":\n # iOS: xcrun simctl io booted screenshot\n result = subprocess.run(\n [\"xcrun\", \"simctl\", \"io\", \"booted\", \"screenshot\", save_path],\n capture_output=True,\n text=True,\n timeout=10\n )\n \n return {\n \"success\": result.returncode == 0,\n \"message\": f\"Screenshot saved to {save_path}\",\n \"path\": save_path,\n \"error\": result.stderr if result.returncode != 0 else None\n }\n \n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # ========================================\n # Tool 5: Get UI Hierarchy\n # ========================================\n @mcp.tool()\n async def get_ui_tree(\n format: Annotated[\n Literal[\"xml\", \"json\"],\n Field(description=\"Output format (xml or json)\")\n ] = \"xml\",\n ) -> Dict[str, Any]:\n \"\"\"\n Get the current UI hierarchy/tree from the device.\n \"\"\"\n try:\n if platform == \"android\":\n # Android: adb shell uiautomator dump\n # Dump to device, then pull\n subprocess.run(\n [\"adb\", \"shell\", \"uiautomator\", \"dump\", \"/sdcard/window_dump.xml\"],\n timeout=10\n )\n \n result = subprocess.run(\n [\"adb\", \"shell\", \"cat\", \"/sdcard/window_dump.xml\"],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n if result.returncode == 0:\n return {\n \"success\": True,\n \"ui_tree\": result.stdout,\n \"format\": \"xml\"\n }\n else:\n return {\"success\": False, \"error\": result.stderr}\n \n elif platform == \"ios\":\n # iOS: requires XCUITest or Appium\n return {\n \"success\": False,\n \"error\": \"iOS UI tree requires XCUITest integration\",\n \"note\": \"Use accessibility inspector or Appium\"\n }\n \n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # ========================================\n # Tool 6: Launch App\n # ========================================\n @mcp.tool()\n async def launch_app(\n package_name: Annotated[\n str,\n Field(description=\"App package name (Android) or bundle ID (iOS)\")\n ],\n ) -> Dict[str, Any]:\n \"\"\"\n Launch an application by package name or bundle ID.\n \"\"\"\n try:\n if platform == \"android\":\n # Android: adb shell monkey\n result = subprocess.run(\n [\n \"adb\", \"shell\", \"monkey\", \"-p\", package_name,\n \"-c\", \"android.intent.category.LAUNCHER\", \"1\"\n ],\n capture_output=True,\n text=True,\n timeout=10\n )\n \n return {\n \"success\": \"monkey\" in result.stdout.lower(),\n \"message\": f\"Launched {package_name}\",\n \"output\": result.stdout\n }\n \n elif platform == \"ios\":\n # iOS: xcrun simctl launch\n result = subprocess.run(\n [\"xcrun\", \"simctl\", \"launch\", \"booted\", package_name],\n capture_output=True,\n text=True,\n timeout=10\n )\n \n return {\n \"success\": result.returncode == 0,\n \"message\": f\"Launched {package_name}\",\n \"error\": result.stderr if result.returncode != 0 else None\n }\n \n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n # Start the server\n mcp.run(transport=\"streamable-http\")\n\n\ndef main():\n \"\"\"CLI entry point for Mobile MCP server.\"\"\"\n parser = argparse.ArgumentParser(description=\"Mobile MCP Server\")\n parser.add_argument(\n \"--port\", type=int, default=8020, help=\"Port to run the server on\"\n )\n parser.add_argument(\n \"--host\", default=\"localhost\", help=\"Host to bind the server to\"\n )\n parser.add_argument(\n \"--platform\",\n choices=[\"android\", \"ios\"],\n default=\"android\",\n help=\"Mobile platform (android or ios)\"\n )\n args = parser.parse_args()\n\n print(\"=\" * 50)\n print(f\"UFO Mobile MCP Server ({args.platform.capitalize()})\")\n print(f\"Mobile device automation via Model Context Protocol\")\n print(f\"Running on {args.host}:{args.port}\")\n print(\"=\" * 50)\n\n create_mobile_mcp_server(host=args.host, port=args.port, platform=args.platform)\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n---\n\n## Tool Definition Best Practices\n\n### 1. Descriptive Tool Names\n\n| ❌ Bad | ✅ Good | Why |\n|--------|---------|-----|\n| `do_thing` | `tap_screen` | Clear action |\n| `cmd` | `execute_command` | Self-documenting |\n| `get` | `get_ui_tree` | Specific purpose |\n\n### 2. Rich Type Annotations\n\n```python\n# ✅ Excellent: Full type hints with descriptions\n@mcp.tool()\nasync def tap_screen(\n x: Annotated[int, Field(description=\"X coordinate in pixels from left edge\")],\n y: Annotated[int, Field(description=\"Y coordinate in pixels from top edge\")],\n duration_ms: Annotated[int, Field(description=\"Tap duration in milliseconds\")] = 100,\n) -> Annotated[Dict[str, Any], Field(description=\"Result dict with 'success' and 'message'\")]:\n```\n\n### 3. Consistent Return Format\n\n```python\n# ✅ Always return structured dict\n{\n \"success\": bool, # Required: operation status\n \"message\": str, # Optional: human-readable result\n \"error\": str, # Optional: error details if success=False\n \"data\": Any, # Optional: additional result data\n}\n\n# ❌ Don't mix return types\nreturn True # Bad: not structured\nraise Exception(\"Error\") # Bad: exceptions not handled by LLM\n```\n\n### 4. Comprehensive Docstrings\n\n```python\n@mcp.tool()\nasync def swipe(start_x: int, start_y: int, end_x: int, end_y: int) -> Dict:\n \"\"\"\n Perform a swipe gesture from start to end coordinates.\n \n Platform support:\n - Android: Uses ADB input swipe\n - iOS: Simulated via multiple taps (requires XCUITest for real swipe)\n \n Args:\n start_x: Starting X coordinate (pixels from left)\n start_y: Starting Y coordinate (pixels from top)\n end_x: Ending X coordinate\n end_y: Ending Y coordinate\n \n Returns:\n Dict with 'success', 'message', and optional 'error'\n \n Example:\n >>> await swipe(100, 500, 100, 100) # Swipe up\n {\"success\": True, \"message\": \"Swiped from (100,500) to (100,100)\"}\n \"\"\"\n```\n\n---\n\n## Error Handling and Validation\n\n### Input Validation Strategies\n\n```python\n@mcp.tool()\nasync def tap_screen(x: int, y: int) -> Dict[str, Any]:\n \"\"\"Tap with validation.\"\"\"\n \n # 1. Range validation\n if x < 0 or y < 0:\n return {\n \"success\": False,\n \"error\": f\"Invalid coordinates: ({x}, {y}). Must be non-negative.\"\n }\n \n # 2. Boundary checks (if screen size known)\n max_x, max_y = 1080, 1920 # Example resolution\n if x > max_x or y > max_y:\n return {\n \"success\": False,\n \"error\": f\"Coordinates out of bounds. Screen: {max_x}x{max_y}\"\n }\n \n # 3. Execute with error handling\n try:\n result = subprocess.run([...], timeout=5)\n return {\"success\": result.returncode == 0}\n except subprocess.TimeoutExpired:\n return {\"success\": False, \"error\": \"Tap command timeout\"}\n except Exception as e:\n return {\"success\": False, \"error\": f\"Unexpected error: {str(e)}\"}\n```\n\n### Security Validation\n\n```python\ndef validate_app_package(package: str) -> bool:\n \"\"\"Validate app package name format.\"\"\"\n import re\n # Android: com.example.app\n android_pattern = r'^[a-z][a-z0-9_]*(\\.[a-z][a-z0-9_]*)+$'\n # iOS: com.example.App\n ios_pattern = r'^[a-zA-Z][a-zA-Z0-9_]*(\\.[a-zA-Z][a-zA-Z0-9_]*)+$'\n \n return bool(re.match(android_pattern, package) or re.match(ios_pattern, package))\n\n@mcp.tool()\nasync def launch_app(package_name: str) -> Dict:\n \"\"\"Launch app with validation.\"\"\"\n if not validate_app_package(package_name):\n return {\n \"success\": False,\n \"error\": f\"Invalid package name format: {package_name}\"\n }\n # ... continue execution\n```\n\n### Timeout Strategies\n\n```python\n# Strategy 1: Command-level timeout\nresult = subprocess.run([...], timeout=5)\n\n# Strategy 2: Async timeout with cleanup\ntry:\n proc = await asyncio.create_subprocess_exec(...)\n stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=10)\nexcept asyncio.TimeoutError:\n proc.kill() # Clean up process\n await proc.wait()\n return {\"success\": False, \"error\": \"Operation timeout\"}\n\n# Strategy 3: Retry with backoff\nasync def execute_with_retry(cmd, max_retries=3):\n for attempt in range(max_retries):\n try:\n return await execute_command(cmd)\n except TimeoutError:\n if attempt == max_retries - 1:\n raise\n await asyncio.sleep(2 ** attempt) # Exponential backoff\n```\n\n---\n\n## Testing Your MCP Server\n\n### Unit Testing\n\n```python\n# tests/test_mobile_mcp_server.py\n\nimport pytest\nfrom unittest.mock import patch, MagicMock\nfrom ufo.client.mcp.http_servers.mobile_mcp_server import (\n create_mobile_mcp_server\n)\n\n\nclass TestMobileMCPServer:\n \"\"\"Unit tests for Mobile MCP Server tools.\"\"\"\n\n @pytest.mark.asyncio\n @patch('subprocess.run')\n async def test_tap_screen_success(self, mock_run):\n \"\"\"Test successful tap execution.\"\"\"\n # Mock subprocess result\n mock_run.return_value = MagicMock(\n returncode=0,\n stdout=\"\",\n stderr=\"\"\n )\n \n # Import tool function (assuming it's exposed)\n from mobile_mcp_server import tap_screen\n \n result = await tap_screen(x=100, y=200)\n \n assert result[\"success\"] == True\n assert \"Tapped at (100, 200)\" in result[\"message\"]\n mock_run.assert_called_once()\n\n @pytest.mark.asyncio\n async def test_tap_screen_invalid_coordinates(self):\n \"\"\"Test tap with invalid coordinates.\"\"\"\n from mobile_mcp_server import tap_screen\n \n result = await tap_screen(x=-10, y=50)\n \n assert result[\"success\"] == False\n assert \"Invalid coordinates\" in result[\"error\"]\n\n @pytest.mark.asyncio\n @patch('subprocess.run')\n async def test_swipe_timeout(self, mock_run):\n \"\"\"Test swipe with timeout.\"\"\"\n mock_run.side_effect = subprocess.TimeoutExpired(cmd=\"adb\", timeout=5)\n \n from mobile_mcp_server import swipe\n \n result = await swipe(0, 0, 100, 100)\n \n assert result[\"success\"] == False\n assert \"timeout\" in result[\"error\"].lower()\n```\n\n### Integration Testing\n\n```python\n# tests/integration/test_mcp_server_integration.py\n\nimport pytest\nimport requests\nfrom ufo.client.mcp.mcp_server_manager import HTTPMCPServer\n\n\nclass TestMCPServerIntegration:\n \"\"\"Integration tests for MCP server.\"\"\"\n\n @pytest.fixture\n def mcp_server(self):\n \"\"\"Start MCP server for testing.\"\"\"\n config = {\n \"host\": \"localhost\",\n \"port\": 8020,\n \"path\": \"/mcp\"\n }\n server = HTTPMCPServer(config)\n server.start()\n yield server\n server.stop()\n\n def test_server_health(self, mcp_server):\n \"\"\"Test server is reachable.\"\"\"\n response = requests.get(f\"{mcp_server.server}/health\")\n assert response.status_code == 200\n\n def test_tap_screen_end_to_end(self, mcp_server):\n \"\"\"Test tap screen tool end-to-end.\"\"\"\n payload = {\n \"tool\": \"tap_screen\",\n \"parameters\": {\"x\": 100, \"y\": 200}\n }\n response = requests.post(\n f\"{mcp_server.server}/execute\",\n json=payload\n )\n \n assert response.status_code == 200\n result = response.json()\n assert \"success\" in result\n```\n\n### Manual Testing\n\n```bash\n# 1. Start MCP server\npython -m ufo.client.mcp.http_servers.mobile_mcp_server \\\n --host localhost \\\n --port 8020 \\\n --platform android\n\n# 2. Test with curl\ncurl -X POST http://localhost:8020/mcp \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"tool\": \"tap_screen\",\n \"parameters\": {\"x\": 500, \"y\": 1000}\n }'\n\n# 3. Expected response\n{\n \"success\": true,\n \"message\": \"Tapped at (500, 1000)\",\n \"platform\": \"android\"\n}\n```\n\n---\n\n## Summary\n\n**What You've Built**:\n\n- ✅ Platform-specific MCP server with FastMCP\n- ✅ Type-safe tool definitions with Pydantic\n- ✅ Async execution with timeout handling\n- ✅ Security validation and error handling\n- ✅ Comprehensive testing strategy\n\n**Key Takeaways**:\n\n| Concept | Best Practice |\n|---------|---------------|\n| **Tool Design** | Atomic, single-purpose operations |\n| **Type Safety** | Use `Annotated[T, Field(description=...)]` |\n| **Error Handling** | Always return structured dicts, never raise |\n| **Security** | Validate inputs, block dangerous operations |\n| **Async** | Use `asyncio` for non-blocking execution |\n| **Testing** | Unit + integration tests for all tools |\n\n---\n\n## Next Steps\n\n**Continue to**: [Part 3: Client Setup →](client_setup.md)\n\nLearn how to configure the UFO client to connect to your MCP server and enable device agent execution.\n\n---\n\n## Related Documentation\n\n- **[MCP Overview](../../mcp/overview.md)** - Model Context Protocol fundamentals\n- **[Creating MCP Servers](../creating_mcp_servers.md)** - General MCP server tutorial\n- **[FastMCP Documentation](https://github.com/jlowin/fastmcp)** - FastMCP library reference\n- **[AIP Protocol](../../aip/overview.md)** - Agent Interaction Protocol\n\n---\n\n**Previous**: [← Part 1: Core Components](core_components.md) \n**Next**: [Part 3: Client Setup →](client_setup.md)\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/overview.md",
"content": "# Creating a New Device Agent - Complete Tutorial\n\nThis comprehensive tutorial teaches you how to create a new device agent (like `MobileAgent`, `AndroidAgent`, or `iOSAgent`) and integrate it with UFO³'s multi-device orchestration system. We'll use **LinuxAgent** as our primary reference implementation.\n\n---\n\n## 📋 Table of Contents\n\n1. [Introduction](#introduction)\n2. [Prerequisites](#prerequisites)\n3. [Understanding Device Agents](#understanding-device-agents)\n4. [LinuxAgent: Reference Implementation](#linuxagent-reference-implementation)\n5. [Architecture Overview](#architecture-overview)\n6. [Tutorial Roadmap](#tutorial-roadmap)\n\n---\n\n## Introduction\n\n### What is a Device Agent?\n\nA **Device Agent** is a specialized AI agent that controls and automates tasks on a specific type of device or platform. Unlike traditional third-party agents that extend specific functionality, device agents represent entire computing platforms with their own:\n\n- **Execution Environment**: Device-specific OS, runtime, and APIs\n- **Control Mechanism**: UI automation, CLI commands, or platform APIs\n- **Communication Protocol**: Client-server architecture via WebSocket\n- **MCP Integration**: Device-specific MCP servers for command execution\n\n### Device Agent vs Third-Party Agent\n\n| Aspect | Device Agent | Third-Party Agent |\n|--------|--------------|-------------------|\n| **Scope** | Full platform control (Windows, Linux, Mobile) | Specific functionality (Hardware, Web) |\n| **Architecture** | Client-Server separation | Runs on orchestrator server |\n| **Communication** | WebSocket + AIP Protocol | Direct method calls |\n| **MCP Servers** | Platform-specific MCP servers | Shares MCP servers |\n| **Examples** | WindowsAgent, LinuxAgent, MobileAgent | HardwareAgent, WebAgent |\n| **Deployment** | Separate client process on device | Part of orchestrator |\n\n### When to Create a Device Agent\n\nCreate a **Device Agent** when you need to:\n\n- Control an entirely new platform (mobile, IoT, embedded)\n- Execute tasks on remote or distributed devices\n- Integrate with Galaxy multi-device orchestration\n- Isolate execution for security or scalability\n\nCreate a **Third-Party Agent** when you need to:\n\n- Extend existing platform with new capabilities\n- Add specialized tools or APIs\n- Run alongside existing agents\n\n---\n\n## Prerequisites\n\nBefore starting this tutorial, ensure you have:\n\n### Knowledge Requirements\n\n- ✅ **Python 3.10+**: Intermediate Python programming skills\n- ✅ **Async Programming**: Understanding of `async`/`await` patterns\n- ✅ **UFO³ Basics**: Familiarity with [Agent Architecture](../../infrastructure/agents/overview.md)\n- ✅ **MCP Protocol**: Understanding of [Model Context Protocol](../../mcp/overview.md)\n- ✅ **WebSocket**: Basic knowledge of WebSocket communication\n\n### Recommended Reading\n\n| Priority | Topic | Link | Time |\n|----------|-------|------|------|\n| 🥇 | **Agent Architecture Overview** | [Infrastructure/Agents](../../infrastructure/agents/overview.md) | 20 min |\n| 🥇 | **LinuxAgent Quick Start** | [Quick Start: Linux](../../getting_started/quick_start_linux.md) | 15 min |\n| 🥈 | **Server-Client Architecture** | [Server Overview](../../server/overview.md), [Client Overview](../../client/overview.md) | 30 min |\n| 🥈 | **MCP Integration** | [MCP Overview](../../mcp/overview.md) | 20 min |\n| 🥉 | **AIP Protocol** | [AIP Protocol](../../aip/overview.md) | 15 min |\n\n### Development Environment\n\n```bash\n# Clone UFO³ repository\ngit clone https://github.com/microsoft/UFO.git\ncd UFO\n\n# Install dependencies\npip install -r requirements.txt\n\n# Verify installation\npython -c \"import ufo; print('UFO³ installed successfully')\"\n```\n\n---\n\n## Understanding Device Agents\n\n### Three-Layer Architecture\n\nAll device agents in UFO³ follow a **unified three-layer architecture**:\n\n```mermaid\ngraph TB\n subgraph \"Device Agent Architecture\"\n subgraph \"Level-1: State Layer (FSM)\"\n S1[AgentState]\n S2[State Machine]\n S3[State Transitions]\n S1 --> S2 --> S3\n end\n \n subgraph \"Level-2: Strategy Layer (Execution Logic)\"\n P1[ProcessorTemplate]\n P2[DATA_COLLECTION]\n P3[LLM_INTERACTION]\n P4[ACTION_EXECUTION]\n P5[MEMORY_UPDATE]\n P1 --> P2 --> P3 --> P4 --> P5\n end\n \n subgraph \"Level-3: Command Layer (System Interface)\"\n C1[CommandDispatcher]\n C2[MCP Tools]\n C3[Device Commands]\n C1 --> C2 --> C3\n end\n \n S3 -->|delegates to| P1\n P5 -->|executes via| C1\n end\n \n style S1 fill:#e1f5ff\n style P1 fill:#fff3e0\n style C1 fill:#f3e5f5\n```\n\n**Key Layers**:\n\n1. **State Layer (Level-1)**: Finite State Machine controlling agent lifecycle\n2. **Strategy Layer (Level-2)**: Processing pipeline with modular strategies\n3. **Command Layer (Level-3)**: Atomic system operations via MCP\n\nFor detailed architecture, see [Agent Architecture Documentation](../../infrastructure/agents/overview.md).\n\n---\n\n### Server-Client Separation\n\nDevice agents use a **server-client architecture** for security and scalability:\n\n```mermaid\ngraph LR\n subgraph \"Server Side (Orchestrator)\"\n Server[Device Agent Server]\n State[State Machine]\n Processor[Strategy Processor]\n LLM[LLM Service]\n \n Server --> State\n Server --> Processor\n Processor -.-> LLM\n end\n \n subgraph \"Communication\"\n AIP[AIP Protocol WebSocket]\n end\n \n subgraph \"Client Side (Device)\"\n Client[Device Client]\n MCP[MCP Server Manager]\n Tools[Platform Tools]\n OS[Device OS]\n \n Client --> MCP\n MCP --> Tools\n Tools --> OS\n end\n \n Server <-->|Commands/Results| AIP\n AIP <-->|Commands/Results| Client\n \n style Server fill:#e1f5ff\n style Client fill:#c8e6c9\n style AIP fill:#fff3e0\n```\n\n**Separation Benefits**:\n\n| Component | Location | Responsibilities | Security |\n|-----------|----------|------------------|----------|\n| **Agent Server** | Orchestrator | Reasoning, planning, state management | Untrusted (LLM-driven) |\n| **Device Client** | Target Device | Command execution, resource access | Trusted (validated operations) |\n| **AIP Protocol** | Network | Message transport, serialization | Encrypted channel |\n\n**Separation Benefits**:\n\n- **Security**: Isolates LLM reasoning from system-level execution\n- **Scalability**: Single orchestrator manages multiple devices\n- **Flexibility**: Clients run on resource-constrained devices (mobile, IoT)\n- **Safety**: Client validates all commands before execution\n\n---\n\n## LinuxAgent: Reference Implementation\n\n### Why LinuxAgent as Reference?\n\n**LinuxAgent** is the ideal reference for creating new device agents because:\n\n- ✅ **Simple Architecture**: Single-tier agent (no HostAgent delegation)\n- ✅ **Clear Separation**: Clean server-client boundary\n- ✅ **Well-Documented**: Comprehensive code and documentation\n- ✅ **Production-Ready**: Battle-tested in real deployments\n- ✅ **Minimal Complexity**: Focuses on core device agent patterns\n\n### LinuxAgent Components\n\n```mermaid\ngraph TB\n subgraph \"Server Side (ufo/agents/)\"\n LA[LinuxAgent Class customized_agent.py]\n LAP[LinuxAgentProcessor customized_agent_processor.py]\n LAS[LinuxAgent Strategies linux_agent_strategy.py]\n LAST[LinuxAgent States linux_agent_state.py]\n \n LA --> LAP\n LAP --> LAS\n LA --> LAST\n end\n \n subgraph \"Client Side (ufo/client/)\"\n Client[UFO Client client.py]\n MCP[MCP Server Manager mcp_server_manager.py]\n LinuxMCP[Linux MCP Server linux_mcp_server.py]\n \n Client --> MCP\n MCP --> LinuxMCP\n end\n \n subgraph \"Configuration\"\n Config[third_party.yaml]\n Devices[devices.yaml]\n Prompts[Prompt Templates]\n end\n \n LA -.reads.-> Config\n Client -.reads.-> Devices\n LA -.uses.-> Prompts\n \n style LA fill:#c8e6c9\n style LAP fill:#c8e6c9\n style LAS fill:#c8e6c9\n style LAST fill:#c8e6c9\n style Client fill:#e1f5ff\n style MCP fill:#e1f5ff\n style LinuxMCP fill:#e1f5ff\n```\n\n**File Locations**:\n\n| Component | File Path | Purpose |\n|-----------|-----------|---------|\n| **Agent Class** | `ufo/agents/agent/customized_agent.py` | LinuxAgent definition |\n| **Processor** | `ufo/agents/processors/customized/customized_agent_processor.py` | LinuxAgentProcessor |\n| **Strategies** | `ufo/agents/processors/strategies/linux_agent_strategy.py` | LLM & Action strategies |\n| **States** | `ufo/agents/states/linux_agent_state.py` | State machine states |\n| **Prompter** | `ufo/prompter/customized/linux_agent_prompter.py` | Prompt construction |\n| **Client** | `ufo/client/client.py` | Device client entry point |\n| **MCP Server** | `ufo/client/mcp/http_servers/linux_mcp_server.py` | Command execution |\n\n---\n\n### LinuxAgent Architecture Diagram\n\n```mermaid\nsequenceDiagram\n participant User\n participant Server as LinuxAgent Server\n participant AIP as AIP Protocol\n participant Client as Linux Client\n participant MCP as Linux MCP Server\n participant Shell as Bash Shell\n \n User->>Server: User Request: \"List files in /tmp\"\n \n Server->>Server: State: ContinueLinuxAgentState\n Server->>Server: Processor: LinuxAgentProcessor\n \n Server->>Server: Strategy: LLM_INTERACTION\n Note over Server: Construct prompt, call LLM\n Server->>Server: LLM Response: execute_command(\"ls -la /tmp\")\n \n Server->>Server: Strategy: ACTION_EXECUTION\n Server->>AIP: COMMAND: execute_command\n AIP->>Client: WebSocket: COMMAND\n \n Client->>MCP: Call MCP Tool: execute_command\n MCP->>Shell: Execute: ls -la /tmp\n Shell-->>MCP: stdout, stderr, exit_code\n MCP-->>Client: Result\n Client->>AIP: WebSocket: RESULT\n AIP->>Server: RESULT\n \n Server->>Server: Strategy: MEMORY_UPDATE\n Server->>Server: Update memory & blackboard\n \n Server->>Server: State Transition: FINISH\n Server->>User: Task Complete\n```\n\n**Key Execution Flow**:\n\n1. **User Request** → LinuxAgent Server receives request\n2. **State Machine** → Activates `ContinueLinuxAgentState`\n3. **Processor** → Executes `LinuxAgentProcessor` strategies\n4. **LLM Interaction** → Generates shell command\n5. **Action Execution** → Sends command via AIP to client\n6. **MCP Execution** → Client executes via Linux MCP Server\n7. **Result Handling** → Server receives result, updates memory\n8. **State Transition** → Moves to `FINISH` state\n\n---\n\n## Architecture Overview\n\n### Complete Device Agent Architecture\n\nWhen creating a new device agent (e.g., `MobileAgent`), you'll implement these components:\n\n```mermaid\ngraph TB\n subgraph \"1. Agent Definition\"\n A1[Agent Class MobileAgent]\n A2[Processor MobileAgentProcessor]\n A3[State Manager MobileAgentStateManager]\n end\n \n subgraph \"2. Processing Strategies\"\n S1[DATA_COLLECTION Screenshot, UI Tree]\n S2[LLM_INTERACTION Prompt Construction]\n S3[ACTION_EXECUTION Command Dispatch]\n S4[MEMORY_UPDATE Context Update]\n end\n \n subgraph \"3. MCP Server\"\n M1[MCP Server mobile_mcp_server.py]\n M2[MCP Tools tap, swipe, type, etc.]\n end\n \n subgraph \"4. Configuration\"\n C1[third_party.yaml Agent Config]\n C2[devices.yaml Device Registry]\n C3[Prompt Templates LLM Prompts]\n end\n \n subgraph \"5. Client\"\n CL1[Device Client client.py]\n CL2[MCP Manager mcp_server_manager.py]\n end\n \n A1 --> A2\n A2 --> S1 & S2 & S3 & S4\n S3 --> M1\n M1 --> M2\n A1 -.reads.-> C1\n CL1 --> CL2\n CL2 --> M1\n CL1 -.reads.-> C2\n A2 -.uses.-> C3\n \n style A1 fill:#c8e6c9\n style A2 fill:#c8e6c9\n style A3 fill:#c8e6c9\n style M1 fill:#e1f5ff\n style CL1 fill:#e1f5ff\n```\n\n**Implementation Checklist**:\n\n- [ ] **Agent Class**: Define `MobileAgent` inheriting from `CustomizedAgent`\n- [ ] **Processor**: Create `MobileAgentProcessor` with custom strategies\n- [ ] **State Manager**: Implement `MobileAgentStateManager` and states\n- [ ] **Strategies**: Build platform-specific LLM and action strategies\n- [ ] **MCP Server**: Develop MCP server with platform tools\n- [ ] **Prompter**: Create custom prompter for mobile context\n- [ ] **Client Setup**: Configure client to run on mobile device\n- [ ] **Configuration**: Add agent config to `third_party.yaml`\n- [ ] **Device Registry**: Register device in `devices.yaml`\n- [ ] **Prompt Templates**: Write LLM prompt templates\n\n---\n\n## Tutorial Roadmap\n\nThis tutorial is split into **6 detailed guides**:\n\n### 📘 Part 1: [Core Components](core_components.md)\n\nLearn to implement the **server-side components**:\n\n- Agent Class definition\n- Processor and strategies\n- State Manager and states\n- Prompter for LLM interaction\n\n**Time**: 45 minutes \n**Difficulty**: ⭐⭐⭐\n\n---\n\n### 📘 Part 2: [MCP Server Development](mcp_server.md)\n\nCreate a **platform-specific MCP server**:\n\n- MCP server architecture\n- Defining MCP tools\n- Command execution logic\n- Error handling and validation\n\n**Time**: 30 minutes \n**Difficulty**: ⭐⭐\n\n---\n\n### 📘 Part 3: [Client Configuration](client_setup.md)\n\nSet up the **device client**:\n\n- Client initialization\n- MCP server manager integration\n- WebSocket connection setup\n- Platform detection\n\n**Time**: 20 minutes \n**Difficulty**: ⭐⭐\n\n---\n\n### 📘 Part 4: [Configuration & Deployment](configuration.md)\n\nConfigure and deploy your agent:\n\n- `third_party.yaml` configuration\n- `devices.yaml` device registration\n- Prompt template creation\n- Galaxy integration\n\n**Time**: 25 minutes \n**Difficulty**: ⭐⭐\n\n---\n\n### 📘 Part 5: [Testing & Debugging](testing.md)\n\nTest and debug your implementation:\n\n- Unit testing strategies\n- Integration testing\n- Debugging techniques\n- Common issues and solutions\n\n**Time**: 30 minutes \n**Difficulty**: ⭐⭐⭐\n\n---\n\n### 📘 Part 6: [Complete Example: MobileAgent](example_mobile_agent.md)\n\n**Hands-on walkthrough** creating `MobileAgent`:\n\n- Step-by-step implementation\n- Android/iOS platform specifics\n- UI Automator integration\n- Complete working example\n\n**Time**: 60 minutes \n**Difficulty**: ⭐⭐⭐⭐\n\n---\n\n## Quick Start Guide\n\nFor experienced developers, here's a **minimal implementation checklist**:\n\n### 1️⃣ Create Agent Class\n\n```python\n# ufo/agents/agent/customized_agent.py\n\n@AgentRegistry.register(\n agent_name=\"MobileAgent\",\n third_party=True,\n processor_cls=MobileAgentProcessor\n)\nclass MobileAgent(CustomizedAgent):\n def __init__(self, name, main_prompt, example_prompt):\n super().__init__(name, main_prompt, example_prompt,\n process_name=None, app_root_name=None, is_visual=None)\n self._blackboard = Blackboard()\n self.set_state(self.default_state)\n self._context_provision_executed = False\n \n @property\n def default_state(self):\n return ContinueMobileAgentState()\n```\n\n### 2️⃣ Create Processor\n\n```python\n# ufo/agents/processors/customized/customized_agent_processor.py\n\nclass MobileAgentProcessor(CustomizedProcessor):\n def _setup_strategies(self):\n # Compose multiple data collection strategies\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(\n strategies=[\n MobileScreenshotCaptureStrategy(fail_fast=True),\n MobileAppsCollectionStrategy(fail_fast=False),\n MobileControlsCollectionStrategy(fail_fast=False),\n ],\n name=\"MobileDataCollectionStrategy\",\n fail_fast=True,\n )\n \n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n MobileLLMInteractionStrategy(fail_fast=True)\n )\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n MobileActionExecutionStrategy(fail_fast=False)\n )\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(fail_fast=False)\n )\n```\n\n### 3️⃣ Create MCP Server\n\n```python\n# ufo/client/mcp/http_servers/mobile_mcp_server.py\n\ndef create_mobile_mcp_server(host=\"localhost\", port=8020):\n mcp = FastMCP(\"Mobile MCP Server\", stateless_http=False, \n json_response=True, host=host, port=port)\n \n @mcp.tool()\n async def tap_element(x: int, y: int) -> dict:\n # Execute tap via ADB or platform API\n pass\n \n mcp.run(transport=\"streamable-http\")\n```\n\n### 4️⃣ Configure Agent\n\n```yaml\n# config/ufo/third_party.yaml\n\nENABLED_THIRD_PARTY_AGENTS: [\"MobileAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n MobileAgent:\n VISUAL_MODE: True\n AGENT_NAME: \"MobileAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/mobile_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/mobile_agent_example.yaml\"\n INTRODUCTION: \"MobileAgent controls Android/iOS devices...\"\n```\n\n### 5️⃣ Register Device\n\n```yaml\n# config/galaxy/devices.yaml\n\ndevices:\n - device_id: \"mobile_agent_1\"\n server_url: \"ws://localhost:5010/ws\"\n os: \"android\"\n capabilities: [\"ui_automation\", \"app_testing\"]\n metadata:\n device_model: \"Pixel 6\"\n android_version: \"13\"\n max_retries: 5\n```\n\n### 6️⃣ Start Server & Client\n\n```bash\n# Terminal 1: Start Agent Server\npython -m ufo.server.app --port 5010\n\n# Terminal 2: Start Device Client\npython -m ufo.client.client \\\n --ws --ws-server ws://localhost:5010/ws \\\n --client-id mobile_agent_1 \\\n --platform android\n\n# Terminal 3: Start MCP Server (on device or accessible endpoint)\npython -m ufo.client.mcp.http_servers.mobile_mcp_server --port 8020\n```\n\n---\n\n## Next Steps\n\n**Ready to Build Your Device Agent?**\n\nStart with Part 1: [Core Components →](core_components.md)\n\nOr jump to a specific topic:\n\n- [MCP Server Development](mcp_server.md)\n- [Configuration & Deployment](configuration.md)\n- [Complete Example: MobileAgent](example_mobile_agent.md)\n\n---\n\n## Related Documentation\n\n- **[Agent Architecture](../../infrastructure/agents/overview.md)** - Three-layer architecture deep dive\n- **[Linux Agent Quick Start](../../getting_started/quick_start_linux.md)** - LinuxAgent deployment guide\n- **[Server Overview](../../server/overview.md)** - Server-side orchestration\n- **[Client Overview](../../client/overview.md)** - Client-side execution\n- **[MCP Overview](../../mcp/overview.md)** - Model Context Protocol\n- **[AIP Protocol](../../aip/overview.md)** - Agent Interaction Protocol\n- **[Creating Third-Party Agents](../creating_third_party_agents.md)** - Third-party agent tutorial\n\n---\n\n## Summary\n\n**Key Takeaways**:\n\n- **Device Agents** control entire platforms (Windows, Linux, Mobile)\n- **Server-Client Architecture** separates reasoning from execution\n- **Three-Layer Design** provides modular, extensible framework\n- **LinuxAgent** is the best reference implementation\n- **6-Part Tutorial** covers all aspects of device agent creation\n- **MCP Integration** enables platform-specific command execution\n- **Galaxy Integration** supports multi-device orchestration\n\n**Ready to build your first device agent? Let's get started!** 🚀\n\n"
},
{
"path": "documents/docs/tutorials/creating_device_agent/testing.md",
"content": "# Part 5: Testing & Debugging\n\n**Note**: This tutorial is currently under development. Check back soon for comprehensive testing and debugging guidance.\n\n## What You'll Learn\n\n- Unit testing strategies\n- Integration testing\n- Debugging techniques\n- Common issues and solutions\n- Performance optimization\n\n## Temporary Quick Guide\n\n### Basic Testing\n\n```python\n# tests/test_mobile_agent.py\n\nimport pytest\nfrom ufo.agents.agent.customized_agent import MobileAgent\n\ndef test_agent_initialization():\n agent = MobileAgent(\n name=\"test_agent\",\n main_prompt=\"ufo/prompts/third_party/mobile_agent.yaml\",\n example_prompt=\"ufo/prompts/third_party/mobile_agent_example.yaml\",\n platform=\"android\",\n )\n assert agent.name == \"test_agent\"\n assert agent.platform == \"android\"\n```\n\n### Common Issues\n\n| Issue | Solution |\n|-------|----------|\n| Agent not registered | Check `@AgentRegistry.register()` decorator |\n| MCP server not responding | Verify MCP server is running on correct port |\n| WebSocket connection failed | Check server URL and network connectivity |\n\n## Related Documentation\n\n- **[Testing Best Practices](../../infrastructure/agents/overview.md#best-practices)** - Agent testing\n- **[Troubleshooting](../../getting_started/quick_start_linux.md#common-issues-troubleshooting)** - Common issues\n\n---\n\n**Previous**: [← Part 4: Configuration](configuration.md) \n**Next**: [Part 6: Complete Example →](example_mobile_agent.md)\n"
},
{
"path": "documents/docs/tutorials/creating_mcp_servers.md",
"content": "# Creating Custom MCP Servers - Complete Tutorial\n\nThis tutorial teaches you how to create, register, and deploy custom MCP servers for UFO² agents. You'll learn to build **local**, **HTTP**, and **stdio** MCP servers, and how to register them with different agents.\n\n**Prerequisites**: Basic Python knowledge, familiarity with [MCP Overview](../mcp/overview.md) and [MCP Configuration](../mcp/configuration.md). Review [Built-in Local Servers](../mcp/local_servers.md) as examples.\n\n---\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Local MCP Servers](#local-mcp-servers)\n3. [HTTP MCP Servers](#http-mcp-servers)\n4. [Stdio MCP Servers](#stdio-mcp-servers)\n5. [Registering Servers with Agents](#registering-servers-with-agents)\n6. [Best Practices](#best-practices)\n7. [Troubleshooting](#troubleshooting)\n\n---\n\n## Overview\n\n### MCP Server Types\n\nUFO² supports three deployment models:\n\n| Type | Deployment | Use Case | Complexity |\n|------|------------|----------|------------|\n| **Local** | In-process with agent | Fast, built-in tools | ⭐ Simple |\n| **HTTP** | Standalone HTTP server | Cross-platform, remote control | ⭐⭐ Moderate |\n| **Stdio** | Child process (stdin/stdout) | Process isolation, third-party tools | ⭐⭐⭐ Advanced |\n\n### Server Categories\n\nAll MCP servers fall into two categories:\n\n| Category | Purpose | LLM Selectable? | Auto-Invoked? |\n|----------|---------|-----------------|---------------|\n| **Data Collection** | Read-only observation | ❌ No | ✅ Yes |\n| **Action** | State-changing execution | ✅ Yes | ❌ No |\n\n**Tool Selection:**\n- **Data Collection tools**: Automatically invoked by the framework to build observation prompts\n- **Action tools**: LLM agent actively selects which tool to execute at each step\n\n**Important**: Write clear docstrings and type annotations - they become LLM instructions!\n\n---\n\n## Local MCP Servers\n\nLocal servers run **in-process** with the UFO² agent, providing the fastest tool access.\n\n### Step 1: Create Your Server\n\nCreate a Python file in `ufo/client/mcp/local_servers/` (or your custom location):\n\n```python\n# File: ufo/client/mcp/local_servers/my_custom_server.py\n\nfrom typing import Annotated\nfrom fastmcp import FastMCP\nfrom pydantic import Field\nfrom ufo.client.mcp.mcp_registry import MCPRegistry\n\n\n@MCPRegistry.register_factory_decorator(\"MyCustomExecutor\")\ndef create_my_custom_server(*args, **kwargs) -> FastMCP:\n \"\"\"\n Create a custom MCP server for specialized automation.\n Factory function registered with MCPRegistry for lazy initialization.\n \n :return: FastMCP instance with custom tools.\n \"\"\"\n \n # Create FastMCP instance\n mcp = FastMCP(\"My Custom MCP Server\")\n \n # Define tools using @mcp.tool() decorator\n @mcp.tool()\n def greet_user(\n name: Annotated[str, Field(description=\"The name of the user to greet.\")],\n formal: Annotated[bool, Field(description=\"Use formal greeting?\")] = False,\n ) -> Annotated[str, Field(description=\"The greeting message.\")]:\n \"\"\"\n Greet a user with a customized message.\n Use formal=True for business contexts, False for casual.\n \"\"\"\n if formal:\n return f\"Good day, {name}. How may I assist you?\"\n else:\n return f\"Hey {name}! What's up?\"\n \n @mcp.tool()\n def calculate_sum(\n numbers: Annotated[\n list[int], \n Field(description=\"List of integers to sum.\")\n ],\n ) -> Annotated[int, Field(description=\"The sum of all numbers.\")]:\n \"\"\"\n Calculate the sum of a list of numbers.\n Useful for quick arithmetic operations.\n \"\"\"\n return sum(numbers)\n \n return mcp\n```\n\n!!!warning \"Critical Design Rules\"\n 1. **Use `@MCPRegistry.register_factory_decorator(\"Namespace\")`** to register the factory\n 2. **Factory function must return a `FastMCP` instance**\n 3. **Use `@mcp.tool()` decorator** for each tool\n 4. **Write detailed docstrings** - they become LLM instructions\n 5. **Use `Annotated[Type, Field(description=\"...\")]`** for all parameters and returns\n 6. **Namespace must be unique** across all servers\n\n### Step 2: Import the Server\n\nAdd your server to `ufo/client/mcp/local_servers/__init__.py`:\n\n```python\n# File: ufo/client/mcp/local_servers/__init__.py\n\nfrom .my_custom_server import create_my_custom_server\n# ... other imports\n\n__all__ = [\n \"create_my_custom_server\",\n # ... other exports\n]\n```\n\n### Step 3: Configure in mcp.yaml\n\nAdd your server to the appropriate agent in `config/ufo/mcp.yaml`:\n\n```yaml\n# For action server (LLM-selectable)\nCustomAgent:\n default:\n action:\n - namespace: MyCustomExecutor\n type: local\n reset: false\n\n# For data collection server (auto-invoked)\nCustomAgent:\n default:\n data_collection:\n - namespace: MyCustomCollector\n type: local\n reset: false\n```\n\n### Step 4: Test Your Server\n\nTest locally before integration:\n\n```python\n# File: test_my_server.py\n\nimport asyncio\nfrom fastmcp.client import Client\nfrom ufo.client.mcp.local_servers.my_custom_server import create_my_custom_server\n\n\nasync def test_server():\n \"\"\"Test the custom MCP server.\"\"\"\n server = create_my_custom_server()\n \n async with Client(server) as client:\n # List available tools\n tools = await client.list_tools()\n print(f\"Available tools: {[t.name for t in tools]}\")\n \n # Test greet_user tool\n result = await client.call_tool(\n \"greet_user\",\n arguments={\"name\": \"Alice\", \"formal\": True}\n )\n print(f\"Greeting: {result.data}\")\n \n # Test calculate_sum tool\n result = await client.call_tool(\n \"calculate_sum\",\n arguments={\"numbers\": [1, 2, 3, 4, 5]}\n )\n print(f\"Sum: {result.data}\")\n\n\nif __name__ == \"__main__\":\n asyncio.run(test_server())\n```\n\n### Example: Application-Specific Server\n\nHere's a real-world example - a server for Chrome browser automation. For more details on wrapping application native APIs, see [Wrapping App Native API](creating_app_agent/warpping_app_native_api.md).\n\n```python\n# File: ufo/client/mcp/local_servers/chrome_executor.py\n\nfrom typing import Annotated, Optional\nfrom fastmcp import FastMCP\nfrom pydantic import Field\nfrom ufo.client.mcp.mcp_registry import MCPRegistry\nfrom ufo.automator.puppeteer import AppPuppeteer\nfrom ufo.automator.action_execution import ActionExecutor\nfrom ufo.agents.processors.schemas.actions import ActionCommandInfo\n\n\n@MCPRegistry.register_factory_decorator(\"ChromeExecutor\")\ndef create_chrome_executor(process_name: str, *args, **kwargs) -> FastMCP:\n \"\"\"\n Create a Chrome-specific automation server.\n \n :param process_name: Chrome process name for UI automation.\n :return: FastMCP instance for Chrome automation.\n \"\"\"\n \n # Initialize puppeteer for Chrome\n puppeteer = AppPuppeteer(\n process_name=process_name,\n app_root_name=\"chrome.exe\",\n )\n executor = ActionExecutor()\n \n def _execute(action: ActionCommandInfo) -> dict:\n \"\"\"Execute action via puppeteer.\"\"\"\n return executor.execute(action, puppeteer, control_dict={})\n \n mcp = FastMCP(\"Chrome Automation MCP Server\")\n \n @mcp.tool()\n def navigate_to_url(\n url: Annotated[str, Field(description=\"The URL to navigate to.\")],\n ) -> Annotated[str, Field(description=\"Navigation result message.\")]:\n \"\"\"\n Navigate Chrome to a specific URL.\n Example: navigate_to_url(url=\"https://www.google.com\")\n \"\"\"\n action = ActionCommandInfo(\n function=\"navigate\",\n arguments={\"url\": url},\n )\n return _execute(action)\n \n @mcp.tool()\n def search_in_page(\n query: Annotated[str, Field(description=\"Search query text.\")],\n case_sensitive: Annotated[\n bool, Field(description=\"Case-sensitive search?\")\n ] = False,\n ) -> Annotated[str, Field(description=\"Search results.\")]:\n \"\"\"\n Search for text in the current Chrome page.\n Returns the number of matches found.\n \"\"\"\n action = ActionCommandInfo(\n function=\"find_in_page\",\n arguments={\"query\": query, \"case_sensitive\": case_sensitive},\n )\n return _execute(action)\n \n @mcp.tool()\n def get_page_title() -> Annotated[str, Field(description=\"The page title.\")]:\n \"\"\"\n Get the title of the current Chrome page.\n Useful for verifying page navigation.\n \"\"\"\n action = ActionCommandInfo(function=\"get_title\", arguments={})\n return _execute(action)\n \n return mcp\n```\n\n**Configuration:**\n\n```yaml\nAppAgent:\n chrome.exe:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: AppUIExecutor # Generic UI automation\n type: local\n - namespace: ChromeExecutor # Chrome-specific tools\n type: local\n reset: true # Reset when switching tabs/windows\n```\n\n---\n\n## HTTP MCP Servers\n\nHTTP servers run as **standalone services**, enabling cross-platform automation and distributed workflows.\n\n### Step 1: Create HTTP Server\n\nCreate a standalone Python script:\n\n```python\n# File: ufo/client/mcp/http_servers/my_http_server.py\n\nimport argparse\nfrom typing import Annotated, Any, Dict\nfrom fastmcp import FastMCP\nfrom pydantic import Field\n\n\ndef create_my_http_server(host: str = \"localhost\", port: int = 8020) -> None:\n \"\"\"\n Create and run an HTTP MCP server.\n \n :param host: Host address to bind the server.\n :param port: Port number for the server.\n \"\"\"\n \n # Create FastMCP with HTTP transport\n mcp = FastMCP(\n \"My Custom HTTP MCP Server\",\n instructions=\"Custom automation server via HTTP.\",\n stateless_http=True, # Stateless HTTP (one-shot JSON)\n json_response=True, # Return pure JSON bodies\n host=host,\n port=port,\n )\n \n @mcp.tool()\n async def process_data(\n data: Annotated[str, Field(description=\"Data to process.\")],\n transform: Annotated[\n str, Field(description=\"Transformation type: 'upper', 'lower', 'reverse'.\")\n ] = \"upper\",\n ) -> Annotated[Dict[str, Any], Field(description=\"Processing result.\")]:\n \"\"\"\n Process text data with various transformations.\n Supports: 'upper' (uppercase), 'lower' (lowercase), 'reverse' (reverse string).\n \"\"\"\n try:\n if transform == \"upper\":\n result = data.upper()\n elif transform == \"lower\":\n result = data.lower()\n elif transform == \"reverse\":\n result = data[::-1]\n else:\n return {\"success\": False, \"error\": f\"Unknown transform: {transform}\"}\n \n return {\n \"success\": True,\n \"original\": data,\n \"transformed\": result,\n \"transform_type\": transform,\n }\n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n \n @mcp.tool()\n async def get_server_info() -> Annotated[\n Dict[str, Any], Field(description=\"Server information.\")\n ]:\n \"\"\"\n Get information about the HTTP MCP server.\n Returns server name, version, and status.\n \"\"\"\n import platform\n return {\n \"server\": \"My Custom HTTP MCP Server\",\n \"version\": \"1.0.0\",\n \"platform\": platform.system(),\n \"status\": \"running\",\n }\n \n # Start the HTTP server\n mcp.run(transport=\"streamable-http\")\n\n\ndef main():\n \"\"\"Main entry point for the HTTP server.\"\"\"\n parser = argparse.ArgumentParser(description=\"My Custom HTTP MCP Server\")\n parser.add_argument(\"--port\", type=int, default=8020, help=\"Server port\")\n parser.add_argument(\"--host\", default=\"localhost\", help=\"Server host\")\n args = parser.parse_args()\n \n print(\"=\" * 60)\n print(\"My Custom HTTP MCP Server\")\n print(f\"Running on {args.host}:{args.port}\")\n print(\"=\" * 60)\n \n create_my_http_server(host=args.host, port=args.port)\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n### Step 2: Start the HTTP Server\n\nRun the server as a standalone process:\n\n```bash\n# Start on localhost\npython -m ufo.client.mcp.http_servers.my_http_server --host localhost --port 8020\n\n# Start on all interfaces (for remote access)\npython -m ufo.client.mcp.http_servers.my_http_server --host 0.0.0.0 --port 8020\n```\n\n**For production, run as a background service:**\n\n**Linux/macOS:**\n```bash\nnohup python -m ufo.client.mcp.http_servers.my_http_server --host 0.0.0.0 --port 8020 &\n```\n\n**Windows:**\n```powershell\nStart-Process python -ArgumentList \"-m\", \"ufo.client.mcp.http_servers.my_http_server\", \"--host\", \"0.0.0.0\", \"--port\", \"8020\" -WindowStyle Hidden\n```\n\n### Step 3: Configure HTTP Server in mcp.yaml\n\n```yaml\nRemoteAgent:\n default:\n action:\n - namespace: MyHTTPExecutor\n type: http\n host: \"localhost\" # Or remote IP: \"192.168.1.100\"\n port: 8020\n path: \"/mcp\"\n reset: false\n```\n\n### Step 4: Test HTTP Server\n\nTest connectivity before integration:\n\n```python\n# File: test_http_server.py\n\nimport asyncio\nfrom fastmcp.client import Client\n\n\nasync def test_http_server():\n \"\"\"Test the HTTP MCP server.\"\"\"\n server_url = \"http://localhost:8020/mcp\"\n \n async with Client(server_url) as client:\n # List tools\n tools = await client.list_tools()\n print(f\"Available tools: {[t.name for t in tools]}\")\n \n # Test process_data\n result = await client.call_tool(\n \"process_data\",\n arguments={\"data\": \"Hello World\", \"transform\": \"reverse\"}\n )\n print(f\"Process result: {result.data}\")\n \n # Test get_server_info\n result = await client.call_tool(\"get_server_info\", arguments={})\n print(f\"Server info: {result.data}\")\n\n\nif __name__ == \"__main__\":\n asyncio.run(test_http_server())\n```\n\n### Example: Cross-Platform Linux Executor\n\nReal-world example - controlling Linux systems from Windows:\n\n```python\n# File: ufo/client/mcp/http_servers/linux_executor.py\n\nimport argparse\nimport asyncio\nfrom typing import Annotated, Any, Dict, Optional\nfrom fastmcp import FastMCP\nfrom pydantic import Field\n\n\ndef create_linux_executor(host: str = \"0.0.0.0\", port: int = 8010) -> None:\n \"\"\"Linux command execution MCP server.\"\"\"\n \n mcp = FastMCP(\n \"Linux Executor MCP Server\",\n instructions=\"Execute shell commands on Linux.\",\n stateless_http=True,\n json_response=True,\n host=host,\n port=port,\n )\n \n @mcp.tool()\n async def execute_command(\n command: Annotated[str, Field(description=\"Shell command to execute.\")],\n timeout: Annotated[int, Field(description=\"Timeout in seconds.\")] = 30,\n cwd: Annotated[\n Optional[str], Field(description=\"Working directory.\")\n ] = None,\n ) -> Annotated[Dict[str, Any], Field(description=\"Execution result.\")]:\n \"\"\"\n Execute a shell command on Linux and return stdout/stderr.\n Dangerous commands (rm -rf /, shutdown, etc.) are blocked.\n \"\"\"\n # Security check\n dangerous = [\"rm -rf /\", \"shutdown\", \"reboot\", \"mkfs\"]\n if any(d in command.lower() for d in dangerous):\n return {\"success\": False, \"error\": \"Blocked dangerous command.\"}\n \n try:\n proc = await asyncio.create_subprocess_shell(\n command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n cwd=cwd,\n )\n \n try:\n stdout, stderr = await asyncio.wait_for(\n proc.communicate(), timeout=timeout\n )\n except asyncio.TimeoutError:\n proc.kill()\n await proc.wait()\n return {\"success\": False, \"error\": f\"Timeout after {timeout}s.\"}\n \n return {\n \"success\": proc.returncode == 0,\n \"exit_code\": proc.returncode,\n \"stdout\": stdout.decode(\"utf-8\", errors=\"replace\"),\n \"stderr\": stderr.decode(\"utf-8\", errors=\"replace\"),\n }\n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n \n @mcp.tool()\n async def get_system_info() -> Annotated[\n Dict[str, Any], Field(description=\"System information.\")\n ]:\n \"\"\"Get basic Linux system information.\"\"\"\n info = {}\n cmds = {\n \"uname\": \"uname -a\",\n \"uptime\": \"uptime\",\n \"memory\": \"free -h\",\n }\n for key, cmd in cmds.items():\n try:\n proc = await asyncio.create_subprocess_shell(\n cmd, stdout=asyncio.subprocess.PIPE\n )\n out, _ = await proc.communicate()\n info[key] = out.decode(\"utf-8\", errors=\"replace\").strip()\n except Exception as e:\n info[key] = f\"Error: {e}\"\n return info\n \n mcp.run(transport=\"streamable-http\")\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Linux Executor MCP Server\")\n parser.add_argument(\"--port\", type=int, default=8010)\n parser.add_argument(\"--host\", default=\"0.0.0.0\")\n args = parser.parse_args()\n \n print(f\"Linux Executor running on {args.host}:{args.port}\")\n create_linux_executor(host=args.host, port=args.port)\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n**Deploy on Linux:**\n\n```bash\n# Start server on Linux machine\npython -m ufo.client.mcp.http_servers.linux_executor --host 0.0.0.0 --port 8010\n```\n\n**Configure on Windows UFO²:**\n\n```yaml\nLinuxAgent:\n default:\n action:\n - namespace: LinuxExecutor\n type: http\n host: \"192.168.1.50\" # Linux machine IP\n port: 8010\n path: \"/mcp\"\n```\n\n**Cross-Platform Workflow**: Now your Windows UFO² agent can execute Linux commands remotely! The LLM will select `execute_command` or `get_system_info` as needed.\n\n---\n\n## Stdio MCP Servers\n\nStdio servers run as **child processes**, communicating via stdin/stdout. They provide process isolation and work with any language.\n\n### Step 1: Create Stdio Server\n\nCreate a standalone script that reads JSON-RPC from stdin and writes to stdout:\n\n```python\n# File: custom_stdio_server.py\n\nimport sys\nimport json\nfrom typing import Any, Dict\n\n\ndef handle_request(request: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"\n Handle incoming MCP request.\n \n :param request: JSON-RPC request from stdin.\n :return: JSON-RPC response.\n \"\"\"\n method = request.get(\"method\", \"\")\n params = request.get(\"params\", {})\n \n if method == \"tools/list\":\n # Return available tools\n return {\n \"jsonrpc\": \"2.0\",\n \"id\": request.get(\"id\"),\n \"result\": {\n \"tools\": [\n {\n \"name\": \"echo\",\n \"description\": \"Echo back a message.\",\n \"inputSchema\": {\n \"type\": \"object\",\n \"properties\": {\n \"message\": {\n \"type\": \"string\",\n \"description\": \"Message to echo.\",\n }\n },\n \"required\": [\"message\"],\n },\n }\n ]\n },\n }\n \n elif method == \"tools/call\":\n tool_name = params.get(\"name\", \"\")\n arguments = params.get(\"arguments\", {})\n \n if tool_name == \"echo\":\n message = arguments.get(\"message\", \"\")\n return {\n \"jsonrpc\": \"2.0\",\n \"id\": request.get(\"id\"),\n \"result\": {\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": f\"Echo: {message}\",\n }\n ]\n },\n }\n else:\n return {\n \"jsonrpc\": \"2.0\",\n \"id\": request.get(\"id\"),\n \"error\": {\n \"code\": -32601,\n \"message\": f\"Unknown tool: {tool_name}\",\n },\n }\n \n else:\n return {\n \"jsonrpc\": \"2.0\",\n \"id\": request.get(\"id\"),\n \"error\": {\n \"code\": -32601,\n \"message\": f\"Unknown method: {method}\",\n },\n }\n\n\ndef main():\n \"\"\"Main stdio loop.\"\"\"\n for line in sys.stdin:\n try:\n request = json.loads(line)\n response = handle_request(request)\n print(json.dumps(response), flush=True)\n except Exception as e:\n error_response = {\n \"jsonrpc\": \"2.0\",\n \"id\": None,\n \"error\": {\n \"code\": -32603,\n \"message\": str(e),\n },\n }\n print(json.dumps(error_response), flush=True)\n\n\nif __name__ == \"__main__\":\n main()\n```\n\n### Step 2: Configure Stdio Server in mcp.yaml\n\n```yaml\nCustomAgent:\n default:\n action:\n - namespace: CustomStdioExecutor\n type: stdio\n command: \"python\"\n start_args: [\"custom_stdio_server.py\"]\n env:\n API_KEY: \"secret_key\"\n LOG_LEVEL: \"INFO\"\n cwd: \"/path/to/server/directory\"\n reset: false\n```\n\n!!!warning \"Stdio Limitations\"\n - **More complex** than local/HTTP servers\n - Requires implementing **JSON-RPC protocol** manually\n - Better suited for **third-party MCP servers** than custom tools\n - For custom Python tools, **prefer local or HTTP servers**\n\n### Example: Third-Party Node.js Server\n\nStdio is ideal for integrating existing MCP servers written in other languages:\n\n```yaml\nCustomAgent:\n default:\n action:\n - namespace: NodeJSTools\n type: stdio\n command: \"node\"\n start_args: [\"./node_mcp_server/index.js\"]\n env:\n NODE_ENV: \"production\"\n cwd: \"/path/to/node_mcp_server\"\n```\n\n---\n\n## Registering Servers with Agents\n\n### Agent-Specific Registration\n\nDifferent agents can use different MCP server configurations:\n\n```yaml\n# HostAgent: System-level automation\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: HostUIExecutor\n type: local\n - namespace: CommandLineExecutor\n type: local\n\n# AppAgent: Application-specific automation\nAppAgent:\n # Default configuration for all apps\n default:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: CommandLineExecutor\n type: local\n \n # Word-specific configuration\n WINWORD.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: WordCOMExecutor # Word COM API\n type: local\n reset: true\n - namespace: CommandLineExecutor\n type: local\n \n # Excel-specific configuration\n EXCEL.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: ExcelCOMExecutor # Excel COM API\n type: local\n reset: true\n \n # Chrome-specific configuration\n chrome.exe:\n data_collection:\n - namespace: UICollector\n type: local\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: ChromeExecutor # Custom Chrome tools\n type: local\n reset: true\n\n# Custom Agent: Specialized automation\nCustomAutomationAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n - namespace: MyCustomCollector # Custom data collection\n type: local\n action:\n - namespace: MyCustomExecutor # Custom actions\n type: local\n - namespace: MyHTTPExecutor # Remote HTTP actions\n type: http\n host: \"192.168.1.100\"\n port: 8020\n path: \"/mcp\"\n```\n\n### Multi-Server Agent Configuration\n\nAgents can register **multiple servers** of the same category:\n\n```yaml\nHybridAgent:\n default:\n # Multiple data collection sources\n data_collection:\n - namespace: UICollector\n type: local\n - namespace: HardwareCollector # Remote hardware monitoring\n type: http\n host: \"192.168.1.50\"\n port: 8006\n path: \"/mcp\"\n - namespace: SystemMetrics # Custom metrics\n type: local\n \n # Multiple action executors (LLM chooses best tool)\n action:\n - namespace: AppUIExecutor # GUI automation\n type: local\n - namespace: WordCOMExecutor # API automation\n type: local\n reset: true\n - namespace: LinuxExecutor # Remote Linux control\n type: http\n host: \"192.168.1.100\"\n port: 8010\n path: \"/mcp\"\n - namespace: CustomExecutor # Custom actions\n type: local\n```\n\n**How it works:**\n\n1. **Data collection tools**: All servers are invoked automatically to build observation\n2. **Action tools**: LLM sees tools from ALL action servers and selects the best one\n\n**Example LLM decision:**\n\n```\nTask: \"Create a Word document with sales data from the Linux database\"\n\nStep 1: Get data from Linux\n → LLM selects: LinuxExecutor::execute_command(\n command=\"mysql -e 'SELECT * FROM sales'\"\n )\n\nStep 2: Create Word document\n → LLM selects: WordCOMExecutor::insert_table(rows=10, columns=3)\n\nStep 3: Format the table\n → LLM selects: WordCOMExecutor::select_table(number=1)\n → AppUIExecutor::click_input(name=\"Table Design\")\n```\n\n### Configuration Hierarchy\n\nAgent configurations follow this **inheritance hierarchy**:\n\n```\nAgentName\n ├─ default (fallback configuration)\n │ ├─ data_collection\n │ └─ action\n └─ SubType (e.g., \"WINWORD.EXE\")\n ├─ data_collection\n └─ action\n```\n\n**Lookup logic:**\n\n1. Check for `AgentName.SubType`\n2. If not found, use `AgentName.default`\n3. If neither exists, raise error\n\n**Example:**\n\n```yaml\nAppAgent:\n # Fallback for all apps\n default:\n action:\n - namespace: AppUIExecutor\n type: local\n \n # Overrides default for Word\n WINWORD.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: WordCOMExecutor\n type: local\n```\n\n---\n\n## Best Practices\n\n### 1. Write Comprehensive Docstrings\n\nYour docstrings are **directly converted to LLM prompts**. The LLM uses them to understand:\n- **What** the tool does\n- **When** to use it\n- **How** to use it correctly\n\n**Bad Example:**\n```python\n@mcp.tool()\ndef process(data: str) -> str:\n \"\"\"Process data.\"\"\" # ❌ Too vague\n return data.upper()\n```\n\n**Good Example:**\n```python\n@mcp.tool()\ndef process_text_to_uppercase(\n text: Annotated[str, Field(description=\"The input text to convert.\")],\n) -> Annotated[str, Field(description=\"The text converted to uppercase.\")]:\n \"\"\"\n Convert text to uppercase letters.\n \n Use this tool when you need to standardize text formatting or make text\n more prominent. Works with all Unicode characters.\n \n Examples:\n - \"hello world\" → \"HELLO WORLD\"\n - \"Café\" → \"CAFÉ\"\n \"\"\" # ✅ Clear, detailed, with examples\n return text.upper()\n```\n\n### 2. Use Descriptive Parameter Names\n\n```python\n# ❌ Bad: Unclear parameter names\n@mcp.tool()\ndef func(a: str, b: int, c: bool) -> str:\n ...\n\n# ✅ Good: Self-documenting parameter names\n@mcp.tool()\ndef send_email(\n recipient_address: str,\n message_body: str,\n use_html_format: bool = False,\n) -> str:\n ...\n```\n\n### 3. Provide Default Values\n\n```python\n@mcp.tool()\ndef search_files(\n query: Annotated[str, Field(description=\"Search query.\")],\n case_sensitive: Annotated[\n bool, Field(description=\"Case-sensitive search?\")\n ] = False, # ✅ Sensible default\n max_results: Annotated[\n int, Field(description=\"Maximum results to return.\")\n ] = 10, # ✅ Sensible default\n) -> list[str]:\n \"\"\"Search for files matching the query.\"\"\"\n ...\n```\n\n### 4. Handle Errors Gracefully\n\n```python\n@mcp.tool()\ndef divide_numbers(\n dividend: Annotated[float, Field(description=\"Number to divide.\")],\n divisor: Annotated[float, Field(description=\"Number to divide by.\")],\n) -> Annotated[dict, Field(description=\"Division result or error.\")]:\n \"\"\"\n Divide two numbers and return the result.\n Returns an error if divisor is zero.\n \"\"\"\n try:\n if divisor == 0:\n return {\n \"success\": False,\n \"error\": \"Cannot divide by zero.\",\n }\n \n result = dividend / divisor\n return {\n \"success\": True,\n \"result\": result,\n }\n except Exception as e:\n return {\n \"success\": False,\n \"error\": f\"Division failed: {str(e)}\",\n }\n```\n\n### 5. Use Reset for Stateful Servers\n\n```yaml\n# ✅ Good: Reset COM servers when switching contexts\nAppAgent:\n WINWORD.EXE:\n action:\n - namespace: WordCOMExecutor\n type: local\n reset: true # Prevents state leakage between documents\n\n# ❌ Bad: Not resetting can cause issues\nAppAgent:\n WINWORD.EXE:\n action:\n - namespace: WordCOMExecutor\n type: local\n reset: false # May retain state from previous document\n```\n\n### 6. Validate Remote Server Connectivity\n\nBefore deploying, test connectivity:\n\n```python\nimport asyncio\nfrom fastmcp.client import Client\n\n\nasync def validate_server(url: str):\n \"\"\"Validate HTTP server is accessible.\"\"\"\n try:\n async with Client(url) as client:\n tools = await client.list_tools()\n print(f\"✅ Server {url} is accessible\")\n print(f\" Tools: {[t.name for t in tools]}\")\n return True\n except Exception as e:\n print(f\"❌ Server {url} is NOT accessible: {e}\")\n return False\n\n\n# Test before adding to mcp.yaml\nasyncio.run(validate_server(\"http://192.168.1.100:8020/mcp\"))\n```\n\n### 7. Use Environment Variables for Secrets\n\n```yaml\n# ❌ Bad: Hardcoded secrets\nCustomAgent:\n default:\n action:\n - namespace: APIExecutor\n type: http\n host: \"api.example.com\"\n port: 443\n auth_token: \"sk-1234567890\" # Don't commit this!\n\n# ✅ Good: Use environment variables\nCustomAgent:\n default:\n action:\n - namespace: APIExecutor\n type: http\n host: \"${API_HOST}\"\n port: \"${API_PORT}\"\n auth_token: \"${API_TOKEN}\"\n```\n\nSet environment variables before running UFO²:\n\n```bash\nexport API_HOST=\"api.example.com\"\nexport API_PORT=\"443\"\nexport API_TOKEN=\"sk-1234567890\"\n```\n\n---\n\n## Troubleshooting\n\n### Common Issues\n\n#### 1. \"No MCP server found for name 'MyServer'\"\n\n**Cause**: Server not registered in MCPRegistry.\n\n**Solution**:\n```python\n# Ensure you're using the decorator\n@MCPRegistry.register_factory_decorator(\"MyServer\")\ndef create_my_server(*args, **kwargs) -> FastMCP:\n ...\n\n# Or manually register\nMCPRegistry.register_factory(\"MyServer\", create_my_server)\n```\n\n#### 2. \"Connection refused\" for HTTP Server\n\n**Cause**: HTTP server not running or wrong host/port.\n\n**Solution**:\n```bash\n# Verify server is running\ncurl http://localhost:8020/mcp\n\n# Check firewall rules\n# Windows:\nnetsh advfirewall firewall add rule name=\"MCP Server\" dir=in action=allow protocol=TCP localport=8020\n\n# Linux:\nsudo ufw allow 8020/tcp\n```\n\n#### 3. Tools Not Appearing in LLM Prompt\n\n**Cause**: Server registered in wrong category (data_collection vs action).\n\n**Solution**:\n```yaml\n# For LLM-selectable tools, use 'action'\nCustomAgent:\n default:\n action: # ✅ Correct for LLM-selectable tools\n - namespace: MyExecutor\n type: local\n\n# For auto-invoked observation, use 'data_collection'\nCustomAgent:\n default:\n data_collection: # ✅ Correct for automatic observation\n - namespace: MyCollector\n type: local\n```\n\n#### 4. Server State Leaking Between Contexts\n\n**Cause**: `reset: false` for stateful servers.\n\n**Solution**:\n```yaml\n# Set reset: true for stateful servers\nAppAgent:\n WINWORD.EXE:\n action:\n - namespace: WordCOMExecutor\n type: local\n reset: true # ✅ Reset COM state when switching documents\n```\n\n#### 5. Timeout Errors for Long-Running Tools\n\n**Cause**: Default timeout is 6000 seconds (100 minutes).\n\n**Solution**:\n```python\n# In Computer class, adjust timeout\nself._tool_timeout = 12000 # 200 minutes\n```\n\n### Debugging Tips\n\n#### Enable Debug Logging\n\n```python\nimport logging\n\nlogging.basicConfig(level=logging.DEBUG)\nlogger = logging.getLogger(\"ufo.client.mcp\")\n```\n\n#### Check Registered Servers\n\n```python\nfrom ufo.client.mcp.mcp_server_manager import MCPServerManager\n\n# List all registered servers\nfor namespace, server in MCPServerManager._servers_mapping.items():\n print(f\"Server: {namespace}, Type: {type(server).__name__}\")\n```\n\n#### Test Server in Isolation\n\n```python\n# Test local server\nfrom ufo.client.mcp.local_servers.my_custom_server import create_my_custom_server\nimport asyncio\nfrom fastmcp.client import Client\n\n\nasync def test():\n server = create_my_custom_server()\n async with Client(server) as client:\n tools = await client.list_tools()\n print(f\"Tools: {[t.name for t in tools]}\")\n\n\nasyncio.run(test())\n```\n\n---\n\n## Next Steps\n\nNow that you've learned to create MCP servers, explore these related topics:\n\n1. **Review Built-in Servers**: See [Local Servers](../mcp/local_servers.md) for production examples\n2. **Explore HTTP Deployment**: Read [Remote Servers](../mcp/remote_servers.md) for cross-platform automation\n3. **Understand Agent Configuration**: Study [MCP Configuration](../mcp/configuration.md) for advanced setups\n4. **Learn about Computer Class**: Review [Computer](../client/computer.md) to understand the MCP client integration\n5. **Create Your First Agent**: Follow [Creating App Agent](creating_app_agent/overview.md) to build custom agents\n\n---\n\n## Related Documentation\n\n- [MCP Overview](../mcp/overview.md) - MCP architecture and concepts\n- [MCP Configuration](../mcp/configuration.md) - Complete configuration reference\n- [Local Servers](../mcp/local_servers.md) - Built-in local servers\n- [Remote Servers](../mcp/remote_servers.md) - HTTP/Stdio deployment\n- [Data Collection Servers](../mcp/data_collection.md) - Observation tools\n- [Action Servers](../mcp/action.md) - Execution tools\n- [MCP Reference](../configuration/system/mcp_reference.md) - Quick reference guide\n\n---\n\n## Best Practices Summary\n\n- ✅ **Write clear docstrings** - they become LLM instructions\n- ✅ **Use descriptive names** - for tools, parameters, and namespaces\n- ✅ **Handle errors gracefully** - return structured error messages\n- ✅ **Test in isolation** - before integrating with agents\n- ✅ **Use `reset: true`** - for stateful servers (COM, API clients)\n- ✅ **Validate connectivity** - for HTTP/Stdio servers before deployment\n"
},
{
"path": "documents/docs/tutorials/creating_third_party_agents.md",
"content": "# Creating Custom Third-Party Agents - Complete Tutorial\n\nThis tutorial teaches you how to create, register, and deploy custom third-party agents that extend UFO²'s capabilities beyond Windows GUI automation. You'll learn the complete process using **HardwareAgent** as a reference implementation.\n\n**Prerequisites**: Basic Python knowledge, familiarity with UFO² agent architecture, [Agent Configuration](../configuration/system/agents_config.md), and [Third-Party Configuration](../configuration/system/third_party_config.md).\n\n---\n\n## Table of Contents\n\n1. [Overview](#overview)\n2. [Understanding Third-Party Agents](#understanding-third-party-agents)\n3. [Step-by-Step Implementation](#step-by-step-implementation)\n4. [Complete Example: HardwareAgent](#complete-example-hardwareagent)\n5. [Registering with HostAgent](#registering-with-hostagent)\n6. [Configuration and Deployment](#configuration-and-deployment)\n7. [Best Practices](#best-practices)\n8. [Troubleshooting](#troubleshooting)\n\n---\n\n## Overview\n\n### What are Third-Party Agents?\n\nThird-party agents are specialized agents that extend UFO²'s capabilities to handle tasks beyond standard Windows GUI automation. They work alongside the core agents (HostAgent and AppAgent) to provide domain-specific functionality.\n\n**Key Characteristics**:\n- ✅ Independent agent implementation with custom logic\n- ✅ Registered and managed by HostAgent\n- ✅ Selectable as execution targets by the LLM\n- ✅ Can use MCP servers and custom tools\n- ✅ Configurable via YAML files\n\n**Common Use Cases**:\n- 🔧 **Hardware Control**: Physical device manipulation (HardwareAgent)\n- 🐧 **Linux CLI**: Server and CLI command execution (LinuxAgent)\n- 🌐 **Web Automation**: Browser-based tasks without GUI\n- 📡 **IoT Integration**: Smart device control\n- 🤖 **Robotic Process Automation**: Custom automation workflows\n\n---\n\n## Understanding Third-Party Agents\n\n### Architecture Overview\n\nThird-party agents integrate with UFO² through a well-defined architecture:\n\n```mermaid\ngraph TB\n HostAgent[\"HostAgent - Orchestrates all agents - Registers third-party agents as selectable targets - Routes tasks to appropriate agents\"]\n \n AppAgent[\"AppAgent (GUI tasks)\"]\n HardwareAgent[\"HardwareAgent (Hardware)\"]\n YourAgent[\"YourAgent (Custom)\"]\n \n Strategies[\"Processing Strategies - LLM Interaction - Action Execution - Memory Updates\"]\n \n HostAgent --> AppAgent\n HostAgent --> HardwareAgent\n HostAgent --> YourAgent\n \n AppAgent --> Strategies\n HardwareAgent --> Strategies\n YourAgent --> Strategies\n \n style HostAgent fill:#e1f5ff,stroke:#0288d1,stroke-width:2px\n style AppAgent fill:#f3e5f5,stroke:#9c27b0,stroke-width:2px\n style HardwareAgent fill:#fff3e0,stroke:#ff9800,stroke-width:2px\n style YourAgent fill:#e8f5e9,stroke:#4caf50,stroke-width:2px\n style Strategies fill:#fce4ec,stroke:#e91e63,stroke-width:2px\n```\n\n### Agent Registry System\n\nUFO² uses a registry pattern to dynamically load and manage agents:\n\n```python\n@AgentRegistry.register(\n agent_name=\"YourAgent\", # Unique identifier\n third_party=True, # Mark as third-party\n processor_cls=YourProcessor # Processing logic\n)\nclass YourAgent(CustomizedAgent):\n \"\"\"Your custom agent implementation.\"\"\"\n pass\n```\n\n**How it works**:\n\n1. **Registration**: `@AgentRegistry.register()` decorator registers your agent class\n2. **Filtering**: Registry checks if agent is in `ENABLED_THIRD_PARTY_AGENTS` config\n3. **Instantiation**: HostAgent creates instances when needed\n4. **Target Selection**: LLM can select your agent as an execution target\n\n---\n\n## Step-by-Step Implementation\n\n### Step 1: Create Agent Class\n\nCreate your agent class by inheriting from `CustomizedAgent`:\n\n```python\n# File: ufo/agents/agent/customized_agent.py\n\nfrom ufo.agents.agent.app_agent import AppAgent\nfrom ufo.agents.agent.basic import AgentRegistry\nfrom ufo.agents.processors.customized.customized_agent_processor import (\n CustomizedProcessor,\n YourAgentProcessor, # Import your processor\n)\n\n@AgentRegistry.register(\n agent_name=\"YourAgent\",\n third_party=True,\n processor_cls=YourAgentProcessor\n)\nclass YourAgent(CustomizedAgent):\n \"\"\"\n YourAgent is a specialized agent that handles [specific functionality].\n \n This agent extends CustomizedAgent to provide:\n - Custom domain logic (e.g., hardware control, web automation)\n - Specialized action execution\n - Domain-specific tool integration\n \"\"\"\n \n def __init__(\n self,\n name: str,\n main_prompt: str,\n example_prompt: str,\n api_prompt: str = None,\n ) -> None:\n \"\"\"\n Initialize YourAgent.\n \n :param name: The name of the agent instance\n :param main_prompt: Path to main prompt template YAML\n :param example_prompt: Path to example prompt template YAML\n :param api_prompt: Optional path to API prompt template YAML\n \"\"\"\n super().__init__(\n name=name,\n main_prompt=main_prompt,\n example_prompt=example_prompt,\n process_name=None,\n app_root_name=None,\n is_visual=None, # Set True if your agent uses screenshots\n )\n \n # Optional: Add custom initialization\n self._custom_state = {}\n self.logger.info(f\"YourAgent initialized with prompts: {main_prompt}\")\n\n # Optional: Override methods for custom behavior\n def get_prompter(self, is_visual: bool, main_prompt: str, example_prompt: str):\n \"\"\"Get the prompter for your agent.\"\"\"\n # Use default or create custom prompter\n return super().get_prompter(is_visual, main_prompt, example_prompt)\n```\n\n**Key Points**:\n- ✅ **Inherit from `CustomizedAgent`**: Provides base functionality\n- ✅ **Use `@AgentRegistry.register()`**: Enables dynamic loading\n- ✅ **Set `third_party=True`**: Triggers configuration filtering\n- ✅ **Specify `processor_cls`**: Links to your processing logic\n\n---\n\n### Step 2: Create Processor Class\n\nCreate a processor that defines how your agent processes tasks. For detailed information about processors and strategies, see [Agent Architecture](../infrastructure/agents/overview.md).\n\n```python\n# File: ufo/agents/processors/customized/customized_agent_processor.py\n\nfrom typing import TYPE_CHECKING\nfrom ufo.agents.processors.app_agent_processor import AppAgentProcessor\nfrom ufo.agents.processors.context.processing_context import (\n ProcessingContext,\n ProcessingPhase,\n)\nfrom ufo.agents.processors.strategies.app_agent_processing_strategy import (\n AppActionExecutionStrategy,\n AppMemoryUpdateStrategy,\n)\nfrom ufo.agents.processors.strategies.customized_agent_processing_strategy import (\n CustomizedLLMInteractionStrategy,\n CustomizedScreenshotCaptureStrategy,\n)\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import YourAgent\n\n\nclass YourAgentProcessor(CustomizedProcessor):\n \"\"\"\n Processor for YourAgent - defines processing pipeline and strategies.\n \"\"\"\n\n def __init__(self, agent: \"YourAgent\", global_context: \"Context\") -> None:\n \"\"\"\n Initialize YourAgent processor.\n \n :param agent: The YourAgent instance\n :param global_context: Global context shared across processing\n \"\"\"\n super().__init__(agent, global_context)\n\n def _setup_strategies(self) -> None:\n \"\"\"\n Setup processing strategies for YourAgent.\n \n Define how your agent processes each phase:\n - DATA_COLLECTION: Gather observations (screenshots, data)\n - LLM_INTERACTION: Communicate with LLM to get actions\n - ACTION_EXECUTION: Execute the selected action\n - MEMORY_UPDATE: Update agent memory and history\n \"\"\"\n \n # Phase 1: Data Collection (if your agent uses visual input)\n self.strategies[ProcessingPhase.DATA_COLLECTION] = (\n CustomizedScreenshotCaptureStrategy(\n fail_fast=True, # Stop if screenshot capture fails\n )\n )\n\n # Phase 2: LLM Interaction\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n CustomizedLLMInteractionStrategy(\n fail_fast=True # LLM failures should halt processing\n )\n )\n\n # Phase 3: Action Execution\n # Option A: Use default strategy\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n AppActionExecutionStrategy(\n fail_fast=False # Continue on action failures\n )\n )\n \n # Option B: Create custom strategy (see Step 3)\n # self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n # YourActionExecutionStrategy(fail_fast=False)\n # )\n\n # Phase 4: Memory Update\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(\n fail_fast=False # Memory failures shouldn't stop agent\n )\n )\n\n def _setup_middleware(self) -> None:\n \"\"\"\n Optional: Setup middleware for logging, error handling, etc.\n \"\"\"\n # Use default middleware or add custom middleware\n super()._setup_middleware()\n \n # Example: Add custom middleware\n # self.middleware_chain.append(YourCustomMiddleware())\n```\n\n**Strategy Setup Guidelines**:\n\n| Phase | Purpose | fail_fast | Strategy Options |\n|-------|---------|-----------|------------------|\n| **DATA_COLLECTION** | Capture observations | `True` | Screenshot, sensor data, API calls |\n| **LLM_INTERACTION** | Get LLM decision | `True` | Custom prompts, function calling |\n| **ACTION_EXECUTION** | Execute action | `False` | Custom tools, API calls, commands |\n| **MEMORY_UPDATE** | Save history | `False` | Standard or custom memory logic |\n\n---\n\n### Step 3: Create Custom Strategies (Optional)\n\nIf you need custom processing logic, create strategy classes:\n\n```python\n# File: ufo/agents/processors/strategies/your_agent_strategy.py\n\nfrom typing import TYPE_CHECKING\nfrom ufo.agents.processors.strategies.base import (\n BaseProcessingStrategy,\n ProcessingResult,\n)\nfrom ufo.agents.processors.context.processing_context import ProcessingContext\n\nif TYPE_CHECKING:\n from ufo.agents.agent.customized_agent import YourAgent\n\n\nclass YourActionExecutionStrategy(BaseProcessingStrategy):\n \"\"\"\n Custom action execution strategy for YourAgent.\n \"\"\"\n\n def __init__(self, fail_fast: bool = False) -> None:\n super().__init__(name=\"your_action_execution\", fail_fast=fail_fast)\n\n async def execute(\n self, \n agent: \"YourAgent\", \n context: ProcessingContext\n ) -> ProcessingResult:\n \"\"\"\n Execute custom actions for your agent.\n \n :param agent: YourAgent instance\n :param context: Processing context with LLM response\n :return: ProcessingResult with execution outcome\n \"\"\"\n try:\n # Extract action from LLM response\n parsed_response = context.get_local(\"parsed_response\")\n function_name = parsed_response.get(\"function\")\n arguments = parsed_response.get(\"arguments\", {})\n \n self.logger.info(f\"Executing action: {function_name}\")\n\n # Execute your custom action logic\n if function_name == \"your_custom_action\":\n result = self._execute_custom_action(arguments)\n else:\n # Fallback to standard action execution\n result = await self._execute_standard_action(\n agent, function_name, arguments\n )\n\n # Store results in context\n context.set_local(\"action_result\", result)\n context.set_local(\"action_status\", \"success\")\n\n return ProcessingResult(\n success=True,\n data={\"result\": result},\n error=None\n )\n\n except Exception as e:\n self.logger.error(f\"Action execution failed: {str(e)}\")\n \n return ProcessingResult(\n success=False,\n data={},\n error=str(e)\n )\n\n def _execute_custom_action(self, arguments: dict) -> dict:\n \"\"\"\n Implement your custom action logic here.\n \n Example: Hardware control, API calls, CLI commands, etc.\n \"\"\"\n # Your custom implementation\n return {\"status\": \"executed\", \"details\": arguments}\n```\n\n**When to Create Custom Strategies**:\n- ✅ Need domain-specific action execution (e.g., hardware APIs)\n- ✅ Special LLM interaction patterns (e.g., multi-turn dialogs)\n- ✅ Custom data collection (e.g., sensor readings, external APIs)\n- ❌ Standard GUI automation (use default strategies)\n\n---\n\n### Step 4: Create Prompt Templates\n\nCreate YAML prompt templates to guide your agent's LLM interactions:\n\n```yaml\n# File: ufo/prompts/third_party/your_agent.yaml\n\nsystem: |\n You are YourAgent, a specialized AI agent that handles [specific domain tasks].\n \n Your capabilities include:\n - [Capability 1]: Description\n - [Capability 2]: Description\n - [Capability 3]: Description\n \n You have access to the following tools:\n {apis}\n \n Guidelines:\n 1. Analyze the user's request carefully\n 2. Select the most appropriate tool for the task\n 3. Provide clear reasoning for your decisions\n 4. Handle errors gracefully\n \n Available actions:\n - your_action_1: Description and usage\n - your_action_2: Description and usage\n - finish: Complete the task\n\nuser: |\n ## Previous Actions\n {previous_actions}\n \n ## Current Task\n User Request: {request}\n \n ## Available Tools\n {tool_list}\n \n ## Instructions\n Based on the above information:\n 1. Analyze what needs to be done\n 2. Select the appropriate action\n 3. Provide the action parameters\n \n Respond with:\n - Thought: Your reasoning\n - Action: The action to take\n - Arguments: Parameters for the action\n```\n\n```yaml\n# File: ufo/prompts/third_party/your_agent_example.yaml\n\nexample_1: |\n User Request: [Example request]\n \n Thought: [Agent's reasoning]\n Action: your_action_1\n Arguments:\n param1: value1\n param2: value2\n\nexample_2: |\n User Request: [Another example]\n \n Thought: [Agent's reasoning]\n Action: finish\n Arguments:\n summary: Task completed successfully\n```\n\n**Prompt Design Best Practices**:\n- ✅ **Clear role definition**: Explain what your agent does\n- ✅ **Tool descriptions**: List available actions with usage\n- ✅ **Examples**: Provide concrete examples of interactions\n- ✅ **Error handling**: Include guidance for error scenarios\n- ✅ **Output format**: Specify expected response structure\n\n---\n\n## Complete Example: HardwareAgent\n\nLet's examine the complete implementation of **HardwareAgent** as a reference:\n\n### Agent Class\n\n```python\n# File: ufo/agents/agent/customized_agent.py\n\n@AgentRegistry.register(\n agent_name=\"HardwareAgent\", \n third_party=True, \n processor_cls=HardwareAgentProcessor\n)\nclass HardwareAgent(CustomizedAgent):\n \"\"\"\n HardwareAgent is a specialized agent that interacts with hardware components.\n It extends CustomizedAgent to provide additional functionality specific to hardware.\n \n Use cases:\n - Robotic arm control for keyboard/mouse input\n - USB device plug/unplug automation\n - Physical hardware testing\n - Sensor data collection\n \"\"\"\n pass # Inherits all functionality from CustomizedAgent\n```\n\n**Why so simple?**\n- ✅ **Inheritance**: Gets all functionality from `CustomizedAgent`\n- ✅ **Composition**: Custom logic goes in the Processor\n- ✅ **Separation of Concerns**: Agent defines \"what\", Processor defines \"how\"\n\n---\n\n### Processor Class\n\n```python\n# File: ufo/agents/processors/customized/customized_agent_processor.py\n\nclass HardwareAgentProcessor(CustomizedProcessor):\n \"\"\"\n Processor for Hardware Agent.\n \n Handles hardware-specific processing logic including:\n - Visual mode for screenshot understanding\n - Custom action execution for hardware APIs\n - Hardware-specific error handling\n \"\"\"\n pass # Uses default strategy setup from CustomizedProcessor\n```\n\n**Default Strategy Setup**:\n```python\n# From CustomizedProcessor._setup_strategies()\ndef _setup_strategies(self) -> None:\n # Data collection with screenshots\n self.strategies[ProcessingPhase.DATA_COLLECTION] = (\n CustomizedScreenshotCaptureStrategy(fail_fast=True)\n )\n\n # LLM interaction with custom prompts\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n CustomizedLLMInteractionStrategy(fail_fast=True)\n )\n\n # Action execution using standard tools\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n AppActionExecutionStrategy(fail_fast=False)\n )\n\n # Memory updates\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(fail_fast=False)\n )\n```\n\n---\n\n### Configuration\n\n```yaml\n# File: config/ufo/third_party.yaml\n\nENABLED_THIRD_PARTY_AGENTS: [\"HardwareAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n HardwareAgent:\n # Enable visual mode for screenshot understanding\n VISUAL_MODE: True\n \n # Agent identifier (must match @AgentRegistry.register name)\n AGENT_NAME: \"HardwareAgent\"\n \n # Prompt templates\n APPAGENT_PROMPT: \"ufo/prompts/share/base/app_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/examples/visual/app_agent_example.yaml\"\n API_PROMPT: \"ufo/prompts/third_party/hardware_agent_api.yaml\"\n \n # Description for LLM context\n INTRODUCTION: \"The HardwareAgent is used to manipulate hardware components of the computer without using GUI, such as robotic arms for keyboard input and mouse control, plug and unplug devices such as USB drives, and other hardware-related tasks.\"\n```\n\n**Configuration Fields**:\n\n| Field | Required | Description |\n|-------|----------|-------------|\n| `VISUAL_MODE` | Optional | Enable screenshot-based reasoning |\n| `AGENT_NAME` | **Required** | Must match registry name exactly |\n| `APPAGENT_PROMPT` | **Required** | Main prompt template path |\n| `APPAGENT_EXAMPLE_PROMPT` | **Required** | Example prompt template path |\n| `API_PROMPT` | Optional | API/tool description prompt |\n| `INTRODUCTION` | **Required** | Agent description for LLM |\n\n---\n\n## Registering with HostAgent\n\n### How HostAgent Discovers Third-Party Agents\n\nThe registration process is automatic through the Agent Registry system:\n\n```python\n# File: ufo/agents/processors/strategies/host_agent_processing_strategy.py\n\ndef _register_third_party_agents(\n self, target_registry: TargetRegistry, start_index: int\n) -> int:\n \"\"\"\n Register enabled third-party agents with HostAgent.\n \n This method:\n 1. Reads ENABLED_THIRD_PARTY_AGENTS from config\n 2. Creates TargetInfo entries for each agent\n 3. Registers them as selectable targets for the LLM\n \"\"\"\n try:\n # Get enabled third-party agent names from configuration\n third_party_agent_names = ufo_config.system.enabled_third_party_agents\n\n if not third_party_agent_names:\n self.logger.info(\"No third-party agents configured\")\n return 0\n\n # Create third-party agent entries\n third_party_agent_list = []\n for i, agent_name in enumerate(third_party_agent_names):\n agent_id = str(i + start_index + 1) # Unique ID for selection\n third_party_agent_list.append(\n TargetInfo(\n kind=TargetKind.THIRD_PARTY_AGENT.value,\n id=agent_id,\n type=\"ThirdPartyAgent\",\n name=agent_name, # e.g., \"HardwareAgent\"\n )\n )\n\n # Register third-party agents in target registry\n target_registry.register(third_party_agent_list)\n\n return len(third_party_agent_list)\n\n except Exception as e:\n self.logger.warning(f\"Failed to register third-party agents: {str(e)}\")\n return 0\n```\n\n**Target Registry Flow**:\n\n```\n1. HostAgent starts processing\n ↓\n2. _register_applications_and_agents() called\n ↓\n3. _register_third_party_agents() called\n ↓\n4. Read ENABLED_THIRD_PARTY_AGENTS from config\n ↓\n5. Create TargetInfo for each agent\n ↓\n6. Register in TargetRegistry\n ↓\n7. LLM can now select third-party agents as targets\n```\n\n### LLM Target Selection\n\nWhen HostAgent presents targets to the LLM:\n\n```json\n{\n \"available_targets\": [\n {\"id\": \"1\", \"name\": \"Microsoft Word\", \"kind\": \"APPLICATION\"},\n {\"id\": \"2\", \"name\": \"Google Chrome\", \"kind\": \"APPLICATION\"},\n {\"id\": \"3\", \"name\": \"HardwareAgent\", \"kind\": \"THIRD_PARTY_AGENT\"},\n {\"id\": \"4\", \"name\": \"LinuxAgent\", \"kind\": \"THIRD_PARTY_AGENT\"}\n ]\n}\n```\n\nThe LLM selects a target based on the task:\n\n```json\n{\n \"thought\": \"Need to control physical hardware for USB operations\",\n \"selected_target\": \"3\", // HardwareAgent\n \"action\": \"delegate_to_agent\"\n}\n```\n\n### Agent Instantiation\n\nWhen LLM selects your agent, HostAgent creates an instance:\n\n```python\n# File: ufo/agents/agent/host_agent.py\n\n@staticmethod\ndef create_agent(agent_type: str, *args, **kwargs) -> BasicAgent:\n \"\"\"\n Create an agent based on the given type.\n \"\"\"\n if agent_type == \"host\":\n return HostAgent(*args, **kwargs)\n elif agent_type == \"app\":\n return AppAgent(*args, **kwargs)\n elif agent_type in AgentRegistry.list_agents():\n # Third-party agents are retrieved from registry\n return AgentRegistry.get(agent_type)(*args, **kwargs)\n else:\n raise ValueError(\"Invalid agent type: {}\".format(agent_type))\n```\n\n**Instantiation Flow**:\n\n```\n1. LLM selects \"HardwareAgent\"\n ↓\n2. HostAgent calls create_agent(\"HardwareAgent\")\n ↓\n3. AgentRegistry.get(\"HardwareAgent\") retrieves class\n ↓\n4. Class instantiated with config parameters\n ↓\n5. Agent executes task\n ↓\n6. Results returned to HostAgent\n```\n\n---\n\n## Configuration and Deployment\n\n### Step 1: Enable Your Agent\n\nEdit `config/ufo/third_party.yaml`:\n\n```yaml\nENABLED_THIRD_PARTY_AGENTS: [\"YourAgent\"]\n\nTHIRD_PARTY_AGENT_CONFIG:\n YourAgent:\n VISUAL_MODE: False # Set True if using screenshots\n AGENT_NAME: \"YourAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/your_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/your_agent_example.yaml\"\n INTRODUCTION: \"YourAgent handles [specific tasks] by [method]. Use this agent when you need to [use case].\"\n```\n\n**Configuration Checklist**:\n- ✅ Add agent name to `ENABLED_THIRD_PARTY_AGENTS`\n- ✅ Create config block under `THIRD_PARTY_AGENT_CONFIG`\n- ✅ Set `AGENT_NAME` to match registry name\n- ✅ Provide paths to prompt templates\n- ✅ Write clear `INTRODUCTION` for LLM context\n\n---\n\n### Step 2: Add Prompt Templates\n\nCreate your prompt files:\n\n```\nufo/prompts/third_party/\n├── your_agent.yaml # Main prompt template\n└── your_agent_example.yaml # Example interactions\n```\n\n**Template Requirements**:\n- ✅ Define agent role and capabilities\n- ✅ List available actions/tools\n- ✅ Provide clear output format\n- ✅ Include error handling guidance\n- ✅ Add concrete examples\n\n---\n\n### Step 3: Test Configuration\n\nTest that your agent loads correctly:\n\n```python\n# test_your_agent.py\n\nfrom config.config_loader import get_ufo_config\nfrom ufo.agents.agent.basic import AgentRegistry\n\ndef test_agent_registration():\n \"\"\"Test that YourAgent is registered correctly.\"\"\"\n config = get_ufo_config()\n \n # Check if agent is enabled\n assert \"YourAgent\" in config.system.enabled_third_party_agents\n print(\"✅ Agent is enabled in config\")\n \n # Check if agent is registered\n registered_agents = AgentRegistry.list_agents()\n assert \"YourAgent\" in registered_agents\n print(\"✅ Agent is registered in AgentRegistry\")\n \n # Test agent instantiation\n agent_cls = AgentRegistry.get(\"YourAgent\")\n agent_config = config.system.third_party_agent_config[\"YourAgent\"]\n \n agent = agent_cls(\n name=\"test_agent\",\n main_prompt=agent_config[\"APPAGENT_PROMPT\"],\n example_prompt=agent_config[\"APPAGENT_EXAMPLE_PROMPT\"],\n )\n print(f\"✅ Agent instantiated: {agent}\")\n \n # Check processor\n assert hasattr(agent, \"_processor_cls\")\n print(f\"✅ Processor registered: {agent._processor_cls}\")\n\nif __name__ == \"__main__\":\n test_agent_registration()\n```\n\nRun test:\n```powershell\npython test_your_agent.py\n```\n\n---\n\n### Step 4: Integration Testing\n\nTest your agent in a full UFO² session:\n\n```python\n# integration_test.py\n\nfrom ufo.agents.agent.host_agent import HostAgent\nfrom config.config_loader import get_ufo_config\n\ndef test_agent_selection():\n \"\"\"Test that HostAgent can discover and select YourAgent.\"\"\"\n config = get_ufo_config()\n \n # Create HostAgent\n host_agent = HostAgent(\n name=\"host\",\n is_visual=True,\n main_prompt=\"ufo/prompts/share/base/host_agent.yaml\",\n example_prompt=\"ufo/prompts/examples/visual/host_agent_example.yaml\",\n api_prompt=\"ufo/prompts/share/base/api.yaml\",\n )\n \n # Verify third-party agents are in target registry\n # (This happens during HostAgent processing)\n print(\"✅ HostAgent created successfully\")\n print(f\"Enabled third-party agents: {config.system.enabled_third_party_agents}\")\n\nif __name__ == \"__main__\":\n test_agent_selection()\n```\n\n---\n\n## Best Practices\n\n### Code Organization\n\n```\nufo/\n├── agents/\n│ ├── agent/\n│ │ └── customized_agent.py # Agent classes\n│ └── processors/\n│ ├── customized/\n│ │ └── customized_agent_processor.py # Processors\n│ └── strategies/\n│ └── your_agent_strategy.py # Custom strategies\n├── prompts/\n│ └── third_party/\n│ ├── your_agent.yaml # Main prompt\n│ └── your_agent_example.yaml # Examples\nconfig/\n└── ufo/\n └── third_party.yaml # Configuration\n```\n\n**Organization Guidelines**:\n- ✅ **Agent classes** → `ufo/agents/agent/customized_agent.py`\n- ✅ **Processors** → `ufo/agents/processors/customized/`\n- ✅ **Custom strategies** → `ufo/agents/processors/strategies/`\n- ✅ **Prompts** → `ufo/prompts/third_party/`\n- ✅ **Configuration** → `config/ufo/third_party.yaml`\n\n---\n\n### Naming Conventions\n\n| Component | Naming Pattern | Example |\n|-----------|----------------|---------|\n| Agent Class | `{Name}Agent` | `HardwareAgent`, `WebAgent` |\n| Processor Class | `{Name}AgentProcessor` | `HardwareAgentProcessor` |\n| Strategy Class | `{Name}{Phase}Strategy` | `HardwareActionExecutionStrategy` |\n| Registry Name | Same as class (no suffix) | `\"HardwareAgent\"` |\n| Config Key | Same as registry name | `HardwareAgent:` |\n\n---\n\n### Error Handling\n\nImplement robust error handling in your strategies:\n\n```python\nasync def execute(self, agent, context) -> ProcessingResult:\n try:\n # Main execution logic\n result = await self._do_work(agent, context)\n \n return ProcessingResult(\n success=True,\n data=result,\n error=None\n )\n \n except SpecificError as e:\n # Handle expected errors gracefully\n self.logger.warning(f\"Expected error: {e}\")\n return ProcessingResult(\n success=False,\n data={\"partial_result\": \"...\"},\n error=f\"Recoverable error: {str(e)}\"\n )\n \n except Exception as e:\n # Log unexpected errors\n self.logger.error(f\"Unexpected error: {e}\", exc_info=True)\n \n if self.fail_fast:\n raise # Re-raise if configured to fail fast\n \n return ProcessingResult(\n success=False,\n data={},\n error=f\"Fatal error: {str(e)}\"\n )\n```\n\n**Error Handling Guidelines**:\n- ✅ Use `ProcessingResult` to communicate outcomes\n- ✅ Log errors at appropriate levels (warning/error)\n- ✅ Respect `fail_fast` setting\n- ✅ Provide actionable error messages\n- ✅ Return partial results when possible\n\n---\n\n### Logging\n\nUse structured logging throughout your agent:\n\n```python\nimport logging\n\nclass YourAgentProcessor(CustomizedProcessor):\n def __init__(self, agent, global_context):\n super().__init__(agent, global_context)\n self.logger = logging.getLogger(__name__)\n \n async def execute(self, agent, context):\n # Info: Normal operation flow\n self.logger.info(f\"Processing task: {context.get_local('task')}\")\n \n # Debug: Detailed debugging info\n self.logger.debug(f\"Context state: {context.get_all_local()}\")\n \n # Warning: Recoverable issues\n self.logger.warning(f\"Retrying action after failure\")\n \n # Error: Serious problems\n self.logger.error(f\"Action failed: {error}\", exc_info=True)\n```\n\n**Logging Best Practices**:\n- ✅ Use `self.logger` from base class\n- ✅ Log at appropriate levels (debug/info/warning/error)\n- ✅ Include context in log messages\n- ✅ Use `exc_info=True` for exceptions\n- ✅ Avoid logging sensitive data\n\n---\n\n### Testing\n\nCreate comprehensive tests for your agent:\n\n```python\n# tests/test_your_agent.py\n\nimport pytest\nfrom ufo.agents.agent.customized_agent import YourAgent\nfrom ufo.agents.processors.customized.customized_agent_processor import (\n YourAgentProcessor\n)\n\nclass TestYourAgent:\n @pytest.fixture\n def agent(self):\n \"\"\"Create test agent instance.\"\"\"\n return YourAgent(\n name=\"test_agent\",\n main_prompt=\"ufo/prompts/third_party/your_agent.yaml\",\n example_prompt=\"ufo/prompts/third_party/your_agent_example.yaml\",\n )\n \n def test_agent_initialization(self, agent):\n \"\"\"Test agent initializes correctly.\"\"\"\n assert agent.name == \"test_agent\"\n assert agent.prompter is not None\n \n def test_processor_registration(self, agent):\n \"\"\"Test processor is registered.\"\"\"\n assert hasattr(agent, \"_processor_cls\")\n assert agent._processor_cls == YourAgentProcessor\n \n @pytest.mark.asyncio\n async def test_action_execution(self, agent, mock_context):\n \"\"\"Test action execution logic.\"\"\"\n processor = YourAgentProcessor(agent, mock_context)\n result = await processor.execute_phase(\n ProcessingPhase.ACTION_EXECUTION,\n agent,\n mock_context\n )\n assert result.success == True\n```\n\n**Test Coverage Checklist**:\n- ✅ Agent initialization\n- ✅ Processor registration\n- ✅ Strategy execution\n- ✅ Error handling\n- ✅ Configuration loading\n- ✅ Integration with HostAgent\n\n---\n\n## Troubleshooting\n\n### Issue 1: Agent Not Registered\n\n!!!bug \"Error Message\"\n ```\n ValueError: No agent class registered under 'YourAgent'\n ```\n \n **Diagnosis**: Agent is not enabled in configuration or decorator is missing.\n \n **Solutions**:\n \n 1. Check configuration:\n ```yaml\n # config/ufo/third_party.yaml\n ENABLED_THIRD_PARTY_AGENTS: [\"YourAgent\"] # ← Must include your agent\n ```\n \n 2. Verify decorator:\n ```python\n @AgentRegistry.register(\n agent_name=\"YourAgent\", # ← Must match config\n third_party=True, # ← Must be True\n processor_cls=YourAgentProcessor\n )\n class YourAgent(CustomizedAgent):\n pass\n ```\n \n 3. Check import:\n ```python\n # Ensure your agent module is imported\n # In ufo/agents/agent/__init__.py or customized_agent.py\n from ufo.agents.agent.customized_agent import YourAgent\n ```\n\n---\n\n### Issue 2: Prompt Files Not Found\n\n!!!bug \"Error Message\"\n ```\n FileNotFoundError: ufo/prompts/third_party/your_agent.yaml\n ```\n \n **Diagnosis**: Prompt template files don't exist or paths are incorrect.\n \n **Solutions**:\n \n 1. Create prompt files:\n ```powershell\n # Create directory if needed\n New-Item -ItemType Directory -Force -Path \"ufo\\prompts\\third_party\"\n \n # Create prompt files\n New-Item -ItemType File -Path \"ufo\\prompts\\third_party\\your_agent.yaml\"\n New-Item -ItemType File -Path \"ufo\\prompts\\third_party\\your_agent_example.yaml\"\n ```\n \n 2. Verify paths in configuration:\n ```yaml\n THIRD_PARTY_AGENT_CONFIG:\n YourAgent:\n APPAGENT_PROMPT: \"ufo/prompts/third_party/your_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/your_agent_example.yaml\"\n ```\n \n 3. Check file permissions:\n ```powershell\n # Verify files are readable\n Test-Path \"ufo\\prompts\\third_party\\your_agent.yaml\"\n ```\n\n---\n\n### Issue 3: Agent Not Appearing in Target List\n\n!!!bug \"Symptom\"\n HostAgent doesn't show your third-party agent as a selectable target.\n \n **Diagnosis**: Agent is registered but not appearing in TargetRegistry.\n \n **Solutions**:\n \n 1. Check enabled agents:\n ```python\n from config.config_loader import get_ufo_config\n config = get_ufo_config()\n print(config.system.enabled_third_party_agents)\n # Should include \"YourAgent\"\n ```\n \n 2. Verify TargetKind:\n ```python\n # In your registration code\n TargetInfo(\n kind=TargetKind.THIRD_PARTY_AGENT.value, # ← Correct kind\n name=agent_name,\n )\n ```\n \n 3. Check HostAgent logs:\n ```\n [INFO] Registered 2 third-party agents\n ```\n \n 4. Test target registry directly:\n ```python\n from ufo.agents.processors.schemas.target import TargetRegistry, TargetKind\n registry = TargetRegistry()\n targets = registry.get_by_kind(TargetKind.THIRD_PARTY_AGENT)\n print(targets) # Should include your agent\n ```\n\n---\n\n### Issue 4: Processor Not Executing\n\n!!!bug \"Symptom\"\n Agent instantiates but processor strategies don't execute.\n \n **Diagnosis**: Processor class not properly linked or strategies not set up.\n \n **Solutions**:\n \n 1. Verify processor_cls in decorator:\n ```python\n @AgentRegistry.register(\n agent_name=\"YourAgent\",\n third_party=True,\n processor_cls=YourAgentProcessor # ← Must be specified\n )\n ```\n \n 2. Check processor initialization:\n ```python\n class YourAgentProcessor(CustomizedProcessor):\n def __init__(self, agent, global_context):\n super().__init__(agent, global_context) # ← Must call super\n # Your custom init\n ```\n \n 3. Verify strategy setup:\n ```python\n def _setup_strategies(self) -> None:\n # Must populate self.strategies dict\n self.strategies[ProcessingPhase.LLM_INTERACTION] = ...\n ```\n \n 4. Check processor is created:\n ```python\n # In your test\n assert hasattr(agent, \"_processor_cls\")\n processor = agent._processor_cls(agent, global_context)\n assert processor is not None\n ```\n\n---\n\n### Issue 5: LLM Not Selecting Your Agent\n\n!!!bug \"Symptom\"\n Agent is registered but LLM never selects it.\n \n **Diagnosis**: Agent description unclear or not suitable for user requests.\n \n **Solutions**:\n \n 1. Improve `INTRODUCTION`:\n ```yaml\n INTRODUCTION: \"Use YourAgent when you need to [clear use case]. It provides [specific capabilities] through [method]. Examples: [concrete examples].\"\n ```\n \n 2. Add clear examples in prompt:\n ```yaml\n # your_agent_example.yaml\n example_1: |\n User: [Clear example request]\n Agent: [Clear example response]\n ```\n \n 3. Test with explicit requests:\n ```python\n # Test with request that clearly needs your agent\n user_request = \"Use YourAgent to [specific task]\"\n ```\n \n 4. Check HostAgent prompt includes your agent:\n ```\n Available targets:\n - YourAgent: [Your INTRODUCTION text should appear here]\n ```\n\n---\n\n## Advanced Topics\n\n### Multi-MCP Integration\n\nIntegrate multiple MCP servers with your agent:\n\n```yaml\n# config/ufo/agent_mcp.yaml\n\nYourAgent:\n mcp_servers:\n hardware_control:\n type: \"local\"\n module: \"your_package.hardware_mcp\"\n config:\n device_port: \"/dev/ttyUSB0\"\n \n data_collection:\n type: \"http\"\n url: \"http://localhost:8080/mcp\"\n config:\n api_key: \"${SENSOR_API_KEY}\"\n```\n\nSee [Creating Custom MCP Servers](./creating_mcp_servers.md) for details.\n\n---\n\n### State Management\n\nMaintain agent state across invocations:\n\n```python\nfrom ufo.agents.memory.blackboard import Blackboard\n\nclass YourAgent(CustomizedAgent):\n def __init__(self, name, main_prompt, example_prompt):\n super().__init__(name, main_prompt, example_prompt)\n \n # Use blackboard for persistent state\n self._blackboard = Blackboard()\n \n @property\n def blackboard(self) -> Blackboard:\n return self._blackboard\n \n def save_state(self, key: str, value: Any):\n \"\"\"Save state to blackboard.\"\"\"\n self.blackboard.add_entry(key, value)\n \n def load_state(self, key: str) -> Any:\n \"\"\"Load state from blackboard.\"\"\"\n return self.blackboard.get_entry(key)\n```\n\n---\n\n### Custom Prompter\n\nCreate a custom prompter for specialized LLM interactions:\n\n```python\nfrom ufo.prompter.app_prompter import AppPrompter\n\nclass YourAgentPrompter(AppPrompter):\n \"\"\"Custom prompter for YourAgent.\"\"\"\n \n def user_content_construction(\n self,\n prev_plan: List[str],\n user_request: str,\n retrieved_docs: str,\n last_success_actions: List[Dict],\n **kwargs\n ) -> List[Dict[str, str]]:\n \"\"\"\n Construct custom user message content.\n \"\"\"\n # Add custom context\n custom_context = self._build_custom_context(**kwargs)\n \n # Call parent method\n base_content = super().user_content_construction(\n prev_plan=prev_plan,\n user_request=user_request,\n retrieved_docs=retrieved_docs,\n last_success_actions=last_success_actions\n )\n \n # Insert custom content\n base_content.insert(0, {\n \"type\": \"text\",\n \"text\": custom_context\n })\n \n return base_content\n```\n\nUse custom prompter in your agent:\n\n```python\nclass YourAgent(CustomizedAgent):\n def get_prompter(self, is_visual, main_prompt, example_prompt):\n return YourAgentPrompter(main_prompt, example_prompt)\n```\n\n---\n\n## Related Documentation\n\n- **[Third-Party Agent Configuration](../configuration/system/third_party_config.md)** - Configuration reference\n- **[Agent Configuration](../configuration/system/agents_config.md)** - Core agent LLM settings \n- **[Creating Custom MCP Servers](./creating_mcp_servers.md)** - MCP server development for custom tools\n- **[Agent Architecture](../infrastructure/agents/overview.md)** - Understanding agent design patterns\n- **[HostAgent Strategy](../ufo2/host_agent/strategy.md)** - Learn how HostAgent orchestrates third-party agents\n- **[AppAgent Strategy](../ufo2/app_agent/strategy.md)** - Processing strategies reference\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n✅ **Third-party agents extend UFO²** with specialized capabilities \n✅ **Use `@AgentRegistry.register()`** to register your agent \n✅ **Create processor classes** to define processing logic \n✅ **Configure in third_party.yaml** to enable your agent \n✅ **HostAgent automatically discovers** enabled third-party agents \n✅ **LLM selects agents** based on task requirements \n✅ **Follow HardwareAgent** as a reference implementation \n\n**Build powerful third-party agents to extend UFO²!** 🚀\n\n---\n\n## Quick Reference\n\n### Minimal Agent Implementation\n\n```python\n# 1. Agent class\n@AgentRegistry.register(\n agent_name=\"MyAgent\", third_party=True, processor_cls=MyProcessor\n)\nclass MyAgent(CustomizedAgent):\n pass\n\n# 2. Processor class\nclass MyProcessor(CustomizedProcessor):\n pass # Use default strategies\n\n# 3. Configuration\n# config/ufo/third_party.yaml\nENABLED_THIRD_PARTY_AGENTS: [\"MyAgent\"]\nTHIRD_PARTY_AGENT_CONFIG:\n MyAgent:\n AGENT_NAME: \"MyAgent\"\n APPAGENT_PROMPT: \"ufo/prompts/third_party/my_agent.yaml\"\n APPAGENT_EXAMPLE_PROMPT: \"ufo/prompts/third_party/my_agent_example.yaml\"\n INTRODUCTION: \"MyAgent handles [tasks].\"\n\n# 4. Prompt templates\n# Create: ufo/prompts/third_party/my_agent.yaml\n# Create: ufo/prompts/third_party/my_agent_example.yaml\n```\n\n**That's all you need to get started!** 🎉\n"
},
{
"path": "documents/docs/ufo2/advanced_usage/batch_mode.md",
"content": "# Batch Mode\n\nBatch mode allows automated execution of tasks on specific applications or files using predefined plan files. This mode is particularly useful for repetitive tasks on Microsoft Office applications (Word, Excel, PowerPoint).\n\n## Quick Start\n\n### Step 1: Create a Plan File\n\nCreate a JSON plan file that defines the task to be automated. The plan file should contain the following fields:\n\n| Field | Description | Type |\n| ------ | -------------------------------------------------------------------------------------------- | ------- |\n| task | The task description. | String |\n| object | The application or file to interact with. | String |\n| close | Determines whether to close the corresponding application or file after completing the task. | Boolean |\n\nExample plan file:\n\n```json\n{\n \"task\": \"Type in a text of 'Test For Fun' with heading 1 level\",\n \"object\": \"draft.docx\",\n \"close\": false\n}\n```\n\n**Important:** The `close` field should be a boolean value (`true` or `false`), not a Python boolean (`True` or `False`).\n\nThe file structure should be organized as follows:\n\n```\nParent/\n├── tasks/\n│ └── plan.json\n└── files/\n └── draft.docx\n```\n\nThe `object` field in the plan file refers to files in the `files` directory. The plan reader will automatically resolve the full file path by replacing `tasks` with `files` in the directory structure.\n\n### Step 2: Start Batch Mode\n\nRun the following command to start batch mode:\n\n```bash\n# Assume you are in the cloned UFO folder\npython -m ufo --task {task_name} --mode batch_normal --plan {plan_file}\n```\n\n**Parameters:**\n- `{task_name}`: Name for this task execution (used for logging)\n- `{plan_file}`: Full path to the plan JSON file (e.g., `C:/Parent/tasks/plan.json`)\n\n### Supported Applications\n\nBatch mode currently supports the following Microsoft Office applications:\n\n- **Word** (`.docx` files) - `WINWORD.EXE`\n- **Excel** (`.xlsx` files) - `EXCEL.EXE`\n- **PowerPoint** (`.pptx` files) - `POWERPNT.EXE`\n\nThe application will be automatically launched when the batch mode starts, and the specified file will be opened and maximized.\n\n## Evaluation\n\nUFO can automatically evaluate whether the task was completed successfully. To enable evaluation, ensure `EVA_SESSION` is set to `True` in the `config/ufo/system.yaml` file.\n\nCheck the evaluation results in `logs/{task_name}/evaluation.log`.\n\n## References\n\nThe batch mode uses a `PlanReader` to parse the plan file and creates a `FromFileSession` to execute the plan.\n\n### PlanReader\n\nThe `PlanReader` is located at `ufo/module/sessions/plan_reader.py`.\n\n:::module.sessions.plan_reader.PlanReader\n\n### FromFileSession\n\nThe `FromFileSession` is located at `ufo/module/sessions/session.py`.\n\n:::module.sessions.session.FromFileSession"
},
{
"path": "documents/docs/ufo2/advanced_usage/customization.md",
"content": "# Customization\n\nUFO can ask users for additional context or information when needed and save it in local memory for future reference. This customization feature enables a more personalized user experience by remembering user-specific information across sessions.\n\n## Example Scenario\n\nConsider a task where UFO needs to book a cab. To complete this task, UFO requires the user's address. UFO will:\n\n1. Ask the user for their address\n2. Save the address in local memory\n3. Use the saved address automatically in future tasks that require it\n\nThis eliminates the need to repeatedly provide the same information.\n\n## How It Works\n\nThe customization feature is implemented across multiple agent types (`HostAgent`, `AppAgent`, and `OpenAIOperatorAgent`). When an agent needs additional information:\n\n1. The agent transitions to the `PENDING` state\n2. The agent asks the user for the required information (if `ASK_QUESTION` is enabled)\n3. The user's response is saved to the `blackboard` in the QA pairs file\n4. All agents in the session can access this information from the shared `blackboard`\n\nThe saved QA pairs are stored locally as JSON lines in the file specified by `QA_PAIR_FILE`. Privacy is preserved as this information never leaves the local machine.\n\n## Configuration\n\nConfigure the customization feature in `config/ufo/system.yaml`:\n\n| Configuration Option | Description | Type | Default Value |\n|------------------------|------------------------------------------------------------------|---------|---------------------------------------|\n| `ASK_QUESTION` | Whether to allow agents to ask users questions | Boolean | False |\n| `USE_CUSTOMIZATION` | Whether to load and use saved QA pairs from previous sessions | Boolean | False |\n| `QA_PAIR_FILE` | Path to the file storing historical QA pairs | String | \"customization/global_memory.jsonl\" |\n| `QA_PAIR_NUM` | Maximum number of recent QA pairs to load into memory | Integer | 20 |\n\n**Note:** Both `ASK_QUESTION` and `USE_CUSTOMIZATION` need to be enabled for the full customization experience. `ASK_QUESTION` controls whether agents can prompt users for information, while `USE_CUSTOMIZATION` controls whether previously saved information is loaded.\n"
},
{
"path": "documents/docs/ufo2/advanced_usage/follower_mode.md",
"content": "# Follower Mode\n\nFollower mode enables UFO to execute a predefined list of steps in natural language. Unlike normal mode where the agent generates its own plan, follower mode creates an `AppAgent` that follows user-provided steps to interact with applications. This mode is particularly useful for debugging, software testing, and verification.\n\n## Quick Start\n\n### Step 1: Create a Plan File\n\nCreate a JSON plan file containing the steps for the agent to follow:\n\n| Field | Description | Type |\n| --- | --- | --- |\n| task | The task description. | String |\n| steps | The list of steps for the agent to follow. | List of Strings |\n| object | The application or file to interact with. | String |\n\nExample plan file:\n\n```json\n{\n \"task\": \"Type in a text of 'Test For Fun' with heading 1 level\",\n \"steps\": \n [\n \"1.type in 'Test For Fun'\", \n \"2.Select the 'Test For Fun' text\",\n \"3.Click 'Home' tab to show the 'Styles' ribbon tab\",\n \"4.Click 'Styles' ribbon tab to show the style 'Heading 1'\",\n \"5.Click 'Heading 1' style to apply the style to the selected text\"\n ],\n \"object\": \"draft.docx\"\n}\n```\n\nThe `object` field specifies the application or file the agent will interact with. This object should be opened and accessible before starting follower mode.\n\n### Step 2: Start Follower Mode\n\nRun the following command:\n\n```bash\n# Assume you are in the cloned UFO folder\npython -m ufo --task {task_name} --mode follower --plan {plan_file}\n```\n\n**Parameters:**\n- `{task_name}`: Name for this task execution (used for logging)\n- `{plan_file}`: Path to the plan JSON file\n\n### Step 3: Run in Batch (Optional)\n\nTo execute multiple plan files sequentially, provide a folder containing multiple plan files:\n\n```bash\n# Assume you are in the cloned UFO folder\npython -m ufo --task {task_name} --mode follower --plan {plan_folder}\n``` \n\nUFO will automatically detect and execute all plan files in the folder sequentially.\n\n**Parameters:**\n- `{task_name}`: Name for this batch execution (used for logging)\n- `{plan_folder}`: Path to the folder containing plan JSON files\n\n## Evaluation\n\nUFO can automatically evaluate task completion. To enable evaluation, ensure `EVA_SESSION` is set to `True` in `config/ufo/system.yaml`.\n\nCheck the evaluation results in `logs/{task_name}/evaluation.log`.\n\n## References\n\nFollower mode uses a `PlanReader` to parse the plan file and creates a `FollowerSession` to execute the steps.\n\n### PlanReader\n\nThe `PlanReader` is located at `ufo/module/sessions/plan_reader.py`.\n\n:::module.sessions.plan_reader.PlanReader\n\n### FollowerSession\n\nThe `FollowerSession` is located at `ufo/module/sessions/session.py`.\n\n:::module.sessions.session.FollowerSession"
},
{
"path": "documents/docs/ufo2/advanced_usage/operator_as_app_agent.md",
"content": "# Operator as an AppAgent\n\nUFO² supports wrapping third-party agents as AppAgents, enabling them to be orchestrated by the HostAgent in multi-agent workflows. This guide demonstrates how to run **Operator**, an OpenAI-based Conversational UI Agent (CUA), within the UFO² ecosystem.\n\n\n\n## Prerequisites\n\nBefore proceeding, ensure that Operator has been properly configured. Follow the setup instructions in the [OpenAI CUA (Operator) guide](../../configuration/models/operator.md).\n\n## Running the Operator\n\nUFO² provides two modes for running Operator:\n\n1. **Single Agent Mode (`operator`)** — Run Operator independently through UFO² as a launcher\n2. **AppAgent Mode (`normal_operator`)** — Run Operator as an `AppAgent` orchestrated by the `HostAgent`\n\n### Single Agent Mode\n\nIn single agent mode, Operator functions independently but is launched through UFO². This mode is useful for debugging or quick prototyping.\n\n```powershell\npython -m ufo --mode operator --task --request \n```\n\n**Example:**\n```powershell\npython -m ufo --mode operator --task test_operator --request \"Open Notepad and type Hello World\"\n```\n\n### AppAgent Mode\n\nIn AppAgent mode, Operator is wrapped as an `AppAgent` and can be triggered as a sub-agent within the HostAgent workflow. This enables task decomposition where the HostAgent coordinates multiple agents including Operator.\n\n```powershell\npython -m ufo --mode normal_operator --task --request \n```\n\n**Example:**\n```powershell\npython -m ufo --mode normal_operator --task test_integration --request \"Search for Python documentation and open the first result\"\n```\n\n## Logs\n\nIn both modes, execution logs are saved in:\n\n```\nlogs//\n```\n\nThese logs follow the same structure and conventions as other UFO² sessions."
},
{
"path": "documents/docs/ufo2/app_agent/commands.md",
"content": "# AppAgent Command System\n\nAppAgent executes application-level commands through the **MCP (Model-Context Protocol)** system. Commands are dynamically provided by MCP servers and executed through the `CommandDispatcher` interface. This document describes the MCP configuration for AppAgent commands.\n\n---\n\n## Command Execution Architecture\n\n```mermaid\ngraph LR\n Agent[AppAgent] --> Dispatcher[CommandDispatcher]\n Dispatcher --> MCPClient[MCP Client]\n MCPClient --> UICollector[UICollector Server]\n MCPClient --> AppUIExecutor[AppUIExecutor Server]\n MCPClient --> COMExecutor[COM Executor Servers]\n MCPClient --> CLIExecutor[CommandLine Executor]\n \n UICollector --> DataCollection[Data Collection Commands]\n AppUIExecutor --> UIActions[UI Automation Commands]\n COMExecutor --> APIActions[Application API Commands]\n CLIExecutor --> ShellActions[Shell Commands]\n \n style Agent fill:#e3f2fd\n style Dispatcher fill:#fff3e0\n style MCPClient fill:#f1f8e9\n style UICollector fill:#c8e6c9\n style AppUIExecutor fill:#fff9c4\n style COMExecutor fill:#ffccbc\n style CLIExecutor fill:#d1c4e9\n```\n\n!!!note \"Dynamic Commands\"\n AppAgent commands are **not hardcoded**. They are dynamically discovered from configured MCP servers. The available commands depend on:\n \n - **MCP server configuration** in `config/ufo/mcp.yaml`\n - **Application context** (e.g., Word, Excel, PowerPoint)\n - **Installed MCP servers** (local, HTTP, or stdio)\n\n---\n\n## MCP Server Configuration\n\n### Configuration File\n\nAppAgent commands are configured in **`config/ufo/mcp.yaml`**:\n\n```yaml\n# Default configuration for all applications\nAppAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: AppUIExecutor\n type: local\n start_args: []\n reset: false\n - namespace: CommandLineExecutor\n type: local\n start_args: []\n reset: false\n \n # Application-specific configurations\n WINWORD.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: WordCOMExecutor\n type: local\n reset: true # Reset on document switch\n \n EXCEL.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: ExcelCOMExecutor\n type: local\n reset: true\n \n POWERPNT.EXE:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: PowerPointCOMExecutor\n type: local\n reset: true\n \n explorer.exe:\n action:\n - namespace: AppUIExecutor\n type: local\n - namespace: PDFReaderExecutor\n type: local\n reset: true\n```\n\n### MCP Servers Used by AppAgent\n\n| Server | Namespace | Type | Purpose | Command Categories |\n|--------|-----------|------|---------|-------------------|\n| **UICollector** | `UICollector` | Local | Data collection | Screenshot capture, control detection, UI tree |\n| **AppUIExecutor** | `AppUIExecutor` | Local | UI automation | Mouse clicks, keyboard input, text entry |\n| **CommandLineExecutor** | `CommandLineExecutor` | Local | Shell execution | PowerShell, Bash commands |\n| **WordCOMExecutor** | `WordCOMExecutor` | Local | Word automation | Document creation, text manipulation, formatting |\n| **ExcelCOMExecutor** | `ExcelCOMExecutor` | Local | Excel automation | Workbook creation, data entry, charts |\n| **PowerPointCOMExecutor** | `PowerPointCOMExecutor` | Local | PowerPoint automation | Presentation creation, slides, shapes |\n| **PDFReaderExecutor** | `PDFReaderExecutor` | Local | PDF operations | Text extraction, page navigation |\n\nWhen AppAgent works with specific applications (Word, Excel, PowerPoint), additional **COM executor servers** are automatically loaded to provide native API access alongside UI automation commands. These servers have `reset: true` to prevent state leakage between documents.\n\n---\n\n## Command Discovery\n\n### Listing Available Commands\n\nAppAgent dynamically discovers available commands from MCP servers:\n\n```python\n# Get all available tools from MCP servers\nresult = await command_dispatcher.execute_commands([\n Command(tool_name=\"list_tools\", parameters={})\n])\n\ntools = result[0].result\n# Returns list of all available commands with their schemas\n```\n\n### Command Categories\n\nCommands are categorized by purpose:\n\n| Category | Server | Examples |\n|----------|--------|----------|\n| **Data Collection** | UICollector | `capture_window_screenshot`, `get_app_window_controls_target_info`, `get_ui_tree` |\n| **Mouse Actions** | AppUIExecutor | `click_input`, `click_on_coordinates`, `drag_on_coordinates`, `wheel_mouse_input` |\n| **Keyboard Actions** | AppUIExecutor | `set_edit_text`, `keyboard_input` |\n| **Data Retrieval** | AppUIExecutor | `texts`, `get_text` |\n| **Document API** | WordCOMExecutor | `create_document`, `insert_text`, `save_document` |\n| **Spreadsheet API** | ExcelCOMExecutor | `create_workbook`, `insert_data`, `create_chart` |\n| **Presentation API** | PowerPointCOMExecutor | `create_presentation`, `add_slide`, `insert_shape` |\n| **Shell Execution** | CommandLineExecutor | `execute_command` |\n\n---\n\n## Command Execution\n\n### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Executor as ActionExecutor\n participant Dispatcher as CommandDispatcher\n participant MCP as MCP Server\n \n Strategy->>Executor: execute(action_info)\n Executor->>Dispatcher: execute_commands([Command(...)])\n Dispatcher->>MCP: Invoke tool\n MCP->>MCP: Execute command logic\n MCP-->>Dispatcher: Result\n Dispatcher-->>Executor: Result\n Executor-->>Strategy: Success/Error\n```\n\n### Example: Execute UI Command\n\n```python\nfrom aip.messages import Command\n\n# Create command\ncommand = Command(\n tool_name=\"click_input\",\n parameters={\n \"id\": \"12\",\n \"name\": \"Export\",\n \"button\": \"left\",\n \"double\": False\n },\n tool_type=\"action\",\n)\n\n# Execute command\nresults = await command_dispatcher.execute_commands([command])\n\n# Check result\nif results[0].status == \"SUCCESS\":\n print(f\"Command executed: {results[0].result}\")\n```\n\n---\n\n## Configuration Resources\n\nFor detailed MCP configuration, server setup, and command reference:\n\n**Quick References:**\n\n- **[MCP Configuration Reference](../../configuration/system/mcp_reference.md)** - Quick MCP settings reference\n- **[MCP Overview](../../mcp/overview.md)** - MCP architecture and concepts\n\n**Configuration Guides:**\n\n- **[MCP Configuration Guide](../../mcp/configuration.md)** - Complete configuration documentation\n- **[Local Servers](../../mcp/local_servers.md)** - Built-in MCP servers\n- **[Remote Servers](../../mcp/remote_servers.md)** - HTTP and stdio servers\n- **[Creating MCP Servers](../../tutorials/creating_mcp_servers.md)** - Creating custom MCP servers\n\n**Server Type Documentation:**\n\n- **[Action Servers](../../mcp/action.md)** - Action server documentation\n- **[Data Collection Servers](../../mcp/data_collection.md)** - Data collection server documentation\n\n### Detailed Server Documentation\n\nEach MCP server has comprehensive documentation:\n\n| Server | Documentation | Command Details |\n|--------|--------------|----------------|\n| UICollector | [UICollector Server](../../mcp/servers/ui_collector.md) | Screenshot, control detection, UI tree commands |\n| AppUIExecutor | [AppUIExecutor Server](../../mcp/servers/app_ui_executor.md) | UI automation commands with parameters |\n| WordCOMExecutor | [Word COM Executor](../../mcp/servers/word_com_executor.md) | Microsoft Word API commands |\n| ExcelCOMExecutor | [Excel COM Executor](../../mcp/servers/excel_com_executor.md) | Microsoft Excel API commands |\n| PowerPointCOMExecutor | [PowerPoint COM Executor](../../mcp/servers/ppt_com_executor.md) | Microsoft PowerPoint API commands |\n| PDFReaderExecutor | [PDF Reader Executor](../../mcp/servers/pdf_reader_executor.md) | PDF reading commands |\n| CommandLineExecutor | [CommandLine Executor](../../mcp/servers/command_line_executor.md) | Shell command execution |\n\n!!!warning \"Command Details Subject to Change\"\n Specific command parameters, names, and behaviors may change as MCP servers evolve. Always refer to the **server-specific documentation** for the most up-to-date command reference.\n\n---\n\n## Agent Configuration Settings\n\n### AppAgent Configuration\n\n```yaml\n# config/ufo/app_agent_config.yaml\nsystem:\n # Control detection backend\n control_backend:\n - \"uia\" # Windows UI Automation\n - \"omniparser\" # Vision-based detection\n \n # Screenshot settings\n save_full_screen: true # Also capture desktop\n save_ui_tree: true # Save UI tree JSON\n include_last_screenshot: true # Include previous step\n concat_screenshot: true # Concatenate clean + annotated\n \n # Window behavior\n maximize_window: false # Maximize on selection\n show_visual_outline_on_screen: true # Draw red outline\n```\n\nSee **[Configuration Overview](../../configuration/system/overview.md)** and **[System Configuration](../../configuration/system/system_config.md)** for complete configuration options.\n\n---\n\n## Related Documentation\n\n**Architecture & Design:**\n\n- **[AppAgent Overview](overview.md)** - High-level AppAgent architecture\n- **[State Machine](state.md)** - State machine documentation\n- **[Processing Strategy](strategy.md)** - 4-phase processing pipeline\n- **[HostAgent Commands](../host_agent/commands.md)** - Desktop-level commands\n\n**Core Features:**\n\n- **[Hybrid Actions](../core_features/hybrid_actions.md)** - MCP command system architecture\n- **[Control Detection](../core_features/control_detection/overview.md)** - UIA and OmniParser backends\n- **[Command Dispatcher](../../infrastructure/modules/dispatcher.md)** - Command routing\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n✅ **MCP-Based**: All commands provided by MCP servers configured in `mcp.yaml` \n✅ **Dynamic Discovery**: Commands discovered at runtime via `list_tools` \n✅ **Application-Specific**: COM executors auto-loaded for Word, Excel, PowerPoint \n✅ **Hybrid Approach**: UI automation + native API commands \n✅ **Configurable**: Extensive MCP server configuration options \n✅ **Documented**: Each server has detailed command reference\n\n!!!warning \"Command Details Subject to Change\"\n Specific command parameters, names, and behaviors may change as MCP servers evolve. Always refer to the **server-specific documentation** for the most up-to-date command reference.\n\n**Next Steps:**\n\n1. **Review MCP Configuration**: [MCP Configuration Reference](../../configuration/system/mcp_reference.md)\n2. **Explore Server Documentation**: Click server links above for command details\n3. **Understand Processing**: [Processing Strategy](strategy.md) shows commands in action\n4. **Learn State Machine**: [State Machine](state.md) explains when commands execute\n"
},
{
"path": "documents/docs/ufo2/app_agent/overview.md",
"content": "# AppAgent: Application Execution Agent\n\n**AppAgent** is the core execution runtime in UFO, responsible for carrying out individual subtasks within a specific Windows application. Each AppAgent functions as an isolated, application-specialized worker process launched and orchestrated by the central HostAgent.\n\n---\n\n## What is AppAgent?\n\n\n \n AppAgent Architecture: Application-specialized worker process for subtask execution\n\n\n**AppAgent** operates as a **child agent** under the HostAgent's orchestration:\n\n- **Isolated Runtime**: Each AppAgent is dedicated to a single Windows application\n- **Subtask Executor**: Executes specific subtasks delegated by HostAgent\n- **Application Expert**: Tailored with deep knowledge of the target app's API surface, control semantics, and domain logic\n- **Hybrid Execution**: Leverages both GUI automation and API-based actions through MCP commands\n\nUnlike monolithic Computer-Using Agents (CUAs) that treat all GUI contexts uniformly, each AppAgent is tailored to a single application and operates with specialized knowledge of its interface and capabilities.\n\n---\n\n## Core Responsibilities\n\n```mermaid\ngraph TB\n subgraph \"AppAgent Core Responsibilities\"\n SR[Sense: Capture Application State]\n RE[Reason: Analyze Next Action]\n EX[Execute: GUI or API Action]\n RP[Report: Write Results to Blackboard]\n end\n \n SR --> RE\n RE --> EX\n EX --> RP\n RP --> SR\n \n style SR fill:#e3f2fd\n style RE fill:#fff3e0\n style EX fill:#f1f8e9\n style RP fill:#fce4ec\n```\n\n| Responsibility | Description | Example |\n|---------------|-------------|---------|\n| **State Sensing** | Capture application UI, detect controls, understand current state | Screenshot Word window → Detect 50 controls → Annotate UI elements |\n| **Reasoning** | Analyze state and determine next action using LLM | \"Table visible with Export button [12] → Click to export data\" |\n| **Action Execution** | Execute GUI clicks or API calls via MCP commands | `click_input(control_id=12)` or `execute_word_command(\"export_table\")` |\n| **Result Reporting** | Write execution results to shared Blackboard | Write extracted data to `subtask_result_1` for HostAgent |\n\n---\n\n## ReAct-Style Control Loop\n\nUpon receiving a subtask and execution context from the HostAgent, the AppAgent initializes a **ReAct-style control loop** where it iteratively:\n\n1. **Observes** the current application state (screenshot + control detection)\n2. **Thinks** about the next step (LLM reasoning)\n3. **Acts** by executing either a GUI or API-based action (MCP commands)\n\n```mermaid\nsequenceDiagram\n participant HostAgent\n participant AppAgent\n participant Application\n participant Blackboard\n \n HostAgent->>AppAgent: Delegate subtask \"Extract table from Word\"\n \n loop ReAct Loop\n AppAgent->>Application: Observe (screenshot + controls)\n Application-->>AppAgent: UI state\n AppAgent->>AppAgent: Think (LLM reasoning)\n AppAgent->>Application: Act (click/API call)\n Application-->>AppAgent: Action result\n end\n \n AppAgent->>Blackboard: Write result\n AppAgent->>HostAgent: Return control\n```\n\nThe MCP command system enables **reliable control** over dynamic and complex UIs by favoring structured API commands whenever available, while retaining fallback to GUI-based interaction commands when necessary.\n\n---\n\n## Execution Architecture\n\n### Finite State Machine\n\nAppAgent uses a finite state machine with 7 states to control its execution flow:\n\n- **CONTINUE**: Continue processing the current subtask\n- **FINISH**: Successfully complete the subtask\n- **ERROR**: Encounter an unrecoverable error\n- **FAIL**: Fail to complete the subtask\n- **PENDING**: Wait for user input or clarification\n- **CONFIRM**: Request user confirmation for sensitive actions\n- **SCREENSHOT**: Capture and re-annotate the application screenshot\n\n**State Details**: See [State Machine Documentation](state.md) for complete state definitions and transitions.\n\n### 4-Phase Processing Pipeline\n\nEach execution round follows a 4-phase pipeline:\n\n```mermaid\ngraph LR\n DC[Phase 1: DATA_COLLECTION Screenshot + Controls] --> LLM[Phase 2: LLM_INTERACTION Reasoning]\n LLM --> AE[Phase 3: ACTION_EXECUTION GUI/API Action]\n AE --> MU[Phase 4: MEMORY_UPDATE Record Action]\n \n style DC fill:#e1f5ff\n style LLM fill:#fff4e6\n style AE fill:#e8f5e9\n style MU fill:#fce4ec\n```\n\n**Strategy Details**: See [Processing Strategy Documentation](strategy.md) for complete pipeline implementation.\n\n---\n\n## Hybrid GUI–API Execution\n\nAppAgent executes actions through the **MCP (Model-Context Protocol) command system**, which provides a unified interface for both GUI automation and native API calls:\n\n```python\n# GUI-based command (fallback)\ncommand = Command(\n tool_name=\"click_input\",\n parameters={\"control_id\": \"12\", \"button\": \"left\"}\n)\nawait command_dispatcher.execute_commands([command])\n\n# API-based command (preferred when available)\ncommand = Command(\n tool_name=\"word_export_table\",\n parameters={\"format\": \"csv\", \"path\": \"output.csv\"}\n)\nawait command_dispatcher.execute_commands([command])\n```\n\n**Implementation**: See [Hybrid Actions](../core_features/hybrid_actions.md) for details on the MCP command system.\n\n---\n\n## Knowledge Enhancement\n\nAppAgent is enhanced with **Retrieval Augmented Generation (RAG)** from heterogeneous sources:\n\n| Knowledge Source | Purpose | Configuration |\n|-----------------|---------|---------------|\n| **Help Documents** | Application-specific documentation | [Learning from Help Documents](../core_features/knowledge_substrate/learning_from_help_document.md) |\n| **Bing Search** | Latest information and updates | [Learning from Bing Search](../core_features/knowledge_substrate/learning_from_bing_search.md) |\n| **Self-Demonstrations** | Successful action trajectories | [Experience Learning](../core_features/knowledge_substrate/experience_learning.md) |\n| **Human Demonstrations** | Expert-provided workflows | [Learning from Demonstrations](../core_features/knowledge_substrate/learning_from_demonstration.md) |\n\n**Knowledge Substrate Overview**: See [Knowledge Substrate](../core_features/knowledge_substrate/overview.md) for the complete RAG architecture.\n\n---\n\n## Command System\n\nAppAgent executes actions through the **MCP (Model-Context Protocol)** command system:\n\n**Application-Level Commands**:\n\n- `capture_window_screenshot` - Capture application window\n- `get_control_info` - Detect UI controls via UIA/OmniParser\n- `click_input` - Click on UI control\n- `set_edit_text` - Type text into input field\n- `annotation` - Annotate screenshot with control labels\n\n**Command Details**: See [Command System Documentation](commands.md) for complete command reference.\n\n---\n\n## Control Detection Backends\n\nAppAgent supports multiple control detection backends for comprehensive UI understanding:\n\n**UIA (UI Automation):** \nNative Windows UI Automation API for standard controls\n\n- ✅ Fast and accurate\n- ✅ Works with most Windows applications\n- ❌ May miss custom controls\n\n**OmniParser (Visual Detection):** \nVision-based grounding model for visual elements\n\n- ✅ Detects icons, images, custom controls\n- ✅ Works with web content\n- ❌ Requires external service\n\n**Hybrid (UIA + OmniParser):** \nBest of both worlds - maximum coverage\n\n- ✅ Native controls + visual elements\n- ✅ Comprehensive UI understanding\n\n**Control Detection Details**: See [Control Detection Overview](../core_features/control_detection/overview.md).\n\n---\n\n## Input and Output\n\n### AppAgent Input\n\n| Input | Description | Source |\n|-------|-------------|--------|\n| **User Request** | Original user request in natural language | HostAgent |\n| **Sub-Task** | Specific subtask to execute | HostAgent delegation |\n| **Application Context** | Target app name, window info | HostAgent |\n| **Control Information** | Detected UI controls with labels | Data collection phase |\n| **Screenshots** | Clean, annotated, previous step images | Data collection phase |\n| **Blackboard** | Shared memory for inter-agent communication | Global context |\n| **Retrieved Knowledge** | Help docs, demos, search results | RAG system |\n\n### AppAgent Output\n\n| Output | Description | Consumer |\n|--------|-------------|----------|\n| **Observation** | Current UI state description | LLM context |\n| **Thought** | Reasoning about next action | Execution log |\n| **ControlLabel** | Selected control to interact with | Action executor |\n| **Function** | MCP command to execute (click_input, set_edit_text, etc.) | Command dispatcher |\n| **Args** | Command parameters | Command dispatcher |\n| **Status** | Agent state (CONTINUE, FINISH, etc.) | State machine |\n| **Blackboard Update** | Execution results | HostAgent |\n\n**Example Output**:\n```json\n{\n \"Observation\": \"Word document with table, Export button at [12]\",\n \"Thought\": \"Click Export to extract table data\",\n \"ControlLabel\": \"12\",\n \"Function\": \"click_input\",\n \"Args\": {\"button\": \"left\"},\n \"Status\": \"CONTINUE\"\n}\n```\n\n---\n\n## Related Documentation\n\n**Detailed Documentation:**\n\n- **[State Machine](state.md)**: Complete FSM with state definitions and transitions\n- **[Processing Strategy](strategy.md)**: 4-phase pipeline implementation details\n- **[Command System](commands.md)**: Application-level MCP commands reference\n\n**Core Features:**\n\n- **[Hybrid Actions](../core_features/hybrid_actions.md)**: MCP command system for GUI–API execution\n- **[Control Detection](../core_features/control_detection/overview.md)**: UIA and visual detection\n- **[Knowledge Substrate](../core_features/knowledge_substrate/overview.md)**: RAG system overview\n\n**Tutorials:**\n\n- **[Creating AppAgent](../../tutorials/creating_app_agent/overview.md)**: Step-by-step guide\n- **[Help Document Provision](../../tutorials/creating_app_agent/help_document_provision.md)**: Add help docs\n- **[Demonstration Provision](../../tutorials/creating_app_agent/demonstration_provision.md)**: Add demos\n- **[Wrapping App-Native API](../../tutorials/creating_app_agent/warpping_app_native_api.md)**: Integrate APIs\n\n---\n\n## API Reference\n\n:::agents.agent.app_agent.AppAgent\n\n---\n\n## Summary\n\n**AppAgent Key Characteristics:**\n\n✅ **Application-Specialized Worker**: Dedicated to single Windows application \n✅ **ReAct Control Loop**: Iterative observe → think → act execution \n✅ **Hybrid Execution**: GUI automation + API calls via MCP commands \n✅ **7-State FSM**: Robust state management for execution control \n✅ **4-Phase Pipeline**: Structured data collection → reasoning → action → memory \n✅ **Knowledge-Enhanced**: RAG from docs, demos, and search \n✅ **Orchestrated by HostAgent**: Child agent in hierarchical architecture\n\n**Next Steps:**\n\n1. **Deep Dive**: Read [State Machine](state.md) and [Processing Strategy](strategy.md) for implementation details\n2. **Learn Features**: Explore [Core Features](../core_features/hybrid_actions.md) for advanced capabilities\n3. **Hands-On Tutorial**: Follow [Creating AppAgent](../../tutorials/creating_app_agent/overview.md) guide\n"
},
{
"path": "documents/docs/ufo2/app_agent/state.md",
"content": "# AppAgent State Machine\n\nAppAgent uses a **7-state finite state machine (FSM)** to control execution flow within a specific Windows application. The state machine manages subtask execution, UI re-annotation, user confirmations, error handling, and handoff back to HostAgent.\n\n---\n\n## State Overview\n\nAppAgent implements a robust 7-state FSM defined in `ufo/agents/states/app_agent_state.py`:\n\n```mermaid\ngraph TB\n subgraph \"Execution States\"\n CONTINUE[CONTINUE Main Execution]\n SCREENSHOT[SCREENSHOT UI Re-annotation]\n end\n \n subgraph \"Interaction States\"\n PENDING[PENDING Await User Input]\n CONFIRM[CONFIRM Safety Confirmation]\n end\n \n subgraph \"Terminal States\"\n FINISH[FINISH Success Return]\n FAIL[FAIL Failed Return]\n ERROR[ERROR Error Return]\n end\n \n style CONTINUE fill:#e3f2fd\n style SCREENSHOT fill:#fff3e0\n style PENDING fill:#f1f8e9\n style CONFIRM fill:#fce4ec\n style FINISH fill:#c8e6c9\n style FAIL fill:#ffe0b2\n style ERROR fill:#ffcdd2\n```\n\n### State Enumeration\n\n```python\nclass AppAgentStatus(Enum):\n \"\"\"Store the status of the app agent.\"\"\"\n \n CONTINUE = \"CONTINUE\" # Main execution state\n SCREENSHOT = \"SCREENSHOT\" # Re-annotation state\n FINISH = \"FINISH\" # Subtask completed successfully\n FAIL = \"FAIL\" # Subtask failed but recoverable\n PENDING = \"PENDING\" # Awaiting user input\n CONFIRM = \"CONFIRM\" # Safety confirmation required\n ERROR = \"ERROR\" # Critical failure\n```\n\n| State | Purpose | Processor Executed | Subtask Ends | Returns to HostAgent |\n|-------|---------|-------------------|--------------|---------------------|\n| **CONTINUE** | Main execution - interact with app controls | ✅ Yes (4 phases) | ❌ No | ❌ No |\n| **SCREENSHOT** | Re-capture and re-annotate UI after changes | ✅ Yes (4 phases) | ❌ No | ❌ No |\n| **FINISH** | Subtask completed successfully | ❌ No | ✅ Yes | ✅ Yes |\n| **FAIL** | Subtask failed but can be retried | ❌ No | ✅ Yes | ✅ Yes |\n| **PENDING** | Await user input for clarification | ✅ Yes (ask user) | ❌ No | ❌ No |\n| **CONFIRM** | Request user approval for safety-critical action | ✅ Yes (present dialog) | ❌ No | ❌ No |\n| **ERROR** | Unhandled exception or critical failure | ❌ No | ✅ Yes | ✅ Yes |\n\n---\n\n## State Definitions\n\n### CONTINUE State\n\n**Purpose**: Main execution state where AppAgent iteratively interacts with the application.\n\n```python\n@AppAgentStateManager.register\nclass ContinueAppAgentState(AppAgentState):\n \"\"\"The class for the continue app agent state.\"\"\"\n \n async def handle(\n self, agent: \"AppAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"\n Handle the agent for the current step.\n :param agent: The agent for the current step.\n :param context: The context for the agent and session.\n \"\"\"\n await agent.process(context)\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return False\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.CONTINUE.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Execution |\n| **Processor Executed** | ✓ Yes (4-phase pipeline) |\n| **Subtask Ends** | No |\n| **Round Ends** | No |\n| **Next States** | CONTINUE / SCREENSHOT / FINISH / PENDING / CONFIRM / ERROR |\n\n**Behavior**:\n\n- Executes 4-phase processing pipeline (DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE)\n- LLM analyzes UI and selects control to interact with\n- Executes action on selected control\n- Records action in memory and Blackboard\n- Transitions based on LLM's `Status` field in response\n\n**Example Flow**:\n```\nCONTINUE → Capture UI → LLM selects \"Export [12]\" → Click control 12 \n→ LLM returns Status: \"SCREENSHOT\" → Transition to SCREENSHOT\n```\n\nCONTINUE is the primary execution state where AppAgent spends most of its time during subtask execution.\n\n---\n\n### SCREENSHOT State\n\n**Purpose**: Re-capture and re-annotate UI after control interactions that change the interface.\n\n```python\n@AppAgentStateManager.register\nclass ScreenshotAppAgentState(ContinueAppAgentState):\n \"\"\"The class for the screenshot app agent state.\"\"\"\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.SCREENSHOT.value\n \n def next_state(self, agent: BasicAgent) -> AgentState:\n \"\"\"Determine next state based on control_reannotate.\"\"\"\n agent_processor = agent.processor\n \n if agent_processor is None:\n agent.status = AppAgentStatus.CONTINUE.value\n return ContinueAppAgentState()\n \n control_reannotate = agent_processor.control_reannotate\n \n if control_reannotate is None or len(control_reannotate) == 0:\n agent.status = AppAgentStatus.CONTINUE.value\n return ContinueAppAgentState()\n else:\n return super().next_state(agent)\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return False\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Execution |\n| **Processor Executed** | ✓ Yes (same as CONTINUE) |\n| **Subtask Ends** | No |\n| **Duration** | Single re-annotation cycle |\n| **Next States** | SCREENSHOT (if controls need re-annotation) / CONTINUE (if complete) |\n\n**Behavior**:\n\n- Inherits from `ContinueAppAgentState` - executes same 4-phase pipeline\n- Re-captures screenshot after UI changes (dialog opened, menu expanded, etc.)\n- Re-detects and re-annotates controls with updated labels\n- Checks `control_reannotate` to determine if more re-annotation needed\n- Transitions to CONTINUE once UI stabilizes\n\n**When to Use**:\n\n- LLM sets `Status: \"SCREENSHOT\"` when it expects UI changes\n- After clicking buttons that open dialogs\n- After expanding dropdown menus or combo boxes\n- After any action that significantly alters the UI\n\n**Screenshot Example:**\n\n```\nAction: Click \"Export\" button [12]\n→ Dialog opens with new controls\n→ LLM sets Status: \"SCREENSHOT\"\n→ SCREENSHOT state re-annotates dialog controls as [1], [2], [3]...\n→ Transitions to CONTINUE with fresh annotations\n```\n\n---\n\n### FINISH State\n\n**Purpose**: Subtask completed successfully - archive results and return control to HostAgent.\n\n```python\n@AppAgentStateManager.register\nclass FinishAppAgentState(AppAgentState):\n \"\"\"The class for the finish app agent state.\"\"\"\n \n async def handle(\n self, agent: \"AppAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Archive subtask result.\"\"\"\n if agent.processor:\n result = agent.processor.processing_context.get_local(\"result\")\n else:\n result = None\n \n await self.archive_subtask(context, result)\n \n def next_agent(self, agent: \"AppAgent\") -> HostAgent:\n \"\"\"Get the agent for the next step.\"\"\"\n return agent.host\n \n def next_state(self, agent: \"AppAgent\") -> HostAgentState:\n \"\"\"Get the next state of the agent.\"\"\"\n if agent.mode == \"follower\":\n return FinishHostAgentState()\n else:\n return ContinueHostAgentState()\n```\n\nFINISH indicates successful completion. The subtask result is available in the Blackboard for HostAgent to access and use in subsequent orchestration decisions.\n\n---\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return True\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.FINISH.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Subtask Ends** | ✓ Yes |\n| **Round Ends** | No (HostAgent continues) |\n| **Next Agent** | HostAgent |\n| **Next States** | HostAgent.CONTINUE (normal) / HostAgent.FINISH (follower mode) |\n\n**Behavior**:\n\n- Archives subtask to `previous_subtasks` with status and result\n- Writes execution results to Blackboard for HostAgent\n- Returns control to HostAgent\n- HostAgent determines next action (new subtask, finish, etc.)\n\n**Transition Logic**:\n\n```python\n# In LLM response\n{\n \"Status\": \"FINISH\",\n \"Comment\": \"Table data successfully extracted and saved\"\n}\n\n# Next agent and state\nnext_agent = agent.host # HostAgent\nnext_state = ContinueHostAgentState() # HostAgent continues orchestration\n```\n\n!!!success \"Subtask Completion\"\n FINISH indicates successful completion. The subtask result is available in the Blackboard for HostAgent to access and use in subsequent orchestration decisions.\n\n---\n\n### PENDING State\n\n**Purpose**: Await user input to clarify ambiguous situations or provide additional information.\n\n```python\n@AppAgentStateManager.register\nclass PendingAppAgentState(AppAgentState):\n \"\"\"The class for the pending app agent state.\"\"\"\n \n async def handle(\n self, agent: \"AppAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Ask the user questions to help the agent proceed.\"\"\"\n agent.process_asker(ask_user=ufo_config.system.ask_question)\n \n def next_state(self, agent: AppAgent) -> AppAgentState:\n \"\"\"Get the next state of the agent.\"\"\"\n agent.status = AppAgentStatus.CONTINUE.value\n return ContinueAppAgentState()\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return False\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.PENDING.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Interaction |\n| **Processor Executed** | ✓ Yes (ask user) |\n| **Subtask Ends** | No |\n| **Duration** | Until user responds |\n| **Next States** | CONTINUE (user provided input) |\n\n**Behavior**:\n\n- Displays question to user via `process_asker`\n- Waits for user response (configurable via `ask_question` setting)\n- User input is added to context for next CONTINUE execution\n- Always transitions to CONTINUE after user responds\n\n**Use Cases**:\n\n- Ambiguous control selection: \"Which 'Export' button should I click?\"\n- Missing information: \"What filename should I use for the export?\"\n- Clarification needed: \"Should I overwrite the existing file?\"\n\n!!!warning \"Configuration Required\"\n Set `system.ask_question = true` in configuration to enable PENDING state user interaction. If disabled, the agent will skip asking and make a best-effort decision.\n\n---\n\n### CONFIRM State\n\n**Purpose**: Request user approval before executing safety-critical or irreversible actions.\n\n```python\n@AppAgentStateManager.register\nclass ConfirmAppAgentState(AppAgentState):\n \"\"\"The class for the confirm app agent state.\"\"\"\n \n def __init__(self) -> None:\n \"\"\"Initialize the confirm state.\"\"\"\n self._confirm = None\n \n async def handle(\n self, agent: \"AppAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Request user confirmation for the action.\"\"\"\n # If safe guard disabled, proceed automatically\n if not ufo_config.system.safe_guard:\n await agent.process_resume()\n self._confirm = True\n return\n \n # Ask user for confirmation\n self._confirm = agent.process_confirmation()\n \n # If user confirms, resume the task\n if self._confirm:\n await agent.process_resume()\n \n def next_state(self, agent: AppAgent) -> AppAgentState:\n \"\"\"Get the next state based on user decision.\"\"\"\n if self._confirm:\n agent.status = AppAgentStatus.CONTINUE.value\n return ContinueAppAgentState()\n else:\n agent.status = AppAgentStatus.FINISH.value\n return FinishAppAgentState()\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return False\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.CONFIRM.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Interaction |\n| **Processor Executed** | ✓ Yes (present confirmation) |\n| **Subtask Ends** | No |\n| **Duration** | Until user approves/rejects |\n| **Next States** | CONTINUE (approved) / FINISH (rejected) |\n\n**Behavior**:\n\n- Presents action for user approval via `process_confirmation`\n- Waits for user decision (approve/reject)\n- If approved: Resumes processing via `process_resume` → CONTINUE\n- If rejected: Archives subtask → FINISH\n- Bypassed if `safe_guard` configuration is disabled\n\n**Safety-Critical Actions**:\n\n- File deletions: \"About to delete file.txt - Confirm?\"\n- Application launches: \"Launch Calculator.exe?\"\n- System configuration changes: \"Modify registry key?\"\n\n!!!warning \"Safety Mechanism\"\n CONFIRM provides a safety net for potentially destructive operations. Configure `system.safe_guard = true` to enable confirmation prompts.\n\n---\n\n### ERROR State\n\n**Purpose**: Handle unrecoverable exceptions and critical failures - archive error and return to HostAgent.\n\n```python\n@AppAgentStateManager.register\nclass ErrorAppAgentState(AppAgentState):\n \"\"\"The class for the error app agent state.\"\"\"\n \n async def handle(\n self, agent: \"AppAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Archive subtask with error result.\"\"\"\n if agent.processor:\n result = agent.processor.processing_context.get_local(\"result\")\n else:\n result = None\n \n await self.archive_subtask(context, result)\n \n def next_agent(self, agent: \"AppAgent\") -> HostAgent:\n \"\"\"Get the agent for the next step.\"\"\"\n return agent.host\n \n def next_state(self, agent: \"AppAgent\") -> HostAgentState:\n \"\"\"Get the next state of the agent.\"\"\"\n return FinishHostAgentState()\n \n def is_round_end(self) -> bool:\n \"\"\"Check if the round ends.\"\"\"\n return True\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return True\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.ERROR.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Subtask Ends** | ✓ Yes |\n| **Round Ends** | ✓ Yes |\n| **Next Agent** | HostAgent |\n| **Next States** | HostAgent.FINISH (terminate round) |\n\n**Behavior**:\n\n- Archives subtask with error status and error details\n- Returns control to HostAgent\n- HostAgent transitions to FINISH (ends current round)\n- Error details logged for debugging\n\n**Error Scenarios**:\n\n- Unhandled Python exceptions during processing\n- Critical LLM failures (timeout, invalid response)\n- Command dispatcher failures\n- Unrecoverable application crashes\n\n!!!danger \"Terminal State\"\n ERROR terminates both the subtask and the current round. HostAgent will end the session or start a new round depending on configuration.\n\n---\n\n### FAIL State\n\n**Purpose**: Handle recoverable failures - archive failed subtask and return to HostAgent for retry or alternative approach.\n\n```python\n@AppAgentStateManager.register\nclass FailAppAgentState(AppAgentState):\n \"\"\"The class for the fail app agent state.\"\"\"\n \n async def handle(\n self, agent: \"AppAgent\", context: Optional[\"Context\"] = None\n ) -> None:\n \"\"\"Archive subtask with failure result.\"\"\"\n if agent.processor:\n result = agent.processor.processing_context.get_local(\"result\")\n else:\n result = None\n \n await self.archive_subtask(context, result)\n \n def next_agent(self, agent: \"AppAgent\") -> HostAgent:\n \"\"\"Get the agent for the next step.\"\"\"\n return agent.host\n \n def next_state(self, agent: \"AppAgent\") -> HostAgentState:\n \"\"\"Get the next state of the agent.\"\"\"\n return FinishHostAgentState()\n \n def is_round_end(self) -> bool:\n \"\"\"Check if the round ends.\"\"\"\n return False\n \n def is_subtask_end(self) -> bool:\n \"\"\"Check if the subtask ends.\"\"\"\n return True\n \n @classmethod\n def name(cls) -> str:\n \"\"\"The class name of the state.\"\"\"\n return AppAgentStatus.FAIL.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Subtask Ends** | ✓ Yes |\n| **Round Ends** | ✗ No (unlike ERROR) |\n| **Next Agent** | HostAgent |\n| **Next States** | HostAgent.FINISH (but round doesn't end) |\n\n**Behavior**:\n\n- Archives subtask with FAIL status and failure details\n- Returns control to HostAgent\n- HostAgent can retry subtask or try alternative approach\n- Unlike ERROR, does not terminate the round\n- Allows for graceful degradation and recovery\n\n**Failure Scenarios**:\n\n- Control not found but task can be retried\n- Action timeout but application still responsive\n- Partial completion with known issues\n- Expected failure conditions\n\n!!!info \"Recoverable Failures\"\n FAIL indicates a recoverable failure that the HostAgent can handle gracefully, unlike ERROR which terminates the entire round. Use FAIL when the task failed but the system is still in a valid state.\n\n---\n\n## State Transition Diagram\n\n```mermaid\nstateDiagram-v2\n [*] --> CONTINUE: HostAgent Delegates Subtask\n\n CONTINUE --> CONTINUE: LLM: More actions Status: CONTINUE\n CONTINUE --> SCREENSHOT: LLM: UI changed Status: SCREENSHOT\n CONTINUE --> FINISH: LLM: Complete Status: FINISH\n CONTINUE --> FAIL: LLM: Failed Status: FAIL\n CONTINUE --> CONFIRM: LLM: Need approval Status: CONFIRM\n CONTINUE --> PENDING: LLM: Need info Status: PENDING\n CONTINUE --> ERROR: System: Exception Status: ERROR\n \n SCREENSHOT --> SCREENSHOT: System: More re-annotation\n SCREENSHOT --> CONTINUE: System: Re-annotation done\n \n CONFIRM --> CONTINUE: User: Approved\n CONFIRM --> FINISH: User: Rejected\n \n PENDING --> CONTINUE: User: Provided input\n \n FINISH --> HostAgent_CONTINUE: Return to HostAgent\n FAIL --> HostAgent_CONTINUE: Return to HostAgent (Can retry)\n ERROR --> HostAgent_FINISH: Return to HostAgent\n \n HostAgent_CONTINUE --> [*]: HostAgent Takes Control\n HostAgent_FINISH --> [*]: Round Terminated\n \n note right of CONTINUE: Main execution 4-phase pipeline\n note right of SCREENSHOT: UI re-annotation after changes\n note left of CONFIRM: Safety check for critical actions\n note left of PENDING: User input for clarification\n```\n\n\n \n AppAgent State Machine: Visual representation of the 6-state FSM with transitions and conditions\n\n\n---\n\n## State Transition Control\n\n### LLM-Driven Transitions\n\nMost state transitions are controlled by the LLM through the `Status` field in its response:\n\n```json\n{\n \"Observation\": \"Word document with Export button [12] visible\",\n \"Thought\": \"I should click the Export button to extract table data\",\n \"ControlLabel\": \"12\",\n \"ControlText\": \"Export\",\n \"Function\": \"click_input\",\n \"Args\": {\"button\": \"left\"},\n \"Status\": \"SCREENSHOT\",\n \"Comment\": \"Clicking Export will open a dialog\"\n}\n```\n\n**Status Mapping**:\n\n| LLM Status Value | Next State | Decision Logic |\n|-----------------|------------|----------------|\n| `\"CONTINUE\"` | CONTINUE | More actions needed, continue execution |\n| `\"SCREENSHOT\"` | SCREENSHOT | UI will change, re-annotate controls |\n| `\"FINISH\"` | FINISH | Subtask complete, return to HostAgent |\n| `\"FAIL\"` | FAIL | Subtask failed but recoverable |\n| `\"PENDING\"` | PENDING | Need user clarification |\n| `\"CONFIRM\"` | CONFIRM | Safety-critical action needs approval |\n| `\"ERROR\"` | ERROR | Manually triggered error (rare) |\n\n### System-Driven Transitions\n\nSome transitions are triggered by system conditions:\n\n```python\n# Exception handling in processor\ntry:\n result = await processor.process(agent, context)\nexcept Exception as e:\n agent.status = AppAgentStatus.ERROR.value\n # Transitions to ERROR state\n\n# Screenshot re-annotation check\nif control_reannotate and len(control_reannotate) > 0:\n # Stay in SCREENSHOT state\n return ScreenshotAppAgentState()\nelse:\n # Transition to CONTINUE\n agent.status = AppAgentStatus.CONTINUE.value\n return ContinueAppAgentState()\n```\n\n---\n\n## Implementation Details\n\n### State Class Hierarchy\n\n```mermaid\nclassDiagram\n class AgentState {\n <>\n +handle(agent, context)*\n +next_agent(agent)*\n +next_state(agent)*\n +is_subtask_end()*\n +is_round_end()\n +name()*\n }\n \n class AppAgentState {\n <>\n +agent_class() AppAgent\n +archive_subtask(context, result)\n }\n \n class ContinueAppAgentState {\n +handle() await agent.process()\n +is_subtask_end() False\n +name() \"CONTINUE\"\n }\n \n class ScreenshotAppAgentState {\n +next_state() check control_reannotate\n +name() \"SCREENSHOT\"\n }\n \n class FinishAppAgentState {\n +handle() archive_subtask\n +next_agent() HostAgent\n +next_state() HostAgent.CONTINUE\n +is_subtask_end() True\n +name() \"FINISH\"\n }\n \n class PendingAppAgentState {\n +handle() process_asker\n +next_state() CONTINUE\n +name() \"PENDING\"\n }\n \n class ConfirmAppAgentState {\n -_confirm: bool\n +handle() process_confirmation\n +next_state() CONTINUE or FINISH\n +name() \"CONFIRM\"\n }\n \n class ErrorAppAgentState {\n +handle() archive_subtask\n +next_agent() HostAgent\n +next_state() HostAgent.FINISH\n +is_round_end() True\n +is_subtask_end() True\n +name() \"ERROR\"\n }\n \n class FailAppAgentState {\n +handle() archive_subtask\n +next_agent() HostAgent\n +next_state() HostAgent.FINISH\n +is_round_end() False\n +is_subtask_end() True\n +name() \"FAIL\"\n }\n \n AgentState <|-- AppAgentState\n AppAgentState <|-- ContinueAppAgentState\n AppAgentState <|-- FinishAppAgentState\n AppAgentState <|-- PendingAppAgentState\n AppAgentState <|-- ConfirmAppAgentState\n AppAgentState <|-- ErrorAppAgentState\n AppAgentState <|-- FailAppAgentState\n ContinueAppAgentState <|-- ScreenshotAppAgentState\n```\n\n### State Manager Registry\n\n```python\nclass AppAgentStateManager(AgentStateManager):\n \"\"\"State manager for AppAgent with registration system.\"\"\"\n \n _state_mapping: Dict[str, Type[AppAgentState]] = {}\n \n @property\n def none_state(self) -> AgentState:\n \"\"\"The none state of the state manager.\"\"\"\n return NoneAppAgentState()\n\n# States are registered via decorator\n@AppAgentStateManager.register\nclass ContinueAppAgentState(AppAgentState):\n ...\n```\n\n**Registration Benefits**:\n\n- Automatic state mapping by name\n- Centralized state lookup via `get_state(status)`\n- Type-safe state retrieval\n- Easy to add new states\n\n---\n\n## Execution Flow Example\n\n### Multi-Step Subtask Execution\n\n```mermaid\nsequenceDiagram\n participant HostAgent\n participant AppAgent\n participant CONTINUE\n participant SCREENSHOT\n participant FINISH\n participant Application\n \n HostAgent->>AppAgent: Delegate subtask \"Extract table from Word\"\n AppAgent->>CONTINUE: Set state\n \n rect rgb(230, 240, 255)\n Note over CONTINUE, Application: Step 1: Capture and analyze\n CONTINUE->>Application: Capture screenshot\n Application-->>CONTINUE: Screenshot + 50 controls\n CONTINUE->>CONTINUE: LLM: \"Click Export [12]\"\n CONTINUE->>Application: click_input(12)\n Application-->>CONTINUE: Dialog opened\n CONTINUE->>SCREENSHOT: Status: \"SCREENSHOT\"\n end\n \n rect rgb(255, 250, 230)\n Note over SCREENSHOT, Application: Step 2: Re-annotate\n SCREENSHOT->>Application: Re-capture screenshot\n Application-->>SCREENSHOT: Screenshot + 30 dialog controls\n SCREENSHOT->>SCREENSHOT: LLM: \"Select CSV [5]\"\n SCREENSHOT->>Application: click_input(5)\n Application-->>SCREENSHOT: Format selected\n SCREENSHOT->>CONTINUE: Re-annotation done\n end\n \n rect rgb(230, 255, 240)\n Note over CONTINUE, Application: Step 3: Complete export\n CONTINUE->>Application: Capture screenshot\n Application-->>CONTINUE: Screenshot + updated controls\n CONTINUE->>CONTINUE: LLM: \"Click OK [1]\"\n CONTINUE->>Application: click_input(1)\n Application-->>CONTINUE: Export complete\n CONTINUE->>FINISH: Status: \"FINISH\"\n end\n \n FINISH->>HostAgent: Return control subtask result in Blackboard\n```\n\n---\n\n## Related Documentation\n\n**Architecture:**\n\n- **[AppAgent Overview](overview.md)**: High-level architecture and responsibilities\n- **[Processing Strategy](strategy.md)**: 4-phase processing pipeline details\n- **[HostAgent State Machine](../host_agent/state.md)**: Parent agent FSM\n\n**Design Patterns:**\n\n- **[State Layer Design](../../infrastructure/agents/design/state.md)**: FSM design principles\n- **[Processor Framework](../../infrastructure/agents/design/processor.md)**: Processing architecture\n\n---\n\n## API Reference\n\n:::agents.states.app_agent_state.AppAgentState\n:::agents.states.app_agent_state.AppAgentStateManager\n\n---\n\n## Summary\n\n**AppAgent State Machine Key Features:**\n\n✅ **7-State FSM**: CONTINUE, SCREENSHOT, FINISH, FAIL, PENDING, CONFIRM, ERROR \n✅ **LLM-Driven**: Most transitions controlled by LLM's `Status` field \n✅ **UI Re-annotation**: SCREENSHOT state handles dynamic UI changes \n✅ **User Interaction**: PENDING and CONFIRM states for human input \n✅ **Error Handling**: ERROR and FAIL states for graceful failure recovery \n✅ **HostAgent Integration**: FINISH/FAIL/ERROR return control to parent agent \n✅ **Subtask Archiving**: Execution history tracked in `previous_subtasks`\n\n**Next Steps:**\n\n1. **Understand Processing**: Read [Processing Strategy](strategy.md) for pipeline details\n2. **Learn Commands**: Check [Command System](commands.md) for available actions\n3. **Explore Patterns**: Review [State Layer Design](../../infrastructure/agents/design/state.md) for FSM principles\n"
},
{
"path": "documents/docs/ufo2/app_agent/strategy.md",
"content": "# AppAgent Processing Strategy\n\nAppAgent executes a **4-phase processing pipeline** in **CONTINUE** and **SCREENSHOT** states. Each phase handles a specific aspect of application-level automation: **data collection** (screenshot + controls), **LLM reasoning**, **action execution**, and **memory recording**. This document details the implementation of each strategy based on the actual codebase.\n\n---\n\n## Strategy Assembly\n\nProcessing strategies are **assembled and orchestrated** by the `AppAgentProcessor` class defined in `ufo/agents/processors/app_agent_processor.py`. The processor acts as the **coordinator** that initializes, configures, and executes the 4-phase pipeline for application-level automation.\n\n### AppAgentProcessor Overview\n\nThe `AppAgentProcessor` extends `ProcessorTemplate` and serves as the main orchestrator for AppAgent workflows:\n\n```python\nclass AppAgentProcessor(ProcessorTemplate):\n \"\"\"\n App Agent Processor - Modern, extensible App Agent processing implementation.\n \n Processing Pipeline:\n 1. Data Collection: Screenshot capture and UI control information (composed strategy)\n 2. LLM Interaction: Context-aware prompting and response parsing\n 3. Action Execution: UI automation and control interaction\n 4. Memory Update: Agent memory and blackboard synchronization\n \n Middleware Stack:\n - Structured logging and debugging middleware\n \"\"\"\n \n processor_context_class = AppAgentProcessorContext\n \n def __init__(self, agent: \"AppAgent\", global_context: \"Context\"):\n super().__init__(agent, global_context)\n```\n\n### Strategy Registration\n\nDuring initialization, `AppAgentProcessor._setup_strategies()` registers all four processing strategies:\n\n```python\ndef _setup_strategies(self) -> None:\n \"\"\"Setup processing strategies for App Agent.\"\"\"\n \n # Phase 1: Data collection (COMPOSED: Screenshot + Control Info)\n self.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(\n strategies=[\n AppScreenshotCaptureStrategy(),\n AppControlInfoStrategy(),\n ],\n name=\"AppDataCollectionStrategy\",\n fail_fast=True, # Data collection is critical\n )\n \n # Phase 2: LLM interaction (critical - fail_fast=True)\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n AppLLMInteractionStrategy(\n fail_fast=True # LLM failure should trigger recovery\n )\n )\n \n # Phase 3: Action execution (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n AppActionExecutionStrategy(\n fail_fast=False # Action failures can be handled gracefully\n )\n )\n \n # Phase 4: Memory update (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n AppMemoryUpdateStrategy(\n fail_fast=False # Memory update failures shouldn't stop process\n )\n )\n```\n\n| Phase | Strategy Class | fail_fast | Composition | Rationale |\n|-------|---------------|-----------|-------------|-----------|\n| **DATA_COLLECTION** | `ComposedStrategy` (Screenshot + Control Info) | ✓ True | ✓ Composed | Screenshot and control detection are critical for LLM context |\n| **LLM_INTERACTION** | `AppLLMInteractionStrategy` | ✓ True | ✗ Single | LLM response failure requires immediate recovery |\n| **ACTION_EXECUTION** | `AppActionExecutionStrategy` | ✗ False | ✗ Single | Action failures can be gracefully handled and retried |\n| **MEMORY_UPDATE** | `AppMemoryUpdateStrategy` | ✗ False | ✗ Single | Memory failures shouldn't block the main execution flow |\n\n**Composed Strategy Pattern:** \nPhase 1 uses **ComposedStrategy** to execute two sub-strategies sequentially:\n\n1. **AppScreenshotCaptureStrategy**: Captures application window + desktop screenshots\n2. **AppControlInfoStrategy**: Detects UI controls via UIA/OmniParser and creates annotations\n\nThis ensures both screenshot and control data are available together for the LLM analysis phase.\n\n### Middleware Configuration\n\nThe processor configures specialized logging middleware:\n\n```python\ndef _setup_middleware(self) -> None:\n \"\"\"Setup middleware pipeline for App Agent.\"\"\"\n self.middleware_chain = [AppAgentLoggingMiddleware()]\n```\n\n**AppAgentLoggingMiddleware** provides:\n\n- Subtask and application context tracking\n- Rich Panel displays with color coding\n- Action execution logging\n- Performance metrics and cost tracking\n\n---\n\n## Processing Pipeline Architecture\n\n```mermaid\ngraph TB\n subgraph \"Phase 1: DATA_COLLECTION (ComposedStrategy)\"\n SS[AppScreenshotCaptureStrategy Capture Screenshots]\n CI[AppControlInfoStrategy Detect & Annotate Controls]\n SS --> CI\n end\n \n subgraph \"Phase 2: LLM_INTERACTION\"\n LLM[AppLLMInteractionStrategy LLM Reasoning]\n end\n \n subgraph \"Phase 3: ACTION_EXECUTION\"\n AE[AppActionExecutionStrategy Execute UI Action]\n end\n \n subgraph \"Phase 4: MEMORY_UPDATE\"\n MU[AppMemoryUpdateStrategy Record in Memory & Blackboard]\n end\n \n CI --> LLM\n LLM --> AE\n AE --> MU\n \n style SS fill:#e1f5ff\n style CI fill:#e1f5ff\n style LLM fill:#fff4e6\n style AE fill:#e8f5e9\n style MU fill:#fce4ec\n```\n\n---\n\n## Phase 1: DATA_COLLECTION\n\n### Strategy: `ComposedStrategy` (Screenshot + Control Info)\n\n**Purpose**: Gather comprehensive application UI context including screenshots and control information for LLM decision making.\n\n```python\n# Composed strategy combines two sub-strategies\nself.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(\n strategies=[\n AppScreenshotCaptureStrategy(),\n AppControlInfoStrategy(),\n ],\n name=\"AppDataCollectionStrategy\",\n fail_fast=True,\n)\n```\n\n### Sub-Strategy 1: AppScreenshotCaptureStrategy\n\n**Purpose**: Capture application window and desktop screenshots.\n\n```python\n@depends_on(\"app_root\", \"log_path\", \"session_step\")\n@provides(\n \"clean_screenshot_path\",\n \"annotated_screenshot_path\",\n \"desktop_screenshot_path\",\n \"ui_tree_path\",\n \"clean_screenshot_url\",\n \"desktop_screenshot_url\",\n \"application_window_info\",\n \"screenshot_saved_time\",\n)\nclass AppScreenshotCaptureStrategy(BaseProcessingStrategy):\n \"\"\"Strategy for capturing application screenshots and desktop screenshots.\"\"\"\n \n async def execute(self, agent, context) -> ProcessingResult:\n # 1. Capture application window screenshot\n clean_screenshot_url = await self._capture_app_screenshot(\n clean_screenshot_path, command_dispatcher\n )\n \n # 2. Capture desktop screenshot if needed\n if ufo_config.system.save_full_screen:\n desktop_screenshot_url = await self._capture_desktop_screenshot(\n desktop_screenshot_path, command_dispatcher\n )\n \n # 3. Capture UI tree if needed\n if ufo_config.system.save_ui_tree:\n await self._capture_ui_tree(ui_tree_path, command_dispatcher)\n \n # 4. Get application window information\n application_window_info = await self._get_application_window_info(\n command_dispatcher\n )\n \n return ProcessingResult(success=True, data={...})\n```\n\n**Execution Steps**:\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant CommandDispatcher\n participant Application\n \n Strategy->>CommandDispatcher: capture_window_screenshot()\n CommandDispatcher->>Application: Screenshot app window\n Application-->>Strategy: clean_screenshot_url\n Strategy->>Strategy: Save to log_path/action_stepN.png\n \n alt save_full_screen=True\n Strategy->>CommandDispatcher: capture_desktop_screenshot(all_screens=True)\n CommandDispatcher-->>Strategy: desktop_screenshot_url\n Strategy->>Strategy: Save to log_path/desktop_stepN.png\n end\n \n alt save_ui_tree=True\n Strategy->>CommandDispatcher: get_ui_tree()\n CommandDispatcher-->>Strategy: ui_tree JSON\n Strategy->>Strategy: Save to log_path/ui_trees/ui_tree_stepN.json\n end\n \n Strategy->>CommandDispatcher: get_app_window_info()\n CommandDispatcher-->>Strategy: application_window_info\n```\n\n**Key Outputs**:\n\n| Output | Type | Description | Example |\n|--------|------|-------------|---------|\n| `clean_screenshot_url` | str | Base64 image of app window | `data:image/png;base64,iVBORw0K...` |\n| `clean_screenshot_path` | str | File path to screenshot | `logs/action_step5.png` |\n| `desktop_screenshot_url` | str | Base64 image of desktop | `data:image/png;base64,iVBORw0K...` |\n| `application_window_info` | TargetInfo | Window metadata (name, rect, type) | `TargetInfo(name=\"Word\", rect=[0,0,1920,1080])` |\n| `screenshot_saved_time` | float | Performance timing (seconds) | `0.324` |\n\n### Sub-Strategy 2: AppControlInfoStrategy\n\n**Purpose**: Detect, filter, and annotate UI controls using UIA and/or OmniParser.\n\n```python\n@depends_on(\"clean_screenshot_path\", \"application_window_info\")\n@provides(\n \"control_info\",\n \"annotation_dict\",\n \"control_filter_time\",\n \"control_recorder\",\n \"annotated_screenshot_path\",\n \"annotated_screenshot_url\",\n)\nclass AppControlInfoStrategy(BaseProcessingStrategy):\n \"\"\"Strategy for collecting and filtering UI control information.\"\"\"\n \n def __init__(self, fail_fast: bool = True):\n super().__init__(name=\"app_control_info\", fail_fast=fail_fast)\n self.control_detection_backend = ufo_config.system.control_backend\n self.photographer = PhotographerFacade()\n \n if \"omniparser\" in self.control_detection_backend:\n self.grounding_service = OmniparserGrounding(...)\n```\n\n**Execution Steps**:\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant UIA\n participant OmniParser\n participant Photographer\n \n alt UIA Backend Enabled\n Strategy->>UIA: get_app_window_controls_target_info()\n UIA-->>Strategy: api_control_list (50 controls)\n end\n \n alt OmniParser Backend Enabled\n Strategy->>OmniParser: screen_parsing(screenshot)\n OmniParser-->>Strategy: grounding_control_list (12 controls)\n end\n \n Strategy->>Strategy: Merge UIA + OmniParser lists (deduplicate by IoU overlap)\n Strategy->>Strategy: Create annotation_dict {id: TargetInfo}\n \n Strategy->>Photographer: capture_with_target_list() (draw labels [1], [2], [3]...)\n Photographer-->>Strategy: annotated_screenshot_url\n```\n\n**Control Detection Backends**:\n\n**UIA (UI Automation):**\n\n```python\nasync def _collect_uia_controls(self, command_dispatcher) -> List[TargetInfo]:\n \"\"\"Collect UIA controls from the application window.\"\"\"\n result = await command_dispatcher.execute_commands([\n Command(\n tool_name=\"get_app_window_controls_target_info\",\n parameters={\"field_list\": [\"id\", \"name\", \"type\", \"rect\", ...]},\n )\n ])\n \n target_info_list = [TargetInfo(**control) for control in result[0].result]\n return target_info_list\n```\n \n**Advantages**: Fast, accurate, native Windows controls\n**Limitations**: May miss custom controls, web content, icons\n\n**OmniParser (Visual):**\n\n```python\nasync def _collect_grounding_controls(\n self, clean_screenshot_path, application_window_info\n) -> List[TargetInfo]:\n \"\"\"Collect controls using grounding service.\"\"\"\n grounding_controls = self.grounding_service.screen_parsing(\n clean_screenshot_path, application_window_info\n )\n return grounding_controls\n```\n \n**Advantages**: Detects visual elements (icons, images, custom controls)\n**Limitations**: Slower, requires external service\n\n**Hybrid (UIA + OmniParser):**\n\n```python\ndef _collect_merged_control_list(\n self, api_control_list, grounding_control_list\n) -> List[TargetInfo]:\n \"\"\"Merge UIA and grounding sources with IoU deduplication.\"\"\"\n merged_controls = self.photographer.merge_target_info_list(\n api_control_list,\n grounding_control_list,\n iou_overlap_threshold=ufo_config.system.iou_threshold_for_merge,\n )\n return merged_controls\n```\n \n**Advantage**: Maximum coverage - native + visual elements\n\n**Annotation Process**:\n\n```python\n# Create annotation dictionary mapping IDs to controls\nannotation_dict = {\n \"1\": TargetInfo(id=\"1\", name=\"Export\", type=\"Button\", rect=[100, 200, 150, 230]),\n \"2\": TargetInfo(id=\"2\", name=\"Save\", type=\"Button\", rect=[160, 200, 210, 230]),\n # ... more controls\n}\n\n# Draw labels on screenshot\nannotated_screenshot_url = self._save_annotated_screenshot(\n application_window_info,\n clean_screenshot_path,\n merged_control_list,\n annotated_screenshot_path,\n)\n```\n\n!!!example \"Control Detection Example\"\n ```\n UIA detects: 45 controls (buttons, textboxes, menus)\n OmniParser detects: 12 visual elements (icons, images)\n IoU deduplication removes: 3 overlapping controls\n Final merged list: 54 annotated controls [1] to [54]\n ```\n\n---\n\n## Phase 2: LLM_INTERACTION\n\n### Strategy: `AppLLMInteractionStrategy`\n\n**Purpose**: Build context-aware prompts with app-specific data and get LLM reasoning for next action.\n\n```python\n@provides(\n \"parsed_response\",\n \"response_text\",\n \"llm_cost\",\n \"prompt_message\",\n \"save_screenshot\",\n \"comment\",\n \"concat_screenshot_path\",\n \"plan\",\n \"observation\",\n \"last_control_screenshot_path\",\n \"action\",\n \"thought\",\n)\nclass AppLLMInteractionStrategy(BaseProcessingStrategy):\n \"\"\"Strategy for LLM interaction with App Agent specific prompting.\"\"\"\n \n async def execute(self, agent, context) -> ProcessingResult:\n # 1. Collect image strings (last step + current clean + annotated)\n image_string_list = self._collect_image_strings(...)\n \n # 2. Retrieve knowledge from RAG system\n knowledge_retrieved = self._knowledge_retrieval(agent, subtask)\n \n # 3. Build comprehensive prompt\n prompt_message = await self._build_app_prompt(...)\n \n # 4. Get LLM response with retry logic\n response_text, llm_cost = await self._get_llm_response(agent, prompt_message)\n \n # 5. Parse and validate response\n parsed_response = self._parse_app_response(agent, response_text)\n \n return ProcessingResult(success=True, data={...})\n```\n\n**Execution Flow**:\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Photographer\n participant RAG\n participant LLM\n \n Strategy->>Photographer: Collect image strings\n Note over Strategy: - Last step screenshot (selected control) - Clean screenshot - Annotated screenshot - Concatenated clean+annotated\n Photographer-->>Strategy: image_string_list\n \n Strategy->>RAG: Retrieve knowledge for subtask\n Note over RAG: - Experience examples - Demonstration examples - Offline docs - Online search results\n RAG-->>Strategy: knowledge_retrieved\n \n Strategy->>Strategy: Build comprehensive prompt (images + controls + knowledge + history)\n \n Strategy->>LLM: Get response with retry (max 3 attempts)\n LLM-->>Strategy: response_text\n \n Strategy->>Strategy: Parse JSON response to AppAgentResponse\n Strategy-->>Strategy: Return parsed_response\n```\n\n**Prompt Construction**:\n\n```python\nasync def _build_app_prompt(\n self,\n agent,\n control_info, # List of detected controls\n image_string_list, # Screenshots\n knowledge_retrieved, # RAG results\n request, # User request\n subtask, # Current subtask\n plan, # Previous plan\n prev_subtask, # Previous subtasks\n application_process_name,\n host_message, # Message from HostAgent\n session_step,\n request_logger,\n) -> List[Dict]:\n \"\"\"Build comprehensive prompt for App Agent.\"\"\"\n \n # Get blackboard context\n blackboard_prompt = agent.blackboard.blackboard_to_prompt()\n \n # Get last successful actions\n last_success_actions = self._get_last_success_actions(agent)\n \n # Extract knowledge\n retrieved_examples = (\n knowledge_retrieved[\"experience_examples\"] +\n knowledge_retrieved[\"demonstration_examples\"]\n )\n retrieved_knowledge = (\n knowledge_retrieved[\"offline_docs\"] +\n knowledge_retrieved[\"online_docs\"]\n )\n \n # Build prompt using agent's message constructor\n prompt_message = agent.message_constructor(\n dynamic_examples=retrieved_examples,\n dynamic_knowledge=retrieved_knowledge,\n image_list=image_string_list,\n control_info=control_info,\n prev_subtask=prev_subtask,\n plan=plan,\n request=request,\n subtask=subtask,\n current_application=application_process_name,\n host_message=host_message,\n blackboard_prompt=blackboard_prompt,\n last_success_actions=last_success_actions,\n )\n \n return prompt_message\n```\n\n**LLM Response Parsing**:\n\n```python\ndef _parse_app_response(self, agent, response_text: str) -> AppAgentResponse:\n \"\"\"Parse LLM response into structured AppAgentResponse.\"\"\"\n response_dict = agent.response_to_dict(response_text)\n parsed_response = AppAgentResponse.model_validate(response_dict)\n return parsed_response\n```\n\n**AppAgentResponse Schema**:\n\n```python\n{\n \"Observation\": \"Word document with Export button at label [12]\",\n \"Thought\": \"I should click Export to extract table data\",\n \"ControlLabel\": \"12\",\n \"ControlText\": \"Export\",\n \"Function\": \"click_input\",\n \"Args\": {\"button\": \"left\", \"double\": false},\n \"Status\": \"SCREENSHOT\",\n \"Plan\": [\"Click Export\", \"Select CSV format\", \"Choose save location\"],\n \"Comment\": \"Clicking Export will open a dialog\",\n \"SaveScreenshot\": {\"save\": false, \"reason\": \"\"}\n}\n```\n\n!!!tip \"Retry Logic\"\n LLM interaction includes automatic retry (configurable, default 3 attempts) to handle transient failures or JSON parsing errors.\n\n---\n\n## Phase 3: ACTION_EXECUTION\n\n### Strategy: `AppActionExecutionStrategy`\n\n**Purpose**: Execute UI actions on selected controls based on LLM response.\n\n```python\n@depends_on(\"parsed_response\", \"log_path\", \"session_step\")\n@provides(\n \"execution_result\",\n \"action_info\",\n \"control_log\",\n \"status\",\n \"selected_control_screenshot_path\",\n)\nclass AppActionExecutionStrategy(BaseProcessingStrategy):\n \"\"\"Strategy for executing App Agent actions.\"\"\"\n \n async def execute(self, agent, context) -> ProcessingResult:\n # 1. Extract parsed response\n parsed_response = context.get_local(\"parsed_response\")\n \n # 2. Execute the action via command dispatcher\n execution_results = await self._execute_app_action(\n command_dispatcher,\n parsed_response.action\n )\n \n # 3. Create action info for memory\n actions = self._create_action_info(\n annotation_dict,\n parsed_response.action,\n execution_results,\n )\n \n # 4. Save annotated screenshot with selected control highlighted\n self._save_annotated_screenshot(...)\n \n return ProcessingResult(success=True, data={...})\n```\n\n**Execution Flow**:\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant CommandDispatcher\n participant Application\n participant Photographer\n \n Strategy->>Strategy: Extract action from parsed_response\n Note over Strategy: ControlLabel: \"12\" Function: \"click_input\" Args: {\"button\": \"left\"}\n \n Strategy->>Strategy: Convert action to Command\n Note over Strategy: Command(tool_name=\"click_input\", parameters={\"id\": \"12\", \"button\": \"left\"})\n \n Strategy->>CommandDispatcher: execute_commands([command])\n CommandDispatcher->>Application: Perform UI automation\n Application-->>CommandDispatcher: Result (status, message)\n CommandDispatcher-->>Strategy: execution_results\n \n Strategy->>Strategy: Create action_info (merge control, action, result)\n Strategy->>Strategy: Print action to console\n \n Strategy->>Photographer: Save screenshot with selected control\n Photographer-->>Strategy: selected_control_screenshot_path\n```\n\n**Action to Command Conversion**:\n\n```python\ndef _action_to_command(self, action: ActionCommandInfo) -> Command:\n \"\"\"Convert ActionCommandInfo to Command for execution.\"\"\"\n return Command(\n tool_name=action.function, # e.g., \"click_input\"\n parameters=action.arguments or {}, # e.g., {\"id\": \"12\", \"button\": \"left\"}\n tool_type=\"action\",\n )\n```\n\n**Action Info Creation**:\n\n```python\ndef _create_action_info(\n self,\n annotation_dict,\n actions,\n execution_results,\n) -> List[ActionCommandInfo]:\n \"\"\"Create action information for memory tracking.\"\"\"\n \n # Handle single or multiple actions\n if isinstance(actions, ActionCommandInfo):\n actions = [actions]\n \n # Merge control info with action results\n for i, action in enumerate(actions):\n if action.arguments and \"id\" in action.arguments:\n control_id = action.arguments[\"id\"]\n target_control = annotation_dict.get(control_id)\n action.target = target_control # Link to TargetInfo\n \n action.result = execution_results[i] # Link to execution result\n \n return actions\n```\n\n**Example Action Execution**:\n\n```\nInput: ControlLabel=\"12\", Function=\"click_input\", Args={\"button\": \"left\"}\n↓\nCommand: Command(tool_name=\"click_input\", parameters={\"id\": \"12\", \"button\": \"left\"})\n↓\nExecution: Click control [12] (Export button) with left mouse button\n↓\nResult: ResultStatus.SUCCESS, message=\"Clicked control successfully\"\n↓\nAction Info: ActionCommandInfo(\n function=\"click_input\",\n target=TargetInfo(name=\"Export\", type=\"Button\"),\n result=Result(status=SUCCESS),\n action_string=\"click_input on [12]Export\"\n)\n```\n\n!!!warning \"Error Handling\"\n Action execution uses `fail_fast=False`, allowing graceful handling of failures. Failed actions are logged but don't halt the pipeline.\n\n---\n\n## Phase 4: MEMORY_UPDATE\n\n### Strategy: `AppMemoryUpdateStrategy`\n\n**Purpose**: Record execution history in agent memory and update shared Blackboard.\n\n```python\n@depends_on(\"session_step\", \"parsed_response\")\n@provides(\"additional_memory\", \"memory_item\", \"updated_blackboard\")\nclass AppMemoryUpdateStrategy(BaseProcessingStrategy):\n \"\"\"Strategy for updating App Agent memory and blackboard.\"\"\"\n \n async def execute(self, agent, context) -> ProcessingResult:\n # 1. Create additional memory data\n additional_memory = self._create_additional_memory_data(agent, context)\n \n # 2. Create and populate memory item\n memory_item = self._create_and_populate_memory_item(\n parsed_response,\n additional_memory\n )\n \n # 3. Add memory to agent\n agent.add_memory(memory_item)\n \n # 4. Update blackboard\n self._update_blackboard(agent, save_screenshot, ...)\n \n # 5. Update structural logs\n self._update_structural_logs(context, memory_item)\n \n return ProcessingResult(success=True, data={...})\n```\n\n**Execution Flow**:\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Memory\n participant Blackboard\n participant Logs\n \n Strategy->>Strategy: Create additional_memory (step, cost, actions, results)\n Strategy->>Strategy: Create memory_item (merge response + additional data)\n \n Strategy->>Memory: agent.add_memory(memory_item)\n Memory-->>Strategy: Memory updated\n \n alt save_screenshot=True\n Strategy->>Blackboard: add_image(screenshot, metadata)\n Blackboard-->>Strategy: Image saved\n end\n \n Strategy->>Blackboard: add_trajectories(memorized_action)\n Blackboard-->>Strategy: Trajectories updated\n \n Strategy->>Logs: Update structural logs\n Logs-->>Strategy: Logs updated\n```\n\n**Memory Item Creation**:\n\n```python\ndef _create_and_populate_memory_item(\n self,\n parsed_response: AppAgentResponse,\n additional_memory: AppAgentProcessorContext,\n) -> MemoryItem:\n \"\"\"Create and populate memory item.\"\"\"\n memory_item = MemoryItem()\n \n # Add LLM response data\n if parsed_response:\n memory_item.add_values_from_dict(parsed_response.model_dump())\n \n # Add additional context data\n memory_item.add_values_from_dict(additional_memory.to_dict(selective=True))\n \n return memory_item\n```\n\n**Additional Memory Data**:\n\n```python\ndef _create_additional_memory_data(self, agent, context):\n \"\"\"Create additional memory data for App Agent.\"\"\"\n app_context = AppAgentProcessorContext()\n \n # Action information\n action_info = context.get(\"action_info\")\n if action_info:\n app_context.function_call = action_info.get_function_calls()\n app_context.action = action_info.to_list_of_dicts()\n app_context.action_success = action_info.to_list_of_dicts(success_only=True)\n app_context.action_type = [action.result.namespace for action in action_info.actions]\n app_context.action_representation = action_info.to_representation()\n \n # Step information\n app_context.session_step = context.get_global(\"SESSION_STEP\", 0)\n app_context.round_step = context.get_global(\"CURRENT_ROUND_STEP\", 0)\n app_context.round_num = context.get_global(\"CURRENT_ROUND_ID\", 0)\n app_context.agent_step = agent.step\n \n # Task information\n app_context.subtask = context.get(\"subtask\", \"\")\n app_context.request = context.get(\"request\", \"\")\n app_context.app_root = context.get(\"app_root\", \"\")\n \n # Cost and results\n app_context.cost = context.get(\"llm_cost\", 0.0)\n app_context.results = context.get(\"execution_result\", [])\n \n return app_context\n```\n\n**Blackboard Update**:\n\n```python\ndef _update_blackboard(\n self,\n agent,\n save_screenshot,\n save_reason,\n screenshot_path,\n memory_item,\n application_process_name,\n):\n \"\"\"Update agent blackboard with screenshots and actions.\"\"\"\n \n # Add action trajectories\n history_keys = ufo_config.system.history_keys\n if history_keys:\n memory_dict = memory_item.to_dict()\n memorized_action = {\n key: memory_dict.get(key)\n for key in history_keys\n if key in memory_dict\n }\n if memorized_action:\n agent.blackboard.add_trajectories(memorized_action)\n \n # Add screenshot if requested\n if save_screenshot:\n metadata = {\n \"screenshot application\": application_process_name,\n \"saving reason\": save_reason,\n }\n agent.blackboard.add_image(screenshot_path, metadata)\n```\n\n**Memory Item Example**:\n\n```python\n{\n \"observation\": \"Word document with Export button at [12]\",\n \"thought\": \"Click Export to extract table\",\n \"control_label\": \"12\",\n \"function_call\": [\"click_input\"],\n \"action\": [{\"function\": \"click_input\", \"target\": {...}, \"result\": {...}}],\n \"action_success\": [{\"action_string\": \"click_input on [12]Export\", ...}],\n \"status\": \"SCREENSHOT\",\n \"plan\": [\"Click Export\", \"Select CSV\", \"Save file\"],\n \"cost\": 0.0023,\n \"session_step\": 5,\n \"round_step\": 2,\n \"subtask\": \"Extract table from Word document\",\n}\n```\n\n!!!info \"Selective Memory\"\n The `history_keys` configuration controls which fields are added to Blackboard trajectories. This prevents information overload while maintaining essential context for cross-agent communication.\n\n---\n\n## Complete Execution Example\n\n### Single Action Cycle\n\n```mermaid\nsequenceDiagram\n participant AppAgent\n participant DC as DATA_COLLECTION\n participant LLM as LLM_INTERACTION\n participant AE as ACTION_EXECUTION\n participant MU as MEMORY_UPDATE\n participant Application\n \n rect rgb(230, 240, 255)\n Note over AppAgent, DC: Phase 1: Data Collection\n AppAgent->>DC: Start processing\n DC->>Application: capture_window_screenshot()\n Application-->>DC: clean_screenshot_url\n DC->>Application: get_app_window_controls_target_info()\n Application-->>DC: 50 controls detected\n DC->>DC: Annotate screenshot [1] to [50]\n DC-->>AppAgent: Screenshots + Controls ready\n end\n \n rect rgb(255, 250, 230)\n Note over AppAgent, LLM: Phase 2: LLM Interaction\n AppAgent->>LLM: Process with controls + images\n LLM->>LLM: Build prompt (RAG + history)\n LLM->>LLM: Get LLM response\n LLM->>LLM: Parse JSON response\n LLM-->>AppAgent: Action: click_input([12], left)\n end\n \n rect rgb(230, 255, 240)\n Note over AppAgent, AE: Phase 3: Action Execution\n AppAgent->>AE: Execute action\n AE->>Application: click_input(id=\"12\")\n Application-->>AE: SUCCESS: Clicked Export button\n AE->>AE: Create action_info\n AE-->>AppAgent: Action completed\n end\n \n rect rgb(255, 240, 245)\n Note over AppAgent, MU: Phase 4: Memory Update\n AppAgent->>MU: Update memory\n MU->>MU: Create memory_item\n MU->>MU: Add to agent.memory\n MU->>MU: Update blackboard\n MU-->>AppAgent: Memory updated\n end\n```\n\n---\n\n## Error Handling\n\n### Fail-Fast vs Graceful\n\n```python\n# DATA_COLLECTION: fail_fast=True\n# Critical failure stops pipeline immediately\ntry:\n result = await screenshot_strategy.execute(agent, context)\nexcept Exception as e:\n # Propagate immediately - cannot proceed without screenshots\n raise ProcessingError(f\"Data collection failed: {e}\")\n\n# ACTION_EXECUTION: fail_fast=False\n# Failures are logged but don't stop pipeline\ntry:\n result = await action_strategy.execute(agent, context)\nexcept Exception as e:\n # Log error, return partial result, continue to memory phase\n logger.error(f\"Action execution failed: {e}\")\n return ProcessingResult(success=False, error=str(e), data={})\n```\n\n### Retry Mechanisms\n\n**LLM Interaction Retry**:\n\n```python\nasync def _get_llm_response(self, agent, prompt_message):\n \"\"\"Get response from LLM with retry logic.\"\"\"\n max_retries = ufo_config.system.json_parsing_retry # Default: 3\n \n for retry_count in range(max_retries):\n try:\n # Run LLM call in thread executor to avoid blocking\n loop = asyncio.get_event_loop()\n response_text, cost = await loop.run_in_executor(\n None,\n agent.get_response,\n prompt_message,\n AgentType.APP,\n True, # use_backup_engine\n )\n \n # Validate response can be parsed\n agent.response_to_dict(response_text)\n return response_text, cost\n \n except Exception as e:\n if retry_count < max_retries - 1:\n logger.warning(f\"LLM retry {retry_count + 1}/{max_retries}: {e}\")\n else:\n raise\n```\n\n---\n\n## Performance Optimization\n\n### Composed Strategy Benefits\n\n```python\n# Sequential execution with shared context\nself.strategies[ProcessingPhase.DATA_COLLECTION] = ComposedStrategy(\n strategies=[\n AppScreenshotCaptureStrategy(), # Provides: screenshots, window_info\n AppControlInfoStrategy(), # Depends on: screenshots, window_info\n ],\n name=\"AppDataCollectionStrategy\",\n fail_fast=True,\n)\n```\n\n**Benefits**:\n\n- **Context Sharing**: Screenshot output immediately available to Control Info strategy\n- **Atomic Failure**: If screenshot fails, control detection is skipped\n- **Performance**: Avoids redundant window queries\n\n### Dependency Injection\n\n```python\n@depends_on(\"clean_screenshot_path\", \"application_window_info\")\n@provides(\"control_info\", \"annotation_dict\", \"annotated_screenshot_url\")\nclass AppControlInfoStrategy(BaseProcessingStrategy):\n # Automatically receives dependencies from previous strategies\n pass\n```\n\n**Benefits**:\n\n- Type-safe dependency declaration\n- Automatic data flow between strategies\n- Easy to add new strategies without refactoring\n\n---\n\n## Related Documentation\n\n**Architecture:**\n\n- **[AppAgent Overview](overview.md)**: High-level architecture and responsibilities\n- **[State Machine](state.md)**: State machine that invokes this pipeline\n- **[Command System](commands.md)**: MCP command details\n- **[HostAgent Processing Strategy](../host_agent/strategy.md)**: Parent agent pipeline\n\n**Core Features:**\n\n- **[Hybrid Actions](../core_features/hybrid_actions.md)**: MCP command system\n- **[Control Detection](../core_features/control_detection/overview.md)**: UIA + OmniParser backends\n- **[Knowledge Substrate](../core_features/knowledge_substrate/overview.md)**: RAG system integration\n\n**Design Patterns:**\n\n- **[Processor Framework](../../infrastructure/agents/design/processor.md)**: ProcessorTemplate architecture\n- **[Strategy Pattern](../../infrastructure/agents/design/processor.md)**: BaseProcessingStrategy design\n\n---\n\n## Summary\n\n**AppAgent Processing Pipeline Key Features:**\n\n✅ **4-Phase Pipeline**: DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE \n✅ **Composed Strategy**: Phase 1 combines Screenshot + Control Info strategies \n✅ **Multi-Backend Control Detection**: UIA + OmniParser with hybrid merging \n✅ **Knowledge-Enhanced Prompting**: RAG integration from docs, demos, and search \n✅ **Retry Logic**: Automatic LLM retry with configurable attempts \n✅ **Memory & Blackboard**: Comprehensive execution tracking and inter-agent communication \n✅ **Graceful Error Handling**: fail_fast configuration per phase\n\n**Next Steps:**\n\n1. **Study Commands**: Read [Command System](commands.md) for MCP command details\n2. **Explore States**: Review [State Machine](state.md) for FSM that invokes pipeline\n3. **Learn Patterns**: Check [Processor Framework](../../infrastructure/agents/design/processor.md) for architecture details\n"
},
{
"path": "documents/docs/ufo2/as_galaxy_device.md",
"content": "# UFO² as UFO³ Galaxy Device\n\nIntegrate **UFO² (Windows Desktop Automation Agent)** into the **UFO³ Galaxy framework** as a managed sub-agent device. This enables Galaxy to orchestrate complex cross-platform workflows combining Windows desktop automation with Linux server operations and other heterogeneous devices.\n\n## Overview\n\nUFO² can function as a **device agent** within the UFO³ Galaxy multi-tier orchestration framework. When configured as a Galaxy device, UFO² operates in **server-client mode**, allowing the Galaxy ConstellationAgent to:\n\n- Dispatch Windows automation subtasks to UFO² devices\n- Coordinate cross-platform workflows (Windows desktop + Linux servers)\n- Leverage UFO²'s HostAgent and AppAgent capabilities at scale\n- Manage multiple Windows devices from a unified control plane\n- Dynamically select devices based on capabilities and installed applications\n\nUFO² integration follows the **server-client architecture** pattern where the UFO² Server manages task orchestration and state machines, the UFO² Client executes Windows automation commands via MCP tools, and the Galaxy ConstellationAgent acts as the top-level orchestrator. Communication is enabled through the Agent Interaction Protocol (AIP). For detailed architecture information, see [Server-Client Architecture](../infrastructure/agents/server_client_architecture.md).\n\n## Galaxy Integration Architecture\n\n```mermaid\ngraph TB\n User[User Request]\n Galaxy[Galaxy ConstellationAgent Top-Level Orchestrator]\n \n subgraph \"Device Pool\"\n subgraph \"Windows Devices (UFO²)\"\n Win1[UFO² Device 1 Office Desktop]\n Win2[UFO² Device 2 Dev Workstation]\n Win3[UFO² Device 3 Test Machine]\n end\n \n subgraph \"Linux Devices\"\n Linux1[Linux Agent 1 Web Server]\n Linux2[Linux Agent 2 Database Server]\n end\n \n subgraph \"Other Devices\"\n Mobile1[Mobile Device]\n Cloud1[Cloud Service]\n end\n end\n \n User -->|Complex Cross-Platform Task| Galaxy\n \n Galaxy -->|Windows Automation Subtask| Win1\n Galaxy -->|Desktop Application Task| Win2\n Galaxy -->|Testing Task| Win3\n \n Galaxy -->|Server Management Task| Linux1\n Galaxy -->|Database Query Task| Linux2\n \n Galaxy -->|Mobile Automation| Mobile1\n Galaxy -->|API Integration| Cloud1\n \n style Galaxy fill:#ffe1e1\n style Win1 fill:#e1f5ff\n style Win2 fill:#e1f5ff\n style Win3 fill:#e1f5ff\n style Linux1 fill:#f0ffe1\n style Linux2 fill:#f0ffe1\n```\n\n**Example Multi-Device Workflow:**\n\n> **User Request:** \"Generate a sales report from the database, create an Excel dashboard, and email it to the team\"\n\n**Galaxy orchestrates:**\n\n1. **Linux DB Server**: Extract sales data from PostgreSQL → CSV export\n2. **UFO² Desktop**: Open Excel, import CSV, create visualizations and pivot tables\n3. **UFO² Desktop**: Open Outlook, compose email with Excel attachment\n4. **UFO² Desktop**: Send email to distribution list\n\n## Prerequisites\n\nBefore configuring UFO² as a Galaxy device, ensure you have:\n\n| Component | Requirement | Verification |\n|-----------|-------------|--------------|\n| **UFO Repository** | Cloned and up-to-date | `git pull origin main` |\n| **Python** | 3.10+ installed | `python --version` |\n| **Dependencies** | All packages installed | `pip install -r requirements.txt` |\n| **LLM Configuration** | API keys configured | Check `config/ufo/agents.yaml` |\n| **Network** | Server-client connectivity | `ping ` |\n| **Windows Machine** | UFO² will run here | Windows 10/11 |\n\n### Configure Agent Configuration\n\n**Before proceeding with Galaxy integration**, you must configure your agent settings in `config/ufo/agents.yaml`:\n\n1. Copy the template file:\n ```powershell\n Copy-Item config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\n ```\n\n2. Configure your LLM provider (OpenAI, Azure OpenAI, etc.) and add API keys\n\nWithout proper agent configuration, UFO² cannot function as a Galaxy device. See [Agents Configuration Guide](../configuration/system/agents_config.md) for detailed setup instructions.\n\n## Server-Client Mode Setup\n\nUFO² **must** operate in **server-client mode** when integrated into Galaxy. This architecture separates orchestration (server) from execution (client), enabling Galaxy to manage multiple UFO² devices efficiently. Unlike standalone UFO² usage (local mode), Galaxy integration requires running UFO² in distributed server-client mode to ensure Galaxy can communicate with UFO² via Agent Interaction Protocol (AIP), multiple UFO² clients can be managed by a single server, task state is managed server-side for reliability, and clients remain stateless execution endpoints.\n\n## Step 1: Start UFO² Server\n\nThe **UFO² Server** handles task orchestration, state management, and LLM-driven decision-making. It communicates with Galaxy and dispatches commands to UFO² clients.\n\n### Basic Server Startup\n\nLaunch UFO² Server on the machine that will host the server (can be any Windows/Linux machine):\n\n```powershell\npython -m ufo.server.app --port 5000\n```\n\n**Expected Output:**\n\n```console\n2025-11-06 10:30:22 - ufo.server.app - INFO - Starting UFO Server on 0.0.0.0:5000\nINFO: Started server process [12345]\nINFO: Waiting for application startup.\nINFO: Application startup complete.\nINFO: Uvicorn running on http://0.0.0.0:5000 (Press CTRL+C to quit)\n```\n\nOnce you see \"Uvicorn running\", the server is ready at `ws://0.0.0.0:5000/ws`.\n\n### Server Configuration Options\n\n| Argument | Default | Description | Example |\n|----------|---------|-------------|---------|\n| `--port` | `5000` | Server listening port | `--port 5000` |\n| `--host` | `0.0.0.0` | Bind address (0.0.0.0 = all interfaces) | `--host 192.168.1.100` |\n| `--log-level` | `WARNING` | Logging verbosity | `--log-level DEBUG` |\n| `--local` | `False` | Run server in local mode | `--local` |\n\n**Examples:**\n\nSpecific port:\n```powershell\npython -m ufo.server.app --port 5000\n```\n\nSpecific IP binding:\n```powershell\npython -m ufo.server.app --host 192.168.1.100 --port 5000\n```\n\nDebug mode:\n```powershell\npython -m ufo.server.app --port 5000 --log-level DEBUG\n```\n\n### Verify Server Health\n\n```powershell\n# Test server health endpoint\ncurl http://localhost:5000/api/health\n```\n\n**Expected Response:**\n\n```json\n{\n \"status\": \"healthy\",\n \"online_clients\": []\n}\n```\n\n## Step 2: Start UFO² Client (Windows Machine)\n\nThe **UFO² Client** runs on the Windows machine where you want to perform desktop automation. It connects to the UFO² server via WebSocket and executes automation commands through MCP tools.\n\n### Basic Client Startup\n\nConnect UFO² Client to Server on the **Windows machine** where you want to run desktop automation:\n\n```powershell\npython -m ufo.client.client `\n --ws `\n --ws-server ws://192.168.1.100:5000/ws `\n --client-id ufo2_desktop_1 `\n --platform windows\n```\n\n**Note:** In PowerShell, use backtick `` ` `` for line continuation. In Command Prompt, use `^`.\n\n### Client Parameters Explained\n\n| Parameter | Required | Description | Example |\n|-----------|----------|-------------|---------|\n| `--ws` | ✅ Yes | Enable WebSocket mode | `--ws` |\n| `--ws-server` | ✅ Yes | Server WebSocket URL | `ws://192.168.1.100:5000/ws` |\n| `--client-id` | ✅ Yes | **Unique** device identifier | `ufo2_desktop_1` |\n| `--platform` | ✅ Yes | Platform type (must be `windows` for UFO²) | `--platform windows` |\n\n**Important:**\n- `--client-id` must be globally unique - No two devices can share the same ID\n- `--platform windows` is mandatory - Without this flag, UFO² won't work correctly\n- Server address must be correct - Replace `192.168.1.100:5000` with your actual server IP and port\n\n### Understanding the WebSocket URL\n\nThe `--ws-server` parameter format is:\n\n```\nws://:/ws\n```\n\nExamples:\n\n| Scenario | WebSocket URL | Description |\n|----------|---------------|-------------|\n| **Localhost** | `ws://localhost:5000/ws` | Server and client on same machine |\n| **Same Network** | `ws://192.168.1.100:5000/ws` | Server on local network |\n| **Remote Server** | `ws://203.0.113.50:5000/ws` | Server on internet (public IP) |\n\n### Connection Success Indicators\n\n**Client Logs:**\n\n```log\nINFO - Platform detected/specified: windows\nINFO - UFO Client initialized for platform: windows\nINFO - [WS] Connecting to ws://192.168.1.100:5000/ws (attempt 1/5)\nINFO - [WS] [AIP] Successfully registered as ufo2_desktop_1\nINFO - [WS] Heartbeat loop started (interval: 30s)\n```\n\n**Server Logs:**\n\n```log\nINFO - [WS] ✅ Registered device client: ufo2_desktop_1\nINFO - [WS] Device ufo2_desktop_1 platform: windows\n```\n\nWhen you see \"Successfully registered\", the UFO² client is connected and ready to receive tasks.\n\n### Verify Connection\n\n```powershell\n# Check connected clients on server\ncurl http://192.168.1.100:5000/api/clients\n```\n\n**Expected Response:**\n\n```json\n{\n \"clients\": [\n {\n \"client_id\": \"ufo2_desktop_1\",\n \"type\": \"device\",\n \"platform\": \"windows\",\n \"connected_at\": 1730899822.0,\n \"uptime_seconds\": 45\n }\n ]\n}\n```\n\n## Step 3: Configure MCP Services\n\nUFO² relies on **MCP (Model Context Protocol) servers** to provide Windows automation capabilities. Unlike Linux agents that may require separate HTTP MCP servers, UFO² MCP servers are primarily **local** and start automatically with the client.\n\nUFO² uses **local MCP servers** that run in-process with the client:\n\n- **UI Automation MCP**: Click, type, screenshot, control detection\n- **File Operations MCP**: Read, write, copy, delete files\n- **Application Control MCP**: Launch apps, switch windows, close processes\n\nThese are **automatically initialized** when the UFO² client starts.\n\n### Default MCP Configuration\n\nBy default, UFO² client automatically starts all necessary **local MCP servers**. No additional configuration is required for standard Windows automation.\n\nWhen you start the UFO² client, it automatically initializes UI automation tools, registers file operation handlers, configures application control interfaces, and sets up screenshot and OCR capabilities.\n\n### Optional: HTTP MCP Server (Advanced)\n\nFor specialized scenarios requiring **remote MCP access** (e.g., hardware automation via external tools), you can optionally start HTTP-based MCP servers. However, note that there is no `windows_mcp_server.py` in the codebase. Available HTTP MCP servers are:\n\n- `hardware_mcp_server.py` - For hardware-level operations\n- `linux_mcp_server.py` - For Linux-specific operations\n\nStart an HTTP MCP server if needed:\n\n```powershell\npython -m ufo.client.mcp.http_servers.hardware_mcp_server\n```\n\n**Note:** For standard Galaxy integration with UFO², local MCP servers are sufficient and HTTP MCP servers are not required.\n\n## Step 4: Configure as Galaxy Device\n\nTo integrate UFO² into the Galaxy framework, register it in the Galaxy device configuration file.\n\n### Device Configuration File\n\nThe Galaxy device pool is configured in `config/galaxy/devices.yaml`.\n\n### Add UFO² Device Configuration\n\nEdit `config/galaxy/devices.yaml` and add your UFO² device(s) under the `devices` section:\n\n```yaml\ndevices:\n - device_id: \"ufo2_desktop_1\"\n server_url: \"ws://192.168.1.100:5000/ws\"\n os: \"windows\"\n capabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"web_browsing\"\n - \"email\"\n - \"file_management\"\n metadata:\n os: \"windows\"\n version: \"11\"\n performance: \"high\"\n installed_apps:\n - \"Microsoft Excel\"\n - \"Microsoft Word\"\n - \"Microsoft PowerPoint\"\n - \"Microsoft Outlook\"\n - \"Google Chrome\"\n - \"Adobe Acrobat\"\n description: \"Primary office workstation for document automation\"\n auto_connect: true\n max_retries: 5\n```\n\n### Configuration Fields Explained\n\n| Field | Required | Type | Description | Example |\n|-------|----------|------|-------------|---------|\n| `device_id` | ✅ Yes | string | **Must match client `--client-id`** | `\"ufo2_desktop_1\"` |\n| `server_url` | ✅ Yes | string | **Must match server WebSocket URL** | `\"ws://192.168.1.100:5000/ws\"` |\n| `os` | ✅ Yes | string | Operating system | `\"windows\"` |\n| `capabilities` | ❌ Optional | list | Device capabilities (for task routing) | `[\"desktop_automation\", \"office\"]` |\n| `metadata` | ❌ Optional | dict | Custom metadata for task context | See below |\n| `auto_connect` | ❌ Optional | boolean | Auto-connect on Galaxy startup | `true` |\n| `max_retries` | ❌ Optional | integer | Connection retry attempts | `5` |\n\n### Capabilities-Based Task Routing\n\nGalaxy uses the `capabilities` field to intelligently route subtasks to appropriate UFO² devices. Define capabilities based on application categories (e.g., `\"office_applications\"`, `\"web_browsing\"`), task types (e.g., `\"desktop_automation\"`, `\"data_entry\"`), specific software (e.g., `\"excel\"`, `\"outlook\"`), and user workflows (e.g., `\"email\"`, `\"reporting\"`).\n\n**Example capability configurations:**\n\n**Office Workstation:**\n```yaml\ncapabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"excel\"\n - \"word\"\n - \"powerpoint\"\n - \"outlook\"\n - \"email\"\n - \"reporting\"\n```\n\n**Web Development Machine:**\n```yaml\ncapabilities:\n - \"desktop_automation\"\n - \"web_browsing\"\n - \"chrome\"\n - \"visual_studio_code\"\n - \"git\"\n - \"development\"\n```\n\n**Testing Workstation:**\n```yaml\ncapabilities:\n - \"desktop_automation\"\n - \"ui_testing\"\n - \"web_browsing\"\n - \"screenshot_comparison\"\n - \"quality_assurance\"\n```\n\n**Media Production:**\n```yaml\ncapabilities:\n - \"desktop_automation\"\n - \"media_editing\"\n - \"photoshop\"\n - \"premiere\"\n - \"video_processing\"\n - \"image_manipulation\"\n```\n\nThe `metadata` field provides **contextual information** that the LLM can use when generating automation commands.\n\n**Metadata Examples:**\n\n**Office Workstation Metadata:**\n```yaml\nmetadata:\n os: \"windows\"\n version: \"11\"\n performance: \"high\"\n installed_apps:\n - \"Microsoft Excel\"\n - \"Microsoft Word\"\n - \"Microsoft Outlook\"\n - \"Adobe Acrobat Reader\"\n default_paths:\n documents: \"C:\\\\Users\\\\user\\\\Documents\"\n downloads: \"C:\\\\Users\\\\user\\\\Downloads\"\n desktop: \"C:\\\\Users\\\\user\\\\Desktop\"\n email_account: \"user@company.com\"\n description: \"Primary office workstation\"\n```\n\n**Development Workstation Metadata:**\n```yaml\nmetadata:\n os: \"windows\"\n version: \"11\"\n performance: \"high\"\n installed_apps:\n - \"Visual Studio Code\"\n - \"Google Chrome\"\n - \"Git\"\n - \"Node.js\"\n - \"Python\"\n default_paths:\n projects: \"C:\\\\Users\\\\dev\\\\Projects\"\n repos: \"C:\\\\Users\\\\dev\\\\Repos\"\n git_username: \"developer\"\n description: \"Development environment\"\n```\n\n**Testing Workstation Metadata:**\n```yaml\nmetadata:\n os: \"windows\"\n version: \"10\"\n performance: \"medium\"\n installed_apps:\n - \"Google Chrome\"\n - \"Microsoft Edge\"\n - \"Firefox\"\n - \"Selenium\"\n test_data_path: \"C:\\\\TestData\"\n screenshot_path: \"C:\\\\Screenshots\"\n description: \"Automated testing environment\"\n```\n\n**How Metadata is Used:**\n\nThe LLM receives metadata in the system prompt, enabling context-aware automation:\n\n```\nSystem Context:\n- Device: ufo2_desktop_1\n- OS: Windows 11\n- Installed Apps: Microsoft Excel, Microsoft Word, Microsoft Outlook\n- Documents Path: C:\\Users\\user\\Documents\n\nUser Request: \"Create a new Excel spreadsheet and save it as Q4_Report.xlsx\"\n\nUFO² Output: \n1. Launch Microsoft Excel\n2. Create new workbook\n3. Save as C:\\Users\\user\\Documents\\Q4_Report.xlsx\n```\n\n## Step 5: Multiple UFO² Devices Configuration\n\nGalaxy can manage **multiple UFO² devices** simultaneously, enabling parallel Windows automation across different machines.\n\n**Multi-Device Galaxy Configuration Example:**\n\n```yaml\ndevices:\n # UFO² Office Desktop 1\n - device_id: \"ufo2_office_1\"\n server_url: \"ws://192.168.1.100:5000/ws\"\n os: \"windows\"\n capabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"excel\"\n - \"word\"\n - \"outlook\"\n - \"email\"\n metadata:\n os: \"windows\"\n version: \"11\"\n installed_apps: [\"Microsoft Excel\", \"Microsoft Word\", \"Microsoft Outlook\"]\n description: \"Primary office desktop\"\n auto_connect: true\n max_retries: 5\n \n # UFO² Office Desktop 2\n - device_id: \"ufo2_office_2\"\n server_url: \"ws://192.168.1.101:5001/ws\"\n os: \"windows\"\n capabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"excel\"\n - \"powerpoint\"\n - \"web_browsing\"\n metadata:\n os: \"windows\"\n version: \"11\"\n installed_apps: [\"Microsoft Excel\", \"Microsoft PowerPoint\", \"Google Chrome\"]\n description: \"Secondary office desktop\"\n auto_connect: true\n max_retries: 5\n \n # UFO² Development Workstation\n - device_id: \"ufo2_dev_1\"\n server_url: \"ws://192.168.1.102:5002/ws\"\n os: \"windows\"\n capabilities:\n - \"desktop_automation\"\n - \"development\"\n - \"web_browsing\"\n - \"code_editing\"\n metadata:\n os: \"windows\"\n version: \"11\"\n installed_apps: [\"Visual Studio Code\", \"Google Chrome\", \"Git\"]\n description: \"Development workstation\"\n auto_connect: true\n max_retries: 5\n \n # Linux Database Server (for cross-platform workflows)\n - device_id: \"linux_db_server\"\n server_url: \"ws://192.168.1.200:5010/ws\"\n os: \"linux\"\n capabilities:\n - \"database_server\"\n - \"postgresql\"\n - \"data_export\"\n metadata:\n os: \"linux\"\n logs_file_path: \"/var/log/postgresql/postgresql.log\"\n description: \"Production database server\"\n auto_connect: true\n max_retries: 5\n```\n\n## Step 6: Launch Galaxy with UFO² Devices\n\nOnce all components are configured, launch Galaxy to begin orchestrating multi-device workflows.\n\n### Prerequisites Checklist\n\nEnsure all components are running **before** starting Galaxy:\n\n1. ✅ **UFO² Server(s)** running on configured ports\n2. ✅ **UFO² Client(s)** connected to their respective servers\n3. ✅ **MCP Services** initialized (automatic with UFO² client)\n4. ✅ **LLM configured** in `config/ufo/agents.yaml`\n5. ✅ **Network connectivity** between all components\n\n### Launch Sequence\n\n**Step 1: Start all UFO² Servers**\n\n```powershell\n# On first Windows machine (192.168.1.100)\npython -m ufo.server.app --port 5000\n\n# On second Windows machine (192.168.1.101)\npython -m ufo.server.app --port 5001\n\n# On third Windows machine (192.168.1.102)\npython -m ufo.server.app --port 5002\n```\n\n**Step 2: Start all UFO² Clients**\n\n```powershell\n# On first Windows desktop\npython -m ufo.client.client `\n --ws `\n --ws-server ws://192.168.1.100:5000/ws `\n --client-id ufo2_office_1 `\n --platform windows\n\n# On second Windows desktop\npython -m ufo.client.client `\n --ws `\n --ws-server ws://192.168.1.101:5001/ws `\n --client-id ufo2_office_2 `\n --platform windows\n\n# On development workstation\npython -m ufo.client.client `\n --ws `\n --ws-server ws://192.168.1.102:5002/ws `\n --client-id ufo2_dev_1 `\n --platform windows\n```\n\n**Step 3: Launch Galaxy**\n\n```powershell\n# On your control machine (interactive mode)\npython -m galaxy --interactive\n```\n\n**Or launch with a specific request:**\n\n```powershell\npython -m galaxy \"Your task description here\"\n```\n\nGalaxy will automatically connect to all configured UFO² devices (based on `config/galaxy/devices.yaml`) and display the orchestration interface.\n\n## Example Multi-Device Workflows\n\n### Workflow 1: Cross-Platform Report Generation\n\n**User Request:**\n> \"Generate a weekly sales report: extract data from PostgreSQL, create Excel dashboard, and email to management\"\n\n**Galaxy Orchestration:**\n\n```mermaid\nsequenceDiagram\n participant User\n participant Galaxy\n participant LinuxDB as Linux DB Server\n participant UFO2 as UFO² Desktop\n \n User->>Galaxy: Request sales report\n Galaxy->>Galaxy: Decompose task\n \n Note over Galaxy,LinuxDB: Subtask 1: Extract data\n Galaxy->>LinuxDB: \"Export sales data from PostgreSQL to CSV\"\n LinuxDB->>LinuxDB: Execute SQL query\n LinuxDB->>LinuxDB: Generate CSV file\n LinuxDB-->>Galaxy: CSV file location\n \n Note over Galaxy,UFO2: Subtask 2: Create Excel report\n Galaxy->>UFO2: \"Create Excel dashboard from CSV\"\n UFO2->>UFO2: Open Excel\n UFO2->>UFO2: Import CSV data\n UFO2->>UFO2: Create pivot tables\n UFO2->>UFO2: Add charts and formatting\n UFO2-->>Galaxy: Excel file created\n \n Note over Galaxy,UFO2: Subtask 3: Send email\n Galaxy->>UFO2: \"Email report to management\"\n UFO2->>UFO2: Open Outlook\n UFO2->>UFO2: Compose email with attachment\n UFO2->>UFO2: Send email\n UFO2-->>Galaxy: Email sent\n \n Galaxy-->>User: Task completed\n```\n\n### Workflow 2: Parallel Document Processing\n\n**User Request:**\n> \"Process all invoices in the shared folder: convert PDFs to Excel, categorize by vendor, and summarize totals\"\n\n**Galaxy Orchestration:**\n\n1. **UFO² Desktop 1**: Process invoices A-M (parallel batch 1)\n2. **UFO² Desktop 2**: Process invoices N-Z (parallel batch 2)\n3. **UFO² Desktop 1**: Consolidate results into master Excel file\n4. **UFO² Desktop 1**: Generate summary report\n5. **UFO² Desktop 1**: Send notification email\n\n### Workflow 3: Development Workflow Automation\n\n**User Request:**\n> \"Pull latest code, run tests, and create deployment package\"\n\n**Galaxy Orchestration:**\n\n1. **UFO² Dev Workstation**: Open VS Code, pull from Git repository\n2. **UFO² Dev Workstation**: Run automated tests, capture results\n3. **Linux Build Server**: Build deployment package\n4. **UFO² Dev Workstation**: Open browser, upload to staging server\n5. **UFO² Desktop**: Send deployment notification email\n\n---\n\n## Task Assignment Behavior\n\n### How Galaxy Routes Tasks to UFO² Devices\n\nGalaxy's ConstellationAgent uses several factors to select the appropriate UFO² device for each subtask:\n\n| Factor | Description | Example |\n|--------|-------------|---------|\n| **Capabilities** | Match subtask requirements to device capabilities | `\"excel\"` → Office workstation |\n| **OS Requirement** | Platform-specific tasks routed to correct OS | Windows automation → UFO² devices |\n| **Metadata Context** | Use device-specific apps and configurations | Email task → device with Outlook |\n| **Device Status** | Only assign to online, healthy devices | Skip offline or failing devices |\n| **Load Balancing** | Distribute tasks across similar devices | Round-robin across office desktops |\n\n### Example Task Decomposition\n\n**User Request:**\n> \"Prepare quarterly financial reports and distribute to stakeholders\"\n\n**Galaxy Decomposition:**\n\n```yaml\nTask 1:\n Description: \"Extract financial data from database\"\n Target: linux_db_server\n Reason: Has \"database_server\" capability\n \nTask 2:\n Description: \"Create Excel financial dashboard\"\n Target: ufo2_office_1\n Reason: Has \"excel\" capability, device is idle\n \nTask 3:\n Description: \"Generate PowerPoint presentation\"\n Target: ufo2_office_2\n Reason: Has \"powerpoint\" capability\n \nTask 4:\n Description: \"Email reports to stakeholders\"\n Target: ufo2_office_1\n Reason: Has \"outlook\" and \"email\" capabilities\n```\n\n## Critical Configuration Requirements\n\n!!!danger \"Configuration Validation Checklist\"\n Ensure these match **exactly** or Galaxy cannot control the UFO² device:\n \n **Device ID Match:**\n - In `devices.yaml`: `device_id: \"ufo2_desktop_1\"`\n - In client command: `--client-id ufo2_desktop_1`\n \n **Server URL Match:**\n - In `devices.yaml`: `server_url: \"ws://192.168.1.100:5000/ws\"`\n - In client command: `--ws-server ws://192.168.1.100:5000/ws`\n \n **Platform Specification:**\n - Must include `--platform windows` for UFO² devices\n\n## Monitoring & Debugging\n\n### Verify Device Registration\n\nCheck if clients are connected to UFO² server:\n\n```powershell\ncurl http://192.168.1.100:5000/api/clients\n```\n\n**Expected response:**\n\n```json\n{\n \"online_clients\": [\n {\n \"client_id\": \"ufo2_office_1\",\n \"type\": \"device\",\n \"platform\": \"windows\",\n \"connected_at\": 1730899822.0,\n \"uptime_seconds\": 45\n },\n {\n \"client_id\": \"ufo2_office_2\",\n \"type\": \"device\",\n \"platform\": \"windows\",\n \"connected_at\": 1730899850.0,\n \"uptime_seconds\": 17\n }\n ]\n}\n```\n\n### View Task Assignments\n\nGalaxy logs show task routing decisions:\n\n```log\nINFO - [Galaxy] Task decomposition: 3 subtasks created\nINFO - [Galaxy] Subtask 1 → linux_db_server (capability match: database_server)\nINFO - [Galaxy] Subtask 2 → ufo2_office_1 (capability match: excel)\nINFO - [Galaxy] Subtask 3 → ufo2_office_1 (capability match: email)\n```\n\n### Troubleshooting Device Connection\n\n**Issue**: UFO² device not appearing in Galaxy device pool\n\n**Diagnosis:**\n\n1. Check if client is connected to server:\n ```powershell\n curl http://192.168.1.100:5000/api/clients\n ```\n\n2. Verify `devices.yaml` configuration matches client parameters\n\n3. Check Galaxy logs for connection errors\n\n4. Ensure `auto_connect: true` in `devices.yaml`\n\n5. Verify UFO² server is running and accessible\n\n## Common Issues & Troubleshooting\n\n### Issue 1: UFO² Client Cannot Connect to Server\n\n!!!bug \"Error: Connection Refused\"\n **Symptoms:**\n ```log\n ERROR - [WS] Failed to connect to ws://192.168.1.100:5000/ws\n Connection refused\n ```\n \n **Diagnosis Checklist:**\n \n - [ ] Is the UFO² server running? (`curl http://192.168.1.100:5000/api/health`)\n - [ ] Is the port correct? (Check server startup logs)\n - [ ] Can client reach server IP? (`ping 192.168.1.100`)\n - [ ] Is Windows Firewall blocking port 5000?\n - [ ] Is the WebSocket URL correct? (should start with `ws://`)\n \n **Solutions:**\n \n **Verify Server:**\n ```powershell\n # On server machine\n curl http://localhost:5000/api/health\n \n # From client machine\n curl http://192.168.1.100:5000/api/health\n ```\n \n **Check Network:**\n ```powershell\n # Test connectivity\n ping 192.168.1.100\n \n # Test port accessibility (requires telnet client)\n Test-NetConnection -ComputerName 192.168.1.100 -Port 5000\n ```\n \n **Check Windows Firewall:**\n ```powershell\n # Allow port through firewall\n New-NetFirewallRule -DisplayName \"UFO Server\" `\n -Direction Inbound `\n -LocalPort 5000 `\n -Protocol TCP `\n -Action Allow\n ```\n\n### Issue 2: Missing `--platform windows` Flag\n\n!!!bug \"Error: Incorrect Agent Type\"\n **Symptoms:**\n - Client connects but cannot execute Windows automation\n - Server logs show wrong platform type\n - Tasks fail with \"unsupported operation\" errors\n \n **Cause:**\n Forgot to add `--platform windows` flag when starting the client.\n \n **Solution:**\n ```powershell\n # Wrong (missing platform)\n python -m ufo.client.client --ws --client-id ufo2_desktop_1\n \n # Correct\n python -m ufo.client.client `\n --ws `\n --client-id ufo2_desktop_1 `\n --platform windows\n ```\n\n### Issue 3: Duplicate Client ID\n\n!!!bug \"Error: Registration Failed\"\n **Symptoms:**\n ```log\n ERROR - [WS] Registration failed: client_id already exists\n ERROR - Another device is using ID 'ufo2_desktop_1'\n ```\n \n **Cause:**\n Multiple UFO² clients trying to use the same `client_id`.\n \n **Solutions:**\n \n 1. **Use unique client IDs:**\n ```powershell\n # Device 1\n --client-id ufo2_desktop_1\n \n # Device 2\n --client-id ufo2_desktop_2\n \n # Device 3\n --client-id ufo2_dev_1\n ```\n \n 2. **Check currently connected clients:**\n ```powershell\n curl http://192.168.1.100:5000/api/clients\n ```\n\n### Issue 4: Galaxy Cannot Find UFO² Device\n\n!!!bug \"Error: Device Not Configured\"\n **Symptoms:**\n ```log\n ERROR - Device 'ufo2_desktop_1' not found in configuration\n WARNING - Cannot dispatch task to unknown device\n ```\n \n **Cause:**\n Mismatch between `devices.yaml` configuration and actual client setup.\n \n **Diagnosis:**\n \n Check that these match **exactly**:\n \n | Location | Field | Example |\n |----------|-------|---------|\n | `devices.yaml` | `device_id` | `\"ufo2_desktop_1\"` |\n | Client command | `--client-id` | `ufo2_desktop_1` |\n | `devices.yaml` | `server_url` | `\"ws://192.168.1.100:5000/ws\"` |\n | Client command | `--ws-server` | `ws://192.168.1.100:5000/ws` |\n \n **Solution:**\n \n Update `devices.yaml` to match your client configuration, or vice versa.\n\n### Issue 5: MCP Tools Not Available\n\n!!!bug \"Error: Tool Execution Failed\"\n **Symptoms:**\n ```log\n ERROR - MCP tool 'click' not found\n ERROR - Cannot execute Windows automation command\n ```\n \n **Diagnosis:**\n \n - [ ] Is UFO² client running properly?\n - [ ] Are local MCP servers initialized?\n - [ ] Check client startup logs for MCP initialization errors\n \n **Solution:**\n \n Restart UFO² client and verify MCP initialization:\n \n ```powershell\n python -m ufo.client.client `\n --ws `\n --ws-server ws://192.168.1.100:5000/ws `\n --client-id ufo2_desktop_1 `\n --platform windows\n ```\n \n Look for:\n ```log\n INFO - MCP servers initialized: ui_automation, file_operations, app_control\n INFO - UFO Client ready with 15 available tools\n ```\n\n---\n\n## Comparison with Standalone UFO²\n\n| Aspect | Standalone UFO² | UFO² as Galaxy Device |\n|--------|----------------|----------------------|\n| **Architecture** | Local mode (single process) | Server-client mode (distributed) |\n| **Control** | Direct user interaction | Galaxy orchestration |\n| **Multi-Device** | Single device only | Multiple UFO² devices |\n| **Cross-Platform** | Windows only | Windows + Linux + others |\n| **Task Distribution** | Manual | Automatic (capabilities-based) |\n| **Scalability** | Limited to one machine | Scales to device pool |\n| **Use Case** | Individual automation tasks | Enterprise multi-tier workflows |\n| **Configuration** | Simple (no server/client setup) | Requires server-client + Galaxy config |\n\n**When to use Standalone UFO²:**\n\n- Simple, single-device Windows automation\n- Development and testing\n- Personal productivity tasks\n- No need for cross-platform workflows\n\n**When to use UFO² as Galaxy Device:**\n\n- Enterprise-scale automation\n- Multi-device orchestration\n- Cross-platform workflows (Windows + Linux)\n- Centralized management and monitoring\n- Parallel task execution across multiple machines\n\n## Related Documentation\n\n- **[UFO² Overview](overview.md)** - Architecture and core concepts\n- **[HostAgent](host_agent/overview.md)** - Desktop-level automation\n- **[AppAgent](app_agent/overview.md)** - Application-specific automation\n- **[Galaxy Overview](../galaxy/overview.md)** - Multi-tier orchestration framework\n- **[Server-Client Architecture](../infrastructure/agents/server_client_architecture.md)** - Distributed agent design\n- **[Linux as Galaxy Device](../linux/as_galaxy_device.md)** - Linux agent integration (similar pattern)\n- **[Quick Start Linux](../getting_started/quick_start_linux.md)** - Similar server-client setup for Linux\n\n## Summary\n\nIntegrating UFO² into UFO³ Galaxy enables:\n\n- **Multi-tier orchestration** - Galaxy coordinates UFO² + Linux + other devices\n- **Cross-platform workflows** - Seamlessly combine Windows desktop + Linux servers\n- **Capability-based routing** - Intelligent task assignment to appropriate devices\n- **Scalable automation** - Manage multiple UFO² devices from unified control plane\n- **Enterprise-ready** - Centralized monitoring, fault isolation, load balancing\n- **Server-client architecture** - Separation of orchestration and execution\n- **Local MCP services** - Automatic initialization, no manual setup required\n\n**Next Steps:**\n\n1. Start with a single UFO² device to verify the setup\n2. Add more UFO² devices as needed for parallel execution\n3. Integrate Linux agents for cross-platform workflows\n4. Define custom capabilities for your specific use cases\n5. Monitor Galaxy logs to understand task routing decisions\n"
},
{
"path": "documents/docs/ufo2/core_features/control_detection/hybrid_detection.md",
"content": "# Hybrid Control Detection\n\nHybrid control detection combines both UIA and OmniParser to provide comprehensive UI coverage. It merges standard Windows controls detected via UIA with visual elements detected through OmniParser, removing duplicates based on Intersection over Union (IoU) overlap.\n\n\n\n## How It Works\n\nThe hybrid detection process follows these steps:\n\n```mermaid\ngraph LR\n A[Screenshot] --> B[UIA Detection]\n A --> C[OmniParser Detection]\n B --> D[UIA Controls Standard UI Elements]\n C --> E[Visual Controls Icons, Images, Custom UI]\n D --> F[Merge & Deduplicate IoU Threshold: 0.1]\n E --> F\n F --> G[Final Control List Annotated [1] to [N]]\n \n style D fill:#e3f2fd\n style E fill:#fff3e0\n style F fill:#e8f5e9\n style G fill:#f3e5f5\n```\n\n**Deduplication Algorithm:**\n\n1. Keep all UIA-detected controls (main list)\n2. For each OmniParser-detected control (additional list):\n - Calculate IoU with all UIA controls\n - If IoU > threshold (default 0.1), discard as duplicate\n - Otherwise, add to merged list\n3. Result: Maximum coverage with minimal duplicates\n\n## Benefits\n\n- **Maximum Coverage**: Detects both standard and custom UI elements\n- **No Gaps**: Visual detection fills in UIA blind spots\n- **Efficiency**: Deduplication prevents redundant annotations\n- **Flexibility**: Works across diverse application types\n\n## Configuration\n\n### Prerequisites\n\nBefore enabling hybrid detection, you must deploy and configure OmniParser. See [Visual Detection - Deployment](./visual_detection.md#deployment) for instructions.\n\n### Enable Hybrid Mode\n\nConfigure both backends in `config/ufo/system.yaml`:\n\n```yaml\n# Enable hybrid detection\nCONTROL_BACKEND: [\"uia\", \"omniparser\"]\n\n# IoU threshold for merging (controls with IoU > threshold are considered duplicates)\nIOU_THRESHOLD_FOR_MERGE: 0.1 # Default: 0.1\n\n# OmniParser configuration\nOMNIPARSER:\n ENDPOINT: \"\"\n BOX_THRESHOLD: 0.05\n IOU_THRESHOLD: 0.1\n USE_PADDLEOCR: True\n IMGSZ: 640\n```\n\n### Configuration Options\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `CONTROL_BACKEND` | List[str] | `[\"uia\"]` | List of detection backends to use |\n| `IOU_THRESHOLD_FOR_MERGE` | float | `0.1` | IoU threshold for duplicate detection (0.0-1.0) |\n\n**Tuning Guidelines:**\n\n- **Lower threshold (< 0.1)**: More aggressive deduplication, may miss some controls\n- **Higher threshold (> 0.1)**: Keep more overlapping controls, may have duplicates\n- **Recommended**: Keep default 0.1 for optimal balance\n\nSee [System Configuration](../../../configuration/system/system_config.md#control-backend) for complete configuration details.\n\n## Implementation\n\nThe hybrid detection is implemented through:\n\n- **`AppControlInfoStrategy`**: Orchestrates control collection from multiple backends\n- **`PhotographerFacade.merge_target_info_list()`**: Performs IoU-based deduplication\n- **`OmniparserGrounding`**: Handles visual detection and parsing\n\n## Reference\n\n:::automator.ui_control.grounding.omniparser.OmniparserGrounding"
},
{
"path": "documents/docs/ufo2/core_features/control_detection/overview.md",
"content": "# Control Detection\n\nWe support different control detection methods to detect controls in the application to accommodate both standard (UIA) and custom controls (Visual).\n\n## Detection Methods\n\n| Method | Description | Use Case |\n|--------|-------------|----------|\n| [**UIA**](./uia_detection.md) | Uses Windows UI Automation framework to detect standard controls. Provides APIs to access and manipulate UI elements in Windows applications. | Standard Windows applications with native controls |\n| [**Visual (OmniParser)**](./visual_detection.md) | Uses OmniParser vision-based detection to identify custom controls through computer vision techniques based on visual appearance. | Applications with custom controls, icons, or visual elements not accessible via UIA |\n| [**Hybrid**](./hybrid_detection.md) | Combines both UIA and OmniParser detection methods. Merges results from both approaches, removing duplicates based on IoU overlap. | Maximum coverage for applications with both standard and custom controls |\n\n## Configuration\n\nConfigure the control detection method by setting the `CONTROL_BACKEND` parameter in `config/ufo/system.yaml`:\n\n```yaml\n# Use UIA only (default, recommended)\nCONTROL_BACKEND: [\"uia\"]\n\n# Use OmniParser only\nCONTROL_BACKEND: [\"omniparser\"]\n\n# Use hybrid mode (UIA + OmniParser)\nCONTROL_BACKEND: [\"uia\", \"omniparser\"]\n```\n\nSee [System Configuration](../../../configuration/system/system_config.md#control-backend) for detailed configuration options.\n\n"
},
{
"path": "documents/docs/ufo2/core_features/control_detection/uia_detection.md",
"content": "# UIA Control Detection\n\nUIA control detection uses the Windows UI Automation (UIA) framework to detect and interact with standard controls in Windows applications. It provides a robust set of APIs to access and manipulate UI elements programmatically.\n\n## Features\n\n- **Fast and Reliable**: Native Windows API with optimal performance\n- **Standard Controls**: Works with most Windows applications using standard controls\n- **Rich Metadata**: Provides detailed control information (type, name, position, state, etc.)\n\n## Limitations\n\nUIA control detection may not detect non-standard controls, custom-rendered UI elements, or visual components that don't expose UIA interfaces (e.g., canvas-based controls, game UIs, some web content).\n\n## Configuration\n\nUIA is the default control detection backend. Configure it in `config/ufo/system.yaml`:\n\n```yaml\nCONTROL_BACKEND: [\"uia\"]\n```\n\nFor applications with custom controls, consider using [hybrid detection](./hybrid_detection.md) which combines UIA with visual detection.\n\n## Implementation\n\nUFO² uses the `ControlInspectorFacade` class to interact with the UIA framework. The facade pattern provides a simplified interface to:\n\n- Enumerate desktop windows\n- Find control elements in window hierarchies\n- Filter controls by type, visibility, and state\n- Extract control metadata and positions\n\nSee [System Configuration](../../../configuration/system/system_config.md#control-backend) for additional options.\n\n## Reference\n\n:::automator.ui_control.inspector.ControlInspectorFacade"
},
{
"path": "documents/docs/ufo2/core_features/control_detection/visual_detection.md",
"content": "# Visual Control Detection (OmniParser)\n\nVisual control detection uses [OmniParser-v2](https://github.com/microsoft/OmniParser), a vision-based grounding model that detects UI elements through computer vision. This method is particularly effective for custom controls, icons, images, and visual elements that may not be accessible through standard UIA.\n\n## Use Cases\n\n- **Custom Controls**: Detects proprietary or non-standard UI elements\n- **Visual Elements**: Icons, images, and graphics-based controls\n- **Web Content**: Elements within browser windows or web views\n- **Canvas-based UIs**: Applications that render custom graphics\n\n## Deployment\n\n### 1. Clone the OmniParser Repository\n\nOn your remote GPU server:\n\n```bash\ngit clone https://github.com/microsoft/OmniParser.git\ncd OmniParser/omnitool/omniparserserver\n```\n\n### 2. Start the OmniParser Service\n\n```bash\npython gradio_demo.py\n```\n\nThis will generate output similar to:\n\n```\n* Running on local URL: http://0.0.0.0:7861\n* Running on public URL: https://xxxxxxxxxxxxxxxxxx.gradio.live\n```\n\nFor detailed deployment instructions, refer to the [OmniParser README](https://github.com/microsoft/OmniParser/tree/master/omnitool).\n\n## Configuration\n\n### OmniParser Settings\n\nConfigure the OmniParser endpoint and parameters in `config/ufo/system.yaml`:\n\n```yaml\nOMNIPARSER:\n ENDPOINT: \"\" # The endpoint URL from deployment\n BOX_THRESHOLD: 0.05 # Bounding box confidence threshold\n IOU_THRESHOLD: 0.1 # IoU threshold for non-max suppression\n USE_PADDLEOCR: True # Enable OCR for text detection\n IMGSZ: 640 # Input image size for the model\n```\n\n### Enable Visual Detection\n\nSet `CONTROL_BACKEND` to use OmniParser:\n\n```yaml\n# Use OmniParser only\nCONTROL_BACKEND: [\"omniparser\"]\n\n# Or use hybrid mode (recommended for maximum coverage)\nCONTROL_BACKEND: [\"uia\", \"omniparser\"]\n```\n\nSee [Hybrid Detection](./hybrid_detection.md) for combining UIA and OmniParser, or [System Configuration](../../../configuration/system/system_config.md#control-backend) for detailed options.\n\n## Reference\n\n:::automator.ui_control.grounding.omniparser.OmniparserGrounding"
},
{
"path": "documents/docs/ufo2/core_features/hybrid_actions.md",
"content": "# Hybrid GUI–API Action Layer\n\nUFO² introduces a **hybrid action layer** that seamlessly combines traditional GUI automation with native application APIs, enabling agents to dynamically select the optimal execution method for each task. This design bridges the gap between universal GUI availability and high-fidelity API control, achieving both robustness and efficiency.\n\n## The Two-Interface Problem\n\nApplication environments typically expose two complementary classes of interfaces, each with distinct trade-offs:\n\n### GUI Frontends (Traditional Approach)\n\n**Characteristics:** \n✅ **Universally Available** — Works with any application, even without API documentation \n✅ **Visual Compatibility** — Follows actual UI layout users see \n✅ **No Integration Required** — Works out-of-the-box with UI Automation\n\n**Limitations:** \n❌ **Brittle to UI Changes** — Layout modifications break automation \n❌ **Slow Execution** — Requires screenshot capture, OCR, and simulated input \n❌ **Limited Precision** — Pixel-based targeting prone to errors \n❌ **High Cognitive Load** — LLMs must interpret visual information at each step\n\n### Native APIs (Preferred Approach)\n\n**Characteristics:** \n✅ **High-Fidelity Control** — Direct manipulation of application state \n✅ **Fast Execution** — No screenshot analysis or UI rendering delays \n✅ **Precise Operations** — Programmatic access to exact data structures \n✅ **Robust to UI Changes** — API contracts remain stable across versions\n\n**Limitations:** \n❌ **Requires Explicit Integration** — Must implement API wrappers for each app \n❌ **Limited Availability** — Not all applications expose comprehensive APIs \n❌ **Maintenance Overhead** — API changes require code updates \n❌ **Documentation Dependency** — Requires accurate API references\n\n!!! info \"Research Finding\"\n Studies show that **API-based agents outperform GUI-only agents** by 15–30% on tasks where APIs are available, but **GUI fallback is essential** for broad application coverage and handling edge cases where APIs are insufficient. \n 📄 Reference: [API Agents vs. GUI Agents](https://arxiv.org/abs/2501.05446)\n\n## UFO²'s Hybrid Solution\n\nUFO² addresses this dilemma through a **unified action layer** that:\n\n1. **Dynamically selects** between GUI and API execution based on availability and task requirements\n2. **Composes hybrid workflows** that mix GUI and API actions within a single task\n3. **Provides graceful fallback** from API to GUI when APIs are unavailable or insufficient\n4. **Leverages MCP servers** for extensible, modular integration of application-specific APIs\n\n\n*UFO²'s hybrid action architecture powered by Model Context Protocol (MCP) servers. Agents dynamically select between GUI automation (via UI Automation/Win32 APIs) and native application APIs (via MCP servers like Excel COM, Outlook API, PowerPoint), enabling optimal execution strategies for each task.*\n\n## MCP-Powered Action Execution\n\nUFO² implements the hybrid action layer through the **Model Context Protocol (MCP)** framework:\n\n### Architecture Components\n\n| Component | Role | Examples |\n|-----------|------|----------|\n| **MCP Servers** | Expose application-specific APIs as standardized tools | Excel COM Server, Outlook API Server, PowerPoint Server |\n| **GUI Automation Servers** | Provide universal UI interaction commands | UICollector, HostUIExecutor, AppUIExecutor |\n| **Command Dispatcher** | Routes agent requests to appropriate MCP server | Selects Excel API for cell operations, GUI for unlabeled buttons |\n| **Action Strategies** | Determine execution method based on context | Prefer API for bulk operations, GUI for visual verification |\n\n### Execution Flow\n\n```mermaid\ngraph TB\n Agent[AppAgent Action Decision] --> Decision{API Available & Preferred?}\n \n Decision -->|Yes| API[MCP API Server]\n Decision -->|No/Fallback| GUI[GUI Automation Server]\n \n API --> ExcelAPI[Excel COM]\n API --> OutlookAPI[Outlook COM]\n API --> PowerPointAPI[PowerPoint COM]\n \n GUI --> UIA[UI Automation]\n GUI --> Win32[Win32 APIs]\n \n ExcelAPI --> Result[Execution Result]\n OutlookAPI --> Result\n PowerPointAPI --> Result\n UIA --> Result\n Win32 --> Result\n \n style API fill:#e8f5e9\n style GUI fill:#fff3e0\n style Result fill:#e3f2fd\n```\n\n### Example: Excel Chart Creation\n\n**Scenario:** Create a column chart from data in cells A1:B10\n\n**API-First Execution:**\n\n```python\n# Agent decision: Use Excel API (fast, precise)\ncommand = ExcelCreateChartCommand(\n data_range=\"A1:B10\",\n chart_type=\"column\",\n chart_title=\"Sales Data\"\n)\n \n# MCP Server: Excel COM\nresult = mcp_server.execute(command)\n# → Direct API call: workbook.charts.add(...)\n# → Execution time: ~0.5s\n```\n\n**GUI Fallback Execution:**\n\n```python\n# Agent decision: API unavailable, use GUI\ncommands = [\n SelectControlCommand(control=\"A1:B10\"),\n ClickCommand(control=\"Insert > Chart\"),\n SelectChartTypeCommand(type=\"Column\"),\n SetTextCommand(control=\"Chart Title\", text=\"Sales Data\"),\n ClickCommand(control=\"OK\")\n]\n \n# MCP Server: UICollector\nfor cmd in commands:\n result = mcp_server.execute(cmd)\n# → UI Automation: capture, annotate, click sequence\n# → Execution time: ~8s\n```\n\n**Hybrid Execution:**\n\n```python\n# Agent decision: Mix API + GUI for optimal workflow\n \n# Step 1: API for data manipulation (fast)\napi_command = ExcelSetRangeCommand(\n range=\"A1:B10\",\n values=processed_data\n)\nmcp_api_server.execute(api_command)\n \n# Step 2: GUI for chart insertion (visual verification)\ngui_commands = [\n SelectControlCommand(control=\"A1:B10\"),\n ClickCommand(control=\"Insert > Recommended Charts\"),\n # Visual confirmation before finalizing\n ScreenshotCommand(),\n ClickCommand(control=\"OK\")\n]\nfor cmd in gui_commands:\n mcp_gui_server.execute(cmd)\n```\n\n---\n\n## Dynamic Action Selection\n\nUFO²'s agents use a **strategy-based decision process** to select execution methods:\n\n### Selection Criteria\n\nUFO² agents dynamically select between GUI and API execution based on:\n\n| Factor | API Preference | GUI Preference |\n|--------|---------------|---------------|\n| **Operation Type** | Bulk data operations, calculations | Visual layout, custom UI elements |\n| **Performance Requirement** | Time-critical tasks | Tasks requiring visual verification |\n| **API Availability** | Application has MCP server configured | Application only has GUI automation |\n| **Precision Requirement** | Exact data manipulation | Approximate interactions (e.g., scrolling) |\n| **Error Handling** | Predictable state changes | Exploratory interactions |\n\n**How Agents Decide:**\n\nThe agent **reasoning process** determines execution method based on:\n\n1. **Available MCP servers** — Check if application has API-based MCP servers configured\n2. **Task characteristics** — Bulk operations favor API, visual tasks favor GUI\n3. **Tool availability** — Each MCP server exposes specific capabilities as tools\n4. **LLM decision** — Agent reasons about which available tool best fits the task\n\n**Real-World Decision Examples:**\n\n**Task: \"Fill 1000 Excel cells with sequential numbers\"** \n→ **Decision: ExcelCOMExecutor** (COM API bulk operation ~2s vs. GUI 1000 clicks ~300s)\n\n**Task: \"Click the blue 'Submit' button in custom dialog\"** \n→ **Decision: AppUIExecutor** (No API for custom dialogs, visual grounding needed)\n\n**Task: \"Create presentation from Excel data, verify slide layout\"** \n→ **Decision: Both servers** (PowerPointCOMExecutor for data, AppUIExecutor for verification)\n\n## MCP Server Configuration\n\nUFO² agents discover available MCP servers through the `config/ufo/mcp.yaml` configuration:\n\n### Server Registration\n\n```yaml\n# config/ufo/mcp.yaml\n# MCP servers are organized by agent type and application\n\nAppAgent:\n # Default configuration for all applications\n default:\n data_collection:\n - namespace: UICollector # Screenshot capture, UI tree extraction\n type: local # Local in-memory server\n start_args: []\n reset: false\n action:\n - namespace: AppUIExecutor # GUI automation (click, type, scroll)\n type: local\n start_args: []\n reset: false\n - namespace: CommandLineExecutor # Command-line execution\n type: local\n start_args: []\n reset: false\n \n # Excel-specific configuration (adds COM API)\n EXCEL.EXE:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: AppUIExecutor # GUI fallback\n type: local\n start_args: []\n reset: false\n - namespace: ExcelCOMExecutor # Excel COM API\n type: local\n start_args: []\n reset: true # Reset when switching apps\n \n # Word-specific configuration\n WINWORD.EXE:\n action:\n - namespace: WordCOMExecutor # Word COM API\n type: local\n start_args: []\n reset: true\n \n # PowerPoint-specific configuration\n POWERPNT.EXE:\n action:\n - namespace: PowerPointCOMExecutor # PowerPoint COM API\n type: local\n start_args: []\n reset: true\n\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: HostUIExecutor # Desktop-level GUI automation\n type: local\n start_args: []\n reset: false\n - namespace: CommandLineExecutor\n type: local\n start_args: []\n reset: false\n```\n\n### How Agents Load MCP Servers\n\nWhen an agent is initialized for a specific application, the system:\n\n1. **Matches application** — Uses process name (e.g., `EXCEL.EXE`) to find configuration\n2. **Creates MCP servers** — Initializes servers via `MCPServerManager.create_or_get_server()`\n3. **Registers tools** — Each MCP server exposes tools (e.g., `excel_write_cell`, `ui_click`)\n4. **Agent discovers capabilities** — LLM sees available tools in system prompt\n\n**Example: Available Tools for Excel**\n\nWhen AppAgent opens Excel, it gets tools from:\n\n**ExcelCOMExecutor (API):**\n- `excel_write_cell` — Write to specific cell\n- `excel_read_range` — Read cell range\n- `excel_create_chart` — Create chart\n- `excel_run_macro` — Run VBA macro\n\n**AppUIExecutor (GUI):**\n- `ui_click` — Click UI element\n- `ui_type_text` — Type text\n- `ui_select` — Select from dropdown\n\n**UICollector (Data):**\n- `capture_screenshot` — Capture screen\n- `get_ui_tree` — Get UI element tree\n\nFor complete MCP documentation, see:\n\n- [MCP Overview](../../mcp/overview.md) — Model Context Protocol architecture\n- [MCP Configuration Reference](../../configuration/system/mcp_reference.md) — Complete configuration options\n- [MCP Server Documentation](../../mcp/local_servers.md) — All available MCP servers\n\n## Best Practices\n\n### When to Use API\n\n✅ **Bulk data operations** — Filling cells, processing records \n✅ **Precise calculations** — Formula application, data transformations \n✅ **Programmatic workflows** — Email automation, calendar scheduling \n✅ **Time-critical tasks** — High-volume operations with strict SLAs\n\n### When to Use GUI\n\n✅ **Visual verification** — Layout checking, color validation \n✅ **Custom UI elements** — Application-specific dialogs, unlabeled controls \n✅ **Exploratory tasks** — Navigating unfamiliar applications \n✅ **Legacy applications** — Apps without accessible APIs\n\n### When to Use Hybrid\n\n✅ **Complex workflows** — Combine API efficiency with GUI verification \n✅ **Partial API coverage** — Use API where available, GUI for gaps \n✅ **User-facing demos** — API for backend, GUI for visible interactions \n✅ **Debugging** — API for state setup, GUI for manual inspection\n\n!!! warning \"Common Pitfalls\"\n - **Over-relying on APIs** — Some UI states only visible through screenshots \n - **Ignoring API errors** — Always implement GUI fallback for resilience \n - **Static execution plans** — Use dynamic selection based on runtime context \n - **Inadequate verification** — Combine API execution with screenshot validation\n\n## Related Documentation\n\n### Core Concepts\n\n- [**MCP Overview**](../../mcp/overview.md) — Model Context Protocol architecture \n- [**AppAgent**](../app_agent/overview.md) — Application-level agent implementation \n- [**HostAgent**](../host_agent/overview.md) — Desktop-level agent implementation\n\n### Configuration\n\n- [**MCP Configuration Reference**](../../configuration/system/mcp_reference.md) — Complete MCP server configuration options \n- [**Configuration Guide**](../../configuration/system/overview.md) — System configuration overview\n\n### MCP Servers\n\n- [**UICollector**](../../mcp/servers/ui_collector.md) — Screenshot and UI tree capture \n- [**AppUIExecutor**](../../mcp/servers/app_ui_executor.md) — GUI automation server \n- [**ExcelCOMExecutor**](../../mcp/servers/excel_com_executor.md) — Excel COM API integration \n- [**WordCOMExecutor**](../../mcp/servers/word_com_executor.md) — Word COM API integration \n- [**PowerPointCOMExecutor**](../../mcp/servers/ppt_com_executor.md) — PowerPoint COM API integration \n- [**CommandLineExecutor**](../../mcp/servers/command_line_executor.md) — Command-line execution\n\n---\n\n## Next Steps\n\n1. **Explore MCP Architecture**: Read [MCP Overview](../../mcp/overview.md) to understand the protocol design \n2. **Configure MCP Servers**: Review [MCP Configuration](../../configuration/system/mcp_reference.md) for setup options \n3. **Study MCP Servers**: Check built-in implementations in [MCP Server Documentation](../../mcp/local_servers.md) \n4. **Build Custom Agents**: Follow [Creating AppAgent](../../tutorials/creating_app_agent/overview.md) to use hybrid actions\n\nWant to see hybrid actions in practice?\n\n- [Quick Start Guide](../../getting_started/quick_start_ufo2.md) — Run UFO² with default MCP servers \n- [Creating AppAgent Tutorial](../../tutorials/creating_app_agent/overview.md) — Build custom agents with hybrid actions\n- [Speculative Multi-Action Execution](multi_action.md) — Optimize performance with batch action prediction\n"
},
{
"path": "documents/docs/ufo2/core_features/knowledge_substrate/experience_learning.md",
"content": "# Learning from Self-Experience\n\nWhen UFO successfully completes a task, users can save the successful experience to enhance the AppAgent's future performance. The AppAgent learns from its own successful experiences to improve task execution.\n\n## Mechanism\n\n```mermaid\ngraph TD\n A[Complete Session] --> B[Prompt User to Save Experience]\n B --> C{User Saves?}\n C -->|Yes| D[Summarize with ExperienceSummarizer]\n C -->|No| I[End]\n D --> E[Save to Experience Database]\n F[AppAgent Encounters Similar Task] --> G[Retrieve Saved Experience]\n G --> H[Generate Plan Using Retrieved Experience]\n```\n\n### Workflow Steps\n\n1. **Complete a Session**: UFO finishes executing a task successfully\n\n2. **Prompt User to Save**: The system asks whether to save the experience\n\n \n\n3. **Summarize Experience**: If the user chooses to save, the `ExperienceSummarizer` processes the session:\n - Extracts key information from the execution trajectory\n - Summarizes the experience into a structured demonstration example\n - Saves it to the experience database at the configured path\n - The demonstration example includes fields similar to those in the [AppAgent's prompt examples](../../prompts/examples_prompts.md)\n\n4. **Retrieve and Utilize**: When encountering similar tasks in the future:\n - The AppAgent queries the experience database\n - Retrieves relevant past experiences\n - Uses them to inform plan generation\n\n## Configuration\n\nConfigure the following parameters in `config.yaml` to enable self-experience learning:\n\n| Configuration Option | Description | Type | Default |\n|---------------------|-------------|------|---------|\n| `RAG_EXPERIENCE` | Enable experience-based learning | Boolean | `False` |\n| `RAG_EXPERIENCE_RETRIEVED_TOPK` | Number of top experiences to retrieve | Integer | `5` |\n| `EXPERIENCE_SAVED_PATH` | Database path for storing experiences | String | `\"vectordb/experience/\"` |\n\nFor more details on RAG configuration, see the [RAG Configuration Guide](../../../configuration/system/rag_config.md).\n\n## API Reference\n\n### Experience Summarizer\n\nThe `ExperienceSummarizer` class in `ufo/experience/summarizer.py` handles experience summarization:\n\n:::experience.summarizer.ExperienceSummarizer\n\n### Experience Retriever\n\nThe `ExperienceRetriever` class in `ufo/rag/retriever.py` handles experience retrieval:\n\n:::rag.retriever.ExperienceRetriever\n"
},
{
"path": "documents/docs/ufo2/core_features/knowledge_substrate/learning_from_bing_search.md",
"content": "# Learning from Bing Search\n\nUFO can enhance the AppAgent by searching for information on Bing to obtain up-to-date knowledge for niche tasks or applications beyond the AppAgent's existing knowledge base.\n\n## Mechanism\n\nWhen processing a request, the AppAgent:\n\n1. Constructs a Bing search query based on the request context\n2. Retrieves top-k search results from Bing\n3. Extracts relevant information from the search results\n4. Generates a plan informed by the retrieved information\n\nThis mechanism is particularly useful for:\n- Tasks requiring current information (e.g., latest software features, current events)\n- Applications or domains not covered by the agent's training data\n- Dynamic information that changes frequently\n\n## Configuration\n\nTo enable Bing search integration:\n\n1. **Obtain Bing API Key**: Get your API key from [Microsoft Azure Bing Search API](https://www.microsoft.com/en-us/bing/apis/bing-web-search-api)\n\n2. **Configure Parameters**: Set the following options in `config.yaml`:\n\n| Configuration Option | Description | Type | Default |\n|---------------------|-------------|------|---------|\n| `RAG_ONLINE_SEARCH` | Enable Bing search integration | Boolean | `False` |\n| `BING_API_KEY` | Bing Search API key | String | `\"\"` |\n| `RAG_ONLINE_SEARCH_TOPK` | Number of search results to retrieve | Integer | `5` |\n| `RAG_ONLINE_RETRIEVED_TOPK` | Number of retrieved results to include in prompt | Integer | `5` |\n\nFor more details on RAG configuration, see the [RAG Configuration Guide](../../../configuration/system/rag_config.md).\n\n## API Reference\n\n:::rag.retriever.OnlineDocRetriever"
},
{
"path": "documents/docs/ufo2/core_features/knowledge_substrate/learning_from_demonstration.md",
"content": "# Learning from User Demonstration\n\nFor complex tasks, users can demonstrate the task execution process to help UFO learn effective action patterns. UFO uses Windows [Step Recorder](https://support.microsoft.com/en-us/windows/record-steps-to-reproduce-a-problem-46582a9b-620f-2e36-00c9-04e25d784e47) to capture user action trajectories, which are then processed and stored for future reference.\n\n## Mechanism\n\nUFO leverages the Windows Step Recorder tool to capture task demonstrations. The workflow operates as follows:\n\n1. **Record**: User performs the task while Step Recorder captures the action sequence\n2. **Process**: The `DemonstrationSummarizer` extracts and summarizes the recorded demonstration from the zip file\n3. **Store**: Summarized demonstrations are saved to the configured demonstration database\n4. **Retrieve**: When encountering similar tasks, the `DemonstrationRetriever` queries relevant demonstrations\n5. **Apply**: Retrieved demonstrations guide the AppAgent's plan generation\n\nSee the [User Demonstration Provision](../../../tutorials/creating_app_agent/demonstration_provision.md) guide for detailed recording instructions.\n\n**Demo Video:**\n\n\n## Configuration\n\nTo enable learning from user demonstrations:\n\n1. **Provide Demonstrations**: Follow the [User Demonstration Provision](../../../tutorials/creating_app_agent/demonstration_provision.md) guide to record demonstrations\n\n2. **Configure Parameters**: Set the following options in `config.yaml`:\n\n| Configuration Option | Description | Type | Default |\n|---------------------|-------------|------|---------|\n| `RAG_DEMONSTRATION` | Enable demonstration-based learning | Boolean | `False` |\n| `RAG_DEMONSTRATION_RETRIEVED_TOPK` | Number of top demonstrations to retrieve | Integer | `5` |\n| `RAG_DEMONSTRATION_COMPLETION_N` | Number of completion choices for demonstration results | Integer | `3` |\n| `DEMONSTRATION_SAVED_PATH` | Database path for storing demonstrations | String | `\"vectordb/demonstration/\"` |\n\nFor more details on RAG configuration, see the [RAG Configuration Guide](../../../configuration/system/rag_config.md).\n\n## API Reference\n\n### Demonstration Summarizer\n\nThe `DemonstrationSummarizer` class in `record_processor/summarizer/summarizer.py` handles demonstration summarization:\n\n:::summarizer.summarizer.DemonstrationSummarizer\n\n### Demonstration Retriever\n\nThe `DemonstrationRetriever` class in `ufo/rag/retriever.py` handles demonstration retrieval:\n\n:::rag.retriever.DemonstrationRetriever"
},
{
"path": "documents/docs/ufo2/core_features/knowledge_substrate/learning_from_help_document.md",
"content": "# Learning from Help Documents\n\nUsers or applications can provide help documents to enhance the AppAgent's capabilities. The AppAgent retrieves relevant knowledge from these documents to improve task understanding, plan quality, and application interaction efficiency.\n\nFor instructions on providing help documents, see the [Help Document Provision](../../../tutorials/creating_app_agent/help_document_provision.md) guide.\n\n## Mechanism\n\nHelp documents are structured as **task-solution pairs**. When processing a request, the AppAgent:\n\n1. Retrieves relevant help documents by matching the request against task descriptions\n2. Uses the retrieved solutions as references for plan generation\n3. Adapts the solutions to the specific context\n\nSince retrieved documents may not be perfectly relevant, the AppAgent treats them as references rather than strict instructions, allowing for flexible adaptation to the actual task requirements.\n\n## Configuration\n\nTo enable learning from help documents:\n\n1. **Provide Help Documents**: Follow the [Help Document Provision](../../../tutorials/creating_app_agent/help_document_provision.md) guide to prepare and index help documents\n\n2. **Configure Parameters**: Set the following options in `config.yaml`:\n\n| Configuration Option | Description | Type | Default |\n|---------------------|-------------|------|---------|\n| `RAG_OFFLINE_DOCS` | Enable offline help document retrieval | Boolean | `False` |\n| `RAG_OFFLINE_DOCS_RETRIEVED_TOPK` | Number of top documents to retrieve | Integer | `1` |\n\nFor more details on RAG configuration, see the [RAG Configuration Guide](../../../configuration/system/rag_config.md).\n\n## API Reference\n\n:::rag.retriever.OfflineDocRetriever"
},
{
"path": "documents/docs/ufo2/core_features/knowledge_substrate/overview.md",
"content": "# Knowledge Substrate\n\nUFO provides versatile mechanisms to enhance the AppAgent's capabilities through RAG (Retrieval-Augmented Generation) and other knowledge retrieval techniques. These mechanisms improve the AppAgent's task understanding, plan quality, and interaction efficiency with applications.\n\n## Supported Knowledge Sources\n\nUFO currently supports the following knowledge retrieval methods:\n\n| Knowledge Source | Description |\n|------------------|-------------|\n| [Help Documents](./learning_from_help_document.md) | Retrieve knowledge from offline help documentation indexed for specific applications. |\n| [Bing Search](./learning_from_bing_search.md) | Search online information via Bing to obtain up-to-date knowledge. |\n| [Self-Experience](./experience_learning.md) | Learn from the agent's own successful task execution history. |\n| [User Demonstrations](./learning_from_demonstration.md) | Learn from action trajectories demonstrated by users. |\n\n## Context Provision\n\nUFO provides knowledge to the AppAgent through the `context_provision` method defined in the `AppAgent` class:\n\n```python\nasync def context_provision(\n self, request: str = \"\", context: Context = None\n) -> None:\n \"\"\"\n Provision the context for the app agent.\n :param request: The request sent to the Bing search retriever.\n \"\"\"\n\n ufo_config = get_ufo_config()\n\n # Load the offline document indexer for the app agent if available.\n if ufo_config.rag.offline_docs:\n console.print(\n f\"📚 Loading offline help document indexer for {self._process_name}...\",\n style=\"magenta\",\n )\n self.build_offline_docs_retriever()\n\n # Load the online search indexer for the app agent if available.\n\n if ufo_config.rag.online_search and request:\n console.print(\"🔍 Creating a Bing search indexer...\", style=\"magenta\")\n self.build_online_search_retriever(\n request, ufo_config.rag.online_search_topk\n )\n\n # Load the experience indexer for the app agent if available.\n if ufo_config.rag.experience:\n console.print(\"📖 Creating an experience indexer...\", style=\"magenta\")\n experience_path = ufo_config.rag.experience_saved_path\n db_path = os.path.join(experience_path, \"experience_db\")\n self.build_experience_retriever(db_path)\n\n # Load the demonstration indexer for the app agent if available.\n if ufo_config.rag.demonstration:\n console.print(\"🎬 Creating an demonstration indexer...\", style=\"magenta\")\n demonstration_path = ufo_config.rag.demonstration_saved_path\n db_path = os.path.join(demonstration_path, \"demonstration_db\")\n self.build_human_demonstration_retriever(db_path)\n\n await self._load_mcp_context(context)\n```\n\nThe `context_provision` method loads various knowledge retrievers based on the configuration settings in `config.yaml`:\n\n- **Offline document retriever**: Loads indexed help documentation for the target application\n- **Online search retriever**: Creates a Bing search indexer when a search request is provided\n- **Experience retriever**: Loads the agent's historical successful experiences\n- **Demonstration retriever**: Loads user-demonstrated action trajectories\n- **MCP context**: Loads Model Context Protocol tool information for the current application\n\n## Retriever API Reference\n\nUFO employs the `Retriever` class located in `ufo/rag/retriever.py` to retrieve knowledge from various sources. For detailed API documentation, see:\n\n:::rag.retriever.Retriever\n"
},
{
"path": "documents/docs/ufo2/core_features/multi_action.md",
"content": "# Speculative Multi-Action Execution\n\nUFO² introduces **Speculative Multi-Action Execution**, a feature that allows agents to bundle multiple predicted steps into a single LLM call and validate them against the live application state. This approach can reduce LLM queries by up to **51%** compared to inferring each action separately.\n\n## Overview\n\nTraditional agent execution follows a sequential pattern: **think → act → observe → think → act → observe**. Each cycle requires a separate LLM inference, making complex tasks slow and expensive.\n\nSpeculative multi-action execution optimizes this by predicting a **batch of likely actions** upfront, then validating them against the live UI Automation state in a single execution pass:\n\n\n\n**Key Benefits:**\n\n- **Reduced LLM Calls**: Up to 51% fewer inference requests for multi-step tasks\n- **Faster Execution**: Batch prediction eliminates per-action round-trips\n- **Lower Costs**: Fewer API calls reduce operational expenses\n- **Maintained Accuracy**: Live validation ensures actions remain correct\n\n## How It Works\n\nWhen enabled, the agent:\n\n1. **Predicts Action Sequence**: Uses contextual understanding to forecast likely next steps (e.g., \"Open Excel → Navigate to cell A1 → Enter value → Save\")\n2. **Validates Against Live State**: Checks each predicted action against current UI Automation state\n3. **Executes Valid Actions**: Runs all validated actions in sequence\n4. **Handles Failures Gracefully**: Falls back to single-action mode if predictions fail validation\n\n## Configuration\n\nEnable speculative multi-action execution in `config/ufo/system.yaml`:\n\n```yaml\n# Action Configuration\nACTION_SEQUENCE: true # Enable multi-action prediction and execution\n```\n\n**Configuration Location**: `config/ufo/system.yaml` (migrated from legacy `config_dev.yaml`)\n\nFor configuration migration details, see [Configuration Migration Guide](../../configuration/system/migration.md).\n\n## Implementation Details\n\nThe multi-action system is implemented through two core classes in `ufo/agents/processors/schemas/actions.py`:\n\n### ActionCommandInfo\n\nRepresents a single action with execution metadata:\n\n:::agents.processors.schemas.actions.ActionCommandInfo\n\n**Key Properties:**\n\n- `function`: Action name (e.g., `click`, `type_text`)\n- `arguments`: Action parameters\n- `target`: UI element information\n- `result`: Execution result with status and error details\n- `action_string`: Human-readable representation\n\n### ListActionCommandInfo\n\nManages sequences of multiple actions:\n\n:::agents.processors.schemas.actions.ListActionCommandInfo\n\n**Key Methods:**\n\n- `add_action()`: Append action to sequence\n- `to_list_of_dicts()`: Serialize for logging/debugging\n- `to_representation()`: Generate human-readable summary\n- `count_repeat_times()`: Track repeated actions for loop detection\n- `get_results()`: Extract execution outcomes\n\n## Example Scenarios\n\n**Scenario 1: Excel Data Entry**\n\nWithout multi-action:\n```\nThink → Open Excel → Observe → Think → Click A1 → Observe → Think → Type \"Sales\" → Observe → Think → Save → Observe\n```\n**5 LLM calls**\n\nWith multi-action:\n```\nThink → [Open Excel, Click A1, Type \"Sales\", Save] → Observe\n```\n**1 LLM call** (80% reduction)\n\n**Scenario 2: Email Composition**\n\nSingle-action mode:\n```\nThink → Open Outlook → Think → Click New → Think → Enter recipient → Think → Enter subject → Think → Type body → Think → Send\n```\n**7 LLM calls**\n\nMulti-action mode:\n```\nThink → [Open Outlook, Click New, Enter recipient, Enter subject, Type body, Send] → Observe\n```\n**1 LLM call** (85% reduction)\n\n## When to Use\n\n**Best for:**\n\n✅ Predictable workflows with clear action sequences \n✅ Repetitive tasks (data entry, form filling) \n✅ Applications with stable UI structures \n✅ Cost-sensitive deployments requiring fewer LLM calls\n\n**Not recommended for:**\n\n❌ Highly dynamic UIs with frequent state changes \n❌ Exploratory tasks requiring frequent observation \n❌ Error-prone applications where validation is critical per step \n❌ Tasks requiring user confirmation between actions\n\n## Related Documentation\n\n- [AppAgent Processing Strategy](../app_agent/strategy.md) — How agents process and execute actions\n- [Hybrid GUI-API Actions](hybrid_actions.md) — Combining GUI automation with native APIs\n- [System Configuration Reference](../../configuration/system/system_config.md) — Complete `system.yaml` options\n- [Configuration Migration](../../configuration/system/migration.md) — Migrating from legacy `config_dev.yaml`\n\n## Performance Considerations\n\n**Trade-offs:**\n\n- **Accuracy vs. Speed**: Multi-action sacrifices per-step validation for batch efficiency\n- **Memory Usage**: Larger context windows needed to predict action sequences\n- **Failure Recovery**: Invalid predictions require full sequence rollback and retry\n\n**Optimization Tips:**\n\n1. **Start Conservative**: Test with `ACTION_SEQUENCE: false` before enabling\n2. **Monitor Validation Rates**: High rejection rates indicate poor prediction quality\n3. **Combine with Hybrid Actions**: Use [API-based execution](hybrid_actions.md) where possible for fastest performance\n4. **Tune MAX_STEP**: Set appropriate `MAX_STEP` limits in `system.yaml` to prevent runaway sequences\n"
},
{
"path": "documents/docs/ufo2/dataflow/execution.md",
"content": "# Execution\n\nThe instantiated plans will be executed by a `execute` task. In this phase, given the task-action data, the execution process will match the real controller based on word environment and execute the plan step by step. After execution, `evalution` agent will evaluation the quality of the entire execution process.\n\n
\n \n
\n\n## ExecuteFlow\n\nThe `ExecuteFlow` class is designed to facilitate the execution and evaluation of tasks in a Windows application environment. It provides functionality to interact with the application's UI, execute predefined tasks, capture screenshots, and evaluate the results of the execution. The class also handles logging and error management for the tasks.\n\n### Task Execution\n\nThe **task execution** in the `ExecuteFlow` class follows a structured sequence to ensure accurate and traceable task performance:\n\n1. **Initialization**:\n\n - Load configuration settings and log paths.\n - Find the application window matching the task.\n - Retrieve or create an `ExecuteAgent` for executing the task.\n2. **Plan Execution**:\n\n - Loop through each step in the `instantiated_plan`.\n - Parse the step to extract information like subtasks, control text, and the required operation.\n3. **Action Execution**:\n\n - Find the control in the application window that matches the specified control text.\n - If no matching control is found, raise an error.\n - Perform the specified action (e.g., click, input text) using the agent's Puppeteer framework.\n - Capture screenshots of the application window and selected controls for logging and debugging.\n4. **Result Logging**:\n\n - Log details of the step execution, including control information, performed action, and results.\n5. **Finalization**:\n\n - Save the final state of the application window.\n - Quit the application client gracefully.\n\n---\n\n## Evaluation\n\nThe **evaluation** process in the `ExecuteFlow` class is designed to assess the performance of the executed task based on predefined prompts:\n\n1. **Start Evaluation**:\n\n - Evaluation begins immediately after task execution.\n - It uses an `ExecuteEvalAgent` initialized during class construction.\n2. **Perform Evaluation**:\n\n - The `ExecuteEvalAgent` evaluates the task using a combination of input prompts (e.g., main prompt and API prompt) and logs generated during task execution.\n - The evaluation process outputs a result summary (e.g., quality flag, comments, and task type).\n3. **Log and Output Results**:\n\n - Display the evaluation results in the console.\n - Return the evaluation summary alongside the executed plan for further analysis or reporting.\n\n# Reference\n\n## ExecuteFlow\n\n::: execution.workflow.execute_flow.ExecuteFlow\n\n## ExecuteAgent\n\n::: execution.agent.execute_agent.ExecuteAgent\n\n## ExecuteEvalAgent\n\n::: execution.agent.execute_eval_agent.ExecuteEvalAgent\n"
},
{
"path": "documents/docs/ufo2/dataflow/instantiation.md",
"content": "# Instantiation\n\nThere are three key steps in the instantiation process:\n\n1. `Choose a template` file according to the specified app and instruction.\n2. `Prefill` the task using the current screenshot.\n3. `Filter` the established task.\n\nGiven the initial task, the dataflow first choose a template (`Phase 1`), the prefill the initial task based on word envrionment to obtain task-action data (`Phase 2`). Finnally, it will filter the established task to evaluate the quality of task-action data.\n\n
\n \n
\n\n## 1. Choose Template File\n\nTemplates for your app must be defined and described in `dataflow/templates/app`. For instance, if you want to instantiate tasks for the Word application, place the relevant `.docx` files in dataflow `/templates/word `, along with a `description.json` file. The appropriate template will be selected based on how well its description matches the instruction.\n\nThe `ChooseTemplateFlow` uses semantic matching, where task descriptions are compared with template descriptions using embeddings and FAISS for efficient nearest neighbor search. If semantic matching fails, a random template is chosen from the available files.\n\n## 2. Prefill the Task\n\n### PrefillFlow\n\nThe `PrefillFlow` class orchestrates the refinement of task plans and UI interactions by leveraging `PrefillAgent` for task planning and action generation. It automates UI control updates, captures screenshots, and manages logs for messages and responses during execution.\n\n### PrefillAgent\n\nThe `PrefillAgent` class facilitates task instantiation and action sequence generation by constructing tailored prompt messages using the `PrefillPrompter`. It integrates system, user, and dynamic context to generate actionable inputs for down-stream workflows.\n\n## 3. Filter Task\n\n### FilterFlow\n\nThe `FilterFlow` class is designed to process and refine task plans by leveraging a `FilterAgent`. The `FilterFlow` class acts as a bridge between the instantiation of tasks and the execution of a filtering process, aiming to refine task steps and prefill task-related files based on predefined filtering criteria.\n\n### FilterAgent\n\nThe `FilterAgent` class is a specialized agent used to evaluate whether an instantiated task is correct. It inherits from the BasicAgent class and includes several methods and attributes to handle its functionality.\n\n# Reference\n\n## ChooseTemplateFlow\n\n::: instantiation.workflow.choose_template_flow.ChooseTemplateFlow\n\n## PrefillFlow\n\n::: instantiation.workflow.prefill_flow.PrefillFlow\n\n## PrefillAgent\n\n::: instantiation.agent.prefill_agent.PrefillAgent\n\n## FilterFlow\n\n::: instantiation.workflow.filter_flow.FilterFlow\n\n## FilterAgent\n\n::: instantiation.agent.filter_agent.FilterAgent\n"
},
{
"path": "documents/docs/ufo2/dataflow/overview.md",
"content": "# Introduction\n\nThis repository contains the implementation of the **Data Collection** process for training the **Large Action Models** (LAMs) in the paper of [Large Action Models: From Inception to Implementation](https://arxiv.org/abs/2412.10047). The **Data Collection** process is designed to streamline task processing, ensuring that all necessary steps are seamlessly integrated from initialization to execution. This module is part of the [**UFO**](https://arxiv.org/abs/2402.07939) project.\n\n# Dataflow\n\nDataflow uses UFO to implement `instantiation`, `execution`, and `dataflow` for a given task, with options for batch processing and single processing.\n\n1. **[Instantiation](./instantiation.md)**: Instantiation refers to the process of setting up and preparing a task for execution. This step typically involves `choosing template`, `prefill` and `filter`.\n2. **[Execution](./execution.md)**: Execution is the actual process of running the task. This step involves carrying out the actions or operations specified by the `Instantiation`. And after execution, an evaluate agent will evaluate the quality of the whole execution process.\n3. **Dataflow**: Dataflow is the overarching process that combines **instantiation** and **execution** into a single pipeline. It provides an end-to-end solution for processing tasks, ensuring that all necessary steps (from initialization to execution) are seamlessly integrated.\n\nYou can use `instantiation` and `execution` independently if you only need to perform one specific part of the process. When both steps are required for a task, the `dataflow` process streamlines them, allowing you to execute tasks from start to finish in a single pipeline.\n\nThe overall processing of dataflow is as below. Given a task-plan data, the LLMwill instantiatie the task-action data, including choosing template, prefill, filter.\n\n
\n \n
\n\n## How To Use\n\n### 1. Install Packages\n\nYou should install the necessary packages in the UFO root folder:\n\n```bash\npip install -r requirements.txt\n```\n\n### 2. Configure the LLMs\n\nBefore running dataflow, you need to provide your LLM configurations **individually for PrefillAgent and FilterAgent**. You can create your own config file `dataflow/config/config.yaml`, by copying the `dataflow/config/config.yaml.template` and editing config for **PREFILL_AGENT** and **FILTER_AGENT** as follows:\n\n#### OpenAI\n\n```bash\nVISUAL_MODE: True, # Whether to use the visual mode\nAPI_TYPE: \"openai\" , # The API type, \"openai\" for the OpenAI API. \nAPI_BASE: \"https://api.openai.com/v1/chat/completions\", # The the OpenAI API endpoint.\nAPI_KEY: \"sk-\", # The OpenAI API key, begin with sk-\nAPI_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\nAPI_MODEL: \"gpt-4-vision-preview\", # The only OpenAI model\n```\n\n#### Azure OpenAI (AOAI)\n\n```bash\nVISUAL_MODE: True, # Whether to use the visual mode\nAPI_TYPE: \"aoai\" , # The API type, \"aoai\" for the Azure OpenAI. \nAPI_BASE: \"YOUR_ENDPOINT\", # The AOAI API address. Format: https://{your-resource-name}.openai.azure.com\nAPI_KEY: \"YOUR_KEY\", # The aoai API key\nAPI_VERSION: \"2024-02-15-preview\", # \"2024-02-15-preview\" by default\nAPI_MODEL: \"gpt-4-vision-preview\", # The only OpenAI model\nAPI_DEPLOYMENT_ID: \"YOUR_AOAI_DEPLOYMENT\", # The deployment id for the AOAI API\n```\n\nYou can also non-visial model (e.g., GPT-4) for each agent, by setting `VISUAL_MODE: False` and proper `API_MODEL` (openai) and `API_DEPLOYMENT_ID` (aoai).\n\n#### Non-Visual Model Configuration\n\nYou can utilize non-visual models (e.g., GPT-4) for each agent by configuring the following settings in the `config.yaml` file:\n\n- ``VISUAL_MODE: False # To enable non-visual mode.``\n- Specify the appropriate `API_MODEL` (OpenAI) and `API_DEPLOYMENT_ID` (AOAI) for each agent.\n\nEnsure you configure these settings accurately to leverage non-visual models effectively.\n\n#### Other Configurations\n\n`config_dev.yaml` specifies the paths of relevant files and contains default settings. The match strategy for the window match and control filter supports options: `'contains'`, `'fuzzy'`, and `'regex'`, allowing flexible matching strategy for users. The `MAX_STEPS` is the max step for the execute_flow, which can be set by users.\n\n!!!note\n The specific implementation and invocation method of the matching strategy can refer to [windows_app_env](./windows_app_env.md).\n\n!!!note\n **BE CAREFUL!** If you are using GitHub or other open-source tools, do not expose your `config.yaml` online, as it contains your private keys.\n\n### 3. Prepare Files\n\nCertain files need to be prepared before running the task.\n\n#### 3.1. Tasks as JSON\n\nThe tasks that need to be instantiated should be organized in a folder of JSON files, with the default folder path set to dataflow `/tasks `. This path can be changed in the `dataflow/config/config.yaml` file, or you can specify it in the terminal, as mentioned in **4. Start Running**. For example, a task stored in `dataflow/tasks/prefill/` may look like this:\n\n```json\n{\n // The app you want to use\n \"app\": \"word\",\n // A unique ID to distinguish different tasks \n \"unique_id\": \"1\",\n // The task and steps to be instantiated\n \"task\": \"Type 'hello' and set the font type to Arial\",\n \"refined_steps\": [\n \"Type 'hello'\",\n \"Set the font to Arial\"\n ]\n}\n```\n\n#### 3.2. Templates and Descriptions\n\nYou should place an app file as a reference for instantiation in a folder named after the app.\n\nFor example, if you have `template1.docx` for Word, it should be located at `dataflow/templates/word/template1.docx`.\n\nAdditionally, for each app folder, there should be a `description.json` file located at `dataflow/templates/word/description.json`, which describes each template file in detail. It may look like this:\n\n```json\n{\n \"template1.docx\": \"A document with a rectangle shape\",\n \"template2.docx\": \"A document with a line of text\"\n}\n```\n\nIf a `description.json` file is not present, one template file will be selected at random.\n\n#### 3.3. Final Structure\n\nEnsure the following files are in place:\n\n- [X] JSON files to be instantiated\n- [X] Templates as references for instantiation\n- [X] Description file in JSON format\n\nThe structure of the files can be:\n\n```txt\ndataflow/\n|\n├── tasks\n│ └── prefill\n│ ├── bulleted.json\n│ ├── delete.json\n│ ├── draw.json\n│ ├── macro.json\n│ └── rotate.json\n├── templates\n│ └── word\n│ ├── description.json\n│ ├── template1.docx\n│ ├── template2.docx\n│ ├── template3.docx\n│ ├── template4.docx\n│ ├── template5.docx\n│ ├── template6.docx\n│ └── template7.docx\n└── ...\n```\n\n### 4. Start Running\n\nAfter finishing the previous steps, you can use the following commands in the command line. We provide single / batch process, for which you need to give the single file path / folder path. Determine the type of path provided by the user and automatically decide whether to process a single task or batch tasks.\n\nAlso, you can choose to use `instantiation` / `execution` sections individually, or use them as a whole section, which is named as `dataflow`.\n\nThe default task hub is set to be `\"TASKS_HUB\"` in `dataflow/config_dev.yaml`.\n\n* Dataflow Task:\n\n```bash\npython -m dataflow -dataflow --task_path path_to_task_file\n```\n\n* Instantiation Task:\n\n```bash\npython -m dataflow -instantiation --task_path path_to_task_file\n```\n\n* Execution Task:\n\n```bash\npython -m dataflow -execution --task_path path_to_task_file\n```\n\n!!! note\n\n 1. Users should be careful to save the original files while using this project; otherwise, the files will be closed when the app is shut down.\n 2. After starting the project, users should not close the app window while the program is taking screenshots.\n\n## Workflow\n\n### Instantiation\n\nThere are three key steps in the instantiation process:\n\n1. `Choose a template` file according to the specified app and instruction.\n2. `Prefill` the task using the current screenshot.\n3. `Filter` the established task.\n\nGiven the initial task, the dataflow first choose a template (`Phase 1`), the prefill the initial task based on word envrionment to obtain task-action data (`Phase 2`). Finnally, it will filter the established task to evaluate the quality of task-action data. (`Phase 3`)\n\n!!! note\n The more detailed code design documentation for instantiation can be found in [instantiation](./instantiation.md).\n\n### Execution\n\nThe instantiated plans will be executed by a execute task. After execution, evalution agent will evaluation the quality of the entire execution process.\n\n!!! note\n The more detailed code design documentation for execution can be found in [execution](./execution.md).\n\n## Result\n\nThe results will be saved in the `results\\` directory under `instantiation`, `execution`, and `dataflow`, and will be further stored in subdirectories based on the execution outcomes.\n\n!!! note\n The more detailed information of result can be found in [result](./result.md).\n\n## Quick Start\n\nWe prepare two cases to show the dataflow, which can be found in `dataflow\\tasks\\prefill`. So after installing required packages, you can type the following command in the command line:\n\n```\npython -m dataflow -dataflow\n```\n\nAnd you can see the hints showing in the terminal, which means the dataflow is working.\n\n### Structure of related files\n\nAfter the two tasks are finished, the task and output files would appear as follows:\n\n```bash\nUFO/\n├── dataflow/\n│ └── results/\n│ ├── saved_document/ \t# Directory for saved documents\n│ │ ├── bulleted.docx \t# Result of the \"bulleted\" task\n│ │ └── rotate.docx \t# Result of the \"rotate\" task\n│ ├── dataflow/ \t\t # Dataflow results directory\n│ │ ├── execution_pass/ \t# Successfully executed tasks\n│ │ │ ├── bulleted.json \t# Execution result for the \"bulleted\" task\n│ │ │ ├── rotate.json \t # Execution result for the \"rotate\" task\n│ │ │ └── ...\n└── ...\n```\n\nThe specific results can be referenced in the [result](./result.md) in JSON format along with example data.\n\n### Log files\n\nThe corresponding logs can be found in the directories `logs/bulleted` and `logs/rotate`, as shown below. Detailed logs for each workflow are recorded, capturing every step of the execution process.\n\n
\n \n
\n\n# Reference\n\n### AppEnum\n\n::: data_flow_controller.AppEnum\n\n### TaskObject\n\n::: data_flow_controller.TaskObject\n\n### DataFlowController\n\n::: data_flow_controller.DataFlowController\n"
},
{
"path": "documents/docs/ufo2/dataflow/result.md",
"content": "# Result\n\nThe results will be saved in the `\"dataflow/results\"` directory under `instantiation`, `execution`, and `dataflow`, and will be further stored in subdirectories based on the result.\n\nThe results are saved by validating the task information against a `schema` (`instantiation_schema` or `execution_schema.json` in the `\"dataflow/schema\"`) and determining the target directory based on the `task type` and its `evaluation status`, then storing the result in the appropriate location. The structure of the storage and the specific meaning of the schema are as follows.\n\n## Overall Result Struction\n\nThe structure of the results of the task is as below:\n\n```bash\nUFO/\n├── dataflow/ # Root folder for dataflow\n│ └── results/ # Directory for storing task processing results\n│ ├── saved_document/ # Directory for final document results\n│ ├── instantiation/ # Directory for instantiation results\n│ │ ├── instantiation_pass/ # Tasks successfully instantiated\n│ │ └── instantiation_fail/ # Tasks that failed instantiation\n│ ├── execution/ # Directory for execution results\n│ │ ├── execution_pass/ # Tasks successfully executed\n│ │ ├── execution_fail/ # Tasks that failed execution\n│ │ └── execution_unsure/ # Tasks with uncertain execution results\n│ ├── dataflow/ # Directory for dataflow results\n│ │ ├── execution_pass/ # Tasks successfully executed\n│ │ ├── execution_fail/ # Tasks that failed execution\n│ │ └── execution_unsure/ # Tasks with uncertain execution results\n│ └── ...\n└── ...\n```\n\n1. **General Description:**\n This directory structure organizes the results of task processing into specific categories, including `instantiation`, `execution`, and `dataflow` outcomes.\n2. **Instantiation:**\n\n - The `instantiation` directory contains subfolders for tasks that were successfully instantiated (`instantiation_pass`) and those that failed during instantiation (`instantiation_fail`).\n - This corresponds to the result of `instantiation_evaluation`, with the field name `\"judge\"`.\n3. **Execution:**\n\n - Results of task `execution` are stored under the `execution` directory, categorized into successful tasks (`execution_pass`), failed tasks (`execution_fail`), and tasks with uncertain outcomes (`execution_unsure`).\n - This corresponds to the `evaluation` result of `execute_flow`, with the field name `\"complete\"`.\n4. **Dataflow Results:**\n\n - The `dataflow` directory similarly stores the results of tasks based on the execution outcome: `execution_pass` for success, `execution_fail` for failure, or `execution_unsure` for uncertainty.\n - This corresponds to the `evaluation` result of `execute_flow`, with the field name `\"complete\"`.\n5. **Saved Documents:**\n Instantiated results are separately stored in the `saved_document` directory for easy access and reference.\n\n### Overall Description\n\nThe result data include `unique_id`，``app``, `original`, `execution_result`, `instantiation_result`, `time_cost`.\n\nThe result data includes the following fields:\n\n| **Field** | **Description** |\n| ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |\n| **`unique_id`** | A unique identifier for the task. |\n| **`app`** | The name of the application that processes the task. |\n| **`original`** | Contains details about the original task, including: |\n| **`original.original_task`** | A description of the original task. |\n| **`original.original_steps`** | A list of steps involved in the original task. |\n| **`execution_result`** | Stores the result of task `execution`, including any errors encountered and execution evaluation. |\n| **`instantiation_result`** | Provides details of the `instantiation`process, including: |\n| **`instantiation_result.choose_template`** | The template selection result and any associated errors. |\n| **`instantiation_result.prefill`** | Information about pre-filled task, including the instantiated request and plan. |\n| **`instantiation_result.instantiation_evaluation`** | Evaluation results of the instantiated task, including judgments and feedback. |\n| **`time_cost`** | Tracks the time taken for various stages of the process, such as template selection, pre-filling, and evaluation. |\n\n## Instantiation Result Schema\n\nThe instantiation schema in `\"dataflow/schema/instantiation_schema.json\"` defines the structure of a JSON object that is used to validate the results of task `instantiation`.\n\n---\n\n### **Schema Tabular Description**\n\n| **Field** | **Description** |\n| --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |\n| **`unique_id`** | A unique identifier for the task. |\n| **`app`** | The name of the application that processes the task. |\n| **`original`** | Contains details about the original task, including: |\n| **`original.original_task`** | A description of the original task. |\n| **`original.original_steps`** | A list of steps involved in the original task. |\n| **`execution_result`** | Stores the result of task execution, including any errors encountered and execution evaluation. |\n| **`execution_result.result`** | Indicates the execution result (or null if not applicable). |\n| **`execution_result.error`** | Details any errors encountered during task execution. |\n| **`instantiation_result`** | Provides details of the instantiation process, including: |\n| **`instantiation_result.choose_template`** | The template selection result and any associated errors. |\n| **`instantiation_result.prefill`** | Information about pre-filled tasks, including the instantiated request and plan. |\n| **`instantiation_result.prefill.result`** | Contains details of instantiated requests and plans. |\n| **`instantiation_result.prefill.result.instantiated_request`** | The instantiated task request. |\n| **`instantiation_result.prefill.result.instantiated_plan`** | Contains details of the instantiated steps. |\n| **`instantiation_result.prefill.result.instantiated_plan.step`** | The step sequence number. |\n| **`instantiation_result.prefill.result.instantiated_plan.subtask`** | The description of the subtask. |\n| **`instantiation_result.prefill.result.instantiated_plan.control_label`** | Control label for the step (or null if not applicable). |\n| **`instantiation_result.prefill.result.instantiated_plan.control_text`** | Contextual text for the step. |\n| **`instantiation_result.prefill.result.instantiated_plan.function`** | The function executed in this step. |\n| **`instantiation_result.prefill.result.instantiated_plan.args`** | Parameters required for the function. |\n| **`instantiation_result.prefill.error`** | Errors, if any, during the prefill process. |\n| **`instantiation_result.instantiation_evaluation`** | Evaluation results of the instantiated task, including judgments and feedback. |\n| **`instantiation_result.instantiation_evaluation.result`** | Contains detailed evaluation results. |\n| **`instantiation_result.instantiation_evaluation.result.judge`** | Indicates whether the evaluation passed. |\n| **`instantiation_result.instantiation_evaluation.result.thought`** | Feedback or observations from the evaluator. |\n| **`instantiation_result.instantiation_evaluation.result.request_type`** | Classification of the request type. |\n| **`instantiation_result.instantiation_evaluation.error`** | Errors, if any, during the evaluation. |\n| **`time_cost`** | Tracks the time taken for various stages of the process, such as template selection, pre-filling, and evaluation. |\n| **`time_cost.choose_template`** | Time taken for the template selection stage. |\n| **`time_cost.prefill`** | Time taken for the prefill stage. |\n| **`time_cost.instantiation_evaluation`** | Time taken for the evaluation stage. |\n| **`time_cost.total`** | Total time taken for the task. |\n\n---\n\n### Example Data\n\n```json\n{\n \"unique_id\": \"5\",\n \"app\": \"word\",\n \"original\": {\n \"original_task\": \"Turning lines of text into a bulleted list in Word\",\n \"original_steps\": [\n \"1. Place the cursor at the beginning of the line of text you want to turn into a bulleted list\",\n \"2. Click the Bullets button in the Paragraph group on the Home tab and choose a bullet style\"\n ]\n },\n \"execution_result\": {\n \"result\": null,\n \"error\": null\n },\n \"instantiation_result\": {\n \"choose_template\": {\n \"result\": \"dataflow\\\\results\\\\saved_document\\\\bulleted.docx\",\n \"error\": null\n },\n \"prefill\": {\n \"result\": {\n \"instantiated_request\": \"Turn the line of text 'text to edit' into a bulleted list in Word.\",\n \"instantiated_plan\": [\n {\n \"Step\": 1,\n \"Subtask\": \"Place the cursor at the beginning of the text 'text to edit'\",\n \"ControlLabel\": null,\n \"ControlText\": \"\",\n \"Function\": \"select_text\",\n \"Args\": {\n \"text\": \"text to edit\"\n }\n },\n {\n \"Step\": 2,\n \"Subtask\": \"Click the Bullets button in the Paragraph group on the Home tab\",\n \"ControlLabel\": null,\n \"ControlText\": \"Bullets\",\n \"Function\": \"click_input\",\n \"Args\": {\n \"button\": \"left\",\n \"double\": false\n }\n }\n ]\n },\n \"error\": null\n },\n \"instantiation_evaluation\": {\n \"result\": {\n \"judge\": true,\n \"thought\": \"The task is specific and involves a basic function in Word that can be executed locally without any external dependencies.\",\n \"request_type\": \"None\"\n },\n \"error\": null\n }\n },\n \"time_cost\": {\n \"choose_template\": 0.012,\n \"prefill\": 15.649,\n \"instantiation_evaluation\": 2.469,\n \"execute\": null,\n \"execute_eval\": null,\n \"total\": 18.130\n }\n}\n```\n\n## Execution Result Schema\n\nThe execution result schema in the `\"dataflow/schema/execution_schema.json\"` defines the structure of a JSON object that is used to validate the results of task `execution` or `dataflow`.\n\nThe **execution result schema** provides **comprehensive feedback on execution**, emphasizing key success metrics (`reason`, `sub_scores`, `complete`) recorded in the `result` field of `execution_result`.\n\nKey enhancements include:\n\n1. Each step in the `instantiated_plan` has been augmented with:\n\n - **`Success`**: Indicates if the step executed successfully (no errors).\n - **`MatchedControlText`**: Records the name of the last matched control.\n - **`ControlLabel`:** Be updated to reflect the final selected control.\n2. The **`execute`**、**`execute_eval`** and **`total`** in the **`time_cost`** field is updated.\n\n---\n\n### **Schema Tabular Description**\n\n| **Field** | **Description** |\n| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- |\n| **`unique_id`** | A unique identifier for the task. |\n| **`app`** | The name of the application that processes the task. |\n| **`original`** | Contains details about the original task, including: |\n| **`original.original_task`** | A description of the original task. |\n| **`original.original_steps`** | A list of steps involved in the original task. |\n| **`execution_result`** | Represents the result of task execution, including any errors encountered and execution evaluation. |\n| **`execution_result.result`** | Indicates the result of the task execution. |\n| **`execution_result.error`** | Indicates any errors that occurred during execution. |\n| **`instantiation_result`** | Provides details about the task instantiation, including: |\n| **`instantiation_result.choose_template.result`** | The template selection result. |\n| **`instantiation_result.choose_template.error`** | Errors, if any, during template selection. |\n| **`instantiation_result.prefill.result.instantiated_request`** | The instantiated task request. |\n| **`instantiation_result.prefill.result.instantiated_plan.Step`** | The step sequence number. |\n| **`instantiation_result.prefill.result.instantiated_plan.Subtask`** | The description of the subtask. |\n| **`instantiation_result.prefill.result.instantiated_plan.ControlLabel`** | Control label for the step. |\n| **`instantiation_result.prefill.result.instantiated_plan.ControlText`** | Contextual text for the step. |\n| **`instantiation_result.prefill.result.instantiated_plan.Function`** | The function executed in this step. |\n| **`instantiation_result.prefill.result.instantiated_plan.Args`** | Parameters required for the function. |\n| **`instantiation_result.prefill.result.instantiated_plan.Success`** | Indicates if the step was executed successfully without errors. |\n| **`instantiation_result.prefill.result.instantiated_plan.MatchedControlText`** | The final matched control text in the execution flow. |\n| **`instantiation_result.prefill.error`** | Errors, if any, during the prefill process. |\n| **`instantiation_result.instantiation_evaluation.result.judge`** | Indicates whether the evaluation passed. |\n| **`instantiation_result.instantiation_evaluation.result.thought`** | Feedback or observations from the evaluator. |\n| **`instantiation_result.instantiation_evaluation.result.request_type`** | Classification of the request type. |\n| **`instantiation_result.instantiation_evaluation.error`** | Errors, if any, during the evaluation. |\n| **`time_cost`** | Tracks the time taken for various stages of the process, including: |\n| **`time_cost.choose_template`** | Time taken for the template selection stage. |\n| **`time_cost.prefill`** | Time taken for the prefill stage. |\n| **`time_cost.instantiation_evaluation`** | Time taken for the evaluation stage. |\n| **`time_cost.execute`** | Time taken for the execute stage. |\n| **`time_cost.execute_eval`** | Time taken for the execute evaluation stage. |\n| **`time_cost.total`** | Total time taken for the task. |\n\n### Example Data\n\n```json\n{\n \"unique_id\": \"5\",\n \"app\": \"word\",\n \"original\": {\n \"original_task\": \"Turning lines of text into a bulleted list in Word\",\n \"original_steps\": [\n \"1. Place the cursor at the beginning of the line of text you want to turn into a bulleted list\",\n \"2. Click the Bullets button in the Paragraph group on the Home tab and choose a bullet style\"\n ]\n },\n \"execution_result\": {\n \"result\": {\n \"reason\": \"The agent successfully selected the text 'text to edit' and then clicked on the 'Bullets' button in the Word application. The final screenshot shows that the text 'text to edit' has been converted into a bulleted list.\",\n \"sub_scores\": {\n \"text selection\": \"yes\",\n \"bulleted list conversion\": \"yes\"\n },\n \"complete\": \"yes\"\n },\n \"error\": null\n },\n \"instantiation_result\": {\n \"choose_template\": {\n \"result\": \"dataflow\\\\results\\\\saved_document\\\\bulleted.docx\",\n \"error\": null\n },\n \"prefill\": {\n \"result\": {\n \"instantiated_request\": \"Turn the line of text 'text to edit' into a bulleted list in Word.\",\n \"instantiated_plan\": [\n {\n \"Step\": 1,\n \"Subtask\": \"Place the cursor at the beginning of the text 'text to edit'\",\n \"ControlLabel\": null,\n \"ControlText\": \"\",\n \"Function\": \"select_text\",\n \"Args\": {\n \"text\": \"text to edit\"\n },\n \"Success\": true,\n \"MatchedControlText\": null\n },\n {\n \"Step\": 2,\n \"Subtask\": \"Click the Bullets button in the Paragraph group on the Home tab\",\n \"ControlLabel\": \"61\",\n \"ControlText\": \"Bullets\",\n \"Function\": \"click_input\",\n \"Args\": {\n \"button\": \"left\",\n \"double\": false\n },\n \"Success\": true,\n \"MatchedControlText\": \"Bullets\"\n }\n ]\n },\n \"error\": null\n },\n \"instantiation_evaluation\": {\n \"result\": {\n \"judge\": true,\n \"thought\": \"The task is specific and involves a basic function in Word that can be executed locally without any external dependencies.\",\n \"request_type\": \"None\"\n },\n \"error\": null\n }\n },\n \"time_cost\": {\n \"choose_template\": 0.012,\n \"prefill\": 15.649,\n \"instantiation_evaluation\": 2.469,\n \"execute\": 5.824,\n \"execute_eval\": 8.702,\n \"total\": 43.522\n }\n}\n```\n"
},
{
"path": "documents/docs/ufo2/dataflow/windows_app_env.md",
"content": "# WindowsAppEnv\n\nThe usage scenarios for the `WindowsAppEnv` class are as follows:\n\n* **Opening a specified document.**\n* **Matching document windows** using different strategies (`contains`, `fuzzy`, and `regex`).\n* **Matching the controls** required for each step in the instantiated plan using various strategies (`contains`, `fuzzy`, and `regex`).\n* **Closing a specified document.**\n\nThe following sections provide a detailed explanation of the **matching strategies for windows and controls**, as well as their usage methods.\n\n## Matching Strategies\n\nIn the `WindowsAppEnv` class, matching strategies are rules that determine how to match `window` or `control` names with a given document name or target text. Based on the configuration file, three different matching strategies can be selected: `contains`, `fuzzy`, and `regex`.\n\n* `Contains` matching is the simplest strategy, suitable when the window and document names match exactly.\n* `Fuzzy` matching is more flexible and can match even when there are spelling errors or partial matches between the window title and document name.\n* `Regex` matching offers the most flexibility, ideal for complex matching patterns in window titles.\n\n### 1. **Window Matching** Example\n\nThe method `find_matching_window` is responsible for matching windows based on the configured matching strategy. Here's how you can use it to find a window by providing a document name:\n\n#### Example:\n\n```python\n# Initialize your application object (assuming app_object is already defined)\napp_env = WindowsAppEnv(app_object)\n\n# Define the document name you're looking for\ndoc_name = \"example_document_name\"\n\n# Call find_matching_window to find the window that matches the document name\nmatching_window = app_env.find_matching_window(doc_name)\n\nif matching_window:\n print(f\"Found matching window: {matching_window.element_info.name}\")\nelse:\n print(\"No matching window found.\")\n```\n\n#### Explanation:\n\n- `app_env.find_matching_window(doc_name)` will search through all open windows and match the window title using the strategy defined in the configuration (contains, fuzzy, or regex).\n- If a match is found, the `matching_window` object will contain the matched window, and you can print the window's name.\n- If no match is found, it will return `None`.\n\n### 2. **Control Matching** Example\n\nTo find a matching control within a window, you can use the `find_matching_controller` method. This method requires a dictionary of filtered controls and a control text to match against.\n\n#### Example:\n\n```python\n# Initialize your application object (assuming app_object is already defined)\napp_env = WindowsAppEnv(app_object)\n\n# Define a filtered annotation dictionary of controls (control_key, control_object)\n# Here, we assume you have a dictionary of UIAWrapper controls from a window.\nfiltered_annotation_dict = {\n 1: some_control_1, # Example control objects\n 2: some_control_2, # Example control objects\n}\n\n# Define the control text you're searching for\ncontrol_text = \"submit_button\"\n\n# Call find_matching_controller to find the best match\ncontroller_key, control_selected = app_env.find_matching_controller(filtered_annotation_dict, control_text)\n\nif control_selected:\n print(f\"Found matching control with key {controller_key}: {control_selected.window_text()}\")\nelse:\n print(\"No matching control found.\")\n```\n\n#### Explanation:\n\n- `filtered_annotation_dict` is a dictionary where the key represents the control's ID and the value is the control object (`UIAWrapper`).\n- `control_text` is the text you're searching for within those controls.\n- `app_env.find_matching_controller(filtered_annotation_dict, control_text)` will calculate the matching score for each control based on the defined strategy and return the control with the highest match score.\n- If a match is found, it will return the control object (`control_selected`) and its key (`controller_key`), which can be used for further interaction.\n\n# Reference\n\n::: env.env_manager.WindowsAppEnv\n"
},
{
"path": "documents/docs/ufo2/evaluation/benchmark/osworld.md",
"content": "# 🧩 Setting up UFO with OSWorld (Windows)\n\n\nOSWorld is a benchmark suite designed to evaluate the performance of AI agents in real-world scenarios. We select the 49 cases from the original OSWorld benchmark that are compatible with the Windows platform, renamed as OSWorld-W. The tasks cover a wide range of functionalities and interactions that users typically perform on their computers, including Office 365 and browser.\n\n---\n\n## 💻 Deployment Guide (WSL Recommended)\n\n> We strongly recommend reviewing the [original WAA deployment guide](https://github.com/microsoft/WindowsAgentArena) beforehand. The instructions below assume you are familiar with the original setup.\n\n---\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/nice-mee/WindowsAgentArena.git\n```\n\n> 💡 *To run OSWorld cases, switch to the dedicated development branch:*\n```bash\ngit checkout osworld\n```\n\nCreate a `config.json` file in the repo root with a placeholder key (UFO will override this):\n\n```json\n{\n \"OPENAI_API_KEY\": \"placeholder\"\n}\n```\n\n---\n\n### 2. Build the Docker Image\n\nNavigate to the `scripts` directory and build the Docker image:\n\n```bash\ncd scripts\nchmod +x build-container-image.sh prepare-agents.sh # (if needed)\n./build-container-image.sh --build-base-image true\n```\n\nThis will generate the `windowsarena/winarena:latest` image using the latest codebase in `src/`.\n\n---\n\n### 3. Integrate UFO\n\n1. Configure UFO via `ufo/config/config.json` (see [UFO repo](https://github.com/microsoft/UFO) for details).\n2. Copy the entire `ufo` folder into the WAA container client directory:\n\n```bash\ncp -r src/win-arena-container/vm/setup/mm_agents/UFO/ufo src/win-arena-container/client/\n```\n\n> ⚠️ Python 3.9 Compatibility Fix \n> In `ufo/llm/openai.py`, swap the order of `@staticmethod` and `@functools.lru_cache()` to prevent issues due to a known Python 3.9 bug.\n\n---\n\n### 4. Prepare the Windows 11 Virtual Machine\n\n#### 4.1 Download the ISO\n\n1. Go to the [Microsoft Evaluation Center](https://info.microsoft.com/ww-landing-windows-11-enterprise.html)\n2. Accept the terms and download **Windows 11 Enterprise Evaluation (English, 90-day trial)** (~6GB)\n3. Rename the file to `setup.iso` and place it in:\n\n```\nWindowsAgentArena/src/win-arena-container/vm/image\n```\n\n#### 4.2 Generate the Golden Image Snapshot\n\nPrepare the Windows VM snapshot (a fully provisioned 30GB image):\n\n```bash\ncd ./scripts\n./run-local.sh --mode dev --prepare-image true\n```\n\n> ⚠️ **Do not interact with the VM during preparation.** It will shut down automatically when complete.\n\nThe golden image will be saved in:\n\n```\nWindowsAgentArena/src/win-arena-container/vm/storage\n```\n\n---\n\n### 5. Initial Run (First Boot Setup)\n\nLaunch the environment:\n\n```bash\n./run-local.sh --mode dev --json-name \"evaluation_examples_windows/test_custom.json\" --agent UFO --agent-settings '{\"llm_type\": \"azure\", \"llm_endpoint\": \"https://cloudgpt-openai.azure-api.net/openai/deployments/gpt-4o-20240513/chat/completions?api-version=2024-04-01-preview\", \"llm_auth\": {\"type\": \"api-key\", \"token\": \"\"}}'\n```\n\nOnce the VM boots:\n\n1. **Do not** enter the device code (this keeps the WAA server alive indefinitely).\n2. Visit `http://localhost:8006` and perform the following setup actions:\n - Disable **Windows Firewall**\n - Open **Google Chrome** and complete initial setup\n - Open **VLC** and complete initial setup\n - Activate Office 365 (Word, Excel, PowerPoint, etc.) with a Microsoft account (use a temporary one if needed).\n\nAfter setup:\n\n- Stop the client\n- Backup the golden image from the `storage` folder\n\n---\n\n## 🧪 Running Experiments\n\nBefore each experiment:\n\n1. Replace the VM image with your prepared golden snapshot\n2. Clear any previous UFO logs\n\nThen run:\n\n```bash\n./run-local.sh --mode dev --json-name \"evaluation_examples_windows/test_full.json\" --agent UFO --agent-settings '{\"llm_type\": \"azure\", \"llm_endpoint\": \"https://cloudgpt-openai.azure-api.net/openai/deployments/gpt-4o-20240513/chat/completions?api-version=2024-04-01-preview\", \"llm_auth\": {\"type\": \"api-key\", \"token\": \"\"}}'\n```\n\n!!!note\n > - `test_full.json`: Contains all test cases where UIA is available. \n > - `test_all.json`: Includes all test cases, even those incompatible with UIA. \n > - Use `test_full.json` if you're **not** using OmniParser.\n---"
},
{
"path": "documents/docs/ufo2/evaluation/benchmark/overview.md",
"content": "# Benchmark Overview\n\nUFO² is rigorously benchmarked on two publicly‑available live‑task suites:\n\n| Benchmark | Scope |\n|-----------|-------|\n| [**Windows Agent Arena (WAA)**](./windows_agent_arena.md) | 154 real Windows tasks across 15 applications (Office, Edge, File Explorer, VS Code, …) | \n| [**OSWorld (Windows)**](./osworld.md) | 49 cross‑application tasks that mix Office 365, browser and system utilities | \n\nThe integration of these benchmarks into UFO² is in separate repositories. Please follow the above documents for more details.\n\n!!!note\n we have revised the verification scripts of some cases to ensure the correctness of the results."
},
{
"path": "documents/docs/ufo2/evaluation/benchmark/windows_agent_arena.md",
"content": "# 🧩 Setting up UFO with Windows Agent Arena (WAA)\n\nWindows Agent Arena (WAA) is a benchmark suite designed to evaluate the performance of AI agents in executing real-world tasks on Windows operating systems. It consists of 154 tasks across 15 applications, including Microsoft Office, Edge, File Explorer, and VS Code. The tasks are designed to cover a wide range of functionalities and interactions that users typically perform on their computers.\n\nThis repository provides a modified version of [**Windows Agent Arena (WAA) 🪟**](https://github.com/microsoft/WindowsAgentArena), a scalable platform for benchmarking and evaluating multimodal desktop AI agents. This customized fork integrates with [**UFO**](https://github.com/microsoft/UFO), a UI-focused automation agent for Windows OS.\n\n---\n\n## 💻 Deployment Guide (WSL Recommended)\n\n> We strongly recommend reviewing the [original WAA deployment guide](https://github.com/microsoft/WindowsAgentArena) beforehand. The instructions below assume you are familiar with the original setup.\n\n---\n\n### 1. Clone the Repository\n\n```bash\ngit clone https://github.com/nice-mee/WindowsAgentArena.git\n```\n\n> 💡 *To run OSWorld cases, switch to the dedicated development branch:*\n```bash\ngit checkout 2020-qqtcg/dev\n```\n\nCreate a `config.json` file in the repo root with a placeholder key (UFO will override this):\n\n```json\n{\n \"OPENAI_API_KEY\": \"placeholder\"\n}\n```\n\n---\n\n### 2. Build the Docker Image\n\nNavigate to the `scripts` directory and build the Docker image:\n\n```bash\ncd scripts\nchmod +x build-container-image.sh prepare-agents.sh # (if needed)\n./build-container-image.sh --build-base-image true\n```\n\nThis will generate the `windowsarena/winarena:latest` image using the latest codebase in `src/`.\n\n---\n\n### 3. Integrate UFO\n\n1. Configure UFO via `ufo/config/config.json` (see [UFO repo](https://github.com/microsoft/UFO) for details).\n2. Copy the entire `ufo` folder into the WAA container client directory:\n\n```bash\ncp -r src/win-arena-container/vm/setup/mm_agents/UFO/ufo src/win-arena-container/client/\n```\n\n> ⚠️ Python 3.9 Compatibility Fix \n> In `ufo/llm/openai.py`, swap the order of `@staticmethod` and `@functools.lru_cache()` to prevent issues due to a known Python 3.9 bug.\n\n---\n\n### 4. Prepare the Windows 11 Virtual Machine\n\n#### 4.1 Download the ISO\n\n1. Go to the [Microsoft Evaluation Center](https://info.microsoft.com/ww-landing-windows-11-enterprise.html)\n2. Accept the terms and download **Windows 11 Enterprise Evaluation (English, 90-day trial)** (~6GB)\n3. Rename the file to `setup.iso` and place it in:\n\n```\nWindowsAgentArena/src/win-arena-container/vm/image\n```\n\n#### 4.2 Generate the Golden Image Snapshot\n\nPrepare the Windows VM snapshot (a fully provisioned 30GB image):\n\n```bash\ncd ./scripts\n./run-local.sh --mode dev --prepare-image true\n```\n\n> ⚠️ **Do not interact with the VM during preparation.** It will shut down automatically when complete.\n\nThe golden image will be saved in:\n\n```\nWindowsAgentArena/src/win-arena-container/vm/storage\n```\n\n---\n\n### 5. Initial Run (First Boot Setup)\n\nLaunch the environment:\n\n```bash\n./run-local.sh --mode dev --json-name \"evaluation_examples_windows/test_custom.json\" --agent UFO --agent-settings '{\"llm_type\": \"azure\", \"llm_endpoint\": \"https://cloudgpt-openai.azure-api.net/openai/deployments/gpt-4o-20240513/chat/completions?api-version=2024-04-01-preview\", \"llm_auth\": {\"type\": \"api-key\", \"token\": \"\"}}'\n```\n\nOnce the VM boots:\n\n1. **Do not** enter the device code (this keeps the WAA server alive indefinitely).\n2. Visit `http://localhost:8006` and perform the following setup actions:\n - Disable **Windows Firewall**\n - Open **Google Chrome** and complete initial setup\n - Open **VLC** and complete initial setup\n\nAfter setup:\n\n- Stop the client\n- Backup the golden image from the `storage` folder\n\n---\n\n## 🧪 Running Experiments\n\nBefore each experiment:\n\n1. Replace the VM image with your prepared golden snapshot\n2. Clear any previous UFO logs\n\nThen run:\n\n```bash\n./run-local.sh --mode dev --json-name \"evaluation_examples_windows/test_full.json\" --agent UFO --agent-settings '{\"llm_type\": \"azure\", \"llm_endpoint\": \"https://cloudgpt-openai.azure-api.net/openai/deployments/gpt-4o-20240513/chat/completions?api-version=2024-04-01-preview\", \"llm_auth\": {\"type\": \"api-key\", \"token\": \"\"}}'\n```\n\n!!!note\n > - `test_full.json`: Contains all test cases where UIA is available. \n > - `test_all.json`: Includes all test cases, even those incompatible with UIA. \n > - Use `test_full.json` if you're **not** using OmniParser.\n---"
},
{
"path": "documents/docs/ufo2/evaluation/evaluation_agent.md",
"content": "# EvaluationAgent\n\nThe `EvaluationAgent` evaluates whether a `Session` or `Round` has been successfully completed by assessing the performance of the `HostAgent` and `AppAgent` in fulfilling user requests. Configuration options are available in `config/ufo/system.yaml`. For more details, refer to the [System Configuration Guide](../../configuration/system/system_config.md).\n\nThe `EvaluationAgent` is fully LLM-driven and conducts evaluations based on action trajectories and screenshots. Since LLM-based evaluation may not be 100% accurate, the results should be used as guidance rather than absolute truth.\n\n\n\n## Configuration\n\nConfigure the `EvaluationAgent` in `config/ufo/system.yaml`:\n\n| Configuration Option | Description | Type | Default Value |\n|---------------------------|-----------------------------------------------|---------|---------------|\n| `EVA_SESSION` | Whether to evaluate the entire session. | Boolean | True |\n| `EVA_ROUND` | Whether to evaluate each round. | Boolean | False |\n| `EVA_ALL_SCREENSHOTS` | Whether to include all screenshots in evaluation. If `False`, only the first and last screenshots are used. | Boolean | True |\n\n## Evaluation Process\n\nThe `EvaluationAgent` uses a Chain-of-Thought (CoT) mechanism to:\n\n1. Decompose the evaluation into multiple sub-goals based on the user request\n2. Evaluate each sub-goal separately\n3. Aggregate the sub-scores to determine the overall completion status\n\n```mermaid\ngraph TD\n A[User Request] --> B[EvaluationAgent]\n C[Action Trajectories] --> B\n D[Screenshots] --> B\n E[APIs Description] --> B\n \n B --> F[CoT: Decompose into Sub-goals]\n F --> G[Evaluate Sub-goal 1]\n F --> H[Evaluate Sub-goal 2]\n F --> I[Evaluate Sub-goal N]\n \n G --> J[Aggregate Sub-scores]\n H --> J\n I --> J\n \n J --> K{Overall Completion Status}\n K -->|yes| L[Task Completed]\n K -->|no| M[Task Failed]\n K -->|unsure| N[Uncertain Result]\n \n B --> O[Generate Detailed Reason]\n O --> P[Evaluation Report]\n J --> P\n```\n\n### Inputs\n\nThe `EvaluationAgent` takes the following inputs:\n\n| Input | Description | Type |\n| --- | --- | --- |\n| User Request | The user's request to be evaluated. | String |\n| APIs Description | Description of the APIs (tools) used during execution. | String |\n| Action Trajectories | Action trajectories executed by the `HostAgent` and `AppAgent`, including subtask, step, observation, thought, plan, comment, action, and application. | List of Dictionaries |\n| Screenshots | Screenshots captured during execution. | List of Images |\n\nThe input construction is handled by the `EvaluationAgentPrompter` class in `ufo/prompter/eva_prompter.py`.\n\n### Outputs\n\nThe `EvaluationAgent` generates the following outputs:\n\n| Output | Description | Type |\n| --- | --- | --- |\n| reason | Detailed reasoning for the judgment based on screenshot analysis and execution trajectory. | String |\n| sub_scores | List of sub-scoring points evaluating different aspects of the task. Each sub-score contains a name and evaluation result. | List of Dictionaries |\n| complete | Overall completion status: `yes`, `no`, or `unsure`. | String |\n\nExample output:\n\n```json\n{\n \"reason\": \"The agent successfully completed the task of sending 'hello' to Zac on Microsoft Teams. \n The initial screenshot shows the Microsoft Teams application with the chat window of Chaoyun Zhang open. \n The agent then focused on the chat window, input the message 'hello', and clicked the Send button. \n The final screenshot confirms that the message 'hello' was sent to Zac.\", \n \"sub_scores\": [\n { \"name\": \"correct application focus\", \"evaluation\": \"yes\" }, \n { \"name\": \"correct message input\", \"evaluation\": \"yes\" }, \n { \"name\": \"message sent successfully\", \"evaluation\": \"yes\" }\n ], \n \"complete\": \"yes\"\n}\n```\n\nEvaluation logs are saved in `logs/{task_name}/evaluation.log`.\n\n## See Also\n\n- [System Configuration](../../configuration/system/system_config.md) - Configure evaluation settings\n- [Evaluation Logs](logs/evaluation_logs.md) - Understanding evaluation logs structure\n- [Logs Overview](logs/overview.md) - Complete guide to UFO logging system\n- [Benchmark Overview](benchmark/overview.md) - Benchmarking UFO performance using evaluation results\n\n## Reference\n\n:::agents.agent.evaluation_agent.EvaluationAgent\n\n"
},
{
"path": "documents/docs/ufo2/evaluation/logs/evaluation_logs.md",
"content": "# Evaluation Logs\n\nThe evaluation log stores task completion assessment results from the `EvaluationAgent`. The log is saved as `evaluation.log` in JSON format, containing a single entry that evaluates the entire session.\n\n## Log Structure\n\nThe evaluation log contains the following fields:\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `complete` | Overall completion status: `yes`, `no`, or `unsure` | String |\n| `sub_scores` | Breakdown of evaluation into sub-goals, each with name and evaluation status | List of Dictionaries |\n| `reason` | Detailed justification based on screenshots and execution trajectory | String |\n| `level` | Evaluation scope (e.g., `session`) | String |\n| `request` | Original user request being evaluated | String |\n| `type` | Log entry type, set to `evaluation_result` | String |\n\n## Sub-score Structure\n\nEach item in `sub_scores` contains:\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `name` | Name of the sub-goal being evaluated | String |\n| `evaluation` | Completion status: `yes`, `no`, or `unsure` | String |\n\n## Example\n\n```json\n{\n \"complete\": \"yes\",\n \"sub_scores\": [\n {\n \"name\": \"Open application\",\n \"evaluation\": \"yes\"\n },\n {\n \"name\": \"Complete data entry\",\n \"evaluation\": \"yes\"\n }\n ],\n \"reason\": \"All sub-tasks completed successfully. Screenshots show the application was opened and data was correctly entered.\",\n \"level\": \"session\",\n \"request\": \"Open the application and enter data\",\n \"type\": \"evaluation_result\"\n}\n\n\n"
},
{
"path": "documents/docs/ufo2/evaluation/logs/markdown_log_viewer.md",
"content": "# Markdown Log Viewer\n\nUFO provides a Markdown-formatted log viewer that consolidates all execution data into a readable, structured document. This format is ideal for debugging, analysis, and documentation.\n\n## Configuration\n\nEnable Markdown log generation in `config_dev.yaml`:\n\n```yaml\nLOG_TO_MARKDOWN: true\n```\n\n## Output\n\n**File location:** `logs/{task_name}/output.md`\n\nThe generated Markdown file includes:\n\n- Session overview and metadata\n- Step-by-step execution timeline\n- Agent responses and reasoning\n- Screenshots embedded inline\n- Evaluation results\n\n## Use Cases\n\n**Debugging:** Quickly trace through execution flow with visual context\n\n**Documentation:** Share execution logs with human-readable formatting\n\n**Analysis:** Review agent decision-making process with screenshots\n\n**Reporting:** Generate execution reports for evaluation or review\n\n## Implementation\n\nThe Markdown log is automatically generated at session end by the `Trajectory` class (located in `ufo/trajectory/parser.py`), which parses `response.log` and combines it with screenshots and other artifacts.\n"
},
{
"path": "documents/docs/ufo2/evaluation/logs/overview.md",
"content": "# UFO Logs\n\nUFO generates comprehensive logs for debugging, analysis, and evaluation. Understanding these logs is essential for diagnosing issues and improving agent performance.\n\n## Log Types\n\n| Log Type | Description | Location |\n| --- | --- | --- |\n| [Request Log](./request_logs.md) | LLM prompt requests at each step | `logs/{task_name}/request.log` |\n| [Step Log](./step_logs.md) | Agent responses and execution details | `logs/{task_name}/response.log` |\n| [Evaluation Log](./evaluation_logs.md) | Task evaluation results | `logs/{task_name}/evaluation.log` |\n| [Screenshots](./screenshots_logs.md) | UI screenshots and visual captures | `logs/{task_name}/` |\n| [UI Tree](./ui_tree_logs.md) | Application UI structure data | `logs/{task_name}/ui_tree/` |\n\nAll logs are stored in the `logs/{task_name}` directory, where `{task_name}` is auto-generated based on timestamp."
},
{
"path": "documents/docs/ufo2/evaluation/logs/request_logs.md",
"content": "# Request Logs\n\nThe request log stores all prompt messages sent to LLMs during execution. Each line is a JSON entry representing one LLM request at a specific step.\n\n## Location\n\n```\nlogs/{task_name}/request.log\n```\n\n## Log Fields\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `step` | Step number in the session | Integer |\n| `prompt` | Complete prompt message sent to the LLM | Dictionary/List |\n\n## Reading Request Logs\n\n```python\nimport json\n\nwith open('logs/{task_name}/request.log', 'r') as f:\n for line in f:\n log = json.loads(line)\n print(f\"Step {log['step']}: {log['prompt']}\")\n```\n\nThe request log is useful for:\n\n- Debugging LLM interactions\n- Understanding what context was provided at each step\n- Analyzing prompt effectiveness\n- Reproducing agent behavior\n "
},
{
"path": "documents/docs/ufo2/evaluation/logs/screenshots_logs.md",
"content": "# Screenshot Logs\n\nUFO captures screenshots at every step for debugging and evaluation purposes. All screenshots are stored in the `logs/{task_name}/` directory.\n\n## Screenshot Types\n\n### 1. Clean Screenshots\n\nUnmodified screenshots of the desktop or application window.\n\n**File naming:**\n\n- Step screenshots: `action_step{step_number}.png`\n- Subtask completion: `action_round_{round_id}_sub_round_{sub_task_id}_final.png`\n- Round completion: `action_round_{round_id}_final.png`\n- Session completion: `action_step_final.png`\n\n**Example:**\n\n
\n \n
\n\n### 2. Annotated Screenshots\n\nScreenshots with UI controls labeled using the [Set-of-Mark](https://arxiv.org/pdf/2310.11441) paradigm. Each interactive control is marked with a number for reference.\n\n**File naming:** `action_step{step_number}_annotated.png`\n\n**Example:**\n\n
\n \n
\n\nOnly control types configured in `CONTROL_LIST` (in `config_dev.yaml`) are annotated. Different control types use different colors, configurable via `ANNOTATION_COLORS`.\n\n### 3. Concatenated Screenshots\n\nClean and annotated screenshots placed side-by-side for comparison.\n\n**File naming:** `action_step{step_number}_concat.png`\n\n**Example:**\n\n
\n \n
\n\nConfigure whether to feed concatenated or separate screenshots to LLMs using `CONCAT_SCREENSHOT` in `config_dev.yaml`.\n\n### 4. Selected Control Screenshots\n\nClose-up view of the control element selected for interaction in the previous step.\n\n**File naming:** `action_step{step_number}_selected_controls.png`\n\n**Example:**\n\n
\n \n
\n\nEnable/disable sending selected control screenshots to LLM using `INCLUDE_LAST_SCREENSHOT` in `config_dev.yaml`."
},
{
"path": "documents/docs/ufo2/evaluation/logs/step_logs.md",
"content": "# Step Logs\n\nThe step log captures agent responses and execution details at every step. Each line in `response.log` is a JSON entry representing one agent action.\n\n## Location\n\n```\nlogs/{task_name}/response.log\n```\n\n## HostAgent Logs\n\n### LLM Response Fields\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `observation` | Desktop screenshot analysis and current state | String |\n| `thought` | Reasoning process for task decomposition | String |\n| `current_subtask` | Subtask to be executed by AppAgent | String |\n| `message` | Instructions and context for AppAgent | List of Strings |\n| `control_label` | Index of selected application | String |\n| `control_text` | Name of selected application | String |\n| `plan` | Future subtasks after current one | List of Strings |\n| `status` | Agent state: `FINISH`, `CONTINUE`, `PENDING`, or `ASSIGN` | String |\n| `comment` | User-facing summary or progress update | String |\n| `questions` | Questions requiring user clarification | List of Strings |\n| `function` | System command to execute (optional) | String |\n\n### Additional Metadata\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `step` | Global step number in session | Integer |\n| `round_step` | Step number within current round | Integer |\n| `agent_step` | Step number for this agent instance | Integer |\n| `round_num` | Current round number | Integer |\n| `request` | Original user request | String |\n| `agent_type` | Set to `HostAgent` | String |\n| `agent_name` | Agent instance name | String |\n| `application` | Application process name | String |\n| `cost` | LLM cost for this step | Float |\n| `result` | Execution results | String |\n| `screenshot_clean` | Clean desktop screenshot path | String |\n| `screenshot_annotated` | Annotated screenshot path | String |\n| `screenshot_concat` | Concatenated screenshot path | String |\n| `screenshot_selected_control` | Selected control screenshot path | String |\n| `time_cost` | Time spent on each processing phase | Dictionary |\n\n## AppAgent Logs\n\n### LLM Response Fields\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `observation` | Application UI analysis and status | String |\n| `thought` | Reasoning for next action | String |\n| `control_label` | Index of selected control element | String |\n| `control_text` | Name of selected control element | String |\n| `action` | Action details including function and arguments | Dictionary or List |\n| `status` | Agent state (CONTINUE, FINISH, etc.) | String |\n| `plan` | Planned steps after current action | List of Strings |\n| `comment` | Progress summary or completion notes | String |\n| `save_screenshot` | Screenshot save configuration | Dictionary |\n\n### Additional Metadata\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `step` | Global step number in session | Integer |\n| `round_step` | Step number within current round | Integer |\n| `agent_step` | Step number for this agent instance | Integer |\n| `round_num` | Current round number | Integer |\n| `subtask` | Subtask assigned by HostAgent | String |\n| `subtask_index` | Index of subtask in current round | Integer |\n| `action_type` | Type of action performed | String |\n| `request` | Original user request | String |\n| `agent_type` | Set to `AppAgent` | String |\n| `agent_name` | Agent instance name | String |\n| `application` | Application process name | String |\n| `cost` | LLM cost for this step | Float |\n| `result` | Execution results | String |\n| `screenshot_clean` | Clean application screenshot path | String |\n| `screenshot_annotated` | Annotated screenshot path | String |\n| `screenshot_concat` | Concatenated screenshot path | String |\n| `time_cost` | Time spent on each processing phase | Dictionary |\n\n## Reading Step Logs\n\n```python\nimport json\n\nwith open('logs/{task_name}/response.log', 'r') as f:\n for line in f:\n log = json.loads(line)\n print(f\"Step {log['step']} - Agent: {log['agent_type']}\")\n print(f\"Thought: {log['thought']}\")\n```\n"
},
{
"path": "documents/docs/ufo2/evaluation/logs/ui_tree_logs.md",
"content": "# UI Tree Logs\n\nUFO can capture the complete UI control tree of application windows at every step. This structured data represents the hierarchical UI layout and is useful for analysis and debugging.\n\n## Configuration\n\nEnable UI tree logging by setting `SAVE_UI_TREE: true` in `config_dev.yaml`.\n\n**Location:** `logs/{task_name}/ui_tree/`\n\n**File naming:** `step_{step_number}.json`\n\n## Example\n \n```json\n{\n \"id\": \"node_0\",\n \"name\": \"Mail - Chaoyun Zhang - Outlook\",\n \"control_type\": \"Window\",\n \"rectangle\": {\n \"left\": 628,\n \"top\": 258,\n \"right\": 3508,\n \"bottom\": 1795\n },\n \"adjusted_rectangle\": {\n \"left\": 0,\n \"top\": 0,\n \"right\": 2880,\n \"bottom\": 1537\n },\n \"relative_rectangle\": {\n \"left\": 0.0,\n \"top\": 0.0,\n \"right\": 1.0,\n \"bottom\": 1.0\n },\n \"level\": 0,\n \"children\": [\n {\n \"id\": \"node_1\",\n \"name\": \"\",\n \"control_type\": \"Pane\",\n \"rectangle\": {\n \"left\": 3282,\n \"top\": 258,\n \"right\": 3498,\n \"bottom\": 330\n },\n \"adjusted_rectangle\": {\n \"left\": 2654,\n \"top\": 0,\n \"right\": 2870,\n \"bottom\": 72\n },\n \"relative_rectangle\": {\n \"left\": 0.9215277777777777,\n \"top\": 0.0,\n \"right\": 0.9965277777777778,\n \"bottom\": 0.0468445022771633\n },\n \"level\": 1,\n \"children\": []\n }\n ]\n}\n```\n\n\n## Field Reference\n\n| Field | Description | Type |\n| --- | --- | --- |\n| `id` | Unique node identifier in the tree | String |\n| `name` | Control element name/text | String |\n| `control_type` | UI element type (Window, Button, Edit, etc.) | String |\n| `rectangle` | Absolute screen coordinates | Dictionary |\n| `adjusted_rectangle` | Coordinates relative to window | Dictionary |\n| `relative_rectangle` | Normalized coordinates (0.0-1.0) | Dictionary |\n| `level` | Depth in the UI tree hierarchy | Integer |\n| `children` | Child UI elements | List |\n\n### Rectangle Structure\n\nAll rectangle fields contain:\n\n```json\n{\n \"left\": 0,\n \"top\": 0,\n \"right\": 100,\n \"bottom\": 100\n}\n```\n\n## Usage\n\nUI tree logs enable:\n\n- Understanding application structure\n- Analyzing control element hierarchy\n- Debugging control selection issues\n- Training ML models on UI data\n\n!!! note \"Performance Impact\"\n Saving UI trees increases execution latency. Disable when not needed for data collection.\n\n## Reference\n\n:::automator.ui_control.ui_tree.UITree"
},
{
"path": "documents/docs/ufo2/host_agent/commands.md",
"content": "# HostAgent Command System\n\nHostAgent executes desktop-level commands through the **MCP (Model Context Protocol)** system. Commands are dynamically provided by MCP servers and executed through the `CommandDispatcher` interface. This document describes the MCP configuration for HostAgent commands.\n\n---\n\n## Command Execution Architecture\n\n```mermaid\ngraph TB\n HostAgent[HostAgent] --> Dispatcher[CommandDispatcher]\n Dispatcher --> MCPClient[MCP Client]\n MCPClient --> UICollector[UICollector Server]\n MCPClient --> HostUIExecutor[HostUIExecutor Server]\n MCPClient --> CLIExecutor[CommandLine Executor]\n \n UICollector --> DataCollection[Desktop Screenshot Window Info]\n HostUIExecutor --> DesktopActions[Window Selection App Launch]\n CLIExecutor --> ShellActions[Shell Commands]\n \n style HostAgent fill:#e3f2fd\n style Dispatcher fill:#fff3e0\n style MCPClient fill:#f1f8e9\n style UICollector fill:#c8e6c9\n style HostUIExecutor fill:#fff9c4\n style CLIExecutor fill:#d1c4e9\n```\n\n!!!note \"Dynamic Commands\"\n HostAgent commands are **not hardcoded**. They are dynamically discovered from configured MCP servers. Available commands depend on MCP server configuration in `config/ufo/mcp.yaml`, installed MCP servers, and active MCP connections.\n\n---\n\n## MCP Server Configuration\n\n### Configuration File\n\nHostAgent commands are configured in **`config/ufo/mcp.yaml`**:\n\n```yaml\nHostAgent:\n default:\n data_collection:\n - namespace: UICollector\n type: local\n start_args: []\n reset: false\n action:\n - namespace: HostUIExecutor\n type: local\n start_args: []\n reset: false\n - namespace: CommandLineExecutor\n type: local\n start_args: []\n reset: false\n```\n\n### MCP Servers Used by HostAgent\n\n| Server | Namespace | Type | Purpose | Command Categories |\n|--------|-----------|------|---------|-------------------|\n| **UICollector** | `UICollector` | Local | Data collection | Desktop screenshot, window enumeration |\n| **HostUIExecutor** | `HostUIExecutor` | Local | Desktop actions | Window selection, application launch |\n| **CommandLineExecutor** | `CommandLineExecutor` | Local | Shell execution | PowerShell, Bash commands |\n\n---\n\n## Command Discovery\n\n### Listing Available Commands\n\nHostAgent dynamically discovers available commands from MCP servers:\n\n```python\n# Get all available tools from MCP servers\nresult = await command_dispatcher.execute_commands([\n Command(tool_name=\"list_tools\", parameters={})\n])\n\ntools = result[0].result\n# Returns list of all available commands with their schemas\n```\n\n### Command Categories\n\nCommands are categorized by purpose:\n\n| Category | Server | Examples |\n|----------|--------|----------|\n| **Data Collection** | UICollector | `capture_desktop_screenshot`, `get_desktop_app_target_info`, `get_desktop_window_info` |\n| **Window Management** | HostUIExecutor | `select_application_window`, `launch_application` |\n| **Process Control** | HostUIExecutor | `close_application`, `get_process_info` |\n| **Shell Execution** | CommandLineExecutor | `execute_command` |\n| **Tool Discovery** | All Servers | `list_tools` |\n\n---\n\n## Command Execution\n\n### Execution Flow\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Executor as ActionExecutor\n participant Dispatcher as CommandDispatcher\n participant MCP as MCP Server\n \n Strategy->>Executor: execute(action_info)\n Executor->>Dispatcher: execute_commands([Command(...)])\n Dispatcher->>MCP: Invoke tool\n MCP->>MCP: Execute command logic\n MCP-->>Dispatcher: Result\n Dispatcher-->>Executor: Result\n Executor-->>Strategy: Success/Error\n```\n\n### Example: Capture Desktop Screenshot\n\n```python\nfrom aip.messages import Command\n\n# Create command\ncommand = Command(\n tool_name=\"capture_desktop_screenshot\",\n parameters={\"all_screens\": True},\n tool_type=\"data_collection\",\n)\n\n# Execute command\nresults = await command_dispatcher.execute_commands([command])\n\n# Access result\nscreenshot_data = results[0].result # Base64-encoded image\n```\n\n### Example: Select Application Window\n\n```python\n# Select and focus application window\ncommand = Command(\n tool_name=\"select_application_window\",\n parameters={\n \"id\": \"0\",\n \"name\": \"Microsoft Word - Document1\"\n },\n tool_type=\"action\",\n)\n\nresults = await command_dispatcher.execute_commands([command])\napp_info = results[0].result\n```\n\n---\n\n## Configuration Resources\n\nFor detailed MCP configuration, server setup, and command reference:\n\n**Quick References:**\n\n- **[MCP Configuration Reference](../../configuration/system/mcp_reference.md)** - Quick MCP settings reference\n- **[MCP Overview](../../mcp/overview.md)** - MCP architecture and concepts\n\n**Configuration Guides:**\n\n- **[MCP Configuration Guide](../../mcp/configuration.md)** - Complete configuration documentation\n- **[Local Servers](../../mcp/local_servers.md)** - Built-in MCP servers\n- **[Remote Servers](../../mcp/remote_servers.md)** - HTTP and stdio servers\n- **[Creating MCP Servers](../../tutorials/creating_mcp_servers.md)** - Creating custom MCP servers\n\n**Server Type Documentation:**\n\n- **[Action Servers](../../mcp/action.md)** - Action server documentation\n- **[Data Collection Servers](../../mcp/data_collection.md)** - Data collection server documentation\n\n### Detailed Server Documentation\n\nEach MCP server has comprehensive documentation:\n\n| Server | Documentation | Command Details |\n|--------|--------------|----------------|\n| UICollector | [UICollector Server](../../mcp/servers/ui_collector.md) | Screenshot, window info, control detection commands |\n| HostUIExecutor | [HostUIExecutor Server](../../mcp/servers/host_ui_executor.md) | Window management and desktop automation commands |\n| CommandLineExecutor | [CommandLine Executor](../../mcp/servers/command_line_executor.md) | Shell command execution |\n\n!!!warning \"Command Details Subject to Change\"\n Specific command parameters, names, and behaviors may change as MCP servers evolve. Always refer to the server-specific documentation for the most up-to-date command reference.\n\n---\n\n## Agent Configuration Settings\n\n### HostAgent Configuration\n\n```yaml\n# config/ufo/host_agent_config.yaml\nsystem:\n # Control detection backend\n control_backend:\n - \"uia\" # Windows UI Automation\n - \"omniparser\" # Vision-based detection\n \n # Screenshot settings\n save_full_screen: true # Capture desktop screenshots\n save_ui_tree: true # Save UI tree JSON\n include_last_screenshot: true # Include previous step\n concat_screenshot: true # Concatenate clean + annotated\n \n # Window behavior\n maximize_window: false # Maximize on selection\n show_visual_outline_on_screen: true # Draw red outline\n```\n\nSee **[Configuration Overview](../../configuration/system/overview.md)** and **[System Configuration](../../configuration/system/system_config.md)** for complete configuration options.\n\n---\n\n## Related Documentation\n\n**Architecture & Design:**\n\n- **[HostAgent Overview](overview.md)** - High-level HostAgent architecture\n- **[State Machine](state.md)** - 7-state FSM documentation\n- **[Processing Strategy](strategy.md)** - 4-phase processing pipeline\n- **[AppAgent Commands](../app_agent/commands.md)** - Application-level commands\n\n**Core Features:**\n - **[Hybrid Actions](../core_features/hybrid_actions.md)** - MCP command system architecture\n - **[Control Detection](../core_features/control_detection/overview.md)** - UIA and OmniParser backends\n - **[Command Dispatcher](../../infrastructure/modules/dispatcher.md)** - Command routing\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n- **MCP-Based**: All commands provided by MCP servers configured in `mcp.yaml`\n- **Dynamic Discovery**: Commands discovered at runtime via `list_tools`\n- **Desktop-Level**: System-wide operations (screenshots, window management)\n- **Configurable**: Extensive MCP server configuration options\n- **Documented**: Each server has detailed command reference\n\n!!!warning\n Command details subject to change - refer to server documentation for latest information\n\n**Next Steps:**\n\n1. **Review MCP Configuration**: [MCP Configuration Reference](../../configuration/system/mcp_reference.md)\n2. **Explore Server Documentation**: Click server links above for command details\n3. **Understand Processing**: [Processing Strategy](strategy.md) shows commands in action\n4. **Learn State Machine**: [State Machine](state.md) explains when commands execute\n"
},
{
"path": "documents/docs/ufo2/host_agent/overview.md",
"content": "# HostAgent: Desktop Orchestrator\n\n**HostAgent** serves as the centralized control plane of UFO². It interprets user-specified goals, decomposes them into structured subtasks, instantiates and dispatches AppAgent modules, and coordinates their progress across the system. HostAgent provides system-level services for introspection, planning, application lifecycle management, and multi-agent synchronization.\n\n---\n\n## Architecture Overview\n\nOperating atop the native Windows substrate, HostAgent monitors active applications, issues shell commands to spawn new processes as needed, and manages the creation and teardown of application-specific AppAgent instances. All coordination occurs through a persistent state machine, which governs the transitions across execution phases.\n\n\n \n Figure: HostAgent architecture showing the finite state machine, processing pipeline, and interactions with AppAgents through the Blackboard pattern.\n\n\n---\n\n## Core Responsibilities\n\n### Task Decomposition\n\nGiven a user's natural language input, HostAgent identifies the underlying task goal and decomposes it into a dependency-ordered subtask graph.\n\n**Example:** User request \"Extract data from Word and create an Excel chart\" becomes:\n\n1. Extract table from Word document\n2. Create chart in Excel with extracted data\n\n\n \n Figure: HostAgent decomposes user requests into sequential subtasks, assigns each to the appropriate application, and orchestrates AppAgents to complete them in dependency order.\n\n\n### Application Lifecycle Management\n\nFor each subtask, HostAgent inspects system process metadata (via UIA APIs) to determine whether the target application is running. If not, it launches the program and registers it with the runtime.\n\n### AppAgent Instantiation\n\nHostAgent spawns the corresponding AppAgent for each active application, providing it with task context, memory references, and relevant toolchains (e.g., APIs, documentation).\n\n### Task Scheduling and Control\n\nThe global execution plan is serialized into a finite state machine (FSM), allowing HostAgent to enforce execution order, detect failures, and resolve dependencies across agents. See **[State Machine Details](state.md)** for the FSM architecture.\n\n### Shared State Communication\n\nHostAgent reads from and writes to a global blackboard, enabling inter-agent communication and system-level observability for debugging and replay.\n\n---\n\n## Key Characteristics\n\n- **Scope**: Desktop-level orchestrator (system-wide, not application-specific)\n- **Lifecycle**: Single instance per session, persists throughout task execution\n- **Hierarchy**: Parent agent that manages multiple child AppAgents\n- **Communication**: Owns and coordinates the shared Blackboard\n- **Control**: 7-state finite state machine with 4-phase processing pipeline\n\n---\n\n## Execution Workflow\n\n```mermaid\nsequenceDiagram\n participant User\n participant HostAgent\n participant Blackboard\n participant AppAgent1\n participant AppAgent2\n \n User->>HostAgent: \"Extract Word table, create Excel chart\"\n HostAgent->>HostAgent: Decompose into subtasks\n HostAgent->>Blackboard: Write subtask 1\n HostAgent->>AppAgent1: Create/Get Word AppAgent\n AppAgent1->>AppAgent1: Execute Word task\n AppAgent1->>Blackboard: Write result 1\n AppAgent1-->>HostAgent: Return FINISH\n \n HostAgent->>Blackboard: Read result 1\n HostAgent->>Blackboard: Write subtask 2\n HostAgent->>AppAgent2: Create/Get Excel AppAgent\n AppAgent2->>Blackboard: Read result 1\n AppAgent2->>AppAgent2: Execute Excel task\n AppAgent2->>Blackboard: Write result 2\n AppAgent2-->>HostAgent: Return FINISH\n \n HostAgent->>HostAgent: Verify completion\n HostAgent-->>User: Task completed\n```\n\n---\n\n## Deep Dive Topics\n\n- **[State Machine](state.md)**: 7-state FSM architecture and transitions\n- **[Processing Strategy](strategy.md)**: 4-phase processing pipeline\n- **[Command System](commands.md)**: Desktop-level MCP commands\n\n---\n\n## Input and Output\n\n### HostAgent Input\n\n| Input | Description | Type |\n|-------|-------------|------|\n| User Request | Natural language task description | String |\n| Application Information | Active application metadata | List of Dicts |\n| Desktop Screenshots | Visual context of desktop state | Image |\n| Previous Sub-Tasks | Completed subtask history | List of Dicts |\n| Previous Plan | Planned future subtasks | List of Strings |\n| Blackboard | Shared memory space | Dictionary |\n\n### HostAgent Output\n\n| Output | Description | Type |\n|--------|-------------|------|\n| Observation | Desktop screenshot analysis | String |\n| Thought | Reasoning process | String |\n| Current Sub-Task | Active subtask description | String |\n| Message | Information for AppAgent | String |\n| ControlLabel | Selected application index | String |\n| ControlText | Selected application name | String |\n| Plan | Future subtask sequence | List of Strings |\n| Status | Agent state (CONTINUE/ASSIGN/FINISH/etc.) | String |\n| Comment | User-facing information | String |\n| Questions | Clarification requests | List of Strings |\n| Bash | System command to execute | String |\n\n**Example Output:**\n```json\n{\n \"Observation\": \"Desktop shows Microsoft Word with document open containing a table\",\n \"Thought\": \"User wants to extract data from Word first\",\n \"Current Sub-Task\": \"Extract the table data from the document\",\n \"Message\": \"Starting data extraction from Word document\",\n \"ControlLabel\": \"0\",\n \"ControlText\": \"Microsoft Word - Document1\",\n \"Plan\": [\"Extract table from Word\", \"Create chart in Excel\"],\n \"Status\": \"ASSIGN\",\n \"Comment\": \"Delegating table extraction to Word AppAgent\",\n \"Questions\": [],\n \"Bash\": \"\"\n}\n```\n\n---\n\n## Related Documentation\n\n**Architecture & Design:**\n\n- **[Windows Agent Overview](../overview.md)**: Module architecture and hierarchy\n- **[AppAgent](../app_agent/overview.md)**: Application automation agent\n- **[Blackboard](../../infrastructure/agents/design/blackboard.md)**: Inter-agent communication\n- **[Memory System](../../infrastructure/agents/design/memory.md)**: Execution history\n\n**Configuration:**\n\n- **[Configuration System Overview](../../configuration/system/overview.md)**: System configuration structure\n- **[Agents Configuration](../../configuration/system/agents_config.md)**: LLM and agent settings\n- **[System Configuration](../../configuration/system/system_config.md)**: Runtime and execution settings\n- **[MCP Reference](../../configuration/system/mcp_reference.md)**: MCP server configuration\n\n**System Integration:**\n\n- **[Session Management](../../infrastructure/modules/session.md)**: Session lifecycle\n- **[Round Management](../../infrastructure/modules/round.md)**: Execution rounds\n\n---\n\n## API Reference\n\n:::agents.agent.host_agent.HostAgent\n\n---\n\n## Summary\n\nHostAgent is the desktop-level orchestrator that:\n\n- Decomposes tasks and coordinates AppAgents\n- Operates at system level, not application level \n- Uses a 7-state FSM: CONTINUE → ASSIGN → AppAgent → CONTINUE → FINISH\n- Executes a 4-phase pipeline: DATA_COLLECTION → LLM → ACTION → MEMORY\n- Creates, caches, and reuses AppAgent instances\n- Provides shared Blackboard memory for all agents\n- Maintains single instance per session managing multiple AppAgents\n\n**Next Steps:**\n\n1. Read [State Machine](state.md) for FSM details\n2. Read [Processing Strategy](strategy.md) for pipeline architecture \n3. Read [Command System](commands.md) for available desktop operations\n4. Read [AppAgent](../app_agent/overview.md) for application-level execution\n"
},
{
"path": "documents/docs/ufo2/host_agent/state.md",
"content": "# HostAgent State Machine\n\n!!!abstract \"Overview\"\n HostAgent uses a **7-state finite state machine (FSM)** to manage task orchestration flow. The state machine controls task decomposition, application selection, AppAgent delegation, and completion verification. States transition based on LLM decisions and system events.\n\n---\n\n## State Machine Architecture\n\n### State Enumeration\n\n```python\nclass HostAgentStatus(Enum):\n \"\"\"Store the status of the host agent\"\"\"\n ERROR = \"ERROR\" # Unhandled exception or system error\n FINISH = \"FINISH\" # Task completed successfully\n CONTINUE = \"CONTINUE\" # Active processing state\n ASSIGN = \"ASSIGN\" # Delegate to AppAgent\n FAIL = \"FAIL\" # Task failed, cannot proceed\n PENDING = \"PENDING\" # Await external event or user input\n CONFIRM = \"CONFIRM\" # Request user approval\n```\n\n### State Management\n\nHostAgent states are managed by `HostAgentStateManager`, which implements a singleton registry pattern:\n\n```python\nclass HostAgentStateManager(AgentStateManager):\n \"\"\"Manages the states of the host agent\"\"\"\n _state_mapping: Dict[str, Type[HostAgentState]] = {}\n \n @property\n def none_state(self) -> AgentState:\n return NoneHostAgentState()\n```\n\nAll HostAgent states are registered using the `@HostAgentStateManager.register` decorator, enabling dynamic state lookup by name.\n\n---\n\n## State Definitions\n\n### 1. CONTINUE State\n\n**Purpose**: Active orchestration state where HostAgent executes its 4-phase processing pipeline.\n\n```python\n@HostAgentStateManager.register\nclass ContinueHostAgentState(HostAgentState):\n \"\"\"The class for the continue host agent state\"\"\"\n \n async def handle(self, agent: \"HostAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Execute the 4-phase processing pipeline\"\"\"\n await agent.process(context)\n \n def is_round_end(self) -> bool:\n return False # Round continues\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.CONTINUE.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Active |\n| **Processor Executed** | ✓ Yes (4 phases) |\n| **Round Ends** | No |\n| **Duration** | Single round |\n| **Next States** | CONTINUE, ASSIGN, FINISH, CONFIRM, ERROR |\n\n**Behavior**:\n\n1. Captures desktop screenshot\n2. LLM analyzes desktop and selects application\n3. Updates context with selected application\n4. Records orchestration step in memory\n\n**Example Usage:**\n\n```python\n# HostAgent in CONTINUE state\nagent.status = HostAgentStatus.CONTINUE.value\nagent.set_state(ContinueHostAgentState())\n\n# State executes 4-phase pipeline\nawait state.handle(agent, context)\n\n# LLM sets next status in response\n# {\"Status\": \"ASSIGN\", \"ControlText\": \"Microsoft Word\"}\n```\n\n---\n\n### 2. ASSIGN State\n\n**Purpose**: Create or retrieve AppAgent for the selected application and delegate execution.\n\n```python\n@HostAgentStateManager.register\nclass AssignHostAgentState(HostAgentState):\n \"\"\"The class for the assign host agent state\"\"\"\n \n async def handle(self, agent: \"HostAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Create/get AppAgent for selected application\"\"\"\n agent.create_subagent(context)\n \n def next_state(self, agent: \"HostAgent\") -> \"AppAgentState\":\n \"\"\"Transition to AppAgent's CONTINUE state\"\"\"\n next_agent = self.next_agent(agent)\n \n if type(next_agent) == OpenAIOperatorAgent:\n return ContinueOpenAIOperatorState()\n else:\n return ContinueAppAgentState()\n \n def next_agent(self, agent: \"HostAgent\") -> \"AppAgent\":\n \"\"\"Get the active AppAgent for delegation\"\"\"\n return agent.get_active_appagent()\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.ASSIGN.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Transition |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | No |\n| **Duration** | Immediate |\n| **Next States** | AppAgent.CONTINUE |\n| **Next Agent** | AppAgent (switched) |\n\n**Behavior**:\n\n1. Checks if AppAgent for application already exists (cache)\n2. Creates new AppAgent if not cached\n3. Sets parent-child relationship (`app_agent.host = self`)\n4. Shares Blackboard (`app_agent.blackboard = self.blackboard`)\n5. Transitions to `AppAgent.CONTINUE` state\n\n**AppAgent Caching:**\n\n```python\n# HostAgent maintains a cache of created AppAgents\nagent_key = f\"{app_root}/{process_name}\"\n\nif agent_key in self.appagent_dict:\n # Reuse existing AppAgent\n self._active_appagent = self.appagent_dict[agent_key]\nelse:\n # Create new AppAgent\n app_agent = AgentFactory.create_agent(**config)\n self.appagent_dict[agent_key] = app_agent\n self._active_appagent = app_agent\n```\n\n---\n\n### 3. FINISH State\n\n**Purpose**: Task completed successfully, terminate session.\n\n```python\n@HostAgentStateManager.register\nclass FinishHostAgentState(HostAgentState):\n \"\"\"The class for the finish host agent state\"\"\"\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.FINISH.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Duration** | Permanent |\n| **Next States** | None |\n\n**Behavior**:\n\n- Session terminates successfully\n- All subtasks completed\n- Results available in Blackboard\n\n---\n\n### 4. FAIL State\n\n**Purpose**: Task failed, cannot proceed further.\n\n```python\n@HostAgentStateManager.register\nclass FailHostAgentState(HostAgentState):\n \"\"\"The class for the fail host agent state\"\"\"\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n def next_state(self, agent: \"HostAgent\") -> AgentState:\n return FinishHostAgentState() # Transition to FINISH for cleanup\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.FAIL.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Duration** | Permanent |\n| **Next States** | FINISH (for cleanup) |\n\n**Behavior**:\n\n- Task cannot be completed\n- May result from user rejection or irrecoverable error\n- Transitions to FINISH for graceful shutdown\n\n---\n\n### 5. ERROR State\n\n**Purpose**: Unhandled exception or critical system error.\n\n```python\n@HostAgentStateManager.register\nclass ErrorHostAgentState(HostAgentState):\n \"\"\"The class for the error host agent state\"\"\"\n \n def is_round_end(self) -> bool:\n return True # Round ends\n \n def next_state(self, agent: \"HostAgent\") -> AgentState:\n return FinishHostAgentState() # Transition to FINISH for cleanup\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.ERROR.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Terminal |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | Yes |\n| **Duration** | Permanent |\n| **Next States** | FINISH (for cleanup) |\n\n**Behavior**:\n\n- Critical system error occurred\n- Unhandled exception during processing\n- Automatically triggers graceful shutdown\n\n**Error vs Fail:**\n\n- **ERROR**: System/code errors (exceptions, crashes)\n- **FAIL**: Logical task failures (user rejection, impossible task)\n\n---\n\n### 6. PENDING State\n\n**Purpose**: Await external event or user input before continuing.\n\n```python\n@HostAgentStateManager.register\nclass PendingHostAgentState(HostAgentState):\n \"\"\"The class for the pending host agent state\"\"\"\n \n async def handle(self, agent: \"HostAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Ask the user questions to help the agent proceed\"\"\"\n agent.process_asker(ask_user=ufo_config.system.ask_question)\n \n def next_state(self, agent: \"HostAgent\") -> AgentState:\n \"\"\"Return to CONTINUE after receiving input\"\"\"\n agent.status = HostAgentStatus.CONTINUE.value\n return ContinueHostAgentState()\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.PENDING.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Waiting |\n| **Processor Executed** | ✗ No |\n| **Round Ends** | No |\n| **Duration** | Until event/timeout |\n| **Next States** | CONTINUE, FAIL |\n\n**Behavior**:\n\n- Requests additional information from user\n- Waits for external event (async operation)\n- Transitions to CONTINUE after receiving input\n- May timeout and transition to FAIL\n\n---\n\n### 7. CONFIRM State\n\n**Purpose**: Request user approval before proceeding with action.\n\n```python\n@HostAgentStateManager.register\nclass ConfirmHostAgentState(HostAgentState):\n \"\"\"The class for the confirm host agent state\"\"\"\n \n async def handle(self, agent: \"HostAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Request user confirmation\"\"\"\n # Confirmation logic handled by processor\n pass\n \n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.CONFIRM.value\n```\n\n| Property | Value |\n|----------|-------|\n| **Type** | Waiting |\n| **Processor Executed** | ✓ Yes (collect confirmation) |\n| **Round Ends** | No |\n| **Duration** | Until user responds |\n| **Next States** | CONTINUE (approved), FAIL (rejected) |\n\n**Behavior**:\n\n- Displays confirmation request to user\n- Waits for user approval/rejection\n- CONTINUE if approved\n- FAIL if rejected\n\n**Safety Check:**\n\nCONFIRM state provides a safety mechanism for sensitive operations such as application launches, file deletions, and system configuration changes.\n\n---\n\n## State Transition Diagram\n\n\n \n HostAgent State Machine: Visual representation of the 7-state FSM with transitions and conditions\n\n\n---\n\n## State Transition Control\n\n### LLM-Driven Transitions\n\nMost state transitions are controlled by the LLM through the `Status` field in its response:\n\n```json\n{\n \"Observation\": \"Desktop shows Word and Excel. User wants to extract data from Word.\",\n \"Thought\": \"I should start with Word to extract the table data first.\",\n \"Current Sub-Task\": \"Extract table data from Word document\",\n \"ControlLabel\": \"0\",\n \"ControlText\": \"Microsoft Word - Document1\",\n \"Status\": \"ASSIGN\",\n \"Comment\": \"Delegating data extraction to Word AppAgent\"\n}\n```\n\n**Transition Flow**:\n\n1. HostAgent in `CONTINUE` state executes processor\n2. LLM analyzes desktop and decides next action\n3. LLM sets `Status: \"ASSIGN\"` in response\n4. Processor updates `agent.status = \"ASSIGN\"`\n5. State machine transitions: `CONTINUE` → `ASSIGN`\n6. `ASSIGN` state creates/gets AppAgent\n7. Transitions to `AppAgent.CONTINUE`\n\n### System-Driven Transitions\n\nSome transitions are automatic and controlled by the system:\n\n| From State | To State | Trigger | Controller |\n|------------|----------|---------|------------|\n| ASSIGN | AppAgent.CONTINUE | AppAgent created | System |\n| AppAgent.CONTINUE | CONTINUE | AppAgent returns | System |\n| PENDING | FAIL | Timeout | System |\n| CONFIRM | CONTINUE | User approved | User Input |\n| CONFIRM | FAIL | User rejected | User Input |\n| ERROR | FINISH | Exception caught | System |\n| FAIL | FINISH | Cleanup needed | System |\n\n---\n\n## Complete Execution Flow Example\n\n### Multi-Application Task\n\n**User Request**: \"Extract sales table from Word and create bar chart in Excel\"\n\n```mermaid\nsequenceDiagram\n participant User\n participant HostAgent\n participant WordAppAgent\n participant ExcelAppAgent\n \n Note over HostAgent: State: CONTINUE\n User->>HostAgent: \"Extract Word table, create Excel chart\"\n HostAgent->>HostAgent: Phase 1: Capture desktop Phase 2: LLM analyzes\n Note over HostAgent: LLM Decision: Status=ASSIGN\n HostAgent->>HostAgent: Phase 3: Update context Phase 4: Record memory\n \n Note over HostAgent: State: ASSIGN\n HostAgent->>WordAppAgent: create_subagent(\"Word\")\n Note over HostAgent,WordAppAgent: Agent Handoff\n \n Note over WordAppAgent: State: AppAgent.CONTINUE\n WordAppAgent->>WordAppAgent: Capture Word UI Select table Execute copy\n WordAppAgent->>HostAgent: Return Status=FINISH\n \n Note over HostAgent: State: CONTINUE\n HostAgent->>HostAgent: Phase 2: LLM sees Word result Decides Excel next\n Note over HostAgent: LLM Decision: Status=ASSIGN\n \n Note over HostAgent: State: ASSIGN\n HostAgent->>ExcelAppAgent: create_subagent(\"Excel\")\n Note over HostAgent,ExcelAppAgent: Agent Handoff\n \n Note over ExcelAppAgent: State: AppAgent.CONTINUE\n ExcelAppAgent->>ExcelAppAgent: Paste data Insert chart Format\n ExcelAppAgent->>HostAgent: Return Status=FINISH\n \n Note over HostAgent: State: CONTINUE\n HostAgent->>HostAgent: Phase 2: LLM confirms complete\n Note over HostAgent: LLM Decision: Status=FINISH\n \n Note over HostAgent: State: FINISH\n HostAgent->>User: Task completed!\n```\n\n### Step-by-Step State Transitions\n\n| Step | Agent | State | Action | Next State |\n|------|-------|-------|--------|------------|\n| 1 | HostAgent | CONTINUE | Analyze desktop, select Word | ASSIGN |\n| 2 | HostAgent | ASSIGN | Create WordAppAgent | AppAgent.CONTINUE |\n| 3 | WordAppAgent | CONTINUE | Extract table | FINISH |\n| 4 | HostAgent | CONTINUE | Analyze result, select Excel | ASSIGN |\n| 5 | HostAgent | ASSIGN | Create ExcelAppAgent | AppAgent.CONTINUE |\n| 6 | ExcelAppAgent | CONTINUE | Create chart | FINISH |\n| 7 | HostAgent | CONTINUE | Verify completion | FINISH |\n| 8 | HostAgent | FINISH | Session ends | - |\n\n---\n\n## Implementation Details\n\n### State Class Hierarchy\n\n```python\n# Base state interface\nclass HostAgentState(AgentState):\n \"\"\"Abstract class for host agent states\"\"\"\n \n async def handle(self, agent: \"HostAgent\", context: Optional[\"Context\"] = None):\n \"\"\"Execute state-specific logic\"\"\"\n pass\n \n def next_state(self, agent: \"HostAgent\") -> AgentState:\n \"\"\"Determine next state based on agent status\"\"\"\n status = agent.status\n return HostAgentStateManager().get_state(status)\n \n def next_agent(self, agent: \"HostAgent\") -> \"HostAgent\":\n \"\"\"Get agent for next step (usually same agent)\"\"\"\n return agent\n \n def is_round_end(self) -> bool:\n \"\"\"Check if round should end\"\"\"\n return False\n \n @classmethod\n def agent_class(cls) -> Type[\"HostAgent\"]:\n from ufo.agents.agent.host_agent import HostAgent\n return HostAgent\n```\n\n### State Registration Pattern\n\n```python\n# Registration decorator adds state to manager\n@HostAgentStateManager.register\nclass ContinueHostAgentState(HostAgentState):\n @classmethod\n def name(cls) -> str:\n return HostAgentStatus.CONTINUE.value\n\n# Manager can look up states by name\nstate = HostAgentStateManager().get_state(\"CONTINUE\")\n# Returns: ContinueHostAgentState instance\n```\n\n**Lazy Loading:**\n\nStates are loaded lazily by `HostAgentStateManager` only when needed, reducing initialization overhead.\n\n---\n```\n\n### State Transition in Round Execution\n\n```python\n# In Round.run() method\nwhile not state.is_round_end():\n # Execute current state\n await state.handle(agent, context)\n \n # Get next state based on agent.status\n state = state.next_state(agent)\n \n # Check if agent switched (HostAgent → AppAgent)\n agent = state.next_agent(agent)\n```\n\n!!!tip \"Lazy Loading\"\n States are loaded lazily by `HostAgentStateManager` only when needed, reducing initialization overhead.\n\n---\n\n## State Transition Table\n\n### Complete Transition Matrix\n\n| From \\ To | CONTINUE | ASSIGN | FINISH | FAIL | ERROR | PENDING | CONFIRM | AppAgent.CONTINUE |\n|-----------|----------|--------|--------|------|-------|---------|---------|-------------------|\n| **CONTINUE** | ✓ LLM | ✓ LLM | ✓ LLM | ✗ | ✓ System | ✓ LLM | ✓ LLM | ✗ |\n| **ASSIGN** | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ System |\n| **FINISH** | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |\n| **FAIL** | ✗ | ✗ | ✓ System | ✗ | ✗ | ✗ | ✗ | ✗ |\n| **ERROR** | ✗ | ✗ | ✓ System | ✗ | ✗ | ✗ | ✗ | ✗ |\n| **PENDING** | ✓ User | ✗ | ✗ | ✓ Timeout | ✗ | ✗ | ✗ | ✗ |\n| **CONFIRM** | ✓ User | ✗ | ✗ | ✓ User | ✗ | ✗ | ✗ | ✗ |\n| **AppAgent.CONTINUE** | ✓ System | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |\n\n**Legend**:\n- ✓ LLM: Transition controlled by LLM decision\n- ✓ System: Automatic system transition\n- ✓ User: User input required\n- ✓ Timeout: Timeout triggers transition\n- ✗: Transition not allowed\n\n---\n\n## Related Documentation\n\n**Architecture & Design:**\n\n- **[Overview](overview.md)**: HostAgent high-level architecture\n- **[Processing Strategy](strategy.md)**: 4-phase processing pipeline\n- **[State Design Pattern](../../infrastructure/agents/design/state.md)**: General state framework\n- **[AppAgent State Machine](../app_agent/state.md)**: AppAgent FSM comparison\n\n**System Integration:**\n\n- **[Round Management](../../infrastructure/modules/round.md)**: How states execute in rounds\n- **[Session Management](../../infrastructure/modules/session.md)**: Session lifecycle\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n- **7 States**: CONTINUE, ASSIGN, FINISH, FAIL, ERROR, PENDING, CONFIRM\n- **LLM Control**: Most transitions driven by LLM's `Status` field\n- **Agent Handoff**: ASSIGN state transitions to AppAgent.CONTINUE\n- **Terminal States**: FINISH, FAIL, ERROR end the session\n- **Safety Checks**: CONFIRM and PENDING provide user control\n- **State Pattern**: Implements Gang of Four State design pattern\n- **Singleton Registry**: HostAgentStateManager manages all states\n\n**Next Steps:**\n\n- Read [Processing Strategy](strategy.md) to understand what happens in CONTINUE state\n- Read [Command System](commands.md) for available desktop operations\n- Read [AppAgent State Machine](../app_agent/state.md) for comparison\n"
},
{
"path": "documents/docs/ufo2/host_agent/strategy.md",
"content": "# HostAgent Processing Strategy\n\nHostAgent executes a **4-phase processing pipeline** in **CONTINUE** and **CONFIRM** states. Each phase handles a specific aspect of desktop orchestration: **data collection**, **LLM decision making**, **action execution**, and **memory recording**. This document details the implementation of each strategy based on the actual codebase.\n\n---\n\n## Strategy Assembly\n\nProcessing strategies are **assembled and orchestrated** by the `HostAgentProcessor` class defined in `ufo/agents/processors/host_agent_processor.py`. The processor acts as the **coordinator** that initializes, configures, and executes the 4-phase pipeline.\n\n### HostAgentProcessor Overview\n\nThe `HostAgentProcessor` extends `ProcessorTemplate` and serves as the main orchestrator for HostAgent workflows:\n\n```python\nclass HostAgentProcessor(ProcessorTemplate):\n \"\"\"\n Enhanced processor for Host Agent with comprehensive functionality.\n \n Manages the complete workflow including:\n - Desktop environment analysis and screenshot capture\n - Application window detection and registration\n - Third-party agent integration and management\n - LLM-based decision making with context-aware prompting\n - Action execution including application selection and command dispatch\n - Memory management with detailed logging and state tracking\n \"\"\"\n \n processor_context_class = HostAgentProcessorContext\n \n def __init__(self, agent: \"HostAgent\", global_context: Context):\n super().__init__(agent, global_context)\n```\n\n### Strategy Registration\n\nDuring initialization, `HostAgentProcessor._setup_strategies()` registers all four processing strategies:\n\n```python\ndef _setup_strategies(self) -> None:\n \"\"\"Configure processing strategies with error handling and logging.\"\"\"\n \n # Phase 1: Desktop data collection (critical - fail_fast=True)\n self.strategies[ProcessingPhase.DATA_COLLECTION] = (\n DesktopDataCollectionStrategy(\n fail_fast=True # Desktop data collection is critical\n )\n )\n \n # Phase 2: LLM interaction (critical - fail_fast=True)\n self.strategies[ProcessingPhase.LLM_INTERACTION] = (\n HostLLMInteractionStrategy(\n fail_fast=True # LLM failure should trigger recovery\n )\n )\n \n # Phase 3: Action execution (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.ACTION_EXECUTION] = (\n HostActionExecutionStrategy(\n fail_fast=False # Action failures can be handled gracefully\n )\n )\n \n # Phase 4: Memory update (graceful - fail_fast=False)\n self.strategies[ProcessingPhase.MEMORY_UPDATE] = (\n HostMemoryUpdateStrategy(\n fail_fast=False # Memory update failures shouldn't stop process\n )\n )\n```\n\n| Phase | Strategy Class | fail_fast | Rationale |\n|-------|---------------|-----------|-----------|\n| **DATA_COLLECTION** | `DesktopDataCollectionStrategy` | ✓ True | Desktop screenshot and window info are critical for LLM context |\n| **LLM_INTERACTION** | `HostLLMInteractionStrategy` | ✓ True | LLM response failure requires immediate recovery mechanism |\n| **ACTION_EXECUTION** | `HostActionExecutionStrategy` | ✗ False | Action failures can be gracefully handled and reported |\n| **MEMORY_UPDATE** | `HostMemoryUpdateStrategy` | ✗ False | Memory failures shouldn't block the main execution flow |\n\n**Fail-Fast vs Graceful:**\n\nThe `fail_fast` parameter controls error propagation behavior:\n\n- **fail_fast=True**: Errors immediately halt the pipeline and trigger recovery (used for critical phases)\n- **fail_fast=False**: Errors are logged but don't stop execution (used for non-critical phases)\n\n### Middleware Configuration\n\nThe processor also configures specialized logging middleware:\n\n```python\ndef _setup_middleware(self) -> None:\n \"\"\"Set up enhanced middleware chain with comprehensive monitoring.\"\"\"\n self.middleware_chain = [\n HostAgentLoggingMiddleware(), # Specialized logging for Host Agent\n ]\n```\n\n**HostAgentLoggingMiddleware** provides:\n\n- Round and step progress tracking\n- Rich Panel displays with color coding\n- Application selection logging\n- Detailed error context reporting\n\n---\n\n## Processing Pipeline Architecture\n\n```mermaid\ngraph LR\n DC[Phase 1: DATA_COLLECTION DesktopDataCollectionStrategy] --> LLM[Phase 2: LLM_INTERACTION HostLLMInteractionStrategy]\n LLM --> AE[Phase 3: ACTION_EXECUTION HostActionExecutionStrategy]\n AE --> MU[Phase 4: MEMORY_UPDATE HostMemoryUpdateStrategy]\n \n style DC fill:#e1f5ff\n style LLM fill:#fff4e6\n style AE fill:#e8f5e9\n style MU fill:#fce4ec\n```\n\nEach phase is implemented as a separate **strategy class** inheriting from `BaseProcessingStrategy`. Strategies declare their dependencies and outputs using `@depends_on` and `@provides` decorators for automatic data flow management.\n\n---\n\n## Phase 1: DATA_COLLECTION\n\n### Strategy: `DesktopDataCollectionStrategy`\n\n**Purpose**: Gather comprehensive desktop environment context for LLM decision making.\n\n```python\n@depends_on(\"command_dispatcher\", \"log_path\", \"session_step\")\n@provides(\n \"desktop_screenshot_url\",\n \"desktop_screenshot_path\",\n \"application_windows_info\",\n \"target_registry\",\n \"target_info_list\",\n)\nclass DesktopDataCollectionStrategy(BaseProcessingStrategy):\n \"\"\"Enhanced strategy for collecting desktop environment data\"\"\"\n \n def __init__(self, fail_fast: bool = True):\n super().__init__(name=\"desktop_data_collection\", fail_fast=fail_fast)\n```\n\n### Execution Steps\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant CommandDispatcher\n participant Desktop\n participant TargetRegistry\n \n Strategy->>CommandDispatcher: capture_desktop_screenshot\n CommandDispatcher->>Desktop: Screenshot all screens\n Desktop-->>Strategy: screenshot_url\n Strategy->>Strategy: Save to log_path\n \n Strategy->>CommandDispatcher: get_desktop_app_target_info\n CommandDispatcher->>Desktop: Query windows\n Desktop-->>Strategy: app_windows_info[]\n \n Strategy->>TargetRegistry: Register applications\n Strategy->>TargetRegistry: Register third-party agents\n TargetRegistry-->>Strategy: target_registry\n \n Strategy->>Strategy: Prepare target_info_list\n Strategy-->>Strategy: Return ProcessingResult\n```\n\n### Step 1: Capture Desktop Screenshot\n\n**Code**:\n```python\nasync def _capture_desktop_screenshot(\n self,\n command_dispatcher: BasicCommandDispatcher,\n save_path: str,\n) -> str:\n \"\"\"Capture desktop screenshot with error handling\"\"\"\n result = await command_dispatcher.execute_commands([\n Command(\n tool_name=\"capture_desktop_screenshot\",\n parameters={\"all_screens\": True},\n tool_type=\"data_collection\",\n )\n ])\n \n desktop_screenshot_url = result[0].result\n utils.save_image_string(desktop_screenshot_url, save_path)\n return desktop_screenshot_url\n```\n\n**Outputs**:\n- `desktop_screenshot_url`: Base64 encoded screenshot for LLM\n- `desktop_screenshot_path`: File path for logging (`action_step{N}.png`)\n\n**Multi-Screen Support:**\n\nThe `all_screens: True` parameter captures all connected monitors in a single composite image, providing complete desktop context.\n\n### Step 2: Collect Application Window Information\n\n**Code**:\n```python\nasync def _get_desktop_application_info(\n self, command_dispatcher: BasicCommandDispatcher\n) -> List[TargetInfo]:\n \"\"\"Get comprehensive desktop application information\"\"\"\n result = await command_dispatcher.execute_commands([\n Command(\n tool_name=\"get_desktop_app_target_info\",\n parameters={\n \"remove_empty\": True,\n \"refresh_app_windows\": True\n },\n tool_type=\"data_collection\",\n )\n ])\n \n app_windows_info = result[0].result or []\n target_info = [TargetInfo(**control_info) for control_info in app_windows_info]\n return target_info\n```\n\n**Outputs**:\n- List of `TargetInfo` objects containing:\n - `id`: Unique identifier (index-based)\n - `name`: Window title or process name\n - `kind`: Target type (APPLICATION, PROCESS, etc.)\n - `type`: Detailed type information\n - Additional metadata (position, size, state)\n\n**Window Filtering:**\n\n`remove_empty: True` filters out windows without valid handles or titles, reducing noise for LLM decision making.\n\n### Step 3: Register Applications and Third-Party Agents\n\n**Code**:\n```python\ndef _register_applications_and_agents(\n self,\n app_windows_info: List[TargetInfo],\n target_registry: TargetRegistry = None,\n) -> TargetRegistry:\n \"\"\"Register desktop applications and third-party agents\"\"\"\n if not target_registry:\n target_registry = TargetRegistry()\n \n # Register desktop application windows\n target_registry.register(app_windows_info)\n \n # Register third-party agents\n third_party_count = self._register_third_party_agents(\n target_registry, len(app_windows_info)\n )\n \n return target_registry\n\ndef _register_third_party_agents(\n self, target_registry: TargetRegistry, start_index: int\n) -> int:\n \"\"\"Register enabled third-party agents\"\"\"\n third_party_agent_names = ufo_config.system.enabled_third_party_agents\n \n third_party_agent_list = []\n for i, agent_name in enumerate(third_party_agent_names):\n agent_id = str(i + start_index + 1)\n third_party_agent_list.append(\n TargetInfo(\n kind=TargetKind.THIRD_PARTY_AGENT.value,\n id=agent_id,\n type=\"ThirdPartyAgent\",\n name=agent_name,\n )\n )\n \n target_registry.register(third_party_agent_list)\n return len(third_party_agent_list)\n```\n\n**Target Registry**:\n\n| Component | Purpose |\n|-----------|---------|\n| **TargetRegistry** | Centralized registry of all selectable targets |\n| **Applications** | Desktop windows (Word, Excel, browser, etc.) |\n| **Third-Party Agents** | Custom agents from configuration |\n| **Indexing** | Sequential IDs for LLM selection (0, 1, 2, ...) |\n\n**Target Registry Example:**\n\n```json\n[\n {\"id\": \"0\", \"name\": \"Microsoft Word - Document1\", \"kind\": \"APPLICATION\"},\n {\"id\": \"1\", \"name\": \"Microsoft Excel - Workbook1\", \"kind\": \"APPLICATION\"},\n {\"id\": \"2\", \"name\": \"Chrome - GitHub\", \"kind\": \"APPLICATION\"},\n {\"id\": \"3\", \"name\": \"HardwareAgent\", \"kind\": \"THIRD_PARTY_AGENT\"}\n]\n```\n\n### Processing Result\n\n**Outputs**:\n```python\nProcessingResult(\n success=True,\n data={\n \"desktop_screenshot_url\": \"data:image/png;base64,...\",\n \"desktop_screenshot_path\": \"C:/logs/action_step1.png\",\n \"application_windows_info\": [TargetInfo(...), ...],\n \"target_registry\": TargetRegistry(...),\n \"target_info_list\": [{\"id\": \"0\", \"name\": \"Word\", \"kind\": \"APPLICATION\"}, ...]\n },\n phase=ProcessingPhase.DATA_COLLECTION\n)\n```\n\n---\n\n## Phase 2: LLM_INTERACTION\n\n### Strategy: `HostLLMInteractionStrategy`\n\n**Purpose**: Construct context-aware prompts and obtain LLM decisions for application selection and task decomposition.\n\n```python\n@depends_on(\"target_info_list\", \"desktop_screenshot_url\")\n@provides(\n \"parsed_response\",\n \"response_text\",\n \"llm_cost\",\n \"prompt_message\",\n \"subtask\",\n \"plan\",\n \"result\",\n \"host_message\",\n \"status\",\n \"question_list\",\n \"function_name\",\n \"function_arguments\",\n)\nclass HostLLMInteractionStrategy(BaseProcessingStrategy):\n \"\"\"Enhanced LLM interaction strategy for Host Agent\"\"\"\n \n def __init__(self, fail_fast: bool = True):\n super().__init__(name=\"host_llm_interaction\", fail_fast=fail_fast)\n```\n\n### Execution Steps\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant HostAgent\n participant Blackboard\n participant Prompter\n participant LLM\n \n Strategy->>HostAgent: Get previous plan from memory\n Strategy->>Blackboard: Get blackboard context\n Blackboard-->>Strategy: blackboard_prompt[]\n \n Strategy->>Prompter: Build comprehensive prompt\n Prompter->>Prompter: Construct system message\n Prompter->>Prompter: Construct user message\n Prompter-->>Strategy: prompt_message\n \n Strategy->>Strategy: Log request data\n \n Strategy->>LLM: Send prompt with retry logic\n LLM-->>Strategy: response_text, cost\n \n Strategy->>Strategy: Parse & validate response\n Strategy->>HostAgent: print_response()\n \n Strategy->>Strategy: Extract structured data\n Strategy-->>Strategy: Return ProcessingResult\n```\n\n### Step 1: Build Comprehensive Prompt\n\n**Code**:\n```python\nasync def _build_comprehensive_prompt(\n self,\n agent: \"HostAgent\",\n target_info_list: List[Any],\n desktop_screenshot_url: str,\n prev_plan: List[Any],\n previous_subtasks: List[Any],\n request: str,\n session_step: int,\n request_logger,\n) -> Dict[str, Any]:\n \"\"\"Build comprehensive prompt message\"\"\"\n host_agent: \"HostAgent\" = agent\n \n # Get blackboard context if available\n blackboard_prompt = []\n if not host_agent.blackboard.is_empty():\n blackboard_prompt = host_agent.blackboard.blackboard_to_prompt()\n \n # Build complete prompt message\n prompt_message = host_agent.message_constructor(\n image_list=[desktop_screenshot_url] if desktop_screenshot_url else [],\n os_info=target_info_list,\n plan=prev_plan,\n prev_subtask=previous_subtasks,\n request=request,\n blackboard_prompt=blackboard_prompt,\n )\n \n return prompt_message\n```\n\n**Prompt Components**:\n\n| Component | Source | Purpose |\n|-----------|--------|---------|\n| **System Message** | Prompter template | Define agent role and capabilities |\n| **Desktop Screenshot** | Phase 1 | Visual context |\n| **Target List** | Phase 1 | Available applications |\n| **User Request** | Session context | Original task description |\n| **Previous Subtasks** | Session context | Completed steps |\n| **Previous Plan** | Agent memory | Future steps from last round |\n| **Blackboard** | Shared memory | Inter-agent communication |\n\n**Blackboard Integration:**\n\nThe Blackboard provides inter-agent communication by including results from AppAgents in the prompt:\n\n```python\nblackboard_prompt = [\n {\"role\": \"user\", \"content\": \"Previous result from Word AppAgent: Table data extracted\"}\n]\n```\n\n### Step 2: Get LLM Response with Retry\n\n**Code**:\n```python\nasync def _get_llm_response_with_retry(\n self, host_agent: \"HostAgent\", prompt_message: Dict[str, Any]\n) -> tuple[str, float]:\n \"\"\"Get LLM response with retry logic for JSON parsing failures\"\"\"\n max_retries = ufo_config.system.json_parsing_retry\n \n for retry_count in range(max_retries):\n try:\n # Run synchronous LLM call in thread executor\n loop = asyncio.get_event_loop()\n response_text, cost = await loop.run_in_executor(\n None,\n host_agent.get_response,\n prompt_message,\n AgentType.HOST,\n True, # use_backup_engine\n )\n \n # Validate response can be parsed as JSON\n host_agent.response_to_dict(response_text)\n \n return response_text, cost\n \n except Exception as e:\n if retry_count < max_retries - 1:\n self.logger.warning(f\"Retry {retry_count + 1}/{max_retries}: {e}\")\n else:\n raise Exception(f\"Failed after {max_retries} attempts: {e}\")\n```\n\n!!!note \"WebSocket Timeout Fix\"\n The code uses `run_in_executor` to prevent blocking the event loop during long LLM responses, which could cause WebSocket ping/pong timeouts in MCP connections.\n\n### Step 3: Parse and Validate Response\n\n**Code**:\n```python\ndef _parse_and_validate_response(\n self, host_agent: \"HostAgent\", response_text: str\n) -> HostAgentResponse:\n \"\"\"Parse and validate LLM response\"\"\"\n # Parse response to dictionary\n response_dict = host_agent.response_to_dict(response_text)\n \n # Create structured response object\n parsed_response = HostAgentResponse.model_validate(response_dict)\n \n # Validate required fields\n self._validate_response_fields(parsed_response)\n \n # Print response for user feedback\n host_agent.print_response(parsed_response)\n \n return parsed_response\n\ndef _validate_response_fields(self, response: HostAgentResponse):\n \"\"\"Validate response contains required fields\"\"\"\n if not response.observation:\n raise ValueError(\"Response missing required 'observation' field\")\n if not response.thought:\n raise ValueError(\"Response missing required 'thought' field\")\n if not response.status:\n raise ValueError(\"Response missing required 'status' field\")\n \n valid_statuses = [\"CONTINUE\", \"FINISH\", \"CONFIRM\", \"ERROR\", \"ASSIGN\"]\n if response.status.upper() not in valid_statuses:\n self.logger.warning(f\"Unexpected status value: {response.status}\")\n```\n\n**HostAgentResponse Structure**:\n\n```python\nclass HostAgentResponse(BaseModel):\n observation: str # What the agent sees\n thought: str # Reasoning process\n current_subtask: str # Current subtask description\n message: str # Message for AppAgent\n control_label: str # Selected target ID\n control_text: str # Selected target name\n plan: List[str] # Future subtasks\n status: str # Next state (ASSIGN/CONTINUE/FINISH/etc.)\n comment: str # User-facing comment\n questions: List[str] # Clarification questions\n function: str # Command to execute\n arguments: Dict[str, Any] # Command arguments\n result: str # Result description\n```\n\n### Processing Result\n\n**Outputs**:\n```python\nProcessingResult(\n success=True,\n data={\n \"parsed_response\": HostAgentResponse(...),\n \"response_text\": '{\"Observation\": \"...\", ...}',\n \"llm_cost\": 0.025,\n \"prompt_message\": [...],\n \"subtask\": \"Extract table from Word\",\n \"plan\": [\"Create chart in Excel\"],\n \"host_message\": \"Starting extraction\",\n \"status\": \"ASSIGN\",\n \"result\": \"\",\n \"question_list\": [],\n \"function_name\": \"select_application_window\",\n \"function_arguments\": {\"id\": \"0\"}\n },\n phase=ProcessingPhase.LLM_INTERACTION\n)\n```\n\n!!!example \"LLM Response Example\"\n ```json\n {\n \"Observation\": \"Desktop shows Word with table and Excel empty\",\n \"Thought\": \"Need to extract table from Word first before creating chart\",\n \"Current Sub-Task\": \"Extract sales table from Word document\",\n \"Message\": \"Please extract the table data for chart creation\",\n \"ControlLabel\": \"0\",\n \"ControlText\": \"Microsoft Word - Sales Report\",\n \"Plan\": [\"Extract table\", \"Create bar chart in Excel\"],\n \"Status\": \"ASSIGN\",\n \"Comment\": \"Starting data extraction from Word\",\n \"Questions\": [],\n \"Function\": \"select_application_window\",\n \"Args\": {\"id\": \"0\"}\n }\n ```\n\n---\n\n## Phase 3: ACTION_EXECUTION\n\n### Strategy: `HostActionExecutionStrategy`\n\n**Purpose**: Execute LLM-decided actions including application selection, third-party agent assignment, and generic command execution.\n\n```python\n@depends_on(\"target_registry\", \"command_dispatcher\")\n@provides(\n \"execution_result\",\n \"action_info\",\n \"selected_target_id\",\n \"selected_application_root\",\n \"assigned_third_party_agent\",\n \"target\",\n)\nclass HostActionExecutionStrategy(BaseProcessingStrategy):\n \"\"\"Enhanced action execution strategy for Host Agent\"\"\"\n \n SELECT_APPLICATION_COMMAND: str = \"select_application_window\"\n \n def __init__(self, fail_fast: bool = False):\n super().__init__(name=\"host_action_execution\", fail_fast=fail_fast)\n```\n\n### Execution Flow\n\n```mermaid\ngraph TD\n Start[Start Action Execution] --> CheckFunc{Function Name?}\n \n CheckFunc -->|select_application_window| SelectApp[Execute Application Selection]\n CheckFunc -->|Other Command| Generic[Execute Generic Command]\n CheckFunc -->|None| NoAction[No Action]\n \n SelectApp --> CheckKind{Target Kind?}\n \n CheckKind -->|THIRD_PARTY_AGENT| ThirdParty[Assign Third-Party Agent]\n CheckKind -->|APPLICATION| RegularApp[Select Regular Application]\n \n ThirdParty --> CreateAction[Create Action Info]\n RegularApp --> MCP[Execute MCP Command]\n MCP --> CreateAction\n Generic --> CreateAction\n NoAction --> CreateAction\n \n CreateAction --> Return[Return ProcessingResult]\n \n style SelectApp fill:#e3f2fd\n style ThirdParty fill:#fff3e0\n style RegularApp fill:#f1f8e9\n style Generic fill:#fce4ec\n```\n\n### Application Selection\n\n**Code**:\n```python\nasync def _execute_application_selection(\n self,\n parsed_response: HostAgentResponse,\n target_registry: TargetRegistry,\n command_dispatcher: BasicCommandDispatcher,\n) -> List[Result]:\n \"\"\"Execute application selection\"\"\"\n target_id = parsed_response.arguments.get(\"id\")\n target = target_registry.get(target_id)\n \n # Handle third-party agent selection\n if target.kind == TargetKind.THIRD_PARTY_AGENT:\n return await self._select_third_party_agent(target)\n # Handle regular application selection\n else:\n return await self._select_regular_application(target, command_dispatcher)\n```\n\n#### Third-Party Agent Selection\n\n**Code**:\n```python\nasync def _select_third_party_agent(self, target: TargetInfo) -> List[Result]:\n \"\"\"Handle third-party agent selection\"\"\"\n self.logger.info(f\"Assigned third-party agent: {target.name}\")\n \n return [\n Result(\n status=\"success\",\n result={\n \"id\": target.id,\n \"name\": target.name,\n \"type\": \"third_party_agent\",\n },\n )\n ]\n```\n\n!!!info \"Third-Party Agents\"\n Third-party agents are custom agents registered in configuration:\n ```yaml\n enabled_third_party_agents:\n - HardwareAgent\n - NetworkAgent\n ```\n \n They are selected like applications but don't require window management.\n\n#### Regular Application Selection\n\n**Code**:\n```python\nasync def _select_regular_application(\n self, target: TargetInfo, command_dispatcher: BasicCommandDispatcher\n) -> List[Result]:\n \"\"\"Handle regular application selection\"\"\"\n execution_result = await command_dispatcher.execute_commands([\n Command(\n tool_name=\"select_application_window\",\n parameters={\"id\": str(target.id), \"name\": target.name},\n tool_type=\"action\",\n )\n ])\n \n if execution_result and execution_result[0].result:\n app_root = execution_result[0].result.get(\"root_name\", \"\")\n self.logger.info(f\"Selected application: {target.name}, root: {app_root}\")\n \n return execution_result\n```\n\n**Window Selection Actions**:\n1. Focuses application window\n2. Brings window to foreground\n3. Retrieves application root name (for AppAgent configuration)\n4. Updates global context with window information\n\n### Generic Command Execution\n\n**Code**:\n```python\nasync def _execute_generic_command(\n self,\n parsed_response: HostAgentResponse,\n command_dispatcher: BasicCommandDispatcher,\n) -> List[Result]:\n \"\"\"Execute generic command\"\"\"\n function_name = parsed_response.function\n arguments = parsed_response.arguments or {}\n \n execution_result = await command_dispatcher.execute_commands([\n Command(\n tool_name=function_name,\n parameters=arguments,\n tool_type=\"action\",\n )\n ])\n \n return execution_result\n```\n\n**Generic Commands:**\n\n- `launch_application`: Start new application\n- `close_application`: Terminate application\n- `bash_command`: Execute shell command\n- Custom MCP tools\n\n### Action Info Creation\n\n**Code**:\n```python\ndef _create_action_info(\n self,\n parsed_response: HostAgentResponse,\n execution_result: List[Result],\n target_registry: TargetRegistry,\n selected_target_id: str,\n) -> ActionCommandInfo:\n \"\"\"Create action information object for memory\"\"\"\n target_object = None\n if target_registry and selected_target_id:\n target_object = target_registry.get(selected_target_id)\n \n action_info = ActionCommandInfo(\n function=parsed_response.function,\n arguments=parsed_response.arguments or {},\n target=target_object,\n status=parsed_response.status,\n result=execution_result[0] if execution_result else Result(status=\"none\"),\n )\n \n return action_info\n```\n\n**ActionCommandInfo Structure**:\n\n| Field | Type | Description |\n|-------|------|-------------|\n| `function` | str | Command name executed |\n| `arguments` | Dict | Command parameters |\n| `target` | TargetInfo | Selected target object |\n| `status` | str | Agent status after execution |\n| `result` | Result | Execution result |\n\n### Processing Result\n\n**Outputs**:\n```python\nProcessingResult(\n success=True,\n data={\n \"execution_result\": [Result(...)],\n \"action_info\": ActionCommandInfo(...),\n \"target\": TargetInfo(...),\n \"selected_target_id\": \"0\",\n \"selected_application_root\": \"WINWORD\",\n \"assigned_third_party_agent\": \"\",\n },\n phase=ProcessingPhase.ACTION_EXECUTION\n)\n```\n\n---\n\n## Phase 4: MEMORY_UPDATE\n\n### Strategy: `HostMemoryUpdateStrategy`\n\n**Purpose**: Record orchestration step in agent memory, update structural logs, and maintain Blackboard trajectories.\n\n```python\n@depends_on(\"session_step\")\n@provides(\"additional_memory\", \"memory_item\", \"memory_keys_count\")\nclass HostMemoryUpdateStrategy(BaseProcessingStrategy):\n \"\"\"Enhanced memory update strategy for Host Agent\"\"\"\n \n def __init__(self, fail_fast: bool = False):\n super().__init__(name=\"host_memory_update\", fail_fast=fail_fast)\n```\n\n### Execution Steps\n\n```mermaid\nsequenceDiagram\n participant Strategy\n participant Context\n participant MemoryItem\n participant AgentMemory\n participant StructuralLogs\n participant Blackboard\n \n Strategy->>Context: Extract all processing data\n Strategy->>Strategy: Create additional_memory\n \n Strategy->>MemoryItem: new MemoryItem()\n Strategy->>MemoryItem: add_values_from_dict(response)\n Strategy->>MemoryItem: add_values_from_dict(additional_memory)\n \n Strategy->>AgentMemory: add_memory(memory_item)\n Strategy->>StructuralLogs: add_to_structural_logs(memory_dict)\n \n Strategy->>Blackboard: add_trajectories(memorized_action)\n \n Strategy-->>Strategy: Return ProcessingResult\n```\n\n### Step 1: Create Additional Memory Data\n\n**Code**:\n```python\ndef _create_additional_memory_data(\n self, agent: \"HostAgent\", context: ProcessingContext\n) -> \"HostAgentProcessorContext\":\n \"\"\"Create comprehensive additional memory data\"\"\"\n host_context: HostAgentProcessorContext = context.local_context\n \n # Update context with current state\n host_context.session_step = context.get_global(ContextNames.SESSION_STEP.name, 0)\n host_context.round_step = context.get_global(ContextNames.CURRENT_ROUND_STEP.name, 0)\n host_context.round_num = context.get_global(ContextNames.CURRENT_ROUND_ID.name, 0)\n host_context.agent_step = agent.step if agent else 0\n \n action_info: ActionCommandInfo = host_context.action_info\n \n # Update action information\n if action_info:\n host_context.action = [action_info.model_dump()]\n host_context.function_call = action_info.function or \"\"\n host_context.arguments = action_info.arguments\n host_context.action_representation = action_info.to_representation()\n \n if action_info.result and action_info.result.result:\n host_context.results = str(action_info.result.result)\n \n # Update application and agent names\n host_context.application = host_context.selected_application_root or \"\"\n host_context.agent_name = agent.name\n \n return host_context\n```\n\n**Additional Memory Fields**:\n\n| Field | Description |\n|-------|-------------|\n| `session_step` | Global session step counter |\n| `round_step` | Step within current round |\n| `round_num` | Current round number |\n| `agent_step` | HostAgent's own step counter |\n| `action` | Executed action details |\n| `function_call` | Command name |\n| `arguments` | Command parameters |\n| `action_representation` | Human-readable action description |\n| `results` | Execution results |\n| `application` | Selected application root |\n| `agent_name` | \"HostAgent\" |\n\n### Step 2: Create and Populate Memory Item\n\n**Code**:\n```python\ndef _create_and_populate_memory_item(\n self,\n parsed_response: HostAgentResponse,\n additional_memory: \"HostAgentProcessorContext\",\n) -> MemoryItem:\n \"\"\"Create and populate memory item\"\"\"\n memory_item = MemoryItem()\n \n # Add response data\n if parsed_response:\n memory_item.add_values_from_dict(parsed_response.model_dump())\n \n # Add additional memory data\n memory_item.add_values_from_dict(additional_memory.to_dict(selective=True))\n \n return memory_item\n```\n\n**MemoryItem Contents**:\n\n```python\n{\n # From HostAgentResponse\n \"observation\": \"Desktop shows Word and Excel...\",\n \"thought\": \"Need to extract table first...\",\n \"current_subtask\": \"Extract table from Word\",\n \"plan\": [\"Create chart in Excel\"],\n \"status\": \"ASSIGN\",\n \n # From Additional Memory\n \"session_step\": 1,\n \"round_num\": 0,\n \"round_step\": 0,\n \"agent_step\": 0,\n \"action\": [{\"function\": \"select_application_window\", ...}],\n \"application\": \"WINWORD\",\n \"agent_name\": \"HostAgent\",\n ...\n}\n```\n\n### Step 3: Update Structural Logs\n\n**Code**:\n```python\ndef _update_structural_logs(self, memory_item: MemoryItem, global_context):\n \"\"\"Update structural logs for debugging\"\"\"\n global_context.add_to_structural_logs(memory_item.to_dict())\n```\n\n**Structural Logs:**\n\nStructural logs provide machine-readable JSON logs of every agent step for debugging and analysis, replay and reproduction, performance monitoring, and training data collection.\n\n### Step 4: Update Blackboard Trajectories\n\n**Code**:\n```python\ndef _update_blackboard_trajectories(\n self,\n host_agent: \"HostAgent\",\n memory_item: MemoryItem,\n):\n \"\"\"Update blackboard trajectories\"\"\"\n history_keys = ufo_config.system.history_keys\n \n memory_dict = memory_item.to_dict()\n memorized_action = {\n key: memory_dict.get(key) for key in history_keys if key in memory_dict\n }\n \n if memorized_action:\n host_agent.blackboard.add_trajectories(memorized_action)\n```\n\n**Blackboard Trajectories**:\n\n```python\n# Configuration\nhistory_keys = [\"observation\", \"thought\", \"current_subtask\", \"status\", \"result\"]\n\n# Stored in Blackboard\n{\n \"step_0\": {\n \"observation\": \"Desktop shows Word and Excel\",\n \"thought\": \"Extract table first\",\n \"current_subtask\": \"Extract table\",\n \"status\": \"ASSIGN\",\n \"result\": \"\"\n },\n \"step_1\": {\n \"observation\": \"Word AppAgent extracted table\",\n \"thought\": \"Now create chart in Excel\",\n \"current_subtask\": \"Create bar chart\",\n \"status\": \"ASSIGN\",\n \"result\": \"Table data: [...]\"\n }\n}\n```\n\n**Inter-Agent Communication:**\n\nBlackboard trajectories enable AppAgents to access HostAgent's orchestration history, providing context for their execution.\n\n### Processing Result\n\n**Outputs**:\n```python\nProcessingResult(\n success=True,\n data={\n \"additional_memory\": HostAgentProcessorContext(...),\n \"memory_item\": MemoryItem(...),\n \"memory_keys_count\": 25\n },\n phase=ProcessingPhase.MEMORY_UPDATE\n)\n```\n\n---\n\n## Complete Processing Flow\n\n### Multi-Step Example\n\n**User Request**: \"Extract table from Word and create chart in Excel\"\n\n**Round 1**: Select Word\n\n| Phase | Key Operations | Outputs |\n|-------|----------------|---------|\n| DATA_COLLECTION | Capture desktop, list windows | screenshot, [Word, Excel] |\n| LLM_INTERACTION | Analyze, select Word | Status=ASSIGN, target_id=0 |\n| ACTION_EXECUTION | Select Word window | app_root=\"WINWORD\" |\n| MEMORY_UPDATE | Record step | memory_item added |\n\n**Round 2**: Create Excel Chart\n\n| Phase | Key Operations | Outputs |\n|-------|----------------|---------|\n| DATA_COLLECTION | Capture desktop, list windows | screenshot, [Word, Excel] |\n| LLM_INTERACTION | Analyze Word result, select Excel | Status=ASSIGN, target_id=1 |\n| ACTION_EXECUTION | Select Excel window | app_root=\"EXCEL\" |\n| MEMORY_UPDATE | Record step | memory_item added |\n\n**Round 3**: Verify Completion\n\n| Phase | Key Operations | Outputs |\n|-------|----------------|---------|\n| DATA_COLLECTION | Capture desktop | screenshot |\n| LLM_INTERACTION | Verify chart created | Status=FINISH |\n| ACTION_EXECUTION | No action | - |\n| MEMORY_UPDATE | Record completion | memory_item added |\n\n---\n\n## Error Handling\n\n### Strategy-Level Error Handling\n\nEach strategy implements robust error handling:\n\n```python\nasync def execute(self, agent, context) -> ProcessingResult:\n try:\n # Execute strategy logic\n return ProcessingResult(success=True, data={...})\n except Exception as e:\n error_msg = f\"{self.name} failed: {str(e)}\"\n self.logger.error(error_msg)\n return self.handle_error(e, self.phase, context)\n```\n\n**Error Handling Modes**:\n\n| Strategy | `fail_fast` | Behavior |\n|----------|-------------|----------|\n| DATA_COLLECTION | True | Stop immediately on failure |\n| LLM_INTERACTION | True | Stop immediately on failure |\n| ACTION_EXECUTION | False | Log error, continue |\n| MEMORY_UPDATE | False | Log error, continue |\n\n!!!warning \"Critical vs Non-Critical Failures\"\n - **Critical** (fail_fast=True): Desktop capture, LLM interaction\n - **Non-Critical** (fail_fast=False): Action execution, memory update\n \n Critical failures prevent further processing, while non-critical failures are logged but don't stop the pipeline.\n\n---\n\n## Performance Considerations\n\n### Async Execution\n\nAll strategies use async/await for non-blocking I/O:\n\n```python\n# Non-blocking screenshot capture\nresult = await command_dispatcher.execute_commands([...])\n\n# Non-blocking LLM call (with thread executor)\nloop = asyncio.get_event_loop()\nresponse = await loop.run_in_executor(None, llm_call, ...)\n```\n\n### Retry Logic\n\nLLM interaction includes automatic retry for transient failures:\n\n```python\nmax_retries = ufo_config.system.json_parsing_retry # Default: 3\n\nfor retry_count in range(max_retries):\n try:\n response = await get_llm_response(...)\n validate_json(response)\n return response\n except Exception as e:\n if retry_count < max_retries - 1:\n continue\n raise\n```\n\n### Caching\n\nTarget registry can be reused across rounds:\n\n```python\nexisting_target_registry = context.get_local(\"target_registry\")\ntarget_registry = self._register_applications_and_agents(\n app_windows_info, existing_target_registry\n)\n```\n\n---\n\n## Related Documentation\n\n**Architecture & Design:**\n\n- **[Overview](overview.md)**: HostAgent high-level architecture\n- **[State Machine](state.md)**: When strategies are executed\n- **[Processor Framework](../../infrastructure/agents/design/processor.md)**: General processor architecture\n\n**System Integration:**\n\n- **[Command System](commands.md)**: Available desktop commands\n- **[Blackboard](../../infrastructure/agents/design/blackboard.md)**: Inter-agent communication\n- **[Memory System](../../infrastructure/agents/design/memory.md)**: Memory management\n\n---\n\n## Summary\n\n**Key Takeaways:**\n\n- **4 Phases**: DATA_COLLECTION → LLM_INTERACTION → ACTION_EXECUTION → MEMORY_UPDATE\n- **Desktop Context**: Capture screenshot + application list\n- **LLM Decision**: Select application, decompose task, set status\n- **Action Types**: Application selection, third-party agent assignment, generic commands\n- **Memory Persistence**: Record every step for context and replay\n- **Blackboard Integration**: Share trajectories with AppAgents\n- **Error Resilience**: Retry logic, fail-fast configuration, graceful degradation\n\n**Next Steps:**\n\n- Read [Command System](commands.md) for available desktop operations\n- Read [State Machine](state.md) to understand when processing occurs\n- Read [Blackboard](../../infrastructure/agents/design/blackboard.md) for inter-agent communication\n- Learn [Creating Third-Party Agents](../../tutorials/creating_third_party_agents.md) to build custom agents\n"
},
{
"path": "documents/docs/ufo2/overview.md",
"content": "# UFO² — Windows AgentOS\n\n[](https://arxiv.org/abs/2504.14603) \n \n[](https://opensource.org/licenses/MIT) \n[](https://github.com/microsoft/UFO) \n[](https://www.youtube.com/watch?v=QT_OhygMVXU) \n\n\n**UFO²** is a Windows AgentOS that reimagines desktop automation as a first-class operating system abstraction. Unlike traditional Computer-Using Agents (CUAs) that rely on screenshots and simulated inputs, UFO² deeply integrates with Windows OS through UI Automation APIs, application-specific introspection, and hybrid GUI–API execution—enabling robust, efficient, and non-disruptive automation across 20+ real-world applications.\n\n---\n\n## What is UFO²?\n\nUFO² addresses fundamental limitations of existing desktop automation solutions:\n\n**Traditional RPA (UiPath, Power Automate):** \n❌ Fragile scripts that break with UI changes \n❌ Requires extensive manual maintenance \n❌ Limited adaptability to dynamic environments\n\n**Current CUAs (Claude, Operator):** \n❌ Visual-only inputs with high cognitive overhead \n❌ Miss native OS APIs and application internals \n❌ Lock users out during automation (poor UX)\n\n**UFO² AgentOS:** \n✅ **Deep OS Integration** — Windows UIA, Win32, WinCOM APIs \n✅ **Hybrid GUI–API Actions** — Native APIs + fallback GUI automation \n✅ **Continuous Knowledge Learning** — RAG-enhanced from docs & execution history \n✅ **Picture-in-Picture Desktop** — Parallel automation without user disruption \n✅ **10%+ better success rate** than state-of-the-art CUAs\n\n\n \n Figure 1: Comparison between (a) traditional CUAs that rely on screenshots and simulated inputs, and (b) UFO² AgentOS that deeply integrates with OS APIs, application internals, and hybrid GUI–API execution.\n\n\n---\n\n## Core Architecture\n\nUFO² implements a **hierarchical multi-agent system** optimized for Windows desktop automation:\n\n\n \n Figure 2: UFO² system architecture featuring the two-tier agent hierarchy (HostAgent + AppAgents), hybrid control detection pipeline, continuous knowledge substrate integration, and unified GUI–API action layer coordinated through MCP servers.\n\n\n\n### Two-Tier Agent Hierarchy\n\n| Agent Type | Role | Key Capabilities |\n|------------|------|------------------|\n| **HostAgent** | Desktop Orchestrator | Task decomposition • Application selection • Cross-app coordination • AppAgent lifecycle management |\n| **AppAgent** | Application Executor | UI element interaction • Hybrid GUI–API execution • Application-specific automation • Result reporting |\n\n**Design Philosophy:** \n- **HostAgent** handles **WHAT** (which application) and **WHEN** (task sequencing) \n- **AppAgent** handles **HOW** (UI/API interaction) and **WHERE** (control targeting) \n- **Blackboard** facilitates inter-agent communication without tight coupling \n- **State Machines** ensure deterministic execution flow and error recovery\n\n!!!info \"Learn More\"\n - [**HostAgent Documentation**](host_agent/overview.md) — 7-state FSM, desktop orchestration, AppAgent lifecycle \n - [**AppAgent Documentation**](app_agent/overview.md) — 6-state FSM, UI automation, hybrid action execution \n - [**Agent Architecture**](../infrastructure/agents/overview.md) — Three-layer design principles\n\n---\n\n## Key Innovations\n\n### 1. Deep OS Integration 🔧\n\nUFO² embeds directly into Windows OS infrastructure:\n\n- **UI Automation (UIA):** Introspects accessibility trees for standard controls \n- **Win32 APIs:** Low-level window management and process control \n- **WinCOM:** Interacts with Office applications (Excel, Word, Outlook) \n- **Hybrid Detection:** Fuses UIA metadata + visual grounding for non-standard UI elements\n\n!!!tip \"Hybrid Control Detection\"\n Combines Windows UIA APIs with vision models ([OmniParser](https://arxiv.org/abs/2408.00203)) to detect both standard and custom UI controls—bridging structured accessibility trees and pixel-level perception.\n \n 📖 [Control Detection Guide](core_features/control_detection/overview.md)\n\n### 2. Unified GUI–API Action Layer ⚡\n\nTraditional CUAs simulate mouse/keyboard only. UFO² chooses the best execution method:\n\n**GUI Actions** (fallback): \n`click`, `type`, `select`, `scroll` → Reliable for any application\n\n**Native APIs** (preferred): \n- Excel: `xlwings` for direct cell/chart manipulation \n- Outlook: `win32com` for email operations \n- PowerPoint: `python-pptx` for slide editing \n→ **51% fewer LLM calls** via speculative multi-action execution\n\n**Model Context Protocol (MCP) Servers:** \nExtensible framework for adding application-specific APIs without modifying agent code.\n\n!!!info \"Learn More\"\n 📖 [Hybrid Actions Guide](core_features/hybrid_actions.md) • [MCP Integration](../mcp/overview.md)\n\n### 3. Continuous Knowledge Substrate 📚\n\nUFO² learns from three knowledge sources without model retraining:\n\n| Source | Content | Integration Method |\n|--------|---------|-------------------|\n| **Help Documents** | Official app documentation, API references | Vectorized retrieval (RAG) |\n| **Bing Search** | Real-time web knowledge for latest features | Dynamic query expansion |\n| **Execution History** | Past successful/failed action sequences | Experience replay & pattern mining |\n\n**Result:** Agents improve autonomously by retrieving relevant context at execution time.\n\n!!!info \"Knowledge Integration\"\n 📖 [Knowledge Substrate Overview](core_features/knowledge_substrate/overview.md) \n 📖 [Learning from Help Documents](core_features/knowledge_substrate/learning_from_help_document.md) \n 📖 [Experience Learning](core_features/knowledge_substrate/experience_learning.md)\n\n### 4. Speculative Multi-Action Execution 🚀\n\nReduce LLM latency by predicting and validating action sequences:\n\n**Traditional Approach:** \n1 LLM call → 1 action → observe → repeat → **High latency**\n\n**UFO² Speculative Execution:** \n1 LLM call → predict N actions → validate with UI state → execute all → **51% fewer queries**\n\n**Validation Mechanism:** \nLightweight control-state checks ensure predicted actions remain valid before execution.\n\n!!!example \"Efficiency Gain\"\n **Task:** \"Fill form fields A1–A10 with sequential numbers\"\n \n - **Traditional CUA:** 10 LLM calls (1 per field) → ~30 seconds \n - **UFO² Speculative:** 1 LLM call predicts all 10 actions → ~8 seconds\n \n 📖 [Multi-Action Execution Guide](core_features/multi_action.md)\n\n### 5. Picture-in-Picture Desktop 🖼️\n\n**Problem:** Existing CUAs lock users out during automation (poor UX).\n\n**UFO² Solution:** Nested virtual desktop via Windows Remote Desktop loopback:\n\n- **User Desktop:** Continue working normally \n- **Agent Desktop (PiP):** Automation runs in parallel sandboxed environment \n- **Zero Interference:** User and agent don't compete for mouse/keyboard\n\n**Implementation:** \nBuilt on Windows native remote desktop infrastructure—secure, isolated, non-disruptive.\n\n!!!success \"User Experience\"\n Users can continue email, browsing, or coding while UFO² automates Excel reports in the background PiP desktop.\n\n---\n\n## System Components\n\n### Processing Pipeline\n\nBoth HostAgent and AppAgent execute a **4-phase processing cycle**:\n\n| Phase | Purpose | HostAgent Strategy | AppAgent Strategy |\n|-------|---------|-------------------|------------------|\n| **1. Data Collection** | Gather environment state | Desktop screenshot, app list | App screenshot, UI tree, control annotations |\n| **2. LLM Interaction** | Decide next action | Select application, plan subtask | Select control, plan action sequence |\n| **3. Action Execution** | Execute commands | Launch app, create AppAgent | Execute GUI/API actions |\n| **4. Memory Update** | Record execution | Save orchestration step | Save interaction step, update blackboard |\n\n!!!info \"Processing Details\"\n 📖 [Strategy Layer](../infrastructure/agents/design/processor.md) — Processing framework and dependency chain \n 📖 [State Layer](../infrastructure/agents/design/state.md) — FSM design principles\n\n### Command System\n\nCommands are dispatched through **MCP (Model Context Protocol)** servers:\n\n**HostAgent Commands:**\n\n- **Desktop Capture:** `capture_desktop_screenshot` \n- **Window Management:** `get_desktop_app_info`, `get_app_window` \n- **Process Control:** `launch_application`, `close_application`\n\n**AppAgent Commands:**\n\n- **Screenshot:** `capture_screenshot`, `annotate_screenshot` \n- **UI Inspection:** `get_control_info`, `get_ui_tree` \n- **UI Interaction:** `click`, `set_edit_text`, `wheel_mouse_input` \n- **Control Selection:** `select_control_by_index`, `select_control_by_name`\n\n!!!info \"Command Architecture\"\n 📖 [Command Layer](../infrastructure/agents/design/command.md) — MCP integration and command dispatch \n 📖 [MCP Servers](../mcp/overview.md) — Server architecture and custom server creation\n\n---\n\n\n## Configuration\n\nUFO² integrates with a centralized YAML-based configuration system:\n\n```yaml\n# config/ufo/host_agent_config.yaml\nhost_agent:\n visual_mode: true # Enable screenshot-based reasoning\n max_subtasks: 10 # Maximum subtasks per session\n llm_config:\n model: \"gpt-4o\"\n temperature: 0.0\n\n# config/ufo/app_agent_config.yaml\napp_agent:\n visual_mode: true # Enable UI screenshot analysis\n control_backend: \"uia\" # UI Automation (uia) or Win32 (win32)\n max_steps: 20 # Maximum steps per subtask\n```\n\n!!!tip \"Complete Configuration Guide\"\n For detailed configuration options, model setup, and advanced customization:\n \n 📖 **[Configuration & Setup](../configuration/system/overview.md)** — Complete system configuration reference \n 📖 **[Model Setup](../configuration/models/overview.md)** — LLM provider configuration (OpenAI, Azure, Gemini, Claude, etc.) \n 📖 **[MCP Configuration](../configuration/system/mcp_reference.md)** — MCP server and extension configuration\n\n---\n\n## Quick Start\n\n### Basic Usage\n\nUFO² is designed to be run from the command line:\n\n**Interactive Mode:**\n```powershell\n# Start UFO² in interactive mode\npython -m ufo --task \n```\n\n**Example:**\n```powershell\npython -m ufo --task excel_demo\n```\n\nThis will prompt you to enter your request interactively:\n```\nWelcome to use UFO🛸, A UI-focused Agent for Windows OS Interaction.\nPlease enter your request to be completed🛸: Create a chart from Sheet1 data in Excel\n```\n\n**Direct Request Mode:**\n```powershell\n# Execute with a specific request directly\npython -m ufo --task -r \"\"\n```\n\n**Example:**\n```powershell\npython -m ufo --task excel_demo -r \"Open Excel and create a chart from Sheet1 data\"\n```\n\n!!!tip \"Complete Setup Guide\"\n For detailed installation, configuration, and advanced usage options, see the **[Quick Start Guide](../getting_started/quick_start_ufo2.md)**.\n\n### What Happens Under the Hood\n\n1. **Session** creates **HostAgent** with user request \n2. **HostAgent** captures desktop, selects \"Microsoft Excel\", launches app \n3. **HostAgent** creates **AppAgent** for Excel, delegates subtask \n4. **AppAgent** captures Excel UI, identifies chart insertion control \n5. **AppAgent** executes hybrid action (API if available, GUI fallback) \n6. **AppAgent** reports completion to **HostAgent** \n7. **HostAgent** verifies task, returns success to **Session**\n\n!!!tip \"Next Steps\"\n 📖 [Getting Started Guide](../getting_started/quick_start_ufo2.md) \n 📖 [Creating Your AppAgent](../tutorials/creating_app_agent/overview.md)\n\n---\n\n## Documentation Navigation\n\n### Core Concepts\n\n- [**HostAgent**](host_agent/overview.md) — Desktop orchestrator with 7-state FSM \n- [**AppAgent**](app_agent/overview.md) — Application executor with 6-state FSM \n- [**Agent Types**](../infrastructure/agents/agent_types.md) — Platform-specific implementations \n- [**Evaluation Agent**](evaluation/evaluation_agent.md) — Automated testing and benchmarking\n\n### Advanced Features\n\n- [**Hybrid Actions**](core_features/hybrid_actions.md) — GUI–API execution layer \n- [**Control Detection**](core_features/control_detection/overview.md) — UIA + visual grounding \n- [**Knowledge Substrate**](core_features/knowledge_substrate/overview.md) — RAG-enhanced learning \n- [**Multi-Action Execution**](core_features/multi_action.md) — Speculative action planning \n- [**Follower Mode**](advanced_usage/follower_mode.md) — Human-in-the-loop execution \n- [**Batch Mode**](advanced_usage/batch_mode.md) — Bulk task processing\n\n### System Architecture\n\n- [**Device Agent Overview**](../infrastructure/agents/overview.md) — Three-layer architecture \n- [**State Layer**](../infrastructure/agents/design/state.md) — FSM design principles \n- [**Strategy Layer**](../infrastructure/agents/design/processor.md) — Processing framework \n- [**Command Layer**](../infrastructure/agents/design/command.md) — MCP integration \n\n### Development\n\n- [**Creating AppAgent**](../tutorials/creating_app_agent/overview.md) — Custom agent development \n- [**MCP Servers**](../mcp/overview.md) — Building custom MCP servers \n- [**Configuration**](../configuration/system/overview.md) — System configuration reference \n- [**Prompts**](prompts/overview.md) — Prompt engineering guide\n\n### Benchmarking & Logs\n\n- [**Benchmark Overview**](evaluation/benchmark/overview.md) — WindowsAgentArena, OSWorld \n- [**Performance Logs**](evaluation/logs/overview.md) — Execution logs and debugging \n\n---\n\n## Research Impact\n\nUFO² demonstrates that **system-level integration** and **architectural design** matter more than model size alone:\n\n!!!success \"Key Findings\"\n - **10%+ improvement** over Claude/Operator on WindowsAgentArena \n - **51% fewer LLM calls** via speculative multi-action execution \n - **Robust to UI changes** through hybrid UIA + visual detection \n - **Continuous learning** without model retraining via RAG \n - **Non-disruptive UX** via Picture-in-Picture desktop\n\n**Research Paper:** \n📄 [UFO²: A Grounded OS Agent for Windows](https://arxiv.org/abs/2504.14603)\n\n---\n\n## Get Started\n\nReady to explore UFO²? Choose your path:\n\n!!!info \"Learning Paths\"\n **🚀 New Users:** Start with [Quick Start Guide](../getting_started/quick_start_ufo2.md) \n **🔧 Developers:** Read [Creating AppAgent](../tutorials/creating_app_agent/overview.md) \n **🏗️ System Architects:** Study [Device Agent Architecture](../infrastructure/agents/overview.md) \n **📊 Researchers:** Check [Benchmark Results](evaluation/benchmark/overview.md)\n\n**Next:** [HostAgent Deep Dive](host_agent/overview.md) → Understand desktop orchestration\n\n---\n\n## 🌐 Media Coverage\n\nCheck out our official deep dive of UFO on [this Youtube Video](https://www.youtube.com/watch?v=QT_OhygMVXU).\n\nUFO sightings have garnered attention from various media outlets, including:\n\n- [微软正式开源UFO²，Windows桌面迈入「AgentOS 时代」](https://www.jiqizhixin.com/articles/2025-05-06-13)\n- [Microsoft's UFO abducts traditional user interfaces for a smarter Windows experience](https://the-decoder.com/microsofts-ufo-abducts-traditional-user-interfaces-for-a-smarter-windows-experience/)\n- [🚀 UFO & GPT-4-V: Sit back and relax, mientras GPT lo hace todo🌌](https://www.linkedin.com/posts/gutierrezfrancois_ai-ufo-microsoft-activity-7176819900399652865-pLoo?utm_source=share&utm_medium=member_desktop)\n- [The AI PC - The Future of Computers? - Microsoft UFO](https://www.youtube.com/watch?v=1k4LcffCq3E)\n- [下一代Windows系统曝光：基于GPT-4V，Agent跨应用调度，代号UFO](https://baijiahao.baidu.com/s?id=1790938358152188625&wfr=spider&for=pc)\n- [下一代智能版 Windows 要来了？微软推出首个 Windows Agent，命名为 UFO！](https://blog.csdn.net/csdnnews/article/details/136161570)\n- [Microsoft発のオープンソース版「UFO」登場！ Windowsを自動操縦するAIエージェントを試す](https://internet.watch.impress.co.jp/docs/column/shimizu/1570581.html)\n\n---\n\n## 📚 Citation\n\nIf you build on this work, please cite the AgentOS framework:\n\n**UFO² – The Desktop AgentOS (2025)** \n\n\n```bibtex\n@article{zhang2025ufo2,\n title = {{UFO2: The Desktop AgentOS}},\n author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\n journal = {arXiv preprint arXiv:2504.14603},\n year = {2025}\n}\n```\n\n**UFO – A UI‑Focused Agent for Windows OS Interaction (2024)** \n\n\n```bibtex\n@article{zhang2024ufo,\n title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},\n author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},\n journal = {arXiv preprint arXiv:2402.07939},\n year = {2024}\n}\n```\n\n---\n\n## 🎨 Related Projects\n\n- **TaskWeaver** — a code‑first LLM agent for data analytics: \n- **LLM‑Brained GUI Agents: A Survey**: • [GitHub](https://github.com/vyokky/LLM-Brained-GUI-Agents-Survey) • [Interactive site](https://vyokky.github.io/LLM-Brained-GUI-Agents-Survey/)\n\n---\n\n## ❓Get Help\n\n- ❔GitHub Issues (preferred)\n- For other communications, please contact [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)\n\n"
},
{
"path": "documents/docs/ufo2/prompts/basic_template.md",
"content": "# Basic Prompt Template\n\nThe basic prompt template is a fixed format used to generate prompts for the `HostAgent`, `AppAgent`, and `EvaluationAgent`. It includes templates for the `system` and `user` roles to construct each agent's prompt.\n\nDefault file paths for basic prompt templates:\n\n| Agent | File Path |\n| --- | --- |\n| HostAgent | [ufo/prompts/share/base/host_agent.yaml](https://github.com/microsoft/UFO/blob/main/ufo/prompts/share/base/host_agent.yaml) |\n| AppAgent | [ufo/prompts/share/base/app_agent.yaml](https://github.com/microsoft/UFO/blob/main/ufo/prompts/share/base/app_agent.yaml) |\n| EvaluationAgent | [ufo/prompts/evaluation/evaluate.yaml](https://github.com/microsoft/UFO/blob/main/ufo/prompts/evaluation/evaluate.yaml) |\n\nYou can configure the prompt template in the system configuration files. See the [System Configuration Guide](../../configuration/system/system_config.md) for details.\n\n## Template Structure\n\nEach YAML template contains structured sections for the `system` and `user` roles:\n\n- **System role**: Contains agent instructions, capabilities, and output format requirements\n- **User role**: Defines the structure for runtime context injection (observations, tasks, etc.)\n\nThese templates are loaded and populated by the agent's `Prompter` class at runtime. Learn how templates are processed and combined with dynamic content in the [Prompter documentation](../../infrastructure/agents/design/prompter.md).\n"
},
{
"path": "documents/docs/ufo2/prompts/examples_prompts.md",
"content": "# Example Prompts\n\nExample prompts provide demonstration examples for in-context learning. They are stored in the `ufo/prompts/examples` directory with the following subdirectories:\n\n| Directory | Description |\n| --- | --- |\n| `nonvisual` | Examples for non-visual LLMs |\n| `visual` | Examples for visual LLMs |\n\nYou can configure which example prompts to use in the system configuration files. See the [System Configuration Guide](../../configuration/system/system_config.md) for details.\n\n## How Examples Are Used\n\nExample prompts serve as in-context learning demonstrations that help the LLM understand the expected output format and reasoning process. The agent's `Prompter` class:\n\n1. Loads examples from YAML files based on the model type (visual/nonvisual)\n2. Formats them into the system prompt using `examples_prompt_helper()`\n3. Combines them with API documentation and base instructions\n\nSee the [Prompter documentation](../../infrastructure/agents/design/prompter.md) for details on how examples are loaded and formatted into the final prompt.\n\n\n## Example Structure\n\nBelow are examples for the `HostAgent` and `AppAgent`:\n\n### HostAgent Example\n\n```yaml\nRequest: |-\n My name is Zac. Please send a email to jack@outlook.com to thanks his contribution on the open source.\nResponse: \n observation: |-\n I observe that the outlook application is visible in the screenshot, with the title of 'Mail - Outlook - Zac'. I can see a list of emails in the application.\n thought: |-\n The user request can be solely complete on the outlook application. I need to open the outlook application for the current sub-task. If successful, no further sub-tasks are needed.\n current_subtask: |- \n Compose an email to send to Jack (jack@outlook.com) to thank him for his contribution to the open source project on the outlook application, using the name Zac.\n message:\n - (1) The name of the sender is Zac.\n - (2) The email composed should be detailed and professional.\n status: |- \n ASSIGN\n plan: []\n function: select_application_window\n arguments:\n id: \"12\"\n name: \"Mail - Outlook - Zac\"\n comment: |-\n It is time to open the outlook application!\n questions: []\n result: |-\n User request in ASSIGN state. Target window 'Mail - Outlook - Zac' (id:12) identified; will call select_application_window to focus Outlook and begin composing.\n```\n\n### AppAgent Example\n\n```yaml\nRequest: |-\n My name is Zac. Please send a email to jack@outlook.com to thanks his contribution on the open source.\nSub-task: |-\n Compose an email to send to Jack (jack@outlook.com) to thank him for his contribution to the open source project on the outlook application, using the name Zac.\nResponse: \n observation: |-\n The screenshot shows that I am on the Main Page of Outlook. The Main Page has a list of control items and email received. The new email editing window is not opened.\n thought: |-\n Base on the screenshots and the control item list, I need to click the New Email button to open a New Email window for the one-step action.\n action:\n function: |-\n click_input\n arguments: \n {\"id\": \"1\", \"name\": \"New Email\", \"button\": \"left\", \"double\": false}\n status: |-\n CONTINUE\n plan:\n - (1) Input the email address of the receiver.\n - (2) Input the title of the email.\n - (3) Input the content of the email.\n - (4) Click the Send button to send the email.\n comment: |-\n After I click the New Email button, the New Email window will be opened and available for composing the email.\n save_screenshot:\n {\"save\": false, \"reason\": \"\"}\n result: |-\n Successfully clicked the 'New Email' button in Outlook to initiate email composition.\nTips: \n - Sending an email is a sensitive action that needs to be confirmed by the user before the execution.\n - You need to draft the content of the email and send it to the receiver.\n```\n\nThese examples regulate the output format of the agent's response and provide a structured way to generate demonstration examples for in-context learning.\n\n## Related Documentation\n\n- **[Prompter Design](../../infrastructure/agents/design/prompter.md)** - Learn how examples are loaded and formatted\n- **[Basic Template](./basic_template.md)** - Understand the YAML template structure\n- **[System Configuration](../../configuration/system/system_config.md)** - Configure which examples to use "
},
{
"path": "documents/docs/ufo2/prompts/overview.md",
"content": "# Prompts\n\nAll prompts used in UFO are stored in the `ufo/prompts` directory. The folder structure is as follows:\n\n```\n📦prompts\n ┣ 📂demonstration # Prompts for summarizing human demonstrations\n ┣ 📂evaluation # Prompts for the EvaluationAgent\n ┣ 📂examples # Demonstration examples for in-context learning\n ┣ 📂nonvisual # Examples for non-visual LLMs\n ┗ 📂visual # Examples for visual LLMs\n ┣ 📂experience # Prompts for summarizing agent self-experience\n ┣ 📂share # Shared prompt templates\n ┗ 📂base # Basic version of shared prompts\n ┣ 📜api.yaml # Basic API prompt\n ┣ 📜app_agent.yaml # Basic AppAgent prompt template\n ┗ 📜host_agent.yaml # Basic HostAgent prompt template\n ┗ 📂third_party # Third-party integration prompts (e.g., Linux agents)\n```\n\nVisual LLMs can process screenshots while non-visual LLMs rely on text-only control information.\n\n## Agent Prompts\n\nAgent prompts are constructed from the following components:\n\n| Component | Description | Source |\n| --- | --- | --- |\n| **Basic Template** | Base template with system and user roles | YAML files in `share/base/` |\n| **API Documentation** | Skills and APIs available to the agent | Dynamically generated from MCP tools |\n| **Examples** | In-context learning demonstrations | YAML files in `examples/visual/` or `examples/nonvisual/` |\n\nYou can find the base templates in the `share/base` directory.\n\n## How Prompts Are Constructed\n\nThe agent's `Prompter` class is responsible for:\n\n1. **Loading** YAML templates from the file system\n2. **Formatting** API documentation from available tools\n3. **Selecting** appropriate examples based on model type (visual/nonvisual)\n4. **Combining** all components into a structured message list for the LLM\n5. **Injecting** runtime context (observations, screenshots, retrieved knowledge)\n\nEach agent type has its own specialized Prompter:\n\n- **HostAgentPrompter**: Desktop-level orchestration with third-party agent support\n- **AppAgentPrompter**: Application-level interactions with multi-action capabilities\n- **EvaluationAgentPrompter**: Task evaluation and success assessment\n\nFor comprehensive details about the Prompter class architecture, template loading, and prompt construction workflow, see the [Prompter documentation](../../infrastructure/agents/design/prompter.md).\n\n\n"
},
{
"path": "documents/mkdocs.yml",
"content": "site_name: UFO³ Documentation\n\n\nnav:\n - Home: index.md\n - Choose Your Path: choose_path.md\n - Project Directory Structure: project_directory_structure.md\n - Getting Started:\n - Quick Start (UFO³ Agent Galaxy): getting_started/quick_start_galaxy.md\n - Quick Start (UFO²): getting_started/quick_start_ufo2.md\n - Quick Start (Linux Agent): getting_started/quick_start_linux.md\n - Quick Start (Mobile Agent): getting_started/quick_start_mobile.md\n - Migration UFO² → UFO³: getting_started/migration_ufo2_to_galaxy.md\n - More Guidance: getting_started/more_guidance.md\n - Configuration & Setup:\n - Configuration System:\n - Overview: configuration/system/overview.md\n - Agent Configuration: configuration/system/agents_config.md\n - System Configuration: configuration/system/system_config.md\n - RAG Configuration: configuration/system/rag_config.md\n - Pricing Configuration: configuration/system/prices_config.md\n - Third-Party Configuration: configuration/system/third_party_config.md\n - MCP Reference: configuration/system/mcp_reference.md\n - Migration Guide: configuration/system/migration.md\n - Extending Configuration: configuration/system/extending.md\n - Galaxy Configuration:\n - Devices: configuration/system/galaxy_devices.md\n - Constellation: configuration/system/galaxy_constellation.md\n - Agent: configuration/system/galaxy_agent.md\n - Model Setup:\n - Overview: configuration/models/overview.md\n - OpenAI: configuration/models/openai.md\n - Azure OpenAI: configuration/models/azure_openai.md\n - OpenAI CUA (Operator): configuration/models/operator.md\n - Gemini: configuration/models/gemini.md\n - Claude: configuration/models/claude.md\n - Qwen: configuration/models/qwen.md\n - DeepSeek: configuration/models/deepseek.md\n - Ollama: configuration/models/ollama.md\n - Custom Model: configuration/models/custom_model.md\n - UFO³ Agent Galaxy:\n - Overview: galaxy/overview.md\n - WebUI: galaxy/webui.md\n - Galaxy Client:\n - Overview: galaxy/client/overview.md\n - ConstellationClient: galaxy/client/constellation_client.md\n - DeviceManager: galaxy/client/device_manager.md\n - Components: galaxy/client/components.md\n - AIP Integration: galaxy/client/aip_integration.md\n - GalaxyClient: galaxy/client/galaxy_client.md\n - Agent Registration:\n - Overview: galaxy/agent_registration/overview.md\n - Agent Profile: galaxy/agent_registration/agent_profile.md\n - Device Registry: galaxy/agent_registration/device_registry.md\n - Registration Flow: galaxy/agent_registration/registration_flow.md\n - Task Constellation (DAG):\n - Overview: galaxy/constellation/overview.md\n - TaskStar: galaxy/constellation/task_star.md\n - TaskStarLine: galaxy/constellation/task_star_line.md\n - TaskConstellation: galaxy/constellation/task_constellation.md\n - ConstellationEditor: galaxy/constellation/constellation_editor.md\n - Constellation Agent:\n - Overview: galaxy/constellation_agent/overview.md\n - State Machine: galaxy/constellation_agent/state.md\n - Strategy Pattern: galaxy/constellation_agent/strategy.md\n - MCP Commands: galaxy/constellation_agent/command.md\n - Constellation Orchestrator:\n - Overview: galaxy/constellation_orchestrator/overview.md\n - Event-Driven Coordination: galaxy/constellation_orchestrator/event_driven_coordination.md\n - Asynchronous Scheduling: galaxy/constellation_orchestrator/asynchronous_scheduling.md\n - Safe Assignment Locking: galaxy/constellation_orchestrator/safe_assignment_locking.md\n - Consistency Guarantees: galaxy/constellation_orchestrator/consistency_guarantees.md\n - Batched Editing: galaxy/constellation_orchestrator/batched_editing.md\n - Constellation Manager: galaxy/constellation_orchestrator/constellation_manager.md\n - API Reference: galaxy/constellation_orchestrator/api_reference.md\n - Observer System:\n - Overview: galaxy/observer/overview.md\n - Event System: galaxy/observer/event_system.md\n - Progress Observer: galaxy/observer/progress_observer.md\n - Agent Output Observer: galaxy/observer/agent_output_observer.md\n - Synchronizer: galaxy/observer/synchronizer.md\n - Metrics Observer: galaxy/observer/metrics_observer.md\n - Visualization Observer: galaxy/observer/visualization_observer.md\n - Evaluation & Logging:\n - Trajectory Report: galaxy/evaluation/trajectory_report.md\n - Performance Metrics: galaxy/evaluation/performance_metrics.md\n - Result JSON Reference: galaxy/evaluation/result_json.md\n - UFO² Desktop AgentOS:\n - Overview: ufo2/overview.md\n - Using as Galaxy Device: ufo2/as_galaxy_device.md\n - HostAgent:\n - Overview: ufo2/host_agent/overview.md\n - State Machine: ufo2/host_agent/state.md\n - Processing Strategy: ufo2/host_agent/strategy.md\n - Command System: ufo2/host_agent/commands.md\n - AppAgent:\n - Overview: ufo2/app_agent/overview.md\n - State Machine: ufo2/app_agent/state.md\n - Processing Strategy: ufo2/app_agent/strategy.md\n - Command System: ufo2/app_agent/commands.md\n - Core Features:\n - Hybrid GUI–API Actions: ufo2/core_features/hybrid_actions.md\n - Control Detection:\n - Overview: ufo2/core_features/control_detection/overview.md\n - UIA Detection: ufo2/core_features/control_detection/uia_detection.md\n - Visual Detection: ufo2/core_features/control_detection/visual_detection.md\n - Hybrid Detection: ufo2/core_features/control_detection/hybrid_detection.md\n - Knowledge Substrate:\n - Overview: ufo2/core_features/knowledge_substrate/overview.md\n - Help Documents: ufo2/core_features/knowledge_substrate/learning_from_help_document.md\n - Bing Search: ufo2/core_features/knowledge_substrate/learning_from_bing_search.md\n - Experience Learning: ufo2/core_features/knowledge_substrate/experience_learning.md \n - Demos: ufo2/core_features/knowledge_substrate/learning_from_demonstration.md\n - Speculative Multi-Action: ufo2/core_features/multi_action.md\n - Advanced Usage:\n - Follower Mode: ufo2/advanced_usage/follower_mode.md\n - Batch Mode: ufo2/advanced_usage/batch_mode.md\n - Operator Integration: ufo2/advanced_usage/operator_as_app_agent.md\n - Customization: ufo2/advanced_usage/customization.md\n - Prompts:\n - Overview: ufo2/prompts/overview.md\n - Basic Template: ufo2/prompts/basic_template.md\n - Examples: ufo2/prompts/examples_prompts.md\n - Evaluation:\n - EvaluationAgent: ufo2/evaluation/evaluation_agent.md\n - Benchmark Overview: ufo2/evaluation/benchmark/overview.md\n - Windows Agent Arena: ufo2/evaluation/benchmark/windows_agent_arena.md\n - OSWorld (Windows): ufo2/evaluation/benchmark/osworld.md\n - Performance Logs:\n - Overview: ufo2/evaluation/logs/overview.md\n - Evaluation Logs: ufo2/evaluation/logs/evaluation_logs.md\n - Markdown Log Viewer: ufo2/evaluation/logs/markdown_log_viewer.md\n - Request Logs: ufo2/evaluation/logs/request_logs.md\n - Screenshots Logs: ufo2/evaluation/logs/screenshots_logs.md\n - Step Logs: ufo2/evaluation/logs/step_logs.md\n - UI Tree Logs: ufo2/evaluation/logs/ui_tree_logs.md\n - Dataflow:\n - Overview: ufo2/dataflow/overview.md\n - Instantiation: ufo2/dataflow/instantiation.md\n - Execution: ufo2/dataflow/execution.md\n - Windows App Environment: ufo2/dataflow/windows_app_env.md\n - Result: ufo2/dataflow/result.md\n - Linux Agent:\n - Overview: linux/overview.md\n - Using as Galaxy Device: linux/as_galaxy_device.md\n - State Machine: linux/state.md\n - Processing Strategy: linux/strategy.md\n - MCP Commands: linux/commands.md\n - Mobile Agent:\n - Overview: mobile/overview.md\n - Using as Galaxy Device: mobile/as_galaxy_device.md\n - State Machine: mobile/state.md\n - Processing Strategy: mobile/strategy.md\n - MCP Commands: mobile/commands.md\n - Tutorials & Development:\n - Creating Custom MCP Servers: tutorials/creating_mcp_servers.md\n - Creating Custom Third-Party Agents: tutorials/creating_third_party_agents.md\n - Creating Custom Device Agents:\n - Overview: tutorials/creating_device_agent/overview.md\n - Index: tutorials/creating_device_agent/index.md\n - Client Setup: tutorials/creating_device_agent/client_setup.md\n - Core Components: tutorials/creating_device_agent/core_components.md\n - MCP Server: tutorials/creating_device_agent/mcp_server.md\n - Configuration: tutorials/creating_device_agent/configuration.md\n - Testing: tutorials/creating_device_agent/testing.md\n - Example Mobile Agent: tutorials/creating_device_agent/example_mobile_agent.md\n - Enhancing AppAgent Capabilities:\n - Overview: tutorials/creating_app_agent/overview.md\n - Help Document Provision: tutorials/creating_app_agent/help_document_provision.md\n - Demonstration Provision: tutorials/creating_app_agent/demonstration_provision.md\n - Wrapping App-Native APIs: tutorials/creating_app_agent/warpping_app_native_api.md\n - Infrastructure:\n - Basic Modules:\n - Overview: infrastructure/modules/overview.md\n - Session: infrastructure/modules/session.md\n - Round: infrastructure/modules/round.md\n - Context: infrastructure/modules/context.md\n - Dispatcher: infrastructure/modules/dispatcher.md\n - Session Factory & Pool: infrastructure/modules/session_pool.md\n - Platform Sessions: infrastructure/modules/platform_sessions.md\n - Device Agent Architecture:\n - Overview: infrastructure/agents/overview.md\n - Agent Types & Implementation: infrastructure/agents/agent_types.md\n - Server-Client Architecture: infrastructure/agents/server_client_architecture.md\n - Architecture Layers:\n - State Layer (Level-1): infrastructure/agents/design/state.md\n - Strategy Layer (Level-2): infrastructure/agents/design/processor.md\n - Strategy Components: infrastructure/agents/design/strategy.md\n - Command Layer (Level-3): infrastructure/agents/design/command.md\n - Supporting Systems:\n - Memory System: infrastructure/agents/design/memory.md\n - Blackboard: infrastructure/agents/design/blackboard.md\n - Prompter: infrastructure/agents/design/prompter.md\n - Agent Interaction Protocol (AIP):\n - Overview: aip/overview.md\n - Message Reference: aip/messages.md\n - Protocol Guide: aip/protocols.md\n - Transport Layer: aip/transport.md\n - Endpoints: aip/endpoints.md\n - Resilience: aip/resilience.md\n - Agent Server:\n - Overview: server/overview.md\n - Quick Start: server/quick_start.md\n - Session Manager: server/session_manager.md\n - WebSocket Handler: server/websocket_handler.md\n - Client Connection Manager: server/client_connection_manager.md\n - HTTP API: server/api.md\n - Monitoring: server/monitoring.md\n - Agent Client:\n - Overview: client/overview.md\n - Quick Start: client/quick_start.md\n - WebSocket Client: client/websocket_client.md\n - UFO Client: client/ufo_client.md\n - Computer Manager: client/computer_manager.md\n - Computer: client/computer.md\n - Device Info Provider: client/device_info.md\n - MCP Integration: client/mcp_integration.md\n - MCP (Model Context Protocol):\n - Overview: mcp/overview.md\n - Data Collection Servers: mcp/data_collection.md\n - Action Servers: mcp/action.md\n - Configuration Guide: mcp/configuration.md\n - Local Servers: mcp/local_servers.md\n - Remote Servers: mcp/remote_servers.md\n - Server Reference:\n - UICollector: mcp/servers/ui_collector.md\n - HostUIExecutor: mcp/servers/host_ui_executor.md\n - AppUIExecutor: mcp/servers/app_ui_executor.md\n - CommandLineExecutor: mcp/servers/command_line_executor.md\n - WordCOMExecutor: mcp/servers/word_com_executor.md\n - ExcelCOMExecutor: mcp/servers/excel_com_executor.md\n - PowerPointCOMExecutor: mcp/servers/ppt_com_executor.md\n - PDFReaderExecutor: mcp/servers/pdf_reader_executor.md\n - ConstellationEditor: mcp/servers/constellation_editor.md\n - HardwareExecutor: mcp/servers/hardware_executor.md\n - BashExecutor: mcp/servers/bash_executor.md\n - MobileExecutor: mcp/servers/mobile_executor.md\n - About:\n - Contributing: about/CONTRIBUTING.md\n - License: about/LICENSE.md\n - Code of Conduct: about/CODE_OF_CONDUCT.md\n - Disclaimer: about/DISCLAIMER.md\n - Support: about/SUPPORT.md\n - FAQ: faq.md\n\nmarkdown_extensions:\n - pymdownx.tasklist\n - admonition\n\ntheme:\n name: readthedocs\n analytics:\n - gtag: G-FX17ZGJYGC\n favicon: ./assets/ufo_blue.png\n features:\n - content.code.annotate\n - content.code.copy\n - content.code.select\n - content.tooltips\n - content.tabs.link\n\nextra_javascript:\n - https://unpkg.com/mermaid@10.6.1/dist/mermaid.min.js\n - javascripts/mermaid-init.js\n - https://polyfill.io/v3/polyfill.min.js?features=es6\n - https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-mml-chtml.js\n\n\nmarkdown_extensions:\n - admonition\n - attr_list\n - md_in_html\n - pymdownx.arithmatex:\n generic: true\n - pymdownx.superfences:\n custom_fences:\n - name: mermaid\n class: mermaid\n format: !!python/name:pymdownx.superfences.fence_div_format\n\n\n\nplugins:\n - search\n - mkdocstrings:\n handlers:\n python:\n paths: [\"../ufo\", \"../record_processor\", \"../dataflow\", \"../config\", \"../aip\", \"..\"]\n options:\n docstring_style: sphinx\n docstring_section_style: list\n merge_init_into_class: true\n show_docstring_returns: true\n"
},
{
"path": "galaxy/README.md",
"content": "\n\n
\n UFO³ : Weaving the Digital Agent Galaxy\n
\n
\n Cross-Device Orchestration Framework for Ubiquitous Intelligent Automation\n
\n\n---\n\n## 🌟 What is UFO³ Galaxy?\n\n**UFO³ Galaxy** is a revolutionary **cross-device orchestration framework** that transforms isolated device agents into a unified digital ecosystem. It models complex user requests as **Task Constellations** (星座) — dynamic distributed DAGs where nodes represent executable subtasks and edges capture dependencies across heterogeneous devices.\n\n### 🎯 The Vision\n\nBuilding truly ubiquitous intelligent agents requires moving beyond single-device automation. UFO³ Galaxy addresses four fundamental challenges in cross-device agent orchestration:\n\n
\n
\n
\n\n**🔄 Asynchronous Parallelism** \nEnabling concurrent task execution across multiple devices while maintaining correctness through event-driven coordination and safe concurrency control\n\n**⚡ Dynamic Adaptation** \nReal-time workflow evolution in response to intermediate results, transient failures, and runtime observations without workflow abortion\n\n
\n
\n\n**🌐 Distributed Coordination** \nReliable, low-latency communication across heterogeneous devices via WebSocket-based Agent Interaction Protocol with fault tolerance\n\n**🛡️ Safety Guarantees** \nFormal invariants ensuring DAG consistency during concurrent modifications and parallel execution, verified through rigorous proofs\n\n
\n
\n
\n\n---\n\n## ✨ Key Innovations\n\nUFO³ Galaxy realizes cross-device orchestration through five tightly integrated design principles:\n\n---\n\n### 🌟 Declarative Decomposition into Dynamic DAG\n\nUser requests are decomposed by the **ConstellationAgent** into a structured DAG of **TaskStars** (nodes) and **TaskStarLines** (edges) encoding workflow logic, dependencies, and device assignments.\n\n**Key Benefits:** Declarative structure for automated scheduling • Runtime introspection • Dynamic rewriting • Cross-device orchestration\n\n
\n \n
\n\n---\n\n
\n
\n
\n\n### 🔄 Continuous Result-Driven Graph Evolution\n\nThe **TaskConstellation** evolves dynamically in response to execution feedback, intermediate results, and failures through controlled DAG rewrites.\n\n**Adaptation Mechanisms:**\n- 🩺 Diagnostic TaskStars for debugging\n- 🛡️ Fallback creation for error recovery\n- 🔗 Dependency rewiring for optimization\n- ✂️ Node pruning after completion\n\nEnables resilient adaptation instead of workflow abortion.\n\n
\n
\n\n### ⚡ Heterogeneous, Asynchronous & Safe Orchestration\n\nTasks are matched to optimal devices via **AgentProfiles** (OS, hardware, tools) and executed asynchronously in parallel.\n\n**Safety Guarantees:**\n- 🔒 Safe assignment locking (no race conditions)\n- 📅 Event-driven scheduling (DAG readiness)\n- ✅ DAG consistency checks (structural integrity)\n- 🔄 Batched edits (atomicity)\n- 📐 Formal verification (provable correctness)\n\nEnsures high efficiency with reliability.\n\n
\n
\n
\n
\n\n### 🔌 Unified Agent Interaction Protocol (AIP)\n\nPersistent **WebSocket-based** protocol providing unified, secure, fault-tolerant communication for the entire agent ecosystem.\n\n**Core Capabilities:**\n- 📝 Agent registry with capability profiles\n- 🔐 Secure session management\n- 📤 Intelligent task routing\n- 💓 Health monitoring with heartbeats\n- 🔌 Auto-reconnection & retry mechanisms\n\n**Benefits:** Lightweight • Extensible • Fault-tolerant\n\n
\n \n 🎯 Together, these designs enable UFO³ to decompose, schedule, execute, and adapt distributed tasks efficiently while maintaining safety and consistency across heterogeneous devices.\n
\n\n---\n\n## 🎥 Demo Video\n\nSee UFO³ Galaxy in action with this comprehensive demonstration of cross-device orchestration:\n\n
🎬 Click to watch: Multi-device workflow orchestration with UFO³ Galaxy
\n
\n\n---\n\n## 🏗️ Architecture Overview\n\n
\n \n
UFO³ Galaxy Layered Architecture — From natural language to distributed execution
\n
\n\n### Hierarchical Design\n\n
\n
\n
\n\n#### 🎛️ Control Plane\n\n| Component | Role |\n|-----------|------|\n| **🌐 ConstellationClient** | Global device registry with capability profiles |\n| **🖥️ Device Agents** | Local orchestration with unified MCP tools |\n| **🔒 Clean Separation** | Global policies & device independence |\n\n
\n
\n\n#### 🔄 Execution Workflow\n\n
\n \n
\n\n
\n
\n
\n\n---\n\n## 🚀 Quick Start\n\n### 🛠️ Step 1: Installation\n\n```powershell\n# Clone repository\ngit clone https://github.com/microsoft/UFO.git\ncd UFO\n\n# Create environment (recommended)\nconda create -n ufo3 python=3.10\nconda activate ufo3\n\n# Install dependencies\npip install -r requirements.txt\n```\n\n### ⚙️ Step 2: Configure ConstellationAgent LLM\n\nUFO³ Galaxy uses a **ConstellationAgent** that orchestrates all device agents. Configure its LLM settings:\n\n```powershell\n# Create configuration from template\ncopy config\\galaxy\\agent.yaml.template config\\galaxy\\agent.yaml\nnotepad config\\galaxy\\agent.yaml\n```\n\n**Configuration File Location:**\n```\nconfig/galaxy/\n├── agent.yaml.template # Template - COPY THIS\n├── agent.yaml # Your config with API keys (DO NOT commit)\n└── devices.yaml # Device pool configuration (Step 4)\n```\n\n**OpenAI Configuration:**\n```yaml\nCONSTELLATION_AGENT:\n REASONING_MODEL: false\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_VERSION: \"2025-02-01-preview\"\n API_MODEL: \"gpt-5-chat-20251003\"\n # ... (prompt configurations use defaults)\n```\n\n**Azure OpenAI Configuration:**\n```yaml\nCONSTELLATION_AGENT:\n REASONING_MODEL: false\n API_TYPE: \"aoai\"\n API_BASE: \"https://YOUR_RESOURCE.openai.azure.com\"\n API_KEY: \"YOUR_AOAI_KEY\"\n API_VERSION: \"2024-02-15-preview\"\n API_MODEL: \"gpt-5-chat-20251003\"\n API_DEPLOYMENT_ID: \"YOUR_DEPLOYMENT_ID\"\n # ... (prompt configurations use defaults)\n```\n\n### 🖥️ Step 3: Configure Device Agents\n\nEach device agent (Windows/Linux) needs its own LLM configuration to execute tasks.\n\n```powershell\n# Configure device agent LLMs\ncopy config\\ufo\\agents.yaml.template config\\ufo\\agents.yaml\nnotepad config\\ufo\\agents.yaml\n```\n\n**Configuration File Location:**\n```\nconfig/ufo/\n├── agents.yaml.template # Template - COPY THIS\n└── agents.yaml # Device agent LLM config (DO NOT commit)\n```\n\n**Example Configuration:**\n```yaml\nHOST_AGENT:\n VISUAL_MODE: true\n API_TYPE: \"openai\" # or \"aoai\" for Azure OpenAI\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"gpt-4o\"\n\nAPP_AGENT:\n VISUAL_MODE: true\n API_TYPE: \"openai\"\n API_BASE: \"https://api.openai.com/v1/chat/completions\"\n API_KEY: \"sk-YOUR_KEY_HERE\"\n API_MODEL: \"gpt-4o\"\n```\n\n> **💡 Tip:** You can use the same API key and model for both ConstellationAgent (Step 2) and device agents (Step 3).\n\n### 🌐 Step 4: Configure Device Pool\n\n```powershell\n# Configure available devices\ncopy config\\galaxy\\devices.yaml.template config\\galaxy\\devices.yaml\nnotepad config\\galaxy\\devices.yaml\n```\n\n**Example Device Configuration:**\n```yaml\ndevices:\n # Windows Device (UFO²)\n - device_id: \"windows_device_1\" # Must match --client-id\n server_url: \"ws://localhost:5000/ws\" # Must match server WebSocket URL\n os: \"windows\"\n capabilities:\n - \"desktop_automation\"\n - \"office_applications\"\n - \"excel\"\n - \"word\"\n - \"outlook\"\n - \"email\"\n - \"web_browsing\"\n metadata:\n os: \"windows\"\n version: \"11\"\n performance: \"high\"\n installed_apps:\n - \"Microsoft Excel\"\n - \"Microsoft Word\"\n - \"Microsoft Outlook\"\n - \"Google Chrome\"\n description: \"Primary Windows desktop for office automation\"\n auto_connect: true\n max_retries: 5\n\n # Linux Device\n - device_id: \"linux_device_1\" # Must match --client-id\n server_url: \"ws://localhost:5001/ws\" # Must match server WebSocket URL\n os: \"linux\"\n capabilities:\n - \"server_management\"\n - \"log_analysis\"\n - \"file_operations\"\n - \"database_operations\"\n metadata:\n os: \"linux\"\n performance: \"medium\"\n logs_file_path: \"/var/log/myapp/app.log\"\n dev_path: \"/home/user/projects/\"\n warning_log_pattern: \"WARN\"\n error_log_pattern: \"ERROR|FATAL\"\n description: \"Development server for backend operations\"\n auto_connect: true\n max_retries: 5\n```\n\n> **⚠️ Critical: IDs and URLs Must Match**\n> - `device_id` **must exactly match** the `--client-id` flag\n> - `server_url` **must exactly match** the server WebSocket URL\n> - Otherwise, Galaxy cannot control the device!\n\n### 🖥️ Step 5: Start Device Agents\n\nGalaxy orchestrates **device agents** that execute tasks on individual machines. You need to start the appropriate device agents based on your needs.\n\n#### Example: Quick Windows Device Setup\n\n**On your Windows machine:**\n\n```powershell\n# Terminal 1: Start UFO² Server\npython -m ufo.server.app --port 5000\n\n# Terminal 2: Start UFO² Client (connect to server)\npython -m ufo.client.client `\n --ws `\n --ws-server ws://localhost:5000/ws `\n --client-id windows_device_1 `\n --platform windows\n```\n\n> **⚠️ Important: Platform Flag Required**\n> Always include `--platform windows` for Windows devices and `--platform linux` for Linux devices!\n\n#### Example: Quick Linux Device Setup\n\n**On your Linux machine:**\n\n```bash\n# Terminal 1: Start Device Agent Server\npython -m ufo.server.app --port 5001\n\n# Terminal 2: Start Linux Client (connect to server)\npython -m ufo.client.client \\\n --ws \\\n --ws-server ws://localhost:5001/ws \\\n --client-id linux_device_1 \\\n --platform linux\n\n# Terminal 3: Start HTTP MCP Server (for Linux tools)\npython -m ufo.client.mcp.http_servers.linux_mcp_server\n```\n\n**📖 Detailed Setup Instructions:**\n- **For Windows devices (UFO²):** See [UFO² as Galaxy Device](../documents/docs/ufo2/as_galaxy_device.md)\n- **For Linux devices:** See [Linux as Galaxy Device](../documents/docs/linux/as_galaxy_device.md)\n\n### 🌌 Step 6: Launch Galaxy Client\n\n#### 🎨 Interactive WebUI Mode (Recommended)\n\nLaunch Galaxy with an interactive web interface for real-time constellation visualization and monitoring:\n\n```powershell\npython -m galaxy --webui\n```\n\nThis will start the Galaxy server with WebUI and open your browser to the interactive interface:\n\n
\n \n
🎨 Galaxy WebUI - Interactive constellation visualization and chat interface
\n
\n\n**WebUI Features:**\n- 🗣️ **Chat Interface**: Submit requests and interact with ConstellationAgent in real-time\n- 📊 **Live DAG Visualization**: Watch task constellation formation and execution\n- 🎯 **Task Status Tracking**: Monitor each TaskStar's progress and completion\n- 🔄 **Dynamic Updates**: See constellation evolution as tasks complete\n- 📱 **Responsive Design**: Works on desktop and tablet devices\n\n**Default URL:** `http://localhost:8000` (automatically finds next available port if 8000 is occupied)\n\n---\n\n#### 💬 Interactive Terminal Mode\n\nFor command-line interaction:\n\n```powershell\npython -m galaxy --interactive\n```\n\n---\n\n#### ⚡ Direct Request Mode\n\nExecute a single request and exit:\n\n```powershell\npython -m galaxy --request \"Extract data from Excel on Windows, process with Python on Linux, and generate visualization report\"\n```\n\n---\n\n#### 🔧 Programmatic API\n\nEmbed Galaxy in your Python applications:\n\n```python\nfrom galaxy.galaxy_client import GalaxyClient\n\nasync def main():\n # Initialize client\n client = GalaxyClient(session_name=\"data_pipeline\")\n await client.initialize()\n \n # Execute cross-device workflow\n result = await client.process_request(\n \"Download sales data, analyze trends, generate executive summary\"\n )\n \n # Access constellation details\n constellation = client.session.constellation\n print(f\"Tasks executed: {len(constellation.tasks)}\")\n print(f\"Devices used: {set(t.assigned_device for t in constellation.tasks)}\")\n \n await client.shutdown()\n\nimport asyncio\nasyncio.run(main())\n```\n\n---\n\n## 🎯 Use Cases\n\n### 🖥️ Software Development & CI/CD\n\n**Request:** \n*\"Clone repository on Windows, build Docker image on Linux GPU server, deploy to staging, and run test suite on CI cluster\"*\n\n**Constellation Workflow:**\n```\nClone (Windows) → Build (Linux GPU) → Deploy (Linux Server) → Test (Linux CI)\n```\n\n**Benefit:** Parallel execution reduces pipeline time by 60%\n\n---\n\n### 📊 Data Science Workflows\n\n**Request:** \n*\"Fetch dataset from cloud storage, preprocess on Linux workstation, train model on A100 node, visualize results on Windows\"*\n\n**Constellation Workflow:**\n```\nFetch (Any) → Preprocess (Linux) → Train (Linux GPU) → Visualize (Windows)\n```\n\n**Benefit:** Automatic GPU detection and optimal device assignment\n\n---\n\n### 📝 Cross-Platform Document Processing\n\n**Request:** \n*\"Extract data from Excel on Windows, process with Python on Linux, generate PDF report, and email summary\"*\n\n**Constellation Workflow:**\n```\nExtract (Windows) → Process (Linux) ┬→ Generate PDF (Windows)\n └→ Send Email (Windows)\n```\n\n**Benefit:** Parallel report generation and email delivery\n\n---\n\n### 🔬 Distributed System Monitoring\n\n**Request:** \n*\"Collect server logs from all Linux machines, analyze for errors, generate alerts, create consolidated report\"*\n\n**Constellation Workflow:**\n```\n┌→ Collect (Linux 1) ┐\n├→ Collect (Linux 2) ├→ Analyze (Any) → Report (Windows)\n└→ Collect (Linux 3) ┘\n```\n\n**Benefit:** Parallel log collection with automatic aggregation\n\n---\n\n## 🌐 System Capabilities\n\nBuilding on the five design principles, UFO³ Galaxy delivers powerful capabilities for distributed automation:\n\n
\n
\n
\n\n### ⚡ Efficient Parallel Execution\n- **Event-driven scheduling** monitors DAG for ready tasks\n- **Non-blocking execution** with Python `asyncio`\n- **Dynamic task integration** without workflow interruption\n- **Result:** Up to 70% reduction in end-to-end latency compared to sequential execution\n\n---\n\n### 🛡️ Formal Safety Guarantees\n- **Three formal invariants (I1-I3)** ensure DAG correctness\n- **Safe assignment locking** prevents race conditions\n- **Acyclicity validation** eliminates circular dependencies\n- **State merging** preserves progress during runtime modifications\n- **Formally verified** through rigorous mathematical proofs\n\n
\n
\n\n### 🔄 Intelligent Adaptation\n- **Dual-mode ConstellationAgent** (creation/editing) with FSM control\n- **Result-driven evolution** based on execution feedback\n- **LLM-powered reasoning** via ReAct architecture\n- **Automatic error recovery** through diagnostic tasks and fallbacks\n- **Workflow optimization** via dynamic rewiring and pruning\n\n---\n\n### 👁️ Comprehensive Observability\n- **Real-time visualization** of constellation structure and execution\n- **Event-driven updates** via publish-subscribe pattern\n- **Rich execution logs** with markdown trajectories\n- **Status tracking** for each TaskStar and dependency\n- **Interactive WebUI** for monitoring and control\n\n
\n
\n
\n\n---\n\n### 🔌 Extensibility & Platform Independence\n\nUFO³ is designed as a **universal orchestration framework** that seamlessly integrates heterogeneous device agents across platforms.\n\n**Multi-Platform Support:**\n- 🪟 **Windows** — Desktop automation via UFO²\n- 🐧 **Linux** — Server management, DevOps, data processing\n- 📱 **Android** — Mobile device automation via MCP\n- 🌐 **Web** — Browser-based agents (coming soon)\n- 🍎 **macOS** — Desktop automation (coming soon)\n- 🤖 **IoT/Embedded** — Edge devices and sensors (coming soon)\n\n**Developer-Friendly:**\n- 📦 **Lightweight template** for rapid agent development\n- 🧩 **MCP integration** for plug-and-play tool extension\n- 📖 **Comprehensive tutorials** and API documentation\n- 🔌 **AIP protocol** for seamless ecosystem integration\n\n**📖 Want to build your own device agent?** See our [Creating Custom Device Agents tutorial](../documents/docs/tutorials/creating_device_agent/overview.md) to learn how to extend UFO³ to new platforms.\n\n---\n\n## 📚 Documentation\n\n| Component | Description | Link |\n|-----------|-------------|------|\n| **Galaxy Client** | Device coordination and ConstellationClient API | [Learn More](../documents/docs/galaxy/client/overview.md) |\n| **Constellation Agent** | LLM-driven task decomposition and DAG evolution | [Learn More](../documents/docs/galaxy/constellation_agent/overview.md) |\n| **Task Orchestrator** | Asynchronous execution and safety guarantees | [Learn More](../documents/docs/galaxy/constellation_orchestrator/overview.md) |\n| **Task Constellation** | DAG structure and constellation editor | [Learn More](../documents/docs/galaxy/constellation/overview.md) |\n| **Agent Registration** | Device registry and agent profiles | [Learn More](../documents/docs/galaxy/agent_registration/overview.md) |\n| **AIP Protocol** | WebSocket messaging and communication patterns | [Learn More](../documents/docs/aip/overview.md) |\n| **Configuration** | Device pools and orchestration policies | [Learn More](../documents/docs/configuration/system/galaxy_devices.md) |\n| **Creating Device Agents** | Tutorial for building custom device agents | [Learn More](../documents/docs/tutorials/creating_device_agent/overview.md) |\n\n---\n\n## 📊 System Architecture\n\n### Core Components\n\n| Component | Location | Responsibility |\n|-----------|----------|----------------|\n| **GalaxyClient** | `galaxy/galaxy_client.py` | Session management, user interaction |\n| **ConstellationClient** | `galaxy/client/constellation_client.py` | Device registry, connection lifecycle |\n| **ConstellationAgent** | `galaxy/agents/constellation_agent.py` | DAG synthesis and evolution |\n| **TaskConstellationOrchestrator** | `galaxy/constellation/orchestrator/` | Asynchronous execution, safety enforcement |\n| **TaskConstellation** | `galaxy/constellation/task_constellation.py` | DAG data structure and validation |\n| **DeviceManager** | `galaxy/client/device_manager.py` | WebSocket connections, heartbeat monitoring |\n\n### Technology Stack\n\n| Layer | Technologies |\n|-------|-------------|\n| **Language** | Python 3.10+, asyncio, dataclasses |\n| **Communication** | WebSockets, JSON-RPC |\n| **LLM** | OpenAI, Azure OpenAI, Gemini, Claude |\n| **Tools** | Model Context Protocol (MCP) |\n| **Config** | YAML, Pydantic validation |\n| **Logging** | Rich console, Markdown trajectories |\n\n---\n\n## 🌟 From Devices to Galaxy\n\nUFO³ represents a paradigm shift in intelligent automation:\n\n```mermaid\n%%{init: {'theme':'base', 'themeVariables': { 'primaryColor':'#E8F4F8','primaryTextColor':'#1A1A1A','primaryBorderColor':'#7CB9E8','lineColor':'#A8D5E2','secondaryColor':'#B8E6F0','tertiaryColor':'#D4F1F4','fontSize':'16px','fontFamily':'Segoe UI, Arial, sans-serif'}}}%%\ngraph LR\n A[\"🎈 UFO February 2024 GUI Agent for Windows\"] \n B[\"🖥️ UFO² April 2025 Desktop AgentOS\"]\n C[\"🌌 UFO³ Galaxy November 2025 Multi-Device Orchestration\"]\n \n A -->|Evolve| B\n B -->|Scale| C\n \n style A fill:#E8F4F8,stroke:#7CB9E8,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15\n style B fill:#C5E8F5,stroke:#5BA8D0,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15\n style C fill:#A4DBF0,stroke:#3D96BE,stroke-width:2.5px,color:#1A1A1A,rx:15,ry:15\n```\n\nOver time, multiple constellations interconnect, forming a self-organizing **Digital Agent Galaxy** where devices, agents, and capabilities weave together into adaptive, resilient, and intelligent ubiquitous computing systems.\n\n---\n\n## 📄 Citation\n\nIf you use UFO³ Galaxy in your research, please cite:\n\n**UFO³ Galaxy Framework:**\n```bibtex\n@article{zhang2025ufo3,\n title={UFO$^3$: Weaving the Digital Agent Galaxy}, \n author = {Zhang, Chaoyun and Li, Liqun and Huang, He and Ni, Chiming and Qiao, Bo and Qin, Si and Kang, Yu and Ma, Minghua and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\n journal = {arXiv preprint arXiv:2511.11332},\n year = {2025},\n}\n```\n\n**UFO² Desktop AgentOS:**\n```bibtex\n@article{zhang2025ufo2,\n title = {{UFO2: The Desktop AgentOS}},\n author = {Zhang, Chaoyun and Huang, He and Ni, Chiming and Mu, Jian and Qin, Si and He, Shilin and Wang, Lu and Yang, Fangkai and Zhao, Pu and Du, Chao and Li, Liqun and Kang, Yu and Jiang, Zhao and Zheng, Suzhen and Wang, Rujia and Qian, Jiaxu and Ma, Minghua and Lou, Jian-Guang and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei},\n journal = {arXiv preprint arXiv:2504.14603},\n year = {2025}\n}\n```\n\n**First UFO:**\n```bibtex\n@article{zhang2024ufo,\n title = {{UFO: A UI-Focused Agent for Windows OS Interaction}},\n author = {Zhang, Chaoyun and Li, Liqun and He, Shilin and Zhang, Xu and Qiao, Bo and Qin, Si and Ma, Minghua and Kang, Yu and Lin, Qingwei and Rajmohan, Saravan and Zhang, Dongmei and Zhang, Qi},\n journal = {arXiv preprint arXiv:2402.07939},\n year = {2024}\n}\n```\n\n---\n\n## 🤝 Contributing\n\nWe welcome contributions! Whether building new device agents, improving orchestration algorithms, or enhancing the protocol:\n\n- 🐛 [Report Issues](https://github.com/microsoft/UFO/issues)\n- 💡 [Request Features](https://github.com/microsoft/UFO/discussions)\n- 📝 [Improve Documentation](https://github.com/microsoft/UFO/pulls)\n- 🧪 [Submit Pull Requests](../../CONTRIBUTING.md)\n\n---\n\n## 📬 Contact & Support\n\n- 📖 **Documentation**: [https://microsoft.github.io/UFO/](https://microsoft.github.io/UFO/)\n- 💬 **Discussions**: [GitHub Discussions](https://github.com/microsoft/UFO/discussions)\n- 🐛 **Issues**: [GitHub Issues](https://github.com/microsoft/UFO/issues)\n- 📧 **Email**: [ufo-agent@microsoft.com](mailto:ufo-agent@microsoft.com)\n\n---\n\n## ⚖️ License\n\nUFO³ Galaxy is released under the [MIT License](../../LICENSE).\n\nSee [DISCLAIMER.md](../../DISCLAIMER.md) for privacy and safety notices.\n\n---\n\n
\n
Transform your distributed devices into a unified digital collective.
\n
UFO³ Galaxy — Where every device is a star, and every task is a constellation.
![\"UFO](\"assets/logo3.png\"){kind=logo}
: Weaving the Digital Agent Galaxy\r\n![\"UFO²](\"assets/ufo_blue.png\")
**UFO² Desktop AgentOS**\r\n**STABLE & BATTLE-TESTED**\r\n\r\n**Perfect for:**\r\n- 💻 Single Windows automation\r\n- ⚡ Quick task execution\r\n- 🎓 Learning agent basics\r\n- 🛠️ Simple workflows\r\n\r\n**Key Features:**\r\n- Deep Windows OS integration\r\n- Hybrid GUI + API actions\r\n- Proven reliability\r\n- Easy setup\r\n- Can serve as Galaxy device agent\r\n\r\n\r\n**📖 [UFO² Documentation →](./ufo/README.md)**\r\n\r\n![\"UFO³](\"assets/poster_with_play.png\")
\r\n \r\n ![](\"../assets/dataflow/overview.png\"/)
\n![](\"../assets/dataflow/instantiation.png\"/)
\n![](\"../assets/dataflow/execution.png\"/)
\n![](\"../assets/dataflow/result_example.png\"/)
\n![\"Task](\"/img/task_constellation.png\")
\n ![](\"topology_images/step2_after_constellation_xxx.png\")
\n\n##### Task Summary Table\n\n| Task ID | Name | Status | Device | Duration |\n|---------|------|--------|--------|----------|\n| task-1 | Type hi on linux_agent_1 | pending | linux_agent_1 | N/A |\n| task-2 | Type hi on linux_agent_2 | pending | linux_agent_2 | N/A |\n| task-3 | Type hi on linux_agent_3 | pending | linux_agent_3 | N/A |\n```\n\n**Topology Visualization Features:**\n- **Color-coded nodes** by task status:\n - 🟢 Green: Completed\n - 🔵 Cyan: Running\n - ⚫ Gray: Pending\n - 🔴 Red: Failed/Error\n- **Edge styles** for dependencies:\n - Solid green: Satisfied dependencies\n - Dashed orange: Pending dependencies\n- **Automatic layout** with hierarchical spring algorithm\n- **Legend** showing node/edge meanings\n\n##### Detailed Task Information\n\nComprehensive task metadata with execution details:\n\n```markdown\n#### Task task-1: Type hi on linux_agent_1\n\n- **Status**: completed\n- **Target Device**: linux_agent_1\n- **Priority**: 2\n- **Description**: On device linux_agent_1 (Linux), open a terminal and execute the command: echo 'hi'. Return the output text.\n- **Tips**:\n - Ensure CLI access is available.\n - Expected textual result: Return the exact output of the command, which should be 'hi'.\n- **Result**: \n ```\n hi\n ```\n- **Started**: 2025-11-05T05:45:26.395208+00:00\n- **Ended**: 2025-11-05T05:45:42.981859+00:00\n- **Duration**: 16.59s\n```\n\n**Task Fields:**\n- **Status**: Current execution state (`pending`, `running`, `completed`, `failed`, `cancelled`)\n- **Target Device**: Assigned device agent ID\n- **Priority**: Task scheduling priority (1=HIGH, 2=MEDIUM, 3=LOW)\n- **Description**: Natural language task specification for device agent\n- **Tips**: Execution hints and expected output guidance\n- **Result**: Task execution output (truncated if large)\n- **Error**: Error message if task failed\n- **Timing**: Start/end timestamps and duration\n\n##### Dependency Details\n\nShows task relationships and satisfaction status:\n\n```markdown\n| Line ID | From Task | To Task | Type | Satisfied | Condition |\n|---------|-----------|---------|------|-----------|----------|\n| l1 | t1 | t4 | unconditional | [PENDING] | Output from linux_agent_1 collected successfully. |\n| l2 | t2 | t4 | unconditional | [OK] | Output from linux_agent_2 collected successfully. |\n```\n\n**Dependency Types:**\n- `unconditional`: Always active when source task completes\n- `conditional`: Activated based on result evaluation\n\n#### Connected Devices\n\nDevice registry snapshot at step completion:\n\n```markdown\n![\"UFO³](\"/img/poster.png\")
\n ![\"UFO³](\"/img/overview2.png\")
\n ![\"Galaxy](\"../img/webui.png\")
\n ![\"Add](\"../img/add_device.png\")
\n ![\"UFO](\"./img/logo3.png\"){kind=logo}
: Weaving the Digital Agent Galaxy\n ![\"UFO³](\"./img/poster.png\")
\n![\"UFO³](\"./img/overview2.png\")
\n![\"UFO²](\"./img/framework2.png\")
\n![](\"../../img/execution.png\"/)
\n![](\"../../img/instantiation.png\"/)
\n![](\"../../img/overview.png\")
\n![](\"../../img/result_example.png\"/)
\n![\"Clean](\"../../img/action_step2.png\")
\n![\"Annotated](\"../../img/action_step2_annotated.png\")
\n![\"Concatenated](\"../../img/action_step2_concat.png\")
\n![\"Selected](\"../../img/action_step2_selected_controls.png\")
\n![\"UFO³](\"../assets/logo3.png\"){kind=logo}
: Weaving the Digital Agent Galaxy\n![\"Task](\"../assets/task_constellation.png\")
\n![\"UFO³](\"../assets/poster_with_play.png\")
\n \n ![\"UFO³](\"../documents/docs/img/overview2.png\")
\n ![\"Execution](\"../assets/orchestrator.png\")
\n![\"UFO³](\"../assets/webui.png\")
\n