[
{
"path": ".dockerignore",
"content": "# Git\n.git\n.gitignore\n\n# Node\nnew_ui/frontend/node_modules\nnew_ui/frontend/dist\n\n# Python\n__pycache__\n*.pyc\n*.pyo\n*.egg-info\n.eggs\ndist\nbuild\n\n# Virtual environments\n.venv\nvenv\nenv\n\n# IDE\n.vscode\n.idea\n.cursor\n*.swp\n*.swo\n\n# Runtime data\ndeepcode_lab\nuploads\nlogs\n*.log\n\n# Docker\ndeepcode_docker/Dockerfile\ndeepcode_docker/docker-compose.yml\ndeepcode_docker/.dockerignore\ndeepcode_docker/run_docker.sh\n\n# Documentation\nassets\n*.md\nLICENSE\n"
},
{
"path": ".gitattributes",
"content": "# Force LF line endings for shell scripts (prevents CRLF issues in Docker)\n*.sh text eol=lf\ndocker-entrypoint.sh text eol=lf\n"
},
{
"path": ".github/ISSUE_TEMPLATE/bug_report.yml",
"content": "name: Bug Report\ndescription: File a bug report\ntitle: \"[Bug]:\"\nlabels: [\"bug\", \"triage\"]\n\nbody:\n - type: checkboxes\n id: existingcheck\n attributes:\n label: Do you need to file an issue?\n description: Please help us manage our time by avoiding duplicates and common bugs with the steps below.\n options:\n - label: I have searched the existing issues and this bug is not already filed.\n - label: I believe this is a legitimate bug, not just a question or feature request.\n - type: textarea\n id: description\n attributes:\n label: Describe the bug\n description: A clear and concise description of what the bug is.\n placeholder: What went wrong?\n - type: textarea\n id: reproduce\n attributes:\n label: Steps to reproduce\n description: Steps to reproduce the behavior.\n placeholder: How can we replicate the issue?\n - type: textarea\n id: expected_behavior\n attributes:\n label: Expected Behavior\n description: A clear and concise description of what you expected to happen.\n placeholder: What should have happened?\n - type: textarea\n id: configused\n attributes:\n label: DeepCode Config Used\n description: The DeepCode configuration used for the run.\n placeholder: The settings content or DeepCode configuration\n value: |\n # Paste your config here\n - type: textarea\n id: screenshotslogs\n attributes:\n label: Logs and screenshots\n description: If applicable, add screenshots and logs to help explain your problem.\n placeholder: Add logs and screenshots here\n - type: textarea\n id: additional_information\n attributes:\n label: Additional Information\n description: |\n - DeepCode Version: e.g., v0.1.1\n - Operating System: e.g., Windows 10, Ubuntu 20.04\n - Python Version: e.g., 3.8\n - Related Issues: e.g., #1\n - Any other relevant information.\n value: |\n - DeepCode Version:\n - Operating System:\n - Python Version:\n - Related Issues:\n"
},
{
"path": ".github/ISSUE_TEMPLATE/config.yml",
"content": "blank_issues_enabled: false\n"
},
{
"path": ".github/ISSUE_TEMPLATE/feature_request.yml",
"content": "name: Feature Request\ndescription: File a feature request\nlabels: [\"enhancement\"]\ntitle: \"[Feature Request]:\"\n\nbody:\n - type: checkboxes\n id: existingcheck\n attributes:\n label: Do you need to file a feature request?\n description: Please help us manage our time by avoiding duplicates and common feature request with the steps below.\n options:\n - label: I have searched the existing feature request and this feature request is not already filed.\n - label: I believe this is a legitimate feature request, not just a question or bug.\n - type: textarea\n id: feature_request_description\n attributes:\n label: Feature Request Description\n description: A clear and concise description of the feature request you would like.\n placeholder: What this feature request add more or improve?\n - type: textarea\n id: additional_context\n attributes:\n label: Additional Context\n description: Add any other context or screenshots about the feature request here.\n placeholder: Any additional information\n"
},
{
"path": ".github/ISSUE_TEMPLATE/question.yml",
"content": "name: Question\ndescription: Ask a general question\nlabels: [\"question\"]\ntitle: \"[Question]:\"\n\nbody:\n - type: checkboxes\n id: existingcheck\n attributes:\n label: Do you need to ask a question?\n description: Please help us manage our time by avoiding duplicates and common questions with the steps below.\n options:\n - label: I have searched the existing question and discussions and this question is not already answered.\n - label: I believe this is a legitimate question, not just a bug or feature request.\n - type: textarea\n id: question\n attributes:\n label: Your Question\n description: A clear and concise description of your question.\n placeholder: What is your question?\n - type: textarea\n id: context\n attributes:\n label: Additional Context\n description: Provide any additional context or details that might help us understand your question better.\n placeholder: Add any relevant information here\n"
},
{
"path": ".github/dependabot.yml",
"content": "# To get started with Dependabot version updates, you'll need to specify which\n# package ecosystems to update and where the package manifests are located.\n# Please see the documentation for all configuration options:\n# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file\n\nversion: 2\nupdates:\n - package-ecosystem: \"pip\" # See documentation for possible values\n directory: \"/\" # Location of package manifests\n schedule:\n interval: \"weekly\"\n"
},
{
"path": ".github/pull_request_template.md",
"content": "\n\n## Description\n\n[Briefly describe the changes made in this pull request.]\n\n## Related Issues\n\n[Reference any related issues or tasks addressed by this pull request.]\n\n## Changes Made\n\n[List the specific changes made in this pull request.]\n\n## Checklist\n\n- [ ] Changes tested locally\n- [ ] Code reviewed\n- [ ] Documentation updated (if necessary)\n- [ ] Unit tests added (if applicable)\n\n## Additional Notes\n\n[Add any additional notes or context for the reviewer(s).]\n"
},
{
"path": ".github/workflows/linting.yaml",
"content": "name: Linting and Formatting\n\non:\n push:\n branches:\n - main\n pull_request:\n branches:\n - main\n\njobs:\n lint-and-format:\n runs-on: ubuntu-latest\n\n steps:\n - name: Checkout code\n uses: actions/checkout@v2\n\n - name: Set up Python\n uses: actions/setup-python@v2\n with:\n python-version: '3.x'\n\n - name: Install dependencies\n run: |\n python -m pip install --upgrade pip\n pip install pre-commit\n\n - name: Run pre-commit\n run: pre-commit run --all-files --show-diff-on-failure\n"
},
{
"path": ".github/workflows/pypi-publish.yml",
"content": "name: Upload DeepCode Package\n\non:\n release:\n types: [published]\n\npermissions:\n contents: read\n\njobs:\n release-build:\n runs-on: ubuntu-latest\n\n steps:\n - uses: actions/checkout@v4\n\n - uses: actions/setup-python@v5\n with:\n python-version: \"3.x\"\n\n - name: Build release distributions\n run: |\n python -m pip install build\n python -m build\n\n - name: Upload distributions\n uses: actions/upload-artifact@v4\n with:\n name: release-dists\n path: dist/\n\n pypi-publish:\n runs-on: ubuntu-latest\n needs:\n - release-build\n permissions:\n id-token: write\n\n environment:\n name: pypi\n\n steps:\n - name: Retrieve release distributions\n uses: actions/download-artifact@v4\n with:\n name: release-dists\n path: dist/\n\n - name: Publish release distributions to PyPI\n uses: pypa/gh-action-pypi-publish@release/v1\n with:\n packages-dir: dist/\n"
},
{
"path": ".gitignore",
"content": "# Python-related files\n__pycache__/\n*.py[cod]\n*.egg-info/\n.eggs/\n*.tgz\n*.tar.gz\n*.ini\n\n# Virtual Environment\n.venv/\nenv/\nvenv/\n*.env*\n.env_example\n\n\n# Build / Distribution\ndist/\nbuild/\nsite/\n\n# Logs / Reports\n*.log\n*.log.*\n*.logfire\n*.coverage/\nlog/\nlogs/\n\n# Node\nnode_modules/\n\n# Caches\n.cache/\n.mypy_cache/\n.pytest_cache/\n.ruff_cache/\n.gradio/\n.history/\ntemp/\n\n# IDE / Editor Files\n.idea/\n.vscode/\n.vscode/settings.json\n\n# Framework-specific files\nlocal_neo4jWorkDir/\nneo4jWorkDir/\n\n# Data & Storage\ninputs/\nrag_storage/\nexamples/input/\nexamples/output/\ndeepcode-mcp/agent_folders\n\n# Miscellaneous\n.DS_Store\nTODO.md\nignore_this.txt\n*.ignore.*\n\n# unit-test files\ntest_*\nrun_indexer_with_filtering.py\n\n# Cline files\nmemory-bank/\n\n# project files\ndeepcode_lab/\n\n# secrets (use .env or environment variables instead)\nmcp_agent.secrets.yaml\nnanobot_config.json\n"
},
{
"path": ".pre-commit-config.yaml",
"content": "repos:\n - repo: https://github.com/pre-commit/pre-commit-hooks\n rev: v5.0.0\n hooks:\n - id: trailing-whitespace\n - id: end-of-file-fixer\n - id: requirements-txt-fixer\n\n\n - repo: https://github.com/astral-sh/ruff-pre-commit\n rev: v0.6.4\n hooks:\n - id: ruff-format\n - id: ruff\n args: [--fix, --ignore=E402]\n\n - repo: https://github.com/mgedmin/check-manifest\n rev: \"0.49\"\n hooks:\n - id: check-manifest\n stages: [manual]\n"
},
{
"path": "CHANGELOG.md",
"content": "# Changelog\n\nAll notable changes to DeepCode will be documented in this file.\n\n## [1.0.6-jm] - 2025-10-19\n\n### Added\n- **Dynamic Model Limit Detection**: New `utils/model_limits.py` module that automatically detects and adapts to any LLM model's token limits and pricing\n- **Loop Detection System**: `utils/loop_detector.py` prevents infinite loops by detecting repeated tool calls, timeouts, and progress stalls\n- **Progress Tracking**: 8-phase progress tracking (5% → 100%) with file-level progress indicators in both UI and terminal\n- **Abort Mechanism**: \"Stop Processing\" button in UI with global abort flag for clean process termination\n- **Cache Cleanup Scripts**: `start_clean.bat` and `start_clean.ps1` to clear Python cache before starting\n- **Enhanced Error Display**: Real-time error messages in both UI and terminal with timestamps\n- **File Progress Tracking**: Shows files completed/total with estimated time remaining\n\n### Fixed\n- **Critical: False Error Detection**: Fixed overly aggressive error detection that was marking successful operations as failures, causing premature abort and empty file generation\n- **Critical: Empty File Generation**: Files now contain actual code instead of being empty (2-byte files)\n- **Unique Folder Naming**: Each project run now creates `paper_{timestamp}` folders instead of reusing `pdf_output`\n- **PDF Save Location**: PDFs now save to `deepcode_lab/papers/` instead of system temp directory\n- **Duplicate Folder Prevention**: Added session state caching to prevent duplicate folder creation on UI reruns\n- **Token Limit Compliance**: Fixed `max_tokens` to respect model limits dynamically (e.g., gpt-4o-mini's 16,384 token limit)\n- **Empty Plan Detection**: System now fails early with clear error messages when initial plan is empty or invalid\n- **Process Hanging**: Fixed infinite loops and hanging on errors - process now exits cleanly\n- **Token Cost Tracking**: Restored accurate token usage and cost display (was showing $0.0000)\n- **PDF to Markdown Conversion**: Fixed automatic conversion and file location handling\n- **Document Segmentation**: Properly uses configured 50K character threshold from `mcp_agent.config.yaml`\n- **Error Propagation**: Abort mechanism now properly stops process after 10 consecutive real errors\n\n### Changed\n- **Model-Aware Token Management**: Token limits now adapt automatically based on configured model instead of hardcoded values\n- **Cost Calculation**: Dynamic pricing based on actual model rates (OpenAI, Anthropic)\n- **Retry Logic**: Token limits for retries now respect model maximum (87.5% → 95% → 98% of max)\n- **Segmentation Workflow**: Better integration with code implementation phase\n- **Error Handling**: Enhanced error propagation - errors no longer reported as \"success\"\n- **UI Display**: Shows project folder name after PDF conversion for better visibility\n- **Terminal Logging**: Added timestamps to all progress messages\n\n### Technical Improvements\n- Added document-segmentation server to code implementation workflow for better token management\n- Improved error handling in agent orchestration engine with proper cleanup\n- Enhanced subprocess handling on Windows (hide console windows, prevent hanging)\n- Better LibreOffice detection on Windows using direct path checking\n- Fixed input data format consistency (JSON with `paper_path` key)\n- Added comprehensive logging throughout the pipeline\n- Improved resource cleanup on errors and process termination\n\n### Documentation\n- Translated Chinese comments to English in core workflow files\n- Added inline documentation for new utility modules\n- Created startup scripts with clear usage instructions\n\n### Breaking Changes\n- None - all changes are backward compatible\n\n### Known Issues\n- Terminal may show trailing \"Calling Tool...\" line after completion (cosmetic display artifact - process completes successfully)\n- Some Chinese comments remain in non-critical files (cli, tools) - translation in progress\n- tiktoken package optional warning (doesn't affect functionality)\n\n### Success Metrics\n- ✅ Complete end-to-end workflow: DOCX upload → PDF conversion → Markdown → Segmentation → Planning → Code generation\n- ✅ Files generated with actual code content (15+ files with proper implementation)\n- ✅ Single folder per project run (no duplicates)\n- ✅ Dynamic token management working across different models\n- ✅ Accurate cost tracking per model\n- ✅ Clean process termination with proper error handling\n\n---\n\n## [1.0.5] - Previous Release\n\nSee previous releases for earlier changes.\n\n"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2025 ✨Data Intelligence Lab@HKU✨\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "MANIFEST.in",
"content": "include README.md\ninclude LICENSE\ninclude requirements.txt\ninclude __init__.py\ninclude *.png\ninclude *.yaml\nrecursive-include config *.yaml\nrecursive-include prompts *\nrecursive-include schema *\nrecursive-include ui *.py\nrecursive-include cli *.py\nrecursive-include utils *.py\nrecursive-include tools *.py\nrecursive-include workflows *.py\nglobal-exclude *.pyc\nglobal-exclude .git*\nglobal-exclude .history*\nglobal-exclude .ruff_cache*\nglobal-exclude __pycache__*\n"
},
{
"path": "README.md",
"content": "
\n\n---\n\n\n\n\n> *\"Where AI Agents Transform Ideas into Production-Ready Code\"*\n\n
\n\n---\n\n## 📑 Table of Contents\n\n- [📰 News](#-news)\n- [🚀 Key Features](#-key-features)\n- [🏗️ Architecture](#️-architecture)\n- [📊 Experimental Results](#-experimental-results)\n- [🚀 Quick Start](#-quick-start)\n- [🤖 nanobot Integration (Feishu Chatbot)](#-nanobot-integration-feishu-chatbot)\n- [💡 Examples](#-examples)\n - [🎬 Live Demonstrations](#-live-demonstrations)\n- [⭐ Star History](#-star-history)\n- [📄 License](#-license)\n\n\n---\n\n## 📰 News\n\n🎉 **[2026-02] nanobot ✖️ DeepCode. Just chat naturally with openclaw/nanobot to handle your coding tasks:**\n\n
\n
\n
\n
✦
\n
\n
\n
\n\n- [nanobot](https://github.com/HKUDS/nanobot) nanobot now powers your agentic coding & engineering! 🤖💻\n- Step away from your laptop — make vibe coding even more vibe! Code directly from your phone! 📱✨\n- One-command deploy: `./nanobot/run_nanobot.sh` → **[Setup Guide →](#-nanobot-integration-feishu-chatbot)**\n\n
\n
\n
\n \n
\n
\n \n
\n
\nFeishu Bot in Action — Natural language → Full code generation with setup instructions\n
\n\n---\n\n🎉 **[2026-02] New Web UI Experience Upgrade!**\n\n- 🔄 **User-in-Loop Interaction**: Support real-time user interaction during workflows - AI asks clarifying questions directly in the chat\n- 💬 **Inline Interaction Design**: Interaction prompts appear naturally within the chat flow for a seamless experience\n- 🚀 **One-Click Launch**: Simply run `deepcode` to start the new UI (cross-platform: Windows/macOS/Linux)\n- 🔧 **Improved Process Management**: Enhanced service start/stop mechanism with automatic port cleanup\n- 📡 **WebSocket Real-time Communication**: Fixed message loss issues, ensuring proper interaction state synchronization\n\n
\n \n \n DeepCode New Web UI - Modern React-based Interface\n
\n\n---\n\n🎉 **[2025-10-28] DeepCode Achieves SOTA on PaperBench!**\n\nDeepCode sets new benchmarks on OpenAI's PaperBench Code-Dev across all categories:\n\n- 🏆 **Surpasses Human Experts**: **75.9%** (DeepCode) vs Top Machine Learning PhDs 72.4% (+3.5%).\n- 🥇 **Outperforms SOTA Commercial Code Agents**: **84.8%** (DeepCode) vs Leading Commercial Code Agents (+26.1%) (Cursor, Claude Code, and Codex).\n- 🔬 **Advances Scientific Coding**: **73.5%** (DeepCode) vs PaperCoder 51.1% (+22.4%).\n- 🚀 **Beats LLM Agents**: **73.5%** (DeepCode) vs best LLM frameworks 43.3% (+30.2%).\n\n---\n\n## 🚀 Key Features\n\n \n\n
\n
\n
\n\n
\n
🚀 Paper2Code
\n
\n\n
\n \n
\n\n
\n
Automated Implementation of Complex Algorithms
\n
\n\n
\n
Effortlessly converts complex algorithms from research papers into high-quality, production-ready code, accelerating algorithm reproduction.
\n
\n\n\n\n
\n
\n\n
\n
🎨 Text2Web
\n
\n\n
\n \n
\n\n
\n
Automated Front-End Web Development
\n
\n\n
\n
Translates plain textual descriptions into fully functional, visually appealing front-end web code for rapid interface creation.
\n
\n\n\n\n
\n
\n\n
\n
⚙️ Text2Backend
\n
\n\n
\n \n
\n\n
\n
Automated Back-End Development
\n
\n\n
\n
Generates efficient, scalable, and feature-rich back-end code from simple text inputs, streamlining server-side development.
\n
\n\n\n\n
\n
\n
\n\n \n\n---\n\n## 📊 Experimental Results\n\n
\n \n
\n \n\nWe evaluate **DeepCode** on the [*PaperBench*](https://openai.com/index/paperbench/) benchmark (released by OpenAI), a rigorous testbed requiring AI agents to independently reproduce 20 ICML 2024 papers from scratch. The benchmark comprises 8,316 gradable components assessed using SimpleJudge with hierarchical weighting.\n\nOur experiments compare DeepCode against four baseline categories: **(1) Human Experts**, **(2) State-of-the-Art Commercial Code Agents**, **(3) Scientific Code Agents**, and **(4) LLM-Based Agents**.\n\n### ① 🧠 Human Expert Performance (Top Machine Learning PhD)\n\n**DeepCode: 75.9% vs. Top Machine Learning PhD: 72.4% (+3.5%)**\n\nDeepCode achieves **75.9%** on the 3-paper human evaluation subset, **surpassing the best-of-3 human expert baseline (72.4%) by +3.5 percentage points**. This demonstrates that our framework not only matches but exceeds expert-level code reproduction capabilities, representing a significant milestone in autonomous scientific software engineering.\n\n### ② 💼 State-of-the-Art Commercial Code Agents\n\n**DeepCode: 84.8% vs. Best Commercial Agent: 58.7% (+26.1%)**\n\nOn the 5-paper subset, DeepCode substantially outperforms leading commercial coding tools:\n- Cursor: 58.4%\n- Claude Code: 58.7%\n- Codex: 40.0%\n- **DeepCode: 84.8%**\n\nThis represents a **+26.1% improvement** over the leading commercial code agent. All commercial agents utilize Claude Sonnet 4.5 or GPT-5 Codex-high, highlighting that **DeepCode's superior architecture**—rather than base model capability—drives this performance gap.\n\n### ③ 🔬 Scientific Code Agents\n\n**DeepCode: 73.5% vs. PaperCoder: 51.1% (+22.4%)**\n\nCompared to PaperCoder (**51.1%**), the state-of-the-art scientific code reproduction framework, DeepCode achieves **73.5%**, demonstrating a **+22.4% relative improvement**. This substantial margin validates our multi-module architecture combining planning, hierarchical task decomposition, code generation, and iterative debugging over simpler pipeline-based approaches.\n\n### ④ 🤖 LLM-Based Agents\n\n**DeepCode: 73.5% vs. Best LLM Agent: 43.3% (+30.2%)**\n\nDeepCode significantly outperforms all tested LLM agents:\n- Claude 3.5 Sonnet + IterativeAgent: 27.5%\n- o1 + IterativeAgent (36 hours): 42.4%\n- o1 BasicAgent: 43.3%\n- **DeepCode: 73.5%**\n\nThe **+30.2% improvement** over the best-performing LLM agent demonstrates that sophisticated agent scaffolding, rather than extended inference time or larger models, is critical for complex code reproduction tasks.\n\n---\n\n### 🎯 **Autonomous Self-Orchestrating Multi-Agent Architecture**\n\n**The Challenges**:\n\n- 📄 **Implementation Complexity**: Converting academic papers and complex algorithms into working code requires significant technical effort and domain expertise\n\n- 🔬 **Research Bottleneck**: Researchers spend valuable time implementing algorithms instead of focusing on their core research and discovery work\n\n- ⏱️ **Development Delays**: Product teams experience long wait times between concept and testable prototypes, slowing down innovation cycles\n\n- 🔄 **Repetitive Coding**: Developers repeatedly implement similar patterns and functionality instead of building on existing solutions\n\n**DeepCode** addresses these workflow inefficiencies by providing reliable automation for common development tasks, streamlining your development workflow from concept to code.\n\n
\n\n```mermaid\nflowchart LR\n A[\"📄 Research Papers 💬 Text Prompts 🌐 URLs & Document 📎 Files: PDF, DOC, PPTX, TXT, HTML\"] --> B[\"🧠 DeepCode Multi-Agent Engine\"]\n B --> C[\"🚀 Algorithm Implementation 🎨 Frontend Development ⚙️ Backend Development\"]\n\n style A fill:#ff6b6b,stroke:#c0392b,stroke-width:2px,color:#000\n style B fill:#00d4ff,stroke:#0984e3,stroke-width:3px,color:#000\n style C fill:#00b894,stroke:#00a085,stroke-width:2px,color:#000\n```\n\n
\n\n---\n\n## 🏗️ Architecture\n\n### 📊 **System Overview**\n\n**DeepCode** is an AI-powered development platform that automates code generation and implementation tasks. Our multi-agent system handles the complexity of translating requirements into functional, well-structured code, allowing you to focus on innovation rather than implementation details.\n\n🎯 **Technical Capabilities**:\n\n🧬 **Research-to-Production Pipeline** \nMulti-modal document analysis engine that extracts algorithmic logic and mathematical models from academic papers. Generates optimized implementations with proper data structures while preserving computational complexity characteristics.\n\n🪄 **Natural Language Code Synthesis** \nContext-aware code generation using fine-tuned language models trained on curated code repositories. Maintains architectural consistency across modules while supporting multiple programming languages and frameworks.\n\n⚡ **Automated Prototyping Engine** \nIntelligent scaffolding system generating complete application structures including database schemas, API endpoints, and frontend components. Uses dependency analysis to ensure scalable architecture from initial generation.\n\n💎 **Quality Assurance Automation** \nIntegrated static analysis with automated unit test generation and documentation synthesis. Employs AST analysis for code correctness and property-based testing for comprehensive coverage.\n\n🔮 **CodeRAG Integration System** \nAdvanced retrieval-augmented generation combining semantic vector embeddings with graph-based dependency analysis. Automatically discovers optimal libraries and implementation patterns from large-scale code corpus.\n\n---\n\n### 🔧 **Core Techniques**\n\n- 🧠 **Intelligent Orchestration Agent**: Central decision-making system that coordinates workflow phases and analyzes requirements. Employs dynamic planning algorithms to adapt execution strategies in real-time based on evolving project complexity. Dynamically selects optimal processing strategies for each implementation step. \n\n- 💾 **Efficient Memory Mechanism**: Advanced context engineering system that manages large-scale code contexts efficiently. Implements hierarchical memory structures with intelligent compression for handling complex codebases. This component enables instant retrieval of implementation patterns and maintains semantic coherence across extended development sessions. \n\n- 🔍 **Advanced CodeRAG System**: Global code comprehension engine that analyzes complex inter-dependencies across repositories. Performs cross-codebase relationship mapping to understand architectural patterns from a holistic perspective. This module leverages dependency graphs and semantic analysis to provide globally-aware code recommendations during implementation.\n\n---\n\n### 🤖 **Multi-Agent Architecture of DeepCode**:\n\n- **🎯 Central Orchestrating Agent**: Orchestrates entire workflow execution and makes strategic decisions. Coordinates specialized agents based on input complexity analysis. Implements dynamic task planning and resource allocation algorithms. \n\n- **📝 Intent Understanding Agent**: Performs deep semantic analysis of user requirements to decode complex intentions. Extracts functional specifications and technical constraints through advanced NLP processing. Transforms ambiguous human descriptions into precise, actionable development specifications with structured task decomposition. \n\n- **📄 Document Parsing Agent**: Processes complex technical documents and research papers with advanced parsing capabilities. Extracts algorithms and methodologies using document understanding models. Converts academic concepts into practical implementation specifications through intelligent content analysis. \n\n- **🏗️ Code Planning Agent**: Performs architectural design and technology stack optimization. Dynamic planning for adaptive development roadmaps. Enforces coding standards and generates modular structures through automated design pattern selection. \n\n- **🔍 Code Reference Mining Agent**: Discovers relevant repositories and frameworks through intelligent search algorithms. Analyzes codebases for compatibility and integration potential. Provides recommendations based on similarity metrics and automated dependency analysis. \n\n- **📚 Code Indexing Agent**: Builds comprehensive knowledge graphs of discovered codebases. Maintains semantic relationships between code components. Enables intelligent retrieval and cross-reference capabilities. \n\n- **🧬 Code Generation Agent**: Synthesizes gathered information into executable code implementations. Creates functional interfaces and integrates discovered components. Generates comprehensive test suites and documentation for reproducibility.\n\n---\n\n#### 🛠️ **Implementation Tools Matrix**\n\n**🔧 Powered by MCP (Model Context Protocol)**\n\nDeepCode leverages the **Model Context Protocol (MCP)** standard to seamlessly integrate with various tools and services. This standardized approach ensures reliable communication between AI agents and external systems, enabling powerful automation capabilities.\n\n##### 📡 **MCP Servers & Tools**\n\n| 🛠️ **MCP Server** | 🔧 **Primary Function** | 💡 **Purpose & Capabilities** |\n|-------------------|-------------------------|-------------------------------|\n| **🔍 brave** | Web Search Engine | Real-time information retrieval via Brave Search API |\n| **🌐 bocha-mcp** | Alternative Search | Secondary search option with independent API access |\n| **📂 filesystem** | File System Operations | Local file and directory management, read/write operations |\n| **🌐 fetch** | Web Content Retrieval | Fetch and extract content from URLs and web resources |\n| **📥 github-downloader** | Repository Management | Clone and download GitHub repositories for analysis |\n| **📋 file-downloader** | Document Processing | Download and convert files (PDF, DOCX, etc.) to Markdown |\n| **⚡ command-executor** | System Commands | Execute bash/shell commands for environment management |\n| **🧬 code-implementation** | Code Generation Hub | Comprehensive code reproduction with execution and testing |\n| **📚 code-reference-indexer** | Smart Code Search | Intelligent indexing and search of code repositories |\n| **📄 document-segmentation** | Smart Document Analysis | Intelligent document segmentation for large papers and technical documents |\n\n##### 🔧 **Legacy Tool Functions** *(for reference)*\n\n| 🛠️ **Function** | 🎯 **Usage Context** |\n|-----------------|---------------------|\n| **📄 read_code_mem** | Efficient code context retrieval from memory |\n| **✍️ write_file** | Direct file content generation and modification |\n| **🐍 execute_python** | Python code testing and validation |\n| **📁 get_file_structure** | Project structure analysis and organization |\n| **⚙️ set_workspace** | Dynamic workspace and environment configuration |\n| **📊 get_operation_history** | Process monitoring and operation tracking |\n\n\n---\n\n🎛️ **Multi-Interface Framework** \nRESTful API with CLI and web frontends featuring real-time code streaming, interactive debugging, and extensible plugin architecture for CI/CD integration.\n\n**🚀 Multi-Agent Intelligent Pipeline:**\n\n
\n\n### 🌟 **Intelligence Processing Flow**\n\n
\n
\n
\n💡 INPUT LAYER \n📄 Research Papers • 💬 Natural Language • 🌐 URLs • 📋 Requirements\n
\n
\n
\n
\n
\n🎯 CENTRAL ORCHESTRATION \nStrategic Decision Making • Workflow Coordination • Agent Management\n
Intelligent task distribution and parallel processing
\n
\n
\n
\n
\n
🔍 Context Awareness
\n
Deep understanding through CodeRAG integration
\n
\n
\n
\n
\n
⚡ Quality Assurance
\n
Automated testing and validation throughout
\n
\n
\n
\n
\n\n
\n\n---\n\n\n## 🚀 Quick Start\n\n### 📋 **Prerequisites**\n\nBefore installing DeepCode, ensure you have the following:\n\n| Requirement | Version | Purpose |\n|-------------|---------|---------|\n| **Python** | 3.9+ | Core runtime |\n| **Node.js** | 18+ | New UI frontend |\n| **npm** | 8+ | Package management |\n\n```bash\n# Check your versions\npython --version # Should be 3.9+\nnode --version # Should be 18+\nnpm --version # Should be 8+\n```\n\n\n📥 Install Node.js (if not installed)\n\n```bash\n# macOS (using Homebrew)\nbrew install node\n\n# Ubuntu/Debian\ncurl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -\nsudo apt-get install -y nodejs\n\n# Windows\n# Download from https://nodejs.org/\n```\n\n\n\n### 📦 **Step 1: Installation**\n\nChoose one of the following installation methods:\n\n#### ⚡ **Direct Installation (Recommended)**\n\n```bash\n# 🚀 Install DeepCode package directly\npip install deepcode-hku\n\n# 🔑 Download configuration files\ncurl -O https://raw.githubusercontent.com/HKUDS/DeepCode/main/mcp_agent.config.yaml\ncurl -O https://raw.githubusercontent.com/HKUDS/DeepCode/main/mcp_agent.secrets.yaml\n```\n\n#### 🔧 **Development Installation (From Source)**\n\n\n📂 Click to expand development installation options\n\n##### 🔥 **Using UV (Recommended for Development)**\n\n```bash\ngit clone https://github.com/HKUDS/DeepCode.git\ncd DeepCode/\n\ncurl -LsSf https://astral.sh/uv/install.sh | sh\nuv venv --python=3.13\nsource .venv/bin/activate # On Windows: .venv\\Scripts\\activate\nuv pip install -r requirements.txt\n\n# Install frontend dependencies\nnpm install --prefix new_ui/frontend\n```\n\n##### 🐍 **Using Traditional pip**\n\n```bash\ngit clone https://github.com/HKUDS/DeepCode.git\ncd DeepCode/\n\npip install -r requirements.txt\n\n# Install frontend dependencies\nnpm install --prefix new_ui/frontend\n```\n\n\n\n### 🔧 **Step 2: Configuration**\n\n> The following configuration applies to **all installation methods** (pip, UV, source, and Docker).\n\n#### 🔑 API Keys *(required)*\n\nEdit `mcp_agent.secrets.yaml` with your API keys:\n\n```yaml\n# At least ONE provider API key is required\nopenai:\n api_key: \"your_openai_api_key\"\n base_url: \"https://openrouter.ai/api/v1\" # Optional: for OpenRouter or custom endpoints\n\nanthropic:\n api_key: \"your_anthropic_api_key\" # For Claude models\n\ngoogle:\n api_key: \"your_google_api_key\" # For Gemini models\n```\n\n#### 🤖 LLM Provider *(optional)*\n\nEdit `mcp_agent.config.yaml` to choose your preferred LLM provider (line ~106):\n\n```yaml\n# Options: \"google\", \"anthropic\", \"openai\"\n# If not set or unavailable, will automatically fallback to first available provider\nllm_provider: \"google\"\n```\n\n#### 🔍 Search API Keys *(optional)*\n\nConfigure web search in `mcp_agent.config.yaml`:\n\n```yaml\n# For Brave Search (default) — set in brave.env section (line ~28)\nbrave:\n env:\n BRAVE_API_KEY: \"your_brave_api_key_here\"\n\n# For Bocha-MCP (alternative) — set in bocha-mcp.env section (line ~74)\nbocha-mcp:\n env:\n BOCHA_API_KEY: \"your_bocha_api_key_here\"\n```\n\n#### 📄 Document Segmentation *(optional)*\n\nControl document processing in `mcp_agent.config.yaml`:\n\n```yaml\ndocument_segmentation:\n enabled: true # true/false — whether to use intelligent document segmentation\n size_threshold_chars: 50000 # Document size threshold to trigger segmentation\n```\n\n\n🪟 Windows Users: Additional MCP Server Configuration\n\nIf you're using Windows, you may need to configure MCP servers manually in `mcp_agent.config.yaml`:\n\n```bash\n# 1. Install MCP servers globally\nnpm i -g @modelcontextprotocol/server-brave-search\nnpm i -g @modelcontextprotocol/server-filesystem\n\n# 2. Find your global node_modules path\nnpm -g root\n```\n\nThen update your `mcp_agent.config.yaml` to use absolute paths:\n\n```yaml\nmcp:\n servers:\n brave:\n command: \"node\"\n args: [\"C:/Program Files/nodejs/node_modules/@modelcontextprotocol/server-brave-search/dist/index.js\"]\n filesystem:\n command: \"node\"\n args: [\"C:/Program Files/nodejs/node_modules/@modelcontextprotocol/server-filesystem/dist/index.js\", \".\"]\n```\n\n> **Note**: Replace the path with your actual global node_modules path from step 2.\n\n\n\n\n🔍 Search Server Configuration (Optional)\n\nDeepCode supports multiple search servers for web search functionality. You can configure your preferred option in `mcp_agent.config.yaml`:\n\n```yaml\n# Default search server configuration\n# Options: \"brave\" or \"bocha-mcp\"\ndefault_search_server: \"brave\"\n```\n\n**Available Options:**\n- **🔍 Brave Search** (`\"brave\"`): Default option with high-quality search results. Requires `BRAVE_API_KEY`. Recommended for most users.\n- **🌐 Bocha-MCP** (`\"bocha-mcp\"`): Alternative search server. Requires `BOCHA_API_KEY`. Uses local Python server implementation.\n\n**Full MCP server configuration in mcp_agent.config.yaml:**\n```yaml\n# For Brave Search (default) - around line 28\nbrave:\n command: \"npx\"\n args: [\"-y\", \"@modelcontextprotocol/server-brave-search\"]\n env:\n BRAVE_API_KEY: \"your_brave_api_key_here\"\n\n# For Bocha-MCP (alternative) - around line 74\nbocha-mcp:\n command: \"python\"\n args: [\"tools/bocha_search_server.py\"]\n env:\n PYTHONPATH: \".\"\n BOCHA_API_KEY: \"your_bocha_api_key_here\"\n```\n\n> **💡 Tip**: Both search servers require API key configuration. Choose the one that best fits your API access and requirements.\n\n\n\n### ⚡ **Step 3: Launch Application**\n\nChoose your preferred launch method:\n\n
\n
\n
🐳 Docker (Recommended)
\n
🚀 Local (deepcode command)
\n
🛠️ Other Methods
\n
\n
\n\nNo Python/Node needed — everything in container.\n\n```bash\ngit clone https://github.com/HKUDS/DeepCode.git\ncd DeepCode/\ncp mcp_agent.secrets.yaml.example \\\n mcp_agent.secrets.yaml\n# Edit secrets with your API keys\n\n./deepcode_docker/run_docker.sh\n# Access → http://localhost:8000\n```\n\n
\n\nAuto-installs deps on first run.\n\n```bash\ndeepcode\n# Frontend → http://localhost:5173\n# Backend → http://localhost:8000\n# Ctrl+C to stop\n```\n\nFeatures: User-in-Loop, real-time progress, inline chat.\n\n
\n\n\n🐳 Docker Management Commands\n\n```bash\n./deepcode_docker/run_docker.sh stop # Stop\n./deepcode_docker/run_docker.sh restart # Restart (no rebuild needed for config changes)\n./deepcode_docker/run_docker.sh --build # Force rebuild\n./deepcode_docker/run_docker.sh logs # Real-time logs\n./deepcode_docker/run_docker.sh status # Health check\n./deepcode_docker/run_docker.sh clean # Remove containers & images\n```\n\nOr with Docker Compose directly:\n```bash\ndocker compose -f deepcode_docker/docker-compose.yml up --build # Build & start\ndocker compose -f deepcode_docker/docker-compose.yml down # Stop\ndocker compose -f deepcode_docker/docker-compose.yml logs -f # Logs\n```\n\n> **💡** Config files are mounted as volumes — edit and restart, no rebuild needed.\n> **💡** Windows users: run `docker compose` commands directly if shell scripts aren't available.\n\n\n\n### 🎯 **Step 4: Generate Code**\n\n1. **📄 Input** — Upload a research paper, type requirements, or paste a URL\n2. **🤖 Processing** — The multi-agent system analyzes, plans, and generates\n3. **⚡ Output** — Receive production-ready code with tests and documentation\n\n---\n\n### 🔧 **Troubleshooting**\n\n\n❓ Common Issues & Solutions\n\n| Problem | Cause | Fix |\n|---|---|---|\n| Docker build fails with `tsc: not found` | Corrupted build cache | `docker builder prune -f` then rebuild with `--no-cache` |\n| `error during connect` / `cannot find the file` | Docker Desktop not running | Start Docker Desktop, wait until ready, retry |\n| Frontend blank page | Corrupted `node_modules` | `cd new_ui/frontend && rm -rf node_modules && npm install` |\n| `ERR_CONNECTION_REFUSED` | Wrong port / backend not running | Docker: `http://localhost:8000`. Local: `http://localhost:5173` |\n| `npm install` → `Could not read package.json` | Wrong directory | Use `npm install --prefix new_ui/frontend` |\n| Windows: MCP servers not working | Need absolute paths | See [Windows MCP Configuration](#-step-2-configuration) above |\n\n\n\n ---\n\n## 🤖 nanobot Integration (Feishu Chatbot)\n\n> Chat with DeepCode from **Feishu** — powered by [nanobot](https://github.com/HKUDS/nanobot).\n\n
\n\n```mermaid\nflowchart LR\n subgraph Clients[\"💬 Chat Platforms\"]\n direction TB\n F[\"Feishu WebSocket\"]\n T[\"Telegram Polling\"]\n D[\"Discord Gateway\"]\n end\n\n subgraph Gateway[\"🐈 nanobot Gateway\"]\n direction TB\n A[\"Agent Loop LLM + Tool Calls\"]\n end\n\n subgraph Engine[\"🧠 DeepCode Engine\"]\n direction TB\n P2C[\"Paper → Code\"]\n C2C[\"Chat → Code\"]\n TRK[\"Task Tracking\"]\n end\n\n F & T & D <-->|\"messages\"| A\n A -->|\"HTTP API\"| P2C & C2C & TRK\n A -.->|\"LLM API\"| LLM[\"☁️ OpenRouter\"]\n\n style Clients fill:#1a1a2e,stroke:#00d9ff,color:#fff\n style Gateway fill:#1a1a2e,stroke:#4ecdc4,color:#fff\n style Engine fill:#1a1a2e,stroke:#ff6b6b,color:#fff\n style LLM fill:#1a1a2e,stroke:#9b59b6,color:#fff\n```\n\n
\n\n
\n
\n
\n
✦
\n
\n
\n
\n\nBoth services run inside the same **Docker Compose** network. Prerequisites: **Docker Desktop** + **OpenRouter API Key** ([get one](https://openrouter.ai/keys)) + **Feishu App**.\n\n---\n\n### Step 1 · Create a Feishu Bot\n\n\nFeishu / Lark (Recommended — WebSocket, no public IP needed)\n\n1. Go to [Feishu Open Platform](https://open.feishu.cn/app) → **Create Custom App**\n2. Enable **Bot** capability in App Features\n3. Add permissions: `im:message` · `im:message:send_as_bot`\n4. Event Subscription → select **Long Connection** → add `im.message.receive_v1`\n5. Note your **App ID** (`cli_xxx`) and **App Secret** → Publish the app\n\n> **Note**: Feishu requires an active WebSocket connection before you can save \"Long Connection\" mode. Start nanobot first (Step 3), then come back to configure Event Subscription.\n\n\n\n### Step 2 · Configure\n\n```bash\ncp nanobot_config.json.example nanobot_config.json\n```\n\nEdit `nanobot_config.json` — fill in the 3 required fields:\n\n```jsonc\n{\n \"channels\": {\n \"feishu\": {\n \"enabled\": true,\n \"appId\": \"cli_xxx\", // ← Feishu App ID\n \"appSecret\": \"xxx\", // ← Feishu App Secret\n \"allowFrom\": [] // [] = allow all users\n }\n },\n \"providers\": {\n \"openrouter\": {\n \"apiKey\": \"sk-or-v1-xxx\" // ← OpenRouter API Key\n }\n },\n \"agents\": {\n \"defaults\": {\n \"model\": \"anthropic/claude-sonnet-4-20250514\"\n }\n }\n}\n```\n\n> **Model choice**: Any model on [openrouter.ai/models](https://openrouter.ai/models). Use `anthropic/claude-sonnet-4-20250514` for English, `minimax/minimax-m2.1` for Chinese.\n\n---\n\n### Step 3 · Launch\n\nMake sure `mcp_agent.secrets.yaml` has your DeepCode API keys (see [Configuration](#-step-2-configuration)), then:\n\n```bash\n./nanobot/run_nanobot.sh -d # Start both DeepCode + nanobot in background\n```\n\nThe script checks Docker, validates configs, builds images (first run only), and starts both containers.\n\n```\n✓ DeepCode API: http://localhost:8000\n✓ Nanobot: http://localhost:18790\n```\n\nNow open Feishu → find your bot → send a message!\n\n\nManagement Commands\n\n```bash\n./nanobot/run_nanobot.sh # Start (foreground)\n./nanobot/run_nanobot.sh -d # Start (background)\n./nanobot/run_nanobot.sh stop # Stop all services\n./nanobot/run_nanobot.sh restart # Restart (config changes take effect immediately)\n./nanobot/run_nanobot.sh --build # Force rebuild Docker images\n./nanobot/run_nanobot.sh logs # View real-time logs\n./nanobot/run_nanobot.sh status # Health check\n./nanobot/run_nanobot.sh clean # Remove containers & images\n```\n\n\n\n\nTroubleshooting\n\n| Problem | Fix |\n|---|---|\n| Feishu bot doesn't respond | Check logs (`./nanobot/run_nanobot.sh logs`), verify `appId`/`appSecret`, ensure app is published with Long Connection mode |\n| Can't connect to DeepCode | Verify `deepcode` container is healthy: `curl http://localhost:8000/health` |\n| Wrong language output | Switch model — `minimax-m2.1` defaults to Chinese, use Claude/GPT for English |\n| Config not taking effect | Just restart: `./nanobot/run_nanobot.sh restart` (no rebuild needed) |\n| Clear chat history | Send `/clear` in chat, or: `docker exec nanobot sh -c 'rm -rf /root/.nanobot/sessions/*.jsonl'` |\n\n\n\n---\n\n## 💡 Examples\n\n\n\n### 🎬 **Live Demonstrations**\n\n\n\n
\n
\n
\n\n#### 📄 **Paper2Code Demo**\n**Research to Implementation**\n\n
\n\n---\n\n### 🚀 **Ready to Transform Development?**\n\n
\n\n
\n \n \n \n
\n\n---\n\n
\n\n### 📖 **Citation**\n\n\nIf you find DeepCode useful in your research or applications, please kindly cite:\n\n```\n@misc{li2025deepcodeopenagenticcoding,\n title={DeepCode: Open Agentic Coding},\n author={Zongwei Li and Zhonghang Li and Zirui Guo and Xubin Ren and Chao Huang},\n year={2025},\n eprint={2512.07921},\n archivePrefix={arXiv},\n primaryClass={cs.SE},\n url={https://arxiv.org/abs/2512.07921},\n}\n```\n\n---\n\n\n### 📄 **License**\n\n
\n\n\n\n**MIT License** - Copyright (c) 2025 Data Intelligence Lab, The University of Hong Kong\n\n---\n\n\n\n\n
\n"
},
{
"path": "__init__.py",
"content": "\"\"\"\nDeepCode - AI Research Engine\n\n🧬 Next-Generation AI Research Automation Platform\n⚡ Transform research papers into working code automatically\n\"\"\"\n\n__version__ = \"1.2.0\"\n__author__ = \"DeepCode Team\"\n__url__ = \"https://github.com/HKUDS/DeepCode\"\n__repo__ = \"https://github.com/Jany-M/DeepCode/\"\n\n# Import main components for easy access\nfrom utils import FileProcessor, DialogueLogger\n\n__all__ = [\n \"FileProcessor\",\n \"DialogueLogger\",\n \"__version__\",\n \"__author__\",\n \"__url__\",\n]\n"
},
{
"path": "cli/__init__.py",
"content": "\"\"\"\nCLI Module for DeepCode Agent\nDeepCode智能体CLI模块\n\n包含以下组件 / Contains the following components:\n- cli_app: CLI应用主程序 / CLI application main program\n- cli_interface: CLI界面组件 / CLI interface components\n- cli_launcher: CLI启动器 / CLI launcher\n\"\"\"\n\n__version__ = \"1.0.0\"\n__author__ = \"DeepCode Team - Data Intelligence Lab @ HKU\"\n\nfrom .cli_app import main as cli_main\nfrom .cli_interface import CLIInterface\nfrom .cli_launcher import main as launcher_main\n\n__all__ = [\"cli_main\", \"CLIInterface\", \"launcher_main\"]\n"
},
{
"path": "cli/cli_app.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nDeepCode - CLI Application Main Program\n深度代码 - CLI应用主程序\n\n🧬 Open-Source Code Agent by Data Intelligence Lab @ HKU\n⚡ Revolutionizing research reproducibility through collaborative AI\n\"\"\"\n\nimport os\nimport sys\nimport asyncio\nimport time\nimport json\n\n# 禁止生成.pyc文件\nos.environ[\"PYTHONDONTWRITEBYTECODE\"] = \"1\"\n\n# 添加项目根目录到路径\ncurrent_dir = os.path.dirname(os.path.abspath(__file__))\nparent_dir = os.path.dirname(current_dir)\nif parent_dir not in sys.path:\n sys.path.insert(0, parent_dir)\n\n# 导入MCP应用和工作流\n\nfrom cli.workflows import CLIWorkflowAdapter\nfrom cli.cli_interface import CLIInterface, Colors\n\n\nclass CLIApp:\n \"\"\"CLI应用主类 - 升级版智能体编排引擎\"\"\"\n\n def __init__(self):\n self.cli = CLIInterface()\n self.workflow_adapter = CLIWorkflowAdapter(cli_interface=self.cli)\n self.app = None # Will be initialized by workflow adapter\n self.logger = None\n self.context = None\n # Document segmentation will be managed by CLI interface\n\n async def initialize_mcp_app(self):\n \"\"\"初始化MCP应用 - 使用工作流适配器\"\"\"\n # Workflow adapter will handle MCP initialization\n return await self.workflow_adapter.initialize_mcp_app()\n\n async def cleanup_mcp_app(self):\n \"\"\"清理MCP应用 - 使用工作流适配器\"\"\"\n await self.workflow_adapter.cleanup_mcp_app()\n\n async def process_requirement_analysis_non_interactive(self, initial_idea: str):\n \"\"\"处理需求分析工作流（非交互式，用于命令行参数） (NEW: matching UI version)\"\"\"\n try:\n self.cli.print_separator()\n self.cli.print_status(\n \"🧠 Starting requirement analysis workflow...\", \"info\"\n )\n\n # Step 1: Generate guiding questions\n self.cli.print_status(\n \"🤖 Generating AI-guided questions to refine your requirements...\",\n \"processing\",\n )\n\n questions_result = (\n await self.workflow_adapter.execute_requirement_analysis_workflow(\n user_input=initial_idea, analysis_mode=\"generate_questions\"\n )\n )\n\n if questions_result[\"status\"] != \"success\":\n self.cli.print_status(\n f\"❌ Failed to generate questions: {questions_result.get('error', 'Unknown error')}\",\n \"error\",\n )\n return questions_result\n\n # Step 2: Display questions\n questions_json = questions_result[\"result\"]\n self.cli.display_guiding_questions(questions_json)\n\n # For non-interactive mode, we can't get user answers, so we provide a summary\n self.cli.print_status(\n \"ℹ️ In non-interactive mode, using initial idea for implementation\",\n \"info\",\n )\n self.cli.print_status(\n \"💡 For guided analysis, please use interactive mode (python main_cli.py)\",\n \"info\",\n )\n\n # Proceed directly with the initial idea as the requirement\n self.cli.print_status(\n \"🚀 Starting code implementation based on initial requirements...\",\n \"processing\",\n )\n\n implementation_result = await self.process_input(initial_idea, \"chat\")\n\n return {\n \"status\": \"success\",\n \"questions_generated\": questions_result,\n \"implementation\": implementation_result,\n }\n\n except Exception as e:\n error_msg = str(e)\n self.cli.print_error_box(\"Requirement Analysis Error\", error_msg)\n self.cli.print_status(\n f\"Error during requirement analysis: {error_msg}\", \"error\"\n )\n\n return {\"status\": \"error\", \"error\": error_msg}\n\n async def process_requirement_analysis(self):\n \"\"\"处理需求分析工作流（交互式） (NEW: matching UI version)\"\"\"\n try:\n # Step 1: Get initial requirements from user\n self.cli.print_separator()\n self.cli.print_status(\n \"🧠 Starting requirement analysis workflow...\", \"info\"\n )\n\n user_input = self.cli.get_requirement_analysis_input()\n\n if not user_input:\n self.cli.print_status(\"Requirement analysis cancelled\", \"warning\")\n return {\"status\": \"cancelled\"}\n\n # Step 2: Generate guiding questions\n self.cli.print_status(\n \"🤖 Generating AI-guided questions to refine your requirements...\",\n \"processing\",\n )\n\n questions_result = (\n await self.workflow_adapter.execute_requirement_analysis_workflow(\n user_input=user_input, analysis_mode=\"generate_questions\"\n )\n )\n\n if questions_result[\"status\"] != \"success\":\n self.cli.print_status(\n f\"❌ Failed to generate questions: {questions_result.get('error', 'Unknown error')}\",\n \"error\",\n )\n return questions_result\n\n # Step 3: Display questions and get user answers\n questions_json = questions_result[\"result\"]\n self.cli.display_guiding_questions(questions_json)\n\n # Ask if user wants to answer the questions\n proceed = (\n input(\n f\"\\n{Colors.BOLD}{Colors.YELLOW}Would you like to answer these questions? (y/n):{Colors.ENDC} \"\n )\n .strip()\n .lower()\n )\n\n if proceed != \"y\":\n self.cli.print_status(\n \"You can still use the initial requirements for chat input\",\n \"info\",\n )\n return {\"status\": \"partial\", \"initial_requirements\": user_input}\n\n user_answers = self.cli.get_question_answers(questions_json)\n\n # Step 4: Generate requirement summary\n self.cli.print_status(\n \"📄 Generating detailed requirement document...\", \"processing\"\n )\n\n summary_result = (\n await self.workflow_adapter.execute_requirement_analysis_workflow(\n user_input=user_input,\n analysis_mode=\"summarize_requirements\",\n user_answers=user_answers,\n )\n )\n\n if summary_result[\"status\"] != \"success\":\n self.cli.print_status(\n f\"❌ Failed to generate summary: {summary_result.get('error', 'Unknown error')}\",\n \"error\",\n )\n return summary_result\n\n # Step 5: Display requirement summary\n requirement_summary = summary_result[\"result\"]\n should_proceed = self.cli.display_requirement_summary(requirement_summary)\n\n if should_proceed:\n # Step 6: Proceed with chat-based implementation\n self.cli.print_status(\n \"🚀 Starting code implementation based on analyzed requirements...\",\n \"processing\",\n )\n\n implementation_result = await self.process_input(\n requirement_summary, \"chat\"\n )\n\n return {\n \"status\": \"success\",\n \"requirement_analysis\": summary_result,\n \"implementation\": implementation_result,\n }\n else:\n self.cli.print_status(\n \"Requirement analysis completed. Implementation skipped.\", \"info\"\n )\n return {\n \"status\": \"success\",\n \"requirement_analysis\": summary_result,\n \"implementation\": None,\n }\n\n except Exception as e:\n error_msg = str(e)\n self.cli.print_error_box(\"Requirement Analysis Error\", error_msg)\n self.cli.print_status(\n f\"Error during requirement analysis: {error_msg}\", \"error\"\n )\n\n return {\"status\": \"error\", \"error\": error_msg}\n\n async def process_input(self, input_source: str, input_type: str):\n \"\"\"处理输入源（URL或文件）- 使用升级版智能体编排引擎\"\"\"\n try:\n # Document segmentation configuration is managed by CLI interface\n\n self.cli.print_separator()\n self.cli.print_status(\n \"🚀 Starting intelligent agent orchestration...\", \"processing\"\n )\n\n # 显示处理阶段（根据配置决定）\n chat_mode = input_type == \"chat\"\n self.cli.display_processing_stages(\n 0, self.cli.enable_indexing, chat_mode=chat_mode\n )\n\n # 使用工作流适配器进行处理\n result = await self.workflow_adapter.process_input_with_orchestration(\n input_source=input_source,\n input_type=input_type,\n enable_indexing=self.cli.enable_indexing,\n )\n\n if result[\"status\"] == \"success\":\n # 显示完成状态\n if chat_mode:\n final_stage = 4\n else:\n final_stage = 8 if self.cli.enable_indexing else 5\n self.cli.display_processing_stages(\n final_stage, self.cli.enable_indexing, chat_mode=chat_mode\n )\n self.cli.print_status(\n \"🎉 Agent orchestration completed successfully!\", \"complete\"\n )\n\n # 显示结果\n self.display_results(\n result.get(\"analysis_result\", \"\"),\n result.get(\"download_result\", \"\"),\n result.get(\"repo_result\", \"\"),\n result.get(\"pipeline_mode\", \"comprehensive\"),\n )\n else:\n self.cli.print_status(\n f\"❌ Processing failed: {result.get('error', 'Unknown error')}\",\n \"error\",\n )\n\n # 添加到历史记录\n self.cli.add_to_history(input_source, result)\n\n return result\n\n except Exception as e:\n error_msg = str(e)\n self.cli.print_error_box(\"Agent Orchestration Error\", error_msg)\n self.cli.print_status(f\"Error during orchestration: {error_msg}\", \"error\")\n\n # 添加错误到历史记录\n error_result = {\"status\": \"error\", \"error\": error_msg}\n self.cli.add_to_history(input_source, error_result)\n\n return error_result\n\n def display_results(\n self,\n analysis_result: str,\n download_result: str,\n repo_result: str,\n pipeline_mode: str = \"comprehensive\",\n ):\n \"\"\"显示处理结果\"\"\"\n self.cli.print_results_header()\n\n # 显示流水线模式\n if pipeline_mode == \"chat\":\n mode_display = \"💬 Chat Planning Mode\"\n elif pipeline_mode == \"comprehensive\":\n mode_display = \"🧠 Comprehensive Mode\"\n else:\n mode_display = \"⚡ Optimized Mode\"\n print(\n f\"{Colors.BOLD}{Colors.PURPLE}🤖 PIPELINE MODE: {mode_display}{Colors.ENDC}\"\n )\n self.cli.print_separator(\"─\", 79, Colors.PURPLE)\n\n print(f\"{Colors.BOLD}{Colors.OKCYAN}📊 ANALYSIS PHASE RESULTS:{Colors.ENDC}\")\n self.cli.print_separator(\"─\", 79, Colors.CYAN)\n\n # 尝试解析并格式化分析结果\n try:\n if analysis_result.strip().startswith(\"{\"):\n parsed_analysis = json.loads(analysis_result)\n print(json.dumps(parsed_analysis, indent=2, ensure_ascii=False))\n else:\n print(\n analysis_result[:1000] + \"...\"\n if len(analysis_result) > 1000\n else analysis_result\n )\n except Exception:\n print(\n analysis_result[:1000] + \"...\"\n if len(analysis_result) > 1000\n else analysis_result\n )\n\n print(f\"\\n{Colors.BOLD}{Colors.PURPLE}📥 DOWNLOAD PHASE RESULTS:{Colors.ENDC}\")\n self.cli.print_separator(\"─\", 79, Colors.PURPLE)\n print(\n download_result[:1000] + \"...\"\n if len(download_result) > 1000\n else download_result\n )\n\n print(\n f\"\\n{Colors.BOLD}{Colors.GREEN}⚙️ IMPLEMENTATION PHASE RESULTS:{Colors.ENDC}\"\n )\n self.cli.print_separator(\"─\", 79, Colors.GREEN)\n print(repo_result[:1000] + \"...\" if len(repo_result) > 1000 else repo_result)\n\n # 尝试提取生成的代码目录信息\n if \"Code generated in:\" in repo_result:\n code_dir = (\n repo_result.split(\"Code generated in:\")[-1].strip().split(\"\\n\")[0]\n )\n print(\n f\"\\n{Colors.BOLD}{Colors.YELLOW}📁 Generated Code Directory: {Colors.ENDC}{code_dir}\"\n )\n\n # 显示处理完成的工作流阶段\n print(\n f\"\\n{Colors.BOLD}{Colors.OKCYAN}🔄 COMPLETED WORKFLOW STAGES:{Colors.ENDC}\"\n )\n\n if pipeline_mode == \"chat\":\n stages = [\n \"🚀 Engine Initialization\",\n \"💬 Requirements Analysis\",\n \"🏗️ Workspace Setup\",\n \"📝 Implementation Plan Generation\",\n \"⚙️ Code Implementation\",\n ]\n else:\n stages = [\n \"📄 Document Processing\",\n \"🔍 Reference Analysis\",\n \"📋 Plan Generation\",\n \"📦 Repository Download\",\n \"🗂️ Codebase Indexing\",\n \"⚙️ Code Implementation\",\n ]\n\n for stage in stages:\n print(f\" ✅ {stage}\")\n\n self.cli.print_separator()\n\n async def run_interactive_session(self):\n \"\"\"运行交互式会话\"\"\"\n # 清屏并显示启动界面\n self.cli.clear_screen()\n self.cli.print_logo()\n self.cli.print_welcome_banner()\n\n # 初始化MCP应用\n await self.initialize_mcp_app()\n\n try:\n # 主交互循环\n while self.cli.is_running:\n self.cli.create_menu()\n choice = self.cli.get_user_input()\n\n if choice in [\"q\", \"quit\", \"exit\"]:\n self.cli.print_goodbye()\n break\n\n elif choice in [\"u\", \"url\"]:\n url = self.cli.get_url_input()\n if url:\n await self.process_input(url, \"url\")\n\n elif choice in [\"f\", \"file\"]:\n file_path = self.cli.upload_file_gui()\n if file_path:\n await self.process_input(f\"file://{file_path}\", \"file\")\n\n elif choice in [\"t\", \"chat\", \"text\"]:\n chat_input = self.cli.get_chat_input()\n if chat_input:\n await self.process_input(chat_input, \"chat\")\n\n elif choice in [\"r\", \"req\", \"requirement\", \"requirements\"]:\n # NEW: Requirement Analysis workflow\n await self.process_requirement_analysis()\n\n elif choice in [\"h\", \"history\"]:\n self.cli.show_history()\n\n elif choice in [\"c\", \"config\", \"configure\"]:\n # Show configuration menu - all settings managed by CLI interface\n self.cli.show_configuration_menu()\n\n else:\n self.cli.print_status(\n \"Invalid choice. Please select U, F, T, R, C, H, or Q.\",\n \"warning\",\n )\n\n # 询问是否继续\n if self.cli.is_running and choice in [\n \"u\",\n \"f\",\n \"t\",\n \"r\",\n \"chat\",\n \"text\",\n \"req\",\n \"requirement\",\n \"requirements\",\n ]:\n if not self.cli.ask_continue():\n self.cli.is_running = False\n self.cli.print_status(\"Session ended by user\", \"info\")\n\n except KeyboardInterrupt:\n print(f\"\\n{Colors.WARNING}⚠️ Process interrupted by user{Colors.ENDC}\")\n except Exception as e:\n print(f\"\\n{Colors.FAIL}❌ Unexpected error: {str(e)}{Colors.ENDC}\")\n finally:\n # 清理资源\n await self.cleanup_mcp_app()\n\n\nasync def main():\n \"\"\"主函数\"\"\"\n start_time = time.time()\n\n try:\n # 创建并运行CLI应用\n app = CLIApp()\n await app.run_interactive_session()\n\n except KeyboardInterrupt:\n print(f\"\\n{Colors.WARNING}⚠️ Application interrupted by user{Colors.ENDC}\")\n except Exception as e:\n print(f\"\\n{Colors.FAIL}❌ Application error: {str(e)}{Colors.ENDC}\")\n finally:\n end_time = time.time()\n print(\n f\"\\n{Colors.BOLD}{Colors.CYAN}⏱️ Total runtime: {end_time - start_time:.2f} seconds{Colors.ENDC}\"\n )\n\n # 清理缓存文件\n print(f\"{Colors.YELLOW}🧹 Cleaning up cache files...{Colors.ENDC}\")\n if os.name == \"nt\": # Windows\n os.system(\n \"powershell -Command \\\"Get-ChildItem -Path . -Filter '__pycache__' -Recurse -Directory | Remove-Item -Recurse -Force\\\" 2>nul\"\n )\n else: # Unix/Linux/macOS\n os.system('find . -type d -name \"__pycache__\" -exec rm -r {} + 2>/dev/null')\n\n print(\n f\"{Colors.OKGREEN}✨ Goodbye! Thanks for using DeepCode CLI! ✨{Colors.ENDC}\"\n )\n\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n"
},
{
"path": "cli/cli_interface.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nEnhanced CLI Interface Module for DeepCode\n增强版CLI界面模块 - 专为DeepCode设计\n\"\"\"\n\nimport os\nimport time\nimport platform\nfrom typing import Optional\n\n\nclass Colors:\n \"\"\"ANSI color codes for terminal styling\"\"\"\n\n HEADER = \"\\033[95m\"\n OKBLUE = \"\\033[94m\"\n OKCYAN = \"\\033[96m\"\n OKGREEN = \"\\033[92m\"\n WARNING = \"\\033[93m\"\n FAIL = \"\\033[91m\"\n ENDC = \"\\033[0m\"\n BOLD = \"\\033[1m\"\n UNDERLINE = \"\\033[4m\"\n\n # Gradient colors\n PURPLE = \"\\033[35m\"\n MAGENTA = \"\\033[95m\"\n BLUE = \"\\033[34m\"\n CYAN = \"\\033[36m\"\n GREEN = \"\\033[32m\"\n YELLOW = \"\\033[33m\"\n\n\nclass CLIInterface:\n \"\"\"Enhanced CLI interface with modern styling for DeepCode\"\"\"\n\n def __init__(self):\n self.uploaded_file = None\n self.is_running = True\n self.processing_history = []\n self.enable_indexing = (\n False # Default configuration (matching UI: fast mode by default)\n )\n\n # Load segmentation config from the same source as UI\n self._load_segmentation_config()\n\n # Initialize tkinter availability\n self._init_tkinter()\n\n def _load_segmentation_config(self):\n \"\"\"Load segmentation configuration from mcp_agent.config.yaml\"\"\"\n try:\n from utils.llm_utils import get_document_segmentation_config\n\n seg_config = get_document_segmentation_config()\n self.segmentation_enabled = seg_config.get(\"enabled\", True)\n self.segmentation_threshold = seg_config.get(\"size_threshold_chars\", 50000)\n except Exception as e:\n print(f\"⚠️ Warning: Failed to load segmentation config: {e}\")\n # Fall back to defaults\n self.segmentation_enabled = True\n self.segmentation_threshold = 50000\n\n def _save_segmentation_config(self):\n \"\"\"Save segmentation configuration to mcp_agent.config.yaml\"\"\"\n import yaml\n import os\n\n # Get the project root directory (where mcp_agent.config.yaml is located)\n current_file = os.path.abspath(__file__)\n cli_dir = os.path.dirname(current_file) # cli directory\n project_root = os.path.dirname(cli_dir) # project root\n config_path = os.path.join(project_root, \"mcp_agent.config.yaml\")\n\n try:\n # Read current config\n with open(config_path, \"r\", encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n\n # Update document segmentation settings\n if \"document_segmentation\" not in config:\n config[\"document_segmentation\"] = {}\n\n config[\"document_segmentation\"][\"enabled\"] = self.segmentation_enabled\n config[\"document_segmentation\"][\"size_threshold_chars\"] = (\n self.segmentation_threshold\n )\n\n # Write updated config\n with open(config_path, \"w\", encoding=\"utf-8\") as f:\n yaml.dump(config, f, default_flow_style=False, allow_unicode=True)\n\n print(\n f\"{Colors.OKGREEN}✅ Document segmentation configuration updated{Colors.ENDC}\"\n )\n\n except Exception as e:\n print(\n f\"{Colors.WARNING}⚠️ Failed to update segmentation config: {str(e)}{Colors.ENDC}\"\n )\n\n def _init_tkinter(self):\n \"\"\"Initialize tkinter availability check\"\"\"\n # Check tkinter availability for file dialogs\n self.tkinter_available = True\n try:\n import tkinter as tk\n\n # Test if tkinter can create a window\n test_root = tk.Tk()\n test_root.withdraw()\n test_root.destroy()\n except Exception:\n self.tkinter_available = False\n\n def clear_screen(self):\n \"\"\"Clear terminal screen\"\"\"\n os.system(\"cls\" if os.name == \"nt\" else \"clear\")\n\n def print_logo(self):\n \"\"\"Print enhanced ASCII logo for DeepCode CLI\"\"\"\n logo = f\"\"\"\n{Colors.CYAN}╔═══════════════════════════════════════════════════════════════════════════════╗\n║ ║\n║ {Colors.BOLD}{Colors.MAGENTA}██████╗ ███████╗███████╗██████╗ ██████╗ ██████╗ ██████╗ ███████╗{Colors.CYAN} ║\n║ {Colors.BOLD}{Colors.PURPLE}██╔══██╗██╔════╝██╔════╝██╔══██╗██╔════╝██╔═══██╗██╔══██╗██╔════╝{Colors.CYAN} ║\n║ {Colors.BOLD}{Colors.BLUE}██║ ██║█████╗ █████╗ ██████╔╝██║ ██║ ██║██║ ██║█████╗ {Colors.CYAN} ║\n║ {Colors.BOLD}{Colors.OKBLUE}██║ ██║██╔══╝ ██╔══╝ ██╔═══╝ ██║ ██║ ██║██║ ██║██╔══╝ {Colors.CYAN} ║\n║ {Colors.BOLD}{Colors.OKCYAN}██████╔╝███████╗███████╗██║ ╚██████╗╚██████╔╝██████╔╝███████╗{Colors.CYAN} ║\n║ {Colors.BOLD}{Colors.GREEN}╚═════╝ ╚══════╝╚══════╝╚═╝ ╚═════╝ ╚═════╝ ╚═════╝ ╚══════╝{Colors.CYAN} ║\n║ ║\n║ {Colors.BOLD}{Colors.GREEN}🧬 OPEN-SOURCE CODE AGENT • DATA INTELLIGENCE LAB @ HKU 🚀 {Colors.CYAN}║\n║ {Colors.BOLD}{Colors.GREEN}⚡ REVOLUTIONIZING RESEARCH REPRODUCIBILITY ⚡ {Colors.CYAN}║\n║ ║\n╚═══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\"\n print(logo)\n\n def print_welcome_banner(self):\n \"\"\"Print enhanced welcome banner\"\"\"\n banner = f\"\"\"\n{Colors.BOLD}{Colors.CYAN}╔═══════════════════════════════════════════════════════════════════════════════╗\n║ WELCOME TO DEEPCODE CLI ║\n╠═══════════════════════════════════════════════════════════════════════════════╣\n║ {Colors.YELLOW}Open-Source Code Agent | Data Intelligence Lab @ HKU | MIT License {Colors.CYAN}║\n║ {Colors.GREEN}Status: Ready | Engine: Multi-Agent Architecture Initialized {Colors.CYAN}║\n║ {Colors.PURPLE}Mission: Revolutionizing Research Reproducibility {Colors.CYAN}║\n║ ║\n║ {Colors.BOLD}{Colors.OKCYAN}💎 CORE CAPABILITIES:{Colors.ENDC} {Colors.CYAN}║\n║ {Colors.BOLD}{Colors.OKCYAN}▶ Automated Paper-to-Code Reproduction {Colors.CYAN}║\n║ {Colors.BOLD}{Colors.OKCYAN}▶ Collaborative Multi-Agent Architecture {Colors.CYAN}║\n║ {Colors.BOLD}{Colors.OKCYAN}▶ Intelligent Code Implementation & Validation {Colors.CYAN}║\n║ {Colors.BOLD}{Colors.OKCYAN}▶ Future Vision: One Sentence → Complete Codebase {Colors.CYAN}║\n╚═══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\"\n print(banner)\n\n def print_separator(self, char=\"═\", length=79, color=Colors.CYAN):\n \"\"\"Print a styled separator line\"\"\"\n print(f\"{color}{char * length}{Colors.ENDC}\")\n\n def print_status(self, message: str, status_type: str = \"info\"):\n \"\"\"Print status message with appropriate styling\"\"\"\n status_styles = {\n \"success\": f\"{Colors.OKGREEN}✅\",\n \"error\": f\"{Colors.FAIL}❌\",\n \"warning\": f\"{Colors.WARNING}⚠️ \",\n \"info\": f\"{Colors.OKBLUE}ℹ️ \",\n \"processing\": f\"{Colors.YELLOW}⏳\",\n \"upload\": f\"{Colors.PURPLE}📁\",\n \"download\": f\"{Colors.CYAN}📥\",\n \"analysis\": f\"{Colors.MAGENTA}🔍\",\n \"implementation\": f\"{Colors.GREEN}⚙️ \",\n \"complete\": f\"{Colors.OKGREEN}🎉\",\n }\n\n icon = status_styles.get(status_type, status_styles[\"info\"])\n timestamp = time.strftime(\"%H:%M:%S\")\n print(\n f\"[{Colors.BOLD}{timestamp}{Colors.ENDC}] {icon} {Colors.BOLD}{message}{Colors.ENDC}\"\n )\n\n def create_menu(self):\n \"\"\"Create enhanced interactive menu\"\"\"\n # Display current configuration\n pipeline_mode = \"🧠 COMPREHENSIVE\" if self.enable_indexing else \"⚡ OPTIMIZED\"\n index_status = \"✅ Enabled\" if self.enable_indexing else \"🔶 Disabled\"\n segmentation_mode = (\n \"📄 SMART\" if self.segmentation_enabled else \"📋 TRADITIONAL\"\n )\n\n menu = f\"\"\"\n{Colors.BOLD}{Colors.CYAN}╔═══════════════════════════════════════════════════════════════════════════════╗\n║ MAIN MENU ║\n╠═══════════════════════════════════════════════════════════════════════════════╣\n║ {Colors.OKGREEN}🌐 [U] Process URL {Colors.CYAN}│ {Colors.PURPLE}📁 [F] Upload File {Colors.CYAN}│ {Colors.MAGENTA}💬 [T] Chat Input{Colors.CYAN} ║\n║ {Colors.BLUE}🧠 [R] Req. Analysis {Colors.CYAN}│ {Colors.OKCYAN}⚙️ [C] Configure {Colors.CYAN}│ {Colors.YELLOW}📊 [H] History{Colors.CYAN} ║\n║ {Colors.FAIL}❌ [Q] Quit{Colors.CYAN} ║\n║ ║\n║ {Colors.BOLD}🤖 Current Pipeline Mode: {pipeline_mode}{Colors.CYAN} ║\n║ {Colors.BOLD}🗂️ Codebase Indexing: {index_status}{Colors.CYAN} ║\n║ {Colors.BOLD}📄 Document Processing: {segmentation_mode}{Colors.CYAN} ║\n║ ║\n║ {Colors.YELLOW}📝 URL Processing:{Colors.CYAN} ║\n║ {Colors.YELLOW} ▶ Enter research paper URL (arXiv, IEEE, ACM, etc.) {Colors.CYAN}║\n║ {Colors.YELLOW} ▶ Supports direct PDF links and academic paper pages {Colors.CYAN}║\n║ ║\n║ {Colors.PURPLE}📁 File Processing:{Colors.CYAN} ║\n║ {Colors.PURPLE} ▶ Upload PDF, DOCX, PPTX, HTML, or TXT files {Colors.CYAN}║\n║ {Colors.PURPLE} ▶ Intelligent file format detection and processing {Colors.CYAN}║\n║ ║\n║ {Colors.MAGENTA}💬 Chat Input:{Colors.CYAN} ║\n║ {Colors.MAGENTA} ▶ Describe your coding requirements in natural language {Colors.CYAN}║\n║ {Colors.MAGENTA} ▶ AI generates implementation plan and code automatically {Colors.CYAN}║\n║ ║\n║ {Colors.BLUE}🧠 Requirement Analysis (NEW):{Colors.CYAN} ║\n║ {Colors.BLUE} ▶ Get AI-guided questions to refine your requirements {Colors.CYAN}║\n║ {Colors.BLUE} ▶ Generate detailed requirement documents from your answers {Colors.CYAN}║\n║ ║\n║ {Colors.OKCYAN}🔄 Processing Pipeline:{Colors.CYAN} ║\n║ {Colors.OKCYAN} ▶ Intelligent agent orchestration → Code synthesis {Colors.CYAN}║\n║ {Colors.OKCYAN} ▶ Multi-agent coordination with progress tracking {Colors.CYAN}║\n╚═══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\"\n print(menu)\n\n def get_user_input(self):\n \"\"\"Get user input with styled prompt\"\"\"\n print(f\"\\n{Colors.BOLD}{Colors.OKCYAN}➤ Your choice: {Colors.ENDC}\", end=\"\")\n return input().strip().lower()\n\n def upload_file_gui(self) -> Optional[str]:\n \"\"\"Enhanced file upload interface with better error handling\"\"\"\n if not self.tkinter_available:\n self.print_status(\n \"GUI file dialog not available - using manual input\", \"warning\"\n )\n return self._get_manual_file_path()\n\n def select_file():\n try:\n import tkinter as tk\n from tkinter import filedialog\n\n root = tk.Tk()\n root.withdraw()\n root.attributes(\"-topmost\", True)\n\n file_types = [\n (\"Research Papers\", \"*.pdf;*.docx;*.doc\"),\n (\"PDF Files\", \"*.pdf\"),\n (\"Word Documents\", \"*.docx;*.doc\"),\n (\"PowerPoint Files\", \"*.pptx;*.ppt\"),\n (\"HTML Files\", \"*.html;*.htm\"),\n (\"Text Files\", \"*.txt;*.md\"),\n (\"All Files\", \"*.*\"),\n ]\n\n if platform.system() == \"Darwin\":\n file_types = [\n (\"Research Papers\", \".pdf .docx .doc\"),\n (\"PDF Files\", \".pdf\"),\n (\"Word Documents\", \".docx .doc\"),\n (\"PowerPoint Files\", \".pptx .ppt\"),\n (\"HTML Files\", \".html .htm\"),\n (\"Text Files\", \".txt .md\"),\n (\"All Files\", \".*\"),\n ]\n\n file_path = filedialog.askopenfilename(\n title=\"Select Research File - DeepCode CLI\",\n filetypes=file_types,\n initialdir=os.getcwd(),\n )\n\n root.destroy()\n return file_path\n\n except Exception as e:\n self.print_status(f\"File dialog error: {str(e)}\", \"error\")\n return self._get_manual_file_path()\n\n self.print_status(\"Opening file browser dialog...\", \"upload\")\n file_path = select_file()\n\n if file_path:\n self.print_status(\n f\"File selected: {os.path.basename(file_path)}\", \"success\"\n )\n return file_path\n else:\n self.print_status(\"No file selected\", \"warning\")\n return None\n\n def _get_manual_file_path(self) -> Optional[str]:\n \"\"\"Get file path through manual input with validation\"\"\"\n self.print_separator(\"─\", 79, Colors.YELLOW)\n print(f\"{Colors.BOLD}{Colors.YELLOW}📁 Manual File Path Input{Colors.ENDC}\")\n print(\n f\"{Colors.CYAN}Please enter the full path to your research paper file:{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}Supported formats: PDF, DOCX, PPTX, HTML, TXT, MD{Colors.ENDC}\"\n )\n self.print_separator(\"─\", 79, Colors.YELLOW)\n\n while True:\n print(f\"\\n{Colors.BOLD}{Colors.OKCYAN}📂 File path: {Colors.ENDC}\", end=\"\")\n file_path = input().strip()\n\n if not file_path:\n self.print_status(\n \"Empty path entered. Please try again or press Ctrl+C to cancel.\",\n \"warning\",\n )\n continue\n\n file_path = os.path.expanduser(file_path)\n file_path = os.path.abspath(file_path)\n\n if not os.path.exists(file_path):\n self.print_status(f\"File not found: {file_path}\", \"error\")\n retry = (\n input(f\"{Colors.YELLOW}Try again? (y/n): {Colors.ENDC}\")\n .strip()\n .lower()\n )\n if retry != \"y\":\n return None\n continue\n\n if not os.path.isfile(file_path):\n self.print_status(f\"Path is not a file: {file_path}\", \"error\")\n continue\n\n supported_extensions = {\n \".pdf\",\n \".docx\",\n \".doc\",\n \".pptx\",\n \".ppt\",\n \".html\",\n \".htm\",\n \".txt\",\n \".md\",\n }\n file_ext = os.path.splitext(file_path)[1].lower()\n\n if file_ext not in supported_extensions:\n self.print_status(f\"Unsupported file format: {file_ext}\", \"warning\")\n proceed = (\n input(f\"{Colors.YELLOW}Process anyway? (y/n): {Colors.ENDC}\")\n .strip()\n .lower()\n )\n if proceed != \"y\":\n continue\n\n self.print_status(\n f\"File validated: {os.path.basename(file_path)}\", \"success\"\n )\n return file_path\n\n def get_url_input(self) -> str:\n \"\"\"Enhanced URL input with validation\"\"\"\n self.print_separator(\"─\", 79, Colors.GREEN)\n print(f\"{Colors.BOLD}{Colors.GREEN}🌐 URL Input Interface{Colors.ENDC}\")\n print(\n f\"{Colors.CYAN}Enter a research paper URL from supported platforms:{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}• arXiv (arxiv.org) • IEEE Xplore (ieeexplore.ieee.org){Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}• ACM Digital Library • SpringerLink • Nature • Science{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}• Direct PDF links • Academic publisher websites{Colors.ENDC}\"\n )\n self.print_separator(\"─\", 79, Colors.GREEN)\n\n while True:\n print(f\"\\n{Colors.BOLD}{Colors.OKCYAN}🔗 URL: {Colors.ENDC}\", end=\"\")\n url = input().strip()\n\n if not url:\n self.print_status(\n \"Empty URL entered. Please try again or press Ctrl+C to cancel.\",\n \"warning\",\n )\n continue\n\n if not url.startswith((\"http://\", \"https://\")):\n self.print_status(\"URL must start with http:// or https://\", \"error\")\n retry = (\n input(f\"{Colors.YELLOW}Try again? (y/n): {Colors.ENDC}\")\n .strip()\n .lower()\n )\n if retry != \"y\":\n return \"\"\n continue\n\n academic_domains = [\n \"arxiv.org\",\n \"ieeexplore.ieee.org\",\n \"dl.acm.org\",\n \"link.springer.com\",\n \"nature.com\",\n \"science.org\",\n \"scholar.google.com\",\n \"researchgate.net\",\n \"semanticscholar.org\",\n ]\n\n is_academic = any(domain in url.lower() for domain in academic_domains)\n if not is_academic and not url.lower().endswith(\".pdf\"):\n self.print_status(\n \"URL doesn't appear to be from a known academic platform\", \"warning\"\n )\n proceed = (\n input(f\"{Colors.YELLOW}Process anyway? (y/n): {Colors.ENDC}\")\n .strip()\n .lower()\n )\n if proceed != \"y\":\n continue\n\n self.print_status(f\"URL validated: {url}\", \"success\")\n return url\n\n def get_chat_input(self) -> str:\n \"\"\"Enhanced chat input interface for coding requirements\"\"\"\n self.print_separator(\"─\", 79, Colors.PURPLE)\n print(f\"{Colors.BOLD}{Colors.PURPLE}💬 Chat Input Interface{Colors.ENDC}\")\n print(\n f\"{Colors.CYAN}Describe your coding requirements in natural language.{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}Our AI will analyze your needs and generate a comprehensive implementation plan.{Colors.ENDC}\"\n )\n self.print_separator(\"─\", 79, Colors.PURPLE)\n\n # Display examples to help users\n print(f\"\\n{Colors.BOLD}{Colors.YELLOW}💡 Examples:{Colors.ENDC}\")\n print(f\"{Colors.CYAN}Academic Research:{Colors.ENDC}\")\n print(\n \" • 'I need to implement a reinforcement learning algorithm for robotic control'\"\n )\n print(\n \" • 'Create a neural network for image classification with attention mechanisms'\"\n )\n print(f\"{Colors.CYAN}Engineering Projects:{Colors.ENDC}\")\n print(\n \" • 'Develop a web application for project management with user authentication'\"\n )\n print(\" • 'Create a data visualization dashboard for sales analytics'\")\n print(f\"{Colors.CYAN}Mixed Projects:{Colors.ENDC}\")\n print(\n \" • 'Implement a machine learning model with a web interface for real-time predictions'\"\n )\n\n self.print_separator(\"─\", 79, Colors.PURPLE)\n\n print(\n f\"\\n{Colors.BOLD}{Colors.OKCYAN}✏️ Enter your coding requirements below:{Colors.ENDC}\"\n )\n print(\n f\"{Colors.YELLOW}(Type your description, press Enter twice when finished, or Ctrl+C to cancel){Colors.ENDC}\"\n )\n\n lines = []\n empty_line_count = 0\n\n while True:\n try:\n if len(lines) == 0:\n print(f\"{Colors.BOLD}> {Colors.ENDC}\", end=\"\")\n else:\n print(f\"{Colors.BOLD} {Colors.ENDC}\", end=\"\")\n\n line = input()\n\n if line.strip() == \"\":\n empty_line_count += 1\n if empty_line_count >= 2:\n # Two consecutive empty lines means user finished input\n break\n lines.append(\"\") # Keep empty line for formatting\n else:\n empty_line_count = 0\n lines.append(line)\n\n except KeyboardInterrupt:\n print(f\"\\n{Colors.WARNING}Input cancelled by user{Colors.ENDC}\")\n return \"\"\n\n # Join all lines and clean up\n user_input = \"\\n\".join(lines).strip()\n\n if not user_input:\n self.print_status(\"No input provided\", \"warning\")\n return \"\"\n\n if len(user_input) < 20:\n self.print_status(\n \"Input too short. Please provide more detailed requirements (at least 20 characters)\",\n \"warning\",\n )\n retry = (\n input(f\"{Colors.YELLOW}Try again? (y/n): {Colors.ENDC}\").strip().lower()\n )\n if retry == \"y\":\n return self.get_chat_input() # Recursive call for retry\n return \"\"\n\n # Display input summary\n word_count = len(user_input.split())\n char_count = len(user_input)\n\n print(f\"\\n{Colors.BOLD}{Colors.GREEN}📋 Input Summary:{Colors.ENDC}\")\n print(f\" • {Colors.CYAN}Word count: {word_count}{Colors.ENDC}\")\n print(f\" • {Colors.CYAN}Character count: {char_count}{Colors.ENDC}\")\n\n # Show preview\n preview = user_input[:200] + \"...\" if len(user_input) > 200 else user_input\n print(f\"\\n{Colors.BOLD}{Colors.CYAN}📄 Preview:{Colors.ENDC}\")\n print(f\"{Colors.YELLOW}{preview}{Colors.ENDC}\")\n\n # Confirm with user\n confirm = (\n input(\n f\"\\n{Colors.BOLD}{Colors.OKCYAN}Proceed with this input? (y/n): {Colors.ENDC}\"\n )\n .strip()\n .lower()\n )\n if confirm != \"y\":\n retry = (\n input(f\"{Colors.YELLOW}Edit input? (y/n): {Colors.ENDC}\")\n .strip()\n .lower()\n )\n if retry == \"y\":\n return self.get_chat_input() # Recursive call for retry\n return \"\"\n\n self.print_status(\n f\"Chat input captured: {word_count} words, {char_count} characters\",\n \"success\",\n )\n return user_input\n\n def show_progress_bar(self, message: str, duration: float = 2.0):\n \"\"\"Show animated progress bar\"\"\"\n print(f\"\\n{Colors.BOLD}{Colors.CYAN}{message}{Colors.ENDC}\")\n\n bar_length = 50\n for i in range(bar_length + 1):\n percent = (i / bar_length) * 100\n filled = \"█\" * i\n empty = \"░\" * (bar_length - i)\n\n print(\n f\"\\r{Colors.OKGREEN}[{filled}{empty}] {percent:3.0f}%{Colors.ENDC}\",\n end=\"\",\n flush=True,\n )\n time.sleep(duration / bar_length)\n\n print(f\"\\n{Colors.OKGREEN}✓ {message} completed{Colors.ENDC}\")\n\n def show_spinner(self, message: str, duration: float = 1.0):\n \"\"\"Show spinner animation\"\"\"\n spinner_chars = \"⠋⠙⠹⠸⠼⠴⠦⠧⠇⠏\"\n end_time = time.time() + duration\n\n print(\n f\"{Colors.BOLD}{Colors.CYAN}{message}... {Colors.ENDC}\", end=\"\", flush=True\n )\n\n i = 0\n while time.time() < end_time:\n print(\n f\"\\r{Colors.BOLD}{Colors.CYAN}{message}... {Colors.YELLOW}{spinner_chars[i % len(spinner_chars)]}{Colors.ENDC}\",\n end=\"\",\n flush=True,\n )\n time.sleep(0.1)\n i += 1\n\n print(\n f\"\\r{Colors.BOLD}{Colors.CYAN}{message}... {Colors.OKGREEN}✓{Colors.ENDC}\"\n )\n\n def display_processing_stages(\n self,\n current_stage: int = 0,\n enable_indexing: bool = True,\n chat_mode: bool = False,\n ):\n \"\"\"Display processing pipeline stages with current progress\"\"\"\n if chat_mode:\n # Chat mode - simplified workflow for user requirements\n stages = [\n (\"🚀\", \"Initialize\", \"Setting up chat engine\"),\n (\"💬\", \"Planning\", \"Analyzing requirements\"),\n (\"🏗️\", \"Setup\", \"Creating workspace\"),\n (\"📝\", \"Save Plan\", \"Saving implementation plan\"),\n (\"⚙️\", \"Implement\", \"Generating code\"),\n ]\n pipeline_mode = \"CHAT PLANNING\"\n elif enable_indexing:\n # Full pipeline with all stages\n stages = [\n (\"🚀\", \"Initialize\", \"Setting up AI engine\"),\n (\"📊\", \"Analyze\", \"Analyzing research content\"),\n (\"📥\", \"Download\", \"Processing document\"),\n (\"📋\", \"Plan\", \"Generating code architecture\"),\n (\"🔍\", \"References\", \"Analyzing references\"),\n (\"📦\", \"Repos\", \"Downloading repositories\"),\n (\"🗂️\", \"Index\", \"Building code index\"),\n (\"⚙️\", \"Implement\", \"Implementing code\"),\n ]\n pipeline_mode = \"COMPREHENSIVE\"\n else:\n # Fast mode - skip indexing related stages\n stages = [\n (\"🚀\", \"Initialize\", \"Setting up AI engine\"),\n (\"📊\", \"Analyze\", \"Analyzing research content\"),\n (\"📥\", \"Download\", \"Processing document\"),\n (\"📋\", \"Plan\", \"Generating code architecture\"),\n (\"⚙️\", \"Implement\", \"Implementing code\"),\n ]\n pipeline_mode = \"OPTIMIZED\"\n\n print(\n f\"\\n{Colors.BOLD}{Colors.CYAN}📋 {pipeline_mode} PIPELINE STATUS{Colors.ENDC}\"\n )\n self.print_separator(\"─\", 79, Colors.CYAN)\n\n for i, (icon, name, desc) in enumerate(stages):\n if i < current_stage:\n status = f\"{Colors.OKGREEN}✓ COMPLETED{Colors.ENDC}\"\n elif i == current_stage:\n status = f\"{Colors.YELLOW}⏳ IN PROGRESS{Colors.ENDC}\"\n else:\n status = f\"{Colors.CYAN}⏸️ PENDING{Colors.ENDC}\"\n\n print(\n f\"{icon} {Colors.BOLD}{name:<12}{Colors.ENDC} │ {desc:<25} │ {status}\"\n )\n\n self.print_separator(\"─\", 79, Colors.CYAN)\n\n def print_results_header(self):\n \"\"\"Print results section header\"\"\"\n header = f\"\"\"\n{Colors.BOLD}{Colors.OKGREEN}╔═══════════════════════════════════════════════════════════════════════════════╗\n║ PROCESSING RESULTS ║\n╚═══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\"\n print(header)\n\n def print_error_box(self, title: str, error_msg: str):\n \"\"\"Print formatted error box\"\"\"\n print(\n f\"\\n{Colors.FAIL}╔══════════════════════════════════════════════════════════════╗\"\n )\n print(f\"║ {Colors.BOLD}ERROR: {title:<50}{Colors.FAIL} ║\")\n print(\"╠══════════════════════════════════════════════════════════════╣\")\n\n words = error_msg.split()\n lines = []\n current_line = \"\"\n\n for word in words:\n if len(current_line + word) <= 54:\n current_line += word + \" \"\n else:\n lines.append(current_line.strip())\n current_line = word + \" \"\n if current_line:\n lines.append(current_line.strip())\n\n for line in lines:\n print(f\"║ {line:<56} ║\")\n\n print(\n f\"╚══════════════════════════════════════════════════════════════╝{Colors.ENDC}\"\n )\n\n def cleanup_cache(self):\n \"\"\"清理Python缓存文件 / Clean up Python cache files\"\"\"\n try:\n self.print_status(\"Cleaning up cache files...\", \"info\")\n # 清理__pycache__目录\n os.system('find . -type d -name \"__pycache__\" -exec rm -r {} + 2>/dev/null')\n # 清理.pyc文件\n os.system('find . -name \"*.pyc\" -delete 2>/dev/null')\n self.print_status(\"Cache cleanup completed\", \"success\")\n except Exception as e:\n self.print_status(f\"Cache cleanup failed: {e}\", \"warning\")\n\n def print_goodbye(self):\n \"\"\"Print goodbye message\"\"\"\n # 清理缓存文件\n self.cleanup_cache()\n\n goodbye = f\"\"\"\n{Colors.BOLD}{Colors.CYAN}╔═══════════════════════════════════════════════════════════════════════════════╗\n║ GOODBYE ║\n╠═══════════════════════════════════════════════════════════════════════════════╣\n║ {Colors.OKGREEN}🎉 Thank you for using DeepCode CLI! {Colors.CYAN}║\n║ ║\n║ {Colors.YELLOW}🧬 Join our community in revolutionizing research reproducibility {Colors.CYAN}║\n║ {Colors.PURPLE}⚡ Together, we're building the future of automated code generation {Colors.CYAN}║\n║ ║\n║ {Colors.OKCYAN}💡 Questions? Contribute to our open-source mission at GitHub {Colors.CYAN}║\n║ {Colors.GREEN}🧹 Cache files cleaned up for optimal performance {Colors.CYAN}║\n║ ║\n╚═══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\"\n print(goodbye)\n\n def get_requirement_analysis_input(self) -> str:\n \"\"\"Enhanced requirement analysis input interface (NEW: matching UI version)\"\"\"\n self.print_separator(\"─\", 79, Colors.BLUE)\n print(\n f\"{Colors.BOLD}{Colors.BLUE}🧠 Requirement Analysis Interface{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}Describe your project idea or requirements briefly.{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}Our AI will generate guiding questions to help you refine your vision.{Colors.ENDC}\"\n )\n self.print_separator(\"─\", 79, Colors.BLUE)\n\n # Display examples\n print(f\"\\n{Colors.BOLD}{Colors.YELLOW}💡 Examples:{Colors.ENDC}\")\n print(\n f\"{Colors.CYAN} • 'I want to build a machine learning system for image recognition'{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN} • 'Create a web app for project management with real-time collaboration'{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN} • 'Develop a data analysis pipeline for financial forecasting'{Colors.ENDC}\"\n )\n\n self.print_separator(\"─\", 79, Colors.BLUE)\n\n print(\n f\"\\n{Colors.BOLD}{Colors.OKCYAN}✏️ Enter your initial requirements below:{Colors.ENDC}\"\n )\n print(\n f\"{Colors.YELLOW}(Type your description, press Enter twice when finished, or Ctrl+C to cancel){Colors.ENDC}\"\n )\n\n lines = []\n empty_line_count = 0\n\n while True:\n try:\n if len(lines) == 0:\n print(f\"{Colors.BOLD}> {Colors.ENDC}\", end=\"\")\n else:\n print(f\"{Colors.BOLD} {Colors.ENDC}\", end=\"\")\n\n line = input()\n\n if line.strip() == \"\":\n empty_line_count += 1\n if empty_line_count >= 2:\n break\n lines.append(\"\")\n else:\n empty_line_count = 0\n lines.append(line)\n\n except KeyboardInterrupt:\n print(f\"\\n{Colors.WARNING}Input cancelled by user{Colors.ENDC}\")\n return \"\"\n\n user_input = \"\\n\".join(lines).strip()\n\n if not user_input:\n self.print_status(\"No input provided\", \"warning\")\n return \"\"\n\n if len(user_input) < 20:\n self.print_status(\n \"Input too short. Please provide more details (at least 20 characters)\",\n \"warning\",\n )\n retry = (\n input(f\"{Colors.YELLOW}Try again? (y/n): {Colors.ENDC}\").strip().lower()\n )\n if retry == \"y\":\n return self.get_requirement_analysis_input()\n return \"\"\n\n # Display input summary\n word_count = len(user_input.split())\n char_count = len(user_input)\n\n print(f\"\\n{Colors.BOLD}{Colors.GREEN}📋 Input Summary:{Colors.ENDC}\")\n print(f\" • {Colors.CYAN}Word count: {word_count}{Colors.ENDC}\")\n print(f\" • {Colors.CYAN}Character count: {char_count}{Colors.ENDC}\")\n\n # Show preview\n preview = user_input[:200] + \"...\" if len(user_input) > 200 else user_input\n print(f\"\\n{Colors.BOLD}{Colors.CYAN}📄 Preview:{Colors.ENDC}\")\n print(f\"{Colors.YELLOW}{preview}{Colors.ENDC}\")\n\n # Confirm\n confirm = (\n input(\n f\"\\n{Colors.BOLD}{Colors.OKCYAN}Proceed with this input? (y/n): {Colors.ENDC}\"\n )\n .strip()\n .lower()\n )\n if confirm != \"y\":\n retry = (\n input(f\"{Colors.YELLOW}Edit input? (y/n): {Colors.ENDC}\")\n .strip()\n .lower()\n )\n if retry == \"y\":\n return self.get_requirement_analysis_input()\n return \"\"\n\n self.print_status(\n f\"Requirement input captured: {word_count} words, {char_count} characters\",\n \"success\",\n )\n return user_input\n\n def display_guiding_questions(self, questions_json: str):\n \"\"\"Display AI-generated guiding questions (NEW: matching UI version)\"\"\"\n import json\n\n try:\n questions = json.loads(questions_json)\n\n self.print_separator(\"═\", 79, Colors.GREEN)\n print(\n f\"\\n{Colors.BOLD}{Colors.GREEN}🤖 AI-Generated Guiding Questions{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}Please answer these questions to help refine your requirements:{Colors.ENDC}\\n\"\n )\n self.print_separator(\"─\", 79, Colors.GREEN)\n\n for i, q in enumerate(questions, 1):\n print(\n f\"\\n{Colors.BOLD}{Colors.YELLOW}Question {i}:{Colors.ENDC} {Colors.CYAN}{q}{Colors.ENDC}\"\n )\n\n self.print_separator(\"═\", 79, Colors.GREEN)\n\n except json.JSONDecodeError:\n self.print_status(\"Failed to parse questions\", \"error\")\n print(questions_json)\n\n def get_question_answers(self, questions_json: str) -> dict:\n \"\"\"Get user answers to guiding questions (NEW: matching UI version)\"\"\"\n import json\n\n try:\n questions = json.loads(questions_json)\n answers = {}\n\n print(\n f\"\\n{Colors.BOLD}{Colors.BLUE}📝 Answer the following questions:{Colors.ENDC}\"\n )\n print(\n f\"{Colors.CYAN}(Type your answer and press Enter for each question){Colors.ENDC}\\n\"\n )\n\n for i, question in enumerate(questions, 1):\n print(\n f\"\\n{Colors.BOLD}{Colors.YELLOW}Q{i}:{Colors.ENDC} {Colors.CYAN}{question}{Colors.ENDC}\"\n )\n print(f\"{Colors.BOLD}{Colors.OKCYAN}Your answer:{Colors.ENDC} \", end=\"\")\n\n answer = input().strip()\n answers[f\"question_{i}\"] = answer\n\n if answer:\n self.print_status(f\"Answer {i} recorded\", \"success\")\n else:\n self.print_status(f\"Answer {i} left blank\", \"warning\")\n\n return answers\n\n except json.JSONDecodeError:\n self.print_status(\"Failed to parse questions\", \"error\")\n return {}\n\n def display_requirement_summary(self, summary: str):\n \"\"\"Display generated requirement document (NEW: matching UI version)\"\"\"\n self.print_separator(\"═\", 79, Colors.GREEN)\n print(\n f\"\\n{Colors.BOLD}{Colors.GREEN}📄 Generated Requirement Document{Colors.ENDC}\\n\"\n )\n self.print_separator(\"─\", 79, Colors.GREEN)\n\n print(f\"{Colors.CYAN}{summary}{Colors.ENDC}\")\n\n self.print_separator(\"═\", 79, Colors.GREEN)\n\n # Ask if user wants to proceed with implementation\n proceed = (\n input(\n f\"\\n{Colors.BOLD}{Colors.YELLOW}Would you like to proceed with code implementation based on these requirements? (y/n):{Colors.ENDC} \"\n )\n .strip()\n .lower()\n )\n\n return proceed == \"y\"\n\n def ask_continue(self) -> bool:\n \"\"\"Ask if user wants to continue with another paper\"\"\"\n self.print_separator(\"─\", 79, Colors.YELLOW)\n print(f\"\\n{Colors.BOLD}{Colors.YELLOW}🔄 Process another paper?{Colors.ENDC}\")\n choice = input(f\"{Colors.OKCYAN}Continue? (y/n): {Colors.ENDC}\").strip().lower()\n return choice in [\"y\", \"yes\", \"1\", \"true\"]\n\n def add_to_history(self, input_source: str, result: dict):\n \"\"\"Add processing result to history\"\"\"\n entry = {\n \"timestamp\": time.strftime(\"%Y-%m-%d %H:%M:%S\"),\n \"input_source\": input_source,\n \"status\": result.get(\"status\", \"unknown\"),\n \"result\": result,\n }\n self.processing_history.append(entry)\n\n def show_history(self):\n \"\"\"Display processing history\"\"\"\n if not self.processing_history:\n self.print_status(\"No processing history available\", \"info\")\n return\n\n print(f\"\\n{Colors.BOLD}{Colors.CYAN}📚 PROCESSING HISTORY{Colors.ENDC}\")\n self.print_separator(\"─\", 79, Colors.CYAN)\n\n for i, entry in enumerate(self.processing_history, 1):\n status_icon = \"✅\" if entry[\"status\"] == \"success\" else \"❌\"\n source = entry[\"input_source\"]\n if len(source) > 50:\n source = source[:47] + \"...\"\n\n print(f\"{i}. {status_icon} {entry['timestamp']} | {source}\")\n\n self.print_separator(\"─\", 79, Colors.CYAN)\n\n def show_configuration_menu(self):\n \"\"\"Show configuration options menu\"\"\"\n self.clear_screen()\n\n # Get segmentation config status\n segmentation_enabled = getattr(self, \"segmentation_enabled\", True)\n segmentation_threshold = getattr(self, \"segmentation_threshold\", 50000)\n\n print(f\"\"\"\n{Colors.BOLD}{Colors.CYAN}╔═══════════════════════════════════════════════════════════════════════════════╗\n║ CONFIGURATION MENU ║\n╠═══════════════════════════════════════════════════════════════════════════════╣\n║ ║\n║ {Colors.BOLD}🤖 Agent Orchestration Engine Configuration{Colors.CYAN} ║\n║ ║\n║ {Colors.OKCYAN}[1] Pipeline Mode:{Colors.CYAN} ║\n║ {Colors.BOLD}🧠 Comprehensive Mode{Colors.CYAN} - Full intelligence analysis (Default) ║\n║ ✓ Research Analysis + Resource Processing ║\n║ ✓ Reference Intelligence Discovery ║\n║ ✓ Automated Repository Acquisition ║\n║ ✓ Codebase Intelligence Orchestration ║\n║ ✓ Intelligent Code Implementation Synthesis ║\n║ ║\n║ {Colors.BOLD}⚡ Optimized Mode{Colors.CYAN} - Fast processing (Skip indexing) ║\n║ ✓ Research Analysis + Resource Processing ║\n║ ✓ Code Architecture Synthesis ║\n║ ✓ Intelligent Code Implementation Synthesis ║\n║ ✗ Reference Intelligence Discovery (Skipped) ║\n║ ✗ Repository Acquisition (Skipped) ║\n║ ✗ Codebase Intelligence Orchestration (Skipped) ║\n║ ║\n║ {Colors.OKCYAN}[2] Document Processing:{Colors.CYAN} ║\n║ {Colors.BOLD}📄 Smart Segmentation{Colors.CYAN} - Intelligent document analysis (Default) ║\n║ ✓ Semantic boundary detection ║\n║ ✓ Algorithm integrity preservation ║\n║ ✓ Formula chain recognition ║\n║ ✓ Adaptive character limits ║\n║ ║\n║ {Colors.BOLD}📋 Traditional Processing{Colors.CYAN} - Full document reading ║\n║ ✓ Complete document analysis ║\n║ ✗ Smart segmentation (Disabled) ║\n║ ║\n║ {Colors.YELLOW}Current Settings:{Colors.CYAN} ║\n║ Pipeline: {'🧠 Comprehensive Mode' if self.enable_indexing else '⚡ Optimized Mode'} ║\n║ Document: {'📄 Smart Segmentation' if segmentation_enabled else '📋 Traditional Processing'} ║\n║ Threshold: {segmentation_threshold} characters ║\n║ ║\n║ {Colors.OKGREEN}[T] Toggle Pipeline {Colors.BLUE}[S] Toggle Segmentation {Colors.FAIL}[B] Back{Colors.CYAN} ║\n╚═══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\")\n\n while True:\n print(\n f\"\\n{Colors.BOLD}{Colors.OKCYAN}➤ Configuration choice: {Colors.ENDC}\",\n end=\"\",\n )\n choice = input().strip().lower()\n\n if choice in [\"t\", \"toggle\"]:\n self.enable_indexing = not self.enable_indexing\n mode = \"🧠 Comprehensive\" if self.enable_indexing else \"⚡ Optimized\"\n self.print_status(f\"Pipeline mode switched to: {mode}\", \"success\")\n time.sleep(1)\n self.show_configuration_menu()\n return\n\n elif choice in [\"s\", \"segmentation\"]:\n current_state = getattr(self, \"segmentation_enabled\", True)\n self.segmentation_enabled = not current_state\n # Save the configuration to file\n self._save_segmentation_config()\n seg_mode = (\n \"📄 Smart Segmentation\"\n if self.segmentation_enabled\n else \"📋 Traditional Processing\"\n )\n self.print_status(\n f\"Document processing switched to: {seg_mode}\", \"success\"\n )\n time.sleep(1)\n self.show_configuration_menu()\n return\n\n elif choice in [\"b\", \"back\"]:\n return\n\n else:\n self.print_status(\n \"Invalid choice. Please enter 'T', 'S', or 'B'.\", \"warning\"\n )\n"
},
{
"path": "cli/cli_launcher.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nDeepCode - CLI Research Engine Launcher\nDeepCode - CLI研究引擎启动器\n\n🧬 Open-Source Code Agent by Data Intelligence Lab @ HKU (CLI Edition)\n⚡ Revolutionizing research reproducibility through collaborative AI via command line\n\"\"\"\n\nimport sys\nfrom pathlib import Path\n\n\ndef check_dependencies():\n \"\"\"检查必要的依赖是否已安装 / Check if necessary dependencies are installed\"\"\"\n import importlib.util\n\n print(\"🔍 Checking CLI dependencies...\")\n\n missing_deps = []\n\n # Check asyncio availability\n if importlib.util.find_spec(\"asyncio\") is not None:\n print(\"✅ Asyncio is available\")\n else:\n missing_deps.append(\"asyncio\")\n\n # Check PyYAML availability\n if importlib.util.find_spec(\"yaml\") is not None:\n print(\"✅ PyYAML is installed\")\n else:\n missing_deps.append(\"pyyaml\")\n\n # Check Tkinter availability\n if importlib.util.find_spec(\"tkinter\") is not None:\n print(\"✅ Tkinter is available (for file dialogs)\")\n else:\n print(\"⚠️ Tkinter not available - file dialogs will use manual input\")\n\n # Check for MCP agent dependencies\n if importlib.util.find_spec(\"mcp_agent.app\") is not None:\n print(\"✅ MCP Agent framework is available\")\n else:\n missing_deps.append(\"mcp-agent\")\n\n # Check for workflow dependencies\n # 添加项目根目录到路径\n current_dir = Path(__file__).parent\n project_root = current_dir.parent\n if str(project_root) not in sys.path:\n sys.path.insert(0, str(project_root))\n\n if importlib.util.find_spec(\"workflows.agent_orchestration_engine\") is not None:\n print(\"✅ Workflow modules are available\")\n else:\n print(\"⚠️ Workflow modules may not be properly configured\")\n\n # Check for CLI components\n if importlib.util.find_spec(\"cli.cli_app\") is not None:\n print(\"✅ CLI application components are available\")\n else:\n print(\"❌ CLI application components missing\")\n missing_deps.append(\"cli-components\")\n\n if missing_deps:\n print(\"\\n❌ Missing dependencies:\")\n for dep in missing_deps:\n print(f\" - {dep}\")\n print(\"\\nPlease install missing dependencies using:\")\n print(\n f\"pip install {' '.join([d for d in missing_deps if d != 'cli-components'])}\"\n )\n if \"cli-components\" in missing_deps:\n print(\n \"CLI components appear to be missing - please check the cli/ directory\"\n )\n return False\n\n print(\"✅ All CLI dependencies satisfied\")\n return True\n\n\ndef print_banner():\n \"\"\"显示CLI启动横幅 / Display CLI startup banner\"\"\"\n banner = \"\"\"\n╔══════════════════════════════════════════════════════════════╗\n║ ║\n║ 🧬 DeepCode - Open-Source Code Agent ║\n║ ║\n║ ⚡ DATA INTELLIGENCE LAB @ HKU ⚡ ║\n║ ║\n║ ║\n║ ║\n╚══════════════════════════════════════════════════════════════╝\n\"\"\"\n print(banner)\n\n\ndef main():\n \"\"\"主函数 / Main function\"\"\"\n print_banner()\n\n # 检查依赖 / Check dependencies\n if not check_dependencies():\n print(\"\\n🚨 Please install missing dependencies and try again.\")\n sys.exit(1)\n\n # 获取当前脚本目录 / Get current script directory\n current_dir = Path(__file__).parent\n project_root = current_dir.parent\n cli_app_path = current_dir / \"cli_app.py\"\n\n # 检查cli_app.py是否存在 / Check if cli_app.py exists\n if not cli_app_path.exists():\n print(f\"❌ CLI application file not found: {cli_app_path}\")\n print(\"Please ensure the cli/cli_app.py file exists.\")\n sys.exit(1)\n\n print(f\"\\n📁 CLI App location: {cli_app_path}\")\n print(\"🖥️ Starting DeepCode CLI interface...\")\n print(\"🚀 Initializing command line application\")\n print(\"=\" * 70)\n print(\"💡 Tip: Follow the interactive prompts to process your research\")\n print(\"🛑 Press Ctrl+C to exit at any time\")\n print(\"=\" * 70)\n\n # 启动CLI应用 / Launch CLI application\n try:\n # 导入并运行CLI应用\n if str(project_root) not in sys.path:\n sys.path.insert(0, str(project_root)) # 添加项目根目录到路径\n from cli.cli_app import main as cli_main\n\n print(\"\\n🎯 Launching CLI application...\")\n\n # 使用asyncio运行主函数\n import asyncio\n\n asyncio.run(cli_main())\n\n except KeyboardInterrupt:\n print(\"\\n\\n🛑 DeepCode CLI stopped by user\")\n print(\"Thank you for using DeepCode CLI! 🧬\")\n except ImportError as e:\n print(f\"\\n❌ Failed to import CLI application: {e}\")\n print(\"Please check if all modules are properly installed.\")\n sys.exit(1)\n except Exception as e:\n print(f\"\\n❌ Unexpected error: {e}\")\n print(\"Please check your Python environment and try again.\")\n sys.exit(1)\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "cli/main_cli.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nDeepCode CLI - Open-Source Code Agent\n深度代码CLI - 开源代码智能体\n\n🧬 Data Intelligence Lab @ HKU\n⚡ Revolutionizing Research Reproducibility through Multi-Agent Architecture\n\"\"\"\n\nimport os\nimport sys\nimport asyncio\nimport argparse\n\n# 禁止生成.pyc文件\nos.environ[\"PYTHONDONTWRITEBYTECODE\"] = \"1\"\n\n# 添加项目根目录到路径\ncurrent_dir = os.path.dirname(os.path.abspath(__file__))\nparent_dir = os.path.dirname(current_dir)\nif parent_dir not in sys.path:\n sys.path.insert(0, parent_dir)\n\n# 导入CLI应用\nfrom cli.cli_app import CLIApp, Colors\n\n\ndef print_enhanced_banner():\n \"\"\"显示增强版启动横幅\"\"\"\n banner = f\"\"\"\n{Colors.CYAN}╔══════════════════════════════════════════════════════════════════════════════╗\n║ ║\n║ {Colors.BOLD}{Colors.MAGENTA}🧬 DeepCode - Open-Source Code Agent{Colors.CYAN} ║\n║ ║\n║ {Colors.BOLD}{Colors.YELLOW}⚡ DATA INTELLIGENCE LAB @ HKU ⚡{Colors.CYAN} ║\n║ ║\n║ Revolutionizing research reproducibility through collaborative AI ║\n║ Building the future where code is reproduced from natural language ║\n║ ║\n║ {Colors.BOLD}{Colors.GREEN}🤖 Key Features:{Colors.CYAN} ║\n║ • Automated paper-to-code reproduction ║\n║ • Multi-agent collaborative architecture ║\n║ • Open-source and extensible design ║\n║ • Join our growing research community ║\n║ ║\n╚══════════════════════════════════════════════════════════════════════════════╝{Colors.ENDC}\n\"\"\"\n print(banner)\n\n\ndef check_environment():\n \"\"\"检查运行环境\"\"\"\n print(f\"{Colors.CYAN}🔍 Checking environment...{Colors.ENDC}\")\n\n # 检查Python版本\n if sys.version_info < (3, 8):\n print(\n f\"{Colors.FAIL}❌ Python 3.8+ required. Current: {sys.version}{Colors.ENDC}\"\n )\n return False\n\n print(f\"{Colors.OKGREEN}✅ Python {sys.version.split()[0]} - OK{Colors.ENDC}\")\n\n # 检查必要模块\n required_modules = [\n (\"asyncio\", \"Async IO support\"),\n (\"pathlib\", \"Path handling\"),\n (\"typing\", \"Type hints\"),\n ]\n\n missing_modules = []\n for module, desc in required_modules:\n try:\n __import__(module)\n print(f\"{Colors.OKGREEN}✅ {desc} - OK{Colors.ENDC}\")\n except ImportError:\n missing_modules.append(module)\n print(f\"{Colors.FAIL}❌ {desc} - Missing{Colors.ENDC}\")\n\n if missing_modules:\n print(\n f\"{Colors.FAIL}❌ Missing required modules: {', '.join(missing_modules)}{Colors.ENDC}\"\n )\n return False\n\n print(f\"{Colors.OKGREEN}✅ Environment check passed{Colors.ENDC}\")\n return True\n\n\ndef parse_arguments():\n \"\"\"解析命令行参数\"\"\"\n parser = argparse.ArgumentParser(\n description=\"DeepCode CLI - Open-Source Code Agent by Data Intelligence Lab @ HKU\",\n formatter_class=argparse.RawDescriptionHelpFormatter,\n epilog=f\"\"\"\n{Colors.BOLD}Examples:{Colors.ENDC}\n {Colors.CYAN}python main_cli.py{Colors.ENDC} # Interactive mode\n {Colors.CYAN}python main_cli.py --file paper.pdf{Colors.ENDC} # Process file directly\n {Colors.CYAN}python main_cli.py --url https://...{Colors.ENDC} # Process URL directly\n {Colors.CYAN}python main_cli.py --chat \"Build a web app...\"{Colors.ENDC} # Process chat requirements\n {Colors.CYAN}python main_cli.py --requirement \"ML system for...\"{Colors.ENDC} # Guided requirement analysis (NEW)\n {Colors.CYAN}python main_cli.py --optimized{Colors.ENDC} # Use optimized mode\n {Colors.CYAN}python main_cli.py --disable-segmentation{Colors.ENDC} # Disable document segmentation\n {Colors.CYAN}python main_cli.py --segmentation-threshold 30000{Colors.ENDC} # Custom segmentation threshold\n\n{Colors.BOLD}Pipeline Modes:{Colors.ENDC}\n {Colors.GREEN}Comprehensive{Colors.ENDC}: Full intelligence analysis with indexing\n {Colors.YELLOW}Optimized{Colors.ENDC}: Fast processing without indexing\n {Colors.BLUE}Requirement Analysis{Colors.ENDC}: Guided Q&A to refine requirements (NEW)\n\n{Colors.BOLD}Document Processing:{Colors.ENDC}\n {Colors.BLUE}Smart Segmentation{Colors.ENDC}: Intelligent document segmentation for large papers\n {Colors.MAGENTA}Supported Formats{Colors.ENDC}: PDF, DOCX, DOC, PPT, PPTX, XLS, XLSX, HTML, TXT, MD\n \"\"\",\n )\n\n parser.add_argument(\n \"--file\", \"-f\", type=str, help=\"Process a specific file (PDF, DOCX, TXT, etc.)\"\n )\n\n parser.add_argument(\n \"--url\", \"-u\", type=str, help=\"Process a research paper from URL\"\n )\n\n parser.add_argument(\n \"--chat\",\n \"-t\",\n type=str,\n help=\"Process coding requirements via chat input (provide requirements as argument)\",\n )\n\n parser.add_argument(\n \"--requirement\",\n \"-r\",\n type=str,\n help=\"Process requirements via guided analysis (provide initial idea as argument)\",\n )\n\n parser.add_argument(\n \"--optimized\",\n \"-o\",\n action=\"store_true\",\n help=\"Use optimized mode (skip indexing for faster processing)\",\n )\n\n parser.add_argument(\n \"--disable-segmentation\",\n action=\"store_true\",\n help=\"Disable intelligent document segmentation (use traditional full-document processing)\",\n )\n\n parser.add_argument(\n \"--segmentation-threshold\",\n type=int,\n default=50000,\n help=\"Document size threshold (characters) to trigger segmentation (default: 50000)\",\n )\n\n parser.add_argument(\n \"--verbose\", \"-v\", action=\"store_true\", help=\"Enable verbose output\"\n )\n\n return parser.parse_args()\n\n\nasync def run_direct_processing(app: CLIApp, input_source: str, input_type: str):\n \"\"\"直接处理模式（非交互式）\"\"\"\n try:\n print(\n f\"\\n{Colors.BOLD}{Colors.CYAN}🚀 Starting direct processing mode...{Colors.ENDC}\"\n )\n print(f\"{Colors.CYAN}Input: {input_source}{Colors.ENDC}\")\n print(f\"{Colors.CYAN}Type: {input_type}{Colors.ENDC}\")\n print(\n f\"{Colors.CYAN}Mode: {'🧠 Comprehensive' if app.cli.enable_indexing else '⚡ Optimized'}{Colors.ENDC}\"\n )\n\n # 初始化应用\n init_result = await app.initialize_mcp_app()\n if init_result[\"status\"] != \"success\":\n print(\n f\"{Colors.FAIL}❌ Initialization failed: {init_result['message']}{Colors.ENDC}\"\n )\n return False\n\n # 处理输入\n result = await app.process_input(input_source, input_type)\n\n if result[\"status\"] == \"success\":\n print(\n f\"\\n{Colors.BOLD}{Colors.OKGREEN}🎉 Processing completed successfully!{Colors.ENDC}\"\n )\n return True\n else:\n print(\n f\"\\n{Colors.BOLD}{Colors.FAIL}❌ Processing failed: {result.get('error', 'Unknown error')}{Colors.ENDC}\"\n )\n return False\n\n except Exception as e:\n print(f\"\\n{Colors.FAIL}❌ Direct processing error: {str(e)}{Colors.ENDC}\")\n return False\n finally:\n await app.cleanup_mcp_app()\n\n\nasync def run_requirement_analysis(app: CLIApp, initial_idea: str):\n \"\"\"需求分析模式（非交互式） - NEW: matching UI version\"\"\"\n try:\n print(\n f\"\\n{Colors.BOLD}{Colors.BLUE}🧠 Starting requirement analysis mode...{Colors.ENDC}\"\n )\n print(f\"{Colors.CYAN}Initial Idea: {initial_idea}{Colors.ENDC}\")\n\n # 初始化应用\n init_result = await app.initialize_mcp_app()\n if init_result[\"status\"] != \"success\":\n print(\n f\"{Colors.FAIL}❌ Initialization failed: {init_result['message']}{Colors.ENDC}\"\n )\n return False\n\n # 执行需求分析工作流\n result = await app.process_requirement_analysis_non_interactive(initial_idea)\n\n if result[\"status\"] == \"success\":\n print(\n f\"\\n{Colors.BOLD}{Colors.OKGREEN}🎉 Requirement analysis completed successfully!{Colors.ENDC}\"\n )\n return True\n else:\n print(\n f\"\\n{Colors.BOLD}{Colors.FAIL}❌ Requirement analysis failed: {result.get('error', 'Unknown error')}{Colors.ENDC}\"\n )\n return False\n\n except Exception as e:\n print(f\"\\n{Colors.FAIL}❌ Requirement analysis error: {str(e)}{Colors.ENDC}\")\n return False\n finally:\n await app.cleanup_mcp_app()\n\n\nasync def main():\n \"\"\"主函数\"\"\"\n # 解析命令行参数\n args = parse_arguments()\n\n # 显示横幅\n print_enhanced_banner()\n\n # 检查环境\n if not check_environment():\n print(\n f\"\\n{Colors.FAIL}🚨 Environment check failed. Please fix the issues and try again.{Colors.ENDC}\"\n )\n sys.exit(1)\n\n try:\n # 创建CLI应用\n app = CLIApp()\n\n # 设置配置 - 默认禁用索引功能以加快处理速度\n if args.optimized:\n app.cli.enable_indexing = False\n print(\n f\"\\n{Colors.YELLOW}⚡ Optimized mode enabled - indexing disabled{Colors.ENDC}\"\n )\n else:\n # 默认也禁用索引功能\n app.cli.enable_indexing = False\n print(\n f\"\\n{Colors.YELLOW}⚡ Fast mode enabled - indexing disabled by default{Colors.ENDC}\"\n )\n\n # Configure document segmentation settings\n if hasattr(args, \"disable_segmentation\") and args.disable_segmentation:\n print(\n f\"\\n{Colors.MAGENTA}📄 Document segmentation disabled - using traditional processing{Colors.ENDC}\"\n )\n app.cli.segmentation_enabled = False\n app.cli.segmentation_threshold = args.segmentation_threshold\n app.cli._save_segmentation_config()\n else:\n print(\n f\"\\n{Colors.BLUE}📄 Smart document segmentation enabled (threshold: {args.segmentation_threshold} chars){Colors.ENDC}\"\n )\n app.cli.segmentation_enabled = True\n app.cli.segmentation_threshold = args.segmentation_threshold\n app.cli._save_segmentation_config()\n\n # 检查是否为直接处理模式\n if args.file or args.url or args.chat or args.requirement:\n if args.file:\n # 验证文件存在\n if not os.path.exists(args.file):\n print(f\"{Colors.FAIL}❌ File not found: {args.file}{Colors.ENDC}\")\n sys.exit(1)\n # 使用 file:// 前缀保持与交互模式一致，确保文件被复制而非移动\n file_url = f\"file://{os.path.abspath(args.file)}\"\n success = await run_direct_processing(app, file_url, \"file\")\n elif args.url:\n success = await run_direct_processing(app, args.url, \"url\")\n elif args.chat:\n # 验证chat输入长度\n if len(args.chat.strip()) < 20:\n print(\n f\"{Colors.FAIL}❌ Chat input too short. Please provide more detailed requirements (at least 20 characters){Colors.ENDC}\"\n )\n sys.exit(1)\n success = await run_direct_processing(app, args.chat, \"chat\")\n elif args.requirement:\n # NEW: Requirement analysis mode\n # 验证需求输入长度\n if len(args.requirement.strip()) < 10:\n print(\n f\"{Colors.FAIL}❌ Requirement input too short. Please provide more details (at least 10 characters){Colors.ENDC}\"\n )\n sys.exit(1)\n success = await run_requirement_analysis(app, args.requirement)\n\n sys.exit(0 if success else 1)\n else:\n # 交互式模式\n print(f\"\\n{Colors.CYAN}🎮 Starting interactive mode...{Colors.ENDC}\")\n await app.run_interactive_session()\n\n except KeyboardInterrupt:\n print(f\"\\n{Colors.WARNING}⚠️ Application interrupted by user{Colors.ENDC}\")\n sys.exit(1)\n except Exception as e:\n print(f\"\\n{Colors.FAIL}❌ Application errors: {str(e)}{Colors.ENDC}\")\n sys.exit(1)\n\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n"
},
{
"path": "cli/workflows/__init__.py",
"content": "\"\"\"\nCLI-specific Workflow Adapters\nCLI专用工作流适配器\n\nThis module provides CLI-optimized versions of workflow components that are\nspecifically adapted for command-line interface usage patterns.\n\"\"\"\n\nfrom .cli_workflow_adapter import CLIWorkflowAdapter\n\n__all__ = [\"CLIWorkflowAdapter\"]\n"
},
{
"path": "cli/workflows/cli_workflow_adapter.py",
"content": "\"\"\"\nCLI Workflow Adapter for Agent Orchestration Engine\nCLI工作流适配器 - 智能体编排引擎\n\nThis adapter provides CLI-optimized interface to the latest agent orchestration engine,\nwith enhanced progress reporting, error handling, and CLI-specific optimizations.\n\nVersion: 2.1 (Updated to match UI version - Added Requirement Analysis)\nChanges:\n- Default enable_indexing=False for faster processing (matching UI defaults)\n- Mode-aware progress callback with detailed stage mapping\n- Chat pipeline now accepts enable_indexing parameter\n- Improved error handling and resource management\n- Enhanced progress display for different modes (fast/comprehensive/chat)\n- NEW: Added requirement analysis workflow support\n\"\"\"\n\nimport os\nfrom typing import Callable, Dict, Any\nfrom mcp_agent.app import MCPApp\n\n\nclass CLIWorkflowAdapter:\n \"\"\"\n CLI-optimized workflow adapter for the intelligent agent orchestration engine.\n\n This adapter provides:\n - Enhanced CLI progress reporting\n - Optimized error handling for CLI environments\n - Streamlined interface for command-line usage\n - Integration with the latest agent orchestration engine\n \"\"\"\n\n def __init__(self, cli_interface=None):\n \"\"\"\n Initialize CLI workflow adapter.\n\n Args:\n cli_interface: CLI interface instance for progress reporting\n \"\"\"\n self.cli_interface = cli_interface\n self.app = None\n self.logger = None\n self.context = None\n\n async def initialize_mcp_app(self) -> Dict[str, Any]:\n \"\"\"\n Initialize MCP application for CLI usage (improved version matching UI).\n\n Returns:\n dict: Initialization result\n \"\"\"\n try:\n if self.cli_interface:\n self.cli_interface.show_spinner(\n \"🚀 Initializing Agent Orchestration Engine\", 2.0\n )\n\n # Initialize MCP application using async context manager (matching UI pattern)\n self.app = MCPApp(name=\"cli_agent_orchestration\")\n self.app_context = self.app.run()\n agent_app = await self.app_context.__aenter__()\n\n self.logger = agent_app.logger\n self.context = agent_app.context\n\n # Configure filesystem access\n self.context.config.mcp.servers[\"filesystem\"].args.extend([os.getcwd()])\n\n if self.cli_interface:\n self.cli_interface.print_status(\n \"🧠 Agent Orchestration Engine initialized successfully\", \"success\"\n )\n\n return {\n \"status\": \"success\",\n \"message\": \"MCP application initialized successfully\",\n }\n\n except Exception as e:\n error_msg = f\"Failed to initialize MCP application: {str(e)}\"\n if self.cli_interface:\n self.cli_interface.print_status(error_msg, \"error\")\n return {\"status\": \"error\", \"message\": error_msg}\n\n async def cleanup_mcp_app(self):\n \"\"\"\n Clean up MCP application resources.\n \"\"\"\n if hasattr(self, \"app_context\"):\n try:\n await self.app_context.__aexit__(None, None, None)\n if self.cli_interface:\n self.cli_interface.print_status(\n \"🧹 Resources cleaned up successfully\", \"info\"\n )\n except Exception as e:\n if self.cli_interface:\n self.cli_interface.print_status(\n f\"⚠️ Cleanup warning: {str(e)}\", \"warning\"\n )\n\n def create_cli_progress_callback(self, enable_indexing: bool = True) -> Callable:\n \"\"\"\n Create CLI-optimized progress callback function with mode-aware stage mapping.\n\n This matches the UI version's detailed progress mapping logic.\n\n Args:\n enable_indexing: Whether indexing is enabled (affects stage mapping)\n\n Returns:\n Callable: Progress callback function\n \"\"\"\n\n def progress_callback(progress: int, message: str):\n if self.cli_interface:\n # Mode-aware stage mapping (matching UI version logic)\n if enable_indexing:\n # Full workflow mapping: Initialize -> Analyze -> Download -> Plan -> References -> Repos -> Index -> Implement\n if progress <= 5:\n stage = 0 # Initialize\n elif progress <= 10:\n stage = 1 # Analyze\n elif progress <= 25:\n stage = 2 # Download\n elif progress <= 40:\n stage = 3 # Plan\n elif progress <= 50:\n stage = 4 # References\n elif progress <= 60:\n stage = 5 # Repos\n elif progress <= 70:\n stage = 6 # Index\n elif progress <= 85:\n stage = 7 # Implement\n else:\n stage = 8 # Complete\n else:\n # Fast mode mapping: Initialize -> Analyze -> Download -> Plan -> Implement\n if progress <= 5:\n stage = 0 # Initialize\n elif progress <= 10:\n stage = 1 # Analyze\n elif progress <= 25:\n stage = 2 # Download\n elif progress <= 40:\n stage = 3 # Plan\n elif progress <= 85:\n stage = 4 # Implement (skip References, Repos, Index)\n else:\n stage = 4 # Complete\n\n self.cli_interface.display_processing_stages(stage, enable_indexing)\n\n # Display status message\n self.cli_interface.print_status(message, \"processing\")\n\n return progress_callback\n\n async def execute_full_pipeline(\n self, input_source: str, enable_indexing: bool = False\n ) -> Dict[str, Any]:\n \"\"\"\n Execute the complete intelligent multi-agent research orchestration pipeline.\n\n Updated to match UI version: default enable_indexing=False for faster processing.\n\n Args:\n input_source: Research input source (file path, URL, or preprocessed analysis)\n enable_indexing: Whether to enable advanced intelligence analysis (default: False)\n\n Returns:\n dict: Comprehensive pipeline execution result\n \"\"\"\n try:\n # Import the latest agent orchestration engine\n from workflows.agent_orchestration_engine import (\n execute_multi_agent_research_pipeline,\n )\n\n # Create CLI progress callback with mode awareness\n progress_callback = self.create_cli_progress_callback(enable_indexing)\n\n # Display pipeline start\n if self.cli_interface:\n if enable_indexing:\n mode_msg = \"🧠 comprehensive (with indexing)\"\n else:\n mode_msg = \"⚡ fast (indexing disabled)\"\n self.cli_interface.print_status(\n f\"🚀 Starting {mode_msg} agent orchestration pipeline...\",\n \"processing\",\n )\n self.cli_interface.display_processing_stages(0, enable_indexing)\n\n # Execute the pipeline\n result = await execute_multi_agent_research_pipeline(\n input_source=input_source,\n logger=self.logger,\n progress_callback=progress_callback,\n enable_indexing=enable_indexing,\n )\n\n # Display completion\n if self.cli_interface:\n final_stage = 8 if enable_indexing else 4\n self.cli_interface.display_processing_stages(\n final_stage, enable_indexing\n )\n self.cli_interface.print_status(\n \"🎉 Agent orchestration pipeline completed successfully!\",\n \"complete\",\n )\n\n return {\n \"status\": \"success\",\n \"result\": result,\n \"pipeline_mode\": \"comprehensive\" if enable_indexing else \"optimized\",\n }\n\n except Exception as e:\n error_msg = f\"Pipeline execution failed: {str(e)}\"\n if self.cli_interface:\n self.cli_interface.print_status(error_msg, \"error\")\n\n return {\n \"status\": \"error\",\n \"error\": error_msg,\n \"pipeline_mode\": \"comprehensive\" if enable_indexing else \"optimized\",\n }\n\n async def execute_requirement_analysis_workflow(\n self, user_input: str, analysis_mode: str, user_answers: Dict[str, str] = None\n ) -> Dict[str, Any]:\n \"\"\"\n Execute requirement analysis workflow (NEW: matching UI version).\n\n This workflow helps users refine their requirements through guided questions\n and intelligent analysis before starting code implementation.\n\n Args:\n user_input: User's initial requirements or description\n analysis_mode: Analysis mode (\"generate_questions\" or \"summarize_requirements\")\n user_answers: Dictionary of user answers to guiding questions (for summarize mode)\n\n Returns:\n dict: Analysis result with questions or requirement summary\n \"\"\"\n try:\n # Import the requirement analysis workflow\n from workflows.agent_orchestration_engine import (\n execute_requirement_analysis_workflow,\n )\n\n # Create CLI progress callback\n def analysis_progress_callback(progress: int, message: str):\n if self.cli_interface:\n self.cli_interface.print_status(message, \"processing\")\n\n # Display workflow start\n if self.cli_interface:\n if analysis_mode == \"generate_questions\":\n self.cli_interface.print_status(\n \"🤖 Generating guiding questions for your requirements...\",\n \"processing\",\n )\n else:\n self.cli_interface.print_status(\n \"📄 Analyzing and summarizing your detailed requirements...\",\n \"processing\",\n )\n\n # Execute the requirement analysis workflow\n result = await execute_requirement_analysis_workflow(\n user_input=user_input,\n analysis_mode=analysis_mode,\n user_answers=user_answers,\n logger=self.logger,\n progress_callback=analysis_progress_callback,\n )\n\n # Display completion\n if self.cli_interface:\n if result[\"status\"] == \"success\":\n if analysis_mode == \"generate_questions\":\n self.cli_interface.print_status(\n \"✅ Guiding questions generated successfully!\", \"success\"\n )\n else:\n self.cli_interface.print_status(\n \"✅ Requirements analysis completed successfully!\",\n \"success\",\n )\n else:\n self.cli_interface.print_status(\n f\"❌ Analysis failed: {result.get('error', 'Unknown error')}\",\n \"error\",\n )\n\n return result\n\n except Exception as e:\n error_msg = f\"Requirement analysis workflow failed: {str(e)}\"\n if self.cli_interface:\n self.cli_interface.print_status(error_msg, \"error\")\n\n return {\"status\": \"error\", \"error\": error_msg}\n\n async def execute_chat_pipeline(\n self, user_input: str, enable_indexing: bool = False\n ) -> Dict[str, Any]:\n \"\"\"\n Execute the chat-based planning and implementation pipeline.\n\n Updated to match UI version: accepts enable_indexing parameter.\n\n Args:\n user_input: User's coding requirements and description\n enable_indexing: Whether to enable indexing for enhanced code understanding (default: False)\n\n Returns:\n dict: Chat pipeline execution result\n \"\"\"\n try:\n # Import the chat-based pipeline\n from workflows.agent_orchestration_engine import (\n execute_chat_based_planning_pipeline,\n )\n\n # Create CLI progress callback for chat mode\n def chat_progress_callback(progress: int, message: str):\n if self.cli_interface:\n # Map progress to CLI stages for chat mode (matching UI logic)\n if progress <= 5:\n stage = 0 # Initialize\n elif progress <= 30:\n stage = 1 # Planning\n elif progress <= 50:\n stage = 2 # Setup\n elif progress <= 70:\n stage = 3 # Save Plan\n else:\n stage = 4 # Implement\n\n self.cli_interface.display_processing_stages(stage, chat_mode=True)\n\n # Display status message\n self.cli_interface.print_status(message, \"processing\")\n\n # Display pipeline start\n if self.cli_interface:\n indexing_note = (\n \" (with indexing)\" if enable_indexing else \" (fast mode)\"\n )\n self.cli_interface.print_status(\n f\"🚀 Starting chat-based planning pipeline{indexing_note}...\",\n \"processing\",\n )\n self.cli_interface.display_processing_stages(0, chat_mode=True)\n\n # Execute the chat pipeline with configurable indexing\n result = await execute_chat_based_planning_pipeline(\n user_input=user_input,\n logger=self.logger,\n progress_callback=chat_progress_callback,\n enable_indexing=enable_indexing, # Pass through enable_indexing parameter\n )\n\n # Display completion\n if self.cli_interface:\n self.cli_interface.display_processing_stages(4, chat_mode=True)\n self.cli_interface.print_status(\n \"🎉 Chat-based planning pipeline completed successfully!\",\n \"complete\",\n )\n\n return {\"status\": \"success\", \"result\": result, \"pipeline_mode\": \"chat\"}\n\n except Exception as e:\n error_msg = f\"Chat pipeline execution failed: {str(e)}\"\n if self.cli_interface:\n self.cli_interface.print_status(error_msg, \"error\")\n\n return {\"status\": \"error\", \"error\": error_msg, \"pipeline_mode\": \"chat\"}\n\n async def process_input_with_orchestration(\n self, input_source: str, input_type: str, enable_indexing: bool = False\n ) -> Dict[str, Any]:\n \"\"\"\n Process input using the intelligent agent orchestration engine.\n\n This is the main CLI interface to the latest agent orchestration capabilities.\n Updated to match UI version: default enable_indexing=False.\n\n Args:\n input_source: Input source (file path, URL, or chat input)\n input_type: Type of input ('file', 'url', or 'chat')\n enable_indexing: Whether to enable advanced intelligence analysis (default: False)\n\n Returns:\n dict: Processing result with status and details\n \"\"\"\n pipeline_result = None\n\n try:\n # Initialize MCP app\n init_result = await self.initialize_mcp_app()\n if init_result[\"status\"] != \"success\":\n return init_result\n\n # Process file:// URLs for traditional file/URL inputs\n if input_source.startswith(\"file://\"):\n file_path = input_source[7:]\n if os.name == \"nt\" and file_path.startswith(\"/\"):\n file_path = file_path.lstrip(\"/\")\n input_source = file_path\n\n # Execute appropriate pipeline based on input type\n if input_type == \"chat\":\n # Use chat-based planning pipeline for user requirements\n # Pass enable_indexing to chat pipeline as well\n pipeline_result = await self.execute_chat_pipeline(\n input_source, enable_indexing=enable_indexing\n )\n else:\n # Use traditional multi-agent research pipeline for files/URLs\n pipeline_result = await self.execute_full_pipeline(\n input_source, enable_indexing=enable_indexing\n )\n\n return {\n \"status\": pipeline_result[\"status\"],\n \"analysis_result\": \"Integrated into agent orchestration pipeline\",\n \"download_result\": \"Integrated into agent orchestration pipeline\",\n \"repo_result\": pipeline_result.get(\"result\", \"\"),\n \"pipeline_mode\": pipeline_result.get(\"pipeline_mode\", \"comprehensive\"),\n \"error\": pipeline_result.get(\"error\"),\n }\n\n except Exception as e:\n error_msg = f\"Error during orchestrated processing: {str(e)}\"\n if self.cli_interface:\n self.cli_interface.print_status(error_msg, \"error\")\n\n return {\n \"status\": \"error\",\n \"error\": error_msg,\n \"analysis_result\": \"\",\n \"download_result\": \"\",\n \"repo_result\": \"\",\n \"pipeline_mode\": \"comprehensive\" if enable_indexing else \"optimized\",\n }\n\n finally:\n # Clean up resources\n await self.cleanup_mcp_app()\n"
},
{
"path": "config/mcp_tool_definitions.py",
"content": "\"\"\"\nMCP工具定义配置模块\nMCP Tool Definitions Configuration Module\n\n将工具定义从主程序逻辑中分离，提供标准化的工具定义格式\nSeparate tool definitions from main program logic, providing standardized tool definition format\n\n支持的工具类型：\n- 文件操作工具 (File Operations)\n- 代码执行工具 (Code Execution)\n- 搜索工具 (Search Tools)\n- 项目结构工具 (Project Structure Tools)\n\"\"\"\n\nfrom typing import Dict, List, Any\n\n\nclass MCPToolDefinitions:\n \"\"\"MCP工具定义管理器\"\"\"\n\n @staticmethod\n def get_code_implementation_tools() -> List[Dict[str, Any]]:\n \"\"\"\n 获取代码实现相关的工具定义\n Get tool definitions for code implementation\n \"\"\"\n return [\n # MCPToolDefinitions._get_read_file_tool(),\n # MCPToolDefinitions._get_read_multiple_files_tool(),\n # MCPToolDefinitions._get_read_code_mem_tool(),\n MCPToolDefinitions._get_write_file_tool(),\n # MCPToolDefinitions._get_write_multiple_files_tool(),\n # MCPToolDefinitions._get_execute_python_tool(),\n # MCPToolDefinitions._get_execute_bash_tool(),\n ]\n\n @staticmethod\n def _get_read_file_tool() -> Dict[str, Any]:\n \"\"\"读取文件工具定义\"\"\"\n return {\n \"name\": \"read_file\",\n \"description\": \"Read file content, supports specifying line number range\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_path\": {\n \"type\": \"string\",\n \"description\": \"File path, relative to workspace\",\n },\n \"start_line\": {\n \"type\": \"integer\",\n \"description\": \"Start line number (starting from 1, optional)\",\n },\n \"end_line\": {\n \"type\": \"integer\",\n \"description\": \"End line number (starting from 1, optional)\",\n },\n },\n \"required\": [\"file_path\"],\n },\n }\n\n @staticmethod\n def _get_read_multiple_files_tool() -> Dict[str, Any]:\n \"\"\"批量读取多个文件工具定义\"\"\"\n return {\n \"name\": \"read_multiple_files\",\n \"description\": \"Read multiple files in a single operation (for batch reading)\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_requests\": {\n \"type\": \"string\",\n \"description\": 'JSON string with file requests, e.g., \\'{\"file1.py\": {}, \"file2.py\": {\"start_line\": 1, \"end_line\": 10}}\\' or simple array \\'[\"file1.py\", \"file2.py\"]\\'',\n },\n \"max_files\": {\n \"type\": \"integer\",\n \"description\": \"Maximum number of files to read in one operation\",\n \"default\": 5,\n \"minimum\": 1,\n \"maximum\": 10,\n },\n },\n \"required\": [\"file_requests\"],\n },\n }\n\n @staticmethod\n def _get_read_code_mem_tool() -> Dict[str, Any]:\n \"\"\"Read code memory tool definition - reads from implement_code_summary.md\"\"\"\n return {\n \"name\": \"read_code_mem\",\n \"description\": \"Check if file summaries exist in implement_code_summary.md for multiple files in a single call. Returns summaries for all requested files if available.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_paths\": {\n \"type\": \"array\",\n \"items\": {\"type\": \"string\"},\n \"description\": \"List of file paths to check for summary information in implement_code_summary.md\",\n }\n },\n \"required\": [\"file_paths\"],\n },\n }\n\n @staticmethod\n def _get_write_file_tool() -> Dict[str, Any]:\n \"\"\"写入文件工具定义\"\"\"\n return {\n \"name\": \"write_file\",\n \"description\": \"Write content to file\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_path\": {\n \"type\": \"string\",\n \"description\": \"File path, relative to workspace\",\n },\n \"content\": {\n \"type\": \"string\",\n \"description\": \"Content to write to file\",\n },\n \"create_dirs\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create directories if they don't exist\",\n \"default\": True,\n },\n \"create_backup\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create backup file if file already exists\",\n \"default\": False,\n },\n },\n \"required\": [\"file_path\", \"content\"],\n },\n }\n\n @staticmethod\n def _get_write_multiple_files_tool() -> Dict[str, Any]:\n \"\"\"批量写入多个文件工具定义\"\"\"\n return {\n \"name\": \"write_multiple_files\",\n \"description\": \"Write multiple files in a single operation (for batch implementation)\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_implementations\": {\n \"type\": \"string\",\n \"description\": 'JSON string mapping file paths to content, e.g., \\'{\"file1.py\": \"content1\", \"file2.py\": \"content2\"}\\'',\n },\n \"create_dirs\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create directories if they don't exist\",\n \"default\": True,\n },\n \"create_backup\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create backup files if they already exist\",\n \"default\": False,\n },\n \"max_files\": {\n \"type\": \"integer\",\n \"description\": \"Maximum number of files to write in one operation\",\n \"default\": 5,\n \"minimum\": 1,\n \"maximum\": 10,\n },\n },\n \"required\": [\"file_implementations\"],\n },\n }\n\n @staticmethod\n def _get_execute_python_tool() -> Dict[str, Any]:\n \"\"\"Python执行工具定义\"\"\"\n return {\n \"name\": \"execute_python\",\n \"description\": \"Execute Python code and return output\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"code\": {\"type\": \"string\", \"description\": \"Python code to execute\"},\n \"timeout\": {\n \"type\": \"integer\",\n \"description\": \"Timeout in seconds\",\n \"default\": 30,\n },\n },\n \"required\": [\"code\"],\n },\n }\n\n @staticmethod\n def _get_execute_bash_tool() -> Dict[str, Any]:\n \"\"\"Bash执行工具定义\"\"\"\n return {\n \"name\": \"execute_bash\",\n \"description\": \"Execute bash command\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"command\": {\n \"type\": \"string\",\n \"description\": \"Bash command to execute\",\n },\n \"timeout\": {\n \"type\": \"integer\",\n \"description\": \"Timeout in seconds\",\n \"default\": 30,\n },\n },\n \"required\": [\"command\"],\n },\n }\n\n @staticmethod\n def _get_file_structure_tool() -> Dict[str, Any]:\n \"\"\"文件结构获取工具定义\"\"\"\n return {\n \"name\": \"get_file_structure\",\n \"description\": \"Get directory file structure\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"directory\": {\n \"type\": \"string\",\n \"description\": \"Directory path, relative to workspace\",\n \"default\": \".\",\n },\n \"max_depth\": {\n \"type\": \"integer\",\n \"description\": \"Maximum traversal depth\",\n \"default\": 5,\n },\n },\n },\n }\n\n @staticmethod\n def _get_search_code_references_tool() -> Dict[str, Any]:\n \"\"\"统一代码参考搜索工具定义 - 合并了三个步骤为一个工具\"\"\"\n return {\n \"name\": \"search_code_references\",\n \"description\": \"UNIFIED TOOL: Search relevant reference code from index files. Combines directory setup, index loading, and searching in a single call.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"indexes_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the indexes directory containing JSON index files\",\n },\n \"target_file\": {\n \"type\": \"string\",\n \"description\": \"Target file path to be implemented\",\n },\n \"keywords\": {\n \"type\": \"string\",\n \"description\": \"Search keywords, comma-separated\",\n \"default\": \"\",\n },\n \"max_results\": {\n \"type\": \"integer\",\n \"description\": \"Maximum number of results to return\",\n \"default\": 10,\n },\n },\n \"required\": [\"indexes_path\", \"target_file\"],\n },\n }\n\n @staticmethod\n def _get_get_indexes_overview_tool() -> Dict[str, Any]:\n \"\"\"获取索引概览工具定义\"\"\"\n return {\n \"name\": \"get_indexes_overview\",\n \"description\": \"Get overview of all available reference code index information from specified directory\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"indexes_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the indexes directory containing JSON index files\",\n }\n },\n \"required\": [\"indexes_path\"],\n },\n }\n\n @staticmethod\n def _get_set_workspace_tool() -> Dict[str, Any]:\n \"\"\"Set workspace directory tool definition\"\"\"\n return {\n \"name\": \"set_workspace\",\n \"description\": \"Set the workspace directory for file operations\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"workspace_path\": {\n \"type\": \"string\",\n \"description\": \"Directory path for the workspace\",\n }\n },\n \"required\": [\"workspace_path\"],\n },\n }\n\n # @staticmethod\n # def _get_set_indexes_directory_tool() -> Dict[str, Any]:\n # \"\"\"Set indexes directory tool definition - DEPRECATED: Use unified search_code_references instead\"\"\"\n # return {\n # \"name\": \"set_indexes_directory\",\n # \"description\": \"Set the directory path for code reference indexes\",\n # \"input_schema\": {\n # \"type\": \"object\",\n # \"properties\": {\n # \"indexes_path\": {\n # \"type\": \"string\",\n # \"description\": \"Directory path containing index JSON files\"\n # }\n # },\n # \"required\": [\"indexes_path\"]\n # }\n # }\n\n @staticmethod\n def get_available_tool_sets() -> Dict[str, str]:\n \"\"\"\n 获取可用的工具集合\n Get available tool sets\n \"\"\"\n return {\n \"code_implementation\": \"代码实现相关工具集 / Code implementation tool set\",\n # 可以在这里添加更多工具集\n # \"data_analysis\": \"数据分析工具集 / Data analysis tool set\",\n # \"web_scraping\": \"网页爬取工具集 / Web scraping tool set\",\n }\n\n @staticmethod\n def get_tool_set(tool_set_name: str) -> List[Dict[str, Any]]:\n \"\"\"\n 根据名称获取特定的工具集\n Get specific tool set by name\n \"\"\"\n tool_sets = {\n \"code_implementation\": MCPToolDefinitions.get_code_implementation_tools(),\n }\n\n return tool_sets.get(tool_set_name, [])\n\n @staticmethod\n def get_all_tools() -> List[Dict[str, Any]]:\n \"\"\"\n 获取所有可用工具\n Get all available tools\n \"\"\"\n all_tools = []\n for tool_set_name in MCPToolDefinitions.get_available_tool_sets().keys():\n all_tools.extend(MCPToolDefinitions.get_tool_set(tool_set_name))\n return all_tools\n\n\n# 便捷访问函数\ndef get_mcp_tools(tool_set: str = \"code_implementation\") -> List[Dict[str, Any]]:\n \"\"\"\n 便捷函数：获取MCP工具定义\n Convenience function: Get MCP tool definitions\n\n Args:\n tool_set: 工具集名称 (默认: \"code_implementation\")\n\n Returns:\n 工具定义列表\n \"\"\"\n return MCPToolDefinitions.get_tool_set(tool_set)\n"
},
{
"path": "config/mcp_tool_definitions_index.py",
"content": "\"\"\"\nMCP工具定义配置模块\nMCP Tool Definitions Configuration Module\n\n将工具定义从主程序逻辑中分离，提供标准化的工具定义格式\nSeparate tool definitions from main program logic, providing standardized tool definition format\n\n支持的工具类型：\n- 文件操作工具 (File Operations)\n- 代码执行工具 (Code Execution)\n- 搜索工具 (Search Tools)\n- 项目结构工具 (Project Structure Tools)\n\"\"\"\n\nfrom typing import Dict, List, Any\n\n\nclass MCPToolDefinitions:\n \"\"\"MCP工具定义管理器\"\"\"\n\n @staticmethod\n def get_code_implementation_tools() -> List[Dict[str, Any]]:\n \"\"\"\n 获取代码实现相关的工具定义\n Get tool definitions for code implementation\n \"\"\"\n return [\n # MCPToolDefinitions._get_read_file_tool(),\n # MCPToolDefinitions._get_read_multiple_files_tool(),\n # MCPToolDefinitions._get_read_code_mem_tool(),\n MCPToolDefinitions._get_write_file_tool(),\n # MCPToolDefinitions._get_write_multiple_files_tool(),\n # MCPToolDefinitions._get_execute_python_tool(),\n # MCPToolDefinitions._get_execute_bash_tool(),\n MCPToolDefinitions._get_search_code_references_tool(),\n # MCPToolDefinitions._get_search_code_tool(),\n # MCPToolDefinitions._get_file_structure_tool(),\n # MCPToolDefinitions._get_set_workspace_tool(),\n # MCPToolDefinitions._get_operation_history_tool(),\n ]\n\n @staticmethod\n def get_code_evaluation_tools() -> List[Dict[str, Any]]:\n \"\"\"\n 获取代码评估相关的工具定义\n Get tool definitions for code evaluation\n \"\"\"\n return [\n MCPToolDefinitions._get_analyze_repo_structure_tool(),\n MCPToolDefinitions._get_detect_dependencies_tool(),\n MCPToolDefinitions._get_assess_code_quality_tool(),\n MCPToolDefinitions._get_evaluate_documentation_tool(),\n MCPToolDefinitions._get_check_reproduction_readiness_tool(),\n MCPToolDefinitions._get_generate_evaluation_summary_tool(),\n MCPToolDefinitions._get_detect_empty_files_tool(),\n MCPToolDefinitions._get_detect_missing_files_tool(),\n MCPToolDefinitions._get_generate_code_revision_report_tool(),\n ]\n\n @staticmethod\n def _get_read_file_tool() -> Dict[str, Any]:\n \"\"\"读取文件工具定义\"\"\"\n return {\n \"name\": \"read_file\",\n \"description\": \"Read file content, supports specifying line number range\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_path\": {\n \"type\": \"string\",\n \"description\": \"File path, relative to workspace\",\n },\n \"start_line\": {\n \"type\": \"integer\",\n \"description\": \"Start line number (starting from 1, optional)\",\n },\n \"end_line\": {\n \"type\": \"integer\",\n \"description\": \"End line number (starting from 1, optional)\",\n },\n },\n \"required\": [\"file_path\"],\n },\n }\n\n @staticmethod\n def _get_read_multiple_files_tool() -> Dict[str, Any]:\n \"\"\"批量读取多个文件工具定义\"\"\"\n return {\n \"name\": \"read_multiple_files\",\n \"description\": \"Read multiple files in a single operation (for batch reading)\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_requests\": {\n \"type\": \"string\",\n \"description\": 'JSON string with file requests, e.g., \\'{\"file1.py\": {}, \"file2.py\": {\"start_line\": 1, \"end_line\": 10}}\\' or simple array \\'[\"file1.py\", \"file2.py\"]\\'',\n },\n \"max_files\": {\n \"type\": \"integer\",\n \"description\": \"Maximum number of files to read in one operation\",\n \"default\": 5,\n \"minimum\": 1,\n \"maximum\": 10,\n },\n },\n \"required\": [\"file_requests\"],\n },\n }\n\n @staticmethod\n def _get_read_code_mem_tool() -> Dict[str, Any]:\n \"\"\"Read code memory tool definition - reads from implement_code_summary.md\"\"\"\n return {\n \"name\": \"read_code_mem\",\n \"description\": \"Check if file summaries exist in implement_code_summary.md for multiple files in a single call. Returns summaries for all requested files if available.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_paths\": {\n \"type\": \"array\",\n \"items\": {\"type\": \"string\"},\n \"description\": \"List of file paths to check for summary information in implement_code_summary.md\",\n }\n },\n \"required\": [\"file_paths\"],\n },\n }\n\n @staticmethod\n def _get_write_file_tool() -> Dict[str, Any]:\n \"\"\"写入文件工具定义\"\"\"\n return {\n \"name\": \"write_file\",\n \"description\": \"Write content to file\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_path\": {\n \"type\": \"string\",\n \"description\": \"File path, relative to workspace\",\n },\n \"content\": {\n \"type\": \"string\",\n \"description\": \"Content to write to file\",\n },\n \"create_dirs\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create directories if they don't exist\",\n \"default\": True,\n },\n \"create_backup\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create backup file if file already exists\",\n \"default\": False,\n },\n },\n \"required\": [\"file_path\", \"content\"],\n },\n }\n\n @staticmethod\n def _get_write_multiple_files_tool() -> Dict[str, Any]:\n \"\"\"批量写入多个文件工具定义\"\"\"\n return {\n \"name\": \"write_multiple_files\",\n \"description\": \"Write multiple files in a single operation (for batch implementation)\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"file_implementations\": {\n \"type\": \"string\",\n \"description\": 'JSON string mapping file paths to content, e.g., \\'{\"file1.py\": \"content1\", \"file2.py\": \"content2\"}\\'',\n },\n \"create_dirs\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create directories if they don't exist\",\n \"default\": True,\n },\n \"create_backup\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to create backup files if they already exist\",\n \"default\": False,\n },\n \"max_files\": {\n \"type\": \"integer\",\n \"description\": \"Maximum number of files to write in one operation\",\n \"default\": 5,\n \"minimum\": 1,\n \"maximum\": 10,\n },\n },\n \"required\": [\"file_implementations\"],\n },\n }\n\n @staticmethod\n def _get_execute_python_tool() -> Dict[str, Any]:\n \"\"\"Python执行工具定义\"\"\"\n return {\n \"name\": \"execute_python\",\n \"description\": \"Execute Python code and return output\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"code\": {\"type\": \"string\", \"description\": \"Python code to execute\"},\n \"timeout\": {\n \"type\": \"integer\",\n \"description\": \"Timeout in seconds\",\n \"default\": 30,\n },\n },\n \"required\": [\"code\"],\n },\n }\n\n @staticmethod\n def _get_execute_bash_tool() -> Dict[str, Any]:\n \"\"\"Bash执行工具定义\"\"\"\n return {\n \"name\": \"execute_bash\",\n \"description\": \"Execute bash command\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"command\": {\n \"type\": \"string\",\n \"description\": \"Bash command to execute\",\n },\n \"timeout\": {\n \"type\": \"integer\",\n \"description\": \"Timeout in seconds\",\n \"default\": 30,\n },\n },\n \"required\": [\"command\"],\n },\n }\n\n @staticmethod\n def _get_file_structure_tool() -> Dict[str, Any]:\n \"\"\"文件结构获取工具定义\"\"\"\n return {\n \"name\": \"get_file_structure\",\n \"description\": \"Get directory file structure\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"directory\": {\n \"type\": \"string\",\n \"description\": \"Directory path, relative to workspace\",\n \"default\": \".\",\n },\n \"max_depth\": {\n \"type\": \"integer\",\n \"description\": \"Maximum traversal depth\",\n \"default\": 5,\n },\n },\n },\n }\n\n @staticmethod\n def _get_search_code_references_tool() -> Dict[str, Any]:\n \"\"\"统一代码参考搜索工具定义 - 合并了三个步骤为一个工具\"\"\"\n return {\n \"name\": \"search_code_references\",\n \"description\": \"UNIFIED TOOL: Search relevant reference code from index files. Combines directory setup, index loading, and searching in a single call.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"indexes_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the indexes directory containing JSON index files\",\n },\n \"target_file\": {\n \"type\": \"string\",\n \"description\": \"Target file path to be implemented\",\n },\n \"keywords\": {\n \"type\": \"string\",\n \"description\": \"Search keywords, comma-separated\",\n \"default\": \"\",\n },\n \"max_results\": {\n \"type\": \"integer\",\n \"description\": \"Maximum number of results to return\",\n \"default\": 10,\n },\n },\n \"required\": [\"indexes_path\", \"target_file\"],\n },\n }\n\n @staticmethod\n def _get_search_code_tool() -> Dict[str, Any]:\n \"\"\"代码搜索工具定义 - 在当前代码库中搜索模式\"\"\"\n return {\n \"name\": \"search_code\",\n \"description\": \"Search patterns in code files within the current repository\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"pattern\": {\n \"type\": \"string\",\n \"description\": \"Search pattern\",\n },\n \"file_pattern\": {\n \"type\": \"string\",\n \"description\": \"File pattern (e.g., '*.py')\",\n \"default\": \"*.py\",\n },\n \"use_regex\": {\n \"type\": \"boolean\",\n \"description\": \"Whether to use regular expressions\",\n \"default\": False,\n },\n \"search_directory\": {\n \"type\": \"string\",\n \"description\": \"Specify search directory (optional)\",\n },\n },\n \"required\": [\"pattern\"],\n },\n }\n\n @staticmethod\n def _get_operation_history_tool() -> Dict[str, Any]:\n \"\"\"操作历史工具定义\"\"\"\n return {\n \"name\": \"get_operation_history\",\n \"description\": \"Get operation history\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"last_n\": {\n \"type\": \"integer\",\n \"description\": \"Return the last N operations\",\n \"default\": 10,\n },\n },\n },\n }\n\n @staticmethod\n def _get_get_indexes_overview_tool() -> Dict[str, Any]:\n \"\"\"获取索引概览工具定义\"\"\"\n return {\n \"name\": \"get_indexes_overview\",\n \"description\": \"Get overview of all available reference code index information from specified directory\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"indexes_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the indexes directory containing JSON index files\",\n }\n },\n \"required\": [\"indexes_path\"],\n },\n }\n\n @staticmethod\n def _get_set_workspace_tool() -> Dict[str, Any]:\n \"\"\"Set workspace directory tool definition\"\"\"\n return {\n \"name\": \"set_workspace\",\n \"description\": \"Set the workspace directory for file operations\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"workspace_path\": {\n \"type\": \"string\",\n \"description\": \"Directory path for the workspace\",\n }\n },\n \"required\": [\"workspace_path\"],\n },\n }\n\n # @staticmethod\n # def _get_set_indexes_directory_tool() -> Dict[str, Any]:\n # \"\"\"Set indexes directory tool definition - DEPRECATED: Use unified search_code_references instead\"\"\"\n # return {\n # \"name\": \"set_indexes_directory\",\n # \"description\": \"Set the directory path for code reference indexes\",\n # \"input_schema\": {\n # \"type\": \"object\",\n # \"properties\": {\n # \"indexes_path\": {\n # \"type\": \"string\",\n # \"description\": \"Directory path containing index JSON files\"\n # }\n # },\n # \"required\": [\"indexes_path\"]\n # }\n # }\n\n # Code evaluation tool definitions\n @staticmethod\n def _get_analyze_repo_structure_tool() -> Dict[str, Any]:\n return {\n \"name\": \"analyze_repo_structure\",\n \"description\": \"Perform comprehensive repository structure analysis\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository to analyze\",\n }\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_detect_dependencies_tool() -> Dict[str, Any]:\n return {\n \"name\": \"detect_dependencies\",\n \"description\": \"Detect and analyze project dependencies across multiple languages\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository\",\n }\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_assess_code_quality_tool() -> Dict[str, Any]:\n return {\n \"name\": \"assess_code_quality\",\n \"description\": \"Assess code quality metrics and identify potential issues\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository\",\n }\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_evaluate_documentation_tool() -> Dict[str, Any]:\n return {\n \"name\": \"evaluate_documentation\",\n \"description\": \"Evaluate documentation completeness and quality\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository\",\n },\n \"docs_path\": {\n \"type\": \"string\",\n \"description\": \"Optional path to external documentation\",\n },\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_check_reproduction_readiness_tool() -> Dict[str, Any]:\n return {\n \"name\": \"check_reproduction_readiness\",\n \"description\": \"Assess repository readiness for reproduction and validation\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository\",\n },\n \"docs_path\": {\n \"type\": \"string\",\n \"description\": \"Optional path to reproduction documentation\",\n },\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_generate_evaluation_summary_tool() -> Dict[str, Any]:\n return {\n \"name\": \"generate_evaluation_summary\",\n \"description\": \"Generate comprehensive evaluation summary combining all analysis results\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository\",\n },\n \"docs_path\": {\n \"type\": \"string\",\n \"description\": \"Optional path to reproduction documentation\",\n },\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_detect_empty_files_tool() -> Dict[str, Any]:\n return {\n \"name\": \"detect_empty_files\",\n \"description\": \"Detect empty files in the repository that may need implementation\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository to analyze\",\n }\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_detect_missing_files_tool() -> Dict[str, Any]:\n return {\n \"name\": \"detect_missing_files\",\n \"description\": \"Detect missing essential files like main programs, tests, requirements, etc.\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository to analyze\",\n }\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def _get_generate_code_revision_report_tool() -> Dict[str, Any]:\n return {\n \"name\": \"generate_code_revision_report\",\n \"description\": \"Generate comprehensive code revision report combining empty files, missing files, and quality analysis\",\n \"input_schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"repo_path\": {\n \"type\": \"string\",\n \"description\": \"Path to the repository to analyze\",\n },\n \"docs_path\": {\n \"type\": \"string\",\n \"description\": \"Optional path to documentation\",\n },\n },\n \"required\": [\"repo_path\"],\n },\n }\n\n @staticmethod\n def get_available_tool_sets() -> Dict[str, str]:\n \"\"\"\n 获取可用的工具集合\n Get available tool sets\n \"\"\"\n return {\n \"code_implementation\": \"代码实现相关工具集 / Code implementation tool set\",\n \"code_evaluation\": \"代码评估相关工具集 / Code evaluation tool set\",\n # 可以在这里添加更多工具集\n # \"data_analysis\": \"数据分析工具集 / Data analysis tool set\",\n # \"web_scraping\": \"网页爬取工具集 / Web scraping tool set\",\n }\n\n @staticmethod\n def get_tool_set(tool_set_name: str) -> List[Dict[str, Any]]:\n \"\"\"\n 根据名称获取特定的工具集\n Get specific tool set by name\n \"\"\"\n tool_sets = {\n \"code_implementation\": MCPToolDefinitions.get_code_implementation_tools(),\n \"code_evaluation\": MCPToolDefinitions.get_code_evaluation_tools(),\n }\n\n return tool_sets.get(tool_set_name, [])\n\n @staticmethod\n def get_all_tools() -> List[Dict[str, Any]]:\n \"\"\"\n 获取所有可用工具\n Get all available tools\n \"\"\"\n all_tools = []\n for tool_set_name in MCPToolDefinitions.get_available_tool_sets().keys():\n all_tools.extend(MCPToolDefinitions.get_tool_set(tool_set_name))\n return all_tools\n\n\n# 便捷访问函数\ndef get_mcp_tools(tool_set: str = \"code_implementation\") -> List[Dict[str, Any]]:\n \"\"\"\n 便捷函数：获取MCP工具定义\n Convenience function: Get MCP tool definitions\n\n Args:\n tool_set: 工具集名称 (默认: \"code_implementation\")\n\n Returns:\n 工具定义列表\n \"\"\"\n return MCPToolDefinitions.get_tool_set(tool_set)\n"
},
{
"path": "deepcode.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nDeepCode - AI Research Engine Launcher\n\n🧬 Next-Generation AI Research Automation Platform\n⚡ Transform research papers into working code automatically\n\nCross-platform support: Windows, macOS, Linux\n\"\"\"\n\nimport os\nimport sys\nimport subprocess\nimport signal\nimport platform\nimport socket\nimport time\nfrom pathlib import Path\n\n\n# Global process references for cleanup\n_backend_process = None\n_frontend_process = None\n\n\ndef get_platform():\n \"\"\"Get current platform\"\"\"\n system = platform.system().lower()\n if system == \"darwin\":\n return \"macos\"\n elif system == \"windows\":\n return \"windows\"\n else:\n return \"linux\"\n\n\ndef check_dependencies():\n \"\"\"Check if necessary dependencies are installed for new UI\"\"\"\n import importlib.util\n import shutil\n\n print(\"🔍 Checking dependencies...\")\n\n missing_deps = []\n missing_system_deps = []\n\n # Check FastAPI availability (for backend)\n if importlib.util.find_spec(\"fastapi\") is not None:\n print(\"✅ FastAPI is installed\")\n else:\n missing_deps.append(\"fastapi>=0.104.0\")\n\n # Check uvicorn availability (for backend server)\n if importlib.util.find_spec(\"uvicorn\") is not None:\n print(\"✅ Uvicorn is installed\")\n else:\n missing_deps.append(\"uvicorn>=0.24.0\")\n\n # Check PyYAML availability\n if importlib.util.find_spec(\"yaml\") is not None:\n print(\"✅ PyYAML is installed\")\n else:\n missing_deps.append(\"pyyaml>=6.0\")\n\n # Check pydantic-settings availability\n if importlib.util.find_spec(\"pydantic_settings\") is not None:\n print(\"✅ Pydantic-settings is installed\")\n else:\n missing_deps.append(\"pydantic-settings>=2.0.0\")\n\n # Check Node.js availability (for frontend)\n node_cmd = \"node.exe\" if get_platform() == \"windows\" else \"node\"\n if shutil.which(node_cmd) or shutil.which(\"node\"):\n try:\n result = subprocess.run(\n [\"node\", \"--version\"],\n capture_output=True,\n text=True,\n timeout=5,\n shell=(get_platform() == \"windows\"),\n )\n if result.returncode == 0:\n print(f\"✅ Node.js is installed ({result.stdout.strip()})\")\n except Exception:\n missing_system_deps.append(\"Node.js\")\n else:\n missing_system_deps.append(\"Node.js\")\n print(\"❌ Node.js not found (required for frontend)\")\n\n # Check npm availability\n npm_cmd = \"npm.cmd\" if get_platform() == \"windows\" else \"npm\"\n if shutil.which(npm_cmd) or shutil.which(\"npm\"):\n print(\"✅ npm is available\")\n else:\n missing_system_deps.append(\"npm\")\n print(\"❌ npm not found (required for frontend)\")\n\n # Display missing dependencies\n if missing_deps or missing_system_deps:\n print(\"\\n📋 Dependency Status:\")\n\n if missing_deps:\n print(\"❌ Missing Python dependencies:\")\n for dep in missing_deps:\n print(f\" - {dep}\")\n print(f\"\\nInstall with: pip install {' '.join(missing_deps)}\")\n\n if missing_system_deps:\n print(\"\\n❌ Missing system dependencies:\")\n for dep in missing_system_deps:\n print(f\" - {dep}\")\n print(\"\\nInstall Node.js:\")\n print(\" - Windows/macOS: https://nodejs.org/\")\n print(\" - macOS: brew install node\")\n print(\" - Ubuntu/Debian: sudo apt-get install nodejs npm\")\n\n # Fail if critical dependencies are missing\n if missing_deps or missing_system_deps:\n return False\n else:\n print(\"✅ All dependencies satisfied\")\n\n return True\n\n\ndef is_port_in_use(port: int) -> bool:\n \"\"\"Check if a port is in use (cross-platform)\"\"\"\n with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:\n return s.connect_ex((\"localhost\", port)) == 0\n\n\ndef kill_process_on_port(port: int):\n \"\"\"Kill process using a specific port (cross-platform)\"\"\"\n current_platform = get_platform()\n\n try:\n if current_platform == \"windows\":\n # Windows: use netstat and taskkill\n result = subprocess.run(\n f\"netstat -ano | findstr :{port}\",\n capture_output=True,\n text=True,\n shell=True,\n )\n if result.stdout:\n for line in result.stdout.strip().split(\"\\n\"):\n parts = line.split()\n if len(parts) >= 5:\n pid = parts[-1]\n if pid.isdigit():\n subprocess.run(\n f\"taskkill /F /PID {pid}\",\n shell=True,\n capture_output=True,\n )\n print(f\" ✓ Killed process on port {port} (PID: {pid})\")\n else:\n # macOS/Linux: use lsof\n result = subprocess.run(\n f\"lsof -ti :{port}\", capture_output=True, text=True, shell=True\n )\n if result.stdout:\n pids = result.stdout.strip().split(\"\\n\")\n for pid in pids:\n if pid.isdigit():\n os.kill(int(pid), signal.SIGKILL)\n print(f\" ✓ Killed process on port {port} (PID: {pid})\")\n except Exception as e:\n print(f\" ⚠️ Could not kill process on port {port}: {e}\")\n\n\ndef cleanup_ports():\n \"\"\"Clean up ports 8000 and 5173 if in use\"\"\"\n for port in [8000, 5173]:\n if is_port_in_use(port):\n print(f\"⚠️ Port {port} is in use, cleaning up...\")\n kill_process_on_port(port)\n time.sleep(1)\n\n\ndef install_backend_deps():\n \"\"\"Install backend dependencies if needed\"\"\"\n import importlib.util\n\n if importlib.util.find_spec(\"fastapi\") is None:\n print(\"📦 Installing backend dependencies...\")\n deps = [\n \"fastapi\",\n \"uvicorn\",\n \"pydantic-settings\",\n \"python-multipart\",\n \"aiofiles\",\n \"websockets\",\n \"pyyaml\",\n ]\n subprocess.run(\n [sys.executable, \"-m\", \"pip\", \"install\", \"-q\"] + deps, check=True\n )\n print(\"✅ Backend dependencies installed\")\n\n\ndef install_frontend_deps(frontend_dir: Path):\n \"\"\"Install frontend dependencies if needed\"\"\"\n node_modules = frontend_dir / \"node_modules\"\n\n if not node_modules.exists():\n print(\"📦 Installing frontend dependencies (first run)...\")\n npm_cmd = \"npm.cmd\" if get_platform() == \"windows\" else \"npm\"\n subprocess.run(\n [npm_cmd, \"install\"],\n cwd=frontend_dir,\n check=True,\n shell=(get_platform() == \"windows\"),\n )\n print(\"✅ Frontend dependencies installed\")\n\n\ndef start_backend(backend_dir: Path):\n \"\"\"Start the backend server\"\"\"\n global _backend_process\n\n print(\"🔧 Starting backend server...\")\n\n # Use shell=True on Windows for proper command handling\n if get_platform() == \"windows\":\n _backend_process = subprocess.Popen(\n f'\"{sys.executable}\" -m uvicorn main:app --host 0.0.0.0 --port 8000 --reload',\n cwd=backend_dir,\n shell=True,\n creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,\n )\n else:\n _backend_process = subprocess.Popen(\n [\n sys.executable,\n \"-m\",\n \"uvicorn\",\n \"main:app\",\n \"--host\",\n \"0.0.0.0\",\n \"--port\",\n \"8000\",\n \"--reload\",\n ],\n cwd=backend_dir,\n start_new_session=True, # Create new process group\n )\n\n # Wait for backend to start\n time.sleep(2)\n\n if _backend_process.poll() is None:\n print(\"✅ Backend started: http://localhost:8000\")\n return True\n else:\n print(\"❌ Backend failed to start\")\n return False\n\n\ndef start_frontend(frontend_dir: Path):\n \"\"\"Start the frontend dev server\"\"\"\n global _frontend_process\n\n print(\"🎨 Starting frontend server...\")\n\n npm_cmd = \"npm.cmd\" if get_platform() == \"windows\" else \"npm\"\n\n if get_platform() == \"windows\":\n _frontend_process = subprocess.Popen(\n f\"{npm_cmd} run dev\",\n cwd=frontend_dir,\n shell=True,\n creationflags=subprocess.CREATE_NEW_PROCESS_GROUP,\n )\n else:\n _frontend_process = subprocess.Popen(\n [npm_cmd, \"run\", \"dev\"],\n cwd=frontend_dir,\n start_new_session=True, # Create new process group\n )\n\n # Wait for frontend to start\n time.sleep(3)\n\n if _frontend_process.poll() is None:\n print(\"✅ Frontend started: http://localhost:5173\")\n return True\n else:\n print(\"❌ Frontend failed to start\")\n return False\n\n\ndef cleanup_processes():\n \"\"\"Clean up running processes\"\"\"\n global _backend_process, _frontend_process\n\n print(\"\\n🛑 Stopping services...\")\n\n for name, proc in [(\"Backend\", _backend_process), (\"Frontend\", _frontend_process)]:\n if proc and proc.poll() is None:\n try:\n if get_platform() == \"windows\":\n # Windows: use taskkill with /T to kill tree\n subprocess.run(\n f\"taskkill /F /T /PID {proc.pid}\",\n shell=True,\n capture_output=True,\n )\n else:\n # Unix: kill the process group\n try:\n os.killpg(os.getpgid(proc.pid), signal.SIGTERM)\n proc.wait(timeout=5)\n except Exception:\n os.killpg(os.getpgid(proc.pid), signal.SIGKILL)\n print(f\" ✓ {name} stopped\")\n except Exception:\n # Fallback: try direct terminate\n try:\n proc.terminate()\n proc.wait(timeout=3)\n print(f\" ✓ {name} stopped\")\n except Exception:\n try:\n proc.kill()\n print(f\" ✓ {name} killed\")\n except Exception:\n print(f\" ⚠️ Could not stop {name}\")\n\n # Also clean up any orphaned processes on ports\n time.sleep(0.5)\n for port in [8000, 5173]:\n if is_port_in_use(port):\n kill_process_on_port(port)\n\n print(\"✅ All services stopped\")\n\n\ndef cleanup_cache():\n \"\"\"Clean up Python cache files\"\"\"\n try:\n print(\"🧹 Cleaning up cache files...\")\n # Clean up __pycache__ directories\n os.system('find . -type d -name \"__pycache__\" -exec rm -r {} + 2>/dev/null')\n # Clean up .pyc files\n os.system('find . -name \"*.pyc\" -delete 2>/dev/null')\n print(\"✅ Cache cleanup completed\")\n except Exception as e:\n print(f\"⚠️ Cache cleanup failed: {e}\")\n\n\ndef print_banner():\n \"\"\"Display startup banner\"\"\"\n banner = \"\"\"\n╔══════════════════════════════════════════════════════════════╗\n║ ║\n║ 🧬 DeepCode - AI Research Engine ║\n║ ║\n║ ⚡ NEURAL • AUTONOMOUS • REVOLUTIONARY ⚡ ║\n║ ║\n║ Transform research papers into working code ║\n║ Next-generation AI automation platform ║\n║ ║\n╚══════════════════════════════════════════════════════════════╝\n\"\"\"\n print(banner)\n\n\ndef launch_classic_ui():\n \"\"\"Launch classic Streamlit UI\"\"\"\n import importlib.util\n\n print(\"🌐 Launching Classic Streamlit UI...\")\n\n # Check if Streamlit is installed\n if importlib.util.find_spec(\"streamlit\") is None:\n print(\"❌ Streamlit is not installed.\")\n print(\"Install with: pip install streamlit\")\n sys.exit(1)\n\n current_dir = Path(__file__).parent\n streamlit_app_path = current_dir / \"ui\" / \"streamlit_app.py\"\n\n if not streamlit_app_path.exists():\n print(f\"❌ Streamlit app not found: {streamlit_app_path}\")\n sys.exit(1)\n\n print(f\"📁 UI App: {streamlit_app_path}\")\n print(\"🚀 Launching on http://localhost:8501\")\n print(\"=\" * 70)\n\n try:\n cmd = [\n sys.executable,\n \"-m\",\n \"streamlit\",\n \"run\",\n str(streamlit_app_path),\n \"--server.port\",\n \"8501\",\n \"--server.address\",\n \"localhost\",\n \"--browser.gatherUsageStats\",\n \"false\",\n ]\n subprocess.run(cmd, check=True)\n except KeyboardInterrupt:\n print(\"\\n\\n🛑 Streamlit server stopped by user\")\n except Exception as e:\n print(f\"\\n❌ Error: {e}\")\n sys.exit(1)\n\n\ndef _check_docker_prerequisites():\n \"\"\"Check Docker prerequisites and config files. Returns (current_dir, compose_file, compose_args).\"\"\"\n import shutil\n\n current_dir = Path(__file__).parent\n compose_file = current_dir / \"deepcode_docker\" / \"docker-compose.yml\"\n\n if not compose_file.exists():\n print(\"❌ deepcode_docker/docker-compose.yml not found\")\n print(\" Make sure you are running from the DeepCode project root.\")\n sys.exit(1)\n\n # Check Docker is installed\n if not shutil.which(\"docker\"):\n print(\"❌ Docker not found. Please install Docker Desktop first.\")\n print(\" https://www.docker.com/products/docker-desktop\")\n sys.exit(1)\n\n # Check Docker daemon is running\n result = subprocess.run([\"docker\", \"info\"], capture_output=True, text=True)\n if result.returncode != 0:\n print(\"❌ Docker is installed but not running.\")\n print(\" Please start Docker Desktop and try again.\")\n sys.exit(1)\n\n # Check/create secrets file\n secrets_file = current_dir / \"mcp_agent.secrets.yaml\"\n if not secrets_file.exists():\n example = current_dir / \"mcp_agent.secrets.yaml.example\"\n if example.exists():\n print(\"⚠️ mcp_agent.secrets.yaml not found.\")\n print(\" Creating from template...\")\n import shutil as sh\n\n sh.copy2(example, secrets_file)\n print(f\" ✅ Created {secrets_file}\")\n print(\"\")\n print(\" ⚠️ Please edit mcp_agent.secrets.yaml and fill in your API keys:\")\n print(f\" {secrets_file}\")\n print(\"\")\n print(\n \" At least ONE LLM provider key is required (OpenAI/Anthropic/Google).\"\n )\n print(\" Then run 'deepcode' again.\")\n sys.exit(0)\n else:\n print(\n \"❌ mcp_agent.secrets.yaml not found. Please create it with your API keys.\"\n )\n sys.exit(1)\n\n # Check config file\n config_file = current_dir / \"mcp_agent.config.yaml\"\n if not config_file.exists():\n print(\"❌ mcp_agent.config.yaml not found.\")\n print(\" This file should be in the project root.\")\n sys.exit(1)\n\n # Ensure data directories exist\n for d in [\"deepcode_lab\", \"uploads\", \"logs\"]:\n (current_dir / d).mkdir(exist_ok=True)\n\n os.chdir(current_dir)\n compose_args = [\"docker\", \"compose\", \"-f\", str(compose_file)]\n\n return current_dir, compose_file, compose_args\n\n\ndef launch_docker():\n \"\"\"Launch DeepCode via Docker\"\"\"\n current_dir, compose_file, compose_args = _check_docker_prerequisites()\n\n print(\"🐳 Starting DeepCode with Docker...\")\n print(\"=\" * 50)\n\n try:\n # Check if image exists (auto-build on first run)\n result = subprocess.run(\n compose_args + [\"images\", \"-q\"], capture_output=True, text=True\n )\n if not result.stdout.strip():\n print(\n \"📦 First run detected — building Docker image (may take a few minutes)...\"\n )\n subprocess.run(compose_args + [\"build\"], check=True)\n\n # Start (if already running, docker compose will detect and skip)\n subprocess.run(compose_args + [\"up\", \"-d\"], check=True)\n\n print(\"\")\n print(\"=\" * 50)\n print(\"✅ DeepCode is running!\")\n print(\"\")\n print(\" 🌐 Open: http://localhost:8000\")\n print(\" 📚 Docs: http://localhost:8000/docs\")\n print(\"\")\n print(\" 📋 View logs: docker logs deepcode -f\")\n print(\n \" 🛑 Stop: docker compose -f deepcode_docker/docker-compose.yml down\"\n )\n print(\"=\" * 50)\n\n except subprocess.CalledProcessError as e:\n print(f\"\\n❌ Docker failed: {e}\")\n sys.exit(1)\n except KeyboardInterrupt:\n print(\"\\n🛑 Cancelled\")\n\n\ndef launch_docker_cli():\n \"\"\"Launch DeepCode CLI inside Docker container\"\"\"\n current_dir, compose_file, compose_args = _check_docker_prerequisites()\n\n print(\"🖥️ Starting DeepCode CLI in Docker...\")\n print(\"=\" * 50)\n\n try:\n # Check if image exists (auto-build on first run)\n result = subprocess.run(\n compose_args + [\"images\", \"-q\"], capture_output=True, text=True\n )\n if not result.stdout.strip():\n print(\n \"📦 First run detected — building Docker image (may take a few minutes)...\"\n )\n subprocess.run(compose_args + [\"build\"], check=True)\n\n # Run CLI interactively\n subprocess.run(\n compose_args + [\"run\", \"--rm\", \"-it\", \"deepcode\", \"cli\"], check=True\n )\n\n except subprocess.CalledProcessError as e:\n print(f\"\\n❌ Docker failed: {e}\")\n sys.exit(1)\n except KeyboardInterrupt:\n print(\"\\n🛑 Cancelled\")\n\n\ndef launch_paper_test(paper_name: str, fast_mode: bool = False):\n \"\"\"Launch paper testing mode\"\"\"\n try:\n print(\"\\n🧪 Launching Paper Test Mode\")\n print(f\"📄 Paper: {paper_name}\")\n print(f\"⚡ Fast mode: {'enabled' if fast_mode else 'disabled'}\")\n print(\"=\" * 60)\n\n # Run the test setup\n setup_cmd = [sys.executable, \"test_paper.py\", paper_name]\n if fast_mode:\n setup_cmd.append(\"--fast\")\n\n result = subprocess.run(setup_cmd, check=True)\n\n if result.returncode == 0:\n print(\"\\n✅ Paper test setup completed successfully!\")\n print(\"📁 Files are ready in deepcode_lab/papers/\")\n print(\"\\n💡 Next steps:\")\n print(\" 1. Install MCP dependencies: pip install -r requirements.txt\")\n print(\n f\" 2. Run full pipeline: python -m workflows.paper_test_engine --paper {paper_name}\"\n + (\" --fast\" if fast_mode else \"\")\n )\n\n except subprocess.CalledProcessError as e:\n print(f\"\\n❌ Paper test setup failed: {e}\")\n sys.exit(1)\n except Exception as e:\n print(f\"\\n❌ Unexpected error: {e}\")\n sys.exit(1)\n\n\ndef main():\n \"\"\"Main function\"\"\"\n # Parse command line arguments\n if len(sys.argv) > 1:\n if sys.argv[1] == \"test\" and len(sys.argv) >= 3:\n # Paper testing mode: python deepcode.py test rice [--fast]\n paper_name = sys.argv[2]\n fast_mode = \"--fast\" in sys.argv or \"-f\" in sys.argv\n\n print_banner()\n launch_paper_test(paper_name, fast_mode)\n return\n elif sys.argv[1] == \"--local\":\n # Launch locally (without Docker) — fall through to local launch below\n print_banner()\n pass\n elif sys.argv[1] == \"--docker\":\n # Explicit Docker launch (same as default)\n print_banner()\n launch_docker()\n return\n elif sys.argv[1] == \"--cli\":\n # Launch CLI inside Docker container\n print_banner()\n launch_docker_cli()\n return\n elif sys.argv[1] == \"--classic\":\n # Launch classic Streamlit UI\n print_banner()\n launch_classic_ui()\n return\n elif sys.argv[1] in [\"--help\", \"-h\", \"help\"]:\n print_banner()\n print(\"\"\"\n🔧 Usage:\n deepcode - Launch via Docker (default, recommended)\n deepcode --docker - Same as above (launch via Docker)\n deepcode --cli - Launch interactive CLI in Docker\n deepcode --local - Launch locally (requires Python + Node.js)\n deepcode test - Test paper reproduction\n deepcode test --fast - Test paper (fast mode)\n deepcode --classic - Launch classic Streamlit UI\n\n📄 Examples:\n deepcode - Start with Docker (one command)\n deepcode --cli - Interactive CLI in Docker\n deepcode --local - Start the new UI locally\n deepcode test rice - Test RICE paper reproduction\n deepcode test rice --fast - Test RICE paper (fast mode)\n\n🌐 New UI Features:\n • User-in-Loop interaction\n • Real-time progress tracking\n • Inline chat interaction\n • Modern React-based interface\n\n📁 Available papers:\"\"\")\n\n # List available papers\n papers_dir = \"papers\"\n if os.path.exists(papers_dir):\n for item in os.listdir(papers_dir):\n item_path = os.path.join(papers_dir, item)\n if os.path.isdir(item_path):\n paper_md = os.path.join(item_path, \"paper.md\")\n addendum_md = os.path.join(item_path, \"addendum.md\")\n status = \"✅\" if os.path.exists(paper_md) else \"❌\"\n addendum_status = \"📄\" if os.path.exists(addendum_md) else \"➖\"\n print(f\" {status} {item} {addendum_status}\")\n print(\n \"\\n Legend: ✅ = paper.md exists, 📄 = addendum.md exists, ➖ = no addendum\"\n )\n return\n else:\n # Unknown argument — show help hint\n print(f\"Unknown option: {sys.argv[1]}\")\n print(\"Run 'deepcode --help' for usage information.\")\n sys.exit(1)\n else:\n # Default (no arguments) → Docker\n print_banner()\n launch_docker()\n return\n\n # --- Local launch (only reached via --local) ---\n\n # Show platform info\n current_platform = get_platform()\n print(f\"🖥️ Platform: {current_platform.capitalize()}\")\n\n # Check dependencies\n if not check_dependencies():\n print(\"\\n🚨 Please install missing dependencies and try again.\")\n sys.exit(1)\n\n # Get paths\n current_dir = Path(__file__).parent\n new_ui_dir = current_dir / \"new_ui\"\n backend_dir = new_ui_dir / \"backend\"\n frontend_dir = new_ui_dir / \"frontend\"\n\n # Check if new_ui directory exists\n if not new_ui_dir.exists():\n print(f\"❌ New UI directory not found: {new_ui_dir}\")\n sys.exit(1)\n\n print(\"\\n🚀 Starting DeepCode New UI...\")\n print(\"=\" * 70)\n print(\"🎨 Frontend: http://localhost:5173\")\n print(\"🔧 Backend: http://localhost:8000\")\n print(\"📚 API Docs: http://localhost:8000/docs\")\n print(\"=\" * 70)\n print(\"💡 Tip: Keep this terminal open while using the application\")\n print(\"🛑 Press Ctrl+C to stop all services\")\n print(\"=\" * 70)\n\n try:\n # Clean up ports if in use\n cleanup_ports()\n\n # Install dependencies if needed\n install_backend_deps()\n install_frontend_deps(frontend_dir)\n\n # Start services\n if not start_backend(backend_dir):\n print(\"❌ Failed to start backend\")\n sys.exit(1)\n\n if not start_frontend(frontend_dir):\n print(\"❌ Failed to start frontend\")\n cleanup_processes()\n sys.exit(1)\n\n print(\"\\n\" + \"=\" * 70)\n print(\"╔════════════════════════════════════════╗\")\n print(\"║ 🎉 DeepCode New UI is running! ║\")\n print(\"╠════════════════════════════════════════╣\")\n print(\"║ ║\")\n print(\"║ 🌐 Frontend: http://localhost:5173 ║\")\n print(\"║ 🔧 Backend: http://localhost:8000 ║\")\n print(\"║ 📚 API Docs: http://localhost:8000/docs║\")\n print(\"║ ║\")\n print(\"║ Press Ctrl+C to stop all services ║\")\n print(\"╚════════════════════════════════════════╝\")\n print(\"=\" * 70 + \"\\n\")\n\n # Wait for processes\n while True:\n # Check if processes are still running\n if _backend_process and _backend_process.poll() is not None:\n print(\"⚠️ Backend process exited unexpectedly\")\n break\n if _frontend_process and _frontend_process.poll() is not None:\n print(\"⚠️ Frontend process exited unexpectedly\")\n break\n time.sleep(1)\n\n except KeyboardInterrupt:\n print(\"\\n\")\n except Exception as e:\n print(f\"\\n❌ Unexpected error: {e}\")\n finally:\n cleanup_processes()\n cleanup_cache()\n print(\"Thank you for using DeepCode! 🧬\")\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "deepcode_docker/.dockerignore",
"content": "# Git\n.git\n.gitignore\n\n# Node\nnew_ui/frontend/node_modules\nnew_ui/frontend/dist\n\n# Python\n__pycache__\n*.pyc\n*.pyo\n*.egg-info\n.eggs\ndist\nbuild\n\n# Virtual environments\n.venv\nvenv\nenv\n\n# IDE\n.vscode\n.idea\n.cursor\n*.swp\n*.swo\n\n# Runtime data\ndeepcode_lab\nuploads\nlogs\n*.log\n\n# Docker\ndeepcode_docker/Dockerfile\ndeepcode_docker/docker-compose.yml\ndeepcode_docker/.dockerignore\ndeepcode_docker/run_docker.sh\n\n# Documentation\nassets\n*.md\nLICENSE\n"
},
{
"path": "deepcode_docker/Dockerfile",
"content": "# =============================================================\n# DeepCode - Docker Build\n# Multi-stage: Frontend build → Final image with Python + Node\n# =============================================================\n\n# ------ Stage 1: Build frontend static assets ------\nFROM node:18-alpine AS frontend-builder\n\nWORKDIR /build\nCOPY new_ui/frontend/package*.json ./\nRUN npm ci --no-audit --no-fund\nCOPY new_ui/frontend/ ./\nRUN npm run build\n\n\n# ------ Stage 2: Final image ------\nFROM python:3.10-slim\n\n# Metadata\nLABEL maintainer=\"DeepCode Team\"\nLABEL description=\"DeepCode - AI Research Engine\"\nLABEL version=\"1.0\"\n\n# Environment\nENV PYTHONDONTWRITEBYTECODE=1 \\\n PYTHONUNBUFFERED=1 \\\n DEEPCODE_ENV=docker \\\n DEEPCODE_HOST=0.0.0.0 \\\n DEEPCODE_PORT=8000\n\n# Install system dependencies:\n# - git: for git clone operations in workflows\n# - nodejs/npm/npx: for MCP servers (brave-search, filesystem, fetch)\n# - curl: for health checks\nRUN apt-get update && \\\n apt-get install -y --no-install-recommends \\\n git \\\n curl \\\n ca-certificates && \\\n # Install Node.js 18 via official binary (includes npm + npx)\n ARCH=$(dpkg --print-architecture) && \\\n if [ \"$ARCH\" = \"arm64\" ]; then NODE_ARCH=\"arm64\"; else NODE_ARCH=\"x64\"; fi && \\\n curl -fsSL https://nodejs.org/dist/v18.20.8/node-v18.20.8-linux-${NODE_ARCH}.tar.gz \\\n | tar -xz -C /usr/local --strip-components=1 && \\\n # Install uv (Python package installer, used by mcp-server-fetch)\n pip install --no-cache-dir uv && \\\n # Cleanup\n apt-get clean && \\\n rm -rf /var/lib/apt/lists/* && \\\n # Verify\n node --version && npm --version && npx --version\n\nWORKDIR /app\n\n# Install Python dependencies first (cache layer)\nCOPY requirements.txt .\nRUN pip install --no-cache-dir -r requirements.txt\n\n# Pre-install npx MCP server packages (avoid download at runtime)\nRUN npx -y @modelcontextprotocol/server-brave-search --help 2>/dev/null || true && \\\n npx -y @modelcontextprotocol/server-filesystem --help 2>/dev/null || true\n\n# Copy project source code\nCOPY __init__.py setup.py deepcode.py ./\nCOPY config/ ./config/\nCOPY prompts/ ./prompts/\nCOPY schema/ ./schema/\nCOPY tools/ ./tools/\nCOPY utils/ ./utils/\nCOPY workflows/ ./workflows/\nCOPY cli/ ./cli/\nCOPY ui/ ./ui/\nCOPY new_ui/backend/ ./new_ui/backend/\n\n# Copy frontend build output from Stage 1\nCOPY --from=frontend-builder /build/dist ./new_ui/frontend/dist\n\n# Create runtime directories\nRUN mkdir -p deepcode_lab uploads logs\n\n# Copy entrypoint script\nCOPY deepcode_docker/docker-entrypoint.sh /docker-entrypoint.sh\nRUN chmod +x /docker-entrypoint.sh\n\nEXPOSE 8000\n\nHEALTHCHECK --interval=30s --timeout=10s --start-period=15s --retries=3 \\\n CMD curl -f http://localhost:8000/health || exit 1\n\nENTRYPOINT [\"/docker-entrypoint.sh\"]\n"
},
{
"path": "deepcode_docker/docker-compose.yml",
"content": "services:\n deepcode:\n build:\n context: ..\n dockerfile: deepcode_docker/Dockerfile\n container_name: deepcode\n ports:\n - \"8000:8000\"\n volumes:\n # Configuration (required)\n - ../mcp_agent.config.yaml:/app/mcp_agent.config.yaml:ro\n - ../mcp_agent.secrets.yaml:/app/mcp_agent.secrets.yaml:ro\n\n # Persistent data\n - ../deepcode_lab:/app/deepcode_lab\n - ../uploads:/app/uploads\n - ../logs:/app/logs\n environment:\n - DEEPCODE_ENV=docker\n - DEEPCODE_PORT=8000\n restart: unless-stopped\n\n nanobot:\n build:\n context: ..\n dockerfile: nanobot/Dockerfile\n container_name: nanobot\n ports:\n - \"18790:18790\"\n volumes:\n # nanobot configuration (飞书/Telegram token 等)\n - ../nanobot_config.json:/root/.nanobot/config.json:ro\n\n # Persistent workspace data\n - nanobot-workspace:/root/.nanobot/workspace\n - nanobot-sessions:/root/.nanobot/sessions\n\n # Shared with DeepCode: nanobot can access generated code\n - ../deepcode_lab:/app/deepcode_lab\n environment:\n - NANOBOT_ENV=docker\n # Internal API URL for nanobot -> DeepCode communication\n - DEEPCODE_API_URL=http://deepcode:8000\n depends_on:\n - deepcode\n restart: unless-stopped\n\nvolumes:\n nanobot-workspace:\n nanobot-sessions:\n"
},
{
"path": "deepcode_docker/docker-entrypoint.sh",
"content": "#!/bin/bash\nset -e\n\necho \"============================================\"\necho \" DeepCode - AI Research Engine (Docker)\"\necho \"============================================\"\n\n# ------ Validate configuration ------\nif [ ! -f \"mcp_agent.config.yaml\" ]; then\n echo \"⚠️ mcp_agent.config.yaml not found, using default config\"\nfi\n\nif [ ! -f \"mcp_agent.secrets.yaml\" ]; then\n echo \"\"\n echo \"❌ ERROR: mcp_agent.secrets.yaml not found!\"\n echo \"\"\n echo \"Please mount your secrets file:\"\n echo \" docker run -v ./mcp_agent.secrets.yaml:/app/mcp_agent.secrets.yaml ...\"\n echo \"\"\n echo \"Or use docker-compose with the provided template.\"\n echo \"\"\n exit 1\nfi\n\n# ------ Ensure directories exist ------\nmkdir -p deepcode_lab uploads logs\n\n# ------ CLI mode: launch interactive CLI ------\nif [ \"$1\" = \"cli\" ]; then\n shift\n echo \"\"\n echo \"🖥️ Starting DeepCode CLI...\"\n echo \"============================================\"\n echo \"\"\n exec python cli/main_cli.py \"$@\"\nfi\n\n# ------ Web mode (default): start backend + frontend ------\necho \"\"\necho \"🚀 Starting DeepCode...\"\necho \" API: http://localhost:${DEEPCODE_PORT:-8000}\"\necho \" Docs: http://localhost:${DEEPCODE_PORT:-8000}/docs\"\necho \"============================================\"\necho \"\"\n\nexec python -m uvicorn new_ui.backend.main:app \\\n --host \"${DEEPCODE_HOST:-0.0.0.0}\" \\\n --port \"${DEEPCODE_PORT:-8000}\" \\\n --workers 1 \\\n --log-level info\n"
},
{
"path": "deepcode_docker/run_docker.sh",
"content": "#!/bin/bash\n# DeepCode Docker 一键启动脚本\n\nset -e\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nPROJECT_ROOT=\"$(cd \"$SCRIPT_DIR/..\" && pwd)\"\nCOMPOSE_FILE=\"$SCRIPT_DIR/docker-compose.yml\"\n\n# 颜色定义\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nBLUE='\\033[0;34m'\nYELLOW='\\033[1;33m'\nCYAN='\\033[0;36m'\nNC='\\033[0m'\n\n# docker compose wrapper — always use the correct compose file\ndc() {\n docker compose -f \"$COMPOSE_FILE\" \"$@\"\n}\n\necho \"\"\necho \"╔════════════════════════════════════════╗\"\necho \"║ DeepCode - Docker 启动脚本 ║\"\necho \"╚════════════════════════════════════════╝\"\necho \"\"\n\n# ============ 检查 Docker 环境 ============\ncheck_docker() {\n if ! command -v docker &> /dev/null; then\n echo -e \"${RED}❌ 未检测到 Docker，请先安装 Docker Desktop${NC}\"\n echo \" 下载地址: https://www.docker.com/products/docker-desktop\"\n exit 1\n fi\n\n if ! docker info &> /dev/null 2>&1; then\n echo -e \"${RED}❌ Docker 服务未运行，请先启动 Docker Desktop${NC}\"\n exit 1\n fi\n\n echo -e \"${GREEN}✓ Docker 环境正常${NC}\"\n}\n\n# ============ 检查配置文件 ============\ncheck_config() {\n if [ ! -f \"$PROJECT_ROOT/mcp_agent.config.yaml\" ]; then\n echo -e \"${RED}❌ 缺少 mcp_agent.config.yaml 配置文件${NC}\"\n exit 1\n fi\n echo -e \"${GREEN}✓ mcp_agent.config.yaml 已找到${NC}\"\n\n if [ ! -f \"$PROJECT_ROOT/mcp_agent.secrets.yaml\" ]; then\n if [ -f \"$PROJECT_ROOT/mcp_agent.secrets.yaml.example\" ]; then\n echo -e \"${YELLOW}⚠ 未找到 mcp_agent.secrets.yaml${NC}\"\n echo -e \"${YELLOW} 正在从模板创建...${NC}\"\n cp \"$PROJECT_ROOT/mcp_agent.secrets.yaml.example\" \"$PROJECT_ROOT/mcp_agent.secrets.yaml\"\n echo -e \"${YELLOW} ⚡ 请编辑 mcp_agent.secrets.yaml 填入你的 API Key，然后重新运行此脚本${NC}\"\n exit 1\n else\n echo -e \"${RED}❌ 缺少 mcp_agent.secrets.yaml，且未找到模板文件${NC}\"\n exit 1\n fi\n fi\n echo -e \"${GREEN}✓ mcp_agent.secrets.yaml 已找到${NC}\"\n}\n\n# ============ 创建必要目录 ============\nensure_dirs() {\n mkdir -p \"$PROJECT_ROOT/deepcode_lab\" \"$PROJECT_ROOT/uploads\" \"$PROJECT_ROOT/logs\"\n echo -e \"${GREEN}✓ 数据目录已就绪 (deepcode_lab/, uploads/, logs/)${NC}\"\n}\n\n# ============ 解析命令行参数 ============\nACTION=\"up\"\nBUILD_FLAG=\"\"\nDETACH_FLAG=\"\"\n\nusage() {\n echo \"用法: $0 [选项]\"\n echo \"\"\n echo \"选项:\"\n echo \" --build 强制重新构建镜像\"\n echo \" -d, --detach 后台运行（不占用终端）\"\n echo \" stop 停止容器\"\n echo \" restart 重启容器\"\n echo \" logs 查看容器日志\"\n echo \" status 查看容器状态\"\n echo \" cli 在 Docker 容器内启动交互式 CLI\"\n echo \" clean 停止并删除容器和镜像\"\n echo \" -h, --help 显示帮助信息\"\n echo \"\"\n echo \"示例:\"\n echo \" $0 # 构建并启动（首次会自动构建）\"\n echo \" $0 --build # 强制重新构建后启动\"\n echo \" $0 -d # 后台启动\"\n echo \" $0 stop # 停止服务\"\n echo \" $0 logs # 查看实时日志\"\n echo \" $0 cli # 启动交互式 CLI\"\n echo \" $0 clean # 完全清理\"\n}\n\nwhile [[ $# -gt 0 ]]; do\n case $1 in\n --build)\n BUILD_FLAG=\"--build\"\n shift\n ;;\n -d|--detach)\n DETACH_FLAG=\"-d\"\n shift\n ;;\n stop)\n ACTION=\"stop\"\n shift\n ;;\n restart)\n ACTION=\"restart\"\n shift\n ;;\n logs)\n ACTION=\"logs\"\n shift\n ;;\n status)\n ACTION=\"status\"\n shift\n ;;\n clean)\n ACTION=\"clean\"\n shift\n ;;\n cli)\n ACTION=\"cli\"\n shift\n break # Remaining args passed to CLI\n ;;\n -h|--help)\n usage\n exit 0\n ;;\n *)\n echo -e \"${RED}未知参数: $1${NC}\"\n usage\n exit 1\n ;;\n esac\ndone\n\n# ============ 执行操作 ============\ncase $ACTION in\n up)\n check_docker\n check_config\n ensure_dirs\n\n echo \"\"\n echo -e \"${BLUE}🐳 启动 DeepCode Docker 容器...${NC}\"\n\n # 检查镜像是否存在，首次运行自动构建\n if [ -z \"$BUILD_FLAG\" ]; then\n if ! docker images | grep -q \"deepcode\"; then\n echo -e \"${YELLOW}⚡ 首次运行，自动构建镜像（可能需要几分钟）...${NC}\"\n BUILD_FLAG=\"--build\"\n fi\n fi\n\n dc up $BUILD_FLAG $DETACH_FLAG\n\n if [ -n \"$DETACH_FLAG\" ]; then\n # 后台模式，等待容器启动后显示信息\n echo \"\"\n echo -e \"${YELLOW}⏳ 等待服务启动...${NC}\"\n for i in $(seq 1 30); do\n if curl -sf http://localhost:8000/health > /dev/null 2>&1; then\n echo \"\"\n echo \"╔════════════════════════════════════════╗\"\n echo -e \"║ ${GREEN}DeepCode 已启动! (Docker)${NC} ║\"\n echo \"╠════════════════════════════════════════╣\"\n echo \"║ ║\"\n echo \"║ 🌐 访问: http://localhost:8000 ║\"\n echo \"║ 📚 API: http://localhost:8000/docs ║\"\n echo \"║ ║\"\n echo \"║ 查看日志: $0 logs ║\"\n echo \"║ 停止服务: $0 stop ║\"\n echo \"╚════════════════════════════════════════╝\"\n echo \"\"\n exit 0\n fi\n sleep 2\n done\n echo -e \"${YELLOW}⚠ 服务仍在启动中，请稍后访问 http://localhost:8000${NC}\"\n echo -e \" 使用 ${CYAN}$0 logs${NC} 查看启动日志\"\n fi\n ;;\n\n stop)\n check_docker\n echo -e \"${BLUE}🛑 停止 DeepCode 容器...${NC}\"\n dc down\n echo -e \"${GREEN}✓ 服务已停止${NC}\"\n ;;\n\n restart)\n check_docker\n echo -e \"${BLUE}🔄 重启 DeepCode 容器...${NC}\"\n dc down\n dc up -d $BUILD_FLAG\n echo -e \"${GREEN}✓ 服务已重启${NC}\"\n echo -e \" 访问: http://localhost:8000\"\n ;;\n\n logs)\n check_docker\n echo -e \"${BLUE}📋 DeepCode 容器日志 (Ctrl+C 退出):${NC}\"\n echo \"\"\n dc logs -f\n ;;\n\n status)\n check_docker\n echo -e \"${BLUE}📊 DeepCode 容器状态:${NC}\"\n echo \"\"\n dc ps\n echo \"\"\n # 检查健康状态\n if curl -sf http://localhost:8000/health > /dev/null 2>&1; then\n echo -e \"${GREEN}✓ 服务运行正常 (http://localhost:8000)${NC}\"\n else\n echo -e \"${YELLOW}⚠ 服务未响应或未启动${NC}\"\n fi\n ;;\n\n cli)\n check_docker\n check_config\n ensure_dirs\n echo \"\"\n echo -e \"${BLUE}🖥️ 启动 DeepCode CLI (Docker)...${NC}\"\n echo \"\"\n dc run --rm -it deepcode cli \"$@\"\n ;;\n\n clean)\n check_docker\n echo -e \"${YELLOW}⚠ 即将停止并删除 DeepCode 容器和镜像${NC}\"\n echo -e \"${YELLOW} (数据目录 deepcode_lab/, uploads/, logs/ 不会被删除)${NC}\"\n read -p \"确认? [y/N] \" confirm\n if [[ \"$confirm\" =~ ^[Yy]$ ]]; then\n dc down --rmi local --remove-orphans\n echo -e \"${GREEN}✓ 已清理完成${NC}\"\n else\n echo \"已取消\"\n fi\n ;;\nesac\n"
},
{
"path": "mcp_agent.config.yaml",
"content": "$schema: ./schema/mcp-agent.config.schema.json\nanthropic: null\ndefault_search_server: filesystem\ndocument_segmentation:\n enabled: false\n size_threshold_chars: 50000\nexecution_engine: asyncio\nlogger:\n level: info\n path_settings:\n path_pattern: logs/mcp-agent-{unique_id}.jsonl\n timestamp_format: '%Y%m%d_%H%M%S'\n unique_id: timestamp\n progress_display: false\n transports:\n - console\n - file\nmcp:\n servers:\n bocha-mcp:\n args:\n - tools/bocha_search_server.py\n command: python\n env:\n BOCHA_API_KEY: ''\n PYTHONPATH: .\n brave:\n # macos and linux should use this\n args:\n - -y\n - '@modelcontextprotocol/server-brave-search'\n command: npx\n\n # windows should use this\n # args:\n # # please use the correct path for your system\n # - C:/Users/LEGION/AppData/Roaming/npm/node_modules/@modelcontextprotocol/server-brave-search/dist/index.js\n # command: node\n env:\n BRAVE_API_KEY: ''\n filesystem:\n # macos and linux should use this\n # Note: \"No valid root directories\" warning is harmless - connection still works\n args:\n - -y\n - '@modelcontextprotocol/server-filesystem'\n - .\n - ./deepcode_lab\n command: npx\n\n # windows should use this\n # args:\n # # please use the correct path for your system\n # - C:/Users/LEGION/AppData/Roaming/npm/node_modules/@modelcontextprotocol/server-filesystem/dist/index.js\n # - .\n # command: node\n\n\n code-implementation:\n args:\n - tools/code_implementation_server.py\n command: python\n description: Paper code reproduction tool server - provides file operations,\n code execution, search and other functions\n env:\n PYTHONPATH: .\n code-reference-indexer:\n args:\n - tools/code_reference_indexer.py\n command: python\n description: Code reference indexer server - Provides intelligent code reference\n search from indexed repositories\n env:\n PYTHONPATH: .\n command-executor:\n args:\n - tools/command_executor.py\n command: python\n env:\n PYTHONPATH: .\n document-segmentation:\n args:\n - tools/document_segmentation_server.py\n command: python\n description: Document segmentation server - Provides intelligent document analysis\n and segmented reading to optimize token usage\n env:\n PYTHONPATH: .\n fetch:\n args:\n - mcp-server-fetch\n command: uvx\n file-downloader:\n args:\n - tools/pdf_downloader.py\n command: python\n env:\n PYTHONPATH: .\n github-downloader:\n args:\n - tools/git_command.py\n command: python\n env:\n PYTHONPATH: .\n# LLM Provider Priority (选择使用哪个LLM / Choose which LLM to use)\n# Options: \"anthropic\", \"google\", \"openai\"\n# If not set or provider unavailable, will fallback to first available provider\nllm_provider: \"openai\" # 设置为 \"google\", \"anthropic\", 或 \"openai\"\n\n#openrouter can be used here and openai professional key\nopenai:\n base_max_tokens: 40000\n default_model: \"google/gemini-3-flash-preview\"\n planning_model: \"google/gemini-3-flash-preview\"\n implementation_model: \"google/gemini-3-flash-preview\"\n reasoning_effort: low # Only for thinking models\n max_tokens_policy: adaptive\n retry_max_tokens: 32768\n\n# Provider configurations\n# default_model is used by mcp_agent for planning/analysis phases\n# implementation_model is used by code_implementation_workflow for code generation\ngoogle:\n default_model: \"gemini-3-pro-preview\"\n planning_model: \"gemini-3-pro-preview\"\n implementation_model: \"gemini-2.5-flash\"\n\nanthropic:\n default_model: \"claude-sonnet-4.5\"\n planning_model: \"claude-sonnet-4.5\"\n implementation_model: \"claude-sonnet-3.5\"\n\n\nplanning_mode: traditional\n"
},
{
"path": "mcp_agent.secrets.yaml.example",
"content": "# =============================================================\n# DeepCode - API Keys Configuration\n# =============================================================\n# Copy this file to mcp_agent.secrets.yaml and fill in your keys.\n#\n# At least ONE LLM provider API key is required.\n# Config file takes priority over environment variables.\n# =============================================================\n\n# OpenAI / OpenRouter\nopenai:\n api_key: \"\"\n # For OpenRouter (recommended - access multiple models via one key):\n # base_url: \"https://openrouter.ai/api/v1\"\n\n# Anthropic (Claude)\nanthropic:\n api_key: \"\"\n\n# Google (Gemini)\ngoogle:\n api_key: \"\"\n"
},
{
"path": "nanobot/.dockerignore",
"content": "__pycache__\n*.pyc\n*.pyo\n*.pyd\n*.egg-info\ndist/\nbuild/\n.git\n.env\n.assets\nnode_modules/\nbridge/dist/\nworkspace/\n"
},
{
"path": "nanobot/.gitignore",
"content": ".assets\n.env\n*.pyc\ndist/\nbuild/\ndocs/\n*.egg-info/\n*.egg\n*.pyc\n*.pyo\n*.pyd\n*.pyw\n*.pyz\n*.pywz\n*.pyzz\n.venv/\n__pycache__/\npoetry.lock\n.pytest_cache/\ntests/\nbotpy.log\n"
},
{
"path": "nanobot/COMMUNICATION.md",
"content": "We provide QR codes for joining the HKUDS discussion groups on **WeChat** and **Feishu**.\n\nYou can join by scanning the QR codes below:\n\n\n"
},
{
"path": "nanobot/Dockerfile",
"content": "FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim\n\n# Install Node.js 20 for the WhatsApp bridge\nRUN apt-get update && \\\n apt-get install -y --no-install-recommends curl ca-certificates gnupg git && \\\n mkdir -p /etc/apt/keyrings && \\\n curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \\\n echo \"deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_20.x nodistro main\" > /etc/apt/sources.list.d/nodesource.list && \\\n apt-get update && \\\n apt-get install -y --no-install-recommends nodejs && \\\n apt-get purge -y gnupg && \\\n apt-get autoremove -y && \\\n rm -rf /var/lib/apt/lists/*\n\nWORKDIR /app\n\n# Install Python dependencies first (cached layer)\n# Note: build context is DeepCode root, so paths start with nanobot/\nCOPY nanobot/pyproject.toml nanobot/README.md nanobot/LICENSE ./\nRUN mkdir -p nanobot bridge && touch nanobot/__init__.py && \\\n uv pip install --system --no-cache . && \\\n rm -rf nanobot bridge\n\n# Copy the full source and install\nCOPY nanobot/nanobot/ nanobot/\nCOPY nanobot/bridge/ bridge/\nRUN uv pip install --system --no-cache .\n\n# Build the WhatsApp bridge\nWORKDIR /app/bridge\nRUN npm install && npm run build\nWORKDIR /app\n\n# Create config directory\nRUN mkdir -p /root/.nanobot\n\n# Gateway default port\nEXPOSE 18790\n\nENTRYPOINT [\"nanobot\"]\nCMD [\"gateway\"]\n"
},
{
"path": "nanobot/LICENSE",
"content": "MIT License\n\nCopyright (c) 2025 nanobot contributors\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "nanobot/README.md",
"content": "
\n \n
nanobot: Ultra-Lightweight Personal AI Assistant
\n
\n \n \n \n \n \n \n \n
\n
\n\n🐈 **nanobot** is an **ultra-lightweight** personal AI assistant inspired by [Clawdbot](https://github.com/openclaw/openclaw)\n\n⚡️ Delivers core agent functionality in just **~4,000** lines of code — **99% smaller** than Clawdbot's 430k+ lines.\n\n📏 Real-time line count: **3,510 lines** (run `bash core_agent_lines.sh` to verify anytime)\n\n## 📢 News\n\n- **2026-02-09** 💬 Added Slack, Email, and QQ support — nanobot now supports multiple chat platforms!\n- **2026-02-08** 🔧 Refactored Providers—adding a new LLM provider now takes just 2 simple steps! Check [here](#providers).\n- **2026-02-07** 🚀 Released v0.1.3.post5 with Qwen support & several key improvements! Check [here](https://github.com/HKUDS/nanobot/releases/tag/v0.1.3.post5) for details.\n- **2026-02-06** ✨ Added Moonshot/Kimi provider, Discord integration, and enhanced security hardening!\n- **2026-02-05** ✨ Added Feishu channel, DeepSeek provider, and enhanced scheduled tasks support!\n- **2026-02-04** 🚀 Released v0.1.3.post4 with multi-provider & Docker support! Check [here](https://github.com/HKUDS/nanobot/releases/tag/v0.1.3.post4) for details.\n- **2026-02-03** ⚡ Integrated vLLM for local LLM support and improved natural language task scheduling!\n- **2026-02-02** 🎉 nanobot officially launched! Welcome to try 🐈 nanobot!\n\n## Key Features of nanobot:\n\n🪶 **Ultra-Lightweight**: Just ~4,000 lines of core agent code — 99% smaller than Clawdbot.\n\n🔬 **Research-Ready**: Clean, readable code that's easy to understand, modify, and extend for research.\n\n⚡️ **Lightning Fast**: Minimal footprint means faster startup, lower resource usage, and quicker iterations.\n\n💎 **Easy-to-Use**: One-click to deploy and you're ready to go.\n\n## 🏗️ Architecture\n\n
\n \n
\n\n## ✨ Features\n\n
\n
\n
📈 24/7 Real-Time Market Analysis
\n
🚀 Full-Stack Software Engineer
\n
📅 Smart Daily Routine Manager
\n
📚 Personal Knowledge Assistant
\n
\n
\n
\n
\n
\n
\n
\n
\n
Discovery • Insights • Trends
\n
Develop • Deploy • Scale
\n
Schedule • Automate • Organize
\n
Learn • Memory • Reasoning
\n
\n
\n\n## 📦 Install\n\n**Install from source** (latest features, recommended for development)\n\n```bash\ngit clone https://github.com/HKUDS/nanobot.git\ncd nanobot\npip install -e .\n```\n\n**Install with [uv](https://github.com/astral-sh/uv)** (stable, fast)\n\n```bash\nuv tool install nanobot-ai\n```\n\n**Install from PyPI** (stable)\n\n```bash\npip install nanobot-ai\n```\n\n## 🚀 Quick Start\n\n> [!TIP]\n> Set your API key in `~/.nanobot/config.json`.\n> Get API keys: [OpenRouter](https://openrouter.ai/keys) (Global) · [DashScope](https://dashscope.console.aliyun.com) (Qwen) · [Brave Search](https://brave.com/search/api/) (optional, for web search)\n\n**1. Initialize**\n\n```bash\nnanobot onboard\n```\n\n**2. Configure** (`~/.nanobot/config.json`)\n\nFor OpenRouter - recommended for global users:\n```json\n{\n \"providers\": {\n \"openrouter\": {\n \"apiKey\": \"sk-or-v1-xxx\"\n }\n },\n \"agents\": {\n \"defaults\": {\n \"model\": \"anthropic/claude-opus-4-5\"\n }\n }\n}\n```\n\n**3. Chat**\n\n```bash\nnanobot agent -m \"What is 2+2?\"\n```\n\nThat's it! You have a working AI assistant in 2 minutes.\n\n## 🖥️ Local Models (vLLM)\n\nRun nanobot with your own local models using vLLM or any OpenAI-compatible server.\n\n**1. Start your vLLM server**\n\n```bash\nvllm serve meta-llama/Llama-3.1-8B-Instruct --port 8000\n```\n\n**2. Configure** (`~/.nanobot/config.json`)\n\n```json\n{\n \"providers\": {\n \"vllm\": {\n \"apiKey\": \"dummy\",\n \"apiBase\": \"http://localhost:8000/v1\"\n }\n },\n \"agents\": {\n \"defaults\": {\n \"model\": \"meta-llama/Llama-3.1-8B-Instruct\"\n }\n }\n}\n```\n\n**3. Chat**\n\n```bash\nnanobot agent -m \"Hello from my local LLM!\"\n```\n\n> [!TIP]\n> The `apiKey` can be any non-empty string for local servers that don't require authentication.\n\n## 💬 Chat Apps\n\nTalk to your nanobot through Telegram, Discord, WhatsApp, Feishu, DingTalk, Slack, Email, or QQ — anytime, anywhere.\n\n| Channel | Setup |\n|---------|-------|\n| **Telegram** | Easy (just a token) |\n| **Discord** | Easy (bot token + intents) |\n| **WhatsApp** | Medium (scan QR) |\n| **Feishu** | Medium (app credentials) |\n| **DingTalk** | Medium (app credentials) |\n| **Slack** | Medium (bot + app tokens) |\n| **Email** | Medium (IMAP/SMTP credentials) |\n| **QQ** | Easy (app credentials) |\n\n\nTelegram (Recommended)\n\n**1. Create a bot**\n- Open Telegram, search `@BotFather`\n- Send `/newbot`, follow prompts\n- Copy the token\n\n**2. Configure**\n\n```json\n{\n \"channels\": {\n \"telegram\": {\n \"enabled\": true,\n \"token\": \"YOUR_BOT_TOKEN\",\n \"allowFrom\": [\"YOUR_USER_ID\"]\n }\n }\n}\n```\n\n> You can find your **User ID** in Telegram settings. It is shown as `@yourUserId`.\n> Copy this value **without the `@` symbol** and paste it into the config file.\n\n\n**3. Run**\n\n```bash\nnanobot gateway\n```\n\n\n\n\nDiscord\n\n**1. Create a bot**\n- Go to https://discord.com/developers/applications\n- Create an application → Bot → Add Bot\n- Copy the bot token\n\n**2. Enable intents**\n- In the Bot settings, enable **MESSAGE CONTENT INTENT**\n- (Optional) Enable **SERVER MEMBERS INTENT** if you plan to use allow lists based on member data\n\n**3. Get your User ID**\n- Discord Settings → Advanced → enable **Developer Mode**\n- Right-click your avatar → **Copy User ID**\n\n**4. Configure**\n\n```json\n{\n \"channels\": {\n \"discord\": {\n \"enabled\": true,\n \"token\": \"YOUR_BOT_TOKEN\",\n \"allowFrom\": [\"YOUR_USER_ID\"]\n }\n }\n}\n```\n\n**5. Invite the bot**\n- OAuth2 → URL Generator\n- Scopes: `bot`\n- Bot Permissions: `Send Messages`, `Read Message History`\n- Open the generated invite URL and add the bot to your server\n\n**6. Run**\n\n```bash\nnanobot gateway\n```\n\n\n\n\nWhatsApp\n\nRequires **Node.js ≥18**.\n\n**1. Link device**\n\n```bash\nnanobot channels login\n# Scan QR with WhatsApp → Settings → Linked Devices\n```\n\n**2. Configure**\n\n```json\n{\n \"channels\": {\n \"whatsapp\": {\n \"enabled\": true,\n \"allowFrom\": [\"+1234567890\"]\n }\n }\n}\n```\n\n**3. Run** (two terminals)\n\n```bash\n# Terminal 1\nnanobot channels login\n\n# Terminal 2\nnanobot gateway\n```\n\n\n\n\nFeishu (飞书)\n\nUses **WebSocket** long connection — no public IP required.\n\n**1. Create a Feishu bot**\n- Visit [Feishu Open Platform](https://open.feishu.cn/app)\n- Create a new app → Enable **Bot** capability\n- **Permissions**: Add `im:message` (send messages)\n- **Events**: Add `im.message.receive_v1` (receive messages)\n - Select **Long Connection** mode (requires running nanobot first to establish connection)\n- Get **App ID** and **App Secret** from \"Credentials & Basic Info\"\n- Publish the app\n\n**2. Configure**\n\n```json\n{\n \"channels\": {\n \"feishu\": {\n \"enabled\": true,\n \"appId\": \"cli_xxx\",\n \"appSecret\": \"xxx\",\n \"encryptKey\": \"\",\n \"verificationToken\": \"\",\n \"allowFrom\": []\n }\n }\n}\n```\n\n> `encryptKey` and `verificationToken` are optional for Long Connection mode.\n> `allowFrom`: Leave empty to allow all users, or add `[\"ou_xxx\"]` to restrict access.\n\n**3. Run**\n\n```bash\nnanobot gateway\n```\n\n> [!TIP]\n> Feishu uses WebSocket to receive messages — no webhook or public IP needed!\n\n\n\n\nQQ (QQ私聊)\n\nUses **botpy SDK** with WebSocket — no public IP required.\n\n**1. Create a QQ bot**\n- Visit [QQ Open Platform](https://q.qq.com)\n- Create a new bot application\n- Get **AppID** and **Secret** from \"Developer Settings\"\n\n**2. Configure**\n\n```json\n{\n \"channels\": {\n \"qq\": {\n \"enabled\": true,\n \"appId\": \"YOUR_APP_ID\",\n \"secret\": \"YOUR_APP_SECRET\",\n \"allowFrom\": []\n }\n }\n}\n```\n\n> `allowFrom`: Leave empty for public access, or add user openids to restrict access.\n> Example: `\"allowFrom\": [\"user_openid_1\", \"user_openid_2\"]`\n\n**3. Run**\n\n```bash\nnanobot gateway\n```\n\n> [!TIP]\n> QQ bot currently supports **private messages only**. Group chat support coming soon!\n\n\n\n\nDingTalk (钉钉)\n\nUses **Stream Mode** — no public IP required.\n\n**1. Create a DingTalk bot**\n- Visit [DingTalk Open Platform](https://open-dev.dingtalk.com/)\n- Create a new app -> Add **Robot** capability\n- **Configuration**:\n - Toggle **Stream Mode** ON\n- **Permissions**: Add necessary permissions for sending messages\n- Get **AppKey** (Client ID) and **AppSecret** (Client Secret) from \"Credentials\"\n- Publish the app\n\n**2. Configure**\n\n```json\n{\n \"channels\": {\n \"dingtalk\": {\n \"enabled\": true,\n \"clientId\": \"YOUR_APP_KEY\",\n \"clientSecret\": \"YOUR_APP_SECRET\",\n \"allowFrom\": []\n }\n }\n}\n```\n\n> `allowFrom`: Leave empty to allow all users, or add `[\"staffId\"]` to restrict access.\n\n**3. Run**\n\n```bash\nnanobot gateway\n```\n\n\n\n\nSlack\n\nUses **Socket Mode** — no public URL required.\n\n**1. Create a Slack app**\n- Go to [Slack API](https://api.slack.com/apps) → Create New App\n- **OAuth & Permissions**: Add bot scopes: `chat:write`, `reactions:write`, `app_mentions:read`\n- Install to your workspace and copy the **Bot Token** (`xoxb-...`)\n- **Socket Mode**: Enable it and generate an **App-Level Token** (`xapp-...`) with `connections:write` scope\n- **Event Subscriptions**: Subscribe to `message.im`, `message.channels`, `app_mention`\n\n**2. Configure**\n\n```json\n{\n \"channels\": {\n \"slack\": {\n \"enabled\": true,\n \"botToken\": \"xoxb-...\",\n \"appToken\": \"xapp-...\",\n \"groupPolicy\": \"mention\"\n }\n }\n}\n```\n\n> `groupPolicy`: `\"mention\"` (respond only when @mentioned), `\"open\"` (respond to all messages), or `\"allowlist\"` (restrict to specific channels).\n> DM policy defaults to open. Set `\"dm\": {\"enabled\": false}` to disable DMs.\n\n**3. Run**\n\n```bash\nnanobot gateway\n```\n\n\n\n\nEmail\n\nGive nanobot its own email account. It polls **IMAP** for incoming mail and replies via **SMTP** — like a personal email assistant.\n\n**1. Get credentials (Gmail example)**\n- Create a dedicated Gmail account for your bot (e.g. `my-nanobot@gmail.com`)\n- Enable 2-Step Verification → Create an [App Password](https://myaccount.google.com/apppasswords)\n- Use this app password for both IMAP and SMTP\n\n**2. Configure**\n\n> - `consentGranted` must be `true` to allow mailbox access. This is a safety gate — set `false` to fully disable.\n> - `allowFrom`: Leave empty to accept emails from anyone, or restrict to specific senders.\n> - `smtpUseTls` and `smtpUseSsl` default to `true` / `false` respectively, which is correct for Gmail (port 587 + STARTTLS). No need to set them explicitly.\n> - Set `\"autoReplyEnabled\": false` if you only want to read/analyze emails without sending automatic replies.\n\n```json\n{\n \"channels\": {\n \"email\": {\n \"enabled\": true,\n \"consentGranted\": true,\n \"imapHost\": \"imap.gmail.com\",\n \"imapPort\": 993,\n \"imapUsername\": \"my-nanobot@gmail.com\",\n \"imapPassword\": \"your-app-password\",\n \"smtpHost\": \"smtp.gmail.com\",\n \"smtpPort\": 587,\n \"smtpUsername\": \"my-nanobot@gmail.com\",\n \"smtpPassword\": \"your-app-password\",\n \"fromAddress\": \"my-nanobot@gmail.com\",\n \"allowFrom\": [\"your-real-email@gmail.com\"]\n }\n }\n}\n```\n\n\n**3. Run**\n\n```bash\nnanobot gateway\n```\n\n\n\n## ⚙️ Configuration\n\nConfig file: `~/.nanobot/config.json`\n\n### Providers\n\n> [!TIP]\n> - **Groq** provides free voice transcription via Whisper. If configured, Telegram voice messages will be automatically transcribed.\n> - **Zhipu Coding Plan**: If you're on Zhipu's coding plan, set `\"apiBase\": \"https://open.bigmodel.cn/api/coding/paas/v4\"` in your zhipu provider config.\n\n| Provider | Purpose | Get API Key |\n|----------|---------|-------------|\n| `openrouter` | LLM (recommended, access to all models) | [openrouter.ai](https://openrouter.ai) |\n| `anthropic` | LLM (Claude direct) | [console.anthropic.com](https://console.anthropic.com) |\n| `openai` | LLM (GPT direct) | [platform.openai.com](https://platform.openai.com) |\n| `deepseek` | LLM (DeepSeek direct) | [platform.deepseek.com](https://platform.deepseek.com) |\n| `groq` | LLM + **Voice transcription** (Whisper) | [console.groq.com](https://console.groq.com) |\n| `gemini` | LLM (Gemini direct) | [aistudio.google.com](https://aistudio.google.com) |\n| `aihubmix` | LLM (API gateway, access to all models) | [aihubmix.com](https://aihubmix.com) |\n| `dashscope` | LLM (Qwen) | [dashscope.console.aliyun.com](https://dashscope.console.aliyun.com) |\n| `moonshot` | LLM (Moonshot/Kimi) | [platform.moonshot.cn](https://platform.moonshot.cn) |\n| `zhipu` | LLM (Zhipu GLM) | [open.bigmodel.cn](https://open.bigmodel.cn) |\n| `vllm` | LLM (local, any OpenAI-compatible server) | — |\n\n\nAdding a New Provider (Developer Guide)\n\nnanobot uses a **Provider Registry** (`nanobot/providers/registry.py`) as the single source of truth.\nAdding a new provider only takes **2 steps** — no if-elif chains to touch.\n\n**Step 1.** Add a `ProviderSpec` entry to `PROVIDERS` in `nanobot/providers/registry.py`:\n\n```python\nProviderSpec(\n name=\"myprovider\", # config field name\n keywords=(\"myprovider\", \"mymodel\"), # model-name keywords for auto-matching\n env_key=\"MYPROVIDER_API_KEY\", # env var for LiteLLM\n display_name=\"My Provider\", # shown in `nanobot status`\n litellm_prefix=\"myprovider\", # auto-prefix: model → myprovider/model\n skip_prefixes=(\"myprovider/\",), # don't double-prefix\n)\n```\n\n**Step 2.** Add a field to `ProvidersConfig` in `nanobot/config/schema.py`:\n\n```python\nclass ProvidersConfig(BaseModel):\n ...\n myprovider: ProviderConfig = ProviderConfig()\n```\n\nThat's it! Environment variables, model prefixing, config matching, and `nanobot status` display will all work automatically.\n\n**Common `ProviderSpec` options:**\n\n| Field | Description | Example |\n|-------|-------------|---------|\n| `litellm_prefix` | Auto-prefix model names for LiteLLM | `\"dashscope\"` → `dashscope/qwen-max` |\n| `skip_prefixes` | Don't prefix if model already starts with these | `(\"dashscope/\", \"openrouter/\")` |\n| `env_extras` | Additional env vars to set | `((\"ZHIPUAI_API_KEY\", \"{api_key}\"),)` |\n| `model_overrides` | Per-model parameter overrides | `((\"kimi-k2.5\", {\"temperature\": 1.0}),)` |\n| `is_gateway` | Can route any model (like OpenRouter) | `True` |\n| `detect_by_key_prefix` | Detect gateway by API key prefix | `\"sk-or-\"` |\n| `detect_by_base_keyword` | Detect gateway by API base URL | `\"openrouter\"` |\n| `strip_model_prefix` | Strip existing prefix before re-prefixing | `True` (for AiHubMix) |\n\n\n\n\n### Security\n\n> For production deployments, set `\"restrictToWorkspace\": true` in your config to sandbox the agent.\n\n| Option | Default | Description |\n|--------|---------|-------------|\n| `tools.restrictToWorkspace` | `false` | When `true`, restricts **all** agent tools (shell, file read/write/edit, list) to the workspace directory. Prevents path traversal and out-of-scope access. |\n| `channels.*.allowFrom` | `[]` (allow all) | Whitelist of user IDs. Empty = allow everyone; non-empty = only listed users can interact. |\n\n\n## CLI Reference\n\n| Command | Description |\n|---------|-------------|\n| `nanobot onboard` | Initialize config & workspace |\n| `nanobot agent -m \"...\"` | Chat with the agent |\n| `nanobot agent` | Interactive chat mode |\n| `nanobot agent --no-markdown` | Show plain-text replies |\n| `nanobot agent --logs` | Show runtime logs during chat |\n| `nanobot gateway` | Start the gateway |\n| `nanobot status` | Show status |\n| `nanobot channels login` | Link WhatsApp (scan QR) |\n| `nanobot channels status` | Show channel status |\n\nInteractive mode exits: `exit`, `quit`, `/exit`, `/quit`, `:q`, or `Ctrl+D`.\n\n\nScheduled Tasks (Cron)\n\n```bash\n# Add a job\nnanobot cron add --name \"daily\" --message \"Good morning!\" --cron \"0 9 * * *\"\nnanobot cron add --name \"hourly\" --message \"Check status\" --every 3600\n\n# List jobs\nnanobot cron list\n\n# Remove a job\nnanobot cron remove \n```\n\n\n\n## 🐳 Docker\n\n> [!TIP]\n> The `-v ~/.nanobot:/root/.nanobot` flag mounts your local config directory into the container, so your config and workspace persist across container restarts.\n\nBuild and run nanobot in a container:\n\n```bash\n# Build the image\ndocker build -t nanobot .\n\n# Initialize config (first time only)\ndocker run -v ~/.nanobot:/root/.nanobot --rm nanobot onboard\n\n# Edit config on host to add API keys\nvim ~/.nanobot/config.json\n\n# Run gateway (connects to Telegram/WhatsApp)\ndocker run -v ~/.nanobot:/root/.nanobot -p 18790:18790 nanobot gateway\n\n# Or run a single command\ndocker run -v ~/.nanobot:/root/.nanobot --rm nanobot agent -m \"Hello!\"\ndocker run -v ~/.nanobot:/root/.nanobot --rm nanobot status\n```\n\n## 📁 Project Structure\n\n```\nnanobot/\n├── agent/ # 🧠 Core agent logic\n│ ├── loop.py # Agent loop (LLM ↔ tool execution)\n│ ├── context.py # Prompt builder\n│ ├── memory.py # Persistent memory\n│ ├── skills.py # Skills loader\n│ ├── subagent.py # Background task execution\n│ └── tools/ # Built-in tools (incl. spawn)\n├── skills/ # 🎯 Bundled skills (github, weather, tmux...)\n├── channels/ # 📱 WhatsApp integration\n├── bus/ # 🚌 Message routing\n├── cron/ # ⏰ Scheduled tasks\n├── heartbeat/ # 💓 Proactive wake-up\n├── providers/ # 🤖 LLM providers (OpenRouter, etc.)\n├── session/ # 💬 Conversation sessions\n├── config/ # ⚙️ Configuration\n└── cli/ # 🖥️ Commands\n```\n\n## 🤝 Contribute & Roadmap\n\nPRs welcome! The codebase is intentionally small and readable. 🤗\n\n**Roadmap** — Pick an item and [open a PR](https://github.com/HKUDS/nanobot/pulls)!\n\n- [x] **Voice Transcription** — Support for Groq Whisper (Issue #13)\n- [ ] **Multi-modal** — See and hear (images, voice, video)\n- [ ] **Long-term memory** — Never forget important context\n- [ ] **Better reasoning** — Multi-step planning and reflection\n- [ ] **More integrations** — Calendar and more\n- [ ] **Self-improvement** — Learn from feedback and mistakes\n\n### Contributors\n\n\n \n\n\n\n## ⭐ Star History\n\n
\n\n {\n const index = steps.findIndex((s) => s.id === stepId);\n if (index !== -1) setCurrentStep(index);\n }}\n />\n \n\n {/* Info */}\n \n
About This View
\n
\n The workflow editor allows you to visualize the code generation pipeline.\n Each node represents a step in the process, and the connections show the\n data flow between steps. Use this view to understand how DeepCode processes\n your inputs and generates code.\n
\n
\n
\n \n Pending steps\n
\n
\n \n Active step\n
\n
\n \n Completed steps\n
\n
\n \n
\n );\n}\n"
},
{
"path": "new_ui/frontend/src/pages/index.ts",
"content": "export { default as HomePage } from './HomePage';\nexport { default as PaperToCodePage } from './PaperToCodePage';\nexport { default as ChatPlanningPage } from './ChatPlanningPage';\nexport { default as WorkflowEditorPage } from './WorkflowEditorPage';\nexport { default as SettingsPage } from './SettingsPage';\n"
},
{
"path": "new_ui/frontend/src/services/api.ts",
"content": "import axios from 'axios';\nimport type {\n TaskResponse,\n WorkflowStatusResponse,\n QuestionsResponse,\n RequirementsSummaryResponse,\n ConfigResponse,\n SettingsResponse,\n FileUploadResponse,\n} from '../types/api';\n\nconst api = axios.create({\n baseURL: '/api/v1',\n timeout: 30000,\n headers: {\n 'Content-Type': 'application/json',\n },\n});\n\n// Workflows API\nexport const workflowsApi = {\n startPaperToCode: async (\n inputSource: string,\n inputType: 'file' | 'url',\n enableIndexing: boolean = false\n ): Promise => {\n const response = await api.post('/workflows/paper-to-code', {\n input_source: inputSource,\n input_type: inputType,\n enable_indexing: enableIndexing,\n });\n return response.data;\n },\n\n startChatPlanning: async (\n requirements: string,\n enableIndexing: boolean = false\n ): Promise => {\n const response = await api.post('/workflows/chat-planning', {\n requirements,\n enable_indexing: enableIndexing,\n });\n return response.data;\n },\n\n getStatus: async (taskId: string): Promise => {\n const response = await api.get(\n `/workflows/status/${taskId}`\n );\n return response.data;\n },\n\n cancel: async (taskId: string): Promise => {\n await api.post(`/workflows/cancel/${taskId}`);\n },\n\n getActiveTasks: async (): Promise<{ tasks: Array<{\n task_id: string;\n status: string;\n progress: number;\n message: string;\n started_at: string | null;\n }> }> => {\n const response = await api.get('/workflows/active');\n return response.data;\n },\n\n getRecentTasks: async (limit: number = 10): Promise<{ tasks: Array<{\n task_id: string;\n status: string;\n progress: number;\n message: string;\n result: Record | null;\n error: string | null;\n started_at: string | null;\n completed_at: string | null;\n }> }> => {\n const response = await api.get(`/workflows/recent?limit=${limit}`);\n return response.data;\n },\n\n // User-in-Loop interaction APIs\n respondToInteraction: async (\n taskId: string,\n action: string,\n data: Record = {},\n skipped: boolean = false\n ): Promise<{ status: string; task_id: string; action: string }> => {\n const response = await api.post(`/workflows/respond/${taskId}`, {\n action,\n data,\n skipped,\n });\n return response.data;\n },\n\n getInteraction: async (taskId: string): Promise<{\n has_interaction: boolean;\n task_id: string;\n status: string;\n interaction?: {\n type: string;\n title: string;\n description: string;\n data: Record;\n options: Record;\n required: boolean;\n };\n }> => {\n const response = await api.get(`/workflows/interaction/${taskId}`);\n return response.data;\n },\n};\n\n// Requirements API\nexport const requirementsApi = {\n generateQuestions: async (\n initialRequirement: string\n ): Promise => {\n const response = await api.post('/requirements/questions', {\n initial_requirement: initialRequirement,\n });\n return response.data;\n },\n\n summarize: async (\n initialRequirement: string,\n userAnswers: Record\n ): Promise => {\n const response = await api.post(\n '/requirements/summarize',\n {\n initial_requirement: initialRequirement,\n user_answers: userAnswers,\n }\n );\n return response.data;\n },\n\n modify: async (\n currentRequirements: string,\n modificationFeedback: string\n ): Promise => {\n const response = await api.put(\n '/requirements/modify',\n {\n current_requirements: currentRequirements,\n modification_feedback: modificationFeedback,\n }\n );\n return response.data;\n },\n};\n\n// Config API\nexport const configApi = {\n getSettings: async (): Promise => {\n const response = await api.get('/config/settings');\n return response.data;\n },\n\n getLLMProviders: async (): Promise => {\n const response = await api.get('/config/llm-providers');\n return response.data;\n },\n\n setLLMProvider: async (provider: string): Promise => {\n await api.put('/config/llm-provider', { provider });\n },\n};\n\n// Files API\nexport const filesApi = {\n upload: async (file: File): Promise => {\n const formData = new FormData();\n formData.append('file', file);\n\n const response = await api.post('/files/upload', formData, {\n headers: {\n 'Content-Type': 'multipart/form-data',\n },\n });\n return response.data;\n },\n\n delete: async (fileId: string): Promise => {\n await api.delete(`/files/delete/${fileId}`);\n },\n\n getInfo: async (fileId: string): Promise => {\n const response = await api.get(`/files/info/${fileId}`);\n return response.data;\n },\n};\n\nexport default api;\n"
},
{
"path": "new_ui/frontend/src/stores/index.ts",
"content": "export { useWorkflowStore } from './workflowStore';\nexport { useSessionStore } from './sessionStore';\n"
},
{
"path": "new_ui/frontend/src/stores/sessionStore.ts",
"content": "import { create } from 'zustand';\nimport { persist } from 'zustand/middleware';\nimport type { Message } from '../types/common';\n\ninterface SessionState {\n // Session\n sessionId: string | null;\n\n // Conversation history\n conversationHistory: Message[];\n\n // User preferences\n preferences: {\n llmProvider: string;\n enableIndexing: boolean;\n theme: 'light' | 'dark';\n };\n\n // Recent projects\n recentProjects: {\n id: string;\n name: string;\n type: string;\n timestamp: string;\n }[];\n\n // Actions\n setSessionId: (id: string | null) => void;\n addMessage: (message: Omit) => void;\n clearHistory: () => void;\n updatePreferences: (prefs: Partial) => void;\n addRecentProject: (project: Omit) => void;\n}\n\nexport const useSessionStore = create()(\n persist(\n (set, _get) => ({\n sessionId: null,\n conversationHistory: [],\n preferences: {\n llmProvider: 'google',\n enableIndexing: false,\n theme: 'light',\n },\n recentProjects: [],\n\n setSessionId: (id) => set({ sessionId: id }),\n\n addMessage: (message) => {\n const newMessage: Message = {\n ...message,\n id: crypto.randomUUID(),\n timestamp: new Date().toISOString(),\n };\n set((state) => ({\n conversationHistory: [...state.conversationHistory, newMessage],\n }));\n },\n\n clearHistory: () => set({ conversationHistory: [] }),\n\n updatePreferences: (prefs) =>\n set((state) => ({\n preferences: { ...state.preferences, ...prefs },\n })),\n\n addRecentProject: (project) => {\n const newProject = {\n ...project,\n timestamp: new Date().toISOString(),\n };\n set((state) => ({\n recentProjects: [newProject, ...state.recentProjects.slice(0, 9)],\n }));\n },\n }),\n {\n name: 'deepcode-session',\n partialize: (state) => ({\n preferences: state.preferences,\n recentProjects: state.recentProjects,\n }),\n }\n )\n);\n"
},
{
"path": "new_ui/frontend/src/stores/workflowStore.ts",
"content": "import { create } from 'zustand';\nimport { persist } from 'zustand/middleware';\nimport type {\n WorkflowStatus,\n WorkflowStep,\n} from '../types/workflow';\n\n// Activity log entry type\ninterface ActivityLogEntry {\n id: string;\n timestamp: Date;\n message: string;\n progress: number;\n type: 'info' | 'success' | 'warning' | 'error' | 'progress';\n}\n\n// User-in-Loop interaction types\nexport interface PendingInteraction {\n type: string; // 'requirement_questions' | 'plan_review' | etc.\n title: string;\n description: string;\n data: {\n questions?: Array<{\n id: string;\n question: string;\n category?: string;\n importance?: string;\n hint?: string;\n }>;\n plan?: string;\n plan_preview?: string;\n original_input?: string;\n [key: string]: unknown;\n };\n options: Record;\n required: boolean;\n}\n\ninterface WorkflowState {\n // Current task\n activeTaskId: string | null;\n workflowType: 'paper-to-code' | 'chat-planning' | null; // Track workflow type\n status: WorkflowStatus;\n progress: number;\n message: string;\n\n // Steps\n steps: WorkflowStep[];\n currentStepIndex: number;\n\n // Streaming data\n streamedCode: string;\n currentFile: string | null;\n generatedFiles: string[];\n\n // Activity logs\n activityLogs: ActivityLogEntry[];\n\n // User-in-Loop interaction\n pendingInteraction: PendingInteraction | null;\n isWaitingForInput: boolean;\n\n // Results\n result: Record | null;\n error: string | null;\n\n // Recovery\n needsRecovery: boolean; // Flag to indicate if we need to recover a task\n\n // Actions\n setActiveTask: (taskId: string | null, workflowType?: 'paper-to-code' | 'chat-planning') => void;\n setStatus: (status: WorkflowStatus) => void;\n updateProgress: (progress: number, message: string) => void;\n setSteps: (steps: WorkflowStep[]) => void;\n updateStepStatus: (stepId: string, status: WorkflowStep['status']) => void;\n appendStreamedCode: (chunk: string) => void;\n setCurrentFile: (filename: string | null) => void;\n addGeneratedFile: (filename: string) => void;\n addActivityLog: (message: string, progress: number, type?: ActivityLogEntry['type']) => void;\n setPendingInteraction: (interaction: PendingInteraction | null) => void;\n clearInteraction: () => void;\n setResult: (result: Record | null) => void;\n setError: (error: string | null) => void;\n setNeedsRecovery: (needs: boolean) => void;\n reset: () => void;\n}\n\nconst initialState = {\n activeTaskId: null,\n workflowType: null as 'paper-to-code' | 'chat-planning' | null,\n status: 'idle' as WorkflowStatus,\n progress: 0,\n message: '',\n steps: [],\n currentStepIndex: -1,\n streamedCode: '',\n currentFile: null,\n generatedFiles: [],\n activityLogs: [] as ActivityLogEntry[],\n pendingInteraction: null as PendingInteraction | null,\n isWaitingForInput: false,\n result: null,\n error: null,\n needsRecovery: false,\n};\n\nexport const useWorkflowStore = create()(\n persist(\n (set, get) => ({\n ...initialState,\n\n setActiveTask: (taskId, workflowType) => set({\n activeTaskId: taskId,\n workflowType: workflowType ?? get().workflowType\n }),\n\n setStatus: (status) => {\n console.log('[workflowStore] setStatus:', status);\n set({ status });\n },\n\n updateProgress: (progress, message) => {\n const { steps } = get();\n\n // Find current step based on progress\n let currentStepIndex = -1;\n for (let i = steps.length - 1; i >= 0; i--) {\n if (progress >= steps[i].progress) {\n currentStepIndex = i;\n break;\n }\n }\n\n // Check if workflow is complete (progress >= 100)\n const isComplete = progress >= 100;\n\n // Update step statuses\n const updatedSteps = steps.map((step, index) => ({\n ...step,\n status:\n isComplete\n ? 'completed' // All steps completed when progress >= 100\n : index < currentStepIndex\n ? 'completed'\n : index === currentStepIndex\n ? 'active'\n : 'pending',\n })) as WorkflowStep[];\n\n set({\n progress,\n message,\n currentStepIndex: isComplete ? steps.length - 1 : currentStepIndex,\n steps: updatedSteps,\n });\n },\n\n setSteps: (steps) => set({ steps }),\n\n updateStepStatus: (stepId, status) => {\n const { steps } = get();\n const updatedSteps = steps.map((step) =>\n step.id === stepId ? { ...step, status } : step\n );\n set({ steps: updatedSteps });\n },\n\n appendStreamedCode: (chunk) =>\n set((state) => ({\n streamedCode: state.streamedCode + chunk,\n })),\n\n setCurrentFile: (filename) => set({ currentFile: filename }),\n\n addGeneratedFile: (filename) =>\n set((state) => ({\n generatedFiles: [...state.generatedFiles, filename],\n })),\n\n addActivityLog: (message, progress, type = 'progress') =>\n set((state) => ({\n activityLogs: [\n ...state.activityLogs,\n {\n id: `log-${Date.now()}-${Math.random().toString(36).substr(2, 9)}`,\n timestamp: new Date(),\n message,\n progress,\n type,\n },\n ],\n })),\n\n setPendingInteraction: (interaction) => {\n console.log('[workflowStore] setPendingInteraction:', interaction?.type);\n set({\n pendingInteraction: interaction,\n isWaitingForInput: interaction !== null,\n });\n },\n\n clearInteraction: () => {\n console.log('[workflowStore] clearInteraction');\n set({\n pendingInteraction: null,\n isWaitingForInput: false,\n });\n },\n\n setResult: (result) => {\n console.log('[workflowStore] setResult:', result);\n set({ result });\n },\n\n setError: (error) => set({ error, status: error ? 'error' : get().status }),\n\n setNeedsRecovery: (needs) => set({ needsRecovery: needs }),\n\n reset: () => {\n console.log('[workflowStore] Resetting state and clearing localStorage');\n // Clear localStorage explicitly to ensure clean state\n try {\n localStorage.removeItem('deepcode-workflow');\n } catch (e) {\n console.error('[workflowStore] Failed to clear localStorage:', e);\n }\n set(initialState);\n },\n }),\n {\n name: 'deepcode-workflow',\n // Only persist task-related data for recovery when task is running or waiting\n partialize: (state) => {\n const isActive = state.status === 'running' || state.isWaitingForInput;\n return {\n // Only persist activeTaskId if task is still running or waiting for input\n // This prevents trying to recover completed/errored tasks\n activeTaskId: isActive ? state.activeTaskId : null,\n workflowType: isActive ? state.workflowType : null,\n status: isActive ? state.status : 'idle',\n progress: isActive ? state.progress : 0,\n steps: isActive ? state.steps : [],\n isWaitingForInput: state.isWaitingForInput,\n };\n },\n }\n )\n);\n"
},
{
"path": "new_ui/frontend/src/types/api.ts",
"content": "// API types\n\nexport interface TaskResponse {\n task_id: string;\n status: string;\n message: string;\n created_at?: string;\n}\n\nexport interface WorkflowStatusResponse {\n task_id: string;\n status: string;\n progress: number;\n message: string;\n result?: Record;\n error?: string;\n started_at?: string;\n completed_at?: string;\n}\n\nexport interface QuestionsResponse {\n questions: Question[];\n status: string;\n}\n\nexport interface Question {\n id: string;\n question: string;\n category?: string;\n importance?: string;\n hint?: string;\n}\n\nexport interface RequirementsSummaryResponse {\n summary: string;\n status: string;\n}\n\nexport interface ConfigResponse {\n llm_provider: string;\n available_providers: string[];\n models: Record;\n indexing_enabled: boolean;\n}\n\nexport interface SettingsResponse {\n llm_provider: string;\n models: Record;\n indexing_enabled: boolean;\n document_segmentation: Record;\n}\n\nexport interface FileUploadResponse {\n file_id: string;\n filename: string;\n path: string;\n size: number;\n}\n\nexport interface ErrorResponse {\n error: string;\n detail?: string;\n code?: string;\n}\n\n// WebSocket message types\nexport interface WSProgressMessage {\n type: 'progress' | 'status' | 'heartbeat';\n task_id: string;\n progress?: number;\n message?: string;\n status?: string;\n timestamp: string;\n}\n\nexport interface WSCompleteMessage {\n type: 'complete';\n task_id: string;\n status: string;\n result: Record;\n timestamp: string;\n}\n\nexport interface WSErrorMessage {\n type: 'error';\n task_id: string;\n error: string;\n timestamp: string;\n}\n\nexport interface WSCodeChunkMessage {\n type: 'code_chunk' | 'file_start' | 'file_end';\n task_id: string;\n content?: string;\n filename?: string;\n timestamp: string;\n}\n\nexport interface WSLogMessage {\n type: 'log';\n level: 'INFO' | 'WARNING' | 'ERROR' | 'DEBUG';\n message: string;\n namespace: string;\n timestamp: string;\n}\n\n// User-in-Loop interaction message\nexport interface WSInteractionMessage {\n type: 'interaction_required';\n task_id: string;\n interaction_type: 'requirement_questions' | 'plan_review' | 'code_review' | string;\n title: string;\n description: string;\n data: {\n questions?: Question[];\n plan?: string;\n plan_preview?: string;\n original_input?: string;\n [key: string]: unknown;\n };\n options: Record;\n required: boolean;\n timestamp: string;\n}\n\nexport type WSMessage =\n | WSProgressMessage\n | WSCompleteMessage\n | WSErrorMessage\n | WSCodeChunkMessage\n | WSLogMessage\n | WSInteractionMessage;\n"
},
{
"path": "new_ui/frontend/src/types/common.ts",
"content": "// Common types\n\nexport interface Message {\n id: string;\n role: 'user' | 'assistant' | 'system';\n content: string;\n timestamp: string;\n metadata?: Record;\n}\n\nexport interface Notification {\n id: string;\n type: 'success' | 'error' | 'warning' | 'info';\n title: string;\n description?: string;\n duration?: number;\n}\n\nexport interface LayoutConfig {\n sidebarWidth: number;\n showCodePreview: boolean;\n showWorkflowCanvas: boolean;\n splitRatio: number;\n}\n\nexport type TaskType = 'paper-to-code' | 'chat-planning' | 'workflow-editor' | 'settings';\n"
},
{
"path": "new_ui/frontend/src/types/index.ts",
"content": "export * from './workflow';\nexport * from './api';\nexport * from './common';\n"
},
{
"path": "new_ui/frontend/src/types/workflow.ts",
"content": "// Workflow types\n\nexport type WorkflowStatus = 'idle' | 'running' | 'completed' | 'error' | 'cancelled';\n\nexport interface WorkflowStep {\n id: string;\n title: string;\n subtitle: string;\n progress: number;\n status: 'pending' | 'active' | 'completed' | 'error';\n}\n\nexport interface WorkflowTask {\n taskId: string;\n status: WorkflowStatus;\n progress: number;\n message: string;\n result?: Record;\n error?: string;\n startedAt?: string;\n completedAt?: string;\n}\n\nexport interface WorkflowInput {\n type: 'paper-to-code' | 'chat-planning';\n inputSource: string;\n inputType: 'file' | 'url' | 'chat';\n enableIndexing: boolean;\n}\n\n// Workflow step definitions\nexport const PAPER_TO_CODE_STEPS: WorkflowStep[] = [\n { id: 'init', title: 'Initialize', subtitle: 'Load systems', progress: 5, status: 'pending' },\n { id: 'analyze', title: 'Analyze', subtitle: 'Parse paper', progress: 10, status: 'pending' },\n { id: 'download', title: 'Download', subtitle: 'Collect refs', progress: 25, status: 'pending' },\n { id: 'plan', title: 'Plan', subtitle: 'Blueprint', progress: 40, status: 'pending' },\n { id: 'references', title: 'References', subtitle: 'Key refs', progress: 50, status: 'pending' },\n { id: 'repos', title: 'Repos', subtitle: 'GitHub sync', progress: 60, status: 'pending' },\n { id: 'index', title: 'Index', subtitle: 'Vectorize', progress: 70, status: 'pending' },\n { id: 'implement', title: 'Implement', subtitle: 'Code gen', progress: 85, status: 'pending' },\n];\n\nexport const CHAT_PLANNING_STEPS: WorkflowStep[] = [\n { id: 'init', title: 'Initialize', subtitle: 'Boot agents', progress: 5, status: 'pending' },\n { id: 'plan', title: 'Plan', subtitle: 'Analyze intent', progress: 30, status: 'pending' },\n { id: 'setup', title: 'Setup', subtitle: 'Workspace', progress: 50, status: 'pending' },\n { id: 'draft', title: 'Draft', subtitle: 'Generate plan', progress: 70, status: 'pending' },\n { id: 'implement', title: 'Implement', subtitle: 'Code gen', progress: 85, status: 'pending' },\n];\n"
},
{
"path": "new_ui/frontend/tailwind.config.js",
"content": "/** @type {import('tailwindcss').Config} */\nexport default {\n content: [\n \"./index.html\",\n \"./src/**/*.{js,ts,jsx,tsx}\",\n ],\n theme: {\n extend: {\n colors: {\n primary: {\n 50: '#eff6ff',\n 100: '#dbeafe',\n 200: '#bfdbfe',\n 300: '#93c5fd',\n 400: '#60a5fa',\n 500: '#3b82f6',\n 600: '#2563eb',\n 700: '#1d4ed8',\n 800: '#1e40af',\n 900: '#1e3a8a',\n },\n gray: {\n 50: '#f9fafb',\n 100: '#f3f4f6',\n 200: '#e5e7eb',\n 300: '#d1d5db',\n 400: '#9ca3af',\n 500: '#6b7280',\n 600: '#4b5563',\n 700: '#374151',\n 800: '#1f2937',\n 900: '#111827',\n },\n },\n fontFamily: {\n sans: ['Inter', 'system-ui', 'sans-serif'],\n mono: ['JetBrains Mono', 'Menlo', 'Monaco', 'monospace'],\n },\n animation: {\n 'pulse-slow': 'pulse 3s cubic-bezier(0.4, 0, 0.6, 1) infinite',\n 'slide-in': 'slideIn 0.3s ease-out',\n 'fade-in': 'fadeIn 0.2s ease-out',\n },\n keyframes: {\n slideIn: {\n '0%': { transform: 'translateX(-10px)', opacity: '0' },\n '100%': { transform: 'translateX(0)', opacity: '1' },\n },\n fadeIn: {\n '0%': { opacity: '0' },\n '100%': { opacity: '1' },\n },\n },\n },\n },\n plugins: [],\n}\n"
},
{
"path": "new_ui/frontend/tsconfig.json",
"content": "{\n \"compilerOptions\": {\n \"target\": \"ES2020\",\n \"useDefineForClassFields\": true,\n \"lib\": [\"ES2020\", \"DOM\", \"DOM.Iterable\"],\n \"module\": \"ESNext\",\n \"skipLibCheck\": true,\n \"moduleResolution\": \"bundler\",\n \"allowImportingTsExtensions\": true,\n \"resolveJsonModule\": true,\n \"isolatedModules\": true,\n \"noEmit\": true,\n \"jsx\": \"react-jsx\",\n \"strict\": true,\n \"noUnusedLocals\": true,\n \"noUnusedParameters\": true,\n \"noFallthroughCasesInSwitch\": true,\n \"baseUrl\": \".\",\n \"paths\": {\n \"@/*\": [\"src/*\"]\n }\n },\n \"include\": [\"src\"],\n \"references\": [{ \"path\": \"./tsconfig.node.json\" }]\n}\n"
},
{
"path": "new_ui/frontend/tsconfig.node.json",
"content": "{\n \"compilerOptions\": {\n \"composite\": true,\n \"skipLibCheck\": true,\n \"module\": \"ESNext\",\n \"moduleResolution\": \"bundler\",\n \"allowSyntheticDefaultImports\": true\n },\n \"include\": [\"vite.config.ts\"]\n}\n"
},
{
"path": "new_ui/frontend/vite.config.ts",
"content": "import { defineConfig } from 'vite'\nimport react from '@vitejs/plugin-react'\nimport path from 'path'\n\n// https://vitejs.dev/config/\nexport default defineConfig({\n plugins: [react()],\n resolve: {\n alias: {\n '@': path.resolve(__dirname, './src'),\n },\n },\n server: {\n port: 5173,\n proxy: {\n '/api': {\n target: 'http://localhost:8000',\n changeOrigin: true,\n },\n '/ws': {\n target: 'ws://localhost:8000',\n ws: true,\n },\n },\n },\n})\n"
},
{
"path": "new_ui/scripts/build.sh",
"content": "#!/bin/bash\n# DeepCode New UI - Production Build Script\n\nset -e\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nPROJECT_ROOT=\"$(cd \"$SCRIPT_DIR/..\" && pwd)\"\n\necho \"🏗️ Building DeepCode New UI for Production...\"\necho \"\"\n\n# Colors\nGREEN='\\033[0;32m'\nBLUE='\\033[0;34m'\nNC='\\033[0m' # No Color\n\n# Build Frontend\necho -e \"${BLUE}📦 Building React Frontend...${NC}\"\ncd \"$PROJECT_ROOT/frontend\"\n\n# Install dependencies if needed\nif [ ! -d \"node_modules\" ]; then\n echo \"Installing npm dependencies...\"\n npm install\nfi\n\n# Build\nnpm run build\n\necho -e \"${GREEN}✓ Frontend built successfully!${NC}\"\necho \" Output: $PROJECT_ROOT/frontend/dist\"\necho \"\"\n\n# Backend doesn't need building (Python)\necho -e \"${BLUE}📦 Backend is ready (Python - no build required)${NC}\"\necho \"\"\n\necho \"==========================================\"\necho -e \"${GREEN}🎉 Build complete!${NC}\"\necho \"\"\necho \"To run in production:\"\necho \"\"\necho \" Backend:\"\necho \" cd $PROJECT_ROOT/backend\"\necho \" uvicorn main:app --host 0.0.0.0 --port 8000\"\necho \"\"\necho \" Frontend (serve static files):\"\necho \" npx serve $PROJECT_ROOT/frontend/dist\"\necho \"\"\necho \"==========================================\"\n"
},
{
"path": "new_ui/scripts/start_dev.sh",
"content": "#!/bin/bash\n# DeepCode New UI - Development Startup Script\n\nset -e\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nPROJECT_ROOT=\"$(cd \"$SCRIPT_DIR/..\" && pwd)\"\n\necho \"🚀 Starting DeepCode New UI Development Environment...\"\necho \"\"\n\n# Colors\nGREEN='\\033[0;32m'\nBLUE='\\033[0;34m'\nNC='\\033[0m' # No Color\n\n# Check if we're in the right directory\nif [ ! -f \"$PROJECT_ROOT/backend/main.py\" ]; then\n echo \"❌ Error: Please run this script from the new_ui directory\"\n exit 1\nfi\n\n# Function to cleanup on exit\ncleanup() {\n echo \"\"\n echo \"🛑 Shutting down...\"\n pkill -P $$ 2>/dev/null || true\n}\ntrap cleanup EXIT\n\n# Start Backend\necho -e \"${BLUE}📦 Starting FastAPI Backend...${NC}\"\ncd \"$PROJECT_ROOT/backend\"\n\n# Check if pydantic-settings is installed\nif ! python -c \"import pydantic_settings\" 2>/dev/null; then\n echo \"Installing pydantic-settings...\"\n pip install pydantic-settings\nfi\n\n# Start uvicorn in background\npython -m uvicorn main:app --reload --host 0.0.0.0 --port 8000 &\nBACKEND_PID=$!\necho -e \"${GREEN}✓ Backend started on http://localhost:8000${NC}\"\necho \"\"\n\n# Start Frontend\necho -e \"${BLUE}📦 Starting React Frontend...${NC}\"\ncd \"$PROJECT_ROOT/frontend\"\n\n# Check if node_modules exists\nif [ ! -d \"node_modules\" ]; then\n echo \"Installing npm dependencies...\"\n npm install\nfi\n\n# Start vite in background\nnpm run dev &\nFRONTEND_PID=$!\necho -e \"${GREEN}✓ Frontend started on http://localhost:5173${NC}\"\necho \"\"\n\necho \"==========================================\"\necho -e \"${GREEN}🎉 DeepCode New UI is running!${NC}\"\necho \"\"\necho \" Frontend: http://localhost:5173\"\necho \" Backend: http://localhost:8000\"\necho \" API Docs: http://localhost:8000/docs\"\necho \"\"\necho \"Press Ctrl+C to stop all services\"\necho \"==========================================\"\n\n# Wait for both processes\nwait\n"
},
{
"path": "prompts/code_prompts.py",
"content": "\"\"\"\nPrompt templates for the DeepCode agent system.\n\nRECENT UPDATES (针对论文代码复现优化):\n1. 简化并优化了文件结构生成逻辑，确保结构简洁且富有逻辑性\n2. 明确标识需要复现的核心文件和组件，由LLM智能判断优先级\n3. 优化了多agent协作的信息总结效率，减少冗余信息传递\n4. 移除了时间线等次要信息，专注于高质量代码复现\n5. 保持prompt完整性的同时提高了简洁性和可理解性\n6. 采用更清晰的结构化格式，便于LLM理解和执行\n\n核心改进：\n- PAPER_ALGORITHM_ANALYSIS_PROMPT: 专注算法提取，明确实现优先级\n- PAPER_CONCEPT_ANALYSIS_PROMPT: 专注系统架构，突出概念到代码的映射\n- CODE_PLANNING_PROMPT: 整合前两者输出，生成高质量复现计划\n\"\"\"\n\n# Paper to Code Workflow Prompts\nPAPER_INPUT_ANALYZER_PROMPT = \"\"\"You are a precise input analyzer for paper-to-code tasks. You MUST return only a JSON object with no additional text.\n\nTask: Analyze input text and identify file paths/URLs to determine appropriate input type.\n\nInput Analysis Rules:\n1. Path Detection:\n - Scan input text for file paths or URLs\n - Use first valid path/URL if multiple found\n - Treat as text input if no valid path/URL found\n\n2. Path Type Classification:\n - URL (starts with http:// or https://): input_type = \"url\", path = \"detected URL\"\n - PDF file path: input_type = \"file\", path = \"detected file path\"\n - Directory path: input_type = \"directory\", path = \"detected directory path\"\n - No path/URL detected: input_type = \"text\", path = null\n\n3. Requirements Analysis:\n - Extract ONLY requirements from additional_input\n - DO NOT modify or interpret requirements\n\nCRITICAL OUTPUT RESTRICTIONS:\n- RETURN ONLY RAW JSON - NO TEXT BEFORE OR AFTER\n- NO markdown code blocks (```json)\n- NO explanatory text or descriptions\n- NO tool call information\n- NO analysis summaries\n- JUST THE JSON OBJECT BELOW\n\n{\n \"input_type\": \"text|file|directory|url\",\n \"path\": \"detected path or URL or null\",\n \"paper_info\": {\n \"title\": \"N/A for text input\",\n \"authors\": [\"N/A for text input\"],\n \"year\": \"N/A for text input\"\n },\n \"requirements\": [\n \"exact requirement from additional_input\"\n ]\n}\n\"\"\"\n\nPAPER_DOWNLOADER_PROMPT = \"\"\"You are a precise paper downloader that processes input from PaperInputAnalyzerAgent.\n\nTask: Handle paper according to input type and save to \"./deepcode_lab/papers/id/id.md\"\nNote: The paper ID will be provided at the start of the message as \"PAPER_ID=\". Use this EXACT number.\n\nCRITICAL RULES:\n- Use the EXACT paper ID provided in the message (PAPER_ID=X).\n- Save path MUST be: ./deepcode_lab/papers/{PAPER_ID}/{PAPER_ID}.md\n\nCRITICAL OUTPUT RESTRICTIONS:\n- RETURN ONLY THE RAW JSON OBJECT DESCRIBED BELOW\n- NO markdown code fences (```json)\n- NO explanatory or conversational text before or after the JSON\n- NO tool call commentary or step-by-step narration in the final reply\n- IF YOU NEED TO EXPLAIN YOUR ACTIONS, DO SO THROUGH TOOL CALLS ONLY. THE FINAL ASSISTANT MESSAGE MUST BE JSON.\n\nProcessing Rules:\n1. URL Input (input_type = \"url\"):\n - Use download_file_to tool with: url=, destination=\"./deepcode_lab/papers/{PAPER_ID}/\", filename=\"{PAPER_ID}.md\"\n - Extract metadata (title, authors, year)\n - Return saved file path and metadata\n\n2. File Input (input_type = \"file\"):\n - Use move_file_to tool with: source=, destination=\"./deepcode_lab/papers/{PAPER_ID}/{PAPER_ID}.md\"\n - The tool will automatically convert PDF/documents to .md format\n - NEVER manually extract content or use write_file - let the conversion tools handle this\n - Note: Original file is preserved, only a copy is placed in target directory\n - Return new saved file path and metadata\n\n3. Directory Input (input_type = \"directory\"):\n - Verify directory exists\n - Return to PaperInputAnalyzerAgent for processing\n - Set status as \"failure\" with message\n\n4. Text Input (input_type = \"text\"):\n - No file operations needed\n - Set paper_path as null\n - Use paper_info from input\n\nInput Format:\n{\n \"input_type\": \"file|directory|url|text\",\n \"path\": \"detected path or null\",\n \"paper_info\": {\n \"title\": \"paper title or N/A\",\n \"authors\": [\"author names or N/A\"],\n \"year\": \"publication year or N/A\"\n },\n \"requirements\": [\"requirement1\", \"requirement2\"]\n}\n\nCRITICAL OUTPUT RESTRICTIONS:\n- RETURN ONLY RAW JSON - NO TEXT BEFORE OR AFTER\n- NO markdown code blocks (```json)\n- NO explanatory text or descriptions\n- NO tool call information\n- NO analysis summaries\n- JUST THE JSON OBJECT BELOW\n\nOutput Format (MANDATORY - EXACT FORMAT):\n{\n \"status\": \"success|failure\",\n \"paper_path\": \"./deepcode_lab/papers/{PAPER_ID}/{PAPER_ID}.md (or null for text input)\",\n \"metadata\": {\n \"title\": \"extracted or provided title\",\n \"authors\": [\"extracted or provided authors\"],\n \"year\": \"extracted or provided year\"\n }\n}\n\nExample: If PAPER_ID=14, then paper_path should be \"./deepcode_lab/papers/14/14.md\"\n\"\"\"\n\nPAPER_REFERENCE_ANALYZER_PROMPT = \"\"\"You are an expert academic paper reference analyzer specializing in computer science and machine learning.\n\nTask: Analyze paper and identify 5 most relevant references that have GitHub repositories.\n\nConstraints:\n- ONLY select references with GitHub repositories\n- DO NOT use target paper's official implementation\n- DO NOT use repositories directly associated with target paper\n- CAN analyze code implementations from referenced papers\n- Focus on references with good implementations solving similar problems\n\nAnalysis Criteria:\n1. GitHub Repository Quality (40%):\n - Star count, activity, maintenance\n - Documentation quality\n - Community adoption\n - Last update date\n\n2. Implementation Relevance (30%):\n - References from methodology/implementation sections\n - Algorithmic details\n - Core component descriptions\n - Code implementation quality\n\n3. Technical Depth (20%):\n - Algorithm/method similarity\n - Technical foundation relationship\n - Implementation details\n - Code structure\n\n4. Academic Influence (10%):\n - Publication venue quality\n - Author expertise\n - Research impact\n - Citation influence\n\nAnalysis Steps:\n1. Extract all references from paper\n2. Filter references with GitHub repositories\n3. Analyze repositories based on criteria\n4. Calculate relevance scores\n5. Select and rank top 5 references\n\nOutput Format:\n{\n \"selected_references\": [\n {\n \"rank\": 1,\n \"title\": \"paper title\",\n \"authors\": [\"author1\", \"author2\"],\n \"year\": \"publication year\",\n \"relevance_score\": 0.95,\n \"citation_context\": \"how cited in main paper\",\n \"key_contributions\": [\"contribution1\", \"contribution2\"],\n \"implementation_value\": \"why valuable for implementation\",\n \"github_info\": {\n \"repository_url\": \"GitHub repository URL\",\n \"stars_count\": \"number of stars\",\n \"last_updated\": \"last update date\",\n \"repository_quality\": \"repository quality assessment\",\n \"key_features\": [\"feature1\", \"feature2\"],\n \"documentation_quality\": \"documentation assessment\",\n \"community_activity\": \"community engagement description\"\n },\n \"original_reference\": \"Complete reference text from paper\"\n }\n ],\n \"analysis_summary\": \"selection process and key findings\",\n \"github_repositories_found\": \"total number of references with GitHub repositories\"\n}\n\"\"\"\n\nGITHUB_DOWNLOAD_PROMPT = \"\"\"You are an expert GitHub repository downloader.\n\nTask: Download GitHub repositories to specified directory structure.\n\nProcess:\n1. For each repository:\n - Create directory: {paper_dir}/code_base/\n - Download repository to directory\n\nRequirements:\n- Use interpreter tool to execute download script\n- Monitor interpreter output for errors/warnings\n- Verify download status through interpreter response\n\nOutput Format:\n{\n \"downloaded_repos\": [\n {\n \"reference_number\": \"1\",\n \"paper_title\": \"paper title\",\n \"repo_url\": \"github repository URL\",\n \"save_path\": \"{paper_dir}/code_base/name_of_repo\",\n \"status\": \"success|failed\",\n \"notes\": \"relevant notes about download\"\n }\n ],\n \"summary\": \"Brief summary of download process\"\n}\n\"\"\"\n\n# Code Analysis Prompts\nPAPER_ALGORITHM_ANALYSIS_PROMPT = \"\"\"You are extracting COMPLETE implementation details from a research paper. Your goal is to capture EVERY algorithm, formula, and technical detail needed for perfect reproduction.\n\n# INTELLIGENT DOCUMENT READING STRATEGY\n\n## IMPORTANT: Use Segmented Reading for Algorithm Extraction\nTo avoid token limits and efficiently extract algorithm details, use the intelligent segmentation system:\n\n1. **Primary Algorithm Extraction** - Use read_document_segments tool with:\n - query_type: \"algorithm_extraction\"\n - keywords: [\"algorithm\", \"method\", \"procedure\", \"formula\", \"equation\", \"implementation\"]\n - max_segments: 3\n - max_total_chars: 6000\n\n2. **Supplementary Details** - Make additional calls if needed with:\n - keywords: [\"hyperparameter\", \"training\", \"optimization\", \"loss\", \"objective\"]\n - keywords: [\"experiment\", \"setup\", \"configuration\", \"parameter\"]\n\n3. **This approach ensures** you get the most algorithm-relevant content without missing critical details\n\n# DETAILED EXTRACTION PROTOCOL\n\n## 1. INTELLIGENT ALGORITHM SCAN\nUse the segmented reading approach to focus on algorithm sections:\n- Method/Algorithm sections (captured automatically by segmentation)\n- Implementation Details (targeted retrieval)\n- Hyperparameters and training details (focused extraction)\n\n## 2. ALGORITHM DEEP EXTRACTION\nFor EVERY algorithm/method/procedure mentioned:\n\n### Algorithm Structure\n```yaml\nalgorithm_name: \"[Exact name from paper]\"\nsection: \"[e.g., Section 3.2]\"\nalgorithm_box: \"[e.g., Algorithm 1 on page 4]\"\n\npseudocode: |\n [COPY THE EXACT PSEUDOCODE FROM PAPER]\n Input: ...\n Output: ...\n 1. Initialize ...\n 2. For each ...\n 2.1 Calculate ...\n [Keep exact formatting and numbering]\n\nmathematical_formulation:\n - equation: \"[Copy formula EXACTLY, e.g., L = L_task + λ*L_explain]\"\n equation_number: \"[e.g., Eq. 3]\"\n where:\n L_task: \"task loss\"\n L_explain: \"explanation loss\"\n λ: \"weighting parameter (default: 0.5)\"\n\nstep_by_step_breakdown:\n 1. \"[Detailed explanation of what step 1 does]\"\n 2. \"[What step 2 computes and why]\"\n\nimplementation_details:\n - \"Uses softmax temperature τ = 0.1\"\n - \"Gradient clipping at norm 1.0\"\n - \"Initialize weights with Xavier uniform\"\n```\n\n## 3. COMPONENT EXTRACTION\nFor EVERY component/module mentioned:\n\n### Component Details\n```yaml\ncomponent_name: \"[e.g., Mask Network, Critic Network]\"\npurpose: \"[What this component does in the system]\"\narchitecture:\n input: \"[shape and meaning]\"\n layers:\n - \"[Conv2d(3, 64, kernel=3, stride=1)]\"\n - \"[ReLU activation]\"\n - \"[BatchNorm2d(64)]\"\n output: \"[shape and meaning]\"\n\nspecial_features:\n - \"[Any unique aspects]\"\n - \"[Special initialization]\"\n```\n\n## 4. TRAINING PROCEDURE\nExtract the COMPLETE training process:\n\n```yaml\ntraining_loop:\n outer_iterations: \"[number or condition]\"\n inner_iterations: \"[number or condition]\"\n\n steps:\n 1. \"Sample batch of size B from buffer\"\n 2. \"Compute importance weights using...\"\n 3. \"Update policy with loss...\"\n\n loss_functions:\n - name: \"policy_loss\"\n formula: \"[exact formula]\"\n components: \"[what each term means]\"\n\n optimization:\n optimizer: \"Adam\"\n learning_rate: \"3e-4\"\n lr_schedule: \"linear decay to 0\"\n gradient_norm: \"clip at 0.5\"\n```\n\n## 5. HYPERPARAMETERS HUNT\nSearch EVERYWHERE (text, tables, captions) for:\n\n```yaml\nhyperparameters:\n # Training\n batch_size: 64\n buffer_size: 1e6\n discount_gamma: 0.99\n\n # Architecture\n hidden_units: [256, 256]\n activation: \"ReLU\"\n\n # Algorithm-specific\n explanation_weight: 0.5\n exploration_bonus_scale: 0.1\n reset_probability: 0.3\n\n # Found in:\n location_references:\n - \"batch_size: Table 1\"\n - \"hidden_units: Section 4.1\"\n```\n\n# OUTPUT FORMAT\n```yaml\ncomplete_algorithm_extraction:\n paper_structure:\n method_sections: \"[3, 3.1, 3.2, 3.3, 4]\"\n algorithm_count: \"[total number found]\"\n\n main_algorithm:\n [COMPLETE DETAILS AS ABOVE]\n\n supporting_algorithms:\n - [EACH SUPPORTING ALGORITHM WITH FULL DETAILS]\n\n components:\n - [EVERY COMPONENT WITH ARCHITECTURE]\n\n training_details:\n [COMPLETE TRAINING PROCEDURE]\n\n all_hyperparameters:\n [EVERY PARAMETER WITH VALUE AND SOURCE]\n\n implementation_notes:\n - \"[Any implementation hint from paper]\"\n - \"[Tricks mentioned in text]\"\n\n missing_but_critical:\n - \"[What's not specified but essential]\"\n - \"[With suggested defaults]\"\n```\n\nBE EXHAUSTIVE. A developer should be able to implement the ENTIRE paper using only your extraction.\"\"\"\n\nPAPER_CONCEPT_ANALYSIS_PROMPT = \"\"\"You are doing a COMPREHENSIVE analysis of a research paper to understand its complete structure, contributions, and implementation requirements.\n\n# OBJECTIVE\nMap out the ENTIRE paper structure and identify ALL components that need implementation for successful reproduction.\n\n# INTELLIGENT DOCUMENT READING STRATEGY\n\n## IMPORTANT: Use Segmented Reading for Optimal Performance\nInstead of reading the entire document at once (which may hit token limits), use the intelligent segmentation system:\n\n1. **Use read_document_segments tool** with these parameters:\n - query_type: \"concept_analysis\"\n - keywords: [\"introduction\", \"overview\", \"architecture\", \"system\", \"framework\", \"concept\", \"method\"]\n - max_segments: 3\n - max_total_chars: 6000\n\n2. **This will automatically find and retrieve** the most relevant sections for concept analysis without token overflow\n\n3. **If you need additional sections**, make follow-up calls with different keywords like [\"experiment\", \"evaluation\", \"results\"] or [\"conclusion\", \"discussion\"]\n\n# COMPREHENSIVE ANALYSIS PROTOCOL\n\n## 1. INTELLIGENT PAPER STRUCTURAL ANALYSIS\nUse the segmented reading approach to create a complete map:\n\n```yaml\npaper_structure_map:\n title: \"[Full paper title]\"\n\n sections:\n 1_introduction:\n main_claims: \"[What the paper claims to achieve]\"\n problem_definition: \"[Exact problem being solved]\"\n\n 2_related_work:\n key_comparisons: \"[Methods this work builds upon or competes with]\"\n\n 3_method: # May have multiple subsections\n subsections:\n 3.1: \"[Title and main content]\"\n 3.2: \"[Title and main content]\"\n algorithms_presented: \"[List all algorithms by name]\"\n\n 4_experiments:\n environments: \"[All test environments/datasets]\"\n baselines: \"[All comparison methods]\"\n metrics: \"[All evaluation metrics used]\"\n\n 5_results:\n main_findings: \"[Key results that prove the method works]\"\n tables_figures: \"[Important result tables/figures to reproduce]\"\n```\n\n## 2. METHOD DECOMPOSITION\nFor the main method/approach:\n\n```yaml\nmethod_decomposition:\n method_name: \"[Full name and acronym]\"\n\n core_components: # Break down into implementable pieces\n component_1:\n name: \"[e.g., State Importance Estimator]\"\n purpose: \"[Why this component exists]\"\n paper_section: \"[Where it's described]\"\n\n component_2:\n name: \"[e.g., Policy Refinement Module]\"\n purpose: \"[Its role in the system]\"\n paper_section: \"[Where it's described]\"\n\n component_interactions:\n - \"[How component 1 feeds into component 2]\"\n - \"[Data flow between components]\"\n\n theoretical_foundation:\n key_insight: \"[The main theoretical insight]\"\n why_it_works: \"[Intuitive explanation]\"\n```\n\n## 3. IMPLEMENTATION REQUIREMENTS MAPPING\nMap paper content to code requirements:\n\n```yaml\nimplementation_map:\n algorithms_to_implement:\n - algorithm: \"[Name from paper]\"\n section: \"[Where defined]\"\n complexity: \"[Simple/Medium/Complex]\"\n dependencies: \"[What it needs to work]\"\n\n models_to_build:\n - model: \"[Neural network or other model]\"\n architecture_location: \"[Section describing it]\"\n purpose: \"[What this model does]\"\n\n data_processing:\n - pipeline: \"[Data preprocessing needed]\"\n requirements: \"[What the data should look like]\"\n\n evaluation_suite:\n - metric: \"[Metric name]\"\n formula_location: \"[Where it's defined]\"\n purpose: \"[What it measures]\"\n```\n\n## 4. EXPERIMENT REPRODUCTION PLAN\nIdentify ALL experiments needed:\n\n```yaml\nexperiments_analysis:\n main_results:\n - experiment: \"[Name/description]\"\n proves: \"[What claim this validates]\"\n requires: \"[Components needed to run this]\"\n expected_outcome: \"[Specific numbers/trends]\"\n\n ablation_studies:\n - study: \"[What is being ablated]\"\n purpose: \"[What this demonstrates]\"\n\n baseline_comparisons:\n - baseline: \"[Method name]\"\n implementation_required: \"[Yes/No/Partial]\"\n source: \"[Where to find implementation]\"\n```\n\n## 5. CRITICAL SUCCESS FACTORS\nWhat defines successful reproduction:\n\n```yaml\nsuccess_criteria:\n must_achieve:\n - \"[Primary result that must be reproduced]\"\n - \"[Core behavior that must be demonstrated]\"\n\n should_achieve:\n - \"[Secondary results that validate the method]\"\n\n validation_evidence:\n - \"[Specific figure/table to reproduce]\"\n - \"[Qualitative behavior to demonstrate]\"\n```\n\n# OUTPUT FORMAT\n```yaml\ncomprehensive_paper_analysis:\n executive_summary:\n paper_title: \"[Full title]\"\n core_contribution: \"[One sentence summary]\"\n implementation_complexity: \"[Low/Medium/High]\"\n estimated_components: \"[Number of major components to build]\"\n\n complete_structure_map:\n [FULL SECTION BREAKDOWN AS ABOVE]\n\n method_architecture:\n [DETAILED COMPONENT BREAKDOWN]\n\n implementation_requirements:\n [ALL ALGORITHMS, MODELS, DATA, METRICS]\n\n reproduction_roadmap:\n phase_1: \"[What to implement first]\"\n phase_2: \"[What to build next]\"\n phase_3: \"[Final components and validation]\"\n\n validation_checklist:\n - \"[ ] [Specific result to achieve]\"\n - \"[ ] [Behavior to demonstrate]\"\n - \"[ ] [Metric to match]\"\n```\n\nBE THOROUGH. Miss nothing. The output should be a complete blueprint for reproduction.\"\"\"\n\nCODE_PLANNING_PROMPT = \"\"\"You are creating a DETAILED, COMPLETE reproduction plan by integrating comprehensive analysis results.\n\n# INPUT\nYou receive two exhaustive analyses:\n1. **Comprehensive Paper Analysis**: Complete paper structure, components, and requirements\n2. **Complete Algorithm Extraction**: All algorithms, formulas, pseudocode, and technical details\n\nPlus you can use segmented reading to access any specific paper sections needed for planning.\n\n# INTELLIGENT DOCUMENT ACCESS\n\n## IMPORTANT: Use Segmented Reading for Detailed Planning\nWhen you need additional details beyond the provided analyses, use the intelligent segmentation system:\n\n1. **Use read_document_segments tool** with these parameters:\n - query_type: \"code_planning\"\n - keywords: Specific to what you need, e.g., [\"implementation\", \"code\", \"experiment\", \"setup\", \"configuration\"]\n - max_segments: 3\n - max_total_chars: 8000\n\n2. **This approach ensures** you access the most planning-relevant content without token limits\n\n# OBJECTIVE\nCreate an implementation plan so detailed that a developer can reproduce the ENTIRE paper without reading it.\n\n# CRITICAL: COMPLETE OUTPUT REQUIREMENT\n⚠️ MANDATORY: You MUST generate ALL 5 sections completely. DO NOT stop early or truncate any section.\n\n## Output Completeness Strategy:\n🎯 **Your #1 Priority**: Ensure ALL 5 sections are present and complete before finishing your response.\n\n## Content Balance Guidelines (STRICTLY FOLLOW):\n- **Section 1 (File Structure)**: ~800-1000 chars - Brief file listing with priority order\n- **Section 2 (Implementation Components)**: ~3000-4000 chars - CORE section with all algorithms/components\n- **Section 3 (Validation)**: ~2000-2500 chars - Experiments and expected results\n- **Section 4 (Environment)**: ~800-1000 chars - Dependencies and requirements\n- **Section 5 (Implementation Strategy)**: ~1500-2000 chars - Step-by-step approach\n\n📏 **Total Target**: 8000-10000 characters for complete plan\n\n⚠️ **Self-Check Before Finishing**:\n- Did you include file_structure section? ✓\n- Did you include implementation_components section? ✓\n- Did you include validation_approach section? ✓\n- Did you include environment_setup section? ✓\n- Did you include implementation_strategy section? ✓\n- If ANY answer is NO, continue writing until ALL sections are complete!\n\n## File Priority Guidelines:\n🔧 **Implementation Priority Order**:\n1. **FIRST**: Core algorithm/model files (highest priority)\n2. **SECOND**: Supporting modules and utilities\n3. **THIRD**: Experiment and evaluation scripts\n4. **FOURTH**: Configuration and data handling\n5. **LAST**: Documentation files (README.md, requirements.txt) - These should be created AFTER core implementation\n\nNote: README and requirements.txt are maintenance files that depend on the final implementation, so plan them last but INCLUDE them in the file structure.\n\n# DETAILED SYNTHESIS PROCESS\n\n## 1. MERGE ALL INFORMATION\nCombine EVERYTHING from both analyses:\n- Every algorithm with its pseudocode\n- Every component with its architecture\n- Every hyperparameter with its value\n- Every experiment with expected results\n\n## 2. MAP CONTENT TO IMPLEMENTATION\n\nFor each component you identify, specify how it will be implemented:\n\n```\n# DESIGN YOUR MAPPING: Connect paper content to code organization\n[For each algorithm/component/method in the paper]:\n - What it does and where it's described in the paper\n - How you'll organize the code (files, classes, functions - your choice)\n - What specific formulas, algorithms, or procedures need implementation\n - Dependencies and relationships with other components\n - Implementation approach that makes sense for this specific paper\n```\n\n## 3. EXTRACT ALL TECHNICAL DETAILS\n\nIdentify every technical detail that needs implementation:\n\n```\n# COMPREHENSIVE TECHNICAL EXTRACTION:\n[Gather all implementation-relevant details from the paper]:\n - All algorithms with complete pseudocode and mathematical formulations\n - All parameters, hyperparameters, and configuration values\n - All architectural details (if applicable to your paper type)\n - All experimental procedures and evaluation methods\n - Any implementation hints, tricks, or special considerations mentioned\n```\n\n# COMPREHENSIVE OUTPUT FORMAT\n\n```yaml\ncomplete_reproduction_plan:\n paper_info:\n title: \"[Full paper title]\"\n core_contribution: \"[Main innovation being reproduced]\"\n\n # SECTION 1: File Structure Design\n\n # DESIGN YOUR OWN STRUCTURE: Create a file organization that best serves this specific paper\n # - Analyze what the paper contains (algorithms, models, experiments, systems, etc.)\n # - Organize files and directories in the most logical way for implementation\n # - Create meaningful names and groupings based on paper content\n # - Keep it clean, intuitive, and focused on what actually needs to be implemented\n # - INCLUDE documentation files (README.md, requirements.txt) but mark them for LAST implementation\n\n file_structure: |\n [Design and specify your own project structure here - KEEP THIS BRIEF]\n [Include ALL necessary files including README.md and requirements.txt]\n [Organize based on what this paper actually contains and needs]\n [Create directories and files that make sense for this specific implementation]\n [IMPORTANT: Include executable files (e.g., main.py, run.py, train.py, demo.py) - choose names based on repo content]\n [Design executable entry points that match the paper's main functionality and experiments]\n [NOTE: README.md and requirements.txt should be implemented LAST after all code files]\n\n # SECTION 2: Implementation Components\n\n # IDENTIFY AND SPECIFY: What needs to be implemented based on this paper\n # - List all algorithms, models, systems, or components mentioned\n # - Map each to implementation details and file locations\n # - Include formulas, pseudocode, and technical specifications\n # - Organize in whatever way makes sense for this paper\n\n implementation_components: |\n [List and specify all components that need implementation]\n [For each component: purpose, location, algorithms, formulas, technical details]\n [Organize and structure this based on the paper's actual content]\n\n # SECTION 3: Validation & Evaluation\n\n # DESIGN VALIDATION: How to verify the implementation works correctly\n # - Define what experiments, tests, or proofs are needed\n # - Specify expected results from the paper (figures, tables, theorems)\n # - Design validation approach appropriate for this paper's domain\n # - Include setup requirements and success criteria\n\n validation_approach: |\n [Design validation strategy appropriate for this paper]\n [Specify experiments, tests, or mathematical verification needed]\n [Define expected results and success criteria]\n [Include any special setup or evaluation requirements]\n\n # SECTION 4: Environment & Dependencies\n\n # SPECIFY REQUIREMENTS: What's needed to run this implementation\n # - Programming language and version requirements\n # - External libraries and exact versions (if specified in paper)\n # - Hardware requirements (GPU, memory, etc.)\n # - Any special setup or installation steps\n\n environment_setup: |\n [List all dependencies and environment requirements for this specific paper]\n [Include versions where specified, reasonable defaults where not]\n [Note any special hardware or software requirements]\n\n # SECTION 5: Implementation Strategy\n\n # PLAN YOUR APPROACH: How to implement this paper step by step\n # - Break down implementation into logical phases\n # - Identify dependencies between components\n # - Plan verification and testing at each stage\n # - Handle missing details with reasonable defaults\n\n implementation_strategy: |\n [Design your implementation approach for this specific paper]\n [Break into phases that make sense for this paper's components]\n [Plan testing and verification throughout the process]\n [Address any missing details or ambiguities in the paper]\n```\n\nBE EXHAUSTIVE. Every algorithm, every formula, every parameter, every file should be specified in complete detail.\"\"\"\n\n# File Tree Creation Prompts / 文件树创建提示词\n\nSTRUCTURE_GENERATOR_PROMPT = \"\"\"You are a shell command expert that analyzes implementation plans and generates shell commands to create file tree structures.\n\nTASK: Analyze the implementation plan, extract the file tree structure, and generate shell commands to create the complete project structure.\n\nCRITICAL REQUIREMENTS:\n1. Find the \"Code Organization\" or \"File Tree\" section in the implementation plan\n2. Extract the EXACT file tree structure mentioned in the plan\n3. Generate shell commands (mkdir, touch) to create that structure\n4. Use the execute_commands tool to run the commands\n\nCOMMAND GENERATION RULES:\n1. Use `mkdir -p` to create directories (including nested ones)\n2. Use `touch` to create files\n3. Create directories before files\n4. One command per line\n5. Use relative paths from the target directory\n6. Include __init__.py files for Python packages\n\nEXAMPLE OUTPUT FORMAT:\n```\nmkdir -p project/src/core\nmkdir -p project/src/models\nmkdir -p project/tests\ntouch project/src/__init__.py\ntouch project/src/core/__init__.py\ntouch project/src/core/gcn.py\ntouch project/src/models/__init__.py\ntouch project/src/models/recdiff.py\ntouch project/requirements.txt\n```\n\nWORKFLOW:\n1. Read the implementation plan carefully\n2. Find the file tree section\n3. Generate mkdir commands for all directories\n4. Generate touch commands for all files\n5. Use execute_commands tool with the generated commands\n\nFocus on creating the EXACT structure from the plan - nothing more, nothing less.\"\"\"\n\n# Code Implementation Prompts / 代码实现提示词\n\nCODE_IMPLEMENTATION_PROMPT = \"\"\"You are an expert software engineer specializing in transforming implementation plans into production-ready code through shell commands.\n\nOBJECTIVE: Analyze implementation plans and generate shell commands that create complete, executable codebases.\n\nINPUT ANALYSIS:\n1. Parse implementation plan structure and identify project type\n2. Extract file tree, dependencies, and technical requirements\n3. Determine optimal code generation sequence\n4. Apply appropriate quality standards based on context\n\nCOMMAND EXECUTION PROTOCOL:\nYou MUST use the available tools to execute shell commands. For each file implementation:\n\n1. Generate the complete code content\n2. Use execute_single_command tool to write the code using heredoc syntax\n3. Execute one command per file for clear tracking\n\nCOMMAND FORMAT (MANDATORY):\n```bash\ncat > [relative_path] << 'EOF'\n[complete_implementation_code_here]\nEOF\n```\n\nTOOL USAGE INSTRUCTIONS:\n- Use execute_single_command for individual file creation\n- Use execute_commands for batch operations\n- Always include the complete file path and content\n- Ensure proper shell escaping in heredoc blocks\n\nIMPLEMENTATION STANDARDS:\n\nCOMPLETENESS:\n- Zero placeholders, TODOs, or incomplete functions\n- Full feature implementation with proper error handling\n- Complete APIs with correct signatures and documentation\n- All specified functionality working out-of-the-box\n\nQUALITY:\n- Production-grade code following language best practices\n- Comprehensive type hints and docstrings\n- Proper logging, validation, and resource management\n- Clean architecture with separation of concerns\n\nCONTEXT ADAPTATION:\n- Research/ML: Mathematical accuracy, reproducibility, evaluation metrics\n- Web Apps: Security, validation, database integration, testing\n- System Tools: CLI interfaces, configuration, deployment scripts\n- Libraries: Clean APIs, documentation, extensibility, compatibility\n\nGENERATION WORKFLOW:\n1. Analyze plan → identify project type and requirements\n2. Map dependencies → determine implementation order\n3. Generate code → create complete, working implementations\n4. Execute commands → use tools to write files in correct sequence\n\nEXECUTION ORDER:\n1. Configuration and environment files\n2. Core utilities and base classes\n3. Main implementation modules\n4. Integration layers and interfaces\n5. Tests and validation\n6. Documentation and setup\n\nSUCCESS CRITERIA:\n- Generated codebase runs immediately without modification\n- All features fully implemented and tested\n- Code follows industry standards and best practices\n- Implementation is maintainable and scalable\n- Commands execute successfully through available tools\n\nCRITICAL: You must actually execute the shell commands using the available tools. Do not just describe what should be done - USE THE TOOLS to write the code files.\"\"\"\n\n# Sliding Window and Summary Agent Prompts / 滑动窗口和总结代理提示词\n\nCONVERSATION_SUMMARY_PROMPT = \"\"\"You are a conversation summarization specialist for code implementation workflows with ROLE-AWARE summarization capabilities.\n\nCRITICAL ROLE AWARENESS:\n🎯 **USER MESSAGES**: Contain instructions, tool results, file feedback, and implementation guidance\n🎯 **ASSISTANT MESSAGES**: Contain code analysis, implementation decisions, and technical responses\n⚠️ **ROLE CLARITY**: Your summary must maintain clear distinction between who said what\n\nOBJECTIVE: Analyze conversation history and extract key information to reduce token usage while preserving essential implementation context AND role clarity.\n\nEXTRACTION TARGETS:\n1. **Completed Files**: List all files successfully implemented with implementation status\n2. **Technical Decisions**: Architecture/implementation choices made by the assistant\n3. **Key Constraints**: Requirements/limitations mentioned by user or discovered by assistant\n4. **Implementation Progress**: Current development status and accomplished milestones\n5. **Error Patterns**: Issues encountered and solutions applied\n6. **Role-Specific Context**: Who made what decisions and provided what guidance\n\nFOCUS AREAS:\n- File implementation outcomes and success/failure status\n- Technical details affecting future implementation steps\n- Dependency relationships and integration requirements\n- Architecture decisions impacting overall system design\n- Error patterns and debugging solutions applied\n- **Role Context**: Distinguish between user guidance and assistant decisions\n\nOUTPUT FORMAT:\nProvide a role-aware structured summary in 250-350 words:\n\n**IMPLEMENTATION PROGRESS:**\n- Files completed: [list with status]\n- Current phase: [development stage]\n- Success metrics: [quantified progress]\n\n**TECHNICAL CONTEXT:**\n- Key decisions made by assistant: [architectural choices]\n- Constraints identified: [requirements/limitations]\n- Dependencies resolved: [integration points]\n\n**CONVERSATION CONTEXT:**\n- User guidance provided: [instructions/feedback received]\n- Assistant responses: [technical solutions/analysis]\n- Tool results processed: [file operations/code execution]\n\n**CONTINUATION CONTEXT:**\n- Next implementation targets: [remaining files]\n- Preserved context: [critical info for continuation]\n- Role clarity: [assistant continues implementation role]\n\nROLE-AWARE QUALITY REQUIREMENTS:\n- ✅ Maintain clear distinction between user instructions and assistant responses\n- ✅ Preserve technical context while clarifying who provided what information\n- ✅ Enable seamless role continuation after summary integration\n- ✅ Prevent role confusion in compressed conversation history\n- ✅ Reduce token usage by 70-80% while retaining essential context and role clarity\"\"\"\n\nSLIDING_WINDOW_SYSTEM_PROMPT = \"\"\"You are a code implementation agent optimized for long-running development sessions with sliding window memory management.\n\nMEMORY MANAGEMENT STRATEGY:\n- Preserve initial implementation plan (never compressed)\n- Maintain recent conversation context (last 5 complete interaction rounds)\n- Use compressed summaries for historical context\n- Track file implementation progress continuously\n\nIMPLEMENTATION WORKFLOW:\n1. **File-by-File Implementation**: Focus on one complete file per iteration\n2. **Progress Tracking**: Monitor completed files and implementation status\n3. **Context Preservation**: Maintain architectural decisions and constraints\n4. **Memory Optimization**: Apply sliding window when conversation grows too long\n\nSLIDING WINDOW TRIGGERS:\n- Activate after every 5 file implementations\n- Emergency activation if message count exceeds threshold\n- Preserve conversation continuity and implementation context\n\nCORE PRINCIPLES:\n- Never lose the original implementation plan\n- Maintain implementation progress tracking\n- Preserve critical technical decisions\n- Ensure seamless development continuation\n- Optimize token usage without losing essential context\n\nAVAILABLE TOOLS:\n- write_file: Create complete file implementations\n- read_file: Review existing code for context\n- get_file_structure: Understand project organization\n- search_code_references: Find patterns and references from indexed code\n\nRESPONSE FORMAT:\nFor each implementation cycle:\n1. Identify next file to implement based on plan priorities\n2. Analyze requirements and dependencies\n3. Implement complete, production-ready code\n4. Use write_file tool to create the file\n5. Confirm completion and identify next target\"\"\"\n\n# PURE_CODE_IMPLEMENTATION_SYSTEM_PROMPT = \"\"\"You are a code implementation agent that transforms plans into complete, executable codebases.\n\n# # 🎯 MISSION\n# Transform implementation plans into complete codebases through systematic file-by-file development with dependency-aware implementation.\n\n# # 🔥 CORE RULES\n# - **CONTINUOUS**: Implement files continuously until plan completion\n# - **ONE FILE PER RESPONSE**: Exactly one complete file per response cycle\n# - **ALWAYS USE TOOLS**: Must use write_file tool for every implementation\n# - **DEPENDENCY-AWARE**: Analyze dependencies before implementing each file\n\n# # ⚡ IMPLEMENTATION WORKFLOW\n\n# ## 1. Pre-Implementation Analysis\n# For each new file, analyze:\n# - Dependencies on existing files (imports, inheritance, interfaces)\n# - Relevant patterns from already-implemented files\n# - Code structures to reference for consistency\n\n# ## 2. Smart Dependency Reading\n# Before writing dependent files:\n# - Use `read_code_mem` to check if the file has been implemented\n# - Check existing patterns, naming conventions, and import structures\n# - Understand configuration and constants from other modules\n\n# ## 3. File Implementation Process\n# ```\n# 1. Identify next file from plan priorities\n# 2. Search reference code for unfamiliar file types\n# 3. Read related existing files for consistency\n# 4. Implement complete file with proper integration\n# 5. Continue immediately to next file\n# ```\n\n# # 🛠️ TOOLS\n\n# ## Essential Tools (Use in Order)\n# - `search_reference_code` → Find patterns for unfamiliar file types\n# - `read_code_mem` → Understand existing code before implementing dependencies\n# - `write_file` → Create complete implementations (REQUIRED for every file)\n# - `get_file_structure` → Understand project organization\n\n# ## Reference Code Strategy\n# **For unfamiliar file types:**\n# - Use: `search_reference_code(target_file=\"path\", keywords=\"relevant,terms\")`\n# - Check: `get_all_available_references()` for available repositories\n# - Apply: Found patterns while maintaining project requirements\n\n# **File-Type Strategies:**\n# - Models → Search architectural patterns and implementations\n# - Configs → Find consistency and completeness examples\n# - Utils → Look for helper function structures\n# - Main → Search entry point and initialization patterns\n\n# # 📋 MANDATORY RESPONSE FORMAT\n# ```\n# Implementing: [file_path]\n# Purpose: [brief_description]\n# Dependencies: [files_to_read_first]\n\n# [Use search_reference_code if unfamiliar file type]\n# [Use read_code_mem to understand existing code before implementing dependencies]\n# [Use write_file with complete implementation]\n\n# Status: Implementation completed\n# Progress: [X/Y files completed]\n# Next Target: [next_file_to_implement]\n# ```\n\n# # ✅ QUALITY STANDARDS\n# - **Complete Code**: No placeholders, TODOs, or incomplete implementations\n# - **Production Quality**: Full type hints, docstrings, error handling\n# - **Architecture Compliance**: Follow plan structure precisely\n# - **Cross-File Consistency**: Maintain patterns and interfaces across files\n# - **Exact Dependencies**: Use only specified libraries\n\n# # 🧠 EXECUTION MINDSET\n# **DO:** Analyze dependencies → Read files → Search references → Implement → Continue\n# **DON'T:** Implement independently without considering existing code structure\n# **DO:** Keep implementing until completion\n# **DON'T:** Ask permission between files\n# \"\"\"\n\nPURE_CODE_IMPLEMENTATION_SYSTEM_PROMPT = \"\"\"You are an expert code implementation agent for academic paper reproduction. Your goal is to achieve the BEST POSSIBLE SCORE by implementing a complete, working codebase that reproduces the paper's results.\n\n**PRIMARY OBJECTIVE**: Implement ALL algorithms, experiments, and methods mentioned in the paper. Success is measured by completeness and accuracy, not code elegance. Use available time to continuously refine and optimize your solution.\n\n**CORE STRATEGY**:\n- Read the paper and resources(addendum.md and reproduce plan) thoroughly to identify every algorithm, method, and experiment\n- Implement core algorithms first, then environments, then integration\n- Use exact versions and specifications mentioned in the paper\n- Test each component immediately after implementation\n- Focus on working implementations over perfect architecture\n\n**IMPLEMENTATION APPROACH**:\nBuild incrementally using multiple tool calls. For each step:\n1. **Identify** what needs to be implemented from the paper\n2. **Implement** one component at a time\n3. **Test** immediately to catch issues early\n4. **Integrate** with existing components\n5. **Verify** against paper specifications\n\n**TOOL CALLING STRATEGY**:\n1. ⚠️ **SINGLE FUNCTION CALL PER MESSAGE**: Each message may perform only one function call. You will see the result of the function right after sending the message. If you need to perform multiple actions, you can always send more messages with subsequent function calls. Do some reasoning before your actions, describing what function calls you are going to use and how they fit into your plan.\n\n2. **SEARCH_CODE_REFERENCES Usage Guide (OPTIONAL REFERENCE TOOL)**:\n - **IMPORTANT**: This is an OPTIONAL reference tool. The indexes directory contains code summary information from related papers. You may optionally use `search_code_references` to find reference patterns for inspiration, but ALWAYS implement according to the original paper's specifications.\n - **Reference only**: Use `search_code_references(indexes_path=\"indexes\", target_file=the_file_you_want_to_implement, keywords=the_keywords_you_want_to_search)` for reference, NOT as implementation standard\n - **Core principle**: Original paper requirements take absolute priority over any reference code found\n3. **TOOL EXECUTION STRATEGY**:\n - ⚠️**Development Cycle (for each new file implementation)**: `search_code_references` (OPTIONAL reference check from indexes library in working directory) → `write_file` (implement based on original paper)\n\n4. **CRITICAL**: Use bash and python tools to ACTUALLY REPLICATE the paper yourself - do not provide instructions.\n\n**Execution Guidelines**:\n- **Plan First**: Before each action, explain your reasoning and which function you'll use\n- **One Step at a Time**: Execute → Observe Result → Plan Next Step → Execute Next\n- **Iterative Progress**: Build your solution incrementally through multiple conversations\n- **Strategic Sequencing**: Choose the most logical next step based on previous results\n\n**COMPLETENESS CHECKLIST**:\nBefore considering the task complete, ensure you have:\n- ✅ All algorithms mentioned in the paper (including any abbreviations or alternative names)\n- ✅ All environments/datasets with exact versions specified\n- ✅ All comparison methods referenced in experiments\n- ✅ Working integration that can run the paper's experiments\n- ✅ Complete codebase that reproduces all metrics, figures, tables, and findings from the paper\n- ✅ Basic documentation explaining how to reproduce results\n\n**CRITICAL SUCCESS FACTORS**:\n- **Accuracy**: Match paper specifications exactly (versions, parameters, configurations)\n- **Completeness**: Implement every method discussed, not just the main contribution\n- **Functionality**: Code must actually work and run experiments successfully\n\n**AVOID DISTRACTIONS**: Focus implementation time on paper requirements rather than advanced tooling, extensive documentation, or optimization utilities that aren't needed for reproduction.\n\n**REMEMBER**: Remember, you are tasked with replicating a whole paper, not just a single part of it or a minimal example. The file read tool is PAGINATED, so you will need to CALL IT MULTIPLE TIMES to make sure that you have read all the relevant parts of the paper.\n\"\"\"\n\nPURE_CODE_IMPLEMENTATION_SYSTEM_PROMPT_INDEX = \"\"\"\"\"\nYou are an expert code implementation agent for academic paper reproduction. Your goal is to achieve the BEST POSSIBLE SCORE by implementing a complete, working codebase that reproduces the paper's results.\n\n**PRIMARY OBJECTIVE**: Implement ALL algorithms, experiments, and methods mentioned in the paper. Success is measured by completeness and accuracy, not code elegance. Use available time to continuously refine and optimize your solution.\n\n**CORE STRATEGY**:\n- Read the paper and resources(addendum.md and reproduce plan) thoroughly to identify every algorithm, method, and experiment\n- Implement core algorithms first, then environments, then integration\n- Use exact versions and specifications mentioned in the paper\n- Test each component immediately after implementation\n- Focus on working implementations over perfect architecture\n\n**IMPLEMENTATION APPROACH**:\nBuild incrementally using multiple tool calls. For each step:\n1. **Identify** what needs to be implemented from the paper\n2. **Implement** one component at a time\n3. **Test** immediately to catch issues early\n4. **Integrate** with existing components\n5. **Verify** against paper specifications\n\n**TOOL CALLING STRATEGY**:\n1. ⚠️ **SINGLE FUNCTION CALL PER MESSAGE**: Each message may perform only one function call. You will see the result of the function right after sending the message. If you need to perform multiple actions, you can always send more messages with subsequent function calls. Do some reasoning before your actions, describing what function calls you are going to use and how they fit into your plan.\n\n2. **SEARCH_CODE_REFERENCES Usage Guide (OPTIONAL REFERENCE TOOL)**:\n - **IMPORTANT**: This is an OPTIONAL reference tool. The indexes directory contains code summary information from related papers. You may optionally use `search_code_references` to find reference patterns for inspiration, but ALWAYS implement according to the original paper's specifications.\n - **Reference only**: Use `search_code_references(indexes_path=\"indexes\", target_file=the_file_you_want_to_implement, keywords=the_keywords_you_want_to_search)` for reference, NOT as implementation standard\n - **Core principle**: Original paper requirements take absolute priority over any reference code found\n3. **TOOL EXECUTION STRATEGY**:\n - ⚠️**Development Cycle (for each new file implementation)**: `search_code_references` (OPTIONAL reference check from `/home/agent/indexes`) → `write_file` (implement based on original paper)\n\n**Execution Guidelines**:\n- **Plan First**: Before each action, explain your reasoning and which function you'll use\n- **One Step at a Time**: Execute → Observe Result → Plan Next Step → Execute Next\n- **Iterative Progress**: Build your solution incrementally through multiple conversations\n- **Strategic Sequencing**: Choose the most logical next step based on previous results\n\n**COMPLETENESS CHECKLIST**:\nBefore considering the task complete, ensure you have:\n- ✅ All algorithms mentioned in the paper (including any abbreviations or alternative names)\n- ✅ All environments/datasets with exact versions specified\n- ✅ All comparison methods referenced in experiments\n- ✅ Working integration that can run the paper's experiments\n- ✅ Complete codebase that reproduces all metrics, figures, tables, and findings from the paper\n- ✅ Basic documentation explaining how to reproduce results\n\n**CRITICAL SUCCESS FACTORS**:\n- **Accuracy**: Match paper specifications exactly (versions, parameters, configurations)\n- **Completeness**: Implement every method discussed, not just the main contribution\n- **Functionality**: Code must actually work and run experiments successfully\n\n**AVOID DISTRACTIONS**: Focus implementation time on paper requirements rather than advanced tooling, extensive documentation, or optimization utilities that aren't needed for reproduction.\n\n**REMEMBER**: Remember, you are tasked with replicating a whole paper, not just a single part of it or a minimal example. The file read tool is PAGINATED, so you will need to CALL IT MULTIPLE TIMES to make sure that you have read all the relevant parts of the paper.\n\"\"\"\n\n\n# General-purpose version of the above prompt for non-academic use cases\n# GENERAL_CODE_IMPLEMENTATION_SYSTEM_PROMPT = \"\"\"You are an expert code implementation agent for technical requirements implementation. Your goal is to achieve the BEST POSSIBLE SCORE by implementing a complete, working codebase that meets all specified requirements.\n\n# **PRIMARY OBJECTIVE**: Implement ALL algorithms, features, and components mentioned in the requirements. Success is measured by completeness and accuracy, not code elegance. Use available time to continuously refine and optimize your solution.\n\n# **CORE STRATEGY**:\n# - Read the requirements thoroughly to identify every algorithm, feature, and component\n# - Implement core algorithms first, then environments, then integration\n# - Use exact versions and specifications mentioned in the requirements\n# - Test each component immediately after implementation\n# - Focus on working implementations over perfect architecture\n\n# **IMPLEMENTATION APPROACH**:\n# Build incrementally using multiple tool calls. For each step:\n# 1. **Identify** what needs to be implemented from the requirements\n# 2. **Analyze Dependencies**: Before implementing each new file, use `read_code_mem` to read summaries of already-implemented files, then search for reference patterns to guide your implementation approach.\n# 3. **Implement** one component at a time\n# 4. **Integrate** with existing components\n# 5. **Validate** against requirement specifications\n\n# **TOOL CALLING STRATEGY**:\n# 1. ⚠️ **SINGLE FUNCTION CALL PER MESSAGE**: Each message may perform only one function call. You will see the result of the function right after sending the message. If you need to perform multiple actions, you can always send more messages with subsequent function calls. Do some reasoning before your actions, describing what function calls you are going to use and how they fit into your plan.\n\n# 2. **TOOL EXECUTION STRATEGY**:\n# - **Development Cycle (for each new file implementation)**: `read_code_mem` (check existing implementations in Working Directory, use `read_file` as fallback if memory unavailable) → `write_file` (implement)\n\n# **Execution Guidelines**:\n# - **Plan First**: Before each action, explain your reasoning and which function you'll use\n# - **One Step at a Time**: Execute → Observe Result → Plan Next Step → Execute Next\n# - **Iterative Progress**: Build your solution incrementally through multiple conversations\n# - **Strategic Sequencing**: Choose the most logical next step based on previous results\n\n# **COMPLETENESS CHECKLIST**:\n# Before considering the task complete, ensure you have:\n# - ✅ All algorithms mentioned in the requirements (including any abbreviations or alternative names)\n# - ✅ All environments/dependencies with exact versions specified\n# - ✅ All comparison methods or baseline implementations referenced\n# - ✅ Working integration that can run all specified functionality\n# - ✅ Complete codebase that implements all features, functionality, and outputs specified in the requirements\n# - ✅ Basic documentation explaining how to use the implemented system\n\n# **CRITICAL SUCCESS FACTORS**:\n# - **Accuracy**: Match requirement specifications exactly (versions, parameters, configurations)\n# - **Completeness**: Implement every component discussed, not just the main functionality\n# - **Functionality**: Code must actually work and run all specified features successfully\n\n# **AVOID DISTRACTIONS**: Focus implementation time on requirement fulfillment rather than advanced tooling, extensive documentation, or optimization utilities that aren't needed for the core functionality.\n\n# **REMEMBER**: Remember, you are tasked with implementing a complete system, not just a single part of it or a minimal example. The file read tool is PAGINATED, so you will need to CALL IT MULTIPLE TIMES to make sure that you have read all the relevant parts of the requirements.\n# \"\"\"\nGENERAL_CODE_IMPLEMENTATION_SYSTEM_PROMPT = \"\"\"You are an expert code implementation agent for technical requirements implementation. Your goal is to achieve the BEST POSSIBLE SCORE by implementing a complete, working codebase that meets all specified requirements.\n\n**PRIMARY OBJECTIVE**: Implement ALL algorithms, features, and components mentioned in the requirements. Success is measured by completeness and accuracy, not code elegance. Use available time to continuously refine and optimize your solution.\n\n**CORE STRATEGY**:\n- Read the requirements thoroughly to identify every algorithm, feature, and component\n- Implement core algorithms first, then environments, then integration\n- Use exact versions and specifications mentioned in the requirements\n- Test each component immediately after implementation\n- Focus on working implementations over perfect architecture\n\n**IMPLEMENTATION APPROACH**:\nBuild incrementally using multiple tool calls. For each step:\n1. **Identify** what needs to be implemented from the requirements\n2. **Implement** one component at a time\n3. **Verify** optionally using `execute_python` or `execute_bash` to check implementation completeness if needed\n4. **Integrate** with existing components\n5. **Validate** against requirement specifications\n\n**TOOL CALLING STRATEGY**:\n1. ⚠️ **SINGLE FUNCTION CALL PER MESSAGE**: Each message may perform only one function call. You will see the result of the function right after sending the message. If you need to perform multiple actions, you can always send more messages with subsequent function calls. Do some reasoning before your actions, describing what function calls you are going to use and how they fit into your plan.\n\n2. **TOOL EXECUTION STRATEGY**:\n - **Development Cycle (for each new file implementation)**: `write_file` (implement)\n\n**Execution Guidelines**:\n- **Plan First**: Before each action, explain your reasoning and which function you'll use\n- **One Step at a Time**: Execute → Observe Result → Plan Next Step → Execute Next\n- **Iterative Progress**: Build your solution incrementally through multiple conversations\n- **Strategic Sequencing**: Choose the most logical next step based on previous results\n\n**COMPLETENESS CHECKLIST**:\nBefore considering the task complete, ensure you have:\n- ✅ All algorithms mentioned in the requirements (including any abbreviations or alternative names)\n- ✅ All environments/dependencies with exact versions specified\n- ✅ All comparison methods or baseline implementations referenced\n- ✅ Working integration that can run all specified functionality\n- ✅ Complete codebase that implements all features, functionality, and outputs specified in the requirements\n- ✅ Basic documentation explaining how to use the implemented system\n\n**CRITICAL SUCCESS FACTORS**:\n- **Accuracy**: Match requirement specifications exactly (versions, parameters, configurations)\n- **Completeness**: Implement every component discussed, not just the main functionality\n- **Functionality**: Code must actually work and run all specified features successfully\n\n**AVOID DISTRACTIONS**: Focus implementation time on requirement fulfillment rather than advanced tooling, extensive documentation, or optimization utilities that aren't needed for the core functionality.\n\n**REMEMBER**: Remember, you are tasked with implementing a complete system, not just a single part of it or a minimal example. The file read tool is PAGINATED, so you will need to CALL IT MULTIPLE TIMES to make sure that you have read all the relevant parts of the requirements.\n\"\"\"\n\n# Chat Agent Planning Prompt (Universal for Academic and Engineering Use)\nCHAT_AGENT_PLANNING_PROMPT = \"\"\"You are a universal project planning agent that creates implementation plans for any coding project: web apps, games, academic research, tools, etc.\n\n# 🎯 OBJECTIVE\nTransform user requirements into a clear, actionable implementation plan with optimal file structure and dependencies.\n\n# 📋 OUTPUT FORMAT\n\n```yaml\nproject_plan:\n title: \"[Project Name]\"\n description: \"[Brief description]\"\n project_type: \"[web_app|game|academic|tool|api|other]\"\n\n # CUSTOM FILE TREE STRUCTURE (max 15 files, design as needed)\n file_structure: |\n project_root/\n ├── main.py # Entry point\n ├── [specific_files] # Core files based on project type\n ├── [folder]/ # Organized folders if needed\n │ ├── __init__.py\n │ └── [module].py\n ├── requirements.txt # Dependencies\n └── README.md # Basic documentation\n\n # IMPORTANT: Output ACTUAL file tree structure above, not placeholder text\n # Examples by project type:\n # Web App: app.py, templates/, static/, models.py, config.py\n # Game: main.py, game/, assets/, sprites/, config.yaml\n # Academic: algorithm.py, experiments/, data/, utils.py, config.json\n # Tool: cli.py, core/, utils.py, tests/, setup.py\n\n # CORE IMPLEMENTATION PLAN\n implementation_steps:\n 1. \"[First step - usually setup/core structure]\"\n 2. \"[Second step - main functionality]\"\n 3. \"[Third step - integration/interface]\"\n 4. \"[Fourth step - testing/refinement]\"\n\n # DEPENDENCIES & SETUP\n dependencies:\n required_packages:\n - \"[package1==version]\"\n - \"[package2>=version]\"\n optional_packages:\n - \"[optional1]: [purpose]\"\n setup_commands:\n - \"[command to setup environment]\"\n - \"[command to install dependencies]\"\n\n # KEY TECHNICAL DETAILS\n tech_stack:\n language: \"[primary language]\"\n frameworks: [\"[framework1]\", \"[framework2]\"]\n key_libraries: [\"[lib1]\", \"[lib2]\"]\n\n main_features:\n - \"[core feature 1]\"\n - \"[core feature 2]\"\n - \"[core feature 3]\"\n```\n\n# 🎯 PLANNING PRINCIPLES\n- **Flexibility**: Adapt file structure to project type (no fixed templates)\n- **Simplicity**: Keep under 15 files, focus on essentials\n- **Practicality**: Include specific packages/versions needed\n- **Clarity**: Clear implementation steps that can be directly coded\n- **Universality**: Work for any project type (web, game, academic, etc.)\n\n# 📝 FILE STRUCTURE GUIDELINES\n- **MUST OUTPUT**: Actual file tree with specific filenames (not placeholder text)\n- Design structure based on project needs, not templates\n- Group related functionality logically\n- Include main entry point (main.py, app.py, etc.)\n- Add config/settings files if needed\n- Include requirements.txt or equivalent\n- Keep it minimal but complete (max 15 files)\n- Use tree format: ├── ─ │ symbols for visual hierarchy\"\"\"\n\n# =============================================================================\n# TRADITIONAL PROMPTS (Non-segmented versions for smaller documents)\n# =============================================================================\n\n# Traditional Algorithm Analysis Prompt (No Segmentation)\nPAPER_ALGORITHM_ANALYSIS_PROMPT_TRADITIONAL = \"\"\"You are extracting COMPLETE implementation details from a research paper. Your goal is to capture EVERY algorithm, formula, and technical detail needed for perfect reproduction.\n\n# DOCUMENT READING STRATEGY\n\n## TRADITIONAL APPROACH: Full Document Reading\nRead the complete document to ensure comprehensive coverage of all algorithmic details:\n\n# DETAILED EXTRACTION PROTOCOL\n\n## 1. COMPREHENSIVE ALGORITHM SCAN\nRead through the entire document systematically:\n- Method/Algorithm sections\n- Implementation Details\n- Hyperparameters and training details\n- Mathematical formulations\n\n## 2. ALGORITHM DEEP EXTRACTION\nFor EVERY algorithm/method/procedure mentioned:\n\n### Algorithm Structure\n```yaml\nalgorithm_name: \"[Exact name from paper]\"\nsection: \"[e.g., Section 3.2]\"\nalgorithm_box: \"[e.g., Algorithm 1 on page 4]\"\n\npseudocode: |\n [COPY THE EXACT PSEUDOCODE FROM PAPER]\n Input: ...\n Output: ...\n 1. Initialize ...\n 2. For each ...\n 2.1 Calculate ...\n [Keep exact formatting and numbering]\n\nmathematical_formulation:\n - equation: \"[Copy formula EXACTLY, e.g., L = L_task + λ*L_explain]\"\n equation_number: \"[e.g., Eq. 3]\"\n where:\n L_task: \"task loss\"\n L_explain: \"explanation loss\"\n λ: \"weighting parameter (default: 0.5)\"\n\nstep_by_step_breakdown:\n 1. \"[Detailed explanation of what step 1 does]\"\n 2. \"[What step 2 computes and why]\"\n\nimplementation_details:\n - \"Uses softmax temperature τ = 0.1\"\n - \"Gradient clipping at norm 1.0\"\n - \"Initialize weights with Xavier uniform\"\n```\n\n## 3. COMPONENT EXTRACTION\nFor EVERY component/module mentioned:\n\n### Component Details\n```yaml\ncomponent_name: \"[e.g., Mask Network, Critic Network]\"\npurpose: \"[What this component does in the system]\"\narchitecture:\n input: \"[shape and meaning]\"\n layers:\n - \"[Conv2d(3, 64, kernel=3, stride=1)]\"\n - \"[ReLU activation]\"\n - \"[BatchNorm2d(64)]\"\n output: \"[shape and meaning]\"\n\nspecial_features:\n - \"[Any unique aspects]\"\n - \"[Special initialization]\"\n```\n\n## 4. TRAINING PROCEDURE\nExtract the COMPLETE training process:\n\n```yaml\ntraining_loop:\n outer_iterations: \"[number or condition]\"\n inner_iterations: \"[number or condition]\"\n\n steps:\n 1. \"Sample batch of size B from buffer\"\n 2. \"Compute importance weights using...\"\n 3. \"Update policy with loss...\"\n\n loss_functions:\n - name: \"policy_loss\"\n formula: \"[exact formula]\"\n components: \"[what each term means]\"\n\n optimization:\n optimizer: \"Adam\"\n learning_rate: \"3e-4\"\n lr_schedule: \"linear decay to 0\"\n gradient_norm: \"clip at 0.5\"\n```\n\n## 5. HYPERPARAMETERS HUNT\nSearch EVERYWHERE (text, tables, captions) for:\n\n```yaml\nhyperparameters:\n # Training\n batch_size: 64\n buffer_size: 1e6\n discount_gamma: 0.99\n\n # Architecture\n hidden_units: [256, 256]\n activation: \"ReLU\"\n\n # Algorithm-specific\n explanation_weight: 0.5\n exploration_bonus_scale: 0.1\n reset_probability: 0.3\n\n # Found in:\n location_references:\n - \"batch_size: Table 1\"\n - \"hidden_units: Section 4.1\"\n```\n\n# OUTPUT FORMAT\n```yaml\ncomplete_algorithm_extraction:\n paper_structure:\n method_sections: \"[3, 3.1, 3.2, 3.3, 4]\"\n algorithm_count: \"[total number found]\"\n\n main_algorithm:\n [COMPLETE DETAILS AS ABOVE]\n\n supporting_algorithms:\n - [EACH SUPPORTING ALGORITHM WITH FULL DETAILS]\n\n components:\n - [EVERY COMPONENT WITH ARCHITECTURE]\n\n training_details:\n [COMPLETE TRAINING PROCEDURE]\n\n all_hyperparameters:\n [EVERY PARAMETER WITH VALUE AND SOURCE]\n\n implementation_notes:\n - \"[Any implementation hint from paper]\"\n - \"[Tricks mentioned in text]\"\n\n missing_but_critical:\n - \"[What's not specified but essential]\"\n - \"[With suggested defaults]\"\n```\n\nBE EXHAUSTIVE. A developer should be able to implement the ENTIRE paper using only your extraction.\"\"\"\n\n# Traditional Concept Analysis Prompt (No Segmentation)\nPAPER_CONCEPT_ANALYSIS_PROMPT_TRADITIONAL = \"\"\"You are doing a COMPREHENSIVE analysis of a research paper to understand its complete structure, contributions, and implementation requirements.\n\n# OBJECTIVE\nMap out the ENTIRE paper structure and identify ALL components that need implementation for successful reproduction.\n\n# DOCUMENT READING STRATEGY\n\n## TRADITIONAL APPROACH: Complete Document Analysis\nRead the entire document systematically to ensure comprehensive understanding:\n\n# COMPREHENSIVE ANALYSIS PROTOCOL\n\n## 1. COMPLETE PAPER STRUCTURAL ANALYSIS\nCreate a full map of the document:\n\n```yaml\npaper_structure_map:\n title: \"[Full paper title]\"\n\n sections:\n 1_introduction:\n main_claims: \"[What the paper claims to achieve]\"\n problem_definition: \"[Exact problem being solved]\"\n\n 2_related_work:\n key_comparisons: \"[Methods this work builds upon or competes with]\"\n\n 3_method: # May have multiple subsections\n subsections:\n 3.1: \"[Title and main content]\"\n 3.2: \"[Title and main content]\"\n algorithms_presented: \"[List all algorithms by name]\"\n\n 4_experiments:\n environments: \"[All test environments/datasets]\"\n baselines: \"[All comparison methods]\"\n metrics: \"[All evaluation metrics used]\"\n\n 5_results:\n main_findings: \"[Key results that prove the method works]\"\n tables_figures: \"[Important result tables/figures to reproduce]\"\n```\n\n## 2. METHOD DECOMPOSITION\nFor the main method/approach:\n\n```yaml\nmethod_decomposition:\n method_name: \"[Full name and acronym]\"\n\n core_components: # Break down into implementable pieces\n component_1:\n name: \"[e.g., State Importance Estimator]\"\n purpose: \"[Why this component exists]\"\n paper_section: \"[Where it's described]\"\n\n component_2:\n name: \"[e.g., Policy Refinement Module]\"\n purpose: \"[Its role in the system]\"\n paper_section: \"[Where it's described]\"\n\n component_interactions:\n - \"[How component 1 feeds into component 2]\"\n - \"[Data flow between components]\"\n\n theoretical_foundation:\n key_insight: \"[The main theoretical insight]\"\n why_it_works: \"[Intuitive explanation]\"\n```\n\n## 3. IMPLEMENTATION REQUIREMENTS MAPPING\nMap paper content to code requirements:\n\n```yaml\nimplementation_map:\n algorithms_to_implement:\n - algorithm: \"[Name from paper]\"\n section: \"[Where defined]\"\n complexity: \"[Simple/Medium/Complex]\"\n dependencies: \"[What it needs to work]\"\n\n models_to_build:\n - model: \"[Neural network or other model]\"\n architecture_location: \"[Section describing it]\"\n purpose: \"[What this model does]\"\n\n data_processing:\n - pipeline: \"[Data preprocessing needed]\"\n requirements: \"[What the data should look like]\"\n\n evaluation_suite:\n - metric: \"[Metric name]\"\n formula_location: \"[Where it's defined]\"\n purpose: \"[What it measures]\"\n```\n\n## 4. EXPERIMENT REPRODUCTION PLAN\nIdentify ALL experiments needed:\n\n```yaml\nexperiments_analysis:\n main_results:\n - experiment: \"[Name/description]\"\n proves: \"[What claim this validates]\"\n requires: \"[Components needed to run this]\"\n expected_outcome: \"[Specific numbers/trends]\"\n\n ablation_studies:\n - study: \"[What is being ablated]\"\n purpose: \"[What this demonstrates]\"\n\n baseline_comparisons:\n - baseline: \"[Method name]\"\n implementation_required: \"[Yes/No/Partial]\"\n source: \"[Where to find implementation]\"\n```\n\n## 5. CRITICAL SUCCESS FACTORS\nWhat defines successful reproduction:\n\n```yaml\nsuccess_criteria:\n must_achieve:\n - \"[Primary result that must be reproduced]\"\n - \"[Core behavior that must be demonstrated]\"\n\n should_achieve:\n - \"[Secondary results that validate the method]\"\n\n validation_evidence:\n - \"[Specific figure/table to reproduce]\"\n - \"[Qualitative behavior to demonstrate]\"\n```\n\n# OUTPUT FORMAT\n```yaml\ncomprehensive_paper_analysis:\n executive_summary:\n paper_title: \"[Full title]\"\n core_contribution: \"[One sentence summary]\"\n implementation_complexity: \"[Low/Medium/High]\"\n estimated_components: \"[Number of major components to build]\"\n\n complete_structure_map:\n [FULL SECTION BREAKDOWN AS ABOVE]\n\n method_architecture:\n [DETAILED COMPONENT BREAKDOWN]\n\n implementation_requirements:\n [ALL ALGORITHMS, MODELS, DATA, METRICS]\n\n reproduction_roadmap:\n phase_1: \"[What to implement first]\"\n phase_2: \"[What to build next]\"\n phase_3: \"[Final components and validation]\"\n\n validation_checklist:\n - \"[ ] [Specific result to achieve]\"\n - \"[ ] [Behavior to demonstrate]\"\n - \"[ ] [Metric to match]\"\n```\n\nBE THOROUGH. Miss nothing. The output should be a complete blueprint for reproduction.\"\"\"\n\n# Traditional Code Planning Prompt (No Segmentation)\nCODE_PLANNING_PROMPT_TRADITIONAL = \"\"\"You are creating a DETAILED, COMPLETE reproduction plan by integrating comprehensive analysis results.\n\n# INPUT\nYou receive two exhaustive analyses:\n1. **Comprehensive Paper Analysis**: Complete paper structure, components, and requirements\n2. **Complete Algorithm Extraction**: All algorithms, formulas, pseudocode, and technical details\n\n# OBJECTIVE\nCreate an implementation plan so detailed that a developer can reproduce the ENTIRE paper without reading it.\n\n# CRITICAL: COMPLETE OUTPUT REQUIREMENT\n⚠️ MANDATORY: You MUST generate ALL 5 sections completely. DO NOT stop early or truncate any section.\n\n## Output Completeness Strategy:\n🎯 **Your #1 Priority**: Ensure ALL 5 sections are present and complete before finishing your response.\n\n## Content Balance Guidelines (STRICTLY FOLLOW):\n- **Section 1 (File Structure)**: ~800-1000 chars - Brief file listing with priority order\n- **Section 2 (Implementation Components)**: ~3000-4000 chars - CORE section with all algorithms/components\n- **Section 3 (Validation)**: ~2000-2500 chars - Experiments and expected results\n- **Section 4 (Environment)**: ~800-1000 chars - Dependencies and requirements\n- **Section 5 (Implementation Strategy)**: ~1500-2000 chars - Step-by-step approach\n\n📏 **Total Target**: 8000-10000 characters for complete plan\n\n⚠️ **Self-Check Before Finishing**:\n- Did you include file_structure section? ✓\n- Did you include implementation_components section? ✓\n- Did you include validation_approach section? ✓\n- Did you include environment_setup section? ✓\n- Did you include implementation_strategy section? ✓\n- If ANY answer is NO, continue writing until ALL sections are complete!\n\n## File Priority Guidelines:\n🔧 **Implementation Priority Order**:\n1. **FIRST**: Core algorithm/model files (highest priority)\n2. **SECOND**: Supporting modules and utilities\n3. **THIRD**: Experiment and evaluation scripts\n4. **FOURTH**: Configuration and data handling\n5. **LAST**: Documentation files (README.md, requirements.txt) - These should be created AFTER core implementation\n\nNote: README and requirements.txt are maintenance files that depend on the final implementation, so plan them last but INCLUDE them in the file structure.\n\n# DETAILED SYNTHESIS PROCESS\n\n## 1. MERGE ALL INFORMATION\nCombine EVERYTHING from both analyses:\n- Every algorithm with its pseudocode\n- Every component with its architecture\n- Every hyperparameter with its value\n- Every experiment with expected results\n\n## 2. MAP CONTENT TO IMPLEMENTATION\n\nFor each component you identify, specify how it will be implemented:\n\n```\n# DESIGN YOUR MAPPING: Connect paper content to code organization\n[For each algorithm/component/method in the paper]:\n - What it does and where it's described in the paper\n - How you'll organize the code (files, classes, functions - your choice)\n - What specific formulas, algorithms, or procedures need implementation\n - Dependencies and relationships with other components\n - Implementation approach that makes sense for this specific paper\n```\n\n## 3. EXTRACT ALL TECHNICAL DETAILS\n\nIdentify every technical detail that needs implementation:\n\n```\n# COMPREHENSIVE TECHNICAL EXTRACTION:\n[Gather all implementation-relevant details from the paper]:\n - All algorithms with complete pseudocode and mathematical formulations\n - All parameters, hyperparameters, and configuration values\n - All architectural details (if applicable to your paper type)\n - All experimental procedures and evaluation methods\n - Any implementation hints, tricks, or special considerations mentioned\n```\n\n# COMPREHENSIVE OUTPUT FORMAT\n\n```yaml\ncomplete_reproduction_plan:\n paper_info:\n title: \"[Full paper title]\"\n core_contribution: \"[Main innovation being reproduced]\"\n\n # SECTION 1: File Structure Design\n\n # DESIGN YOUR OWN STRUCTURE: Create a file organization that best serves this specific paper\n # - Analyze what the paper contains (algorithms, models, experiments, systems, etc.)\n # - Organize files and directories in the most logical way for implementation\n # - Create meaningful names and groupings based on paper content\n # - Keep it clean, intuitive, and focused on what actually needs to be implemented\n # - INCLUDE documentation files (README.md, requirements.txt) but mark them for LAST implementation\n\n file_structure: |\n [Design and specify your own project structure here - KEEP THIS BRIEF]\n [Include ALL necessary files including README.md and requirements.txt]\n [Organize based on what this paper actually contains and needs]\n [Create directories and files that make sense for this specific implementation]\n [IMPORTANT: Include executable files (e.g., main.py, run.py, train.py, demo.py) - choose names based on repo content]\n [Design executable entry points that match the paper's main functionality and experiments]\n [FILE COUNT LIMIT: Keep total file count around 20 files - not too many, focus on essential components only]\n [NOTE: README.md and requirements.txt should be implemented LAST after all code files]\n\n # SECTION 2: Implementation Components\n\n # IDENTIFY AND SPECIFY: What needs to be implemented based on this paper\n # - List all algorithms, models, systems, or components mentioned\n # - Map each to implementation details and file locations\n # - Include formulas, pseudocode, and technical specifications\n # - Organize in whatever way makes sense for this paper\n\n implementation_components: |\n [List and specify all components that need implementation]\n [For each component: purpose, location, algorithms, formulas, technical details]\n [Organize and structure this based on the paper's actual content]\n\n # SECTION 3: Validation & Evaluation\n\n # DESIGN VALIDATION: How to verify the implementation works correctly\n # - Define what experiments, tests, or proofs are needed\n # - Specify expected results from the paper (figures, tables, theorems)\n # - Design validation approach appropriate for this paper's domain\n # - Include setup requirements and success criteria\n\n validation_approach: |\n [Design validation strategy appropriate for this paper]\n [Specify experiments, tests, or mathematical verification needed]\n [Define expected results and success criteria]\n [Include any special setup or evaluation requirements]\n\n # SECTION 4: Environment & Dependencies\n\n # SPECIFY REQUIREMENTS: What's needed to run this implementation\n # - Programming language and version requirements\n # - External libraries and exact versions (if specified in paper)\n # - Hardware requirements (GPU, memory, etc.)\n # - Any special setup or installation steps\n\n environment_setup: |\n [List all dependencies and environment requirements for this specific paper]\n [Include versions where specified, reasonable defaults where not]\n [Note any special hardware or software requirements]\n\n # SECTION 5: Implementation Strategy\n\n # PLAN YOUR APPROACH: How to implement this paper step by step\n # - Break down implementation into logical phases\n # - Identify dependencies between components\n # - Plan verification and testing at each stage\n # - Handle missing details with reasonable defaults\n\n implementation_strategy: |\n [Design your implementation approach for this specific paper]\n [Break into phases that make sense for this paper's components]\n [Plan testing and verification throughout the process]\n [Address any missing details or ambiguities in the paper]\n```\n\nBE EXHAUSTIVE. Every algorithm, every formula, every parameter, every file should be specified in complete detail.\"\"\"\n"
},
{
"path": "requirements.txt",
"content": "# Core Dependencies\naiofiles>=0.8.0\naiohttp>=3.8.0\nanthropic\nasyncio-mqtt\ndocling\n\n# New UI Backend Dependencies\nfastapi>=0.104.0\ngoogle-genai\nmcp-agent\nmcp-server-git\nopenapi\nnest_asyncio\nopenai\npathlib2\npydantic-settings>=2.0.0\nPyPDF2>=2.0.0\npython-multipart>=0.0.6\nPyYAML>=6.0\nreportlab>=3.5.0\nstreamlit\nuvicorn>=0.24.0\nwebsockets>=12.0\n"
},
{
"path": "run.bat",
"content": "@echo off\nREM DeepCode New UI - Windows Launcher\nREM 深度代码新UI - Windows启动脚本\n\necho.\necho ========================================\necho DeepCode New UI - Windows Launcher\necho ========================================\necho.\n\nREM Check Python\npython --version >nul 2>&1\nif errorlevel 1 (\n echo [ERROR] Python not found. Please install Python 3.9+\n pause\n exit /b 1\n)\n\nREM Check Node.js\nnode --version >nul 2>&1\nif errorlevel 1 (\n echo [ERROR] Node.js not found. Please install Node.js 18+\n echo Download from: https://nodejs.org/\n pause\n exit /b 1\n)\n\necho [OK] Python found\necho [OK] Node.js found\necho.\n\nREM Run the Python launcher\npython \"%~dp0deepcode.py\"\n\npause\n"
},
{
"path": "run.sh",
"content": "#!/bin/bash\n# DeepCode New UI 一键启动脚本\n\nset -e\n\nSCRIPT_DIR=\"$(cd \"$(dirname \"${BASH_SOURCE[0]}\")\" && pwd)\"\nNEW_UI_DIR=\"$SCRIPT_DIR/new_ui\"\n\n# 颜色定义\nRED='\\033[0;31m'\nGREEN='\\033[0;32m'\nBLUE='\\033[0;34m'\nYELLOW='\\033[1;33m'\nNC='\\033[0m'\n\necho \"🚀 启动 DeepCode New UI...\"\necho \"\"\n\n# ============ 自动设置 Python 环境 ============\nsetup_python_env() {\n # 优先级: 已激活的 conda > 已激活的 venv > 本地 .venv > 本地 venv > 自动激活 conda deepcode\n\n if [ -n \"$CONDA_PREFIX\" ]; then\n echo -e \"${GREEN}✓ 使用 conda 环境: $(basename $CONDA_PREFIX)${NC}\"\n export PATH=\"$CONDA_PREFIX/bin:$PATH\"\n return 0\n fi\n\n if [ -n \"$VIRTUAL_ENV\" ]; then\n echo -e \"${GREEN}✓ 使用 virtualenv: $(basename $VIRTUAL_ENV)${NC}\"\n export PATH=\"$VIRTUAL_ENV/bin:$PATH\"\n return 0\n fi\n\n # 尝试自动激活本地虚拟环境\n if [ -d \"$SCRIPT_DIR/.venv\" ]; then\n echo -e \"${YELLOW}⚡ 自动激活 .venv 环境${NC}\"\n source \"$SCRIPT_DIR/.venv/bin/activate\"\n return 0\n fi\n\n if [ -d \"$SCRIPT_DIR/venv\" ]; then\n echo -e \"${YELLOW}⚡ 自动激活 venv 环境${NC}\"\n source \"$SCRIPT_DIR/venv/bin/activate\"\n return 0\n fi\n\n # 尝试自动激活 conda deepcode 环境\n if command -v conda &> /dev/null; then\n if conda env list 2>/dev/null | grep -q \"deepcode\"; then\n echo -e \"${YELLOW}⚡ 自动激活 conda deepcode 环境${NC}\"\n eval \"$(conda shell.bash hook)\"\n conda activate deepcode\n export PATH=\"$CONDA_PREFIX/bin:$PATH\"\n return 0\n fi\n fi\n\n echo -e \"${YELLOW}⚠ 未检测到虚拟环境，使用系统 Python${NC}\"\n return 1\n}\n\nsetup_python_env\necho -e \"📍 Python: $(which python)\"\necho \"\"\n# ============================================\n\n# 清理函数 - 使用进程组确保所有子进程都被终止\ncleanup() {\n echo \"\"\n echo \"🛑 正在关闭服务...\"\n # 杀死后端进程及其子进程\n if [ -n \"$BACKEND_PID\" ]; then\n kill -- -$BACKEND_PID 2>/dev/null || kill $BACKEND_PID 2>/dev/null || true\n fi\n # 杀死前端进程及其子进程\n if [ -n \"$FRONTEND_PID\" ]; then\n kill -- -$FRONTEND_PID 2>/dev/null || kill $FRONTEND_PID 2>/dev/null || true\n fi\n # 额外清理: 确保端口被释放\n pkill -f \"uvicorn main:app.*--port 8000\" 2>/dev/null || true\n pkill -f \"vite.*5173\" 2>/dev/null || true\n echo \"✓ 所有服务已停止\"\n exit 0\n}\ntrap cleanup SIGINT SIGTERM EXIT\n\n# 检查目录\nif [ ! -d \"$NEW_UI_DIR\" ]; then\n echo \"❌ 错误: new_ui 目录不存在\"\n exit 1\nfi\n\n# 清理被占用的端口\ncleanup_ports() {\n local port=$1\n local pid=$(lsof -ti :$port 2>/dev/null)\n if [ -n \"$pid\" ]; then\n echo -e \"${YELLOW}⚠ 端口 $port 被占用 (PID: $pid)，正在清理...${NC}\"\n kill -9 $pid 2>/dev/null || true\n sleep 1\n echo -e \"${GREEN}✓ 端口 $port 已释放${NC}\"\n fi\n}\n\ncleanup_ports 8000\ncleanup_ports 5173\n\n# 启动后端\necho -e \"${BLUE}[1/2] 启动后端服务...${NC}\"\ncd \"$NEW_UI_DIR/backend\"\n\n# 安装依赖（如果需要）\nif ! python -c \"import fastapi\" 2>/dev/null; then\n echo -e \"${YELLOW}安装后端依赖...${NC}\"\n pip install fastapi uvicorn pydantic-settings python-multipart aiofiles websockets -q\nfi\n\n# 使用 setsid 创建新进程组（如果可用），否则直接后台运行\nif command -v setsid &> /dev/null; then\n setsid python -m uvicorn main:app --host 0.0.0.0 --port 8000 --reload &\nelse\n python -m uvicorn main:app --host 0.0.0.0 --port 8000 --reload &\nfi\nBACKEND_PID=$!\nsleep 2\n\n# 检查后端是否真正启动成功\nif ! kill -0 $BACKEND_PID 2>/dev/null; then\n echo -e \"${RED}✗ 后端启动失败，可能端口被占用${NC}\"\n echo -e \"${YELLOW} 尝试: lsof -i :8000 查看占用端口的进程${NC}\"\nelse\n echo -e \"${GREEN}✓ 后端已启动: http://localhost:8000${NC}\"\nfi\n\n# 启动前端\necho -e \"${BLUE}[2/2] 启动前端服务...${NC}\"\ncd \"$NEW_UI_DIR/frontend\"\n\nif [ ! -d \"node_modules\" ]; then\n echo -e \"${YELLOW}安装前端依赖 (首次运行)...${NC}\"\n npm install\nfi\n\n# 使用 setsid 创建新进程组（如果可用）\nif command -v setsid &> /dev/null; then\n setsid npm run dev &\nelse\n npm run dev &\nfi\nFRONTEND_PID=$!\nsleep 3\n\necho \"\"\necho \"╔════════════════════════════════════════╗\"\necho -e \"║ ${GREEN}DeepCode New UI 已启动!${NC} ║\"\necho \"╠════════════════════════════════════════╣\"\necho \"║ ║\"\necho \"║ 🌐 前端: http://localhost:5173 ║\"\necho \"║ 🔧 后端: http://localhost:8000 ║\"\necho \"║ 📚 API: http://localhost:8000/docs ║\"\necho \"║ ║\"\necho \"║ 按 Ctrl+C 停止所有服务 ║\"\necho \"╚════════════════════════════════════════╝\"\necho \"\"\n\nwait\n"
},
{
"path": "schema/mcp-agent.config.schema.json",
"content": "{\n \"$defs\": {\n \"LogPathSettings\": {\n \"description\": \"Settings for configuring log file paths with dynamic elements like timestamps or session IDs.\",\n \"properties\": {\n \"path_pattern\": {\n \"default\": \"logs/mcp-agent-{unique_id}.jsonl\",\n \"title\": \"Path Pattern\",\n \"type\": \"string\",\n \"description\": \"Path pattern for log files with a {unique_id} placeholder\"\n },\n \"unique_id\": {\n \"default\": \"timestamp\",\n \"enum\": [\n \"timestamp\",\n \"session_id\"\n ],\n \"title\": \"Unique Id\",\n \"type\": \"string\",\n \"description\": \"Type of unique identifier to use in the log filename\"\n },\n \"timestamp_format\": {\n \"default\": \"%Y%m%d_%H%M%S\",\n \"title\": \"Timestamp Format\",\n \"type\": \"string\",\n \"description\": \"Format string for timestamps when unique_id is set to timestamp\"\n }\n },\n \"title\": \"LogPathSettings\",\n \"type\": \"object\"\n },\n \"AnthropicSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Settings for using Anthropic models in the MCP Agent application.\",\n \"properties\": {\n \"api_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Api Key\"\n }\n },\n \"title\": \"AnthropicSettings\",\n \"type\": \"object\"\n },\n \"CohereSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Settings for using Cohere models in the MCP Agent application.\",\n \"properties\": {\n \"api_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Api Key\"\n }\n },\n \"title\": \"CohereSettings\",\n \"type\": \"object\"\n },\n \"LoggerSettings\": {\n \"description\": \"Logger settings for the MCP Agent application.\",\n \"properties\": {\n \"type\": {\n \"default\": \"console\",\n \"enum\": [\n \"none\",\n \"console\",\n \"file\",\n \"http\"\n ],\n \"title\": \"Type\",\n \"type\": \"string\"\n },\n \"transports\": {\n \"default\": [\n \"console\"\n ],\n \"items\": {\n \"enum\": [\n \"none\",\n \"console\",\n \"file\",\n \"http\"\n ],\n \"type\": \"string\"\n },\n \"title\": \"Transports\",\n \"type\": \"array\",\n \"description\": \"List of transports to use (can enable multiple simultaneously)\"\n },\n \"level\": {\n \"default\": \"info\",\n \"enum\": [\n \"debug\",\n \"info\",\n \"warning\",\n \"error\"\n ],\n \"title\": \"Level\",\n \"type\": \"string\",\n \"description\": \"Minimum logging level\"\n },\n \"progress_display\": {\n \"default\": true,\n \"title\": \"Progress Display\",\n \"type\": \"boolean\",\n \"description\": \"Enable or disable the progress display\"\n },\n \"path\": {\n \"default\": \"mcp-agent.jsonl\",\n \"title\": \"Path\",\n \"type\": \"string\",\n \"description\": \"Path to log file, if logger 'type' is 'file'.\"\n },\n \"path_settings\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/LogPathSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Path Settings\",\n \"description\": \"Advanced settings for log file paths with dynamic elements like timestamps or session IDs\"\n },\n \"batch_size\": {\n \"default\": 100,\n \"title\": \"Batch Size\",\n \"type\": \"integer\",\n \"description\": \"Number of events to accumulate before processing\"\n },\n \"flush_interval\": {\n \"default\": 2.0,\n \"title\": \"Flush Interval\",\n \"type\": \"number\",\n \"description\": \"How often to flush events in seconds\"\n },\n \"max_queue_size\": {\n \"default\": 2048,\n \"title\": \"Max Queue Size\",\n \"type\": \"integer\",\n \"description\": \"Maximum queue size for event processing\"\n },\n \"http_endpoint\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Http Endpoint\",\n \"description\": \"HTTP endpoint for event transport\"\n },\n \"http_headers\": {\n \"anyOf\": [\n {\n \"additionalProperties\": {\n \"type\": \"string\"\n },\n \"type\": \"object\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Http Headers\",\n \"description\": \"HTTP headers for event transport\"\n },\n \"http_timeout\": {\n \"default\": 5.0,\n \"title\": \"Http Timeout\",\n \"type\": \"number\",\n \"description\": \"HTTP timeout seconds for event transport\"\n }\n },\n \"title\": \"LoggerSettings\",\n \"type\": \"object\"\n },\n \"MCPRootSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Represents a root directory configuration for an MCP server.\",\n \"properties\": {\n \"uri\": {\n \"title\": \"Uri\",\n \"type\": \"string\",\n \"description\": \"The URI identifying the root. Must start with file://\"\n },\n \"name\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Name\",\n \"description\": \"Optional name for the root.\"\n },\n \"server_uri_alias\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Server Uri Alias\",\n \"description\": \"Optional URI alias for presentation to the server\"\n }\n },\n \"required\": [\n \"uri\"\n ],\n \"title\": \"MCPRootSettings\",\n \"type\": \"object\"\n },\n \"MCPServerAuthSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Represents authentication configuration for a server.\",\n \"properties\": {\n \"api_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Api Key\"\n }\n },\n \"title\": \"MCPServerAuthSettings\",\n \"type\": \"object\"\n },\n \"MCPServerSettings\": {\n \"description\": \"Represents the configuration for an individual server.\",\n \"properties\": {\n \"name\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Name\",\n \"description\": \"The name of the server.\"\n },\n \"description\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Description\",\n \"description\": \"The description of the server.\"\n },\n \"transport\": {\n \"default\": \"stdio\",\n \"enum\": [\n \"stdio\",\n \"sse\"\n ],\n \"title\": \"Transport\",\n \"type\": \"string\",\n \"description\": \"The transport mechanism.\"\n },\n \"command\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Command\",\n \"description\": \"The command to execute the server (e.g. npx).\"\n },\n \"args\": {\n \"anyOf\": [\n {\n \"items\": {\n \"type\": \"string\"\n },\n \"type\": \"array\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Args\",\n \"description\": \"The arguments for the server command.\"\n },\n \"read_timeout_seconds\": {\n \"anyOf\": [\n {\n \"type\": \"integer\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Read Timeout Seconds\",\n \"description\": \"The timeout in seconds for the server connection.\"\n },\n \"url\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Url\",\n \"description\": \"The URL for the server (e.g. for SSE transport).\"\n },\n \"auth\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/MCPServerAuthSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"The authentication configuration for the server.\"\n },\n \"roots\": {\n \"anyOf\": [\n {\n \"items\": {\n \"$ref\": \"#/$defs/MCPRootSettings\"\n },\n \"type\": \"array\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Roots\",\n \"description\": \"Root directories this server has access to.\"\n },\n \"env\": {\n \"anyOf\": [\n {\n \"additionalProperties\": {\n \"type\": \"string\"\n },\n \"type\": \"object\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Env\",\n \"description\": \"Environment variables to pass to the server process.\"\n }\n },\n \"title\": \"MCPServerSettings\",\n \"type\": \"object\"\n },\n \"MCPSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Configuration for all MCP servers.\",\n \"properties\": {\n \"servers\": {\n \"additionalProperties\": {\n \"$ref\": \"#/$defs/MCPServerSettings\"\n },\n \"default\": {},\n \"title\": \"Servers\",\n \"type\": \"object\"\n }\n },\n \"title\": \"MCPSettings\",\n \"type\": \"object\"\n },\n \"OpenAISettings\": {\n \"additionalProperties\": true,\n \"description\": \"Settings for using OpenAI models in the MCP Agent application.\",\n \"properties\": {\n \"api_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Api Key\"\n },\n \"reasoning_effort\": {\n \"default\": \"medium\",\n \"enum\": [\n \"low\",\n \"medium\",\n \"high\"\n ],\n \"title\": \"Reasoning Effort\",\n \"type\": \"string\"\n },\n \"base_url\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Base Url\"\n }\n },\n \"title\": \"OpenAISettings\",\n \"type\": \"object\"\n },\n \"AzureSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Settings for using Azure models in the MCP Agent application.\",\n \"properties\": {\n \"api_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n }\n ],\n \"default\": null,\n \"title\": \"Api Key\"\n },\n \"endpoint\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n }\n ],\n \"default\": null,\n \"title\": \"Azure Endpoint\"\n },\n \"api_version\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"API Version\"\n }\n },\n \"required\": [\n \"api_key\",\n \"endpoint\"\n ],\n \"title\": \"AzureSettings\",\n \"type\": \"object\"\n },\n \"BedrockSettings\": {\n \"additionalProperties\": true,\n \"description\": \"Settings for using AWS Bedrock models in the MCP Agent application.\",\n \"properties\": {\n \"aws_region\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Region\"\n },\n \"aws_access_key_id\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Access Key Id\"\n },\n \"aws_secret_access_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Secret Access Key\"\n },\n \"aws_session_token\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Session Token\"\n },\n \"profile\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Profile\"\n }\n },\n \"title\": \"BedrockSettings\",\n \"type\": \"object\"\n },\n \"OpenTelemetrySettings\": {\n \"description\": \"OTEL settings for the MCP Agent application.\",\n \"properties\": {\n \"enabled\": {\n \"default\": true,\n \"title\": \"Enabled\",\n \"type\": \"boolean\"\n },\n \"service_name\": {\n \"default\": \"mcp-agent\",\n \"title\": \"Service Name\",\n \"type\": \"string\"\n },\n \"service_instance_id\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Service Instance Id\"\n },\n \"service_version\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Service Version\"\n },\n \"otlp_endpoint\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Otlp Endpoint\",\n \"description\": \"OTLP endpoint for OpenTelemetry tracing\"\n },\n \"console_debug\": {\n \"default\": false,\n \"title\": \"Console Debug\",\n \"type\": \"boolean\",\n \"description\": \"Log spans to console\"\n },\n \"sample_rate\": {\n \"default\": 1.0,\n \"title\": \"Sample Rate\",\n \"type\": \"number\",\n \"description\": \"Sample rate for tracing (1.0 = sample everything)\"\n }\n },\n \"title\": \"OpenTelemetrySettings\",\n \"type\": \"object\"\n },\n \"TemporalSettings\": {\n \"description\": \"Temporal settings for the MCP Agent application.\",\n \"properties\": {\n \"host\": {\n \"title\": \"Host\",\n \"type\": \"string\"\n },\n \"namespace\": {\n \"default\": \"default\",\n \"title\": \"Namespace\",\n \"type\": \"string\"\n },\n \"task_queue\": {\n \"title\": \"Task Queue\",\n \"type\": \"string\"\n },\n \"api_key\": {\n \"anyOf\": [\n {\n \"type\": \"string\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"title\": \"Api Key\"\n }\n },\n \"required\": [\n \"host\",\n \"task_queue\"\n ],\n \"title\": \"TemporalSettings\",\n \"type\": \"object\"\n }\n },\n \"additionalProperties\": true,\n \"description\": \"Configuration schema for MCP Agent applications\",\n \"properties\": {\n \"mcp\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/MCPSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": {\n \"servers\": {}\n },\n \"description\": \"MCP config, such as MCP servers\"\n },\n \"execution_engine\": {\n \"default\": \"asyncio\",\n \"enum\": [\n \"asyncio\",\n \"temporal\"\n ],\n \"title\": \"Execution Engine\",\n \"type\": \"string\",\n \"description\": \"Execution engine for the MCP Agent application\"\n },\n \"temporal\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/TemporalSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"Settings for Temporal workflow orchestration\"\n },\n \"anthropic\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/AnthropicSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"Settings for using Anthropic models in the MCP Agent application\"\n },\n \"cohere\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/CohereSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"Settings for using Cohere models in the MCP Agent application\"\n },\n \"openai\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/OpenAISettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"Settings for using OpenAI models in the MCP Agent application\"\n },\n \"azure\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/AzureSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"Settings for using Azure models in the MCP Agent application\"\n },\n \"bedrock\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/BedrockSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": null,\n \"description\": \"Settings for using Bedrock models in the MCP Agent application\"\n },\n \"otel\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/OpenTelemetrySettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": {\n \"enabled\": true,\n \"service_name\": \"mcp-agent\",\n \"service_instance_id\": null,\n \"service_version\": null,\n \"otlp_endpoint\": null,\n \"console_debug\": false,\n \"sample_rate\": 1.0\n },\n \"description\": \"OpenTelemetry logging settings for the MCP Agent application\"\n },\n \"logger\": {\n \"anyOf\": [\n {\n \"$ref\": \"#/$defs/LoggerSettings\"\n },\n {\n \"type\": \"null\"\n }\n ],\n \"default\": {\n \"type\": \"console\",\n \"transports\": [],\n \"level\": \"info\",\n \"progress_display\": true,\n \"path\": \"mcp-agent.jsonl\",\n \"path_settings\": null,\n \"batch_size\": 100,\n \"flush_interval\": 2.0,\n \"max_queue_size\": 2048,\n \"http_endpoint\": null,\n \"http_headers\": null,\n \"http_timeout\": 5.0\n },\n \"description\": \"Logger settings for the MCP Agent application\"\n }\n },\n \"title\": \"MCP Agent Configuration Schema\",\n \"type\": \"object\",\n \"$schema\": \"http://json-schema.org/draft-07/schema#\"\n}\n"
},
{
"path": "setup.py",
"content": "import setuptools\nfrom pathlib import Path\nimport os\n\n\n# Reading the long description from README.md\ndef read_long_description():\n try:\n return Path(\"README.md\").read_text(encoding=\"utf-8\")\n except FileNotFoundError:\n return \"DeepCode: Open Agentic Coding (Paper2Code & Text2Web & Text2Backend)\"\n\n\n# Retrieving metadata from __init__.py\ndef retrieve_metadata():\n vars2find = [\"__author__\", \"__version__\", \"__url__\"]\n vars2readme = {}\n\n # Use definitive path relative to setup.py location\n init_file_path = os.path.join(os.path.dirname(__file__), \"__init__.py\")\n\n with open(init_file_path, encoding=\"utf-8\") as f:\n for line in f.readlines():\n for v in vars2find:\n if line.startswith(v):\n line = (\n line.replace(\" \", \"\").replace('\"', \"\").replace(\"'\", \"\").strip()\n )\n vars2readme[v] = line.split(\"=\")[1]\n\n # Checking if all required variables are found\n missing_vars = [v for v in vars2find if v not in vars2readme]\n if missing_vars:\n raise ValueError(\n f\"Missing required metadata variables in __init__.py: {missing_vars}\"\n )\n\n return vars2readme\n\n\n# Reading dependencies from requirements.txt\ndef read_requirements():\n deps = []\n try:\n with open(\"./requirements.txt\", encoding=\"utf-8\") as f:\n deps = [\n line.strip() for line in f if line.strip() and not line.startswith(\"#\")\n ]\n except FileNotFoundError:\n print(\n \"Warning: 'requirements.txt' not found. No dependencies will be installed.\"\n )\n return deps\n\n\nmetadata = retrieve_metadata()\nlong_description = read_long_description()\nrequirements = read_requirements()\n\nsetuptools.setup(\n name=\"deepcode-hku\",\n url=metadata[\"__url__\"],\n version=metadata[\"__version__\"],\n author=metadata[\"__author__\"],\n description=\"AI Research Engine - Transform research papers into working code automatically\",\n long_description=long_description,\n long_description_content_type=\"text/markdown\",\n packages=setuptools.find_packages(\n exclude=(\"tests*\", \"docs*\", \".history*\", \".git*\", \".ruff_cache*\")\n ),\n py_modules=[\"deepcode\"],\n classifiers=[\n \"Development Status :: 4 - Beta\",\n \"Programming Language :: Python :: 3\",\n \"License :: OSI Approved :: MIT License\",\n \"Operating System :: OS Independent\",\n \"Intended Audience :: Developers\",\n \"Intended Audience :: Science/Research\",\n \"Topic :: Software Development :: Libraries :: Python Modules\",\n \"Topic :: Scientific/Engineering :: Artificial Intelligence\",\n \"Topic :: Text Processing :: Linguistic\",\n ],\n python_requires=\">=3.9\",\n install_requires=requirements,\n include_package_data=True,\n entry_points={\n \"console_scripts\": [\n \"deepcode=deepcode:main\",\n ],\n },\n project_urls={\n \"Documentation\": metadata.get(\"__url__\", \"\"),\n \"Source\": metadata.get(\"__url__\", \"\"),\n \"Tracker\": f\"{metadata.get('__url__', '')}/issues\"\n if metadata.get(\"__url__\")\n else \"\",\n },\n)\n"
},
{
"path": "tools/__init__.py",
"content": ""
},
{
"path": "tools/bocha_search_server.py",
"content": "import os\nimport sys\nimport json\n\nimport httpx\nfrom dotenv import load_dotenv\nfrom mcp.server.fastmcp import FastMCP\n\nload_dotenv()\n\n\n# Initialize FastMCP server\nserver = FastMCP(\n \"bocha-search-mcp\",\n prompt=\"\"\"\n# Bocha Search MCP Server\n\nBocha is a Chinese search engine for AI, This server provides tools for searching the web using Bocha Search API.\nIt allows you to get enhanced search details from billions of web documents, including weather, news, wikis, healthcare, train tickets, images, and more.\n\n## Available Tools\n\n### 1. bocha_web_search\nSearch with Bocha Web Search and get enhanced search details from billions of web documents, including page titles, urls, summaries, site names, site icons, publication dates, image links, and more.\n\n### 2. bocha_ai_search\nSearch with Bocha AI Search, recognizes the semantics of search terms and additionally returns structured modal cards with content from vertical domains.\n\n## Output Format\n\nAll search results will be formatted as text with clear sections for each\nresult item, including:\n\n- Bocha Web search: Title, URL, Description, Published date and Site name\n- Bocha AI search: Title, URL, Description, Published date, Site name, and structured data card\n\nIf the API key is missing or invalid, appropriate error messages will be returned.\n\"\"\",\n)\n\n\n@server.tool()\nasync def bocha_web_search(\n query: str, freshness: str = \"noLimit\", count: int = 10\n) -> str:\n \"\"\"Search with Bocha Web Search and get enhanced search details from billions of web documents,\n including page titles, urls, summaries, site names, site icons, publication dates, image links, and more.\n\n Args:\n query: Search query (required)\n freshness: The time range for the search results. (Available options YYYY-MM-DD, YYYY-MM-DD..YYYY-MM-DD, noLimit, oneYear, oneMonth, oneWeek, oneDay. Default is noLimit)\n count: Number of results (1-50, default 10)\n \"\"\"\n # Get API key from environment\n boch_api_key = os.environ.get(\"BOCHA_API_KEY\", \"\")\n\n if not boch_api_key:\n return (\n \"Error: Bocha API key is not configured. Please set the \"\n \"BOCHA_API_KEY environment variable.\"\n )\n\n # Endpoint\n endpoint = \"https://api.bochaai.com/v1/web-search?utm_source=bocha-mcp-local\"\n\n try:\n payload = {\n \"query\": query,\n \"summary\": True,\n \"freshness\": freshness,\n \"count\": count,\n }\n\n headers = {\n \"Authorization\": f\"Bearer {boch_api_key}\",\n \"Content-Type\": \"application/json\",\n }\n\n async with httpx.AsyncClient() as client:\n response = await client.post(\n endpoint, headers=headers, json=payload, timeout=10.0\n )\n\n response.raise_for_status()\n resp = response.json()\n if \"data\" not in resp:\n return \"Search error.\"\n\n data = resp[\"data\"]\n\n if \"webPages\" not in data:\n return \"No results found.\"\n\n results = []\n for result in data[\"webPages\"][\"value\"]:\n results.append(\n f\"Title: {result['name']}\\n\"\n f\"URL: {result['url']}\\n\"\n f\"Description: {result['summary']}\\n\"\n f\"Published date: {result['datePublished']}\\n\"\n f\"Site name: {result['siteName']}\"\n )\n\n return \"\\n\\n\".join(results)\n\n except httpx.HTTPStatusError as e:\n return f\"Bocha Web Search API HTTP error occurred: {e.response.status_code} - {e.response.text}\"\n except httpx.RequestError as e:\n return f\"Error communicating with Bocha Web Search API: {str(e)}\"\n except Exception as e:\n return f\"Unexpected error: {str(e)}\"\n\n\n@server.tool()\nasync def bocha_ai_search(\n query: str, freshness: str = \"noLimit\", count: int = 10\n) -> str:\n \"\"\"Search with Bocha AI Search, recognizes the semantics of search terms\n and additionally returns structured modal cards with content from vertical domains.\n\n Args:\n query: Search query (required)\n freshness: The time range for the search results. (Available options noLimit, oneYear, oneMonth, oneWeek, oneDay. Default is noLimit)\n count: Number of results (1-50, default 10)\n \"\"\"\n # Get API key from environment\n boch_api_key = os.environ.get(\"BOCHA_API_KEY\", \"\")\n\n if not boch_api_key:\n return (\n \"Error: Bocha API key is not configured. Please set the \"\n \"BOCHA_API_KEY environment variable.\"\n )\n\n # Endpoint\n endpoint = \"https://api.bochaai.com/v1/ai-search?utm_source=bocha-mcp-local\"\n\n try:\n payload = {\n \"query\": query,\n \"freshness\": freshness,\n \"count\": count,\n \"answer\": False,\n \"stream\": False,\n }\n\n headers = {\n \"Authorization\": f\"Bearer {boch_api_key}\",\n \"Content-Type\": \"application/json\",\n }\n\n async with httpx.AsyncClient() as client:\n response = await client.post(\n endpoint, headers=headers, json=payload, timeout=10.0\n )\n\n response.raise_for_status()\n response = response.json()\n results = []\n if \"messages\" in response:\n for message in response[\"messages\"]:\n content = {}\n try:\n content = json.loads(message[\"content\"])\n except (json.JSONDecodeError, TypeError):\n content = {}\n\n # 网页\n if message[\"content_type\"] == \"webpage\":\n if \"value\" in content:\n for item in content[\"value\"]:\n results.append(\n f\"Title: {item['name']}\\n\"\n f\"URL: {item['url']}\\n\"\n f\"Description: {item['summary']}\\n\"\n f\"Published date: {item['datePublished']}\\n\"\n f\"Site name: {item['siteName']}\"\n )\n elif (\n message[\"content_type\"] != \"image\"\n and message[\"content\"] != \"{}\"\n ):\n results.append(message[\"content\"])\n\n if not results:\n return \"No results found.\"\n\n return \"\\n\\n\".join(results)\n\n except httpx.HTTPStatusError as e:\n return f\"Bocha AI Search API HTTP error occurred: {e.response.status_code} - {e.response.text}\"\n except httpx.RequestError as e:\n return f\"Error communicating with Bocha AI Search API: {str(e)}\"\n except Exception as e:\n return f\"Unexpected error: {str(e)}\"\n\n\ndef main():\n \"\"\"Initialize and run the MCP server.\"\"\"\n\n # Check for required environment variables\n if \"BOCHA_API_KEY\" not in os.environ:\n print(\n \"Error: BOCHA_API_KEY environment variable is required\",\n file=sys.stderr,\n )\n print(\n \"Get a Bocha API key from: \" \"https://open.bochaai.com\",\n file=sys.stderr,\n )\n sys.exit(1)\n\n print(\"Starting Bocha Search MCP server...\", file=sys.stderr)\n\n server.run(transport=\"stdio\")\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "tools/code_implementation_server.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nCode Implementation MCP Server\n\nThis MCP server provides core functions needed for paper code reproduction:\n1. File read/write operations\n2. Code execution and testing\n3. Code search and analysis\n4. Iterative improvement support\n\nUsage:\npython tools/code_implementation_server.py\n\"\"\"\n\nimport os\nimport subprocess\nimport json\nimport sys\nimport io\nfrom pathlib import Path\nimport re\nfrom typing import Dict, Any, List\nimport tempfile\nimport shutil\nimport logging\nfrom datetime import datetime\n\n# Set standard output encoding to UTF-8\nif sys.stdout.encoding != \"utf-8\":\n try:\n if hasattr(sys.stdout, \"reconfigure\"):\n sys.stdout.reconfigure(encoding=\"utf-8\")\n sys.stderr.reconfigure(encoding=\"utf-8\")\n else:\n sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding=\"utf-8\")\n sys.stderr = io.TextIOWrapper(sys.stderr.detach(), encoding=\"utf-8\")\n except Exception as e:\n print(f\"Warning: Could not set UTF-8 encoding: {e}\")\n\n# Import MCP related modules\nfrom mcp.server.fastmcp import FastMCP\n\n# Setup logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\n# Create FastMCP server instance\nmcp = FastMCP(\"code-implementation-server\")\n\n# Global variables: workspace directory and operation history\nWORKSPACE_DIR = None\nOPERATION_HISTORY = []\nCURRENT_FILES = {}\n\n\ndef initialize_workspace(workspace_dir: str = None):\n \"\"\"\n Initialize workspace\n\n By default, the workspace will be set by the workflow via the set_workspace tool to:\n {plan_file_parent}/generate_code\n\n Args:\n workspace_dir: Optional workspace directory path\n \"\"\"\n global WORKSPACE_DIR\n if workspace_dir is None:\n # Default to generate_code directory under current directory, but don't create immediately\n # This default value will be overridden by workflow via set_workspace tool\n WORKSPACE_DIR = Path.cwd() / \"generate_code\"\n # logger.info(f\"Workspace initialized (default value, will be overridden by workflow): {WORKSPACE_DIR}\")\n # logger.info(\"Note: Actual workspace will be set by workflow via set_workspace tool to {plan_file_parent}/generate_code\")\n else:\n WORKSPACE_DIR = Path(workspace_dir).resolve()\n # Only create when explicitly specified\n WORKSPACE_DIR.mkdir(parents=True, exist_ok=True)\n logger.info(f\"Workspace initialized: {WORKSPACE_DIR}\")\n\n\ndef ensure_workspace_exists():\n \"\"\"Ensure workspace directory exists\"\"\"\n global WORKSPACE_DIR\n if WORKSPACE_DIR is None:\n initialize_workspace()\n\n # Create workspace directory (if it doesn't exist)\n if not WORKSPACE_DIR.exists():\n WORKSPACE_DIR.mkdir(parents=True, exist_ok=True)\n logger.info(f\"Workspace directory created: {WORKSPACE_DIR}\")\n\n\ndef validate_path(path: str) -> Path:\n \"\"\"Validate if path is within workspace\"\"\"\n if WORKSPACE_DIR is None:\n initialize_workspace()\n\n full_path = (WORKSPACE_DIR / path).resolve()\n if not str(full_path).startswith(str(WORKSPACE_DIR)):\n raise ValueError(f\"Path {path} is outside workspace scope\")\n return full_path\n\n\ndef log_operation(action: str, details: Dict[str, Any]):\n \"\"\"Log operation history\"\"\"\n OPERATION_HISTORY.append(\n {\"timestamp\": datetime.now().isoformat(), \"action\": action, \"details\": details}\n )\n\n\n# ==================== File Operation Tools ====================\n\n\n@mcp.tool()\nasync def read_file(\n file_path: str, start_line: int = None, end_line: int = None\n) -> str:\n \"\"\"\n Read file content, supports specifying line number range\n\n Args:\n file_path: File path, relative to workspace\n start_line: Starting line number (1-based, optional)\n end_line: Ending line number (1-based, optional)\n\n Returns:\n JSON string of file content or error message\n \"\"\"\n try:\n full_path = validate_path(file_path)\n\n if not full_path.exists():\n result = {\"status\": \"error\", \"message\": f\"File does not exist: {file_path}\"}\n log_operation(\n \"read_file_error\", {\"file_path\": file_path, \"error\": \"file_not_found\"}\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n with open(full_path, \"r\", encoding=\"utf-8\") as f:\n lines = f.readlines()\n\n # 处理行号范围\n if start_line is not None or end_line is not None:\n start_idx = (start_line - 1) if start_line else 0\n end_idx = end_line if end_line else len(lines)\n lines = lines[start_idx:end_idx]\n\n content = \"\".join(lines)\n\n result = {\n \"status\": \"success\",\n \"content\": content,\n \"file_path\": file_path,\n \"total_lines\": len(lines),\n \"size_bytes\": len(content.encode(\"utf-8\")),\n }\n\n log_operation(\n \"read_file\",\n {\n \"file_path\": file_path,\n \"start_line\": start_line,\n \"end_line\": end_line,\n \"lines_read\": len(lines),\n },\n )\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to read file: {str(e)}\",\n \"file_path\": file_path,\n }\n log_operation(\"read_file_error\", {\"file_path\": file_path, \"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def read_multiple_files(file_requests: str, max_files: int = 5) -> str:\n \"\"\"\n Read multiple files in a single operation (for batch reading)\n\n Args:\n file_requests: JSON string with file requests, e.g.,\n '{\"file1.py\": {}, \"file2.py\": {\"start_line\": 1, \"end_line\": 10}}'\n or simple array: '[\"file1.py\", \"file2.py\"]'\n max_files: Maximum number of files to read in one operation (default: 5)\n\n Returns:\n JSON string of operation results for all files\n \"\"\"\n try:\n # Parse the file requests\n try:\n requests_data = json.loads(file_requests)\n except json.JSONDecodeError as e:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"Invalid JSON format for file_requests: {str(e)}\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n # Normalize requests format\n if isinstance(requests_data, list):\n # Convert simple array to dict format\n normalized_requests = {file_path: {} for file_path in requests_data}\n elif isinstance(requests_data, dict):\n normalized_requests = requests_data\n else:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": \"file_requests must be a JSON object or array\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n # Validate input\n if len(normalized_requests) == 0:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": \"No files provided for reading\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n if len(normalized_requests) > max_files:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"Too many files provided ({len(normalized_requests)}), maximum is {max_files}\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n # Process each file\n results = {\n \"status\": \"success\",\n \"message\": f\"Successfully processed {len(normalized_requests)} files\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n \"files_processed\": len(normalized_requests),\n \"files\": {},\n \"summary\": {\n \"successful\": 0,\n \"failed\": 0,\n \"total_size_bytes\": 0,\n \"total_lines\": 0,\n \"files_not_found\": 0,\n },\n }\n\n # Process each file individually\n for file_path, options in normalized_requests.items():\n try:\n full_path = validate_path(file_path)\n start_line = options.get(\"start_line\")\n end_line = options.get(\"end_line\")\n\n if not full_path.exists():\n results[\"files\"][file_path] = {\n \"status\": \"error\",\n \"message\": f\"File does not exist: {file_path}\",\n \"file_path\": file_path,\n \"content\": \"\",\n \"total_lines\": 0,\n \"size_bytes\": 0,\n \"start_line\": start_line,\n \"end_line\": end_line,\n }\n results[\"summary\"][\"failed\"] += 1\n results[\"summary\"][\"files_not_found\"] += 1\n continue\n\n with open(full_path, \"r\", encoding=\"utf-8\") as f:\n lines = f.readlines()\n\n # Handle line range\n original_line_count = len(lines)\n if start_line is not None or end_line is not None:\n start_idx = (start_line - 1) if start_line else 0\n end_idx = end_line if end_line else len(lines)\n lines = lines[start_idx:end_idx]\n\n content = \"\".join(lines)\n size_bytes = len(content.encode(\"utf-8\"))\n lines_count = len(lines)\n\n # Record individual file result\n results[\"files\"][file_path] = {\n \"status\": \"success\",\n \"message\": f\"File read successfully: {file_path}\",\n \"file_path\": file_path,\n \"content\": content,\n \"total_lines\": lines_count,\n \"original_total_lines\": original_line_count,\n \"size_bytes\": size_bytes,\n \"start_line\": start_line,\n \"end_line\": end_line,\n \"line_range_applied\": start_line is not None\n or end_line is not None,\n }\n\n # Update summary\n results[\"summary\"][\"successful\"] += 1\n results[\"summary\"][\"total_size_bytes\"] += size_bytes\n results[\"summary\"][\"total_lines\"] += lines_count\n\n # Log individual file operation\n log_operation(\n \"read_file_multi\",\n {\n \"file_path\": file_path,\n \"start_line\": start_line,\n \"end_line\": end_line,\n \"lines_read\": lines_count,\n \"size_bytes\": size_bytes,\n \"batch_operation\": True,\n },\n )\n\n except Exception as file_error:\n # Record individual file error\n results[\"files\"][file_path] = {\n \"status\": \"error\",\n \"message\": f\"Failed to read file: {str(file_error)}\",\n \"file_path\": file_path,\n \"content\": \"\",\n \"total_lines\": 0,\n \"size_bytes\": 0,\n \"start_line\": options.get(\"start_line\"),\n \"end_line\": options.get(\"end_line\"),\n }\n\n results[\"summary\"][\"failed\"] += 1\n\n # Log individual file error\n log_operation(\n \"read_file_multi_error\",\n {\n \"file_path\": file_path,\n \"error\": str(file_error),\n \"batch_operation\": True,\n },\n )\n\n # Determine overall status\n if results[\"summary\"][\"failed\"] > 0:\n if results[\"summary\"][\"successful\"] > 0:\n results[\"status\"] = \"partial_success\"\n results[\"message\"] = (\n f\"Read {results['summary']['successful']} files successfully, {results['summary']['failed']} failed\"\n )\n else:\n results[\"status\"] = \"failed\"\n results[\"message\"] = (\n f\"All {results['summary']['failed']} files failed to read\"\n )\n\n # Log overall operation\n log_operation(\n \"read_multiple_files\",\n {\n \"files_count\": len(normalized_requests),\n \"successful\": results[\"summary\"][\"successful\"],\n \"failed\": results[\"summary\"][\"failed\"],\n \"total_size_bytes\": results[\"summary\"][\"total_size_bytes\"],\n \"status\": results[\"status\"],\n },\n )\n\n return json.dumps(results, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to read multiple files: {str(e)}\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n \"files_processed\": 0,\n }\n log_operation(\"read_multiple_files_error\", {\"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def write_file(\n file_path: str, content: str, create_dirs: bool = True, create_backup: bool = False\n) -> str:\n \"\"\"\n Write content to file\n\n Args:\n file_path: File path, relative to workspace\n content: Content to write to file\n create_dirs: Whether to create directories if they don't exist\n create_backup: Whether to create backup file if file already exists\n\n Returns:\n JSON string of operation result\n \"\"\"\n try:\n full_path = validate_path(file_path)\n\n # Create directories (if needed)\n if create_dirs:\n full_path.parent.mkdir(parents=True, exist_ok=True)\n\n # Backup existing file (only when explicitly requested)\n backup_created = False\n if full_path.exists() and create_backup:\n backup_path = full_path.with_suffix(full_path.suffix + \".backup\")\n shutil.copy2(full_path, backup_path)\n backup_created = True\n\n # Write file\n with open(full_path, \"w\", encoding=\"utf-8\") as f:\n f.write(content)\n\n # Update current file record\n CURRENT_FILES[file_path] = {\n \"last_modified\": datetime.now().isoformat(),\n \"size_bytes\": len(content.encode(\"utf-8\")),\n \"lines\": len(content.split(\"\\n\")),\n }\n\n result = {\n \"status\": \"success\",\n \"message\": f\"File written successfully: {file_path}\",\n \"file_path\": file_path,\n \"size_bytes\": len(content.encode(\"utf-8\")),\n \"lines_written\": len(content.split(\"\\n\")),\n \"backup_created\": backup_created,\n }\n\n log_operation(\n \"write_file\",\n {\n \"file_path\": file_path,\n \"size_bytes\": len(content.encode(\"utf-8\")),\n \"lines\": len(content.split(\"\\n\")),\n \"backup_created\": backup_created,\n },\n )\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to write file: {str(e)}\",\n \"file_path\": file_path,\n }\n log_operation(\"write_file_error\", {\"file_path\": file_path, \"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def write_multiple_files(\n file_implementations: str,\n create_dirs: bool = True,\n create_backup: bool = False,\n max_files: int = 5,\n) -> str:\n \"\"\"\n Write multiple files in a single operation (for batch implementation)\n\n Args:\n file_implementations: JSON string mapping file paths to content, e.g.,\n '{\"file1.py\": \"content1\", \"file2.py\": \"content2\"}'\n create_dirs: Whether to create directories if they don't exist\n create_backup: Whether to create backup files if they already exist\n max_files: Maximum number of files to write in one operation (default: 5)\n\n Returns:\n JSON string of operation results for all files\n \"\"\"\n try:\n # Parse the file implementations\n try:\n files_dict = json.loads(file_implementations)\n except json.JSONDecodeError as e:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"Invalid JSON format for file_implementations: {str(e)}\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n # Validate input\n if not isinstance(files_dict, dict):\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": \"file_implementations must be a JSON object mapping file paths to content\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n if len(files_dict) == 0:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": \"No files provided for writing\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n if len(files_dict) > max_files:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"Too many files provided ({len(files_dict)}), maximum is {max_files}\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n # Process each file\n results = {\n \"status\": \"success\",\n \"message\": f\"Successfully processed {len(files_dict)} files\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n \"files_processed\": len(files_dict),\n \"files\": {},\n \"summary\": {\n \"successful\": 0,\n \"failed\": 0,\n \"total_size_bytes\": 0,\n \"total_lines\": 0,\n \"backups_created\": 0,\n },\n }\n\n # Process each file individually\n for file_path, content in files_dict.items():\n try:\n full_path = validate_path(file_path)\n\n # Create directories (if needed)\n if create_dirs:\n full_path.parent.mkdir(parents=True, exist_ok=True)\n\n # Backup existing file (only when explicitly requested)\n backup_created = False\n if full_path.exists() and create_backup:\n backup_path = full_path.with_suffix(full_path.suffix + \".backup\")\n shutil.copy2(full_path, backup_path)\n backup_created = True\n results[\"summary\"][\"backups_created\"] += 1\n\n # Write file\n with open(full_path, \"w\", encoding=\"utf-8\") as f:\n f.write(content)\n\n # Calculate file metrics\n size_bytes = len(content.encode(\"utf-8\"))\n lines_count = len(content.split(\"\\n\"))\n\n # Update current file record\n CURRENT_FILES[file_path] = {\n \"last_modified\": datetime.now().isoformat(),\n \"size_bytes\": size_bytes,\n \"lines\": lines_count,\n }\n\n # Record individual file result\n results[\"files\"][file_path] = {\n \"status\": \"success\",\n \"message\": f\"File written successfully: {file_path}\",\n \"size_bytes\": size_bytes,\n \"lines_written\": lines_count,\n \"backup_created\": backup_created,\n }\n\n # Update summary\n results[\"summary\"][\"successful\"] += 1\n results[\"summary\"][\"total_size_bytes\"] += size_bytes\n results[\"summary\"][\"total_lines\"] += lines_count\n\n # Log individual file operation\n log_operation(\n \"write_file_multi\",\n {\n \"file_path\": file_path,\n \"size_bytes\": size_bytes,\n \"lines\": lines_count,\n \"backup_created\": backup_created,\n \"batch_operation\": True,\n },\n )\n\n except Exception as file_error:\n # Record individual file error\n results[\"files\"][file_path] = {\n \"status\": \"error\",\n \"message\": f\"Failed to write file: {str(file_error)}\",\n \"size_bytes\": 0,\n \"lines_written\": 0,\n \"backup_created\": False,\n }\n\n results[\"summary\"][\"failed\"] += 1\n\n # Log individual file error\n log_operation(\n \"write_file_multi_error\",\n {\n \"file_path\": file_path,\n \"error\": str(file_error),\n \"batch_operation\": True,\n },\n )\n\n # Determine overall status\n if results[\"summary\"][\"failed\"] > 0:\n if results[\"summary\"][\"successful\"] > 0:\n results[\"status\"] = \"partial_success\"\n results[\"message\"] = (\n f\"Processed {results['summary']['successful']} files successfully, {results['summary']['failed']} failed\"\n )\n else:\n results[\"status\"] = \"failed\"\n results[\"message\"] = (\n f\"All {results['summary']['failed']} files failed to write\"\n )\n\n # Log overall operation\n log_operation(\n \"write_multiple_files\",\n {\n \"files_count\": len(files_dict),\n \"successful\": results[\"summary\"][\"successful\"],\n \"failed\": results[\"summary\"][\"failed\"],\n \"total_size_bytes\": results[\"summary\"][\"total_size_bytes\"],\n \"status\": results[\"status\"],\n },\n )\n\n return json.dumps(results, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to write multiple files: {str(e)}\",\n \"operation_type\": \"multi_file\",\n \"timestamp\": datetime.now().isoformat(),\n \"files_processed\": 0,\n }\n log_operation(\"write_multiple_files_error\", {\"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n# ==================== Code Execution Tools ====================\n\n\n@mcp.tool()\nasync def execute_python(code: str, timeout: int = 30) -> str:\n \"\"\"\n Execute Python code and return output\n\n Args:\n code: Python code to execute\n timeout: Timeout in seconds\n\n Returns:\n JSON string of execution result\n \"\"\"\n try:\n # Create temporary file\n with tempfile.NamedTemporaryFile(\n mode=\"w\", suffix=\".py\", delete=False, encoding=\"utf-8\"\n ) as f:\n f.write(code)\n temp_file = f.name\n\n try:\n # Ensure workspace directory exists\n ensure_workspace_exists()\n\n # Execute Python code\n result = subprocess.run(\n [sys.executable, temp_file],\n cwd=WORKSPACE_DIR,\n capture_output=True,\n text=True,\n timeout=timeout,\n encoding=\"utf-8\",\n )\n\n execution_result = {\n \"status\": \"success\" if result.returncode == 0 else \"error\",\n \"return_code\": result.returncode,\n \"stdout\": result.stdout,\n \"stderr\": result.stderr,\n \"timeout\": timeout,\n }\n\n if result.returncode != 0:\n execution_result[\"message\"] = \"Python code execution failed\"\n else:\n execution_result[\"message\"] = \"Python code execution successful\"\n\n log_operation(\n \"execute_python\",\n {\n \"return_code\": result.returncode,\n \"stdout_length\": len(result.stdout),\n \"stderr_length\": len(result.stderr),\n },\n )\n\n return json.dumps(execution_result, ensure_ascii=False, indent=2)\n\n finally:\n # Clean up temporary file\n os.unlink(temp_file)\n\n except subprocess.TimeoutExpired:\n result = {\n \"status\": \"error\",\n \"message\": f\"Python code execution timeout ({timeout}秒)\",\n \"timeout\": timeout,\n }\n log_operation(\"execute_python_timeout\", {\"timeout\": timeout})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Python code execution failed: {str(e)}\",\n }\n log_operation(\"execute_python_error\", {\"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def execute_bash(command: str, timeout: int = 30) -> str:\n \"\"\"\n Execute bash command\n\n Args:\n command: Bash command to execute\n timeout: Timeout in seconds\n\n Returns:\n JSON string of execution result\n \"\"\"\n try:\n # 安全检查：禁止危险命令\n dangerous_commands = [\"rm -rf\", \"sudo\", \"chmod 777\", \"mkfs\", \"dd if=\"]\n if any(dangerous in command.lower() for dangerous in dangerous_commands):\n result = {\n \"status\": \"error\",\n \"message\": f\"Dangerous command execution prohibited: {command}\",\n }\n log_operation(\n \"execute_bash_blocked\",\n {\"command\": command, \"reason\": \"dangerous_command\"},\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n # Ensure workspace directory exists\n ensure_workspace_exists()\n\n # Execute command\n result = subprocess.run(\n command,\n shell=True,\n cwd=WORKSPACE_DIR,\n capture_output=True,\n text=True,\n timeout=timeout,\n encoding=\"utf-8\",\n )\n\n execution_result = {\n \"status\": \"success\" if result.returncode == 0 else \"error\",\n \"return_code\": result.returncode,\n \"stdout\": result.stdout,\n \"stderr\": result.stderr,\n \"command\": command,\n \"timeout\": timeout,\n }\n\n if result.returncode != 0:\n execution_result[\"message\"] = \"Bash command execution failed\"\n else:\n execution_result[\"message\"] = \"Bash command execution successful\"\n\n log_operation(\n \"execute_bash\",\n {\n \"command\": command,\n \"return_code\": result.returncode,\n \"stdout_length\": len(result.stdout),\n \"stderr_length\": len(result.stderr),\n },\n )\n\n return json.dumps(execution_result, ensure_ascii=False, indent=2)\n\n except subprocess.TimeoutExpired:\n result = {\n \"status\": \"error\",\n \"message\": f\"Bash command execution timeout ({timeout} seconds)\",\n \"command\": command,\n \"timeout\": timeout,\n }\n log_operation(\"execute_bash_timeout\", {\"command\": command, \"timeout\": timeout})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to execute bash command: {str(e)}\",\n \"command\": command,\n }\n log_operation(\"execute_bash_error\", {\"command\": command, \"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def read_code_mem(file_paths: List[str]) -> str:\n \"\"\"\n Check if file summaries exist in implement_code_summary.md for multiple files\n\n Args:\n file_paths: List of file paths to check for summary information in implement_code_summary.md\n\n Returns:\n Summary information for all requested files if available\n \"\"\"\n try:\n if not file_paths or not isinstance(file_paths, list):\n result = {\n \"status\": \"error\",\n \"message\": \"file_paths parameter is required and must be a list\",\n }\n log_operation(\n \"read_code_mem_error\", {\"error\": \"missing_or_invalid_file_paths\"}\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n # Remove duplicates while preserving order\n unique_file_paths = list(dict.fromkeys(file_paths))\n\n # Ensure workspace exists\n ensure_workspace_exists()\n\n # Look for implement_code_summary.md in the workspace\n current_path = Path(WORKSPACE_DIR)\n summary_file_path = current_path.parent / \"implement_code_summary.md\"\n\n if not summary_file_path.exists():\n result = {\n \"status\": \"no_summary\",\n \"file_paths\": unique_file_paths,\n \"message\": \"No summary file found.\",\n \"results\": [],\n }\n log_operation(\n \"read_code_mem\",\n {\"file_paths\": unique_file_paths, \"status\": \"no_summary_file\"},\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n # Read the summary file\n with open(summary_file_path, \"r\", encoding=\"utf-8\") as f:\n summary_content = f.read()\n\n if not summary_content.strip():\n result = {\n \"status\": \"no_summary\",\n \"file_paths\": unique_file_paths,\n \"message\": \"Summary file is empty.\",\n \"results\": [],\n }\n log_operation(\n \"read_code_mem\",\n {\"file_paths\": unique_file_paths, \"status\": \"empty_summary\"},\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n # Process each file path and collect results\n results = []\n summaries_found = 0\n\n for file_path in unique_file_paths:\n # Extract file-specific section from summary\n file_section = _extract_file_section_from_summary(\n summary_content, file_path\n )\n\n if file_section:\n file_result = {\n \"file_path\": file_path,\n \"status\": \"summary_found\",\n \"summary_content\": file_section,\n \"message\": f\"Summary information found for {file_path}\",\n }\n summaries_found += 1\n else:\n file_result = {\n \"file_path\": file_path,\n \"status\": \"no_summary\",\n \"summary_content\": None,\n \"message\": f\"No summary found for {file_path}\",\n }\n\n results.append(file_result)\n\n # Determine overall status\n if summaries_found == len(unique_file_paths):\n overall_status = \"all_summaries_found\"\n elif summaries_found > 0:\n overall_status = \"partial_summaries_found\"\n else:\n overall_status = \"no_summaries_found\"\n\n result = {\n \"status\": overall_status,\n \"file_paths\": unique_file_paths,\n \"total_requested\": len(unique_file_paths),\n \"summaries_found\": summaries_found,\n \"message\": f\"Found summaries for {summaries_found}/{len(unique_file_paths)} files\",\n \"results\": results,\n }\n\n log_operation(\n \"read_code_mem\",\n {\n \"file_paths\": unique_file_paths,\n \"status\": overall_status,\n \"total_requested\": len(unique_file_paths),\n \"summaries_found\": summaries_found,\n },\n )\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to check code memory: {str(e)}\",\n \"file_paths\": file_paths\n if isinstance(file_paths, list)\n else [str(file_paths)],\n \"results\": [],\n }\n log_operation(\n \"read_code_mem_error\", {\"file_paths\": file_paths, \"error\": str(e)}\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\ndef _extract_file_section_from_summary(\n summary_content: str, target_file_path: str\n) -> str:\n \"\"\"\n Extract the specific section for a file from the summary content\n\n Args:\n summary_content: Full summary content\n target_file_path: Path of the target file\n\n Returns:\n File-specific section or None if not found\n \"\"\"\n import re\n\n # Normalize the target path for comparison\n normalized_target = _normalize_file_path(target_file_path)\n\n # Pattern to match implementation sections with separator lines\n section_pattern = r\"={80}\\s*\\n## IMPLEMENTATION File ([^;]+); ROUND \\d+\\s*\\n={80}(.*?)(?=\\n={80}|\\Z)\"\n\n matches = re.findall(section_pattern, summary_content, re.DOTALL)\n\n for file_path_in_summary, section_content in matches:\n file_path_in_summary = file_path_in_summary.strip()\n section_content = section_content.strip()\n\n # Normalize the path from summary for comparison\n normalized_summary_path = _normalize_file_path(file_path_in_summary)\n\n # Check if paths match using multiple strategies\n if _paths_match(\n normalized_target,\n normalized_summary_path,\n target_file_path,\n file_path_in_summary,\n ):\n # Return the complete section with proper formatting\n file_section = f\"\"\"================================================================================\n## IMPLEMENTATION File {file_path_in_summary}; ROUND [X]\n================================================================================\n\n{section_content}\n\n---\n*Extracted from implement_code_summary.md*\"\"\"\n return file_section\n\n # If no section-based match, try alternative parsing method\n return _extract_file_section_alternative(summary_content, target_file_path)\n\n\ndef _normalize_file_path(file_path: str) -> str:\n \"\"\"Normalize file path for comparison\"\"\"\n # Remove leading/trailing slashes and convert to lowercase\n normalized = file_path.strip(\"/\").lower()\n # Replace backslashes with forward slashes\n normalized = normalized.replace(\"\\\\\", \"/\")\n\n # Remove common prefixes to make matching more flexible\n common_prefixes = [\"src/\", \"./src/\", \"./\", \"core/\", \"lib/\", \"main/\"]\n for prefix in common_prefixes:\n if normalized.startswith(prefix):\n normalized = normalized[len(prefix) :]\n break\n\n return normalized\n\n\ndef _paths_match(\n normalized_target: str,\n normalized_summary: str,\n original_target: str,\n original_summary: str,\n) -> bool:\n \"\"\"Check if two file paths match using multiple strategies\"\"\"\n\n # Strategy 1: Exact normalized match\n if normalized_target == normalized_summary:\n return True\n\n # Strategy 2: Basename match (filename only)\n target_basename = os.path.basename(original_target)\n summary_basename = os.path.basename(original_summary)\n if target_basename == summary_basename and len(target_basename) > 4:\n return True\n\n # Strategy 3: Suffix match (remove common prefixes and compare)\n target_suffix = _remove_common_prefixes(normalized_target)\n summary_suffix = _remove_common_prefixes(normalized_summary)\n if target_suffix == summary_suffix:\n return True\n\n # Strategy 4: Ends with match\n if normalized_target.endswith(normalized_summary) or normalized_summary.endswith(\n normalized_target\n ):\n return True\n\n # Strategy 5: Contains match for longer paths\n if len(normalized_target) > 10 and normalized_target in normalized_summary:\n return True\n if len(normalized_summary) > 10 and normalized_summary in normalized_target:\n return True\n\n return False\n\n\ndef _remove_common_prefixes(file_path: str) -> str:\n \"\"\"Remove common prefixes from file path\"\"\"\n prefixes_to_remove = [\"src/\", \"core/\", \"./\", \"lib/\", \"main/\"]\n path = file_path\n\n for prefix in prefixes_to_remove:\n if path.startswith(prefix):\n path = path[len(prefix) :]\n\n return path\n\n\ndef _extract_file_section_alternative(\n summary_content: str, target_file_path: str\n) -> str:\n \"\"\"Alternative method to extract file section using simpler pattern matching\"\"\"\n\n # Get the basename for fallback matching\n target_basename = os.path.basename(target_file_path)\n\n # Split by separator lines to get individual sections\n sections = summary_content.split(\"=\" * 80)\n\n for i, section in enumerate(sections):\n if \"## IMPLEMENTATION File\" in section:\n # Extract the file path from the header\n lines = section.strip().split(\"\\n\")\n for line in lines:\n if \"## IMPLEMENTATION File\" in line:\n # Extract file path between \"File \" and \"; ROUND\"\n try:\n file_part = line.split(\"File \")[1].split(\"; ROUND\")[0].strip()\n\n # Check if this matches our target\n if (\n _normalize_file_path(target_file_path)\n == _normalize_file_path(file_part)\n or target_basename == os.path.basename(file_part)\n or target_file_path in file_part\n or file_part.endswith(target_file_path)\n ):\n # Get the next section which contains the content\n if i + 1 < len(sections):\n content_section = sections[i + 1].strip()\n return f\"\"\"================================================================================\n## IMPLEMENTATION File {file_part}\n================================================================================\n\n{content_section}\n\n---\n*Extracted from implement_code_summary.md using alternative method*\"\"\"\n except (IndexError, AttributeError):\n continue\n\n return None\n\n\n# ==================== Code Search Tools ====================\n\n\n@mcp.tool()\nasync def search_code(\n pattern: str,\n file_pattern: str = \"*.json\",\n use_regex: bool = False,\n search_directory: str = None,\n) -> str:\n \"\"\"\n Search patterns in code files\n\n Args:\n pattern: Search pattern\n file_pattern: File pattern (e.g., '*.py')\n use_regex: Whether to use regular expressions\n search_directory: Specify search directory (optional, uses WORKSPACE_DIR if not specified)\n\n Returns:\n JSON string of search results\n \"\"\"\n try:\n # Determine search directory\n if search_directory:\n # If search directory is specified, use the specified directory\n if os.path.isabs(search_directory):\n search_path = Path(search_directory)\n else:\n # Relative path, relative to current working directory\n search_path = Path.cwd() / search_directory\n else:\n # 如果没有指定Search directory，使用默认的WORKSPACE_DIR\n ensure_workspace_exists()\n search_path = WORKSPACE_DIR\n\n # 检查Search directory是否存在\n if not search_path.exists():\n result = {\n \"status\": \"error\",\n \"message\": f\"Search directory不存在: {search_path}\",\n \"pattern\": pattern,\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n import glob\n\n # Get matching files\n file_paths = glob.glob(str(search_path / \"**\" / file_pattern), recursive=True)\n\n matches = []\n total_files_searched = 0\n\n for file_path in file_paths:\n try:\n with open(file_path, \"r\", encoding=\"utf-8\") as f:\n lines = f.readlines()\n\n total_files_searched += 1\n relative_path = os.path.relpath(file_path, search_path)\n\n for line_num, line in enumerate(lines, 1):\n if use_regex:\n if re.search(pattern, line):\n matches.append(\n {\n \"file\": relative_path,\n \"line_number\": line_num,\n \"line_content\": line.strip(),\n \"match_type\": \"regex\",\n }\n )\n else:\n if pattern.lower() in line.lower():\n matches.append(\n {\n \"file\": relative_path,\n \"line_number\": line_num,\n \"line_content\": line.strip(),\n \"match_type\": \"substring\",\n }\n )\n\n except Exception as e:\n logger.warning(f\"Error searching file {file_path}: {e}\")\n continue\n\n result = {\n \"status\": \"success\",\n \"pattern\": pattern,\n \"file_pattern\": file_pattern,\n \"use_regex\": use_regex,\n \"search_directory\": str(search_path),\n \"total_matches\": len(matches),\n \"total_files_searched\": total_files_searched,\n \"matches\": matches[:50], # 限制返回前50个匹配\n }\n\n if len(matches) > 50:\n result[\"note\"] = f\"显示前50个匹配，总共找到{len(matches)}个匹配\"\n\n log_operation(\n \"search_code\",\n {\n \"pattern\": pattern,\n \"file_pattern\": file_pattern,\n \"use_regex\": use_regex,\n \"search_directory\": str(search_path),\n \"total_matches\": len(matches),\n \"files_searched\": total_files_searched,\n },\n )\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Code search failed: {str(e)}\",\n \"pattern\": pattern,\n }\n log_operation(\"search_code_error\", {\"pattern\": pattern, \"error\": str(e)})\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n# ==================== File Structure Tools ====================\n\n\n@mcp.tool()\nasync def get_file_structure(directory: str = \".\", max_depth: int = 5) -> str:\n \"\"\"\n Get directory file structure\n\n Args:\n directory: Directory path, relative to workspace\n max_depth: 最大遍历深度\n\n Returns:\n JSON string of file structure\n \"\"\"\n try:\n ensure_workspace_exists()\n\n if directory == \".\":\n target_dir = WORKSPACE_DIR\n else:\n target_dir = validate_path(directory)\n\n if not target_dir.exists():\n result = {\n \"status\": \"error\",\n \"message\": f\"Directory does not exist: {directory}\",\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n def scan_directory(path: Path, current_depth: int = 0) -> Dict[str, Any]:\n \"\"\"Recursively scan directory\"\"\"\n if current_depth >= max_depth:\n return {\"type\": \"directory\", \"name\": path.name, \"truncated\": True}\n\n items = []\n try:\n for item in sorted(path.iterdir()):\n relative_path = os.path.relpath(item, WORKSPACE_DIR)\n\n if item.is_file():\n file_info = {\n \"type\": \"file\",\n \"name\": item.name,\n \"path\": relative_path,\n \"size_bytes\": item.stat().st_size,\n \"extension\": item.suffix,\n }\n items.append(file_info)\n elif item.is_dir() and not item.name.startswith(\".\"):\n dir_info = scan_directory(item, current_depth + 1)\n dir_info[\"path\"] = relative_path\n items.append(dir_info)\n except PermissionError:\n pass\n\n return {\n \"type\": \"directory\",\n \"name\": path.name,\n \"items\": items,\n \"item_count\": len(items),\n }\n\n structure = scan_directory(target_dir)\n\n # 统计信息\n def count_items(node):\n if node[\"type\"] == \"file\":\n return {\"files\": 1, \"directories\": 0}\n else:\n counts = {\"files\": 0, \"directories\": 1}\n for item in node.get(\"items\", []):\n item_counts = count_items(item)\n counts[\"files\"] += item_counts[\"files\"]\n counts[\"directories\"] += item_counts[\"directories\"]\n return counts\n\n counts = count_items(structure)\n\n result = {\n \"status\": \"success\",\n \"directory\": directory,\n \"max_depth\": max_depth,\n \"structure\": structure,\n \"summary\": {\n \"total_files\": counts[\"files\"],\n \"total_directories\": counts[\"directories\"]\n - 1, # Exclude root directory\n },\n }\n\n log_operation(\n \"get_file_structure\",\n {\n \"directory\": directory,\n \"max_depth\": max_depth,\n \"total_files\": counts[\"files\"],\n \"total_directories\": counts[\"directories\"] - 1,\n },\n )\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to get file structure: {str(e)}\",\n \"directory\": directory,\n }\n log_operation(\n \"get_file_structure_error\", {\"directory\": directory, \"error\": str(e)}\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n# ==================== Workspace Management Tools ====================\n\n\n@mcp.tool()\nasync def set_workspace(workspace_path: str) -> str:\n \"\"\"\n Set workspace directory\n\n Called by workflow to set workspace to: {plan_file_parent}/generate_code\n This ensures all file operations are executed relative to the correct project directory\n\n Args:\n workspace_path: Workspace path (Usually {plan_file_parent}/generate_code)\n\n Returns:\n JSON string of operation result\n \"\"\"\n try:\n global WORKSPACE_DIR\n new_workspace = Path(workspace_path).resolve()\n\n # Create directory (if it does not exist)\n new_workspace.mkdir(parents=True, exist_ok=True)\n\n old_workspace = WORKSPACE_DIR\n WORKSPACE_DIR = new_workspace\n\n logger.info(f\"New Workspace: {WORKSPACE_DIR}\")\n\n result = {\n \"status\": \"success\",\n \"message\": f\"Workspace setup successful: {workspace_path}\",\n \"new_workspace\": str(WORKSPACE_DIR),\n }\n\n log_operation(\n \"set_workspace\",\n {\n \"old_workspace\": str(old_workspace) if old_workspace else None,\n \"new_workspace\": str(WORKSPACE_DIR),\n \"workspace_alignment\": \"plan_file_parent/generate_code\",\n },\n )\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to set workspace: {str(e)}\",\n \"workspace_path\": workspace_path,\n }\n log_operation(\n \"set_workspace_error\", {\"workspace_path\": workspace_path, \"error\": str(e)}\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def get_operation_history(last_n: int = 10) -> str:\n \"\"\"\n Get operation history\n\n Args:\n last_n: Return the last N operations\n\n Returns:\n JSON string of operation history\n \"\"\"\n try:\n recent_history = (\n OPERATION_HISTORY[-last_n:] if last_n > 0 else OPERATION_HISTORY\n )\n\n result = {\n \"status\": \"success\",\n \"total_operations\": len(OPERATION_HISTORY),\n \"returned_operations\": len(recent_history),\n \"workspace\": str(WORKSPACE_DIR) if WORKSPACE_DIR else None,\n \"history\": recent_history,\n }\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to get operation history: {str(e)}\",\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n# ==================== Server Initialization ====================\n\n\ndef main():\n \"\"\"Start MCP server\"\"\"\n print(\"🚀 Code Implementation MCP Server\")\n print(\n \"📝 Paper Code Implementation Tool Server / Paper Code Implementation Tool Server\"\n )\n print(\"\")\n print(\"Available tools / Available tools:\")\n # print(\" • read_file - Read file contents / Read file contents\")\n print(\n \" • read_code_mem - Read code summary from implement_code_summary.md / Read code summary from implement_code_summary.md\"\n )\n print(\" • write_file - Write file contents / Write file contents\")\n print(\" • execute_python - Execute Python code / Execute Python code\")\n print(\" • execute_bash - Execute bash command / Execute bash commands\")\n print(\" • search_code - Search code patterns / Search code patterns\")\n print(\" • get_file_structure - Get file structure / Get file structure\")\n print(\" • set_workspace - Set workspace / Set workspace\")\n print(\" • get_operation_history - Get operation history / Get operation history\")\n print(\"\")\n print(\"🔧 Server starting...\")\n\n # Initialize default workspace\n initialize_workspace()\n\n # Start server\n mcp.run()\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "tools/code_indexer.py",
"content": "\"\"\"\nCode Indexer for Repository Analysis\n\nAnalyzes code repositories to build comprehensive indexes for each subdirectory,\nidentifying file relationships and reusable components for implementation.\n\nFeatures:\n- Recursive file traversal\n- LLM-powered code similarity analysis using augmented LLM classes\n- JSON-based relationship storage\n- Configurable matching strategies\n- Progress tracking and error handling\n- Automatic LLM provider selection based on API key availability\n\"\"\"\n\nimport asyncio\nimport json\nimport logging\nimport os\nimport re\nfrom datetime import datetime\nfrom pathlib import Path\nfrom dataclasses import dataclass, asdict\nfrom typing import List, Dict, Any\n\n# MCP Agent imports for LLM\nfrom utils.llm_utils import get_preferred_llm_class, get_default_models\n\n\n@dataclass\nclass FileRelationship:\n \"\"\"Represents a relationship between a repo file and target structure file\"\"\"\n\n repo_file_path: str\n target_file_path: str\n relationship_type: str # 'direct_match', 'partial_match', 'reference', 'utility'\n confidence_score: float # 0.0 to 1.0\n helpful_aspects: List[str]\n potential_contributions: List[str]\n usage_suggestions: str\n\n\n@dataclass\nclass FileSummary:\n \"\"\"Summary information for a repository file\"\"\"\n\n file_path: str\n file_type: str\n main_functions: List[str]\n key_concepts: List[str]\n dependencies: List[str]\n summary: str\n lines_of_code: int\n last_modified: str\n\n\n@dataclass\nclass RepoIndex:\n \"\"\"Complete index for a repository\"\"\"\n\n repo_name: str\n total_files: int\n file_summaries: List[FileSummary]\n relationships: List[FileRelationship]\n analysis_metadata: Dict[str, Any]\n\n\nclass CodeIndexer:\n \"\"\"Main class for building code repository indexes\"\"\"\n\n def __init__(\n self,\n code_base_path: str = None,\n target_structure: str = None,\n output_dir: str = None,\n config_path: str = \"mcp_agent.secrets.yaml\",\n indexer_config_path: str = None,\n enable_pre_filtering: bool = True,\n ):\n # Load configurations first\n self.config_path = config_path\n self.indexer_config_path = indexer_config_path\n # Derive main config path from secrets path (same directory)\n secrets_dir = os.path.dirname(os.path.abspath(config_path))\n self.main_config_path = os.path.join(secrets_dir, \"mcp_agent.config.yaml\")\n self.api_config = self._load_api_config()\n self.indexer_config = self._load_indexer_config()\n self.default_models = get_default_models(self.main_config_path)\n\n # Use config paths if not provided as parameters\n paths_config = self.indexer_config.get(\"paths\", {})\n self.code_base_path = Path(\n code_base_path or paths_config.get(\"code_base_path\", \"code_base\")\n )\n self.output_dir = Path(output_dir or paths_config.get(\"output_dir\", \"indexes\"))\n self.target_structure = (\n target_structure # This must be provided as it's project-specific\n )\n self.enable_pre_filtering = enable_pre_filtering\n\n # LLM clients\n self.llm_client = None\n self.llm_client_type = None\n\n # Initialize logger early\n self.logger = self._setup_logger()\n\n # Create output directory if it doesn't exist\n self.output_dir.mkdir(parents=True, exist_ok=True)\n\n # Load file analysis configuration\n file_analysis_config = self.indexer_config.get(\"file_analysis\", {})\n self.supported_extensions = set(\n file_analysis_config.get(\n \"supported_extensions\",\n [\n \".py\",\n \".js\",\n \".ts\",\n \".java\",\n \".cpp\",\n \".c\",\n \".h\",\n \".hpp\",\n \".cs\",\n \".php\",\n \".rb\",\n \".go\",\n \".rs\",\n \".scala\",\n \".kt\",\n \".swift\",\n \".m\",\n \".mm\",\n \".r\",\n \".matlab\",\n \".sql\",\n \".sh\",\n \".bat\",\n \".ps1\",\n \".yaml\",\n \".yml\",\n \".json\",\n \".xml\",\n \".toml\",\n ],\n )\n )\n\n self.skip_directories = set(\n file_analysis_config.get(\n \"skip_directories\",\n [\n \"__pycache__\",\n \"node_modules\",\n \"target\",\n \"build\",\n \"dist\",\n \"venv\",\n \"env\",\n ],\n )\n )\n\n self.max_file_size = file_analysis_config.get(\"max_file_size\", 1048576) # 1MB\n self.max_content_length = file_analysis_config.get(\"max_content_length\", 3000)\n\n # Load LLM configuration\n llm_config = self.indexer_config.get(\"llm\", {})\n self.model_provider = llm_config.get(\"model_provider\", \"anthropic\")\n self.llm_max_tokens = llm_config.get(\"max_tokens\", 4000)\n self.llm_temperature = llm_config.get(\"temperature\", 0.3)\n self.llm_system_prompt = llm_config.get(\n \"system_prompt\",\n \"You are a code analysis expert. Provide precise, structured analysis of code relationships and similarities.\",\n )\n self.request_delay = llm_config.get(\"request_delay\", 0.1)\n self.max_retries = llm_config.get(\"max_retries\", 3)\n self.retry_delay = llm_config.get(\"retry_delay\", 1.0)\n\n # Load relationship configuration\n relationship_config = self.indexer_config.get(\"relationships\", {})\n self.min_confidence_score = relationship_config.get(\"min_confidence_score\", 0.3)\n self.high_confidence_threshold = relationship_config.get(\n \"high_confidence_threshold\", 0.7\n )\n self.relationship_types = relationship_config.get(\n \"relationship_types\",\n {\n \"direct_match\": 1.0,\n \"partial_match\": 0.8,\n \"reference\": 0.6,\n \"utility\": 0.4,\n },\n )\n\n # Load performance configuration\n performance_config = self.indexer_config.get(\"performance\", {})\n self.enable_concurrent_analysis = performance_config.get(\n \"enable_concurrent_analysis\", False\n )\n self.max_concurrent_files = performance_config.get(\"max_concurrent_files\", 5)\n self.enable_content_caching = performance_config.get(\n \"enable_content_caching\", False\n )\n self.max_cache_size = performance_config.get(\"max_cache_size\", 100)\n\n # Load debug configuration\n debug_config = self.indexer_config.get(\"debug\", {})\n self.save_raw_responses = debug_config.get(\"save_raw_responses\", False)\n self.raw_responses_dir = debug_config.get(\n \"raw_responses_dir\", \"debug_responses\"\n )\n self.verbose_output = debug_config.get(\"verbose_output\", False)\n self.mock_llm_responses = debug_config.get(\"mock_llm_responses\", False)\n\n # Load output configuration\n output_config = self.indexer_config.get(\"output\", {})\n self.generate_summary = output_config.get(\"generate_summary\", True)\n self.generate_statistics = output_config.get(\"generate_statistics\", True)\n self.include_metadata = output_config.get(\"include_metadata\", True)\n self.index_filename_pattern = output_config.get(\n \"index_filename_pattern\", \"{repo_name}_index.json\"\n )\n self.summary_filename = output_config.get(\n \"summary_filename\", \"indexing_summary.json\"\n )\n self.stats_filename = output_config.get(\n \"stats_filename\", \"indexing_statistics.json\"\n )\n\n # Initialize caching if enabled\n self.content_cache = {} if self.enable_content_caching else None\n\n # Create debug directory if needed\n if self.save_raw_responses:\n Path(self.raw_responses_dir).mkdir(parents=True, exist_ok=True)\n\n # Debug logging\n if self.verbose_output:\n self.logger.info(\n f\"Initialized CodeIndexer with config: {self.indexer_config_path}\"\n )\n self.logger.info(f\"Code base path: {self.code_base_path}\")\n self.logger.info(f\"Output directory: {self.output_dir}\")\n self.logger.info(f\"Model provider: {self.model_provider}\")\n self.logger.info(f\"Concurrent analysis: {self.enable_concurrent_analysis}\")\n self.logger.info(f\"Content caching: {self.enable_content_caching}\")\n self.logger.info(f\"Mock LLM responses: {self.mock_llm_responses}\")\n\n def _setup_logger(self) -> logging.Logger:\n \"\"\"Setup logging configuration from config file\"\"\"\n logger = logging.getLogger(\"CodeIndexer\")\n\n # Get logging config\n logging_config = self.indexer_config.get(\"logging\", {})\n log_level = logging_config.get(\"level\", \"INFO\")\n log_format = logging_config.get(\n \"log_format\", \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n )\n\n logger.setLevel(getattr(logging, log_level.upper(), logging.INFO))\n\n # Clear existing handlers\n logger.handlers.clear()\n\n # Console handler\n handler = logging.StreamHandler()\n formatter = logging.Formatter(log_format)\n handler.setFormatter(formatter)\n logger.addHandler(handler)\n\n # File handler if enabled\n if logging_config.get(\"log_to_file\", False):\n log_file = logging_config.get(\"log_file\", \"indexer.log\")\n file_handler = logging.FileHandler(log_file, encoding=\"utf-8\")\n file_handler.setFormatter(formatter)\n logger.addHandler(file_handler)\n\n return logger\n\n def _load_api_config(self) -> Dict[str, Any]:\n \"\"\"Load API configuration from YAML file\"\"\"\n try:\n import yaml\n\n with open(self.config_path, \"r\", encoding=\"utf-8\") as f:\n return yaml.safe_load(f)\n except Exception as e:\n # Create a basic logger for this error since self.logger doesn't exist yet\n print(f\"Warning: Failed to load API config from {self.config_path}: {e}\")\n return {}\n\n def _load_indexer_config(self) -> Dict[str, Any]:\n \"\"\"Load indexer configuration from YAML file\"\"\"\n try:\n import yaml\n\n with open(self.indexer_config_path, \"r\", encoding=\"utf-8\") as f:\n config = yaml.safe_load(f)\n if config is None:\n config = {}\n return config\n except Exception as e:\n print(\n f\"Warning: Failed to load indexer config from {self.indexer_config_path}: {e}\"\n )\n print(\"Using default configuration values\")\n return {}\n\n async def _initialize_llm_client(self):\n \"\"\"Initialize LLM client (Anthropic or OpenAI) based on API key availability\"\"\"\n if self.llm_client is not None:\n return self.llm_client, self.llm_client_type\n\n # Check if mock responses are enabled\n if self.mock_llm_responses:\n self.logger.info(\"Using mock LLM responses for testing\")\n self.llm_client = \"mock\"\n self.llm_client_type = \"mock\"\n return \"mock\", \"mock\"\n\n # Check which API has available key and try that first\n anthropic_key = self.api_config.get(\"anthropic\", {}).get(\"api_key\", \"\")\n openai_key = self.api_config.get(\"openai\", {}).get(\"api_key\", \"\")\n\n # Try Anthropic API first if key is available\n if anthropic_key and anthropic_key.strip():\n try:\n from anthropic import AsyncAnthropic\n\n client = AsyncAnthropic(api_key=anthropic_key)\n # Test connection with default model from config\n await client.messages.create(\n model=self.default_models[\"anthropic\"],\n max_tokens=10,\n messages=[{\"role\": \"user\", \"content\": \"test\"}],\n )\n self.logger.info(\n f\"Using Anthropic API with model: {self.default_models['anthropic']}\"\n )\n self.llm_client = client\n self.llm_client_type = \"anthropic\"\n return client, \"anthropic\"\n except Exception as e:\n self.logger.warning(f\"Anthropic API unavailable: {e}\")\n\n # Try OpenAI API if Anthropic failed or key not available\n if openai_key and openai_key.strip():\n try:\n from openai import AsyncOpenAI\n\n # Handle custom base_url if specified\n openai_config = self.api_config.get(\"openai\", {})\n base_url = openai_config.get(\"base_url\")\n\n if base_url:\n client = AsyncOpenAI(api_key=openai_key, base_url=base_url)\n else:\n client = AsyncOpenAI(api_key=openai_key)\n\n # Test connection with default model from config\n await client.chat.completions.create(\n model=self.default_models[\"openai\"],\n max_tokens=10,\n messages=[{\"role\": \"user\", \"content\": \"test\"}],\n )\n self.logger.info(\n f\"Using OpenAI API with model: {self.default_models['openai']}\"\n )\n if base_url:\n self.logger.info(f\"Using custom base URL: {base_url}\")\n self.llm_client = client\n self.llm_client_type = \"openai\"\n return client, \"openai\"\n except Exception as e:\n self.logger.warning(f\"OpenAI API unavailable: {e}\")\n\n raise ValueError(\n \"No available LLM API - please check your API keys in configuration\"\n )\n\n async def _call_llm(\n self, prompt: str, system_prompt: str = None, max_tokens: int = None\n ) -> str:\n \"\"\"Call LLM for code analysis with retry mechanism and debugging support\"\"\"\n if system_prompt is None:\n system_prompt = self.llm_system_prompt\n if max_tokens is None:\n max_tokens = self.llm_max_tokens\n\n # Mock response for testing\n if self.mock_llm_responses:\n mock_response = self._generate_mock_response(prompt)\n if self.save_raw_responses:\n self._save_debug_response(\"mock\", prompt, mock_response)\n return mock_response\n\n last_error = None\n\n # Retry mechanism\n for attempt in range(self.max_retries):\n try:\n if self.verbose_output and attempt > 0:\n self.logger.info(\n f\"LLM call attempt {attempt + 1}/{self.max_retries}\"\n )\n\n client, client_type = await self._initialize_llm_client()\n\n if client_type == \"anthropic\":\n response = await client.messages.create(\n model=self.default_models[\"anthropic\"],\n system=system_prompt,\n messages=[{\"role\": \"user\", \"content\": prompt}],\n max_tokens=max_tokens,\n temperature=self.llm_temperature,\n )\n\n content = \"\"\n for block in response.content:\n if block.type == \"text\":\n content += block.text\n\n # Save debug response if enabled\n if self.save_raw_responses:\n self._save_debug_response(\"anthropic\", prompt, content)\n\n return content\n\n elif client_type == \"openai\":\n messages = [\n {\"role\": \"system\", \"content\": system_prompt},\n {\"role\": \"user\", \"content\": prompt},\n ]\n\n response = await client.chat.completions.create(\n model=self.default_models[\"openai\"],\n messages=messages,\n max_tokens=max_tokens,\n temperature=self.llm_temperature,\n )\n\n content = response.choices[0].message.content or \"\"\n\n # Save debug response if enabled\n if self.save_raw_responses:\n self._save_debug_response(\"openai\", prompt, content)\n\n return content\n else:\n raise ValueError(f\"Unsupported client type: {client_type}\")\n\n except Exception as e:\n last_error = e\n self.logger.warning(f\"LLM call attempt {attempt + 1} failed: {e}\")\n\n if attempt < self.max_retries - 1:\n await asyncio.sleep(\n self.retry_delay * (attempt + 1)\n ) # Exponential backoff\n\n # All retries failed\n error_msg = f\"LLM call failed after {self.max_retries} attempts. Last error: {str(last_error)}\"\n self.logger.error(error_msg)\n return f\"Error in LLM analysis: {error_msg}\"\n\n def _generate_mock_response(self, prompt: str) -> str:\n \"\"\"Generate mock LLM response for testing\"\"\"\n if \"JSON format\" in prompt and \"file_type\" in prompt:\n # File analysis mock\n return \"\"\"\n {\n \"file_type\": \"Python module\",\n \"main_functions\": [\"main_function\", \"helper_function\"],\n \"key_concepts\": [\"data_processing\", \"algorithm\"],\n \"dependencies\": [\"numpy\", \"pandas\"],\n \"summary\": \"Mock analysis of code file functionality.\"\n }\n \"\"\"\n elif \"relationships\" in prompt:\n # Relationship analysis mock\n return \"\"\"\n {\n \"relationships\": [\n {\n \"target_file_path\": \"src/core/mock.py\",\n \"relationship_type\": \"partial_match\",\n \"confidence_score\": 0.8,\n \"helpful_aspects\": [\"algorithm implementation\", \"data structures\"],\n \"potential_contributions\": [\"core functionality\", \"utility methods\"],\n \"usage_suggestions\": \"Mock relationship suggestion for testing.\"\n }\n ]\n }\n \"\"\"\n elif \"relevant_files\" in prompt:\n # File filtering mock\n return \"\"\"\n {\n \"relevant_files\": [\n {\n \"file_path\": \"mock_file.py\",\n \"relevance_reason\": \"Mock relevance reason\",\n \"confidence\": 0.9,\n \"expected_contribution\": \"Mock contribution\"\n }\n ],\n \"summary\": {\n \"total_files_analyzed\": \"10\",\n \"relevant_files_count\": \"1\",\n \"filtering_strategy\": \"Mock filtering strategy\"\n }\n }\n \"\"\"\n else:\n return \"Mock LLM response for testing purposes.\"\n\n def _save_debug_response(self, provider: str, prompt: str, response: str):\n \"\"\"Save LLM response for debugging\"\"\"\n try:\n import hashlib\n from datetime import datetime\n\n # Create a hash of the prompt for filename\n prompt_hash = hashlib.md5(prompt.encode()).hexdigest()[:8]\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n filename = f\"{provider}_{timestamp}_{prompt_hash}.json\"\n\n debug_data = {\n \"timestamp\": datetime.now().isoformat(),\n \"provider\": provider,\n \"prompt\": prompt[:500] + \"...\" if len(prompt) > 500 else prompt,\n \"response\": response,\n \"full_prompt_length\": len(prompt),\n }\n\n debug_file = Path(self.raw_responses_dir) / filename\n with open(debug_file, \"w\", encoding=\"utf-8\") as f:\n json.dump(debug_data, f, indent=2, ensure_ascii=False)\n\n except Exception as e:\n self.logger.warning(f\"Failed to save debug response: {e}\")\n\n def get_all_repo_files(self, repo_path: Path) -> List[Path]:\n \"\"\"Recursively get all supported files in a repository\"\"\"\n files = []\n\n try:\n for root, dirs, filenames in os.walk(repo_path):\n # Skip common non-code directories\n dirs[:] = [\n d\n for d in dirs\n if not d.startswith(\".\") and d not in self.skip_directories\n ]\n\n for filename in filenames:\n file_path = Path(root) / filename\n if file_path.suffix.lower() in self.supported_extensions:\n files.append(file_path)\n\n except Exception as e:\n self.logger.error(f\"Error traversing {repo_path}: {e}\")\n\n return files\n\n def generate_file_tree(self, repo_path: Path, max_depth: int = 5) -> str:\n \"\"\"Generate file tree structure string for the repository\"\"\"\n tree_lines = []\n\n def add_to_tree(current_path: Path, prefix: str = \"\", depth: int = 0):\n if depth > max_depth:\n return\n\n try:\n items = sorted(\n current_path.iterdir(), key=lambda x: (x.is_file(), x.name.lower())\n )\n # Filter out irrelevant directories and files\n items = [\n item\n for item in items\n if not item.name.startswith(\".\")\n and item.name not in self.skip_directories\n ]\n\n for i, item in enumerate(items):\n is_last = i == len(items) - 1\n current_prefix = \"└── \" if is_last else \"├── \"\n tree_lines.append(f\"{prefix}{current_prefix}{item.name}\")\n\n if item.is_dir():\n extension_prefix = \" \" if is_last else \"│ \"\n add_to_tree(item, prefix + extension_prefix, depth + 1)\n elif item.suffix.lower() in self.supported_extensions:\n # Add file size information\n try:\n size = item.stat().st_size\n if size > 1024:\n size_str = f\" ({size // 1024}KB)\"\n else:\n size_str = f\" ({size}B)\"\n tree_lines[-1] += size_str\n except (OSError, PermissionError):\n pass\n\n except PermissionError:\n tree_lines.append(f\"{prefix}├── [Permission Denied]\")\n except Exception as e:\n tree_lines.append(f\"{prefix}├── [Error: {str(e)}]\")\n\n tree_lines.append(f\"{repo_path.name}/\")\n add_to_tree(repo_path)\n return \"\\n\".join(tree_lines)\n\n async def pre_filter_files(self, repo_path: Path, file_tree: str) -> List[str]:\n \"\"\"Use LLM to pre-filter relevant files based on target structure\"\"\"\n filter_prompt = f\"\"\"\n You are a code analysis expert. Please analyze the following code repository file tree based on the target project structure and filter out files that may be relevant to the target project.\n\n Target Project Structure:\n {self.target_structure}\n\n Code Repository File Tree:\n {file_tree}\n\n Please analyze which files might be helpful for implementing the target project structure, including:\n - Core algorithm implementation files (such as GCN, recommendation systems, graph neural networks, etc.)\n - Data processing and preprocessing files\n - Loss functions and evaluation metric files\n - Configuration and utility files\n - Test files\n - Documentation files\n\n Please return the filtering results in JSON format:\n {{\n \"relevant_files\": [\n {{\n \"file_path\": \"file path relative to repository root\",\n \"relevance_reason\": \"why this file is relevant\",\n \"confidence\": 0.0-1.0,\n \"expected_contribution\": \"expected contribution to the target project\"\n }}\n ],\n \"summary\": {{\n \"total_files_analyzed\": \"total number of files analyzed\",\n \"relevant_files_count\": \"number of relevant files\",\n \"filtering_strategy\": \"explanation of filtering strategy\"\n }}\n }}\n\n Only return files with confidence > {self.min_confidence_score}. Focus on files related to recommendation systems, graph neural networks, and diffusion models.\n \"\"\"\n\n try:\n self.logger.info(\"Starting LLM pre-filtering of files...\")\n llm_response = await self._call_llm(\n filter_prompt,\n system_prompt=\"You are a professional code analysis and project architecture expert, skilled at identifying code file functionality and relevance.\",\n max_tokens=2000,\n )\n\n # Parse JSON response\n match = re.search(r\"\\{.*\\}\", llm_response, re.DOTALL)\n if not match:\n self.logger.warning(\n \"Unable to parse LLM filtering response, will use all files\"\n )\n return []\n\n filter_data = json.loads(match.group(0))\n relevant_files = filter_data.get(\"relevant_files\", [])\n\n # Extract file paths\n selected_files = []\n for file_info in relevant_files:\n file_path = file_info.get(\"file_path\", \"\")\n confidence = file_info.get(\"confidence\", 0.0)\n # Use configured minimum confidence threshold\n if file_path and confidence > self.min_confidence_score:\n selected_files.append(file_path)\n\n summary = filter_data.get(\"summary\", {})\n self.logger.info(\n f\"LLM filtering completed: {summary.get('relevant_files_count', len(selected_files))} relevant files selected\"\n )\n self.logger.info(\n f\"Filtering strategy: {summary.get('filtering_strategy', 'Not provided')}\"\n )\n\n return selected_files\n\n except Exception as e:\n self.logger.error(f\"LLM pre-filtering failed: {e}\")\n self.logger.info(\"Will fallback to analyzing all files\")\n return []\n\n def filter_files_by_paths(\n self, all_files: List[Path], selected_paths: List[str], repo_path: Path\n ) -> List[Path]:\n \"\"\"Filter file list based on LLM-selected paths\"\"\"\n if not selected_paths:\n return all_files\n\n filtered_files = []\n\n for file_path in all_files:\n # Get path relative to repository root\n relative_path = str(file_path.relative_to(repo_path))\n\n # Check if it's in the selected list\n for selected_path in selected_paths:\n # Normalize path comparison\n if (\n relative_path == selected_path\n or relative_path.replace(\"\\\\\", \"/\")\n == selected_path.replace(\"\\\\\", \"/\")\n or selected_path in relative_path\n or relative_path in selected_path\n ):\n filtered_files.append(file_path)\n break\n\n return filtered_files\n\n def _get_cache_key(self, file_path: Path) -> str:\n \"\"\"Generate cache key for file content\"\"\"\n try:\n stats = file_path.stat()\n return f\"{file_path}:{stats.st_mtime}:{stats.st_size}\"\n except (OSError, PermissionError):\n return str(file_path)\n\n def _manage_cache_size(self):\n \"\"\"Manage cache size to stay within limits\"\"\"\n if not self.enable_content_caching or not self.content_cache:\n return\n\n if len(self.content_cache) > self.max_cache_size:\n # Remove oldest entries (simple FIFO strategy)\n excess_count = len(self.content_cache) - self.max_cache_size + 10\n keys_to_remove = list(self.content_cache.keys())[:excess_count]\n\n for key in keys_to_remove:\n del self.content_cache[key]\n\n if self.verbose_output:\n self.logger.info(\n f\"Cache cleaned: removed {excess_count} entries, {len(self.content_cache)} entries remaining\"\n )\n\n async def analyze_file_content(self, file_path: Path) -> FileSummary:\n \"\"\"Analyze a single file and create summary with caching support\"\"\"\n try:\n # Check file size before reading\n file_size = file_path.stat().st_size\n if file_size > self.max_file_size:\n self.logger.warning(\n f\"Skipping file {file_path} - size {file_size} bytes exceeds limit {self.max_file_size}\"\n )\n return FileSummary(\n file_path=str(file_path.relative_to(self.code_base_path)),\n file_type=\"skipped - too large\",\n main_functions=[],\n key_concepts=[],\n dependencies=[],\n summary=f\"File skipped - size {file_size} bytes exceeds {self.max_file_size} byte limit\",\n lines_of_code=0,\n last_modified=datetime.fromtimestamp(\n file_path.stat().st_mtime\n ).isoformat(),\n )\n\n # Check cache if enabled\n cache_key = None\n if self.enable_content_caching:\n cache_key = self._get_cache_key(file_path)\n if cache_key in self.content_cache:\n if self.verbose_output:\n self.logger.info(f\"Using cached analysis for {file_path.name}\")\n return self.content_cache[cache_key]\n\n with open(file_path, \"r\", encoding=\"utf-8\", errors=\"ignore\") as f:\n content = f.read()\n\n # Get file stats\n stats = file_path.stat()\n lines_of_code = len([line for line in content.split(\"\\n\") if line.strip()])\n\n # Truncate content based on config\n content_for_analysis = content[: self.max_content_length]\n content_suffix = \"...\" if len(content) > self.max_content_length else \"\"\n\n # Create analysis prompt\n analysis_prompt = f\"\"\"\n Analyze this code file and provide a structured summary:\n\n File: {file_path.name}\n Content:\n ```\n {content_for_analysis}{content_suffix}\n ```\n\n Please provide analysis in this JSON format:\n {{\n \"file_type\": \"description of what type of file this is\",\n \"main_functions\": [\"list\", \"of\", \"main\", \"functions\", \"or\", \"classes\"],\n \"key_concepts\": [\"important\", \"concepts\", \"algorithms\", \"patterns\"],\n \"dependencies\": [\"external\", \"libraries\", \"or\", \"imports\"],\n \"summary\": \"2-3 sentence summary of what this file does\"\n }}\n\n Focus on the core functionality and potential reusability.\n \"\"\"\n\n # Get LLM analysis with configured parameters\n llm_response = await self._call_llm(analysis_prompt, max_tokens=1000)\n\n try:\n # Try to parse JSON response\n match = re.search(r\"\\{.*\\}\", llm_response, re.DOTALL)\n analysis_data = json.loads(match.group(0))\n except json.JSONDecodeError:\n # Fallback to basic analysis if JSON parsing fails\n analysis_data = {\n \"file_type\": f\"{file_path.suffix} file\",\n \"main_functions\": [],\n \"key_concepts\": [],\n \"dependencies\": [],\n \"summary\": \"File analysis failed - JSON parsing error\",\n }\n\n file_summary = FileSummary(\n file_path=str(file_path.relative_to(self.code_base_path)),\n file_type=analysis_data.get(\"file_type\", \"unknown\"),\n main_functions=analysis_data.get(\"main_functions\", []),\n key_concepts=analysis_data.get(\"key_concepts\", []),\n dependencies=analysis_data.get(\"dependencies\", []),\n summary=analysis_data.get(\"summary\", \"No summary available\"),\n lines_of_code=lines_of_code,\n last_modified=datetime.fromtimestamp(stats.st_mtime).isoformat(),\n )\n\n # Cache the result if caching is enabled\n if self.enable_content_caching and cache_key:\n self.content_cache[cache_key] = file_summary\n self._manage_cache_size()\n\n return file_summary\n\n except Exception as e:\n self.logger.error(f\"Error analyzing file {file_path}: {e}\")\n return FileSummary(\n file_path=str(file_path.relative_to(self.code_base_path)),\n file_type=\"error\",\n main_functions=[],\n key_concepts=[],\n dependencies=[],\n summary=f\"Analysis failed: {str(e)}\",\n lines_of_code=0,\n last_modified=\"\",\n )\n\n async def find_relationships(\n self, file_summary: FileSummary\n ) -> List[FileRelationship]:\n \"\"\"Find relationships between a repo file and target structure\"\"\"\n\n # Build relationship type description from config\n relationship_type_desc = []\n for rel_type, weight in self.relationship_types.items():\n relationship_type_desc.append(f\"- {rel_type} (priority: {weight})\")\n\n relationship_prompt = f\"\"\"\n Analyze the relationship between this existing code file and the target project structure.\n\n Existing File Analysis:\n - Path: {file_summary.file_path}\n - Type: {file_summary.file_type}\n - Functions: {', '.join(file_summary.main_functions)}\n - Concepts: {', '.join(file_summary.key_concepts)}\n - Summary: {file_summary.summary}\n\n Target Project Structure:\n {self.target_structure}\n\n Available relationship types (with priority weights):\n {chr(10).join(relationship_type_desc)}\n\n Identify potential relationships and provide analysis in this JSON format:\n {{\n \"relationships\": [\n {{\n \"target_file_path\": \"path/in/target/structure\",\n \"relationship_type\": \"direct_match|partial_match|reference|utility\",\n \"confidence_score\": 0.0-1.0,\n \"helpful_aspects\": [\"specific\", \"aspects\", \"that\", \"could\", \"help\"],\n \"potential_contributions\": [\"how\", \"this\", \"could\", \"contribute\"],\n \"usage_suggestions\": \"detailed suggestion on how to use this file\"\n }}\n ]\n }}\n\n Consider the priority weights when determining relationship types. Higher weight types should be preferred when multiple types apply.\n Only include relationships with confidence > {self.min_confidence_score}. Focus on concrete, actionable connections.\n \"\"\"\n\n try:\n llm_response = await self._call_llm(relationship_prompt, max_tokens=1500)\n\n match = re.search(r\"\\{.*\\}\", llm_response, re.DOTALL)\n relationship_data = json.loads(match.group(0))\n\n relationships = []\n for rel_data in relationship_data.get(\"relationships\", []):\n confidence_score = float(rel_data.get(\"confidence_score\", 0.0))\n relationship_type = rel_data.get(\"relationship_type\", \"reference\")\n\n # Validate relationship type is in config\n if relationship_type not in self.relationship_types:\n if self.verbose_output:\n self.logger.warning(\n f\"Unknown relationship type '{relationship_type}', using 'reference'\"\n )\n relationship_type = \"reference\"\n\n # Apply configured minimum confidence filter\n if confidence_score > self.min_confidence_score:\n relationship = FileRelationship(\n repo_file_path=file_summary.file_path,\n target_file_path=rel_data.get(\"target_file_path\", \"\"),\n relationship_type=relationship_type,\n confidence_score=confidence_score,\n helpful_aspects=rel_data.get(\"helpful_aspects\", []),\n potential_contributions=rel_data.get(\n \"potential_contributions\", []\n ),\n usage_suggestions=rel_data.get(\"usage_suggestions\", \"\"),\n )\n relationships.append(relationship)\n\n return relationships\n\n except Exception as e:\n self.logger.error(\n f\"Error finding relationships for {file_summary.file_path}: {e}\"\n )\n return []\n\n async def _analyze_single_file_with_relationships(\n self, file_path: Path, index: int, total: int\n ) -> tuple:\n \"\"\"Analyze a single file and its relationships (for concurrent processing)\"\"\"\n if self.verbose_output:\n self.logger.info(f\"Analyzing file {index}/{total}: {file_path.name}\")\n\n # Get file summary\n file_summary = await self.analyze_file_content(file_path)\n\n # Find relationships\n relationships = await self.find_relationships(file_summary)\n\n return file_summary, relationships\n\n async def process_repository(self, repo_path: Path) -> RepoIndex:\n \"\"\"Process a single repository and create complete index with optional concurrent processing\"\"\"\n repo_name = repo_path.name\n self.logger.info(f\"Processing repository: {repo_name}\")\n\n # Step 1: Generate file tree\n self.logger.info(\"Generating file tree structure...\")\n file_tree = self.generate_file_tree(repo_path)\n\n # Step 2: Get all files\n all_files = self.get_all_repo_files(repo_path)\n self.logger.info(f\"Found {len(all_files)} files in {repo_name}\")\n\n # Step 3: LLM pre-filtering of relevant files\n if self.enable_pre_filtering:\n self.logger.info(\"Using LLM for file pre-filtering...\")\n selected_file_paths = await self.pre_filter_files(repo_path, file_tree)\n else:\n self.logger.info(\"Pre-filtering is disabled, will analyze all files\")\n selected_file_paths = []\n\n # Step 4: Filter file list based on filtering results\n if selected_file_paths:\n files_to_analyze = self.filter_files_by_paths(\n all_files, selected_file_paths, repo_path\n )\n self.logger.info(\n f\"After LLM filtering, will analyze {len(files_to_analyze)} relevant files (from {len(all_files)} total)\"\n )\n else:\n files_to_analyze = all_files\n self.logger.info(\"LLM filtering failed, will analyze all files\")\n\n # Step 5: Analyze filtered files (concurrent or sequential)\n if self.enable_concurrent_analysis and len(files_to_analyze) > 1:\n self.logger.info(\n f\"Using concurrent analysis with max {self.max_concurrent_files} parallel files\"\n )\n file_summaries, all_relationships = await self._process_files_concurrently(\n files_to_analyze\n )\n else:\n self.logger.info(\"Using sequential file analysis\")\n file_summaries, all_relationships = await self._process_files_sequentially(\n files_to_analyze\n )\n\n # Step 6: Create repository index\n repo_index = RepoIndex(\n repo_name=repo_name,\n total_files=len(all_files), # Record original file count\n file_summaries=file_summaries,\n relationships=all_relationships,\n analysis_metadata={\n \"analysis_date\": datetime.now().isoformat(),\n \"target_structure_analyzed\": self.target_structure[:200] + \"...\",\n \"total_relationships_found\": len(all_relationships),\n \"high_confidence_relationships\": len(\n [\n r\n for r in all_relationships\n if r.confidence_score > self.high_confidence_threshold\n ]\n ),\n \"analyzer_version\": \"1.4.0\", # Updated version to reflect augmented LLM support\n \"pre_filtering_enabled\": self.enable_pre_filtering,\n \"files_before_filtering\": len(all_files),\n \"files_after_filtering\": len(files_to_analyze),\n \"filtering_efficiency\": round(\n (1 - len(files_to_analyze) / len(all_files)) * 100, 2\n )\n if all_files\n else 0,\n \"config_file_used\": self.indexer_config_path,\n \"min_confidence_score\": self.min_confidence_score,\n \"high_confidence_threshold\": self.high_confidence_threshold,\n \"concurrent_analysis_used\": self.enable_concurrent_analysis,\n \"content_caching_enabled\": self.enable_content_caching,\n \"cache_hits\": len(self.content_cache) if self.content_cache else 0,\n },\n )\n\n return repo_index\n\n async def _process_files_sequentially(self, files_to_analyze: list) -> tuple:\n \"\"\"Process files sequentially (original method)\"\"\"\n file_summaries = []\n all_relationships = []\n\n for i, file_path in enumerate(files_to_analyze, 1):\n (\n file_summary,\n relationships,\n ) = await self._analyze_single_file_with_relationships(\n file_path, i, len(files_to_analyze)\n )\n file_summaries.append(file_summary)\n all_relationships.extend(relationships)\n\n # Add configured delay to avoid overwhelming the LLM API\n await asyncio.sleep(self.request_delay)\n\n return file_summaries, all_relationships\n\n async def _process_files_concurrently(self, files_to_analyze: list) -> tuple:\n \"\"\"Process files concurrently with semaphore limiting\"\"\"\n file_summaries = []\n all_relationships = []\n\n # Create semaphore to limit concurrent tasks\n semaphore = asyncio.Semaphore(self.max_concurrent_files)\n tasks = []\n\n async def _process_with_semaphore(file_path: Path, index: int, total: int):\n async with semaphore:\n # Add a small delay to space out concurrent requests\n if index > 1:\n await asyncio.sleep(\n self.request_delay * 0.5\n ) # Reduced delay for concurrent processing\n return await self._analyze_single_file_with_relationships(\n file_path, index, total\n )\n\n try:\n # Create tasks for all files\n tasks = [\n _process_with_semaphore(file_path, i, len(files_to_analyze))\n for i, file_path in enumerate(files_to_analyze, 1)\n ]\n\n # Process tasks and collect results\n if self.verbose_output:\n self.logger.info(\n f\"Starting concurrent analysis of {len(tasks)} files...\"\n )\n\n try:\n results = await asyncio.gather(*tasks, return_exceptions=True)\n\n for i, result in enumerate(results):\n if isinstance(result, Exception):\n self.logger.error(\n f\"Failed to analyze file {files_to_analyze[i]}: {result}\"\n )\n # Create error summary\n error_summary = FileSummary(\n file_path=str(\n files_to_analyze[i].relative_to(self.code_base_path)\n ),\n file_type=\"error\",\n main_functions=[],\n key_concepts=[],\n dependencies=[],\n summary=f\"Concurrent analysis failed: {str(result)}\",\n lines_of_code=0,\n last_modified=\"\",\n )\n file_summaries.append(error_summary)\n else:\n file_summary, relationships = result\n file_summaries.append(file_summary)\n all_relationships.extend(relationships)\n\n except Exception as e:\n self.logger.error(f\"Concurrent processing failed: {e}\")\n # Cancel any remaining tasks\n for task in tasks:\n if not task.done() and not task.cancelled():\n task.cancel()\n\n # Wait for cancelled tasks to complete\n try:\n await asyncio.sleep(0.1) # Brief wait for cancellation\n except Exception:\n pass\n\n # Fallback to sequential processing\n self.logger.info(\"Falling back to sequential processing...\")\n return await self._process_files_sequentially(files_to_analyze)\n\n if self.verbose_output:\n self.logger.info(\n f\"Concurrent analysis completed: {len(file_summaries)} files processed\"\n )\n\n return file_summaries, all_relationships\n\n except Exception as e:\n # Ensure all tasks are cancelled in case of unexpected errors\n if tasks:\n for task in tasks:\n if not task.done() and not task.cancelled():\n task.cancel()\n\n # Wait briefly for cancellation to complete\n try:\n await asyncio.sleep(0.1)\n except Exception:\n pass\n\n self.logger.error(f\"Critical error in concurrent processing: {e}\")\n # Fallback to sequential processing\n self.logger.info(\n \"Falling back to sequential processing due to critical error...\"\n )\n return await self._process_files_sequentially(files_to_analyze)\n\n finally:\n # Final cleanup: ensure all tasks are properly finished\n if tasks:\n for task in tasks:\n if not task.done() and not task.cancelled():\n task.cancel()\n\n # Clear task references to help with garbage collection\n tasks.clear()\n\n # Force garbage collection to help clean up semaphore and related resources\n import gc\n\n gc.collect()\n\n async def build_all_indexes(self) -> Dict[str, str]:\n \"\"\"Build indexes for all repositories in code_base\"\"\"\n if not self.code_base_path.exists():\n raise FileNotFoundError(\n f\"Code base path does not exist: {self.code_base_path}\"\n )\n\n # Get all repository directories\n repo_dirs = [\n d\n for d in self.code_base_path.iterdir()\n if d.is_dir() and not d.name.startswith(\".\")\n ]\n\n if not repo_dirs:\n raise ValueError(f\"No repositories found in {self.code_base_path}\")\n\n self.logger.info(f\"Found {len(repo_dirs)} repositories to process\")\n\n # Process each repository\n output_files = {}\n statistics_data = []\n\n for repo_dir in repo_dirs:\n try:\n # Process repository\n repo_index = await self.process_repository(repo_dir)\n\n # Generate output filename using configured pattern\n output_filename = self.index_filename_pattern.format(\n repo_name=repo_index.repo_name\n )\n output_file = self.output_dir / output_filename\n\n # Get output configuration\n output_config = self.indexer_config.get(\"output\", {})\n json_indent = output_config.get(\"json_indent\", 2)\n ensure_ascii = not output_config.get(\"ensure_ascii\", False)\n\n # Save to JSON file\n with open(output_file, \"w\", encoding=\"utf-8\") as f:\n if self.include_metadata:\n json.dump(\n asdict(repo_index),\n f,\n indent=json_indent,\n ensure_ascii=ensure_ascii,\n )\n else:\n # Save without metadata if disabled\n index_data = asdict(repo_index)\n index_data.pop(\"analysis_metadata\", None)\n json.dump(\n index_data, f, indent=json_indent, ensure_ascii=ensure_ascii\n )\n\n output_files[repo_index.repo_name] = str(output_file)\n self.logger.info(\n f\"Saved index for {repo_index.repo_name} to {output_file}\"\n )\n\n # Collect statistics for report\n if self.generate_statistics:\n stats = self._extract_repository_statistics(repo_index)\n statistics_data.append(stats)\n\n except Exception as e:\n self.logger.error(f\"Failed to process repository {repo_dir.name}: {e}\")\n continue\n\n # Generate additional reports if configured\n if self.generate_summary:\n summary_path = self.generate_summary_report(output_files)\n self.logger.info(f\"Generated summary report: {summary_path}\")\n\n if self.generate_statistics:\n stats_path = self.generate_statistics_report(statistics_data)\n self.logger.info(f\"Generated statistics report: {stats_path}\")\n\n return output_files\n\n def _extract_repository_statistics(self, repo_index: RepoIndex) -> Dict[str, Any]:\n \"\"\"Extract statistical information from a repository index\"\"\"\n metadata = repo_index.analysis_metadata\n\n # Count relationship types\n relationship_type_counts = {}\n for rel in repo_index.relationships:\n rel_type = rel.relationship_type\n relationship_type_counts[rel_type] = (\n relationship_type_counts.get(rel_type, 0) + 1\n )\n\n # Count file types\n file_type_counts = {}\n for file_summary in repo_index.file_summaries:\n file_type = file_summary.file_type\n file_type_counts[file_type] = file_type_counts.get(file_type, 0) + 1\n\n # Calculate statistics\n total_lines = sum(fs.lines_of_code for fs in repo_index.file_summaries)\n avg_lines = (\n total_lines / len(repo_index.file_summaries)\n if repo_index.file_summaries\n else 0\n )\n\n avg_confidence = (\n sum(r.confidence_score for r in repo_index.relationships)\n / len(repo_index.relationships)\n if repo_index.relationships\n else 0\n )\n\n return {\n \"repo_name\": repo_index.repo_name,\n \"total_files\": repo_index.total_files,\n \"analyzed_files\": len(repo_index.file_summaries),\n \"total_relationships\": len(repo_index.relationships),\n \"high_confidence_relationships\": metadata.get(\n \"high_confidence_relationships\", 0\n ),\n \"relationship_type_counts\": relationship_type_counts,\n \"file_type_counts\": file_type_counts,\n \"total_lines_of_code\": total_lines,\n \"average_lines_per_file\": round(avg_lines, 2),\n \"average_confidence_score\": round(avg_confidence, 3),\n \"filtering_efficiency\": metadata.get(\"filtering_efficiency\", 0),\n \"concurrent_analysis_used\": metadata.get(\"concurrent_analysis_used\", False),\n \"cache_hits\": metadata.get(\"cache_hits\", 0),\n \"analysis_date\": metadata.get(\"analysis_date\", \"unknown\"),\n }\n\n def generate_statistics_report(self, statistics_data: List[Dict[str, Any]]) -> str:\n \"\"\"Generate a detailed statistics report\"\"\"\n stats_path = self.output_dir / self.stats_filename\n\n # Calculate aggregate statistics\n total_repos = len(statistics_data)\n total_files_analyzed = sum(stat[\"analyzed_files\"] for stat in statistics_data)\n total_relationships = sum(\n stat[\"total_relationships\"] for stat in statistics_data\n )\n total_lines = sum(stat[\"total_lines_of_code\"] for stat in statistics_data)\n\n # Aggregate relationship types\n aggregated_rel_types = {}\n for stat in statistics_data:\n for rel_type, count in stat[\"relationship_type_counts\"].items():\n aggregated_rel_types[rel_type] = (\n aggregated_rel_types.get(rel_type, 0) + count\n )\n\n # Aggregate file types\n aggregated_file_types = {}\n for stat in statistics_data:\n for file_type, count in stat[\"file_type_counts\"].items():\n aggregated_file_types[file_type] = (\n aggregated_file_types.get(file_type, 0) + count\n )\n\n # Calculate averages\n avg_files_per_repo = total_files_analyzed / total_repos if total_repos else 0\n avg_relationships_per_repo = (\n total_relationships / total_repos if total_repos else 0\n )\n avg_lines_per_repo = total_lines / total_repos if total_repos else 0\n\n # Build statistics report\n statistics_report = {\n \"report_generation_time\": datetime.now().isoformat(),\n \"analyzer_version\": \"1.4.0\",\n \"configuration_used\": {\n \"config_file\": self.indexer_config_path,\n \"concurrent_analysis_enabled\": self.enable_concurrent_analysis,\n \"content_caching_enabled\": self.enable_content_caching,\n \"pre_filtering_enabled\": self.enable_pre_filtering,\n \"min_confidence_score\": self.min_confidence_score,\n \"high_confidence_threshold\": self.high_confidence_threshold,\n },\n \"aggregate_statistics\": {\n \"total_repositories_processed\": total_repos,\n \"total_files_analyzed\": total_files_analyzed,\n \"total_relationships_found\": total_relationships,\n \"total_lines_of_code\": total_lines,\n \"average_files_per_repository\": round(avg_files_per_repo, 2),\n \"average_relationships_per_repository\": round(\n avg_relationships_per_repo, 2\n ),\n \"average_lines_per_repository\": round(avg_lines_per_repo, 2),\n },\n \"relationship_type_distribution\": aggregated_rel_types,\n \"file_type_distribution\": aggregated_file_types,\n \"repository_details\": statistics_data,\n \"performance_metrics\": {\n \"concurrent_processing_repos\": sum(\n 1\n for s in statistics_data\n if s.get(\"concurrent_analysis_used\", False)\n ),\n \"cache_efficiency\": {\n \"total_cache_hits\": sum(\n s.get(\"cache_hits\", 0) for s in statistics_data\n ),\n \"repositories_with_caching\": sum(\n 1 for s in statistics_data if s.get(\"cache_hits\", 0) > 0\n ),\n },\n \"filtering_efficiency\": {\n \"average_filtering_efficiency\": round(\n sum(s.get(\"filtering_efficiency\", 0) for s in statistics_data)\n / total_repos,\n 2,\n )\n if total_repos\n else 0,\n \"max_filtering_efficiency\": max(\n (s.get(\"filtering_efficiency\", 0) for s in statistics_data),\n default=0,\n ),\n \"min_filtering_efficiency\": min(\n (s.get(\"filtering_efficiency\", 0) for s in statistics_data),\n default=0,\n ),\n },\n },\n }\n\n # Get output configuration\n output_config = self.indexer_config.get(\"output\", {})\n json_indent = output_config.get(\"json_indent\", 2)\n ensure_ascii = not output_config.get(\"ensure_ascii\", False)\n\n with open(stats_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(\n statistics_report, f, indent=json_indent, ensure_ascii=ensure_ascii\n )\n\n return str(stats_path)\n\n def generate_summary_report(self, output_files: Dict[str, str]) -> str:\n \"\"\"Generate a summary report of all indexes created\"\"\"\n report_path = self.output_dir / \"indexing_summary.json\"\n\n # Get output configuration from config file\n output_config = self.indexer_config.get(\"output\", {})\n json_indent = output_config.get(\"json_indent\", 2)\n ensure_ascii = not output_config.get(\"ensure_ascii\", False)\n\n summary_data = {\n \"indexing_completion_time\": datetime.now().isoformat(),\n \"total_repositories_processed\": len(output_files),\n \"output_files\": output_files,\n \"target_structure\": self.target_structure,\n \"code_base_path\": str(self.code_base_path),\n \"configuration\": {\n \"config_file_used\": self.indexer_config_path,\n \"api_config_file\": self.config_path,\n \"pre_filtering_enabled\": self.enable_pre_filtering,\n \"min_confidence_score\": self.min_confidence_score,\n \"high_confidence_threshold\": self.high_confidence_threshold,\n \"max_file_size\": self.max_file_size,\n \"max_content_length\": self.max_content_length,\n \"request_delay\": self.request_delay,\n \"supported_extensions_count\": len(self.supported_extensions),\n \"skip_directories_count\": len(self.skip_directories),\n },\n }\n\n with open(report_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(summary_data, f, indent=json_indent, ensure_ascii=ensure_ascii)\n\n return str(report_path)\n\n\nasync def main():\n \"\"\"Main function to run the code indexer with full configuration support\"\"\"\n\n # Configuration - can be overridden by config file\n config_file = \"DeepCode/tools/indexer_config.yaml\"\n api_config_file = \"DeepCode/mcp_agent.secrets.yaml\"\n\n # You can override these parameters or let them be read from config\n code_base_path = \"DeepCode/deepcode_lab/papers/1/code_base/\" # Will use config file value if None\n output_dir = (\n \"DeepCode/deepcode_lab/papers/1/indexes/\" # Will use config file value if None\n )\n\n # Target structure - this should be customized for your specific project\n target_structure = \"\"\"\n project/\n ├── src/\n │ ├── core/\n │ │ ├── gcn.py # GCN encoder\n │ │ ├── diffusion.py # forward/reverse processes\n │ │ ├── denoiser.py # denoising MLP\n │ │ └── fusion.py # fusion combiner\n │ ├── models/ # model wrapper classes\n │ │ └── recdiff.py\n │ ├── utils/\n │ │ ├── data.py # loading & preprocessing\n │ │ ├── predictor.py # scoring functions\n │ │ ├── loss.py # loss functions\n │ │ ├── metrics.py # NDCG, Recall etc.\n │ │ └── sched.py # beta/alpha schedule utils\n │ └── configs/\n │ └── default.yaml # hyperparameters, paths\n ├── tests/\n │ ├── test_gcn.py\n │ ├── test_diffusion.py\n │ ├── test_denoiser.py\n │ ├── test_loss.py\n │ └── test_pipeline.py\n ├── docs/\n │ ├── architecture.md\n │ ├── api_reference.md\n │ └── README.md\n ├── experiments/\n │ ├── run_experiment.py\n │ └── notebooks/\n │ └── analysis.ipynb\n ├── requirements.txt\n └── setup.py\n \"\"\"\n\n print(\"🚀 Starting Code Indexer with Enhanced Configuration Support\")\n print(f\"📋 Configuration file: {config_file}\")\n print(f\"🔑 API configuration file: {api_config_file}\")\n\n # Create indexer with full configuration support\n try:\n indexer = CodeIndexer(\n code_base_path=code_base_path, # None = read from config\n target_structure=target_structure, # Required - project specific\n output_dir=output_dir, # None = read from config\n config_path=api_config_file, # API configuration file\n indexer_config_path=config_file, # Configuration file\n enable_pre_filtering=True, # Can be overridden in config\n )\n\n # Display configuration information\n print(f\"📁 Code base path: {indexer.code_base_path}\")\n print(f\"📂 Output directory: {indexer.output_dir}\")\n print(\n f\"🤖 Default models: Anthropic={indexer.default_models['anthropic']}, OpenAI={indexer.default_models['openai']}\"\n )\n print(f\"🔧 Preferred LLM: {get_preferred_llm_class(api_config_file).__name__}\")\n print(\n f\"⚡ Concurrent analysis: {'enabled' if indexer.enable_concurrent_analysis else 'disabled'}\"\n )\n print(\n f\"🗄️ Content caching: {'enabled' if indexer.enable_content_caching else 'disabled'}\"\n )\n print(\n f\"🔍 Pre-filtering: {'enabled' if indexer.enable_pre_filtering else 'disabled'}\"\n )\n print(f\"🐛 Debug mode: {'enabled' if indexer.verbose_output else 'disabled'}\")\n print(\n f\"🎭 Mock responses: {'enabled' if indexer.mock_llm_responses else 'disabled'}\"\n )\n\n # Validate configuration\n if not indexer.code_base_path.exists():\n raise FileNotFoundError(\n f\"Code base path does not exist: {indexer.code_base_path}\"\n )\n\n if not target_structure:\n raise ValueError(\"Target structure is required for analysis\")\n\n print(\"\\n🔧 Starting indexing process...\")\n\n # Build all indexes\n output_files = await indexer.build_all_indexes()\n\n # Display results\n print(\"\\n✅ Indexing completed successfully!\")\n print(f\"📊 Processed {len(output_files)} repositories\")\n print(\"📁 Output files:\")\n for repo_name, file_path in output_files.items():\n print(f\" - {repo_name}: {file_path}\")\n\n # Display additional reports generated\n if indexer.generate_summary:\n summary_file = indexer.output_dir / indexer.summary_filename\n if summary_file.exists():\n print(f\"📋 Summary report: {summary_file}\")\n\n if indexer.generate_statistics:\n stats_file = indexer.output_dir / indexer.stats_filename\n if stats_file.exists():\n print(f\"📈 Statistics report: {stats_file}\")\n\n # Performance information\n if indexer.enable_content_caching and indexer.content_cache:\n print(f\"🗄️ Cache performance: {len(indexer.content_cache)} items cached\")\n\n print(\"\\n🎉 Code indexing process completed successfully!\")\n\n except FileNotFoundError as e:\n print(f\"❌ File not found error: {e}\")\n print(\"💡 Please check your configuration file paths\")\n except ValueError as e:\n print(f\"❌ Configuration error: {e}\")\n print(\"💡 Please check your configuration file settings\")\n except Exception as e:\n print(f\"❌ Indexing failed: {e}\")\n print(\"💡 Check the logs for more details\")\n\n # Print debug information if available\n try:\n indexer\n if indexer.verbose_output:\n import traceback\n\n print(\"\\n🐛 Debug information:\")\n traceback.print_exc()\n except NameError:\n pass\n\n\ndef print_usage_example():\n \"\"\"Print usage examples for different scenarios\"\"\"\n print(\"\"\"\n 📖 Code Indexer Usage Examples:\n\n 1. Basic usage with config file:\n - Update paths in indexer_config.yaml\n - Run: python code_indexer.py\n\n 2. Enable debugging:\n - Set debug.verbose_output: true in config\n - Set debug.save_raw_responses: true to save LLM responses\n\n 3. Enable concurrent processing:\n - Set performance.enable_concurrent_analysis: true\n - Adjust performance.max_concurrent_files as needed\n\n 4. Enable caching:\n - Set performance.enable_content_caching: true\n - Adjust performance.max_cache_size as needed\n\n 5. Mock mode for testing:\n - Set debug.mock_llm_responses: true\n - No API calls will be made\n\n 6. Custom output:\n - Modify output.index_filename_pattern\n - Set output.generate_statistics: true for detailed reports\n\n 📋 Configuration file location: tools/indexer_config.yaml\n \"\"\")\n\n\nif __name__ == \"__main__\":\n import sys\n\n if len(sys.argv) > 1 and sys.argv[1] in [\"--help\", \"-h\", \"help\"]:\n print_usage_example()\n else:\n asyncio.run(main())\n"
},
{
"path": "tools/code_reference_indexer.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nCode Reference Indexer MCP Tool - Unified Version\n\nSpecialized MCP tool for searching relevant index content in indexes folder\nand formatting it for LLM code implementation reference.\n\nCore Features:\n1. **UNIFIED TOOL**: Combined search_code_references that handles directory setup, loading, and searching in one call\n2. Match relevant reference code based on target file path and functionality requirements\n3. Format output of relevant code examples, functions and concepts\n4. Provide structured reference information for LLM use\n\nKey Improvement:\n- Single tool call that handles all steps internally\n- Agent only needs to provide indexes_path and target_file\n- No dependency on calling order or global state management\n\"\"\"\n\nimport json\nfrom pathlib import Path\nfrom typing import Dict, List, Tuple\nfrom dataclasses import dataclass\nimport logging\n\n# Import MCP modules\nfrom mcp.server.fastmcp import FastMCP\n\n# Setup logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\n# Create FastMCP server instance\nmcp = FastMCP(\"code-reference-indexer\")\n\n\n@dataclass\nclass CodeReference:\n \"\"\"Code reference information structure\"\"\"\n\n file_path: str\n file_type: str\n main_functions: List[str]\n key_concepts: List[str]\n dependencies: List[str]\n summary: str\n lines_of_code: int\n repo_name: str\n confidence_score: float = 0.0\n\n\n@dataclass\nclass RelationshipInfo:\n \"\"\"Relationship information structure\"\"\"\n\n repo_file_path: str\n target_file_path: str\n relationship_type: str\n confidence_score: float\n helpful_aspects: List[str]\n potential_contributions: List[str]\n usage_suggestions: str\n\n\ndef load_index_files_from_directory(indexes_directory: str) -> Dict[str, Dict]:\n \"\"\"Load all index files from specified directory\"\"\"\n indexes_path = Path(indexes_directory).resolve()\n\n if not indexes_path.exists():\n logger.warning(f\"Indexes directory does not exist: {indexes_path}\")\n return {}\n\n index_cache = {}\n\n for index_file in indexes_path.glob(\"*.json\"):\n try:\n with open(index_file, \"r\", encoding=\"utf-8\") as f:\n index_data = json.load(f)\n index_cache[index_file.stem] = index_data\n logger.info(f\"Loaded index file: {index_file.name}\")\n except Exception as e:\n logger.error(f\"Failed to load index file {index_file.name}: {e}\")\n\n logger.info(f\"Loaded {len(index_cache)} index files from {indexes_path}\")\n return index_cache\n\n\ndef extract_code_references(index_data: Dict) -> List[CodeReference]:\n \"\"\"Extract code reference information from index data\"\"\"\n references = []\n\n repo_name = index_data.get(\"repo_name\", \"Unknown\")\n file_summaries = index_data.get(\"file_summaries\", [])\n\n for file_summary in file_summaries:\n reference = CodeReference(\n file_path=file_summary.get(\"file_path\", \"\"),\n file_type=file_summary.get(\"file_type\", \"\"),\n main_functions=file_summary.get(\"main_functions\", []),\n key_concepts=file_summary.get(\"key_concepts\", []),\n dependencies=file_summary.get(\"dependencies\", []),\n summary=file_summary.get(\"summary\", \"\"),\n lines_of_code=file_summary.get(\"lines_of_code\", 0),\n repo_name=repo_name,\n )\n references.append(reference)\n\n return references\n\n\ndef extract_relationships(index_data: Dict) -> List[RelationshipInfo]:\n \"\"\"Extract relationship information from index data\"\"\"\n relationships = []\n\n relationship_list = index_data.get(\"relationships\", [])\n\n for rel in relationship_list:\n relationship = RelationshipInfo(\n repo_file_path=rel.get(\"repo_file_path\", \"\"),\n target_file_path=rel.get(\"target_file_path\", \"\"),\n relationship_type=rel.get(\"relationship_type\", \"\"),\n confidence_score=rel.get(\"confidence_score\", 0.0),\n helpful_aspects=rel.get(\"helpful_aspects\", []),\n potential_contributions=rel.get(\"potential_contributions\", []),\n usage_suggestions=rel.get(\"usage_suggestions\", \"\"),\n )\n relationships.append(relationship)\n\n return relationships\n\n\ndef calculate_relevance_score(\n target_file: str, reference: CodeReference, keywords: List[str] = None\n) -> float:\n \"\"\"Calculate relevance score between reference code and target file\"\"\"\n score = 0.0\n\n # File name similarity\n target_name = Path(target_file).stem.lower()\n ref_name = Path(reference.file_path).stem.lower()\n\n if target_name in ref_name or ref_name in target_name:\n score += 0.3\n\n # File type matching\n target_extension = Path(target_file).suffix\n ref_extension = Path(reference.file_path).suffix\n\n if target_extension == ref_extension:\n score += 0.2\n\n # Keyword matching\n if keywords:\n keyword_matches = 0\n total_searchable_text = (\n \" \".join(reference.key_concepts)\n + \" \"\n + \" \".join(reference.main_functions)\n + \" \"\n + reference.summary\n + \" \"\n + reference.file_type\n ).lower()\n\n for keyword in keywords:\n if keyword.lower() in total_searchable_text:\n keyword_matches += 1\n\n if keywords:\n score += (keyword_matches / len(keywords)) * 0.5\n\n return min(score, 1.0)\n\n\ndef find_relevant_references_in_cache(\n target_file: str,\n index_cache: Dict[str, Dict],\n keywords: List[str] = None,\n max_results: int = 10,\n) -> List[Tuple[CodeReference, float]]:\n \"\"\"Find reference code relevant to target file from provided cache\"\"\"\n all_references = []\n\n # Collect reference information from all index files\n for repo_name, index_data in index_cache.items():\n references = extract_code_references(index_data)\n for ref in references:\n relevance_score = calculate_relevance_score(target_file, ref, keywords)\n if relevance_score > 0.1: # Only keep results with certain relevance\n all_references.append((ref, relevance_score))\n\n # Sort by relevance score\n all_references.sort(key=lambda x: x[1], reverse=True)\n\n return all_references[:max_results]\n\n\ndef find_direct_relationships_in_cache(\n target_file: str, index_cache: Dict[str, Dict]\n) -> List[RelationshipInfo]:\n \"\"\"Find direct relationships with target file from provided cache\"\"\"\n relationships = []\n\n # Normalize target file path (remove common prefixes if exists)\n common_prefixes = [\"src/\", \"core/\", \"lib/\", \"main/\", \"./\"]\n normalized_target = target_file.strip(\"/\")\n for prefix in common_prefixes:\n if normalized_target.startswith(prefix):\n normalized_target = normalized_target[len(prefix) :]\n break\n\n # Collect relationship information from all index files\n for repo_name, index_data in index_cache.items():\n repo_relationships = extract_relationships(index_data)\n for rel in repo_relationships:\n # Normalize target file path in relationship\n normalized_rel_target = rel.target_file_path.strip(\"/\")\n for prefix in common_prefixes:\n if normalized_rel_target.startswith(prefix):\n normalized_rel_target = normalized_rel_target[len(prefix) :]\n break\n\n # Check target file path matching (support multiple matching methods)\n if (\n normalized_target == normalized_rel_target\n or normalized_target in normalized_rel_target\n or normalized_rel_target in normalized_target\n or target_file in rel.target_file_path\n or rel.target_file_path in target_file\n ):\n relationships.append(rel)\n\n # Sort by confidence score\n relationships.sort(key=lambda x: x.confidence_score, reverse=True)\n\n return relationships\n\n\ndef format_reference_output(\n target_file: str,\n relevant_refs: List[Tuple[CodeReference, float]],\n relationships: List[RelationshipInfo],\n) -> str:\n \"\"\"Format reference information output\"\"\"\n output_lines = []\n\n output_lines.append(f\"# Code Reference Information - {target_file}\")\n output_lines.append(\"=\" * 80)\n output_lines.append(\"\")\n\n # Direct relationship information\n if relationships:\n output_lines.append(\"## 🎯 Direct Relationships\")\n output_lines.append(\"\")\n\n for i, rel in enumerate(relationships[:5], 1):\n output_lines.append(f\"### {i}. {rel.repo_file_path}\")\n output_lines.append(f\"**Relationship Type**: {rel.relationship_type}\")\n output_lines.append(f\"**Confidence Score**: {rel.confidence_score:.2f}\")\n output_lines.append(\n f\"**Helpful Aspects**: {', '.join(rel.helpful_aspects)}\"\n )\n output_lines.append(\n f\"**Potential Contributions**: {', '.join(rel.potential_contributions)}\"\n )\n output_lines.append(f\"**Usage Suggestions**: {rel.usage_suggestions}\")\n output_lines.append(\"\")\n\n # Relevant code references\n if relevant_refs:\n output_lines.append(\"## 📚 Relevant Code References\")\n output_lines.append(\"\")\n\n for i, (ref, score) in enumerate(relevant_refs[:8], 1):\n output_lines.append(f\"### {i}. {ref.file_path} (Relevance: {score:.2f})\")\n output_lines.append(f\"**Repository**: {ref.repo_name}\")\n output_lines.append(f\"**File Type**: {ref.file_type}\")\n output_lines.append(\n f\"**Main Functions**: {', '.join(ref.main_functions[:5])}\"\n )\n output_lines.append(f\"**Key Concepts**: {', '.join(ref.key_concepts[:8])}\")\n output_lines.append(f\"**Dependencies**: {', '.join(ref.dependencies[:6])}\")\n output_lines.append(f\"**Lines of Code**: {ref.lines_of_code}\")\n output_lines.append(f\"**Summary**: {ref.summary[:300]}...\")\n output_lines.append(\"\")\n\n # Implementation suggestions\n output_lines.append(\"## 💡 Implementation Suggestions\")\n output_lines.append(\"\")\n\n if relevant_refs:\n # Collect all function names and concepts\n all_functions = set()\n all_concepts = set()\n all_dependencies = set()\n\n for ref, _ in relevant_refs[:5]:\n all_functions.update(ref.main_functions)\n all_concepts.update(ref.key_concepts)\n all_dependencies.update(ref.dependencies)\n\n output_lines.append(\"**Reference Function Name Patterns**:\")\n for func in sorted(list(all_functions))[:10]:\n output_lines.append(f\"- {func}\")\n output_lines.append(\"\")\n\n output_lines.append(\"**Important Concepts and Patterns**:\")\n for concept in sorted(list(all_concepts))[:15]:\n output_lines.append(f\"- {concept}\")\n output_lines.append(\"\")\n\n output_lines.append(\"**Potential Dependencies Needed**:\")\n for dep in sorted(list(all_dependencies))[:10]:\n output_lines.append(f\"- {dep}\")\n output_lines.append(\"\")\n\n output_lines.append(\"## 🚀 Next Actions\")\n output_lines.append(\n \"1. Analyze design patterns and architectural styles from the above reference code\"\n )\n output_lines.append(\"2. Determine core functionalities and interfaces to implement\")\n output_lines.append(\"3. Choose appropriate dependency libraries and tools\")\n output_lines.append(\n \"4. Design implementation solution consistent with existing code style\"\n )\n output_lines.append(\"5. Start writing specific code implementation\")\n\n return \"\\n\".join(output_lines)\n\n\n# ==================== MCP Tool Definitions ====================\n\n\n@mcp.tool()\nasync def search_code_references(\n indexes_path: str, target_file: str, keywords: str = \"\", max_results: int = 10\n) -> str:\n \"\"\"\n **UNIFIED TOOL**: Search relevant reference code from index files for target file implementation.\n This tool combines directory setup, index loading, and searching in a single call.\n\n Args:\n indexes_path: Path to the indexes directory containing JSON index files\n target_file: Target file path (file to be implemented)\n keywords: Search keywords, comma-separated\n max_results: Maximum number of results to return\n\n Returns:\n Formatted reference code information JSON string\n \"\"\"\n try:\n # Step 1: Load index files from specified directory\n logger.info(f\"Loading index files from: {indexes_path}\")\n index_cache = load_index_files_from_directory(indexes_path)\n\n if not index_cache:\n result = {\n \"status\": \"error\",\n \"message\": f\"No index files found or failed to load from: {indexes_path}\",\n \"target_file\": target_file,\n \"indexes_path\": indexes_path,\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n # Step 2: Parse keywords\n keyword_list = (\n [kw.strip() for kw in keywords.split(\",\") if kw.strip()] if keywords else []\n )\n\n # Step 3: Find relevant reference code\n relevant_refs = find_relevant_references_in_cache(\n target_file, index_cache, keyword_list, max_results\n )\n\n # Step 4: Find direct relationships\n relationships = find_direct_relationships_in_cache(target_file, index_cache)\n\n # Step 5: Format output\n formatted_output = format_reference_output(\n target_file, relevant_refs, relationships\n )\n\n result = {\n \"status\": \"success\",\n \"target_file\": target_file,\n \"indexes_path\": indexes_path,\n \"keywords_used\": keyword_list,\n \"total_references_found\": len(relevant_refs),\n \"total_relationships_found\": len(relationships),\n \"formatted_content\": formatted_output,\n \"indexes_loaded\": list(index_cache.keys()),\n \"total_indexes_loaded\": len(index_cache),\n }\n\n logger.info(\n f\"Successfully found {len(relevant_refs)} references and {len(relationships)} relationships for {target_file}\"\n )\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n logger.error(f\"Error in search_code_references: {str(e)}\")\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to search reference code: {str(e)}\",\n \"target_file\": target_file,\n \"indexes_path\": indexes_path,\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\n@mcp.tool()\nasync def get_indexes_overview(indexes_path: str) -> str:\n \"\"\"\n Get overview of all available reference code index information from specified directory\n\n Args:\n indexes_path: Path to the indexes directory containing JSON index files\n\n Returns:\n Overview information of all available reference code JSON string\n \"\"\"\n try:\n # Load index files from specified directory\n index_cache = load_index_files_from_directory(indexes_path)\n\n if not index_cache:\n result = {\n \"status\": \"error\",\n \"message\": f\"No index files found in: {indexes_path}\",\n \"indexes_path\": indexes_path,\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n overview = {\"total_repos\": len(index_cache), \"repositories\": {}}\n\n for repo_name, index_data in index_cache.items():\n repo_info = {\n \"repo_name\": index_data.get(\"repo_name\", repo_name),\n \"total_files\": index_data.get(\"total_files\", 0),\n \"file_types\": [],\n \"main_concepts\": [],\n \"total_relationships\": len(index_data.get(\"relationships\", [])),\n }\n\n # Collect file types and concepts\n file_summaries = index_data.get(\"file_summaries\", [])\n file_types = set()\n concepts = set()\n\n for file_summary in file_summaries:\n file_types.add(file_summary.get(\"file_type\", \"Unknown\"))\n concepts.update(file_summary.get(\"key_concepts\", []))\n\n repo_info[\"file_types\"] = sorted(list(file_types))\n repo_info[\"main_concepts\"] = sorted(list(concepts))[\n :20\n ] # Limit concept count\n\n overview[\"repositories\"][repo_name] = repo_info\n\n result = {\n \"status\": \"success\",\n \"overview\": overview,\n \"indexes_directory\": str(Path(indexes_path).resolve()),\n \"total_indexes_loaded\": len(index_cache),\n }\n\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n except Exception as e:\n result = {\n \"status\": \"error\",\n \"message\": f\"Failed to get indexes overview: {str(e)}\",\n \"indexes_path\": indexes_path,\n }\n return json.dumps(result, ensure_ascii=False, indent=2)\n\n\ndef main():\n \"\"\"Main function\"\"\"\n logger.info(\"Starting unified Code Reference Indexer MCP server\")\n logger.info(\"Available tools:\")\n logger.info(\n \"1. search_code_references(indexes_path, target_file, keywords, max_results) - UNIFIED TOOL\"\n )\n logger.info(\n \"2. get_indexes_overview(indexes_path) - Get overview of available indexes\"\n )\n\n # Run MCP server\n mcp.run()\n\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "tools/command_executor.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nCommand Executor MCP Tool / 命令执行器 MCP 工具\n\n专门负责执行LLM生成的shell命令来创建文件树结构\nSpecialized in executing LLM-generated shell commands to create file tree structures\n\"\"\"\n\nimport subprocess\nfrom pathlib import Path\nfrom typing import List, Dict\nfrom mcp.server.models import InitializationOptions\nimport mcp.types as types\nfrom mcp.server import NotificationOptions, Server\nimport mcp.server.stdio\n\n# 创建MCP服务器实例 / Create MCP server instance\napp = Server(\"command-executor\")\n\n\n@app.list_tools()\nasync def handle_list_tools() -> list[types.Tool]:\n \"\"\"\n 列出可用工具 / List available tools\n \"\"\"\n return [\n types.Tool(\n name=\"execute_commands\",\n description=\"\"\"\n 执行shell命令列表来创建文件树结构\n Execute shell command list to create file tree structure\n\n Args:\n commands: 要执行的shell命令列表（每行一个命令）\n working_directory: 执行命令的工作目录\n\n Returns:\n 命令执行结果和详细报告\n \"\"\",\n inputSchema={\n \"type\": \"object\",\n \"properties\": {\n \"commands\": {\n \"type\": \"string\",\n \"title\": \"Commands\",\n \"description\": \"要执行的shell命令列表，每行一个命令\",\n },\n \"working_directory\": {\n \"type\": \"string\",\n \"title\": \"Working Directory\",\n \"description\": \"执行命令的工作目录\",\n },\n },\n \"required\": [\"commands\", \"working_directory\"],\n },\n ),\n types.Tool(\n name=\"execute_single_command\",\n description=\"\"\"\n 执行单个shell命令\n Execute single shell command\n\n Args:\n command: 要执行的单个命令\n working_directory: 执行命令的工作目录\n\n Returns:\n 命令执行结果\n \"\"\",\n inputSchema={\n \"type\": \"object\",\n \"properties\": {\n \"command\": {\n \"type\": \"string\",\n \"title\": \"Command\",\n \"description\": \"要执行的单个shell命令\",\n },\n \"working_directory\": {\n \"type\": \"string\",\n \"title\": \"Working Directory\",\n \"description\": \"执行命令的工作目录\",\n },\n },\n \"required\": [\"command\", \"working_directory\"],\n },\n ),\n ]\n\n\n@app.call_tool()\nasync def handle_call_tool(name: str, arguments: dict) -> list[types.TextContent]:\n \"\"\"\n 处理工具调用 / Handle tool calls\n \"\"\"\n try:\n if name == \"execute_commands\":\n return await execute_command_batch(\n arguments.get(\"commands\", \"\"), arguments.get(\"working_directory\", \".\")\n )\n elif name == \"execute_single_command\":\n return await execute_single_command(\n arguments.get(\"command\", \"\"), arguments.get(\"working_directory\", \".\")\n )\n else:\n raise ValueError(f\"未知工具 / Unknown tool: {name}\")\n\n except Exception as e:\n return [\n types.TextContent(\n type=\"text\",\n text=f\"工具执行错误 / Error executing tool {name}: {str(e)}\",\n )\n ]\n\n\nasync def execute_command_batch(\n commands: str, working_directory: str\n) -> list[types.TextContent]:\n \"\"\"\n 执行多个shell命令 / Execute multiple shell commands\n\n Args:\n commands: 命令列表，每行一个命令 / Command list, one command per line\n working_directory: 工作目录 / Working directory\n\n Returns:\n 执行结果 / Execution results\n \"\"\"\n try:\n # 确保工作目录存在 / Ensure working directory exists\n Path(working_directory).mkdir(parents=True, exist_ok=True)\n\n # 分割命令行 / Split command lines\n command_lines = [\n cmd.strip() for cmd in commands.strip().split(\"\\n\") if cmd.strip()\n ]\n\n if not command_lines:\n return [\n types.TextContent(\n type=\"text\", text=\"没有提供有效命令 / No valid commands provided\"\n )\n ]\n\n results = []\n stats = {\"successful\": 0, \"failed\": 0, \"timeout\": 0}\n\n for i, command in enumerate(command_lines, 1):\n try:\n # 执行命令 / Execute command\n result = subprocess.run(\n command,\n shell=True,\n cwd=working_directory,\n capture_output=True,\n text=True,\n timeout=30, # 30秒超时\n )\n\n if result.returncode == 0:\n results.append(f\"✅ Command {i}: {command}\")\n if result.stdout.strip():\n results.append(f\" 输出 / Output: {result.stdout.strip()}\")\n stats[\"successful\"] += 1\n else:\n results.append(f\"❌ Command {i}: {command}\")\n if result.stderr.strip():\n results.append(f\" 错误 / Error: {result.stderr.strip()}\")\n stats[\"failed\"] += 1\n\n except subprocess.TimeoutExpired:\n results.append(f\"⏱️ Command {i} 超时 / timeout: {command}\")\n stats[\"timeout\"] += 1\n except Exception as e:\n results.append(f\"💥 Command {i} 异常 / exception: {command} - {str(e)}\")\n stats[\"failed\"] += 1\n\n # 生成执行报告 / Generate execution report\n summary = generate_execution_summary(working_directory, command_lines, stats)\n final_result = summary + \"\\n\" + \"\\n\".join(results)\n\n return [types.TextContent(type=\"text\", text=final_result)]\n\n except Exception as e:\n return [\n types.TextContent(\n type=\"text\",\n text=f\"批量命令执行失败 / Failed to execute command batch: {str(e)}\",\n )\n ]\n\n\nasync def execute_single_command(\n command: str, working_directory: str\n) -> list[types.TextContent]:\n \"\"\"\n 执行单个shell命令 / Execute single shell command\n\n Args:\n command: 要执行的命令 / Command to execute\n working_directory: 工作目录 / Working directory\n\n Returns:\n 执行结果 / Execution result\n \"\"\"\n try:\n # 确保工作目录存在 / Ensure working directory exists\n Path(working_directory).mkdir(parents=True, exist_ok=True)\n\n # 执行命令 / Execute command\n result = subprocess.run(\n command,\n shell=True,\n cwd=working_directory,\n capture_output=True,\n text=True,\n timeout=30,\n )\n\n # 格式化输出 / Format output\n output = format_single_command_result(command, working_directory, result)\n\n return [types.TextContent(type=\"text\", text=output)]\n\n except subprocess.TimeoutExpired:\n return [\n types.TextContent(\n type=\"text\", text=f\"⏱️ 命令超时 / Command timeout: {command}\"\n )\n ]\n except Exception as e:\n return [\n types.TextContent(\n type=\"text\", text=f\"💥 命令执行错误 / Command execution error: {str(e)}\"\n )\n ]\n\n\ndef generate_execution_summary(\n working_directory: str, command_lines: List[str], stats: Dict[str, int]\n) -> str:\n \"\"\"\n 生成执行总结 / Generate execution summary\n\n Args:\n working_directory: 工作目录 / Working directory\n command_lines: 命令列表 / Command list\n stats: 统计信息 / Statistics\n\n Returns:\n 格式化的总结 / Formatted summary\n \"\"\"\n return f\"\"\"\n命令执行总结 / Command Execution Summary:\n{'='*50}\n工作目录 / Working Directory: {working_directory}\n总命令数 / Total Commands: {len(command_lines)}\n成功 / Successful: {stats['successful']}\n失败 / Failed: {stats['failed']}\n超时 / Timeout: {stats['timeout']}\n\n详细结果 / Detailed Results:\n{'-'*50}\"\"\"\n\n\ndef format_single_command_result(\n command: str, working_directory: str, result: subprocess.CompletedProcess\n) -> str:\n \"\"\"\n 格式化单命令执行结果 / Format single command execution result\n\n Args:\n command: 执行的命令 / Executed command\n working_directory: 工作目录 / Working directory\n result: 执行结果 / Execution result\n\n Returns:\n 格式化的结果 / Formatted result\n \"\"\"\n output = f\"\"\"\n单命令执行 / Single Command Execution:\n{'='*40}\n工作目录 / Working Directory: {working_directory}\n命令 / Command: {command}\n返回码 / Return Code: {result.returncode}\n\n\"\"\"\n\n if result.returncode == 0:\n output += \"✅ 状态 / Status: SUCCESS / 成功\\n\"\n if result.stdout.strip():\n output += f\"输出 / Output:\\n{result.stdout.strip()}\\n\"\n else:\n output += \"❌ 状态 / Status: FAILED / 失败\\n\"\n if result.stderr.strip():\n output += f\"错误 / Error:\\n{result.stderr.strip()}\\n\"\n\n return output\n\n\nasync def main():\n \"\"\"\n 运行MCP服务器 / Run MCP server\n \"\"\"\n # 通过stdio运行服务器 / Run server via stdio\n async with mcp.server.stdio.stdio_server() as (read_stream, write_stream):\n await app.run(\n read_stream,\n write_stream,\n InitializationOptions(\n server_name=\"command-executor\",\n server_version=\"1.0.0\",\n capabilities=app.get_capabilities(\n notification_options=NotificationOptions(),\n experimental_capabilities={},\n ),\n ),\n )\n\n\nif __name__ == \"__main__\":\n import asyncio\n\n asyncio.run(main())\n"
},
{
"path": "tools/document_segmentation_server.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nDocument Segmentation MCP Server\n\nThis MCP server provides intelligent document segmentation and retrieval functions for handling\nlarge research papers and technical documents that exceed LLM token limits.\n\n==== CORE FUNCTIONALITY ====\n1. Analyze document structure and type using semantic content analysis\n2. Create intelligent segments based on content semantics, not just structure\n3. Provide query-aware segment retrieval with relevance scoring\n4. Support both structured (papers with headers) and unstructured documents\n5. Configurable segmentation strategies based on document complexity\n\n==== MCP TOOLS PROVIDED ====\n\n📄 analyze_and_segment_document(paper_dir: str, force_refresh: bool = False)\n Purpose: Analyzes document structure and creates intelligent segments\n - Detects document type (research paper, technical doc, algorithm-focused, etc.)\n - Selects optimal segmentation strategy based on content analysis\n - Creates semantic segments preserving algorithm and concept integrity\n - Stores segmentation index for efficient retrieval\n - Returns: JSON with segmentation status, strategy used, and segment count\n\n📖 read_document_segments(paper_dir: str, query_type: str, keywords: List[str] = None,\n max_segments: int = 3, max_total_chars: int = None)\n Purpose: Intelligently retrieves relevant document segments based on query context\n - query_type: \"concept_analysis\", \"algorithm_extraction\", or \"code_planning\"\n - Uses semantic relevance scoring to rank segments\n - Applies query-specific filtering and keyword matching\n - Dynamically calculates optimal character limits based on content complexity\n - Returns: JSON with selected segments optimized for the specific query type\n\n📋 get_document_overview(paper_dir: str)\n Purpose: Provides high-level overview of document structure and available segments\n - Shows document type and segmentation strategy used\n - Lists all segments with titles, content types, and relevance scores\n - Displays segment statistics (character counts, keyword summaries)\n - Returns: JSON with complete document analysis metadata\n\n==== SEGMENTATION STRATEGIES ====\n- semantic_research_focused: For academic papers with complex algorithmic content\n- algorithm_preserve_integrity: Maintains algorithm blocks and formula chains intact\n- concept_implementation_hybrid: Merges related concepts with implementation details\n- semantic_chunking_enhanced: Advanced boundary detection for long documents\n- content_aware_segmentation: Adaptive chunking based on content density\n\n==== INTELLIGENT FEATURES ====\n- Semantic boundary detection (not just structural)\n- Algorithm block identification and preservation\n- Formula chain recognition and grouping\n- Concept-implementation relationship mapping\n- Multi-level relevance scoring (content type, importance, keyword matching)\n- Backward compatibility with existing document indexes\n- Configurable via mcp_agent.config.yaml (enabled/disabled, size thresholds)\n\nUsage:\npython tools/document_segmentation_server.py\n\"\"\"\n\nimport os\nimport re\nimport json\nimport sys\nimport io\nfrom typing import Dict, List, Tuple\nimport hashlib\nimport logging\nfrom datetime import datetime\nfrom dataclasses import dataclass, asdict\n\n# Set standard output encoding to UTF-8\nif sys.stdout.encoding != \"utf-8\":\n try:\n if hasattr(sys.stdout, \"reconfigure\"):\n sys.stdout.reconfigure(encoding=\"utf-8\")\n sys.stderr.reconfigure(encoding=\"utf-8\")\n else:\n sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding=\"utf-8\")\n sys.stderr = io.TextIOWrapper(sys.stderr.detach(), encoding=\"utf-8\")\n except Exception as e:\n print(f\"Warning: Could not set UTF-8 encoding: {e}\")\n\n# Import MCP related modules\nfrom mcp.server.fastmcp import FastMCP\n\n# Setup logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\n# Create FastMCP server instance\nmcp = FastMCP(\"document-segmentation-server\")\n\n\n@dataclass\nclass DocumentSegment:\n \"\"\"Represents a document segment with metadata\"\"\"\n\n id: str\n title: str\n content: str\n content_type: str # \"introduction\", \"methodology\", \"algorithm\", \"results\", etc.\n keywords: List[str]\n char_start: int\n char_end: int\n char_count: int\n relevance_scores: Dict[str, float] # Scores for different query types\n section_path: str # e.g., \"3.2.1\" for nested sections\n\n\n@dataclass\nclass DocumentIndex:\n \"\"\"Document index containing all segments and metadata\"\"\"\n\n document_path: str\n document_type: str # \"academic_paper\", \"technical_doc\", \"code_doc\", \"general\"\n segmentation_strategy: str\n total_segments: int\n total_chars: int\n segments: List[DocumentSegment]\n created_at: str\n\n\nclass DocumentAnalyzer:\n \"\"\"Enhanced document analyzer using semantic content analysis instead of mechanical structure detection\"\"\"\n\n # More precise semantic indicators, weighted by importance\n ALGORITHM_INDICATORS = {\n \"high\": [\n \"algorithm\",\n \"procedure\",\n \"method\",\n \"approach\",\n \"technique\",\n \"framework\",\n ],\n \"medium\": [\"step\", \"process\", \"implementation\", \"computation\", \"calculation\"],\n \"low\": [\"example\", \"illustration\", \"demonstration\"],\n }\n\n TECHNICAL_CONCEPT_INDICATORS = {\n \"high\": [\"formula\", \"equation\", \"theorem\", \"lemma\", \"proof\", \"definition\"],\n \"medium\": [\"parameter\", \"variable\", \"function\", \"model\", \"architecture\"],\n \"low\": [\"notation\", \"symbol\", \"term\"],\n }\n\n IMPLEMENTATION_INDICATORS = {\n \"high\": [\"code\", \"implementation\", \"programming\", \"software\", \"system\"],\n \"medium\": [\"design\", \"structure\", \"module\", \"component\", \"interface\"],\n \"low\": [\"tool\", \"library\", \"package\"],\n }\n\n # Semantic features of document types (not just based on titles)\n RESEARCH_PAPER_PATTERNS = [\n r\"(?i)\\babstract\\b.*?\\n.*?(introduction|motivation|background)\",\n r\"(?i)(methodology|method).*?(experiment|evaluation|result)\",\n r\"(?i)(conclusion|future work|limitation).*?(reference|bibliography)\",\n r\"(?i)(related work|literature review|prior art)\",\n ]\n\n TECHNICAL_DOC_PATTERNS = [\n r\"(?i)(getting started|installation|setup).*?(usage|example)\",\n r\"(?i)(api|interface|specification).*?(parameter|endpoint)\",\n r\"(?i)(tutorial|guide|walkthrough).*?(step|instruction)\",\n r\"(?i)(troubleshooting|faq|common issues)\",\n ]\n\n def analyze_document_type(self, content: str) -> Tuple[str, float]:\n \"\"\"\n Enhanced document type analysis based on semantic content patterns\n\n Returns:\n Tuple[str, float]: (document_type, confidence_score)\n \"\"\"\n content_lower = content.lower()\n\n # Calculate weighted semantic indicator scores\n algorithm_score = self._calculate_weighted_score(\n content_lower, self.ALGORITHM_INDICATORS\n )\n concept_score = self._calculate_weighted_score(\n content_lower, self.TECHNICAL_CONCEPT_INDICATORS\n )\n implementation_score = self._calculate_weighted_score(\n content_lower, self.IMPLEMENTATION_INDICATORS\n )\n\n # Detect semantic patterns of document types\n research_pattern_score = self._detect_pattern_score(\n content, self.RESEARCH_PAPER_PATTERNS\n )\n technical_pattern_score = self._detect_pattern_score(\n content, self.TECHNICAL_DOC_PATTERNS\n )\n\n # Comprehensive evaluation of document type\n total_research_score = (\n algorithm_score + concept_score + research_pattern_score * 2\n )\n total_technical_score = implementation_score + technical_pattern_score * 2\n\n # Determine document type based on content density and pattern matching\n if research_pattern_score > 0.5 and total_research_score > 3.0:\n return \"research_paper\", min(0.95, 0.6 + research_pattern_score * 0.35)\n elif algorithm_score > 2.0 and concept_score > 1.5:\n return \"algorithm_focused\", 0.85\n elif total_technical_score > 2.5:\n return \"technical_doc\", 0.8\n elif implementation_score > 1.5:\n return \"implementation_guide\", 0.75\n else:\n return \"general_document\", 0.5\n\n def _calculate_weighted_score(\n self, content: str, indicators: Dict[str, List[str]]\n ) -> float:\n \"\"\"Calculate weighted semantic indicator scores\"\"\"\n score = 0.0\n for weight_level, terms in indicators.items():\n weight = {\"high\": 3.0, \"medium\": 2.0, \"low\": 1.0}[weight_level]\n for term in terms:\n if term in content:\n score += weight * (\n content.count(term) * 0.5 + 1\n ) # Consider term frequency\n return score\n\n def _detect_pattern_score(self, content: str, patterns: List[str]) -> float:\n \"\"\"Detect semantic pattern matching scores\"\"\"\n matches = 0\n for pattern in patterns:\n if re.search(pattern, content, re.DOTALL):\n matches += 1\n return matches / len(patterns)\n\n def detect_segmentation_strategy(self, content: str, doc_type: str) -> str:\n \"\"\"\n Intelligently determine the best segmentation strategy based on content semantics rather than mechanical structure\n \"\"\"\n # Analyze content characteristics\n algorithm_density = self._calculate_algorithm_density(content)\n concept_complexity = self._calculate_concept_complexity(content)\n implementation_detail_level = self._calculate_implementation_detail_level(\n content\n )\n\n # Select strategy based on document type and content characteristics\n if doc_type == \"research_paper\" and algorithm_density > 0.3:\n return \"semantic_research_focused\"\n elif doc_type == \"algorithm_focused\" or algorithm_density > 0.5:\n return \"algorithm_preserve_integrity\"\n elif concept_complexity > 0.4 and implementation_detail_level > 0.3:\n return \"concept_implementation_hybrid\"\n elif len(content) > 15000: # Long documents\n return \"semantic_chunking_enhanced\"\n else:\n return \"content_aware_segmentation\"\n\n def _calculate_algorithm_density(self, content: str) -> float:\n \"\"\"Calculate algorithm content density\"\"\"\n total_chars = len(content)\n algorithm_chars = 0\n\n # Identify algorithm blocks\n algorithm_patterns = [\n r\"(?i)(algorithm\\s+\\d+|procedure\\s+\\d+)\",\n r\"(?i)(step\\s+\\d+|phase\\s+\\d+)\",\n r\"(?i)(input:|output:|return:|initialize:)\",\n r\"(?i)(for\\s+each|while|if.*then|else)\",\n r\"(?i)(function|method|procedure).*\\(\",\n ]\n\n for pattern in algorithm_patterns:\n matches = re.finditer(pattern, content)\n for match in matches:\n # Estimate algorithm block size (expand forward and backward from match point)\n start = max(0, match.start() - 200)\n end = min(len(content), match.end() + 800)\n algorithm_chars += end - start\n\n return min(1.0, algorithm_chars / total_chars)\n\n def _calculate_concept_complexity(self, content: str) -> float:\n \"\"\"Calculate concept complexity\"\"\"\n concept_indicators = self.TECHNICAL_CONCEPT_INDICATORS\n complexity_score = 0.0\n\n for level, terms in concept_indicators.items():\n weight = {\"high\": 3.0, \"medium\": 2.0, \"low\": 1.0}[level]\n for term in terms:\n complexity_score += content.lower().count(term) * weight\n\n # Normalize to 0-1 range\n return min(1.0, complexity_score / 100)\n\n def _calculate_implementation_detail_level(self, content: str) -> float:\n \"\"\"Calculate implementation detail level\"\"\"\n implementation_patterns = [\n r\"(?i)(code|implementation|programming)\",\n r\"(?i)(class|function|method|variable)\",\n r\"(?i)(import|include|library)\",\n r\"(?i)(parameter|argument|return)\",\n r\"(?i)(example|demo|tutorial)\",\n ]\n\n detail_score = 0\n for pattern in implementation_patterns:\n detail_score += len(re.findall(pattern, content))\n\n return min(1.0, detail_score / 50)\n\n\nclass DocumentSegmenter:\n \"\"\"Creates intelligent segments from documents\"\"\"\n\n def __init__(self):\n self.analyzer = DocumentAnalyzer()\n\n def segment_document(self, content: str, strategy: str) -> List[DocumentSegment]:\n \"\"\"\n Perform intelligent segmentation using the specified strategy\n \"\"\"\n if strategy == \"semantic_research_focused\":\n return self._segment_research_paper_semantically(content)\n elif strategy == \"algorithm_preserve_integrity\":\n return self._segment_preserve_algorithm_integrity(content)\n elif strategy == \"concept_implementation_hybrid\":\n return self._segment_concept_implementation_hybrid(content)\n elif strategy == \"semantic_chunking_enhanced\":\n return self._segment_by_enhanced_semantic_chunks(content)\n elif strategy == \"content_aware_segmentation\":\n return self._segment_content_aware(content)\n else:\n # Compatibility with legacy strategies\n return self._segment_by_enhanced_semantic_chunks(content)\n\n def _segment_by_headers(self, content: str) -> List[DocumentSegment]:\n \"\"\"Segment document based on markdown headers\"\"\"\n segments = []\n lines = content.split(\"\\n\")\n current_segment = []\n current_header = None\n char_pos = 0\n\n for line in lines:\n line_with_newline = line + \"\\n\"\n\n # Check if line is a header\n header_match = re.match(r\"^(#{1,6})\\s+(.+)$\", line)\n\n if header_match:\n # Save previous segment if exists\n if current_segment and current_header:\n segment_content = \"\\n\".join(current_segment).strip()\n if segment_content:\n # Analyze content type and importance\n content_type = self._classify_content_type(\n current_header, segment_content\n )\n importance_score = (\n 0.8 if content_type in [\"algorithm\", \"formula\"] else 0.7\n )\n\n segment = self._create_enhanced_segment(\n segment_content,\n current_header,\n char_pos - len(segment_content.encode(\"utf-8\")),\n char_pos,\n importance_score,\n content_type,\n )\n segments.append(segment)\n\n # Start new segment\n current_header = header_match.group(2).strip()\n current_segment = [line]\n else:\n if current_segment is not None:\n current_segment.append(line)\n\n char_pos += len(line_with_newline.encode(\"utf-8\"))\n\n # Add final segment\n if current_segment and current_header:\n segment_content = \"\\n\".join(current_segment).strip()\n if segment_content:\n # Analyze content type and importance\n content_type = self._classify_content_type(\n current_header, segment_content\n )\n importance_score = (\n 0.8 if content_type in [\"algorithm\", \"formula\"] else 0.7\n )\n\n segment = self._create_enhanced_segment(\n segment_content,\n current_header,\n char_pos - len(segment_content.encode(\"utf-8\")),\n char_pos,\n importance_score,\n content_type,\n )\n segments.append(segment)\n\n return segments\n\n def _segment_preserve_algorithm_integrity(\n self, content: str\n ) -> List[DocumentSegment]:\n \"\"\"Smart segmentation strategy that preserves algorithm integrity\"\"\"\n segments = []\n\n # 1. Identify algorithm blocks and related descriptions\n algorithm_blocks = self._identify_algorithm_blocks(content)\n\n # 2. Identify concept definition groups\n concept_groups = self._identify_concept_groups(content)\n\n # 3. Identify formula derivation chains\n formula_chains = self._identify_formula_chains(content)\n\n # 4. Merge related content blocks to ensure integrity\n content_blocks = self._merge_related_content_blocks(\n algorithm_blocks, concept_groups, formula_chains, content\n )\n\n # 5. Convert to DocumentSegment\n for i, block in enumerate(content_blocks):\n segment = self._create_enhanced_segment(\n block[\"content\"],\n block[\"title\"],\n block[\"start_pos\"],\n block[\"end_pos\"],\n block[\"importance_score\"],\n block[\"content_type\"],\n )\n segments.append(segment)\n\n return segments\n\n def _segment_research_paper_semantically(\n self, content: str\n ) -> List[DocumentSegment]:\n \"\"\"Semantic segmentation specifically for research papers\"\"\"\n segments = []\n\n # Identify semantic structure of research papers\n paper_sections = self._identify_research_paper_sections(content)\n\n for section in paper_sections:\n # Ensure each section contains sufficient context\n enhanced_content = self._enhance_section_with_context(section, content)\n\n segment = self._create_enhanced_segment(\n enhanced_content[\"content\"],\n enhanced_content[\"title\"],\n enhanced_content[\"start_pos\"],\n enhanced_content[\"end_pos\"],\n enhanced_content[\"importance_score\"],\n enhanced_content[\"content_type\"],\n )\n segments.append(segment)\n\n return segments\n\n def _segment_concept_implementation_hybrid(\n self, content: str\n ) -> List[DocumentSegment]:\n \"\"\"Intelligent segmentation combining concepts and implementation\"\"\"\n segments = []\n\n # Identify concept-implementation correspondence\n concept_impl_pairs = self._identify_concept_implementation_pairs(content)\n\n for pair in concept_impl_pairs:\n # Merge related concepts and implementations into one segment\n merged_content = self._merge_concept_with_implementation(pair, content)\n\n segment = self._create_enhanced_segment(\n merged_content[\"content\"],\n merged_content[\"title\"],\n merged_content[\"start_pos\"],\n merged_content[\"end_pos\"],\n merged_content[\"importance_score\"],\n merged_content[\"content_type\"],\n )\n segments.append(segment)\n\n return segments\n\n def _segment_by_enhanced_semantic_chunks(\n self, content: str\n ) -> List[DocumentSegment]:\n \"\"\"Enhanced semantic chunk segmentation\"\"\"\n segments = []\n\n # Use improved semantic boundary detection\n semantic_boundaries = self._detect_semantic_boundaries(content)\n\n current_start = 0\n for i, boundary in enumerate(semantic_boundaries):\n chunk_content = content[current_start : boundary[\"position\"]]\n\n if len(chunk_content.strip()) > 200: # Minimum content threshold\n segment = self._create_enhanced_segment(\n chunk_content,\n boundary[\"suggested_title\"],\n current_start,\n boundary[\"position\"],\n boundary[\"importance_score\"],\n boundary[\"content_type\"],\n )\n segments.append(segment)\n\n current_start = boundary[\"position\"]\n\n # Handle the final segment\n if current_start < len(content):\n final_content = content[current_start:]\n if len(final_content.strip()) > 200:\n segment = self._create_enhanced_segment(\n final_content,\n \"Final Section\",\n current_start,\n len(content),\n 0.7,\n \"general\",\n )\n segments.append(segment)\n\n return segments\n\n def _segment_content_aware(self, content: str) -> List[DocumentSegment]:\n \"\"\"Content-aware intelligent segmentation\"\"\"\n segments = []\n\n # Adaptive segmentation size\n optimal_chunk_size = self._calculate_optimal_chunk_size(content)\n\n # Segment based on content density\n content_chunks = self._create_content_aware_chunks(content, optimal_chunk_size)\n\n for chunk in content_chunks:\n segment = self._create_enhanced_segment(\n chunk[\"content\"],\n chunk[\"title\"],\n chunk[\"start_pos\"],\n chunk[\"end_pos\"],\n chunk[\"importance_score\"],\n chunk[\"content_type\"],\n )\n segments.append(segment)\n\n return segments\n\n def _segment_academic_paper(self, content: str) -> List[DocumentSegment]:\n \"\"\"Segment academic paper using semantic understanding\"\"\"\n # First try header-based segmentation\n headers = re.findall(r\"^(#{1,6})\\s+(.+)$\", content, re.MULTILINE)\n if len(headers) >= 2:\n return self._segment_by_headers(content)\n\n # Fallback to semantic detection of academic sections\n sections = self._detect_academic_sections(content)\n segments = []\n\n for section in sections:\n # Determine importance based on section type\n section_type = section.get(\"type\", \"general\")\n content_type = (\n section_type\n if section_type\n in [\"algorithm\", \"formula\", \"introduction\", \"conclusion\"]\n else \"general\"\n )\n importance_score = {\n \"algorithm\": 0.95,\n \"formula\": 0.9,\n \"introduction\": 0.85,\n \"conclusion\": 0.8,\n }.get(content_type, 0.7)\n\n segment = self._create_enhanced_segment(\n section[\"content\"],\n section[\"title\"],\n section[\"start_pos\"],\n section[\"end_pos\"],\n importance_score,\n content_type,\n )\n segments.append(segment)\n\n return segments\n\n def _detect_academic_sections(self, content: str) -> List[Dict]:\n \"\"\"Detect academic paper sections even without clear headers\"\"\"\n sections = []\n\n # Common academic section patterns\n section_patterns = [\n (r\"(?i)(abstract|摘要)\", \"introduction\"),\n (r\"(?i)(introduction|引言|简介)\", \"introduction\"),\n (r\"(?i)(related work|相关工作|背景)\", \"background\"),\n (r\"(?i)(method|methodology|approach|方法)\", \"methodology\"),\n (r\"(?i)(algorithm|算法)\", \"algorithm\"),\n (r\"(?i)(experiment|实验|evaluation|评估)\", \"experiment\"),\n (r\"(?i)(result|结果|finding)\", \"results\"),\n (r\"(?i)(conclusion|结论|总结)\", \"conclusion\"),\n (r\"(?i)(reference|参考文献|bibliography)\", \"references\"),\n ]\n\n current_pos = 0\n for i, (pattern, section_type) in enumerate(section_patterns):\n match = re.search(pattern, content[current_pos:], re.IGNORECASE)\n if match:\n start_pos = current_pos + match.start()\n\n # Find end position (next section or end of document)\n next_pos = len(content)\n for next_pattern, _ in section_patterns[i + 1 :]:\n next_match = re.search(\n next_pattern, content[start_pos + 100 :], re.IGNORECASE\n )\n if next_match:\n next_pos = start_pos + 100 + next_match.start()\n break\n\n section_content = content[start_pos:next_pos].strip()\n if len(section_content) > 50: # Minimum content length\n # Calculate importance score and content type\n importance_score = self._calculate_paragraph_importance(\n section_content, section_type\n )\n content_type = self._classify_content_type(\n match.group(1), section_content\n )\n\n sections.append(\n {\n \"title\": match.group(1),\n \"content\": section_content,\n \"start_pos\": start_pos,\n \"end_pos\": next_pos,\n \"type\": section_type,\n \"importance_score\": importance_score,\n \"content_type\": content_type,\n }\n )\n\n current_pos = next_pos\n\n return sections\n\n def _segment_by_semantic_chunks(self, content: str) -> List[DocumentSegment]:\n \"\"\"Segment long documents into semantic chunks\"\"\"\n # Split into paragraphs first\n paragraphs = [p.strip() for p in content.split(\"\\n\\n\") if p.strip()]\n\n segments = []\n current_chunk = []\n current_chunk_size = 0\n chunk_size_limit = 3000 # characters\n overlap_size = 200\n\n char_pos = 0\n\n for para in paragraphs:\n para_size = len(para)\n\n # If adding this paragraph exceeds limit, create a segment\n if current_chunk_size + para_size > chunk_size_limit and current_chunk:\n chunk_content = \"\\n\\n\".join(current_chunk)\n # Analyze semantic chunk content type\n content_type = self._classify_paragraph_type(chunk_content)\n importance_score = self._calculate_paragraph_importance(\n chunk_content, content_type\n )\n\n segment = self._create_enhanced_segment(\n chunk_content,\n f\"Section {len(segments) + 1}\",\n char_pos - len(chunk_content.encode(\"utf-8\")),\n char_pos,\n importance_score,\n content_type,\n )\n segments.append(segment)\n\n # Keep last part for overlap\n overlap_content = (\n chunk_content[-overlap_size:]\n if len(chunk_content) > overlap_size\n else \"\"\n )\n current_chunk = [overlap_content, para] if overlap_content else [para]\n current_chunk_size = len(overlap_content) + para_size\n else:\n current_chunk.append(para)\n current_chunk_size += para_size\n\n char_pos += para_size + 2 # +2 for \\n\\n\n\n # Add final chunk\n if current_chunk:\n chunk_content = \"\\n\\n\".join(current_chunk)\n # Analyze final chunk content type\n content_type = self._classify_paragraph_type(chunk_content)\n importance_score = self._calculate_paragraph_importance(\n chunk_content, content_type\n )\n\n segment = self._create_enhanced_segment(\n chunk_content,\n f\"Section {len(segments) + 1}\",\n char_pos - len(chunk_content.encode(\"utf-8\")),\n char_pos,\n importance_score,\n content_type,\n )\n segments.append(segment)\n\n return segments\n\n def _segment_by_paragraphs(self, content: str) -> List[DocumentSegment]:\n \"\"\"Simple paragraph-based segmentation for short documents\"\"\"\n paragraphs = [p.strip() for p in content.split(\"\\n\\n\") if p.strip()]\n segments = []\n char_pos = 0\n\n for i, para in enumerate(paragraphs):\n if len(para) > 100: # Only include substantial paragraphs\n # Analyze paragraph type and importance\n content_type = self._classify_paragraph_type(para)\n importance_score = self._calculate_paragraph_importance(\n para, content_type\n )\n\n segment = self._create_enhanced_segment(\n para,\n f\"Paragraph {i + 1}\",\n char_pos,\n char_pos + len(para.encode(\"utf-8\")),\n importance_score,\n content_type,\n )\n segments.append(segment)\n char_pos += len(para.encode(\"utf-8\")) + 2\n\n return segments\n\n # =============== Enhanced intelligent segmentation helper methods ===============\n\n def _identify_algorithm_blocks(self, content: str) -> List[Dict]:\n \"\"\"Identify algorithm blocks and related descriptions\"\"\"\n algorithm_blocks = []\n\n # Algorithm block identification patterns\n algorithm_patterns = [\n r\"(?i)(algorithm\\s+\\d+|procedure\\s+\\d+|method\\s+\\d+).*?(?=algorithm\\s+\\d+|procedure\\s+\\d+|method\\s+\\d+|$)\",\n r\"(?i)(input:|output:|returns?:|require:|ensure:).*?(?=\\n\\s*\\n|\\n\\s*(?:input:|output:|returns?:|require:|ensure:)|$)\",\n r\"(?i)(for\\s+each|while|if.*then|repeat.*until).*?(?=\\n\\s*\\n|$)\",\n r\"(?i)(step\\s+\\d+|phase\\s+\\d+).*?(?=step\\s+\\d+|phase\\s+\\d+|\\n\\s*\\n|$)\",\n ]\n\n for pattern in algorithm_patterns:\n matches = re.finditer(pattern, content, re.DOTALL)\n for match in matches:\n # Expand context to include complete descriptions\n start = max(0, match.start() - 300)\n end = min(len(content), match.end() + 500)\n\n # Find natural boundaries\n while start > 0 and content[start] not in \"\\n.!?\":\n start -= 1\n while end < len(content) and content[end] not in \"\\n.!?\":\n end += 1\n\n algorithm_blocks.append(\n {\n \"start_pos\": start,\n \"end_pos\": end,\n \"content\": content[start:end].strip(),\n \"title\": self._extract_algorithm_title(\n content[match.start() : match.end()]\n ),\n \"importance_score\": 0.95, # High importance for algorithm blocks\n \"content_type\": \"algorithm\",\n }\n )\n\n return algorithm_blocks\n\n def _identify_concept_groups(self, content: str) -> List[Dict]:\n \"\"\"Identify concept definition groups\"\"\"\n concept_groups = []\n\n # Concept definition patterns\n concept_patterns = [\n r\"(?i)(definition|define|let|denote|given).*?(?=\\n\\s*\\n|definition|define|let|denote|$)\",\n r\"(?i)(theorem|lemma|proposition|corollary).*?(?=\\n\\s*\\n|theorem|lemma|proposition|corollary|$)\",\n r\"(?i)(notation|symbol|parameter).*?(?=\\n\\s*\\n|notation|symbol|parameter|$)\",\n ]\n\n for pattern in concept_patterns:\n matches = re.finditer(pattern, content, re.DOTALL)\n for match in matches:\n # Expand context\n start = max(0, match.start() - 200)\n end = min(len(content), match.end() + 300)\n\n concept_groups.append(\n {\n \"start_pos\": start,\n \"end_pos\": end,\n \"content\": content[start:end].strip(),\n \"title\": self._extract_concept_title(\n content[match.start() : match.end()]\n ),\n \"importance_score\": 0.85,\n \"content_type\": \"concept\",\n }\n )\n\n return concept_groups\n\n def _identify_formula_chains(self, content: str) -> List[Dict]:\n \"\"\"Identify formula derivation chains\"\"\"\n formula_chains = []\n\n # Formula patterns\n formula_patterns = [\n r\"\\$\\$.*?\\$\\$\", # Block-level mathematical formulas\n r\"\\$[^$]+\\$\", # Inline mathematical formulas\n r\"(?i)(equation|formula).*?(?=\\n\\s*\\n|equation|formula|$)\",\n r\"(?i)(where|such that|given that).*?(?=\\n\\s*\\n|where|such that|given that|$)\",\n ]\n\n # Find dense formula regions\n formula_positions = []\n for pattern in formula_patterns:\n matches = re.finditer(pattern, content, re.DOTALL)\n for match in matches:\n formula_positions.append((match.start(), match.end()))\n\n # Merge nearby formulas into formula chains\n formula_positions.sort()\n if formula_positions:\n current_chain_start = formula_positions[0][0]\n current_chain_end = formula_positions[0][1]\n\n for start, end in formula_positions[1:]:\n if (\n start - current_chain_end < 500\n ): # Merge formulas within 500 characters\n current_chain_end = end\n else:\n # Save current chain\n formula_chains.append(\n {\n \"start_pos\": max(0, current_chain_start - 200),\n \"end_pos\": min(len(content), current_chain_end + 200),\n \"content\": content[\n max(0, current_chain_start - 200) : min(\n len(content), current_chain_end + 200\n )\n ].strip(),\n \"title\": \"Mathematical Formulation\",\n \"importance_score\": 0.9,\n \"content_type\": \"formula\",\n }\n )\n current_chain_start = start\n current_chain_end = end\n\n # Add the last chain\n formula_chains.append(\n {\n \"start_pos\": max(0, current_chain_start - 200),\n \"end_pos\": min(len(content), current_chain_end + 200),\n \"content\": content[\n max(0, current_chain_start - 200) : min(\n len(content), current_chain_end + 200\n )\n ].strip(),\n \"title\": \"Mathematical Formulation\",\n \"importance_score\": 0.9,\n \"content_type\": \"formula\",\n }\n )\n\n return formula_chains\n\n def _merge_related_content_blocks(\n self,\n algorithm_blocks: List[Dict],\n concept_groups: List[Dict],\n formula_chains: List[Dict],\n content: str,\n ) -> List[Dict]:\n \"\"\"Merge related content blocks to ensure integrity\"\"\"\n all_blocks = algorithm_blocks + concept_groups + formula_chains\n all_blocks.sort(key=lambda x: x[\"start_pos\"])\n\n merged_blocks = []\n i = 0\n\n while i < len(all_blocks):\n current_block = all_blocks[i]\n\n # Check if can merge with the next block\n while i + 1 < len(all_blocks):\n next_block = all_blocks[i + 1]\n\n # If blocks are close or content related, merge them\n if next_block[\"start_pos\"] - current_block[\n \"end_pos\"\n ] < 300 or self._are_blocks_related(current_block, next_block):\n # Merge blocks\n merged_content = content[\n current_block[\"start_pos\"] : next_block[\"end_pos\"]\n ]\n current_block = {\n \"start_pos\": current_block[\"start_pos\"],\n \"end_pos\": next_block[\"end_pos\"],\n \"content\": merged_content.strip(),\n \"title\": f\"{current_block['title']} & {next_block['title']}\",\n \"importance_score\": max(\n current_block[\"importance_score\"],\n next_block[\"importance_score\"],\n ),\n \"content_type\": \"merged\",\n }\n i += 1\n else:\n break\n\n merged_blocks.append(current_block)\n i += 1\n\n return merged_blocks\n\n def _are_blocks_related(self, block1: Dict, block2: Dict) -> bool:\n \"\"\"Determine if two content blocks are related\"\"\"\n # Check content type associations\n related_types = [\n (\"algorithm\", \"formula\"),\n (\"concept\", \"algorithm\"),\n (\"formula\", \"concept\"),\n ]\n\n for type1, type2 in related_types:\n if (\n block1[\"content_type\"] == type1 and block2[\"content_type\"] == type2\n ) or (block1[\"content_type\"] == type2 and block2[\"content_type\"] == type1):\n return True\n\n return False\n\n def _extract_algorithm_title(self, text: str) -> str:\n \"\"\"Extract title from algorithm text\"\"\"\n lines = text.split(\"\\n\")[:3] # First 3 lines\n for line in lines:\n line = line.strip()\n if line and len(line) < 100: # Reasonable title length\n # Clean title\n title = re.sub(r\"[^\\w\\s-]\", \"\", line)\n if title:\n return title[:50] # Limit title length\n return \"Algorithm Block\"\n\n def _extract_concept_title(self, text: str) -> str:\n \"\"\"Extract title from concept text\"\"\"\n lines = text.split(\"\\n\")[:2]\n for line in lines:\n line = line.strip()\n if line and len(line) < 80:\n title = re.sub(r\"[^\\w\\s-]\", \"\", line)\n if title:\n return title[:50]\n return \"Concept Definition\"\n\n def _create_enhanced_segment(\n self,\n content: str,\n title: str,\n start_pos: int,\n end_pos: int,\n importance_score: float,\n content_type: str,\n ) -> DocumentSegment:\n \"\"\"Create enhanced document segment\"\"\"\n # Generate unique ID\n segment_id = hashlib.md5(\n f\"{title}_{start_pos}_{end_pos}_{importance_score}\".encode()\n ).hexdigest()[:8]\n\n # Extract keywords\n keywords = self._extract_enhanced_keywords(content, content_type)\n\n # Calculate enhanced relevance scores\n relevance_scores = self._calculate_enhanced_relevance_scores(\n content, content_type, importance_score\n )\n\n return DocumentSegment(\n id=segment_id,\n title=title,\n content=content,\n content_type=content_type,\n keywords=keywords,\n char_start=start_pos,\n char_end=end_pos,\n char_count=len(content),\n relevance_scores=relevance_scores,\n section_path=title,\n )\n\n def _extract_enhanced_keywords(self, content: str, content_type: str) -> List[str]:\n \"\"\"Extract enhanced keywords based on content type\"\"\"\n words = re.findall(r\"\\b[a-zA-Z]{3,}\\b\", content.lower())\n\n # Adjust stopwords based on content type\n if content_type == \"algorithm\":\n algorithm_stopwords = {\n \"step\",\n \"then\",\n \"else\",\n \"end\",\n \"begin\",\n \"start\",\n \"stop\",\n }\n words = [w for w in words if w not in algorithm_stopwords]\n elif content_type == \"formula\":\n formula_keywords = [\"equation\", \"formula\", \"where\", \"given\", \"such\", \"that\"]\n words.extend(formula_keywords)\n\n # General stopwords\n general_stopwords = {\n \"the\",\n \"and\",\n \"for\",\n \"are\",\n \"but\",\n \"not\",\n \"you\",\n \"all\",\n \"can\",\n \"her\",\n \"was\",\n \"one\",\n \"our\",\n \"had\",\n \"but\",\n \"have\",\n \"this\",\n \"that\",\n \"with\",\n \"from\",\n \"they\",\n \"she\",\n \"been\",\n \"were\",\n \"said\",\n \"each\",\n \"which\",\n \"their\",\n }\n\n keywords = [w for w in set(words) if w not in general_stopwords and len(w) > 3]\n return keywords[:25] # Increase keyword count\n\n def _calculate_enhanced_relevance_scores(\n self, content: str, content_type: str, importance_score: float\n ) -> Dict[str, float]:\n \"\"\"Calculate enhanced relevance scores\"\"\"\n content_lower = content.lower()\n\n base_scores = {\n \"concept_analysis\": 0.5,\n \"algorithm_extraction\": 0.5,\n \"code_planning\": 0.5,\n }\n\n # Adjust base scores based on content type and importance\n if content_type == \"algorithm\":\n base_scores[\"algorithm_extraction\"] = importance_score\n base_scores[\"code_planning\"] = importance_score * 0.9\n base_scores[\"concept_analysis\"] = importance_score * 0.7\n elif content_type == \"concept\":\n base_scores[\"concept_analysis\"] = importance_score\n base_scores[\"algorithm_extraction\"] = importance_score * 0.8\n base_scores[\"code_planning\"] = importance_score * 0.6\n elif content_type == \"formula\":\n base_scores[\"algorithm_extraction\"] = importance_score\n base_scores[\"concept_analysis\"] = importance_score * 0.8\n base_scores[\"code_planning\"] = importance_score * 0.9\n elif content_type == \"merged\":\n # Merged content is usually important\n base_scores = {k: importance_score * 0.95 for k in base_scores}\n\n # Additional bonus based on content density\n algorithm_indicators = [\"algorithm\", \"method\", \"procedure\", \"step\", \"process\"]\n concept_indicators = [\"definition\", \"concept\", \"framework\", \"approach\"]\n implementation_indicators = [\"implementation\", \"code\", \"function\", \"design\"]\n\n for query_type, indicators in [\n (\"algorithm_extraction\", algorithm_indicators),\n (\"concept_analysis\", concept_indicators),\n (\"code_planning\", implementation_indicators),\n ]:\n density_bonus = (\n sum(1 for indicator in indicators if indicator in content_lower) * 0.1\n )\n base_scores[query_type] = min(1.0, base_scores[query_type] + density_bonus)\n\n return base_scores\n\n # Placeholder methods - can be further implemented later\n def _identify_research_paper_sections(self, content: str) -> List[Dict]:\n \"\"\"Identify research paper sections - simplified implementation\"\"\"\n # Temporarily use improved semantic detection\n return self._detect_academic_sections(content)\n\n def _enhance_section_with_context(self, section: Dict, content: str) -> Dict:\n \"\"\"Add context to sections - simplified implementation\"\"\"\n return section\n\n def _identify_concept_implementation_pairs(self, content: str) -> List[Dict]:\n \"\"\"Identify concept-implementation pairs - simplified implementation\"\"\"\n return []\n\n def _merge_concept_with_implementation(self, pair: Dict, content: str) -> Dict:\n \"\"\"Merge concepts with implementation - simplified implementation\"\"\"\n return pair\n\n def _detect_semantic_boundaries(self, content: str) -> List[Dict]:\n \"\"\"Detect semantic boundaries - based on paragraphs and logical separators\"\"\"\n boundaries = []\n\n # Split paragraphs by double line breaks\n paragraphs = content.split(\"\\n\\n\")\n current_pos = 0\n\n for i, para in enumerate(paragraphs):\n if len(para.strip()) > 100: # Valid paragraph\n # Analyze paragraph type\n content_type = self._classify_paragraph_type(para)\n importance_score = self._calculate_paragraph_importance(\n para, content_type\n )\n\n boundaries.append(\n {\n \"position\": current_pos + len(para),\n \"suggested_title\": self._extract_paragraph_title(para, i + 1),\n \"importance_score\": importance_score,\n \"content_type\": content_type,\n }\n )\n\n current_pos += len(para) + 2 # +2 for \\n\\n\n\n return boundaries\n\n def _classify_paragraph_type(self, paragraph: str) -> str:\n \"\"\"Classify paragraph type\"\"\"\n para_lower = paragraph.lower()\n\n if \"algorithm\" in para_lower or \"procedure\" in para_lower:\n return \"algorithm\"\n elif \"formula\" in para_lower or \"$$\" in paragraph:\n return \"formula\"\n elif any(\n word in para_lower for word in [\"introduction\", \"overview\", \"abstract\"]\n ):\n return \"introduction\"\n elif any(word in para_lower for word in [\"conclusion\", \"summary\", \"result\"]):\n return \"conclusion\"\n else:\n return \"general\"\n\n def _calculate_paragraph_importance(\n self, paragraph: str, content_type: str\n ) -> float:\n \"\"\"Calculate paragraph importance\"\"\"\n if content_type == \"algorithm\":\n return 0.95\n elif content_type == \"formula\":\n return 0.9\n elif content_type == \"introduction\":\n return 0.85\n elif content_type == \"conclusion\":\n return 0.8\n else:\n return 0.7\n\n def _extract_paragraph_title(self, paragraph: str, index: int) -> str:\n \"\"\"Extract paragraph title\"\"\"\n lines = paragraph.split(\"\\n\")\n for line in lines[:2]:\n if line.startswith(\"#\"):\n return line.strip(\"# \")\n elif len(line) < 80 and line.strip():\n return line.strip()\n return f\"Section {index}\"\n\n def _calculate_optimal_chunk_size(self, content: str) -> int:\n \"\"\"Calculate optimal chunk size\"\"\"\n # Dynamically adjust based on content complexity\n complexity = self.analyzer._calculate_concept_complexity(content)\n if complexity > 0.7:\n return 4000 # Complex content needs larger chunks\n elif complexity > 0.4:\n return 3000\n else:\n return 2000\n\n def _create_content_aware_chunks(self, content: str, chunk_size: int) -> List[Dict]:\n \"\"\"Create content-aware chunks - simplified implementation\"\"\"\n chunks = []\n paragraphs = [p.strip() for p in content.split(\"\\n\\n\") if p.strip()]\n\n current_chunk = []\n current_size = 0\n start_pos = 0\n\n for para in paragraphs:\n para_size = len(para)\n\n if current_size + para_size > chunk_size and current_chunk:\n chunk_content = \"\\n\\n\".join(current_chunk)\n chunks.append(\n {\n \"content\": chunk_content,\n \"title\": f\"Section {len(chunks) + 1}\",\n \"start_pos\": start_pos,\n \"end_pos\": start_pos + len(chunk_content),\n \"importance_score\": 0.7,\n \"content_type\": \"general\",\n }\n )\n\n current_chunk = [para]\n current_size = para_size\n start_pos += len(chunk_content) + 2\n else:\n current_chunk.append(para)\n current_size += para_size\n\n # Add the last chunk\n if current_chunk:\n chunk_content = \"\\n\\n\".join(current_chunk)\n chunks.append(\n {\n \"content\": chunk_content,\n \"title\": f\"Section {len(chunks) + 1}\",\n \"start_pos\": start_pos,\n \"end_pos\": start_pos + len(chunk_content),\n \"importance_score\": 0.7,\n \"content_type\": \"general\",\n }\n )\n\n return chunks\n\n def _create_segment(\n self, content: str, title: str, start_pos: int, end_pos: int\n ) -> DocumentSegment:\n \"\"\"Create a DocumentSegment with metadata\"\"\"\n # Generate unique ID\n segment_id = hashlib.md5(f\"{title}_{start_pos}_{end_pos}\".encode()).hexdigest()[\n :8\n ]\n\n # Extract keywords from content\n keywords = self._extract_keywords(content)\n\n # Determine content type\n content_type = self._classify_content_type(title, content)\n\n # Calculate relevance scores for different query types\n relevance_scores = self._calculate_relevance_scores(content, content_type)\n\n return DocumentSegment(\n id=segment_id,\n title=title,\n content=content,\n content_type=content_type,\n keywords=keywords,\n char_start=start_pos,\n char_end=end_pos,\n char_count=len(content),\n relevance_scores=relevance_scores,\n section_path=title, # Simplified for now\n )\n\n def _extract_keywords(self, content: str) -> List[str]:\n \"\"\"Extract relevant keywords from content\"\"\"\n # Simple keyword extraction - could be enhanced with NLP\n words = re.findall(r\"\\b[a-zA-Z]{3,}\\b\", content.lower())\n\n # Remove common words\n stopwords = {\n \"the\",\n \"and\",\n \"for\",\n \"are\",\n \"but\",\n \"not\",\n \"you\",\n \"all\",\n \"can\",\n \"her\",\n \"was\",\n \"one\",\n \"our\",\n \"had\",\n \"but\",\n \"have\",\n \"this\",\n \"that\",\n \"with\",\n \"from\",\n \"they\",\n \"she\",\n \"been\",\n \"were\",\n \"said\",\n \"each\",\n \"which\",\n \"their\",\n }\n\n keywords = [w for w in set(words) if w not in stopwords and len(w) > 3]\n return keywords[:20] # Top 20 keywords\n\n def _classify_content_type(self, title: str, content: str) -> str:\n \"\"\"Classify the type of content based on title and content\"\"\"\n title_lower = title.lower()\n content_lower = content.lower()\n\n if any(\n word in title_lower for word in [\"introduction\", \"abstract\", \"overview\"]\n ):\n return \"introduction\"\n elif any(word in title_lower for word in [\"method\", \"approach\", \"algorithm\"]):\n return \"methodology\"\n elif any(\n word in title_lower for word in [\"experiment\", \"evaluation\", \"result\"]\n ):\n return \"experiment\"\n elif any(\n word in title_lower for word in [\"conclusion\", \"discussion\", \"summary\"]\n ):\n return \"conclusion\"\n elif any(word in title_lower for word in [\"reference\", \"bibliography\"]):\n return \"references\"\n elif \"algorithm\" in content_lower or \"procedure\" in content_lower:\n return \"algorithm\"\n else:\n return \"general\"\n\n def _calculate_relevance_scores(\n self, content: str, content_type: str\n ) -> Dict[str, float]:\n \"\"\"Calculate relevance scores for different query types\"\"\"\n content_lower = content.lower()\n\n scores = {\n \"concept_analysis\": 0.5,\n \"algorithm_extraction\": 0.5,\n \"code_planning\": 0.5,\n }\n\n # Concept analysis relevance\n concept_indicators = [\n \"introduction\",\n \"overview\",\n \"architecture\",\n \"system\",\n \"framework\",\n \"concept\",\n \"approach\",\n ]\n concept_score = sum(\n 1 for indicator in concept_indicators if indicator in content_lower\n ) / len(concept_indicators)\n scores[\"concept_analysis\"] = min(\n 1.0, concept_score + (0.8 if content_type == \"introduction\" else 0)\n )\n\n # Algorithm extraction relevance\n algorithm_indicators = [\n \"algorithm\",\n \"method\",\n \"procedure\",\n \"formula\",\n \"equation\",\n \"step\",\n \"process\",\n ]\n algorithm_score = sum(\n 1 for indicator in algorithm_indicators if indicator in content_lower\n ) / len(algorithm_indicators)\n scores[\"algorithm_extraction\"] = min(\n 1.0, algorithm_score + (0.9 if content_type == \"methodology\" else 0)\n )\n\n # Code planning relevance\n code_indicators = [\n \"implementation\",\n \"code\",\n \"function\",\n \"class\",\n \"module\",\n \"structure\",\n \"design\",\n ]\n code_score = sum(\n 1 for indicator in code_indicators if indicator in content_lower\n ) / len(code_indicators)\n scores[\"code_planning\"] = min(\n 1.0,\n code_score + (0.7 if content_type in [\"methodology\", \"algorithm\"] else 0),\n )\n\n return scores\n\n\n# Global variables\nDOCUMENT_INDEXES: Dict[str, DocumentIndex] = {}\nsegmenter = DocumentSegmenter()\n\n\ndef get_segments_dir(paper_dir: str) -> str:\n \"\"\"Get the segments directory path\"\"\"\n return os.path.join(paper_dir, \"document_segments\")\n\n\ndef ensure_segments_dir_exists(segments_dir: str):\n \"\"\"Ensure segments directory exists\"\"\"\n os.makedirs(segments_dir, exist_ok=True)\n\n\n@mcp.tool()\nasync def analyze_and_segment_document(\n paper_dir: str, force_refresh: bool = False\n) -> str:\n \"\"\"\n Analyze document structure and create intelligent segments\n\n Args:\n paper_dir: Path to the paper directory\n force_refresh: Whether to force re-analysis even if segments exist\n\n Returns:\n JSON string with segmentation results\n \"\"\"\n try:\n # Find markdown file in paper directory\n md_files = [f for f in os.listdir(paper_dir) if f.endswith(\".md\")]\n if not md_files:\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"No markdown file found in {paper_dir}\",\n },\n ensure_ascii=False,\n indent=2,\n )\n\n md_file_path = os.path.join(paper_dir, md_files[0])\n segments_dir = get_segments_dir(paper_dir)\n index_file_path = os.path.join(segments_dir, \"document_index.json\")\n\n # Check if analysis already exists and is recent\n if not force_refresh and os.path.exists(index_file_path):\n try:\n with open(index_file_path, \"r\", encoding=\"utf-8\") as f:\n existing_index = json.load(f)\n\n # Compatibility handling: ensure segments data structure is correct\n if \"segments\" in existing_index:\n segments_data = []\n for seg_data in existing_index[\"segments\"]:\n # Ensure all required fields exist\n segment_dict = dict(seg_data)\n\n if \"content_type\" not in segment_dict:\n segment_dict[\"content_type\"] = \"general\"\n if \"keywords\" not in segment_dict:\n segment_dict[\"keywords\"] = []\n if \"relevance_scores\" not in segment_dict:\n segment_dict[\"relevance_scores\"] = {\n \"concept_analysis\": 0.5,\n \"algorithm_extraction\": 0.5,\n \"code_planning\": 0.5,\n }\n if \"section_path\" not in segment_dict:\n segment_dict[\"section_path\"] = segment_dict.get(\n \"title\", \"Unknown\"\n )\n\n segments_data.append(DocumentSegment(**segment_dict))\n\n existing_index[\"segments\"] = segments_data\n\n DOCUMENT_INDEXES[paper_dir] = DocumentIndex(**existing_index)\n return json.dumps(\n {\n \"status\": \"success\",\n \"message\": \"Using existing document analysis\",\n \"segments_dir\": segments_dir,\n \"total_segments\": existing_index[\"total_segments\"],\n },\n ensure_ascii=False,\n indent=2,\n )\n\n except Exception as e:\n logger.error(f\"Failed to load existing index: {e}\")\n logger.info(\"Will perform fresh analysis instead\")\n # Remove corrupted index file and continue with new analysis\n try:\n os.remove(index_file_path)\n except Exception as e:\n pass\n\n # Read document content\n with open(md_file_path, \"r\", encoding=\"utf-8\") as f:\n content = f.read()\n\n # Analyze document\n analyzer = DocumentAnalyzer()\n doc_type, confidence = analyzer.analyze_document_type(content)\n strategy = analyzer.detect_segmentation_strategy(content, doc_type)\n\n # Create segments\n segments = segmenter.segment_document(content, strategy)\n\n # Create document index\n document_index = DocumentIndex(\n document_path=md_file_path,\n document_type=doc_type,\n segmentation_strategy=strategy,\n total_segments=len(segments),\n total_chars=len(content),\n segments=segments,\n created_at=datetime.now().isoformat(),\n )\n\n # Save segments\n ensure_segments_dir_exists(segments_dir)\n\n # Save document index\n with open(index_file_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(\n asdict(document_index), f, ensure_ascii=False, indent=2, default=str\n )\n\n # Save individual segment files for fallback\n for segment in segments:\n segment_file_path = os.path.join(segments_dir, f\"segment_{segment.id}.md\")\n with open(segment_file_path, \"w\", encoding=\"utf-8\") as f:\n f.write(f\"# {segment.title}\\n\\n\")\n f.write(f\"**Content Type:** {segment.content_type}\\n\")\n f.write(f\"**Keywords:** {', '.join(segment.keywords[:10])}\\n\\n\")\n f.write(segment.content)\n\n # Store in memory\n DOCUMENT_INDEXES[paper_dir] = document_index\n\n logger.info(\n f\"Document segmentation completed: {len(segments)} segments created\"\n )\n\n return json.dumps(\n {\n \"status\": \"success\",\n \"message\": f\"Document analysis completed with {strategy} strategy\",\n \"document_type\": doc_type,\n \"segmentation_strategy\": strategy,\n \"segments_dir\": segments_dir,\n \"total_segments\": len(segments),\n \"total_chars\": len(content),\n },\n ensure_ascii=False,\n indent=2,\n )\n\n except Exception as e:\n logger.error(f\"Error in analyze_and_segment_document: {e}\")\n return json.dumps(\n {\"status\": \"error\", \"message\": f\"Failed to analyze document: {str(e)}\"},\n ensure_ascii=False,\n indent=2,\n )\n\n\n@mcp.tool()\nasync def read_document_segments(\n paper_dir: str,\n query_type: str,\n keywords: List[str] = None,\n max_segments: int = 3,\n max_total_chars: int = None,\n) -> str:\n \"\"\"\n Intelligently retrieve relevant document segments based on query type\n\n Args:\n paper_dir: Path to the paper directory\n query_type: Type of query - \"concept_analysis\", \"algorithm_extraction\", or \"code_planning\"\n keywords: Optional list of keywords to search for\n max_segments: Maximum number of segments to return\n max_total_chars: Maximum total characters to return\n\n Returns:\n JSON string with selected segments\n \"\"\"\n try:\n # Ensure document is analyzed\n if paper_dir not in DOCUMENT_INDEXES:\n segments_dir = get_segments_dir(paper_dir)\n index_file_path = os.path.join(segments_dir, \"document_index.json\")\n\n if os.path.exists(index_file_path):\n with open(index_file_path, \"r\", encoding=\"utf-8\") as f:\n index_data = json.load(f)\n # Convert dict back to DocumentIndex with backward compatibility\n segments_data = []\n for seg_data in index_data.get(\"segments\", []):\n # Ensure all required fields exist, provide default values\n segment_dict = dict(seg_data)\n\n # Compatibility handling: add missing fields\n if \"content_type\" not in segment_dict:\n segment_dict[\"content_type\"] = \"general\"\n if \"keywords\" not in segment_dict:\n segment_dict[\"keywords\"] = []\n if \"relevance_scores\" not in segment_dict:\n segment_dict[\"relevance_scores\"] = {\n \"concept_analysis\": 0.5,\n \"algorithm_extraction\": 0.5,\n \"code_planning\": 0.5,\n }\n if \"section_path\" not in segment_dict:\n segment_dict[\"section_path\"] = segment_dict.get(\n \"title\", \"Unknown\"\n )\n\n segment = DocumentSegment(**segment_dict)\n segments_data.append(segment)\n\n index_data[\"segments\"] = segments_data\n DOCUMENT_INDEXES[paper_dir] = DocumentIndex(**index_data)\n else:\n # Auto-analyze if not found\n await analyze_and_segment_document(paper_dir)\n\n document_index = DOCUMENT_INDEXES[paper_dir]\n\n # Dynamically calculate character limit\n if max_total_chars is None:\n max_total_chars = _calculate_adaptive_char_limit(document_index, query_type)\n\n # Score and rank segments with enhanced algorithm\n scored_segments = []\n for segment in document_index.segments:\n # Base relevance score (already enhanced in new system)\n relevance_score = segment.relevance_scores.get(query_type, 0.5)\n\n # Enhanced keyword matching with position weighting\n if keywords:\n keyword_score = _calculate_enhanced_keyword_score(segment, keywords)\n relevance_score += keyword_score\n\n # Content completeness bonus\n completeness_bonus = _calculate_completeness_bonus(segment, document_index)\n relevance_score += completeness_bonus\n\n scored_segments.append((segment, relevance_score))\n\n # Sort by enhanced relevance score\n scored_segments.sort(key=lambda x: x[1], reverse=True)\n\n # Intelligent segment selection with integrity preservation\n selected_segments = _select_segments_with_integrity(\n scored_segments, max_segments, max_total_chars, query_type\n )\n\n total_chars = sum(seg[\"char_count\"] for seg in selected_segments)\n\n logger.info(\n f\"Selected {len(selected_segments)} segments for {query_type} query\"\n )\n\n return json.dumps(\n {\n \"status\": \"success\",\n \"query_type\": query_type,\n \"keywords\": keywords or [],\n \"total_segments_available\": len(document_index.segments),\n \"segments_selected\": len(selected_segments),\n \"total_chars\": total_chars,\n \"max_chars_used\": max_total_chars,\n \"segments\": selected_segments,\n },\n ensure_ascii=False,\n indent=2,\n )\n\n except Exception as e:\n logger.error(f\"Error in read_document_segments: {e}\")\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"Failed to read document segments: {str(e)}\",\n },\n ensure_ascii=False,\n indent=2,\n )\n\n\n@mcp.tool()\nasync def get_document_overview(paper_dir: str) -> str:\n \"\"\"\n Get overview of document structure and available segments\n\n Args:\n paper_dir: Path to the paper directory\n\n Returns:\n JSON string with document overview\n \"\"\"\n try:\n # Ensure document is analyzed\n if paper_dir not in DOCUMENT_INDEXES:\n await analyze_and_segment_document(paper_dir)\n\n document_index = DOCUMENT_INDEXES[paper_dir]\n\n # Create overview\n segment_summaries = []\n for segment in document_index.segments:\n segment_summaries.append(\n {\n \"id\": segment.id,\n \"title\": segment.title,\n \"content_type\": segment.content_type,\n \"char_count\": segment.char_count,\n \"keywords\": segment.keywords[:5], # Top 5 keywords\n \"relevance_scores\": segment.relevance_scores,\n }\n )\n\n return json.dumps(\n {\n \"status\": \"success\",\n \"document_path\": document_index.document_path,\n \"document_type\": document_index.document_type,\n \"segmentation_strategy\": document_index.segmentation_strategy,\n \"total_segments\": document_index.total_segments,\n \"total_chars\": document_index.total_chars,\n \"created_at\": document_index.created_at,\n \"segments_overview\": segment_summaries,\n },\n ensure_ascii=False,\n indent=2,\n )\n\n except Exception as e:\n logger.error(f\"Error in get_document_overview: {e}\")\n return json.dumps(\n {\n \"status\": \"error\",\n \"message\": f\"Failed to get document overview: {str(e)}\",\n },\n ensure_ascii=False,\n indent=2,\n )\n\n\n# =============== Enhanced retrieval system helper methods ===============\n\n\ndef _calculate_adaptive_char_limit(\n document_index: DocumentIndex, query_type: str\n) -> int:\n \"\"\"Dynamically calculate character limit based on document complexity and query type\"\"\"\n base_limit = 6000\n\n # Adjust based on document type\n if document_index.document_type == \"research_paper\":\n base_limit = 10000\n elif document_index.document_type == \"algorithm_focused\":\n base_limit = 12000\n elif document_index.segmentation_strategy == \"algorithm_preserve_integrity\":\n base_limit = 15000\n\n # Adjust based on query type\n query_multipliers = {\n \"algorithm_extraction\": 1.5, # Algorithms need more context\n \"concept_analysis\": 1.2,\n \"code_planning\": 1.3,\n }\n\n multiplier = query_multipliers.get(query_type, 1.0)\n return int(base_limit * multiplier)\n\n\ndef _calculate_enhanced_keyword_score(\n segment: DocumentSegment, keywords: List[str]\n) -> float:\n \"\"\"Calculate enhanced keyword matching score\"\"\"\n score = 0.0\n content_lower = segment.content.lower()\n title_lower = segment.title.lower()\n\n for keyword in keywords:\n keyword_lower = keyword.lower()\n\n # Title matching has higher weight\n if keyword_lower in title_lower:\n score += 0.3\n\n # Content matching\n content_matches = content_lower.count(keyword_lower)\n if content_matches > 0:\n # Consider term frequency and position\n frequency_score = min(0.2, content_matches * 0.05)\n\n # Check if in important position (first 25% of content)\n early_content = content_lower[: len(content_lower) // 4]\n if keyword_lower in early_content:\n frequency_score += 0.1\n\n score += frequency_score\n\n return min(0.6, score) # Limit maximum bonus\n\n\ndef _calculate_completeness_bonus(\n segment: DocumentSegment, document_index: DocumentIndex\n) -> float:\n \"\"\"Calculate content completeness bonus\"\"\"\n bonus = 0.0\n\n # Completeness bonus for algorithm and formula content\n if segment.content_type in [\"algorithm\", \"formula\", \"merged\"]:\n bonus += 0.2\n\n # Long paragraphs usually contain more complete information\n if segment.char_count > 2000:\n bonus += 0.1\n elif segment.char_count > 4000:\n bonus += 0.15\n\n # High importance paragraph bonus\n if segment.relevance_scores.get(\"algorithm_extraction\", 0) > 0.8:\n bonus += 0.1\n\n return min(0.3, bonus)\n\n\ndef _select_segments_with_integrity(\n scored_segments: List[Tuple],\n max_segments: int,\n max_total_chars: int,\n query_type: str,\n) -> List[Dict]:\n \"\"\"Intelligently select segments while maintaining content integrity\"\"\"\n selected_segments = []\n total_chars = 0\n\n # First select the highest scoring segments\n for segment, score in scored_segments:\n if len(selected_segments) >= max_segments:\n break\n\n if total_chars + segment.char_count <= max_total_chars:\n selected_segments.append(\n {\n \"id\": segment.id,\n \"title\": segment.title,\n \"content\": segment.content,\n \"content_type\": segment.content_type,\n \"relevance_score\": score,\n \"char_count\": segment.char_count,\n }\n )\n total_chars += segment.char_count\n elif len(selected_segments) == 0:\n # If the first segment exceeds the limit, truncate but preserve it\n truncated_content = (\n segment.content[: max_total_chars - 200]\n + \"\\n\\n[Content truncated for length...]\"\n )\n selected_segments.append(\n {\n \"id\": segment.id,\n \"title\": segment.title,\n \"content\": truncated_content,\n \"content_type\": segment.content_type,\n \"relevance_score\": score,\n \"char_count\": len(truncated_content),\n }\n )\n break\n\n # If there's remaining space, try to add relevant small segments\n remaining_chars = max_total_chars - total_chars\n if remaining_chars > 500 and len(selected_segments) < max_segments:\n for segment, score in scored_segments[len(selected_segments) :]:\n if (\n segment.char_count <= remaining_chars\n and len(selected_segments) < max_segments\n ):\n selected_segments.append(\n {\n \"id\": segment.id,\n \"title\": segment.title,\n \"content\": segment.content,\n \"content_type\": segment.content_type,\n \"relevance_score\": score,\n \"char_count\": segment.char_count,\n }\n )\n remaining_chars -= segment.char_count\n\n return selected_segments\n\n\nif __name__ == \"__main__\":\n # Run the MCP server\n mcp.run()\n"
},
{
"path": "tools/git_command.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nGitHub Repository Downloader MCP Tool using FastMCP\n\"\"\"\n\nimport asyncio\nimport os\nimport re\nfrom typing import Dict, List, Optional\nfrom pathlib import Path\n\nfrom mcp.server import FastMCP\n\n# 创建 FastMCP 实例\nmcp = FastMCP(\"github-downloader\")\n\n\nclass GitHubURLExtractor:\n \"\"\"提取GitHub URL的工具类\"\"\"\n\n @staticmethod\n def extract_github_urls(text: str) -> List[str]:\n \"\"\"从文本中提取GitHub URLs\"\"\"\n patterns = [\n # 标准HTTPS URL\n r\"https?://github\\.com/[\\w\\-\\.]+/[\\w\\-\\.]+(?:\\.git)?\",\n # SSH URL\n r\"git@github\\.com:[\\w\\-\\.]+/[\\w\\-\\.]+(?:\\.git)?\",\n # 短格式 owner/repo - 更严格的匹配\n r\"(? Optional[str]:\n \"\"\"从文本中提取目标路径\"\"\"\n # 路径指示词模式\n patterns = [\n r'(?:to|into|in|at)\\s+(?:folder|directory|path)?\\s*[\"\\']?([^\\s\"\\']+)[\"\\']?',\n r'(?:save|download|clone)\\s+(?:to|into|at)\\s+[\"\\']?([^\\s\"\\']+)[\"\\']?',\n # 中文支持\n r'(?:到|在|保存到|下载到|克隆到)\\s*[\"\\']?([^\\s\"\\']+)[\"\\']?',\n ]\n\n for pattern in patterns:\n match = re.search(pattern, text, re.IGNORECASE)\n if match:\n path = match.group(1).strip(\"。，,.\")\n # 过滤掉通用词\n if path and path.lower() not in [\n \"here\",\n \"there\",\n \"current\",\n \"local\",\n \"这里\",\n \"当前\",\n \"本地\",\n ]:\n return path\n\n return None\n\n @staticmethod\n def infer_repo_name(url: str) -> str:\n \"\"\"从URL推断仓库名称\"\"\"\n url = url.rstrip(\".git\")\n if \"github.com\" in url:\n parts = url.split(\"/\")\n if len(parts) >= 2:\n return parts[-1]\n return \"repository\"\n\n\nasync def check_git_installed() -> bool:\n \"\"\"检查Git是否安装\"\"\"\n try:\n proc = await asyncio.create_subprocess_exec(\n \"git\",\n \"--version\",\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n )\n await proc.wait()\n return proc.returncode == 0\n except Exception:\n return False\n\n\nasync def clone_repository(repo_url: str, target_path: str) -> Dict[str, any]:\n \"\"\"执行git clone命令\"\"\"\n try:\n proc = await asyncio.create_subprocess_exec(\n \"git\",\n \"clone\",\n repo_url,\n target_path,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n )\n\n stdout, stderr = await proc.communicate()\n\n return {\n \"success\": proc.returncode == 0,\n \"stdout\": stdout.decode(\"utf-8\", errors=\"replace\"),\n \"stderr\": stderr.decode(\"utf-8\", errors=\"replace\"),\n \"returncode\": proc.returncode,\n }\n except Exception as e:\n return {\"success\": False, \"error\": str(e)}\n\n\n@mcp.tool()\nasync def download_github_repo(instruction: str) -> str:\n \"\"\"\n Download GitHub repositories from natural language instructions.\n\n Args:\n instruction: Natural language text containing GitHub URLs and optional target paths\n\n Returns:\n Status message about the download operation\n\n Examples:\n - \"Download https://github.com/openai/gpt-3\"\n - \"Clone microsoft/vscode to my-projects folder\"\n - \"Get https://github.com/facebook/react\"\n \"\"\"\n # 检查Git是否安装\n if not await check_git_installed():\n return \"❌ Error: Git is not installed or not in system PATH\"\n\n extractor = GitHubURLExtractor()\n\n # 提取GitHub URLs\n urls = extractor.extract_github_urls(instruction)\n if not urls:\n return \"❌ No GitHub URLs found in the instruction\"\n\n # 提取目标路径\n target_path = extractor.extract_target_path(instruction)\n\n # 下载仓库\n results = []\n for url in urls:\n try:\n # 准备目标路径\n if target_path:\n # 判断是否为绝对路径\n if os.path.isabs(target_path):\n # 如果是绝对路径，直接使用\n final_path = target_path\n # 如果目标路径是目录，添加仓库名\n if os.path.basename(target_path) == \"\" or target_path.endswith(\"/\"):\n final_path = os.path.join(\n target_path, extractor.infer_repo_name(url)\n )\n else:\n # 如果是相对路径，保持相对路径\n final_path = target_path\n # 如果目标路径是目录，添加仓库名\n if os.path.basename(target_path) == \"\" or target_path.endswith(\"/\"):\n final_path = os.path.join(\n target_path, extractor.infer_repo_name(url)\n )\n else:\n final_path = extractor.infer_repo_name(url)\n\n # 如果是相对路径，确保使用相对路径格式\n if not os.path.isabs(final_path):\n final_path = os.path.normpath(final_path)\n if final_path.startswith(\"/\"):\n final_path = final_path.lstrip(\"/\")\n\n # 确保父目录存在\n parent_dir = os.path.dirname(final_path)\n if parent_dir:\n os.makedirs(parent_dir, exist_ok=True)\n\n # 检查目标路径是否已存在\n if os.path.exists(final_path):\n results.append(\n f\"❌ Failed to download {url}: Target path already exists: {final_path}\"\n )\n continue\n\n # 执行克隆\n result = await clone_repository(url, final_path)\n\n if result[\"success\"]:\n msg = f\"✅ Successfully downloaded: {url}\\n\"\n msg += f\" Location: {final_path}\"\n if result.get(\"stdout\"):\n msg += f\"\\n {result['stdout'].strip()}\"\n else:\n msg = f\"❌ Failed to download: {url}\\n\"\n msg += f\" Error: {result.get('error', result.get('stderr', 'Unknown error'))}\"\n\n except Exception as e:\n msg = f\"❌ Failed to download: {url}\\n\"\n msg += f\" Error: {str(e)}\"\n\n results.append(msg)\n\n return \"\\n\\n\".join(results)\n\n\n@mcp.tool()\nasync def parse_github_urls(text: str) -> str:\n \"\"\"\n Extract GitHub URLs and target paths from text.\n\n Args:\n text: Text containing GitHub URLs\n\n Returns:\n Parsed GitHub URLs and target path information\n \"\"\"\n extractor = GitHubURLExtractor()\n\n urls = extractor.extract_github_urls(text)\n target_path = extractor.extract_target_path(text)\n\n content = \"📝 Parsed information:\\n\\n\"\n\n if urls:\n content += \"GitHub URLs found:\\n\"\n for url in urls:\n content += f\" • {url}\\n\"\n else:\n content += \"No GitHub URLs found\\n\"\n\n if target_path:\n content += f\"\\nTarget path: {target_path}\"\n else:\n content += \"\\nTarget path: Not specified (will use repository name)\"\n\n return content\n\n\n@mcp.tool()\nasync def git_clone(\n repo_url: str, target_path: Optional[str] = None, branch: Optional[str] = None\n) -> str:\n \"\"\"\n Clone a specific GitHub repository.\n\n Args:\n repo_url: GitHub repository URL\n target_path: Optional target directory path\n branch: Optional branch name to clone\n\n Returns:\n Status message about the clone operation\n \"\"\"\n # 检查Git是否安装\n if not await check_git_installed():\n return \"❌ Error: Git is not installed or not in system PATH\"\n\n # 准备目标路径\n if not target_path:\n extractor = GitHubURLExtractor()\n target_path = extractor.infer_repo_name(repo_url)\n\n # 转换为绝对路径\n if not os.path.isabs(target_path):\n target_path = str(Path.cwd() / target_path)\n\n # 检查目标路径\n if os.path.exists(target_path):\n return f\"❌ Error: Target path already exists: {target_path}\"\n\n # 构建命令\n cmd = [\"git\", \"clone\"]\n if branch:\n cmd.extend([\"-b\", branch])\n cmd.extend([repo_url, target_path])\n\n # 执行克隆\n try:\n proc = await asyncio.create_subprocess_exec(\n *cmd, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE\n )\n\n stdout, stderr = await proc.communicate()\n\n if proc.returncode == 0:\n result = \"✅ Successfully cloned repository\\n\"\n result += f\"Repository: {repo_url}\\n\"\n result += f\"Location: {target_path}\"\n if branch:\n result += f\"\\nBranch: {branch}\"\n return result\n else:\n return f\"❌ Clone failed\\nError: {stderr.decode('utf-8', errors='replace')}\"\n\n except Exception as e:\n return f\"❌ Clone failed\\nError: {str(e)}\"\n\n\n# 主程序入口\nif __name__ == \"__main__\":\n print(\"🚀 GitHub Repository Downloader MCP Tool\")\n print(\"📝 Starting server with FastMCP...\")\n print(\"\\nAvailable tools:\")\n print(\" • download_github_repo - Download repos from natural language\")\n print(\" • parse_github_urls - Extract GitHub URLs from text\")\n print(\" • git_clone - Clone a specific repository\")\n print(\"\")\n\n # 运行服务器\n mcp.run()\n"
},
{
"path": "tools/indexer_config.yaml",
"content": "# Code Indexer Configuration File\n# Configure various aspects of the code indexing process\n\n# Paths Configuration\npaths:\n code_base_path: \"D:/Documents/GitHub/Code-Agent/examples/input/paper1/code_base\"\n output_dir: \"D:/Documents/GitHub/Code-Agent/examples/input/paper1/indexes\"\n\n# File Analysis Settings\nfile_analysis:\n # Supported file extensions for analysis\n supported_extensions:\n - \".py\" # Python\n - \".js\" # JavaScript\n - \".ts\" # TypeScript\n - \".java\" # Java\n - \".cpp\" # C++\n - \".c\" # C\n - \".h\" # C Header\n - \".hpp\" # C++ Header\n - \".cs\" # C#\n - \".php\" # PHP\n - \".rb\" # Ruby\n - \".go\" # Go\n - \".rs\" # Rust\n - \".scala\" # Scala\n - \".kt\" # Kotlin\n - \".swift\" # Swift\n - \".r\" # R\n - \".sql\" # SQL\n - \".sh\" # Shell Script\n - \".bat\" # Batch File\n - \".ps1\" # PowerShell\n - \".yaml\" # YAML\n - \".yml\" # YAML\n - \".json\" # JSON\n - \".xml\" # XML\n - \".toml\" # TOML\n\n # Directories to skip during traversal\n skip_directories:\n - \"__pycache__\"\n - \"node_modules\"\n - \"target\"\n - \"build\"\n - \"dist\"\n - \"venv\"\n - \"env\"\n - \".git\"\n - \".svn\"\n - \".hg\"\n - \"coverage\"\n - \".pytest_cache\"\n - \".mypy_cache\"\n\n # Maximum file size to analyze (in bytes)\n max_file_size: 1048576 # 1MB\n\n # Maximum content length to send to LLM (in characters)\n max_content_length: 3000\n\n# LLM Configuration\nllm:\n # Model selection: \"anthropic\" or \"openai\"\n model_provider: \"openai\"\n\n # Request parameters\n max_tokens: 4000\n temperature: 0.3\n\n # System prompt for analysis\n system_prompt: \"You are a code analysis expert. Provide precise, structured analysis of code relationships and similarities.\"\n\n # Rate limiting (seconds between requests)\n request_delay: 0.1\n\n # Retry configuration\n max_retries: 3\n retry_delay: 1.0\n\n# Relationship Analysis Settings\nrelationships:\n # Minimum confidence score to include a relationship\n min_confidence_score: 0.3\n\n # High confidence threshold for reporting\n high_confidence_threshold: 0.7\n\n # Relationship types and their priorities\n relationship_types:\n direct_match: 1.0 # Direct implementation match\n partial_match: 0.8 # Partial functionality match\n reference: 0.6 # Reference or utility function\n utility: 0.4 # General utility or helper\n\n# Output Configuration\noutput:\n # JSON formatting options\n json_indent: 2\n ensure_ascii: false\n\n # Generate additional report files\n generate_summary: true\n generate_statistics: true\n\n # Include metadata in output\n include_metadata: true\n\n # File naming pattern (use {repo_name} placeholder)\n index_filename_pattern: \"{repo_name}_index.json\"\n summary_filename: \"indexing_summary.json\"\n stats_filename: \"indexing_statistics.json\"\n\n# Logging Configuration\nlogging:\n level: \"INFO\" # DEBUG, INFO, WARNING, ERROR\n log_to_file: true\n log_file: \"indexer.log\"\n log_format: \"%(asctime)s - %(name)s - %(levelname)s - %(message)s\"\n\n# Performance Settings\nperformance:\n # Enable concurrent processing of files within a repository\n enable_concurrent_analysis: true\n max_concurrent_files: 5\n\n # Memory optimization\n enable_content_caching: false\n max_cache_size: 100\n\n# Debug and Development Settings\ndebug:\n # Save raw LLM responses for debugging\n save_raw_responses: false\n raw_responses_dir: \"debug_responses\"\n\n # Verbose output during processing\n verbose_output: false\n\n # Skip LLM calls for testing (uses mock responses)\n mock_llm_responses: false\n"
},
{
"path": "tools/pdf_converter.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nPDF Converter Utility\n\nThis module provides functionality for converting various document formats to PDF,\nincluding Office documents (.doc, .docx, .ppt, .pptx, .xls, .xlsx) and text files (.txt, .md).\n\nRequirements:\n- LibreOffice for Office document conversion\n- ReportLab for text-to-PDF conversion\n\"\"\"\n\nfrom __future__ import annotations\n\nimport argparse\nimport logging\nimport subprocess\nimport tempfile\nimport shutil\nimport platform\nimport os\nfrom pathlib import Path\nfrom typing import Union, Optional, Dict, Any, List\n\n\nclass PDFConverter:\n \"\"\"\n PDF conversion utility class.\n\n Provides methods to convert Office documents and text files to PDF format.\n \"\"\"\n\n # Define supported file formats\n OFFICE_FORMATS = {\".doc\", \".docx\", \".ppt\", \".pptx\", \".xls\", \".xlsx\"}\n TEXT_FORMATS = {\".txt\", \".md\"}\n\n # Class-level logger\n logger = logging.getLogger(__name__)\n\n def __init__(self) -> None:\n \"\"\"Initialize the PDF converter.\"\"\"\n pass\n\n @staticmethod\n def find_libreoffice_windows() -> Optional[str]:\n \"\"\"\n Find LibreOffice installation on Windows.\n \n Returns:\n Path to soffice.exe if found, None otherwise\n \"\"\"\n if platform.system() != \"Windows\":\n return None\n \n # Common LibreOffice installation paths on Windows\n possible_paths = [\n r\"C:\\Program Files\\LibreOffice\\program\\soffice.exe\",\n r\"C:\\Program Files (x86)\\LibreOffice\\program\\soffice.exe\",\n ]\n \n # Also check PROGRAMFILES environment variables\n program_files = os.environ.get(\"PROGRAMFILES\")\n program_files_x86 = os.environ.get(\"PROGRAMFILES(X86)\")\n \n if program_files:\n possible_paths.append(os.path.join(program_files, \"LibreOffice\", \"program\", \"soffice.exe\"))\n if program_files_x86:\n possible_paths.append(os.path.join(program_files_x86, \"LibreOffice\", \"program\", \"soffice.exe\"))\n \n # Check each path\n for path in possible_paths:\n if os.path.exists(path):\n return path\n \n return None\n\n @staticmethod\n def convert_office_to_pdf(\n doc_path: Union[str, Path], output_dir: Optional[str] = None\n ) -> Path:\n \"\"\"\n Convert Office document (.doc, .docx, .ppt, .pptx, .xls, .xlsx) to PDF.\n Requires LibreOffice to be installed.\n\n Args:\n doc_path: Path to the Office document file\n output_dir: Output directory for the PDF file\n\n Returns:\n Path to the generated PDF file\n \"\"\"\n try:\n # Convert to Path object for easier handling\n doc_path = Path(doc_path)\n if not doc_path.exists():\n raise FileNotFoundError(f\"Office document does not exist: {doc_path}\")\n\n name_without_suff = doc_path.stem\n\n # Prepare output directory\n if output_dir:\n base_output_dir = Path(output_dir)\n else:\n # Generate unique folder name with timestamp to avoid conflicts\n import time\n timestamp = int(time.time())\n folder_name = f\"paper_{timestamp}\"\n \n # Save to workspace instead of temp directory\n workspace_base = Path(os.getcwd()) / \"deepcode_lab\" / \"papers\"\n workspace_base.mkdir(parents=True, exist_ok=True)\n base_output_dir = workspace_base / folder_name\n\n base_output_dir.mkdir(parents=True, exist_ok=True)\n\n # Check if LibreOffice is available\n libreoffice_available = False\n working_libreoffice_cmd: Optional[str] = None\n\n # Prepare subprocess parameters to hide console window on Windows\n subprocess_kwargs: Dict[str, Any] = {\n \"capture_output\": True,\n \"check\": True,\n \"timeout\": 10,\n \"encoding\": \"utf-8\",\n \"errors\": \"ignore\",\n }\n\n # Hide console window on Windows\n if platform.system() == \"Windows\":\n # Use CREATE_NO_WINDOW to prevent console window from appearing\n subprocess_kwargs[\"creationflags\"] = 0x08000000\n # Also configure startupinfo to hide window\n startupinfo = subprocess.STARTUPINFO()\n startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW\n startupinfo.wShowWindow = subprocess.SW_HIDE\n subprocess_kwargs[\"startupinfo\"] = startupinfo\n\n # On Windows, try to find LibreOffice in standard installation paths first\n # Don't run --version check on Windows as it can cause window/hanging issues\n if platform.system() == \"Windows\":\n windows_path = PDFConverter.find_libreoffice_windows()\n if windows_path:\n libreoffice_available = True\n working_libreoffice_cmd = windows_path\n logging.info(f\"LibreOffice detected at {windows_path}\")\n\n # On non-Windows systems, try standard commands\n if not libreoffice_available and platform.system() != \"Windows\":\n try:\n result = subprocess.run(\n [\"libreoffice\", \"--version\"], **subprocess_kwargs\n )\n libreoffice_available = True\n working_libreoffice_cmd = \"libreoffice\"\n logging.info(f\"LibreOffice detected: {result.stdout.strip()}\") # type: ignore\n except (\n subprocess.CalledProcessError,\n FileNotFoundError,\n subprocess.TimeoutExpired,\n ):\n pass\n\n # Try alternative commands for LibreOffice (non-Windows)\n if not libreoffice_available and platform.system() != \"Windows\":\n for cmd in [\"soffice\", \"libreoffice\"]:\n try:\n result = subprocess.run([cmd, \"--version\"], **subprocess_kwargs)\n libreoffice_available = True\n working_libreoffice_cmd = cmd\n logging.info(\n f\"LibreOffice detected with command '{cmd}': {result.stdout.strip()}\" # type: ignore\n )\n break\n except (\n subprocess.CalledProcessError,\n FileNotFoundError,\n subprocess.TimeoutExpired,\n ):\n continue\n\n if not libreoffice_available:\n raise RuntimeError(\n \"LibreOffice is required for Office document conversion but was not found.\\n\"\n \"Please install LibreOffice:\\n\"\n \"- Windows: Download from https://www.libreoffice.org/download/download/\\n\"\n \"- macOS: brew install --cask libreoffice\\n\"\n \"- Ubuntu/Debian: sudo apt-get install libreoffice\\n\"\n \"- CentOS/RHEL: sudo yum install libreoffice\\n\"\n \"Alternatively, convert the document to PDF manually.\"\n )\n\n # Create temporary directory for PDF conversion\n with tempfile.TemporaryDirectory() as temp_dir:\n temp_path = Path(temp_dir)\n\n # Convert to PDF using LibreOffice\n logging.info(f\"Converting {doc_path.name} to PDF using LibreOffice...\")\n\n # Use the working LibreOffice command first, then try alternatives if it fails\n commands_to_try = [working_libreoffice_cmd]\n \n # Add alternative commands based on what was found\n if platform.system() == \"Windows\" and working_libreoffice_cmd:\n # If we're using the full Windows path, also try standard commands\n if \"Program Files\" in working_libreoffice_cmd:\n commands_to_try.extend([\"soffice\", \"libreoffice\"])\n elif working_libreoffice_cmd == \"libreoffice\":\n commands_to_try.append(\"soffice\")\n else:\n commands_to_try.append(\"libreoffice\")\n\n conversion_successful = False\n for cmd in commands_to_try:\n if cmd is None:\n continue\n try:\n convert_cmd = [\n cmd,\n \"--headless\",\n \"--convert-to\",\n \"pdf\",\n \"--outdir\",\n str(temp_path),\n str(doc_path),\n ]\n\n # Prepare conversion subprocess parameters\n convert_subprocess_kwargs: Dict[str, Any] = {\n \"capture_output\": True,\n \"text\": True,\n \"timeout\": 60, # 60 second timeout\n \"encoding\": \"utf-8\",\n \"errors\": \"ignore\",\n }\n\n # Hide console window on Windows\n if platform.system() == \"Windows\":\n convert_subprocess_kwargs[\"creationflags\"] = 0x08000000\n # Also configure startupinfo to hide window\n startupinfo = subprocess.STARTUPINFO()\n startupinfo.dwFlags |= subprocess.STARTF_USESHOWWINDOW\n startupinfo.wShowWindow = subprocess.SW_HIDE\n convert_subprocess_kwargs[\"startupinfo\"] = startupinfo\n\n result = subprocess.run(\n convert_cmd, **convert_subprocess_kwargs\n )\n\n if result.returncode == 0: # type: ignore\n conversion_successful = True\n logging.info(\n f\"Successfully converted {doc_path.name} to PDF\"\n )\n break\n else:\n logging.warning(\n f\"LibreOffice command '{cmd}' failed: {result.stderr}\" # type: ignore\n )\n except subprocess.TimeoutExpired:\n logging.warning(f\"LibreOffice command '{cmd}' timed out\")\n except Exception as e:\n logging.error(\n f\"LibreOffice command '{cmd}' failed with exception: {e}\"\n )\n\n if not conversion_successful:\n raise RuntimeError(\n f\"LibreOffice conversion failed for {doc_path.name}. \"\n f\"Please check if the file is corrupted or try converting manually.\"\n )\n\n # Find the generated PDF\n pdf_files = list(temp_path.glob(\"*.pdf\"))\n if not pdf_files:\n raise RuntimeError(\n f\"PDF conversion failed for {doc_path.name} - no PDF file generated. \"\n f\"Please check LibreOffice installation or try manual conversion.\"\n )\n\n pdf_path = pdf_files[0]\n logging.info(\n f\"Generated PDF: {pdf_path.name} ({pdf_path.stat().st_size} bytes)\"\n )\n\n # Validate the generated PDF\n if pdf_path.stat().st_size < 100: # Very small file, likely empty\n raise RuntimeError(\n \"Generated PDF appears to be empty or corrupted. \"\n \"Original file may have issues or LibreOffice conversion failed.\"\n )\n\n # Copy PDF to final output directory\n final_pdf_path = base_output_dir / f\"{name_without_suff}.pdf\"\n shutil.copy2(pdf_path, final_pdf_path)\n \n print(f\"✅ PDF saved to: {final_pdf_path}\")\n print(f\" File size: {final_pdf_path.stat().st_size} bytes\")\n print(f\" Parent folder: {base_output_dir}\")\n\n return final_pdf_path\n\n except Exception as e:\n logging.error(f\"Error in convert_office_to_pdf: {str(e)}\")\n raise\n\n @staticmethod\n def convert_text_to_pdf(\n text_path: Union[str, Path], output_dir: Optional[str] = None\n ) -> Path:\n \"\"\"\n Convert text file (.txt, .md) to PDF using ReportLab with full markdown support.\n\n Args:\n text_path: Path to the text file\n output_dir: Output directory for the PDF file\n\n Returns:\n Path to the generated PDF file\n \"\"\"\n try:\n text_path = Path(text_path)\n if not text_path.exists():\n raise FileNotFoundError(f\"Text file does not exist: {text_path}\")\n\n # Supported text formats\n supported_text_formats = {\".txt\", \".md\"}\n if text_path.suffix.lower() not in supported_text_formats:\n raise ValueError(f\"Unsupported text format: {text_path.suffix}\")\n\n # Read the text content\n try:\n with open(text_path, \"r\", encoding=\"utf-8\") as f:\n text_content = f.read()\n except UnicodeDecodeError:\n # Try with different encodings\n for encoding in [\"gbk\", \"latin-1\", \"cp1252\"]:\n try:\n with open(text_path, \"r\", encoding=encoding) as f:\n text_content = f.read()\n logging.info(f\"Successfully read file with {encoding} encoding\")\n break\n except UnicodeDecodeError:\n continue\n else:\n raise RuntimeError(\n f\"Could not decode text file {text_path.name} with any supported encoding\"\n )\n\n # Prepare output directory\n if output_dir:\n base_output_dir = Path(output_dir)\n else:\n # Generate unique folder name with timestamp to avoid conflicts\n import time\n timestamp = int(time.time())\n folder_name = f\"paper_{timestamp}\"\n \n # Save to workspace instead of temp directory\n workspace_base = Path(os.getcwd()) / \"deepcode_lab\" / \"papers\"\n workspace_base.mkdir(parents=True, exist_ok=True)\n base_output_dir = workspace_base / folder_name\n\n base_output_dir.mkdir(parents=True, exist_ok=True)\n pdf_path = base_output_dir / f\"{text_path.stem}.pdf\"\n\n # Convert text to PDF\n logging.info(f\"Converting {text_path.name} to PDF...\")\n\n try:\n from reportlab.lib.pagesizes import A4\n from reportlab.platypus import SimpleDocTemplate, Paragraph, Spacer\n from reportlab.lib.styles import getSampleStyleSheet, ParagraphStyle\n from reportlab.lib.units import inch\n from reportlab.pdfbase import pdfmetrics\n\n # Create PDF document\n doc = SimpleDocTemplate(\n str(pdf_path),\n pagesize=A4,\n leftMargin=inch,\n rightMargin=inch,\n topMargin=inch,\n bottomMargin=inch,\n )\n\n # Get styles\n styles = getSampleStyleSheet()\n normal_style = styles[\"Normal\"]\n heading_style = styles[\"Heading1\"]\n\n # Try to register a font that supports Chinese characters\n try:\n # Try to use system fonts that support Chinese\n system = platform.system()\n if system == \"Windows\":\n # Try common Windows fonts\n for font_name in [\"SimSun\", \"SimHei\", \"Microsoft YaHei\"]:\n try:\n from reportlab.pdfbase.cidfonts import (\n UnicodeCIDFont,\n )\n\n pdfmetrics.registerFont(UnicodeCIDFont(font_name)) # type: ignore\n normal_style.fontName = font_name\n heading_style.fontName = font_name\n break\n except Exception:\n continue\n elif system == \"Darwin\": # macOS\n for font_name in [\"STSong-Light\", \"STHeiti\"]:\n try:\n from reportlab.pdfbase.cidfonts import (\n UnicodeCIDFont,\n )\n\n pdfmetrics.registerFont(UnicodeCIDFont(font_name)) # type: ignore\n normal_style.fontName = font_name\n heading_style.fontName = font_name\n break\n except Exception:\n continue\n except Exception:\n pass # Use default fonts if Chinese font setup fails\n\n # Build content\n story = []\n\n # Handle markdown or plain text\n if text_path.suffix.lower() == \".md\":\n # Handle markdown content - simplified implementation\n lines = text_content.split(\"\\n\")\n for line in lines:\n line = line.strip()\n if not line:\n story.append(Spacer(1, 12))\n continue\n\n # Headers\n if line.startswith(\"#\"):\n level = len(line) - len(line.lstrip(\"#\"))\n header_text = line.lstrip(\"#\").strip()\n if header_text:\n header_style = ParagraphStyle(\n name=f\"Heading{level}\",\n parent=heading_style,\n fontSize=max(16 - level, 10),\n spaceAfter=8,\n spaceBefore=16 if level <= 2 else 12,\n )\n story.append(Paragraph(header_text, header_style))\n else:\n # Regular text\n processed_line = PDFConverter._process_inline_markdown(line)\n story.append(Paragraph(processed_line, normal_style))\n story.append(Spacer(1, 6))\n else:\n # Handle plain text files (.txt)\n logging.info(\n f\"Processing plain text file with {len(text_content)} characters...\"\n )\n\n # Split text into lines and process each line\n lines = text_content.split(\"\\n\")\n line_count = 0\n\n for line in lines:\n line = line.rstrip()\n line_count += 1\n\n # Empty lines\n if not line.strip():\n story.append(Spacer(1, 6))\n continue\n\n # Regular text lines\n # Escape special characters for ReportLab\n safe_line = (\n line.replace(\"&\", \"&\")\n .replace(\"<\", \"<\")\n .replace(\">\", \">\")\n )\n\n # Create paragraph\n story.append(Paragraph(safe_line, normal_style))\n story.append(Spacer(1, 3))\n\n logging.info(f\"Added {line_count} lines to PDF\")\n\n # If no content was added, add a placeholder\n if not story:\n story.append(Paragraph(\"(Empty text file)\", normal_style))\n\n # Build PDF\n doc.build(story)\n logging.info(\n f\"Successfully converted {text_path.name} to PDF ({pdf_path.stat().st_size / 1024:.1f} KB)\"\n )\n\n except ImportError:\n raise RuntimeError(\n \"reportlab is required for text-to-PDF conversion. \"\n \"Please install it using: pip install reportlab\"\n )\n except Exception as e:\n raise RuntimeError(\n f\"Failed to convert text file {text_path.name} to PDF: {str(e)}\"\n )\n\n # Validate the generated PDF\n if not pdf_path.exists() or pdf_path.stat().st_size < 100:\n raise RuntimeError(\n f\"PDF conversion failed for {text_path.name} - generated PDF is empty or corrupted.\"\n )\n\n print(f\"✅ PDF saved to: {pdf_path}\")\n print(f\" File size: {pdf_path.stat().st_size} bytes\")\n print(f\" Parent folder: {base_output_dir}\")\n \n return pdf_path\n\n except Exception as e:\n logging.error(f\"Error in convert_text_to_pdf: {str(e)}\")\n raise\n\n @staticmethod\n def _process_inline_markdown(text: str) -> str:\n \"\"\"\n Process inline markdown formatting (bold, italic, code, links)\n\n Args:\n text: Raw text with markdown formatting\n\n Returns:\n Text with ReportLab markup\n \"\"\"\n import re\n\n # Escape special characters for ReportLab\n text = text.replace(\"&\", \"&\").replace(\"<\", \"<\").replace(\">\", \">\")\n\n # Bold text: **text** or __text__\n text = re.sub(r\"\\*\\*(.*?)\\*\\*\", r\"\\1\", text)\n text = re.sub(r\"__(.*?)__\", r\"\\1\", text)\n\n # Italic text: *text* or _text_ (but not in the middle of words)\n text = re.sub(r\"(?\\1\", text)\n text = re.sub(r\"(?\\1\", text)\n\n # Inline code: `code`\n text = re.sub(\n r\"`([^`]+?)`\",\n r'\\1',\n text,\n )\n\n # Links: [text](url) - convert to text with URL annotation\n def link_replacer(match):\n link_text = match.group(1)\n url = match.group(2)\n return f'{link_text}'\n\n text = re.sub(r\"\\[([^\\]]+?)\\]\\(([^)]+?)\\)\", link_replacer, text)\n\n # Strikethrough: ~~text~~\n text = re.sub(r\"~~(.*?)~~\", r\"\\1\", text)\n\n return text\n\n def convert_to_pdf(\n self,\n file_path: Union[str, Path],\n output_dir: Optional[str] = None,\n ) -> Path:\n \"\"\"\n Convert document to PDF based on file extension\n\n Args:\n file_path: Path to the file to be converted\n output_dir: Output directory path\n\n Returns:\n Path to the generated PDF file\n \"\"\"\n # Convert to Path object\n file_path = Path(file_path)\n if not file_path.exists():\n raise FileNotFoundError(f\"File does not exist: {file_path}\")\n\n # Get file extension\n ext = file_path.suffix.lower()\n\n # Choose appropriate conversion method based on file type\n if ext in self.OFFICE_FORMATS:\n return self.convert_office_to_pdf(file_path, output_dir)\n elif ext in self.TEXT_FORMATS:\n return self.convert_text_to_pdf(file_path, output_dir)\n else:\n raise ValueError(\n f\"Unsupported file format: {ext}. \"\n f\"Supported formats: {', '.join(self.OFFICE_FORMATS | self.TEXT_FORMATS)}\"\n )\n\n def check_dependencies(self) -> dict:\n \"\"\"\n Check if required dependencies are available\n\n Returns:\n dict: Dictionary with dependency check results\n \"\"\"\n results = {\n \"libreoffice\": False,\n \"reportlab\": False,\n }\n\n # Check LibreOffice\n # On Windows, just check if the executable exists (don't run it to avoid window issues)\n if platform.system() == \"Windows\":\n windows_path = PDFConverter.find_libreoffice_windows()\n if windows_path:\n results[\"libreoffice\"] = True\n else:\n # On non-Windows systems, try running the version command\n try:\n subprocess_kwargs: Dict[str, Any] = {\n \"capture_output\": True,\n \"text\": True,\n \"check\": True,\n \"timeout\": 5,\n \"encoding\": \"utf-8\",\n \"errors\": \"ignore\",\n }\n\n try:\n subprocess.run([\"libreoffice\", \"--version\"], **subprocess_kwargs)\n results[\"libreoffice\"] = True\n except (subprocess.CalledProcessError, FileNotFoundError, subprocess.TimeoutExpired):\n try:\n subprocess.run([\"soffice\", \"--version\"], **subprocess_kwargs)\n results[\"libreoffice\"] = True\n except (subprocess.CalledProcessError, FileNotFoundError, subprocess.TimeoutExpired):\n pass\n except Exception:\n # If any unexpected error occurs during LibreOffice check, silently pass\n pass\n\n # Check ReportLab\n import importlib.util\n\n if importlib.util.find_spec(\"reportlab\") is not None:\n results[\"reportlab\"] = True\n\n return results\n\n\ndef main():\n \"\"\"\n Main function to run the PDF converter from command line\n \"\"\"\n parser = argparse.ArgumentParser(description=\"Convert documents to PDF format\")\n parser.add_argument(\"file_path\", nargs=\"?\", help=\"Path to the document to convert\")\n parser.add_argument(\"--output\", \"-o\", help=\"Output directory path\")\n parser.add_argument(\n \"--check\",\n action=\"store_true\",\n help=\"Check dependencies installation\",\n )\n parser.add_argument(\n \"--verbose\", \"-v\", action=\"store_true\", help=\"Enable verbose logging\"\n )\n\n args = parser.parse_args()\n\n # Configure logging\n log_level = logging.INFO if args.verbose else logging.WARNING\n logging.basicConfig(\n level=log_level,\n format=\"%(asctime)s - %(levelname)s - %(message)s\",\n datefmt=\"%Y-%m-%d %H:%M:%S\",\n )\n\n # Initialize converter\n converter = PDFConverter()\n\n # Check dependencies if requested\n if args.check:\n print(\"🔍 Checking dependencies...\")\n deps = converter.check_dependencies()\n\n print(\n f\"LibreOffice: {'✅ Available' if deps['libreoffice'] else '❌ Not found'}\"\n )\n print(f\"ReportLab: {'✅ Available' if deps['reportlab'] else '❌ Not found'}\")\n\n if not deps[\"libreoffice\"]:\n print(\"\\n📋 To install LibreOffice:\")\n print(\" - Windows: Download from https://www.libreoffice.org/\")\n print(\" - macOS: brew install --cask libreoffice\")\n print(\" - Ubuntu/Debian: sudo apt-get install libreoffice\")\n\n if not deps[\"reportlab\"]:\n print(\"\\n📋 To install ReportLab:\")\n print(\" pip install reportlab\")\n\n return 0\n\n # If not checking dependencies, file_path is required\n if not args.file_path:\n parser.error(\"file_path is required when not using --check\")\n\n try:\n # Convert the file\n output_pdf = converter.convert_to_pdf(\n file_path=args.file_path,\n output_dir=args.output,\n )\n\n print(f\"✅ Successfully converted to PDF: {output_pdf}\")\n print(f\"📄 File size: {output_pdf.stat().st_size / 1024:.1f} KB\")\n\n except Exception as e:\n print(f\"❌ Error: {str(e)}\")\n return 1\n\n return 0\n\n\nif __name__ == \"__main__\":\n exit(main())\n"
},
{
"path": "tools/pdf_downloader.py",
"content": "#!/usr/bin/env python3\n\"\"\"\nSmart PDF Downloader MCP Tool\n\nA standardized MCP tool using FastMCP for intelligent file downloading and document conversion.\nSupports natural language instructions for downloading files from URLs, moving local files,\nand automatic conversion to Markdown format with image extraction.\n\nFeatures:\n- Natural language instruction parsing\n- URL and local path extraction\n- Automatic document conversion (PDF, DOCX, PPTX, HTML, etc.)\n- Image extraction and preservation\n- Multi-format support with fallback options\n\"\"\"\n\nimport os\nimport re\nimport aiohttp\nimport aiofiles\nimport shutil\nimport sys\nimport io\nfrom typing import List, Dict, Optional, Any\nfrom urllib.parse import urlparse, unquote\nfrom datetime import datetime\n\nfrom mcp.server import FastMCP\n\n# Docling imports for document conversion\ntry:\n from docling.document_converter import DocumentConverter\n from docling.datamodel.base_models import InputFormat\n from docling.datamodel.pipeline_options import PdfPipelineOptions\n from docling.document_converter import PdfFormatOption\n\n DOCLING_AVAILABLE = True\nexcept ImportError:\n DOCLING_AVAILABLE = False\n print(\n \"Warning: docling package not available. Document conversion will be disabled.\"\n )\n\n# Fallback PDF text extraction\ntry:\n import PyPDF2\n\n PYPDF2_AVAILABLE = True\nexcept ImportError:\n PYPDF2_AVAILABLE = False\n print(\n \"Warning: PyPDF2 package not available. Fallback PDF extraction will be disabled.\"\n )\n\n# 设置标准输出编码为UTF-8\nif sys.stdout.encoding != \"utf-8\":\n try:\n if hasattr(sys.stdout, \"reconfigure\"):\n sys.stdout.reconfigure(encoding=\"utf-8\")\n sys.stderr.reconfigure(encoding=\"utf-8\")\n else:\n sys.stdout = io.TextIOWrapper(sys.stdout.detach(), encoding=\"utf-8\")\n sys.stderr = io.TextIOWrapper(sys.stderr.detach(), encoding=\"utf-8\")\n except Exception as e:\n print(f\"Warning: Could not set UTF-8 encoding: {e}\")\n\n# 创建 FastMCP 实例\nmcp = FastMCP(\"smart-pdf-downloader\")\n\n\n# 辅助函数\ndef format_success_message(action: str, details: Dict[str, Any]) -> str:\n \"\"\"格式化成功消息\"\"\"\n return f\"✅ {action}\\n\" + \"\\n\".join(f\" {k}: {v}\" for k, v in details.items())\n\n\ndef format_error_message(action: str, error: str) -> str:\n \"\"\"格式化错误消息\"\"\"\n return f\"❌ {action}\\n Error: {error}\"\n\n\ndef format_warning_message(action: str, warning: str) -> str:\n \"\"\"格式化警告消息\"\"\"\n return f\"⚠️ {action}\\n Warning: {warning}\"\n\n\nasync def perform_document_conversion(\n file_path: str, extract_images: bool = True\n) -> Optional[str]:\n \"\"\"\n 执行文档转换的共用逻辑\n\n Args:\n file_path: 文件路径\n extract_images: 是否提取图片\n\n Returns:\n 转换信息字符串，如果没有转换则返回None\n \"\"\"\n if not file_path:\n return None\n\n conversion_msg = \"\"\n\n # 首先尝试使用简单的PDF转换器（对于PDF文件）\n # 检查文件是否实际为PDF（无论扩展名如何）\n is_pdf_file = False\n if PYPDF2_AVAILABLE:\n try:\n with open(file_path, \"rb\") as f:\n header = f.read(8)\n is_pdf_file = header.startswith(b\"%PDF\")\n except Exception:\n is_pdf_file = file_path.lower().endswith(\".pdf\")\n\n if is_pdf_file and PYPDF2_AVAILABLE:\n try:\n simple_converter = SimplePdfConverter()\n conversion_result = simple_converter.convert_pdf_to_markdown(file_path)\n if conversion_result[\"success\"]:\n conversion_msg = \"\\n [INFO] PDF converted to Markdown (PyPDF2)\"\n conversion_msg += (\n f\"\\n Markdown file: {conversion_result['output_file']}\"\n )\n conversion_msg += (\n f\"\\n Conversion time: {conversion_result['duration']:.2f} seconds\"\n )\n conversion_msg += (\n f\"\\n Pages extracted: {conversion_result['pages_extracted']}\"\n )\n\n else:\n conversion_msg = f\"\\n [WARNING] PDF conversion failed: {conversion_result['error']}\"\n except Exception as conv_error:\n conversion_msg = f\"\\n [WARNING] PDF conversion error: {str(conv_error)}\"\n\n # 如果简单转换失败，尝试使用docling（支持图片提取）\n # if not conversion_success and DOCLING_AVAILABLE:\n # try:\n # converter = DoclingConverter()\n # if converter.is_supported_format(file_path):\n # conversion_result = converter.convert_to_markdown(\n # file_path, extract_images=extract_images\n # )\n # if conversion_result[\"success\"]:\n # conversion_msg = (\n # \"\\n [INFO] Document converted to Markdown (docling)\"\n # )\n # conversion_msg += (\n # f\"\\n Markdown file: {conversion_result['output_file']}\"\n # )\n # conversion_msg += f\"\\n Conversion time: {conversion_result['duration']:.2f} seconds\"\n # if conversion_result.get(\"images_extracted\", 0) > 0:\n # conversion_msg += f\"\\n Images extracted: {conversion_result['images_extracted']}\"\n # images_dir = os.path.join(\n # os.path.dirname(conversion_result[\"output_file\"]), \"images\"\n # )\n # conversion_msg += f\"\\n Images saved to: {images_dir}\"\n # else:\n # conversion_msg = f\"\\n [WARNING] Docling conversion failed: {conversion_result['error']}\"\n # except Exception as conv_error:\n # conversion_msg = (\n # f\"\\n [WARNING] Docling conversion error: {str(conv_error)}\"\n # )\n\n return conversion_msg if conversion_msg else None\n\n\ndef format_file_operation_result(\n operation: str,\n source: str,\n destination: str,\n result: Dict[str, Any],\n conversion_msg: Optional[str] = None,\n) -> str:\n \"\"\"\n 格式化文件操作结果的共用逻辑\n\n Args:\n operation: 操作类型 (\"download\", \"copy\", 或 \"move\")\n source: 源文件/URL\n destination: 目标路径\n result: 操作结果字典\n conversion_msg: 转换消息\n\n Returns:\n 格式化的结果消息\n \"\"\"\n if result[\"success\"]:\n size_mb = result[\"size\"] / (1024 * 1024)\n\n # 处理不同操作类型的动词形式\n if operation == \"copy\":\n operation_verb = \"copied\"\n elif operation == \"download\":\n operation_verb = \"downloaded\"\n else: # move\n operation_verb = \"moved\"\n\n msg = f\"[SUCCESS] Successfully {operation_verb}: {source}\\n\"\n\n if operation == \"download\":\n msg += f\" File: {destination}\\n\"\n msg += f\" Size: {size_mb:.2f} MB\\n\"\n msg += f\" Time: {result['duration']:.2f} seconds\\n\"\n speed_mb = result.get(\"speed\", 0) / (1024 * 1024)\n msg += f\" Speed: {speed_mb:.2f} MB/s\"\n else: # copy or move\n msg += f\" To: {destination}\\n\"\n msg += f\" Size: {size_mb:.2f} MB\\n\"\n msg += f\" Time: {result['duration']:.2f} seconds\"\n if operation == \"copy\":\n msg += \"\\n Note: Original file preserved\"\n\n if conversion_msg:\n msg += conversion_msg\n\n return msg\n else:\n return f\"[ERROR] Failed to {operation}: {source}\\n Error: {result.get('error', 'Unknown error')}\"\n\n\nclass LocalPathExtractor:\n \"\"\"本地路径提取器\"\"\"\n\n @staticmethod\n def is_local_path(path: str) -> bool:\n \"\"\"判断是否为本地路径\"\"\"\n path = path.strip(\"\\\"'\")\n\n # 检查是否为URL\n if re.match(r\"^https?://\", path, re.IGNORECASE) or re.match(\n r\"^ftp://\", path, re.IGNORECASE\n ):\n return False\n\n # 路径指示符\n path_indicators = [os.path.sep, \"/\", \"\\\\\", \"~\", \".\", \"..\"]\n has_extension = bool(os.path.splitext(path)[1])\n\n if any(indicator in path for indicator in path_indicators) or has_extension:\n expanded_path = os.path.expanduser(path)\n return os.path.exists(expanded_path) or any(\n indicator in path for indicator in path_indicators\n )\n\n return False\n\n @staticmethod\n def extract_local_paths(text: str) -> List[str]:\n \"\"\"从文本中提取本地文件路径\"\"\"\n patterns = [\n r'\"([^\"]+)\"',\n r\"'([^']+)'\",\n r\"(?:^|\\s)((?:[~./\\\\]|[A-Za-z]:)?(?:[^/\\\\\\s]+[/\\\\])*[^/\\\\\\s]+\\.[A-Za-z0-9]+)(?:\\s|$)\",\n r\"(?:^|\\s)((?:~|\\.{1,2})?/[^\\s]+)(?:\\s|$)\",\n r\"(?:^|\\s)([A-Za-z]:[/\\\\][^\\s]+)(?:\\s|$)\",\n r\"(?:^|\\s)(\\.{1,2}[/\\\\][^\\s]+)(?:\\s|$)\",\n ]\n\n local_paths = []\n potential_paths = []\n\n for pattern in patterns:\n matches = re.findall(pattern, text, re.MULTILINE)\n potential_paths.extend(matches)\n\n for path in potential_paths:\n path = path.strip()\n if path and LocalPathExtractor.is_local_path(path):\n expanded_path = os.path.expanduser(path)\n if expanded_path not in local_paths:\n local_paths.append(expanded_path)\n\n return local_paths\n\n\nclass URLExtractor:\n \"\"\"URL提取器\"\"\"\n\n URL_PATTERNS = [\n r\"https?://(?:[-\\w.]|(?:%[\\da-fA-F]{2}))+(?:/(?:[-\\w._~!$&\\'()*+,;=:@]|%[\\da-fA-F]{2})*)*(?:\\?(?:[-\\w._~!$&\\'()*+,;=:@/?]|%[\\da-fA-F]{2})*)?(?:#(?:[-\\w._~!$&\\'()*+,;=:@/?]|%[\\da-fA-F]{2})*)?\",\n r\"ftp://(?:[-\\w.]|(?:%[\\da-fA-F]{2}))+(?:/(?:[-\\w._~!$&\\'()*+,;=:@]|%[\\da-fA-F]{2})*)*\",\n r\"(? str:\n \"\"\"将arXiv网页链接转换为PDF下载链接\"\"\"\n # 匹配arXiv论文ID的正则表达式\n arxiv_pattern = r\"arxiv\\.org/abs/(\\d+\\.\\d+)(?:v\\d+)?\"\n match = re.search(arxiv_pattern, url, re.IGNORECASE)\n if match:\n paper_id = match.group(1)\n return f\"https://arxiv.org/pdf/{paper_id}.pdf\"\n return url\n\n @classmethod\n def extract_urls(cls, text: str) -> List[str]:\n \"\"\"从文本中提取URL\"\"\"\n urls = []\n\n # 首先处理特殊情况：@开头的URL\n at_url_pattern = r\"@(https?://[^\\s]+)\"\n at_matches = re.findall(at_url_pattern, text, re.IGNORECASE)\n for match in at_matches:\n # 处理arXiv链接\n url = cls.convert_arxiv_url(match.rstrip(\"/\"))\n urls.append(url)\n\n # 然后使用原有的正则模式\n for pattern in cls.URL_PATTERNS:\n matches = re.findall(pattern, text, re.IGNORECASE)\n for match in matches:\n # 处理可能缺少协议的URL\n if not match.startswith((\"http://\", \"https://\", \"ftp://\")):\n # 检查是否是 www 开头\n if match.startswith(\"www.\"):\n match = \"https://\" + match\n else:\n # 其他情况也添加 https\n match = \"https://\" + match\n\n # 处理arXiv链接\n url = cls.convert_arxiv_url(match.rstrip(\"/\"))\n urls.append(url)\n\n # 去重并保持顺序\n seen = set()\n unique_urls = []\n for url in urls:\n if url not in seen:\n seen.add(url)\n unique_urls.append(url)\n\n return unique_urls\n\n @staticmethod\n def infer_filename_from_url(url: str) -> str:\n \"\"\"从URL推断文件名\"\"\"\n parsed = urlparse(url)\n path = unquote(parsed.path)\n\n # 从路径中提取文件名\n filename = os.path.basename(path)\n\n # 特殊处理：arxiv PDF链接\n if \"arxiv.org\" in parsed.netloc and \"/pdf/\" in path:\n if filename:\n # 检查是否已经有合适的文件扩展名\n if not filename.lower().endswith((\".pdf\", \".doc\", \".docx\", \".txt\")):\n filename = f\"{filename}.pdf\"\n else:\n path_parts = [p for p in path.split(\"/\") if p]\n if path_parts and path_parts[-1]:\n filename = f\"{path_parts[-1]}.pdf\"\n else:\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n filename = f\"arxiv_paper_{timestamp}.pdf\"\n\n # 如果没有文件名或没有扩展名，生成一个\n elif not filename or \".\" not in filename:\n # 尝试从URL生成有意义的文件名\n domain = parsed.netloc.replace(\"www.\", \"\").replace(\".\", \"_\")\n timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n\n # 尝试根据路径推断文件类型\n if not path or path == \"/\":\n filename = f\"{domain}_{timestamp}.html\"\n else:\n # 使用路径的最后一部分\n path_parts = [p for p in path.split(\"/\") if p]\n if path_parts:\n filename = f\"{path_parts[-1]}_{timestamp}\"\n else:\n filename = f\"{domain}_{timestamp}\"\n\n # 如果还是没有扩展名，根据路径推断\n if \".\" not in filename:\n # 根据路径中的关键词推断文件类型\n if \"/pdf/\" in path.lower() or path.lower().endswith(\"pdf\"):\n filename += \".pdf\"\n elif any(\n ext in path.lower() for ext in [\"/doc/\", \"/word/\", \".docx\"]\n ):\n filename += \".docx\"\n elif any(\n ext in path.lower()\n for ext in [\"/ppt/\", \"/powerpoint/\", \".pptx\"]\n ):\n filename += \".pptx\"\n elif any(ext in path.lower() for ext in [\"/csv/\", \".csv\"]):\n filename += \".csv\"\n elif any(ext in path.lower() for ext in [\"/zip/\", \".zip\"]):\n filename += \".zip\"\n else:\n filename += \".html\"\n\n return filename\n\n\nclass PathExtractor:\n \"\"\"路径提取器\"\"\"\n\n @staticmethod\n def extract_target_path(text: str) -> Optional[str]:\n \"\"\"从文本中提取目标路径\"\"\"\n patterns = [\n r'(?:save|download|store|put|place|write|copy|move)\\s+(?:to|into|in|at)\\s+[\"\\']?([^\\s\"\\']+)[\"\\']?',\n r'(?:to|into|in|at)\\s+(?:folder|directory|dir|path|location)\\s*[\"\\']?([^\\s\"\\']+)[\"\\']?',\n r'(?:destination|target|output)\\s*(?:is|:)?\\s*[\"\\']?([^\\s\"\\']+)[\"\\']?',\n r'(?:保存|下载|存储|放到|写入|复制|移动)(?:到|至|去)\\s*[\"\\']?([^\\s\"\\']+)[\"\\']?',\n r'(?:到|在|至)\\s*[\"\\']?([^\\s\"\\']+)[\"\\']?\\s*(?:文件夹|目录|路径|位置)',\n ]\n\n filter_words = {\n \"here\",\n \"there\",\n \"current\",\n \"local\",\n \"this\",\n \"that\",\n \"这里\",\n \"那里\",\n \"当前\",\n \"本地\",\n \"这个\",\n \"那个\",\n }\n\n for pattern in patterns:\n match = re.search(pattern, text, re.IGNORECASE)\n if match:\n path = match.group(1).strip(\"。，,.、\")\n if path and path.lower() not in filter_words:\n return path\n\n return None\n\n\nclass SimplePdfConverter:\n \"\"\"简单的PDF转换器，使用PyPDF2提取文本\"\"\"\n\n def convert_pdf_to_markdown(\n self, input_file: str, output_file: Optional[str] = None\n ) -> Dict[str, Any]:\n \"\"\"\n 使用PyPDF2将PDF转换为Markdown格式\n\n Args:\n input_file: 输入PDF文件路径\n output_file: 输出Markdown文件路径（可选）\n\n Returns:\n 转换结果字典\n \"\"\"\n if not PYPDF2_AVAILABLE:\n return {\"success\": False, \"error\": \"PyPDF2 package is not available\"}\n\n try:\n # 检查输入文件是否存在\n if not os.path.exists(input_file):\n return {\n \"success\": False,\n \"error\": f\"Input file not found: {input_file}\",\n }\n\n # 如果没有指定输出文件，自动生成\n if not output_file:\n base_name = os.path.splitext(input_file)[0]\n output_file = f\"{base_name}.md\"\n\n # 确保输出目录存在\n output_dir = os.path.dirname(output_file)\n if output_dir:\n os.makedirs(output_dir, exist_ok=True)\n\n # 执行转换\n start_time = datetime.now()\n\n # 读取PDF文件\n with open(input_file, \"rb\") as file:\n pdf_reader = PyPDF2.PdfReader(file)\n text_content = []\n\n # 提取每页文本\n for page_num, page in enumerate(pdf_reader.pages, 1):\n text = page.extract_text()\n if text.strip():\n text_content.append(f\"## Page {page_num}\\n\\n{text.strip()}\\n\\n\")\n\n # 生成Markdown内容\n markdown_content = f\"# Extracted from {os.path.basename(input_file)}\\n\\n\"\n markdown_content += f\"*Total pages: {len(pdf_reader.pages)}*\\n\\n\"\n markdown_content += \"---\\n\\n\"\n markdown_content += \"\".join(text_content)\n\n # 保存到文件\n with open(output_file, \"w\", encoding=\"utf-8\") as f:\n f.write(markdown_content)\n\n # 计算转换时间\n duration = (datetime.now() - start_time).total_seconds()\n\n # 获取文件大小\n input_size = os.path.getsize(input_file)\n output_size = os.path.getsize(output_file)\n\n return {\n \"success\": True,\n \"input_file\": input_file,\n \"output_file\": output_file,\n \"input_size\": input_size,\n \"output_size\": output_size,\n \"duration\": duration,\n \"markdown_content\": markdown_content,\n \"pages_extracted\": len(pdf_reader.pages),\n }\n\n except Exception as e:\n return {\n \"success\": False,\n \"input_file\": input_file,\n \"error\": f\"Conversion failed: {str(e)}\",\n }\n\n\nclass DoclingConverter:\n \"\"\"文档转换器，使用docling将文档转换为Markdown格式，支持图片提取\"\"\"\n\n def __init__(self):\n if not DOCLING_AVAILABLE:\n raise ImportError(\n \"docling package is not available. Please install it first.\"\n )\n\n # 配置PDF处理选项\n pdf_pipeline_options = PdfPipelineOptions()\n pdf_pipeline_options.do_ocr = False # 暂时禁用OCR以避免认证问题\n pdf_pipeline_options.do_table_structure = False # 暂时禁用表格结构识别\n\n # 创建文档转换器（使用基础模式）\n try:\n self.converter = DocumentConverter(\n format_options={\n InputFormat.PDF: PdfFormatOption(\n pipeline_options=pdf_pipeline_options\n )\n }\n )\n except Exception:\n # 如果失败，尝试更简单的配置\n self.converter = DocumentConverter()\n\n def is_supported_format(self, file_path: str) -> bool:\n \"\"\"检查文件格式是否支持转换\"\"\"\n if not DOCLING_AVAILABLE:\n return False\n\n supported_extensions = {\".pdf\", \".docx\", \".pptx\", \".html\", \".md\", \".txt\"}\n file_extension = os.path.splitext(file_path)[1].lower()\n return file_extension in supported_extensions\n\n def is_url(self, path: str) -> bool:\n \"\"\"检查路径是否为URL\"\"\"\n try:\n result = urlparse(path)\n return result.scheme in (\"http\", \"https\")\n except Exception:\n return False\n\n def extract_images(self, doc, output_dir: str) -> Dict[str, str]:\n \"\"\"\n 提取文档中的图片并保存到本地\n\n Args:\n doc: docling文档对象\n output_dir: 输出目录\n\n Returns:\n 图片ID到本地文件路径的映射\n \"\"\"\n images_dir = os.path.join(output_dir, \"images\")\n os.makedirs(images_dir, exist_ok=True)\n image_map = {} # docling图片id -> 本地文件名\n\n try:\n # 获取文档中的图片\n images = getattr(doc, \"images\", [])\n\n for idx, img in enumerate(images):\n try:\n # 获取图片格式，默认为png\n ext = getattr(img, \"format\", None) or \"png\"\n if ext.lower() not in [\"png\", \"jpg\", \"jpeg\", \"gif\", \"bmp\", \"webp\"]:\n ext = \"png\"\n\n # 生成文件名\n filename = f\"image_{idx+1}.{ext}\"\n filepath = os.path.join(images_dir, filename)\n\n # 保存图片数据\n img_data = getattr(img, \"data\", None)\n if img_data:\n with open(filepath, \"wb\") as f:\n f.write(img_data)\n\n # 计算相对路径\n rel_path = os.path.relpath(filepath, output_dir)\n img_id = getattr(img, \"id\", str(idx + 1))\n image_map[img_id] = rel_path\n\n except Exception as img_error:\n print(f\"Warning: Failed to extract image {idx+1}: {img_error}\")\n continue\n\n except Exception as e:\n print(f\"Warning: Failed to extract images: {e}\")\n\n return image_map\n\n def process_markdown_with_images(\n self, markdown_content: str, image_map: Dict[str, str]\n ) -> str:\n \"\"\"\n 处理Markdown内容，替换图片占位符为实际的图片路径\n\n Args:\n markdown_content: 原始Markdown内容\n image_map: 图片ID到本地路径的映射\n\n Returns:\n 处理后的Markdown内容\n \"\"\"\n\n def replace_img(match):\n img_id = match.group(1)\n if img_id in image_map:\n return f\"\"\n else:\n return match.group(0)\n\n # 替换docling的图片占位符\n processed_content = re.sub(\n r\"!\\[Image\\]\\(docling://image/([^)]+)\\)\", replace_img, markdown_content\n )\n\n return processed_content\n\n def convert_to_markdown(\n self,\n input_file: str,\n output_file: Optional[str] = None,\n extract_images: bool = True,\n ) -> Dict[str, Any]:\n \"\"\"\n 将文档转换为Markdown格式，支持图片提取\n\n Args:\n input_file: 输入文件路径或URL\n output_file: 输出Markdown文件路径（可选）\n extract_images: 是否提取图片（默认True）\n\n Returns:\n 转换结果字典\n \"\"\"\n if not DOCLING_AVAILABLE:\n return {\"success\": False, \"error\": \"docling package is not available\"}\n\n try:\n # 检查输入文件（如果不是URL）\n if not self.is_url(input_file):\n if not os.path.exists(input_file):\n return {\n \"success\": False,\n \"error\": f\"Input file not found: {input_file}\",\n }\n\n # 检查文件格式是否支持\n if not self.is_supported_format(input_file):\n return {\n \"success\": False,\n \"error\": f\"Unsupported file format: {os.path.splitext(input_file)[1]}\",\n }\n else:\n # 对于URL，检查是否为支持的格式\n if not input_file.lower().endswith(\n (\".pdf\", \".docx\", \".pptx\", \".html\", \".md\", \".txt\")\n ):\n return {\n \"success\": False,\n \"error\": f\"Unsupported URL format: {input_file}\",\n }\n\n # 如果没有指定输出文件，自动生成\n if not output_file:\n if self.is_url(input_file):\n # 从URL生成文件名\n filename = URLExtractor.infer_filename_from_url(input_file)\n base_name = os.path.splitext(filename)[0]\n else:\n base_name = os.path.splitext(input_file)[0]\n output_file = f\"{base_name}.md\"\n\n # 确保输出目录存在\n output_dir = os.path.dirname(output_file) or \".\"\n os.makedirs(output_dir, exist_ok=True)\n\n # 执行转换\n start_time = datetime.now()\n result = self.converter.convert(input_file)\n doc = result.document\n\n # 提取图片（如果启用）\n image_map = {}\n images_extracted = 0\n if extract_images:\n image_map = self.extract_images(doc, output_dir)\n images_extracted = len(image_map)\n\n # 获取Markdown内容\n markdown_content = doc.export_to_markdown()\n\n # 处理图片占位符\n if extract_images and image_map:\n markdown_content = self.process_markdown_with_images(\n markdown_content, image_map\n )\n\n # 保存到文件\n with open(output_file, \"w\", encoding=\"utf-8\") as f:\n f.write(markdown_content)\n\n # 计算转换时间\n duration = (datetime.now() - start_time).total_seconds()\n\n # 获取文件大小\n if self.is_url(input_file):\n input_size = 0 # URL无法直接获取大小\n else:\n input_size = os.path.getsize(input_file)\n output_size = os.path.getsize(output_file)\n\n return {\n \"success\": True,\n \"input_file\": input_file,\n \"output_file\": output_file,\n \"input_size\": input_size,\n \"output_size\": output_size,\n \"duration\": duration,\n \"markdown_content\": markdown_content,\n \"images_extracted\": images_extracted,\n \"image_map\": image_map,\n }\n\n except Exception as e:\n return {\n \"success\": False,\n \"input_file\": input_file,\n \"error\": f\"Conversion failed: {str(e)}\",\n }\n\n\nasync def check_url_accessible(url: str) -> Dict[str, Any]:\n \"\"\"检查URL是否可访问\"\"\"\n try:\n timeout = aiohttp.ClientTimeout(total=10)\n async with aiohttp.ClientSession(timeout=timeout) as session:\n async with session.head(url, allow_redirects=True) as response:\n return {\n \"accessible\": response.status < 400,\n \"status\": response.status,\n \"content_type\": response.headers.get(\"Content-Type\", \"\"),\n \"content_length\": response.headers.get(\"Content-Length\", 0),\n }\n except Exception:\n return {\n \"accessible\": False,\n \"status\": 0,\n \"content_type\": \"\",\n \"content_length\": 0,\n }\n\n\nasync def download_file(url: str, destination: str) -> Dict[str, Any]:\n \"\"\"下载单个文件\"\"\"\n start_time = datetime.now()\n chunk_size = 8192\n\n try:\n timeout = aiohttp.ClientTimeout(total=300) # 5分钟超时\n async with aiohttp.ClientSession(timeout=timeout) as session:\n async with session.get(url) as response:\n # 检查响应状态\n response.raise_for_status()\n\n # 获取文件信息\n content_type = response.headers.get(\n \"Content-Type\", \"application/octet-stream\"\n )\n\n # 确保目标目录存在\n parent_dir = os.path.dirname(destination)\n if parent_dir:\n os.makedirs(parent_dir, exist_ok=True)\n\n # 下载文件\n downloaded = 0\n async with aiofiles.open(destination, \"wb\") as file:\n async for chunk in response.content.iter_chunked(chunk_size):\n await file.write(chunk)\n downloaded += len(chunk)\n\n # 计算下载时间\n duration = (datetime.now() - start_time).total_seconds()\n\n return {\n \"success\": True,\n \"url\": url,\n \"destination\": destination,\n \"size\": downloaded,\n \"content_type\": content_type,\n \"duration\": duration,\n \"speed\": downloaded / duration if duration > 0 else 0,\n }\n\n except aiohttp.ClientError as e:\n return {\n \"success\": False,\n \"url\": url,\n \"destination\": destination,\n \"error\": f\"Network error: {str(e)}\",\n }\n except Exception as e:\n return {\n \"success\": False,\n \"url\": url,\n \"destination\": destination,\n \"error\": f\"Download error: {str(e)}\",\n }\n\n\nasync def move_local_file(source_path: str, destination: str) -> Dict[str, Any]:\n \"\"\"复制本地文件到目标位置（保留原文件）\"\"\"\n start_time = datetime.now()\n\n try:\n # 检查源文件是否存在\n if not os.path.exists(source_path):\n return {\n \"success\": False,\n \"source\": source_path,\n \"destination\": destination,\n \"error\": f\"Source file not found: {source_path}\",\n }\n\n # 获取源文件信息\n source_size = os.path.getsize(source_path)\n\n # 确保目标目录存在\n parent_dir = os.path.dirname(destination)\n if parent_dir:\n os.makedirs(parent_dir, exist_ok=True)\n\n # 执行复制操作（保留原文件，防止数据丢失）\n shutil.copy2(source_path, destination)\n\n # 计算操作时间\n duration = (datetime.now() - start_time).total_seconds()\n\n return {\n \"success\": True,\n \"source\": source_path,\n \"destination\": destination,\n \"size\": source_size,\n \"duration\": duration,\n \"operation\": \"copy\", # 改为copy\n }\n\n except Exception as e:\n return {\n \"success\": False,\n \"source\": source_path,\n \"destination\": destination,\n \"error\": f\"Copy error: {str(e)}\",\n }\n\n\n@mcp.tool()\nasync def download_files(instruction: str) -> str:\n \"\"\"\n Download files from URLs or move local files mentioned in natural language instructions.\n\n Args:\n instruction: Natural language instruction containing URLs/local paths and optional destination paths\n\n Returns:\n Status message about the download/move operations\n\n Examples:\n - \"Download https://example.com/file.pdf to documents folder\"\n - \"Move /home/user/file.pdf to documents folder\"\n - \"Please get https://raw.githubusercontent.com/user/repo/main/data.csv and save it to ~/downloads\"\n - \"移动 ~/Desktop/report.docx 到 /tmp/documents/\"\n - \"Download www.example.com/report.xlsx\"\n \"\"\"\n urls = URLExtractor.extract_urls(instruction)\n local_paths = LocalPathExtractor.extract_local_paths(instruction)\n\n if not urls and not local_paths:\n return format_error_message(\n \"Failed to parse instruction\",\n \"No downloadable URLs or movable local files found\",\n )\n\n target_path = PathExtractor.extract_target_path(instruction)\n\n # 处理文件\n results = []\n\n # 处理URL下载\n for url in urls:\n try:\n # 推断文件名\n filename = URLExtractor.infer_filename_from_url(url)\n\n # 构建完整的目标路径\n if target_path:\n # 处理路径\n if target_path.startswith(\"~\"):\n target_path = os.path.expanduser(target_path)\n\n # 确保使用相对路径（如果不是绝对路径）\n if not os.path.isabs(target_path):\n target_path = os.path.normpath(target_path)\n\n # 判断是文件路径还是目录路径\n if os.path.splitext(target_path)[1]: # 有扩展名，是文件\n destination = target_path\n else: # 是目录\n destination = os.path.join(target_path, filename)\n else:\n # 默认下载到当前目录\n destination = filename\n\n # 检查文件是否已存在\n if os.path.exists(destination):\n results.append(\n f\"[WARNING] Skipped {url}: File already exists at {destination}\"\n )\n continue\n\n # 先检查URL是否可访问\n check_result = await check_url_accessible(url)\n if not check_result[\"accessible\"]:\n results.append(\n f\"[ERROR] Failed to access {url}: HTTP {check_result['status'] or 'Connection failed'}\"\n )\n continue\n\n # 执行下载\n result = await download_file(url, destination)\n\n # 执行转换（如果成功下载）\n conversion_msg = None\n if result[\"success\"]:\n conversion_msg = await perform_document_conversion(\n destination, extract_images=True\n )\n\n # 格式化结果\n msg = format_file_operation_result(\n \"download\", url, destination, result, conversion_msg\n )\n\n except Exception as e:\n msg = f\"[ERROR] Failed to download: {url}\\n\"\n msg += f\" Error: {str(e)}\"\n\n results.append(msg)\n\n # 处理本地文件移动\n for local_path in local_paths:\n try:\n # 获取文件名\n filename = os.path.basename(local_path)\n\n # 构建完整的目标路径\n if target_path:\n # 处理路径\n if target_path.startswith(\"~\"):\n target_path = os.path.expanduser(target_path)\n\n # 确保使用相对路径（如果不是绝对路径）\n if not os.path.isabs(target_path):\n target_path = os.path.normpath(target_path)\n\n # 判断是文件路径还是目录路径\n if os.path.splitext(target_path)[1]: # 有扩展名，是文件\n destination = target_path\n else: # 是目录\n destination = os.path.join(target_path, filename)\n else:\n # 默认移动到当前目录\n destination = filename\n\n # 检查目标文件是否已存在\n if os.path.exists(destination):\n results.append(\n f\"[WARNING] Skipped {local_path}: File already exists at {destination}\"\n )\n continue\n\n # 执行复制（保留原文件）\n result = await move_local_file(local_path, destination)\n\n # 执行转换（如果成功复制）\n conversion_msg = None\n if result[\"success\"]:\n conversion_msg = await perform_document_conversion(\n destination, extract_images=True\n )\n\n # 格式化结果\n msg = format_file_operation_result(\n \"copy\", local_path, destination, result, conversion_msg\n )\n\n except Exception as e:\n msg = f\"[ERROR] Failed to copy: {local_path}\\n\"\n msg += f\" Error: {str(e)}\"\n\n results.append(msg)\n\n return \"\\n\\n\".join(results)\n\n\n@mcp.tool()\nasync def parse_download_urls(text: str) -> str:\n \"\"\"\n Extract URLs, local paths and target paths from text without downloading or moving.\n\n Args:\n text: Text containing URLs, local paths and optional destination paths\n\n Returns:\n Parsed URLs, local paths and target path information\n \"\"\"\n urls = URLExtractor.extract_urls(text)\n local_paths = LocalPathExtractor.extract_local_paths(text)\n target_path = PathExtractor.extract_target_path(text)\n\n content = \"📋 Parsed file operation information:\\n\\n\"\n\n if urls:\n content += f\"🔗 URLs found ({len(urls)}):\\n\"\n for i, url in enumerate(urls, 1):\n filename = URLExtractor.infer_filename_from_url(url)\n content += f\" {i}. {url}\\n 📄 Filename: {filename}\\n\"\n else:\n content += \"🔗 No URLs found\\n\"\n\n if local_paths:\n content += f\"\\n📁 Local files found ({len(local_paths)}):\\n\"\n for i, path in enumerate(local_paths, 1):\n exists = os.path.exists(path)\n content += f\" {i}. {path}\\n\"\n content += f\" ✅ Exists: {'Yes' if exists else 'No'}\\n\"\n if exists:\n size_mb = os.path.getsize(path) / (1024 * 1024)\n content += f\" 📊 Size: {size_mb:.2f} MB\\n\"\n else:\n content += \"\\n📁 No local files found\\n\"\n\n if target_path:\n content += f\"\\n🎯 Target path: {target_path}\"\n if target_path.startswith(\"~\"):\n content += f\"\\n (Expanded: {os.path.expanduser(target_path)})\"\n else:\n content += \"\\n🎯 Target path: Not specified (will use current directory)\"\n\n return content\n\n\n@mcp.tool()\nasync def download_file_to(\n url: str, destination: Optional[str] = None, filename: Optional[str] = None\n) -> str:\n \"\"\"\n Download a specific file with detailed options.\n\n Args:\n url: URL to download from\n destination: Target directory or full file path (optional)\n filename: Specific filename to use (optional, ignored if destination is a full file path)\n\n Returns:\n Status message about the download operation\n \"\"\"\n # 确定文件名\n\n url = URLExtractor.extract_urls(url)[0]\n\n if not filename:\n filename = URLExtractor.infer_filename_from_url(url)\n\n if not filename:\n filename = URLExtractor.infer_filename_from_url(url)\n else:\n name_source, extension_source = os.path.splitext(\n os.path.basename(URLExtractor.infer_filename_from_url(url))\n )\n name_destination, extension_destination = os.path.splitext(\n os.path.basename(filename)\n )\n if extension_source:\n filename = name_destination + extension_source\n else:\n filename = name_destination + extension_destination\n\n # 确定完整路径\n if destination:\n # 展开用户目录\n if destination.startswith(\"~\"):\n destination = os.path.expanduser(destination)\n\n # 检查是否是完整文件路径\n if os.path.splitext(destination)[1]: # 有扩展名\n target_path = destination\n else: # 是目录\n target_path = os.path.join(destination, filename)\n else:\n target_path = filename\n\n # 确保使用相对路径（如果不是绝对路径）\n if not os.path.isabs(target_path):\n target_path = os.path.normpath(target_path)\n\n # 检查文件是否已存在\n if os.path.exists(target_path):\n return format_error_message(\n \"Download aborted\", f\"File already exists at {target_path}\"\n )\n\n # 先检查URL\n check_result = await check_url_accessible(url)\n if not check_result[\"accessible\"]:\n return format_error_message(\n \"Cannot access URL\",\n f\"{url} (HTTP {check_result['status'] or 'Connection failed'})\",\n )\n\n # 显示下载信息\n size_mb = (\n int(check_result[\"content_length\"]) / (1024 * 1024)\n if check_result[\"content_length\"]\n else 0\n )\n msg = \"[INFO] Downloading file:\\n\"\n msg += f\" URL: {url}\\n\"\n msg += f\" Target: {target_path}\\n\"\n if size_mb > 0:\n msg += f\" Expected size: {size_mb:.2f} MB\\n\"\n msg += \"\\n\"\n\n # 执行下载\n result = await download_file(url, target_path)\n\n # 执行转换（如果成功下载）\n conversion_msg = None\n if result[\"success\"]:\n conversion_msg = await perform_document_conversion(\n target_path, extract_images=True\n )\n\n # 添加下载信息前缀\n actual_size_mb = result[\"size\"] / (1024 * 1024)\n speed_mb = result[\"speed\"] / (1024 * 1024)\n info_msg = \"[SUCCESS] Download completed!\\n\"\n info_msg += f\" Saved to: {target_path}\\n\"\n info_msg += f\" Size: {actual_size_mb:.2f} MB\\n\"\n info_msg += f\" Duration: {result['duration']:.2f} seconds\\n\"\n info_msg += f\" Speed: {speed_mb:.2f} MB/s\\n\"\n info_msg += f\" Type: {result['content_type']}\"\n\n if conversion_msg:\n info_msg += conversion_msg\n\n return msg + info_msg\n else:\n return msg + f\"[ERROR] Download failed!\\n Error: {result['error']}\"\n\n\n@mcp.tool()\nasync def move_file_to(\n source: str, destination: Optional[str] = None, filename: Optional[str] = None\n) -> str:\n \"\"\"\n Copy a local file to a new location (preserves original file).\n\n Note: Despite the name \"move_file_to\", this tool COPIES the file to preserve the original.\n This prevents data loss during file processing workflows.\n\n Args:\n source: Source file path to copy\n destination: Target directory or full file path (optional)\n filename: Specific filename to use (optional, ignored if destination is a full file path)\n\n Returns:\n Status message about the copy operation\n \"\"\"\n # 展开源路径\n if source.startswith(\"~\"):\n source = os.path.expanduser(source)\n\n # 检查源文件是否存在\n if not os.path.exists(source):\n return format_error_message(\"Copy aborted\", f\"Source file not found: {source}\")\n\n # 确定文件名\n if not filename:\n filename = os.path.basename(source)\n else:\n name_source, extension_source = os.path.splitext(os.path.basename(source))\n name_destination, extension_destination = os.path.splitext(\n os.path.basename(filename)\n )\n if extension_source:\n filename = name_destination + extension_source\n else:\n filename = name_destination + extension_destination\n\n # 确定完整路径\n if destination:\n # 展开用户目录\n if destination.startswith(\"~\"):\n destination = os.path.expanduser(destination)\n\n # 检查是否是完整文件路径\n if os.path.splitext(destination)[1]: # 有扩展名\n target_path = destination\n else: # 是目录\n target_path = os.path.join(destination, filename)\n\n else:\n target_path = filename\n\n # 确保使用相对路径（如果不是绝对路径）\n if not os.path.isabs(target_path):\n target_path = os.path.normpath(target_path)\n\n # 检查目标文件是否已存在\n if os.path.exists(target_path):\n return f\"[ERROR] Target file already exists: {target_path}\"\n\n # 显示复制信息\n source_size_mb = os.path.getsize(source) / (1024 * 1024)\n msg = \"[INFO] Copying file (original preserved):\\n\"\n msg += f\" Source: {source}\\n\"\n msg += f\" Target: {target_path}\\n\"\n msg += f\" Size: {source_size_mb:.2f} MB\\n\"\n msg += \"\\n\"\n\n # 执行复制（保留原文件）\n result = await move_local_file(source, target_path)\n\n # 执行转换（如果成功复制）\n conversion_msg = None\n if result[\"success\"]:\n conversion_msg = await perform_document_conversion(\n target_path, extract_images=True\n )\n\n # 添加复制信息前缀\n info_msg = \"[SUCCESS] File copied successfully (original preserved)!\\n\"\n info_msg += f\" From: {source}\\n\"\n info_msg += f\" To: {target_path}\\n\"\n info_msg += f\" Duration: {result['duration']:.2f} seconds\"\n\n if conversion_msg:\n info_msg += conversion_msg\n\n return msg + info_msg\n else:\n return msg + f\"[ERROR] Copy failed!\\n Error: {result['error']}\"\n\n\n# @mcp.tool()\n# async def convert_document_to_markdown(\n# file_path: str, output_path: Optional[str] = None, extract_images: bool = True\n# ) -> str:\n# \"\"\"\n# Convert a document to Markdown format with image extraction support.\n\n# Supports both local files and URLs. Uses docling for advanced conversion with image extraction,\n# or falls back to PyPDF2 for simple PDF text extraction.\n\n# Args:\n# file_path: Path to the input document file or URL (supports PDF, DOCX, PPTX, HTML, TXT, MD)\n# output_path: Path for the output Markdown file (optional, auto-generated if not provided)\n# extract_images: Whether to extract images from the document (default: True)\n\n# Returns:\n# Status message about the conversion operation with preview of converted content\n\n# Examples:\n# - \"convert_document_to_markdown('paper.pdf')\"\n# - \"convert_document_to_markdown('https://example.com/doc.pdf', 'output.md')\"\n# - \"convert_document_to_markdown('presentation.pptx', extract_images=False)\"\n# \"\"\"\n# # 检查是否为URL\n# is_url_input = False\n# try:\n# parsed = urlparse(file_path)\n# is_url_input = parsed.scheme in (\"http\", \"https\")\n# except Exception:\n# is_url_input = False\n\n# # 检查文件是否存在（如果不是URL）\n# if not is_url_input and not os.path.exists(file_path):\n# return f\"[ERROR] Input file not found: {file_path}\"\n\n# # 检查是否是PDF文件，优先使用简单转换器（仅对本地文件）\n# if (\n# not is_url_input\n# and file_path.lower().endswith(\".pdf\")\n# and PYPDF2_AVAILABLE\n# and not extract_images\n# ):\n# try:\n# simple_converter = SimplePdfConverter()\n# result = simple_converter.convert_pdf_to_markdown(file_path, output_path)\n# except Exception as e:\n# return f\"[ERROR] PDF conversion error: {str(e)}\"\n# elif DOCLING_AVAILABLE:\n# try:\n# converter = DoclingConverter()\n\n# # 检查文件格式是否支持\n# if not is_url_input and not converter.is_supported_format(file_path):\n# supported_formats = [\".pdf\", \".docx\", \".pptx\", \".html\", \".md\", \".txt\"]\n# return f\"[ERROR] Unsupported file format. Supported formats: {', '.join(supported_formats)}\"\n# elif is_url_input and not file_path.lower().endswith(\n# (\".pdf\", \".docx\", \".pptx\", \".html\", \".md\", \".txt\")\n# ):\n# return f\"[ERROR] Unsupported URL format: {file_path}\"\n\n# # 执行转换（支持图片提取）\n# result = converter.convert_to_markdown(\n# file_path, output_path, extract_images\n# )\n# except Exception as e:\n# return f\"[ERROR] Docling conversion error: {str(e)}\"\n# else:\n# return (\n# \"[ERROR] No conversion tools available. Please install docling or PyPDF2.\"\n# )\n\n# if result[\"success\"]:\n# msg = \"[SUCCESS] Document converted successfully!\\n\"\n# msg += f\" Input: {result['input_file']}\\n\"\n# msg += f\" Output file: {result['output_file']}\\n\"\n# msg += f\" Conversion time: {result['duration']:.2f} seconds\\n\"\n\n# if result[\"input_size\"] > 0:\n# msg += f\" Original size: {result['input_size'] / 1024:.1f} KB\\n\"\n# msg += f\" Markdown size: {result['output_size'] / 1024:.1f} KB\\n\"\n\n# # 显示图片提取信息\n# if extract_images and \"images_extracted\" in result:\n# images_count = result[\"images_extracted\"]\n# if images_count > 0:\n# msg += f\" Images extracted: {images_count}\\n\"\n# msg += f\" Images saved to: {os.path.join(os.path.dirname(result['output_file']), 'images')}\\n\"\n# else:\n# msg += \" No images found in document\\n\"\n\n# # 显示Markdown内容的前几行作为预览\n# content_lines = result[\"markdown_content\"].split(\"\\n\")\n# preview_lines = content_lines[:5]\n# if len(content_lines) > 5:\n# preview_lines.append(\"...\")\n\n# msg += \"\\n[PREVIEW] First few lines of converted Markdown:\\n\"\n# for line in preview_lines:\n# msg += f\" {line}\\n\"\n# else:\n# msg = \"[ERROR] Conversion failed!\\n\"\n# msg += f\" Error: {result['error']}\"\n\n# return msg\n\n\nif __name__ == \"__main__\":\n print(\"📄 Smart PDF Downloader MCP Tool\")\n print(\"📝 Starting server with FastMCP...\")\n\n if DOCLING_AVAILABLE:\n print(\"✅ Document conversion to Markdown is ENABLED (docling available)\")\n else:\n print(\"❌ Document conversion to Markdown is DISABLED (docling not available)\")\n print(\" Install docling to enable: pip install docling\")\n\n print(\"\\nAvailable tools:\")\n print(\n \" • download_files - Download files or move local files from natural language\"\n )\n print(\" • parse_download_urls - Extract URLs, local paths and destination paths\")\n print(\" • download_file_to - Download a specific file with options\")\n print(\" • move_file_to - Move a specific local file with options\")\n print(\" • convert_document_to_markdown - Convert documents to Markdown format\")\n\n if DOCLING_AVAILABLE:\n print(\"\\nSupported formats: PDF, DOCX, PPTX, HTML, TXT, MD\")\n print(\"Features: Image extraction, Layout preservation, Automatic conversion\")\n\n print(\"\")\n\n # 运行服务器\n mcp.run()\n"
},
{
"path": "tools/pdf_utils.py",
"content": "\"\"\"\nPDF utility functions for the DeepCode agent system.\n\"\"\"\n\nfrom pathlib import Path\nimport PyPDF2\n\n\ndef read_pdf_metadata(file_path: Path) -> dict:\n \"\"\"Read PDF metadata with proper encoding handling.\"\"\"\n try:\n print(f\"\\nAttempting to read PDF metadata from: {file_path}\")\n with open(file_path, \"rb\") as file:\n pdf_reader = PyPDF2.PdfReader(file)\n info = pdf_reader.metadata\n first_page = pdf_reader.pages[0]\n text = first_page.extract_text()\n lines = text.split(\"\\n\")[:10]\n\n title = None\n authors = []\n\n if info:\n title = info.get(\"/Title\", \"\").strip().replace(\"\\x00\", \"\")\n author = info.get(\"/Author\", \"\").strip().replace(\"\\x00\", \"\")\n if author:\n authors = [author]\n\n if not title and lines:\n title = lines[0].strip()\n\n if not authors and len(lines) > 1:\n for line in lines[1:3]:\n if \"author\" in line.lower() or \"by\" in line.lower():\n authors = [line.strip()]\n break\n\n return {\n \"title\": title if title else \"Unknown Title\",\n \"authors\": authors if authors else [\"Unknown Author\"],\n \"year\": info.get(\"/CreationDate\", \"\")[:4] if info else \"Unknown Year\",\n \"first_lines\": lines,\n }\n\n except Exception as e:\n print(f\"\\nError reading PDF: {str(e)}\")\n return {\n \"title\": \"Error reading PDF\",\n \"authors\": [\"Unknown\"],\n \"year\": \"Unknown\",\n \"first_lines\": [],\n }\n"
},
{
"path": "ui/__init__.py",
"content": "\"\"\"\nUI Module\n\nStreamlit application user interface components module\n\nContains the following submodules:\n- styles: CSS styles\n- components: UI components\n- layout: Page layout\n- handlers: Event handlers\n- streamlit_app: Main application\n- app: Application entry\n\"\"\"\n\n__version__ = \"1.0.0\"\n__author__ = \"DeepCode Team\"\n\n# Import main components\nfrom .layout import main_layout\nfrom .components import display_header, display_features, display_status\nfrom .handlers import initialize_session_state\nfrom .styles import get_main_styles\n\n# Import application main function\ntry:\n from .streamlit_app import main as streamlit_main\nexcept ImportError:\n # Fallback to absolute import if relative import fails\n import sys\n import os\n\n sys.path.insert(0, os.path.dirname(__file__))\n from streamlit_app import main as streamlit_main\n\n__all__ = [\n \"main_layout\",\n \"display_header\",\n \"display_features\",\n \"display_status\",\n \"initialize_session_state\",\n \"get_main_styles\",\n \"streamlit_main\",\n]\n"
},
{
"path": "ui/app.py",
"content": "\"\"\"\nDeepCode UI Application Entry Point\n\nThis file serves as the unified entry point for the UI module\n\"\"\"\n\nfrom .streamlit_app import main\n\n# Directly export main function for external calls\n__all__ = [\"main\"]\n\nif __name__ == \"__main__\":\n main()\n"
},
{
"path": "ui/components.py",
"content": "# -*- coding: utf-8 -*-\n\"\"\"\nStreamlit UI Components - Cyber Edition\nContains all reusable UI components with new styling plus\nthe operational widgets required by the handlers.\n\"\"\"\n\nfrom __future__ import annotations\n\nimport html\nimport base64\nimport sys\nfrom datetime import datetime\nfrom functools import lru_cache\nfrom pathlib import Path\nfrom typing import Dict, Any, Optional, List, Tuple\n\nimport streamlit as st\n\nfrom utils.cross_platform_file_handler import get_file_handler\n\nBASE_DIR = Path(__file__).resolve().parents[1]\nICON_DIR = BASE_DIR / \"assets\" / \"icons\"\n\n\n@lru_cache(maxsize=64)\ndef _icon_data_uri(name: str) -> str:\n path = ICON_DIR / f\"{name}.png\"\n if not path.exists():\n return \"\"\n\n try:\n data = path.read_bytes()\n except OSError:\n return \"\"\n\n encoded = base64.b64encode(data).decode(\"utf-8\")\n return f\"data:image/png;base64,{encoded}\"\n\n\ndef icon_img(name: str, size: int = 32, extra_style: str = \"\") -> str:\n \"\"\"\n Render an inline tag for icons stored in assets/icons via data URI.\n \"\"\"\n data_uri = _icon_data_uri(name)\n if not data_uri:\n return \"\"\n return f''\n\n\ndef clear_guided_answer_inputs():\n \"\"\"Remove temporary answer widgets from session state.\"\"\"\n keys_to_delete = [\n key for key in st.session_state.keys() if key.startswith(\"guided_answer_\")\n ]\n for key in keys_to_delete:\n del st.session_state[key]\n\n\ndef display_header():\n \"\"\"Display the Cyber-styled header\"\"\"\n st.markdown(\n \"\"\"\n
\n
\n
DEEPCODE
\n
Autonomous Research & Engineering Matrix
\n
\n
\n \n SYSTEM ONLINE\n
\n
\n \"\"\",\n unsafe_allow_html=True,\n )\n\n\ndef display_features():\n \"\"\"Display feature cards grid\"\"\"\n feature_cards = [\n {\n \"icon\": \"feature_synthesis\",\n \"fallback\": \"🧬\",\n \"title\": \"Neural Synthesis\",\n \"desc\": \"Transform research papers directly into executable repositories via multi-agent LLM pipelines.\",\n },\n {\n \"icon\": \"feature_hyper\",\n \"fallback\": \"⚡\",\n \"title\": \"Hyper-Speed Mode\",\n \"desc\": \"Acceleration layer that parallelizes retrieval, planning, and implementation for fastest delivery.\",\n },\n {\n \"icon\": \"feature_cognition\",\n \"fallback\": \"🧠\",\n \"title\": \"Cognitive Context\",\n \"desc\": \"Semantic memory graphs retain methodology, datasets, and evaluation strategy during reasoning.\",\n },\n {\n \"icon\": \"feature_secure\",\n \"fallback\": \"🛡️\",\n \"title\": \"Secure Sandbox(Coming Soon)\",\n \"desc\": \"Isolated execution & validation environment keeps experiments safe and reproducible.\",\n },\n ]\n\n cards_html = \"\"\n for card in feature_cards:\n icon_markup = icon_img(\n card[\"icon\"],\n 48,\n \"filter:drop-shadow(0 0 10px rgba(0,242,255,0.4));\",\n )\n if not icon_markup:\n icon_markup = f'{card[\"fallback\"]}'\n\n cards_html += f\"\"\"\n
\n \"\"\"\n\n\ndef enhanced_progress_display_component(\n enable_indexing: bool, chat_mode: bool\n) -> Tuple[Any, Any, List[Any], List[Dict[str, str]]]:\n \"\"\"\n Render the progress panel required by handlers.handle_processing_workflow.\n \"\"\"\n\n if chat_mode:\n workflow_steps = [\n {\"title\": \"INIT\", \"subtitle\": \"Boot agents\"},\n {\"title\": \"PLAN\", \"subtitle\": \"Analyze intent\"},\n {\"title\": \"SETUP\", \"subtitle\": \"Workspace\"},\n {\"title\": \"DRAFT\", \"subtitle\": \"Generate plan\"},\n {\"title\": \"CODE\", \"subtitle\": \"Implement\"},\n ]\n elif not enable_indexing:\n workflow_steps = [\n {\"title\": \"INIT\", \"subtitle\": \"Load systems\"},\n {\"title\": \"ANALYZE\", \"subtitle\": \"Parse paper\"},\n {\"title\": \"DOWNLOAD\", \"subtitle\": \"Collect refs\"},\n {\"title\": \"PLAN\", \"subtitle\": \"Blueprint\"},\n {\"title\": \"CODE\", \"subtitle\": \"Implement\"},\n ]\n else:\n workflow_steps = [\n {\"title\": \"INIT\", \"subtitle\": \"Load systems\"},\n {\"title\": \"ANALYZE\", \"subtitle\": \"Paper scan\"},\n {\"title\": \"DOWNLOAD\", \"subtitle\": \"Docs & data\"},\n {\"title\": \"PLAN\", \"subtitle\": \"Architect\"},\n {\"title\": \"REF\", \"subtitle\": \"Key refs\"},\n {\"title\": \"REPO\", \"subtitle\": \"GitHub sync\"},\n {\"title\": \"INDEX\", \"subtitle\": \"Vectorize\"},\n {\"title\": \"CODE\", \"subtitle\": \"Implementation\"},\n ]\n\n st.markdown(\"### 🛰️ Workflow Monitor\")\n progress_bar = st.progress(0)\n status_text = st.empty()\n\n cols = st.columns(len(workflow_steps))\n step_indicators: List[Any] = []\n for col, step in zip(cols, workflow_steps):\n with col:\n placeholder = st.empty()\n placeholder.markdown(\n _render_step_card(step[\"title\"], step[\"subtitle\"], \"pending\"),\n unsafe_allow_html=True,\n )\n step_indicators.append(placeholder)\n\n return progress_bar, status_text, step_indicators, workflow_steps\n\n\ndef update_step_indicator(\n step_indicators: List[Any],\n workflow_steps: List[Dict[str, str]],\n current_step: int,\n status: str,\n):\n \"\"\"\n Update the workflow step indicators in-place.\n \"\"\"\n total_steps = len(workflow_steps)\n\n for idx, placeholder in enumerate(step_indicators):\n if status == \"error\" and idx == current_step:\n state = \"error\"\n elif current_step >= total_steps:\n state = \"completed\"\n elif idx < current_step:\n state = \"completed\"\n elif idx == current_step:\n state = \"active\"\n else:\n state = \"pending\"\n\n step = workflow_steps[idx]\n placeholder.markdown(\n _render_step_card(step[\"title\"], step[\"subtitle\"], state),\n unsafe_allow_html=True,\n )\n\n\ndef chat_input_component(task_counter: int = 0) -> Optional[str]:\n \"\"\"Render modern chat input for guided mode\"\"\"\n st.markdown(\"### 💬 Neural Link Interface\")\n\n user_input = st.chat_input(\n placeholder=\"Input research directive or query...\",\n key=f\"chat_input_{task_counter}\",\n )\n return user_input\n\n\ndef _save_uploaded_pdf(uploaded_file) -> Optional[str]:\n \"\"\"Persist uploaded PDF to a temp file and return its path.\"\"\"\n try:\n file_bytes = uploaded_file.read()\n suffix = Path(uploaded_file.name).suffix or \".pdf\"\n handler = get_file_handler()\n temp_path = handler.create_safe_temp_file(\n suffix=suffix, prefix=\"deepcode_upload_\", content=file_bytes\n )\n return str(temp_path)\n except Exception as exc:\n st.error(f\"Failed to save uploaded file: {exc}\")\n return None\n\n\ndef input_method_selector(task_counter: int) -> Tuple[Optional[str], Optional[str]]:\n \"\"\"Render the input method selection tabs with modern styling\"\"\"\n\n tab1, tab2, tab3 = st.tabs([\"📄 PDF UPLOAD\", \"🔗 URL LINK\", \"⚡ QUICK COMMAND\"])\n\n input_source: Optional[str] = None\n input_type: Optional[str] = None\n\n with tab1:\n st.markdown('', unsafe_allow_html=True)\n uploaded_file = st.file_uploader(\n \"Upload Research Paper (PDF)\",\n type=\"pdf\",\n key=f\"file_uploader_{task_counter}\",\n )\n if uploaded_file:\n saved_path = _save_uploaded_pdf(uploaded_file)\n if saved_path:\n st.session_state[\"uploaded_filename\"] = uploaded_file.name\n input_source = saved_path\n input_type = \"file\"\n\n with tab2:\n st.markdown('', unsafe_allow_html=True)\n url = st.text_input(\n \"ArXiv / GitHub Resource URL\",\n placeholder=\"https://arxiv.org/abs/...\",\n key=f\"url_input_{task_counter}\",\n )\n if url:\n input_source = url.strip()\n input_type = \"url\"\n\n with tab3:\n st.markdown('', unsafe_allow_html=True)\n query = st.text_area(\n \"Code Specifications / Abstract\",\n placeholder=\"Describe the algorithm or system requirements...\",\n height=150,\n key=f\"text_input_{task_counter}\",\n )\n if query:\n input_source = query.strip()\n input_type = \"chat\"\n\n return input_source, input_type\n\n\ndef results_display_component(result: Any, task_counter: int):\n \"\"\"Display results in a tech-styled container\"\"\"\n\n status = result.get(\"status\", \"unknown\")\n is_success = status == \"success\"\n status_label = \"Mission Complete\" if is_success else \"Execution Failed\"\n status_color = \"var(--success)\" if is_success else \"var(--error)\"\n status_icon = icon_img(\"status_success\" if is_success else \"status_error\", 56)\n if not status_icon:\n status_icon = \"✅\" if is_success else \"⚠️\"\n status_message = (\n \"Computation sequence completed successfully.\"\n if is_success\n else result.get(\"error\", \"Unknown error occurred during processing.\")\n )\n\n st.markdown('', unsafe_allow_html=True)\n st.markdown(\"### 🚀 Operation Result\")\n\n with st.container():\n if is_success:\n st.success(\"Workflow completed across all stages ✅\")\n else:\n st.error(\"Workflow interrupted. Check the logs below ⚠️\")\n\n col1, col2 = st.columns([2, 1])\n with col1:\n with st.expander(\"📜 Execution Logs & Metadata\", expanded=True):\n st.json(result)\n\n with col2:\n st.markdown(\n f\"\"\"\n
\n
{status_icon}
\n
{status_label}
\n
{status_message}
\n
\n \"\"\",\n unsafe_allow_html=True,\n )\n st.download_button(\n label=\"📥 DOWNLOAD ARTIFACTS\" if is_success else \"📥 DOWNLOAD LOGS\",\n data=str(result),\n file_name=f\"deepcode_result_{task_counter}.json\",\n mime=\"application/json\",\n use_container_width=True,\n )\n\n\ndef system_status_component():\n \"\"\"System status check component\"\"\"\n st.markdown(\"### 🔧 System Diagnostics\")\n\n col1, col2 = st.columns(2)\n\n with col1:\n st.markdown(\"#### 📊 Core Metrics\")\n st.info(f\"**Python:** {sys.version.split()[0]}\")\n st.info(f\"**Platform:** {sys.platform}\")\n\n with col2:\n st.markdown(\"#### ⚙️ Runtime Status\")\n try:\n import asyncio\n\n loop = asyncio.get_event_loop()\n if loop.is_running():\n st.success(\"Event Loop: ACTIVE\")\n else:\n st.warning(\"Event Loop: STANDBY\")\n except Exception:\n st.info(\"Event Loop: MANAGED\")\n\n\ndef error_troubleshooting_component():\n \"\"\"Error troubleshooting component\"\"\"\n with st.expander(\"🛠️ Diagnostics & Troubleshooting\", expanded=False):\n st.warning(\n \"If you encounter issues, please check your API keys in the sidebar.\"\n )\n\n\ndef footer_component():\n \"\"\"Minimal futuristic footer\"\"\"\n st.markdown(\n \"\"\"\n
\n DEEPCODE_SYSTEMS // OPERATIONAL // VERSION 3.0.1\n
\n \"\"\",\n unsafe_allow_html=True,\n )\n\n\ndef render_sidebar_feed(max_items: int = 12):\n \"\"\"Render live mission feed inside sidebar.\"\"\"\n st.markdown(\"#### 📡 Mission Feed\")\n events = list(st.session_state.get(\"sidebar_events\", []))\n\n col1, col2 = st.columns([1, 1])\n with col1:\n st.caption(\"Real-time agent telemetry\")\n with col2:\n if st.button(\"Clear Feed\", key=\"sidebar_clear_feed\"):\n st.session_state.sidebar_events = []\n events = []\n st.session_state.sidebar_feed_last_cleared = datetime.utcnow().strftime(\n \"%H:%M:%S\"\n )\n\n if not events:\n st.caption(\"Awaiting activity...\")\n return\n\n recent_events = list(reversed(events[-max_items:]))\n for event in recent_events:\n stage = event.get(\"stage\", \"STAGE\")\n message = html.escape(str(event.get(\"message\", \"\")))\n timestamp = event.get(\"timestamp\", \"--:--:--\")\n level = event.get(\"level\", \"info\")\n extra = event.get(\"extra\")\n\n st.markdown(\n f\"\"\"\n
\n
\n {stage}\n {timestamp}\n
\n
{message}
\n
\n \"\"\",\n unsafe_allow_html=True,\n )\n\n if isinstance(extra, dict) and extra:\n with st.expander(\"Details\", expanded=False):\n st.json(extra)\n\n\ndef render_system_monitor():\n \"\"\"Display current backend + command telemetry.\"\"\"\n st.markdown(\"#### 🧬 System Monitor\")\n processing = st.session_state.get(\"processing\", False)\n mode = st.session_state.get(\"requirement_analysis_mode\", \"direct\").upper()\n indexing_enabled = st.session_state.get(\"enable_indexing\", True)\n task_counter = st.session_state.get(\"task_counter\", 0)\n last_error = st.session_state.get(\"last_error\")\n events = st.session_state.get(\"sidebar_events\", [])\n latest_event = events[-1] if events else None\n last_stage = latest_event.get(\"stage\") if latest_event else \"--\"\n last_message = (\n html.escape(str(latest_event.get(\"message\", \"\"))) if latest_event else \"\"\n )\n last_progress = (\n latest_event.get(\"extra\", {}).get(\"progress\") if latest_event else None\n )\n state_label = \"ACTIVE\" if processing else \"IDLE\"\n\n st.markdown(\n f\"\"\"\n
\n
\n
STATE{state_label}
\n
MODE{mode}
\n
INDEXING{\"ON\" if indexing_enabled else \"OFF\"}
\n
TASKS{task_counter}
\n
\n
\n {last_stage if last_stage else \"--\"}\n {\"· \" + str(last_progress) + \"%\" if last_progress is not None else \"\"}\n {last_message or \"Awaiting telemetry...\"}\n
\n
\n \"\"\",\n unsafe_allow_html=True,\n )\n\n if last_error:\n st.warning(f\"Last error: {last_error}\")\n\n\ndef render_log_viewer(max_lines: int = 50):\n \"\"\"Display live log stream for current mission in a scrollable container.\"\"\"\n st.markdown(\"#### 📁 Live Log Stream\")\n logs_dir = BASE_DIR / \"logs\"\n if not logs_dir.exists():\n st.info(\"Logs directory not found.\")\n return\n\n log_files = sorted(\n [p for p in logs_dir.glob(\"*.jsonl\") if p.is_file()],\n key=lambda p: p.stat().st_mtime,\n reverse=True,\n )\n if not log_files:\n st.info(\"No log files available yet.\")\n return\n\n start_ts = st.session_state.get(\"workflow_start_time\")\n selected_path = None\n\n waiting_for_new_log = False\n\n if start_ts:\n # Use a tolerance window: accept logs created within 10 seconds before workflow_start_time\n tolerance = 10.0\n for candidate in log_files:\n file_mtime = candidate.stat().st_mtime\n if file_mtime >= (start_ts - tolerance):\n selected_path = candidate\n break\n if selected_path is None:\n waiting_for_new_log = True\n else:\n prev = st.session_state.get(\"active_log_file\")\n if prev:\n prev_path = Path(prev)\n if prev_path.exists():\n selected_path = prev_path\n if selected_path is None:\n selected_path = log_files[0]\n\n if waiting_for_new_log:\n st.caption(\"Waiting for current task log to be created...\")\n return\n\n st.session_state.active_log_file = str(selected_path)\n\n try:\n content = selected_path.read_text(encoding=\"utf-8\", errors=\"ignore\")\n except Exception as exc:\n st.error(f\"Failed to read {selected_path.name}: {exc}\")\n return\n\n lines = content.splitlines()\n tail_lines = lines[-max_lines:]\n\n # Show file info\n processing = st.session_state.get(\"processing\", False)\n status_icon = \"🔄\" if processing else \"✅\"\n st.caption(f\"{status_icon} {selected_path.name} | Last {len(tail_lines)} lines\")\n\n if not tail_lines:\n st.info(\"Log file is empty.\")\n return\n\n # Build log HTML with scrollable container\n import json\n\n log_html_parts = []\n\n for line in tail_lines:\n line = line.strip()\n if not line:\n continue\n\n try:\n event = json.loads(line)\n timestamp = event.get(\"timestamp\", \"\")\n level = event.get(\"level\", \"INFO\")\n message = event.get(\"message\", \"\")\n namespace = event.get(\"namespace\", \"\")\n\n # Color code by level\n if level == \"ERROR\":\n level_color = \"#ff4444\"\n elif level == \"WARNING\":\n level_color = \"#ffaa00\"\n elif \"SUCCESS\" in level.upper():\n level_color = \"#00ff88\"\n else:\n level_color = \"#00d4ff\"\n\n # Format display\n time_str = (\n timestamp.split(\"T\")[-1][:12] if \"T\" in timestamp else timestamp[-12:]\n )\n namespace_short = namespace.split(\".\")[-1] if namespace else \"\"\n\n log_html_parts.append(\n f'
\"\n )\n except json.JSONDecodeError:\n # Raw text fallback\n log_html_parts.append(\n f'
{line[:200]}
'\n )\n\n # Render in scrollable container\n full_log_html = f\"\"\"\n
\n {''.join(log_html_parts)}\n
\n \"\"\"\n\n st.markdown(full_log_html, unsafe_allow_html=True)\n\n\ndef reset_guided_workflow_state(preserve_initial: bool = False):\n \"\"\"\n Reset guided requirement workflow state machine.\n \"\"\"\n if preserve_initial:\n initial_text = st.session_state.get(\n \"guided_initial_requirement\",\n st.session_state.get(\"initial_requirement\", \"\"),\n )\n else:\n initial_text = \"\"\n st.session_state.initial_requirement = \"\"\n\n st.session_state.guided_initial_requirement = initial_text\n st.session_state.guided_edit_feedback = \"\"\n st.session_state.requirement_analysis_step = \"input\"\n st.session_state.generated_questions = []\n st.session_state.user_answers = {}\n st.session_state.detailed_requirements = \"\"\n st.session_state.questions_generating = False\n st.session_state.requirements_generating = False\n st.session_state.requirements_confirmed = False\n st.session_state.requirements_editing = False\n st.session_state.edit_feedback = \"\"\n st.session_state.confirmed_requirement_text = None\n clear_guided_answer_inputs()\n\n\ndef requirement_mode_selector() -> str:\n \"\"\"\n Render the requirement workflow mode selector.\n \"\"\"\n mode_labels = {\"direct\": \"🚀 Direct Mode\", \"guided\": \"🧭 Guided Mode\"}\n current_mode = st.session_state.get(\"requirement_analysis_mode\", \"direct\")\n\n selection = st.radio(\n \"Requirement Intake Mode\",\n options=list(mode_labels.keys()),\n index=0 if current_mode != \"guided\" else 1,\n horizontal=True,\n format_func=lambda key: mode_labels[key],\n key=\"requirement_mode_selector_radio\",\n )\n\n if selection != current_mode:\n st.session_state.requirement_analysis_mode = selection\n if selection == \"direct\":\n reset_guided_workflow_state(preserve_initial=False)\n else:\n st.session_state.requirement_analysis_step = \"input\"\n\n return selection\n\n\ndef guided_requirement_workflow() -> Tuple[Optional[str], bool]:\n \"\"\"\n Render the guided requirement analysis workflow.\n \"\"\"\n\n st.markdown(\"### 🧭 Guided Requirement Workflow\")\n\n step = st.session_state.get(\"requirement_analysis_step\", \"input\")\n st.session_state.setdefault(\n \"guided_initial_requirement\", st.session_state.get(\"initial_requirement\", \"\")\n )\n st.session_state.setdefault(\n \"guided_edit_feedback\", st.session_state.get(\"edit_feedback\", \"\")\n )\n\n step_titles = {\n \"input\": \"Step 1 · Describe Requirements\",\n \"questions\": \"Step 2 · Answer Guiding Questions\",\n \"summary\": \"Step 3 · Review Requirement Document\",\n \"editing\": \"Step 4 · Request Changes\",\n }\n st.caption(\n f\"Current Stage: {step_titles.get(step, 'Step 1 · Describe Requirements')}\"\n )\n\n confirmed_doc = st.session_state.get(\"confirmed_requirement_text\")\n\n if step == \"input\":\n st.markdown(\"#### 1 · Describe your project\")\n st.text_area(\n \"Describe the product scope, tech stack, performance targets, and constraints:\",\n key=\"guided_initial_requirement\",\n height=180,\n )\n initial_text = st.session_state.get(\"guided_initial_requirement\", \"\")\n\n col1, col2 = st.columns(2)\n with col1:\n if st.button(\"Generate guiding questions\", type=\"primary\"):\n if not initial_text.strip():\n st.warning(\"Please enter your project requirements first.\")\n else:\n st.session_state.initial_requirement = initial_text.strip()\n st.session_state.questions_generating = True\n st.session_state.requirement_analysis_step = \"questions\"\n st.session_state.generated_questions = []\n st.session_state.user_answers = {}\n st.session_state.detailed_requirements = \"\"\n st.session_state.confirmed_requirement_text = None\n st.session_state.requirements_generating = False\n st.session_state.requirements_confirmed = False\n st.session_state.requirements_editing = False\n st.session_state.edit_feedback = \"\"\n clear_guided_answer_inputs()\n st.rerun()\n\n with col2:\n if st.button(\"Skip Q&A and use current spec\", type=\"secondary\"):\n if not initial_text.strip():\n st.warning(\"Please enter your project requirements first.\")\n else:\n final_doc = initial_text.strip()\n st.session_state.initial_requirement = final_doc\n st.session_state.confirmed_requirement_text = final_doc\n st.session_state.requirements_confirmed = True\n st.success(\n \"Current description locked as the requirement document. Implementation will proceed next.\"\n )\n\n elif step == \"questions\":\n st.markdown(\"#### 2 · Answer guiding questions\")\n if st.session_state.get(\"questions_generating\"):\n st.info(\"LLM is crafting guiding questions. Please wait...\")\n\n questions = st.session_state.get(\"generated_questions\", [])\n question_ids: List[str] = []\n\n if not questions:\n st.caption(\"Guiding questions will appear once generation is complete.\")\n else:\n for idx, question in enumerate(questions):\n if isinstance(question, dict):\n q_id = str(\n question.get(\"id\")\n or question.get(\"question_id\")\n or question.get(\"qid\")\n or idx\n )\n q_text = question.get(\"question\") or question.get(\"content\") or \"\"\n category = question.get(\"category\")\n importance = question.get(\"importance\")\n hint = question.get(\"hint\")\n else:\n q_id = str(idx)\n q_text = str(question)\n category = importance = hint = None\n\n question_ids.append(q_id)\n\n st.markdown(\n f\"**Q{idx + 1}. {q_text or 'Please answer this question'}**\"\n )\n meta_parts = [part for part in [category, importance] if part]\n if meta_parts:\n st.caption(\" / \".join(meta_parts))\n if hint:\n st.caption(f\"Hint: {hint}\")\n\n answer_key = f\"guided_answer_{idx}\"\n if answer_key not in st.session_state:\n default_answer = st.session_state.user_answers.get(q_id, \"\")\n st.session_state[answer_key] = default_answer\n\n st.text_area(\"Your answer\", key=answer_key, height=100)\n\n col1, col2, col3 = st.columns(3)\n with col1:\n if st.button(\n \"Generate requirement document\", type=\"primary\", disabled=not questions\n ):\n answers_payload = {}\n for idx, q_id in enumerate(question_ids):\n answer_value = st.session_state.get(\n f\"guided_answer_{idx}\", \"\"\n ).strip()\n if answer_value:\n answers_payload[q_id] = answer_value\n\n st.session_state.user_answers = answers_payload\n st.session_state.requirements_generating = True\n st.session_state.requirement_analysis_step = \"summary\"\n st.session_state.detailed_requirements = \"\"\n st.session_state.confirmed_requirement_text = None\n st.session_state.requirements_confirmed = False\n st.rerun()\n\n with col2:\n if st.button(\n \"Generate without answers\", type=\"secondary\", disabled=not questions\n ):\n st.session_state.user_answers = {}\n st.session_state.requirements_generating = True\n st.session_state.requirement_analysis_step = \"summary\"\n st.session_state.detailed_requirements = \"\"\n st.session_state.confirmed_requirement_text = None\n st.session_state.requirements_confirmed = False\n st.rerun()\n\n with col3:\n if st.button(\"Back to Step 1\"):\n reset_guided_workflow_state(preserve_initial=True)\n st.rerun()\n\n elif step == \"summary\":\n st.markdown(\"#### 3 · AI-generated requirement document\")\n if st.session_state.get(\"requirements_generating\"):\n st.info(\"Generating requirement document. Please wait...\")\n\n summary = (st.session_state.get(\"detailed_requirements\") or \"\").strip()\n\n if summary:\n st.markdown(summary)\n st.download_button(\n \"Download requirement document\",\n summary,\n file_name=\"deepcode_requirements.md\",\n mime=\"text/markdown\",\n use_container_width=True,\n )\n else:\n st.caption(\"Waiting for requirement document to be generated...\")\n\n col1, col2, col3 = st.columns(3)\n with col1:\n if st.button(\n \"Confirm and start implementation ✅\",\n type=\"primary\",\n disabled=not summary,\n ):\n final_doc = summary or st.session_state.get(\"initial_requirement\", \"\")\n if final_doc.strip():\n st.session_state.confirmed_requirement_text = final_doc.strip()\n st.session_state.requirements_confirmed = True\n st.success(\n \"Requirement document confirmed. Implementation pipeline will start next.\"\n )\n else:\n st.warning(\"No requirement document available yet.\")\n\n with col2:\n if st.button(\"Request edits\", type=\"secondary\", disabled=not summary):\n st.session_state.requirement_analysis_step = \"editing\"\n st.session_state.guided_edit_feedback = \"\"\n\n with col3:\n if st.button(\"Restart Q&A\", type=\"secondary\"):\n reset_guided_workflow_state(preserve_initial=True)\n st.rerun()\n\n elif step == \"editing\":\n st.markdown(\"#### 4 · Modify requirement document\")\n st.text_area(\n \"Describe the changes or clarifications you need:\",\n key=\"guided_edit_feedback\",\n height=160,\n )\n feedback_value = st.session_state.get(\"guided_edit_feedback\", \"\")\n\n col1, col2 = st.columns(2)\n with col1:\n if st.button(\"Submit change request\", type=\"primary\"):\n if not feedback_value.strip():\n st.warning(\"Please describe the requested changes.\")\n else:\n st.session_state.edit_feedback = feedback_value.strip()\n st.session_state.requirements_editing = True\n st.info(\"Updating requirement document based on your feedback...\")\n\n with col2:\n if st.button(\"Back to requirement document\"):\n st.session_state.requirement_analysis_step = \"summary\"\n st.session_state.guided_edit_feedback = \"\"\n\n if st.session_state.get(\"requirements_editing\"):\n st.info(\"Requirement document is updating...\")\n\n if confirmed_doc:\n st.success(\"Requirement document locked. You can start implementation anytime.\")\n\n return (confirmed_doc if confirmed_doc else None, bool(confirmed_doc))\n\n\ndef sidebar_control_panel():\n \"\"\"Sidebar configuration\"\"\"\n with st.sidebar:\n st.markdown(\n \"\"\"\n
\n
CONTROL DECK
\n
// MISSION CONTROL
\n
\n \"\"\",\n unsafe_allow_html=True,\n )\n\n workflow_start = st.session_state.get(\"workflow_start_time\")\n\n if workflow_start:\n render_log_viewer()\n else:\n st.info(\"Awaiting next mission run to stream logs.\")\n st.markdown(\n \"\"\"\n
![\"English\"](\"https://img.shields.io/badge/English-00d4ff?style=for-the-badge&logo=readme&logoColor=white&labelColor=1a1a2e\"){kind=icon}
\n ![\"中文\"](\"https://img.shields.io/badge/中文-00d4ff?style=for-the-badge&logo=readme&logoColor=white&labelColor=1a1a2e\"){kind=icon}
\n ![\"DeepCode](\"https://img.youtube.com/vi/PRgmP8pOI08/maxresdefault.jpg\"\n)
\n ![\"Watch](\"https://img.shields.io/badge/▶️_Watch_Video-FF0000?style=for-the-badge&logo=youtube&logoColor=white\"){kind=icon}
\n ![\"DeepCode\"](\"./assets/logo.png\"){kind=logo}
![\"nanobot\"](\"./assets/nanobot.png\")
![\"Feishu](\"./assets/IMG_8098.jpeg\")
\n![\"Feishu](\"./assets/IMG_8099.jpeg\")
\n![\"DeepCode](\"./assets/NewUI.png\")
\n ![\"Algorithm](\"https://img.shields.io/badge/ALGORITHM-IMPLEMENTATION-ff6b6b?style=for-the-badge&logo=algorithm&logoColor=white\"){kind=icon}
\n![\"Frontend](\"https://img.shields.io/badge/FRONTEND-DEVELOPMENT-4ecdc4?style=for-the-badge&logo=react&logoColor=white\"){kind=icon}
\n![\"Backend](\"https://img.shields.io/badge/BACKEND-DEVELOPMENT-9b59b6?style=for-the-badge&logo=server&logoColor=white\"){kind=icon}
\n
![\"Paper2Code](\"https://img.youtube.com/vi/MQZYpLkzsbw/maxresdefault.jpg\")
\n ![\"Image](\"https://img.youtube.com/vi/nFt5mLaMEac/maxresdefault.jpg\")
\n ![\"Frontend](\"https://img.youtube.com/vi/78wx3dkTaAU/maxresdefault.jpg\")
\n ![\"Get](\"https://img.shields.io/badge/🚀_Get_Started-00d4ff?style=for-the-badge&logo=rocket&logoColor=white\"){kind=icon}
![\"View](\"https://img.shields.io/badge/🏛️_View_on_GitHub-00d4ff?style=for-the-badge&logo=github&logoColor=white\"){kind=icon}
![\"Star](\"https://img.shields.io/badge/⭐_Star_Project-00d4ff?style=for-the-badge&logo=star&logoColor=white\"){kind=icon}
![\"MIT](\"https://img.shields.io/badge/License-MIT-4ecdc4?style=for-the-badge&logo=opensourceinitiative&logoColor=white\"){kind=icon}
\n\n**MIT License** - Copyright (c) 2025 Data Intelligence Lab, The University of Hong Kong\n\n---\n\n\n![](\"https://img.shields.io/pypi/v/deepcode-hku.svg?style=for-the-badge&logo=pypi&logoColor=white&labelColor=1a1a2e&color=ff6b6b\")
![](\"https://img.shields.io/badge/💬Discord-社区-7289da?style=for-the-badge&logo=discord&logoColor=white&labelColor=1a1a2e\"){kind=icon}
![](\"https://img.shields.io/badge/💬微信-群组-07c160?style=for-the-badge&logo=wechat&logoColor=white&labelColor=1a1a2e\"){kind=icon}
![](\"https://img.shields.io/badge/快速开始-立即开始-00d9ff?style=for-the-badge&logo=rocket&logoColor=white&labelColor=1a1a2e\"){kind=icon}
\n ![\"Watch](\"https://img.shields.io/badge/▶️_观看视频-FF0000?style=for-the-badge&logo=youtube&logoColor=white\"){kind=icon}
\n ![\"Algorithm](\"https://img.shields.io/badge/算法-实现-ff6b6b?style=for-the-badge&logo=algorithm&logoColor=white\"){kind=icon}
\n![\"Frontend](\"https://img.shields.io/badge/前端-开发-4ecdc4?style=for-the-badge&logo=react&logoColor=white\"){kind=icon}
\n![\"Backend](\"https://img.shields.io/badge/后端-开发-9b59b6?style=for-the-badge&logo=server&logoColor=white\"){kind=icon}
\n![\"Get](\"https://img.shields.io/badge/🚀_立即开始-00d4ff?style=for-the-badge&logo=rocket&logoColor=white\"){kind=icon}
![\"View](\"https://img.shields.io/badge/🏛️_在GitHub上查看-00d4ff?style=for-the-badge&logo=github&logoColor=white\"){kind=icon}
![\"Star](\"https://img.shields.io/badge/⭐_星标项目-00d4ff?style=for-the-badge&logo=star&logoColor=white\"){kind=icon}
![\"WeChat](\"https://github.com/HKUDS/.github/blob/main/profile/QR.png\")
\n"
},
{
"path": "nanobot/Dockerfile",
"content": "FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim\n\n# Install Node.js 20 for the WhatsApp bridge\nRUN apt-get update && \\\n apt-get install -y --no-install-recommends curl ca-certificates gnupg git && \\\n mkdir -p /etc/apt/keyrings && \\\n curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg && \\\n echo \"deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_20.x nodistro main\" > /etc/apt/sources.list.d/nodesource.list && \\\n apt-get update && \\\n apt-get install -y --no-install-recommends nodejs && \\\n apt-get purge -y gnupg && \\\n apt-get autoremove -y && \\\n rm -rf /var/lib/apt/lists/*\n\nWORKDIR /app\n\n# Install Python dependencies first (cached layer)\n# Note: build context is DeepCode root, so paths start with nanobot/\nCOPY nanobot/pyproject.toml nanobot/README.md nanobot/LICENSE ./\nRUN mkdir -p nanobot bridge && touch nanobot/__init__.py && \\\n uv pip install --system --no-cache . && \\\n rm -rf nanobot bridge\n\n# Copy the full source and install\nCOPY nanobot/nanobot/ nanobot/\nCOPY nanobot/bridge/ bridge/\nRUN uv pip install --system --no-cache .\n\n# Build the WhatsApp bridge\nWORKDIR /app/bridge\nRUN npm install && npm run build\nWORKDIR /app\n\n# Create config directory\nRUN mkdir -p /root/.nanobot\n\n# Gateway default port\nEXPOSE 18790\n\nENTRYPOINT [\"nanobot\"]\nCMD [\"gateway\"]\n"
},
{
"path": "nanobot/LICENSE",
"content": "MIT License\n\nCopyright (c) 2025 nanobot contributors\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "nanobot/README.md",
"content": "![\"nanobot\"](\"nanobot_logo.png\"){kind=logo}
\n ![\"PyPI\"](\"https://img.shields.io/pypi/v/nanobot-ai\")
![\"Downloads\"](\"https://static.pepy.tech/badge/nanobot-ai\"){kind=icon}
![\"Python\"](\"https://img.shields.io/badge/python-≥3.11-blue\"){kind=icon}
\n ![\"License\"](\"https://img.shields.io/badge/license-MIT-green\"){kind=icon}
\n ![\"Feishu\"](\"https://img.shields.io/badge/Feishu-Group-E9DBFC?style=flat&logo=feishu&logoColor=white\"){kind=icon}
![\"WeChat\"](\"https://img.shields.io/badge/WeChat-Group-C5EAB4?style=flat&logo=wechat&logoColor=white\"){kind=icon}
![\"Discord\"](\"https://img.shields.io/badge/Discord-Community-5865F2?style=flat&logo=discord&logoColor=white\"){kind=icon}
![\"nanobot](\"nanobot_arch.png\")
\n![](\"case/search.gif\")
![](\"case/code.gif\")
![](\"case/scedule.gif\")
![](\"case/memory.gif\")
![\"DeepCode](\"assets/logo.png\"){kind=logo}
\n![\"HKUDS%2FDeepCode](\"https://trendshift.io/api/badge/repositories/14665\")
\n![\"DeepCode](\"https://github.com/Zongwei9888/Experiment_Images/raw/43c585dca3d21b8e4b6390d835cdd34dc4b4b23d/DeepCode_images/title_logo.svg\"){kind=logo}
DeepCode: Open Agentic Coding\n\n### *Advancing Code Generation with Multi-Agent Systems*\n\n\n
![](\"https://img.shields.io/badge/Paper-arXiv-orange?style=for-the-badge&logo=arxiv&logoColor=white&labelColor=1a1a2e\"){kind=icon}
![](\"https://img.shields.io/badge/🐍Python-3.13-4ecdc4?style=for-the-badge&logo=python&logoColor=white&labelColor=1a1a2e\"){kind=icon}
![](\"https://img.shields.io/badge/💬Discord-Community-7289da?style=for-the-badge&logo=discord&logoColor=white&labelColor=1a1a2e\"){kind=icon}
![](\"https://img.shields.io/badge/💬WeChat-Group-07c160?style=for-the-badge&logo=wechat&logoColor=white&labelColor=1a1a2e\"){kind=icon}
![](\"https://img.shields.io/badge/Quick%20Start-Get%20Started%20Now-00d9ff?style=for-the-badge&logo=rocket&logoColor=white&labelColor=1a1a2e\")
\n \n![\"CLI](\"https://github.com/Zongwei9888/Experiment_Images/blob/8882a7313c504ca97ead6e7b36c51aa761b6a4f3/DeepCode_images/CLI.gif\")
\n\n ![\"Web](\"https://github.com/Zongwei9888/Experiment_Images/raw/8882a7313c504ca97ead6e7b36c51aa761b6a4f3/DeepCode_images/UI.gif\")
\n\n