[
{
"path": ".github/workflows/build-agent.yaml",
"content": "name: Build Agent\n\non:\n push:\n branches:\n - main\n paths:\n - \"packages/bytebot-agent/**\"\n - \"packages/shared/**\"\n\npermissions:\n contents: read\n packages: write\n\njobs:\n docker:\n runs-on: ubuntu-22.04\n\n steps:\n - uses: actions/checkout@v4\n - uses: docker/setup-qemu-action@v3\n - uses: docker/setup-buildx-action@v3\n - name: Docker meta\n id: meta\n uses: docker/metadata-action@v5\n with:\n images: ghcr.io/bytebot-ai/bytebot-agent\n tags: type=edge\n - name: Login to GitHub Container Registry\n uses: docker/login-action@v3\n with:\n registry: ghcr.io\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n - name: Build and push\n uses: docker/build-push-action@v6\n env:\n BUILDX_NO_DEFAULT_ATTESTATIONS: 1\n DOCKER_BUILD_SUMMARY: false\n DOCKER_BUILD_RECORD_UPLOAD: false\n with:\n context: ./packages\n file: ./packages/bytebot-agent/Dockerfile\n platforms: linux/amd64,linux/arm64\n push: true\n cache-from: type=gha\n cache-to: type=gha,mode=max\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n"
},
{
"path": ".github/workflows/build-desktop.yaml",
"content": "name: Build Desktop\n\non:\n push:\n branches:\n - main\n paths:\n - \"docker/**\"\n - \"packages/bytebotd/**\"\n\npermissions:\n contents: read\n packages: write\n\njobs:\n docker:\n runs-on: ubuntu-22.04\n\n steps:\n # 1. Check out code\n - uses: actions/checkout@v4\n\n # 2. Enable QEMU so the amd64 runner can cross‑build arm64\n - uses: docker/setup-qemu-action@v3\n\n # 3. Set up Buildx builder\n - uses: docker/setup-buildx-action@v3\n\n # 4. Generate OCI labels + the single \"edge\" tag\n - name: Docker meta\n id: meta\n uses: docker/metadata-action@v5\n with:\n images: ghcr.io/bytebot-ai/bytebot-desktop\n tags: type=edge\n\n # 5. Log in to GHCR\n - name: Login to GitHub Container Registry\n uses: docker/login-action@v3\n with:\n registry: ghcr.io\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n\n # 6. Build & push a multi‑arch image\n - name: Build and push\n uses: docker/build-push-action@v6\n env:\n BUILDX_NO_DEFAULT_ATTESTATIONS: 1 # hide \"unknown/unknown\" in GHCR\n DOCKER_BUILD_SUMMARY: false # keep logs concise\n DOCKER_BUILD_RECORD_UPLOAD: false\n with:\n context: ./packages/\n file: ./packages/bytebotd/Dockerfile\n platforms: linux/amd64,linux/arm64 # build both archs in one go\n push: true\n cache-from: type=gha\n cache-to: type=gha,mode=max\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n"
},
{
"path": ".github/workflows/build-ui.yaml",
"content": "name: Build UI\n\non:\n push:\n branches:\n - main\n paths:\n - \"packages/bytebot-ui/**\"\n - \"packages/shared/**\"\n\npermissions:\n contents: read\n packages: write\n\njobs:\n docker:\n runs-on: ubuntu-22.04\n\n steps:\n - uses: actions/checkout@v4\n - uses: docker/setup-qemu-action@v3\n - uses: docker/setup-buildx-action@v3\n - name: Docker meta\n id: meta\n uses: docker/metadata-action@v5\n with:\n images: ghcr.io/bytebot-ai/bytebot-ui\n tags: type=edge\n - name: Login to GitHub Container Registry\n uses: docker/login-action@v3\n with:\n registry: ghcr.io\n username: ${{ github.actor }}\n password: ${{ secrets.GITHUB_TOKEN }}\n - name: Build and push\n uses: docker/build-push-action@v6\n env:\n BUILDX_NO_DEFAULT_ATTESTATIONS: 1\n DOCKER_BUILD_SUMMARY: false\n DOCKER_BUILD_RECORD_UPLOAD: false\n with:\n context: ./packages\n file: ./packages/bytebot-ui/Dockerfile\n platforms: linux/amd64,linux/arm64\n\n push: true\n cache-from: type=gha\n cache-to: type=gha,mode=max\n tags: ${{ steps.meta.outputs.tags }}\n labels: ${{ steps.meta.outputs.labels }}\n"
},
{
"path": ".gitignore",
"content": "# Logs\nlogs\n*.log\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\nlerna-debug.log*\n.pnpm-debug.log*\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\nreport.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json\n\n# Runtime data\npids\n*.pid\n*.seed\n*.pid.lock\n\n# Directory for instrumented libs generated by jscoverage/JSCover\nlib-cov\n\n# Coverage directory used by tools like istanbul\ncoverage\n*.lcov\n\n# nyc test coverage\n.nyc_output\n\n# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)\n.grunt\n\n# Bower dependency directory (https://bower.io/)\nbower_components\n\n# node-waf configuration\n.lock-wscript\n\n# Compiled binary addons (https://nodejs.org/api/addons.html)\nbuild/Release\n\n# Dependency directories\nnode_modules/\njspm_packages/\n\n# Snowpack dependency directory (https://snowpack.dev/)\nweb_modules/\n\n# TypeScript cache\n*.tsbuildinfo\n\n# Optional npm cache directory\n.npm\n\n# Optional eslint cache\n.eslintcache\n\n# Optional stylelint cache\n.stylelintcache\n\n# Microbundle cache\n.rpt2_cache/\n.rts2_cache_cjs/\n.rts2_cache_es/\n.rts2_cache_umd/\n\n# Optional REPL history\n.node_repl_history\n\n# Output of 'npm pack'\n*.tgz\n\n# Yarn Integrity file\n.yarn-integrity\n\n# dotenv environment variable files\n.env\n.env.development.local\n.env.test.local\n.env.production.local\n.env.local\n\n# parcel-bundler cache (https://parceljs.org/)\n.cache\n.parcel-cache\n\n# Next.js build output\n.next\nout\n\n# Nuxt.js build / generate output\n.nuxt\ndist\n\n# Gatsby files\n.cache/\n# Comment in the public line in if your project uses Gatsby and not Next.js\n# https://nextjs.org/blog/next-9-1#public-directory-support\n# public\n\n# vuepress build output\n.vuepress/dist\n\n# vuepress v2.x temp and cache directory\n.temp\n.cache\n\n# Docusaurus cache and generated files\n.docusaurus\n\n# Serverless directories\n.serverless/\n\n# FuseBox cache\n.fusebox/\n\n# DynamoDB Local files\n.dynamodb/\n\n# TernJS port file\n.tern-port\n\n# Stores VSCode versions used for testing VSCode extensions\n.vscode-test\n\n# yarn v2\n.yarn/cache\n.yarn/unplugged\n.yarn/build-state.yml\n.yarn/install-state.gz\n.pnp.*\n\n\n*.qcow2\n*.iso\n*.img\n*.vdi\n*.vmdk\n*.vhdx\n*.vhd\n\n\n# compiled output\nagent/dist\nagent/node_modules\nagent/build\n\n# Logs\nlogs\n*.log\nnpm-debug.log*\npnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\nlerna-debug.log*\n\n# OS\n.DS_Store\n\n# Tests\nagent/coverage\nagent/.nyc_output\n\n# IDEs and editors\nagent/.idea\nagent/.project\nagent/.classpath\n.c9/\n*.launch\n.settings/\n*.sublime-workspace\n\n# IDE - VSCode\n.vscode/*\n!.vscode/settings.json\n!.vscode/tasks.json\n!.vscode/launch.json\n!.vscode/extensions.json\n\n# dotenv environment variable files\n.env.development.local\n.env.test.local\n.env.production.local\n.env.local\n\n# temp directory\n.temp\n.tmp\n\n# Runtime data\npids\n*.pid\n*.seed\n*.pid.lock\n\n# Diagnostic reports (https://nodejs.org/api/report.html)\nreport.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json\n\n# QEMU\n*.qcow2\n"
},
{
"path": ".prettierignore",
"content": "# Ignore formatting in docs folder\n/docs/**\n"
},
{
"path": "LICENSE",
"content": " Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright [yyyy] [name of copyright owner]\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License.\n"
},
{
"path": "README.md",
"content": "
\n\n\n\n# Bytebot: Open-Source AI Desktop Agent\n\n\n\n**An AI that has its own computer to complete tasks for you**\n\n[](https://railway.com/deploy/bytebot?referralCode=L9lKXQ)\n\n[](https://github.com/bytebot-ai/bytebot/tree/main/docker)\n[](LICENSE)\n[](https://discord.com/invite/d9ewZkWPTP)\n\n[🌐 Website](https://bytebot.ai) • [📚 Documentation](https://docs.bytebot.ai) • [💬 Discord](https://discord.com/invite/d9ewZkWPTP) • [𝕏 Twitter](https://x.com/bytebot_ai)\n\n\n[Deutsch](https://zdoc.app/de/bytebot-ai/bytebot) | \n[Español](https://zdoc.app/es/bytebot-ai/bytebot) | \n[français](https://zdoc.app/fr/bytebot-ai/bytebot) | \n[日本語](https://zdoc.app/ja/bytebot-ai/bytebot) | \n[한국어](https://zdoc.app/ko/bytebot-ai/bytebot) | \n[Português](https://zdoc.app/pt/bytebot-ai/bytebot) | \n[Русский](https://zdoc.app/ru/bytebot-ai/bytebot) | \n[中文](https://zdoc.app/zh/bytebot-ai/bytebot)\n
\n\n---\n\nhttps://github.com/user-attachments/assets/f271282a-27a3-43f3-9b99-b34007fdd169\n\nhttps://github.com/user-attachments/assets/72a43cf2-bd87-44c5-a582-e7cbe176f37f\n\n## What is a Desktop Agent?\n\nA desktop agent is an AI that has its own computer. Unlike browser-only agents or traditional RPA tools, Bytebot comes with a full virtual desktop where it can:\n\n- Use any application (browsers, email clients, office tools, IDEs)\n- Download and organize files with its own file system\n- Log into websites and applications using password managers\n- Read and process documents, PDFs, and spreadsheets\n- Complete complex multi-step workflows across different programs\n\nThink of it as a virtual employee with their own computer who can see the screen, move the mouse, type on the keyboard, and complete tasks just like a human would.\n\n## Why Give AI Its Own Computer?\n\nWhen AI has access to a complete desktop environment, it unlocks capabilities that aren't possible with browser-only agents or API integrations:\n\n### Complete Task Autonomy\n\nGive Bytebot a task like \"Download all invoices from our vendor portals and organize them into a folder\" and it will:\n\n- Open the browser\n- Navigate to each portal\n- Handle authentication (including 2FA via password managers)\n- Download the files to its local file system\n- Organize them into a folder\n\n### Process Documents\n\nUpload files directly to Bytebot's desktop and it can:\n\n- Read entire PDFs into its context\n- Extract data from complex documents\n- Cross-reference information across multiple files\n- Create new documents based on analysis\n- Handle formats that APIs can't access\n\n### Use Real Applications\n\nBytebot isn't limited to web interfaces. It can:\n\n- Use desktop applications like text editors, VS Code, or email clients\n- Run scripts and command-line tools\n- Install new software as needed\n- Configure applications for specific workflows\n\n## Quick Start\n\n### Deploy in 2 Minutes\n\n**Option 1: Railway (Easiest)**\n[](https://railway.com/deploy/bytebot?referralCode=L9lKXQ)\n\nJust click and add your AI provider API key.\n\n**Option 2: Docker Compose**\n\n```bash\ngit clone https://github.com/bytebot-ai/bytebot.git\ncd bytebot\n\n# Add your AI provider key (choose one)\necho \"ANTHROPIC_API_KEY=sk-ant-...\" > docker/.env\n# Or: echo \"OPENAI_API_KEY=sk-...\" > docker/.env\n# Or: echo \"GEMINI_API_KEY=...\" > docker/.env\n\ndocker-compose -f docker/docker-compose.yml up -d\n\n# Open http://localhost:9992\n```\n\n[Full deployment guide →](https://docs.bytebot.ai/quickstart)\n\n## How It Works\n\nBytebot consists of four integrated components:\n\n1. **Virtual Desktop**: A complete Ubuntu Linux environment with pre-installed applications\n2. **AI Agent**: Understands your tasks and controls the desktop to complete them\n3. **Task Interface**: Web UI where you create tasks and watch Bytebot work\n4. **APIs**: REST endpoints for programmatic task creation and desktop control\n\n### Key Features\n\n- **Natural Language Tasks**: Just describe what you need done\n- **File Uploads**: Drop files onto tasks for Bytebot to process\n- **Live Desktop View**: Watch Bytebot work in real-time\n- **Takeover Mode**: Take control when you need to help or configure something\n- **Password Manager Support**: Install 1Password, Bitwarden, etc. for automatic authentication\n- **Persistent Environment**: Install programs and they stay available for future tasks\n\n## Example Tasks\n\n### Basic Examples\n\n```\n\"Go to Wikipedia and create a summary of quantum computing\"\n\"Research flights from NYC to London and create a comparison document\"\n\"Take screenshots of the top 5 news websites\"\n```\n\n### Document Processing\n\n```\n\"Read the uploaded contracts.pdf and extract all payment terms and deadlines\"\n\"Process these 5 invoice PDFs and create a summary report\"\n\"Download and analyze the latest financial report and answer: What were the key risks mentioned?\"\n```\n\n### Multi-Application Workflows\n\n```\n\"Download last month's bank statements from our three banks and consolidate them\"\n\"Check all our vendor portals for new invoices and create a summary report\"\n\"Log into our CRM, export the customer list, and update records in the ERP system\"\n```\n\n## Programmatic Control\n\n### Create Tasks via API\n\n```python\nimport requests\n\n# Simple task\nresponse = requests.post('http://localhost:9991/tasks', json={\n 'description': 'Download the latest sales report and create a summary'\n})\n\n# Task with file upload\nfiles = {'files': open('contracts.pdf', 'rb')}\nresponse = requests.post('http://localhost:9991/tasks',\n data={'description': 'Review these contracts for important dates'},\n files=files\n)\n```\n\n### Direct Desktop Control\n\n```bash\n# Take a screenshot\ncurl -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"screenshot\"}'\n\n# Click at specific coordinates\ncurl -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"click_mouse\", \"coordinate\": [500, 300]}'\n```\n\n[Full API documentation →](https://docs.bytebot.ai/api-reference/introduction)\n\n## Setting Up Your Desktop Agent\n\n### 1. Deploy Bytebot\n\nUse one of the deployment methods above to get Bytebot running.\n\n### 2. Configure the Desktop\n\nUse the Desktop tab in the UI to:\n\n- Install additional programs you need\n- Set up password managers for authentication\n- Configure applications with your preferences\n- Log into websites you want Bytebot to access\n\n### 3. Start Giving Tasks\n\nCreate tasks in natural language and watch Bytebot complete them using the configured desktop.\n\n## Use Cases\n\n### Business Process Automation\n\n- Invoice processing and data extraction\n- Multi-system data synchronization\n- Report generation from multiple sources\n- Compliance checking across platforms\n\n### Development & Testing\n\n- Automated UI testing\n- Cross-browser compatibility checks\n- Documentation generation with screenshots\n- Code deployment verification\n\n### Research & Analysis\n\n- Competitive analysis across websites\n- Data gathering from multiple sources\n- Document analysis and summarization\n- Market research compilation\n\n## Architecture\n\nBytebot is built with:\n\n- **Desktop**: Ubuntu 22.04 with XFCE, Firefox, VS Code, and other tools\n- **Agent**: NestJS service that coordinates AI and desktop actions\n- **UI**: Next.js application for task management\n- **AI Support**: Works with Anthropic Claude, OpenAI GPT, Google Gemini\n- **Deployment**: Docker containers for easy self-hosting\n\n## Why Self-Host?\n\n- **Data Privacy**: Everything runs on your infrastructure\n- **Full Control**: Customize the desktop environment as needed\n- **No Limits**: Use your own AI API keys without platform restrictions\n- **Flexibility**: Install any software, access any systems\n\n## Advanced Features\n\n### Multiple AI Providers\n\nUse any AI provider through our [LiteLLM integration](https://docs.bytebot.ai/deployment/litellm):\n\n- Azure OpenAI\n- AWS Bedrock\n- Local models via Ollama\n- 100+ other providers\n\n### Enterprise Deployment\n\nDeploy on Kubernetes with Helm:\n\n```bash\n# Clone the repository\ngit clone https://github.com/bytebot-ai/bytebot.git\ncd bytebot\n\n# Install with Helm\nhelm install bytebot ./helm \\\n --set agent.env.ANTHROPIC_API_KEY=sk-ant-...\n```\n\n[Enterprise deployment guide →](https://docs.bytebot.ai/deployment/helm)\n\n## Community & Support\n\n- **Discord**: [Join our community](https://discord.com/invite/d9ewZkWPTP) for help and discussions\n- **Documentation**: Comprehensive guides at [docs.bytebot.ai](https://docs.bytebot.ai)\n- **GitHub Issues**: Report bugs and request features\n\n## Contributing\n\nWe welcome contributions! Whether it's:\n\n- 🐛 Bug fixes\n- ✨ New features\n- 📚 Documentation improvements\n- 🌐 Translations\n\nPlease:\n\n1. Check existing [issues](https://github.com/bytebot-ai/bytebot/issues) first\n2. Open an issue to discuss major changes\n3. Submit PRs with clear descriptions\n4. Join our [Discord](https://discord.com/invite/d9ewZkWPTP) to discuss ideas\n\n## License\n\nBytebot is open source under the Apache 2.0 license.\n\n---\n\n
\n\n**Give your AI its own computer. See what it can do.**\n\n[](https://railway.com/deploy/bytebot?referralCode=L9lKXQ)\n\nBuilt by [Tantl Labs](https://tantl.com) and the open source community\n\n
\n"
},
{
"path": "docker/bytebot-desktop.Dockerfile",
"content": "# Extend the pre-built bytebot-desktop image\nFROM ghcr.io/bytebot-ai/bytebot-desktop:edge\n\n# Add additional packages, applications, or customizations here\n\n# Expose the bytebotd service port\nEXPOSE 9990\n\n# Start the bytebotd service\nCMD [\"/usr/bin/supervisord\", \"-c\", \"/etc/supervisor/conf.d/supervisord.conf\", \"-n\"]\n"
},
{
"path": "docker/docker-compose-claude-code.yml",
"content": "name: bytebot\n\nservices:\n bytebot-desktop:\n # Build from source\n build:\n context: ../packages/\n dockerfile: bytebotd/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-desktop:edge\n shm_size: \"2g\"\n container_name: bytebot-desktop\n restart: unless-stopped\n hostname: computer\n privileged: true\n ports:\n - \"9990:9990\" # bytebotd service & noVNC\n environment:\n - DISPLAY=:0\n networks:\n - bytebot-network\n\n postgres:\n image: postgres:16-alpine\n container_name: bytebot-postgres\n restart: unless-stopped\n ports:\n - \"5432:5432\"\n environment:\n - POSTGRES_PASSWORD=postgres\n - POSTGRES_USER=postgres\n - POSTGRES_DB=bytebotdb\n networks:\n - bytebot-network\n volumes:\n - postgres_data:/var/lib/postgresql/data\n\n bytebot-agent-cc:\n build:\n context: ../packages/\n dockerfile: bytebot-agent-cc/Dockerfile\n container_name: bytebot-agent-cc\n restart: unless-stopped\n ports:\n - \"9991:9991\"\n environment:\n - DATABASE_URL=${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/bytebotdb}\n - BYTEBOT_DESKTOP_BASE_URL=${BYTEBOT_DESKTOP_BASE_URL:-http://bytebot-desktop:9990}\n - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}\n depends_on:\n - postgres\n networks:\n - bytebot-network\n\n bytebot-ui:\n build:\n context: ../packages/\n dockerfile: bytebot-ui/Dockerfile\n args:\n - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent-cc:9991}\n - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-ui:edge\n container_name: bytebot-ui\n restart: unless-stopped\n ports:\n - \"9992:9992\"\n environment:\n - NODE_ENV=production\n - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent-cc:9991}\n - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}\n depends_on:\n - bytebot-agent-cc\n networks:\n - bytebot-network\n\nnetworks:\n bytebot-network:\n driver: bridge\n\nvolumes:\n postgres_data:\n"
},
{
"path": "docker/docker-compose.core.yml",
"content": "name: bytebot\n\nservices:\n bytebot-desktop:\n # Build from source\n build:\n context: ../packages/\n dockerfile: bytebotd/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-desktop:edge\n shm_size: \"2g\"\n container_name: bytebot-desktop\n restart: unless-stopped\n hostname: computer\n privileged: true\n ports:\n - \"9990:9990\" # bytebotd service & noVNC\n environment:\n - DISPLAY=:0\n"
},
{
"path": "docker/docker-compose.development.yml",
"content": "## docker-compose file that spins up a bytebot-desktop container\n## and a postgres container. bytebot-ui and bytebot-agent are not included\n## in this file, and can be run separately using npm, allowing for\n## easier local development.\n\nname: bytebot\n\nservices:\n bytebot-desktop:\n # Build from source\n build:\n context: ../packages/\n dockerfile: bytebotd/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-desktop:edge\n shm_size: \"2g\"\n container_name: bytebot-desktop\n restart: unless-stopped\n hostname: computer\n privileged: true\n ports:\n - \"9990:9990\" # bytebotd service & noVNC\n environment:\n - DISPLAY=:0\n networks:\n - bytebot-network\n\n postgres:\n image: postgres:16-alpine\n container_name: bytebot-postgres\n restart: unless-stopped\n ports:\n - \"5432:5432\"\n environment:\n - POSTGRES_PASSWORD=postgres\n - POSTGRES_USER=postgres\n - POSTGRES_DB=bytebotdb\n networks:\n - bytebot-network\n volumes:\n - postgres_data:/var/lib/postgresql/data\n\nnetworks:\n bytebot-network:\n driver: bridge\n\nvolumes:\n postgres_data:\n"
},
{
"path": "docker/docker-compose.proxy.yml",
"content": "name: bytebot\n\nservices:\n bytebot-desktop:\n # Build from source\n build:\n context: ../packages/\n dockerfile: bytebotd/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-desktop:edge\n shm_size: \"2g\"\n container_name: bytebot-desktop\n restart: unless-stopped\n hostname: computer\n privileged: true\n ports:\n - \"9990:9990\" # bytebotd service & noVNC\n environment:\n - DISPLAY=:0\n networks:\n - bytebot-network\n\n postgres:\n image: postgres:16-alpine\n container_name: bytebot-postgres\n restart: unless-stopped\n ports:\n - \"5432:5432\"\n environment:\n - POSTGRES_PASSWORD=postgres\n - POSTGRES_USER=postgres\n - POSTGRES_DB=bytebotdb\n networks:\n - bytebot-network\n volumes:\n - postgres_data:/var/lib/postgresql/data\n\n bytebot-agent:\n build:\n context: ../packages/\n dockerfile: bytebot-agent/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-agent:edge\n container_name: bytebot-agent\n restart: unless-stopped\n ports:\n - \"9991:9991\"\n environment:\n - DATABASE_URL=${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/bytebotdb}\n - BYTEBOT_DESKTOP_BASE_URL=${BYTEBOT_DESKTOP_BASE_URL:-http://bytebot-desktop:9990}\n - BYTEBOT_LLM_PROXY_URL=${BYTEBOT_LLM_PROXY_URL:-http://bytebot-llm-proxy:4000}\n depends_on:\n - postgres\n networks:\n - bytebot-network\n\n bytebot-llm-proxy:\n build:\n context: ../packages/\n dockerfile: bytebot-llm-proxy/Dockerfile\n ports:\n - \"4000:4000\"\n environment:\n - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}\n - OPENAI_API_KEY=${OPENAI_API_KEY}\n - GEMINI_API_KEY=${GEMINI_API_KEY}\n networks:\n - bytebot-network\n\n bytebot-ui:\n build:\n context: ../packages/\n dockerfile: bytebot-ui/Dockerfile\n args:\n - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent:9991}\n - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-ui:edge\n container_name: bytebot-ui\n restart: unless-stopped\n ports:\n - \"9992:9992\"\n environment:\n - NODE_ENV=production\n - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent:9991}\n - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}\n depends_on:\n - bytebot-agent\n networks:\n - bytebot-network\n\nnetworks:\n bytebot-network:\n driver: bridge\n\nvolumes:\n postgres_data:\n"
},
{
"path": "docker/docker-compose.yml",
"content": "name: bytebot\n\nservices:\n bytebot-desktop:\n # Build from source\n build:\n context: ../packages/\n dockerfile: bytebotd/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-desktop:edge\n shm_size: \"2g\"\n container_name: bytebot-desktop\n restart: unless-stopped\n hostname: computer\n privileged: true\n ports:\n - \"9990:9990\" # bytebotd service & noVNC\n environment:\n - DISPLAY=:0\n networks:\n - bytebot-network\n\n postgres:\n image: postgres:16-alpine\n container_name: bytebot-postgres\n restart: unless-stopped\n ports:\n - \"5432:5432\"\n environment:\n - POSTGRES_PASSWORD=postgres\n - POSTGRES_USER=postgres\n - POSTGRES_DB=bytebotdb\n networks:\n - bytebot-network\n volumes:\n - postgres_data:/var/lib/postgresql/data\n\n bytebot-agent:\n build:\n context: ../packages/\n dockerfile: bytebot-agent/Dockerfile\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-agent:edge\n container_name: bytebot-agent\n restart: unless-stopped\n ports:\n - \"9991:9991\"\n environment:\n - DATABASE_URL=${DATABASE_URL:-postgresql://postgres:postgres@postgres:5432/bytebotdb}\n - BYTEBOT_DESKTOP_BASE_URL=${BYTEBOT_DESKTOP_BASE_URL:-http://bytebot-desktop:9990}\n - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY}\n - OPENAI_API_KEY=${OPENAI_API_KEY}\n - GEMINI_API_KEY=${GEMINI_API_KEY}\n depends_on:\n - postgres\n networks:\n - bytebot-network\n\n bytebot-ui:\n build:\n context: ../packages/\n dockerfile: bytebot-ui/Dockerfile\n args:\n - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent:9991}\n - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}\n # Use pre-built image\n image: ghcr.io/bytebot-ai/bytebot-ui:edge\n container_name: bytebot-ui\n restart: unless-stopped\n ports:\n - \"9992:9992\"\n environment:\n - NODE_ENV=production\n - BYTEBOT_AGENT_BASE_URL=${BYTEBOT_AGENT_BASE_URL:-http://bytebot-agent:9991}\n - BYTEBOT_DESKTOP_VNC_URL=${BYTEBOT_DESKTOP_VNC_URL:-http://bytebot-desktop:9990/websockify}\n depends_on:\n - bytebot-agent\n networks:\n - bytebot-network\n\nnetworks:\n bytebot-network:\n driver: bridge\n\nvolumes:\n postgres_data:\n"
},
{
"path": "docs/api-reference/agent/tasks.mdx",
"content": "---\ntitle: 'Tasks API'\ndescription: 'Reference documentation for the Bytebot Agent Tasks API'\n---\n\n## Tasks API\n\nThe Tasks API allows you to manage tasks in the Bytebot agent system. It's available at `http://localhost:9991/tasks` when running the full agent setup.\n\n## Task Model\n\n```typescript\n{\n id: string;\n description: string; \n status: 'PENDING' | 'IN_PROGRESS' | 'NEEDS_HELP' | 'NEEDS_REVIEW' | 'COMPLETED' | 'CANCELLED' | 'FAILED';\n priority: 'LOW' | 'MEDIUM' | 'HIGH' | 'URGENT';\n createdAt: string; \n updatedAt: string; \n}\n```\n\n## Endpoints\n\n### Create Task\n\nCreate a new task for the agent to process.\n\n\n Create a new task\n\n\n#### Request Body\n\n```json\n{\n \"description\": \"This is a description of the task\",\n \"priority\": \"MEDIUM\" // Optional: LOW, MEDIUM, HIGH, URGENT\n}\n```\n\n#### With File Upload\n\nTo upload files with a task, use `multipart/form-data`:\n\n```bash\ncurl -X POST http://localhost:9991/tasks \\\n -F \"description=Analyze the uploaded contracts and extract key terms\" \\\n -F \"priority=HIGH\" \\\n -F \"files=@contract1.pdf\" \\\n -F \"files=@contract2.pdf\"\n```\n\nUploaded files are automatically saved to the desktop and can be referenced in the task description.\n\n#### Response\n\n```json\n{\n \"id\": \"task-123\",\n \"description\": \"This is a description of the task\",\n \"status\": \"PENDING\",\n \"priority\": \"MEDIUM\",\n \"createdAt\": \"2025-04-14T12:00:00Z\",\n \"updatedAt\": \"2025-04-14T12:00:00Z\"\n}\n```\n\n### Get All Tasks\n\nRetrieve a list of all tasks.\n\n\n Get all tasks\n\n\n#### Response\n\n```json\n[\n {\n \"id\": \"task-123\",\n \"description\": \"This is a description of the task\",\n \"status\": \"PENDING\",\n \"priority\": \"MEDIUM\",\n \"createdAt\": \"2025-04-14T12:00:00Z\",\n \"updatedAt\": \"2025-04-14T12:00:00Z\"\n },\n // ...more tasks\n]\n```\n\n### Get In-Progress Task\n\nRetrieve the currently in-progress task, if any.\n\n\n Get the currently in-progress task\n\n\n#### Response\n\n```json\n{\n \"id\": \"task-123\",\n \"description\": \"This is a description of the task\",\n \"status\": \"IN_PROGRESS\",\n \"priority\": \"MEDIUM\",\n \"createdAt\": \"2025-04-14T12:00:00Z\",\n \"updatedAt\": \"2025-04-14T12:00:00Z\"\n}\n```\n\nIf no task is in progress, the response will be `null`.\n\n### Get Task by ID\n\nRetrieve a specific task by its ID.\n\n\n Get a task by ID\n\n\n#### Response\n\n```json\n{\n \"id\": \"task-123\",\n \"description\": \"This is a description of the task\",\n \"status\": \"PENDING\",\n \"priority\": \"MEDIUM\",\n \"createdAt\": \"2025-04-14T12:00:00Z\",\n \"updatedAt\": \"2025-04-14T12:00:00Z\",\n \"messages\": [\n {\n \"id\": \"msg-456\",\n \"content\": [\n {\n \"type\": \"text\",\n \"text\": \"This is a message\"\n }\n ],\n \"role\": \"USER\",\n \"taskId\": \"task-123\",\n \"createdAt\": \"2025-04-14T12:05:00Z\",\n \"updatedAt\": \"2025-04-14T12:05:00Z\"\n }\n // ...more messages\n ]\n}\n```\n\n### Update Task\n\nUpdate an existing task.\n\n\n Update a task\n\n\n#### Request Body\n\n```json\n{\n \"status\": \"COMPLETED\", \n \"priority\": \"HIGH\" \n}\n```\n\n#### Response\n\n```json\n{\n \"id\": \"task-123\",\n \"description\": \"This is a description of the task\",\n \"status\": \"COMPLETED\",\n \"priority\": \"HIGH\",\n \"createdAt\": \"2025-04-14T12:00:00Z\",\n \"updatedAt\": \"2025-04-14T12:01:00Z\"\n}\n```\n\n### Delete Task\n\nDelete a task.\n\n\n Delete a task\n\n\n#### Response\n\nStatus code `204 No Content` with an empty response body.\n\n## Message Content Structure\n\nMessages in the Bytebot agent system use a content block structure compatible with Anthropic's Claude API:\n\n```typescript\ntype MessageContent = MessageContentBlock[];\n\ninterface MessageContentBlock {\n type: string; \n [key: string]: any; \n}\n\ninterface TextContentBlock {\n type: \"text\";\n text: string;\n}\n\ninterface ImageContentBlock {\n type: \"image\";\n source: {\n type: \"base64\";\n media_type: string; \n data: string; \n };\n}\n```\n\n## Error Responses\n\nThe API may return the following error responses:\n\n| Status Code | Description |\n|-------------|--------------------------------------------|\n| `400` | Bad Request - Invalid parameters |\n| `404` | Not Found - Resource does not exist |\n| `500` | Internal Server Error - Server side error |\n\nExample error response:\n\n```json\n{\n \"statusCode\": 404,\n \"message\": \"Task with ID task-123 not found\",\n \"error\": \"Not Found\"\n}\n```\n\n## Code Examples\n\n\n```javascript JavaScript\nconst axios = require('axios');\n\nasync function createTask(description) {\n const response = await axios.post('http://localhost:9991/tasks', {\n description\n });\n return response.data;\n}\n\nasync function findInProgressTask() {\n const response = await axios.get('http://localhost:9991/tasks/in-progress');\n return response.data;\n}\n\n// Example usage\nasync function main() {\n // Create a new task\n const task = await createTask('Compare React, Vue, and Angular for a new project');\n console.log('Created task:', task);\n \n // Get current in-progress task\n const inProgressTask = await findInProgressTask();\n console.log('In progress task:', inProgressTask);\n}\n```\n\n```python Python\nimport requests\n\ndef create_task(description):\n response = requests.post(\n \"http://localhost:9991/tasks\",\n json={\n \"description\": description\n }\n )\n return response.json()\n\ndef find_in_progress_task():\n response = requests.get(\"http://localhost:9991/tasks/in-progress\")\n return response.json()\n\n# Example usage\ndef main():\n # Create a new task\n task = create_task(\"Compare React, Vue, and Angular for a new project\")\n print(f\"Created task: {task}\")\n \n # Get current in-progress task\n in_progress_task = find_in_progress_task()\n print(f\"In progress task: {in_progress_task}\")\n```\n\n```curl cURL\n# Create a new task\ncurl -X POST http://localhost:9991/tasks \\\n -H \"Content-Type: application/json\" \\\n -d '{\n \"description\": \"Compare React, Vue, and Angular for a new project\"\n }'\n\n# Get current in-progress task\ncurl -X GET http://localhost:9991/tasks/in-progress\n```\n\n"
},
{
"path": "docs/api-reference/agent/ui.mdx",
"content": "---\ntitle: 'Task UI'\ndescription: 'Documentation for the Bytebot Task UI'\n---\n\n## Bytebot Task UI\n\nThe Bytebot Task UI provides a web-based interface for interacting with the Bytebot agent system. It combines a action feed with an embedded noVNC viewer, allowing you to watch it perform task on the desktop in real-time.\n\n\n\n## Accessing the UI\n\nWhen running the full Bytebot agent system, the Task UI is available at:\n\n```\nhttp://localhost:9992\n```\n\n## UI Components\n\n### Task Management Panel\n\nThe task management panel allows you to:\n\n- Create new tasks\n- View existing tasks\n- See task status and priority\n- Select a task to work on\n\n\n\n### Task Interface\n\nThe main task interface provides:\n\n- Task history with the agent\n- Support for markdown formatting in messages\n- Automatic scrolling to new messages\n\n### Desktop Viewer\n\nThe embedded noVNC viewer displays:\n\n- Real-time view of the desktop environment\n- Visual feedback of agent actions\n- Option to expand to take over the desktop\n- Connection status indicator\n\n## Features\n\n### Task Creation\n\nTo create a new task:\n\n1. Enter a description for the task\n2. Click \"Start Task\" button (or press Enter)\n\n### Conversation Controls\n\nThe task interface supports:\n\n- Text messages with markdown formatting\n- Viewing image content in messages\n- Displaying tool use actions\n- Showing tool results\n\n### Desktop Interaction\n\nWhile primarily for viewing, the desktop panel allows:\n\n- Taking over the desktop\n- Real-time monitoring of agent actions\n\n## Message Types\n\nThe task interface displays different types of messages based on Bytebot's content block structure:\n\n- **User Messages**: Your instructions and queries\n- **Assistant Messages**: Responses from the agent, which may include:\n - **Text Content Blocks**: Markdown-formatted text responses\n - **Image Content Blocks**: Images generated or captured\n - **Tool Use Content Blocks**: Computer actions being performed\n - **Tool Result Content Blocks**: Results of computer actions\n\nThe message content structure follows this format:\n\n```typescript\ninterface Message {\n id: string;\n content: MessageContentBlock[];\n role: Role; // \"USER\" or \"ASSISTANT\"\n createdAt?: string;\n}\n\ninterface MessageContentBlock {\n type: string;\n [key: string]: any;\n}\n\ninterface TextContentBlock extends MessageContentBlock {\n type: \"text\";\n text: string;\n}\n\ninterface ImageContentBlock extends MessageContentBlock {\n type: \"image\";\n source: {\n type: \"base64\";\n media_type: string;\n data: string;\n };\n}\n```\n\n## Technical Details\n\nThe Bytebot Task UI is built with:\n\n- **Next.js**: React framework for the frontend\n- **Tailwind CSS**: For styling\n- **ReactMarkdown**: For rendering markdown content\n- **noVNC**: For the embedded desktop viewer\n\n## Troubleshooting\n\n### Connection Issues\n\nIf you experience connection issues:\n\n1. Ensure all Bytebot services are running\n2. Check that ports 9990, 9991, and 9992 are accessible\n3. Try refreshing the browser\n4. Check browser console for error messages\n\n### Desktop Viewer Issues\n\nIf the desktop viewer is not displaying:\n\n1. Ensure the Bytebot container is running\n2. Check that the noVNC service is accessible at port 9990\n\n### Message Display Issues\n\nIf messages are not displaying correctly:\n\n1. Check that the message content is properly formatted\n2. Ensure the agent service is processing task correctly\n3. Check the browser console for any rendering errors\n4. Try refreshing the browser\n"
},
{
"path": "docs/api-reference/computer-use/examples.mdx",
"content": "---\ntitle: \"Computer Use API Examples\"\ndescription: \"Code examples for common automation scenarios using the Bytebot API\"\n---\n\n## Basic Examples\n\nHere are some practical examples of how to use the Computer Use API in different programming languages.\n\n### Using cURL\n\n\n```bash Opening a Web Browser\n# Move to Firefox/Chrome icon in the dock and click it\ncurl -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"move_mouse\", \"coordinates\": {\"x\": 100, \"y\": 960}}'\n\ncurl -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"click_mouse\", \"button\": \"left\", \"clickCount\": 1}'\n\n````\n\n```bash Taking and Saving a Screenshot\n# Take a screenshot\nresponse=$(curl -s -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"screenshot\"}')\n\n# Extract the base64 image data and save to a file\necho $response | jq -r '.data.image' | base64 -d > screenshot.png\n````\n\n```bash Typing and Keyboard Shortcuts\n# Type text in a text editor\ncurl -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"type_text\", \"text\": \"Hello, this is an automated test!\", \"delay\": 30}'\n\n# Press Ctrl+S to save\ncurl -X POST http://localhost:9990/computer-use \\\n -H \"Content-Type: application/json\" \\\n -d '{\"action\": \"press_keys\", \"key\": \"s\", \"modifiers\": [\"control\"]}'\n```\n\n\n\n### Python Examples\n\n\n```python Basic Automation\nimport requests\nimport json\nimport base64\nimport time\nfrom io import BytesIO\nfrom PIL import Image\n\ndef control_computer(action, **params):\nurl = \"http://localhost:9990/computer-use\"\ndata = {\"action\": action, **params}\nresponse = requests.post(url, json=data)\nreturn response.json()\n\n# Open a web browser by clicking an icon\n\ncontrol_computer(\"move_mouse\", coordinates={\"x\": 100, \"y\": 960})\ncontrol_computer(\"click_mouse\", button=\"left\")\n\n# Wait for the browser to open\n\ncontrol_computer(\"wait\", duration=2000)\n\n# Type a URL\n\ncontrol_computer(\"type_text\", text=\"https://example.com\")\ncontrol_computer(\"press_keys\", key=\"enter\")\n\n````\n\n```python Screenshot and Analysis\nimport requests\nimport json\nimport base64\nimport cv2\nimport numpy as np\nfrom PIL import Image\nfrom io import BytesIO\n\ndef take_screenshot():\n url = \"http://localhost:9990/computer-use\"\n data = {\"action\": \"screenshot\"}\n response = requests.post(url, json=data)\n\n if response.json()[\"success\"]:\n img_data = base64.b64decode(response.json()[\"data\"][\"image\"])\n image = Image.open(BytesIO(img_data))\n return np.array(image)\n return None\n\n# Take a screenshot\nimg = take_screenshot()\n\n# Convert to grayscale for analysis\nif img is not None:\n gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)\n\n # Save the screenshot\n cv2.imwrite(\"screenshot.png\", img)\n\n # Perform image analysis (example: find edges)\n edges = cv2.Canny(gray, 100, 200)\n cv2.imwrite(\"edges.png\", edges)\n````\n\n```python Web Form Automation\nimport requests\nimport time\n\ndef control_computer(action, **params):\n url = \"http://localhost:9990/computer-use\"\n data = {\"action\": action, **params}\n response = requests.post(url, json=data)\n return response.json()\n\ndef fill_web_form(form_fields):\n # Click on the first form field\n control_computer(\"move_mouse\", coordinates=form_fields[0])\n control_computer(\"click_mouse\", button=\"left\")\n\n # Fill out each field\n for i, field in enumerate(form_fields):\n # Input the field value\n control_computer(\"type_text\", text=field[\"value\"])\n\n # If not the last field, press Tab to move to next field\n if i < len(form_fields) - 1:\n control_computer(\"press_keys\", key=\"tab\")\n time.sleep(0.5)\n\n # Submit the form by pressing Enter\n control_computer(\"press_keys\", key=\"enter\")\n\n# Example form fields with coordinates and values\nform_fields = [\n {\"x\": 500, \"y\": 300, \"value\": \"John Doe\"},\n {\"x\": 500, \"y\": 350, \"value\": \"john@example.com\"},\n {\"x\": 500, \"y\": 400, \"value\": \"Password123\"}\n]\n\nfill_web_form(form_fields)\n```\n\n\n\n### JavaScript/Node.js Examples\n\n\n```javascript Basic Automation\nconst axios = require('axios');\n\nasync function controlComputer(action, params = {}) {\nconst url = \"http://localhost:9990/computer-use\";\nconst data = { action, ...params };\n\ntry {\nconst response = await axios.post(url, data);\nreturn response.data;\n} catch (error) {\nconsole.error('Error:', error.message);\nreturn { success: false, error: error.message };\n}\n}\n\n// Example: Automate opening an application and typing\nasync function automateTextEditor() {\ntry {\n// Open text editor by clicking its icon\nawait controlComputer(\"move_mouse\", { coordinates: { x: 150, y: 960 } });\nawait controlComputer(\"click_mouse\", { button: \"left\" });\n\n // Wait for it to open\n await controlComputer(\"wait\", { duration: 2000 });\n\n // Type some text\n await controlComputer(\"type_text\", {\n text: \"This is an automated test using Node.js and Bytebot\",\n delay: 30\n });\n\n console.log(\"Automation completed successfully\");\n\n} catch (error) {\nconsole.error(\"Automation failed:\", error);\n}\n}\n\nautomateTextEditor();\n\n````\n\n```javascript Advanced: Screenshot Comparison\nconst axios = require('axios');\nconst fs = require('fs');\nconst { createCanvas, loadImage } = require('canvas');\nconst pixelmatch = require('pixelmatch');\n\nasync function controlComputer(action, params = {}) {\n const url = \"http://localhost:9990/computer-use\";\n const data = { action, ...params };\n\n try {\n const response = await axios.post(url, data);\n return response.data;\n } catch (error) {\n console.error('Error:', error.message);\n return { success: false, error: error.message };\n }\n}\n\nasync function compareScreenshots() {\n try {\n // Take first screenshot\n const screenshot1 = await controlComputer(\"screenshot\");\n\n // Do some actions\n await controlComputer(\"move_mouse\", { coordinates: { x: 500, y: 500 } });\n await controlComputer(\"click_mouse\", { button: \"left\" });\n await controlComputer(\"wait\", { duration: 1000 });\n\n // Take second screenshot\n const screenshot2 = await controlComputer(\"screenshot\");\n\n // Compare screenshots\n if (screenshot1.success && screenshot2.success) {\n const img1Data = Buffer.from(screenshot1.data.image, 'base64');\n const img2Data = Buffer.from(screenshot2.data.image, 'base64');\n\n fs.writeFileSync('screenshot1.png', img1Data);\n fs.writeFileSync('screenshot2.png', img2Data);\n\n // Now you could load and compare these images\n // This requires additional image comparison libraries\n console.log('Screenshots saved for comparison');\n }\n } catch (error) {\n console.error(\"Screenshot comparison failed:\", error);\n }\n}\n\ncompareScreenshots();\n````\n\n\n\n## File Operations\n\n### Writing Files\n\nThese examples show how to write files to the desktop environment:\n\n\n```python Python\nimport requests\nimport base64\n\ndef write_file(path, content):\n url = \"http://localhost:9990/computer-use\"\n \n # Encode content to base64\n encoded_content = base64.b64encode(content.encode('utf-8')).decode('utf-8')\n \n data = {\n \"action\": \"write_file\",\n \"path\": path,\n \"data\": encoded_content\n }\n \n response = requests.post(url, json=data)\n return response.json()\n\n# Write a text file\nresult = write_file(\"/home/user/hello.txt\", \"Hello, Bytebot!\")\nprint(result) # {'success': True, 'message': 'File written successfully...'}\n\n# Write to desktop (relative path)\nresult = write_file(\"report.txt\", \"Daily report content\")\nprint(result) # File will be written to /home/user/Desktop/report.txt\n```\n\n```javascript JavaScript\nconst axios = require('axios');\n\nasync function writeFile(path, content) {\n const url = \"http://localhost:9990/computer-use\";\n \n // Encode content to base64\n const encodedContent = Buffer.from(content, 'utf-8').toString('base64');\n \n const data = {\n action: \"write_file\",\n path: path,\n data: encodedContent\n };\n \n const response = await axios.post(url, data);\n return response.data;\n}\n\n// Write a text file\nwriteFile(\"/home/user/notes.txt\", \"Meeting notes...\")\n .then(result => console.log(result))\n .catch(error => console.error(error));\n\n// Write HTML file to desktop\nconst htmlContent = '
\n );\n}\n"
},
{
"path": "packages/bytebot-ui/src/components/messages/content/ImageContent.tsx",
"content": "import React from \"react\";\nimport Image from \"next/image\";\nimport { HugeiconsIcon } from \"@hugeicons/react\";\nimport { Camera01Icon } from \"@hugeicons/core-free-icons\";\nimport { ImageContentBlock } from \"@bytebot/shared\";\n\ninterface ImageContentProps {\n block: ImageContentBlock;\n}\n\nexport function ImageContent({ block }: ImageContentProps) {\n // Use a fixed size for the image since width/height are not available on block.source\n const width = 250;\n const height = 250;\n return (\n
\n
\n \n
\n Screenshot taken\n
\n
\n
\n \n
\n
\n );\n}"
},
{
"path": "packages/bytebot-ui/src/components/messages/content/MessageContent.tsx",
"content": "import React from \"react\";\nimport {\n MessageContentBlock,\n isTextContentBlock,\n isImageContentBlock,\n isComputerToolUseContentBlock,\n isToolResultContentBlock,\n} from \"@bytebot/shared\";\nimport { TextContent } from \"./TextContent\";\nimport { ImageContent } from \"./ImageContent\";\nimport { ComputerToolContent } from \"./ComputerToolContent\";\nimport { ErrorContent } from \"./ErrorContent\";\n\ninterface MessageContentProps {\n content: MessageContentBlock[];\n isTakeOver?: boolean;\n}\n\nexport function MessageContent({\n content,\n isTakeOver = false,\n}: MessageContentProps) {\n // Filter content blocks and check if any visible content remains\n const visibleBlocks = content.filter((block) => {\n // Filter logic from the original code\n if (\n isToolResultContentBlock(block) &&\n block.content &&\n block.content.some((contentBlock) => isImageContentBlock(contentBlock))\n ) {\n return true;\n }\n if (\n isToolResultContentBlock(block) &&\n block.tool_use_id !== \"set_task_status\" &&\n !block.is_error\n ) {\n return false;\n }\n return true;\n });\n\n // Skip rendering if no visible content\n if (visibleBlocks.length === 0) {\n return null;\n }\n\n return (\n
![\"Bytebot](\"docs/images/bytebot-logo.png\"){kind=logo}
\n\n# Bytebot: Open-Source AI Desktop Agent\n\n![\"Bytebot](\"/static/chat-ui-overview.png\")
\n\n## Accessing the UI\n\nWhen running the full Bytebot agent system, the Task UI is available at:\n\n```\nhttp://localhost:9992\n```\n\n## UI Components\n\n### Task Management Panel\n\nThe task management panel allows you to:\n\n- Create new tasks\n- View existing tasks\n- See task status and priority\n- Select a task to work on\n\n![\"Task](\"/static/ui-task-management.png\")
\n\n### Task Interface\n\nThe main task interface provides:\n\n- Task history with the agent\n- Support for markdown formatting in messages\n- Automatic scrolling to new messages\n\n### Desktop Viewer\n\nThe embedded noVNC viewer displays:\n\n- Real-time view of the desktop environment\n- Visual feedback of agent actions\n- Option to expand to take over the desktop\n- Connection status indicator\n\n## Features\n\n### Task Creation\n\nTo create a new task:\n\n1. Enter a description for the task\n2. Click \"Start Task\" button (or press Enter)\n\n### Conversation Controls\n\nThe task interface supports:\n\n- Text messages with markdown formatting\n- Viewing image content in messages\n- Displaying tool use actions\n- Showing tool results\n\n### Desktop Interaction\n\nWhile primarily for viewing, the desktop panel allows:\n\n- Taking over the desktop\n- Real-time monitoring of agent actions\n\n## Message Types\n\nThe task interface displays different types of messages based on Bytebot's content block structure:\n\n- **User Messages**: Your instructions and queries\n- **Assistant Messages**: Responses from the agent, which may include:\n - **Text Content Blocks**: Markdown-formatted text responses\n - **Image Content Blocks**: Images generated or captured\n - **Tool Use Content Blocks**: Computer actions being performed\n - **Tool Result Content Blocks**: Results of computer actions\n\nThe message content structure follows this format:\n\n```typescript\ninterface Message {\n id: string;\n content: MessageContentBlock[];\n role: Role; // \"USER\" or \"ASSISTANT\"\n createdAt?: string;\n}\n\ninterface MessageContentBlock {\n type: string;\n [key: string]: any;\n}\n\ninterface TextContentBlock extends MessageContentBlock {\n type: \"text\";\n text: string;\n}\n\ninterface ImageContentBlock extends MessageContentBlock {\n type: \"image\";\n source: {\n type: \"base64\";\n media_type: string;\n data: string;\n };\n}\n```\n\n## Technical Details\n\nThe Bytebot Task UI is built with:\n\n- **Next.js**: React framework for the frontend\n- **Tailwind CSS**: For styling\n- **ReactMarkdown**: For rendering markdown content\n- **noVNC**: For the embedded desktop viewer\n\n## Troubleshooting\n\n### Connection Issues\n\nIf you experience connection issues:\n\n1. Ensure all Bytebot services are running\n2. Check that ports 9990, 9991, and 9992 are accessible\n3. Try refreshing the browser\n4. Check browser console for error messages\n\n### Desktop Viewer Issues\n\nIf the desktop viewer is not displaying:\n\n1. Ensure the Bytebot container is running\n2. Check that the noVNC service is accessible at port 9990\n\n### Message Display Issues\n\nIf messages are not displaying correctly:\n\n1. Check that the message content is properly formatted\n2. Ensure the agent service is processing task correctly\n3. Check the browser console for any rendering errors\n4. Try refreshing the browser\n"
},
{
"path": "docs/api-reference/computer-use/examples.mdx",
"content": "---\ntitle: \"Computer Use API Examples\"\ndescription: \"Code examples for common automation scenarios using the Bytebot API\"\n---\n\n## Basic Examples\n\nHere are some practical examples of how to use the Computer Use API in different programming languages.\n\n### Using cURL\n\n![\"Task](\"/images/task-lifecycle.png\")
\n\nTasks move through these states:\n\n1. **Created** → Task is defined but not started\n2. **Queued** → Waiting for agent availability\n3. **Running** → Actively being worked on\n4. **Needs Help** → Requires human input\n5. **Completed** → Successfully finished\n6. **Failed** → Could not be completed\n\n### Monitoring Progress\n\n#### Real-time Updates\n\nWatch Bytebot work through the task detail viewer:\n\n- **Green dot**: Task is actively running\n- **Status messages**: Current step being executed\n- **Desktop view**: See what Bytebot sees in real-time\n\n#### Chat Messages\n\nBytebot provides updates like:\n```\nAssistant: I'm now searching for project management tools...\nAssistant: Found 15 options, filtering by your criteria...\nAssistant: Creating the comparison table with 5 tools...\n```\n\n### Interacting with Running Tasks\n\n#### Providing Additional Information\n\n```\nUser: \"Also include free tier options in your research\"\nAssistant: \"I'll add a column for free tier availability to the comparison table.\"\n```\n\n#### Clarifying Instructions\n\n```\nAssistant: \"I found multiple forms on this page. Which one should I fill out?\"\nUser: \"Use the 'Contact Sales' form on the right side\"\n```\n\n#### Modifying Tasks\n\n```\nUser: \"Actually, make it top 10 tools instead of top 5\"\nAssistant: \"I'll expand my research to include 10 tools in the comparison.\"\n```\n\n## Advanced Task Management\n\n### Task Dependencies\n\nChain tasks that depend on each other:\n\n```\nTask 1: \"Download the invoice from the vendor portal\"\nTask 2: \"Open the downloaded invoice and extract the total amount\"\nTask 3: \"Enter the amount into our accounting system\"\n```\n\n## Best Practices\n\n### Do's ✅\n\n1. **Start Simple**: Test with basic tasks before complex ones\n2. **Provide Examples**: \"Format it like the report from last week\"\n3. **Include Credentials Safely**: Use takeover mode for passwords\n4. **Set Realistic Expectations**: Complex tasks take time\n5. **Review Results**: Always verify important outputs\n\n### Don'ts ❌\n\n1. **Overload Single Tasks**: Break complex workflows into steps\n2. **Assume Knowledge**: Explain custom applications\n3. **Skip Context**: Always provide necessary background\n4. **Ignore Errors**: Address issues promptly\n5. **Rush Critical Tasks**: Allow time for careful execution\n\n## Task Examples by Category\n\n### 📄 Document Processing & Analysis\n```\n\"Read the uploaded contract.pdf and extract all key terms including payment schedules, deliverables, and termination clauses. Create a summary document with these details.\"\n\n\"Process all the uploaded invoice PDFs, extract vendor names, amounts, and due dates, then create a consolidated Excel spreadsheet sorted by due date.\"\n\n\"Analyze the uploaded financial_report.pdf and answer these questions: What was the revenue growth? What are the main risk factors mentioned? What is the debt-to-equity ratio?\"\n\n\"Read through the uploaded employee_handbook.pdf and create a checklist of all compliance requirements mentioned in the document.\"\n```\n\n### 🏦 Enterprise Automation (RPA-Style Workflows)\n```\n\"Log into our banking portal, download all transaction files from last month, save them to the Finance/Statements folder, then run the reconciliation script on each file.\"\n\n(Note: Bytebot handles all authentication including 2FA automatically using the built-in password manager)\n\n\"Access the vendor portal at supplier.example.com, navigate to the invoice section, download all pending invoices, extract the data into our standard template, and upload to the AP system.\"\n\n\"Open our legacy ERP system, export the customer list, then for each customer, look them up in the new CRM and update their status and last contact date.\"\n```\n\n### 📊 Financial Operations & Data Analysis\n```\n\"Read the uploaded bank_statements folder containing 12 monthly PDFs, extract all transactions over $10,000, and create a summary report showing patterns and anomalies.\"\n\n\"Log into each of our 5 bank accounts, download the daily statements, consolidate them into a single cash position report, and save to the shared finance folder.\"\n\n\"Process the uploaded expense_reports.zip file, review all reports over $1,000, create a summary with policy violations flagged, and prepare for approval.\"\n\n\"Navigate to the tax authority website, download all GST/VAT returns for Q4, extract the figures, and populate our tax reconciliation spreadsheet.\"\n```\n\n### 🔄 Multi-System Integration\n```\n\"Pull today's orders from Shopify, create corresponding entries in NetSuite, update inventory in our WMS, and trigger shipping labels in ShipStation.\"\n\n\"Extract employee data from Workday, cross-reference with our access control system, identify discrepancies, and create tickets for IT to resolve.\"\n\n\"Log into our insurance portal, download policy documents for all active policies, extract key dates and coverage amounts, update our risk management database.\"\n```\n\n### 📈 Compliance & Reporting\n```\n\"Access all state regulatory websites for our operating regions, check for new compliance updates since last month, download relevant documents, and create a summary report.\"\n\n\"Log into our various SaaS tools (list provided), export user access reports, consolidate into a single audit trail, and flag any terminated employees still with access.\"\n\n\"Navigate to customer portal, download all SLA performance reports, extract metrics, compare against our internal data, and highlight discrepancies.\"\n```\n\n### 🤝 Development & QA Integration\n```\n\"After the code agent deploys the new feature, test the complete user journey from signup to checkout, take screenshots at each step, and verify against the design specs.\"\n\n\"Run through all test scenarios in our QA checklist, but for any failures, have the code agent analyze the error and attempt a fix, then retest automatically.\"\n\n\"Monitor our staging environment, when a new build is deployed, automatically run the smoke test suite and create a visual regression report comparing to production.\"\n```\n\n## Troubleshooting Common Issues\n\n![\"Bytebot](\"/images/agent-architecture.png\")
\n\n