{/* Main container */}
{/* No action buttons for desktop page */}
# Bytebot: Open-Source AI Desktop Agent
# Bytebot: Open-Source AI Desktop Agent
## Accessing the UI
When running the full Bytebot agent system, the Task UI is available at:
```
http://localhost:9992
```
## UI Components
### Task Management Panel
The task management panel allows you to:
- Create new tasks
- View existing tasks
- See task status and priority
- Select a task to work on
### Task Interface
The main task interface provides:
- Task history with the agent
- Support for markdown formatting in messages
- Automatic scrolling to new messages
### Desktop Viewer
The embedded noVNC viewer displays:
- Real-time view of the desktop environment
- Visual feedback of agent actions
- Option to expand to take over the desktop
- Connection status indicator
## Features
### Task Creation
To create a new task:
1. Enter a description for the task
2. Click "Start Task" button (or press Enter)
### Conversation Controls
The task interface supports:
- Text messages with markdown formatting
- Viewing image content in messages
- Displaying tool use actions
- Showing tool results
### Desktop Interaction
While primarily for viewing, the desktop panel allows:
- Taking over the desktop
- Real-time monitoring of agent actions
## Message Types
The task interface displays different types of messages based on Bytebot's content block structure:
- **User Messages**: Your instructions and queries
- **Assistant Messages**: Responses from the agent, which may include:
- **Text Content Blocks**: Markdown-formatted text responses
- **Image Content Blocks**: Images generated or captured
- **Tool Use Content Blocks**: Computer actions being performed
- **Tool Result Content Blocks**: Results of computer actions
The message content structure follows this format:
```typescript
interface Message {
id: string;
content: MessageContentBlock[];
role: Role; // "USER" or "ASSISTANT"
createdAt?: string;
}
interface MessageContentBlock {
type: string;
[key: string]: any;
}
interface TextContentBlock extends MessageContentBlock {
type: "text";
text: string;
}
interface ImageContentBlock extends MessageContentBlock {
type: "image";
source: {
type: "base64";
media_type: string;
data: string;
};
}
```
## Technical Details
The Bytebot Task UI is built with:
- **Next.js**: React framework for the frontend
- **Tailwind CSS**: For styling
- **ReactMarkdown**: For rendering markdown content
- **noVNC**: For the embedded desktop viewer
## Troubleshooting
### Connection Issues
If you experience connection issues:
1. Ensure all Bytebot services are running
2. Check that ports 9990, 9991, and 9992 are accessible
3. Try refreshing the browser
4. Check browser console for error messages
### Desktop Viewer Issues
If the desktop viewer is not displaying:
1. Ensure the Bytebot container is running
2. Check that the noVNC service is accessible at port 9990
### Message Display Issues
If messages are not displaying correctly:
1. Check that the message content is properly formatted
2. Ensure the agent service is processing task correctly
3. Check the browser console for any rendering errors
4. Try refreshing the browser
================================================
FILE: docs/api-reference/computer-use/examples.mdx
================================================
---
title: "Computer Use API Examples"
description: "Code examples for common automation scenarios using the Bytebot API"
---
## Basic Examples
Here are some practical examples of how to use the Computer Use API in different programming languages.
### Using cURL
## How the AI Agent Works
### The Brain: Multi-Model AI Integration
At the heart of Bytebot is a flexible AI integration that supports multiple models. Choose the AI that best fits your needs:
**Anthropic Claude** (Default):
- Best for complex reasoning and visual understanding
- Excellent at following detailed instructions
- Superior performance on desktop automation tasks
**OpenAI GPT Models**:
- Fast and reliable for general automation
- Strong code understanding and generation
- Cost-effective for routine tasks
**Google Gemini**:
- Efficient for high-volume tasks
- Good balance of speed and capability
- Excellent multilingual support
The agent with any model:
1. **Understands Context**: Processes your natural language requests with full conversation history
2. **Plans Actions**: Breaks down complex tasks into executable computer actions
3. **Adapts in Real-time**: Adjusts its approach based on what it sees on screen
4. **Learns from Feedback**: Improves task execution through conversation
### Conversation Flow
## System Architecture
The system consists of four main components that work together:
### 1. Bytebot Desktop Container
The foundation of the system - a virtual Linux desktop that provides:
- **Ubuntu 22.04 LTS** base for stability and compatibility
- **XFCE4 Desktop** for a lightweight, responsive UI
- **bytebotd Daemon** - The automation service built on nutjs that executes computer actions
- **Pre-installed Applications**: Firefox ESR, Thunderbird, text editors, and development tools
- **noVNC** for remote desktop access
**Key Features:**
- Runs completely isolated from your host system
- Consistent environment across different platforms
- Can be customized with additional software
- Accessible via REST API on port 9990
- MCP SSE endpoint available at `/mcp`
- Uses shared types from `@bytebot/shared` package
### 2. AI Agent Service
The brain of the system - orchestrates tasks using an LLM:
- **NestJS Framework** for robust, scalable backend
- **LLM Integration** supporting Anthropic Claude, OpenAI GPT, and Google Gemini models
- **WebSocket Support** for real-time updates
- **Computer Use API Client** to control the desktop
- **Prisma ORM** for database operations
- **Tool definitions** for computer actions (mouse, keyboard, screenshots)
**Responsibilities:**
- Interprets natural language requests
- Plans sequences of computer actions
- Manages task state and progress
- Handles errors and retries
- Provides real-time task updates via WebSocket
### 3. Web Task Interface
The user interface for interacting with your AI agent:
- **Next.js 15 Application** with TypeScript for type safety
- **Embedded VNC Viewer** to watch the desktop in action
- **Task Management** UI with status badges
- **WebSocket Connections** for live updates
- **Reusable components** for consistent UI
- **API utilities** for streamlined server communication
**Features:**
- Task creation and management interface
- Desktop tab for direct manual control
- Real-time desktop viewer with takeover mode
- Task history and status tracking
- Responsive design for all devices
### 4. PostgreSQL Database
Persistent storage for the agent system:
- **Tasks Table**: Stores task details, status, and metadata
- **Messages Table**: Stores AI conversation history
- **Prisma ORM** for type-safe database access
## Data Flow
### Task Execution Flow
## Why a Virtual Desktop?
### Complete Isolation
- **No Risk to Host**: All actions happen inside the container
- **Sandboxed Environment**: Desktop can't access your host system
- **Easy Reset**: Destroy and recreate in seconds
- **Clean Workspace**: Each restart provides a fresh environment
### Consistency Everywhere
- **Platform Independent**: Same environment on Mac, Windows, or Linux
- **Reproducible**: Identical setup every time
- **Version Control**: Pin specific versions for stability
- **No Dependencies**: Everything included in the container
### Built for Automation
- **Predictable UI**: Consistent element positioning
- **Clean Environment**: No popups or distractions
- **Automation-Ready**: Optimized for programmatic control
- **Fast Startup**: Desktop ready in seconds
## Technical Stack
### Base System
- **Ubuntu 22.04 LTS**: Stable, well-supported Linux distribution
- **XFCE4 Desktop**: Lightweight, responsive desktop environment
- **X11 Display Server**: Standard Linux graphics system
- **supervisord**: Service management
### Pre-installed Software
Tasks move through these states:
1. **Created** → Task is defined but not started
2. **Queued** → Waiting for agent availability
3. **Running** → Actively being worked on
4. **Needs Help** → Requires human input
5. **Completed** → Successfully finished
6. **Failed** → Could not be completed
### Monitoring Progress
#### Real-time Updates
Watch Bytebot work through the task detail viewer:
- **Green dot**: Task is actively running
- **Status messages**: Current step being executed
- **Desktop view**: See what Bytebot sees in real-time
#### Chat Messages
Bytebot provides updates like:
```
Assistant: I'm now searching for project management tools...
Assistant: Found 15 options, filtering by your criteria...
Assistant: Creating the comparison table with 5 tools...
```
### Interacting with Running Tasks
#### Providing Additional Information
```
User: "Also include free tier options in your research"
Assistant: "I'll add a column for free tier availability to the comparison table."
```
#### Clarifying Instructions
```
Assistant: "I found multiple forms on this page. Which one should I fill out?"
User: "Use the 'Contact Sales' form on the right side"
```
#### Modifying Tasks
```
User: "Actually, make it top 10 tools instead of top 5"
Assistant: "I'll expand my research to include 10 tools in the comparison."
```
## Advanced Task Management
### Task Dependencies
Chain tasks that depend on each other:
```
Task 1: "Download the invoice from the vendor portal"
Task 2: "Open the downloaded invoice and extract the total amount"
Task 3: "Enter the amount into our accounting system"
```
## Best Practices
### Do's ✅
1. **Start Simple**: Test with basic tasks before complex ones
2. **Provide Examples**: "Format it like the report from last week"
3. **Include Credentials Safely**: Use takeover mode for passwords
4. **Set Realistic Expectations**: Complex tasks take time
5. **Review Results**: Always verify important outputs
### Don'ts ❌
1. **Overload Single Tasks**: Break complex workflows into steps
2. **Assume Knowledge**: Explain custom applications
3. **Skip Context**: Always provide necessary background
4. **Ignore Errors**: Address issues promptly
5. **Rush Critical Tasks**: Allow time for careful execution
## Task Examples by Category
### 📄 Document Processing & Analysis
```
"Read the uploaded contract.pdf and extract all key terms including payment schedules, deliverables, and termination clauses. Create a summary document with these details."
"Process all the uploaded invoice PDFs, extract vendor names, amounts, and due dates, then create a consolidated Excel spreadsheet sorted by due date."
"Analyze the uploaded financial_report.pdf and answer these questions: What was the revenue growth? What are the main risk factors mentioned? What is the debt-to-equity ratio?"
"Read through the uploaded employee_handbook.pdf and create a checklist of all compliance requirements mentioned in the document."
```
### 🏦 Enterprise Automation (RPA-Style Workflows)
```
"Log into our banking portal, download all transaction files from last month, save them to the Finance/Statements folder, then run the reconciliation script on each file."
(Note: Bytebot handles all authentication including 2FA automatically using the built-in password manager)
"Access the vendor portal at supplier.example.com, navigate to the invoice section, download all pending invoices, extract the data into our standard template, and upload to the AP system."
"Open our legacy ERP system, export the customer list, then for each customer, look them up in the new CRM and update their status and last contact date."
```
### 📊 Financial Operations & Data Analysis
```
"Read the uploaded bank_statements folder containing 12 monthly PDFs, extract all transactions over $10,000, and create a summary report showing patterns and anomalies."
"Log into each of our 5 bank accounts, download the daily statements, consolidate them into a single cash position report, and save to the shared finance folder."
"Process the uploaded expense_reports.zip file, review all reports over $1,000, create a summary with policy violations flagged, and prepare for approval."
"Navigate to the tax authority website, download all GST/VAT returns for Q4, extract the figures, and populate our tax reconciliation spreadsheet."
```
### 🔄 Multi-System Integration
```
"Pull today's orders from Shopify, create corresponding entries in NetSuite, update inventory in our WMS, and trigger shipping labels in ShipStation."
"Extract employee data from Workday, cross-reference with our access control system, identify discrepancies, and create tickets for IT to resolve."
"Log into our insurance portal, download policy documents for all active policies, extract key dates and coverage amounts, update our risk management database."
```
### 📈 Compliance & Reporting
```
"Access all state regulatory websites for our operating regions, check for new compliance updates since last month, download relevant documents, and create a summary report."
"Log into our various SaaS tools (list provided), export user access reports, consolidate into a single audit trail, and flag any terminated employees still with access."
"Navigate to customer portal, download all SLA performance reports, extract metrics, compare against our internal data, and highlight discrepancies."
```
### 🤝 Development & QA Integration
```
"After the code agent deploys the new feature, test the complete user journey from signup to checkout, take screenshots at each step, and verify against the design specs."
"Run through all test scenarios in our QA checklist, but for any failures, have the code agent analyze the error and attempt a fix, then retest automatically."
"Monitor our staging environment, when a new build is deployed, automatically run the smoke test suite and create a visual regression report comparing to production."
```
## Troubleshooting Common Issues
Loading tasks...
Get started by creating a first task
Loading tasks...
You took control
No messages yet...