[ { "path": ".gitignore", "content": "# Python\n__pycache__/\n*.py[cod]\n*$py.class\n*.so\n*.egg\n*.egg-info/\ndist/\nbuild/\neggs/\n*.manifest\n*.spec\npip-log.txt\npip-delete-this-directory.txt\n\n# Virtual environments\nvenv/\nenv/\nENV/\n.venv/\n\n# IDEs\n.idea/\n.vscode/\n*.swp\n*.swo\n*~\n.project\n.pydevproject\n.settings/\n\n# SAM Audio original repo (install from pip instead)\nsam_audio/\neval/\nexamples/\nassets/\n.github/\n.checkpoints/\ncheckpoints/\nsam_audio.egg-info/\n\n# Test files and outputs\n*.mp3\n*.wav\n*.mp4\noutput_*.wav\noutput_*.mp4\ntest.mp3\ntest_audio.wav\noffice.mp4\n\n# Test scripts (keep sam_audio_lite.py for reference)\ntest_small.py\ntest_video.py\n\n# Secrets\n.hf_token\n.env\n*.env\n\n# Jupyter\n.ipynb_checkpoints/\n*.ipynb\n\n# OS\n.DS_Store\nThumbs.db\n\n# Node.js (frontend)\nfrontend/node_modules/\nfrontend/.next/\nfrontend/out/\n\n# Redis/Celery\nredis/\n*.rdb\ncelerybeat-schedule\ncelerybeat.pid\n\n# Logs\n*.log\nlogs/\n\n# Uploads/Outputs (runtime generated)\nbackend/uploads/\nbackend/outputs/\n\n# Original repo files (not needed for our fork)\nCODE_OF_CONDUCT.md\nCONTRIBUTING.md\n.pre-commit-config.yaml\npyproject.toml\n" }, { "path": "LICENSE", "content": "MIT License\n\nCopyright (c) 2024 AudioGhost AI Contributors\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n\n---\n\n## Third-Party Licenses\n\nThis project uses the following third-party components:\n\n### SAM-Audio\nSAM-Audio is developed by Meta AI Research and is subject to Meta's research\nlicense. See https://github.com/facebookresearch/sam-audio for more information.\n" }, { "path": "QUICKSTART.md", "content": "# AudioGhost AI 啟動指南\n\n## 快速啟動\n\n### 1. 啟動 Redis (使用 Docker)\n```powershell\ncd d:\\sam_audio\ndocker-compose up -d\n```\n\n### 2. 建立 Anaconda 環境\n```powershell\n# 建立新環境 (Python 3.11+ 必要)\nconda create -n audioghost python=3.11 -y\n\n# 啟動環境\nconda activate audioghost\n```\n\n### 3. 安裝 PyTorch + xformers (CUDA 12.4)\n```powershell\npip install torch==2.9.0+cu126 torchvision==0.24.0+cu126 torchaudio==2.9.0+cu126 --index-url https://download.pytorch.org/whl/cu126 --extra-index-url https://pypi.org/simple\n```\n\n### 4. 安裝 FFmpeg (TorchCodec 需要)\n```powershell\nconda install -c conda-forge ffmpeg -y\n```\n\n### 5. 安裝 SAM Audio\n\n\n```powershell\ncd d:\\sam_audio\npip install .\n```\n\n### 6. 安裝 Backend 依賴\n```powershell\ncd d:\\sam_audio\\backend\npip install -r requirements.txt\n```\n\n### 7. 啟動 Backend API\n```powershell\ncd d:\\sam_audio\\backend\nuvicorn main:app --reload --port 8000\n```\n\n### 8. 啟動 Celery Worker (新終端機)\n\n```powershell\nconda activate audioghost\ncd d:\\sam_audio\\backend\ncelery -A workers.celery_app worker --loglevel=info --pool=solo\n```\n\n### 9. 啟動 Frontend (新終端機)\n```powershell\ncd d:\\sam_audio\\frontend\nnpm run dev\n```\n\n### 10. 開啟瀏覽器\n訪問 http://localhost:3000\n\n\n\n\n## 首次使用\n\n1. 點擊右上角 \"Connect HuggingFace\" 按鈕\n2. 前往 https://huggingface.co/facebook/sam-audio-large 申請存取權限\n3. 建立 Access Token: https://huggingface.co/settings/tokens\n4. 將 Token 貼入並連接\n\n## 功能使用\n\n- **上傳音訊**：拖放或點擊上傳區域\n- **語意分離**：選擇快捷按鈕或輸入自訂描述\n- **時間鎖定**：在波形圖上選取區域\n- **三軌輸出**：Original / Ghost / Clean\n" }, { "path": "README.md", "content": "# AudioGhost AI 🎵👻\n\n![AudioGhost Banner](banner.png)\n\n**AI-Powered Object-Oriented Audio Separation**\n\nDescribe the sound you want to extract or remove using natural language. Powered by Meta's [SAM-Audio](https://github.com/facebookresearch/sam-audio) model.\n\n![Demo](https://img.shields.io/badge/status-MVP%20v1.0-green) ![Python](https://img.shields.io/badge/python-3.11+-blue) ![License](https://img.shields.io/badge/license-MIT-lightgrey)\n\n## 🎬 Demo\n\n### Audio Separation\nhttps://github.com/user-attachments/assets/49248e25-0c56-46ab-a821-2de7f7016bb6\n\n### Video Upload\nhttps://github.com/user-attachments/assets/6b8c08a8-c84f-4fc3-83ad-5703f474fc1b\n\n## Features\n\n- 🎯 **Text-Guided Separation** - Describe what you want to extract: \"vocals\", \"drums\", \"a dog barking\"\n- 🎬 **Video Upload Support** - Upload videos and extract/remove audio sources (audio extraction only, not vision-based)\n- 🚀 **Memory Optimized** - Lite mode reduces VRAM from ~11GB to ~4GB\n- 🎨 **Modern UI** - Glassmorphism design with waveform visualization\n- ⚡ **Real-time Progress** - Track separation progress in real-time\n- 🎛️ **Stem Mixer** - Preview and compare original, extracted, and residual audio\n\n## 🗺️ Roadmap\n\n- 🖱️ **Visual Prompting** - Click on video to select sound sources visually (Integration with [SAM 2](https://github.com/facebookresearch/sam2))\n\n## Architecture\n\n```\n┌─────────────────────────────────────────────────┐\n│ Frontend │\n│ (Next.js + Tailwind v4) │\n└──────────────────────┬──────────────────────────┘\n │\n┌──────────────────────▼──────────────────────────┐\n│ Backend API │\n│ (FastAPI + Python) │\n└──────────────────────┬──────────────────────────┘\n │\n┌──────────────────────▼──────────────────────────┐\n│ Task Queue │\n│ (Celery + Redis) │\n└──────────────────────┬──────────────────────────┘\n │\n┌──────────────────────▼──────────────────────────┐\n│ SAM Audio Lite │\n│ (Memory-optimized Meta SAM-Audio) │\n└─────────────────────────────────────────────────┘\n```\n\n## Requirements\n\n- **Python 3.11+**\n- **CUDA-compatible GPU** (4GB+ VRAM for lite mode, 12GB+ for full mode)\n- **CUDA 12.6** (recommended)\n- **Node.js 18+** (for frontend)\n\n> 💡 FFmpeg and Redis are automatically installed by the installer.\n\n## 🚀 One-Click Installation (Recommended)\n\n### First Time Setup\n```bash\n# Run installer (creates Conda env, downloads Redis, installs all dependencies)\ninstall.bat\n```\n\n### Daily Usage\n```bash\n# Start all services with one click\nstart.bat\n\n# Stop all services\nstop.bat\n```\n\n---\n\n## Manual Setup (Advanced)\n\n### 1. Start Redis\n\nRedis is automatically downloaded to `redis/` folder by `install.bat`. If you prefer Docker:\n```bash\ndocker-compose up -d\n```\n\n### 2. Create Anaconda Environment\n\n```bash\n# Create new environment (Python 3.11+ required)\nconda create -n audioghost python=3.11 -y\n\n# Activate environment\nconda activate audioghost\n```\n\n### 3. Install PyTorch (CUDA 12.6)\n\n```bash\npip install torch==2.9.0+cu126 torchvision==0.24.0+cu126 torchaudio==2.9.0+cu126 --index-url https://download.pytorch.org/whl/cu126 --extra-index-url https://pypi.org/simple\n```\n\n### 4. Install FFmpeg (required by TorchCodec)\n\n```bash\nconda install -c conda-forge ffmpeg -y\n```\n\n### 5. Install SAM Audio\n\n```bash\npip install git+https://github.com/facebookresearch/sam-audio.git\n```\n\n### 6. Install Backend Dependencies\n\n```bash\ncd backend\npip install -r requirements.txt\n```\n\n### 7. Install Frontend Dependencies\n\n```bash\ncd frontend\nnpm install\n```\n\n### 8. Start Services\n\n**Terminal 1 - Backend API:**\n```bash\ncd backend\nuvicorn main:app --reload --port 8000\n```\n\n**Terminal 2 - Celery Worker:**\n```bash\nconda activate audioghost\ncd backend\ncelery -A workers.celery_app worker --loglevel=info --pool=solo\n```\n\n**Terminal 3 - Frontend:**\n```bash\ncd frontend\nnpm run dev\n```\n\n### 9. Open the App\n\nNavigate to `http://localhost:3000`\n\n### 10. Connect HuggingFace\n\n1. Click \"Connect HuggingFace\" button\n2. Request access at https://huggingface.co/facebook/sam-audio-large\n3. Create Access Token: https://huggingface.co/settings/tokens\n4. Paste the token and connect\n\n\n\n## Usage\n\n1. **Upload** an audio file (MP3, WAV, FLAC)\n2. **Describe** what you want to extract or remove:\n - \"vocals\" / \"singing voice\"\n - \"drums\" / \"percussion\"\n - \"background music\"\n - \"a dog barking\"\n - \"crowd noise\"\n3. Click **Extract** or **Remove**\n4. Wait for processing\n5. **Preview** and **download** the results\n\n## Performance Benchmarks\n\n> Tested on RTX 4090 with 4:26 audio (11 chunks @ 25s each)\n\n### VRAM Usage (Lite Mode)\n\n| Model | bfloat16 (Default) | float32 (High Quality) | Recommended GPU |\n|-------|-------------------|------------------------|-----------------|\n| Small | **~6 GB** | **~10 GB** | RTX 3060 6GB / RTX 3070 8GB |\n| Base | **~7 GB** | **~13 GB** | RTX 3070/4060 8GB / RTX 4070 12GB |\n| Large | **~10 GB** | **~20 GB** | RTX 3080/4070 12GB / RTX 4080 16GB |\n\n> 💡 **High Quality Mode (float32)**: Better separation quality but uses +2-3GB more VRAM. Enable via the \"High Quality Mode\" toggle in the UI.\n\n### Processing Time\n\n| Model | First Run (incl. model load) | Subsequent Runs | Speed |\n|-------|------------------------------|-----------------|-------|\n| Small | ~78s | **~25s** | ~10x realtime |\n| Base | ~100s | **~29s** | ~9x realtime |\n| Large | ~130s | **~41s** | ~6.5x realtime |\n\n> 💡 First run includes model download and loading. Subsequent runs use cached models.\n\n### Memory Optimization Details\n\nAudioGhost uses a \"Lite Mode\" that removes unused model components:\n\n| Component Removed | VRAM Saved |\n|-------------------|------------|\n| Vision Encoder | ~2GB |\n| Visual Ranker | ~2GB |\n| Text Ranker | ~2GB |\n| Span Predictor | ~1-2GB |\n\n**Total Reduction**: Up to **40% less VRAM** compared to original SAM-Audio\n\nThis is achieved by:\n- Disabling video-related features (not needed for audio-only)\n- Using `predict_spans=False` and `reranking_candidates=1`\n- Using `bfloat16` precision by default (optional float32 for quality)\n- 25-second chunking for long audio files\n\n## Project Structure\n\n```\naudioghost-ai/\n├── backend/\n│ ├── main.py # FastAPI app\n│ ├── api/ # API routes\n│ │ ├── auth.py # HuggingFace auth\n│ │ └── separate.py # Separation endpoints\n│ └── workers/\n│ ├── celery_app.py # Celery config\n│ └── tasks.py # SAM Audio Lite worker\n├── frontend/\n│ ├── src/\n│ │ ├── app/ # Next.js app\n│ │ └── components/ # React components\n│ └── package.json\n├── sam_audio_lite.py # Standalone lite version\n├── QUICKSTART.md # Quick setup guide\n└── README.md\n```\n\n## API Reference\n\n### POST /api/separate/\n\nCreate a separation task.\n\n**Form Data:**\n- `file` - Audio file\n- `description` - Text prompt (e.g., \"vocals\")\n- `mode` - \"extract\" or \"remove\"\n- `model_size` - \"small\", \"base\", or \"large\" (default: \"base\")\n\n**Response:**\n```json\n{\n \"task_id\": \"uuid\",\n \"status\": \"pending\",\n \"message\": \"Task submitted successfully\"\n}\n```\n\n### GET /api/separate/{task_id}/status\n\nGet task status and progress.\n\n### GET /api/separate/{task_id}/download/{stem}\n\nDownload result audio (ghost, clean, or original).\n\n## Troubleshooting\n\n### CUDA Out of Memory\n- Use `model_size: \"small\"` instead of \"base\" or \"large\"\n- Ensure lite mode is enabled (check for \"Optimizing model for low VRAM\" in logs)\n- Close other GPU applications\n\n### TorchCodec DLL Error\n- Downgrade to FFmpeg 7.x\n- Ensure FFmpeg `bin` directory is in PATH\n\n### HuggingFace 401 Error\n- Re-authenticate via the UI\n- Check that `.hf_token` exists in `backend/`\n\n## License\n\nThis project is licensed under the MIT License. SAM-Audio is licensed by Meta under a research license.\n\n## Credits\n\n- [SAM-Audio](https://github.com/facebookresearch/sam-audio) by Meta AI Research\n- **Core Optimization Logic**: Special thanks to [NilanEkanayake](https://github.com/NilanEkanayake) for providing the initial code modifications in [Issue #24](https://github.com/facebookresearch/sam-audio/issues/24) that made VRAM inference reduction possible.\n- Built with ❤️ using Next.js, FastAPI, and Celery\n" }, { "path": "backend/api/__init__.py", "content": "\"\"\"API Package\"\"\"\n" }, { "path": "backend/api/auth.py", "content": "\"\"\"\nAuthentication API - HuggingFace Token Management\n\"\"\"\nimport os\nfrom pathlib import Path\nfrom typing import Optional\nfrom fastapi import APIRouter, HTTPException\nfrom pydantic import BaseModel\nfrom huggingface_hub import HfApi, hf_hub_download\nfrom huggingface_hub.utils import HfHubHTTPError\n\nrouter = APIRouter()\n\n# Token storage path (use absolute path based on this file's location)\nBACKEND_DIR = Path(__file__).parent.parent\nTOKEN_FILE = BACKEND_DIR / \".hf_token\"\nCHECKPOINTS_DIR = BACKEND_DIR / \"checkpoints\"\n\n\nclass TokenRequest(BaseModel):\n token: str\n\nclass AuthStatus(BaseModel):\n authenticated: bool\n model_downloaded: bool\n model_name: Optional[str] = None\n\n\ndef get_saved_token() -> Optional[str]:\n \"\"\"Get saved HuggingFace token\"\"\"\n if TOKEN_FILE.exists():\n return TOKEN_FILE.read_text().strip()\n return os.environ.get(\"HF_TOKEN\")\n\n\ndef save_token(token: str):\n \"\"\"Save HuggingFace token\"\"\"\n TOKEN_FILE.write_text(token)\n\n\ndef check_model_downloaded() -> bool:\n \"\"\"Check if SAM Audio model is downloaded\"\"\"\n # Check for common model files\n model_files = list(CHECKPOINTS_DIR.glob(\"*.safetensors\")) + \\\n list(CHECKPOINTS_DIR.glob(\"*.bin\"))\n return len(model_files) > 0\n\n\n@router.get(\"/status\", response_model=AuthStatus)\nasync def get_auth_status():\n \"\"\"Check authentication and model status\"\"\"\n token = get_saved_token()\n authenticated = False\n \n if token:\n try:\n api = HfApi(token=token)\n api.whoami()\n authenticated = True\n except Exception:\n authenticated = False\n \n return AuthStatus(\n authenticated=authenticated,\n model_downloaded=check_model_downloaded(),\n model_name=\"facebook/sam-audio-large\" if check_model_downloaded() else None\n )\n\n\n@router.post(\"/login\")\nasync def login(request: TokenRequest):\n \"\"\"Validate and save HuggingFace token\"\"\"\n try:\n # Validate token\n api = HfApi(token=request.token)\n user_info = api.whoami()\n \n # Check if user has access to SAM Audio\n try:\n api.model_info(\"facebook/sam-audio-large\", token=request.token)\n except HfHubHTTPError as e:\n if \"403\" in str(e) or \"401\" in str(e):\n raise HTTPException(\n status_code=403,\n detail=\"You need to request access to facebook/sam-audio-large on HuggingFace first\"\n )\n raise\n \n # Save token\n save_token(request.token)\n \n return {\n \"success\": True,\n \"username\": user_info.get(\"name\", \"Unknown\"),\n \"message\": \"Successfully authenticated\"\n }\n \n except HTTPException:\n raise\n except Exception as e:\n raise HTTPException(status_code=401, detail=f\"Invalid token: {str(e)}\")\n\n\n@router.post(\"/download-model\")\nasync def download_model():\n \"\"\"Download SAM Audio model\"\"\"\n token = get_saved_token()\n \n if not token:\n raise HTTPException(status_code=401, detail=\"Not authenticated\")\n \n try:\n # Note: In production, this should be a background task\n # For MVP, we'll use the HuggingFace auto-download feature\n # which downloads on first use\n \n return {\n \"success\": True,\n \"message\": \"Model will be downloaded automatically on first use\"\n }\n \n except Exception as e:\n raise HTTPException(status_code=500, detail=f\"Download failed: {str(e)}\")\n\n\n@router.post(\"/logout\")\nasync def logout():\n \"\"\"Clear saved token\"\"\"\n if TOKEN_FILE.exists():\n TOKEN_FILE.unlink()\n \n return {\"success\": True, \"message\": \"Logged out\"}\n" }, { "path": "backend/api/separate.py", "content": "\"\"\"\nSeparation API - Audio/Video Separation Endpoints\n\"\"\"\nimport uuid\nfrom pathlib import Path\nfrom typing import Optional, List\nfrom fastapi import APIRouter, UploadFile, File, Form, HTTPException\nfrom pydantic import BaseModel\n\nfrom workers.celery_app import celery_app\nfrom workers.tasks import separate_audio_task\n\nrouter = APIRouter()\n\nUPLOAD_DIR = Path(\"uploads\")\n\n# Supported MIME types\nAUDIO_TYPES = [\"audio/mpeg\", \"audio/wav\", \"audio/mp3\", \"audio/x-wav\", \"audio/flac\", \"audio/m4a\", \"audio/aac\"]\nVIDEO_TYPES = [\"video/mp4\", \"video/webm\", \"video/quicktime\", \"video/x-msvideo\", \"video/mpeg\", \"video/x-matroska\"]\nVIDEO_EXTENSIONS = [\".mp4\", \".webm\", \".mov\", \".avi\", \".mkv\", \".mpeg\"]\n\n\nclass SeparationRequest(BaseModel):\n description: str\n mode: str = \"extract\" # \"extract\" or \"remove\"\n start_time: Optional[float] = None\n end_time: Optional[float] = None\n model_size: str = \"base\" # \"small\", \"base\", \"large\"\n\n\nclass SeparationResponse(BaseModel):\n task_id: str\n status: str\n message: str\n\n\n@router.post(\"/\", response_model=SeparationResponse)\nasync def create_separation_task(\n file: UploadFile = File(...),\n description: str = Form(...),\n mode: str = Form(\"extract\"),\n start_time: Optional[float] = Form(None),\n end_time: Optional[float] = Form(None),\n model_size: str = Form(\"base\"),\n chunk_duration: float = Form(25.0),\n use_float32: str = Form(\"false\")\n):\n \"\"\"\n Create a new audio/video separation task\n \n - **file**: Audio or video file to process (video audio will be extracted)\n - **description**: Text prompt describing the sound to separate\n - **mode**: \"extract\" to isolate the sound, \"remove\" to remove it\n - **start_time**: Optional start time for temporal prompting\n - **end_time**: Optional end time for temporal prompting\n - **model_size**: SAM Audio model size (small/base/large)\n - **chunk_duration**: Audio chunk duration in seconds (5-60, default 25)\n - **use_float32**: Use float32 precision for better quality (default: false)\n \"\"\"\n \n # Validate chunk_duration\n chunk_duration = max(5.0, min(60.0, chunk_duration))\n \n # Parse use_float32 from string to bool\n use_float32_bool = use_float32.lower() == \"true\"\n \n # Detect if file is video\n file_extension = Path(file.filename).suffix.lower() if file.filename else \"\"\n is_video = (\n (file.content_type and file.content_type in VIDEO_TYPES) or\n file_extension in VIDEO_EXTENSIONS\n )\n \n # Generate task ID\n task_id = str(uuid.uuid4())\n \n # Save uploaded file\n file_extension = Path(file.filename).suffix or \".mp3\"\n upload_path = UPLOAD_DIR / f\"{task_id}{file_extension}\"\n \n with open(upload_path, \"wb\") as f:\n content = await file.read()\n f.write(content)\n \n # Build anchors for temporal prompting\n anchors = None\n if start_time is not None and end_time is not None:\n anchors = [[[\"+\", start_time, end_time]]]\n \n # Submit Celery task\n celery_task = separate_audio_task.apply_async(\n args=[\n str(upload_path),\n description,\n mode,\n anchors,\n model_size,\n chunk_duration,\n use_float32_bool,\n is_video # New: flag for video processing\n ],\n task_id=task_id\n )\n \n return SeparationResponse(\n task_id=task_id,\n status=\"pending\",\n message=\"Task submitted successfully\"\n )\n\n\n@router.post(\"/batch\", response_model=List[SeparationResponse])\nasync def create_batch_separation(\n file: UploadFile = File(...),\n descriptions: str = Form(...), # JSON array of descriptions\n mode: str = Form(\"extract\")\n):\n \"\"\"\n Create multiple separation tasks for the same audio file\n Useful for separating multiple stems at once\n \"\"\"\n import json\n \n try:\n desc_list = json.loads(descriptions)\n except json.JSONDecodeError:\n raise HTTPException(status_code=400, detail=\"Invalid descriptions format\")\n \n # Save file once\n base_task_id = str(uuid.uuid4())\n file_extension = Path(file.filename).suffix or \".mp3\"\n upload_path = UPLOAD_DIR / f\"{base_task_id}{file_extension}\"\n \n with open(upload_path, \"wb\") as f:\n content = await file.read()\n f.write(content)\n \n responses = []\n for i, desc in enumerate(desc_list):\n task_id = f\"{base_task_id}-{i}\"\n \n separate_audio_task.apply_async(\n args=[str(upload_path), desc, mode, None, \"small\"],\n task_id=task_id\n )\n \n responses.append(SeparationResponse(\n task_id=task_id,\n status=\"pending\",\n message=f\"Task for '{desc}' submitted\"\n ))\n \n return responses\n" }, { "path": "backend/api/tasks.py", "content": "\"\"\"\nTasks API - Task Status and Results\n\"\"\"\nfrom pathlib import Path\nfrom typing import Optional, List\nfrom fastapi import APIRouter, HTTPException\nfrom fastapi.responses import FileResponse\nfrom pydantic import BaseModel\nfrom celery.result import AsyncResult\n\nfrom workers.celery_app import celery_app\n\nrouter = APIRouter()\n\nOUTPUT_DIR = Path(\"outputs\")\n\n\nclass TaskStatus(BaseModel):\n task_id: str\n status: str # pending, processing, completed, failed\n progress: int # 0-100\n message: Optional[str] = None\n result: Optional[dict] = None\n\n\nclass TaskResult(BaseModel):\n original_url: str\n ghost_url: str # Separated target\n clean_url: str # Residual\n\n\n@router.get(\"/{task_id}\", response_model=TaskStatus)\nasync def get_task_status(task_id: str):\n \"\"\"Get the status of a separation task\"\"\"\n \n result = AsyncResult(task_id, app=celery_app)\n \n if result.state == \"PENDING\":\n return TaskStatus(\n task_id=task_id,\n status=\"pending\",\n progress=0,\n message=\"Task is waiting to be processed\"\n )\n \n elif result.state == \"PROGRESS\":\n info = result.info or {}\n return TaskStatus(\n task_id=task_id,\n status=\"processing\",\n progress=info.get(\"progress\", 0),\n message=info.get(\"message\", \"Processing...\")\n )\n \n elif result.state == \"SUCCESS\":\n return TaskStatus(\n task_id=task_id,\n status=\"completed\",\n progress=100,\n message=\"Task completed successfully\",\n result=result.result\n )\n \n elif result.state == \"FAILURE\":\n return TaskStatus(\n task_id=task_id,\n status=\"failed\",\n progress=0,\n message=str(result.info)\n )\n \n else:\n return TaskStatus(\n task_id=task_id,\n status=result.state.lower(),\n progress=0,\n message=f\"Task state: {result.state}\"\n )\n\n\n@router.get(\"/{task_id}/download/{file_type}\")\nasync def download_result(task_id: str, file_type: str):\n \"\"\"\n Download processed audio or video file\n \n - **file_type**: \"original\", \"ghost\", \"clean\", or \"video\"\n \"\"\"\n \n if file_type not in [\"original\", \"ghost\", \"clean\", \"video\"]:\n raise HTTPException(status_code=400, detail=\"Invalid file type\")\n \n result = AsyncResult(task_id, app=celery_app)\n \n if result.state != \"SUCCESS\":\n raise HTTPException(status_code=404, detail=\"Task not completed\")\n \n # Handle video file separately\n if file_type == \"video\":\n video_path = result.result.get(\"video_path\")\n if not video_path:\n raise HTTPException(status_code=404, detail=\"No video file for this task\")\n \n file_path = Path(video_path)\n if not file_path.exists():\n raise HTTPException(status_code=404, detail=\"Video file not found\")\n \n # Determine media type based on extension\n extension = file_path.suffix.lower()\n media_types = {\n \".mp4\": \"video/mp4\",\n \".webm\": \"video/webm\",\n \".mov\": \"video/quicktime\",\n \".avi\": \"video/x-msvideo\",\n \".mkv\": \"video/x-matroska\"\n }\n media_type = media_types.get(extension, \"video/mp4\")\n \n return FileResponse(\n path=file_path,\n filename=f\"{task_id}_video{extension}\",\n media_type=media_type\n )\n \n # Handle audio files\n file_path = Path(result.result.get(f\"{file_type}_path\", \"\"))\n \n if not file_path.exists():\n raise HTTPException(status_code=404, detail=\"File not found\")\n \n return FileResponse(\n path=file_path,\n filename=f\"{task_id}_{file_type}.wav\",\n media_type=\"audio/wav\"\n )\n\n\n@router.get(\"/{task_id}/download-video-with-audio/{audio_type}\")\nasync def download_video_with_audio(task_id: str, audio_type: str):\n \"\"\"\n Download video with merged audio track\n \n - **audio_type**: \"original\", \"ghost\", or \"clean\"\n \"\"\"\n import subprocess\n import tempfile\n import os\n \n if audio_type not in [\"original\", \"ghost\", \"clean\"]:\n raise HTTPException(status_code=400, detail=\"Invalid audio type. Use 'original', 'ghost', or 'clean'\")\n \n result = AsyncResult(task_id, app=celery_app)\n \n if result.state != \"SUCCESS\":\n raise HTTPException(status_code=404, detail=\"Task not completed\")\n \n # Get video path\n video_path = result.result.get(\"video_path\")\n if not video_path:\n raise HTTPException(status_code=404, detail=\"No video file for this task\")\n \n video_file = Path(video_path)\n if not video_file.exists():\n raise HTTPException(status_code=404, detail=\"Video file not found\")\n \n # Get audio path\n audio_path = Path(result.result.get(f\"{audio_type}_path\", \"\"))\n if not audio_path.exists():\n raise HTTPException(status_code=404, detail=f\"Audio file '{audio_type}' not found\")\n \n # Create output file in the same directory as video\n output_dir = video_file.parent\n extension = video_file.suffix.lower()\n output_filename = f\"{task_id}_{audio_type}_merged{extension}\"\n output_path = output_dir / output_filename\n \n # Use FFmpeg to merge video and audio\n try:\n cmd = [\n \"ffmpeg\", \"-y\",\n \"-i\", str(video_file),\n \"-i\", str(audio_path),\n \"-c:v\", \"copy\", # Copy video stream without re-encoding\n \"-c:a\", \"aac\", # Encode audio to AAC\n \"-b:a\", \"192k\", # Audio bitrate\n \"-map\", \"0:v:0\", # Use video from first input\n \"-map\", \"1:a:0\", # Use audio from second input\n \"-shortest\", # Match shortest stream\n str(output_path)\n ]\n subprocess.run(cmd, check=True, capture_output=True)\n except subprocess.CalledProcessError as e:\n raise HTTPException(status_code=500, detail=f\"FFmpeg error: {e.stderr.decode()}\")\n except FileNotFoundError:\n raise HTTPException(status_code=500, detail=\"FFmpeg not found. Please install FFmpeg.\")\n \n if not output_path.exists():\n raise HTTPException(status_code=500, detail=\"Failed to create merged video\")\n \n # Determine media type\n media_types = {\n \".mp4\": \"video/mp4\",\n \".webm\": \"video/webm\",\n \".mov\": \"video/quicktime\",\n \".avi\": \"video/x-msvideo\",\n \".mkv\": \"video/x-matroska\"\n }\n media_type = media_types.get(extension, \"video/mp4\")\n \n # Map audio type to display name\n audio_labels = {\n \"original\": \"original\",\n \"ghost\": \"isolated\",\n \"clean\": \"without_isolated\"\n }\n \n return FileResponse(\n path=output_path,\n filename=f\"{task_id}_{audio_labels[audio_type]}_video{extension}\",\n media_type=media_type\n )\n\n\n@router.delete(\"/{task_id}\")\nasync def cancel_task(task_id: str):\n \"\"\"Cancel a pending or running task\"\"\"\n \n result = AsyncResult(task_id, app=celery_app)\n result.revoke(terminate=True)\n \n return {\"success\": True, \"message\": \"Task cancelled\"}\n\n\n@router.get(\"/\", response_model=List[TaskStatus])\nasync def list_recent_tasks(limit: int = 10):\n \"\"\"List recent tasks (simplified - in production would use database)\"\"\"\n # Note: This is a simplified implementation\n # In production, you would store task metadata in a database\n \n return []\n" }, { "path": "backend/main.py", "content": "\"\"\"\nAudioGhost AI - FastAPI Backend\n\"\"\"\nimport os\nfrom pathlib import Path\nfrom fastapi import FastAPI\nfrom fastapi.middleware.cors import CORSMiddleware\nfrom fastapi.staticfiles import StaticFiles\n\nfrom api import auth, separate, tasks\n\n# Create necessary directories\nUPLOAD_DIR = Path(\"uploads\")\nOUTPUT_DIR = Path(\"outputs\")\nCHECKPOINTS_DIR = Path(\"../checkpoints\")\n\nUPLOAD_DIR.mkdir(exist_ok=True)\nOUTPUT_DIR.mkdir(exist_ok=True)\nCHECKPOINTS_DIR.mkdir(exist_ok=True)\n\napp = FastAPI(\n title=\"AudioGhost AI\",\n description=\"AI-Powered Audio Separation Tool\",\n version=\"1.0.0\"\n)\n\n# CORS Configuration\napp.add_middleware(\n CORSMiddleware,\n allow_origins=[\"http://localhost:3000\"],\n allow_credentials=True,\n allow_methods=[\"*\"],\n allow_headers=[\"*\"],\n)\n\n# Mount static files for downloads\napp.mount(\"/outputs\", StaticFiles(directory=\"outputs\"), name=\"outputs\")\n\n# Include routers\napp.include_router(auth.router, prefix=\"/api/auth\", tags=[\"Authentication\"])\napp.include_router(separate.router, prefix=\"/api/separate\", tags=[\"Separation\"])\napp.include_router(tasks.router, prefix=\"/api/tasks\", tags=[\"Tasks\"])\n\n\n@app.get(\"/\")\nasync def root():\n return {\n \"name\": \"AudioGhost AI\",\n \"version\": \"1.0.0\",\n \"status\": \"running\"\n }\n\n\n@app.get(\"/health\")\nasync def health():\n return {\"status\": \"healthy\"}\n" }, { "path": "backend/requirements.txt", "content": "# AudioGhost AI - Backend Dependencies\n\n# FastAPI Framework\nfastapi==0.115.6\nuvicorn[standard]==0.34.0\npython-multipart==0.0.19\n\n# Task Queue\ncelery==5.4.0\nredis==5.2.1\n\n# Media Processing\npydub==0.25.1\n\n# AI Dependencies (SAM Audio already installed from parent)\n# huggingface_hub is included in SAM Audio deps\n\n# Utilities\npython-dotenv==1.0.1\naiofiles==24.1.0\n" }, { "path": "backend/workers/__init__.py", "content": "\"\"\"Workers Package\"\"\"\n" }, { "path": "backend/workers/celery_app.py", "content": "\"\"\"\nCelery Application Configuration\n\"\"\"\nfrom celery import Celery\n\ncelery_app = Celery(\n \"audioghost\",\n broker=\"redis://localhost:6379/0\",\n backend=\"redis://localhost:6379/0\",\n include=[\"workers.tasks\"]\n)\n\n# Celery Configuration\ncelery_app.conf.update(\n task_serializer=\"json\",\n accept_content=[\"json\"],\n result_serializer=\"json\",\n timezone=\"UTC\",\n enable_utc=True,\n task_track_started=True,\n task_time_limit=3600, # 1 hour max per task\n worker_prefetch_multiplier=1, # Process one task at a time (GPU memory)\n result_expires=86400, # Results expire after 24 hours\n)\n" }, { "path": "backend/workers/tasks.py", "content": "\"\"\"\nCelery Tasks - Audio Separation Workers\nWith SAM Audio Lite optimization for low VRAM usage\n\"\"\"\nimport os\nimport sys\nimport gc\nfrom pathlib import Path\nfrom typing import Optional, List\n\nfrom celery import current_task\n\nfrom workers.celery_app import celery_app\n\n# Add parent directory to path for SAM Audio imports\nsys.path.insert(0, str(Path(__file__).parent.parent.parent))\n\nOUTPUT_DIR = Path(\"outputs\")\nOUTPUT_DIR.mkdir(exist_ok=True)\n\n# Global model cache to avoid reloading\n_model_cache = {}\n_processor_cache = {}\n\n\ndef update_progress(progress: int, message: str):\n \"\"\"Update task progress\"\"\"\n current_task.update_state(\n state=\"PROGRESS\",\n meta={\"progress\": progress, \"message\": message}\n )\n\n\ndef create_lite_model(model_name: str, hf_token: str = None):\n \"\"\"\n Create a memory-optimized SAM Audio model by removing unused components.\n \n Reduces VRAM usage from ~11GB to ~4-5GB by:\n - Replacing vision_encoder with a dummy\n - Disabling visual_ranker\n - Disabling text_ranker\n - Disabling span_predictor\n \"\"\"\n import torch\n from sam_audio import SAMAudio, SAMAudioProcessor\n \n print(f\"Loading {model_name} (lite mode)...\")\n \n # Load model\n if hf_token:\n model = SAMAudio.from_pretrained(model_name, token=hf_token)\n else:\n model = SAMAudio.from_pretrained(model_name)\n \n processor = SAMAudioProcessor.from_pretrained(model_name)\n \n print(\"Optimizing model for low VRAM...\")\n \n # Get vision encoder dim before deleting\n vision_dim = model.vision_encoder.dim if hasattr(model.vision_encoder, 'dim') else 1024\n \n # Delete heavy components\n del model.vision_encoder\n gc.collect()\n \n # Store the dim for _get_video_features\n model._vision_encoder_dim = vision_dim\n \n # Replace _get_video_features to not use vision_encoder\n def _get_video_features_lite(self, video, audio_features):\n B, T, _ = audio_features.shape\n return audio_features.new_zeros(B, self._vision_encoder_dim, T)\n \n import types\n model._get_video_features = types.MethodType(_get_video_features_lite, model)\n \n # Delete rankers\n if hasattr(model, 'visual_ranker') and model.visual_ranker is not None:\n del model.visual_ranker\n model.visual_ranker = None\n gc.collect()\n \n if hasattr(model, 'text_ranker') and model.text_ranker is not None:\n del model.text_ranker\n model.text_ranker = None\n gc.collect()\n \n # Delete span predictor\n if hasattr(model, 'span_predictor') and model.span_predictor is not None:\n del model.span_predictor\n model.span_predictor = None\n gc.collect()\n \n if hasattr(model, 'span_predictor_transform') and model.span_predictor_transform is not None:\n del model.span_predictor_transform\n model.span_predictor_transform = None\n gc.collect()\n \n # Force garbage collection\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n \n print(\"Model optimization complete!\")\n \n return model, processor\n\n\ndef get_or_load_lite_model(model_name: str, hf_token: str, device: str, dtype):\n \"\"\"Get cached lite model or create it - only keeps ONE model in memory\"\"\"\n import torch\n \n # Include dtype in cache key to ensure correct model is loaded\n dtype_str = \"bf16\" if dtype == torch.bfloat16 else \"fp32\"\n cache_key = f\"{model_name}_lite_{device}_{dtype_str}\"\n \n print(f\"[DEBUG] Looking for cached model with key: {cache_key}\")\n print(f\"[DEBUG] Current cache keys: {list(_model_cache.keys())}\")\n \n if cache_key not in _model_cache:\n print(f\"[DEBUG] Cache miss - creating new lite model\")\n \n # IMPORTANT: Clear any existing models first to free memory\n if len(_model_cache) > 0:\n print(f\"[DEBUG] Clearing {len(_model_cache)} existing model(s) from cache...\")\n for old_key in list(_model_cache.keys()):\n del _model_cache[old_key]\n for old_key in list(_processor_cache.keys()):\n del _processor_cache[old_key]\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n print(f\"[DEBUG] GPU Memory after clearing old models: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n \n model, processor = create_lite_model(model_name, hf_token)\n \n print(f\"[DEBUG] Converting model to {device} with dtype {dtype}\")\n model = model.eval().to(device, dtype)\n \n _model_cache[cache_key] = model\n _processor_cache[model_name] = processor\n \n if torch.cuda.is_available():\n print(f\"[DEBUG] GPU Memory after loading: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n else:\n print(f\"[DEBUG] Cache hit - using existing model\")\n \n return _model_cache[cache_key], _processor_cache[model_name]\n\n\n\n\ndef cleanup_gpu_memory():\n \"\"\"Clean up GPU memory after task\"\"\"\n import torch\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n gc.collect()\n\n\n@celery_app.task(bind=True)\ndef separate_audio_task(\n self,\n audio_path: str,\n description: str,\n mode: str = \"extract\",\n anchors: Optional[List] = None,\n model_size: str = \"base\",\n chunk_duration: float = 25.0,\n use_float32: bool = False,\n is_video: bool = False\n):\n \"\"\"\n Separate audio using SAM Audio Lite (memory optimized)\n \n Args:\n audio_path: Path to input audio or video file\n description: Text prompt for separation\n mode: \"extract\" or \"remove\"\n anchors: Optional temporal anchors [[\"+\", start, end], ...]\n model_size: Model size (small/base/large)\n chunk_duration: Audio chunk duration in seconds (5-60)\n use_float32: Use float32 precision for better quality\n is_video: If True, extract audio from video file first\n \n Returns:\n Dictionary with paths to output files\n \"\"\"\n import torch\n import torchaudio\n import time\n import subprocess\n import shutil\n from huggingface_hub import login\n \n task_id = self.request.id\n device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n video_path = None # Will be set if input is video\n \n # Debug: Show received parameter\n print(f\"[DEBUG] use_float32 parameter received: {use_float32} (type: {type(use_float32).__name__})\")\n print(f\"[DEBUG] is_video parameter received: {is_video}\")\n \n # Handle video files - extract audio using FFmpeg\n if is_video:\n update_progress(2, \"Extracting audio from video...\")\n video_path = Path(audio_path)\n \n # Copy video to output directory for later playback\n output_video_path = OUTPUT_DIR / f\"{task_id}.video{video_path.suffix}\"\n shutil.copy2(video_path, output_video_path)\n print(f\"[DEBUG] Copied video to: {output_video_path}\")\n \n # Extract audio from video using FFmpeg\n extracted_audio_path = OUTPUT_DIR / f\"{task_id}.extracted.wav\"\n ffmpeg_cmd = [\n \"ffmpeg\", \"-y\",\n \"-i\", str(video_path),\n \"-vn\", # No video\n \"-acodec\", \"pcm_s16le\", # PCM 16-bit\n \"-ar\", \"44100\", # 44.1kHz sample rate\n \"-ac\", \"1\", # Mono\n str(extracted_audio_path)\n ]\n \n try:\n result = subprocess.run(\n ffmpeg_cmd,\n capture_output=True,\n text=True,\n check=True\n )\n print(f\"[DEBUG] FFmpeg audio extraction successful\")\n except subprocess.CalledProcessError as e:\n raise Exception(f\"FFmpeg audio extraction failed: {e.stderr}\")\n \n # Use extracted audio for processing\n audio_path = str(extracted_audio_path)\n \n # Set precision based on use_float32 parameter\n if use_float32 or device == \"cpu\":\n dtype = torch.float32\n print(f\"[DEBUG] Using float32 precision (High Quality Mode)\")\n else:\n dtype = torch.bfloat16\n print(f\"[DEBUG] Using bfloat16 precision (Memory Optimized)\")\n \n # Start timing\n start_time = time.time()\n \n try:\n update_progress(5, \"Initializing...\")\n \n # Load HuggingFace token\n backend_dir = Path(__file__).parent.parent\n token_file = backend_dir / \".hf_token\"\n if token_file.exists():\n with open(token_file, \"r\") as f:\n hf_token = f.read().strip()\n login(token=hf_token)\n else:\n raise Exception(\"HuggingFace token not found. Please authenticate first.\")\n \n # Select model based on size\n model_name = f\"facebook/sam-audio-{model_size}\"\n \n update_progress(10, f\"Loading {model_name} (lite mode)...\")\n \n # Clean up before loading\n cleanup_gpu_memory()\n \n # Load lite model (with caching)\n model, processor = get_or_load_lite_model(model_name, hf_token, device, dtype)\n \n update_progress(30, \"Loading audio...\")\n \n # Get sample rate\n sample_rate = processor.audio_sampling_rate\n \n # Load and preprocess audio\n audio, orig_sr = torchaudio.load(audio_path)\n if orig_sr != sample_rate:\n resampler = torchaudio.transforms.Resample(orig_sr, sample_rate)\n audio = resampler(audio)\n \n # Convert to mono if stereo\n if audio.shape[0] > 1:\n audio = audio.mean(dim=0, keepdim=True)\n \n # Calculate audio duration\n audio_duration = audio.shape[1] / sample_rate\n print(f\"[DEBUG] Audio duration: {audio_duration:.2f}s\")\n \n # Chunking settings (from parameter, clamped to 5-60)\n CHUNK_DURATION = max(5.0, min(60.0, chunk_duration))\n MAX_CHUNK_SAMPLES = int(sample_rate * CHUNK_DURATION)\n \n # Check if chunking is needed\n if audio.shape[1] > MAX_CHUNK_SAMPLES:\n print(f\"[DEBUG] Audio is {audio_duration:.1f}s, using chunking ({CHUNK_DURATION}s chunks)\")\n \n # Split audio into chunks\n audio_tensor = audio.squeeze(0).to(device, dtype)\n chunks = torch.split(audio_tensor, MAX_CHUNK_SAMPLES, dim=-1)\n total_chunks = len(chunks)\n \n out_target = []\n out_residual = []\n \n for i, chunk in enumerate(chunks):\n # Update progress\n chunk_progress = 30 + int((i / total_chunks) * 50)\n update_progress(chunk_progress, f\"Processing chunk {i+1}/{total_chunks}...\")\n \n # Skip very short chunks\n if chunk.shape[-1] < sample_rate: # Less than 1 second\n print(f\"[DEBUG] Skipping chunk {i+1} (too short)\")\n continue\n \n # Prepare batch for this chunk\n batch = processor(\n audios=[chunk.unsqueeze(0)],\n descriptions=[description]\n ).to(device)\n \n # Run separation\n with torch.inference_mode():\n with torch.cuda.amp.autocast(enabled=(device == \"cuda\")):\n result = model.separate(\n batch,\n predict_spans=False,\n reranking_candidates=1\n )\n \n out_target.append(result.target[0].cpu())\n out_residual.append(result.residual[0].cpu())\n \n # Clean up chunk results\n del batch, result\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n \n # Concatenate all chunks\n target_audio = torch.cat(out_target, dim=-1).clamp(-1, 1).float().unsqueeze(0)\n residual_audio = torch.cat(out_residual, dim=-1).clamp(-1, 1).float().unsqueeze(0)\n \n del out_target, out_residual, chunks, audio_tensor\n \n else:\n print(f\"[DEBUG] Audio is {audio_duration:.1f}s, processing as single batch\")\n \n update_progress(50, \"Running separation...\")\n \n # Process entire audio at once\n batch = processor(\n audios=[audio_path],\n descriptions=[description]\n ).to(device)\n \n # Run separation\n with torch.inference_mode():\n with torch.cuda.amp.autocast(enabled=(device == \"cuda\")):\n result = model.separate(\n batch,\n predict_spans=False,\n reranking_candidates=1\n )\n \n target_audio = result.target[0].float().unsqueeze(0).cpu()\n residual_audio = result.residual[0].float().unsqueeze(0).cpu()\n \n del batch, result\n \n update_progress(80, \"Saving results...\")\n \n # Output paths\n output_base = OUTPUT_DIR / task_id\n original_path = output_base.with_suffix(\".original.wav\")\n ghost_path = output_base.with_suffix(\".ghost.wav\")\n clean_path = output_base.with_suffix(\".clean.wav\")\n \n # Save original audio\n torchaudio.save(str(original_path), audio.cpu(), sample_rate)\n \n # Save separated audio\n if mode == \"extract\":\n torchaudio.save(str(ghost_path), target_audio, sample_rate)\n torchaudio.save(str(clean_path), residual_audio, sample_rate)\n else:\n torchaudio.save(str(ghost_path), target_audio, sample_rate)\n torchaudio.save(str(clean_path), residual_audio, sample_rate)\n \n update_progress(100, \"Complete!\")\n \n # Aggressive cleanup\n print(f\"[DEBUG] Cleaning up GPU memory...\")\n del target_audio, residual_audio, audio\n \n gc.collect()\n cleanup_gpu_memory()\n \n if torch.cuda.is_available():\n print(f\"[DEBUG] GPU Memory after cleanup: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n \n # Calculate processing time\n processing_time = time.time() - start_time\n print(f\"[DEBUG] Processing completed in {processing_time:.2f}s for {audio_duration:.2f}s audio\")\n \n result = {\n \"original_path\": str(original_path),\n \"ghost_path\": str(ghost_path),\n \"clean_path\": str(clean_path),\n \"description\": description,\n \"mode\": mode,\n \"audio_duration\": round(audio_duration, 2),\n \"processing_time\": round(processing_time, 2),\n \"model_size\": model_size\n }\n \n # Add video path if this was a video file\n if video_path is not None:\n output_video_path = OUTPUT_DIR / f\"{task_id}.video{video_path.suffix}\"\n result[\"video_path\"] = str(output_video_path)\n result[\"is_video\"] = True\n \n return result\n\n \n except Exception as e:\n gc.collect()\n cleanup_gpu_memory()\n raise Exception(f\"Separation failed: {str(e)}\")\n\n\n\n@celery_app.task(bind=True)\ndef match_pattern_task(\n self,\n audio_path: str,\n sample_path: str,\n threshold: float = 0.85,\n model_size: str = \"base\"\n):\n \"\"\"\n Find and remove sounds similar to a sample\n \n Args:\n audio_path: Path to input audio file\n sample_path: Path to sample audio file\n threshold: Similarity threshold (0-1)\n model_size: Model size (small/base/large)\n \n Returns:\n Dictionary with paths to output files and matched segments\n \"\"\"\n # TODO: Implement pattern matching with CLAP embeddings\n # This is a placeholder for MVP v1.0\n \n update_progress(50, \"Pattern matching not yet implemented in MVP\")\n \n return {\n \"status\": \"not_implemented\",\n \"message\": \"Pattern matching will be available in v1.1\"\n }\n" }, { "path": "docker-compose.yml", "content": "version: '3.8'\n\nservices:\n redis:\n image: redis:alpine\n container_name: audioghost-redis\n ports:\n - \"6379:6379\"\n volumes:\n - redis_data:/data\n restart: unless-stopped\n\nvolumes:\n redis_data:\n" }, { "path": "frontend/.gitignore", "content": "# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.\n\n# dependencies\n/node_modules\n/.pnp\n.pnp.*\n.yarn/*\n!.yarn/patches\n!.yarn/plugins\n!.yarn/releases\n!.yarn/versions\n\n# testing\n/coverage\n\n# next.js\n/.next/\n/out/\n\n# production\n/build\n\n# misc\n.DS_Store\n*.pem\n\n# debug\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\n.pnpm-debug.log*\n\n# env files (can opt-in for committing if needed)\n.env*\n\n# vercel\n.vercel\n\n# typescript\n*.tsbuildinfo\nnext-env.d.ts\n" }, { "path": "frontend/README.md", "content": "This is a [Next.js](https://nextjs.org) project bootstrapped with [`create-next-app`](https://nextjs.org/docs/app/api-reference/cli/create-next-app).\n\n## Getting Started\n\nFirst, run the development server:\n\n```bash\nnpm run dev\n# or\nyarn dev\n# or\npnpm dev\n# or\nbun dev\n```\n\nOpen [http://localhost:3000](http://localhost:3000) with your browser to see the result.\n\nYou can start editing the page by modifying `app/page.tsx`. The page auto-updates as you edit the file.\n\nThis project uses [`next/font`](https://nextjs.org/docs/app/building-your-application/optimizing/fonts) to automatically optimize and load [Geist](https://vercel.com/font), a new font family for Vercel.\n\n## Learn More\n\nTo learn more about Next.js, take a look at the following resources:\n\n- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.\n- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.\n\nYou can check out [the Next.js GitHub repository](https://github.com/vercel/next.js) - your feedback and contributions are welcome!\n\n## Deploy on Vercel\n\nThe easiest way to deploy your Next.js app is to use the [Vercel Platform](https://vercel.com/new?utm_medium=default-template&filter=next.js&utm_source=create-next-app&utm_campaign=create-next-app-readme) from the creators of Next.js.\n\nCheck out our [Next.js deployment documentation](https://nextjs.org/docs/app/building-your-application/deploying) for more details.\n" }, { "path": "frontend/eslint.config.mjs", "content": "import { defineConfig, globalIgnores } from \"eslint/config\";\nimport nextVitals from \"eslint-config-next/core-web-vitals\";\nimport nextTs from \"eslint-config-next/typescript\";\n\nconst eslintConfig = defineConfig([\n ...nextVitals,\n ...nextTs,\n // Override default ignores of eslint-config-next.\n globalIgnores([\n // Default ignores of eslint-config-next:\n \".next/**\",\n \"out/**\",\n \"build/**\",\n \"next-env.d.ts\",\n ]),\n]);\n\nexport default eslintConfig;\n" }, { "path": "frontend/next.config.ts", "content": "import type { NextConfig } from \"next\";\n\nconst nextConfig: NextConfig = {\n /* config options here */\n};\n\nexport default nextConfig;\n" }, { "path": "frontend/package.json", "content": "{\n \"name\": \"frontend\",\n \"version\": \"0.1.0\",\n \"private\": true,\n \"scripts\": {\n \"dev\": \"next dev\",\n \"build\": \"next build\",\n \"start\": \"next start\",\n \"lint\": \"eslint\"\n },\n \"dependencies\": {\n \"axios\": \"^1.13.2\",\n \"lucide-react\": \"^0.562.0\",\n \"next\": \"16.1.0\",\n \"react\": \"19.2.3\",\n \"react-dom\": \"19.2.3\",\n \"wavesurfer.js\": \"^7.12.1\"\n },\n \"devDependencies\": {\n \"@tailwindcss/postcss\": \"^4\",\n \"@types/node\": \"^20\",\n \"@types/react\": \"^19\",\n \"@types/react-dom\": \"^19\",\n \"eslint\": \"^9\",\n \"eslint-config-next\": \"16.1.0\",\n \"tailwindcss\": \"^4\",\n \"typescript\": \"^5\"\n }\n}\n" }, { "path": "frontend/postcss.config.mjs", "content": "const config = {\n plugins: {\n \"@tailwindcss/postcss\": {},\n },\n};\n\nexport default config;\n" }, { "path": "frontend/src/app/globals.css", "content": "@import \"tailwindcss\";\n\n:root {\n /* AudioGhost Brand Colors */\n --ghost-primary: #8B5CF6;\n --ghost-secondary: #06B6D4;\n --ghost-accent: #F472B6;\n --ghost-success: #10B981;\n --ghost-warning: #F59E0B;\n --ghost-error: #EF4444;\n\n /* Dark Theme */\n --bg-primary: #0A0A0F;\n --bg-secondary: #12121A;\n --bg-tertiary: #1A1A25;\n --bg-card: #16161F;\n --bg-hover: #1E1E2A;\n\n /* Text Colors */\n --text-primary: #FFFFFF;\n --text-secondary: #A1A1AA;\n --text-muted: #71717A;\n\n /* Glassmorphism */\n --glass-bg: rgba(22, 22, 31, 0.8);\n --glass-border: rgba(139, 92, 246, 0.2);\n}\n\n* {\n box-sizing: border-box;\n padding: 0;\n margin: 0;\n}\n\nhtml {\n scroll-behavior: smooth;\n}\n\nbody {\n min-height: 100vh;\n background: var(--bg-primary);\n color: var(--text-primary);\n font-family: 'Inter', -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;\n}\n\nbody.light-mode {\n --bg-primary: #FAFAFA;\n --bg-secondary: #F5F5F5;\n --bg-tertiary: #EBEBEB;\n --bg-card: #FFFFFF;\n --bg-hover: #F0F0F0;\n --text-primary: #18181B;\n --text-secondary: #52525B;\n --text-muted: #A1A1AA;\n --glass-bg: rgba(255, 255, 255, 0.9);\n --glass-border: rgba(139, 92, 246, 0.3);\n}\n\n/* Custom Scrollbar */\n::-webkit-scrollbar {\n width: 8px;\n height: 8px;\n}\n\n::-webkit-scrollbar-track {\n background: var(--bg-secondary);\n}\n\n::-webkit-scrollbar-thumb {\n background: var(--ghost-primary);\n border-radius: 4px;\n}\n\n::-webkit-scrollbar-thumb:hover {\n background: #7C3AED;\n}\n\n/* Glassmorphism Card */\n.glass-card {\n background: var(--glass-bg);\n backdrop-filter: blur(12px);\n -webkit-backdrop-filter: blur(12px);\n border: 1px solid var(--glass-border);\n border-radius: 16px;\n}\n\n/* Gradient Text */\n.gradient-text {\n background: linear-gradient(135deg, var(--ghost-primary) 0%, var(--ghost-secondary) 50%, var(--ghost-accent) 100%);\n -webkit-background-clip: text;\n -webkit-text-fill-color: transparent;\n background-clip: text;\n}\n\n/* Glow Effects */\n.glow-primary {\n box-shadow: 0 0 20px rgba(139, 92, 246, 0.3);\n}\n\n.glow-secondary {\n box-shadow: 0 0 20px rgba(6, 182, 212, 0.3);\n}\n\n/* Button Styles */\n.btn-primary {\n background: linear-gradient(135deg, var(--ghost-primary), #7C3AED);\n color: white;\n padding: 12px 24px;\n border-radius: 12px;\n font-weight: 600;\n transition: all 0.3s ease;\n border: none;\n cursor: pointer;\n display: inline-flex;\n align-items: center;\n justify-content: center;\n}\n\n.btn-primary:hover {\n transform: translateY(-2px);\n box-shadow: 0 8px 25px rgba(139, 92, 246, 0.4);\n}\n\n.btn-primary:active {\n transform: translateY(0);\n}\n\n.btn-secondary {\n background: var(--bg-tertiary);\n color: var(--text-primary);\n padding: 12px 24px;\n border-radius: 12px;\n font-weight: 600;\n transition: all 0.3s ease;\n border: 1px solid var(--glass-border);\n cursor: pointer;\n display: inline-flex;\n align-items: center;\n justify-content: center;\n}\n\n.btn-secondary:hover {\n background: var(--bg-hover);\n border-color: var(--ghost-primary);\n}\n\n/* Input Styles */\n.input-ghost {\n background: var(--bg-tertiary);\n border: 1px solid var(--glass-border);\n border-radius: 12px;\n padding: 12px 16px;\n color: var(--text-primary);\n transition: all 0.3s ease;\n outline: none;\n width: 100%;\n}\n\n.input-ghost:focus {\n border-color: var(--ghost-primary);\n box-shadow: 0 0 0 3px rgba(139, 92, 246, 0.2);\n}\n\n.input-ghost::placeholder {\n color: var(--text-muted);\n}\n\n/* Waveform Container */\n.waveform-container {\n background: linear-gradient(180deg, var(--bg-secondary) 0%, var(--bg-tertiary) 100%);\n border-radius: 16px;\n padding: 24px;\n border: 1px solid var(--glass-border);\n}\n\n/* Progress Bar */\n.progress-bar {\n height: 6px;\n background: var(--bg-tertiary);\n border-radius: 3px;\n overflow: hidden;\n}\n\n.progress-bar-fill {\n height: 100%;\n background: linear-gradient(90deg, var(--ghost-primary), var(--ghost-secondary));\n border-radius: 3px;\n transition: width 0.3s ease;\n}\n\n/* Animations */\n@keyframes pulse-glow {\n\n 0%,\n 100% {\n box-shadow: 0 0 20px rgba(139, 92, 246, 0.3);\n }\n\n 50% {\n box-shadow: 0 0 40px rgba(139, 92, 246, 0.5);\n }\n}\n\n@keyframes float {\n\n 0%,\n 100% {\n transform: translateY(0);\n }\n\n 50% {\n transform: translateY(-10px);\n }\n}\n\n@keyframes shimmer {\n 0% {\n background-position: -200% 0;\n }\n\n 100% {\n background-position: 200% 0;\n }\n}\n\n.animate-pulse-glow {\n animation: pulse-glow 2s ease-in-out infinite;\n}\n\n.animate-float {\n animation: float 3s ease-in-out infinite;\n}\n\n.shimmer {\n background: linear-gradient(90deg,\n var(--bg-tertiary) 25%,\n var(--bg-hover) 50%,\n var(--bg-tertiary) 75%);\n background-size: 200% 100%;\n animation: shimmer 1.5s infinite;\n}\n\n/* Stem Mixer Track */\n.stem-track {\n background: var(--bg-secondary);\n border-radius: 12px;\n padding: 16px;\n border: 1px solid var(--glass-border);\n transition: all 0.3s ease;\n}\n\n.stem-track:hover {\n border-color: var(--ghost-primary);\n}\n\n.stem-track.original {\n border-left: 3px solid var(--ghost-primary);\n}\n\n.stem-track.ghost {\n border-left: 3px solid var(--ghost-accent);\n}\n\n.stem-track.clean {\n border-left: 3px solid var(--ghost-success);\n}\n\n/* Quick Action Tags */\n.quick-tag {\n display: inline-flex;\n align-items: center;\n gap: 6px;\n padding: 8px 16px;\n background: var(--bg-tertiary);\n border: 1px solid var(--glass-border);\n border-radius: 20px;\n font-size: 14px;\n color: var(--text-secondary);\n cursor: pointer;\n transition: all 0.3s ease;\n}\n\n.quick-tag:hover {\n background: var(--bg-hover);\n color: var(--text-primary);\n border-color: var(--ghost-primary);\n}\n\n.quick-tag.selected {\n background: linear-gradient(135deg, var(--ghost-primary), #7C3AED);\n color: white;\n border-color: transparent;\n}\n\n/* Upload Zone */\n.upload-zone {\n border: 2px dashed var(--glass-border);\n border-radius: 16px;\n padding: 64px 48px;\n text-align: center;\n transition: all 0.3s ease;\n cursor: pointer;\n background: var(--bg-secondary);\n}\n\n.upload-zone:hover {\n border-color: var(--ghost-primary);\n background: rgba(139, 92, 246, 0.05);\n}\n\n.upload-zone.dragover {\n border-color: var(--ghost-primary);\n background: rgba(139, 92, 246, 0.1);\n box-shadow: 0 0 30px rgba(139, 92, 246, 0.2);\n}\n\n/* Responsive */\n@media (max-width: 768px) {\n .upload-zone {\n padding: 48px 24px;\n }\n}" }, { "path": "frontend/src/app/layout.tsx", "content": "import type { Metadata } from \"next\";\nimport { Inter } from \"next/font/google\";\nimport \"./globals.css\";\n\nconst inter = Inter({\n subsets: [\"latin\"],\n variable: \"--font-inter\",\n});\n\nexport const metadata: Metadata = {\n title: \"AudioGhost AI - AI-Powered Audio Separation\",\n description: \"Separate any sound from audio using natural language. Remove vocals, extract instruments, eliminate background noise with state-of-the-art AI.\",\n keywords: [\"audio separation\", \"AI\", \"vocal removal\", \"stem separation\", \"audio editing\"],\n};\n\nexport default function RootLayout({\n children,\n}: Readonly<{\n children: React.ReactNode;\n}>) {\n return (\n \n \n {children}\n \n \n );\n}\n" }, { "path": "frontend/src/app/page.tsx", "content": "\"use client\";\n\nimport { useState, useEffect } from \"react\";\nimport Header from \"@/components/Header\";\nimport AuthModal from \"@/components/AuthModal\";\nimport AudioUploader from \"@/components/AudioUploader\";\nimport WaveformEditor from \"@/components/WaveformEditor\";\nimport SeparationPanel from \"@/components/SeparationPanel\";\nimport ProgressTracker from \"@/components/ProgressTracker\";\nimport StemMixer from \"@/components/StemMixer\";\nimport VideoStemMixer from \"@/components/VideoStemMixer\";\n\ninterface TaskResult {\n original_path: string;\n ghost_path: string;\n clean_path: string;\n description: string;\n mode: string;\n audio_duration?: number;\n processing_time?: number;\n model_size?: string;\n video_path?: string;\n is_video?: boolean;\n}\n\ninterface TaskState {\n taskId: string | null;\n status: \"idle\" | \"pending\" | \"processing\" | \"completed\" | \"failed\";\n progress: number;\n message: string;\n result: TaskResult | null;\n}\n\nexport default function Home() {\n const [isAuthenticated, setIsAuthenticated] = useState(false);\n const [showAuthModal, setShowAuthModal] = useState(false);\n const [audioFile, setAudioFile] = useState(null);\n const [audioUrl, setAudioUrl] = useState(null);\n const [isVideo, setIsVideo] = useState(false);\n const [isDarkMode, setIsDarkMode] = useState(true);\n const [selectedRegion, setSelectedRegion] = useState<{ start: number; end: number } | null>(null);\n\n // Persistent separation settings (won't reset on \"New\")\n const [separationSettings, setSeparationSettings] = useState({\n modelSize: \"base\" as \"small\" | \"base\" | \"large\",\n chunkDuration: 25,\n useFloat32: false,\n });\n\n const [task, setTask] = useState({\n taskId: null,\n status: \"idle\",\n progress: 0,\n message: \"\",\n result: null,\n });\n\n // Check auth status on mount\n useEffect(() => {\n checkAuthStatus();\n }, []);\n\n // Toggle theme\n useEffect(() => {\n document.body.classList.toggle(\"light-mode\", !isDarkMode);\n }, [isDarkMode]);\n\n const checkAuthStatus = async () => {\n try {\n const res = await fetch(\"http://localhost:8000/api/auth/status\");\n const data = await res.json();\n setIsAuthenticated(data.authenticated);\n } catch (error) {\n console.error(\"Failed to check auth status:\", error);\n }\n };\n\n const handleFileUpload = (file: File) => {\n setAudioFile(file);\n const url = URL.createObjectURL(file);\n setAudioUrl(url);\n\n // Detect if file is video\n const isVideoFile = file.type.startsWith(\"video/\") ||\n /\\.(mp4|webm|mov|avi|mkv)$/i.test(file.name);\n setIsVideo(isVideoFile);\n\n // Reset task state\n setTask({\n taskId: null,\n status: \"idle\",\n progress: 0,\n message: \"\",\n result: null,\n });\n };\n\n const handleReset = () => {\n // Clean up the object URL to free memory\n if (audioUrl) {\n URL.revokeObjectURL(audioUrl);\n }\n setAudioFile(null);\n setAudioUrl(null);\n setIsVideo(false);\n setSelectedRegion(null);\n setTask({\n taskId: null,\n status: \"idle\",\n progress: 0,\n message: \"\",\n result: null,\n });\n };\n\n const handleSeparation = async (\n description: string,\n mode: \"extract\" | \"remove\",\n modelSize: string = \"base\",\n chunkDuration: number = 25,\n useFloat32: boolean = false\n ) => {\n if (!audioFile) return;\n\n const formData = new FormData();\n formData.append(\"file\", audioFile);\n formData.append(\"description\", description);\n formData.append(\"mode\", mode);\n formData.append(\"model_size\", modelSize);\n formData.append(\"chunk_duration\", chunkDuration.toString());\n formData.append(\"use_float32\", useFloat32.toString());\n\n if (selectedRegion) {\n formData.append(\"start_time\", selectedRegion.start.toString());\n formData.append(\"end_time\", selectedRegion.end.toString());\n }\n\n try {\n const res = await fetch(\"http://localhost:8000/api/separate/\", {\n method: \"POST\",\n body: formData,\n });\n\n const data = await res.json();\n\n setTask({\n taskId: data.task_id,\n status: \"pending\",\n progress: 0,\n message: \"Task submitted...\",\n result: null,\n });\n\n // Start polling for status\n pollTaskStatus(data.task_id);\n } catch (error) {\n console.error(\"Failed to submit separation task:\", error);\n setTask(prev => ({\n ...prev,\n status: \"failed\",\n message: \"Failed to submit task\",\n }));\n }\n };\n\n const pollTaskStatus = async (taskId: string) => {\n const poll = async () => {\n try {\n const res = await fetch(`http://localhost:8000/api/tasks/${taskId}`);\n const data = await res.json();\n\n setTask({\n taskId,\n status: data.status,\n progress: data.progress,\n message: data.message || \"\",\n result: data.result || null,\n });\n\n if (data.status !== \"completed\" && data.status !== \"failed\") {\n setTimeout(poll, 1000);\n }\n } catch (error) {\n console.error(\"Failed to poll task status:\", error);\n }\n };\n\n poll();\n };\n\n return (\n \n setShowAuthModal(true)}\n isDarkMode={isDarkMode}\n onThemeToggle={() => setIsDarkMode(!isDarkMode)}\n onLogoClick={handleReset}\n />\n\n \n {/* Hero Section */}\n {!audioUrl && (\n
\n

\n AudioGhost{\" \"}\n AI\n

\n

\n AI-Powered Object-Oriented Audio Separation\n

\n

\n Describe the sound you want to extract or remove using natural language\n

\n
\n )}\n\n {/* Main Content */}\n
\n {/* Upload Zone */}\n {!audioUrl && (\n \n )}\n\n {/* Waveform Editor (Audio) or Video Preview - Hide when results are shown */}\n {audioUrl && task.status !== \"completed\" && (\n <>\n {/* Section Header with Upload Button */}\n
\n

\n {isVideo ? \"Video Preview\" : \"Audio Editor\"}\n

\n e.currentTarget.style.background = \"var(--bg-secondary)\"}\n onMouseOut={(e) => e.currentTarget.style.background = \"var(--bg-tertiary)\"}\n >\n ↩ Upload New File\n \n
\n\n {/* Show Video Player or Waveform based on file type */}\n {isVideo ? (\n \n \n

\n Audio will be extracted from this video for separation processing\n

\n
\n ) : (\n \n )}\n \n )}\n\n\n {/* Separation Controls */}\n {audioUrl && task.status === \"idle\" && (\n setShowAuthModal(true)}\n hasRegion={!!selectedRegion}\n settings={separationSettings}\n onSettingsChange={setSeparationSettings}\n />\n )}\n\n {/* Progress Tracker */}\n {(task.status === \"pending\" || task.status === \"processing\") && (\n \n )}\n\n {/* Results - Stem Mixer (Audio) or Video Stem Mixer */}\n {task.status === \"completed\" && task.result && task.taskId && (\n task.result.is_video ? (\n {\n setTask({\n taskId: null,\n status: \"idle\",\n progress: 0,\n message: \"\",\n result: null,\n });\n }}\n />\n ) : (\n {\n setTask({\n taskId: null,\n status: \"idle\",\n progress: 0,\n message: \"\",\n result: null,\n });\n }}\n />\n )\n )}\n\n {/* Error State */}\n {task.status === \"failed\" && (\n
\n
❌ Separation Failed
\n

{task.message}

\n setTask({ taskId: null, status: \"idle\", progress: 0, message: \"\", result: null })}\n >\n Try Again\n \n
\n )}\n \n \n\n {/* Auth Modal */}\n {showAuthModal && (\n setShowAuthModal(false)}\n onSuccess={() => {\n setIsAuthenticated(true);\n setShowAuthModal(false);\n }}\n />\n )}\n \n );\n}\n" }, { "path": "frontend/src/components/AudioUploader.tsx", "content": "\"use client\";\n\nimport { useState, useRef, useCallback } from \"react\";\nimport { Upload, Music, Video } from \"lucide-react\";\n\ninterface AudioUploaderProps {\n onFileUpload: (file: File) => void;\n}\n\nexport default function AudioUploader({ onFileUpload }: AudioUploaderProps) {\n const [isDragOver, setIsDragOver] = useState(false);\n const fileInputRef = useRef(null);\n\n const handleDragOver = useCallback((e: React.DragEvent) => {\n e.preventDefault();\n setIsDragOver(true);\n }, []);\n\n const handleDragLeave = useCallback((e: React.DragEvent) => {\n e.preventDefault();\n setIsDragOver(false);\n }, []);\n\n const handleDrop = useCallback((e: React.DragEvent) => {\n e.preventDefault();\n setIsDragOver(false);\n\n const file = e.dataTransfer.files[0];\n if (file && isMediaFile(file)) {\n onFileUpload(file);\n }\n }, [onFileUpload]);\n\n const handleFileSelect = (e: React.ChangeEvent) => {\n const file = e.target.files?.[0];\n if (file && isMediaFile(file)) {\n onFileUpload(file);\n }\n };\n\n const isMediaFile = (file: File) => {\n // Accept audio files\n if (file.type.startsWith(\"audio/\") ||\n /\\.(mp3|wav|flac|ogg|m4a|aac)$/i.test(file.name)) {\n return true;\n }\n // Accept video files\n if (file.type.startsWith(\"video/\") ||\n /\\.(mp4|webm|mov|avi|mkv)$/i.test(file.name)) {\n return true;\n }\n return false;\n };\n\n return (\n fileInputRef.current?.click()}\n >\n \n\n
\n {/* Icon */}\n \n {isDragOver ? (\n \n ) : (\n \n )}\n
\n\n {/* Text */}\n

\n {isDragOver ? \"Drop your file here\" : \"Upload Audio or Video\"}\n

\n

\n Drag & drop or click to browse\n

\n\n {/* Supported formats */}\n
\n {[\"MP3\", \"WAV\", \"FLAC\", \"MP4\", \"WebM\", \"MOV\"].map((format) => (\n \n {format}\n \n ))}\n
\n \n \n );\n}\n" }, { "path": "frontend/src/components/AuthModal.tsx", "content": "\"use client\";\n\nimport { useState } from \"react\";\nimport { X, Key, ExternalLink, Loader2, CheckCircle } from \"lucide-react\";\n\ninterface AuthModalProps {\n onClose: () => void;\n onSuccess: () => void;\n}\n\nexport default function AuthModal({ onClose, onSuccess }: AuthModalProps) {\n const [token, setToken] = useState(\"\");\n const [isLoading, setIsLoading] = useState(false);\n const [error, setError] = useState(\"\");\n const [step, setStep] = useState<\"input\" | \"success\">(\"input\");\n\n const handleSubmit = async (e: React.FormEvent) => {\n e.preventDefault();\n setIsLoading(true);\n setError(\"\");\n\n try {\n const res = await fetch(\"http://localhost:8000/api/auth/login\", {\n method: \"POST\",\n headers: { \"Content-Type\": \"application/json\" },\n body: JSON.stringify({ token }),\n });\n\n const data = await res.json();\n\n if (!res.ok) {\n throw new Error(data.detail || \"Authentication failed\");\n }\n\n setStep(\"success\");\n setTimeout(() => {\n onSuccess();\n }, 1500);\n } catch (err) {\n setError(err instanceof Error ? err.message : \"Authentication failed\");\n } finally {\n setIsLoading(false);\n }\n };\n\n return (\n \n e.stopPropagation()}\n >\n {/* Close Button */}\n \n \n \n\n {step === \"input\" ? (\n <>\n {/* Header */}\n
\n \n \n
\n

\n Connect HuggingFace\n

\n

\n Enter your HuggingFace token to access SAM Audio models\n

\n \n\n {/* Instructions */}\n \n

\n How to get your token:\n

\n
    \n
  1. \n 1.\n \n Request access to{\" \"}\n \n SAM Audio on HuggingFace\n \n \n \n
    2. \n
  2. \n 2.\n \n Create an{\" \"}\n \n access token\n \n \n \n
    3. \n
  3. \n 3.\n Paste the token below\n
    4. \n
\n \n\n {/* Form */}\n
\n
\n setToken(e.target.value)}\n placeholder=\"hf_xxxxxxxxxxxxxxxxxxxxxxxxxx\"\n className=\"input-ghost font-mono text-sm\"\n disabled={isLoading}\n />\n
\n\n {error && (\n \n {error}\n \n )}\n\n \n {isLoading ? (\n <>\n \n Verifying...\n \n ) : (\n \"Connect\"\n )}\n \n \n \n ) : (\n /* Success State */\n
\n \n \n
\n

\n Connected!\n

\n

\n You can now use SAM Audio models\n

\n \n )}\n \n \n );\n}\n" }, { "path": "frontend/src/components/Header.tsx", "content": "\"use client\";\n\nimport { Sun, Moon, Ghost, User, LogOut } from \"lucide-react\";\n\ninterface HeaderProps {\n isAuthenticated: boolean;\n onAuthClick: () => void;\n isDarkMode: boolean;\n onThemeToggle: () => void;\n onLogoClick?: () => void;\n}\n\nexport default function Header({\n isAuthenticated,\n onAuthClick,\n isDarkMode,\n onThemeToggle,\n onLogoClick\n}: HeaderProps) {\n return (\n \n \n {/* Logo - Clickable */}\n \n \n
\n

\n AudioGhost\n

\n

\n v1.0 MVP\n

\n
\n \n\n {/* Actions */}\n
\n {/* Theme Toggle */}\n \n {isDarkMode ? : }\n \n\n {/* Auth Button */}\n {isAuthenticated ? (\n \n \n \n
\n Connected\n \n ) : (\n \n \n Connect HuggingFace\n \n )}\n \n \n \n );\n\n}\n" }, { "path": "frontend/src/components/ProgressTracker.tsx", "content": "\"use client\";\n\nimport { Ghost, Loader2, CheckCircle2 } from \"lucide-react\";\n\ninterface ProgressTrackerProps {\n status: \"pending\" | \"processing\";\n progress: number;\n message: string;\n}\n\nexport default function ProgressTracker({ status, progress, message }: ProgressTrackerProps) {\n const steps = [\n { step: \"Loading model\", threshold: 10 },\n { step: \"Processing audio\", threshold: 30 },\n { step: \"Running separation\", threshold: 50 },\n { step: \"Saving results\", threshold: 80 },\n ];\n\n return (\n \n {/* Icon */}\n
\n \n {status === \"pending\" ? (\n \n ) : (\n \n )}\n
\n \n\n {/* Title */}\n \n {status === \"pending\" ? \"Waiting in Queue...\" : \"Processing Audio...\"}\n \n\n {/* Message */}\n \n {message}\n

\n\n {/* Progress Bar */}\n
\n \n \n Progress\n \n \n {progress}%\n \n
\n \n \n \n \n\n {/* Steps */}\n \n {steps.map(({ step, threshold }, index) => {\n const isComplete = progress >= threshold;\n const isActive = !isComplete && progress >= threshold - 20;\n\n return (\n \n {/* Status Icon */}\n \n {isComplete ? (\n \n ) : isActive ? (\n \n ) : null}\n \n\n {/* Step Label */}\n \n {step}\n \n \n );\n })}\n \n \n );\n}\n" }, { "path": "frontend/src/components/SeparationPanel.tsx", "content": "\"use client\";\n\nimport { useState } from \"react\";\nimport {\n Mic,\n Music,\n Volume2,\n Sparkles,\n Clock,\n AlertCircle,\n Cpu,\n Search,\n Sliders,\n Zap\n} from \"lucide-react\";\n\ninterface SeparationSettings {\n modelSize: \"small\" | \"base\" | \"large\";\n chunkDuration: number;\n useFloat32: boolean;\n}\n\ninterface SeparationPanelProps {\n onSeparate: (description: string, mode: \"extract\" | \"remove\", modelSize: string, chunkDuration: number, useFloat32: boolean) => void;\n isAuthenticated: boolean;\n onAuthRequired: () => void;\n hasRegion: boolean;\n // Persistent settings from parent\n settings: SeparationSettings;\n onSettingsChange: (settings: SeparationSettings) => void;\n}\n\nconst QUICK_PROMPTS = [\n { icon: Mic, label: \"Voice\", prompt: \"singing voice\", color: \"#8B5CF6\" },\n { icon: Music, label: \"Music\", prompt: \"background music\", color: \"#06B6D4\" },\n { icon: Volume2, label: \"Drums\", prompt: \"drums and percussion\", color: \"#F472B6\" },\n { icon: Sparkles, label: \"Guitar\", prompt: \"acoustic guitar\", color: \"#10B981\" },\n { icon: Volume2, label: \"Bass\", prompt: \"bass\", color: \"#F59E0B\" },\n { icon: Volume2, label: \"Piano\", prompt: \"piano\", color: \"#EF4444\" },\n];\n\nconst MODEL_OPTIONS = [\n { value: \"small\", label: \"Small\", vram: \"~6GB\", vramFp32: \"~9GB\", speed: \"Fast\" },\n { value: \"base\", label: \"Base\", vram: \"~7GB\", vramFp32: \"~10GB\", speed: \"Balanced\" },\n { value: \"large\", label: \"Large\", vram: \"~10GB\", vramFp32: \"~13GB\", speed: \"Best\" },\n] as const;\n\nexport default function SeparationPanel({\n onSeparate,\n isAuthenticated,\n onAuthRequired,\n hasRegion,\n settings,\n onSettingsChange\n}: SeparationPanelProps) {\n // Only prompt and mode are local (reset each time)\n const [customPrompt, setCustomPrompt] = useState(\"\");\n const [selectedPrompt, setSelectedPrompt] = useState(null);\n const [mode, setMode] = useState<\"extract\" | \"remove\">(\"extract\");\n\n // Destructure settings for easier access\n const { modelSize, chunkDuration, useFloat32 } = settings;\n\n // Helper to update a single setting\n const updateSetting = (key: K, value: SeparationSettings[K]) => {\n onSettingsChange({ ...settings, [key]: value });\n };\n\n const handleQuickSelect = (prompt: string) => {\n setSelectedPrompt(prompt);\n setCustomPrompt(prompt);\n };\n\n const handleSeparate = () => {\n if (!isAuthenticated) {\n onAuthRequired();\n return;\n }\n\n const prompt = customPrompt || selectedPrompt;\n if (!prompt) return;\n\n onSeparate(prompt, mode, modelSize, chunkDuration, useFloat32);\n };\n\n const activePrompt = customPrompt || selectedPrompt;\n\n return (\n \n {/* Header */}\n \n

\n Separation Settings\n

\n\n {hasRegion && (\n \n \n Temporal Lock\n \n )}\n \n\n
\n\n {/* ============================================ */}\n {/* MAIN INPUT - Describe the sound (PROMINENT) */}\n {/* ============================================ */}\n \n \n {\n setCustomPrompt(e.target.value);\n setSelectedPrompt(null);\n }}\n placeholder=\"e.g., singing voice, drums, police siren, crowd noise, a dog barking...\"\n style={{\n width: \"100%\",\n padding: \"16px 18px\",\n borderRadius: \"12px\",\n border: \"2px solid var(--ghost-primary)\",\n background: \"var(--bg-primary)\",\n color: \"var(--text-primary)\",\n fontSize: \"1rem\",\n fontWeight: 500,\n outline: \"none\",\n boxShadow: \"0 4px 15px rgba(168, 85, 247, 0.15)\"\n }}\n />\n\n {/* Quick Select Tags */}\n
\n {QUICK_PROMPTS.map(({ icon: Icon, label, prompt, color }) => (\n handleQuickSelect(prompt)}\n style={{\n display: \"flex\",\n alignItems: \"center\",\n gap: \"6px\",\n padding: \"8px 12px\",\n borderRadius: \"20px\",\n border: \"none\",\n cursor: \"pointer\",\n fontSize: \"0.8rem\",\n fontWeight: 500,\n transition: \"all 0.2s\",\n background: selectedPrompt === prompt\n ? `linear-gradient(135deg, ${color}, ${color}dd)`\n : \"var(--bg-tertiary)\",\n color: selectedPrompt === prompt\n ? \"white\"\n : \"var(--text-secondary)\"\n }}\n >\n \n {label}\n \n ))}\n
\n
\n\n {/* Settings Grid */}\n
\n\n {/* Mode Toggle */}\n
\n \n
\n setMode(\"extract\")}\n style={{\n flex: 1,\n padding: \"12px\",\n borderRadius: \"10px\",\n fontWeight: 500,\n fontSize: \"0.85rem\",\n border: \"none\",\n cursor: \"pointer\",\n background: mode === \"extract\"\n ? \"linear-gradient(135deg, var(--ghost-primary), #7C3AED)\"\n : \"var(--bg-tertiary)\",\n color: mode === \"extract\" ? \"white\" : \"var(--text-secondary)\"\n }}\n >\n ✨ Extract\n \n setMode(\"remove\")}\n style={{\n flex: 1,\n padding: \"12px\",\n borderRadius: \"10px\",\n fontWeight: 500,\n fontSize: \"0.85rem\",\n border: \"none\",\n cursor: \"pointer\",\n background: mode === \"remove\"\n ? \"linear-gradient(135deg, var(--ghost-error), #DC2626)\"\n : \"var(--bg-tertiary)\",\n color: mode === \"remove\" ? \"white\" : \"var(--text-secondary)\"\n }}\n >\n 🗑️ Remove\n \n
\n
\n\n {/* Model Selector */}\n
\n \n
\n {MODEL_OPTIONS.map(({ value, label, vram, vramFp32 }) => (\n updateSetting(\"modelSize\", value)}\n style={{\n flex: 1,\n padding: \"10px 8px\",\n borderRadius: \"8px\",\n border: \"none\",\n cursor: \"pointer\",\n textAlign: \"center\",\n background: modelSize === value\n ? \"linear-gradient(135deg, #6366f1, #8b5cf6)\"\n : \"var(--bg-tertiary)\",\n color: modelSize === value ? \"white\" : \"var(--text-secondary)\"\n }}\n >\n
{label}
\n
\n {useFloat32 ? vramFp32 : vram}\n
\n \n ))}\n
\n
\n
\n\n {/* Float32 Precision Toggle */}\n \n
\n \n
\n
\n High Quality Mode (float32)\n
\n
\n Better separation quality, +2-3GB VRAM\n
\n
\n
\n updateSetting(\"useFloat32\", !useFloat32)}\n style={{\n width: \"44px\",\n height: \"24px\",\n borderRadius: \"12px\",\n border: \"none\",\n cursor: \"pointer\",\n position: \"relative\",\n background: useFloat32\n ? \"linear-gradient(135deg, #10B981, #06B6D4)\"\n : \"var(--bg-secondary)\",\n transition: \"all 0.2s ease\"\n }}\n >\n
\n \n
\n\n {/* Chunk Duration Slider */}\n \n
\n \n \n {chunkDuration}s\n \n
\n\n updateSetting(\"chunkDuration\", Number(e.target.value))}\n style={{\n width: \"100%\",\n height: \"6px\",\n borderRadius: \"3px\",\n background: `linear-gradient(to right, var(--ghost-primary) ${((chunkDuration - 5) / 55) * 100}%, var(--bg-secondary) ${((chunkDuration - 5) / 55) * 100}%)`,\n cursor: \"pointer\",\n appearance: \"none\",\n outline: \"none\"\n }}\n />\n\n
\n 5s (Low VRAM)\n 60s (Fast)\n
\n\n

\n ⚡ Smaller chunks = Less VRAM usage, but slower processing & may affect quality at boundaries\n

\n \n\n {/* Auth Warning */}\n {!isAuthenticated && (\n \n \n

\n Connect HuggingFace to continue\n

\n \n )}\n\n {/* Action Button */}\n \n {mode === \"extract\" ? \"✨ Extract\" : \"🗑️ Remove\"} \"{activePrompt || \"...\"}\"\n \n \n \n );\n}\n" }, { "path": "frontend/src/components/StemMixer.tsx", "content": "\"use client\";\n\nimport { useState, useRef, useEffect, useCallback } from \"react\";\nimport {\n Play,\n Pause,\n Download,\n Volume2,\n VolumeX,\n RefreshCw,\n Ghost,\n Leaf,\n SkipBack,\n Music,\n type LucideIcon\n} from \"lucide-react\";\nimport WaveSurfer from \"wavesurfer.js\";\n\ninterface StemMixerProps {\n taskId: string;\n description: string;\n onNewSeparation: () => void;\n onUploadNew?: () => void;\n audioDuration?: number;\n processingTime?: number;\n modelSize?: string;\n}\n\ninterface Track {\n id: \"ghost\" | \"clean\";\n label: string;\n icon: LucideIcon;\n color: string;\n waveColor: string;\n}\n\nconst TRACKS: Track[] = [\n {\n id: \"ghost\",\n label: \"Isolated Sound\",\n icon: Ghost,\n color: \"#F472B6\",\n waveColor: \"#F472B6\"\n },\n {\n id: \"clean\",\n label: \"Without Isolated Sound\",\n icon: Leaf,\n color: \"#60A5FA\",\n waveColor: \"#60A5FA\"\n },\n];\n\nexport default function StemMixer({\n taskId,\n description,\n onNewSeparation,\n onUploadNew,\n audioDuration,\n processingTime,\n modelSize\n}: StemMixerProps) {\n const [isPlaying, setIsPlaying] = useState(false);\n const [currentTime, setCurrentTime] = useState(0);\n const [duration, setDuration] = useState(0);\n const [muted, setMuted] = useState>({\n ghost: false,\n clean: false,\n });\n const [isReady, setIsReady] = useState>({\n ghost: false,\n clean: false,\n });\n\n const wavesurferRefs = useRef>({});\n const containerRefs = useRef>({});\n const isSeeking = useRef(false);\n\n const getAudioUrl = (trackId: string) => {\n return `http://localhost:8000/api/tasks/${taskId}/download/${trackId}`;\n };\n\n useEffect(() => {\n const initWaveSurfers = async () => {\n for (const track of TRACKS) {\n const container = containerRefs.current[track.id];\n if (!container) continue;\n\n if (wavesurferRefs.current[track.id]) {\n wavesurferRefs.current[track.id]?.destroy();\n }\n\n const ws = WaveSurfer.create({\n container,\n waveColor: `${track.waveColor}40`,\n progressColor: track.waveColor,\n cursorColor: \"#ffffff\",\n cursorWidth: 1,\n barWidth: 2,\n barGap: 2,\n barRadius: 2,\n height: 48,\n normalize: true,\n // All tracks are seekable - clicking any will sync all others\n interact: true,\n hideScrollbar: true,\n });\n\n ws.load(getAudioUrl(track.id));\n\n ws.on(\"ready\", () => {\n setIsReady(prev => ({ ...prev, [track.id]: true }));\n if (track.id === \"ghost\") {\n setDuration(ws.getDuration());\n }\n ws.setMuted(muted[track.id]);\n });\n\n ws.on(\"audioprocess\", () => {\n if (!isSeeking.current && track.id === \"ghost\") {\n setCurrentTime(ws.getCurrentTime());\n }\n });\n\n ws.on(\"finish\", () => {\n // Only trigger finish logic from the \"ghost\" track (master)\n // to prevent multiple finish events causing desync\n if (track.id === \"ghost\") {\n setIsPlaying(false);\n setCurrentTime(0);\n // Seek all tracks back to start synchronized\n Object.values(wavesurferRefs.current).forEach(w => {\n if (w) {\n w.pause();\n w.seekTo(0);\n }\n });\n }\n });\n\n // Sync seeking across all tracks - when ANY track is seeked\n ws.on(\"seeking\", () => {\n if (isSeeking.current) return; // Prevent recursive seeking\n\n isSeeking.current = true;\n const progress = ws.getCurrentTime() / ws.getDuration();\n\n // Sync all OTHER tracks to the same position\n Object.entries(wavesurferRefs.current).forEach(([id, w]) => {\n if (w && id !== track.id) {\n w.seekTo(progress);\n }\n });\n\n setCurrentTime(ws.getCurrentTime());\n\n // Reset seeking flag after a short delay\n setTimeout(() => {\n isSeeking.current = false;\n }, 50);\n });\n\n wavesurferRefs.current[track.id] = ws;\n }\n };\n\n initWaveSurfers();\n\n return () => {\n // Use setTimeout to avoid AbortError when component unmounts during loading\n const refs = { ...wavesurferRefs.current };\n setTimeout(() => {\n Object.values(refs).forEach(ws => {\n if (ws) {\n try { ws.destroy(); } catch { /* ignore AbortError */ }\n }\n });\n }, 0);\n };\n }, [taskId]);\n\n // Continuous sync effect - keep all tracks aligned to original during playback\n useEffect(() => {\n let animationFrameId: number;\n const syncInterval = 100; // Sync every 100ms\n let lastSync = 0;\n\n const syncTracks = (timestamp: number) => {\n if (isPlaying && timestamp - lastSync > syncInterval) {\n lastSync = timestamp;\n const originalWs = wavesurferRefs.current[\"original\"];\n if (originalWs) {\n const masterTime = originalWs.getCurrentTime();\n const masterDuration = originalWs.getDuration();\n const progress = masterTime / masterDuration;\n\n // Check if original finished\n if (masterTime >= masterDuration - 0.1) {\n // Stop all tracks\n Object.values(wavesurferRefs.current).forEach(ws => ws?.pause());\n Object.values(wavesurferRefs.current).forEach(ws => ws?.seekTo(0));\n setIsPlaying(false);\n setCurrentTime(0);\n return;\n }\n\n // Sync other tracks to master\n Object.entries(wavesurferRefs.current).forEach(([id, ws]) => {\n if (ws && id !== \"original\") {\n const trackTime = ws.getCurrentTime();\n // Only sync if drift is more than 0.05 seconds\n if (Math.abs(trackTime - masterTime) > 0.05) {\n ws.seekTo(progress);\n }\n }\n });\n\n setCurrentTime(masterTime);\n }\n }\n animationFrameId = requestAnimationFrame(syncTracks);\n };\n\n if (isPlaying) {\n animationFrameId = requestAnimationFrame(syncTracks);\n }\n\n return () => {\n if (animationFrameId) {\n cancelAnimationFrame(animationFrameId);\n }\n };\n }, [isPlaying]);\n\n // Sync play/pause across all tracks\n const togglePlayAll = useCallback(() => {\n const allReady = Object.values(isReady).every(r => r);\n if (!allReady) return;\n\n if (isPlaying) {\n // Pause all\n Object.values(wavesurferRefs.current).forEach(ws => ws?.pause());\n setIsPlaying(false);\n } else {\n // First sync all tracks to the same position\n const originalWs = wavesurferRefs.current[\"original\"];\n if (originalWs) {\n const progress = originalWs.getCurrentTime() / originalWs.getDuration();\n Object.entries(wavesurferRefs.current).forEach(([id, ws]) => {\n if (ws && id !== \"original\") {\n ws.seekTo(progress);\n }\n });\n }\n\n // Then play all together\n Object.values(wavesurferRefs.current).forEach(ws => ws?.play());\n setIsPlaying(true);\n }\n }, [isPlaying, isReady]);\n\n const resetToStart = useCallback(() => {\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) {\n ws.pause();\n ws.seekTo(0);\n }\n });\n setIsPlaying(false);\n setCurrentTime(0);\n }, []);\n\n const toggleMute = useCallback((trackId: string) => {\n const ws = wavesurferRefs.current[trackId];\n if (ws) {\n const newMuted = !muted[trackId];\n ws.setMuted(newMuted);\n setMuted(prev => ({ ...prev, [trackId]: newMuted }));\n }\n }, [muted]);\n\n const handleSeek = useCallback((e: React.MouseEvent) => {\n const rect = e.currentTarget.getBoundingClientRect();\n const progress = Math.max(0, Math.min(1, (e.clientX - rect.left) / rect.width));\n\n isSeeking.current = true;\n // Seek all tracks to the same position\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) ws.seekTo(progress);\n });\n setCurrentTime(progress * duration);\n isSeeking.current = false;\n }, [duration]);\n\n const downloadTrack = (trackId: string, label: string) => {\n const link = document.createElement(\"a\");\n link.href = getAudioUrl(trackId);\n link.download = `${taskId}_${label.toLowerCase().replace(/\\s+/g, \"_\")}.wav`;\n document.body.appendChild(link);\n link.click();\n document.body.removeChild(link);\n };\n\n const formatTime = (seconds: number) => {\n const mins = Math.floor(seconds / 60);\n const secs = Math.floor(seconds % 60);\n return `${mins}:${secs.toString().padStart(2, \"0\")}`;\n };\n\n const allReady = Object.values(isReady).every(r => r);\n\n return (\n \n {/* Header */}\n \n
\n

\n ✨ Separation Complete\n

\n

\n \"{description}\"\n

\n
\n
\n {onUploadNew && (\n \n ↩ Upload New File\n \n )}\n \n \n New\n \n
\n \n\n {/* Stats Bar */}\n {(audioDuration || processingTime || modelSize) && (\n \n {audioDuration !== undefined && (\n
\n \n Audio:\n \n \n {Math.floor(audioDuration / 60)}:{(audioDuration % 60).toFixed(0).padStart(2, \"0\")}\n \n
\n )}\n {processingTime !== undefined && (\n
\n \n Processing:\n \n \n {processingTime.toFixed(1)}s\n \n
\n )}\n {modelSize && (\n
\n \n Model:\n \n \n {modelSize}\n \n
\n )}\n {audioDuration && processingTime && (\n
\n \n Speed:\n \n \n {(audioDuration / processingTime).toFixed(1)}x\n \n
\n )}\n \n )}\n\n {/* Transport Controls */}\n \n \n \n \n\n \n {isPlaying ? (\n \n ) : (\n \n )}\n \n\n
\n \n {formatTime(currentTime)}\n \n\n \n 0 ? (currentTime / duration) * 100 : 0}%`,\n transition: \"width 0.1s\"\n }}\n />\n
\n\n \n {formatTime(duration)}\n \n \n \n\n {/* Tracks */}\n
\n {TRACKS.map((track) => {\n const TrackIcon = track.icon;\n const isMuted = muted[track.id];\n const trackReady = isReady[track.id];\n\n return (\n \n {/* Mute Button */}\n toggleMute(track.id)}\n style={{\n width: \"32px\",\n height: \"32px\",\n borderRadius: \"8px\",\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n background: isMuted ? \"var(--bg-secondary)\" : `${track.color}20`,\n color: isMuted ? \"var(--text-muted)\" : track.color,\n border: \"none\",\n cursor: \"pointer\",\n flexShrink: 0\n }}\n >\n {isMuted ? (\n \n ) : (\n \n )}\n \n\n {/* Track Label */}\n
\n \n \n {track.label}\n \n
\n\n {/* Waveform */}\n { containerRefs.current[track.id] = el; }}\n style={{\n flex: 1,\n borderRadius: \"8px\",\n overflow: \"hidden\",\n background: \"var(--bg-secondary)\",\n minHeight: \"48px\"\n }}\n >\n {!trackReady && (\n
\n \n Loading...\n \n
\n )}\n
\n\n {/* Download */}\n downloadTrack(track.id, track.label)}\n style={{\n width: \"32px\",\n height: \"32px\",\n borderRadius: \"8px\",\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n background: \"var(--bg-secondary)\",\n color: \"var(--text-muted)\",\n border: \"none\",\n cursor: \"pointer\",\n flexShrink: 0\n }}\n >\n \n \n \n );\n })}\n \n\n {/* Download All */}\n
\n TRACKS.forEach((track) => downloadTrack(track.id, track.label))}\n style={{\n width: \"100%\",\n padding: \"14px\",\n borderRadius: \"10px\",\n background: \"linear-gradient(135deg, #6366f1, #8b5cf6)\",\n color: \"white\",\n border: \"none\",\n cursor: \"pointer\",\n fontSize: \"0.9rem\",\n fontWeight: 600,\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n gap: \"8px\",\n boxShadow: \"0 4px 12px rgba(99, 102, 241, 0.3)\"\n }}\n >\n \n Download All Stems\n \n
\n \n );\n}\n" }, { "path": "frontend/src/components/VideoStemMixer.tsx", "content": "\"use client\";\n\nimport { useState, useRef, useEffect, useCallback } from \"react\";\nimport {\n Play,\n Pause,\n Download,\n Volume2,\n VolumeX,\n RefreshCw,\n Ghost,\n Leaf,\n SkipBack,\n Film,\n type LucideIcon\n} from \"lucide-react\";\nimport WaveSurfer from \"wavesurfer.js\";\n\ninterface VideoStemMixerProps {\n taskId: string;\n description: string;\n onNewSeparation: () => void;\n onUploadNew?: () => void;\n audioDuration?: number;\n processingTime?: number;\n modelSize?: string;\n}\n\ninterface Track {\n id: \"ghost\" | \"clean\";\n label: string;\n icon: LucideIcon;\n color: string;\n waveColor: string;\n}\n\nconst TRACKS: Track[] = [\n {\n id: \"ghost\",\n label: \"Isolated Sound\",\n icon: Ghost,\n color: \"#F472B6\",\n waveColor: \"#F472B6\"\n },\n {\n id: \"clean\",\n label: \"Without Isolated Sound\",\n icon: Leaf,\n color: \"#60A5FA\",\n waveColor: \"#60A5FA\"\n },\n];\n\nexport default function VideoStemMixer({\n taskId,\n description,\n onNewSeparation,\n onUploadNew,\n audioDuration,\n processingTime,\n modelSize\n}: VideoStemMixerProps) {\n const [isPlaying, setIsPlaying] = useState(false);\n const [currentTime, setCurrentTime] = useState(0);\n const [duration, setDuration] = useState(0);\n const [muted, setMuted] = useState>({\n ghost: false,\n clean: false,\n });\n const [videoMuted, setVideoMuted] = useState(true);\n const [showVideoDownload, setShowVideoDownload] = useState(false);\n const [isReady, setIsReady] = useState>({\n video: false,\n ghost: false,\n clean: false,\n });\n\n const videoRef = useRef(null);\n const wavesurferRefs = useRef>({});\n const containerRefs = useRef>({});\n const isSeeking = useRef(false);\n\n const getAudioUrl = (trackId: string) => {\n return `http://localhost:8000/api/tasks/${taskId}/download/${trackId}`;\n };\n\n const getVideoUrl = () => {\n return `http://localhost:8000/api/tasks/${taskId}/download/video`;\n };\n\n // Initialize video and wavesurfers\n useEffect(() => {\n const initWaveSurfers = async () => {\n for (const track of TRACKS) {\n const container = containerRefs.current[track.id];\n if (!container) continue;\n\n if (wavesurferRefs.current[track.id]) {\n wavesurferRefs.current[track.id]?.destroy();\n }\n\n const ws = WaveSurfer.create({\n container,\n waveColor: `${track.waveColor}40`,\n progressColor: track.waveColor,\n cursorColor: \"#ffffff\",\n cursorWidth: 1,\n barWidth: 2,\n barGap: 2,\n barRadius: 2,\n height: 48,\n normalize: true,\n interact: true,\n hideScrollbar: true,\n });\n\n ws.load(getAudioUrl(track.id));\n\n ws.on(\"ready\", () => {\n setIsReady(prev => ({ ...prev, [track.id]: true }));\n if (track.id === \"ghost\") {\n setDuration(ws.getDuration());\n }\n ws.setMuted(muted[track.id]);\n });\n\n // When user clicks on waveform - sync video and other tracks\n // Use 'interaction' event to detect actual user clicks vs programmatic seeks\n let isUserInteracting = false;\n\n ws.on(\"interaction\", () => {\n isUserInteracting = true;\n });\n\n ws.on(\"seeking\", () => {\n // Only handle user-initiated seeks, not programmatic ones\n if (!isUserInteracting) return;\n isUserInteracting = false;\n\n if (isSeeking.current) return;\n isSeeking.current = true;\n\n const progress = ws.getCurrentTime() / ws.getDuration();\n const newTime = ws.getCurrentTime();\n\n // Sync video\n if (videoRef.current) {\n videoRef.current.currentTime = newTime;\n }\n\n // Sync other audio tracks\n Object.entries(wavesurferRefs.current).forEach(([id, w]) => {\n if (w && id !== track.id) {\n w.seekTo(progress);\n }\n });\n\n setCurrentTime(newTime);\n\n // Reset seeking flag after short delay\n setTimeout(() => {\n isSeeking.current = false;\n }, 150);\n });\n\n wavesurferRefs.current[track.id] = ws;\n }\n };\n\n initWaveSurfers();\n\n return () => {\n const refs = { ...wavesurferRefs.current };\n setTimeout(() => {\n Object.values(refs).forEach(ws => {\n if (ws) {\n try { ws.destroy(); } catch { /* ignore */ }\n }\n });\n }, 0);\n };\n }, [taskId]);\n\n // Handle video events\n useEffect(() => {\n const video = videoRef.current;\n if (!video) return;\n\n const handleTimeUpdate = () => {\n if (!isSeeking.current) {\n setCurrentTime(video.currentTime);\n }\n };\n\n const handleLoadedMetadata = () => {\n setIsReady(prev => ({ ...prev, video: true }));\n setDuration(video.duration);\n };\n\n const handleEnded = () => {\n setIsPlaying(false);\n setCurrentTime(0);\n // Pause all audio tracks when video ends\n Object.values(wavesurferRefs.current).forEach(w => {\n if (w) {\n w.pause();\n w.seekTo(0);\n }\n });\n };\n\n // When video starts seeking (user dragging video scrubber)\n const handleSeeking = () => {\n isSeeking.current = true;\n };\n\n // When video finishes seeking - sync all audio tracks to video position\n const handleSeeked = () => {\n const progress = video.currentTime / video.duration;\n // Sync all audio tracks to the new video position\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) {\n ws.seekTo(progress);\n }\n });\n setCurrentTime(video.currentTime);\n // Small delay before allowing new syncs\n setTimeout(() => {\n isSeeking.current = false;\n }, 100);\n };\n\n video.addEventListener(\"timeupdate\", handleTimeUpdate);\n video.addEventListener(\"loadedmetadata\", handleLoadedMetadata);\n video.addEventListener(\"ended\", handleEnded);\n video.addEventListener(\"seeking\", handleSeeking);\n video.addEventListener(\"seeked\", handleSeeked);\n\n return () => {\n video.removeEventListener(\"timeupdate\", handleTimeUpdate);\n video.removeEventListener(\"loadedmetadata\", handleLoadedMetadata);\n video.removeEventListener(\"ended\", handleEnded);\n video.removeEventListener(\"seeking\", handleSeeking);\n video.removeEventListener(\"seeked\", handleSeeked);\n };\n }, []);\n\n // Continuous sync effect - keeps audio tracks aligned with video during playback\n useEffect(() => {\n let animationFrameId: number;\n const syncInterval = 150; // Check less frequently\n let lastSync = 0;\n\n const syncTracks = (timestamp: number) => {\n // Skip sync during seeking to prevent interference\n if (isPlaying && !isSeeking.current && timestamp - lastSync > syncInterval) {\n lastSync = timestamp;\n const video = videoRef.current;\n if (video && !video.seeking) {\n const masterTime = video.currentTime;\n const masterDuration = video.duration;\n const progress = masterTime / masterDuration;\n\n // Sync audio tracks to video only if drift is significant\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) {\n const trackTime = ws.getCurrentTime();\n // Only sync if drift is more than 0.15 seconds\n if (Math.abs(trackTime - masterTime) > 0.15) {\n ws.seekTo(progress);\n }\n }\n });\n\n setCurrentTime(masterTime);\n }\n }\n animationFrameId = requestAnimationFrame(syncTracks);\n };\n\n if (isPlaying) {\n animationFrameId = requestAnimationFrame(syncTracks);\n }\n\n return () => {\n if (animationFrameId) {\n cancelAnimationFrame(animationFrameId);\n }\n };\n }, [isPlaying]);\n\n const togglePlayAll = useCallback(() => {\n const allReady = Object.values(isReady).every(r => r);\n if (!allReady) return;\n\n const video = videoRef.current;\n if (!video) return;\n\n if (isPlaying) {\n video.pause();\n Object.values(wavesurferRefs.current).forEach(ws => ws?.pause());\n setIsPlaying(false);\n } else {\n // Sync all tracks to video position first\n const progress = video.currentTime / video.duration;\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) ws.seekTo(progress);\n });\n\n video.play();\n Object.values(wavesurferRefs.current).forEach(ws => ws?.play());\n setIsPlaying(true);\n }\n }, [isPlaying, isReady]);\n\n const resetToStart = useCallback(() => {\n const video = videoRef.current;\n if (video) {\n video.pause();\n video.currentTime = 0;\n }\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) {\n ws.pause();\n ws.seekTo(0);\n }\n });\n setIsPlaying(false);\n setCurrentTime(0);\n }, []);\n\n const toggleMute = useCallback((trackId: string) => {\n const ws = wavesurferRefs.current[trackId];\n if (ws) {\n const newMuted = !muted[trackId];\n ws.setMuted(newMuted);\n setMuted(prev => ({ ...prev, [trackId]: newMuted }));\n }\n }, [muted]);\n\n const handleSeek = useCallback((e: React.MouseEvent) => {\n const rect = e.currentTarget.getBoundingClientRect();\n const progress = Math.max(0, Math.min(1, (e.clientX - rect.left) / rect.width));\n const newTime = progress * duration;\n\n isSeeking.current = true;\n\n // Seek video - video's 'seeked' event will reset isSeeking and sync audio\n if (videoRef.current) {\n videoRef.current.currentTime = newTime;\n }\n\n // Also seek audio tracks immediately for visual feedback\n Object.values(wavesurferRefs.current).forEach(ws => {\n if (ws) ws.seekTo(progress);\n });\n\n setCurrentTime(newTime);\n // Note: isSeeking is reset by video's 'seeked' event\n }, [duration]);\n\n const downloadTrack = (trackId: string, label: string) => {\n const link = document.createElement(\"a\");\n link.href = getAudioUrl(trackId);\n link.download = `${taskId}_${label.toLowerCase().replace(/\\s+/g, \"_\")}.wav`;\n document.body.appendChild(link);\n link.click();\n document.body.removeChild(link);\n };\n\n const downloadVideoWithAudio = (audioType: \"original\" | \"ghost\" | \"clean\") => {\n const link = document.createElement(\"a\");\n link.href = `http://localhost:8000/api/tasks/${taskId}/download-video-with-audio/${audioType}`;\n const labels = { original: \"original\", ghost: \"isolated\", clean: \"without_isolated\" };\n link.download = `${taskId}_${labels[audioType]}_video.mp4`;\n document.body.appendChild(link);\n link.click();\n document.body.removeChild(link);\n setShowVideoDownload(false);\n };\n\n const formatTime = (seconds: number) => {\n const mins = Math.floor(seconds / 60);\n const secs = Math.floor(seconds % 60);\n return `${mins}:${secs.toString().padStart(2, \"0\")}`;\n };\n\n const allReady = Object.values(isReady).every(r => r);\n\n return (\n \n {/* Header */}\n \n
\n

\n \n Video Separation Complete\n

\n

\n \"{description}\"\n

\n
\n
\n {onUploadNew && (\n \n ↩ Upload New File\n \n )}\n \n \n New\n \n
\n \n\n {/* Video Player */}\n
\n
\n \n {/* Video Mute Toggle Button */}\n setVideoMuted(!videoMuted)}\n style={{\n position: \"absolute\",\n bottom: \"12px\",\n right: \"12px\",\n width: \"40px\",\n height: \"40px\",\n borderRadius: \"50%\",\n background: \"rgba(0, 0, 0, 0.7)\",\n border: \"1px solid rgba(255, 255, 255, 0.2)\",\n color: videoMuted ? \"var(--text-muted)\" : \"#fff\",\n cursor: \"pointer\",\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n transition: \"all 0.2s ease\"\n }}\n title={videoMuted ? \"Unmute video\" : \"Mute video\"}\n >\n {videoMuted ? (\n \n ) : (\n \n )}\n \n
\n

\n {videoMuted\n ? \"Video is muted. Audio plays from separated stems below.\"\n : \"Playing original video audio. Stem audio may overlap.\"}\n

\n
\n\n {/* Stats Bar */}\n {(audioDuration || processingTime || modelSize) && (\n \n {audioDuration !== undefined && (\n
\n \n Duration:\n \n \n {Math.floor(audioDuration / 60)}:{(audioDuration % 60).toFixed(0).padStart(2, \"0\")}\n \n
\n )}\n {processingTime !== undefined && (\n
\n \n Processing:\n \n \n {processingTime.toFixed(1)}s\n \n
\n )}\n {modelSize && (\n
\n \n Model:\n \n \n {modelSize}\n \n
\n )}\n \n )}\n\n {/* Transport Controls */}\n \n \n \n \n\n \n {isPlaying ? (\n \n ) : (\n \n )}\n \n\n
\n \n {formatTime(currentTime)}\n \n\n \n 0 ? (currentTime / duration) * 100 : 0}%`,\n transition: \"width 0.1s\"\n }}\n />\n
\n\n \n {formatTime(duration)}\n \n \n \n\n {/* Audio Tracks */}\n
\n {TRACKS.map((track) => {\n const TrackIcon = track.icon;\n const isMuted = muted[track.id];\n const trackReady = isReady[track.id];\n\n return (\n \n {/* Mute Button */}\n toggleMute(track.id)}\n style={{\n width: \"32px\",\n height: \"32px\",\n borderRadius: \"8px\",\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n background: isMuted ? \"var(--bg-secondary)\" : `${track.color}20`,\n color: isMuted ? \"var(--text-muted)\" : track.color,\n border: \"none\",\n cursor: \"pointer\",\n flexShrink: 0\n }}\n >\n {isMuted ? (\n \n ) : (\n \n )}\n \n\n {/* Track Label */}\n
\n \n \n {track.label}\n \n
\n\n {/* Waveform */}\n { containerRefs.current[track.id] = el; }}\n style={{\n flex: 1,\n borderRadius: \"8px\",\n overflow: \"hidden\",\n background: \"var(--bg-secondary)\",\n minHeight: \"48px\"\n }}\n >\n {!trackReady && (\n
\n \n Loading...\n \n
\n )}\n
\n\n {/* Download */}\n downloadTrack(track.id, track.label)}\n style={{\n width: \"32px\",\n height: \"32px\",\n borderRadius: \"8px\",\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n background: \"var(--bg-secondary)\",\n color: \"var(--text-muted)\",\n border: \"none\",\n cursor: \"pointer\",\n flexShrink: 0\n }}\n >\n \n \n \n );\n })}\n \n\n {/* Download All */}\n
\n TRACKS.forEach((track) => downloadTrack(track.id, track.label))}\n style={{\n width: \"100%\",\n padding: \"14px\",\n borderRadius: \"10px\",\n background: \"linear-gradient(135deg, #6366f1, #8b5cf6)\",\n color: \"white\",\n border: \"none\",\n cursor: \"pointer\",\n fontSize: \"0.9rem\",\n fontWeight: 600,\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n gap: \"8px\",\n boxShadow: \"0 4px 12px rgba(99, 102, 241, 0.3)\"\n }}\n >\n \n Download All Stems\n \n\n {/* Download Video with Audio */}\n
\n setShowVideoDownload(!showVideoDownload)}\n style={{\n width: \"100%\",\n padding: \"14px\",\n borderRadius: showVideoDownload ? \"10px 10px 0 0\" : \"10px\",\n background: showVideoDownload\n ? \"var(--bg-tertiary)\"\n : \"linear-gradient(135deg, #059669, #10b981)\",\n color: \"white\",\n border: \"none\",\n cursor: \"pointer\",\n fontSize: \"0.9rem\",\n fontWeight: 600,\n display: \"flex\",\n alignItems: \"center\",\n justifyContent: \"center\",\n gap: \"8px\",\n boxShadow: showVideoDownload\n ? \"none\"\n : \"0 4px 12px rgba(16, 185, 129, 0.3)\"\n }}\n >\n \n Download Video with Audio\n ▼\n \n\n {/* Inline Options */}\n {showVideoDownload && (\n
\n downloadVideoWithAudio(\"original\")}\n style={{\n width: \"100%\",\n padding: \"12px 16px\",\n background: \"transparent\",\n color: \"var(--text-primary)\",\n border: \"none\",\n borderBottom: \"1px solid var(--glass-border)\",\n cursor: \"pointer\",\n fontSize: \"0.85rem\",\n textAlign: \"left\",\n display: \"flex\",\n alignItems: \"center\",\n gap: \"10px\"\n }}\n >\n \n Original Audio\n \n downloadVideoWithAudio(\"ghost\")}\n style={{\n width: \"100%\",\n padding: \"12px 16px\",\n background: \"transparent\",\n color: \"var(--text-primary)\",\n border: \"none\",\n borderBottom: \"1px solid var(--glass-border)\",\n cursor: \"pointer\",\n fontSize: \"0.85rem\",\n textAlign: \"left\",\n display: \"flex\",\n alignItems: \"center\",\n gap: \"10px\"\n }}\n >\n \n Isolated Sound Only\n \n downloadVideoWithAudio(\"clean\")}\n style={{\n width: \"100%\",\n padding: \"12px 16px\",\n background: \"transparent\",\n color: \"var(--text-primary)\",\n border: \"none\",\n cursor: \"pointer\",\n fontSize: \"0.85rem\",\n textAlign: \"left\",\n display: \"flex\",\n alignItems: \"center\",\n gap: \"10px\"\n }}\n >\n \n Without Isolated Sound\n \n
\n )}\n
\n
\n \n );\n}\n" }, { "path": "frontend/src/components/WaveformEditor.tsx", "content": "\"use client\";\n\nimport { useEffect, useRef, useState } from \"react\";\nimport WaveSurfer from \"wavesurfer.js\";\nimport RegionsPlugin from \"wavesurfer.js/dist/plugins/regions.js\";\nimport { Play, Pause, RotateCcw, Scissors, X } from \"lucide-react\";\n\ninterface WaveformEditorProps {\n audioUrl: string;\n onRegionSelect: (region: { start: number; end: number } | null) => void;\n selectedRegion: { start: number; end: number } | null;\n}\n\nexport default function WaveformEditor({\n audioUrl,\n onRegionSelect,\n selectedRegion\n}: WaveformEditorProps) {\n const containerRef = useRef(null);\n const wavesurferRef = useRef(null);\n const regionsRef = useRef(null);\n\n const [isPlaying, setIsPlaying] = useState(false);\n const [currentTime, setCurrentTime] = useState(0);\n const [duration, setDuration] = useState(0);\n const [isLoaded, setIsLoaded] = useState(false);\n\n useEffect(() => {\n if (!containerRef.current) return;\n\n let isMounted = true;\n\n // Create regions plugin\n const regions = RegionsPlugin.create();\n regionsRef.current = regions;\n\n // Create wavesurfer instance\n const wavesurfer = WaveSurfer.create({\n container: containerRef.current,\n waveColor: \"rgba(139, 92, 246, 0.5)\",\n progressColor: \"#8B5CF6\",\n cursorColor: \"#F472B6\",\n cursorWidth: 2,\n barWidth: 3,\n barGap: 2,\n barRadius: 3,\n height: 128,\n normalize: true,\n plugins: [regions],\n });\n\n wavesurferRef.current = wavesurfer;\n\n // Event listeners\n wavesurfer.on(\"ready\", () => {\n if (isMounted) {\n setDuration(wavesurfer.getDuration());\n setIsLoaded(true);\n }\n });\n\n wavesurfer.on(\"audioprocess\", () => {\n if (isMounted) {\n setCurrentTime(wavesurfer.getCurrentTime());\n }\n });\n\n wavesurfer.on(\"seeking\", () => {\n if (isMounted) {\n setCurrentTime(wavesurfer.getCurrentTime());\n }\n });\n\n wavesurfer.on(\"play\", () => isMounted && setIsPlaying(true));\n wavesurfer.on(\"pause\", () => isMounted && setIsPlaying(false));\n\n // Region events\n regions.on(\"region-created\", (region) => {\n // Only allow one region at a time\n regions.getRegions().forEach((r) => {\n if (r.id !== region.id) {\n r.remove();\n }\n });\n\n if (isMounted) {\n onRegionSelect({\n start: region.start,\n end: region.end,\n });\n }\n });\n\n regions.on(\"region-updated\", (region) => {\n if (isMounted) {\n onRegionSelect({\n start: region.start,\n end: region.end,\n });\n }\n });\n\n // Catch loading errors silently (including AbortError when component unmounts)\n wavesurfer.on(\"error\", (error) => {\n // Silently ignore AbortError - this happens when unmounting during load\n if (error?.name === \"AbortError\" || String(error).includes(\"abort\")) {\n return;\n }\n console.warn(\"WaveSurfer error:\", error);\n });\n\n // Load audio\n wavesurfer.load(audioUrl).catch((error) => {\n // Silently ignore AbortError\n if (error?.name === \"AbortError\" || String(error).includes(\"abort\")) {\n return;\n }\n console.warn(\"Failed to load audio:\", error);\n });\n\n return () => {\n isMounted = false;\n // Use setTimeout to ensure any pending operations complete\n // before attempting destruction\n const ws = wavesurfer;\n setTimeout(() => {\n try {\n ws.destroy();\n } catch {\n // Ignore AbortError and other destruction errors\n // This happens when component unmounts during audio loading\n }\n }, 0);\n };\n }, [audioUrl]);\n\n\n const togglePlayPause = () => {\n wavesurferRef.current?.playPause();\n };\n\n const restart = () => {\n wavesurferRef.current?.seekTo(0);\n wavesurferRef.current?.play();\n };\n\n const createRegion = () => {\n if (!regionsRef.current || !wavesurferRef.current) return;\n\n const currentPos = wavesurferRef.current.getCurrentTime();\n const dur = wavesurferRef.current.getDuration();\n\n // Create a region from current position to +5 seconds\n const start = currentPos;\n const end = Math.min(currentPos + 5, dur);\n\n regionsRef.current.addRegion({\n start,\n end,\n color: \"rgba(244, 114, 182, 0.3)\",\n drag: true,\n resize: true,\n });\n };\n\n const clearRegion = () => {\n regionsRef.current?.getRegions().forEach((r) => r.remove());\n onRegionSelect(null);\n };\n\n const formatTime = (seconds: number) => {\n const mins = Math.floor(seconds / 60);\n const secs = Math.floor(seconds % 60);\n return `${mins}:${secs.toString().padStart(2, \"0\")}`;\n };\n\n return (\n
\n {/* Header */}\n
\n

\n Waveform Editor\n

\n
\n {selectedRegion && (\n \n Selected: {formatTime(selectedRegion.start)} - {formatTime(selectedRegion.end)}\n \n )}\n
\n
\n\n {/* Waveform */}\n \n\n {/* Loading skeleton */}\n {!isLoaded && (\n
\n )}\n\n {/* Controls */}\n
\n {/* Playback Controls */}\n
\n \n {isPlaying ? (\n \n ) : (\n \n )}\n \n\n \n \n \n\n \n {formatTime(currentTime)} / {formatTime(duration)}\n \n
\n\n {/* Region Controls */}\n
\n \n \n Select Region\n \n\n {selectedRegion && (\n \n \n \n )}\n
\n
\n\n {/* Instructions */}\n

\n 💡 Tip: Select a region to apply Temporal Lock - the AI will focus on that specific time range.\n

\n
\n );\n}\n" }, { "path": "frontend/tsconfig.json", "content": "{\n \"compilerOptions\": {\n \"target\": \"ES2017\",\n \"lib\": [\"dom\", \"dom.iterable\", \"esnext\"],\n \"allowJs\": true,\n \"skipLibCheck\": true,\n \"strict\": true,\n \"noEmit\": true,\n \"esModuleInterop\": true,\n \"module\": \"esnext\",\n \"moduleResolution\": \"bundler\",\n \"resolveJsonModule\": true,\n \"isolatedModules\": true,\n \"jsx\": \"react-jsx\",\n \"incremental\": true,\n \"plugins\": [\n {\n \"name\": \"next\"\n }\n ],\n \"paths\": {\n \"@/*\": [\"./src/*\"]\n }\n },\n \"include\": [\n \"next-env.d.ts\",\n \"**/*.ts\",\n \"**/*.tsx\",\n \".next/types/**/*.ts\",\n \".next/dev/types/**/*.ts\",\n \"**/*.mts\"\n ],\n \"exclude\": [\"node_modules\"]\n}\n" }, { "path": "install.bat", "content": "@echo off\nchcp 65001 >nul\ntitle AudioGhost AI - One-Click Installer\n\necho.\necho ╔══════════════════════════════════════════════════════════════╗\necho ║ AudioGhost AI - One-Click Installer ║\necho ║ v1.0 MVP ║\necho ╚══════════════════════════════════════════════════════════════╝\necho.\n\n:: Get the directory where this script is located\nset \"SCRIPT_DIR=%~dp0\"\ncd /d \"%SCRIPT_DIR%\"\n\n:: Check if Conda is installed\nwhere conda >nul 2>&1\nif %errorlevel% neq 0 (\n echo [ERROR] Conda not found. Please install Anaconda or Miniconda first.\n echo Download from: https://www.anaconda.com/download\n pause\n exit /b 1\n)\n\necho [1/8] Downloading Redis for Windows...\nif not exist \"redis\\redis-server.exe\" (\n echo Downloading from GitHub...\n powershell -Command \"Invoke-WebRequest -Uri 'https://github.com/tporadowski/redis/releases/download/v5.0.14.1/Redis-x64-5.0.14.1.zip' -OutFile 'redis.zip'\"\n echo Extracting...\n powershell -Command \"Expand-Archive -Path 'redis.zip' -DestinationPath 'redis' -Force\"\n del redis.zip\n echo Redis installed to ./redis/\n) else (\n echo Redis already exists, skipping...\n)\n\necho.\necho [2/8] Creating Conda environment 'audioghost' (Python 3.11)...\ncall conda create -n audioghost python=3.11 -y\nif %errorlevel% neq 0 (\n echo [WARN] Environment may already exist, continuing...\n)\n\necho.\necho [3/8] Activating environment...\ncall conda activate audioghost\n\necho.\necho [4/8] Installing PyTorch (CUDA 12.6)...\necho This may take several minutes...\npip install torch==2.9.0+cu126 torchvision==0.24.0+cu126 torchaudio==2.9.0+cu126 --index-url https://download.pytorch.org/whl/cu126 --extra-index-url https://pypi.org/simple\n\necho.\necho [5/8] Installing FFmpeg...\ncall conda install -c conda-forge ffmpeg -y\n\necho.\necho [6/8] Installing SAM Audio...\npip install git+https://github.com/facebookresearch/sam-audio.git\n\necho.\necho [7/8] Installing Backend dependencies...\ncd backend\npip install -r requirements.txt\ncd ..\n\necho.\necho [8/8] Installing Frontend dependencies...\ncd frontend\ncall npm install\ncd ..\n\necho.\necho ╔══════════════════════════════════════════════════════════════╗\necho ║ Installation Complete! ✓ ║\necho ╠══════════════════════════════════════════════════════════════╣\necho ║ ║\necho ║ To start AudioGhost, run: start.bat ║\necho ║ ║\necho ║ No Docker required! Redis is included. ║\necho ║ ║\necho ╚══════════════════════════════════════════════════════════════╝\necho.\npause\n" }, { "path": "sam_audio_lite.py", "content": "\"\"\"\nSAM Audio Lite - Lightweight version for low VRAM usage (4-6GB)\nDisables vision_encoder, rankers, and span_predictor to reduce memory\nWITH CHUNKING - Same logic as Celery worker for memory comparison\n\"\"\"\nimport torch\nimport gc\nimport time\n\n\ndef show_gpu_memory(label: str = \"\"):\n \"\"\"Show complete GPU memory stats (matches nvidia-smi more closely)\"\"\"\n if torch.cuda.is_available():\n allocated = torch.cuda.memory_allocated() / 1024**3\n reserved = torch.cuda.memory_reserved() / 1024**3\n max_allocated = torch.cuda.max_memory_allocated() / 1024**3\n print(f\"[GPU Memory{' - ' + label if label else ''}] \"\n f\"Allocated: {allocated:.2f}GB | Reserved: {reserved:.2f}GB | Peak: {max_allocated:.2f}GB\")\n\n\ndef create_lite_model(model_name: str = \"facebook/sam-audio-base\", token: str = None):\n \"\"\"\n Create a memory-optimized SAM Audio model by removing unused components.\n \n This can reduce VRAM usage from ~11GB to ~4GB by:\n - Replacing vision_encoder with a dummy (saves ~2GB)\n - Disabling visual_ranker (saves ~2GB)\n - Disabling text_ranker (saves ~2GB)\n - Disabling span_predictor (saves ~1-2GB)\n \n Returns:\n model: Optimized SAM Audio model\n processor: SAM Audio processor\n \"\"\"\n from sam_audio import SAMAudio, SAMAudioProcessor\n \n print(f\"Loading {model_name}...\")\n \n # Load model\n if token:\n model = SAMAudio.from_pretrained(model_name, token=token)\n else:\n model = SAMAudio.from_pretrained(model_name)\n \n processor = SAMAudioProcessor.from_pretrained(model_name)\n \n print(\"Optimizing model for low VRAM...\")\n \n # Get vision encoder dim before deleting\n vision_dim = model.vision_encoder.dim if hasattr(model.vision_encoder, 'dim') else 1024\n \n # Delete heavy components\n del model.vision_encoder\n gc.collect()\n print(\" - Removed vision_encoder\")\n \n # Store the dim for _get_video_features\n model._vision_encoder_dim = vision_dim\n \n # Replace _get_video_features to not use vision_encoder\n def _get_video_features_lite(self, video, audio_features):\n B, T, _ = audio_features.shape\n # Always return zeros since we're not using video\n return audio_features.new_zeros(B, self._vision_encoder_dim, T)\n \n # Bind the new method\n import types\n model._get_video_features = types.MethodType(_get_video_features_lite, model)\n \n # Delete rankers\n if hasattr(model, 'visual_ranker') and model.visual_ranker is not None:\n del model.visual_ranker\n model.visual_ranker = None\n gc.collect()\n print(\" - Removed visual_ranker\")\n \n if hasattr(model, 'text_ranker') and model.text_ranker is not None:\n del model.text_ranker\n model.text_ranker = None\n gc.collect()\n print(\" - Removed text_ranker\")\n \n # Delete span predictor\n if hasattr(model, 'span_predictor') and model.span_predictor is not None:\n del model.span_predictor\n model.span_predictor = None\n gc.collect()\n print(\" - Removed span_predictor\")\n \n if hasattr(model, 'span_predictor_transform') and model.span_predictor_transform is not None:\n del model.span_predictor_transform\n model.span_predictor_transform = None\n gc.collect()\n print(\" - Removed span_predictor_transform\")\n \n # Force garbage collection\n gc.collect()\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n \n print(\"Model optimization complete!\")\n \n return model, processor\n\n\nif __name__ == \"__main__\":\n import torchaudio\n from pathlib import Path\n \n # =====================================\n # CONFIGURATION\n # =====================================\n USE_BF16 = True # Set to False to use float32 (better quality, more VRAM)\n # =====================================\n \n # Setup\n device = 'cuda' if torch.cuda.is_available() else 'cpu'\n dtype = torch.bfloat16 if USE_BF16 else torch.float32\n print(f\"Using device: {device}, dtype: {dtype}\")\n print(f\"USE_BF16: {USE_BF16}\")\n \n # Clear GPU memory and reset peak stats\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n torch.cuda.reset_peak_memory_stats()\n gc.collect()\n show_gpu_memory(\"Before loading model\")\n \n # Create lite model\n model, processor = create_lite_model()\n model = model.eval().to(device, dtype)\n \n show_gpu_memory(\"After loading model\")\n \n # Test audio\n test_audio = \"test_chunk.mp3\"\n description = \"singing voice\"\n \n if not Path(test_audio).exists():\n print(f\"\\nNo test file at {test_audio}, creating test tone...\")\n sample_rate = 16000\n duration = 10\n t = torch.linspace(0, duration, int(sample_rate * duration))\n audio = 0.5 * torch.sin(2 * 3.14159 * 200 * t)\n audio = audio.unsqueeze(0)\n test_audio = \"test_audio.wav\"\n torchaudio.save(test_audio, audio, sample_rate)\n \n print(f\"\\nProcessing: {test_audio}\")\n print(f\"Description: '{description}'\")\n \n # Start timing\n start_time = time.time()\n \n # Load audio\n sample_rate = processor.audio_sampling_rate\n audio, orig_sr = torchaudio.load(test_audio)\n if orig_sr != sample_rate:\n resampler = torchaudio.transforms.Resample(orig_sr, sample_rate)\n audio = resampler(audio)\n \n # Convert to mono if stereo\n if audio.shape[0] > 1:\n audio = audio.mean(dim=0, keepdim=True)\n \n # Calculate audio duration\n audio_duration = audio.shape[1] / sample_rate\n print(f\"\\nAudio duration: {audio_duration:.2f}s\")\n \n # ===============================\n # CHUNKING LOGIC (same as worker)\n # ===============================\n CHUNK_DURATION = 25.0 # 25 seconds per chunk\n MAX_CHUNK_SAMPLES = int(sample_rate * CHUNK_DURATION)\n \n if audio.shape[1] > MAX_CHUNK_SAMPLES:\n print(f\"Audio is {audio_duration:.1f}s, using chunking ({CHUNK_DURATION}s chunks)\")\n \n # Split audio into chunks\n audio_tensor = audio.squeeze(0).to(device, dtype)\n chunks = torch.split(audio_tensor, MAX_CHUNK_SAMPLES, dim=-1)\n total_chunks = len(chunks)\n \n out_target = []\n out_residual = []\n \n show_gpu_memory(\"Before chunked separation\")\n \n for i, chunk in enumerate(chunks):\n print(f\"\\nProcessing chunk {i+1}/{total_chunks}...\")\n \n # Skip very short chunks\n if chunk.shape[-1] < sample_rate: # Less than 1 second\n print(f\" Skipping chunk {i+1} (too short: {chunk.shape[-1]/sample_rate:.2f}s)\")\n continue\n \n # Prepare batch for this chunk\n batch = processor(\n audios=[chunk.unsqueeze(0)],\n descriptions=[description]\n ).to(device)\n \n # Run separation\n with torch.inference_mode():\n with torch.cuda.amp.autocast(enabled=(device == \"cuda\")):\n result = model.separate(\n batch,\n predict_spans=False,\n reranking_candidates=1\n )\n \n out_target.append(result.target[0].cpu())\n out_residual.append(result.residual[0].cpu())\n \n show_gpu_memory(f\"After chunk {i+1}\")\n \n # Clean up chunk results\n del batch, result\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n \n # Concatenate all chunks\n target_audio = torch.cat(out_target, dim=-1).clamp(-1, 1).float().unsqueeze(0)\n residual_audio = torch.cat(out_residual, dim=-1).clamp(-1, 1).float().unsqueeze(0)\n \n del out_target, out_residual, chunks, audio_tensor\n \n else:\n print(f\"Audio is {audio_duration:.1f}s, processing as single batch (no chunking needed)\")\n \n # Prepare inputs\n print(\"\\nPreparing batch...\")\n batch = processor(\n audios=[test_audio],\n descriptions=[description],\n ).to(device)\n \n # Run separation\n print(\"Running separation...\")\n show_gpu_memory(\"Before separation\")\n \n with torch.inference_mode(), torch.autocast(device_type=device, dtype=dtype):\n result = model.separate(\n batch, \n predict_spans=False,\n reranking_candidates=1\n )\n \n show_gpu_memory(\"After separation\")\n \n target_audio = result.target[0].float().unsqueeze(0).cpu()\n residual_audio = result.residual[0].float().unsqueeze(0).cpu()\n \n del batch, result\n \n # Calculate processing time\n processing_time = time.time() - start_time\n \n # Save results\n print(\"\\n\" + \"=\"*50)\n print(\"Saving results...\")\n \n torchaudio.save(\"output_target.wav\", target_audio, sample_rate)\n torchaudio.save(\"output_residual.wav\", residual_audio, sample_rate)\n \n # Cleanup\n del target_audio, residual_audio\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n gc.collect()\n show_gpu_memory(\"After cleanup\")\n \n # Summary\n print(\"\\n\" + \"=\"*60)\n print(\"SUMMARY\")\n print(\"=\"*60)\n print(f\"Audio duration: {audio_duration:.2f}s ({audio_duration/60:.2f} min)\")\n print(f\"Processing time: {processing_time:.2f}s\")\n print(f\"Speed: {audio_duration/processing_time:.2f}x realtime\")\n print(f\"Chunking: {'Yes (' + str(len(chunks) if 'chunks' in dir() else '?') + ' chunks)' if audio_duration > CHUNK_DURATION else 'No (single batch)'}\")\n if torch.cuda.is_available():\n peak_gb = torch.cuda.max_memory_allocated() / 1024**3\n reserved_gb = torch.cuda.max_memory_reserved() / 1024**3\n print(f\"Peak GPU Memory: {peak_gb:.2f}GB (reserved: {reserved_gb:.2f}GB)\")\n print(\"=\"*60)\n print(\"Output files:\")\n print(\" - output_target.wav: Extracted audio\")\n print(\" - output_residual.wav: Residual audio\")\n" }, { "path": "start.bat", "content": "@echo off\nchcp 65001 >nul\ntitle AudioGhost AI - Launcher\n\necho.\necho ╔══════════════════════════════════════════════════════════════╗\necho ║ AudioGhost AI - Launcher ║\necho ║ v1.0 MVP ║\necho ╚══════════════════════════════════════════════════════════════╝\necho.\n\n:: Get the directory where this script is located\nset \"SCRIPT_DIR=%~dp0\"\ncd /d \"%SCRIPT_DIR%\"\n\n:: Check if Docker Redis is already running on port 6379\necho [1/4] Checking Redis...\nnetstat -an | findstr \":6379.*LISTENING\" >nul 2>&1\nif %ERRORLEVEL%==0 (\n echo Docker Redis detected - using existing instance\n goto :redis_ready\n)\n\n:: Docker Redis not running, try offline Redis\nif exist \"redis\\redis-server.exe\" (\n echo Starting offline Redis...\n start \"AudioGhost Redis\" /min cmd /c \"cd /d %SCRIPT_DIR%redis && redis-server.exe\"\n timeout /t 2 /nobreak >nul\n goto :redis_ready\n)\n\n:: Neither available\necho [ERROR] Redis not available. Either:\necho - Start Docker: docker-compose up -d\necho - Or run install.bat to download offline Redis\npause\nexit /b 1\n\n:redis_ready\n\necho [2/4] Starting Backend API...\nstart \"AudioGhost Backend\" cmd /k \"cd /d %SCRIPT_DIR% && conda activate audioghost && cd backend && uvicorn main:app --reload --port 8000\"\n\necho [3/4] Starting Celery Worker...\ntimeout /t 2 /nobreak >nul\nstart \"AudioGhost Worker\" cmd /k \"cd /d %SCRIPT_DIR% && conda activate audioghost && cd backend && celery -A workers.celery_app worker --loglevel=info --pool=solo\"\n\necho [4/4] Starting Frontend...\ntimeout /t 2 /nobreak >nul\nstart \"AudioGhost Frontend\" cmd /k \"cd /d %SCRIPT_DIR% && cd frontend && npm run dev\"\n\necho.\necho ╔══════════════════════════════════════════════════════════════╗\necho ║ All Services Started! ✓ ║\necho ╠══════════════════════════════════════════════════════════════╣\necho ║ ║\necho ║ Frontend: http://localhost:3000 ║\necho ║ Backend: http://localhost:8000 ║\necho ║ API Docs: http://localhost:8000/docs ║\necho ║ ║\necho ║ Four windows opened: ║\necho ║ - AudioGhost Redis (minimized) ║\necho ║ - AudioGhost Backend (FastAPI) ║\necho ║ - AudioGhost Worker (Celery) ║\necho ║ - AudioGhost Frontend (Next.js) ║\necho ║ ║\necho ║ Run stop.bat or close all windows to stop services. ║\necho ╚══════════════════════════════════════════════════════════════╝\necho.\necho Opening browser in 3 seconds...\ntimeout /t 3 /nobreak >nul\n\n:: Open browser automatically\nstart http://localhost:3000\n" }, { "path": "stop.bat", "content": "@echo off\nchcp 65001 >nul\ntitle AudioGhost AI - Stop All Services\n\necho.\necho ╔══════════════════════════════════════════════════════════════╗\necho ║ AudioGhost AI - Shutdown ║\necho ╚══════════════════════════════════════════════════════════════╝\necho.\n\necho Stopping all AudioGhost services...\necho.\n\n:: Kill Node.js (Frontend)\necho [1/4] Stopping Frontend...\ntaskkill /FI \"WINDOWTITLE eq AudioGhost Frontend*\" /F >nul 2>&1\ntaskkill /IM \"node.exe\" /F >nul 2>&1\n\n:: Kill Celery (Worker)\necho [2/4] Stopping Celery Worker...\ntaskkill /FI \"WINDOWTITLE eq AudioGhost Worker*\" /F >nul 2>&1\n\n:: Kill Uvicorn (Backend)\necho [3/4] Stopping Backend API...\ntaskkill /FI \"WINDOWTITLE eq AudioGhost Backend*\" /F >nul 2>&1\n\n:: Stop Redis\necho [4/4] Stopping Redis...\ndocker-compose down >nul 2>&1\n\necho.\necho ╔══════════════════════════════════════════════════════════════╗\necho ║ All Services Stopped! ✓ ║\necho ╚══════════════════════════════════════════════════════════════╝\necho.\npause\n" }, { "path": "test_video_only.py", "content": "\"\"\"\nTest Video Only - Simple video-based audio separation using SAM Audio\nUses small model with bfloat16 for lower memory usage\n\"\"\"\nimport torch\nimport torchaudio\nimport gc\n\ndef main():\n # Setup\n device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n dtype = torch.bfloat16 if device.type == \"cuda\" else torch.float32\n print(f\"Device: {device}, dtype: {dtype}\")\n \n # Clear GPU memory\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n gc.collect()\n print(f\"GPU Memory before loading: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n \n # Load model (small for lower memory)\n from sam_audio import SAMAudio, SAMAudioProcessor\n \n model_name = \"facebook/sam-audio-base\"\n print(f\"Loading {model_name}...\")\n \n model = SAMAudio.from_pretrained(model_name).to(device, dtype).eval()\n processor = SAMAudioProcessor.from_pretrained(model_name)\n \n if torch.cuda.is_available():\n print(f\"GPU Memory after loading: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n \n # Video file\n video_file = \"office.mp4\"\n description = \"walking sound\"\n \n print(f\"\\nProcessing video: {video_file}\")\n print(f\"Description: '{description}'\")\n \n # Process\n inputs = processor(audios=[video_file], descriptions=[description]).to(device)\n \n print(\"Running separation...\")\n if torch.cuda.is_available():\n print(f\"GPU Memory before separation: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n \n with torch.inference_mode(), torch.autocast(device_type=device.type, dtype=dtype):\n result = model.separate(inputs)\n \n if torch.cuda.is_available():\n print(f\"GPU Memory after separation: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n \n # Save results\n sample_rate = processor.audio_sampling_rate\n \n target_audio = result.target[0].float().unsqueeze(0).cpu()\n residual_audio = result.residual[0].float().unsqueeze(0).cpu()\n \n torchaudio.save(\"video_target.wav\", target_audio, sample_rate)\n torchaudio.save(\"video_residual.wav\", residual_audio, sample_rate)\n \n print(\"\\nDone!\")\n print(\"- video_target.wav: Extracted audio (target)\")\n print(\"- video_residual.wav: Remaining audio (residual)\")\n \n # Cleanup\n del model, processor, inputs, result\n if torch.cuda.is_available():\n torch.cuda.empty_cache()\n gc.collect()\n print(f\"GPU Memory after cleanup: {torch.cuda.memory_allocated() / 1024**3:.2f} GB\")\n\n\nif __name__ == \"__main__\":\n main()\n" } ]