[
{
"path": ".gitignore",
"content": "# Python-generated files\n__pycache__/\n*.py[oc]\nbuild/\ndist/\nwheels/\n*.egg-info\n\n# Virtual environments\n.venv\n\n# Keys and secrets\n.env\n\n# Data files\ndata/\n\n# Frontend\nfrontend/node_modules/\nfrontend/dist/\nfrontend/.vite/"
},
{
"path": ".python-version",
"content": "3.10\n"
},
{
"path": "CLAUDE.md",
"content": "# CLAUDE.md - Technical Notes for LLM Council\n\nThis file contains technical details, architectural decisions, and important implementation notes for future development sessions.\n\n## Project Overview\n\nLLM Council is a 3-stage deliberation system where multiple LLMs collaboratively answer user questions. The key innovation is anonymized peer review in Stage 2, preventing models from playing favorites.\n\n## Architecture\n\n### Backend Structure (`backend/`)\n\n**`config.py`**\n- Contains `COUNCIL_MODELS` (list of OpenRouter model identifiers)\n- Contains `CHAIRMAN_MODEL` (model that synthesizes final answer)\n- Uses environment variable `OPENROUTER_API_KEY` from `.env`\n- Backend runs on **port 8001** (NOT 8000 - user had another app on 8000)\n\n**`openrouter.py`**\n- `query_model()`: Single async model query\n- `query_models_parallel()`: Parallel queries using `asyncio.gather()`\n- Returns dict with 'content' and optional 'reasoning_details'\n- Graceful degradation: returns None on failure, continues with successful responses\n\n**`council.py`** - The Core Logic\n- `stage1_collect_responses()`: Parallel queries to all council models\n- `stage2_collect_rankings()`:\n - Anonymizes responses as \"Response A, B, C, etc.\"\n - Creates `label_to_model` mapping for de-anonymization\n - Prompts models to evaluate and rank (with strict format requirements)\n - Returns tuple: (rankings_list, label_to_model_dict)\n - Each ranking includes both raw text and `parsed_ranking` list\n- `stage3_synthesize_final()`: Chairman synthesizes from all responses + rankings\n- `parse_ranking_from_text()`: Extracts \"FINAL RANKING:\" section, handles both numbered lists and plain format\n- `calculate_aggregate_rankings()`: Computes average rank position across all peer evaluations\n\n**`storage.py`**\n- JSON-based conversation storage in `data/conversations/`\n- Each conversation: `{id, created_at, messages[]}`\n- Assistant messages contain: `{role, stage1, stage2, stage3}`\n- Note: metadata (label_to_model, aggregate_rankings) is NOT persisted to storage, only returned via API\n\n**`main.py`**\n- FastAPI app with CORS enabled for localhost:5173 and localhost:3000\n- POST `/api/conversations/{id}/message` returns metadata in addition to stages\n- Metadata includes: label_to_model mapping and aggregate_rankings\n\n### Frontend Structure (`frontend/src/`)\n\n**`App.jsx`**\n- Main orchestration: manages conversations list and current conversation\n- Handles message sending and metadata storage\n- Important: metadata is stored in the UI state for display but not persisted to backend JSON\n\n**`components/ChatInterface.jsx`**\n- Multiline textarea (3 rows, resizable)\n- Enter to send, Shift+Enter for new line\n- User messages wrapped in markdown-content class for padding\n\n**`components/Stage1.jsx`**\n- Tab view of individual model responses\n- ReactMarkdown rendering with markdown-content wrapper\n\n**`components/Stage2.jsx`**\n- **Critical Feature**: Tab view showing RAW evaluation text from each model\n- De-anonymization happens CLIENT-SIDE for display (models receive anonymous labels)\n- Shows \"Extracted Ranking\" below each evaluation so users can validate parsing\n- Aggregate rankings shown with average position and vote count\n- Explanatory text clarifies that boldface model names are for readability only\n\n**`components/Stage3.jsx`**\n- Final synthesized answer from chairman\n- Green-tinted background (#f0fff0) to highlight conclusion\n\n**Styling (`*.css`)**\n- Light mode theme (not dark mode)\n- Primary color: #4a90e2 (blue)\n- Global markdown styling in `index.css` with `.markdown-content` class\n- 12px padding on all markdown content to prevent cluttered appearance\n\n## Key Design Decisions\n\n### Stage 2 Prompt Format\nThe Stage 2 prompt is very specific to ensure parseable output:\n```\n1. Evaluate each response individually first\n2. Provide \"FINAL RANKING:\" header\n3. Numbered list format: \"1. Response C\", \"2. Response A\", etc.\n4. No additional text after ranking section\n```\n\nThis strict format allows reliable parsing while still getting thoughtful evaluations.\n\n### De-anonymization Strategy\n- Models receive: \"Response A\", \"Response B\", etc.\n- Backend creates mapping: `{\"Response A\": \"openai/gpt-5.1\", ...}`\n- Frontend displays model names in **bold** for readability\n- Users see explanation that original evaluation used anonymous labels\n- This prevents bias while maintaining transparency\n\n### Error Handling Philosophy\n- Continue with successful responses if some models fail (graceful degradation)\n- Never fail the entire request due to single model failure\n- Log errors but don't expose to user unless all models fail\n\n### UI/UX Transparency\n- All raw outputs are inspectable via tabs\n- Parsed rankings shown below raw text for validation\n- Users can verify system's interpretation of model outputs\n- This builds trust and allows debugging of edge cases\n\n## Important Implementation Details\n\n### Relative Imports\nAll backend modules use relative imports (e.g., `from .config import ...`) not absolute imports. This is critical for Python's module system to work correctly when running as `python -m backend.main`.\n\n### Port Configuration\n- Backend: 8001 (changed from 8000 to avoid conflict)\n- Frontend: 5173 (Vite default)\n- Update both `backend/main.py` and `frontend/src/api.js` if changing\n\n### Markdown Rendering\nAll ReactMarkdown components must be wrapped in `
` for proper spacing. This class is defined globally in `index.css`.\n\n### Model Configuration\nModels are hardcoded in `backend/config.py`. Chairman can be same or different from council members. The current default is Gemini as chairman per user preference.\n\n## Common Gotchas\n\n1. **Module Import Errors**: Always run backend as `python -m backend.main` from project root, not from backend directory\n2. **CORS Issues**: Frontend must match allowed origins in `main.py` CORS middleware\n3. **Ranking Parse Failures**: If models don't follow format, fallback regex extracts any \"Response X\" patterns in order\n4. **Missing Metadata**: Metadata is ephemeral (not persisted), only available in API responses\n\n## Future Enhancement Ideas\n\n- Configurable council/chairman via UI instead of config file\n- Streaming responses instead of batch loading\n- Export conversations to markdown/PDF\n- Model performance analytics over time\n- Custom ranking criteria (not just accuracy/insight)\n- Support for reasoning models (o1, etc.) with special handling\n\n## Testing Notes\n\nUse `test_openrouter.py` to verify API connectivity and test different model identifiers before adding to council. The script tests both streaming and non-streaming modes.\n\n## Data Flow Summary\n\n```\nUser Query\n ↓\nStage 1: Parallel queries → [individual responses]\n ↓\nStage 2: Anonymize → Parallel ranking queries → [evaluations + parsed rankings]\n ↓\nAggregate Rankings Calculation → [sorted by avg position]\n ↓\nStage 3: Chairman synthesis with full context\n ↓\nReturn: {stage1, stage2, stage3, metadata}\n ↓\nFrontend: Display with tabs + validation UI\n```\n\nThe entire flow is async/parallel where possible to minimize latency.\n"
},
{
"path": "README.md",
"content": "# LLM Council\n\n\n\nThe idea of this repo is that instead of asking a question to your favorite LLM provider (e.g. OpenAI GPT 5.1, Google Gemini 3.0 Pro, Anthropic Claude Sonnet 4.5, xAI Grok 4, eg.c), you can group them into your \"LLM Council\". This repo is a simple, local web app that essentially looks like ChatGPT except it uses OpenRouter to send your query to multiple LLMs, it then asks them to review and rank each other's work, and finally a Chairman LLM produces the final response.\n\nIn a bit more detail, here is what happens when you submit a query:\n\n1. **Stage 1: First opinions**. The user query is given to all LLMs individually, and the responses are collected. The individual responses are shown in a \"tab view\", so that the user can inspect them all one by one.\n2. **Stage 2: Review**. Each individual LLM is given the responses of the other LLMs. Under the hood, the LLM identities are anonymized so that the LLM can't play favorites when judging their outputs. The LLM is asked to rank them in accuracy and insight.\n3. **Stage 3: Final response**. The designated Chairman of the LLM Council takes all of the model's responses and compiles them into a single final answer that is presented to the user.\n\n## Vibe Code Alert\n\nThis project was 99% vibe coded as a fun Saturday hack because I wanted to explore and evaluate a number of LLMs side by side in the process of [reading books together with LLMs](https://x.com/karpathy/status/1990577951671509438). It's nice and useful to see multiple responses side by side, and also the cross-opinions of all LLMs on each other's outputs. I'm not going to support it in any way, it's provided here as is for other people's inspiration and I don't intend to improve it. Code is ephemeral now and libraries are over, ask your LLM to change it in whatever way you like.\n\n## Setup\n\n### 1. Install Dependencies\n\nThe project uses [uv](https://docs.astral.sh/uv/) for project management.\n\n**Backend:**\n```bash\nuv sync\n```\n\n**Frontend:**\n```bash\ncd frontend\nnpm install\ncd ..\n```\n\n### 2. Configure API Key\n\nCreate a `.env` file in the project root:\n\n```bash\nOPENROUTER_API_KEY=sk-or-v1-...\n```\n\nGet your API key at [openrouter.ai](https://openrouter.ai/). Make sure to purchase the credits you need, or sign up for automatic top up.\n\n### 3. Configure Models (Optional)\n\nEdit `backend/config.py` to customize the council:\n\n```python\nCOUNCIL_MODELS = [\n \"openai/gpt-5.1\",\n \"google/gemini-3-pro-preview\",\n \"anthropic/claude-sonnet-4.5\",\n \"x-ai/grok-4\",\n]\n\nCHAIRMAN_MODEL = \"google/gemini-3-pro-preview\"\n```\n\n## Running the Application\n\n**Option 1: Use the start script**\n```bash\n./start.sh\n```\n\n**Option 2: Run manually**\n\nTerminal 1 (Backend):\n```bash\nuv run python -m backend.main\n```\n\nTerminal 2 (Frontend):\n```bash\ncd frontend\nnpm run dev\n```\n\nThen open http://localhost:5173 in your browser.\n\n## Tech Stack\n\n- **Backend:** FastAPI (Python 3.10+), async httpx, OpenRouter API\n- **Frontend:** React + Vite, react-markdown for rendering\n- **Storage:** JSON files in `data/conversations/`\n- **Package Management:** uv for Python, npm for JavaScript\n"
},
{
"path": "backend/__init__.py",
"content": "\"\"\"LLM Council backend package.\"\"\"\n"
},
{
"path": "backend/config.py",
"content": "\"\"\"Configuration for the LLM Council.\"\"\"\n\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\n\n# OpenRouter API key\nOPENROUTER_API_KEY = os.getenv(\"OPENROUTER_API_KEY\")\n\n# Council members - list of OpenRouter model identifiers\nCOUNCIL_MODELS = [\n \"openai/gpt-5.1\",\n \"google/gemini-3-pro-preview\",\n \"anthropic/claude-sonnet-4.5\",\n \"x-ai/grok-4\",\n]\n\n# Chairman model - synthesizes final response\nCHAIRMAN_MODEL = \"google/gemini-3-pro-preview\"\n\n# OpenRouter API endpoint\nOPENROUTER_API_URL = \"https://openrouter.ai/api/v1/chat/completions\"\n\n# Data directory for conversation storage\nDATA_DIR = \"data/conversations\"\n"
},
{
"path": "backend/council.py",
"content": "\"\"\"3-stage LLM Council orchestration.\"\"\"\n\nfrom typing import List, Dict, Any, Tuple\nfrom .openrouter import query_models_parallel, query_model\nfrom .config import COUNCIL_MODELS, CHAIRMAN_MODEL\n\n\nasync def stage1_collect_responses(user_query: str) -> List[Dict[str, Any]]:\n \"\"\"\n Stage 1: Collect individual responses from all council models.\n\n Args:\n user_query: The user's question\n\n Returns:\n List of dicts with 'model' and 'response' keys\n \"\"\"\n messages = [{\"role\": \"user\", \"content\": user_query}]\n\n # Query all models in parallel\n responses = await query_models_parallel(COUNCIL_MODELS, messages)\n\n # Format results\n stage1_results = []\n for model, response in responses.items():\n if response is not None: # Only include successful responses\n stage1_results.append({\n \"model\": model,\n \"response\": response.get('content', '')\n })\n\n return stage1_results\n\n\nasync def stage2_collect_rankings(\n user_query: str,\n stage1_results: List[Dict[str, Any]]\n) -> Tuple[List[Dict[str, Any]], Dict[str, str]]:\n \"\"\"\n Stage 2: Each model ranks the anonymized responses.\n\n Args:\n user_query: The original user query\n stage1_results: Results from Stage 1\n\n Returns:\n Tuple of (rankings list, label_to_model mapping)\n \"\"\"\n # Create anonymized labels for responses (Response A, Response B, etc.)\n labels = [chr(65 + i) for i in range(len(stage1_results))] # A, B, C, ...\n\n # Create mapping from label to model name\n label_to_model = {\n f\"Response {label}\": result['model']\n for label, result in zip(labels, stage1_results)\n }\n\n # Build the ranking prompt\n responses_text = \"\\n\\n\".join([\n f\"Response {label}:\\n{result['response']}\"\n for label, result in zip(labels, stage1_results)\n ])\n\n ranking_prompt = f\"\"\"You are evaluating different responses to the following question:\n\nQuestion: {user_query}\n\nHere are the responses from different models (anonymized):\n\n{responses_text}\n\nYour task:\n1. First, evaluate each response individually. For each response, explain what it does well and what it does poorly.\n2. Then, at the very end of your response, provide a final ranking.\n\nIMPORTANT: Your final ranking MUST be formatted EXACTLY as follows:\n- Start with the line \"FINAL RANKING:\" (all caps, with colon)\n- Then list the responses from best to worst as a numbered list\n- Each line should be: number, period, space, then ONLY the response label (e.g., \"1. Response A\")\n- Do not add any other text or explanations in the ranking section\n\nExample of the correct format for your ENTIRE response:\n\nResponse A provides good detail on X but misses Y...\nResponse B is accurate but lacks depth on Z...\nResponse C offers the most comprehensive answer...\n\nFINAL RANKING:\n1. Response C\n2. Response A\n3. Response B\n\nNow provide your evaluation and ranking:\"\"\"\n\n messages = [{\"role\": \"user\", \"content\": ranking_prompt}]\n\n # Get rankings from all council models in parallel\n responses = await query_models_parallel(COUNCIL_MODELS, messages)\n\n # Format results\n stage2_results = []\n for model, response in responses.items():\n if response is not None:\n full_text = response.get('content', '')\n parsed = parse_ranking_from_text(full_text)\n stage2_results.append({\n \"model\": model,\n \"ranking\": full_text,\n \"parsed_ranking\": parsed\n })\n\n return stage2_results, label_to_model\n\n\nasync def stage3_synthesize_final(\n user_query: str,\n stage1_results: List[Dict[str, Any]],\n stage2_results: List[Dict[str, Any]]\n) -> Dict[str, Any]:\n \"\"\"\n Stage 3: Chairman synthesizes final response.\n\n Args:\n user_query: The original user query\n stage1_results: Individual model responses from Stage 1\n stage2_results: Rankings from Stage 2\n\n Returns:\n Dict with 'model' and 'response' keys\n \"\"\"\n # Build comprehensive context for chairman\n stage1_text = \"\\n\\n\".join([\n f\"Model: {result['model']}\\nResponse: {result['response']}\"\n for result in stage1_results\n ])\n\n stage2_text = \"\\n\\n\".join([\n f\"Model: {result['model']}\\nRanking: {result['ranking']}\"\n for result in stage2_results\n ])\n\n chairman_prompt = f\"\"\"You are the Chairman of an LLM Council. Multiple AI models have provided responses to a user's question, and then ranked each other's responses.\n\nOriginal Question: {user_query}\n\nSTAGE 1 - Individual Responses:\n{stage1_text}\n\nSTAGE 2 - Peer Rankings:\n{stage2_text}\n\nYour task as Chairman is to synthesize all of this information into a single, comprehensive, accurate answer to the user's original question. Consider:\n- The individual responses and their insights\n- The peer rankings and what they reveal about response quality\n- Any patterns of agreement or disagreement\n\nProvide a clear, well-reasoned final answer that represents the council's collective wisdom:\"\"\"\n\n messages = [{\"role\": \"user\", \"content\": chairman_prompt}]\n\n # Query the chairman model\n response = await query_model(CHAIRMAN_MODEL, messages)\n\n if response is None:\n # Fallback if chairman fails\n return {\n \"model\": CHAIRMAN_MODEL,\n \"response\": \"Error: Unable to generate final synthesis.\"\n }\n\n return {\n \"model\": CHAIRMAN_MODEL,\n \"response\": response.get('content', '')\n }\n\n\ndef parse_ranking_from_text(ranking_text: str) -> List[str]:\n \"\"\"\n Parse the FINAL RANKING section from the model's response.\n\n Args:\n ranking_text: The full text response from the model\n\n Returns:\n List of response labels in ranked order\n \"\"\"\n import re\n\n # Look for \"FINAL RANKING:\" section\n if \"FINAL RANKING:\" in ranking_text:\n # Extract everything after \"FINAL RANKING:\"\n parts = ranking_text.split(\"FINAL RANKING:\")\n if len(parts) >= 2:\n ranking_section = parts[1]\n # Try to extract numbered list format (e.g., \"1. Response A\")\n # This pattern looks for: number, period, optional space, \"Response X\"\n numbered_matches = re.findall(r'\\d+\\.\\s*Response [A-Z]', ranking_section)\n if numbered_matches:\n # Extract just the \"Response X\" part\n return [re.search(r'Response [A-Z]', m).group() for m in numbered_matches]\n\n # Fallback: Extract all \"Response X\" patterns in order\n matches = re.findall(r'Response [A-Z]', ranking_section)\n return matches\n\n # Fallback: try to find any \"Response X\" patterns in order\n matches = re.findall(r'Response [A-Z]', ranking_text)\n return matches\n\n\ndef calculate_aggregate_rankings(\n stage2_results: List[Dict[str, Any]],\n label_to_model: Dict[str, str]\n) -> List[Dict[str, Any]]:\n \"\"\"\n Calculate aggregate rankings across all models.\n\n Args:\n stage2_results: Rankings from each model\n label_to_model: Mapping from anonymous labels to model names\n\n Returns:\n List of dicts with model name and average rank, sorted best to worst\n \"\"\"\n from collections import defaultdict\n\n # Track positions for each model\n model_positions = defaultdict(list)\n\n for ranking in stage2_results:\n ranking_text = ranking['ranking']\n\n # Parse the ranking from the structured format\n parsed_ranking = parse_ranking_from_text(ranking_text)\n\n for position, label in enumerate(parsed_ranking, start=1):\n if label in label_to_model:\n model_name = label_to_model[label]\n model_positions[model_name].append(position)\n\n # Calculate average position for each model\n aggregate = []\n for model, positions in model_positions.items():\n if positions:\n avg_rank = sum(positions) / len(positions)\n aggregate.append({\n \"model\": model,\n \"average_rank\": round(avg_rank, 2),\n \"rankings_count\": len(positions)\n })\n\n # Sort by average rank (lower is better)\n aggregate.sort(key=lambda x: x['average_rank'])\n\n return aggregate\n\n\nasync def generate_conversation_title(user_query: str) -> str:\n \"\"\"\n Generate a short title for a conversation based on the first user message.\n\n Args:\n user_query: The first user message\n\n Returns:\n A short title (3-5 words)\n \"\"\"\n title_prompt = f\"\"\"Generate a very short title (3-5 words maximum) that summarizes the following question.\nThe title should be concise and descriptive. Do not use quotes or punctuation in the title.\n\nQuestion: {user_query}\n\nTitle:\"\"\"\n\n messages = [{\"role\": \"user\", \"content\": title_prompt}]\n\n # Use gemini-2.5-flash for title generation (fast and cheap)\n response = await query_model(\"google/gemini-2.5-flash\", messages, timeout=30.0)\n\n if response is None:\n # Fallback to a generic title\n return \"New Conversation\"\n\n title = response.get('content', 'New Conversation').strip()\n\n # Clean up the title - remove quotes, limit length\n title = title.strip('\"\\'')\n\n # Truncate if too long\n if len(title) > 50:\n title = title[:47] + \"...\"\n\n return title\n\n\nasync def run_full_council(user_query: str) -> Tuple[List, List, Dict, Dict]:\n \"\"\"\n Run the complete 3-stage council process.\n\n Args:\n user_query: The user's question\n\n Returns:\n Tuple of (stage1_results, stage2_results, stage3_result, metadata)\n \"\"\"\n # Stage 1: Collect individual responses\n stage1_results = await stage1_collect_responses(user_query)\n\n # If no models responded successfully, return error\n if not stage1_results:\n return [], [], {\n \"model\": \"error\",\n \"response\": \"All models failed to respond. Please try again.\"\n }, {}\n\n # Stage 2: Collect rankings\n stage2_results, label_to_model = await stage2_collect_rankings(user_query, stage1_results)\n\n # Calculate aggregate rankings\n aggregate_rankings = calculate_aggregate_rankings(stage2_results, label_to_model)\n\n # Stage 3: Synthesize final answer\n stage3_result = await stage3_synthesize_final(\n user_query,\n stage1_results,\n stage2_results\n )\n\n # Prepare metadata\n metadata = {\n \"label_to_model\": label_to_model,\n \"aggregate_rankings\": aggregate_rankings\n }\n\n return stage1_results, stage2_results, stage3_result, metadata\n"
},
{
"path": "backend/main.py",
"content": "\"\"\"FastAPI backend for LLM Council.\"\"\"\n\nfrom fastapi import FastAPI, HTTPException\nfrom fastapi.middleware.cors import CORSMiddleware\nfrom fastapi.responses import StreamingResponse\nfrom pydantic import BaseModel\nfrom typing import List, Dict, Any\nimport uuid\nimport json\nimport asyncio\n\nfrom . import storage\nfrom .council import run_full_council, generate_conversation_title, stage1_collect_responses, stage2_collect_rankings, stage3_synthesize_final, calculate_aggregate_rankings\n\napp = FastAPI(title=\"LLM Council API\")\n\n# Enable CORS for local development\napp.add_middleware(\n CORSMiddleware,\n allow_origins=[\"http://localhost:5173\", \"http://localhost:3000\"],\n allow_credentials=True,\n allow_methods=[\"*\"],\n allow_headers=[\"*\"],\n)\n\n\nclass CreateConversationRequest(BaseModel):\n \"\"\"Request to create a new conversation.\"\"\"\n pass\n\n\nclass SendMessageRequest(BaseModel):\n \"\"\"Request to send a message in a conversation.\"\"\"\n content: str\n\n\nclass ConversationMetadata(BaseModel):\n \"\"\"Conversation metadata for list view.\"\"\"\n id: str\n created_at: str\n title: str\n message_count: int\n\n\nclass Conversation(BaseModel):\n \"\"\"Full conversation with all messages.\"\"\"\n id: str\n created_at: str\n title: str\n messages: List[Dict[str, Any]]\n\n\n@app.get(\"/\")\nasync def root():\n \"\"\"Health check endpoint.\"\"\"\n return {\"status\": \"ok\", \"service\": \"LLM Council API\"}\n\n\n@app.get(\"/api/conversations\", response_model=List[ConversationMetadata])\nasync def list_conversations():\n \"\"\"List all conversations (metadata only).\"\"\"\n return storage.list_conversations()\n\n\n@app.post(\"/api/conversations\", response_model=Conversation)\nasync def create_conversation(request: CreateConversationRequest):\n \"\"\"Create a new conversation.\"\"\"\n conversation_id = str(uuid.uuid4())\n conversation = storage.create_conversation(conversation_id)\n return conversation\n\n\n@app.get(\"/api/conversations/{conversation_id}\", response_model=Conversation)\nasync def get_conversation(conversation_id: str):\n \"\"\"Get a specific conversation with all its messages.\"\"\"\n conversation = storage.get_conversation(conversation_id)\n if conversation is None:\n raise HTTPException(status_code=404, detail=\"Conversation not found\")\n return conversation\n\n\n@app.post(\"/api/conversations/{conversation_id}/message\")\nasync def send_message(conversation_id: str, request: SendMessageRequest):\n \"\"\"\n Send a message and run the 3-stage council process.\n Returns the complete response with all stages.\n \"\"\"\n # Check if conversation exists\n conversation = storage.get_conversation(conversation_id)\n if conversation is None:\n raise HTTPException(status_code=404, detail=\"Conversation not found\")\n\n # Check if this is the first message\n is_first_message = len(conversation[\"messages\"]) == 0\n\n # Add user message\n storage.add_user_message(conversation_id, request.content)\n\n # If this is the first message, generate a title\n if is_first_message:\n title = await generate_conversation_title(request.content)\n storage.update_conversation_title(conversation_id, title)\n\n # Run the 3-stage council process\n stage1_results, stage2_results, stage3_result, metadata = await run_full_council(\n request.content\n )\n\n # Add assistant message with all stages\n storage.add_assistant_message(\n conversation_id,\n stage1_results,\n stage2_results,\n stage3_result\n )\n\n # Return the complete response with metadata\n return {\n \"stage1\": stage1_results,\n \"stage2\": stage2_results,\n \"stage3\": stage3_result,\n \"metadata\": metadata\n }\n\n\n@app.post(\"/api/conversations/{conversation_id}/message/stream\")\nasync def send_message_stream(conversation_id: str, request: SendMessageRequest):\n \"\"\"\n Send a message and stream the 3-stage council process.\n Returns Server-Sent Events as each stage completes.\n \"\"\"\n # Check if conversation exists\n conversation = storage.get_conversation(conversation_id)\n if conversation is None:\n raise HTTPException(status_code=404, detail=\"Conversation not found\")\n\n # Check if this is the first message\n is_first_message = len(conversation[\"messages\"]) == 0\n\n async def event_generator():\n try:\n # Add user message\n storage.add_user_message(conversation_id, request.content)\n\n # Start title generation in parallel (don't await yet)\n title_task = None\n if is_first_message:\n title_task = asyncio.create_task(generate_conversation_title(request.content))\n\n # Stage 1: Collect responses\n yield f\"data: {json.dumps({'type': 'stage1_start'})}\\n\\n\"\n stage1_results = await stage1_collect_responses(request.content)\n yield f\"data: {json.dumps({'type': 'stage1_complete', 'data': stage1_results})}\\n\\n\"\n\n # Stage 2: Collect rankings\n yield f\"data: {json.dumps({'type': 'stage2_start'})}\\n\\n\"\n stage2_results, label_to_model = await stage2_collect_rankings(request.content, stage1_results)\n aggregate_rankings = calculate_aggregate_rankings(stage2_results, label_to_model)\n yield f\"data: {json.dumps({'type': 'stage2_complete', 'data': stage2_results, 'metadata': {'label_to_model': label_to_model, 'aggregate_rankings': aggregate_rankings}})}\\n\\n\"\n\n # Stage 3: Synthesize final answer\n yield f\"data: {json.dumps({'type': 'stage3_start'})}\\n\\n\"\n stage3_result = await stage3_synthesize_final(request.content, stage1_results, stage2_results)\n yield f\"data: {json.dumps({'type': 'stage3_complete', 'data': stage3_result})}\\n\\n\"\n\n # Wait for title generation if it was started\n if title_task:\n title = await title_task\n storage.update_conversation_title(conversation_id, title)\n yield f\"data: {json.dumps({'type': 'title_complete', 'data': {'title': title}})}\\n\\n\"\n\n # Save complete assistant message\n storage.add_assistant_message(\n conversation_id,\n stage1_results,\n stage2_results,\n stage3_result\n )\n\n # Send completion event\n yield f\"data: {json.dumps({'type': 'complete'})}\\n\\n\"\n\n except Exception as e:\n # Send error event\n yield f\"data: {json.dumps({'type': 'error', 'message': str(e)})}\\n\\n\"\n\n return StreamingResponse(\n event_generator(),\n media_type=\"text/event-stream\",\n headers={\n \"Cache-Control\": \"no-cache\",\n \"Connection\": \"keep-alive\",\n }\n )\n\n\nif __name__ == \"__main__\":\n import uvicorn\n uvicorn.run(app, host=\"0.0.0.0\", port=8001)\n"
},
{
"path": "backend/openrouter.py",
"content": "\"\"\"OpenRouter API client for making LLM requests.\"\"\"\n\nimport httpx\nfrom typing import List, Dict, Any, Optional\nfrom .config import OPENROUTER_API_KEY, OPENROUTER_API_URL\n\n\nasync def query_model(\n model: str,\n messages: List[Dict[str, str]],\n timeout: float = 120.0\n) -> Optional[Dict[str, Any]]:\n \"\"\"\n Query a single model via OpenRouter API.\n\n Args:\n model: OpenRouter model identifier (e.g., \"openai/gpt-4o\")\n messages: List of message dicts with 'role' and 'content'\n timeout: Request timeout in seconds\n\n Returns:\n Response dict with 'content' and optional 'reasoning_details', or None if failed\n \"\"\"\n headers = {\n \"Authorization\": f\"Bearer {OPENROUTER_API_KEY}\",\n \"Content-Type\": \"application/json\",\n }\n\n payload = {\n \"model\": model,\n \"messages\": messages,\n }\n\n try:\n async with httpx.AsyncClient(timeout=timeout) as client:\n response = await client.post(\n OPENROUTER_API_URL,\n headers=headers,\n json=payload\n )\n response.raise_for_status()\n\n data = response.json()\n message = data['choices'][0]['message']\n\n return {\n 'content': message.get('content'),\n 'reasoning_details': message.get('reasoning_details')\n }\n\n except Exception as e:\n print(f\"Error querying model {model}: {e}\")\n return None\n\n\nasync def query_models_parallel(\n models: List[str],\n messages: List[Dict[str, str]]\n) -> Dict[str, Optional[Dict[str, Any]]]:\n \"\"\"\n Query multiple models in parallel.\n\n Args:\n models: List of OpenRouter model identifiers\n messages: List of message dicts to send to each model\n\n Returns:\n Dict mapping model identifier to response dict (or None if failed)\n \"\"\"\n import asyncio\n\n # Create tasks for all models\n tasks = [query_model(model, messages) for model in models]\n\n # Wait for all to complete\n responses = await asyncio.gather(*tasks)\n\n # Map models to their responses\n return {model: response for model, response in zip(models, responses)}\n"
},
{
"path": "backend/storage.py",
"content": "\"\"\"JSON-based storage for conversations.\"\"\"\n\nimport json\nimport os\nfrom datetime import datetime\nfrom typing import List, Dict, Any, Optional\nfrom pathlib import Path\nfrom .config import DATA_DIR\n\n\ndef ensure_data_dir():\n \"\"\"Ensure the data directory exists.\"\"\"\n Path(DATA_DIR).mkdir(parents=True, exist_ok=True)\n\n\ndef get_conversation_path(conversation_id: str) -> str:\n \"\"\"Get the file path for a conversation.\"\"\"\n return os.path.join(DATA_DIR, f\"{conversation_id}.json\")\n\n\ndef create_conversation(conversation_id: str) -> Dict[str, Any]:\n \"\"\"\n Create a new conversation.\n\n Args:\n conversation_id: Unique identifier for the conversation\n\n Returns:\n New conversation dict\n \"\"\"\n ensure_data_dir()\n\n conversation = {\n \"id\": conversation_id,\n \"created_at\": datetime.utcnow().isoformat(),\n \"title\": \"New Conversation\",\n \"messages\": []\n }\n\n # Save to file\n path = get_conversation_path(conversation_id)\n with open(path, 'w') as f:\n json.dump(conversation, f, indent=2)\n\n return conversation\n\n\ndef get_conversation(conversation_id: str) -> Optional[Dict[str, Any]]:\n \"\"\"\n Load a conversation from storage.\n\n Args:\n conversation_id: Unique identifier for the conversation\n\n Returns:\n Conversation dict or None if not found\n \"\"\"\n path = get_conversation_path(conversation_id)\n\n if not os.path.exists(path):\n return None\n\n with open(path, 'r') as f:\n return json.load(f)\n\n\ndef save_conversation(conversation: Dict[str, Any]):\n \"\"\"\n Save a conversation to storage.\n\n Args:\n conversation: Conversation dict to save\n \"\"\"\n ensure_data_dir()\n\n path = get_conversation_path(conversation['id'])\n with open(path, 'w') as f:\n json.dump(conversation, f, indent=2)\n\n\ndef list_conversations() -> List[Dict[str, Any]]:\n \"\"\"\n List all conversations (metadata only).\n\n Returns:\n List of conversation metadata dicts\n \"\"\"\n ensure_data_dir()\n\n conversations = []\n for filename in os.listdir(DATA_DIR):\n if filename.endswith('.json'):\n path = os.path.join(DATA_DIR, filename)\n with open(path, 'r') as f:\n data = json.load(f)\n # Return metadata only\n conversations.append({\n \"id\": data[\"id\"],\n \"created_at\": data[\"created_at\"],\n \"title\": data.get(\"title\", \"New Conversation\"),\n \"message_count\": len(data[\"messages\"])\n })\n\n # Sort by creation time, newest first\n conversations.sort(key=lambda x: x[\"created_at\"], reverse=True)\n\n return conversations\n\n\ndef add_user_message(conversation_id: str, content: str):\n \"\"\"\n Add a user message to a conversation.\n\n Args:\n conversation_id: Conversation identifier\n content: User message content\n \"\"\"\n conversation = get_conversation(conversation_id)\n if conversation is None:\n raise ValueError(f\"Conversation {conversation_id} not found\")\n\n conversation[\"messages\"].append({\n \"role\": \"user\",\n \"content\": content\n })\n\n save_conversation(conversation)\n\n\ndef add_assistant_message(\n conversation_id: str,\n stage1: List[Dict[str, Any]],\n stage2: List[Dict[str, Any]],\n stage3: Dict[str, Any]\n):\n \"\"\"\n Add an assistant message with all 3 stages to a conversation.\n\n Args:\n conversation_id: Conversation identifier\n stage1: List of individual model responses\n stage2: List of model rankings\n stage3: Final synthesized response\n \"\"\"\n conversation = get_conversation(conversation_id)\n if conversation is None:\n raise ValueError(f\"Conversation {conversation_id} not found\")\n\n conversation[\"messages\"].append({\n \"role\": \"assistant\",\n \"stage1\": stage1,\n \"stage2\": stage2,\n \"stage3\": stage3\n })\n\n save_conversation(conversation)\n\n\ndef update_conversation_title(conversation_id: str, title: str):\n \"\"\"\n Update the title of a conversation.\n\n Args:\n conversation_id: Conversation identifier\n title: New title for the conversation\n \"\"\"\n conversation = get_conversation(conversation_id)\n if conversation is None:\n raise ValueError(f\"Conversation {conversation_id} not found\")\n\n conversation[\"title\"] = title\n save_conversation(conversation)\n"
},
{
"path": "frontend/.gitignore",
"content": "# Logs\nlogs\n*.log\nnpm-debug.log*\nyarn-debug.log*\nyarn-error.log*\npnpm-debug.log*\nlerna-debug.log*\n\nnode_modules\ndist\ndist-ssr\n*.local\n\n# Editor directories and files\n.vscode/*\n!.vscode/extensions.json\n.idea\n.DS_Store\n*.suo\n*.ntvs*\n*.njsproj\n*.sln\n*.sw?\n"
},
{
"path": "frontend/README.md",
"content": "# React + Vite\n\nThis template provides a minimal setup to get React working in Vite with HMR and some ESLint rules.\n\nCurrently, two official plugins are available:\n\n- [@vitejs/plugin-react](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react) uses [Babel](https://babeljs.io/) (or [oxc](https://oxc.rs) when used in [rolldown-vite](https://vite.dev/guide/rolldown)) for Fast Refresh\n- [@vitejs/plugin-react-swc](https://github.com/vitejs/vite-plugin-react/blob/main/packages/plugin-react-swc) uses [SWC](https://swc.rs/) for Fast Refresh\n\n## React Compiler\n\nThe React Compiler is not enabled on this template because of its impact on dev & build performances. To add it, see [this documentation](https://react.dev/learn/react-compiler/installation).\n\n## Expanding the ESLint configuration\n\nIf you are developing a production application, we recommend using TypeScript with type-aware lint rules enabled. Check out the [TS template](https://github.com/vitejs/vite/tree/main/packages/create-vite/template-react-ts) for information on how to integrate TypeScript and [`typescript-eslint`](https://typescript-eslint.io) in your project.\n"
},
{
"path": "frontend/eslint.config.js",
"content": "import js from '@eslint/js'\nimport globals from 'globals'\nimport reactHooks from 'eslint-plugin-react-hooks'\nimport reactRefresh from 'eslint-plugin-react-refresh'\nimport { defineConfig, globalIgnores } from 'eslint/config'\n\nexport default defineConfig([\n globalIgnores(['dist']),\n {\n files: ['**/*.{js,jsx}'],\n extends: [\n js.configs.recommended,\n reactHooks.configs.flat.recommended,\n reactRefresh.configs.vite,\n ],\n languageOptions: {\n ecmaVersion: 2020,\n globals: globals.browser,\n parserOptions: {\n ecmaVersion: 'latest',\n ecmaFeatures: { jsx: true },\n sourceType: 'module',\n },\n },\n rules: {\n 'no-unused-vars': ['error', { varsIgnorePattern: '^[A-Z_]' }],\n },\n },\n])\n"
},
{
"path": "frontend/index.html",
"content": "\n\n \n \n \n \n frontend\n \n \n \n \n \n\n"
},
{
"path": "frontend/package.json",
"content": "{\n \"name\": \"frontend\",\n \"private\": true,\n \"version\": \"0.0.0\",\n \"type\": \"module\",\n \"scripts\": {\n \"dev\": \"vite\",\n \"build\": \"vite build\",\n \"lint\": \"eslint .\",\n \"preview\": \"vite preview\"\n },\n \"dependencies\": {\n \"react\": \"^19.2.0\",\n \"react-dom\": \"^19.2.0\",\n \"react-markdown\": \"^10.1.0\"\n },\n \"devDependencies\": {\n \"@eslint/js\": \"^9.39.1\",\n \"@types/react\": \"^19.2.5\",\n \"@types/react-dom\": \"^19.2.3\",\n \"@vitejs/plugin-react\": \"^5.1.1\",\n \"eslint\": \"^9.39.1\",\n \"eslint-plugin-react-hooks\": \"^7.0.1\",\n \"eslint-plugin-react-refresh\": \"^0.4.24\",\n \"globals\": \"^16.5.0\",\n \"vite\": \"^7.2.4\"\n }\n}\n"
},
{
"path": "frontend/src/App.css",
"content": "* {\n box-sizing: border-box;\n}\n\n.app {\n display: flex;\n height: 100vh;\n width: 100vw;\n overflow: hidden;\n background: #ffffff;\n color: #333;\n font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', 'Roboto', 'Oxygen',\n 'Ubuntu', 'Cantarell', 'Fira Sans', 'Droid Sans', 'Helvetica Neue',\n sans-serif;\n}\n"
},
{
"path": "frontend/src/App.jsx",
"content": "import { useState, useEffect } from 'react';\nimport Sidebar from './components/Sidebar';\nimport ChatInterface from './components/ChatInterface';\nimport { api } from './api';\nimport './App.css';\n\nfunction App() {\n const [conversations, setConversations] = useState([]);\n const [currentConversationId, setCurrentConversationId] = useState(null);\n const [currentConversation, setCurrentConversation] = useState(null);\n const [isLoading, setIsLoading] = useState(false);\n\n // Load conversations on mount\n useEffect(() => {\n loadConversations();\n }, []);\n\n // Load conversation details when selected\n useEffect(() => {\n if (currentConversationId) {\n loadConversation(currentConversationId);\n }\n }, [currentConversationId]);\n\n const loadConversations = async () => {\n try {\n const convs = await api.listConversations();\n setConversations(convs);\n } catch (error) {\n console.error('Failed to load conversations:', error);\n }\n };\n\n const loadConversation = async (id) => {\n try {\n const conv = await api.getConversation(id);\n setCurrentConversation(conv);\n } catch (error) {\n console.error('Failed to load conversation:', error);\n }\n };\n\n const handleNewConversation = async () => {\n try {\n const newConv = await api.createConversation();\n setConversations([\n { id: newConv.id, created_at: newConv.created_at, message_count: 0 },\n ...conversations,\n ]);\n setCurrentConversationId(newConv.id);\n } catch (error) {\n console.error('Failed to create conversation:', error);\n }\n };\n\n const handleSelectConversation = (id) => {\n setCurrentConversationId(id);\n };\n\n const handleSendMessage = async (content) => {\n if (!currentConversationId) return;\n\n setIsLoading(true);\n try {\n // Optimistically add user message to UI\n const userMessage = { role: 'user', content };\n setCurrentConversation((prev) => ({\n ...prev,\n messages: [...prev.messages, userMessage],\n }));\n\n // Create a partial assistant message that will be updated progressively\n const assistantMessage = {\n role: 'assistant',\n stage1: null,\n stage2: null,\n stage3: null,\n metadata: null,\n loading: {\n stage1: false,\n stage2: false,\n stage3: false,\n },\n };\n\n // Add the partial assistant message\n setCurrentConversation((prev) => ({\n ...prev,\n messages: [...prev.messages, assistantMessage],\n }));\n\n // Send message with streaming\n await api.sendMessageStream(currentConversationId, content, (eventType, event) => {\n switch (eventType) {\n case 'stage1_start':\n setCurrentConversation((prev) => {\n const messages = [...prev.messages];\n const lastMsg = messages[messages.length - 1];\n lastMsg.loading.stage1 = true;\n return { ...prev, messages };\n });\n break;\n\n case 'stage1_complete':\n setCurrentConversation((prev) => {\n const messages = [...prev.messages];\n const lastMsg = messages[messages.length - 1];\n lastMsg.stage1 = event.data;\n lastMsg.loading.stage1 = false;\n return { ...prev, messages };\n });\n break;\n\n case 'stage2_start':\n setCurrentConversation((prev) => {\n const messages = [...prev.messages];\n const lastMsg = messages[messages.length - 1];\n lastMsg.loading.stage2 = true;\n return { ...prev, messages };\n });\n break;\n\n case 'stage2_complete':\n setCurrentConversation((prev) => {\n const messages = [...prev.messages];\n const lastMsg = messages[messages.length - 1];\n lastMsg.stage2 = event.data;\n lastMsg.metadata = event.metadata;\n lastMsg.loading.stage2 = false;\n return { ...prev, messages };\n });\n break;\n\n case 'stage3_start':\n setCurrentConversation((prev) => {\n const messages = [...prev.messages];\n const lastMsg = messages[messages.length - 1];\n lastMsg.loading.stage3 = true;\n return { ...prev, messages };\n });\n break;\n\n case 'stage3_complete':\n setCurrentConversation((prev) => {\n const messages = [...prev.messages];\n const lastMsg = messages[messages.length - 1];\n lastMsg.stage3 = event.data;\n lastMsg.loading.stage3 = false;\n return { ...prev, messages };\n });\n break;\n\n case 'title_complete':\n // Reload conversations to get updated title\n loadConversations();\n break;\n\n case 'complete':\n // Stream complete, reload conversations list\n loadConversations();\n setIsLoading(false);\n break;\n\n case 'error':\n console.error('Stream error:', event.message);\n setIsLoading(false);\n break;\n\n default:\n console.log('Unknown event type:', eventType);\n }\n });\n } catch (error) {\n console.error('Failed to send message:', error);\n // Remove optimistic messages on error\n setCurrentConversation((prev) => ({\n ...prev,\n messages: prev.messages.slice(0, -2),\n }));\n setIsLoading(false);\n }\n };\n\n return (\n
\n Each model evaluated all responses (anonymized as Response A, B, C, etc.) and provided rankings.\n Below, model names are shown in bold for readability, but the original evaluation used anonymous labels.\n