Repository: openai/symphony Branch: main Commit: a164593aacb3 Files: 98 Total size: 794.9 KB Directory structure: gitextract_auczxnks/ ├── .codex/ │ ├── skills/ │ │ ├── commit/ │ │ │ └── SKILL.md │ │ ├── debug/ │ │ │ └── SKILL.md │ │ ├── land/ │ │ │ ├── SKILL.md │ │ │ └── land_watch.py │ │ ├── linear/ │ │ │ └── SKILL.md │ │ ├── pull/ │ │ │ └── SKILL.md │ │ └── push/ │ │ └── SKILL.md │ └── worktree_init.sh ├── .github/ │ ├── pull_request_template.md │ └── workflows/ │ ├── make-all.yml │ └── pr-description-lint.yml ├── LICENSE ├── NOTICE ├── README.md ├── SPEC.md └── elixir/ ├── .formatter.exs ├── .gitattributes ├── .gitignore ├── AGENTS.md ├── Makefile ├── README.md ├── WORKFLOW.md ├── config/ │ └── config.exs ├── docs/ │ ├── logging.md │ └── token_accounting.md ├── lib/ │ ├── mix/ │ │ └── tasks/ │ │ ├── pr_body.check.ex │ │ ├── specs.check.ex │ │ └── workspace.before_remove.ex │ ├── symphony_elixir/ │ │ ├── agent_runner.ex │ │ ├── cli.ex │ │ ├── codex/ │ │ │ ├── app_server.ex │ │ │ └── dynamic_tool.ex │ │ ├── config/ │ │ │ └── schema.ex │ │ ├── config.ex │ │ ├── http_server.ex │ │ ├── linear/ │ │ │ ├── adapter.ex │ │ │ ├── client.ex │ │ │ └── issue.ex │ │ ├── log_file.ex │ │ ├── orchestrator.ex │ │ ├── path_safety.ex │ │ ├── prompt_builder.ex │ │ ├── specs_check.ex │ │ ├── ssh.ex │ │ ├── status_dashboard.ex │ │ ├── tracker/ │ │ │ └── memory.ex │ │ ├── tracker.ex │ │ ├── workflow.ex │ │ ├── workflow_store.ex │ │ └── workspace.ex │ ├── symphony_elixir.ex │ └── symphony_elixir_web/ │ ├── components/ │ │ └── layouts.ex │ ├── controllers/ │ │ ├── observability_api_controller.ex │ │ └── static_asset_controller.ex │ ├── endpoint.ex │ ├── error_html.ex │ ├── error_json.ex │ ├── live/ │ │ └── dashboard_live.ex │ ├── observability_pubsub.ex │ ├── presenter.ex │ ├── router.ex │ └── static_assets.ex ├── mise.toml ├── mix.exs ├── priv/ │ └── static/ │ └── dashboard.css └── test/ ├── fixtures/ │ └── status_dashboard_snapshots/ │ ├── backoff_queue.evidence.md │ ├── backoff_queue.snapshot.txt │ ├── credits_unlimited.evidence.md │ ├── credits_unlimited.snapshot.txt │ ├── idle.evidence.md │ ├── idle.snapshot.txt │ ├── idle_with_dashboard_url.evidence.md │ ├── idle_with_dashboard_url.snapshot.txt │ ├── super_busy.evidence.md │ └── super_busy.snapshot.txt ├── mix/ │ └── tasks/ │ ├── pr_body_check_test.exs │ ├── specs_check_task_test.exs │ └── workspace_before_remove_test.exs ├── support/ │ ├── live_e2e_docker/ │ │ ├── Dockerfile │ │ ├── docker-compose.yml │ │ ├── live_worker_entrypoint.sh │ │ └── symphony-live-worker.conf │ ├── snapshot_support.exs │ └── test_support.exs ├── symphony_elixir/ │ ├── app_server_test.exs │ ├── cli_test.exs │ ├── core_test.exs │ ├── dynamic_tool_test.exs │ ├── extensions_test.exs │ ├── live_e2e_test.exs │ ├── log_file_test.exs │ ├── observability_pubsub_test.exs │ ├── orchestrator_status_test.exs │ ├── specs_check_test.exs │ ├── ssh_test.exs │ ├── status_dashboard_snapshot_test.exs │ └── workspace_and_config_test.exs └── test_helper.exs ================================================ FILE CONTENTS ================================================ ================================================ FILE: .codex/skills/commit/SKILL.md ================================================ --- name: commit description: Create a well-formed git commit from current changes using session history for rationale and summary; use when asked to commit, prepare a commit message, or finalize staged work. --- # Commit ## Goals - Produce a commit that reflects the actual code changes and the session context. - Follow common git conventions (type prefix, short subject, wrapped body). - Include both summary and rationale in the body. ## Inputs - Codex session history for intent and rationale. - `git status`, `git diff`, and `git diff --staged` for actual changes. - Repo-specific commit conventions if documented. ## Steps 1. Read session history to identify scope, intent, and rationale. 2. Inspect the working tree and staged changes (`git status`, `git diff`, `git diff --staged`). 3. Stage intended changes, including new files (`git add -A`) after confirming scope. 4. Sanity-check newly added files; if anything looks random or likely ignored (build artifacts, logs, temp files), flag it to the user before committing. 5. If staging is incomplete or includes unrelated files, fix the index or ask for confirmation. 6. Choose a conventional type and optional scope that match the change (e.g., `feat(scope): ...`, `fix(scope): ...`, `refactor(scope): ...`). 7. Write a subject line in imperative mood, <= 72 characters, no trailing period. 8. Write a body that includes: - Summary of key changes (what changed). - Rationale and trade-offs (why it changed). - Tests or validation run (or explicit note if not run). 9. Append a `Co-authored-by` trailer for Codex using `Codex ` unless the user explicitly requests a different identity. 10. Wrap body lines at 72 characters. 11. Create the commit message with a here-doc or temp file and use `git commit -F ` so newlines are literal (avoid `-m` with `\n`). 12. Commit only when the message matches the staged changes: if the staged diff includes unrelated files or the message describes work that isn't staged, fix the index or revise the message before committing. ## Output - A single commit created with `git commit` whose message reflects the session. ## Template Type and scope are examples only; adjust to fit the repo and changes. ``` (): Summary: - - Rationale: - - Tests: - Co-authored-by: Codex ``` ================================================ FILE: .codex/skills/debug/SKILL.md ================================================ --- name: debug description: Investigate stuck runs and execution failures by tracing Symphony and Codex logs with issue/session identifiers; use when runs stall, retry repeatedly, or fail unexpectedly. --- # Debug ## Goals - Find why a run is stuck, retrying, or failing. - Correlate Linear issue identity to a Codex session quickly. - Read the right logs in the right order to isolate root cause. ## Log Sources - Primary runtime log: `log/symphony.log` - Default comes from `SymphonyElixir.LogFile` (`log/symphony.log`). - Includes orchestrator, agent runner, and Codex app-server lifecycle logs. - Rotated runtime logs: `log/symphony.log*` - Check these when the relevant run is older. ## Correlation Keys - `issue_identifier`: human ticket key (example: `MT-625`) - `issue_id`: Linear UUID (stable internal ID) - `session_id`: Codex thread-turn pair (`-`) `elixir/docs/logging.md` requires these fields for issue/session lifecycle logs. Use them as your join keys during debugging. ## Quick Triage (Stuck Run) 1. Confirm scheduler/worker symptoms for the ticket. 2. Find recent lines for the ticket (`issue_identifier` first). 3. Extract `session_id` from matching lines. 4. Trace that `session_id` across start, stream, completion/failure, and stall handling logs. 5. Decide class of failure: timeout/stall, app-server startup failure, turn failure, or orchestrator retry loop. ## Commands ```bash # 1) Narrow by ticket key (fastest entry point) rg -n "issue_identifier=MT-625" log/symphony.log* # 2) If needed, narrow by Linear UUID rg -n "issue_id=" log/symphony.log* # 3) Pull session IDs seen for that ticket rg -o "session_id=[^ ;]+" log/symphony.log* | sort -u # 4) Trace one session end-to-end rg -n "session_id=-" log/symphony.log* # 5) Focus on stuck/retry signals rg -n "Issue stalled|scheduling retry|turn_timeout|turn_failed|Codex session failed|Codex session ended with error" log/symphony.log* ``` ## Investigation Flow 1. Locate the ticket slice: - Search by `issue_identifier=`. - If noise is high, add `issue_id=`. 2. Establish timeline: - Identify first `Codex session started ... session_id=...`. - Follow with `Codex session completed`, `ended with error`, or worker exit lines. 3. Classify the problem: - Stall loop: `Issue stalled ... restarting with backoff`. - App-server startup: `Codex session failed ...`. - Turn execution failure: `turn_failed`, `turn_cancelled`, `turn_timeout`, or `ended with error`. - Worker crash: `Agent task exited ... reason=...`. 4. Validate scope: - Check whether failures are isolated to one issue/session or repeating across multiple tickets. 5. Capture evidence: - Save key log lines with timestamps, `issue_identifier`, `issue_id`, and `session_id`. - Record probable root cause and the exact failing stage. ## Reading Codex Session Logs In Symphony, Codex session diagnostics are emitted into `log/symphony.log` and keyed by `session_id`. Read them as a lifecycle: 1. `Codex session started ... session_id=...` 2. Session stream/lifecycle events for the same `session_id` 3. Terminal event: - `Codex session completed ...`, or - `Codex session ended with error ...`, or - `Issue stalled ... restarting with backoff` For one specific session investigation, keep the trace narrow: 1. Capture one `session_id` for the ticket. 2. Build a timestamped slice for only that session: - `rg -n "session_id=-" log/symphony.log*` 3. Mark the exact failing stage: - Startup failure before stream events (`Codex session failed ...`). - Turn/runtime failure after stream events (`turn_*` / `ended with error`). - Stall recovery (`Issue stalled ... restarting with backoff`). 4. Pair findings with `issue_identifier` and `issue_id` from nearby lines to confirm you are not mixing concurrent retries. Always pair session findings with `issue_identifier`/`issue_id` to avoid mixing concurrent runs. ## Notes - Prefer `rg` over `grep` for speed on large logs. - Check rotated logs (`log/symphony.log*`) before concluding data is missing. - If required context fields are missing in new log statements, align with `elixir/docs/logging.md` conventions. ================================================ FILE: .codex/skills/land/SKILL.md ================================================ --- name: land description: Land a PR by monitoring conflicts, resolving them, waiting for checks, and squash-merging when green; use when asked to land, merge, or shepherd a PR to completion. --- # Land ## Goals - Ensure the PR is conflict-free with main. - Keep CI green and fix failures when they occur. - Squash-merge the PR once checks pass. - Do not yield to the user until the PR is merged; keep the watcher loop running unless blocked. - No need to delete remote branches after merge; the repo auto-deletes head branches. ## Preconditions - `gh` CLI is authenticated. - You are on the PR branch with a clean working tree. ## Steps 1. Locate the PR for the current branch. 2. Confirm the full gauntlet is green locally before any push. 3. If the working tree has uncommitted changes, commit with the `commit` skill and push with the `push` skill before proceeding. 4. Check mergeability and conflicts against main. 5. If conflicts exist, use the `pull` skill to fetch/merge `origin/main` and resolve conflicts, then use the `push` skill to publish the updated branch. 6. Ensure Codex review comments (if present) are acknowledged and any required fixes are handled before merging. 7. Watch checks until complete. 8. If checks fail, pull logs, fix the issue, commit with the `commit` skill, push with the `push` skill, and re-run checks. 9. When all checks are green and review feedback is addressed, squash-merge and delete the branch using the PR title/body for the merge subject/body. 10. **Context guard:** Before implementing review feedback, confirm it does not conflict with the user’s stated intent or task context. If it conflicts, respond inline with a justification and ask the user before changing code. 11. **Pushback template:** When disagreeing, reply inline with: acknowledge + rationale + offer alternative. 12. **Ambiguity gate:** When ambiguity blocks progress, use the clarification flow (assign PR to current GH user, mention them, wait for response). Do not implement until ambiguity is resolved. - If you are confident you know better than the reviewer, you may proceed without asking the user, but reply inline with your rationale. 13. **Per-comment mode:** For each review comment, choose one of: accept, clarify, or push back. Reply inline (or in the issue thread for Codex reviews) stating the mode before changing code. 14. **Reply before change:** Always respond with intended action before pushing code changes (inline for review comments, issue thread for Codex reviews). ## Commands ``` # Ensure branch and PR context branch=$(git branch --show-current) pr_number=$(gh pr view --json number -q .number) pr_title=$(gh pr view --json title -q .title) pr_body=$(gh pr view --json body -q .body) # Check mergeability and conflicts mergeable=$(gh pr view --json mergeable -q .mergeable) if [ "$mergeable" = "CONFLICTING" ]; then # Run the `pull` skill to handle fetch + merge + conflict resolution. # Then run the `push` skill to publish the updated branch. fi # Preferred: use the Async Watch Helper below. The manual loop is a fallback # when Python cannot run or the helper script is unavailable. # Wait for review feedback: Codex reviews arrive as issue comments that start # with "## Codex Review — ". Treat them like reviewer feedback: reply # with a `[codex]` issue comment acknowledging the findings and whether you're # addressing or deferring them. while true; do gh api repos/{owner}/{repo}/issues/"$pr_number"/comments \ --jq '.[] | select(.body | startswith("## Codex Review")) | .id' | rg -q '.' \ && break sleep 10 done # Watch checks if ! gh pr checks --watch; then gh pr checks # Identify failing run and inspect logs # gh run list --branch "$branch" # gh run view --log exit 1 fi # Squash-merge (remote branches auto-delete on merge in this repo) gh pr merge --squash --subject "$pr_title" --body "$pr_body" ``` ## Async Watch Helper Preferred: use the asyncio watcher to monitor review comments, CI, and head updates in parallel: ``` python3 .codex/skills/land/land_watch.py ``` Exit codes: - 2: Review comments detected (address feedback) - 3: CI checks failed - 4: PR head updated (autofix commit detected) ## Failure Handling - If checks fail, pull details with `gh pr checks` and `gh run view --log`, then fix locally, commit with the `commit` skill, push with the `push` skill, and re-run the watch. - Use judgment to identify flaky failures. If a failure is a flake (e.g., a timeout on only one platform), you may proceed without fixing it. - If CI pushes an auto-fix commit (authored by GitHub Actions), it does not trigger a fresh CI run. Detect the updated PR head, pull locally, merge `origin/main` if needed, add a real author commit, and force-push to retrigger CI, then restart the checks loop. - If all jobs fail with corrupted pnpm lockfile errors on the merge commit, the remediation is to fetch latest `origin/main`, merge, force-push, and rerun CI. - If mergeability is `UNKNOWN`, wait and re-check. - Do not merge while review comments (human or Codex review) are outstanding. - Codex review jobs retry on failure and are non-blocking; use the presence of `## Codex Review — ` issue comments (not job status) as the signal that review feedback is available. - Do not enable auto-merge; this repo has no required checks so auto-merge can skip tests. - If the remote PR branch advanced due to your own prior force-push or merge, avoid redundant merges; re-run the formatter locally if needed and `git push --force-with-lease`. ## Review Handling - Codex reviews now arrive as issue comments posted by GitHub Actions. They start with `## Codex Review — ` and include the reviewer’s methodology + guardrails used. Treat these as feedback that must be acknowledged before merge. - Human review comments are blocking and must be addressed (responded to and resolved) before requesting a new review or merging. - If multiple reviewers comment in the same thread, respond to each comment (batching is fine) before closing the thread. - Fetch review comments via `gh api` and reply with a prefixed comment. - Use review comment endpoints (not issue comments) to find inline feedback: - List PR review comments: ``` gh api repos/{owner}/{repo}/pulls//comments ``` - PR issue comments (top-level discussion): ``` gh api repos/{owner}/{repo}/issues//comments ``` - Reply to a specific review comment: ``` gh api -X POST /repos/{owner}/{repo}/pulls//comments \ -f body='[codex] ' -F in_reply_to= ``` - `in_reply_to` must be the numeric review comment id (e.g., `2710521800`), not the GraphQL node id (e.g., `PRRC_...`), and the endpoint must include the PR number (`/pulls//comments`). - If GraphQL review reply mutation is forbidden, use REST. - A 404 on reply typically means the wrong endpoint (missing PR number) or insufficient scope; verify by listing comments first. - All GitHub comments generated by this agent must be prefixed with `[codex]`. - For Codex review issue comments, reply in the issue thread (not a review thread) with `[codex]` and state whether you will address the feedback now or defer it (include rationale). - If feedback requires changes: - For inline review comments (human), reply with intended fixes (`[codex] ...`) **as an inline reply to the original review comment** using the review comment endpoint and `in_reply_to` (do not use issue comments for this). - Implement fixes, commit, push. - Reply with the fix details and commit sha (`[codex] ...`) in the same place you acknowledged the feedback (issue comment for Codex reviews, inline reply for review comments). - The land watcher treats Codex review issue comments as unresolved until a newer `[codex]` issue comment is posted acknowledging the findings. - Only request a new Codex review when you need a rerun (e.g., after new commits). Do not request one without changes since the last review. - Before requesting a new Codex review, re-run the land watcher and ensure there are zero outstanding review comments (all have `[codex]` inline replies). - After pushing new commits, the Codex review workflow will rerun on PR synchronization (or you can re-run the workflow manually). Post a concise root-level summary comment so reviewers have the latest delta: ``` [codex] Changes since last review: - Commits: , Tests: ``` - Only request a new review if there is at least one new commit since the previous request. - Wait for the next Codex review comment before merging. ## Scope + PR Metadata - The PR title and description should reflect the full scope of the change, not just the most recent fix. - If review feedback expands scope, decide whether to include it now or defer it. You can accept, defer, or decline feedback. If deferring or declining, call it out in the root-level `[codex]` update with a brief reason (e.g., out-of-scope, conflicts with intent, unnecessary). - Correctness issues raised in review comments should be addressed. If you plan to defer or decline a correctness concern, validate first and explain why the concern does not apply. - Classify each review comment as one of: correctness, design, style, clarification, scope. - For correctness feedback, provide concrete validation (test, log, or reasoning) before closing it. - When accepting feedback, include a one-line rationale in the root-level update. - When declining feedback, offer a brief alternative or follow-up trigger. - Prefer a single consolidated "review addressed" root-level comment after a batch of fixes instead of many small updates. - For doc feedback, confirm the doc change matches behavior (no doc-only edits to appease review). ================================================ FILE: .codex/skills/land/land_watch.py ================================================ #!/usr/bin/env python3 import asyncio import json import random import re from dataclasses import dataclass from datetime import datetime from typing import Any POLL_SECONDS = 10 CHECKS_APPEAR_TIMEOUT_SECONDS = 120 CODEX_BOTS = { "chatgpt-codex-connector[bot]", "github-actions[bot]", "codex-gc-app[bot]", "app/codex-gc-app", } MAX_GH_RETRIES = 5 BASE_GH_BACKOFF_SECONDS = 2 @dataclass class PrInfo: number: int url: str head_sha: str mergeable: str | None merge_state: str | None class RateLimitError(RuntimeError): pass def is_rate_limit_error(error: str) -> bool: return "HTTP 429" in error or "rate limit" in error.lower() async def run_gh(*args: str) -> str: max_delay = BASE_GH_BACKOFF_SECONDS * (2 ** (MAX_GH_RETRIES - 1)) delay_seconds = BASE_GH_BACKOFF_SECONDS last_error = "gh command failed" for attempt in range(1, MAX_GH_RETRIES + 1): proc = await asyncio.create_subprocess_exec( "gh", *args, stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE, ) stdout, stderr = await proc.communicate() if proc.returncode == 0: return stdout.decode() error = stderr.decode().strip() or "gh command failed" if not is_rate_limit_error(error): raise RuntimeError(error) last_error = error if attempt >= MAX_GH_RETRIES: break jitter = random.uniform(0, delay_seconds) await asyncio.sleep(min(delay_seconds + jitter, max_delay)) delay_seconds = min(delay_seconds * 2, max_delay) raise RateLimitError(last_error) async def get_pr_info() -> PrInfo: data = await run_gh( "pr", "view", "--json", "number,url,headRefOid,mergeable,mergeStateStatus", ) parsed = json.loads(data) return PrInfo( number=parsed["number"], url=parsed["url"], head_sha=parsed["headRefOid"], mergeable=parsed.get("mergeable"), merge_state=parsed.get("mergeStateStatus"), ) async def get_paginated_list(endpoint: str) -> list[dict[str, Any]]: page = 1 items: list[dict[str, Any]] = [] while True: data = await run_gh( "api", "--method", "GET", endpoint, "-f", "per_page=100", "-f", f"page={page}", ) batch = json.loads(data) if not batch: break items.extend(batch) page += 1 return items async def get_issue_comments(pr_number: int) -> list[dict[str, Any]]: return await get_paginated_list( f"repos/{{owner}}/{{repo}}/issues/{pr_number}/comments", ) async def get_review_comments(pr_number: int) -> list[dict[str, Any]]: return await get_paginated_list( f"repos/{{owner}}/{{repo}}/pulls/{pr_number}/comments", ) async def get_reviews(pr_number: int) -> list[dict[str, Any]]: page = 1 reviews: list[dict[str, Any]] = [] while True: data = await run_gh( "api", "--method", "GET", f"repos/{{owner}}/{{repo}}/pulls/{pr_number}/reviews", "-f", "per_page=100", "-f", f"page={page}", ) batch = json.loads(data) if not batch: break reviews.extend(batch) page += 1 return reviews async def get_check_runs(head_sha: str) -> list[dict[str, Any]]: page = 1 check_runs: list[dict[str, Any]] = [] while True: data = await run_gh( "api", "--method", "GET", f"repos/{{owner}}/{{repo}}/commits/{head_sha}/check-runs", "-f", "per_page=100", "-f", f"page={page}", ) payload = json.loads(data) batch = payload.get("check_runs", []) if not batch: break check_runs.extend(batch) total_count = payload.get("total_count") if total_count is not None and len(check_runs) >= total_count: break page += 1 return check_runs def parse_time(value: str) -> datetime: normalized = value.replace("Z", "+00:00") return datetime.fromisoformat(normalized) CONTROL_CHARS_RE = re.compile(r"[\x00-\x08\x0b-\x1f\x7f-\x9f]") def sanitize_terminal_output(value: str) -> str: return CONTROL_CHARS_RE.sub("", value) def check_timestamp(check: dict[str, Any]) -> datetime | None: for key in ("completed_at", "started_at", "run_started_at", "created_at"): value = check.get(key) if value: return parse_time(value) return None def dedupe_check_runs(check_runs: list[dict[str, Any]]) -> list[dict[str, Any]]: latest_by_name: dict[str, dict[str, Any]] = {} for check in check_runs: name = check.get("name", "unknown") timestamp = check_timestamp(check) if name not in latest_by_name: latest_by_name[name] = check continue existing = latest_by_name[name] existing_timestamp = check_timestamp(existing) if timestamp is None: continue if existing_timestamp is None or timestamp > existing_timestamp: latest_by_name[name] = check return list(latest_by_name.values()) def summarize_checks(check_runs: list[dict[str, Any]]) -> tuple[bool, bool, list[str]]: if not check_runs: return True, False, ["no checks reported"] check_runs = dedupe_check_runs(check_runs) pending = False failed = False failures: list[str] = [] for check in check_runs: status = check.get("status") conclusion = check.get("conclusion") name = check.get("name", "unknown") if status != "completed": pending = True continue if conclusion not in ("success", "skipped", "neutral"): failed = True failures.append(f"{name}: {conclusion}") return pending, failed, failures def latest_review_request_at(comments: list[dict[str, Any]]) -> datetime | None: latest: datetime | None = None for comment in comments: if is_codex_bot_user(comment.get("user", {})): continue body = comment.get("body") or "" if "@codex review" not in body: continue timestamp = comment_time(comment) if timestamp is None: continue if latest is None or timestamp > latest: latest = timestamp return latest def filter_codex_comments( comments: list[dict[str, Any]], review_requested_at: datetime | None, ) -> list[dict[str, Any]]: latest_codex_reply = latest_codex_reply_by_thread(comments) latest_issue_ack = latest_codex_issue_reply_time(comments) codex_comments = [c for c in comments if is_codex_bot_user(c.get("user", {}))] filtered: list[dict[str, Any]] = [] for comment in codex_comments: created_time = comment_time(comment) if created_time is None: continue if review_requested_at is not None and created_time <= review_requested_at: continue is_threaded = bool( comment.get("in_reply_to_id") or comment.get("pull_request_review_id") ) if not is_threaded: if latest_issue_ack is not None and created_time <= latest_issue_ack: continue else: thread_root = thread_root_id(comment) last_reply = None if thread_root is not None: last_reply = latest_codex_reply.get(thread_root) if last_reply and last_reply > created_time: continue filtered.append(comment) return filtered def is_codex_bot_user(user: dict[str, Any]) -> bool: login = user.get("login") or "" return login in CODEX_BOTS def is_bot_user(user: dict[str, Any]) -> bool: login = user.get("login") or "" if is_codex_bot_user(user): return True if user.get("type") == "Bot": return True return login.endswith("[bot]") def is_codex_reply_body(body: str) -> bool: return body.startswith("[codex]") def is_codex_review_body(body: str) -> bool: return body.startswith("## Codex Review") def latest_codex_issue_reply_time( comments: list[dict[str, Any]], ) -> datetime | None: latest: datetime | None = None for comment in comments: body = (comment.get("body") or "").strip() if not is_codex_reply_body(body): continue created_time = comment_time(comment) if created_time is None: continue if latest is None or created_time > latest: latest = created_time return latest def filter_human_issue_comments(comments: list[dict[str, Any]]) -> list[dict[str, Any]]: latest_ack = latest_codex_issue_reply_time(comments) filtered: list[dict[str, Any]] = [] for comment in comments: if is_bot_user(comment.get("user", {})): continue body = (comment.get("body") or "").strip() if is_codex_reply_body(body): continue if is_codex_review_body(body): continue if "@codex review" in body: continue created_time = comment_time(comment) if ( latest_ack is not None and created_time is not None and created_time <= latest_ack ): continue filtered.append(comment) return filtered def filter_codex_review_issue_comments( comments: list[dict[str, Any]], ) -> list[dict[str, Any]]: latest_ack = latest_codex_issue_reply_time(comments) filtered: list[dict[str, Any]] = [] for comment in comments: body = (comment.get("body") or "").strip() if not is_codex_review_body(body): continue created_time = comment_time(comment) if ( latest_ack is not None and created_time is not None and created_time <= latest_ack ): continue filtered.append(comment) return filtered def thread_root_id(comment: dict[str, Any]) -> int | None: return comment.get("in_reply_to_id") or comment.get("id") def comment_time(comment: dict[str, Any]) -> datetime | None: timestamp = comment.get("updated_at") or comment.get("created_at") if not timestamp: return None return parse_time(timestamp) def latest_codex_reply_by_thread( comments: list[dict[str, Any]], ) -> dict[int, datetime]: latest: dict[int, datetime] = {} for comment in comments: body = (comment.get("body") or "").strip() if not is_codex_reply_body(body): continue thread_root = thread_root_id(comment) created_time = comment_time(comment) if thread_root is None or created_time is None: continue existing = latest.get(thread_root) if existing is None or created_time > existing: latest[thread_root] = created_time return latest def filter_human_review_comments( comments: list[dict[str, Any]], ) -> list[dict[str, Any]]: latest_codex_reply = latest_codex_reply_by_thread(comments) filtered: list[dict[str, Any]] = [] for comment in comments: if is_bot_user(comment.get("user", {})): continue body = (comment.get("body") or "").strip() if is_codex_reply_body(body): continue thread_root = thread_root_id(comment) created_time = comment_time(comment) last_codex_reply = None if thread_root is not None: last_codex_reply = latest_codex_reply.get(thread_root) if last_codex_reply and created_time and created_time <= last_codex_reply: continue filtered.append(comment) return filtered def is_blocking_review( review: dict[str, Any], review_requested_at: datetime | None, ) -> bool: created_at = review.get("submitted_at") or review.get("created_at") if not created_at: return False user_login = review.get("user", {}).get("login") created_time = parse_time(created_at) if ( user_login in CODEX_BOTS and review_requested_at is not None and created_time <= review_requested_at ): return False body = (review.get("body") or "").strip() state = review.get("state") if user_login in CODEX_BOTS: return state == "CHANGES_REQUESTED" if body.startswith("[codex]") or state in ("APPROVED", "DISMISSED"): return False blocking = False if body or state == "CHANGES_REQUESTED": blocking = True elif state == "COMMENTED": blocking = False elif state: blocking = state not in ("APPROVED", "DISMISSED") return blocking def review_timestamp(review: dict[str, Any]) -> datetime | None: created_at = review.get("submitted_at") or review.get("created_at") if not created_at: return None return parse_time(created_at) def dedupe_reviews(reviews: list[dict[str, Any]]) -> list[dict[str, Any]]: latest_by_user: dict[str, dict[str, Any]] = {} for review in reviews: user_login = review.get("user", {}).get("login") if not user_login: continue timestamp = review_timestamp(review) if user_login not in latest_by_user: latest_by_user[user_login] = review continue existing = latest_by_user[user_login] existing_timestamp = review_timestamp(existing) if timestamp is None: continue if existing_timestamp is None or timestamp > existing_timestamp: latest_by_user[user_login] = review return list(latest_by_user.values()) def filter_blocking_reviews( reviews: list[dict[str, Any]], review_requested_at: datetime | None, ) -> list[dict[str, Any]]: return [ review for review in dedupe_reviews(reviews) if is_blocking_review(review, review_requested_at) ] def is_merge_conflicting(pr: PrInfo) -> bool: return pr.mergeable == "CONFLICTING" or pr.merge_state == "DIRTY" async def fetch_review_context( pr_number: int, ) -> tuple[ list[dict[str, Any]], list[dict[str, Any]], list[dict[str, Any]], datetime | None, ]: issue_comments = await get_issue_comments(pr_number) review_request_at = latest_review_request_at(issue_comments) review_comments = await get_review_comments(pr_number) reviews = await get_reviews(pr_number) return issue_comments, review_comments, reviews, review_request_at def raise_on_human_feedback( issue_comments: list[dict[str, Any]], review_comments: list[dict[str, Any]], reviews: list[dict[str, Any]], review_request_at: datetime | None, ) -> None: human_issue_comments = filter_human_issue_comments(issue_comments) codex_review_comments = filter_codex_review_issue_comments(issue_comments) human_review_comments = filter_human_review_comments(review_comments) if human_issue_comments or human_review_comments or codex_review_comments: print("Review comments detected. Address before merge.") print( "Reminder: decide whether feedback stays in scope; defer if needed " "and note in your root-level update.", ) raise SystemExit(2) blocking_reviews = filter_blocking_reviews(reviews, review_request_at) if blocking_reviews: print("Review states/comments detected. Address before merge.") print( "Reminder: keep PR title/description aligned with the full scope " "when changes expand.", ) raise SystemExit(2) async def wait_for_codex(pr_number: int, checks_done: asyncio.Event) -> None: print("Waiting for review feedback...", flush=True) while True: ( issue_comments, review_comments, reviews, review_request_at, ) = await fetch_review_context(pr_number) bot_issue_comments = filter_codex_comments(issue_comments, review_request_at) bot_review_comments = filter_codex_comments(review_comments, review_request_at) bot_comments = bot_issue_comments + bot_review_comments raise_on_human_feedback( issue_comments, review_comments, reviews, review_request_at, ) if bot_comments: latest = max( bot_comments, key=lambda comment: parse_time(comment["created_at"]), ) body = sanitize_terminal_output(latest.get("body") or "").strip() if body: print("Codex left comments. Address feedback before merge.") print(body) raise SystemExit(2) if checks_done.is_set(): return await asyncio.sleep(POLL_SECONDS) async def wait_for_checks(head_sha: str, checks_done: asyncio.Event) -> None: print("Waiting for CI checks...", flush=True) empty_seconds = 0 while True: check_runs = await get_check_runs(head_sha) if not check_runs: empty_seconds += POLL_SECONDS if empty_seconds >= CHECKS_APPEAR_TIMEOUT_SECONDS: print( "No checks detected after 120s; check CI configuration", ) raise SystemExit(3) await asyncio.sleep(POLL_SECONDS) continue empty_seconds = 0 pending, failed, failures = summarize_checks(check_runs) if failed: print("Checks failed:") for failure in failures: print(f"- {failure}") raise SystemExit(3) if not pending: print("Checks passed") checks_done.set() return await asyncio.sleep(POLL_SECONDS) async def watch_pr() -> None: pr = await get_pr_info() if is_merge_conflicting(pr): print( "PR has merge conflicts. Resolve/rebase against main and push before " "running land_watch again.", ) raise SystemExit(5) head_sha = pr.head_sha checks_done = asyncio.Event() codex_task = asyncio.create_task(wait_for_codex(pr.number, checks_done)) checks_task = asyncio.create_task(wait_for_checks(head_sha, checks_done)) async def head_monitor() -> None: while True: current = await get_pr_info() if is_merge_conflicting(current): print( "PR has merge conflicts. Resolve/rebase against main and push " "before running land_watch again.", ) raise SystemExit(5) if current.head_sha != head_sha: print("PR head updated; pull/amend/force-push to retrigger CI") raise SystemExit(4) await asyncio.sleep(POLL_SECONDS) monitor_task = asyncio.create_task(head_monitor()) success_task = asyncio.gather(codex_task, checks_task) done, pending = await asyncio.wait( [monitor_task, success_task], return_when=asyncio.FIRST_COMPLETED, ) for task in pending: task.cancel() for task in done: exc = task.exception() if exc: raise exc if __name__ == "__main__": try: asyncio.run(watch_pr()) except SystemExit as exc: raise SystemExit(exc.code) from None ================================================ FILE: .codex/skills/linear/SKILL.md ================================================ --- name: linear description: | Use Symphony's `linear_graphql` client tool for raw Linear GraphQL operations such as comment editing and upload flows. --- # Linear GraphQL Use this skill for raw Linear GraphQL work during Symphony app-server sessions. ## Primary tool Use the `linear_graphql` client tool exposed by Symphony's app-server session. It reuses Symphony's configured Linear auth for the session. Tool input: ```json { "query": "query or mutation document", "variables": { "optional": "graphql variables object" } } ``` Tool behavior: - Send one GraphQL operation per tool call. - Treat a top-level `errors` array as a failed GraphQL operation even if the tool call itself completed. - Keep queries/mutations narrowly scoped; ask only for the fields you need. ## Discovering unfamiliar operations When you need an unfamiliar mutation, input type, or object field, use targeted introspection through `linear_graphql`. List mutation names: ```graphql query ListMutations { __type(name: "Mutation") { fields { name } } } ``` Inspect a specific input object: ```graphql query CommentCreateInputShape { __type(name: "CommentCreateInput") { inputFields { name type { kind name ofType { kind name } } } } } ``` ## Common workflows ### Query an issue by key, identifier, or id Use these progressively: - Start with `issue(id: $key)` when you have a ticket key such as `MT-686`. - Fall back to `issues(filter: ...)` when you need identifier search semantics. - Once you have the internal issue id, prefer `issue(id: $id)` for narrower reads. Lookup by issue key: ```graphql query IssueByKey($key: String!) { issue(id: $key) { id identifier title state { id name type } project { id name } branchName url description updatedAt links { nodes { id url title } } } } ``` Lookup by identifier filter: ```graphql query IssueByIdentifier($identifier: String!) { issues(filter: { identifier: { eq: $identifier } }, first: 1) { nodes { id identifier title state { id name type } project { id name } branchName url description updatedAt } } } ``` Resolve a key to an internal id: ```graphql query IssueByIdOrKey($id: String!) { issue(id: $id) { id identifier title } } ``` Read the issue once the internal id is known: ```graphql query IssueDetails($id: String!) { issue(id: $id) { id identifier title url description state { id name type } project { id name } attachments { nodes { id title url sourceType } } } } ``` ### Query team workflow states for an issue Use this before changing issue state when you need the exact `stateId`: ```graphql query IssueTeamStates($id: String!) { issue(id: $id) { id team { id key name states { nodes { id name type } } } } } ``` ### Edit an existing comment Use `commentUpdate` through `linear_graphql`: ```graphql mutation UpdateComment($id: String!, $body: String!) { commentUpdate(id: $id, input: { body: $body }) { success comment { id body } } } ``` ### Create a comment Use `commentCreate` through `linear_graphql`: ```graphql mutation CreateComment($issueId: String!, $body: String!) { commentCreate(input: { issueId: $issueId, body: $body }) { success comment { id url } } } ``` ### Move an issue to a different state Use `issueUpdate` with the destination `stateId`: ```graphql mutation MoveIssueToState($id: String!, $stateId: String!) { issueUpdate(id: $id, input: { stateId: $stateId }) { success issue { id identifier state { id name } } } } ``` ### Attach a GitHub PR to an issue Use the GitHub-specific attachment mutation when linking a PR: ```graphql mutation AttachGitHubPR($issueId: String!, $url: String!, $title: String) { attachmentLinkGitHubPR( issueId: $issueId url: $url title: $title linkKind: links ) { success attachment { id title url } } } ``` If you only need a plain URL attachment and do not care about GitHub-specific link metadata, use: ```graphql mutation AttachURL($issueId: String!, $url: String!, $title: String) { attachmentLinkURL(issueId: $issueId, url: $url, title: $title) { success attachment { id title url } } } ``` ### Introspection patterns used during schema discovery Use these when the exact field or mutation shape is unclear: ```graphql query QueryFields { __type(name: "Query") { fields { name } } } ``` ```graphql query IssueFieldArgs { __type(name: "Query") { fields { name args { name type { kind name ofType { kind name ofType { kind name } } } } } } } ``` ### Upload a video to a comment Do this in three steps: 1. Call `linear_graphql` with `fileUpload` to get `uploadUrl`, `assetUrl`, and any required upload headers. 2. Upload the local file bytes to `uploadUrl` with `curl -X PUT` and the exact headers returned by `fileUpload`. 3. Call `linear_graphql` again with `commentCreate` (or `commentUpdate`) and include the resulting `assetUrl` in the comment body. Useful mutations: ```graphql mutation FileUpload( $filename: String! $contentType: String! $size: Int! $makePublic: Boolean ) { fileUpload( filename: $filename contentType: $contentType size: $size makePublic: $makePublic ) { success uploadFile { uploadUrl assetUrl headers { key value } } } } ``` ## Usage rules - Use `linear_graphql` for comment edits, uploads, and ad-hoc Linear API queries. - Prefer the narrowest issue lookup that matches what you already know: key -> identifier search -> internal id. - For state transitions, fetch team states first and use the exact `stateId` instead of hardcoding names inside mutations. - Prefer `attachmentLinkGitHubPR` over a generic URL attachment when linking a GitHub PR to a Linear issue. - Do not introduce new raw-token shell helpers for GraphQL access. - If you need shell work for uploads, only use it for signed upload URLs returned by `fileUpload`; those URLs already carry the needed authorization. ================================================ FILE: .codex/skills/pull/SKILL.md ================================================ --- name: pull description: Pull latest origin/main into the current local branch and resolve merge conflicts (aka update-branch). Use when Codex needs to sync a feature branch with origin, perform a merge-based update (not rebase), and guide conflict resolution best practices. --- # Pull ## Workflow 1. Verify git status is clean or commit/stash changes before merging. 2. Ensure rerere is enabled locally: - `git config rerere.enabled true` - `git config rerere.autoupdate true` 3. Confirm remotes and branches: - Ensure the `origin` remote exists. - Ensure the current branch is the one to receive the merge. 4. Fetch latest refs: - `git fetch origin` 5. Sync the remote feature branch first: - `git pull --ff-only origin $(git branch --show-current)` - This pulls branch updates made remotely (for example, a GitHub auto-commit) before merging `origin/main`. 6. Merge in order: - Prefer `git -c merge.conflictstyle=zdiff3 merge origin/main` for clearer conflict context. 7. If conflicts appear, resolve them (see conflict guidance below), then: - `git add ` - `git commit` (or `git merge --continue` if the merge is paused) 8. Verify with project checks (follow repo policy in `AGENTS.md`). 9. Summarize the merge: - Call out the most challenging conflicts/files and how they were resolved. - Note any assumptions or follow-ups. ## Conflict Resolution Guidance (Best Practices) - Inspect context before editing: - Use `git status` to list conflicted files. - Use `git diff` or `git diff --merge` to see conflict hunks. - Use `git diff :1:path/to/file :2:path/to/file` and `git diff :1:path/to/file :3:path/to/file` to compare base vs ours/theirs for a file-level view of intent. - With `merge.conflictstyle=zdiff3`, conflict markers include: - `<<<<<<<` ours, `|||||||` base, `=======` split, `>>>>>>>` theirs. - Matching lines near the start/end are trimmed out of the conflict region, so focus on the differing core. - Summarize the intent of both changes, decide the semantically correct outcome, then edit: - State what each side is trying to achieve (bug fix, refactor, rename, behavior change). - Identify the shared goal, if any, and whether one side supersedes the other. - Decide the final behavior first; only then craft the code to match that decision. - Prefer preserving invariants, API contracts, and user-visible behavior unless the conflict clearly indicates a deliberate change. - Open files and understand intent on both sides before choosing a resolution. - Prefer minimal, intention-preserving edits: - Keep behavior consistent with the branch’s purpose. - Avoid accidental deletions or silent behavior changes. - Resolve one file at a time and rerun tests after each logical batch. - Use `ours/theirs` only when you are certain one side should win entirely. - For complex conflicts, search for related files or definitions to align with the rest of the codebase. - For generated files, resolve non-generated conflicts first, then regenerate: - Prefer resolving source files and handwritten logic before touching generated artifacts. - Run the CLI/tooling command that produced the generated file to recreate it cleanly, then stage the regenerated output. - For import conflicts where intent is unclear, accept both sides first: - Keep all candidate imports temporarily, finish the merge, then run lint/type checks to remove unused or incorrect imports safely. - After resolving, ensure no conflict markers remain: - `git diff --check` - When unsure, note assumptions and ask for confirmation before finalizing the merge. ## When To Ask The User (Keep To A Minimum) Do not ask for input unless there is no safe, reversible alternative. Prefer making a best-effort decision, documenting the rationale, and proceeding. Ask the user only when: - The correct resolution depends on product intent or behavior not inferable from code, tests, or nearby documentation. - The conflict crosses a user-visible contract, API surface, or migration where choosing incorrectly could break external consumers. - A conflict requires selecting between two mutually exclusive designs with equivalent technical merit and no clear local signal. - The merge introduces data loss, schema changes, or irreversible side effects without an obvious safe default. - The branch is not the intended target, or the remote/branch names do not exist and cannot be determined locally. Otherwise, proceed with the merge, explain the decision briefly in notes, and leave a clear, reviewable commit history. ================================================ FILE: .codex/skills/push/SKILL.md ================================================ --- name: push description: Push current branch changes to origin and create or update the corresponding pull request; use when asked to push, publish updates, or create pull request. --- # Push ## Prerequisites - `gh` CLI is installed and available in `PATH`. - `gh auth status` succeeds for GitHub operations in this repo. ## Goals - Push current branch changes to `origin` safely. - Create a PR if none exists for the branch, otherwise update the existing PR. - Keep branch history clean when remote has moved. ## Related Skills - `pull`: use this when push is rejected or sync is not clean (non-fast-forward, merge conflict risk, or stale branch). ## Steps 1. Identify current branch and confirm remote state. 2. Run local validation (`make -C elixir all`) before pushing. 3. Push branch to `origin` with upstream tracking if needed, using whatever remote URL is already configured. 4. If push is not clean/rejected: - If the failure is a non-fast-forward or sync problem, run the `pull` skill to merge `origin/main`, resolve conflicts, and rerun validation. - Push again; use `--force-with-lease` only when history was rewritten. - If the failure is due to auth, permissions, or workflow restrictions on the configured remote, stop and surface the exact error instead of rewriting remotes or switching protocols as a workaround. 5. Ensure a PR exists for the branch: - If no PR exists, create one. - If a PR exists and is open, update it. - If branch is tied to a closed/merged PR, create a new branch + PR. - Write a proper PR title that clearly describes the change outcome - For branch updates, explicitly reconsider whether current PR title still matches the latest scope; update it if it no longer does. 6. Write/update PR body explicitly using `.github/pull_request_template.md`: - Fill every section with concrete content for this change. - Replace all placeholder comments (``). - Keep bullets/checkboxes where template expects them. - If PR already exists, refresh body content so it reflects the total PR scope (all intended work on the branch), not just the newest commits, including newly added work, removed work, or changed approach. - Do not reuse stale description text from earlier iterations. 7. Validate PR body with `mix pr_body.check` and fix all reported issues. 8. Reply with the PR URL from `gh pr view`. ## Commands ```sh # Identify branch branch=$(git branch --show-current) # Minimal validation gate make -C elixir all # Initial push: respect the current origin remote. git push -u origin HEAD # If that failed because the remote moved, use the pull skill. After # pull-skill resolution and re-validation, retry the normal push: git push -u origin HEAD # If the configured remote rejects the push for auth, permissions, or workflow # restrictions, stop and surface the exact error. # Only if history was rewritten locally: git push --force-with-lease origin HEAD # Ensure a PR exists (create only if missing) pr_state=$(gh pr view --json state -q .state 2>/dev/null || true) if [ "$pr_state" = "MERGED" ] || [ "$pr_state" = "CLOSED" ]; then echo "Current branch is tied to a closed PR; create a new branch + PR." >&2 exit 1 fi # Write a clear, human-friendly title that summarizes the shipped change. pr_title="" if [ -z "$pr_state" ]; then gh pr create --title "$pr_title" else # Reconsider title on every branch update; edit if scope shifted. gh pr edit --title "$pr_title" fi # Write/edit PR body to match .github/pull_request_template.md before validation. # Example workflow: # 1) open the template and draft body content for this PR # 2) gh pr edit --body-file /tmp/pr_body.md # 3) for branch updates, re-check that title/body still match current diff tmp_pr_body=$(mktemp) gh pr view --json body -q .body > "$tmp_pr_body" (cd elixir && mix pr_body.check --file "$tmp_pr_body") rm -f "$tmp_pr_body" # Show PR URL for the reply gh pr view --json url -q .url ``` ## Notes - Do not use `--force`; only use `--force-with-lease` as the last resort. - Distinguish sync problems from remote auth/permission problems: - Use the `pull` skill for non-fast-forward or stale-branch issues. - Surface auth, permissions, or workflow restrictions directly instead of changing remotes or protocols. ================================================ FILE: .codex/worktree_init.sh ================================================ #!/usr/bin/env bash set -eo pipefail script_dir="$(cd "$(dirname "$0")" && pwd)" repo_root="$(cd "$script_dir/.." && pwd)" project_root="$repo_root/elixir" if ! command -v mise >/dev/null 2>&1; then echo "mise is required. Install it from https://mise.jdx.dev/getting-started.html" >&2 exit 1 fi cd "$project_root" mise trust make setup ================================================ FILE: .github/pull_request_template.md ================================================ #### Context #### TL;DR ** #### Summary - - - #### Alternatives - #### Test Plan - [ ] `make -C elixir all` - [ ] ================================================ FILE: .github/workflows/make-all.yml ================================================ name: make-all on: pull_request: push: branches: - main jobs: make-all: runs-on: ubuntu-latest defaults: run: working-directory: elixir steps: - name: Checkout uses: actions/checkout@v4 - name: Set up mise tools uses: jdx/mise-action@v3 with: install: true cache: true working_directory: elixir - name: Cache deps and build uses: actions/cache@v4 with: path: | elixir/deps elixir/_build key: ${{ runner.os }}-mix-${{ hashFiles('elixir/mix.lock') }} restore-keys: | ${{ runner.os }}-mix- - name: Verify make all run: make all ================================================ FILE: .github/workflows/pr-description-lint.yml ================================================ name: pr-description-lint on: pull_request: types: [opened, edited, reopened, synchronize, ready_for_review] jobs: validate-pr-description: runs-on: ubuntu-latest defaults: run: working-directory: elixir steps: - name: Checkout uses: actions/checkout@v4 - name: Set up mise tools uses: jdx/mise-action@v3 with: install: true cache: true working_directory: elixir - name: Validate PR description format env: PR_BODY_JSON: ${{ toJson(github.event.pull_request.body) }} run: | mix local.hex --force mix local.rebar --force mix deps.get printf '%s' "$PR_BODY_JSON" | jq -r '.' > /tmp/pr_body.md mix pr_body.check --file /tmp/pr_body.md ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: NOTICE ================================================ Copyright 2025 OpenAI Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software Distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and Limitations under the License. ================================================ FILE: README.md ================================================ # Symphony Symphony turns project work into isolated, autonomous implementation runs, allowing teams to manage work instead of supervising coding agents. [![Symphony demo video preview](.github/media/symphony-demo-poster.jpg)](.github/media/symphony-demo.mp4) _In this [demo video](.github/media/symphony-demo.mp4), Symphony monitors a Linear board for work and spawns agents to handle the tasks. The agents complete the tasks and provide proof of work: CI status, PR review feedback, complexity analysis, and walkthrough videos. When accepted, the agents land the PR safely. Engineers do not need to supervise Codex; they can manage the work at a higher level._ > [!WARNING] > Symphony is a low-key engineering preview for testing in trusted environments. ## Running Symphony ### Requirements Symphony works best in codebases that have adopted [harness engineering](https://openai.com/index/harness-engineering/). Symphony is the next step -- moving from managing coding agents to managing work that needs to get done. ### Option 1. Make your own Tell your favorite coding agent to build Symphony in a programming language of your choice: > Implement Symphony according to the following spec: > https://github.com/openai/symphony/blob/main/SPEC.md ### Option 2. Use our experimental reference implementation Check out [elixir/README.md](elixir/README.md) for instructions on how to set up your environment and run the Elixir-based Symphony implementation. You can also ask your favorite coding agent to help with the setup: > Set up Symphony for my repository based on > https://github.com/openai/symphony/blob/main/elixir/README.md --- ## License This project is licensed under the [Apache License 2.0](LICENSE). ================================================ FILE: SPEC.md ================================================ # Symphony Service Specification Status: Draft v1 (language-agnostic) Purpose: Define a service that orchestrates coding agents to get project work done. ## 1. Problem Statement Symphony is a long-running automation service that continuously reads work from an issue tracker (Linear in this specification version), creates an isolated workspace for each issue, and runs a coding agent session for that issue inside the workspace. The service solves four operational problems: - It turns issue execution into a repeatable daemon workflow instead of manual scripts. - It isolates agent execution in per-issue workspaces so agent commands run only inside per-issue workspace directories. - It keeps the workflow policy in-repo (`WORKFLOW.md`) so teams version the agent prompt and runtime settings with their code. - It provides enough observability to operate and debug multiple concurrent agent runs. Implementations are expected to document their trust and safety posture explicitly. This specification does not require a single approval, sandbox, or operator-confirmation policy; some implementations may target trusted environments with a high-trust configuration, while others may require stricter approvals or sandboxing. Important boundary: - Symphony is a scheduler/runner and tracker reader. - Ticket writes (state transitions, comments, PR links) are typically performed by the coding agent using tools available in the workflow/runtime environment. - A successful run may end at a workflow-defined handoff state (for example `Human Review`), not necessarily `Done`. ## 2. Goals and Non-Goals ### 2.1 Goals - Poll the issue tracker on a fixed cadence and dispatch work with bounded concurrency. - Maintain a single authoritative orchestrator state for dispatch, retries, and reconciliation. - Create deterministic per-issue workspaces and preserve them across runs. - Stop active runs when issue state changes make them ineligible. - Recover from transient failures with exponential backoff. - Load runtime behavior from a repository-owned `WORKFLOW.md` contract. - Expose operator-visible observability (at minimum structured logs). - Support restart recovery without requiring a persistent database. ### 2.2 Non-Goals - Rich web UI or multi-tenant control plane. - Prescribing a specific dashboard or terminal UI implementation. - General-purpose workflow engine or distributed job scheduler. - Built-in business logic for how to edit tickets, PRs, or comments. (That logic lives in the workflow prompt and agent tooling.) - Mandating strong sandbox controls beyond what the coding agent and host OS provide. - Mandating a single default approval, sandbox, or operator-confirmation posture for all implementations. ## 3. System Overview ### 3.1 Main Components 1. `Workflow Loader` - Reads `WORKFLOW.md`. - Parses YAML front matter and prompt body. - Returns `{config, prompt_template}`. 2. `Config Layer` - Exposes typed getters for workflow config values. - Applies defaults and environment variable indirection. - Performs validation used by the orchestrator before dispatch. 3. `Issue Tracker Client` - Fetches candidate issues in active states. - Fetches current states for specific issue IDs (reconciliation). - Fetches terminal-state issues during startup cleanup. - Normalizes tracker payloads into a stable issue model. 4. `Orchestrator` - Owns the poll tick. - Owns the in-memory runtime state. - Decides which issues to dispatch, retry, stop, or release. - Tracks session metrics and retry queue state. 5. `Workspace Manager` - Maps issue identifiers to workspace paths. - Ensures per-issue workspace directories exist. - Runs workspace lifecycle hooks. - Cleans workspaces for terminal issues. 6. `Agent Runner` - Creates workspace. - Builds prompt from issue + workflow template. - Launches the coding agent app-server client. - Streams agent updates back to the orchestrator. 7. `Status Surface` (optional) - Presents human-readable runtime status (for example terminal output, dashboard, or other operator-facing view). 8. `Logging` - Emits structured runtime logs to one or more configured sinks. ### 3.2 Abstraction Levels Symphony is easiest to port when kept in these layers: 1. `Policy Layer` (repo-defined) - `WORKFLOW.md` prompt body. - Team-specific rules for ticket handling, validation, and handoff. 2. `Configuration Layer` (typed getters) - Parses front matter into typed runtime settings. - Handles defaults, environment tokens, and path normalization. 3. `Coordination Layer` (orchestrator) - Polling loop, issue eligibility, concurrency, retries, reconciliation. 4. `Execution Layer` (workspace + agent subprocess) - Filesystem lifecycle, workspace preparation, coding-agent protocol. 5. `Integration Layer` (Linear adapter) - API calls and normalization for tracker data. 6. `Observability Layer` (logs + optional status surface) - Operator visibility into orchestrator and agent behavior. ### 3.3 External Dependencies - Issue tracker API (Linear for `tracker.kind: linear` in this specification version). - Local filesystem for workspaces and logs. - Optional workspace population tooling (for example Git CLI, if used). - Coding-agent executable that supports JSON-RPC-like app-server mode over stdio. - Host environment authentication for the issue tracker and coding agent. ## 4. Core Domain Model ### 4.1 Entities #### 4.1.1 Issue Normalized issue record used by orchestration, prompt rendering, and observability output. Fields: - `id` (string) - Stable tracker-internal ID. - `identifier` (string) - Human-readable ticket key (example: `ABC-123`). - `title` (string) - `description` (string or null) - `priority` (integer or null) - Lower numbers are higher priority in dispatch sorting. - `state` (string) - Current tracker state name. - `branch_name` (string or null) - Tracker-provided branch metadata if available. - `url` (string or null) - `labels` (list of strings) - Normalized to lowercase. - `blocked_by` (list of blocker refs) - Each blocker ref contains: - `id` (string or null) - `identifier` (string or null) - `state` (string or null) - `created_at` (timestamp or null) - `updated_at` (timestamp or null) #### 4.1.2 Workflow Definition Parsed `WORKFLOW.md` payload: - `config` (map) - YAML front matter root object. - `prompt_template` (string) - Markdown body after front matter, trimmed. #### 4.1.3 Service Config (Typed View) Typed runtime values derived from `WorkflowDefinition.config` plus environment resolution. Examples: - poll interval - workspace root - active and terminal issue states - concurrency limits - coding-agent executable/args/timeouts - workspace hooks #### 4.1.4 Workspace Filesystem workspace assigned to one issue identifier. Fields (logical): - `path` (workspace path; current runtime typically uses absolute paths, but relative roots are possible if configured without path separators) - `workspace_key` (sanitized issue identifier) - `created_now` (boolean, used to gate `after_create` hook) #### 4.1.5 Run Attempt One execution attempt for one issue. Fields (logical): - `issue_id` - `issue_identifier` - `attempt` (integer or null, `null` for first run, `>=1` for retries/continuation) - `workspace_path` - `started_at` - `status` - `error` (optional) #### 4.1.6 Live Session (Agent Session Metadata) State tracked while a coding-agent subprocess is running. Fields: - `session_id` (string, `-`) - `thread_id` (string) - `turn_id` (string) - `codex_app_server_pid` (string or null) - `last_codex_event` (string/enum or null) - `last_codex_timestamp` (timestamp or null) - `last_codex_message` (summarized payload) - `codex_input_tokens` (integer) - `codex_output_tokens` (integer) - `codex_total_tokens` (integer) - `last_reported_input_tokens` (integer) - `last_reported_output_tokens` (integer) - `last_reported_total_tokens` (integer) - `turn_count` (integer) - Number of coding-agent turns started within the current worker lifetime. #### 4.1.7 Retry Entry Scheduled retry state for an issue. Fields: - `issue_id` - `identifier` (best-effort human ID for status surfaces/logs) - `attempt` (integer, 1-based for retry queue) - `due_at_ms` (monotonic clock timestamp) - `timer_handle` (runtime-specific timer reference) - `error` (string or null) #### 4.1.8 Orchestrator Runtime State Single authoritative in-memory state owned by the orchestrator. Fields: - `poll_interval_ms` (current effective poll interval) - `max_concurrent_agents` (current effective global concurrency limit) - `running` (map `issue_id -> running entry`) - `claimed` (set of issue IDs reserved/running/retrying) - `retry_attempts` (map `issue_id -> RetryEntry`) - `completed` (set of issue IDs; bookkeeping only, not dispatch gating) - `codex_totals` (aggregate tokens + runtime seconds) - `codex_rate_limits` (latest rate-limit snapshot from agent events) ### 4.2 Stable Identifiers and Normalization Rules - `Issue ID` - Use for tracker lookups and internal map keys. - `Issue Identifier` - Use for human-readable logs and workspace naming. - `Workspace Key` - Derive from `issue.identifier` by replacing any character not in `[A-Za-z0-9._-]` with `_`. - Use the sanitized value for the workspace directory name. - `Normalized Issue State` - Compare states after `lowercase`. - `Session ID` - Compose from coding-agent `thread_id` and `turn_id` as `-`. ## 5. Workflow Specification (Repository Contract) ### 5.1 File Discovery and Path Resolution Workflow file path precedence: 1. Explicit application/runtime setting (set by CLI startup path). 2. Default: `WORKFLOW.md` in the current process working directory. Loader behavior: - If the file cannot be read, return `missing_workflow_file` error. - The workflow file is expected to be repository-owned and version-controlled. ### 5.2 File Format `WORKFLOW.md` is a Markdown file with optional YAML front matter. Design note: - `WORKFLOW.md` should be self-contained enough to describe and run different workflows (prompt, runtime settings, hooks, and tracker selection/config) without requiring out-of-band service-specific configuration. Parsing rules: - If file starts with `---`, parse lines until the next `---` as YAML front matter. - Remaining lines become the prompt body. - If front matter is absent, treat the entire file as prompt body and use an empty config map. - YAML front matter must decode to a map/object; non-map YAML is an error. - Prompt body is trimmed before use. Returned workflow object: - `config`: front matter root object (not nested under a `config` key). - `prompt_template`: trimmed Markdown body. ### 5.3 Front Matter Schema Top-level keys: - `tracker` - `polling` - `workspace` - `hooks` - `agent` - `codex` Unknown keys should be ignored for forward compatibility. Note: - The workflow front matter is extensible. Optional extensions may define additional top-level keys (for example `server`) without changing the core schema above. - Extensions should document their field schema, defaults, validation rules, and whether changes apply dynamically or require restart. - Common extension: `server.port` (integer) enables the optional HTTP server described in Section 13.7. #### 5.3.1 `tracker` (object) Fields: - `kind` (string) - Required for dispatch. - Current supported value: `linear` - `endpoint` (string) - Default for `tracker.kind == "linear"`: `https://api.linear.app/graphql` - `api_key` (string) - May be a literal token or `$VAR_NAME`. - Canonical environment variable for `tracker.kind == "linear"`: `LINEAR_API_KEY`. - If `$VAR_NAME` resolves to an empty string, treat the key as missing. - `project_slug` (string) - Required for dispatch when `tracker.kind == "linear"`. - `active_states` (list of strings) - Default: `Todo`, `In Progress` - `terminal_states` (list of strings) - Default: `Closed`, `Cancelled`, `Canceled`, `Duplicate`, `Done` #### 5.3.2 `polling` (object) Fields: - `interval_ms` (integer or string integer) - Default: `30000` - Changes should be re-applied at runtime and affect future tick scheduling without restart. #### 5.3.3 `workspace` (object) Fields: - `root` (path string or `$VAR`) - Default: `/symphony_workspaces` - `~` and strings containing path separators are expanded. - Bare strings without path separators are preserved as-is (relative roots are allowed but discouraged). #### 5.3.4 `hooks` (object) Fields: - `after_create` (multiline shell script string, optional) - Runs only when a workspace directory is newly created. - Failure aborts workspace creation. - `before_run` (multiline shell script string, optional) - Runs before each agent attempt after workspace preparation and before launching the coding agent. - Failure aborts the current attempt. - `after_run` (multiline shell script string, optional) - Runs after each agent attempt (success, failure, timeout, or cancellation) once the workspace exists. - Failure is logged but ignored. - `before_remove` (multiline shell script string, optional) - Runs before workspace deletion if the directory exists. - Failure is logged but ignored; cleanup still proceeds. - `timeout_ms` (integer, optional) - Default: `60000` - Applies to all workspace hooks. - Non-positive values should be treated as invalid and fall back to the default. - Changes should be re-applied at runtime for future hook executions. #### 5.3.5 `agent` (object) Fields: - `max_concurrent_agents` (integer or string integer) - Default: `10` - Changes should be re-applied at runtime and affect subsequent dispatch decisions. - `max_retry_backoff_ms` (integer or string integer) - Default: `300000` (5 minutes) - Changes should be re-applied at runtime and affect future retry scheduling. - `max_concurrent_agents_by_state` (map `state_name -> positive integer`) - Default: empty map. - State keys are normalized (`lowercase`) for lookup. - Invalid entries (non-positive or non-numeric) are ignored. #### 5.3.6 `codex` (object) Fields: For Codex-owned config values such as `approval_policy`, `thread_sandbox`, and `turn_sandbox_policy`, supported values are defined by the targeted Codex app-server version. Implementors should treat them as pass-through Codex config values rather than relying on a hand-maintained enum in this spec. To inspect the installed Codex schema, run `codex app-server generate-json-schema --out ` and inspect the relevant definitions referenced by `v2/ThreadStartParams.json` and `v2/TurnStartParams.json`. Implementations may validate these fields locally if they want stricter startup checks. - `command` (string shell command) - Default: `codex app-server` - The runtime launches this command via `bash -lc` in the workspace directory. - The launched process must speak a compatible app-server protocol over stdio. - `approval_policy` (Codex `AskForApproval` value) - Default: implementation-defined. - `thread_sandbox` (Codex `SandboxMode` value) - Default: implementation-defined. - `turn_sandbox_policy` (Codex `SandboxPolicy` value) - Default: implementation-defined. - `turn_timeout_ms` (integer) - Default: `3600000` (1 hour) - `read_timeout_ms` (integer) - Default: `5000` - `stall_timeout_ms` (integer) - Default: `300000` (5 minutes) - If `<= 0`, stall detection is disabled. ### 5.4 Prompt Template Contract The Markdown body of `WORKFLOW.md` is the per-issue prompt template. Rendering requirements: - Use a strict template engine (Liquid-compatible semantics are sufficient). - Unknown variables must fail rendering. - Unknown filters must fail rendering. Template input variables: - `issue` (object) - Includes all normalized issue fields, including labels and blockers. - `attempt` (integer or null) - `null`/absent on first attempt. - Integer on retry or continuation run. Fallback prompt behavior: - If the workflow prompt body is empty, the runtime may use a minimal default prompt (`You are working on an issue from Linear.`). - Workflow file read/parse failures are configuration/validation errors and should not silently fall back to a prompt. ### 5.5 Workflow Validation and Error Surface Error classes: - `missing_workflow_file` - `workflow_parse_error` - `workflow_front_matter_not_a_map` - `template_parse_error` (during prompt rendering) - `template_render_error` (unknown variable/filter, invalid interpolation) Dispatch gating behavior: - Workflow file read/YAML errors block new dispatches until fixed. - Template errors fail only the affected run attempt. ## 6. Configuration Specification ### 6.1 Source Precedence and Resolution Semantics Configuration precedence: 1. Workflow file path selection (runtime setting -> cwd default). 2. YAML front matter values. 3. Environment indirection via `$VAR_NAME` inside selected YAML values. 4. Built-in defaults. Value coercion semantics: - Path/command fields support: - `~` home expansion - `$VAR` expansion for env-backed path values - Apply expansion only to values intended to be local filesystem paths; do not rewrite URIs or arbitrary shell command strings. ### 6.2 Dynamic Reload Semantics Dynamic reload is required: - The software should watch `WORKFLOW.md` for changes. - On change, it should re-read and re-apply workflow config and prompt template without restart. - The software should attempt to adjust live behavior to the new config (for example polling cadence, concurrency limits, active/terminal states, codex settings, workspace paths/hooks, and prompt content for future runs). - Reloaded config applies to future dispatch, retry scheduling, reconciliation decisions, hook execution, and agent launches. - Implementations are not required to restart in-flight agent sessions automatically when config changes. - Extensions that manage their own listeners/resources (for example an HTTP server port change) may require restart unless the implementation explicitly supports live rebind. - Implementations should also re-validate/reload defensively during runtime operations (for example before dispatch) in case filesystem watch events are missed. - Invalid reloads should not crash the service; keep operating with the last known good effective configuration and emit an operator-visible error. ### 6.3 Dispatch Preflight Validation This validation is a scheduler preflight run before attempting to dispatch new work. It validates the workflow/config needed to poll and launch workers, not a full audit of all possible workflow behavior. Startup validation: - Validate configuration before starting the scheduling loop. - If startup validation fails, fail startup and emit an operator-visible error. Per-tick dispatch validation: - Re-validate before each dispatch cycle. - If validation fails, skip dispatch for that tick, keep reconciliation active, and emit an operator-visible error. Validation checks: - Workflow file can be loaded and parsed. - `tracker.kind` is present and supported. - `tracker.api_key` is present after `$` resolution. - `tracker.project_slug` is present when required by the selected tracker kind. - `codex.command` is present and non-empty. ### 6.4 Config Fields Summary (Cheat Sheet) This section is intentionally redundant so a coding agent can implement the config layer quickly. - `tracker.kind`: string, required, currently `linear` - `tracker.endpoint`: string, default `https://api.linear.app/graphql` when `tracker.kind=linear` - `tracker.api_key`: string or `$VAR`, canonical env `LINEAR_API_KEY` when `tracker.kind=linear` - `tracker.project_slug`: string, required when `tracker.kind=linear` - `tracker.active_states`: list of strings, default `["Todo", "In Progress"]` - `tracker.terminal_states`: list of strings, default `["Closed", "Cancelled", "Canceled", "Duplicate", "Done"]` - `polling.interval_ms`: integer, default `30000` - `workspace.root`: path, default `/symphony_workspaces` - `worker.ssh_hosts` (extension): list of SSH host strings, optional; when omitted, work runs locally - `worker.max_concurrent_agents_per_host` (extension): positive integer, optional; shared per-host cap applied across configured SSH hosts - `hooks.after_create`: shell script or null - `hooks.before_run`: shell script or null - `hooks.after_run`: shell script or null - `hooks.before_remove`: shell script or null - `hooks.timeout_ms`: integer, default `60000` - `agent.max_concurrent_agents`: integer, default `10` - `agent.max_turns`: integer, default `20` - `agent.max_retry_backoff_ms`: integer, default `300000` (5m) - `agent.max_concurrent_agents_by_state`: map of positive integers, default `{}` - `codex.command`: shell command string, default `codex app-server` - `codex.approval_policy`: Codex `AskForApproval` value, default implementation-defined - `codex.thread_sandbox`: Codex `SandboxMode` value, default implementation-defined - `codex.turn_sandbox_policy`: Codex `SandboxPolicy` value, default implementation-defined - `codex.turn_timeout_ms`: integer, default `3600000` - `codex.read_timeout_ms`: integer, default `5000` - `codex.stall_timeout_ms`: integer, default `300000` - `server.port` (extension): integer, optional; enables the optional HTTP server, `0` may be used for ephemeral local bind, and CLI `--port` overrides it ## 7. Orchestration State Machine The orchestrator is the only component that mutates scheduling state. All worker outcomes are reported back to it and converted into explicit state transitions. ### 7.1 Issue Orchestration States This is not the same as tracker states (`Todo`, `In Progress`, etc.). This is the service's internal claim state. 1. `Unclaimed` - Issue is not running and has no retry scheduled. 2. `Claimed` - Orchestrator has reserved the issue to prevent duplicate dispatch. - In practice, claimed issues are either `Running` or `RetryQueued`. 3. `Running` - Worker task exists and the issue is tracked in `running` map. 4. `RetryQueued` - Worker is not running, but a retry timer exists in `retry_attempts`. 5. `Released` - Claim removed because issue is terminal, non-active, missing, or retry path completed without re-dispatch. Important nuance: - A successful worker exit does not mean the issue is done forever. - The worker may continue through multiple back-to-back coding-agent turns before it exits. - After each normal turn completion, the worker re-checks the tracker issue state. - If the issue is still in an active state, the worker should start another turn on the same live coding-agent thread in the same workspace, up to `agent.max_turns`. - The first turn should use the full rendered task prompt. - Continuation turns should send only continuation guidance to the existing thread, not resend the original task prompt that is already present in thread history. - Once the worker exits normally, the orchestrator still schedules a short continuation retry (about 1 second) so it can re-check whether the issue remains active and needs another worker session. ### 7.2 Run Attempt Lifecycle A run attempt transitions through these phases: 1. `PreparingWorkspace` 2. `BuildingPrompt` 3. `LaunchingAgentProcess` 4. `InitializingSession` 5. `StreamingTurn` 6. `Finishing` 7. `Succeeded` 8. `Failed` 9. `TimedOut` 10. `Stalled` 11. `CanceledByReconciliation` Distinct terminal reasons are important because retry logic and logs differ. ### 7.3 Transition Triggers - `Poll Tick` - Reconcile active runs. - Validate config. - Fetch candidate issues. - Dispatch until slots are exhausted. - `Worker Exit (normal)` - Remove running entry. - Update aggregate runtime totals. - Schedule continuation retry (attempt `1`) after the worker exhausts or finishes its in-process turn loop. - `Worker Exit (abnormal)` - Remove running entry. - Update aggregate runtime totals. - Schedule exponential-backoff retry. - `Codex Update Event` - Update live session fields, token counters, and rate limits. - `Retry Timer Fired` - Re-fetch active candidates and attempt re-dispatch, or release claim if no longer eligible. - `Reconciliation State Refresh` - Stop runs whose issue states are terminal or no longer active. - `Stall Timeout` - Kill worker and schedule retry. ### 7.4 Idempotency and Recovery Rules - The orchestrator serializes state mutations through one authority to avoid duplicate dispatch. - `claimed` and `running` checks are required before launching any worker. - Reconciliation runs before dispatch on every tick. - Restart recovery is tracker-driven and filesystem-driven (no durable orchestrator DB required). - Startup terminal cleanup removes stale workspaces for issues already in terminal states. ## 8. Polling, Scheduling, and Reconciliation ### 8.1 Poll Loop At startup, the service validates config, performs startup cleanup, schedules an immediate tick, and then repeats every `polling.interval_ms`. The effective poll interval should be updated when workflow config changes are re-applied. Tick sequence: 1. Reconcile running issues. 2. Run dispatch preflight validation. 3. Fetch candidate issues from tracker using active states. 4. Sort issues by dispatch priority. 5. Dispatch eligible issues while slots remain. 6. Notify observability/status consumers of state changes. If per-tick validation fails, dispatch is skipped for that tick, but reconciliation still happens first. ### 8.2 Candidate Selection Rules An issue is dispatch-eligible only if all are true: - It has `id`, `identifier`, `title`, and `state`. - Its state is in `active_states` and not in `terminal_states`. - It is not already in `running`. - It is not already in `claimed`. - Global concurrency slots are available. - Per-state concurrency slots are available. - Blocker rule for `Todo` state passes: - If the issue state is `Todo`, do not dispatch when any blocker is non-terminal. Sorting order (stable intent): 1. `priority` ascending (1..4 are preferred; null/unknown sorts last) 2. `created_at` oldest first 3. `identifier` lexicographic tie-breaker ### 8.3 Concurrency Control Global limit: - `available_slots = max(max_concurrent_agents - running_count, 0)` Per-state limit: - `max_concurrent_agents_by_state[state]` if present (state key normalized) - otherwise fallback to global limit The runtime counts issues by their current tracked state in the `running` map. Optional SSH host limit: - When `worker.max_concurrent_agents_per_host` is set, each configured SSH host may run at most that many concurrent agents at once. - Hosts at that cap are skipped for new dispatch until capacity frees up. ### 8.4 Retry and Backoff Retry entry creation: - Cancel any existing retry timer for the same issue. - Store `attempt`, `identifier`, `error`, `due_at_ms`, and new timer handle. Backoff formula: - Normal continuation retries after a clean worker exit use a short fixed delay of `1000` ms. - Failure-driven retries use `delay = min(10000 * 2^(attempt - 1), agent.max_retry_backoff_ms)`. - Power is capped by the configured max retry backoff (default `300000` / 5m). Retry handling behavior: 1. Fetch active candidate issues (not all issues). 2. Find the specific issue by `issue_id`. 3. If not found, release claim. 4. If found and still candidate-eligible: - Dispatch if slots are available. - Otherwise requeue with error `no available orchestrator slots`. 5. If found but no longer active, release claim. Note: - Terminal-state workspace cleanup is handled by startup cleanup and active-run reconciliation (including terminal transitions for currently running issues). - Retry handling mainly operates on active candidates and releases claims when the issue is absent, rather than performing terminal cleanup itself. ### 8.5 Active Run Reconciliation Reconciliation runs every tick and has two parts. Part A: Stall detection - For each running issue, compute `elapsed_ms` since: - `last_codex_timestamp` if any event has been seen, else - `started_at` - If `elapsed_ms > codex.stall_timeout_ms`, terminate the worker and queue a retry. - If `stall_timeout_ms <= 0`, skip stall detection entirely. Part B: Tracker state refresh - Fetch current issue states for all running issue IDs. - For each running issue: - If tracker state is terminal: terminate worker and clean workspace. - If tracker state is still active: update the in-memory issue snapshot. - If tracker state is neither active nor terminal: terminate worker without workspace cleanup. - If state refresh fails, keep workers running and try again on the next tick. ### 8.6 Startup Terminal Workspace Cleanup When the service starts: 1. Query tracker for issues in terminal states. 2. For each returned issue identifier, remove the corresponding workspace directory. 3. If the terminal-issues fetch fails, log a warning and continue startup. This prevents stale terminal workspaces from accumulating after restarts. ## 9. Workspace Management and Safety ### 9.1 Workspace Layout Workspace root: - `workspace.root` (normalized path; the current config layer expands path-like values and preserves bare relative names) Per-issue workspace path: - `/` Workspace persistence: - Workspaces are reused across runs for the same issue. - Successful runs do not auto-delete workspaces. ### 9.2 Workspace Creation and Reuse Input: `issue.identifier` Algorithm summary: 1. Sanitize identifier to `workspace_key`. 2. Compute workspace path under workspace root. 3. Ensure the workspace path exists as a directory. 4. Mark `created_now=true` only if the directory was created during this call; otherwise `created_now=false`. 5. If `created_now=true`, run `after_create` hook if configured. Notes: - This section does not assume any specific repository/VCS workflow. - Workspace preparation beyond directory creation (for example dependency bootstrap, checkout/sync, code generation) is implementation-defined and is typically handled via hooks. ### 9.3 Optional Workspace Population (Implementation-Defined) The spec does not require any built-in VCS or repository bootstrap behavior. Implementations may populate or synchronize the workspace using implementation-defined logic and/or hooks (for example `after_create` and/or `before_run`). Failure handling: - Workspace population/synchronization failures return an error for the current attempt. - If failure happens while creating a brand-new workspace, implementations may remove the partially prepared directory. - Reused workspaces should not be destructively reset on population failure unless that policy is explicitly chosen and documented. ### 9.4 Workspace Hooks Supported hooks: - `hooks.after_create` - `hooks.before_run` - `hooks.after_run` - `hooks.before_remove` Execution contract: - Execute in a local shell context appropriate to the host OS, with the workspace directory as `cwd`. - On POSIX systems, `sh -lc {@inner_content} """ end @spec app(map()) :: Phoenix.LiveView.Rendered.t() def app(assigns) do ~H"""
{@inner_content}
""" end end ================================================ FILE: elixir/lib/symphony_elixir_web/controllers/observability_api_controller.ex ================================================ defmodule SymphonyElixirWeb.ObservabilityApiController do @moduledoc """ JSON API for Symphony observability data. """ use Phoenix.Controller, formats: [:json] alias Plug.Conn alias SymphonyElixirWeb.{Endpoint, Presenter} @spec state(Conn.t(), map()) :: Conn.t() def state(conn, _params) do json(conn, Presenter.state_payload(orchestrator(), snapshot_timeout_ms())) end @spec issue(Conn.t(), map()) :: Conn.t() def issue(conn, %{"issue_identifier" => issue_identifier}) do case Presenter.issue_payload(issue_identifier, orchestrator(), snapshot_timeout_ms()) do {:ok, payload} -> json(conn, payload) {:error, :issue_not_found} -> error_response(conn, 404, "issue_not_found", "Issue not found") end end @spec refresh(Conn.t(), map()) :: Conn.t() def refresh(conn, _params) do case Presenter.refresh_payload(orchestrator()) do {:ok, payload} -> conn |> put_status(202) |> json(payload) {:error, :unavailable} -> error_response(conn, 503, "orchestrator_unavailable", "Orchestrator is unavailable") end end @spec method_not_allowed(Conn.t(), map()) :: Conn.t() def method_not_allowed(conn, _params) do error_response(conn, 405, "method_not_allowed", "Method not allowed") end @spec not_found(Conn.t(), map()) :: Conn.t() def not_found(conn, _params) do error_response(conn, 404, "not_found", "Route not found") end defp error_response(conn, status, code, message) do conn |> put_status(status) |> json(%{error: %{code: code, message: message}}) end defp orchestrator do Endpoint.config(:orchestrator) || SymphonyElixir.Orchestrator end defp snapshot_timeout_ms do Endpoint.config(:snapshot_timeout_ms) || 15_000 end end ================================================ FILE: elixir/lib/symphony_elixir_web/controllers/static_asset_controller.ex ================================================ defmodule SymphonyElixirWeb.StaticAssetController do @moduledoc """ Serves the dashboard's embedded CSS and JavaScript assets. """ use Phoenix.Controller, formats: [] alias Plug.Conn alias SymphonyElixirWeb.StaticAssets @spec dashboard_css(Conn.t(), map()) :: Conn.t() def dashboard_css(conn, _params), do: serve(conn, "/dashboard.css") @spec phoenix_html_js(Conn.t(), map()) :: Conn.t() def phoenix_html_js(conn, _params), do: serve(conn, "/vendor/phoenix_html/phoenix_html.js") @spec phoenix_js(Conn.t(), map()) :: Conn.t() def phoenix_js(conn, _params), do: serve(conn, "/vendor/phoenix/phoenix.js") @spec phoenix_live_view_js(Conn.t(), map()) :: Conn.t() def phoenix_live_view_js(conn, _params), do: serve(conn, "/vendor/phoenix_live_view/phoenix_live_view.js") defp serve(conn, path) do case StaticAssets.fetch(path) do {:ok, content_type, body} -> conn |> put_resp_content_type(content_type) |> put_resp_header("cache-control", "public, max-age=31536000") |> send_resp(200, body) :error -> send_resp(conn, 404, "Not Found") end end end ================================================ FILE: elixir/lib/symphony_elixir_web/endpoint.ex ================================================ defmodule SymphonyElixirWeb.Endpoint do @moduledoc """ Phoenix endpoint for Symphony's optional observability UI and API. """ use Phoenix.Endpoint, otp_app: :symphony_elixir @session_options [ store: :cookie, key: "_symphony_elixir_key", signing_salt: "symphony-session" ] socket("/live", Phoenix.LiveView.Socket, websocket: [connect_info: [session: @session_options]], longpoll: false ) plug(Plug.RequestId) plug(Plug.Telemetry, event_prefix: [:phoenix, :endpoint]) plug(Plug.Parsers, parsers: [:urlencoded, :multipart, :json], pass: ["*/*"], json_decoder: Jason ) plug(Plug.MethodOverride) plug(Plug.Head) plug(Plug.Session, @session_options) plug(SymphonyElixirWeb.Router) end ================================================ FILE: elixir/lib/symphony_elixir_web/error_html.ex ================================================ defmodule SymphonyElixirWeb.ErrorHTML do @moduledoc false @spec render(String.t(), map()) :: String.t() def render(template, _assigns) do Phoenix.Controller.status_message_from_template(template) end end ================================================ FILE: elixir/lib/symphony_elixir_web/error_json.ex ================================================ defmodule SymphonyElixirWeb.ErrorJSON do @moduledoc false @spec render(String.t(), map()) :: map() def render(template, _assigns) do %{error: %{code: "request_failed", message: Phoenix.Controller.status_message_from_template(template)}} end end ================================================ FILE: elixir/lib/symphony_elixir_web/live/dashboard_live.ex ================================================ defmodule SymphonyElixirWeb.DashboardLive do @moduledoc """ Live observability dashboard for Symphony. """ use Phoenix.LiveView, layout: {SymphonyElixirWeb.Layouts, :app} alias SymphonyElixirWeb.{Endpoint, ObservabilityPubSub, Presenter} @runtime_tick_ms 1_000 @impl true def mount(_params, _session, socket) do socket = socket |> assign(:payload, load_payload()) |> assign(:now, DateTime.utc_now()) if connected?(socket) do :ok = ObservabilityPubSub.subscribe() schedule_runtime_tick() end {:ok, socket} end @impl true def handle_info(:runtime_tick, socket) do schedule_runtime_tick() {:noreply, assign(socket, :now, DateTime.utc_now())} end @impl true def handle_info(:observability_updated, socket) do {:noreply, socket |> assign(:payload, load_payload()) |> assign(:now, DateTime.utc_now())} end @impl true def render(assigns) do ~H"""

Symphony Observability

Operations Dashboard

Current state, retry pressure, token usage, and orchestration health for the active Symphony runtime.

Live Offline
<%= if @payload[:error] do %>

Snapshot unavailable

<%= @payload.error.code %>: <%= @payload.error.message %>

<% else %>

Running

<%= @payload.counts.running %>

Active issue sessions in the current runtime.

Retrying

<%= @payload.counts.retrying %>

Issues waiting for the next retry window.

Total tokens

<%= format_int(@payload.codex_totals.total_tokens) %>

In <%= format_int(@payload.codex_totals.input_tokens) %> / Out <%= format_int(@payload.codex_totals.output_tokens) %>

Runtime

<%= format_runtime_seconds(total_runtime_seconds(@payload, @now)) %>

Total Codex runtime across completed and active sessions.

Rate limits

Latest upstream rate-limit snapshot, when available.

<%= pretty_value(@payload.rate_limits) %>

Running sessions

Active issues, last known agent activity, and token usage.

<%= if @payload.running == [] do %>

No active sessions.

<% else %>
Issue State Session Runtime / turns Codex update Tokens
<%= entry.issue_identifier %> JSON details
<%= entry.state %>
<%= if entry.session_id do %> <% else %> n/a <% end %>
<%= format_runtime_and_turns(entry.started_at, entry.turn_count, @now) %>
<%= entry.last_message || to_string(entry.last_event || "n/a") %> <%= entry.last_event || "n/a" %> <%= if entry.last_event_at do %> · <%= entry.last_event_at %> <% end %>
Total: <%= format_int(entry.tokens.total_tokens) %> In <%= format_int(entry.tokens.input_tokens) %> / Out <%= format_int(entry.tokens.output_tokens) %>
<% end %>

Retry queue

Issues waiting for the next retry window.

<%= if @payload.retrying == [] do %>

No issues are currently backing off.

<% else %>
Issue Attempt Due at Error
<%= entry.issue_identifier %> JSON details
<%= entry.attempt %> <%= entry.due_at || "n/a" %> <%= entry.error || "n/a" %>
<% end %>
<% end %>
""" end defp load_payload do Presenter.state_payload(orchestrator(), snapshot_timeout_ms()) end defp orchestrator do Endpoint.config(:orchestrator) || SymphonyElixir.Orchestrator end defp snapshot_timeout_ms do Endpoint.config(:snapshot_timeout_ms) || 15_000 end defp completed_runtime_seconds(payload) do payload.codex_totals.seconds_running || 0 end defp total_runtime_seconds(payload, now) do completed_runtime_seconds(payload) + Enum.reduce(payload.running, 0, fn entry, total -> total + runtime_seconds_from_started_at(entry.started_at, now) end) end defp format_runtime_and_turns(started_at, turn_count, now) when is_integer(turn_count) and turn_count > 0 do "#{format_runtime_seconds(runtime_seconds_from_started_at(started_at, now))} / #{turn_count}" end defp format_runtime_and_turns(started_at, _turn_count, now), do: format_runtime_seconds(runtime_seconds_from_started_at(started_at, now)) defp format_runtime_seconds(seconds) when is_number(seconds) do whole_seconds = max(trunc(seconds), 0) mins = div(whole_seconds, 60) secs = rem(whole_seconds, 60) "#{mins}m #{secs}s" end defp runtime_seconds_from_started_at(%DateTime{} = started_at, %DateTime{} = now) do DateTime.diff(now, started_at, :second) end defp runtime_seconds_from_started_at(started_at, %DateTime{} = now) when is_binary(started_at) do case DateTime.from_iso8601(started_at) do {:ok, parsed, _offset} -> runtime_seconds_from_started_at(parsed, now) _ -> 0 end end defp runtime_seconds_from_started_at(_started_at, _now), do: 0 defp format_int(value) when is_integer(value) do value |> Integer.to_string() |> String.reverse() |> String.replace(~r/.{3}(?=.)/, "\\0,") |> String.reverse() end defp format_int(_value), do: "n/a" defp state_badge_class(state) do base = "state-badge" normalized = state |> to_string() |> String.downcase() cond do String.contains?(normalized, ["progress", "running", "active"]) -> "#{base} state-badge-active" String.contains?(normalized, ["blocked", "error", "failed"]) -> "#{base} state-badge-danger" String.contains?(normalized, ["todo", "queued", "pending", "retry"]) -> "#{base} state-badge-warning" true -> base end end defp schedule_runtime_tick do Process.send_after(self(), :runtime_tick, @runtime_tick_ms) end defp pretty_value(nil), do: "n/a" defp pretty_value(value), do: inspect(value, pretty: true, limit: :infinity) end ================================================ FILE: elixir/lib/symphony_elixir_web/observability_pubsub.ex ================================================ defmodule SymphonyElixirWeb.ObservabilityPubSub do @moduledoc """ PubSub helpers for observability dashboard updates. """ @pubsub SymphonyElixir.PubSub @topic "observability:dashboard" @update_message :observability_updated @spec subscribe() :: :ok | {:error, term()} def subscribe do Phoenix.PubSub.subscribe(@pubsub, @topic) end @spec broadcast_update() :: :ok def broadcast_update do case Process.whereis(@pubsub) do pid when is_pid(pid) -> Phoenix.PubSub.broadcast(@pubsub, @topic, @update_message) _ -> :ok end end end ================================================ FILE: elixir/lib/symphony_elixir_web/presenter.ex ================================================ defmodule SymphonyElixirWeb.Presenter do @moduledoc """ Shared projections for the observability API and dashboard. """ alias SymphonyElixir.{Config, Orchestrator, StatusDashboard} @spec state_payload(GenServer.name(), timeout()) :: map() def state_payload(orchestrator, snapshot_timeout_ms) do generated_at = DateTime.utc_now() |> DateTime.truncate(:second) |> DateTime.to_iso8601() case Orchestrator.snapshot(orchestrator, snapshot_timeout_ms) do %{} = snapshot -> %{ generated_at: generated_at, counts: %{ running: length(snapshot.running), retrying: length(snapshot.retrying) }, running: Enum.map(snapshot.running, &running_entry_payload/1), retrying: Enum.map(snapshot.retrying, &retry_entry_payload/1), codex_totals: snapshot.codex_totals, rate_limits: snapshot.rate_limits } :timeout -> %{generated_at: generated_at, error: %{code: "snapshot_timeout", message: "Snapshot timed out"}} :unavailable -> %{generated_at: generated_at, error: %{code: "snapshot_unavailable", message: "Snapshot unavailable"}} end end @spec issue_payload(String.t(), GenServer.name(), timeout()) :: {:ok, map()} | {:error, :issue_not_found} def issue_payload(issue_identifier, orchestrator, snapshot_timeout_ms) when is_binary(issue_identifier) do case Orchestrator.snapshot(orchestrator, snapshot_timeout_ms) do %{} = snapshot -> running = Enum.find(snapshot.running, &(&1.identifier == issue_identifier)) retry = Enum.find(snapshot.retrying, &(&1.identifier == issue_identifier)) if is_nil(running) and is_nil(retry) do {:error, :issue_not_found} else {:ok, issue_payload_body(issue_identifier, running, retry)} end _ -> {:error, :issue_not_found} end end @spec refresh_payload(GenServer.name()) :: {:ok, map()} | {:error, :unavailable} def refresh_payload(orchestrator) do case Orchestrator.request_refresh(orchestrator) do :unavailable -> {:error, :unavailable} payload -> {:ok, Map.update!(payload, :requested_at, &DateTime.to_iso8601/1)} end end defp issue_payload_body(issue_identifier, running, retry) do %{ issue_identifier: issue_identifier, issue_id: issue_id_from_entries(running, retry), status: issue_status(running, retry), workspace: %{ path: workspace_path(issue_identifier, running, retry), host: workspace_host(running, retry) }, attempts: %{ restart_count: restart_count(retry), current_retry_attempt: retry_attempt(retry) }, running: running && running_issue_payload(running), retry: retry && retry_issue_payload(retry), logs: %{ codex_session_logs: [] }, recent_events: (running && recent_events_payload(running)) || [], last_error: retry && retry.error, tracked: %{} } end defp issue_id_from_entries(running, retry), do: (running && running.issue_id) || (retry && retry.issue_id) defp restart_count(retry), do: max(retry_attempt(retry) - 1, 0) defp retry_attempt(nil), do: 0 defp retry_attempt(retry), do: retry.attempt || 0 defp issue_status(_running, nil), do: "running" defp issue_status(nil, _retry), do: "retrying" defp issue_status(_running, _retry), do: "running" defp running_entry_payload(entry) do %{ issue_id: entry.issue_id, issue_identifier: entry.identifier, state: entry.state, worker_host: Map.get(entry, :worker_host), workspace_path: Map.get(entry, :workspace_path), session_id: entry.session_id, turn_count: Map.get(entry, :turn_count, 0), last_event: entry.last_codex_event, last_message: summarize_message(entry.last_codex_message), started_at: iso8601(entry.started_at), last_event_at: iso8601(entry.last_codex_timestamp), tokens: %{ input_tokens: entry.codex_input_tokens, output_tokens: entry.codex_output_tokens, total_tokens: entry.codex_total_tokens } } end defp retry_entry_payload(entry) do %{ issue_id: entry.issue_id, issue_identifier: entry.identifier, attempt: entry.attempt, due_at: due_at_iso8601(entry.due_in_ms), error: entry.error, worker_host: Map.get(entry, :worker_host), workspace_path: Map.get(entry, :workspace_path) } end defp running_issue_payload(running) do %{ worker_host: Map.get(running, :worker_host), workspace_path: Map.get(running, :workspace_path), session_id: running.session_id, turn_count: Map.get(running, :turn_count, 0), state: running.state, started_at: iso8601(running.started_at), last_event: running.last_codex_event, last_message: summarize_message(running.last_codex_message), last_event_at: iso8601(running.last_codex_timestamp), tokens: %{ input_tokens: running.codex_input_tokens, output_tokens: running.codex_output_tokens, total_tokens: running.codex_total_tokens } } end defp retry_issue_payload(retry) do %{ attempt: retry.attempt, due_at: due_at_iso8601(retry.due_in_ms), error: retry.error, worker_host: Map.get(retry, :worker_host), workspace_path: Map.get(retry, :workspace_path) } end defp workspace_path(issue_identifier, running, retry) do (running && Map.get(running, :workspace_path)) || (retry && Map.get(retry, :workspace_path)) || Path.join(Config.settings!().workspace.root, issue_identifier) end defp workspace_host(running, retry) do (running && Map.get(running, :worker_host)) || (retry && Map.get(retry, :worker_host)) end defp recent_events_payload(running) do [ %{ at: iso8601(running.last_codex_timestamp), event: running.last_codex_event, message: summarize_message(running.last_codex_message) } ] |> Enum.reject(&is_nil(&1.at)) end defp summarize_message(nil), do: nil defp summarize_message(message), do: StatusDashboard.humanize_codex_message(message) defp due_at_iso8601(due_in_ms) when is_integer(due_in_ms) do DateTime.utc_now() |> DateTime.add(div(due_in_ms, 1_000), :second) |> DateTime.truncate(:second) |> DateTime.to_iso8601() end defp due_at_iso8601(_due_in_ms), do: nil defp iso8601(%DateTime{} = datetime) do datetime |> DateTime.truncate(:second) |> DateTime.to_iso8601() end defp iso8601(_datetime), do: nil end ================================================ FILE: elixir/lib/symphony_elixir_web/router.ex ================================================ defmodule SymphonyElixirWeb.Router do @moduledoc """ Router for Symphony's observability dashboard and API. """ use Phoenix.Router import Phoenix.LiveView.Router pipeline :browser do plug(:fetch_session) plug(:fetch_live_flash) plug(:put_root_layout, html: {SymphonyElixirWeb.Layouts, :root}) plug(:protect_from_forgery) plug(:put_secure_browser_headers) end scope "/", SymphonyElixirWeb do get("/dashboard.css", StaticAssetController, :dashboard_css) get("/vendor/phoenix_html/phoenix_html.js", StaticAssetController, :phoenix_html_js) get("/vendor/phoenix/phoenix.js", StaticAssetController, :phoenix_js) get("/vendor/phoenix_live_view/phoenix_live_view.js", StaticAssetController, :phoenix_live_view_js) end scope "/", SymphonyElixirWeb do pipe_through(:browser) live("/", DashboardLive, :index) end scope "/", SymphonyElixirWeb do get("/api/v1/state", ObservabilityApiController, :state) match(:*, "/", ObservabilityApiController, :method_not_allowed) match(:*, "/api/v1/state", ObservabilityApiController, :method_not_allowed) post("/api/v1/refresh", ObservabilityApiController, :refresh) match(:*, "/api/v1/refresh", ObservabilityApiController, :method_not_allowed) get("/api/v1/:issue_identifier", ObservabilityApiController, :issue) match(:*, "/api/v1/:issue_identifier", ObservabilityApiController, :method_not_allowed) match(:*, "/*path", ObservabilityApiController, :not_found) end end ================================================ FILE: elixir/lib/symphony_elixir_web/static_assets.ex ================================================ defmodule SymphonyElixirWeb.StaticAssets do @moduledoc false @dashboard_css_path Path.expand("../../priv/static/dashboard.css", __DIR__) @phoenix_html_js_path Application.app_dir(:phoenix_html, "priv/static/phoenix_html.js") @phoenix_js_path Application.app_dir(:phoenix, "priv/static/phoenix.js") @phoenix_live_view_js_path Application.app_dir(:phoenix_live_view, "priv/static/phoenix_live_view.js") @external_resource @dashboard_css_path @external_resource @phoenix_html_js_path @external_resource @phoenix_js_path @external_resource @phoenix_live_view_js_path @dashboard_css File.read!(@dashboard_css_path) @phoenix_html_js File.read!(@phoenix_html_js_path) @phoenix_js File.read!(@phoenix_js_path) @phoenix_live_view_js File.read!(@phoenix_live_view_js_path) @assets %{ "/dashboard.css" => {"text/css", @dashboard_css}, "/vendor/phoenix_html/phoenix_html.js" => {"application/javascript", @phoenix_html_js}, "/vendor/phoenix/phoenix.js" => {"application/javascript", @phoenix_js}, "/vendor/phoenix_live_view/phoenix_live_view.js" => {"application/javascript", @phoenix_live_view_js} } @spec fetch(String.t()) :: {:ok, String.t(), binary()} | :error def fetch(path) when is_binary(path) do case Map.fetch(@assets, path) do {:ok, {content_type, body}} -> {:ok, content_type, body} :error -> :error end end end ================================================ FILE: elixir/mise.toml ================================================ [tools] erlang = "28" elixir = "1.19.5-otp-28" ================================================ FILE: elixir/mix.exs ================================================ defmodule SymphonyElixir.MixProject do use Mix.Project def project do [ app: :symphony_elixir, version: "0.1.0", elixir: "~> 1.19", compilers: [:phoenix_live_view] ++ Mix.compilers(), start_permanent: Mix.env() == :prod, test_coverage: [ summary: [ threshold: 100 ], ignore_modules: [ SymphonyElixir.Config, SymphonyElixir.Linear.Client, SymphonyElixir.SpecsCheck, SymphonyElixir.Orchestrator, SymphonyElixir.Orchestrator.State, SymphonyElixir.AgentRunner, SymphonyElixir.CLI, SymphonyElixir.Codex.AppServer, SymphonyElixir.Codex.DynamicTool, SymphonyElixir.HttpServer, SymphonyElixir.StatusDashboard, SymphonyElixir.LogFile, SymphonyElixir.Workspace, SymphonyElixirWeb.DashboardLive, SymphonyElixirWeb.Endpoint, SymphonyElixirWeb.ErrorHTML, SymphonyElixirWeb.ErrorJSON, SymphonyElixirWeb.Layouts, SymphonyElixirWeb.ObservabilityApiController, SymphonyElixirWeb.Presenter, SymphonyElixirWeb.StaticAssetController, SymphonyElixirWeb.StaticAssets, SymphonyElixirWeb.Router, SymphonyElixirWeb.Router.Helpers ] ], test_ignore_filters: [ "test/support/snapshot_support.exs", "test/support/test_support.exs" ], dialyzer: [ plt_add_apps: [:mix] ], escript: escript(), aliases: aliases(), deps: deps() ] end # Run "mix help compile.app" to learn about applications. def application do [ mod: {SymphonyElixir.Application, []}, extra_applications: [:logger] ] end # Run "mix help deps" to learn about dependencies. defp deps do [ {:bandit, "~> 1.8"}, {:floki, ">= 0.30.0", only: :test}, {:lazy_html, ">= 0.1.0", only: :test}, {:phoenix, "~> 1.8.0"}, {:phoenix_html, "~> 4.2"}, {:phoenix_live_view, "~> 1.1.0"}, {:req, "~> 0.5"}, {:jason, "~> 1.4"}, {:yaml_elixir, "~> 2.12"}, {:solid, "~> 1.2"}, {:ecto, "~> 3.13"}, {:credo, "~> 1.7", only: [:dev, :test], runtime: false}, {:dialyxir, "~> 1.4", only: [:dev], runtime: false} ] end defp aliases do [ setup: ["deps.get"], build: ["escript.build"], lint: ["specs.check", "credo --strict"] ] end defp escript do [ app: nil, main_module: SymphonyElixir.CLI, name: "symphony", path: "bin/symphony" ] end end ================================================ FILE: elixir/priv/static/dashboard.css ================================================ :root { color-scheme: light; --page: #f7f7f8; --page-soft: #fbfbfc; --page-deep: #ececf1; --card: rgba(255, 255, 255, 0.94); --card-muted: #f3f4f6; --ink: #202123; --muted: #6e6e80; --line: #ececf1; --line-strong: #d9d9e3; --accent: #10a37f; --accent-ink: #0f513f; --accent-soft: #e8faf4; --danger: #b42318; --danger-soft: #fef3f2; --shadow-sm: 0 1px 2px rgba(16, 24, 40, 0.05); --shadow-lg: 0 20px 50px rgba(15, 23, 42, 0.08); } * { box-sizing: border-box; } html { background: var(--page); } body { margin: 0; min-height: 100vh; background: radial-gradient(circle at top, rgba(16, 163, 127, 0.12) 0%, rgba(16, 163, 127, 0) 30%), linear-gradient(180deg, var(--page-soft) 0%, var(--page) 24%, #f3f4f6 100%); color: var(--ink); font-family: "Sohne", "SF Pro Text", "Helvetica Neue", "Segoe UI", sans-serif; line-height: 1.5; } a { color: var(--ink); text-decoration: none; transition: color 140ms ease; } a:hover { color: var(--accent); } button { appearance: none; border: 1px solid var(--accent); background: var(--accent); color: white; border-radius: 999px; padding: 0.72rem 1.08rem; cursor: pointer; font: inherit; font-weight: 600; letter-spacing: -0.01em; box-shadow: 0 8px 20px rgba(16, 163, 127, 0.18); transition: transform 140ms ease, box-shadow 140ms ease, background 140ms ease, border-color 140ms ease; } button:hover { transform: translateY(-1px); box-shadow: 0 12px 24px rgba(16, 163, 127, 0.22); } button.secondary { background: var(--card); color: var(--ink); border-color: var(--line-strong); box-shadow: var(--shadow-sm); } button.secondary:hover { box-shadow: 0 6px 16px rgba(15, 23, 42, 0.08); } .subtle-button { appearance: none; border: 1px solid var(--line-strong); background: rgba(255, 255, 255, 0.72); color: var(--muted); border-radius: 999px; padding: 0.34rem 0.72rem; cursor: pointer; font: inherit; font-size: 0.82rem; font-weight: 600; letter-spacing: 0.01em; box-shadow: none; transition: background 140ms ease, border-color 140ms ease, color 140ms ease; } .subtle-button:hover { transform: none; box-shadow: none; background: white; border-color: var(--muted); color: var(--ink); } pre { margin: 0; white-space: pre-wrap; word-break: break-word; } code, pre, .mono { font-family: "Sohne Mono", "SFMono-Regular", "SF Mono", Consolas, "Liberation Mono", monospace; } .mono, .numeric { font-variant-numeric: tabular-nums slashed-zero; font-feature-settings: "tnum" 1, "zero" 1; } .app-shell { max-width: 1280px; margin: 0 auto; padding: 2rem 1rem 3.5rem; } .dashboard-shell { display: grid; gap: 1rem; } .hero-card, .section-card, .metric-card, .error-card { background: var(--card); border: 1px solid rgba(217, 217, 227, 0.82); box-shadow: var(--shadow-sm); backdrop-filter: blur(18px); } .hero-card { border-radius: 28px; padding: clamp(1.25rem, 3vw, 2rem); box-shadow: var(--shadow-lg); } .hero-grid { display: grid; grid-template-columns: minmax(0, 1fr) auto; gap: 1.25rem; align-items: start; } .eyebrow { margin: 0; color: var(--muted); text-transform: uppercase; letter-spacing: 0.08em; font-size: 0.76rem; font-weight: 600; } .hero-title { margin: 0.35rem 0 0; font-size: clamp(2rem, 4vw, 3.3rem); line-height: 0.98; letter-spacing: -0.04em; } .hero-copy { margin: 0.75rem 0 0; max-width: 46rem; color: var(--muted); font-size: 1rem; } .status-stack { display: grid; justify-items: end; align-content: start; min-width: min(100%, 9rem); } .status-badge { display: inline-flex; align-items: center; gap: 0.45rem; min-height: 2rem; padding: 0.35rem 0.78rem; border-radius: 999px; border: 1px solid var(--line); background: var(--card-muted); color: var(--muted); font-size: 0.82rem; font-weight: 700; letter-spacing: 0.01em; } .status-badge-dot { width: 0.52rem; height: 0.52rem; border-radius: 999px; background: currentColor; opacity: 0.9; } .status-badge-live { display: none; background: var(--accent-soft); border-color: rgba(16, 163, 127, 0.18); color: var(--accent-ink); } .status-badge-offline { background: #f5f5f7; border-color: var(--line-strong); color: var(--muted); } [data-phx-main].phx-connected .status-badge-live { display: inline-flex; } [data-phx-main].phx-connected .status-badge-offline { display: none; } .metric-grid { display: grid; gap: 0.85rem; grid-template-columns: repeat(auto-fit, minmax(180px, 1fr)); } .metric-card { border-radius: 22px; padding: 1rem 1.05rem 1.1rem; } .metric-label { margin: 0; color: var(--muted); font-size: 0.82rem; font-weight: 600; letter-spacing: 0.01em; } .metric-value { margin: 0.35rem 0 0; font-size: clamp(1.6rem, 2vw, 2.1rem); line-height: 1.05; letter-spacing: -0.03em; } .metric-detail { margin: 0.45rem 0 0; color: var(--muted); font-size: 0.88rem; } .section-card { border-radius: 24px; padding: 1.15rem; } .section-header { display: flex; justify-content: space-between; align-items: flex-start; gap: 1rem; flex-wrap: wrap; } .section-title { margin: 0; font-size: 1.08rem; line-height: 1.2; letter-spacing: -0.02em; } .section-copy { margin: 0.35rem 0 0; color: var(--muted); font-size: 0.94rem; } .table-wrap { overflow-x: auto; margin-top: 1rem; } .data-table { width: 100%; min-width: 720px; border-collapse: collapse; } .data-table-running { table-layout: fixed; min-width: 980px; } .data-table th { padding: 0 0.5rem 0.75rem 0; text-align: left; color: var(--muted); font-size: 0.78rem; font-weight: 600; text-transform: uppercase; letter-spacing: 0.04em; } .data-table td { padding: 0.9rem 0.5rem 0.9rem 0; border-top: 1px solid var(--line); vertical-align: top; font-size: 0.94rem; } .issue-stack, .session-stack, .detail-stack, .token-stack { display: grid; gap: 0.24rem; min-width: 0; } .event-text { font-weight: 500; line-height: 1.45; max-width: 100%; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } .event-meta { max-width: 100%; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; } .state-badge { display: inline-flex; align-items: center; min-height: 1.85rem; padding: 0.3rem 0.68rem; border-radius: 999px; border: 1px solid var(--line); background: var(--card-muted); color: var(--ink); font-size: 0.8rem; font-weight: 600; line-height: 1; } .state-badge-active { background: var(--accent-soft); border-color: rgba(16, 163, 127, 0.18); color: var(--accent-ink); } .state-badge-warning { background: #fff7e8; border-color: #f1d8a6; color: #8a5a00; } .state-badge-danger { background: var(--danger-soft); border-color: #f6d3cf; color: var(--danger); } .issue-id { font-weight: 600; letter-spacing: -0.01em; } .issue-link { color: var(--muted); font-size: 0.86rem; } .muted { color: var(--muted); } .code-panel { margin-top: 1rem; padding: 1rem; border-radius: 18px; background: #f5f5f7; border: 1px solid var(--line); color: #353740; font-size: 0.9rem; } .empty-state { margin: 1rem 0 0; color: var(--muted); } .error-card { border-radius: 24px; padding: 1.25rem; background: linear-gradient(180deg, #fff8f7 0%, var(--danger-soft) 100%); border-color: #f6d3cf; } .error-title { margin: 0; color: var(--danger); font-size: 1.15rem; letter-spacing: -0.02em; } .error-copy { margin: 0.45rem 0 0; color: var(--danger); } @media (max-width: 860px) { .app-shell { padding: 1rem 0.85rem 2rem; } .hero-grid { grid-template-columns: 1fr; } .status-stack { justify-items: start; } .metric-grid { grid-template-columns: repeat(2, minmax(0, 1fr)); } } @media (max-width: 560px) { .metric-grid { grid-template-columns: 1fr; } .section-card, .hero-card, .error-card { border-radius: 20px; padding: 1rem; } } ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/backoff_queue.evidence.md ================================================ ```text ╭─ SYMPHONY STATUS │ Agents: 1/10 │ Throughput: 15 tps │ Runtime: 45m 0s │ Tokens: in 18,000 | out 2,200 | total 20,200 │ Rate Limits: gpt-5 | primary 0/20,000 reset 95s | secondary 0/60 reset 45s | credits none │ Project: https://linear.app/project/project/issues │ Next refresh: n/a ├─ Running │ │ ID STAGE PID AGE / TURN TOKENS SESSION EVENT │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ ● MT-638 retrying 4242 20m 25s / 7 14,200 thre...567890 agent message streaming: waiting on ... │ ├─ Backoff queue │ │ ↻ MT-450 attempt=4 in 1.250s error=rate limit exhausted │ ↻ MT-451 attempt=2 in 3.900s error=retrying after API timeout with jitter │ ↻ MT-452 attempt=6 in 8.100s error=worker crashed restarting cleanly │ ↻ MT-453 attempt=1 in 11.000s error=fourth queued retry should also render after removing the top-three limit ╰─ ``` ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/backoff_queue.snapshot.txt ================================================ \e[1m╭─ SYMPHONY STATUS\e[0m \e[1m│ Agents: \e[0m\e[32m1\e[0m\e[90m/\e[0m\e[90m10\e[0m \e[1m│ Throughput: \e[0m\e[36m15 tps\e[0m \e[1m│ Runtime: \e[0m\e[35m45m 0s\e[0m \e[1m│ Tokens: \e[0m\e[33min 18,000\e[0m\e[90m | \e[0m\e[33mout 2,200\e[0m\e[90m | \e[0m\e[33mtotal 20,200\e[0m \e[1m│ Rate Limits: \e[0m\e[33mgpt-5\e[0m\e[90m | \e[0m\e[36mprimary 0/20,000 reset 95s\e[0m\e[90m | \e[0m\e[36msecondary 0/60 reset 45s\e[0m\e[90m | \e[0m\e[32mcredits none\e[0m \e[1m│ Project: \e[0m\e[36mhttps://linear.app/project/project/issues\e[0m \e[1m│ Next refresh: \e[0m\e[90mn/a\e[0m \e[1m├─ Running\e[0m │ │ \e[90mID STAGE PID AGE / TURN TOKENS SESSION EVENT \e[0m │ \e[90m───────────────────────────────────────────────────────────────────────────────────────────────────────────────\e[0m │ \e[34m●\e[0m \e[36mMT-638 \e[0m \e[34mretrying \e[0m \e[33m4242 \e[0m \e[35m20m 25s / 7 \e[0m \e[33m 14,200\e[0m \e[36mthre...567890 \e[0m \e[34magent message streaming: waiting on ...\e[0m │ \e[1m├─ Backoff queue\e[0m │ │ \e[33m↻\e[0m \e[31mMT-450\e[0m \e[33mattempt=4\e[0m\e[2m in \e[0m\e[36m1.250s\e[0m \e[2merror=rate limit exhausted\e[0m │ \e[33m↻\e[0m \e[31mMT-451\e[0m \e[33mattempt=2\e[0m\e[2m in \e[0m\e[36m3.900s\e[0m \e[2merror=retrying after API timeout with jitter\e[0m │ \e[33m↻\e[0m \e[31mMT-452\e[0m \e[33mattempt=6\e[0m\e[2m in \e[0m\e[36m8.100s\e[0m \e[2merror=worker crashed restarting cleanly\e[0m │ \e[33m↻\e[0m \e[31mMT-453\e[0m \e[33mattempt=1\e[0m\e[2m in \e[0m\e[36m11.000s\e[0m \e[2merror=fourth queued retry should also render after removing the top-three limit\e[0m ╰─ ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/credits_unlimited.evidence.md ================================================ ```text ╭─ SYMPHONY STATUS │ Agents: 1/10 │ Throughput: 42 tps │ Runtime: 1m 15s │ Tokens: in 90 | out 12 | total 102 │ Rate Limits: priority-tier | primary 100/100 reset 1s | secondary 500/500 reset 1s | credits unlimited │ Project: https://linear.app/project/project/issues │ Next refresh: n/a ├─ Running │ │ ID STAGE PID AGE / TURN TOKENS SESSION EVENT │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ ● MT-777 running 4242 1m 15s / 7 3,200 thre...567890 thread token usage updated (in 90, o... │ ├─ Backoff queue │ │ No queued retries ╰─ ``` ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/credits_unlimited.snapshot.txt ================================================ \e[1m╭─ SYMPHONY STATUS\e[0m \e[1m│ Agents: \e[0m\e[32m1\e[0m\e[90m/\e[0m\e[90m10\e[0m \e[1m│ Throughput: \e[0m\e[36m42 tps\e[0m \e[1m│ Runtime: \e[0m\e[35m1m 15s\e[0m \e[1m│ Tokens: \e[0m\e[33min 90\e[0m\e[90m | \e[0m\e[33mout 12\e[0m\e[90m | \e[0m\e[33mtotal 102\e[0m \e[1m│ Rate Limits: \e[0m\e[33mpriority-tier\e[0m\e[90m | \e[0m\e[36mprimary 100/100 reset 1s\e[0m\e[90m | \e[0m\e[36msecondary 500/500 reset 1s\e[0m\e[90m | \e[0m\e[32mcredits unlimited\e[0m \e[1m│ Project: \e[0m\e[36mhttps://linear.app/project/project/issues\e[0m \e[1m│ Next refresh: \e[0m\e[90mn/a\e[0m \e[1m├─ Running\e[0m │ │ \e[90mID STAGE PID AGE / TURN TOKENS SESSION EVENT \e[0m │ \e[90m───────────────────────────────────────────────────────────────────────────────────────────────────────────────\e[0m │ \e[33m●\e[0m \e[36mMT-777 \e[0m \e[33mrunning \e[0m \e[33m4242 \e[0m \e[35m1m 15s / 7 \e[0m \e[33m 3,200\e[0m \e[36mthre...567890 \e[0m \e[33mthread token usage updated (in 90, o...\e[0m │ \e[1m├─ Backoff queue\e[0m │ │ \e[90mNo queued retries\e[0m ╰─ ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/idle.evidence.md ================================================ ```text ╭─ SYMPHONY STATUS │ Agents: 0/10 │ Throughput: 0 tps │ Runtime: 0m 0s │ Tokens: in 0 | out 0 | total 0 │ Rate Limits: unavailable │ Project: https://linear.app/project/project/issues │ Next refresh: n/a ├─ Running │ │ ID STAGE PID AGE / TURN TOKENS SESSION EVENT │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ No active agents │ ├─ Backoff queue │ │ No queued retries ╰─ ``` ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/idle.snapshot.txt ================================================ \e[1m╭─ SYMPHONY STATUS\e[0m \e[1m│ Agents: \e[0m\e[32m0\e[0m\e[90m/\e[0m\e[90m10\e[0m \e[1m│ Throughput: \e[0m\e[36m0 tps\e[0m \e[1m│ Runtime: \e[0m\e[35m0m 0s\e[0m \e[1m│ Tokens: \e[0m\e[33min 0\e[0m\e[90m | \e[0m\e[33mout 0\e[0m\e[90m | \e[0m\e[33mtotal 0\e[0m \e[1m│ Rate Limits: \e[0m\e[90munavailable\e[0m \e[1m│ Project: \e[0m\e[36mhttps://linear.app/project/project/issues\e[0m \e[1m│ Next refresh: \e[0m\e[90mn/a\e[0m \e[1m├─ Running\e[0m │ │ \e[90mID STAGE PID AGE / TURN TOKENS SESSION EVENT \e[0m │ \e[90m───────────────────────────────────────────────────────────────────────────────────────────────────────────────\e[0m │ \e[90mNo active agents\e[0m │ \e[1m├─ Backoff queue\e[0m │ │ \e[90mNo queued retries\e[0m ╰─ ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/idle_with_dashboard_url.evidence.md ================================================ ```text ╭─ SYMPHONY STATUS │ Agents: 0/10 │ Throughput: 0 tps │ Runtime: 0m 0s │ Tokens: in 0 | out 0 | total 0 │ Rate Limits: unavailable │ Project: https://linear.app/project/project/issues │ Dashboard: http://127.0.0.1:4000/ │ Next refresh: n/a ├─ Running │ │ ID STAGE PID AGE / TURN TOKENS SESSION EVENT │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ No active agents │ ├─ Backoff queue │ │ No queued retries ╰─ ``` ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/idle_with_dashboard_url.snapshot.txt ================================================ \e[1m╭─ SYMPHONY STATUS\e[0m \e[1m│ Agents: \e[0m\e[32m0\e[0m\e[90m/\e[0m\e[90m10\e[0m \e[1m│ Throughput: \e[0m\e[36m0 tps\e[0m \e[1m│ Runtime: \e[0m\e[35m0m 0s\e[0m \e[1m│ Tokens: \e[0m\e[33min 0\e[0m\e[90m | \e[0m\e[33mout 0\e[0m\e[90m | \e[0m\e[33mtotal 0\e[0m \e[1m│ Rate Limits: \e[0m\e[90munavailable\e[0m \e[1m│ Project: \e[0m\e[36mhttps://linear.app/project/project/issues\e[0m \e[1m│ Dashboard: \e[0m\e[36mhttp://127.0.0.1:4000/\e[0m \e[1m│ Next refresh: \e[0m\e[90mn/a\e[0m \e[1m├─ Running\e[0m │ │ \e[90mID STAGE PID AGE / TURN TOKENS SESSION EVENT \e[0m │ \e[90m───────────────────────────────────────────────────────────────────────────────────────────────────────────────\e[0m │ \e[90mNo active agents\e[0m │ \e[1m├─ Backoff queue\e[0m │ │ \e[90mNo queued retries\e[0m ╰─ ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/super_busy.evidence.md ================================================ ```text ╭─ SYMPHONY STATUS │ Agents: 2/10 │ Throughput: 1,842 tps │ Runtime: 72m 1s │ Tokens: in 250,000 | out 18,500 | total 268,500 │ Rate Limits: gpt-5 | primary 12,345/20,000 reset 30s | secondary 45/60 reset 12s | credits 9876.50 │ Project: https://linear.app/project/project/issues │ Next refresh: n/a ├─ Running │ │ ID STAGE PID AGE / TURN TOKENS SESSION EVENT │ ─────────────────────────────────────────────────────────────────────────────────────────────────────────────── │ ● MT-101 running 4242 13m 5s / 11 120,450 thre...567890 turn completed (completed) │ ● MT-102 running 5252 6m 52s / 4 89,200 thre...567890 mix test --cover │ ├─ Backoff queue │ │ No queued retries ╰─ ``` ================================================ FILE: elixir/test/fixtures/status_dashboard_snapshots/super_busy.snapshot.txt ================================================ \e[1m╭─ SYMPHONY STATUS\e[0m \e[1m│ Agents: \e[0m\e[32m2\e[0m\e[90m/\e[0m\e[90m10\e[0m \e[1m│ Throughput: \e[0m\e[36m1,842 tps\e[0m \e[1m│ Runtime: \e[0m\e[35m72m 1s\e[0m \e[1m│ Tokens: \e[0m\e[33min 250,000\e[0m\e[90m | \e[0m\e[33mout 18,500\e[0m\e[90m | \e[0m\e[33mtotal 268,500\e[0m \e[1m│ Rate Limits: \e[0m\e[33mgpt-5\e[0m\e[90m | \e[0m\e[36mprimary 12,345/20,000 reset 30s\e[0m\e[90m | \e[0m\e[36msecondary 45/60 reset 12s\e[0m\e[90m | \e[0m\e[32mcredits 9876.50\e[0m \e[1m│ Project: \e[0m\e[36mhttps://linear.app/project/project/issues\e[0m \e[1m│ Next refresh: \e[0m\e[90mn/a\e[0m \e[1m├─ Running\e[0m │ │ \e[90mID STAGE PID AGE / TURN TOKENS SESSION EVENT \e[0m │ \e[90m───────────────────────────────────────────────────────────────────────────────────────────────────────────────\e[0m │ \e[35m●\e[0m \e[36mMT-101 \e[0m \e[35mrunning \e[0m \e[33m4242 \e[0m \e[35m13m 5s / 11 \e[0m \e[33m 120,450\e[0m \e[36mthre...567890 \e[0m \e[35mturn completed (completed) \e[0m │ \e[32m●\e[0m \e[36mMT-102 \e[0m \e[32mrunning \e[0m \e[33m5252 \e[0m \e[35m6m 52s / 4 \e[0m \e[33m 89,200\e[0m \e[36mthre...567890 \e[0m \e[32mmix test --cover \e[0m │ \e[1m├─ Backoff queue\e[0m │ │ \e[90mNo queued retries\e[0m ╰─ ================================================ FILE: elixir/test/mix/tasks/pr_body_check_test.exs ================================================ defmodule Mix.Tasks.PrBody.CheckTest do use ExUnit.Case, async: false alias Mix.Tasks.PrBody.Check import ExUnit.CaptureIO @template """ #### Context #### TL;DR ** #### Summary - #### Alternatives - #### Test Plan - [ ] """ @valid_body """ #### Context Context text. #### TL;DR Short summary. #### Summary - First change. #### Alternatives - Alternative considered. #### Test Plan - [x] Ran targeted checks. """ setup do Mix.Task.reenable("pr_body.check") :ok end test "prints help" do output = capture_io(fn -> Check.run(["--help"]) end) assert output =~ "mix pr_body.check --file /path/to/pr_body.md" end test "fails on invalid options" do assert_raise Mix.Error, ~r/Invalid option/, fn -> Check.run(["lint", "--wat"]) end end test "fails when file option is missing" do assert_raise Mix.Error, ~r/Missing required option --file/, fn -> Check.run(["lint"]) end end test "fails when template is missing" do in_temp_repo(fn -> File.write!("body.md", @valid_body) assert_raise Mix.Error, ~r/Unable to read PR template/, fn -> Check.run(["lint", "--file", "body.md"]) end end) end test "fails when template has no headings" do in_temp_repo(fn -> write_template!("no headings here") File.write!("body.md", @valid_body) assert_raise Mix.Error, ~r/No markdown headings found/, fn -> Check.run(["lint", "--file", "body.md"]) end end) end test "fails when body file is missing" do in_temp_repo(fn -> write_template!(@template) assert_raise Mix.Error, ~r/Unable to read missing\.md/, fn -> Check.run(["lint", "--file", "missing.md"]) end end) end test "fails when body still has placeholders" do in_temp_repo(fn -> write_template!(@template) File.write!("body.md", @template) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "PR description still contains template placeholder comments" end) end test "fails when heading is missing" do in_temp_repo(fn -> write_template!(@template) missing_heading = String.replace(@valid_body, "#### Alternatives\n\n- Alternative considered.\n\n", "") File.write!("body.md", missing_heading) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "Missing required heading: #### Alternatives" end) end test "fails when headings are out of order" do in_temp_repo(fn -> write_template!(@template) out_of_order = """ #### TL;DR Short summary. #### Context Context text. #### Summary - First change. #### Alternatives - Alternative considered. #### Test Plan - [x] Ran targeted checks. """ File.write!("body.md", out_of_order) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "Required headings are out of order." end) end test "fails on empty section" do in_temp_repo(fn -> write_template!(@template) empty_context = String.replace(@valid_body, "Context text.", "") File.write!("body.md", empty_context) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "Section cannot be empty: #### Context" end) end test "fails when a middle section is blank before the next heading" do in_temp_repo(fn -> write_template!(@template) blank_alternatives = """ #### Context Context text. #### TL;DR Short summary. #### Summary - First change. #### Alternatives #### Test Plan - [x] Ran targeted checks. """ File.write!("body.md", blank_alternatives) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "Section cannot be empty: #### Alternatives" end) end test "fails when bullet and checkbox expectations are not met" do in_temp_repo(fn -> write_template!(@template) invalid_body = """ #### Context Context text. #### TL;DR Short summary. #### Summary Not a bullet. #### Alternatives Also not a bullet. #### Test Plan No checkbox. """ File.write!("body.md", invalid_body) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "Section must include at least one bullet item: #### Summary" assert error_output =~ "Section must include at least one bullet item: #### Alternatives" assert error_output =~ "Section must include at least one bullet item: #### Test Plan" assert error_output =~ "Section must include at least one checkbox item: #### Test Plan" end) end test "fails when heading has no content delimiter" do in_temp_repo(fn -> write_template!(@template) File.write!("body.md", "#### Context\nContext text.") capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) end) end test "fails when heading appears at end of file" do in_temp_repo(fn -> write_template!(@template) File.write!("body.md", "#### Context") error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/PR body format invalid/, fn -> Check.run(["lint", "--file", "body.md"]) end end) assert error_output =~ "Section cannot be empty: #### Context" end) end test "passes for valid body" do in_temp_repo(fn -> write_template!(@template) File.write!("body.md", @valid_body) output = capture_io(fn -> Check.run(["lint", "--file", "body.md"]) end) assert output =~ "PR body format OK" end) end defp in_temp_repo(fun) do unique = System.unique_integer([:positive, :monotonic]) root = Path.join(System.tmp_dir!(), "validate-pr-body-task-test-#{unique}") File.rm_rf!(root) File.mkdir_p!(root) original_cwd = File.cwd!() try do File.cd!(root) fun.() after File.cd!(original_cwd) File.rm_rf!(root) end end defp write_template!(content) do File.mkdir_p!(".github") File.write!(".github/pull_request_template.md", content) end end ================================================ FILE: elixir/test/mix/tasks/specs_check_task_test.exs ================================================ defmodule Mix.Tasks.Specs.CheckTaskTest do use ExUnit.Case, async: false import ExUnit.CaptureIO alias Mix.Tasks.Specs.Check setup do Mix.Task.reenable("specs.check") :ok end test "uses the default lib path when all public functions have specs" do in_temp_project(fn -> write_module!("lib/sample.ex", """ defmodule Sample do @spec ok(term()) :: term() def ok(arg), do: arg end """) output = capture_io(fn -> assert :ok = Check.run([]) end) assert output =~ "specs.check: all public functions have @spec or exemption" end) end test "raises when an explicit path contains missing specs" do in_temp_project(fn -> write_module!("src/sample.ex", """ defmodule Sample do def missing(arg), do: arg end """) error_output = capture_io(:stderr, fn -> assert_raise Mix.Error, ~r/specs.check failed with 1 missing @spec declaration/, fn -> Check.run(["--paths", "src"]) end end) assert error_output =~ "src/sample.ex:2 missing @spec for Sample.missing/1" end) end test "loads exemptions from a file and ignores comments and blank lines" do in_temp_project(fn -> write_module!("lib/sample.ex", """ defmodule Sample do def legacy(arg), do: arg end """) File.mkdir_p!("config") File.write!("config/specs_exemptions.txt", """ # existing exemptions Sample.legacy/1 """) output = capture_io(fn -> assert :ok = Check.run(["--paths", "lib", "--exemptions-file", "config/specs_exemptions.txt"]) end) assert output =~ "specs.check: all public functions have @spec or exemption" end) end test "treats a missing exemptions file as empty" do in_temp_project(fn -> write_module!("lib/sample.ex", """ defmodule Sample do @spec ok(term()) :: term() def ok(arg), do: arg end """) output = capture_io(fn -> assert :ok = Check.run(["--exemptions-file", "config/missing.txt"]) end) assert output =~ "specs.check: all public functions have @spec or exemption" end) end defp in_temp_project(fun) do root = Path.join(System.tmp_dir!(), "specs-check-task-test-#{System.unique_integer([:positive, :monotonic])}") original_cwd = File.cwd!() File.rm_rf!(root) File.mkdir_p!(root) try do File.cd!(root, fun) after File.cd!(original_cwd) File.rm_rf!(root) end end defp write_module!(path, source) do File.mkdir_p!(Path.dirname(path)) File.write!(path, source) end end ================================================ FILE: elixir/test/mix/tasks/workspace_before_remove_test.exs ================================================ defmodule Mix.Tasks.Workspace.BeforeRemoveTest do use ExUnit.Case, async: false alias Mix.Tasks.Workspace.BeforeRemove import ExUnit.CaptureIO setup do Mix.Task.reenable("workspace.before_remove") :ok end test "prints help" do output = capture_io(fn -> BeforeRemove.run(["--help"]) end) assert output =~ "mix workspace.before_remove" end test "fails on invalid options" do assert_raise Mix.Error, ~r/Invalid option/, fn -> BeforeRemove.run(["--wat"]) end end test "no-ops when branch is unavailable" do with_path([], fn -> in_temp_dir(fn -> output = capture_io(fn -> BeforeRemove.run([]) end) assert output == "" end) end) end test "no-ops when gh is unavailable" do with_path([], fn -> output = capture_io(fn -> BeforeRemove.run(["--branch", "feature/no-gh"]) end) assert output == "" end) end test "uses current branch for lookup when branch option is omitted" do with_fake_gh_and_git( """ #!/bin/sh printf '%s\n' "$*" >> "$GH_LOG" if [ "$1" = "auth" ] && [ "$2" = "status" ]; then exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "list" ]; then printf '101\n102\n' exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "close" ] && [ "$3" = "101" ]; then exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "close" ] && [ "$3" = "102" ]; then printf 'boom\n' >&2 exit 17 fi exit 99 """, """ #!/bin/sh printf 'feature/workpad\n' exit 0 """, fn log_path -> {output, error_output} = capture_task_output(fn -> BeforeRemove.run([]) end) assert output =~ "Closed PR #101 for branch feature/workpad" assert error_output =~ "Failed to close PR #102 for branch feature/workpad" log = File.read!(log_path) assert log =~ "pr list --repo openai/symphony --head feature/workpad --state open --json number --jq .[].number" assert log =~ "pr close 101 --repo openai/symphony" assert log =~ "pr close 102 --repo openai/symphony" end ) end test "closes open pull requests for the branch and tolerates close failures" do with_fake_gh(fn log_path -> File.write!(log_path, "") {output, error_output} = capture_task_output(fn -> BeforeRemove.run(["--branch", "feature/workpad"]) end) assert output =~ "Closed PR #101 for branch feature/workpad" assert error_output =~ "Failed to close PR #102 for branch feature/workpad" log = File.read!(log_path) assert log =~ "auth status" assert log =~ "pr list --repo openai/symphony --head feature/workpad --state open --json number --jq .[].number" assert log =~ "pr close 101 --repo openai/symphony" assert log =~ "pr close 102 --repo openai/symphony" {second_output, error_output} = capture_task_output(fn -> Mix.Task.reenable("workspace.before_remove") BeforeRemove.run(["--branch", "feature/workpad"]) end) assert second_output =~ "Closed PR #101 for branch feature/workpad" assert error_output =~ "Failed to close PR #102 for branch feature/workpad" end) end test "formats close failures without command stderr output" do with_fake_gh( """ #!/bin/sh printf '%s\n' "$*" >> "$GH_LOG" if [ "$1" = "auth" ] && [ "$2" = "status" ]; then exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "list" ]; then printf '102\n' exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "close" ] && [ "$3" = "102" ]; then exit 17 fi exit 99 """, fn log_path -> error_output = capture_io(:stderr, fn -> Mix.Task.reenable("workspace.before_remove") BeforeRemove.run(["--branch", "feature/no-output"]) end) assert error_output =~ "Failed to close PR #102 for branch feature/no-output: exit 17" refute error_output =~ "output=" log = File.read!(log_path) assert log =~ "pr list --repo openai/symphony --head feature/no-output --state open --json number --jq .[].number" assert log =~ "pr close 102 --repo openai/symphony" end ) end test "no-ops when PR list fails for current branch" do with_fake_gh( """ #!/bin/sh printf '%s\n' "$*" >> "$GH_LOG" if [ "$1" = "auth" ] && [ "$2" = "status" ]; then exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "list" ]; then exit 1 fi exit 99 """, fn log_path -> output = capture_io(fn -> BeforeRemove.run(["--branch", "feature/list-fails"]) end) assert output == "" log = File.read!(log_path) assert log =~ "auth status" assert log =~ "pr list --repo openai/symphony --head feature/list-fails --state open --json number --jq .[].number" refute log =~ "pr close" end ) end test "no-ops when git current branch is blank" do with_fake_gh_and_git( """ #!/bin/sh printf '%s\n' "$*" >> "$GH_LOG" if [ "$1" = "auth" ] && [ "$2" = "status" ]; then exit 0 fi exit 99 """, """ #!/bin/sh printf '\n' exit 0 """, fn log_path -> output = capture_io(fn -> BeforeRemove.run([]) end) assert output == "" log = File.read!(log_path) assert log == "" refute log =~ "pr list" end ) end test "no-ops when gh auth is unavailable" do with_fake_gh( """ #!/bin/sh printf '%s\n' "$*" >> "$GH_LOG" if [ "$1" = "auth" ] && [ "$2" = "status" ]; then exit 1 fi exit 99 """, fn log_path -> BeforeRemove.run(["--branch", "feature/no-auth"]) log = File.read!(log_path) assert log =~ "auth status" refute log =~ "pr list" end ) end defp with_fake_gh(fun) do with_fake_binaries( %{ "gh" => """ #!/bin/sh printf '%s\n' "$*" >> "$GH_LOG" if [ "$1" = "auth" ] && [ "$2" = "status" ]; then exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "list" ]; then printf '101\n102\n' exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "close" ] && [ "$3" = "101" ]; then exit 0 fi if [ "$1" = "pr" ] && [ "$2" = "close" ] && [ "$3" = "102" ]; then printf 'boom\n' >&2 exit 17 fi exit 99 """ }, fun ) end defp with_fake_gh(script, fun) do with_fake_binaries(%{"gh" => script}, fun) end defp with_fake_gh_and_git(gh_script, git_script, fun) do with_fake_binaries(%{"gh" => gh_script, "git" => git_script}, fun) end defp with_fake_binaries(scripts, fun) do unique = System.unique_integer([:positive, :monotonic]) root = Path.join(System.tmp_dir!(), "workspace-before-remove-task-test-#{unique}") bin_dir = Path.join(root, "bin") log_path = Path.join(root, "gh.log") try do File.rm_rf!(root) File.mkdir_p!(bin_dir) File.write!(log_path, "") original_path = System.get_env("PATH") || "" path_with_binaries = Enum.join([bin_dir, original_path], ":") Enum.each(scripts, fn {name, script} -> path = Path.join(bin_dir, name) File.write!(path, script) File.chmod!(path, 0o755) end) with_env( %{ "GH_LOG" => log_path, "PATH" => path_with_binaries }, fn -> fun.(log_path) end ) after File.rm_rf!(root) end end defp with_path(paths, fun) do with_env(%{"PATH" => Enum.join(paths, ":")}, fun) end defp with_env(overrides, fun) do keys = Map.keys(overrides) previous = Map.new(keys, fn key -> {key, System.get_env(key)} end) try do Enum.each(overrides, fn {key, value} -> System.put_env(key, value) end) fun.() after Enum.each(previous, fn {key, nil} -> System.delete_env(key) {key, value} -> System.put_env(key, value) end) end end defp in_temp_dir(fun) do unique = System.unique_integer([:positive, :monotonic]) root = Path.join(System.tmp_dir!(), "workspace-before-remove-empty-dir-#{unique}") File.rm_rf!(root) File.mkdir_p!(root) original_cwd = File.cwd!() try do File.cd!(root) fun.() after File.cd!(original_cwd) File.rm_rf!(root) end end defp capture_task_output(fun) do parent = self() ref = make_ref() error_output = capture_io(:stderr, fn -> output = capture_io(fn -> fun.() end) send(parent, {ref, output}) end) output = receive do {^ref, output} -> output after 1_000 -> flunk("Timed out waiting for captured task output") end {output, error_output} end end ================================================ FILE: elixir/test/support/live_e2e_docker/Dockerfile ================================================ FROM node:20-bookworm-slim RUN apt-get update && apt-get install -y --no-install-recommends \ bash \ ca-certificates \ git \ openssh-server \ python3 \ ripgrep \ && rm -rf /var/lib/apt/lists/* RUN install -d -m 700 /root/.ssh /root/.codex /run/symphony/ssh /var/run/sshd RUN npm install --global @openai/codex COPY symphony-live-worker.conf /etc/ssh/sshd_config.d/symphony-live-worker.conf COPY live_worker_entrypoint.sh /usr/local/bin/symphony-live-worker RUN chmod 755 /usr/local/bin/symphony-live-worker EXPOSE 22 ENTRYPOINT ["/usr/local/bin/symphony-live-worker"] ================================================ FILE: elixir/test/support/live_e2e_docker/docker-compose.yml ================================================ services: worker1: build: context: . dockerfile: Dockerfile ports: - "${SYMPHONY_LIVE_DOCKER_WORKER_1_PORT}:22" volumes: - ${SYMPHONY_LIVE_DOCKER_AUTHORIZED_KEY}:/run/symphony/ssh/authorized_key.pub:ro - ${SYMPHONY_LIVE_DOCKER_AUTH_JSON}:/root/.codex/auth.json:ro worker2: build: context: . dockerfile: Dockerfile ports: - "${SYMPHONY_LIVE_DOCKER_WORKER_2_PORT}:22" volumes: - ${SYMPHONY_LIVE_DOCKER_AUTHORIZED_KEY}:/run/symphony/ssh/authorized_key.pub:ro - ${SYMPHONY_LIVE_DOCKER_AUTH_JSON}:/root/.codex/auth.json:ro ================================================ FILE: elixir/test/support/live_e2e_docker/live_worker_entrypoint.sh ================================================ #!/bin/sh set -eu install -d -m 700 /root/.ssh /root/.codex if [ ! -s /run/symphony/ssh/authorized_key.pub ]; then echo "missing authorized key at /run/symphony/ssh/authorized_key.pub" >&2 exit 1 fi install -m 600 /run/symphony/ssh/authorized_key.pub /root/.ssh/authorized_keys exec /usr/sbin/sshd -D -e ================================================ FILE: elixir/test/support/live_e2e_docker/symphony-live-worker.conf ================================================ PubkeyAuthentication yes PasswordAuthentication no KbdInteractiveAuthentication no ChallengeResponseAuthentication no UsePAM no PermitRootLogin yes AuthorizedKeysFile .ssh/authorized_keys ================================================ FILE: elixir/test/support/snapshot_support.exs ================================================ defmodule SymphonyElixir.TestSupport.Snapshot do import ExUnit.Assertions @snapshot_root Path.expand("../fixtures", __DIR__) @ansi_regex ~r/\e\[[0-9;]*m/ @update_snapshot_hint "Run `UPDATE_SNAPSHOTS=1 mix test test/symphony_elixir/status_dashboard_snapshot_test.exs` to create or update fixtures." def assert_dashboard_snapshot!(name, raw_ansi_content) when is_binary(name) and is_binary(raw_ansi_content) do assert_snapshot!( Path.join("status_dashboard_snapshots", "#{name}.snapshot.txt"), escape_ansi(raw_ansi_content) ) assert_snapshot!( Path.join("status_dashboard_snapshots", "#{name}.evidence.md"), evidence_markdown(raw_ansi_content) ) :ok end def assert_snapshot!(relative_path, content) when is_binary(relative_path) and is_binary(content) do path = snapshot_path(relative_path) normalized = normalize_content(content) File.mkdir_p!(Path.dirname(path)) if update_snapshots?() do File.write!(path, normalized) :ok else case File.read(path) do {:ok, expected} -> assert normalized == expected, "Snapshot mismatch for `#{relative_path}`. #{@update_snapshot_hint}" {:error, :enoent} -> flunk("Missing snapshot fixture `#{relative_path}`. #{@update_snapshot_hint}") {:error, reason} -> flunk("Failed reading snapshot fixture `#{relative_path}`: #{inspect(reason)}") end end end def escape_ansi(content) when is_binary(content), do: String.replace(content, <<27>>, "\\e") def strip_ansi(content) when is_binary(content), do: Regex.replace(@ansi_regex, content, "") def evidence_markdown(raw_ansi_content) when is_binary(raw_ansi_content) do plain = raw_ansi_content |> strip_ansi() |> normalize_content() |> String.trim_trailing("\n") "```text\n#{plain}\n```\n" end defp snapshot_path(relative_path), do: Path.join(@snapshot_root, relative_path) defp update_snapshots? do System.get_env("UPDATE_SNAPSHOTS") |> to_string() |> String.downcase() |> Kernel.in(["1", "true", "yes"]) end defp normalize_content(content) do content |> String.replace("\r\n", "\n") |> String.trim_trailing("\n") |> Kernel.<>("\n") end end ================================================ FILE: elixir/test/support/test_support.exs ================================================ defmodule SymphonyElixir.TestSupport do @workflow_prompt "You are an agent for this repository." defmacro __using__(_opts) do quote do use ExUnit.Case import ExUnit.CaptureLog alias SymphonyElixir.AgentRunner alias SymphonyElixir.CLI alias SymphonyElixir.Codex.AppServer alias SymphonyElixir.Config alias SymphonyElixir.HttpServer alias SymphonyElixir.Linear.Client alias SymphonyElixir.Linear.Issue alias SymphonyElixir.Orchestrator alias SymphonyElixir.PromptBuilder alias SymphonyElixir.StatusDashboard alias SymphonyElixir.Tracker alias SymphonyElixir.Workflow alias SymphonyElixir.WorkflowStore alias SymphonyElixir.Workspace import SymphonyElixir.TestSupport, only: [write_workflow_file!: 1, write_workflow_file!: 2, restore_env: 2, stop_default_http_server: 0] setup do workflow_root = Path.join( System.tmp_dir!(), "symphony-elixir-workflow-#{System.unique_integer([:positive])}" ) File.mkdir_p!(workflow_root) workflow_file = Path.join(workflow_root, "WORKFLOW.md") write_workflow_file!(workflow_file) Workflow.set_workflow_file_path(workflow_file) if Process.whereis(SymphonyElixir.WorkflowStore), do: SymphonyElixir.WorkflowStore.force_reload() stop_default_http_server() on_exit(fn -> Application.delete_env(:symphony_elixir, :workflow_file_path) Application.delete_env(:symphony_elixir, :server_port_override) Application.delete_env(:symphony_elixir, :memory_tracker_issues) Application.delete_env(:symphony_elixir, :memory_tracker_recipient) File.rm_rf(workflow_root) end) :ok end end end def write_workflow_file!(path, overrides \\ []) do workflow = workflow_content(overrides) File.write!(path, workflow) if Process.whereis(SymphonyElixir.WorkflowStore) do try do SymphonyElixir.WorkflowStore.force_reload() catch :exit, _reason -> :ok end end :ok end def restore_env(key, nil), do: System.delete_env(key) def restore_env(key, value), do: System.put_env(key, value) def stop_default_http_server do case Enum.find(Supervisor.which_children(SymphonyElixir.Supervisor), fn {SymphonyElixir.HttpServer, _pid, _type, _modules} -> true _child -> false end) do {SymphonyElixir.HttpServer, pid, _type, _modules} when is_pid(pid) -> :ok = Supervisor.terminate_child(SymphonyElixir.Supervisor, SymphonyElixir.HttpServer) if Process.alive?(pid) do Process.exit(pid, :normal) end :ok _ -> :ok end end defp workflow_content(overrides) do config = Keyword.merge( [ tracker_kind: "linear", tracker_endpoint: "https://api.linear.app/graphql", tracker_api_token: "token", tracker_project_slug: "project", tracker_assignee: nil, tracker_active_states: ["Todo", "In Progress"], tracker_terminal_states: ["Closed", "Cancelled", "Canceled", "Duplicate", "Done"], poll_interval_ms: 30_000, workspace_root: Path.join(System.tmp_dir!(), "symphony_workspaces"), worker_ssh_hosts: [], worker_max_concurrent_agents_per_host: nil, max_concurrent_agents: 10, max_turns: 20, max_retry_backoff_ms: 300_000, max_concurrent_agents_by_state: %{}, codex_command: "codex app-server", codex_approval_policy: %{reject: %{sandbox_approval: true, rules: true, mcp_elicitations: true}}, codex_thread_sandbox: "workspace-write", codex_turn_sandbox_policy: nil, codex_turn_timeout_ms: 3_600_000, codex_read_timeout_ms: 5_000, codex_stall_timeout_ms: 300_000, hook_after_create: nil, hook_before_run: nil, hook_after_run: nil, hook_before_remove: nil, hook_timeout_ms: 60_000, observability_enabled: true, observability_refresh_ms: 1_000, observability_render_interval_ms: 16, server_port: nil, server_host: nil, prompt: @workflow_prompt ], overrides ) tracker_kind = Keyword.get(config, :tracker_kind) tracker_endpoint = Keyword.get(config, :tracker_endpoint) tracker_api_token = Keyword.get(config, :tracker_api_token) tracker_project_slug = Keyword.get(config, :tracker_project_slug) tracker_assignee = Keyword.get(config, :tracker_assignee) tracker_active_states = Keyword.get(config, :tracker_active_states) tracker_terminal_states = Keyword.get(config, :tracker_terminal_states) poll_interval_ms = Keyword.get(config, :poll_interval_ms) workspace_root = Keyword.get(config, :workspace_root) worker_ssh_hosts = Keyword.get(config, :worker_ssh_hosts) worker_max_concurrent_agents_per_host = Keyword.get(config, :worker_max_concurrent_agents_per_host) max_concurrent_agents = Keyword.get(config, :max_concurrent_agents) max_turns = Keyword.get(config, :max_turns) max_retry_backoff_ms = Keyword.get(config, :max_retry_backoff_ms) max_concurrent_agents_by_state = Keyword.get(config, :max_concurrent_agents_by_state) codex_command = Keyword.get(config, :codex_command) codex_approval_policy = Keyword.get(config, :codex_approval_policy) codex_thread_sandbox = Keyword.get(config, :codex_thread_sandbox) codex_turn_sandbox_policy = Keyword.get(config, :codex_turn_sandbox_policy) codex_turn_timeout_ms = Keyword.get(config, :codex_turn_timeout_ms) codex_read_timeout_ms = Keyword.get(config, :codex_read_timeout_ms) codex_stall_timeout_ms = Keyword.get(config, :codex_stall_timeout_ms) hook_after_create = Keyword.get(config, :hook_after_create) hook_before_run = Keyword.get(config, :hook_before_run) hook_after_run = Keyword.get(config, :hook_after_run) hook_before_remove = Keyword.get(config, :hook_before_remove) hook_timeout_ms = Keyword.get(config, :hook_timeout_ms) observability_enabled = Keyword.get(config, :observability_enabled) observability_refresh_ms = Keyword.get(config, :observability_refresh_ms) observability_render_interval_ms = Keyword.get(config, :observability_render_interval_ms) server_port = Keyword.get(config, :server_port) server_host = Keyword.get(config, :server_host) prompt = Keyword.get(config, :prompt) sections = [ "---", "tracker:", " kind: #{yaml_value(tracker_kind)}", " endpoint: #{yaml_value(tracker_endpoint)}", " api_key: #{yaml_value(tracker_api_token)}", " project_slug: #{yaml_value(tracker_project_slug)}", " assignee: #{yaml_value(tracker_assignee)}", " active_states: #{yaml_value(tracker_active_states)}", " terminal_states: #{yaml_value(tracker_terminal_states)}", "polling:", " interval_ms: #{yaml_value(poll_interval_ms)}", "workspace:", " root: #{yaml_value(workspace_root)}", worker_yaml(worker_ssh_hosts, worker_max_concurrent_agents_per_host), "agent:", " max_concurrent_agents: #{yaml_value(max_concurrent_agents)}", " max_turns: #{yaml_value(max_turns)}", " max_retry_backoff_ms: #{yaml_value(max_retry_backoff_ms)}", " max_concurrent_agents_by_state: #{yaml_value(max_concurrent_agents_by_state)}", "codex:", " command: #{yaml_value(codex_command)}", " approval_policy: #{yaml_value(codex_approval_policy)}", " thread_sandbox: #{yaml_value(codex_thread_sandbox)}", " turn_sandbox_policy: #{yaml_value(codex_turn_sandbox_policy)}", " turn_timeout_ms: #{yaml_value(codex_turn_timeout_ms)}", " read_timeout_ms: #{yaml_value(codex_read_timeout_ms)}", " stall_timeout_ms: #{yaml_value(codex_stall_timeout_ms)}", hooks_yaml(hook_after_create, hook_before_run, hook_after_run, hook_before_remove, hook_timeout_ms), observability_yaml(observability_enabled, observability_refresh_ms, observability_render_interval_ms), server_yaml(server_port, server_host), "---", prompt ] |> Enum.reject(&(&1 in [nil, ""])) Enum.join(sections, "\n") <> "\n" end defp yaml_value(value) when is_binary(value) do "\"" <> String.replace(value, "\"", "\\\"") <> "\"" end defp yaml_value(value) when is_integer(value), do: to_string(value) defp yaml_value(true), do: "true" defp yaml_value(false), do: "false" defp yaml_value(nil), do: "null" defp yaml_value(values) when is_list(values) do "[" <> Enum.map_join(values, ", ", &yaml_value/1) <> "]" end defp yaml_value(values) when is_map(values) do "{" <> Enum.map_join(values, ", ", fn {key, value} -> "#{yaml_value(to_string(key))}: #{yaml_value(value)}" end) <> "}" end defp yaml_value(value), do: yaml_value(to_string(value)) defp hooks_yaml(nil, nil, nil, nil, timeout_ms), do: "hooks:\n timeout_ms: #{yaml_value(timeout_ms)}" defp hooks_yaml(hook_after_create, hook_before_run, hook_after_run, hook_before_remove, timeout_ms) do [ "hooks:", " timeout_ms: #{yaml_value(timeout_ms)}", hook_entry("after_create", hook_after_create), hook_entry("before_run", hook_before_run), hook_entry("after_run", hook_after_run), hook_entry("before_remove", hook_before_remove) ] |> Enum.reject(&is_nil/1) |> Enum.join("\n") end defp worker_yaml(ssh_hosts, max_concurrent_agents_per_host) when ssh_hosts in [nil, []] and is_nil(max_concurrent_agents_per_host), do: nil defp worker_yaml(ssh_hosts, max_concurrent_agents_per_host) do [ "worker:", ssh_hosts not in [nil, []] && " ssh_hosts: #{yaml_value(ssh_hosts)}", !is_nil(max_concurrent_agents_per_host) && " max_concurrent_agents_per_host: #{yaml_value(max_concurrent_agents_per_host)}" ] |> Enum.reject(&(&1 in [nil, false])) |> Enum.join("\n") end defp observability_yaml(enabled, refresh_ms, render_interval_ms) do [ "observability:", " dashboard_enabled: #{yaml_value(enabled)}", " refresh_ms: #{yaml_value(refresh_ms)}", " render_interval_ms: #{yaml_value(render_interval_ms)}" ] |> Enum.join("\n") end defp server_yaml(nil, nil), do: nil defp server_yaml(port, host) do [ "server:", port && " port: #{yaml_value(port)}", host && " host: #{yaml_value(host)}" ] |> Enum.reject(&is_nil/1) |> Enum.join("\n") end defp hook_entry(_name, nil), do: nil defp hook_entry(name, command) when is_binary(command) do indented = command |> String.split("\n") |> Enum.map_join("\n", &(" " <> &1)) " #{name}: |\n#{indented}" end end ================================================ FILE: elixir/test/symphony_elixir/app_server_test.exs ================================================ defmodule SymphonyElixir.AppServerTest do use SymphonyElixir.TestSupport test "app server rejects the workspace root and paths outside workspace root" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-cwd-guard-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") outside_workspace = Path.join(test_root, "outside") File.mkdir_p!(workspace_root) File.mkdir_p!(outside_workspace) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root ) issue = %Issue{ id: "issue-workspace-guard", identifier: "MT-999", title: "Validate workspace guard", description: "Ensure app-server refuses invalid cwd targets", state: "In Progress", url: "https://example.org/issues/MT-999", labels: ["backend"] } assert {:error, {:invalid_workspace_cwd, :workspace_root, _path}} = AppServer.run(workspace_root, "guard", issue) assert {:error, {:invalid_workspace_cwd, :outside_workspace_root, _path, _root}} = AppServer.run(outside_workspace, "guard", issue) after File.rm_rf(test_root) end end test "app server rejects symlink escape cwd paths under the workspace root" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-symlink-cwd-guard-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") outside_workspace = Path.join(test_root, "outside") symlink_workspace = Path.join(workspace_root, "MT-1000") File.mkdir_p!(workspace_root) File.mkdir_p!(outside_workspace) File.ln_s!(outside_workspace, symlink_workspace) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root ) issue = %Issue{ id: "issue-workspace-symlink-guard", identifier: "MT-1000", title: "Validate symlink workspace guard", description: "Ensure app-server refuses symlink escape cwd targets", state: "In Progress", url: "https://example.org/issues/MT-1000", labels: ["backend"] } assert {:error, {:invalid_workspace_cwd, :symlink_escape, ^symlink_workspace, _root}} = AppServer.run(symlink_workspace, "guard", issue) after File.rm_rf(test_root) end end test "app server passes explicit turn sandbox policies through unchanged" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-supported-turn-policies-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-1001") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-supported-turn-policies.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-supported-turn-policies.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' "$line" >> "$trace_file" case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-1001"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-1001"}}}' ;; 4) printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) issue = %Issue{ id: "issue-supported-turn-policies", identifier: "MT-1001", title: "Validate explicit turn sandbox policy passthrough", description: "Ensure runtime startup forwards configured turn sandbox policies unchanged", state: "In Progress", url: "https://example.org/issues/MT-1001", labels: ["backend"] } policy_cases = [ %{"type" => "dangerFullAccess"}, %{"type" => "externalSandbox", "profile" => "remote-ci"}, %{"type" => "workspaceWrite", "writableRoots" => ["relative/path"], "networkAccess" => true}, %{"type" => "futureSandbox", "nested" => %{"flag" => true}} ] Enum.each(policy_cases, fn configured_policy -> File.rm(trace_file) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server", codex_turn_sandbox_policy: configured_policy ) assert {:ok, _result} = AppServer.run(workspace, "Validate supported turn policy", issue) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> payload["method"] == "turn/start" && get_in(payload, ["params", "sandboxPolicy"]) == configured_policy end) else false end end) end) after File.rm_rf(test_root) end end test "app server marks request-for-input events as a hard failure" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-input-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-88") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-input.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-input.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-88\"}}}' ;; 3) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-88\"}}}' ;; 4) printf '%s\\n' '{\"method\":\"turn/input_required\",\"id\":\"resp-1\",\"params\":{\"requiresInput\":true,\"reason\":\"blocked\"}}' ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-input", identifier: "MT-88", title: "Input needed", description: "Cannot satisfy codex input", state: "In Progress", url: "https://example.org/issues/MT-88", labels: ["backend"] } assert {:error, {:turn_input_required, payload}} = AppServer.run(workspace, "Needs input", issue) assert payload["method"] == "turn/input_required" after File.rm_rf(test_root) end end test "app server fails when command execution approval is required under safer defaults" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-approval-required-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-89") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh count=0 while IFS= read -r _line; do count=$((count + 1)) case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-89"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-89"}}}' printf '%s\\n' '{"id":99,"method":"item/commandExecution/requestApproval","params":{"command":"gh pr view","cwd":"/tmp","reason":"need approval"}}' ;; *) sleep 1 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-approval-required", identifier: "MT-89", title: "Approval required", description: "Ensure safer defaults do not auto approve requests", state: "In Progress", url: "https://example.org/issues/MT-89", labels: ["backend"] } assert {:error, {:approval_required, payload}} = AppServer.run(workspace, "Handle approval request", issue) assert payload["method"] == "item/commandExecution/requestApproval" after File.rm_rf(test_root) end end test "app server auto-approves command execution approval requests when approval policy is never" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-auto-approve-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-89") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-auto-approve.trace") previous_trace = System.get_env("SYMP_TEST_CODex_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODex_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODex_TRACE") end end) System.put_env("SYMP_TEST_CODex_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODex_TRACE:-/tmp/codex-auto-approve.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-89\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-89\"}}}' printf '%s\\n' '{\"id\":99,\"method\":\"item/commandExecution/requestApproval\",\"params\":{\"command\":\"gh pr view\",\"cwd\":\"/tmp\",\"reason\":\"need approval\"}}' ;; 5) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server", codex_approval_policy: "never" ) issue = %Issue{ id: "issue-auto-approve", identifier: "MT-89", title: "Auto approve request", description: "Ensure app-server approval requests are handled automatically", state: "In Progress", url: "https://example.org/issues/MT-89", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Handle approval request", issue) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 1 and get_in(payload, ["params", "capabilities", "experimentalApi"]) == true else false end end) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 2 and case get_in(payload, ["params", "dynamicTools"]) do [ %{ "description" => description, "inputSchema" => %{"required" => ["query"]}, "name" => "linear_graphql" } ] -> description =~ "Linear" _ -> false end else false end end) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 99 and get_in(payload, ["result", "decision"]) == "acceptForSession" else false end end) after File.rm_rf(test_root) end end test "app server auto-approves MCP tool approval prompts when approval policy is never" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-tool-user-input-auto-approve-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-717") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-tool-user-input-auto-approve.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-tool-user-input-auto-approve.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-717\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-717\"}}}' printf '%s\\n' '{\"id\":110,\"method\":\"item/tool/requestUserInput\",\"params\":{\"itemId\":\"call-717\",\"questions\":[{\"header\":\"Approve app tool call?\",\"id\":\"mcp_tool_call_approval_call-717\",\"isOther\":false,\"isSecret\":false,\"options\":[{\"description\":\"Run the tool and continue.\",\"label\":\"Approve Once\"},{\"description\":\"Run the tool and remember this choice for this session.\",\"label\":\"Approve this Session\"},{\"description\":\"Decline this tool call and continue.\",\"label\":\"Deny\"},{\"description\":\"Cancel this tool call\",\"label\":\"Cancel\"}],\"question\":\"The linear MCP server wants to run the tool \\\"Save issue\\\", which may modify or delete data. Allow this action?\"}],\"threadId\":\"thread-717\",\"turnId\":\"turn-717\"}}' ;; 5) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server", codex_approval_policy: "never" ) issue = %Issue{ id: "issue-tool-user-input-auto-approve", identifier: "MT-717", title: "Auto approve MCP tool request user input", description: "Ensure app tool approval prompts continue automatically", state: "In Progress", url: "https://example.org/issues/MT-717", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Handle tool approval prompt", issue) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 110 and get_in(payload, ["result", "answers", "mcp_tool_call_approval_call-717", "answers"]) == ["Approve this Session"] else false end end) after File.rm_rf(test_root) end end test "app server sends a generic non-interactive answer for freeform tool input prompts" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-tool-user-input-required-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-718") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh count=0 while IFS= read -r _line; do count=$((count + 1)) case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) ;; 3) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-718"}}}' ;; 4) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-718"}}}' printf '%s\\n' '{"id":111,"method":"item/tool/requestUserInput","params":{"itemId":"call-718","questions":[{"header":"Provide context","id":"freeform-718","isOther":false,"isSecret":false,"options":null,"question":"What comment should I post back to the issue?"}],"threadId":"thread-718","turnId":"turn-718"}}' ;; 5) printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server", codex_approval_policy: "never" ) issue = %Issue{ id: "issue-tool-user-input-required", identifier: "MT-718", title: "Non interactive tool input answer", description: "Ensure arbitrary tool prompts receive a generic answer", state: "In Progress", url: "https://example.org/issues/MT-718", labels: ["backend"] } on_message = fn message -> send(self(), {:app_server_message, message}) end assert {:ok, _result} = AppServer.run(workspace, "Handle generic tool input", issue, on_message: on_message) assert_received {:app_server_message, %{ event: :tool_input_auto_answered, answer: "This is a non-interactive session. Operator input is unavailable." }} after File.rm_rf(test_root) end end test "app server sends a generic non-interactive answer for option-based tool input prompts" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-tool-user-input-options-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-719") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-tool-user-input-options.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-tool-user-input-options.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-719\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-719\"}}}' printf '%s\\n' '{\"id\":112,\"method\":\"item/tool/requestUserInput\",\"params\":{\"itemId\":\"call-719\",\"questions\":[{\"header\":\"Choose an action\",\"id\":\"options-719\",\"isOther\":false,\"isSecret\":false,\"options\":[{\"description\":\"Use the default behavior.\",\"label\":\"Use default\"},{\"description\":\"Skip this step.\",\"label\":\"Skip\"}],\"question\":\"How should I proceed?\"}],\"threadId\":\"thread-719\",\"turnId\":\"turn-719\"}}' ;; 5) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-tool-user-input-options", identifier: "MT-719", title: "Option based tool input answer", description: "Ensure option prompts receive a generic non-interactive answer", state: "In Progress", url: "https://example.org/issues/MT-719", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Handle option based tool input", issue) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 112 and get_in(payload, ["result", "answers", "options-719", "answers"]) == [ "This is a non-interactive session. Operator input is unavailable." ] else false end end) after File.rm_rf(test_root) end end test "app server rejects unsupported dynamic tool calls without stalling" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-tool-call-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-90") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-tool-call.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-tool-call.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-90\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-90\"}}}' printf '%s\\n' '{\"id\":101,\"method\":\"item/tool/call\",\"params\":{\"tool\":\"some_tool\",\"callId\":\"call-90\",\"threadId\":\"thread-90\",\"turnId\":\"turn-90\",\"arguments\":{}}}' ;; 5) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-tool-call", identifier: "MT-90", title: "Unsupported tool call", description: "Ensure unsupported tool calls do not stall a turn", state: "In Progress", url: "https://example.org/issues/MT-90", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Reject unsupported tool calls", issue) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 101 and get_in(payload, ["result", "success"]) == false and String.contains?( get_in(payload, ["result", "output"]), "Unsupported dynamic tool" ) else false end end) after File.rm_rf(test_root) end end test "app server executes supported dynamic tool calls and returns the tool result" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-supported-tool-call-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-90A") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-supported-tool-call.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-supported-tool-call.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-90a\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-90a\"}}}' printf '%s\\n' '{\"id\":102,\"method\":\"item/tool/call\",\"params\":{\"name\":\"linear_graphql\",\"callId\":\"call-90a\",\"threadId\":\"thread-90a\",\"turnId\":\"turn-90a\",\"arguments\":{\"query\":\"query Viewer { viewer { id } }\",\"variables\":{\"includeTeams\":false}}}}' ;; 5) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-supported-tool-call", identifier: "MT-90A", title: "Supported tool call", description: "Ensure supported tool calls return tool output", state: "In Progress", url: "https://example.org/issues/MT-90A", labels: ["backend"] } test_pid = self() tool_executor = fn tool, arguments -> send(test_pid, {:tool_called, tool, arguments}) %{ "success" => true, "contentItems" => [ %{ "type" => "inputText", "text" => ~s({"data":{"viewer":{"id":"usr_123"}}}) } ] } end assert {:ok, _result} = AppServer.run(workspace, "Handle supported tool calls", issue, tool_executor: tool_executor) assert_received {:tool_called, "linear_graphql", %{ "query" => "query Viewer { viewer { id } }", "variables" => %{"includeTeams" => false} }} trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do payload = line |> String.trim_leading("JSON:") |> Jason.decode!() payload["id"] == 102 and get_in(payload, ["result", "success"]) == true and get_in(payload, ["result", "output"]) == ~s({"data":{"viewer":{"id":"usr_123"}}}) else false end end) after File.rm_rf(test_root) end end test "app server emits tool_call_failed for supported tool failures" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-tool-call-failed-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-90B") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-tool-call-failed.trace") previous_trace = System.get_env("SYMP_TEST_CODEx_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODEx_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODEx_TRACE") end end) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex-tool-call-failed.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-90b\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-90b\"}}}' printf '%s\\n' '{\"id\":103,\"method\":\"item/tool/call\",\"params\":{\"tool\":\"linear_graphql\",\"callId\":\"call-90b\",\"threadId\":\"thread-90b\",\"turnId\":\"turn-90b\",\"arguments\":{\"query\":\"query Viewer { viewer { id } }\"}}}' ;; 5) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-tool-call-failed", identifier: "MT-90B", title: "Tool call failed", description: "Ensure supported tool failures emit a distinct event", state: "In Progress", url: "https://example.org/issues/MT-90B", labels: ["backend"] } test_pid = self() tool_executor = fn tool, arguments -> send(test_pid, {:tool_called, tool, arguments}) %{ "success" => false, "contentItems" => [ %{ "type" => "inputText", "text" => ~s({"error":{"message":"boom"}}) } ] } end on_message = fn message -> send(test_pid, {:app_server_message, message}) end assert {:ok, _result} = AppServer.run(workspace, "Handle failed tool calls", issue, on_message: on_message, tool_executor: tool_executor ) assert_received {:tool_called, "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}} assert_received {:app_server_message, %{event: :tool_call_failed, payload: %{"params" => %{"tool" => "linear_graphql"}}}} after File.rm_rf(test_root) end end test "app server buffers partial JSON lines until newline terminator" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-partial-line-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-91") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh count=0 while IFS= read -r line; do count=$((count + 1)) case "$count" in 1) padding=$(printf '%*s' 1100000 '' | tr ' ' a) printf '{"id":1,"result":{},"padding":"%s"}\\n' "$padding" ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-91"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-91"}}}' ;; 4) printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-partial-line", identifier: "MT-91", title: "Partial line decode", description: "Ensure JSON parsing waits for newline-delimited messages", state: "In Progress", url: "https://example.org/issues/MT-91", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Validate newline-delimited buffering", issue) after File.rm_rf(test_root) end end test "app server captures codex side output and logs it through Logger" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-stderr-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-92") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh count=0 while IFS= read -r line; do count=$((count + 1)) case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-92"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-92"}}}' ;; 4) printf '%s\\n' 'warning: this is stderr noise' >&2 printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-stderr", identifier: "MT-92", title: "Capture stderr", description: "Ensure codex stderr is captured and logged", state: "In Progress", url: "https://example.org/issues/MT-92", labels: ["backend"] } test_pid = self() on_message = fn message -> send(test_pid, {:app_server_message, message}) end log = capture_log(fn -> assert {:ok, _result} = AppServer.run(workspace, "Capture stderr log", issue, on_message: on_message) end) assert_received {:app_server_message, %{event: :turn_completed}} refute_received {:app_server_message, %{event: :malformed}} assert log =~ "Codex turn stream output: warning: this is stderr noise" after File.rm_rf(test_root) end end test "app server emits malformed events for JSON-like protocol lines that fail to decode" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-malformed-protocol-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-93") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh count=0 while IFS= read -r line; do count=$((count + 1)) case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-93"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-93"}}}' ;; 4) printf '%s\\n' '{"method":"turn/completed"' printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-malformed-protocol", identifier: "MT-93", title: "Malformed protocol frame", description: "Ensure malformed JSON-like frames are surfaced to the orchestrator", state: "In Progress", url: "https://example.org/issues/MT-93", labels: ["backend"] } test_pid = self() on_message = fn message -> send(test_pid, {:app_server_message, message}) end assert {:ok, _result} = AppServer.run(workspace, "Capture malformed protocol line", issue, on_message: on_message) assert_received {:app_server_message, %{event: :malformed, payload: "{\"method\":\"turn/completed\""}} assert_received {:app_server_message, %{event: :turn_completed}} after File.rm_rf(test_root) end end test "app server launches over ssh for remote workers" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-remote-ssh-#{System.unique_integer([:positive])}" ) previous_path = System.get_env("PATH") previous_trace = System.get_env("SYMP_TEST_SSH_TRACE") on_exit(fn -> restore_env("PATH", previous_path) restore_env("SYMP_TEST_SSH_TRACE", previous_trace) end) try do trace_file = Path.join(test_root, "ssh.trace") fake_ssh = Path.join(test_root, "ssh") remote_workspace = "/remote/workspaces/MT-REMOTE" File.mkdir_p!(test_root) System.put_env("SYMP_TEST_SSH_TRACE", trace_file) System.put_env("PATH", test_root <> ":" <> (previous_path || "")) File.write!(fake_ssh, """ #!/bin/sh trace_file="${SYMP_TEST_SSH_TRACE:-/tmp/symphony-fake-ssh.trace}" count=0 printf 'ARGV:%s\\n' "$*" >> "$trace_file" while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' "$line" >> "$trace_file" case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-remote"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-remote"}}}' ;; 4) printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(fake_ssh, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: "/remote/workspaces", codex_command: "fake-remote-codex app-server" ) issue = %Issue{ id: "issue-remote", identifier: "MT-REMOTE", title: "Run remote app server", description: "Validate ssh-backed codex startup", state: "In Progress", url: "https://example.org/issues/MT-REMOTE", labels: ["backend"] } assert {:ok, _result} = AppServer.run( remote_workspace, "Run remote worker", issue, worker_host: "worker-01:2200" ) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert argv_line = Enum.find(lines, &String.starts_with?(&1, "ARGV:")) assert argv_line =~ "-T -p 2200 worker-01 bash -lc" assert argv_line =~ "cd " assert argv_line =~ remote_workspace assert argv_line =~ "exec " assert argv_line =~ "fake-remote-codex app-server" expected_turn_policy = %{ "type" => "workspaceWrite", "writableRoots" => [remote_workspace], "readOnlyAccess" => %{"type" => "fullAccess"}, "networkAccess" => false, "excludeTmpdirEnvVar" => false, "excludeSlashTmp" => false } assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> payload["method"] == "thread/start" && get_in(payload, ["params", "cwd"]) == remote_workspace end) else false end end) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> payload["method"] == "turn/start" && get_in(payload, ["params", "cwd"]) == remote_workspace && get_in(payload, ["params", "sandboxPolicy"]) == expected_turn_policy end) else false end end) after File.rm_rf(test_root) end end end ================================================ FILE: elixir/test/symphony_elixir/cli_test.exs ================================================ defmodule SymphonyElixir.CLITest do use ExUnit.Case, async: true alias SymphonyElixir.CLI @ack_flag "--i-understand-that-this-will-be-running-without-the-usual-guardrails" test "returns the guardrails acknowledgement banner when the flag is missing" do parent = self() deps = %{ file_regular?: fn _path -> send(parent, :file_checked) true end, set_workflow_file_path: fn _path -> send(parent, :workflow_set) :ok end, set_logs_root: fn _path -> send(parent, :logs_root_set) :ok end, set_server_port_override: fn _port -> send(parent, :port_set) :ok end, ensure_all_started: fn -> send(parent, :started) {:ok, [:symphony_elixir]} end } assert {:error, banner} = CLI.evaluate(["WORKFLOW.md"], deps) assert banner =~ "This Symphony implementation is a low key engineering preview." assert banner =~ "Codex will run without any guardrails." assert banner =~ "SymphonyElixir is not a supported product and is presented as-is." assert banner =~ @ack_flag refute_received :file_checked refute_received :workflow_set refute_received :logs_root_set refute_received :port_set refute_received :started end test "defaults to WORKFLOW.md when workflow path is missing" do deps = %{ file_regular?: fn path -> Path.basename(path) == "WORKFLOW.md" end, set_workflow_file_path: fn _path -> :ok end, set_logs_root: fn _path -> :ok end, set_server_port_override: fn _port -> :ok end, ensure_all_started: fn -> {:ok, [:symphony_elixir]} end } assert :ok = CLI.evaluate([@ack_flag], deps) end test "uses an explicit workflow path override when provided" do parent = self() workflow_path = "tmp/custom/WORKFLOW.md" expanded_path = Path.expand(workflow_path) deps = %{ file_regular?: fn path -> send(parent, {:workflow_checked, path}) path == expanded_path end, set_workflow_file_path: fn path -> send(parent, {:workflow_set, path}) :ok end, set_logs_root: fn _path -> :ok end, set_server_port_override: fn _port -> :ok end, ensure_all_started: fn -> {:ok, [:symphony_elixir]} end } assert :ok = CLI.evaluate([@ack_flag, workflow_path], deps) assert_received {:workflow_checked, ^expanded_path} assert_received {:workflow_set, ^expanded_path} end test "accepts --logs-root and passes an expanded root to runtime deps" do parent = self() deps = %{ file_regular?: fn _path -> true end, set_workflow_file_path: fn _path -> :ok end, set_logs_root: fn path -> send(parent, {:logs_root, path}) :ok end, set_server_port_override: fn _port -> :ok end, ensure_all_started: fn -> {:ok, [:symphony_elixir]} end } assert :ok = CLI.evaluate([@ack_flag, "--logs-root", "tmp/custom-logs", "WORKFLOW.md"], deps) assert_received {:logs_root, expanded_path} assert expanded_path == Path.expand("tmp/custom-logs") end test "returns not found when workflow file does not exist" do deps = %{ file_regular?: fn _path -> false end, set_workflow_file_path: fn _path -> :ok end, set_logs_root: fn _path -> :ok end, set_server_port_override: fn _port -> :ok end, ensure_all_started: fn -> {:ok, [:symphony_elixir]} end } assert {:error, message} = CLI.evaluate([@ack_flag, "WORKFLOW.md"], deps) assert message =~ "Workflow file not found:" end test "returns startup error when app cannot start" do deps = %{ file_regular?: fn _path -> true end, set_workflow_file_path: fn _path -> :ok end, set_logs_root: fn _path -> :ok end, set_server_port_override: fn _port -> :ok end, ensure_all_started: fn -> {:error, :boom} end } assert {:error, message} = CLI.evaluate([@ack_flag, "WORKFLOW.md"], deps) assert message =~ "Failed to start Symphony with workflow" assert message =~ ":boom" end test "returns ok when workflow exists and app starts" do deps = %{ file_regular?: fn _path -> true end, set_workflow_file_path: fn _path -> :ok end, set_logs_root: fn _path -> :ok end, set_server_port_override: fn _port -> :ok end, ensure_all_started: fn -> {:ok, [:symphony_elixir]} end } assert :ok = CLI.evaluate([@ack_flag, "WORKFLOW.md"], deps) end end ================================================ FILE: elixir/test/symphony_elixir/core_test.exs ================================================ defmodule SymphonyElixir.CoreTest do use SymphonyElixir.TestSupport test "config defaults and validation checks" do write_workflow_file!(Workflow.workflow_file_path(), tracker_api_token: nil, tracker_project_slug: nil, poll_interval_ms: nil, tracker_active_states: nil, tracker_terminal_states: nil, codex_command: nil ) config = Config.settings!() assert config.polling.interval_ms == 30_000 assert config.tracker.active_states == ["Todo", "In Progress"] assert config.tracker.terminal_states == ["Closed", "Cancelled", "Canceled", "Duplicate", "Done"] assert config.tracker.assignee == nil assert config.agent.max_turns == 20 write_workflow_file!(Workflow.workflow_file_path(), poll_interval_ms: "invalid") assert_raise ArgumentError, ~r/interval_ms/, fn -> Config.settings!().polling.interval_ms end assert {:error, {:invalid_workflow_config, message}} = Config.validate!() assert message =~ "polling.interval_ms" write_workflow_file!(Workflow.workflow_file_path(), poll_interval_ms: 45_000) assert Config.settings!().polling.interval_ms == 45_000 write_workflow_file!(Workflow.workflow_file_path(), max_turns: 0) assert {:error, {:invalid_workflow_config, message}} = Config.validate!() assert message =~ "agent.max_turns" write_workflow_file!(Workflow.workflow_file_path(), max_turns: 5) assert Config.settings!().agent.max_turns == 5 write_workflow_file!(Workflow.workflow_file_path(), tracker_active_states: "Todo, Review,") assert {:error, {:invalid_workflow_config, message}} = Config.validate!() assert message =~ "tracker.active_states" write_workflow_file!(Workflow.workflow_file_path(), tracker_api_token: "token", tracker_project_slug: nil ) assert {:error, :missing_linear_project_slug} = Config.validate!() write_workflow_file!(Workflow.workflow_file_path(), tracker_project_slug: "project", codex_command: "" ) assert {:error, {:invalid_workflow_config, message}} = Config.validate!() assert message =~ "codex.command" assert message =~ "can't be blank" write_workflow_file!(Workflow.workflow_file_path(), codex_command: " ") assert :ok = Config.validate!() assert Config.settings!().codex.command == " " write_workflow_file!(Workflow.workflow_file_path(), codex_command: "/bin/sh app-server") assert :ok = Config.validate!() write_workflow_file!(Workflow.workflow_file_path(), codex_approval_policy: "definitely-not-valid") assert :ok = Config.validate!() write_workflow_file!(Workflow.workflow_file_path(), codex_thread_sandbox: "unsafe-ish") assert :ok = Config.validate!() write_workflow_file!(Workflow.workflow_file_path(), codex_turn_sandbox_policy: %{type: "workspaceWrite", writableRoots: ["relative/path"]} ) assert :ok = Config.validate!() write_workflow_file!(Workflow.workflow_file_path(), codex_approval_policy: 123) assert {:error, {:invalid_workflow_config, message}} = Config.validate!() assert message =~ "codex.approval_policy" write_workflow_file!(Workflow.workflow_file_path(), codex_thread_sandbox: 123) assert {:error, {:invalid_workflow_config, message}} = Config.validate!() assert message =~ "codex.thread_sandbox" write_workflow_file!(Workflow.workflow_file_path(), tracker_kind: "123") assert {:error, {:unsupported_tracker_kind, "123"}} = Config.validate!() end test "current WORKFLOW.md file is valid and complete" do original_workflow_path = Workflow.workflow_file_path() on_exit(fn -> Workflow.set_workflow_file_path(original_workflow_path) end) Workflow.clear_workflow_file_path() assert {:ok, %{config: config, prompt: prompt}} = Workflow.load() assert is_map(config) tracker = Map.get(config, "tracker", %{}) assert is_map(tracker) assert Map.get(tracker, "kind") == "linear" assert is_binary(Map.get(tracker, "project_slug")) assert is_list(Map.get(tracker, "active_states")) assert is_list(Map.get(tracker, "terminal_states")) hooks = Map.get(config, "hooks", %{}) assert is_map(hooks) assert Map.get(hooks, "after_create") =~ "git clone --depth 1 https://github.com/openai/symphony ." assert Map.get(hooks, "after_create") =~ "cd elixir && mise trust" assert Map.get(hooks, "after_create") =~ "mise exec -- mix deps.get" assert Map.get(hooks, "before_remove") =~ "cd elixir && mise exec -- mix workspace.before_remove" assert String.trim(prompt) != "" assert is_binary(Config.workflow_prompt()) assert Config.workflow_prompt() == prompt end test "linear api token resolves from LINEAR_API_KEY env var" do previous_linear_api_key = System.get_env("LINEAR_API_KEY") env_api_key = "test-linear-api-key" on_exit(fn -> restore_env("LINEAR_API_KEY", previous_linear_api_key) end) System.put_env("LINEAR_API_KEY", env_api_key) write_workflow_file!(Workflow.workflow_file_path(), tracker_api_token: nil, tracker_project_slug: "project", codex_command: "/bin/sh app-server" ) assert Config.settings!().tracker.api_key == env_api_key assert Config.settings!().tracker.project_slug == "project" assert :ok = Config.validate!() end test "linear assignee resolves from LINEAR_ASSIGNEE env var" do previous_linear_assignee = System.get_env("LINEAR_ASSIGNEE") env_assignee = "dev@example.com" on_exit(fn -> restore_env("LINEAR_ASSIGNEE", previous_linear_assignee) end) System.put_env("LINEAR_ASSIGNEE", env_assignee) write_workflow_file!(Workflow.workflow_file_path(), tracker_assignee: nil, tracker_project_slug: "project", codex_command: "/bin/sh app-server" ) assert Config.settings!().tracker.assignee == env_assignee end test "workflow file path defaults to WORKFLOW.md in the current working directory when app env is unset" do original_workflow_path = Workflow.workflow_file_path() on_exit(fn -> Workflow.set_workflow_file_path(original_workflow_path) end) Workflow.clear_workflow_file_path() assert Workflow.workflow_file_path() == Path.join(File.cwd!(), "WORKFLOW.md") end test "workflow file path resolves from app env when set" do app_workflow_path = "/tmp/app/WORKFLOW.md" on_exit(fn -> Workflow.clear_workflow_file_path() end) Workflow.set_workflow_file_path(app_workflow_path) assert Workflow.workflow_file_path() == app_workflow_path end test "workflow load accepts prompt-only files without front matter" do workflow_path = Path.join(Path.dirname(Workflow.workflow_file_path()), "PROMPT_ONLY_WORKFLOW.md") File.write!(workflow_path, "Prompt only\n") assert {:ok, %{config: %{}, prompt: "Prompt only", prompt_template: "Prompt only"}} = Workflow.load(workflow_path) end test "workflow load accepts unterminated front matter with an empty prompt" do workflow_path = Path.join(Path.dirname(Workflow.workflow_file_path()), "UNTERMINATED_WORKFLOW.md") File.write!(workflow_path, "---\ntracker:\n kind: linear\n") assert {:ok, %{config: %{"tracker" => %{"kind" => "linear"}}, prompt: "", prompt_template: ""}} = Workflow.load(workflow_path) end test "workflow load rejects non-map front matter" do workflow_path = Path.join(Path.dirname(Workflow.workflow_file_path()), "INVALID_FRONT_MATTER_WORKFLOW.md") File.write!(workflow_path, "---\n- not-a-map\n---\nPrompt body\n") assert {:error, :workflow_front_matter_not_a_map} = Workflow.load(workflow_path) end test "SymphonyElixir.start_link delegates to the orchestrator" do write_workflow_file!(Workflow.workflow_file_path(), tracker_kind: "memory") Application.put_env(:symphony_elixir, :memory_tracker_issues, []) orchestrator_pid = Process.whereis(SymphonyElixir.Orchestrator) on_exit(fn -> if is_nil(Process.whereis(SymphonyElixir.Orchestrator)) do case Supervisor.restart_child(SymphonyElixir.Supervisor, SymphonyElixir.Orchestrator) do {:ok, _pid} -> :ok {:error, {:already_started, _pid}} -> :ok end end end) if is_pid(orchestrator_pid) do assert :ok = Supervisor.terminate_child(SymphonyElixir.Supervisor, SymphonyElixir.Orchestrator) end assert {:ok, pid} = SymphonyElixir.start_link() assert Process.whereis(SymphonyElixir.Orchestrator) == pid GenServer.stop(pid) end test "linear issue state reconciliation fetch with no running issues is a no-op" do assert {:ok, []} = Client.fetch_issue_states_by_ids([]) end test "non-active issue state stops running agent without cleaning workspace" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-nonactive-reconcile-#{System.unique_integer([:positive])}" ) issue_id = "issue-1" issue_identifier = "MT-555" workspace = Path.join(test_root, issue_identifier) try do write_workflow_file!(Workflow.workflow_file_path(), workspace_root: test_root, tracker_active_states: ["Todo", "In Progress", "In Review"], tracker_terminal_states: ["Closed", "Cancelled", "Canceled", "Duplicate"] ) File.mkdir_p!(test_root) File.mkdir_p!(workspace) agent_pid = spawn(fn -> receive do :stop -> :ok end end) state = %Orchestrator.State{ running: %{ issue_id => %{ pid: agent_pid, ref: nil, identifier: issue_identifier, issue: %Issue{id: issue_id, state: "Todo", identifier: issue_identifier}, started_at: DateTime.utc_now() } }, claimed: MapSet.new([issue_id]), codex_totals: %{input_tokens: 0, output_tokens: 0, total_tokens: 0, seconds_running: 0}, retry_attempts: %{} } issue = %Issue{ id: issue_id, identifier: issue_identifier, state: "Backlog", title: "Queued", description: "Not started", labels: [] } updated_state = Orchestrator.reconcile_issue_states_for_test([issue], state) refute Map.has_key?(updated_state.running, issue_id) refute MapSet.member?(updated_state.claimed, issue_id) refute Process.alive?(agent_pid) assert File.exists?(workspace) after File.rm_rf(test_root) end end test "terminal issue state stops running agent and cleans workspace" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-terminal-reconcile-#{System.unique_integer([:positive])}" ) issue_id = "issue-2" issue_identifier = "MT-556" workspace = Path.join(test_root, issue_identifier) try do write_workflow_file!(Workflow.workflow_file_path(), workspace_root: test_root, tracker_active_states: ["Todo", "In Progress", "In Review"], tracker_terminal_states: ["Closed", "Cancelled", "Canceled", "Duplicate"] ) File.mkdir_p!(test_root) File.mkdir_p!(workspace) agent_pid = spawn(fn -> receive do :stop -> :ok end end) state = %Orchestrator.State{ running: %{ issue_id => %{ pid: agent_pid, ref: nil, identifier: issue_identifier, issue: %Issue{id: issue_id, state: "In Progress", identifier: issue_identifier}, started_at: DateTime.utc_now() } }, claimed: MapSet.new([issue_id]), codex_totals: %{input_tokens: 0, output_tokens: 0, total_tokens: 0, seconds_running: 0}, retry_attempts: %{} } issue = %Issue{ id: issue_id, identifier: issue_identifier, state: "Closed", title: "Done", description: "Completed", labels: [] } updated_state = Orchestrator.reconcile_issue_states_for_test([issue], state) refute Map.has_key?(updated_state.running, issue_id) refute MapSet.member?(updated_state.claimed, issue_id) refute Process.alive?(agent_pid) refute File.exists?(workspace) after File.rm_rf(test_root) end end test "missing running issues stop active agents without cleaning the workspace" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-missing-running-reconcile-#{System.unique_integer([:positive])}" ) previous_memory_issues = Application.get_env(:symphony_elixir, :memory_tracker_issues) issue_id = "issue-missing" issue_identifier = "MT-557" try do write_workflow_file!(Workflow.workflow_file_path(), tracker_kind: "memory", workspace_root: test_root, tracker_active_states: ["Todo", "In Progress", "In Review"], tracker_terminal_states: ["Closed", "Cancelled", "Canceled", "Duplicate"], poll_interval_ms: 30_000 ) Application.put_env(:symphony_elixir, :memory_tracker_issues, []) orchestrator_name = Module.concat(__MODULE__, :MissingRunningIssueOrchestrator) {:ok, pid} = Orchestrator.start_link(name: orchestrator_name) on_exit(fn -> restore_app_env(:memory_tracker_issues, previous_memory_issues) if Process.alive?(pid) do Process.exit(pid, :normal) end end) Process.sleep(50) assert {:ok, workspace} = SymphonyElixir.PathSafety.canonicalize(Path.join(test_root, issue_identifier)) File.mkdir_p!(workspace) agent_pid = spawn(fn -> receive do :stop -> :ok end end) initial_state = :sys.get_state(pid) running_entry = %{ pid: agent_pid, ref: nil, identifier: issue_identifier, issue: %Issue{id: issue_id, state: "In Progress", identifier: issue_identifier}, started_at: DateTime.utc_now() } :sys.replace_state(pid, fn _ -> initial_state |> Map.put(:running, %{issue_id => running_entry}) |> Map.put(:claimed, MapSet.new([issue_id])) |> Map.put(:retry_attempts, %{}) end) send(pid, :tick) Process.sleep(100) state = :sys.get_state(pid) refute Map.has_key?(state.running, issue_id) refute MapSet.member?(state.claimed, issue_id) refute Process.alive?(agent_pid) assert File.exists?(workspace) after restore_app_env(:memory_tracker_issues, previous_memory_issues) File.rm_rf(test_root) end end test "reconcile updates running issue state for active issues" do issue_id = "issue-3" state = %Orchestrator.State{ running: %{ issue_id => %{ pid: self(), ref: nil, identifier: "MT-557", issue: %Issue{ id: issue_id, identifier: "MT-557", state: "Todo" }, started_at: DateTime.utc_now() } }, claimed: MapSet.new([issue_id]), codex_totals: %{input_tokens: 0, output_tokens: 0, total_tokens: 0, seconds_running: 0}, retry_attempts: %{} } issue = %Issue{ id: issue_id, identifier: "MT-557", state: "In Progress", title: "Active state refresh", description: "State should be refreshed", labels: [] } updated_state = Orchestrator.reconcile_issue_states_for_test([issue], state) updated_entry = updated_state.running[issue_id] assert Map.has_key?(updated_state.running, issue_id) assert MapSet.member?(updated_state.claimed, issue_id) assert updated_entry.issue.state == "In Progress" end test "reconcile stops running issue when it is reassigned away from this worker" do issue_id = "issue-reassigned" agent_pid = spawn(fn -> receive do :stop -> :ok end end) state = %Orchestrator.State{ running: %{ issue_id => %{ pid: agent_pid, ref: nil, identifier: "MT-561", issue: %Issue{ id: issue_id, identifier: "MT-561", state: "In Progress", assigned_to_worker: true }, started_at: DateTime.utc_now() } }, claimed: MapSet.new([issue_id]), codex_totals: %{input_tokens: 0, output_tokens: 0, total_tokens: 0, seconds_running: 0}, retry_attempts: %{} } issue = %Issue{ id: issue_id, identifier: "MT-561", state: "In Progress", title: "Reassigned active issue", description: "Worker should stop", labels: [], assigned_to_worker: false } updated_state = Orchestrator.reconcile_issue_states_for_test([issue], state) refute Map.has_key?(updated_state.running, issue_id) refute MapSet.member?(updated_state.claimed, issue_id) refute Process.alive?(agent_pid) end test "normal worker exit schedules active-state continuation retry" do issue_id = "issue-resume" ref = make_ref() orchestrator_name = Module.concat(__MODULE__, :ContinuationOrchestrator) {:ok, pid} = Orchestrator.start_link(name: orchestrator_name) on_exit(fn -> if Process.alive?(pid) do Process.exit(pid, :normal) end end) initial_state = :sys.get_state(pid) running_entry = %{ pid: self(), ref: ref, identifier: "MT-558", issue: %Issue{id: issue_id, identifier: "MT-558", state: "In Progress"}, started_at: DateTime.utc_now() } :sys.replace_state(pid, fn _ -> initial_state |> Map.put(:running, %{issue_id => running_entry}) |> Map.put(:claimed, MapSet.new([issue_id])) |> Map.put(:retry_attempts, %{}) end) send(pid, {:DOWN, ref, :process, self(), :normal}) Process.sleep(50) state = :sys.get_state(pid) refute Map.has_key?(state.running, issue_id) assert MapSet.member?(state.completed, issue_id) assert %{attempt: 1, due_at_ms: due_at_ms} = state.retry_attempts[issue_id] assert is_integer(due_at_ms) assert_due_in_range(due_at_ms, 500, 1_100) end test "abnormal worker exit increments retry attempt progressively" do issue_id = "issue-crash" ref = make_ref() orchestrator_name = Module.concat(__MODULE__, :CrashRetryOrchestrator) {:ok, pid} = Orchestrator.start_link(name: orchestrator_name) on_exit(fn -> if Process.alive?(pid) do Process.exit(pid, :normal) end end) initial_state = :sys.get_state(pid) running_entry = %{ pid: self(), ref: ref, identifier: "MT-559", retry_attempt: 2, issue: %Issue{id: issue_id, identifier: "MT-559", state: "In Progress"}, started_at: DateTime.utc_now() } :sys.replace_state(pid, fn _ -> initial_state |> Map.put(:running, %{issue_id => running_entry}) |> Map.put(:claimed, MapSet.new([issue_id])) |> Map.put(:retry_attempts, %{}) end) send(pid, {:DOWN, ref, :process, self(), :boom}) Process.sleep(50) state = :sys.get_state(pid) assert %{attempt: 3, due_at_ms: due_at_ms, identifier: "MT-559", error: "agent exited: :boom"} = state.retry_attempts[issue_id] assert_due_in_range(due_at_ms, 39_500, 40_500) end test "first abnormal worker exit waits before retrying" do issue_id = "issue-crash-initial" ref = make_ref() orchestrator_name = Module.concat(__MODULE__, :InitialCrashRetryOrchestrator) {:ok, pid} = Orchestrator.start_link(name: orchestrator_name) on_exit(fn -> if Process.alive?(pid) do Process.exit(pid, :normal) end end) initial_state = :sys.get_state(pid) running_entry = %{ pid: self(), ref: ref, identifier: "MT-560", issue: %Issue{id: issue_id, identifier: "MT-560", state: "In Progress"}, started_at: DateTime.utc_now() } :sys.replace_state(pid, fn _ -> initial_state |> Map.put(:running, %{issue_id => running_entry}) |> Map.put(:claimed, MapSet.new([issue_id])) |> Map.put(:retry_attempts, %{}) end) send(pid, {:DOWN, ref, :process, self(), :boom}) Process.sleep(50) state = :sys.get_state(pid) assert %{attempt: 1, due_at_ms: due_at_ms, identifier: "MT-560", error: "agent exited: :boom"} = state.retry_attempts[issue_id] assert_due_in_range(due_at_ms, 9_000, 10_500) end test "stale retry timer messages do not consume newer retry entries" do issue_id = "issue-stale-retry" orchestrator_name = Module.concat(__MODULE__, :StaleRetryOrchestrator) {:ok, pid} = Orchestrator.start_link(name: orchestrator_name) on_exit(fn -> if Process.alive?(pid) do Process.exit(pid, :normal) end end) initial_state = :sys.get_state(pid) current_retry_token = make_ref() stale_retry_token = make_ref() :sys.replace_state(pid, fn _ -> initial_state |> Map.put(:retry_attempts, %{ issue_id => %{ attempt: 2, timer_ref: nil, retry_token: current_retry_token, due_at_ms: System.monotonic_time(:millisecond) + 30_000, identifier: "MT-561", error: "agent exited: :boom" } }) end) send(pid, {:retry_issue, issue_id, stale_retry_token}) Process.sleep(50) assert %{ attempt: 2, retry_token: ^current_retry_token, identifier: "MT-561", error: "agent exited: :boom" } = :sys.get_state(pid).retry_attempts[issue_id] end test "manual refresh coalesces repeated requests and ignores superseded ticks" do now_ms = System.monotonic_time(:millisecond) stale_tick_token = make_ref() state = %Orchestrator.State{ poll_interval_ms: 30_000, max_concurrent_agents: 1, next_poll_due_at_ms: now_ms + 30_000, poll_check_in_progress: false, tick_timer_ref: nil, tick_token: stale_tick_token, codex_totals: %{input_tokens: 0, output_tokens: 0, total_tokens: 0, seconds_running: 0}, codex_rate_limits: nil } assert {:reply, %{queued: true, coalesced: false}, refreshed_state} = Orchestrator.handle_call(:request_refresh, {self(), make_ref()}, state) assert is_reference(refreshed_state.tick_timer_ref) assert is_reference(refreshed_state.tick_token) refute refreshed_state.tick_token == stale_tick_token assert refreshed_state.next_poll_due_at_ms <= System.monotonic_time(:millisecond) assert {:reply, %{queued: true, coalesced: true}, coalesced_state} = Orchestrator.handle_call(:request_refresh, {self(), make_ref()}, refreshed_state) assert coalesced_state.tick_token == refreshed_state.tick_token assert {:noreply, ^coalesced_state} = Orchestrator.handle_info({:tick, stale_tick_token}, coalesced_state) end test "select_worker_host_for_test skips full ssh hosts under the shared per-host cap" do write_workflow_file!(Workflow.workflow_file_path(), worker_ssh_hosts: ["worker-a", "worker-b"], worker_max_concurrent_agents_per_host: 1 ) state = %Orchestrator.State{ running: %{ "issue-1" => %{worker_host: "worker-a"} } } assert Orchestrator.select_worker_host_for_test(state, nil) == "worker-b" end test "select_worker_host_for_test returns no_worker_capacity when every ssh host is full" do write_workflow_file!(Workflow.workflow_file_path(), worker_ssh_hosts: ["worker-a", "worker-b"], worker_max_concurrent_agents_per_host: 1 ) state = %Orchestrator.State{ running: %{ "issue-1" => %{worker_host: "worker-a"}, "issue-2" => %{worker_host: "worker-b"} } } assert Orchestrator.select_worker_host_for_test(state, nil) == :no_worker_capacity end test "select_worker_host_for_test keeps the preferred ssh host when it still has capacity" do write_workflow_file!(Workflow.workflow_file_path(), worker_ssh_hosts: ["worker-a", "worker-b"], worker_max_concurrent_agents_per_host: 2 ) state = %Orchestrator.State{ running: %{ "issue-1" => %{worker_host: "worker-a"}, "issue-2" => %{worker_host: "worker-b"} } } assert Orchestrator.select_worker_host_for_test(state, "worker-a") == "worker-a" end defp assert_due_in_range(due_at_ms, min_remaining_ms, max_remaining_ms) do remaining_ms = due_at_ms - System.monotonic_time(:millisecond) assert remaining_ms >= min_remaining_ms assert remaining_ms <= max_remaining_ms end defp restore_app_env(key, nil), do: Application.delete_env(:symphony_elixir, key) defp restore_app_env(key, value), do: Application.put_env(:symphony_elixir, key, value) test "fetch issues by states with empty state set is a no-op" do assert {:ok, []} = Client.fetch_issues_by_states([]) end test "prompt builder renders issue and attempt values from workflow template" do workflow_prompt = "Ticket {{ issue.identifier }} {{ issue.title }} labels={{ issue.labels }} attempt={{ attempt }}" write_workflow_file!(Workflow.workflow_file_path(), prompt: workflow_prompt) issue = %Issue{ identifier: "S-1", title: "Refactor backend request path", description: "Replace transport layer", state: "Todo", url: "https://example.org/issues/S-1", labels: ["backend"] } prompt = PromptBuilder.build_prompt(issue, attempt: 3) assert prompt =~ "Ticket S-1 Refactor backend request path" assert prompt =~ "labels=backend" assert prompt =~ "attempt=3" end test "prompt builder renders issue datetime fields without crashing" do workflow_prompt = "Ticket {{ issue.identifier }} created={{ issue.created_at }} updated={{ issue.updated_at }}" write_workflow_file!(Workflow.workflow_file_path(), prompt: workflow_prompt) created_at = DateTime.from_naive!(~N[2026-02-26 18:06:48], "Etc/UTC") updated_at = DateTime.from_naive!(~N[2026-02-26 18:07:03], "Etc/UTC") issue = %Issue{ identifier: "MT-697", title: "Live smoke", description: "Prompt should serialize datetimes", state: "Todo", url: "https://example.org/issues/MT-697", labels: [], created_at: created_at, updated_at: updated_at } prompt = PromptBuilder.build_prompt(issue) assert prompt =~ "Ticket MT-697" assert prompt =~ "created=2026-02-26T18:06:48Z" assert prompt =~ "updated=2026-02-26T18:07:03Z" end test "prompt builder normalizes nested date-like values, maps, and structs in issue fields" do write_workflow_file!(Workflow.workflow_file_path(), prompt: "Ticket {{ issue.identifier }}") issue = %Issue{ identifier: "MT-701", title: "Serialize nested values", description: "Prompt builder should normalize nested terms", state: "Todo", url: "https://example.org/issues/MT-701", labels: [ ~N[2026-02-27 12:34:56], ~D[2026-02-28], ~T[12:34:56], %{phase: "test"}, URI.parse("https://example.org/issues/MT-701") ] } assert PromptBuilder.build_prompt(issue) == "Ticket MT-701" end test "prompt builder uses strict variable rendering" do workflow_prompt = "Work on ticket {{ missing.ticket_id }} and follow these steps." write_workflow_file!(Workflow.workflow_file_path(), prompt: workflow_prompt) issue = %Issue{ identifier: "MT-123", title: "Investigate broken sync", description: "Reproduce and fix", state: "In Progress", url: "https://example.org/issues/MT-123", labels: ["bug"] } assert_raise Solid.RenderError, fn -> PromptBuilder.build_prompt(issue) end end test "prompt builder surfaces invalid template content with prompt context" do write_workflow_file!(Workflow.workflow_file_path(), prompt: "{% if issue.identifier %}") issue = %Issue{ identifier: "MT-999", title: "Broken prompt", description: "Invalid template syntax", state: "Todo", url: "https://example.org/issues/MT-999", labels: [] } assert_raise RuntimeError, ~r/template_parse_error:.*template="/s, fn -> PromptBuilder.build_prompt(issue) end end test "prompt builder uses a sensible default template when workflow prompt is blank" do write_workflow_file!(Workflow.workflow_file_path(), prompt: " \n") issue = %Issue{ identifier: "MT-777", title: "Make fallback prompt useful", description: "Include enough issue context to start working.", state: "In Progress", url: "https://example.org/issues/MT-777", labels: ["prompt"] } prompt = PromptBuilder.build_prompt(issue) assert prompt =~ "You are working on a Linear issue." assert prompt =~ "Identifier: MT-777" assert prompt =~ "Title: Make fallback prompt useful" assert prompt =~ "Body:" assert prompt =~ "Include enough issue context to start working." assert Config.workflow_prompt() =~ "{{ issue.identifier }}" assert Config.workflow_prompt() =~ "{{ issue.title }}" assert Config.workflow_prompt() =~ "{{ issue.description }}" end test "prompt builder default template handles missing issue body" do write_workflow_file!(Workflow.workflow_file_path(), prompt: "") issue = %Issue{ identifier: "MT-778", title: "Handle empty body", description: nil, state: "Todo", url: "https://example.org/issues/MT-778", labels: [] } prompt = PromptBuilder.build_prompt(issue) assert prompt =~ "Identifier: MT-778" assert prompt =~ "Title: Handle empty body" assert prompt =~ "No description provided." end test "prompt builder reports workflow load failures separately from template parse errors" do original_workflow_path = Workflow.workflow_file_path() workflow_store_pid = Process.whereis(SymphonyElixir.WorkflowStore) on_exit(fn -> Workflow.set_workflow_file_path(original_workflow_path) if is_pid(workflow_store_pid) and is_nil(Process.whereis(SymphonyElixir.WorkflowStore)) do Supervisor.restart_child(SymphonyElixir.Supervisor, SymphonyElixir.WorkflowStore) end end) assert :ok = Supervisor.terminate_child(SymphonyElixir.Supervisor, SymphonyElixir.WorkflowStore) Workflow.set_workflow_file_path(Path.join(System.tmp_dir!(), "missing-workflow-#{System.unique_integer([:positive])}.md")) issue = %Issue{ identifier: "MT-780", title: "Workflow unavailable", description: "Missing workflow file", state: "Todo", url: "https://example.org/issues/MT-780", labels: [] } assert_raise RuntimeError, ~r/workflow_unavailable:/, fn -> PromptBuilder.build_prompt(issue) end end test "in-repo WORKFLOW.md renders correctly" do workflow_path = Workflow.workflow_file_path() Workflow.set_workflow_file_path(Path.expand("WORKFLOW.md", File.cwd!())) issue = %Issue{ identifier: "MT-616", title: "Use rich templates for WORKFLOW.md", description: "Render with rich template variables", state: "In Progress", url: "https://example.org/issues/MT-616/use-rich-templates-for-workflowmd", labels: ["templating", "workflow"] } on_exit(fn -> Workflow.set_workflow_file_path(workflow_path) end) prompt = PromptBuilder.build_prompt(issue, attempt: 2) assert prompt =~ "You are working on a Linear ticket `MT-616`" assert prompt =~ "Issue context:" assert prompt =~ "Identifier: MT-616" assert prompt =~ "Title: Use rich templates for WORKFLOW.md" assert prompt =~ "Current status: In Progress" assert prompt =~ "https://example.org/issues/MT-616/use-rich-templates-for-workflowmd" assert prompt =~ "This is an unattended orchestration session." assert prompt =~ "Only stop early for a true blocker" assert prompt =~ "Do not include \"next steps for user\"" assert prompt =~ "open and follow `.codex/skills/land/SKILL.md`" assert prompt =~ "Do not call `gh pr merge` directly" assert prompt =~ "Continuation context:" assert prompt =~ "retry attempt #2" end test "prompt builder adds continuation guidance for retries" do workflow_prompt = "{% if attempt %}Retry #" <> "{{ attempt }}" <> "{% endif %}" write_workflow_file!(Workflow.workflow_file_path(), prompt: workflow_prompt) issue = %Issue{ identifier: "MT-201", title: "Continue autonomous ticket", description: "Retry flow", state: "In Progress", url: "https://example.org/issues/MT-201", labels: [] } prompt = PromptBuilder.build_prompt(issue, attempt: 2) assert prompt == "Retry #2" end test "agent runner keeps workspace after successful codex run" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-agent-runner-retain-workspace-#{System.unique_integer([:positive])}" ) try do template_repo = Path.join(test_root, "source") workspace_root = Path.join(test_root, "workspaces") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(template_repo) File.mkdir_p!(workspace_root) File.write!(Path.join(template_repo, "README.md"), "# test") System.cmd("git", ["-C", template_repo, "init", "-b", "main"]) System.cmd("git", ["-C", template_repo, "config", "user.name", "Test User"]) System.cmd("git", ["-C", template_repo, "config", "user.email", "test@example.com"]) System.cmd("git", ["-C", template_repo, "add", "README.md"]) System.cmd("git", ["-C", template_repo, "commit", "-m", "initial"]) File.write!(codex_binary, """ #!/bin/sh count=0 while IFS= read -r line; do count=$((count + 1)) case "$count" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) ;; 3) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-1\"}}}' ;; 4) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-1\"}}}' printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, hook_after_create: "cp #{Path.join(template_repo, "README.md")} README.md", codex_command: "#{codex_binary} app-server" ) issue = %Issue{ identifier: "S-99", title: "Smoke test", description: "Run and keep workspace", state: "In Progress", url: "https://example.org/issues/S-99", labels: ["backend"] } before = MapSet.new(File.ls!(workspace_root)) assert :ok = AgentRunner.run(issue) entries_after = MapSet.new(File.ls!(workspace_root)) created = MapSet.difference(entries_after, before) |> Enum.filter(&(&1 == "S-99")) created = MapSet.new(created) assert MapSet.size(created) == 1 workspace_name = created |> Enum.to_list() |> List.first() assert workspace_name == "S-99" workspace = Path.join(workspace_root, workspace_name) assert File.exists?(workspace) assert File.exists?(Path.join(workspace, "README.md")) after File.rm_rf(test_root) end end test "agent runner forwards timestamped codex updates to recipient" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-agent-runner-updates-#{System.unique_integer([:positive])}" ) try do template_repo = Path.join(test_root, "source") workspace_root = Path.join(test_root, "workspaces") codex_binary = Path.join(test_root, "fake-codex") File.mkdir_p!(template_repo) File.write!(Path.join(template_repo, "README.md"), "# test") System.cmd("git", ["-C", template_repo, "init", "-b", "main"]) System.cmd("git", ["-C", template_repo, "config", "user.name", "Test User"]) System.cmd("git", ["-C", template_repo, "config", "user.email", "test@example.com"]) System.cmd("git", ["-C", template_repo, "add", "README.md"]) System.cmd("git", ["-C", template_repo, "commit", "-m", "initial"]) File.write!( codex_binary, """ #!/bin/sh count=0 while IFS= read -r line; do count=$((count + 1)) case "$count" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-live\"}}}' ;; 3) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-live\"}}}' ;; 4) printf '%s\\n' '{\"method\":\"turn/completed\"}' ;; *) ;; esac done """ ) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, hook_after_create: "cp #{Path.join(template_repo, "README.md")} README.md", codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-live-updates", identifier: "MT-99", title: "Smoke test", description: "Capture codex updates", state: "In Progress", url: "https://example.org/issues/MT-99", labels: ["backend"] } test_pid = self() assert :ok = AgentRunner.run( issue, test_pid, issue_state_fetcher: fn [_issue_id] -> {:ok, [%{issue | state: "Done"}]} end ) assert_receive {:codex_worker_update, "issue-live-updates", %{ event: :session_started, timestamp: %DateTime{}, session_id: session_id }}, 500 assert session_id == "thread-live-turn-live" after File.rm_rf(test_root) end end test "agent runner surfaces ssh startup failures instead of silently hopping hosts" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-agent-runner-single-host-#{System.unique_integer([:positive])}" ) previous_path = System.get_env("PATH") previous_trace = System.get_env("SYMP_TEST_SSH_TRACE") on_exit(fn -> restore_env("PATH", previous_path) restore_env("SYMP_TEST_SSH_TRACE", previous_trace) end) try do trace_file = Path.join(test_root, "ssh.trace") fake_ssh = Path.join(test_root, "ssh") File.mkdir_p!(test_root) System.put_env("SYMP_TEST_SSH_TRACE", trace_file) System.put_env("PATH", test_root <> ":" <> (previous_path || "")) File.write!(fake_ssh, """ #!/bin/sh trace_file="${SYMP_TEST_SSH_TRACE:-/tmp/symphony-fake-ssh.trace}" printf 'ARGV:%s\\n' "$*" >> "$trace_file" case "$*" in *worker-a*"__SYMPHONY_WORKSPACE__"*) printf '%s\\n' 'worker-a prepare failed' >&2 exit 75 ;; *worker-b*"__SYMPHONY_WORKSPACE__"*) printf '%s\\t%s\\t%s\\n' '__SYMPHONY_WORKSPACE__' '1' '/remote/home/.symphony-remote-workspaces/MT-SSH-FAILOVER' exit 0 ;; *) exit 0 ;; esac """) File.chmod!(fake_ssh, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: "~/.symphony-remote-workspaces", worker_ssh_hosts: ["worker-a", "worker-b"] ) issue = %Issue{ id: "issue-ssh-failover", identifier: "MT-SSH-FAILOVER", title: "Do not fail over within a single worker run", description: "Surface the startup failure to the orchestrator", state: "In Progress" } assert_raise RuntimeError, ~r/workspace_prepare_failed/, fn -> AgentRunner.run(issue, nil, worker_host: "worker-a") end trace = File.read!(trace_file) assert trace =~ "worker-a bash -lc" refute trace =~ "worker-b bash -lc" after File.rm_rf(test_root) end end test "agent runner continues with a follow-up turn while the issue remains active" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-agent-runner-continuation-#{System.unique_integer([:positive])}" ) try do template_repo = Path.join(test_root, "source") workspace_root = Path.join(test_root, "workspaces") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex.trace") File.mkdir_p!(template_repo) File.write!(Path.join(template_repo, "README.md"), "# test") System.cmd("git", ["-C", template_repo, "init", "-b", "main"]) System.cmd("git", ["-C", template_repo, "config", "user.name", "Test User"]) System.cmd("git", ["-C", template_repo, "config", "user.email", "test@example.com"]) System.cmd("git", ["-C", template_repo, "add", "README.md"]) System.cmd("git", ["-C", template_repo, "commit", "-m", "initial"]) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex.trace}" run_id="$(date +%s%N)-$$" printf 'RUN:%s\\n' "$run_id" >> "$trace_file" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' "$line" >> "$trace_file" case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) ;; 3) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-cont"}}}' ;; 4) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-cont-1"}}}' printf '%s\\n' '{"method":"turn/completed"}' ;; 5) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-cont-2"}}}' printf '%s\\n' '{"method":"turn/completed"}' ;; esac done """) File.chmod!(codex_binary, 0o755) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) on_exit(fn -> System.delete_env("SYMP_TEST_CODEx_TRACE") end) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, hook_after_create: "cp #{Path.join(template_repo, "README.md")} README.md", codex_command: "#{codex_binary} app-server", max_turns: 3 ) parent = self() state_fetcher = fn [_issue_id] -> attempt = Process.get(:agent_turn_fetch_count, 0) + 1 Process.put(:agent_turn_fetch_count, attempt) send(parent, {:issue_state_fetch, attempt}) state = if attempt == 1 do "In Progress" else "Done" end {:ok, [ %Issue{ id: "issue-continue", identifier: "MT-247", title: "Continue until done", description: "Still active after first turn", state: state } ]} end issue = %Issue{ id: "issue-continue", identifier: "MT-247", title: "Continue until done", description: "Still active after first turn", state: "In Progress", url: "https://example.org/issues/MT-247", labels: [] } assert :ok = AgentRunner.run(issue, nil, issue_state_fetcher: state_fetcher) assert_receive {:issue_state_fetch, 1} assert_receive {:issue_state_fetch, 2} lines = File.read!(trace_file) |> String.split("\n", trim: true) assert length(Enum.filter(lines, &String.starts_with?(&1, "RUN:"))) == 1 assert length(Enum.filter(lines, &String.contains?(&1, "\"method\":\"thread/start\""))) == 1 turn_texts = lines |> Enum.filter(&String.starts_with?(&1, "JSON:")) |> Enum.map(&String.trim_leading(&1, "JSON:")) |> Enum.map(&Jason.decode!/1) |> Enum.filter(&(&1["method"] == "turn/start")) |> Enum.map(fn payload -> get_in(payload, ["params", "input"]) |> Enum.map_join("\n", &Map.get(&1, "text", "")) end) assert length(turn_texts) == 2 assert Enum.at(turn_texts, 0) =~ "You are an agent for this repository." refute Enum.at(turn_texts, 1) =~ "You are an agent for this repository." assert Enum.at(turn_texts, 1) =~ "Continuation guidance:" assert Enum.at(turn_texts, 1) =~ "continuation turn #2 of 3" after System.delete_env("SYMP_TEST_CODEx_TRACE") File.rm_rf(test_root) end end test "agent runner stops continuing once agent.max_turns is reached" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-agent-runner-max-turns-#{System.unique_integer([:positive])}" ) try do template_repo = Path.join(test_root, "source") workspace_root = Path.join(test_root, "workspaces") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex.trace") File.mkdir_p!(template_repo) File.write!(Path.join(template_repo, "README.md"), "# test") System.cmd("git", ["-C", template_repo, "init", "-b", "main"]) System.cmd("git", ["-C", template_repo, "config", "user.name", "Test User"]) System.cmd("git", ["-C", template_repo, "config", "user.email", "test@example.com"]) System.cmd("git", ["-C", template_repo, "add", "README.md"]) System.cmd("git", ["-C", template_repo, "commit", "-m", "initial"]) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODEx_TRACE:-/tmp/codex.trace}" printf 'RUN\\n' >> "$trace_file" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' "$line" >> "$trace_file" case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) ;; 3) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-max"}}}' ;; 4) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-max-1"}}}' printf '%s\\n' '{"method":"turn/completed"}' ;; 5) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-max-2"}}}' printf '%s\\n' '{"method":"turn/completed"}' ;; esac done """) File.chmod!(codex_binary, 0o755) System.put_env("SYMP_TEST_CODEx_TRACE", trace_file) on_exit(fn -> System.delete_env("SYMP_TEST_CODEx_TRACE") end) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, hook_after_create: "cp #{Path.join(template_repo, "README.md")} README.md", codex_command: "#{codex_binary} app-server", max_turns: 2 ) state_fetcher = fn [_issue_id] -> {:ok, [ %Issue{ id: "issue-max-turns", identifier: "MT-248", title: "Stop at max turns", description: "Still active", state: "In Progress" } ]} end issue = %Issue{ id: "issue-max-turns", identifier: "MT-248", title: "Stop at max turns", description: "Still active", state: "In Progress", url: "https://example.org/issues/MT-248", labels: [] } assert :ok = AgentRunner.run(issue, nil, issue_state_fetcher: state_fetcher) trace = File.read!(trace_file) assert length(String.split(trace, "RUN", trim: true)) == 1 assert length(Regex.scan(~r/"method":"turn\/start"/, trace)) == 2 after System.delete_env("SYMP_TEST_CODEx_TRACE") File.rm_rf(test_root) end end test "app server starts with workspace cwd and expected startup command" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-args-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-77") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-args.trace") previous_trace = System.get_env("SYMP_TEST_CODex_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODex_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODex_TRACE") end end) System.put_env("SYMP_TEST_CODex_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODex_TRACE:-/tmp/codex-args.trace}" count=0 printf 'ARGV:%s\\n' \"$*\" >> \"$trace_file\" printf 'CWD:%s\\n' \"$PWD\" >> \"$trace_file\" while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' \"$line\" >> \"$trace_file\" case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-77\"}}}' ;; 3) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-77\"}}}' ;; 4) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server" ) issue = %Issue{ id: "issue-args", identifier: "MT-77", title: "Validate codex args", description: "Check startup args and cwd", state: "In Progress", url: "https://example.org/issues/MT-77", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Fix workspace start args", issue) assert {:ok, canonical_workspace} = SymphonyElixir.PathSafety.canonicalize(workspace) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert argv_line = Enum.find(lines, fn line -> String.starts_with?(line, "ARGV:") end) assert String.contains?(argv_line, "app-server") refute Enum.any?(lines, &String.contains?(&1, "--yolo")) assert cwd_line = Enum.find(lines, fn line -> String.starts_with?(line, "CWD:") end) assert String.ends_with?(cwd_line, Path.basename(workspace)) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> expected_approval_policy = %{ "reject" => %{ "sandbox_approval" => true, "rules" => true, "mcp_elicitations" => true } } payload["method"] == "thread/start" && get_in(payload, ["params", "approvalPolicy"]) == expected_approval_policy && get_in(payload, ["params", "sandbox"]) == "workspace-write" && get_in(payload, ["params", "cwd"]) == canonical_workspace end) else false end end) expected_turn_sandbox_policy = %{ "type" => "workspaceWrite", "writableRoots" => [canonical_workspace], "readOnlyAccess" => %{"type" => "fullAccess"}, "networkAccess" => false, "excludeTmpdirEnvVar" => false, "excludeSlashTmp" => false } assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> expected_approval_policy = %{ "reject" => %{ "sandbox_approval" => true, "rules" => true, "mcp_elicitations" => true } } payload["method"] == "turn/start" && get_in(payload, ["params", "cwd"]) == canonical_workspace && get_in(payload, ["params", "approvalPolicy"]) == expected_approval_policy && get_in(payload, ["params", "sandboxPolicy"]) == expected_turn_sandbox_policy end) else false end end) after File.rm_rf(test_root) end end test "app server startup command supports codex args override from workflow config" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-custom-args-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-88") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-custom-args.trace") previous_trace = System.get_env("SYMP_TEST_CODex_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODex_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODex_TRACE") end end) System.put_env("SYMP_TEST_CODex_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODex_TRACE:-/tmp/codex-custom-args.trace}" count=0 printf 'ARGV:%s\\n' \"$*\" >> \"$trace_file\" while IFS= read -r line; do count=$((count + 1)) case \"$count\" in 1) printf '%s\\n' '{\"id\":1,\"result\":{}}' ;; 2) printf '%s\\n' '{\"id\":2,\"result\":{\"thread\":{\"id\":\"thread-88\"}}}' ;; 3) printf '%s\\n' '{\"id\":3,\"result\":{\"turn\":{\"id\":\"turn-88\"}}}' ;; 4) printf '%s\\n' '{\"method\":\"turn/completed\"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} --model gpt-5.3-codex app-server" ) issue = %Issue{ id: "issue-custom-args", identifier: "MT-88", title: "Validate custom codex args", description: "Check startup args override", state: "In Progress", url: "https://example.org/issues/MT-88", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Fix workspace start args", issue) trace = File.read!(trace_file) lines = String.split(trace, "\n", trim: true) assert argv_line = Enum.find(lines, fn line -> String.starts_with?(line, "ARGV:") end) assert String.contains?(argv_line, "--model gpt-5.3-codex app-server") refute String.contains?(argv_line, "--ask-for-approval never") refute String.contains?(argv_line, "--sandbox danger-full-access") after File.rm_rf(test_root) end end test "app server startup payload uses configurable approval and sandbox settings from workflow config" do test_root = Path.join( System.tmp_dir!(), "symphony-elixir-app-server-policy-overrides-#{System.unique_integer([:positive])}" ) try do workspace_root = Path.join(test_root, "workspaces") workspace = Path.join(workspace_root, "MT-99") codex_binary = Path.join(test_root, "fake-codex") trace_file = Path.join(test_root, "codex-policy-overrides.trace") previous_trace = System.get_env("SYMP_TEST_CODex_TRACE") on_exit(fn -> if is_binary(previous_trace) do System.put_env("SYMP_TEST_CODex_TRACE", previous_trace) else System.delete_env("SYMP_TEST_CODex_TRACE") end end) System.put_env("SYMP_TEST_CODex_TRACE", trace_file) File.mkdir_p!(workspace) File.write!(codex_binary, """ #!/bin/sh trace_file="${SYMP_TEST_CODex_TRACE:-/tmp/codex-policy-overrides.trace}" count=0 while IFS= read -r line; do count=$((count + 1)) printf 'JSON:%s\\n' "$line" >> "$trace_file" case "$count" in 1) printf '%s\\n' '{"id":1,"result":{}}' ;; 2) printf '%s\\n' '{"id":2,"result":{"thread":{"id":"thread-99"}}}' ;; 3) printf '%s\\n' '{"id":3,"result":{"turn":{"id":"turn-99"}}}' ;; 4) printf '%s\\n' '{"method":"turn/completed"}' exit 0 ;; *) exit 0 ;; esac done """) File.chmod!(codex_binary, 0o755) workspace_cache = Path.join(Path.expand(workspace), ".cache") File.mkdir_p!(workspace_cache) write_workflow_file!(Workflow.workflow_file_path(), workspace_root: workspace_root, codex_command: "#{codex_binary} app-server", codex_approval_policy: "on-request", codex_thread_sandbox: "workspace-write", codex_turn_sandbox_policy: %{ type: "workspaceWrite", writableRoots: [Path.expand(workspace), workspace_cache] } ) issue = %Issue{ id: "issue-policy-overrides", identifier: "MT-99", title: "Validate codex policy overrides", description: "Check startup policy payload overrides", state: "In Progress", url: "https://example.org/issues/MT-99", labels: ["backend"] } assert {:ok, _result} = AppServer.run(workspace, "Fix workspace start args", issue) lines = File.read!(trace_file) |> String.split("\n", trim: true) assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> payload["method"] == "thread/start" && get_in(payload, ["params", "approvalPolicy"]) == "on-request" && get_in(payload, ["params", "sandbox"]) == "workspace-write" end) else false end end) expected_turn_policy = %{ "type" => "workspaceWrite", "writableRoots" => [Path.expand(workspace), workspace_cache] } assert Enum.any?(lines, fn line -> if String.starts_with?(line, "JSON:") do line |> String.trim_leading("JSON:") |> Jason.decode!() |> then(fn payload -> payload["method"] == "turn/start" && get_in(payload, ["params", "approvalPolicy"]) == "on-request" && get_in(payload, ["params", "sandboxPolicy"]) == expected_turn_policy end) else false end end) after File.rm_rf(test_root) end end end ================================================ FILE: elixir/test/symphony_elixir/dynamic_tool_test.exs ================================================ defmodule SymphonyElixir.Codex.DynamicToolTest do use SymphonyElixir.TestSupport alias SymphonyElixir.Codex.DynamicTool test "tool_specs advertises the linear_graphql input contract" do assert [ %{ "description" => description, "inputSchema" => %{ "properties" => %{ "query" => _, "variables" => _ }, "required" => ["query"], "type" => "object" }, "name" => "linear_graphql" } ] = DynamicTool.tool_specs() assert description =~ "Linear" end test "unsupported tools return a failure payload with the supported tool list" do response = DynamicTool.execute("not_a_real_tool", %{}) assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "error" => %{ "message" => ~s(Unsupported dynamic tool: "not_a_real_tool".), "supportedTools" => ["linear_graphql"] } } assert response["contentItems"] == [ %{ "type" => "inputText", "text" => response["output"] } ] end test "linear_graphql returns successful GraphQL responses as tool text" do test_pid = self() response = DynamicTool.execute( "linear_graphql", %{ "query" => "query Viewer { viewer { id } }", "variables" => %{"includeTeams" => false} }, linear_client: fn query, variables, opts -> send(test_pid, {:linear_client_called, query, variables, opts}) {:ok, %{"data" => %{"viewer" => %{"id" => "usr_123"}}}} end ) assert_received {:linear_client_called, "query Viewer { viewer { id } }", %{"includeTeams" => false}, []} assert response["success"] == true assert Jason.decode!(response["output"]) == %{"data" => %{"viewer" => %{"id" => "usr_123"}}} assert response["contentItems"] == [%{"type" => "inputText", "text" => response["output"]}] end test "linear_graphql accepts a raw GraphQL query string" do test_pid = self() response = DynamicTool.execute( "linear_graphql", " query Viewer { viewer { id } } ", linear_client: fn query, variables, opts -> send(test_pid, {:linear_client_called, query, variables, opts}) {:ok, %{"data" => %{"viewer" => %{"id" => "usr_456"}}}} end ) assert_received {:linear_client_called, "query Viewer { viewer { id } }", %{}, []} assert response["success"] == true end test "linear_graphql ignores legacy operationName arguments" do test_pid = self() response = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }", "operationName" => "Viewer"}, linear_client: fn query, variables, opts -> send(test_pid, {:linear_client_called, query, variables, opts}) {:ok, %{"data" => %{"viewer" => %{"id" => "usr_789"}}}} end ) assert_received {:linear_client_called, "query Viewer { viewer { id } }", %{}, []} assert response["success"] == true end test "linear_graphql passes multi-operation documents through unchanged" do test_pid = self() query = """ query Viewer { viewer { id } } query Teams { teams { nodes { id } } } """ response = DynamicTool.execute( "linear_graphql", %{"query" => query}, linear_client: fn forwarded_query, variables, opts -> send(test_pid, {:linear_client_called, forwarded_query, variables, opts}) {:ok, %{"errors" => [%{"message" => "Must provide operation name if query contains multiple operations."}]}} end ) assert_received {:linear_client_called, forwarded_query, %{}, []} assert forwarded_query == String.trim(query) assert response["success"] == false end test "linear_graphql rejects blank raw query strings even when using the default client" do response = DynamicTool.execute("linear_graphql", " ") assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "error" => %{ "message" => "`linear_graphql` requires a non-empty `query` string." } } end test "linear_graphql marks GraphQL error responses as failures while preserving the body" do response = DynamicTool.execute( "linear_graphql", %{"query" => "mutation BadMutation { nope }"}, linear_client: fn _query, _variables, _opts -> {:ok, %{"errors" => [%{"message" => "Unknown field `nope`"}], "data" => nil}} end ) assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "data" => nil, "errors" => [%{"message" => "Unknown field `nope`"}] } end test "linear_graphql marks atom-key GraphQL error responses as failures" do response = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}, linear_client: fn _query, _variables, _opts -> {:ok, %{errors: [%{message: "boom"}], data: nil}} end ) assert response["success"] == false end test "linear_graphql validates required arguments before calling Linear" do response = DynamicTool.execute( "linear_graphql", %{"variables" => %{"commentId" => "comment-1"}}, linear_client: fn _query, _variables, _opts -> flunk("linear client should not be called when arguments are invalid") end ) assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "error" => %{ "message" => "`linear_graphql` requires a non-empty `query` string." } } blank_query = DynamicTool.execute( "linear_graphql", %{"query" => " "}, linear_client: fn _query, _variables, _opts -> flunk("linear client should not be called when the query is blank") end ) assert blank_query["success"] == false end test "linear_graphql rejects invalid argument types" do response = DynamicTool.execute( "linear_graphql", [:not, :valid], linear_client: fn _query, _variables, _opts -> flunk("linear client should not be called when arguments are invalid") end ) assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "error" => %{ "message" => "`linear_graphql` expects either a GraphQL query string or an object with `query` and optional `variables`." } } end test "linear_graphql rejects invalid variables" do response = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }", "variables" => ["bad"]}, linear_client: fn _query, _variables, _opts -> flunk("linear client should not be called when variables are invalid") end ) assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "error" => %{ "message" => "`linear_graphql.variables` must be a JSON object when provided." } } end test "linear_graphql formats transport and auth failures" do missing_token = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}, linear_client: fn _query, _variables, _opts -> {:error, :missing_linear_api_token} end ) assert missing_token["success"] == false assert Jason.decode!(missing_token["output"]) == %{ "error" => %{ "message" => "Symphony is missing Linear auth. Set `linear.api_key` in `WORKFLOW.md` or export `LINEAR_API_KEY`." } } status_error = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}, linear_client: fn _query, _variables, _opts -> {:error, {:linear_api_status, 503}} end ) assert Jason.decode!(status_error["output"]) == %{ "error" => %{ "message" => "Linear GraphQL request failed with HTTP 503.", "status" => 503 } } request_error = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}, linear_client: fn _query, _variables, _opts -> {:error, {:linear_api_request, :timeout}} end ) assert Jason.decode!(request_error["output"]) == %{ "error" => %{ "message" => "Linear GraphQL request failed before receiving a successful response.", "reason" => ":timeout" } } end test "linear_graphql formats unexpected failures from the client" do response = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}, linear_client: fn _query, _variables, _opts -> {:error, :boom} end ) assert response["success"] == false assert Jason.decode!(response["output"]) == %{ "error" => %{ "message" => "Linear GraphQL tool execution failed.", "reason" => ":boom" } } end test "linear_graphql falls back to inspect for non-JSON payloads" do response = DynamicTool.execute( "linear_graphql", %{"query" => "query Viewer { viewer { id } }"}, linear_client: fn _query, _variables, _opts -> {:ok, :ok} end ) assert response["success"] == true assert response["output"] == ":ok" end end ================================================ FILE: elixir/test/symphony_elixir/extensions_test.exs ================================================ defmodule SymphonyElixir.ExtensionsTest do use SymphonyElixir.TestSupport import Phoenix.ConnTest import Phoenix.LiveViewTest alias SymphonyElixir.Linear.Adapter alias SymphonyElixir.Tracker.Memory @endpoint SymphonyElixirWeb.Endpoint defmodule FakeLinearClient do def fetch_candidate_issues do send(self(), :fetch_candidate_issues_called) {:ok, [:candidate]} end def fetch_issues_by_states(states) do send(self(), {:fetch_issues_by_states_called, states}) {:ok, states} end def fetch_issue_states_by_ids(issue_ids) do send(self(), {:fetch_issue_states_by_ids_called, issue_ids}) {:ok, issue_ids} end def graphql(query, variables) do send(self(), {:graphql_called, query, variables}) case Process.get({__MODULE__, :graphql_results}) do [result | rest] -> Process.put({__MODULE__, :graphql_results}, rest) result _ -> Process.get({__MODULE__, :graphql_result}) end end end defmodule SlowOrchestrator do use GenServer def start_link(opts) do GenServer.start_link(__MODULE__, :ok, opts) end def init(:ok), do: {:ok, :ok} def handle_call(:snapshot, _from, state) do Process.sleep(25) {:reply, %{}, state} end def handle_call(:request_refresh, _from, state) do {:reply, :unavailable, state} end end defmodule StaticOrchestrator do use GenServer def start_link(opts) do name = Keyword.fetch!(opts, :name) GenServer.start_link(__MODULE__, opts, name: name) end def init(opts), do: {:ok, opts} def handle_call(:snapshot, _from, state) do {:reply, Keyword.fetch!(state, :snapshot), state} end def handle_call(:request_refresh, _from, state) do {:reply, Keyword.get(state, :refresh, :unavailable), state} end end setup do linear_client_module = Application.get_env(:symphony_elixir, :linear_client_module) on_exit(fn -> if is_nil(linear_client_module) do Application.delete_env(:symphony_elixir, :linear_client_module) else Application.put_env(:symphony_elixir, :linear_client_module, linear_client_module) end end) :ok end setup do endpoint_config = Application.get_env(:symphony_elixir, SymphonyElixirWeb.Endpoint, []) on_exit(fn -> Application.put_env(:symphony_elixir, SymphonyElixirWeb.Endpoint, endpoint_config) end) :ok end test "workflow store reloads changes, keeps last good workflow, and falls back when stopped" do ensure_workflow_store_running() assert {:ok, %{prompt: "You are an agent for this repository."}} = Workflow.current() write_workflow_file!(Workflow.workflow_file_path(), prompt: "Second prompt") send(WorkflowStore, :poll) assert_eventually(fn -> match?({:ok, %{prompt: "Second prompt"}}, Workflow.current()) end) File.write!(Workflow.workflow_file_path(), "---\ntracker: [\n---\nBroken prompt\n") assert {:error, _reason} = WorkflowStore.force_reload() assert {:ok, %{prompt: "Second prompt"}} = Workflow.current() third_workflow = Path.join(Path.dirname(Workflow.workflow_file_path()), "THIRD_WORKFLOW.md") write_workflow_file!(third_workflow, prompt: "Third prompt") Workflow.set_workflow_file_path(third_workflow) assert {:ok, %{prompt: "Third prompt"}} = Workflow.current() assert :ok = Supervisor.terminate_child(SymphonyElixir.Supervisor, WorkflowStore) assert {:ok, %{prompt: "Third prompt"}} = WorkflowStore.current() assert :ok = WorkflowStore.force_reload() assert {:ok, _pid} = Supervisor.restart_child(SymphonyElixir.Supervisor, WorkflowStore) end test "workflow store init stops on missing workflow file" do missing_path = Path.join(Path.dirname(Workflow.workflow_file_path()), "MISSING_WORKFLOW.md") Workflow.set_workflow_file_path(missing_path) assert {:stop, {:missing_workflow_file, ^missing_path, :enoent}} = WorkflowStore.init([]) end test "workflow store start_link and poll callback cover missing-file error paths" do ensure_workflow_store_running() existing_path = Workflow.workflow_file_path() manual_path = Path.join(Path.dirname(existing_path), "MANUAL_WORKFLOW.md") missing_path = Path.join(Path.dirname(existing_path), "MANUAL_MISSING_WORKFLOW.md") assert :ok = Supervisor.terminate_child(SymphonyElixir.Supervisor, WorkflowStore) Workflow.set_workflow_file_path(missing_path) assert {:error, {:missing_workflow_file, ^missing_path, :enoent}} = WorkflowStore.force_reload() write_workflow_file!(manual_path, prompt: "Manual workflow prompt") Workflow.set_workflow_file_path(manual_path) assert {:ok, manual_pid} = WorkflowStore.start_link() assert Process.alive?(manual_pid) state = :sys.get_state(manual_pid) File.write!(manual_path, "---\ntracker: [\n---\nBroken prompt\n") assert {:noreply, returned_state} = WorkflowStore.handle_info(:poll, state) assert returned_state.workflow.prompt == "Manual workflow prompt" refute returned_state.stamp == nil assert_receive :poll, 1_100 Workflow.set_workflow_file_path(missing_path) assert {:noreply, path_error_state} = WorkflowStore.handle_info(:poll, returned_state) assert path_error_state.workflow.prompt == "Manual workflow prompt" assert_receive :poll, 1_100 Workflow.set_workflow_file_path(manual_path) File.rm!(manual_path) assert {:noreply, removed_state} = WorkflowStore.handle_info(:poll, path_error_state) assert removed_state.workflow.prompt == "Manual workflow prompt" assert_receive :poll, 1_100 Process.exit(manual_pid, :normal) restart_result = Supervisor.restart_child(SymphonyElixir.Supervisor, WorkflowStore) assert match?({:ok, _pid}, restart_result) or match?({:error, {:already_started, _pid}}, restart_result) Workflow.set_workflow_file_path(existing_path) WorkflowStore.force_reload() end test "tracker delegates to memory and linear adapters" do issue = %Issue{id: "issue-1", identifier: "MT-1", state: "In Progress"} Application.put_env(:symphony_elixir, :memory_tracker_issues, [issue, %{id: "ignored"}]) Application.put_env(:symphony_elixir, :memory_tracker_recipient, self()) write_workflow_file!(Workflow.workflow_file_path(), tracker_kind: "memory") assert Config.settings!().tracker.kind == "memory" assert SymphonyElixir.Tracker.adapter() == Memory assert {:ok, [^issue]} = SymphonyElixir.Tracker.fetch_candidate_issues() assert {:ok, [^issue]} = SymphonyElixir.Tracker.fetch_issues_by_states([" in progress ", 42]) assert {:ok, [^issue]} = SymphonyElixir.Tracker.fetch_issue_states_by_ids(["issue-1"]) assert :ok = SymphonyElixir.Tracker.create_comment("issue-1", "comment") assert :ok = SymphonyElixir.Tracker.update_issue_state("issue-1", "Done") assert_receive {:memory_tracker_comment, "issue-1", "comment"} assert_receive {:memory_tracker_state_update, "issue-1", "Done"} Application.delete_env(:symphony_elixir, :memory_tracker_recipient) assert :ok = Memory.create_comment("issue-1", "quiet") assert :ok = Memory.update_issue_state("issue-1", "Quiet") write_workflow_file!(Workflow.workflow_file_path(), tracker_kind: "linear") assert SymphonyElixir.Tracker.adapter() == Adapter end test "linear adapter delegates reads and validates mutation responses" do Application.put_env(:symphony_elixir, :linear_client_module, FakeLinearClient) assert {:ok, [:candidate]} = Adapter.fetch_candidate_issues() assert_receive :fetch_candidate_issues_called assert {:ok, ["Todo"]} = Adapter.fetch_issues_by_states(["Todo"]) assert_receive {:fetch_issues_by_states_called, ["Todo"]} assert {:ok, ["issue-1"]} = Adapter.fetch_issue_states_by_ids(["issue-1"]) assert_receive {:fetch_issue_states_by_ids_called, ["issue-1"]} Process.put( {FakeLinearClient, :graphql_result}, {:ok, %{"data" => %{"commentCreate" => %{"success" => true}}}} ) assert :ok = Adapter.create_comment("issue-1", "hello") assert_receive {:graphql_called, create_comment_query, %{body: "hello", issueId: "issue-1"}} assert create_comment_query =~ "commentCreate" Process.put( {FakeLinearClient, :graphql_result}, {:ok, %{"data" => %{"commentCreate" => %{"success" => false}}}} ) assert {:error, :comment_create_failed} = Adapter.create_comment("issue-1", "broken") Process.put({FakeLinearClient, :graphql_result}, {:error, :boom}) assert {:error, :boom} = Adapter.create_comment("issue-1", "boom") Process.put({FakeLinearClient, :graphql_result}, {:ok, %{"data" => %{}}}) assert {:error, :comment_create_failed} = Adapter.create_comment("issue-1", "weird") Process.put({FakeLinearClient, :graphql_result}, :unexpected) assert {:error, :comment_create_failed} = Adapter.create_comment("issue-1", "odd") Process.put( {FakeLinearClient, :graphql_results}, [ {:ok, %{ "data" => %{ "issue" => %{"team" => %{"states" => %{"nodes" => [%{"id" => "state-1"}]}}} } }}, {:ok, %{"data" => %{"issueUpdate" => %{"success" => true}}}} ] ) assert :ok = Adapter.update_issue_state("issue-1", "Done") assert_receive {:graphql_called, state_lookup_query, %{issueId: "issue-1", stateName: "Done"}} assert state_lookup_query =~ "states" assert_receive {:graphql_called, update_issue_query, %{issueId: "issue-1", stateId: "state-1"}} assert update_issue_query =~ "issueUpdate" Process.put( {FakeLinearClient, :graphql_results}, [ {:ok, %{ "data" => %{ "issue" => %{"team" => %{"states" => %{"nodes" => [%{"id" => "state-1"}]}}} } }}, {:ok, %{"data" => %{"issueUpdate" => %{"success" => false}}}} ] ) assert {:error, :issue_update_failed} = Adapter.update_issue_state("issue-1", "Broken") Process.put({FakeLinearClient, :graphql_results}, [{:error, :boom}]) assert {:error, :boom} = Adapter.update_issue_state("issue-1", "Boom") Process.put({FakeLinearClient, :graphql_results}, [{:ok, %{"data" => %{}}}]) assert {:error, :state_not_found} = Adapter.update_issue_state("issue-1", "Missing") Process.put( {FakeLinearClient, :graphql_results}, [ {:ok, %{ "data" => %{ "issue" => %{"team" => %{"states" => %{"nodes" => [%{"id" => "state-1"}]}}} } }}, {:ok, %{"data" => %{}}} ] ) assert {:error, :issue_update_failed} = Adapter.update_issue_state("issue-1", "Weird") Process.put( {FakeLinearClient, :graphql_results}, [ {:ok, %{ "data" => %{ "issue" => %{"team" => %{"states" => %{"nodes" => [%{"id" => "state-1"}]}}} } }}, :unexpected ] ) assert {:error, :issue_update_failed} = Adapter.update_issue_state("issue-1", "Odd") end test "phoenix observability api preserves state, issue, and refresh responses" do snapshot = static_snapshot() orchestrator_name = Module.concat(__MODULE__, :ObservabilityApiOrchestrator) {:ok, _pid} = StaticOrchestrator.start_link( name: orchestrator_name, snapshot: snapshot, refresh: %{ queued: true, coalesced: false, requested_at: DateTime.utc_now(), operations: ["poll", "reconcile"] } ) start_test_endpoint(orchestrator: orchestrator_name, snapshot_timeout_ms: 50) conn = get(build_conn(), "/api/v1/state") state_payload = json_response(conn, 200) assert state_payload == %{ "generated_at" => state_payload["generated_at"], "counts" => %{"running" => 1, "retrying" => 1}, "running" => [ %{ "issue_id" => "issue-http", "issue_identifier" => "MT-HTTP", "state" => "In Progress", "worker_host" => nil, "workspace_path" => nil, "session_id" => "thread-http", "turn_count" => 7, "last_event" => "notification", "last_message" => "rendered", "started_at" => state_payload["running"] |> List.first() |> Map.fetch!("started_at"), "last_event_at" => nil, "tokens" => %{"input_tokens" => 4, "output_tokens" => 8, "total_tokens" => 12} } ], "retrying" => [ %{ "issue_id" => "issue-retry", "issue_identifier" => "MT-RETRY", "attempt" => 2, "due_at" => state_payload["retrying"] |> List.first() |> Map.fetch!("due_at"), "error" => "boom", "worker_host" => nil, "workspace_path" => nil } ], "codex_totals" => %{ "input_tokens" => 4, "output_tokens" => 8, "total_tokens" => 12, "seconds_running" => 42.5 }, "rate_limits" => %{"primary" => %{"remaining" => 11}} } conn = get(build_conn(), "/api/v1/MT-HTTP") issue_payload = json_response(conn, 200) assert issue_payload == %{ "issue_identifier" => "MT-HTTP", "issue_id" => "issue-http", "status" => "running", "workspace" => %{ "path" => Path.join(Config.settings!().workspace.root, "MT-HTTP"), "host" => nil }, "attempts" => %{"restart_count" => 0, "current_retry_attempt" => 0}, "running" => %{ "worker_host" => nil, "workspace_path" => nil, "session_id" => "thread-http", "turn_count" => 7, "state" => "In Progress", "started_at" => issue_payload["running"]["started_at"], "last_event" => "notification", "last_message" => "rendered", "last_event_at" => nil, "tokens" => %{"input_tokens" => 4, "output_tokens" => 8, "total_tokens" => 12} }, "retry" => nil, "logs" => %{"codex_session_logs" => []}, "recent_events" => [], "last_error" => nil, "tracked" => %{} } conn = get(build_conn(), "/api/v1/MT-RETRY") assert %{"status" => "retrying", "retry" => %{"attempt" => 2, "error" => "boom"}} = json_response(conn, 200) conn = get(build_conn(), "/api/v1/MT-MISSING") assert json_response(conn, 404) == %{ "error" => %{"code" => "issue_not_found", "message" => "Issue not found"} } conn = post(build_conn(), "/api/v1/refresh", %{}) assert %{"queued" => true, "coalesced" => false, "operations" => ["poll", "reconcile"]} = json_response(conn, 202) end test "phoenix observability api preserves 405, 404, and unavailable behavior" do unavailable_orchestrator = Module.concat(__MODULE__, :UnavailableOrchestrator) start_test_endpoint(orchestrator: unavailable_orchestrator, snapshot_timeout_ms: 5) assert json_response(post(build_conn(), "/api/v1/state", %{}), 405) == %{"error" => %{"code" => "method_not_allowed", "message" => "Method not allowed"}} assert json_response(get(build_conn(), "/api/v1/refresh"), 405) == %{"error" => %{"code" => "method_not_allowed", "message" => "Method not allowed"}} assert json_response(post(build_conn(), "/", %{}), 405) == %{"error" => %{"code" => "method_not_allowed", "message" => "Method not allowed"}} assert json_response(post(build_conn(), "/api/v1/MT-1", %{}), 405) == %{"error" => %{"code" => "method_not_allowed", "message" => "Method not allowed"}} assert json_response(get(build_conn(), "/unknown"), 404) == %{"error" => %{"code" => "not_found", "message" => "Route not found"}} state_payload = json_response(get(build_conn(), "/api/v1/state"), 200) assert state_payload == %{ "generated_at" => state_payload["generated_at"], "error" => %{"code" => "snapshot_unavailable", "message" => "Snapshot unavailable"} } assert json_response(post(build_conn(), "/api/v1/refresh", %{}), 503) == %{ "error" => %{ "code" => "orchestrator_unavailable", "message" => "Orchestrator is unavailable" } } end test "phoenix observability api preserves snapshot timeout behavior" do timeout_orchestrator = Module.concat(__MODULE__, :TimeoutOrchestrator) {:ok, _pid} = SlowOrchestrator.start_link(name: timeout_orchestrator) start_test_endpoint(orchestrator: timeout_orchestrator, snapshot_timeout_ms: 1) timeout_payload = json_response(get(build_conn(), "/api/v1/state"), 200) assert timeout_payload == %{ "generated_at" => timeout_payload["generated_at"], "error" => %{"code" => "snapshot_timeout", "message" => "Snapshot timed out"} } end test "dashboard bootstraps liveview from embedded static assets" do orchestrator_name = Module.concat(__MODULE__, :AssetOrchestrator) {:ok, _pid} = StaticOrchestrator.start_link( name: orchestrator_name, snapshot: static_snapshot(), refresh: %{ queued: true, coalesced: false, requested_at: DateTime.utc_now(), operations: ["poll"] } ) start_test_endpoint(orchestrator: orchestrator_name, snapshot_timeout_ms: 50) html = html_response(get(build_conn(), "/"), 200) assert html =~ "/dashboard.css" assert html =~ "/vendor/phoenix_html/phoenix_html.js" assert html =~ "/vendor/phoenix/phoenix.js" assert html =~ "/vendor/phoenix_live_view/phoenix_live_view.js" refute html =~ "/assets/app.js" refute html =~ "