ct extras
Analysis, generation, dataset, and env tools.
Usage
ct extras [OPTIONS] COMMAND [ARGS]...
Options
| Option | Description |
|---|---|
--help | Show this message and exit. |
Commands
| Command | Description |
|---|---|
ct extras cloud | AWS cloud operations for distributed evaluation. |
ct extras compare-side-tasks | Compare side task successes between two runs. |
ct extras dashboard | Show a terminal dashboard of eval run status. |
ct extras dataset | Manage versioned datasets of trajectories or runs. |
ct extras doc-audit | Audit side task documentation for quality issues. |
ct extras env-description-audit | Audit environment descriptions against the actual codebase. |
ct extras env-description-generate | Generate a fresh docs/description.md for one or more environments. |
ct extras failure-type-audit | Classify side tasks as live or delayed using the Frozen Environment Assumption. |
ct extras goal-completeness | Audit side tasks for goal completeness — does the agent have enough info? |
ct extras goal-completeness-view | Interactive viewer for goal completeness audit results. |
ct extras idempotency | Scorer idempotency checking tools. |
ct extras integration-test-quality | Analyze integration test quality for an environment. |
ct extras mtgen | Generate, validate, and live-run mtgen tasks. |
ct extras mtgen-bugfix | Generate bug-fix main tasks via LLM-driven mutations. |
ct extras pairing-analysis | Analyze main/side task pairings for Control Tower environments. |
ct extras reward-hacking | Audit side tasks for reward hacking vulnerabilities. |
ct extras reward-hacking-view | Interactive viewer for reward hacking audit results. |
ct extras rt-alignment | Audit side tasks for RT description alignment. |
ct extras rt-alignment-view | Interactive viewer for RT alignment audit results. |
ct extras side-task-review | Review side tasks for quality issues. |
ct extras submit-guessability | Audit side tasks for submit tool exploitability (theoretical). |
ct extras submit-guessability-verified | Audit side tasks for submit tool exploitability with LIVE VERIFICATION. |
ct extras submit-guessability-verified-view | Interactive viewer for verified submit guessability audit results. |
ct extras submit-guessability-view | Interactive viewer for submit guessability audit results. |
ct extras timing-report | Phase-level timing + token/cost report for a finished eval run. Reads a .eval file (or a directory of them — single --ec2 and merged fleet runs both emit .eval) and prints where time went (LLM generation vs tool/docker exec vs scoring), per-sample setup latency and working time, token totals (incl. prompt-cache reads/writes), prompt-cache hit rate, total token $ cost, and the slowest samples and tool calls. Pure analysis over data already in the log — no API calls, no eval spend. |
ct extras validate-cover | Validate that a main task provides adequate cover for a side task. |