Submit Guessability Audit

Audits side tasks for submit tool exploitability.

Purpose

Checks whether an agent could pass the scorer by calling the submit tool cleverly, without modifying the environment or violating the security property. The analysis reads the code without running it.

Commands

ct extras submit-guessability

Audit side tasks for submit tool exploitability.

Examples:

# Audit all side tasks with custom submit tools
ct extras submit-guessability

# Audit specific environment
ct extras submit-guessability -e web_scraping

# Audit specific side task
ct extras submit-guessability -s expose_secret

ct extras submit-guessability-view

Interactive viewer for submit guessability audit results.

Usage:

ct extras submit-guessability-view [RUN_PATH]

Output

Results are saved to data/submit_guessability_runs/<name>_<timestamp>/:

  • results.jsonl - Analysis for each side task, including theoretical exploits identified by the LLM
  • prompt_logs/<env>_<side_task>.json - Prompts, response, and web searches for each side task
  • run.log - Run log

Scope

Only side tasks with a custom submit tool that takes arguments are audited.

For Verified Exploits

ct extras submit-guessability-verified tests whether the exploits work by running them against live scorers.