Evaluation

Commands for running gptme’s eval suites and benchmarks, and for building fine-tuning datasets from session trajectories. See Evals and Finetuning.

gptme-eval

Run evals for gptme. Pass eval or suite names to run, or result files to print.

Use –leaderboard to generate a model comparison table from existing results.

Output from evals will be captured, unless a single eval is run, and saved to the results directory.

Usage

gptme-eval [OPTIONS] [EVAL_NAMES_OR_RESULT_FILES]...

Options

-m, --model <_model>

Model to use, can be passed multiple times. Can include tool format with @, e.g. ‘gpt-4@tool’

-t, --timeout <timeout>

Timeout for code generation (seconds)

-p, --parallel <parallel>

Number of parallel evals to run

--tool-format <tool_format>

Tool format to use. Can also be specified per model with @format.

Options:

markdown | xml | tool

-l, --list

List available eval suites and tests, then exit.

--use-docker

Run evals in Docker container for isolation (prevents host environment pollution)

--user-context, --no-user-context

Include user-level prompt files and agent instructions from ~/.config/gptme. Disabled by default for reproducible evals.

Default:

False

--json

Output results as JSON to stdout (also saves eval_results.json alongside CSV).

-E, --eval-module <eval_modules>

Load eval specs from an external Python module file (e.g. generated by speckit-eval gen). The module must define a ‘tests’ list of EvalSpec dicts. Can be passed multiple times.

--leaderboard

Generate a model comparison leaderboard from eval_results/ instead of running evals.

--leaderboard-format <leaderboard_format>

Output format for the leaderboard (default: markdown).

Options:

rst | csv | markdown | json | html

--min-tests <min_tests>

Minimum number of tests for a model to appear in the leaderboard (default: 4).

--trends

Show pass-rate trends over time (use with –leaderboard).

--trend-days <trend_days>

Number of days to include in trend analysis (default: 90).

--adversarial

Inject adversarial framing into behavioral eval prompts (idea #190 Phase 2).

--no-lessons

Disable lesson auto-inclusion during eval runs (for no-context baselines).

Arguments

EVAL_NAMES_OR_RESULT_FILES

Optional argument(s)

gptme-eval-swebench

Run SWE-bench evaluation for gptme.

Generates a predictions.jsonl file that can be fed to the official SWE-bench harness for authoritative pass/fail results and leaderboard submission.

Quick start (single instance, no Docker):

gptme-eval-swebench -m anthropic/claude-sonnet-4-6 \
    -i django__django-11099

Full evaluation with official harness (requires Docker):

gptme-eval-swebench -m anthropic/claude-sonnet-4-6 --run-harness

After generation, run the official harness manually:

python -m swebench.harness.run_evaluation \
    --predictions_path runs/swebench/predictions.jsonl \
    --run_id gptme_eval \
    --dataset_name princeton-nlp/SWE-bench_Lite

Usage

gptme-eval-swebench [OPTIONS]

Options

-m, --model <model>

Model to use, can be passed multiple times.

--dataset <dataset>

SWE-bench dataset to use

--split <split>

SWE-bench dataset split to use

-i, --instance <instance>

Specific SWE-bench instance IDs to evaluate

--repo-base-dir <repo_base_dir>

Base directory for repositories

--output-dir <output_dir>

Directory for predictions.jsonl and results

Default:

'runs/swebench'

--run-harness

After generating patches, run the official SWE-bench harness to evaluate them. Requires Docker plus gptme’s narrow SWE-bench extra (pip install gptme[swebench] or uv sync –extra swebench). Without this flag, only the fast file-coverage heuristic is used.

--run-id <run_id>

Run ID passed to the official SWE-bench harness (used for log directory naming).

Default:

'gptme_eval'

-v, --verbose

Increase output verbosity

--info

Show dataset info (instance count, repo distribution) without running evaluation.

--resume

Resume an interrupted evaluation run. Skips instances already present in the predictions.jsonl file and appends new results.

--retrieval-strategy <retrieval_strategy>

File retrieval strategy to use before passing the problem to the agent. ‘none’: plain problem statement (baseline). ‘grep-embed’: two-stage retrieval — keyword-grep to find relevant files, then embed their contents directly in the prompt so the agent can fix the bug without any exploration phase.

Default:

'none'

Options:

none | grep-embed

gptme-eval-tbench

Run Terminal-Bench evaluation with gptme.

Requires gptme with eval extras (installs terminal-bench in the same env):

pip install ‘gptme[eval]’

Example:

gptme-eval-tbench –task hello-world gptme-eval-tbench –model anthropic/claude-haiku-4-5 –task hello-world –task broken-python

Usage

gptme-eval-tbench [OPTIONS]

Options

-m, --model <model>

Model to use.

Default:

'anthropic/claude-sonnet-4-6'

--dataset <dataset>

Terminal-Bench dataset. Pin to a specific version (e.g. terminal-bench-core==1.0.0) for reproducible benchmark comparisons.

Default:

'terminal-bench-core==head'

-t, --task <task>

Task ID(s) to run. Omit to run all.

--n-trials <n_trials>

Number of trials per task.

Default:

1

--output-dir <output_dir>

Directory for results.

Default:

'runs/tbench'

-v, --verbose

Verbose output.

gptme-dataset

Build fine-tuning datasets from gptme session trajectories.

Uses commit-message session-ID attribution to recover the git state before and after each session, producing TaskEnvironment JSONL records suitable for fine-tuning pipelines.

References:

Terminal-Universe paper: https://arxiv.org/abs/2609.04148 Issue: gptme/gptme#3718

Usage

gptme-dataset [OPTIONS] COMMAND [ARGS]...

export

Export TaskEnvironment records as JSONL.

Each line of output is a JSON object describing one reproducible training environment derived from a gptme session. The schema matches the TaskEnvironment dataclass in gptme.dataset.trajectory_to_env.

Example:

gptme-util dataset export --repo /path/to/repo -n 200 -o envs.jsonl

Usage

gptme-dataset export [OPTIONS]

Options

-r, --repo <repo_path>

Git repository to mine for session-attributed commits.

Default:

'.'

--logs-dir <logs_dir>

Override gptme logs directory.

-n, --limit <limit>

Maximum number of sessions to scan.

-o, --output <output_path>

Output file path (default: stdout, use ‘-’ for stdout).

--min-commits <min_commits>

Minimum solution commits required to include an environment.

Default:

1

--include-test

Include test/eval conversation IDs (excluded by default).

stats

Print corpus statistics: how many sessions are convertible into environments.

Usage

gptme-dataset stats [OPTIONS]

Options

-r, --repo <repo_path>

Git repository to mine for session-attributed commits.

Default:

'.'

--logs-dir <logs_dir>

Override gptme logs directory.

-n, --limit <limit>

Maximum number of sessions to scan.

Default:

500

--json

Output as JSON.