ALE-Bench

​

Usage

# List available problems
PYTHONPATH=. python -m benchmarks.ale.runner --list

# List lite benchmark problems
PYTHONPATH=. python -m benchmarks.ale.runner --lite

# Solve a problem
PYTHONPATH=. python -m benchmarks.ale.runner -p ahc039

# With options
PYTHONPATH=. python -m benchmarks.ale.runner \
    -p ahc039 \
    -i 14 \
    -m ALE_CONFIGS \
    -d aider

​

CLI Options

​

Available Problems

​

Output Structure

experiment_workspace/{uuid}/
├── main.cpp          # C++ solution
├── pre_run.cpp       # Optional precomputation (max 1 min)
└── sessions/         # Experiment branches

​

Evaluation

1. Code Submission: The main.cpp file is read from the experiment workspace
2. Docker Evaluation: Code is sent to ale_bench.public_eval() which compiles and runs in an isolated Docker container
3. Test Execution: Solution runs against all test cases with strict time limits
4. Validation: Each test case must return ACCEPTED with a non-zero score
5. Score Stabilization: If all tests pass, the solution runs 4 additional times and scores are averaged for stability
6. Final Ranking: Private evaluation compares against original contest participants

​

Code Requirements

• Be time-aware (limit: time_limit - 100ms for I/O)
• Handle all input constraints
• Use efficient algorithms and data structures
• Include compiler optimization pragmas if helpful

​

Built-in Domain Knowledge