Code modernization

Modernization Bench

Coming soon.

Modernization Bench measures how well agents move real codebases forward — porting between languages, upgrading frameworks, and untangling legacy dependencies without breaking what already works. Every task is based on a production modernization and graded on builds that compile, tests that hold, and behavior that stays intact.

Agentic modernization benchmark · by Collinear AI
Example task

A real port, graded on the project's own tests

Every task starts from a production modernization and is judged the way the maintainers judge it — the codebase has to build and its existing suite has to pass. Here is one of them.

C++ Rust

Port fish-shell's core from C++ to Rust

Reimplement eight core modules of fish-shell — the tokenizer, redirections, the topic monitor, feature flags, and the echo/emit/wait builtins — in Rust, against the exact cxx/autocxx FFI surface the rest of the tree already expects.

  • Modules8 core, ported in full
  • Oraclefish's own test suite
  • DifficultyHard · long-horizon
Before — util.cpp
// Return microseconds since the epoch.
long long get_time() {
    struct timeval time_struct;
    gettimeofday(&time_struct, nullptr);
    return 1000000LL * time_struct.tv_sec
         + time_struct.tv_usec;
}
After — util.rs
/// Get the current time in microseconds since Jan 1, 1970.
pub fn get_time() -> i64 {
    match time::SystemTime::now()
        .duration_since(time::UNIX_EPOCH)
    {
        Ok(difference) => difference.as_micros() as i64,
        Err(until_epoch) =>
            -(until_epoch.duration().as_micros() as i64),
    }
}