Contribute

Seed programs

We are looking for private programs that seed our compilation, obfuscation, and packing pipeline and become new reverse engineering targets. A program qualifies when it meets both conditions.

• Private, and older than 2024. The code has never been public, and its last commit predates January 1, 2024. Models have read most of public GitHub; a program they have never seen is what keeps the benchmark contamination-free.
• Builds to a native binary. The program compiles to a standalone executable, ideally in C, C++, Rust, or Go, so it can go through our compilation, obfuscation, and packing pipeline.

You do not need to write tasks or graders. We turn each accepted program into several independently graded tasks with deterministic, multi-score verification.

Programs between roughly 5,000 and 30,000 lines with real functionality work best. Single-file utilities are too small to hide anything, and anything already on the public web is disqualified however good it is. Send seed programs with a subject that starts with [SRE-Bench Program].

CTF challenges

If you write reverse engineering challenges in the CTF style, we also take finished binaries as they are. A challenge skips the compilation pipeline and appears on the Tasks page as its own cell, with its name, application type, and source language. Send us five things.

• The prompt. The task text exactly as the agent should see it, including what to submit and in which form.
• The binary, together with the name, application type, and source language we show on the Tasks page.
• The source code, so we can confirm the challenge is solvable: we give the source to GPT-5.6-sol and check that it reaches the answer. Without it, a binary could demand something nobody can do, such as factoring RSA-2048.
• A verifier or the expected answer. A flag, a key, or a script that checks a submission. Grading has to give the same answer on every run.
• A recommended tool set. Every challenge runs inside a Linux Docker container, so tell us which decompilers, debuggers, emulators, or runtimes the agent needs. This matters most for binaries that are not native to Linux; other platforms may be added later.

A challenge is considered when it meets all four conditions.

• Not solved by GPT-5.6-Sol Max. Run it yourself before submitting. A challenge the model already solves adds nothing to the benchmark.
• No public writeup. The solution has not been published anywhere: no blog post, no video, no solved thread on the platform that hosts the challenge. A model that has read the writeup is recalling, not reverse engineering.
• No anti-AI tricks. Nothing in the binary or the prompt may target the model instead of the analyst: no prompt injection, no instructions addressed to the agent, no content designed to derail it.
• A verifiable author. We run contributed binaries on our own infrastructure, so we need a valid affiliation and a track record: an institution, a company, or an established handle on a platform such as crackmes.one.

Send challenges with a subject that starts with [SRE-Bench CTF]. We run GPT-6 against every submission and use the outcome to decide whether it joins the benchmark.

Obfuscators and packers

The protections in v0 are in-house. We want to add tooling from the community: obfuscators, packers, virtualizers, anti-debugging layers, or anything else that makes a binary harder to analyse.

• Supports C, C++, Rust, or Go. At least one of them, ideally all four. Source-level, IR-level, and binary-level approaches are all welcome.
• Reproducible. Given the same input and configuration, the tool produces a binary we can rebuild and re-grade.
• Named presets. A documented configuration we can cite as a column on the Tasks page.

Send protections with a subject that starts with [SRE-Bench Obfuscation].

How submissions are reviewed

Every submission is evaluated internally before we decide whether to accept it. For seed programs, we build the binaries, write the grading tasks, and run the current models to confirm the target is neither trivial nor broken. For CTF challenges, we confirm from the source that the challenge is solvable, check the verifier, and run GPT-6. For protections, we verify that the protected binaries still pass their graders and measure how much harder the tasks become. We try to reply to every submission, and accepted contributions are credited on the Contributors page.