SGU-Editorial: A Small Dataset of Competitive Programming Problems with LLM-Enhanced Editorials

@misc{dimitrov2025sgu,
  title={SGU-Editorial: A Small Dataset of Competitive Programming Problems
         with LLM-Enhanced Editorials},
  author={Radoslav Dimitrov},
  year={2025},
  url={https://radoslav11.com/sgu-dataset/sgu-editorial.pdf}
}

Benchmark: sgu-bench

The same 452 problems double as an offline, verifiable benchmark for LLMs and coding agents. You write one function, statement → source code, and sgu-bench compiles it, runs it in a resource-limited sandbox over hidden tests, and returns a real verdict (AC / WA / TLE / MLE / RE / CE) per problem. Grading is deterministic and exact: every expected output comes from a reference solution accepted on the real judge, and the tests include maximal-size cases so time and memory limits actually bite.

pip install sgu-bench

from sgu_bench import Benchmark
bench = Benchmark()

def solver(statement: str) -> str:   # your model or agent
    return my_model_generate_cpp(statement)

report = bench.evaluate(solver, problems="all")
print(report)   # solved N/452 + verdict breakdown

The benchmark data (~310 MB) downloads automatically on first use. Repository, examples, and traces: github.com/radoslav11/sgu-bench.

Results so far

Accepted rate over all 452 problems, low reasoning effort, one sample per problem (pass@1).

Single-sample, low-effort numbers, meant as a reference point rather than a definitive ranking.

← Go back