HiPhO: How Far Are (M)LLMs from Humans in the Latest High School Physics Olympiad Benchmark?

Abstract

Recently, the physics reasoning capabilities of (M)LLMs have attracted growing attention. However, existing physics benchmarks lack systematic coverage of recent physics Olympiads and direct comparison with human contestants. We present HiPhO, the first benchmark dedicated to high school physics Olympiads with human-aligned evaluation. HiPhO highlights three key innovations. (1) Comprehensive data: it compiles 13 latest Olympiads from 2024--2025, covering international and regional competitions and spanning mixed modalities from text-only to diagram-based problems. (2) Professional evaluation: it adopts official rubrics for fine-grained answer- and step-level grading aligned with human examiners. (3) Human-level comparison: it assigns gold, silver, and bronze medals to models based on official medal scores, enabling direct comparison with human contestants. Evaluating 30 (M)LLMs across 13 exams, we find that most open-source MLLMs remain at or below the bronze level, open-source LLMs demonstrate notable progress with multiple gold medals, and closed-source MLLMs achieve 6-13 gold medals, while most models still fall well short of full marks. These results underscore the substantial gap between current (M)LLMs and top human contestants, as well as the room for further improvement. The dataset and leaderboard are available at https://huggingface.co/datasets/SciYu/HiPhO and https://phyarena.github.io, respectively.