ScientifyMDArena
Full MD workflows · 50 tasks
INCARBench Generation
Write VASP INCARs · 192 tasks
INCARBench Repair
Fix VASP INCARs · 576 tasks
CompMat-Bench
Materials research steps · 94 tasks
The benchmarks
They test what computational chemists do every day
None of the three is multiple choice. Every task supplies a real system, input files and a research goal, and expects output that a simulation code or a grader can check directly.
End-to-end molecular dynamics research
MDArena
Proposed by：University of Oxford
93.24%
Average correctness
Its 50 tasks come from real biomolecular simulation projects and cover every step from a raw structure to a physical conclusion: cleaning crystal structures, parameterizing ligands and non-standard residues, building GPCR membrane systems, setting up relative and absolute binding free energies, configuring expanded ensembles and alchemical metadynamics, and measuring hydrogen bonds, conformational principal components and ion conductance from trajectories. Grading checks output files and numerical conclusions item by item.
- System building and membrane proteins
- Force field parameterization (GAFF, OpenFF, residue patches)
- Free energy (OpenFE RBFE/ABFE, GROMACS ABFE)
- Enhanced sampling (PLUMED, expanded ensemble)
- Trajectory analysis and QM cluster models
- Tasks
- 50
- Hard
- 16
- Perfect
- 43
Parameter judgment for first-principles calculations
INCARBench
Proposed by：Nankai University
95.04% / 99.32%
Generation / Repair
The INCAR file decides what a VASP calculation computes and whether it computes it correctly. Generation tasks give a material, a POSCAR and a goal and ask for an INCAR from scratch. Repair tasks give a flawed draft: task errors (a static run written as a relaxation, a band structure written as SCF) and materials-physics errors (wrong smearing for a semiconductor, spin or DFT+U dropped for a magnetic system, spin-orbit coupling dropped for heavy elements), mixed with clean drafts that must not be broken.
- Static SCF, relaxation, DOS and band structure
- Spin and DFT+U for magnetic and correlated systems
- Smearing choice for metals and semiconductors
- Recognizing settings that are already right
- Tasks
- 768
- Materials
- 48
- Families
- 16
Single steps of frontier materials research
CompMat-Bench
Proposed by：Rice University
97.87%
Pass rate
Its 94 tasks are drawn from recently published computational materials studies: Heusler alloys, WS₂/WSe₂ moiré heterostructures, the MnTe altermagnet, HfZrO ferroelectrics, BaTiO₃ dislocations, phonon transport in BAs and antiferromagnetic order in CrN. Tasks ask for inputs to VASP, Quantum ESPRESSO, ABACUS, LAMMPS and VAMPIRE, or for physical quantities such as band gaps, exchange constants, thermal conductivity and Néel temperatures extracted from computed data, all checked by the official grader.
- DFT, DFT+U and spin-orbit inputs
- Phonons, thermal transport and dielectric response
- Magnetic exchange and Monte Carlo
- Molecular dynamics and saddle points
- Tasks
- 94
- Passed
- 92
- Grader checks
- 2349 / 2362
Leaderboards
First place on all four evaluations
Comparison scores come from each benchmark's paper and public leaderboard. Scientify leads INCARBench Repair by more than 25 points and MDArena and INCARBench Generation by about 10 points.
MDArena v0.2.0
Average correctness
Lead over #2
+9.44 pt
- Scientify93.24
- GPT-5.5 XHigh83.80
- GPT-5.5 Medium83.60
- Gemini 3.5 Flash68.40
- Gemini 3.1 Pro64.80
- DeepSeek-V3.259.90
- Qwen3-235B21.40
Comparison: paper Table 1
INCARBench v1 · Generation
Average must-match
Lead over #2
+10.86 pt
- Scientify95.04
- GPT-5.4 XHigh84.18
- GPT-5.482.39
- GPT-5.4 No reasoning81.49
- Qwen3.6+ thinking74.82
- Claude Opus 4.674.02
- Gemini 3.1 Pro73.08
- DeepSeek Reasoner70.86
- Claude Sonnet 4.667.74
- DeepSeek Chat65.25
- Llama 4 Maverick63.51
- Claude Haiku 4.561.10
- Qwen3.6+ no-thinking59.85
Comparison: public leaderboard
INCARBench v1 · Repair
Repair total
Lead over #2
+25.55 pt
- Scientify99.32
- Claude Haiku 4.573.77
- Gemini 3.1 Pro72.45
- Claude Sonnet 4.671.31
- DeepSeek Reasoner71.16
- GPT-5.470.87
- Qwen3.6+ thinking70.74
- Qwen3.6+ no-thinking69.89
- GPT-5.4 No reasoning69.67
- Claude Opus 4.669.60
- GPT-5.4 XHigh67.68
- DeepSeek Chat66.59
- Llama 4 Maverick65.95
Comparison: public leaderboard
CompMat-Bench v1.0.0 · Single-step
Pass rate over 94 tasks
Lead over #2
+6.38 pt
- Scientify97.87
- Codex GPT-5.691.49
- GPT-5.6-sol90.43
- Gemini 3.8 Flash89.36
- DeepSeek V4.1 Flash65.96
Comparison: paper
Sample tasks
One task from each benchmark, with the prompt and answer to download
Each task below comes from a different benchmark. Download the prompt, compare it with Scientify's submitted answer and check every step yourself.
MDArena · Hard
Na⁺ conductance of the 6NQ0 channel from a 168 ns trajectory
Given an MD trajectory of the 6NQ0 membrane channel, reproduce the original analysis: find complete Na⁺ permeation events from the cytosolic gate through the selectivity filter and compute current and conductance.
Scientify narrowed 180 Na⁺ ions to 167 pore candidates, tracked their state frame by frame, found 5 complete permeation events with entry and exit times, and computed a conductance of 6.36 pS at −750 mV. Two independent implementations agreed exactly on ion indices, event times and conductance.
- Trajectory
- 1681 frames · 168 ns
- Events
- 5
- Conductance
- 6.36 pS
- Score
- Full marks
INCARBench · Repair · L3
Restoring spin polarization and DFT+U for NiO
NiO is a textbook correlated antiferromagnetic oxide. The static SCF draft set ISPIN = 1 and had no DFT+U at all, which would produce a spurious non-magnetic metal.
Scientify restored spin polarization, set an initial magnetic moment on Ni, applied a Hubbard U to Ni d states and added LMAXMIX and LASPH, while keeping the draft's correct tetrahedron integration, cutoff and static-run settings.
- Difficulty
- L3
- Family
- Correlated oxide
- Key fixes
- ISPIN · MAGMOM · LDAU
- Score
- Full marks
CompMat-Bench · Single-step
The Néel temperature of CrN from a Monte Carlo sweep
CrN is a double-layered antiferromagnet. The task supplies a VAMPIRE deck and Monte Carlo observables over a temperature grid with three seeds, without the software itself, and asks for the normalized staggered magnetization curve and the Néel temperature.
Scientify mapped magnetic sublattices to output columns from the deck and the manual, built the antiferromagnetic order parameter, averaged the three seeds into a full temperature curve and extracted a Néel temperature of 282.5 K, passing every check of the official grader.
- Software
- VAMPIRE
- Néel temperature
- 282.5 K
- Grader checks
- 2 / 2
- Score
- Full marks
Per-task scores
Scores for all 912 tasks, one row each
The table lists benchmark, version, split, task ID, difficulty and score for every task, so each row can be matched to the original benchmark problem.
Bring your computational chemistry project to Scientify
The cloud computer comes with GROMACS, LAMMPS, OpenMM and AmberTools installed, and the agent rents a GPU when the job needs one. Describe the research goal and Scientify prepares inputs, runs the calculations and analyzes the results.