Scientify

2024 MCM Problem C · independent case study

Beats the O-award paper: Scientify completed 2024 MCM Problem C

Scientify completed the 2024 MCM Problem C tennis momentum task. On a strictly held-out test set its accuracy is higher than both accuracies reported by the Outstanding Winner paper on the same problem.

Reference paper

Problem C O-award paper 2401445

Comparison

Scientify scores higher

Delivery

Paper, code, and the full workspace

Comparison with the O-award paper

Scientify scores higher than the Problem C O-award paper

Problem C O-award paper 2401445 reports two accuracies: 82.40% for player performance judgement and 77.60% for momentum swing prediction. On the strictly held-out test set Scientify measures 96.02% accuracy and 91.51% balanced accuracy over 201 prediction points.

Comparison result

Scientify scores higher

Scientify version nine · accuracy96.02%
Scientify version nine · balanced accuracy91.51%
O-award paper 2401445 · player performance82.40%
O-award paper 2401445 · swing prediction77.60%

All four accuracies share one axis; each value comes from the published report of that solution.

Scientify version nineHeld-out test set · 201 points

96.02%

91.51%

Baseline

O-award paper 2401445Player performance judgement

82.40%

Not reported

13.62 points below Scientify

O-award paper 2401445Momentum swing prediction

77.60%

Not reported

18.42 points below Scientify

On the strictly held-out test set Scientify reaches 96.02% accuracy over 201 prediction points, 13.62 and 18.42 points above the 82.40% and 77.60% reported by the Problem C O-award paper.

REF 01COMAP 2024 resultsTeam 2401445 · Problem C Outstanding WinnerREF 02Unveiling the Dynamics of TennisAbstract reports 82.4% and 77.6%REF 032024 MCM Problem COriginal problem statement and data specification

Autonomous execution

Scientify runs the long exploration on its own and iterates the algorithm to version nine

Starting from a baseline model, Scientify replaced the state definition, the serve prior, the direction gate, and the decision rule inside one workspace until it reached the version nine counterfactual state machine. Development macro F1 moved from 0.79287 in version six to 0.93722, and the worst single-match macro F1 from 0.74163 to 0.91971.

Subject
2024 MCM Problem C, tennis momentum
Input
Problem PDF, data dictionary, 7,284 points
Constraint
Only information through point t
Output
Report, code, predictions, and result records
  1. 01Complete

    Read and check

    Check the problem, data dictionary, and 7,284 points, then rebuild score and server state for every point.

  2. 02Complete

    Iterate hypotheses

    Move from the baseline model to a confirmed state machine and a serve-adjusted direction gate, comparing candidates every round; the version seven joint-state route scored 0.69958 development macro F1 and is kept in the workspace.

  3. 03Complete

    Counterfactual state machine

    Version nine drops the gate and the direction classifier: it simulates both outcomes of the next point and estimates each branch with the training-only player serve prior.

  4. 04Complete

    Deliver and test

    Record the development, tuning, and calibration metrics, then run the independent test on the held-out test set.

Complete records keep the result checkable

  • The evaluation runs in a separate script, and the workspace records metrics for development, tuning, and calibration.
  • The workspace keeps the replaced modeling route and the full record of checks.

Auditable workspace

Records of the task execution

Open a file to see the hypotheses, code, results, and checks from each round.

Preparing your workspace
01

mcm2024c_paper.pdf

Complete paper, including the LaTeX source

Generated
02

outer_result_v9.json

201 prediction points from the held-out test set

Generated
03

frozen_directional_counterfactual_v9.json

Settings and metrics from development and calibration

Recorded
04

directional_counterfactual_state.py

Counterfactual state machine of version nine

Runnable
05

audit.json

Point-level data and field checks

Generated
06

predictions.csv

Per-point predictions with three-class probabilities

Generated
07

report.md

Complete research report

Generated

Pricing

New users get a free cloud computer and $5 in model credit

Choose a plan by model credit. The cloud computer is free, and the problem, data, code, logs, and results stay in one workspace.

View complete pricing

Starter

$19/month

$60 model credit

Researcher

$29/month

$90 model credit

Professional

$99/month

$350 model credit

At equivalent model usage, Scientify costs about 30% of standard API pricing. Credits are deducted against standard API token prices.

Run a similar project

Give a four-day competition problem to an agent that keeps computing

The problem, data, code, logs, and evaluation results stay on an isolated cloud computer. You can return to the same workspace at any time.

Open the Scientify workspace