iMINDBench

Before you start

• Your complete population_*.json results, in one directory outside the repository.
• A model ID: lowercase letters, digits, and underscores, such as example_model. Use it everywhere below.
• A clone of the repository. Every path and command below is relative to its leaderboard/ directory.

1. Describe your model

Create data/submissions/example_model.json:

{
  "schema_version": 1,
  "model_id": "example_model",
  "display_name": "Example Model",
  "pretrained": true,
  "pretrained_on": "Public Dataset",
  "description": "Short public description of the evaluated model.",
  "coverage_note": null
}

Use null for pretrained_on if the model is not pretrained. Leave coverage_note as null unless the coverage section below applies.

2. Generate the artifact

python add_model.py \
  --model-outputs /path/to/evaluation_outputs/example_model \
  --submission data/submissions/example_model.json \
  --dry-run

This writes nothing. Read the summary it prints, in particular the coverage and cohort lines. If it looks right, run the same command again without --dry-run. That creates data/models/example_model.json and updates data/manifest.json and data_bundle.js.

3. Check, then open the pull request

python validate_data.py

Commit these four files and nothing else:

• data/submissions/example_model.json
• data/models/example_model.json
• data/manifest.json
• data_bundle.js

No raw outputs, no caches, no edits to another model. Open the pull request with the link at the top of this page; the template asks for your submission file, the coverage lines from step 2, and where the model comes from. The same validator runs on your pull request.

Coverage

Evaluate on the whole benchmark if you can. If you cannot, cover exactly the Main cohort or exactly the Challenge cohort, and use the same choice throughout. Any other subset is rejected; step 2 tells you which one you have.

A cohort submission is listed with an automatic footnote about the subject-sessions it skipped. Name your cohort in coverage_note and it is added to that footnote.

Updating a listed model, viewing the site locally, and the full schema: leaderboard/README.md.