Small language model benchmark

Micro-ARC Leaderboard

MARC-1 · MARC-2 · MARC-3 — a code-only, contamination-proof test of rule induction for ≤ 2B models. No dataset. Just a harness, a rule grammar, and your model.

Download harness

Run it on any Hugging Face model in one command.
python marc.py --hf HuggingFaceTB/SmolLM2-360M --chat

EM = exact grid match. fmt = output parses as a grid. Overall = mean across MARC-1/2/3. pass@n > EM means the model has latent rule inference it doesn't always land. Highlighted cells mark the best value in the current view.

Understanding the graphs: Small models often fall back on statistical guessing rather than solving the rule.

The Red dashed line is the mathematical random-guessing baseline.
The Yellow dashed line is 2× the random baseline. Scoring below this generally means the model is just guessing, or worse, copying the input.

Pattern

Periodic sequence continuation.

Object

Cross-grid spatial movement and transformation.

Counting

Read and tally marked cells.

Compositional

Chain two or more rules together.

#	Model	Org	pattern	object	counting	compositional	AVG

Cell value = EM for that family. Color scales from clay (low) to sage (high). pattern tests periodicity, object tests cross-grid comparison, counting tests read-and-tally, compositional tests two- to three-step rule chains.

Self-reported · paste results.json into the discussion to appear here

How to use Micro-ARC

One command, one file, one JSON. No dataset download, no inference server, no config YAML.

TL;DR: download marc.py, run it against any HF model or OpenAI-compatible endpoint, point --out at results.json, paste that file into the discussion.

1. Setup

pip install torch transformers

Add openai only if you'll test a remote endpoint. GPU strongly recommended.

2. Download the harness

Click Download harness in the header. It saves marc.py (~400 lines). That is the whole benchmark.

3. Run it

# base model (few-shot text completion)
python marc.py --hf HuggingFaceTB/SmolLM2-360M

# instruct model (chat template), write a results.json entry
python marc.py --hf Qwen/Qwen3-0.5B-Instruct --chat \
  --org Alibaba --params 0.5B --out results.json

# remote reference model through any OpenAI-compatible endpoint
python marc.py --openai Qwen/Qwen3-32B \
  --base-url http://localhost:8000/v1 --chat

4. Options

--chat — use the chat template. Set for instruct models, leave off for base models.
--families — subset of pattern object counting compositional.
--difficulties — subset of 1 2 3.
--n-per-cell — tasks per (family, difficulty) cell. Default 20, use 100+ for a stable estimate.
--n-samples — samples per task for pass@n. Set 5–8 with --temperature 0.7 to measure latent rule inference.
--seed — tasks are generated from a seed. Different seeds sample different tasks.
--org / --params / --out — metadata written into the results JSON.

5. What you get

Per-cell console lines: MARC-1 pattern EM=60.00% fmt=100.00% pass=60.00% and a per-tier summary. With --out, a results.json entry that's ready for the leaderboard:

{
  "models": [{
    "name": "Qwen/Qwen3-0.5B-Instruct",
    "org": "Alibaba",
    "params": "0.5B",
    "chat": true,
    "seed": "0",
    "nPerCell": 20,
    "nSamples": 1,
    "temperature": 0.0,
    "tiers": {
      "1": { "em": 0.43, "fmt": 0.89, "pass": 0.43 },
      "2": { "em": 0.22, "fmt": 0.76, "pass": 0.22 },
      "3": { "em": 0.07, "fmt": 0.61, "pass": 0.07 }
    },
    "families": {
      "pattern":       { "1": { "em": 0.60, "fmt": 1.0, "pass": 0.60 }, "2": { ... }, "3": { ... } },
      "object":        { ... },
      "counting":      { ... },
      "compositional": { ... }
    }
  }]
}

6. Submit

Open the discussion link, attach your results.json (or paste it in a code block). One entry per model per run. Include the command line you used so others can reproduce it.