Skip to main content

Accuracy and fidelity

model.yaml declares two kinds of quality evidence for a build profile's whole model:

SectionQuestion it answersMeasured byStage
numerical_fidelityDoes the compiled model produce the same numbers as the source model?scripts/compile.pycompile
accuracyHow well does the compiled model do its task?scripts/accuracy.py on a DevKitaccuracy

Fidelity is a numerical diagnostic, not task accuracy. Neither is calibration: calibration data feeds quantization and proves nothing about quality.

Complete Datasets for local model development first. Fidelity and accuracy datasets resolve through Dataset Registry and need working Artifactory access.

Compile-time fidelity​

numerical_fidelity asks the recipe to compare every build profile's compiled model with the source model on a fixed sample of a registered dataset:

numerical_fidelity:
status: diagnostic
dataset: {id: imagenet-2012, variant: cal, sha256: e6c18b28…, size_bytes: 112710756}
selection: {method: seeded_random_without_replacement, seed: 42, offset: 0, samples: 50}
outputs: [logits]

dataset pins a Dataset Registry archive, outputs names outputs of the model, and status is diagnostic, provisional, or ready; see numerical_fidelity. The section is a compile input: changing it recompiles the model's builds.

scripts/run_model_compilation.sh resolves the dataset with model-registry fetch-fidelity-dataset and passes it to the recipe as --fidelity-dataset. The recipe selects the declared samples, runs the reference and the candidate, and writes fidelity.json in the profile's output directory, beside the component directories.

The report follows numerical-fidelity.schema.json. For each declared output it records cosine similarity, normalized RMSE, and mean and maximum absolute error. The profile's default package carries the report, and CI publishes it with the build. A build must write the report when model.yaml declares numerical_fidelity, and must not write one when it does not. CI rejects the package otherwise.

Accuracy on the DevKit​

Use DevKit accuracy when a task metric needs the model's own decoding and scoring, such as top-1 accuracy for a classifier or COCO mAP for a detector. The model's scripts/accuracy.py runs every compiled MPK of a build profile on a DevKit with PyNeat, on each dataset accuracy names:

accuracy:
datasets:
- {id: imagenet-2012, variant: val, sha256: 018bf95e…, size_bytes: 6666407650}

Each datasets entry pins a Dataset Registry archive, which goes by its source ID, here imagenet-2012-val. The metrics the report must hold come from datasets/<registry id>.yaml for the model's tasks; see Required accuracy metrics. Without build_profiles, every build profile is measured; see accuracy. Record the source model's published scores in the top-level reference_metrics, not in accuracy, so the model page shows them before the first DevKit run.

Optional scripts/requirements-accuracy.txt pins extra wheels. They install into a separate folder beside PyNeat's environment, never into it.

The accuracy script​

scripts/accuracy.py receives:

  • --mpk <component>=<path> once for each component, model=<path> for a single-component model;
  • --dataset <source id>=<directory> for each dataset accuracy pins, the extracted archive;
  • --resources <directory> when the compile declares supporting runtime_files, the verified profile root with the same relative asset paths as the package; and
  • --output <directory>.

It scores the whole model on each dataset and hands the scores to tools/model_report, keyed by the dataset source IDs. model_report writes model-report.json into the output directory:

dataset_id, dataset_dir = args.dataset.split("=", 1) # imagenet-2012-val, <directory>
...
report = model_report.from_accuracy({
dataset_id: {
"metrics": {"top1_accuracy": top1 / len(names), "top5_accuracy": top5 / len(names)},
"split": VALIDATION_SPLIT,
"evaluated_samples": len(names),
"available_samples": len(available),
"details": {"inference_seconds": round(time.monotonic() - started, 1)},
},
})
report.save(args.output)

How the script evaluates each dataset is up to the model. Every score is named as an accuracy metric in tools/model_report/metrics.json and given in that metric's unit: a ratio from 0 to 1 for most metrics, a nonnegative ratio without an upper bound for abs_rel, a percentage from 0 to 100 for d1_all, and pixels for epe. Each dataset's result must hold every metric its requirements file lists for the model's tasks. Other known accuracy metrics are allowed. Existing scripts also accept --limit <images> to evaluate a subset for a quick local check; CI never passes it.

Run it on a DevKit​

scripts/run_model_accuracy.sh runs the stage the way CI does, against the output of scripts/run_model_compilation.sh:

REGISTRY_PYTHON=<python with model-registry and dataset-registry> \
PYNEAT_PYTHON=~/pyneat/bin/python \
scripts/run_model_accuracy.sh models/<model-id> <build-profile> <compile-output-root> <result-directory>

<compile-output-root> holds <model-id>/<build-profile>/compile-outputs.json and the MPKs it lists. The script does this:

  1. It resolves each declared dataset into DATASET_CACHE (default /media/nvme/datasets/artifacts). A lock makes concurrent runs share one download.
  2. It lists each component's MPK with model-registry model-component-mpks.
  3. It finds the build's supporting files with model-registry model-runtime-directory. When the compile declared runtime_files, it passes the verified profile root to accuracy.py as --resources.
  4. It extracts each dataset under WORK_DIR and removes that directory when it exits. The default is a folder for this run under /media/nvme/neat-tests/, which a local run names model-accuracy-local-1.
  5. It installs requirements-accuracy.txt beside PyNeat.
  6. It runs accuracy.py with --dataset <source id>=<directory> for each dataset and PYTHONPATH including tools. The script writes model-report.json, and the runner saves its output to accuracy.log.
  7. It runs model-registry check-report, which validates the report against model-report.schema.json, requires a result for exactly the declared datasets, each with its required metrics, and writes the board's installation to environment.json.

REGISTRY_PYTHON needs the full model-registry tooling from this checkout (python -m pip install .) and Dataset Registry, because the dataset resolution and report checks are not in the author-only model-registry package. It exits without running anything when the build profile is not measured.

For a quicker check, resolve the dataset with Dataset Registry, extract it, and run the script directly:

PYTHONPATH=tools ~/pyneat/bin/python models/resnet_50/scripts/accuracy.py \
--mpk model=<path to MPK> --dataset imagenet-2012-val=<extracted dataset> \
--output /tmp/resnet50-accuracy --limit 500

A model whose accuracy.py imports another file under scripts/, such as ResNet-50's calibration.py, needs that file beside it.

FP32 baseline​

When no published score matches the package's setup, such as SuperGlue's 512 keypoints from a 640x480 image, the model page can show an FP32 baseline instead: what the unquantized source model scores on the same inputs and protocol. A standalone scripts/fp32_baseline_accuracy.py measures it by hand on a host. It runs an FP32 ONNX with ONNX Runtime and scores it with the model's own accuracy.py, so only the model execution differs from the DevKit stage. Its docstring names the ONNX it reads: the output of the model's own convert.py, such as DETR's, or that output after surgery.py, such as SuperGlue's:

PYTHONPATH=tools python models/superglue_outdoor/scripts/fp32_baseline_accuracy.py \
--onnx <model.onnx from surgery.py> --dataset yfcc100m-keypoints-val=<extracted dataset> \
--output fp32-baseline.json

Copy its metrics into a reference_metrics entry with origin: fp32_baseline (reference metrics), and give the command and its output in the pull request. No stage runs the script and no fingerprint includes it, so adding or editing it reruns nothing, and the ONNX you convert locally need not match the sha256 of the file Vulcan Model Convert published.

Report and publication​

model-report.json follows model-report.schema.json: the stage, the PyNeat version and build, and datasets, which gives for each dataset source ID its evaluated split and sample counts, one number per metric, and free-form details. The model, build profile, MPK digests, and board are not report fields. CI records them when it publishes the report.

On the DevKit, scripts/run_model_tests.sh runs the model's tests/benchmark/test_accuracy.py on the report, which compares each dataset's metrics with the build profile's expected values for that dataset in tests/tests.yaml (set up tests/tests.yaml). CI then publishes the report with the test outcome: passed, failed, or skipped. A failed test still publishes the report, then fails the workflow run. See CI and registry publication for the shared flow.

The model browser shows each dataset's result in the accuracy section. The first four metrics appear as tiles and the rest in a table. Each metric shows its reference score when reference_metrics records one for the same dataset. The section also shows the evaluated sample count. The catalog and the Compare page use the first declared dataset.