Skip to main content

Concepts

This page defines the terms the other pages use, starting with the ones you meet first when you reproduce or add a model. Look here when a page, a script, or a CI message uses a name you do not know. Each example comes from models/resnet_50.

Two tools appear in every definition. The Model Compiler turns an ONNX model into the file a board runs. A Modalix DevKit is the board that runs the compiled model.

Model​

A model is one folder under models/, and the folder name is the model ID. Its model.yaml declares the model, and its scripts/ folder holds the code that converts, compiles, and measures it.

Example: models/resnet_50 is the model with the ID resnet_50. The Model Browser shows it under its display_name, ResNet-50.

Build profile​

A build profile is one named way to compile a model. model.yaml lists the build profiles under build_profiles, and each sets the parameters its components compile with, such as the precision.

Example: resnet_50 has three build profiles. w8a8 stores weights and activations as 8-bit integers (INT8). w8a16 keeps INT8 weights and stores activations in BF16, a 16-bit float format. w16a16 uses BF16 for both. The model page shows one tab per build profile, such as W8A8.

Build​

A build is one build profile of one model. It compiles in one compile invocation, which has the same name as the build profile.

Example: resnet_50 has three builds, one per build profile. The scripts name the w8a8 build with --build-profile w8a8 --compile-invocation w8a8.

Stage​

A stage is one step a build passes through. There are three: compile, performance, and accuracy. Every build compiles. The performance and accuracy stages run when model.yaml declares them.

StageWhere it runsModel scriptWhat it produces
compileModel Compilerscripts/compile.pyone MPK per model component, plus fidelity.json when model.yaml declares numerical_fidelity, a comparison of the compiled model's outputs with the original's
performanceModalix DevKitscripts/performance.pya model report with the performance of the whole model
accuracyModalix DevKitscripts/accuracy.pya model report with the accuracy on each declared dataset

A language model compiles with scripts/compile.sh instead, and its compile stage produces a deployment folder instead of MPKs.

Example: resnet_50 declares performance and accuracy in model.yaml, so each of its builds passes through all three stages.

Benchmark​

The benchmarks are the performance and accuracy stages. Both run on a DevKit and measure a compiled build.

Example: the benchmarks of the resnet_50 w8a8 build are a performance run and an accuracy run on the dataset imagenet-2012-val.

Model component​

A model component is a separately compiled part of a model. A model with one component names it model. A model with several components lists them in model.yaml in the order they run.

Example: resnet_50 has the one component model. By contrast, rf_detr_nano has two, backbone and decoder.

MPK​

MPK is the file format of a compiled model component: a .tar.gz archive that a DevKit loads and runs. The compile stage writes one MPK per model component.

Example: the w8a8 build of resnet_50 compiles to resnet_50_w8a8_mpk.tar.gz.

Model package​

A model package is what a build profile ships. It is one ZIP file with the MPK of each model component and the supporting files an application needs, such as the license file that model.yaml pins under resources. A language model's package is its deployment folder instead. The download on a model page is the model package of the selected build profile.

Example: the package of the w8a8 build is resnet_50_w8a8_default.zip. It holds resnet_50_w8a8_mpk.tar.gz and the license file licenses/LICENSE.

Conversion​

Conversion exports the official checkpoint of a model to ONNX, the format the compile stage reads. The model's scripts/convert.py does it, started by scripts/run_model_conversion.sh. A model needs no conversion when its model.yaml pins an ONNX file.

Example: resnet_50 has no conversion. Its compilation.source pins onnx/model.onnx from a Hugging Face repository, with the file's SHA-256 and size.

Calibration​

Calibration runs sample inputs through the model so that the compiler can choose the value ranges of an INT8 build. model.yaml pins each calibration dataset under calibrations, and a build profile names the one it uses.

Example: the w8a8 build profile of resnet_50 sets calibration_dataset: imagenet-2012-cal and calibration_samples: 50. The w8a16 and w16a16 build profiles use no calibration dataset.

Fingerprint​

A fingerprint is a SHA-256 of every input that decides the result of a stage, such as the source, the scripts, and the compiler version. CI reruns a stage only when one of its inputs changed, which it detects from a changed fingerprint. A sha256 field is different: it is the SHA-256 of one file.

Example: a change to models/resnet_50/scripts/performance.py changes only the performance fingerprint, so CI measures performance again without compiling. A change to compile.py changes the compile fingerprint, so CI compiles again and then reruns both benchmarks. What a change reruns lists the inputs of each fingerprint.

Model report​

A model report is the file model-report.json that a benchmark writes. A performance report holds one number per metric under metrics. An accuracy report holds, under datasets, the metrics measured on each dataset. model-report.schema.json defines the format.

Example: the performance report of resnet_50 holds metrics.throughput_fps. Its accuracy report holds top1_accuracy and top5_accuracy for imagenet-2012-val.

Metric​

A metric is what a benchmark measures, with its unit and the direction that is better. metrics.json names every metric a model report may hold.

Example: throughput_fps counts inferences per second, and higher is better. top1_accuracy is a ratio from 0 to 1, and higher is better.

Measurement​

A measurement is one measured value of a metric, for one build.

Example: models/resnet_50/tests/tests.yaml records 1508.0 as the expected throughput_fps of the w8a8 build. That number is a measurement of the metric throughput_fps.

On-demand build​

An on-demand build belongs to a model that sets build-on-demand: true in model.yaml. CI checks such a model on every push, but compiles and benchmarks it only when a maintainer starts the build for chosen build profiles. Models that are expensive to compile use this setting. Promote a model describes how such a build becomes available for download.

Example: resnet_50 does not set the field, and its default is false. CI therefore compiles and benchmarks its builds after each change that affects them.