Skip to main content

Request expensive or on-demand compile runners

Use this guide when your model needs more compilation capacity or more predictable completion than the default Spot runner provides. It explains what makes a model eligible for an exception, what to submit for approval, and how to use the approved runner. You request resources in model.yaml; the Vulcan infrastructure owner reviews and deploys the permission. Start with the default size on Spot.

What qualifies as a special model?​

A special model is one with a demonstrated compilation need that justifies an exception. It is not a schema category or an automatic entitlement for a model family. The following are reasons to request review, not automatic approvals:

Demonstrated needExamples worth investigatingAppropriate request
Default capacity cannot compile the model reliably because of memory or resource pressure.Large LLM/VLM checkpoints, long-context builds, or large vision graphs with measured peak-memory pressure or out-of-memory failures.The smallest sufficient elevated class, initially on Spot.
A larger runner materially improves an otherwise impractical compile.CPU-heavy compilation or expensive graph optimization, with measured runtime and cost comparisons.Elevated class supported by the comparison; more cores alone do not establish a speedup.
Long, costly-to-repeat compilation makes Spot reclamation unacceptable for the use case.A long detector, language, vision-language or audio build with lost work, verified interruptions, or a documented completion deadline and reliability requirement.On-demand at the existing size unless separate evidence supports a size upgrade.
Both capacity and completion reliability are limiting.A resource-heavy compile that also loses substantial work to Spot interruptions.Elevated class plus on-demand, with evidence for each permission.

A model name, parameter count, large input resolution, or membership in the LLM, VLM, audio or RF-DETR families is not sufficient on its own. Existing grants are for exact folder IDs, not whole families. A new model can request review before registration using representative measurements or documented resource estimates; label estimates clearly and explain what still needs measurement.

There is no universal parameter-count or compile-duration cutoff in this policy. Explain why the default runner is unsuitable for this build and why the added cost is justified. First rule out invalid model inputs, compiler errors, avoidable recipe work, and queue delays: on-demand does not fix those. A canceled GitHub job is not evidence of Spot reclamation unless corroborated by EC2 events. If there have been no observed interruptions, say so and justify the reliability need without claiming a measured interruption rate.

Choose size and market independently​

RequestPurposeVulcan approval
runner_class: default, runner_market: spotNormal compilation; both fields may be omitted.No model-specific grant.
runner_class: moderate, runner_market: spot64 GiB compile capacity while retaining Spot pricing.Maximum class of moderate or higher.
runner_class: expensive, runner_market: spotMore compile capacity while retaining Spot pricing.Maximum class of expensive or higher.
runner_class: most-expensive, runner_market: spot384 GiB compile capacity.Maximum class of most-expensive.
runner_class: default, runner_market: on-demandAvoid Spot reclamation without increasing runner size.Published policy v3 with allow_on_demand: true.
runner_class: expensive, runner_market: on-demandMore capacity and avoidance of Spot reclamation.Both a sufficient maximum class and allow_on_demand: true.

Class order is default < moderate < expensive < most-expensive.

Vulcan owns the mapping from class to EC2 instance types. Do not put instance types in the model manifest. Permission does not select a larger runner or a market automatically, trigger a build, or retry an interrupted Spot job on on-demand. Neither class nor market changes invalidate a reusable compile.

runner_market: on-demand selects EC2 purchasing. The separate top-level build-on-demand: true controls scheduling: it excludes the model from normal push builds and requires an explicit manual selection. Either purchasing market can be used with that scheduling setting.

Declare the request​

Set the request on the existing profile in models/<model_id>/model.yaml. These are partial examples; retain the model's real parameters and calibration. For a build approved for both an expensive runner and on-demand purchasing:

build_profiles:
w8a8:
runner_class: expensive
runner_market: on-demand
components:
model:
parameters: {precision: w8a8}

If only on-demand is justified, keep the default size:

runner_class: default
runner_market: on-demand

For an expensive Spot build, use runner_class: expensive with runner_market: spot (or omit the market). Request only the combination approved for your model.

Place both fields directly under the profile, alongside components, not inside parameters. A component binding can override either profile value. A missing market inherits the profile value, then defaults to spot; a missing class inherits the profile value, then defaults to default.

A profile's components compile together on one runner. Prefer a profile-level class and market for that shared invocation; do not use component overrides to try to allocate separate machines. All effective markets must agree or validation rejects the grouped build. For example, a profile requesting on-demand with one component overriding it to Spot conflicts unless every component resolves to Spot.

Obtain access through a Vulcan policy PR​

Before enabling the request on a branch that CI compiles:

  1. Measure the existing build. For a larger class, provide peak-memory or runtime evidence and explain why recipe improvements are insufficient. For on-demand, provide observed durations and interruption evidence; a canceled GitHub job alone does not establish Spot reclamation.
  2. Open a PR against the shared Vulcan approval catalog. Give exact model folder IDs, an owner, reason, evidence links, expected build frequency, the requested class/market, and the cost tradeoff. Measure size upgrades separately instead of assuming a larger runner is cheaper overall.
  3. Have the infrastructure owner review and deploy the grant. A model PR cannot approve itself. Existing size grants do not permit on-demand; an on-demand grant needs published policy v3 with allow_on_demand: true. Ask the owner to confirm that compatible Models workflows and Vulcan scaler code are deployed before enabling it; a merged policy PR alone does not prove it is active.
  4. Link the Vulcan PR in the Models PR, explain the runner request in the Models PR description, and validate the manifest. Enable only the reviewed request.

Use this checklist in the approval request so the owner can review it without reconstructing the build history:

Model folder ID(s):
Owner:
Profiles/components measured:
Requested class and market:
Why the default Spot runner is unsuitable:
Compiler version, current runner type, and build/run links:
Observed duration and peak memory (or clearly labeled estimates):
Interruption evidence and lost work, or the required completion window:
Recipe/resource reductions already considered:
Expected builds per week/month, including retries:
Estimated current versus requested cost per build and per month:
Why this is the smallest sufficient exception:
Validation plan and links to the Models/Vulcan PRs:

Estimate compute cost as hourly price × billed hours, including startup and retries; state the region, pricing date, assumptions, and excluded charges. On-demand can be worthwhile for predictable completion without being cheaper. For a size upgrade, compare runtime as well as hourly price. Do not claim a speedup or cost saving that has not been measured.

You do not need to operate Terraform or edit deployed specifications to use this feature. The infrastructure owner maintains the shared approval catalog and publishes it to both admission and provisioning. For size-only access, the grant sets maximum_runner_class high enough for the request. For on-demand it must also explicitly allow the market, including when the maximum class is default. The owner handles any required policy-version migration; existing size-only grants must not silently acquire on-demand permission. Infrastructure details belong in Vulcan's approval documentation.

The grant identity is sima-neat/models:<model_id>, where model_id is the exact folder name, without models/. It covers every current and future profile and component in that folder. Staging and production share the source-controlled catalog; there are no wildcard, expiration, or environment-scoped grants. A rename needs a policy update. Grants may precede model registration. Revocation affects future admission after deployment; it does not cancel an existing job. Compile grants do not authorize ONNX conversion runners.

Validate and run​

With contributor tooling installed, run from the repository root, replacing example_detector and w8a8 with the exact model/profile IDs:

model-registry validate models/example_detector
model-registry matrix --repository . --models example_detector --profiles w8a8

The matrix reports requested_runner_class and requested_runner_market. These local commands validate and describe the request. They do not grant access or read the deployed approvals, and neither do the pull request checks. After a push, CI checks each request against the deployed approvals before it starts a compile, and Vulcan checks it again before it launches a runner.

A push compiles each build whose compile fingerprint has no published result. To select a build yourself, run Vulcan Model Orchestrator with the model ID and profile ID. For example, from this repository with GitHub CLI access:

gh workflow run vulcan-model-orchestrator.yml --ref develop \
-f models=example_detector -f profiles=w8a8

The selected ref must contain the request. A change to the market alone may reuse an existing build and start no runner. To compile again on purpose, add -f force_rebuild_all=true to that dispatch. A dispatch that names profiles compiles the selected builds but publishes no package and runs no benchmark, unless the model sets build-on-demand: true. To rebuild and publish, dispatch with models only. If the package is already published for the current commit, push a new Git commit first. A manual selection is also how a model with build-on-demand: true gets built.

Verify the result and diagnose failures​

The compile job's Compile runner summary shows the requested, selected and actual market, the runner name, the instance type, the approval identity, and the policy revision. The published build keeps the same fields in its package manifest's provenance. They describe the runner that compiled the build, not a later run that reuses it, and older builds can omit them. Recording the runner does not change the compile fingerprint. A default Spot request needs no approval identity, but still records the policy revision. Link the first run and its summary in your Models PR, and ask the infrastructure owner to confirm that the instance was cleaned up. Local validation or admission alone does not prove that a runner started or that compilation succeeded.

SymptomAction
Missing approval or requested class exceeds the maximumCheck the exact folder ID and the deployed grant, not only whether its PR merged.
No Vulcan on-demand approvalCheck runtime policy v3 and allow_on_demand: true; a size grant is insufficient.
Conflicting marketsMake all bindings in the grouped invocation resolve to one market.
Unsupported schema or malformed grantAsk the infrastructure owner to check reader compatibility and policy publication. Do not bypass admission.
Job waits for a runnerCheck that the deployed scaler supports the requested market label and that pool capacity is available.
Runner market mismatchStop and investigate runner registration versus EC2 lifecycle; the job fails before recording identity.
No new compile jobCheck fingerprint reuse and build-on-demand; an approval never forces a rebuild.

There is no silent downgrade. On-demand avoids Spot reclamation but not cancellation, compiler errors, workflow timeouts, or Vulcan cleanup. The job timeout is 360 minutes; the compiler profile's hard lifetime is seven hours, including startup and cleanup margin. A larger class does not extend these limits.

Revisit the exception when the compiler, recipe or model changes. Remove an unneeded manifest request and ask the infrastructure owner to reduce or revoke the grant. Policy changes affect future admission, not a job already running.