Run a MIPI Camera Model

Field	Value
Difficulty	Intermediate
Estimated Read Time	10-15 minutes
Labels	mipi, camera, live-input, model, ev74

This chapter assumes the camera already works through the board overlay and libcamera. Neat does not select .dtbo files or tune the ISP; it consumes frames once libcamerasrc can produce them. Before running the tutorial, validate the camera with the hardware MIPI guide and a GStreamer caps check.

Think of the tutorial as gate 2. Gate 1 is camera bring-up: overlay, driver, libcamera, ISP, and exact caps. Gate 2 is the Neat graph: camera frames into CVU preprocessing, MLA inference, optional EV74 BoxDecode, and output pulling.

Walkthrough

Configure the camera source

CameraInputOptions describes the source caps Neat requests from libcamerasrc: resolution, frame rate, format, and an optional libcamera camera name. Set allow_cpu_fallback = true for current camera stacks that do not expose SiMaAI zero-copy buffers yet. Strict zero-copy is still available through --strict-zero-copy when your libcamerasrc supports it.

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

neat::CameraInputOptions camera;
camera.width = static_cast<std::uint32_t>(int_arg(argc, argv, "--width", 1920));
camera.height = static_cast<std::uint32_t>(int_arg(argc, argv, "--height", 1080));
camera.framerate_num = static_cast<std::uint32_t>(int_arg(argc, argv, "--fps", 30));
camera.framerate_den = 1;
camera.format = "NV12";
camera.buffer_name = "camera0";
camera.allow_cpu_fallback = !has_flag(argc, argv, "--strict-zero-copy");
std::string camera_name;
if (get_arg(argc, argv, "--camera-name", camera_name)) {
  camera.camera_name = camera_name;
}

Configure the model route

The model sees camera frames as NV12 images. Configure model-managed preprocessing for color conversion, resize, normalization, quantization, and tessellation. The example pins model-managed CVU preprocessing to EV74 so a production graph does not quietly become a CPU image pipeline. With --decode none, the route terminates at the MLA and returns raw model tensors. With a YOLO --decode token, BoxDecode runs as the model-managed EV74 postprocess stage.

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

const neat::BoxDecodeType decode_type = decode_type_from_token(decode_token);
neat::Model model(model_path, model_options_for_camera(camera, decode_type));

neat::Model::RouteOptions route;
route.include_input = false;
route.include_output = true;
route.upstream_name = camera.buffer_name;
route.buffer_name = camera.buffer_name;
route.name_suffix = "_camera0";
route.advanced_execution.preprocess_target = "EV74";
if (decode_type != neat::BoxDecodeType::Unspecified) {
  route.advanced_execution.postprocess_target = "EV74";
}

Compose the source-owned graph

Add CameraInput first, then add the model route with include_input = false. There is no public Input node because frames originate inside the running pipeline. include_output = true keeps a pull endpoint for detections or tensors.

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

neat::Graph graph("mipi_camera_model");
graph.add(neat::nodes::CameraInput(camera));
graph.add(model.graph(route));

if (has_flag(argc, argv, "--print-backend")) {
  std::cout << graph.describe_backend(false) << "\n";
}

neat::Run run = graph.build();

Pull outputs

Build the graph and pull a fixed number of outputs. A timeout means no model output reached the app before --pull-timeout-ms; the camera may have stopped, caps may not have negotiated, or a downstream stage such as BoxDecode may be backpressured. Print tensor counts and the first tensor shape so you can confirm data is moving before adding application logic.

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

for (int i = 0; i < frames; ++i) {
  std::optional<neat::Sample> sample = run.pull(/*timeout_ms=*/pull_timeout_ms);
  if (!sample.has_value()) {
    std::cout << "frame=" << i << " output_timeout timeout_ms=" << pull_timeout_ms;
    const std::string last_error = run.last_error();
    if (!last_error.empty())
      std::cout << " last_error=" << last_error;
    std::cout << "\n";
    return 2;
  }
  const neat::TensorList tensors = neat::tensors_from_sample(*sample, true);
  std::cout << "frame=" << i << " tensors=" << tensors.size();
  if (!tensors.empty())
    std::cout << " first_shape=" << shape_string(tensors.front().shape);
  std::cout << "\n";
}

Run

Run this tutorial directly on a Modalix DevKit with a configured MIPI camera. Run prebuilt commands from the Neat install root; run build-from-source commands from the repo root. The model archive must match the preprocessing and optional --decode mode you request.

The default pull timeout is 15 seconds. Increase --pull-timeout-ms when you are collecting first-run diagnostics on a cold board.

C++ (prebuilt):

devkit$./lib/sima-neat/tutorials/tutorial_023_run_mipi_camera_model devkit$--model /path/to/model.tar.gz --frames 5 --decode none

For YOLO-style models with a supported BoxDecode route, choose a decode token such as yolov8 or yolov9seg:

devkit$python3 share/sima-neat/tutorials/023_run_mipi_camera_model/run_mipi_camera_model.py \devkit$  --model /path/to/yolo.tar.gz --frames 5 --decode yolov8

devkit$./lib/sima-neat/tutorials/tutorial_023_run_mipi_camera_model \devkit$  --model /path/to/yolo.tar.gz --frames 5 --decode yolov8

C++ (build from source):

devkit$./build.sh --target tutorial_023_run_mipi_camera_model

devkit$./build/tutorials-standalone/tutorial_023_run_mipi_camera_model \devkit$  --model /path/to/model.tar.gz --frames 5 --decode none

Expected output shape depends on the model and decode route. Raw MLA output usually contains model-specific tensors:

frame=0 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=1 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=2 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=3 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
frame=4 tensors=<raw_tensor_count> first_shape=[<model_specific_shape>]
[OK] 023_run_mipi_camera_model

With a supported BoxDecode route, the output changes to decoded detection or segmentation tensors. Use the tensor count and first shape as a movement check, not as a universal contract.

If you see output_timeout, validate the camera with gst-launch-1.0, then inspect the generated backend with --print-backend. For BoxDecode routes, confirm the model archive, --decode token, and thresholds match the model.

In Practice

Use --print-backend when you need to inspect the generated GStreamer path. The production path should contain libcamerasrc, neatcamerabridge when fallback is enabled, neatprocesscvu, neatprocessmla, optional EV74 postprocess, and appsink. It should not contain appsrc, ostosima, videoconvert, or videoscale unless you intentionally added a debug-only path.

Full source

Show the complete source programs

tutorials/023_run_mipi_camera_model/run_mipi_camera_model.cpp

// Run a model from a MIPI/libcamera camera source.
//
// Usage:
//   tutorial_023_run_mipi_camera_model --model /path/to/model.tar.gz [--frames 5]

#include <neat.h>

#include <algorithm>
#include <cctype>
#include <cstdint>
#include <filesystem>
#include <iostream>
#include <optional>
#include <stdexcept>
#include <string>
#include <vector>

namespace neat = simaai::neat;
namespace fs = std::filesystem;

namespace {

bool get_arg(int argc, char** argv, const std::string& key, std::string& out) {
  for (int i = 1; i + 1 < argc; ++i) {
    if (key == argv[i]) {
      out = argv[i + 1];
      return true;
    }
  }
  return false;
}

bool has_flag(int argc, char** argv, const std::string& key) {
  for (int i = 1; i < argc; ++i) {
    if (key == argv[i])
      return true;
  }
  return false;
}

int int_arg(int argc, char** argv, const std::string& key, int def) {
  std::string value;
  if (!get_arg(argc, argv, key, value))
    return def;
  return std::stoi(value);
}

std::string lower_copy(std::string value) {
  std::transform(value.begin(), value.end(), value.begin(),
                 [](unsigned char c) { return static_cast<char>(std::tolower(c)); });
  return value;
}

neat::BoxDecodeType decode_type_from_token(const std::string& token) {
  const std::string v = lower_copy(token);
  if (v.empty() || v == "none" || v == "raw")
    return neat::BoxDecodeType::Unspecified;
  if (v == "yolo")
    return neat::BoxDecodeType::Yolo;
  if (v == "yolov5")
    return neat::BoxDecodeType::YoloV5;
  if (v == "yolov8")
    return neat::BoxDecodeType::YoloV8;
  if (v == "yolov8seg" || v == "yolov8-seg")
    return neat::BoxDecodeType::YoloV8Seg;
  if (v == "yolov9")
    return neat::BoxDecodeType::YoloV9;
  if (v == "yolov9seg" || v == "yolov9-seg")
    return neat::BoxDecodeType::YoloV9Seg;
  throw std::runtime_error("unsupported --decode token: " + token);
}

template <typename Shape> std::string shape_string(const Shape& shape) {
  std::string out = "[";
  for (std::size_t i = 0; i < shape.size(); ++i) {
    out += std::to_string(shape[i]);
    if (i + 1 < shape.size())
      out += ",";
  }
  out += "]";
  return out;
}

neat::Model::Options model_options_for_camera(const neat::CameraInputOptions& camera,
                                              neat::BoxDecodeType decode_type) {
  neat::Model::Options options;
  options.preprocess.kind = neat::InputKind::Image;
  options.preprocess.input_max_width = static_cast<int>(camera.width);
  options.preprocess.input_max_height = static_cast<int>(camera.height);
  options.preprocess.input_max_depth = 3;
  options.preprocess.color_convert.input_format = neat::PreprocessColorFormat::NV12;
  options.preprocess.color_convert.output_format = neat::PreprocessColorFormat::RGB;
  options.preprocess.resize.enable = neat::AutoFlag::On;
  options.preprocess.resize.width = 640;
  options.preprocess.resize.height = 640;
  options.preprocess.resize.mode = neat::ResizeMode::Letterbox;
  options.preprocess.resize.pad_value = 114;
  options.preprocess.preset = neat::NormalizePreset::COCO_YOLO;
  options.advanced_execution.preprocess_target = "EV74";
  options.decode_type = decode_type;
  if (decode_type == neat::BoxDecodeType::Unspecified) {
    options.inference_terminal.mla_only = true;
  } else {
    options.advanced_execution.postprocess_target = "EV74";
    options.score_threshold = 0.25f;
    options.nms_iou_threshold = 0.45f;
    options.top_k = 100;
  }
  return options;
}

void usage(const char* argv0) {
  std::cerr << "Usage: " << argv0
            << " --model <model.tar.gz> [--frames 5] [--width 1920] [--height 1080] "
               "[--fps 30] [--camera-name NAME] [--decode none|yolov8|yolov9seg] "
               "[--pull-timeout-ms 15000] [--strict-zero-copy] [--print-backend]\n";
}

} // namespace

int main(int argc, char** argv) {
  try {
    std::string model_path;
    if (!get_arg(argc, argv, "--model", model_path)) {
      usage(argv[0]);
      return 1;
    }
    if (!fs::exists(model_path))
      throw std::runtime_error("model archive not found: " + model_path);

    const int frames = int_arg(argc, argv, "--frames", 5);
    if (frames <= 0)
      throw std::runtime_error("--frames must be positive");
    const int pull_timeout_ms = int_arg(argc, argv, "--pull-timeout-ms", 15000);
    if (pull_timeout_ms <= 0)
      throw std::runtime_error("--pull-timeout-ms must be positive");

    std::string decode_token = "none";
    get_arg(argc, argv, "--decode", decode_token);

    // CORE LOGIC
    neat::CameraInputOptions camera;
    camera.width = static_cast<std::uint32_t>(int_arg(argc, argv, "--width", 1920));
    camera.height = static_cast<std::uint32_t>(int_arg(argc, argv, "--height", 1080));
    camera.framerate_num = static_cast<std::uint32_t>(int_arg(argc, argv, "--fps", 30));
    camera.framerate_den = 1;
    camera.format = "NV12";
    camera.buffer_name = "camera0";
    camera.allow_cpu_fallback = !has_flag(argc, argv, "--strict-zero-copy");
    std::string camera_name;
    if (get_arg(argc, argv, "--camera-name", camera_name)) {
      camera.camera_name = camera_name;
    }

    const neat::BoxDecodeType decode_type = decode_type_from_token(decode_token);
    neat::Model model(model_path, model_options_for_camera(camera, decode_type));

    neat::Model::RouteOptions route;
    route.include_input = false;
    route.include_output = true;
    route.upstream_name = camera.buffer_name;
    route.buffer_name = camera.buffer_name;
    route.name_suffix = "_camera0";
    route.advanced_execution.preprocess_target = "EV74";
    if (decode_type != neat::BoxDecodeType::Unspecified) {
      route.advanced_execution.postprocess_target = "EV74";
    }

    neat::Graph graph("mipi_camera_model");
    graph.add(neat::nodes::CameraInput(camera));
    graph.add(model.graph(route));

    if (has_flag(argc, argv, "--print-backend")) {
      std::cout << graph.describe_backend(false) << "\n";
    }

    neat::Run run = graph.build();

    for (int i = 0; i < frames; ++i) {
      std::optional<neat::Sample> sample = run.pull(/*timeout_ms=*/pull_timeout_ms);
      if (!sample.has_value()) {
        std::cout << "frame=" << i << " output_timeout timeout_ms=" << pull_timeout_ms;
        const std::string last_error = run.last_error();
        if (!last_error.empty())
          std::cout << " last_error=" << last_error;
        std::cout << "\n";
        return 2;
      }
      const neat::TensorList tensors = neat::tensors_from_sample(*sample, true);
      std::cout << "frame=" << i << " tensors=" << tensors.size();
      if (!tensors.empty())
        std::cout << " first_shape=" << shape_string(tensors.front().shape);
      std::cout << "\n";
    }

    std::cout << "[OK] 023_run_mipi_camera_model\n";
    return 0;
  } catch (const std::exception& e) {
    std::cerr << "[FAIL] " << e.what() << "\n";
    return 1;
  }
}

Source

• C++
• Python
• README