CameraInput Node

The following libcamera guide describes the default backend. For Metoak SIMOR owned raw tensors, see the raw V4L2 section below.

CameraInput is a source node for MIPI cameras exposed through libcamera and GStreamer libcamerasrc. Use it at the head of a source-owned graph when frames should come from the DevKit camera stack instead of from run.push(...).

For a task guide, see Use a MIPI Camera.

Prerequisites

Before you use CameraInput, bring up the camera outside Neat:

• attach the camera while the Modalix DevKit is powered off;
• select a .dtbo overlay that matches the carrier board, camera vendor, sensor, and port;
• confirm libcamera lists the camera;
• confirm libcamerasrc can stream the exact format,width,height,framerate you plan to request.

camera_name is the libcamera camera name. Leave it unset for the default camera, or copy the name from cam -l when the board exposes more than one camera.

Quick start

#include <neat.h>

namespace neat = simaai::neat;

neat::CameraInputOptions opt;
opt.width = 1920;
opt.height = 1080;
opt.framerate_num = 30;
opt.framerate_den = 1;
opt.format = "NV12";
opt.buffer_name = "camera0";
opt.allow_cpu_fallback = true;

neat::Graph graph;
graph.add(neat::nodes::CameraInput(opt));
graph.add(neat::nodes::Output("frames"));

neat::Run run = graph.build();
std::optional<neat::Sample> frame = run.pull(/*timeout_ms=*/5000);

Because CameraInput owns the source, build the graph without a public Input node unless your application really has another app-pushed input.

API surface

namespace simaai::neat {
struct CameraInputOptions;
class CameraInput;
}

namespace simaai::neat::nodes {
std::shared_ptr<simaai::neat::Node> CameraInput(
    simaai::neat::CameraInputOptions opt = {});
std::shared_ptr<simaai::neat::Node> CameraInputWithCaptureBuffers(
    simaai::neat::CameraInputOptions opt,
    std::uint32_t capture_buffer_count);
}

Python:

opt = pyneat.CameraInputOptions()
node = pyneat.nodes.camera_input(opt, capture_buffer_count=32)

Options

Field	Default	Meaning
camera_name	unset	Optional libcamera camera name, usually copied from cam -l. Leave unset to let libcamera choose the default camera.
width	1920	Requested frame width in pixels. Must be supported by the camera mode and overlay.
height	1080	Requested frame height in pixels.
framerate_num	30	Framerate numerator.
framerate_den	1	Framerate denominator. 0 is normalized to 1.
format	"NV12"	Requested video/x-raw format. NV12 is the recommended model-preproc path.
buffer_name	"camera"	Logical buffer name used in downstream metadata and model route naming.
insert_queue	true	Insert a small live-source queue after allocator negotiation and zero-copy validation.
leaky_queue	true	Make the queue leaky downstream so live graphs prefer recent frames over stale backlog.
queue_depth	2	Maximum queued buffers when insert_queue is true.
allow_cpu_fallback	false	If false, require camera/device zero-copy support. Set it to true only to opt into Neat's private adaptive bridge into SiMaAI memory.

capture_buffer_count is now also a CameraInputOptions field. Zero preserves the backend default; a nonzero value overrides its capture count. The existing C++ factory and Python keyword remain supported. The libcamera limit is 128; raw V4L2 requires 4–128 buffers.

Input and output contract

Contract item	Behavior
Input role	Source node. The graph pulls frames from the camera; the app does not push samples into this node.
Output media type	video/x-raw.
Output format	The requested format, usually NV12.
Memory contract	Prefers device/SiMaAI zero-copy. With fallback enabled, OS/libcamera buffers are adapted into SiMaAI memory for downstream CVU/MLA stages.
Caps certainty	Static hint based on the requested options. Runtime caps negotiation still depends on the camera stack.

Zero-copy and fallback

allow_cpu_fallback = false is the default and requests strict camera/device zero-copy. The build fails with an actionable error unless the installed libcamerasrc exposes the generic external-buffer-mode property and the memory library can export the camera allocation as DMA-BUFs.

Neat always places its private camera bridge directly after the capsfilter and before the live queue. The bridge proposes a standard pool through GStreamer's downstream GST_QUERY_ALLOCATION. The pool derives plane sizes and strides from the validated caps, allocates them from one packed SiMaAI allocation, and exports one DMA-BUF per plane. libcamerasrc imports those DMA-BUFs into the ISP capture queue. The bridge verifies the result and unwraps the same packed allocation without a copy in strict mode.

Pass capture_buffer_count to the camera factory based on the downstream owner. A temporal encoder or deep asynchronous ML graph may need 32 buffers; a low-latency CV graph may use fewer and drop stale frames. This setting controls the ISP-output/application cycle only. It does not enlarge the kernel's private CSI-to-ISP RAW transit ring, and it is independent of the GStreamer queue_depth drop policy.

Set allow_cpu_fallback = true explicitly to permit the already-present bridge to copy when the camera stack cannot honor the proposed allocator. The bridge passes through SiMaAI/EV74 buffers when upstream provides them. Otherwise, it copies the camera frame into a pooled SiMaAI buffer and attaches the metadata downstream stages expect. That private copy pool grows on demand instead of being capped by queue_depth, allowing a downstream leaky queue to discard stale frames without the bridge stalling first.

The fallback name means the bridge can accept OS/libcamera buffers. It does not mean you should move resize, color conversion, or normalization to the CPU. Keep those operations in model-managed CVU preprocessing whenever the route supports it.

Do not add an OsToSima node yourself for CameraInput. The adaptation belongs to the camera source path.

Common graph shapes

Pull camera frames directly:

CameraInput -> Output

Run a camera through a model:

CameraInput -> model.graph({include_input=false, include_output=true})

For model pipelines, configure preprocessing through Model::Options::preprocess or pyneat.ModelOptions.preprocess so resize, color conversion, normalization, quantization, and tessellation stay in the model-managed CVU route.

Failure modes

Error	Likely cause
build.plugin_missing	libcamerasrc or neatcamerabridge is missing from the runtime plugin set.
misconfig.media_caps / not-negotiated	The camera does not support the requested format, resolution, or framerate.
Strict zero-copy error	allow_cpu_fallback is false and the camera source does not expose SiMaAI zero-copy properties.
Pull timeout	The camera did not deliver frames, the graph is backpressured, or the camera stack stopped streaming.
Green, purple, or heavily tinted frames	The frame format or model preprocess color conversion is wrong, or the tint already exists in the camera ISP/libcamera output. Validate an NV12 capture outside Neat first.

Owned raw V4L2 capture

Use the existing CameraInput(options) factory (Python: camera_input) for both backends. The following example selects the Linux-only Metoak SIMOR raw-copy profile. Choose the actual capture device; /dev/video1 is only an example. Other platforms report an unsupported-backend error.

#include <neat.h>
namespace n = simaai::neat;

n::CameraInputOptions camera;
camera.device = "/dev/video1";
camera.profile = n::CameraProfile::MetoakSimor;

n::Graph capture;
capture.add(n::nodes::CameraInput(camera));
capture.add(n::nodes::Output());

import pyneat as neat

camera = neat.CameraInputOptions()
camera.device = "/dev/video1"
camera.profile = neat.CameraProfile.MetoakSimor

capture = neat.Graph()
capture.add(neat.nodes.camera_input(camera))
capture.add(neat.nodes.output())

MetoakSimor defaults to owned-copy capture when zero_copy is unset. Each dequeued MMAP frame is copied to independent pooled system memory before requeueing the camera buffer. Explicit zero_copy=false is equivalent; zero_copy=true is rejected because this profile does not support DMA-BUF export/import. There is no silent fallback. camera_name and allow_cpu_fallback are libcamera-only. The device cadence is retained; framerate_num/framerate_den do not configure this profile.

The output is a flat UInt8 tensor [bytesused], including valid row padding and trailer bytes, not a decoded image. Read negotiated fourcc, width, height, row-stride and sizeimage from Sample.caps_string. Do not reshape to decoded image dimensions or discard a trailer before interpreting the device protocol. Delivery timestamps remain metadata, not tensor bytes.

capture_buffer_count defaults to 8 (range 4–128), output_buffer_count to 8 (range 2–128), and frame_timeout_ms to 2000 (range 1–60000). Retained outputs consume pool slots. Release them to resume capture when the bounded pool fills; held tensors are never overwritten. An enabled queue must have a positive queue_depth smaller than output_buffer_count. Stopping interrupts pool waits.

Configure the device's media-controller links and pad formats using its BSP before starting capture; Neat does not rewrite the media graph. Do not run a competing capture client. Platform and USB devices use the same V4L2 MMAP lifecycle. The driver must honor the STREAMOFF contract: copy mode does not make faulty kernel DMA teardown safe. Failed retirement requires device recovery, not an automatic reopen.

For ROS/SIMOR, the application still performs CPU unpacking, calibration and ROS publication. It may feed the six original inputs to MetoakDepth graph 20 in a separate processing graph. This capture API does not add SIMOR processing or require graph 21, new firmware, or changes to an existing ROS integration.

Startup prepares output caps and the CPU buffer pool before queueing camera buffers. Stop interrupts frame and pool waits, retires capture, then releases the output pool. Tensors already delivered remain valid after stop. A capture-retirement error is reported on the GStreamer bus and prevents restarting that source; process exit is not a substitute for device recovery.

Profile selection and compatibility

• profile=Default (the default) always uses libcamera and preserves camera_name selection. Setting device without profile=MetoakSimor is rejected; it never implicitly selects Metoak.
• profile=MetoakSimor privately selects owned-copy V4L2 with RAW8, BA81, and 1920×360 wire geometry. It replaces the legacy NV12/1080 image defaults; other conflicting format or geometry overrides are rejected. This profile does not describe every Metoak camera.
• A SIMOR profile without device requires exactly one matching endpoint in the media topology. Missing, ambiguous or mixed-sensor routes fail; no capture device is opened during discovery. Explicit profile plus device avoids discovery.
• Pixel format is not a backend selector. Generic raw/Bayer requests stay on the libcamera route; this change does not add or qualify Bayer caps negotiation. The profile owns its wire FourCC; no public FourCC option is exposed.
• For MetoakSimor, unset zero_copy resolves to false; explicit true fails. For libcamera, unset preserves allow_cpu_fallback; false permits the existing copy fallback, while true requires zero-copy. Contradictory settings fail.
• The implementation is selected once during node construction, before graph contract propagation. There is no public backend option, and a failed implementation never switches to another one. Discovery does not configure media links or pad formats.

Only a MetoakSimor profile without device scans media controllers. These metadata queries are synchronous kernel calls, not deadline-bounded operations. Set both device and profile=MetoakSimor to bypass discovery when a controller is unresponsive.

Use only CameraInput(options) and CameraInputOptions; V4L2 configuration is private. The manifest and SONAME remain at Core ABI 5. The public options and node layouts changed for this unreleased integration: rebuild C++ consumers and use the matching Python wheel. Do not mix older ABI-5 binaries with the new headers or runtime.