Raspberry Pi AI Camera

The Raspberry Pi AI Camera pairs Sony’s IMX500 image sensor with an on-chip neural-network accelerator. The sensor runs inference concurrently with video capture — at no host-CPU cost — by loading a compiled network file (.rpk) onto the sensor at startup.

QVideo supports the AI Camera through QIMXCamera, a subclass of QPicamera that adds one extra signal:

QPicamera                   QIMXCamera
──────────────────────────  ──────────────────────────────────────
newFrame (ndarray)          newFrame (ndarray)     ← inherited
all picamera2 controls      all picamera2 controls ← inherited
                            newOutput (list|None)  ← new

newOutput is emitted once per frame. Its payload is either a list of numpy.ndarray objects (the raw tensor outputs produced by the sensor’s AI accelerator) or None when the sensor has not yet produced results (typically the first few frames after start-up).

Installation

Install picamera2 with IMX500 support on Raspberry Pi OS:

pip install "picamera2[imx500]"

Then obtain a pre-compiled network (see Model format below).

Basic usage

from QVideo.cameras.Picamera import QIMXCamera, QIMXSource

MODEL = (
    '/usr/share/imx500-models/'
    'imx500_network_nanodet_plus_416x416.rpk'
)

source = QIMXSource(model=MODEL)
source.source.newOutput.connect(my_widget.setOutputs)
source.start()

Because QIMXCamera is a full QPicamera subclass, all standard controls (exposure, gain, white balance, etc.) remain accessible through QPicameraTree as usual.

Writing an inference consumer

Connect newOutput to a slot that interprets the tensor list for the specific model loaded. The slot runs in the Qt main thread, so processing should be lightweight. For heavy post-processing, offload to a QThread or follow the same drop-frame strategy used by AsyncVideoFilter.

The example below reads scores and bounding boxes from a NanoDet-Plus model:

import numpy as np
from qtpy import QtCore, QtWidgets


class DetectionWidget(QtWidgets.QGroupBox):
    '''Display object-detection results from an IMX500 AI Camera.'''

    def __init__(self, parent=None) -> None:
        super().__init__('Detections', parent)
        self.setCheckable(True)
        self.setChecked(True)
        self._source = None

    @property
    def source(self):
        return self._source

    @source.setter
    def source(self, src) -> None:
        if self._source is not None:
            self._source.source.newOutput.disconnect(
                self._setOutputs)
        self._source = src
        if src is not None:
            src.source.newOutput.connect(self._setOutputs)

    @QtCore.Slot(object)
    def _setOutputs(self, outputs) -> None:
        if outputs is None or not self.isChecked():
            return
        scores = outputs[0]    # model-specific layout
        boxes  = outputs[1]
        # ... update graphics items here

The tensor layout (which index holds scores, boxes, class IDs, etc.) is model-specific. Refer to the model’s documentation or the picamera2 examples for the correct interpretation.

Wiring into QCamcorder

Subclass QCamcorder and add DetectionWidget alongside the camera tree:

from QVideo import QCamcorder
from QVideo.cameras.Picamera import QIMXSource

MODEL = (
    '/usr/share/imx500-models/'
    'imx500_network_nanodet_plus_416x416.rpk'
)


class AIDemo(QCamcorder):

    def __init__(self, source=None, parent=None) -> None:
        source = source or QIMXSource(model=MODEL)
        super().__init__(source, parent)
        self._detections = DetectionWidget(self)
        self._detections.source = source
        self._layout.addWidget(self._detections)


if __name__ == '__main__':  # pragma: no cover
    import pyqtgraph as pg
    from qtpy import QtGui, QtWidgets

    app = pg.mkQApp('AIDemo')
    widget = AIDemo()
    QtWidgets.QShortcut(
        QtGui.QKeySequence('Ctrl+Q'), widget
    ).activated.connect(app.quit)
    widget.show()
    pg.exec()

Model format

The IMX500 requires a packed-network file (.rpk) compiled for the sensor’s AI accelerator. Sources:

• Pre-built models — installed to /usr/share/imx500-models/ on Raspberry Pi OS via the imx500-all apt package.

• Raspberry Pi model zoo — available via the picamera2 GitHub repository examples.

• Custom models — convert any supported ONNX or TFLite model with the IMX500 model-conversion tools provided by Sony.

Note

QIMXCamera is intentionally excluded from the automatic camera-discovery backend registry because it requires a model path at construction time. Create it directly rather than via Camera().