1059b1c5bSMauro Carvalho Chehab.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later 254f38fcaSMauro Carvalho Chehab 354f38fcaSMauro Carvalho Chehab.. _subdev: 454f38fcaSMauro Carvalho Chehab 554f38fcaSMauro Carvalho Chehab******************** 654f38fcaSMauro Carvalho ChehabSub-device Interface 754f38fcaSMauro Carvalho Chehab******************** 854f38fcaSMauro Carvalho Chehab 954f38fcaSMauro Carvalho ChehabThe complex nature of V4L2 devices, where hardware is often made of 1054f38fcaSMauro Carvalho Chehabseveral integrated circuits that need to interact with each other in a 1154f38fcaSMauro Carvalho Chehabcontrolled way, leads to complex V4L2 drivers. The drivers usually 1254f38fcaSMauro Carvalho Chehabreflect the hardware model in software, and model the different hardware 1354f38fcaSMauro Carvalho Chehabcomponents as software blocks called sub-devices. 1454f38fcaSMauro Carvalho Chehab 1554f38fcaSMauro Carvalho ChehabV4L2 sub-devices are usually kernel-only objects. If the V4L2 driver 1654f38fcaSMauro Carvalho Chehabimplements the media device API, they will automatically inherit from 1754f38fcaSMauro Carvalho Chehabmedia entities. Applications will be able to enumerate the sub-devices 1854f38fcaSMauro Carvalho Chehaband discover the hardware topology using the media entities, pads and 1954f38fcaSMauro Carvalho Chehablinks enumeration API. 2054f38fcaSMauro Carvalho Chehab 2154f38fcaSMauro Carvalho ChehabIn addition to make sub-devices discoverable, drivers can also choose to 2254f38fcaSMauro Carvalho Chehabmake them directly configurable by applications. When both the 2354f38fcaSMauro Carvalho Chehabsub-device driver and the V4L2 device driver support this, sub-devices 2454f38fcaSMauro Carvalho Chehabwill feature a character device node on which ioctls can be called to 2554f38fcaSMauro Carvalho Chehab 2654f38fcaSMauro Carvalho Chehab- query, read and write sub-devices controls 2754f38fcaSMauro Carvalho Chehab 2854f38fcaSMauro Carvalho Chehab- subscribe and unsubscribe to events and retrieve them 2954f38fcaSMauro Carvalho Chehab 3054f38fcaSMauro Carvalho Chehab- negotiate image formats on individual pads 3154f38fcaSMauro Carvalho Chehab 32ea73eda5SJacopo Mondi- inspect and modify internal data routing between pads of the same entity 33ea73eda5SJacopo Mondi 3454f38fcaSMauro Carvalho ChehabSub-device character device nodes, conventionally named 3554f38fcaSMauro Carvalho Chehab``/dev/v4l-subdev*``, use major number 81. 3654f38fcaSMauro Carvalho Chehab 373fb0ee8bSJacopo MondiDrivers may opt to limit the sub-device character devices to only expose 383fb0ee8bSJacopo Mondioperations that do not modify the device state. In such a case the sub-devices 393fb0ee8bSJacopo Mondiare referred to as ``read-only`` in the rest of this documentation, and the 403fb0ee8bSJacopo Mondirelated restrictions are documented in individual ioctls. 413fb0ee8bSJacopo Mondi 4254f38fcaSMauro Carvalho Chehab 4354f38fcaSMauro Carvalho ChehabControls 4454f38fcaSMauro Carvalho Chehab======== 4554f38fcaSMauro Carvalho Chehab 4654f38fcaSMauro Carvalho ChehabMost V4L2 controls are implemented by sub-device hardware. Drivers 4754f38fcaSMauro Carvalho Chehabusually merge all controls and expose them through video device nodes. 4854f38fcaSMauro Carvalho ChehabApplications can control all sub-devices through a single interface. 4954f38fcaSMauro Carvalho Chehab 5054f38fcaSMauro Carvalho ChehabComplex devices sometimes implement the same control in different pieces 5154f38fcaSMauro Carvalho Chehabof hardware. This situation is common in embedded platforms, where both 5254f38fcaSMauro Carvalho Chehabsensors and image processing hardware implement identical functions, 5354f38fcaSMauro Carvalho Chehabsuch as contrast adjustment, white balance or faulty pixels correction. 5454f38fcaSMauro Carvalho ChehabAs the V4L2 controls API doesn't support several identical controls in a 5554f38fcaSMauro Carvalho Chehabsingle device, all but one of the identical controls are hidden. 5654f38fcaSMauro Carvalho Chehab 5754f38fcaSMauro Carvalho ChehabApplications can access those hidden controls through the sub-device 5854f38fcaSMauro Carvalho Chehabnode with the V4L2 control API described in :ref:`control`. The ioctls 5954f38fcaSMauro Carvalho Chehabbehave identically as when issued on V4L2 device nodes, with the 6054f38fcaSMauro Carvalho Chehabexception that they deal only with controls implemented in the 6154f38fcaSMauro Carvalho Chehabsub-device. 6254f38fcaSMauro Carvalho Chehab 6354f38fcaSMauro Carvalho ChehabDepending on the driver, those controls might also be exposed through 6454f38fcaSMauro Carvalho Chehabone (or several) V4L2 device nodes. 6554f38fcaSMauro Carvalho Chehab 6654f38fcaSMauro Carvalho Chehab 6754f38fcaSMauro Carvalho ChehabEvents 6854f38fcaSMauro Carvalho Chehab====== 6954f38fcaSMauro Carvalho Chehab 7054f38fcaSMauro Carvalho ChehabV4L2 sub-devices can notify applications of events as described in 7154f38fcaSMauro Carvalho Chehab:ref:`event`. The API behaves identically as when used on V4L2 device 7254f38fcaSMauro Carvalho Chehabnodes, with the exception that it only deals with events generated by 7354f38fcaSMauro Carvalho Chehabthe sub-device. Depending on the driver, those events might also be 7454f38fcaSMauro Carvalho Chehabreported on one (or several) V4L2 device nodes. 7554f38fcaSMauro Carvalho Chehab 7654f38fcaSMauro Carvalho Chehab 7754f38fcaSMauro Carvalho Chehab.. _pad-level-formats: 7854f38fcaSMauro Carvalho Chehab 7954f38fcaSMauro Carvalho ChehabPad-level Formats 8054f38fcaSMauro Carvalho Chehab================= 8154f38fcaSMauro Carvalho Chehab 8254f38fcaSMauro Carvalho Chehab.. warning:: 8354f38fcaSMauro Carvalho Chehab 8454f38fcaSMauro Carvalho Chehab Pad-level formats are only applicable to very complex devices that 8554f38fcaSMauro Carvalho Chehab need to expose low-level format configuration to user space. Generic 8654f38fcaSMauro Carvalho Chehab V4L2 applications do *not* need to use the API described in this 8754f38fcaSMauro Carvalho Chehab section. 8854f38fcaSMauro Carvalho Chehab 8954f38fcaSMauro Carvalho Chehab.. note:: 9054f38fcaSMauro Carvalho Chehab 9154f38fcaSMauro Carvalho Chehab For the purpose of this section, the term *format* means the 9254f38fcaSMauro Carvalho Chehab combination of media bus data format, frame width and frame height. 9354f38fcaSMauro Carvalho Chehab 9454f38fcaSMauro Carvalho ChehabImage formats are typically negotiated on video capture and output 9554f38fcaSMauro Carvalho Chehabdevices using the format and 9654f38fcaSMauro Carvalho Chehab:ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` ioctls. The driver is 9754f38fcaSMauro Carvalho Chehabresponsible for configuring every block in the video pipeline according 9854f38fcaSMauro Carvalho Chehabto the requested format at the pipeline input and/or output. 9954f38fcaSMauro Carvalho Chehab 10054f38fcaSMauro Carvalho ChehabFor complex devices, such as often found in embedded systems, identical 10154f38fcaSMauro Carvalho Chehabimage sizes at the output of a pipeline can be achieved using different 10254f38fcaSMauro Carvalho Chehabhardware configurations. One such example is shown on 10354f38fcaSMauro Carvalho Chehab:ref:`pipeline-scaling`, where image scaling can be performed on both 10454f38fcaSMauro Carvalho Chehabthe video sensor and the host image processing hardware. 10554f38fcaSMauro Carvalho Chehab 10654f38fcaSMauro Carvalho Chehab 10754f38fcaSMauro Carvalho Chehab.. _pipeline-scaling: 10854f38fcaSMauro Carvalho Chehab 10954f38fcaSMauro Carvalho Chehab.. kernel-figure:: pipeline.dot 11054f38fcaSMauro Carvalho Chehab :alt: pipeline.dot 11154f38fcaSMauro Carvalho Chehab :align: center 11254f38fcaSMauro Carvalho Chehab 11354f38fcaSMauro Carvalho Chehab Image Format Negotiation on Pipelines 11454f38fcaSMauro Carvalho Chehab 11554f38fcaSMauro Carvalho Chehab High quality and high speed pipeline configuration 11654f38fcaSMauro Carvalho Chehab 11754f38fcaSMauro Carvalho Chehab 11854f38fcaSMauro Carvalho Chehab 11954f38fcaSMauro Carvalho ChehabThe sensor scaler is usually of less quality than the host scaler, but 12054f38fcaSMauro Carvalho Chehabscaling on the sensor is required to achieve higher frame rates. 12154f38fcaSMauro Carvalho ChehabDepending on the use case (quality vs. speed), the pipeline must be 12254f38fcaSMauro Carvalho Chehabconfigured differently. Applications need to configure the formats at 12354f38fcaSMauro Carvalho Chehabevery point in the pipeline explicitly. 12454f38fcaSMauro Carvalho Chehab 12554f38fcaSMauro Carvalho ChehabDrivers that implement the :ref:`media API <media-controller-intro>` 12654f38fcaSMauro Carvalho Chehabcan expose pad-level image format configuration to applications. When 12754f38fcaSMauro Carvalho Chehabthey do, applications can use the 12854f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and 12954f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls. to 13054f38fcaSMauro Carvalho Chehabnegotiate formats on a per-pad basis. 13154f38fcaSMauro Carvalho Chehab 13254f38fcaSMauro Carvalho ChehabApplications are responsible for configuring coherent parameters on the 13354f38fcaSMauro Carvalho Chehabwhole pipeline and making sure that connected pads have compatible 13454f38fcaSMauro Carvalho Chehabformats. The pipeline is checked for formats mismatch at 13554f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time, and an ``EPIPE`` error 13654f38fcaSMauro Carvalho Chehabcode is then returned if the configuration is invalid. 13754f38fcaSMauro Carvalho Chehab 13854f38fcaSMauro Carvalho ChehabPad-level image format configuration support can be tested by calling 13954f38fcaSMauro Carvalho Chehabthe :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad 14054f38fcaSMauro Carvalho Chehab0. If the driver returns an ``EINVAL`` error code pad-level format 14154f38fcaSMauro Carvalho Chehabconfiguration is not supported by the sub-device. 14254f38fcaSMauro Carvalho Chehab 14354f38fcaSMauro Carvalho Chehab 14454f38fcaSMauro Carvalho ChehabFormat Negotiation 14554f38fcaSMauro Carvalho Chehab------------------ 14654f38fcaSMauro Carvalho Chehab 14754f38fcaSMauro Carvalho ChehabAcceptable formats on pads can (and usually do) depend on a number of 14854f38fcaSMauro Carvalho Chehabexternal parameters, such as formats on other pads, active links, or 14954f38fcaSMauro Carvalho Chehabeven controls. Finding a combination of formats on all pads in a video 15054f38fcaSMauro Carvalho Chehabpipeline, acceptable to both application and driver, can't rely on 15154f38fcaSMauro Carvalho Chehabformats enumeration only. A format negotiation mechanism is required. 15254f38fcaSMauro Carvalho Chehab 15354f38fcaSMauro Carvalho ChehabCentral to the format negotiation mechanism are the get/set format 15454f38fcaSMauro Carvalho Chehaboperations. When called with the ``which`` argument set to 15554f38fcaSMauro Carvalho Chehab:ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_FMT>`, the 15654f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and 15754f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls operate on 15854f38fcaSMauro Carvalho Chehaba set of formats parameters that are not connected to the hardware 15954f38fcaSMauro Carvalho Chehabconfiguration. Modifying those 'try' formats leaves the device state 16054f38fcaSMauro Carvalho Chehabuntouched (this applies to both the software state stored in the driver 16154f38fcaSMauro Carvalho Chehaband the hardware state stored in the device itself). 16254f38fcaSMauro Carvalho Chehab 16354f38fcaSMauro Carvalho ChehabWhile not kept as part of the device state, try formats are stored in 16454f38fcaSMauro Carvalho Chehabthe sub-device file handles. A 16554f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` call will return 16654f38fcaSMauro Carvalho Chehabthe last try format set *on the same sub-device file handle*. Several 16754f38fcaSMauro Carvalho Chehabapplications querying the same sub-device at the same time will thus not 16854f38fcaSMauro Carvalho Chehabinteract with each other. 16954f38fcaSMauro Carvalho Chehab 17054f38fcaSMauro Carvalho ChehabTo find out whether a particular format is supported by the device, 17154f38fcaSMauro Carvalho Chehabapplications use the 17254f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctl. Drivers 17354f38fcaSMauro Carvalho Chehabverify and, if needed, change the requested ``format`` based on device 17454f38fcaSMauro Carvalho Chehabrequirements and return the possibly modified value. Applications can 17554f38fcaSMauro Carvalho Chehabthen choose to try a different format or accept the returned value and 17654f38fcaSMauro Carvalho Chehabcontinue. 17754f38fcaSMauro Carvalho Chehab 17854f38fcaSMauro Carvalho ChehabFormats returned by the driver during a negotiation iteration are 17954f38fcaSMauro Carvalho Chehabguaranteed to be supported by the device. In particular, drivers 18054f38fcaSMauro Carvalho Chehabguarantee that a returned format will not be further changed if passed 18154f38fcaSMauro Carvalho Chehabto an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` call as-is 18254f38fcaSMauro Carvalho Chehab(as long as external parameters, such as formats on other pads or links' 18354f38fcaSMauro Carvalho Chehabconfiguration are not changed). 18454f38fcaSMauro Carvalho Chehab 18554f38fcaSMauro Carvalho ChehabDrivers automatically propagate formats inside sub-devices. When a try 18654f38fcaSMauro Carvalho Chehabor active format is set on a pad, corresponding formats on other pads of 18754f38fcaSMauro Carvalho Chehabthe same sub-device can be modified by the driver. Drivers are free to 18854f38fcaSMauro Carvalho Chehabmodify formats as required by the device. However, they should comply 18954f38fcaSMauro Carvalho Chehabwith the following rules when possible: 19054f38fcaSMauro Carvalho Chehab 19154f38fcaSMauro Carvalho Chehab- Formats should be propagated from sink pads to source pads. Modifying 19254f38fcaSMauro Carvalho Chehab a format on a source pad should not modify the format on any sink 19354f38fcaSMauro Carvalho Chehab pad. 19454f38fcaSMauro Carvalho Chehab 19554f38fcaSMauro Carvalho Chehab- Sub-devices that scale frames using variable scaling factors should 19654f38fcaSMauro Carvalho Chehab reset the scale factors to default values when sink pads formats are 19754f38fcaSMauro Carvalho Chehab modified. If the 1:1 scaling ratio is supported, this means that 19854f38fcaSMauro Carvalho Chehab source pads formats should be reset to the sink pads formats. 19954f38fcaSMauro Carvalho Chehab 20054f38fcaSMauro Carvalho ChehabFormats are not propagated across links, as that would involve 20154f38fcaSMauro Carvalho Chehabpropagating them from one sub-device file handle to another. 20254f38fcaSMauro Carvalho ChehabApplications must then take care to configure both ends of every link 20354f38fcaSMauro Carvalho Chehabexplicitly with compatible formats. Identical formats on the two ends of 20454f38fcaSMauro Carvalho Chehaba link are guaranteed to be compatible. Drivers are free to accept 20554f38fcaSMauro Carvalho Chehabdifferent formats matching device requirements as being compatible. 20654f38fcaSMauro Carvalho Chehab 20754f38fcaSMauro Carvalho Chehab:ref:`sample-pipeline-config` shows a sample configuration sequence 20854f38fcaSMauro Carvalho Chehabfor the pipeline described in :ref:`pipeline-scaling` (table columns 20954f38fcaSMauro Carvalho Chehablist entity names and pad numbers). 21054f38fcaSMauro Carvalho Chehab 21154f38fcaSMauro Carvalho Chehab 21254f38fcaSMauro Carvalho Chehab.. raw:: latex 21354f38fcaSMauro Carvalho Chehab 214fea13a69SMauro Carvalho Chehab \begingroup 21554f38fcaSMauro Carvalho Chehab \scriptsize 216fea13a69SMauro Carvalho Chehab \setlength{\tabcolsep}{2pt} 21754f38fcaSMauro Carvalho Chehab 218fea13a69SMauro Carvalho Chehab.. tabularcolumns:: |p{2.0cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}| 21954f38fcaSMauro Carvalho Chehab 22054f38fcaSMauro Carvalho Chehab.. _sample-pipeline-config: 22154f38fcaSMauro Carvalho Chehab 22254f38fcaSMauro Carvalho Chehab.. flat-table:: Sample Pipeline Configuration 22354f38fcaSMauro Carvalho Chehab :header-rows: 1 22454f38fcaSMauro Carvalho Chehab :stub-columns: 0 22554f38fcaSMauro Carvalho Chehab :widths: 5 5 5 5 5 5 5 22654f38fcaSMauro Carvalho Chehab 22754f38fcaSMauro Carvalho Chehab * - 22854f38fcaSMauro Carvalho Chehab - Sensor/0 22954f38fcaSMauro Carvalho Chehab 23054f38fcaSMauro Carvalho Chehab format 23154f38fcaSMauro Carvalho Chehab - Frontend/0 23254f38fcaSMauro Carvalho Chehab 23354f38fcaSMauro Carvalho Chehab format 23454f38fcaSMauro Carvalho Chehab - Frontend/1 23554f38fcaSMauro Carvalho Chehab 23654f38fcaSMauro Carvalho Chehab format 23754f38fcaSMauro Carvalho Chehab - Scaler/0 23854f38fcaSMauro Carvalho Chehab 23954f38fcaSMauro Carvalho Chehab format 24054f38fcaSMauro Carvalho Chehab - Scaler/0 24154f38fcaSMauro Carvalho Chehab 24254f38fcaSMauro Carvalho Chehab compose selection rectangle 24354f38fcaSMauro Carvalho Chehab - Scaler/1 24454f38fcaSMauro Carvalho Chehab 24554f38fcaSMauro Carvalho Chehab format 24654f38fcaSMauro Carvalho Chehab * - Initial state 24754f38fcaSMauro Carvalho Chehab - 2048x1536 24854f38fcaSMauro Carvalho Chehab 24954f38fcaSMauro Carvalho Chehab SGRBG8_1X8 25054f38fcaSMauro Carvalho Chehab - (default) 25154f38fcaSMauro Carvalho Chehab - (default) 25254f38fcaSMauro Carvalho Chehab - (default) 25354f38fcaSMauro Carvalho Chehab - (default) 25454f38fcaSMauro Carvalho Chehab - (default) 25554f38fcaSMauro Carvalho Chehab * - Configure frontend sink format 25654f38fcaSMauro Carvalho Chehab - 2048x1536 25754f38fcaSMauro Carvalho Chehab 25854f38fcaSMauro Carvalho Chehab SGRBG8_1X8 25954f38fcaSMauro Carvalho Chehab - *2048x1536* 26054f38fcaSMauro Carvalho Chehab 26154f38fcaSMauro Carvalho Chehab *SGRBG8_1X8* 26254f38fcaSMauro Carvalho Chehab - *2046x1534* 26354f38fcaSMauro Carvalho Chehab 26454f38fcaSMauro Carvalho Chehab *SGRBG8_1X8* 26554f38fcaSMauro Carvalho Chehab - (default) 26654f38fcaSMauro Carvalho Chehab - (default) 26754f38fcaSMauro Carvalho Chehab - (default) 26854f38fcaSMauro Carvalho Chehab * - Configure scaler sink format 26954f38fcaSMauro Carvalho Chehab - 2048x1536 27054f38fcaSMauro Carvalho Chehab 27154f38fcaSMauro Carvalho Chehab SGRBG8_1X8 27254f38fcaSMauro Carvalho Chehab - 2048x1536 27354f38fcaSMauro Carvalho Chehab 27454f38fcaSMauro Carvalho Chehab SGRBG8_1X8 27554f38fcaSMauro Carvalho Chehab - 2046x1534 27654f38fcaSMauro Carvalho Chehab 27754f38fcaSMauro Carvalho Chehab SGRBG8_1X8 27854f38fcaSMauro Carvalho Chehab - *2046x1534* 27954f38fcaSMauro Carvalho Chehab 28054f38fcaSMauro Carvalho Chehab *SGRBG8_1X8* 28154f38fcaSMauro Carvalho Chehab - *0,0/2046x1534* 28254f38fcaSMauro Carvalho Chehab - *2046x1534* 28354f38fcaSMauro Carvalho Chehab 28454f38fcaSMauro Carvalho Chehab *SGRBG8_1X8* 28554f38fcaSMauro Carvalho Chehab * - Configure scaler sink compose selection 28654f38fcaSMauro Carvalho Chehab - 2048x1536 28754f38fcaSMauro Carvalho Chehab 28854f38fcaSMauro Carvalho Chehab SGRBG8_1X8 28954f38fcaSMauro Carvalho Chehab - 2048x1536 29054f38fcaSMauro Carvalho Chehab 29154f38fcaSMauro Carvalho Chehab SGRBG8_1X8 29254f38fcaSMauro Carvalho Chehab - 2046x1534 29354f38fcaSMauro Carvalho Chehab 29454f38fcaSMauro Carvalho Chehab SGRBG8_1X8 29554f38fcaSMauro Carvalho Chehab - 2046x1534 29654f38fcaSMauro Carvalho Chehab 29754f38fcaSMauro Carvalho Chehab SGRBG8_1X8 29854f38fcaSMauro Carvalho Chehab - *0,0/1280x960* 29954f38fcaSMauro Carvalho Chehab - *1280x960* 30054f38fcaSMauro Carvalho Chehab 30154f38fcaSMauro Carvalho Chehab *SGRBG8_1X8* 30254f38fcaSMauro Carvalho Chehab 30354f38fcaSMauro Carvalho Chehab.. raw:: latex 30454f38fcaSMauro Carvalho Chehab 305fea13a69SMauro Carvalho Chehab \endgroup 30654f38fcaSMauro Carvalho Chehab 30754f38fcaSMauro Carvalho Chehab1. Initial state. The sensor source pad format is set to its native 3MP 30854f38fcaSMauro Carvalho Chehab size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the 30954f38fcaSMauro Carvalho Chehab host frontend and scaler sink and source pads have the default 31054f38fcaSMauro Carvalho Chehab values, as well as the compose rectangle on the scaler's sink pad. 31154f38fcaSMauro Carvalho Chehab 31254f38fcaSMauro Carvalho Chehab2. The application configures the frontend sink pad format's size to 31354f38fcaSMauro Carvalho Chehab 2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The 31454f38fcaSMauro Carvalho Chehab driver propagates the format to the frontend source pad. 31554f38fcaSMauro Carvalho Chehab 31654f38fcaSMauro Carvalho Chehab3. The application configures the scaler sink pad format's size to 31754f38fcaSMauro Carvalho Chehab 2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to 31854f38fcaSMauro Carvalho Chehab match the frontend source size and media bus code. The media bus code 31954f38fcaSMauro Carvalho Chehab on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver 32054f38fcaSMauro Carvalho Chehab propagates the size to the compose selection rectangle on the 32154f38fcaSMauro Carvalho Chehab scaler's sink pad, and the format to the scaler source pad. 32254f38fcaSMauro Carvalho Chehab 32354f38fcaSMauro Carvalho Chehab4. The application configures the size of the compose selection 32454f38fcaSMauro Carvalho Chehab rectangle of the scaler's sink pad 1280x960. The driver propagates 32554f38fcaSMauro Carvalho Chehab the size to the scaler's source pad format. 32654f38fcaSMauro Carvalho Chehab 32754f38fcaSMauro Carvalho ChehabWhen satisfied with the try results, applications can set the active 32854f38fcaSMauro Carvalho Chehabformats by setting the ``which`` argument to 32954f38fcaSMauro Carvalho Chehab``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try 33054f38fcaSMauro Carvalho Chehabformats by drivers. To avoid modifying the hardware state during format 33154f38fcaSMauro Carvalho Chehabnegotiation, applications should negotiate try formats first and then 33254f38fcaSMauro Carvalho Chehabmodify the active settings using the try formats returned during the 33354f38fcaSMauro Carvalho Chehablast negotiation iteration. This guarantees that the active format will 33454f38fcaSMauro Carvalho Chehabbe applied as-is by the driver without being modified. 33554f38fcaSMauro Carvalho Chehab 33654f38fcaSMauro Carvalho Chehab 33754f38fcaSMauro Carvalho Chehab.. _v4l2-subdev-selections: 33854f38fcaSMauro Carvalho Chehab 33954f38fcaSMauro Carvalho ChehabSelections: cropping, scaling and composition 34054f38fcaSMauro Carvalho Chehab--------------------------------------------- 34154f38fcaSMauro Carvalho Chehab 34254f38fcaSMauro Carvalho ChehabMany sub-devices support cropping frames on their input or output pads 34354f38fcaSMauro Carvalho Chehab(or possible even on both). Cropping is used to select the area of 34454f38fcaSMauro Carvalho Chehabinterest in an image, typically on an image sensor or a video decoder. 34554f38fcaSMauro Carvalho ChehabIt can also be used as part of digital zoom implementations to select 34654f38fcaSMauro Carvalho Chehabthe area of the image that will be scaled up. 34754f38fcaSMauro Carvalho Chehab 34854f38fcaSMauro Carvalho ChehabCrop settings are defined by a crop rectangle and represented in a 34954f38fcaSMauro Carvalho Chehabstruct :c:type:`v4l2_rect` by the coordinates of the top 35054f38fcaSMauro Carvalho Chehableft corner and the rectangle size. Both the coordinates and sizes are 35154f38fcaSMauro Carvalho Chehabexpressed in pixels. 35254f38fcaSMauro Carvalho Chehab 35354f38fcaSMauro Carvalho ChehabAs for pad formats, drivers store try and active rectangles for the 35454f38fcaSMauro Carvalho Chehabselection targets :ref:`v4l2-selections-common`. 35554f38fcaSMauro Carvalho Chehab 35654f38fcaSMauro Carvalho ChehabOn sink pads, cropping is applied relative to the current pad format. 35754f38fcaSMauro Carvalho ChehabThe pad format represents the image size as received by the sub-device 35854f38fcaSMauro Carvalho Chehabfrom the previous block in the pipeline, and the crop rectangle 35954f38fcaSMauro Carvalho Chehabrepresents the sub-image that will be transmitted further inside the 36054f38fcaSMauro Carvalho Chehabsub-device for processing. 36154f38fcaSMauro Carvalho Chehab 36254f38fcaSMauro Carvalho ChehabThe scaling operation changes the size of the image by scaling it to new 36354f38fcaSMauro Carvalho Chehabdimensions. The scaling ratio isn't specified explicitly, but is implied 36454f38fcaSMauro Carvalho Chehabfrom the original and scaled image sizes. Both sizes are represented by 36554f38fcaSMauro Carvalho Chehabstruct :c:type:`v4l2_rect`. 36654f38fcaSMauro Carvalho Chehab 36754f38fcaSMauro Carvalho ChehabScaling support is optional. When supported by a subdev, the crop 36854f38fcaSMauro Carvalho Chehabrectangle on the subdev's sink pad is scaled to the size configured 36954f38fcaSMauro Carvalho Chehabusing the 37054f38fcaSMauro Carvalho Chehab:ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV_G_SELECTION>` IOCTL 37154f38fcaSMauro Carvalho Chehabusing ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the 37254f38fcaSMauro Carvalho Chehabsubdev supports scaling but not composing, the top and left values are 37354f38fcaSMauro Carvalho Chehabnot used and must always be set to zero. 37454f38fcaSMauro Carvalho Chehab 37554f38fcaSMauro Carvalho ChehabOn source pads, cropping is similar to sink pads, with the exception 37654f38fcaSMauro Carvalho Chehabthat the source size from which the cropping is performed, is the 37754f38fcaSMauro Carvalho ChehabCOMPOSE rectangle on the sink pad. In both sink and source pads, the 37854f38fcaSMauro Carvalho Chehabcrop rectangle must be entirely contained inside the source image size 37954f38fcaSMauro Carvalho Chehabfor the crop operation. 38054f38fcaSMauro Carvalho Chehab 38154f38fcaSMauro Carvalho ChehabThe drivers should always use the closest possible rectangle the user 38254f38fcaSMauro Carvalho Chehabrequests on all selection targets, unless specifically told otherwise. 38354f38fcaSMauro Carvalho Chehab``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round 38454f38fcaSMauro Carvalho Chehabthe image size either up or down. :ref:`v4l2-selection-flags` 38554f38fcaSMauro Carvalho Chehab 38654f38fcaSMauro Carvalho Chehab 38754f38fcaSMauro Carvalho ChehabTypes of selection targets 38854f38fcaSMauro Carvalho Chehab-------------------------- 38954f38fcaSMauro Carvalho Chehab 39054f38fcaSMauro Carvalho Chehab 39154f38fcaSMauro Carvalho ChehabActual targets 39254f38fcaSMauro Carvalho Chehab^^^^^^^^^^^^^^ 39354f38fcaSMauro Carvalho Chehab 39454f38fcaSMauro Carvalho ChehabActual targets (without a postfix) reflect the actual hardware 39554f38fcaSMauro Carvalho Chehabconfiguration at any point of time. There is a BOUNDS target 39654f38fcaSMauro Carvalho Chehabcorresponding to every actual target. 39754f38fcaSMauro Carvalho Chehab 39854f38fcaSMauro Carvalho Chehab 39954f38fcaSMauro Carvalho ChehabBOUNDS targets 40054f38fcaSMauro Carvalho Chehab^^^^^^^^^^^^^^ 40154f38fcaSMauro Carvalho Chehab 40254f38fcaSMauro Carvalho ChehabBOUNDS targets is the smallest rectangle that contains all valid actual 40354f38fcaSMauro Carvalho Chehabrectangles. It may not be possible to set the actual rectangle as large 40454f38fcaSMauro Carvalho Chehabas the BOUNDS rectangle, however. This may be because e.g. a sensor's 40554f38fcaSMauro Carvalho Chehabpixel array is not rectangular but cross-shaped or round. The maximum 40654f38fcaSMauro Carvalho Chehabsize may also be smaller than the BOUNDS rectangle. 40754f38fcaSMauro Carvalho Chehab 40854f38fcaSMauro Carvalho Chehab 409*58388bd7SSakari Ailus.. _format-propagation: 410*58388bd7SSakari Ailus 41154f38fcaSMauro Carvalho ChehabOrder of configuration and format propagation 41254f38fcaSMauro Carvalho Chehab--------------------------------------------- 41354f38fcaSMauro Carvalho Chehab 41454f38fcaSMauro Carvalho ChehabInside subdevs, the order of image processing steps will always be from 41554f38fcaSMauro Carvalho Chehabthe sink pad towards the source pad. This is also reflected in the order 41654f38fcaSMauro Carvalho Chehabin which the configuration must be performed by the user: the changes 41754f38fcaSMauro Carvalho Chehabmade will be propagated to any subsequent stages. If this behaviour is 41854f38fcaSMauro Carvalho Chehabnot desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This 41954f38fcaSMauro Carvalho Chehabflag causes no propagation of the changes are allowed in any 42054f38fcaSMauro Carvalho Chehabcircumstances. This may also cause the accessed rectangle to be adjusted 42154f38fcaSMauro Carvalho Chehabby the driver, depending on the properties of the underlying hardware. 42254f38fcaSMauro Carvalho Chehab 42354f38fcaSMauro Carvalho ChehabThe coordinates to a step always refer to the actual size of the 42454f38fcaSMauro Carvalho Chehabprevious step. The exception to this rule is the sink compose 42554f38fcaSMauro Carvalho Chehabrectangle, which refers to the sink compose bounds rectangle --- if it 42654f38fcaSMauro Carvalho Chehabis supported by the hardware. 42754f38fcaSMauro Carvalho Chehab 42854f38fcaSMauro Carvalho Chehab1. Sink pad format. The user configures the sink pad format. This format 42954f38fcaSMauro Carvalho Chehab defines the parameters of the image the entity receives through the 43054f38fcaSMauro Carvalho Chehab pad for further processing. 43154f38fcaSMauro Carvalho Chehab 43254f38fcaSMauro Carvalho Chehab2. Sink pad actual crop selection. The sink pad crop defines the crop 43354f38fcaSMauro Carvalho Chehab performed to the sink pad format. 43454f38fcaSMauro Carvalho Chehab 43554f38fcaSMauro Carvalho Chehab3. Sink pad actual compose selection. The size of the sink pad compose 43654f38fcaSMauro Carvalho Chehab rectangle defines the scaling ratio compared to the size of the sink 43754f38fcaSMauro Carvalho Chehab pad crop rectangle. The location of the compose rectangle specifies 43854f38fcaSMauro Carvalho Chehab the location of the actual sink compose rectangle in the sink compose 43954f38fcaSMauro Carvalho Chehab bounds rectangle. 44054f38fcaSMauro Carvalho Chehab 44154f38fcaSMauro Carvalho Chehab4. Source pad actual crop selection. Crop on the source pad defines crop 44254f38fcaSMauro Carvalho Chehab performed to the image in the sink compose bounds rectangle. 44354f38fcaSMauro Carvalho Chehab 44454f38fcaSMauro Carvalho Chehab5. Source pad format. The source pad format defines the output pixel 44554f38fcaSMauro Carvalho Chehab format of the subdev, as well as the other parameters with the 44654f38fcaSMauro Carvalho Chehab exception of the image width and height. Width and height are defined 44754f38fcaSMauro Carvalho Chehab by the size of the source pad actual crop selection. 44854f38fcaSMauro Carvalho Chehab 44954f38fcaSMauro Carvalho ChehabAccessing any of the above rectangles not supported by the subdev will 45054f38fcaSMauro Carvalho Chehabreturn ``EINVAL``. Any rectangle referring to a previous unsupported 45154f38fcaSMauro Carvalho Chehabrectangle coordinates will instead refer to the previous supported 45254f38fcaSMauro Carvalho Chehabrectangle. For example, if sink crop is not supported, the compose 45354f38fcaSMauro Carvalho Chehabselection will refer to the sink pad format dimensions instead. 45454f38fcaSMauro Carvalho Chehab 45554f38fcaSMauro Carvalho Chehab 45654f38fcaSMauro Carvalho Chehab.. _subdev-image-processing-crop: 45754f38fcaSMauro Carvalho Chehab 45854f38fcaSMauro Carvalho Chehab.. kernel-figure:: subdev-image-processing-crop.svg 45954f38fcaSMauro Carvalho Chehab :alt: subdev-image-processing-crop.svg 46054f38fcaSMauro Carvalho Chehab :align: center 46154f38fcaSMauro Carvalho Chehab 46254f38fcaSMauro Carvalho Chehab **Figure 4.5. Image processing in subdevs: simple crop example** 46354f38fcaSMauro Carvalho Chehab 46454f38fcaSMauro Carvalho ChehabIn the above example, the subdev supports cropping on its sink pad. To 46554f38fcaSMauro Carvalho Chehabconfigure it, the user sets the media bus format on the subdev's sink 46654f38fcaSMauro Carvalho Chehabpad. Now the actual crop rectangle can be set on the sink pad --- the 46754f38fcaSMauro Carvalho Chehablocation and size of this rectangle reflect the location and size of a 46854f38fcaSMauro Carvalho Chehabrectangle to be cropped from the sink format. The size of the sink crop 46954f38fcaSMauro Carvalho Chehabrectangle will also be the size of the format of the subdev's source 47054f38fcaSMauro Carvalho Chehabpad. 47154f38fcaSMauro Carvalho Chehab 47254f38fcaSMauro Carvalho Chehab 47354f38fcaSMauro Carvalho Chehab.. _subdev-image-processing-scaling-multi-source: 47454f38fcaSMauro Carvalho Chehab 47554f38fcaSMauro Carvalho Chehab.. kernel-figure:: subdev-image-processing-scaling-multi-source.svg 47654f38fcaSMauro Carvalho Chehab :alt: subdev-image-processing-scaling-multi-source.svg 47754f38fcaSMauro Carvalho Chehab :align: center 47854f38fcaSMauro Carvalho Chehab 47954f38fcaSMauro Carvalho Chehab **Figure 4.6. Image processing in subdevs: scaling with multiple sources** 48054f38fcaSMauro Carvalho Chehab 48154f38fcaSMauro Carvalho ChehabIn this example, the subdev is capable of first cropping, then scaling 48254f38fcaSMauro Carvalho Chehaband finally cropping for two source pads individually from the resulting 48354f38fcaSMauro Carvalho Chehabscaled image. The location of the scaled image in the cropped image is 48454f38fcaSMauro Carvalho Chehabignored in sink compose target. Both of the locations of the source crop 48554f38fcaSMauro Carvalho Chehabrectangles refer to the sink scaling rectangle, independently cropping 48654f38fcaSMauro Carvalho Chehaban area at location specified by the source crop rectangle from it. 48754f38fcaSMauro Carvalho Chehab 48854f38fcaSMauro Carvalho Chehab 48954f38fcaSMauro Carvalho Chehab.. _subdev-image-processing-full: 49054f38fcaSMauro Carvalho Chehab 49154f38fcaSMauro Carvalho Chehab.. kernel-figure:: subdev-image-processing-full.svg 49254f38fcaSMauro Carvalho Chehab :alt: subdev-image-processing-full.svg 49354f38fcaSMauro Carvalho Chehab :align: center 49454f38fcaSMauro Carvalho Chehab 49554f38fcaSMauro Carvalho Chehab **Figure 4.7. Image processing in subdevs: scaling and composition with multiple sinks and sources** 49654f38fcaSMauro Carvalho Chehab 49754f38fcaSMauro Carvalho ChehabThe subdev driver supports two sink pads and two source pads. The images 49854f38fcaSMauro Carvalho Chehabfrom both of the sink pads are individually cropped, then scaled and 49954f38fcaSMauro Carvalho Chehabfurther composed on the composition bounds rectangle. From that, two 50054f38fcaSMauro Carvalho Chehabindependent streams are cropped and sent out of the subdev from the 50154f38fcaSMauro Carvalho Chehabsource pads. 50254f38fcaSMauro Carvalho Chehab 50354f38fcaSMauro Carvalho Chehab 50454f38fcaSMauro Carvalho Chehab.. toctree:: 50554f38fcaSMauro Carvalho Chehab :maxdepth: 1 50654f38fcaSMauro Carvalho Chehab 50754f38fcaSMauro Carvalho Chehab subdev-formats 50870283e99STomi Valkeinen 50970283e99STomi ValkeinenStreams, multiplexed media pads and internal routing 51070283e99STomi Valkeinen---------------------------------------------------- 51170283e99STomi Valkeinen 512*58388bd7SSakari AilusSimple V4L2 sub-devices do not support multiple, unrelated video streams, 513*58388bd7SSakari Ailusand only a single stream can pass through a media link and a media pad. 514*58388bd7SSakari AilusThus each pad contains a format and selection configuration for that 515*58388bd7SSakari Ailussingle stream. A subdev can do stream processing and split a stream into 516*58388bd7SSakari Ailustwo or compose two streams into one, but the inputs and outputs for the 517*58388bd7SSakari Ailussubdev are still a single stream per pad. 51870283e99STomi Valkeinen 51970283e99STomi ValkeinenSome hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple 52070283e99STomi Valkeinendata streams are transmitted on the same bus, which is represented by a media 52170283e99STomi Valkeinenlink connecting a transmitter source pad with a sink pad on the receiver. For 52270283e99STomi Valkeinenexample, a camera sensor can produce two distinct streams, a pixel stream and a 52370283e99STomi Valkeinenmetadata stream, which are transmitted on the multiplexed data bus, represented 52470283e99STomi Valkeinenby a media link which connects the single sensor's source pad with the receiver 52570283e99STomi Valkeinensink pad. The stream-aware receiver will de-multiplex the streams received on 52670283e99STomi Valkeinenthe its sink pad and allows to route them individually to one of its source 52770283e99STomi Valkeinenpads. 52870283e99STomi Valkeinen 52970283e99STomi ValkeinenSubdevice drivers that support multiplexed streams are compatible with 53070283e99STomi Valkeinennon-multiplexed subdev drivers, but, of course, require a routing configuration 53170283e99STomi Valkeinenwhere the link between those two types of drivers contains only a single 53270283e99STomi Valkeinenstream. 53370283e99STomi Valkeinen 53470283e99STomi ValkeinenUnderstanding streams 53570283e99STomi Valkeinen^^^^^^^^^^^^^^^^^^^^^ 53670283e99STomi Valkeinen 53770283e99STomi ValkeinenA stream is a stream of content (e.g. pixel data or metadata) flowing through 53870283e99STomi Valkeinenthe media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a 53970283e99STomi Valkeinenreceiver and demultiplexer in a SoC). Each media link carries all the enabled 540*58388bd7SSakari Ailusstreams from one end of the link to the other, and sub-devices have routing 54170283e99STomi Valkeinentables which describe how the incoming streams from sink pads are routed to the 54270283e99STomi Valkeinensource pads. 54370283e99STomi Valkeinen 544*58388bd7SSakari AilusA stream ID is a media pad-local identifier for a stream. Streams IDs of 545*58388bd7SSakari Ailusthe same stream must be equal on both ends of a link. In other words, 546*58388bd7SSakari Ailusa particular stream ID must exist on both sides of a media 54770283e99STomi Valkeinenlink, but another stream ID can be used for the same stream at the other side 548*58388bd7SSakari Ailusof the sub-device. 54970283e99STomi Valkeinen 550*58388bd7SSakari AilusA stream at a specific point in the media pipeline is identified by the 551*58388bd7SSakari Ailussub-device and a (pad, stream) pair. For sub-devices that do not support 552*58388bd7SSakari Ailusmultiplexed streams the 'stream' field is always 0. 553*58388bd7SSakari Ailus 554*58388bd7SSakari AilusInteraction between routes, streams, formats and selections 555*58388bd7SSakari Ailus^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 556*58388bd7SSakari Ailus 557*58388bd7SSakari AilusThe addition of streams to the V4L2 sub-device interface moves the sub-device 558*58388bd7SSakari Ailusformats and selections from pads to (pad, stream) pairs. Besides the 559*58388bd7SSakari Ailususual pad, also the stream ID needs to be provided for setting formats and 560*58388bd7SSakari Ailusselections. The order of configuring formats and selections along a stream is 561*58388bd7SSakari Ailusthe same as without streams (see :ref:`format-propagation`). 562*58388bd7SSakari Ailus 563*58388bd7SSakari AilusInstead of the sub-device wide merging of streams from all sink pads 564*58388bd7SSakari Ailustowards all source pads, data flows for each route are separate from each 565*58388bd7SSakari Ailusother. Any number of routes from streams on sink pads towards streams on 566*58388bd7SSakari Ailussource pads is allowed, to the extent supported by drivers. For every 567*58388bd7SSakari Ailusstream on a source pad, however, only a single route is allowed. 568*58388bd7SSakari Ailus 569*58388bd7SSakari AilusAny configurations of a stream within a pad, such as format or selections, 570*58388bd7SSakari Ailusare independent of similar configurations on other streams. This is 571*58388bd7SSakari Ailussubject to change in the future. 57270283e99STomi Valkeinen 57370283e99STomi ValkeinenConfiguring streams 57470283e99STomi Valkeinen^^^^^^^^^^^^^^^^^^^ 57570283e99STomi Valkeinen 576*58388bd7SSakari AilusThe configuration of the streams is done individually for each sub-device and 577*58388bd7SSakari Ailusthe validity of the streams between sub-devices is validated when the pipeline 57870283e99STomi Valkeinenis started. 57970283e99STomi Valkeinen 58070283e99STomi ValkeinenThere are three steps in configuring the streams: 58170283e99STomi Valkeinen 582*58388bd7SSakari Ailus1) Set up links. Connect the pads between sub-devices using the :ref:`Media 58370283e99STomi ValkeinenController API <media_controller>` 58470283e99STomi Valkeinen 585*58388bd7SSakari Ailus2) Streams. Streams are declared and their routing is configured by 586*58388bd7SSakari Ailussetting the routing table for the sub-device using 58770283e99STomi Valkeinen:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that 588*58388bd7SSakari Ailussetting the routing table will reset formats and selections in the 589*58388bd7SSakari Ailussub-device to default values. 59070283e99STomi Valkeinen 591*58388bd7SSakari Ailus3) Configure formats and selections. Formats and selections of each stream 592*58388bd7SSakari Ailusare configured separately as documented for plain sub-devices in 593*58388bd7SSakari Ailus:ref:`format-propagation`. The stream ID is set to the same stream ID 594*58388bd7SSakari Ailusassociated with either sink or source pads of routes configured using the 595*58388bd7SSakari Ailus:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. 59670283e99STomi Valkeinen 59770283e99STomi ValkeinenMultiplexed streams setup example 59870283e99STomi Valkeinen^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 59970283e99STomi Valkeinen 60070283e99STomi ValkeinenA simple example of a multiplexed stream setup might be as follows: 60170283e99STomi Valkeinen 60270283e99STomi Valkeinen- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source 60370283e99STomi Valkeinen pad (pad 0) which carries a pixel data stream. 60470283e99STomi Valkeinen 60570283e99STomi Valkeinen- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the 60670283e99STomi Valkeinen sensors (pads 0, 1), and one source pad (pad 2), which outputs two streams. 60770283e99STomi Valkeinen 60870283e99STomi Valkeinen- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), 60970283e99STomi Valkeinen connected to the bridge, and two source pads (pads 1-2), going to the DMA 61070283e99STomi Valkeinen engine. The receiver demultiplexes the incoming streams to the source pads. 61170283e99STomi Valkeinen 61270283e99STomi Valkeinen- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is 61370283e99STomi Valkeinen connected to a single source pad in the receiver. 61470283e99STomi Valkeinen 615*58388bd7SSakari AilusThe sensors, the bridge and the receiver are modeled as V4L2 sub-devices, 61670283e99STomi Valkeinenexposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are 61770283e99STomi Valkeinenmodeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. 61870283e99STomi Valkeinen 61970283e99STomi ValkeinenTo configure this pipeline, the userspace must take the following steps: 62070283e99STomi Valkeinen 62170283e99STomi Valkeinen1) Set up media links between entities: connect the sensors to the bridge, 62270283e99STomi Valkeinenbridge to the receiver, and the receiver to the DMA engines. This step does 62370283e99STomi Valkeinennot differ from normal non-multiplexed media controller setup. 62470283e99STomi Valkeinen 625*58388bd7SSakari Ailus2) Configure routing 62670283e99STomi Valkeinen 62770283e99STomi Valkeinen.. flat-table:: Bridge routing table 62870283e99STomi Valkeinen :header-rows: 1 62970283e99STomi Valkeinen 63070283e99STomi Valkeinen * - Sink Pad/Stream 63170283e99STomi Valkeinen - Source Pad/Stream 63270283e99STomi Valkeinen - Routing Flags 63370283e99STomi Valkeinen - Comments 63470283e99STomi Valkeinen * - 0/0 63570283e99STomi Valkeinen - 2/0 63670283e99STomi Valkeinen - V4L2_SUBDEV_ROUTE_FL_ACTIVE 63770283e99STomi Valkeinen - Pixel data stream from Sensor A 63870283e99STomi Valkeinen * - 1/0 63970283e99STomi Valkeinen - 2/1 64070283e99STomi Valkeinen - V4L2_SUBDEV_ROUTE_FL_ACTIVE 64170283e99STomi Valkeinen - Pixel data stream from Sensor B 64270283e99STomi Valkeinen 64370283e99STomi Valkeinen.. flat-table:: Receiver routing table 64470283e99STomi Valkeinen :header-rows: 1 64570283e99STomi Valkeinen 64670283e99STomi Valkeinen * - Sink Pad/Stream 64770283e99STomi Valkeinen - Source Pad/Stream 64870283e99STomi Valkeinen - Routing Flags 64970283e99STomi Valkeinen - Comments 65070283e99STomi Valkeinen * - 0/0 65170283e99STomi Valkeinen - 1/0 65270283e99STomi Valkeinen - V4L2_SUBDEV_ROUTE_FL_ACTIVE 65370283e99STomi Valkeinen - Pixel data stream from Sensor A 65470283e99STomi Valkeinen * - 0/1 65570283e99STomi Valkeinen - 2/0 65670283e99STomi Valkeinen - V4L2_SUBDEV_ROUTE_FL_ACTIVE 65770283e99STomi Valkeinen - Pixel data stream from Sensor B 65870283e99STomi Valkeinen 659*58388bd7SSakari Ailus3) Configure formats and selections 66070283e99STomi Valkeinen 661*58388bd7SSakari AilusAfter configuring routing, the next step is configuring the formats and 662*58388bd7SSakari Ailusselections for the streams. This is similar to performing this step without 663*58388bd7SSakari Ailusstreams, with just one exception: the ``stream`` field needs to be assigned 664*58388bd7SSakari Ailusto the value of the stream ID. 66570283e99STomi Valkeinen 66670283e99STomi ValkeinenA common way to accomplish this is to start from the sensors and propagate the 66770283e99STomi Valkeinenconfigurations along the stream towards the receiver, 66870283e99STomi Valkeinenusing :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each 669*58388bd7SSakari Ailusstream endpoint in each sub-device. 670