1.. SPDX-License-Identifier: GPL-2.0 2 3.. include:: <isonum.txt> 4 5========================================= 6Rockchip Image Signal Processor (rkisp1) 7========================================= 8 9Introduction 10============ 11 12This file documents the driver for the Rockchip ISP1 that is part of RK3288 13and RK3399 SoCs. The driver is located under drivers/staging/media/rkisp1 14and uses the Media-Controller API. 15 16Topology 17======== 18.. _rkisp1_topology_graph: 19 20.. kernel-figure:: rkisp1.dot 21 :alt: Diagram of the default media pipeline topology 22 :align: center 23 24 25The driver has 4 video devices: 26 27- rkisp1_mainpath: capture device for retrieving images, usually in higher 28 resolution. 29- rkisp1_selfpath: capture device for retrieving images. 30- rkisp1_stats: a metadata capture device that sends statistics. 31- rkisp1_params: a metadata output device that receives parameters 32 configurations from userspace. 33 34The driver has 3 subdevices: 35 36- rkisp1_resizer_mainpath: used to resize and downsample frames for the 37 mainpath capture device. 38- rkisp1_resizer_selfpath: used to resize and downsample frames for the 39 selfpath capture device. 40- rkisp1_isp: is connected to the sensor and is responsible for all the isp 41 operations. 42 43 44rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes 45------------------------------------------------------------- 46Those are the `mainpath` and `selfpath` capture devices to capture frames. 47Those entities are the DMA engines that write the frames to memory. 48The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded 49stream and it is able to convert it to RGB. The selfpath is not able to 50capture bayer formats. 51The mainpath can capture both bayer and YUV formats but it is not able to 52capture RGB formats. 53Both capture videos support 54the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`. 55 56 57rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes 58---------------------------------------------------------------------------- 59Those are resizer entities for the mainpath and the selfpath. Those entities 60can scale the frames up and down and also change the YUV sampling (for example 61YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad. 62The resizers entities can only operate on YUV:4:2:2 format 63(MEDIA_BUS_FMT_YUYV8_2X8). 64The mainpath capture device supports capturing video in bayer formats. In that 65case the resizer of the mainpath is set to 'bypass' mode - it just forward the 66frame without operating on it. 67 68rkisp1_isp - Image Signal Processing Subdevice Node 69--------------------------------------------------- 70This is the isp entity. It is connected to the sensor on sink pad 0 and 71receives the frames using the CSI-2 protocol. It is responsible of configuring 72the CSI-2 protocol. It has a cropping capability on sink pad 0 that is 73connected to the sensor and on source pad 2 connected to the resizer entities. 74Cropping on sink pad 0 defines the image region from the sensor. 75Cropping on source pad 2 defines the region for the Image Stabilizer (IS). 76 77.. _rkisp1_stats: 78 79rkisp1_stats - Statistics Video Node 80------------------------------------ 81The statistics video node outputs the 3A (auto focus, auto exposure and auto 82white balance) statistics, and also histogram statistics for the frames that 83are being processed by the rkisp1 to userspace applications. 84Using these data, applications can implement algorithms and re-parameterize 85the driver through the rkisp_params node to improve image quality during a 86video stream. 87The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and 88userspace should set 89:ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-rk-isp1-stat-3a>` as the 90dataformat. 91 92.. _rkisp1_params: 93 94rkisp1_params - Parameters Video Node 95------------------------------------- 96The rkisp1_params video node receives a set of parameters from userspace 97to be applied to the hardware during a video stream, allowing userspace 98to dynamically modify values such as black level, cross talk corrections 99and others. 100 101The buffer format is defined by struct :c:type:`rkisp1_params_cfg`, and 102userspace should set 103:ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-rk-isp1-params>` as the 104dataformat. 105 106 107Capturing Video Frames Example 108============================== 109 110In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is 111imx219. 112 113The following commands can be used to capture video from the selfpath video 114node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping 115capabilities possible, (see explanation right below) 116 117.. code-block:: bash 118 119 # set the links 120 "media-ctl" "-d" "platform:rkisp1" "-r" 121 "media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]" 122 "media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]" 123 "media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]" 124 125 # set format for imx219 4-0010:0 126 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]' 127 128 # set format for rkisp1_isp pads: 129 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]' 130 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]' 131 132 # set format for rkisp1_resizer_selfpath pads: 133 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]' 134 "media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]' 135 136 # set format for rkisp1_selfpath: 137 "v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800," 138 "v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P" 139 140 # start streaming: 141 v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10" 142 143 144In the above example the sensor is configured to bayer format: 145`SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the 146same mbus format and dimensions as the sensor, otherwise streaming will fail 147with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`. 148In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`. 149 150The cropping dimensions are automatically propagated to be the format of the 151isp source pad `rkisp1_isp:2`. Another cropping operation is configured on 152the isp source pad: `(0,0)/1500x1100`. 153 154The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format 155`YUYV8_2X8/1500x1100` in order to match the format on the other side of the 156link. In addition a cropping `(300,400)/1400x1000` is configured on it. 157 158The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to 159format `YUYV8_2X8/900x800`. That means that the resizer first crop a window 160of `(300,400)/1400x100` from the received frame and then scales this window 161to dimension `900x800`. 162 163Note that the above example does not uses the stats-params control loop. 164Therefore the capture frames will not go through the 3A algorithms and 165probably won't have a good quality, and can even look dark and greenish. 166 167Configuring Quantization 168======================== 169 170The driver supports limited and full range quantization on YUV formats, 171where limited is the default. 172To switch between one or the other, userspace should use the Colorspace 173Conversion API (CSC) for subdevices on source pad 2 of the 174isp (`rkisp1_isp:2`). The quantization configured on this pad is the 175quantization of the captured video frames on the mainpath and selfpath 176video nodes. 177Note that the resizer and capture entities will always report 178``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full 179range on `rkisp1_isp:2`. So in order to get the configured quantization, 180application should get it from pad `rkisp1_isp:2`. 181 182