1.. SPDX-License-Identifier: GPL-2.0
2
3.. include:: <isonum.txt>
4
5=========================================
6Rockchip Image Signal Processor (rkisp1)
7=========================================
8
9Introduction
10============
11
12This file documents the driver for the Rockchip ISP1 that is part of RK3288
13and RK3399 SoCs. The driver is located under drivers/staging/media/rkisp1
14and uses the Media-Controller API.
15
16Revisions
17=========
18
19There exist multiple smaller revisions to this ISP that got introduced in
20later SoCs. Revisions can be found in the enum :c:type:`rkisp1_cif_isp_version`
21in the UAPI and the revision of the ISP inside the running SoC can be read
22in the field hw_revision of struct media_device_info as returned by
23ioctl MEDIA_IOC_DEVICE_INFO.
24
25Versions in use are:
26
27- RKISP1_V10: used at least in rk3288 and rk3399
28- RKISP1_V11: declared in the original vendor code, but not used
29- RKISP1_V12: used at least in rk3326 and px30
30- RKISP1_V13: used at least in rk1808
31
32Topology
33========
34.. _rkisp1_topology_graph:
35
36.. kernel-figure:: rkisp1.dot
37    :alt:   Diagram of the default media pipeline topology
38    :align: center
39
40
41The driver has 4 video devices:
42
43- rkisp1_mainpath: capture device for retrieving images, usually in higher
44  resolution.
45- rkisp1_selfpath: capture device for retrieving images.
46- rkisp1_stats: a metadata capture device that sends statistics.
47- rkisp1_params: a metadata output device that receives parameters
48  configurations from userspace.
49
50The driver has 3 subdevices:
51
52- rkisp1_resizer_mainpath: used to resize and downsample frames for the
53  mainpath capture device.
54- rkisp1_resizer_selfpath: used to resize and downsample frames for the
55  selfpath capture device.
56- rkisp1_isp: is connected to the sensor and is responsible for all the isp
57  operations.
58
59
60rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes
61-------------------------------------------------------------
62Those are the `mainpath` and `selfpath` capture devices to capture frames.
63Those entities are the DMA engines that write the frames to memory.
64The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded
65stream and it is able to convert it to RGB. The selfpath is not able to
66capture bayer formats.
67The mainpath can capture both bayer and YUV formats but it is not able to
68capture RGB formats.
69Both capture videos support
70the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`.
71
72
73rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes
74----------------------------------------------------------------------------
75Those are resizer entities for the mainpath and the selfpath. Those entities
76can scale the frames up and down and also change the YUV sampling (for example
77YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad.
78The resizers entities can only operate on YUV:4:2:2 format
79(MEDIA_BUS_FMT_YUYV8_2X8).
80The mainpath capture device supports capturing video in bayer formats. In that
81case the resizer of the mainpath is set to 'bypass' mode - it just forward the
82frame without operating on it.
83
84rkisp1_isp - Image Signal Processing Subdevice Node
85---------------------------------------------------
86This is the isp entity. It is connected to the sensor on sink pad 0 and
87receives the frames using the CSI-2 protocol. It is responsible of configuring
88the CSI-2 protocol. It has a cropping capability on sink pad 0 that is
89connected to the sensor and on source pad 2 connected to the resizer entities.
90Cropping on sink pad 0 defines the image region from the sensor.
91Cropping on source pad 2 defines the region for the Image Stabilizer (IS).
92
93.. _rkisp1_stats:
94
95rkisp1_stats - Statistics Video Node
96------------------------------------
97The statistics video node outputs the 3A (auto focus, auto exposure and auto
98white balance) statistics, and also histogram statistics for the frames that
99are being processed by the rkisp1 to userspace applications.
100Using these data, applications can implement algorithms and re-parameterize
101the driver through the rkisp_params node to improve image quality during a
102video stream.
103The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and
104userspace should set
105:ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-rk-isp1-stat-3a>` as the
106dataformat.
107
108.. _rkisp1_params:
109
110rkisp1_params - Parameters Video Node
111-------------------------------------
112The rkisp1_params video node receives a set of parameters from userspace
113to be applied to the hardware during a video stream, allowing userspace
114to dynamically modify values such as black level, cross talk corrections
115and others.
116
117The buffer format is defined by struct :c:type:`rkisp1_params_cfg`, and
118userspace should set
119:ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-rk-isp1-params>` as the
120dataformat.
121
122
123Capturing Video Frames Example
124==============================
125
126In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is
127imx219.
128
129The following commands can be used to capture video from the selfpath video
130node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping
131capabilities possible, (see explanation right below)
132
133.. code-block:: bash
134
135	# set the links
136	"media-ctl" "-d" "platform:rkisp1" "-r"
137	"media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]"
138	"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]"
139	"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]"
140
141	# set format for imx219 4-0010:0
142	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]'
143
144	# set format for rkisp1_isp pads:
145	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]'
146	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]'
147
148	# set format for rkisp1_resizer_selfpath pads:
149	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]'
150	"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]'
151
152	# set format for rkisp1_selfpath:
153	"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800,"
154	"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P"
155
156	# start streaming:
157	v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10"
158
159
160In the above example the sensor is configured to bayer format:
161`SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the
162same mbus format and dimensions as the sensor, otherwise streaming will fail
163with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`.
164In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`.
165
166The cropping dimensions are automatically propagated to be the format of the
167isp source pad `rkisp1_isp:2`. Another cropping operation is configured on
168the isp source pad: `(0,0)/1500x1100`.
169
170The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format
171`YUYV8_2X8/1500x1100` in order to match the format on the other side of the
172link. In addition a cropping `(300,400)/1400x1000` is configured on it.
173
174The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to
175format `YUYV8_2X8/900x800`. That means that the resizer first crop a window
176of `(300,400)/1400x100` from the received frame and then scales this window
177to dimension `900x800`.
178
179Note that the above example does not uses the stats-params control loop.
180Therefore the capture frames will not go through the 3A algorithms and
181probably won't have a good quality, and can even look dark and greenish.
182
183Configuring Quantization
184========================
185
186The driver supports limited and full range quantization on YUV formats,
187where limited is the default.
188To switch between one or the other, userspace should use the Colorspace
189Conversion API (CSC) for subdevices on source pad 2 of the
190isp (`rkisp1_isp:2`). The quantization configured on this pad is the
191quantization of the captured video frames on the mainpath and selfpath
192video nodes.
193Note that the resizer and capture entities will always report
194``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full
195range on `rkisp1_isp:2`. So in order to get the configured quantization,
196application should get it from pad `rkisp1_isp:2`.
197
198