1.. SPDX-License-Identifier: GPL-2.0
2
3.. include:: <isonum.txt>
4
5===============================================================
6Intel Image Processing Unit 3 (IPU3) Imaging Unit (ImgU) driver
7===============================================================
8
9Copyright |copy| 2018 Intel Corporation
10
11Introduction
12============
13
14This file documents the Intel IPU3 (3rd generation Image Processing Unit)
15Imaging Unit drivers located under drivers/media/pci/intel/ipu3 (CIO2) as well
16as under drivers/staging/media/ipu3 (ImgU).
17
18The Intel IPU3 found in certain Kaby Lake (as well as certain Sky Lake)
19platforms (U/Y processor lines) is made up of two parts namely the Imaging Unit
20(ImgU) and the CIO2 device (MIPI CSI2 receiver).
21
22The CIO2 device receives the raw Bayer data from the sensors and outputs the
23frames in a format that is specific to the IPU3 (for consumption by the IPU3
24ImgU). The CIO2 driver is available as drivers/media/pci/intel/ipu3/ipu3-cio2*
25and is enabled through the CONFIG_VIDEO_IPU3_CIO2 config option.
26
27The Imaging Unit (ImgU) is responsible for processing images captured
28by the IPU3 CIO2 device. The ImgU driver sources can be found under
29drivers/staging/media/ipu3 directory. The driver is enabled through the
30CONFIG_VIDEO_IPU3_IMGU config option.
31
32The two driver modules are named ipu3_csi2 and ipu3_imgu, respectively.
33
34The drivers has been tested on Kaby Lake platforms (U/Y processor lines).
35
36Both of the drivers implement V4L2, Media Controller and V4L2 sub-device
37interfaces. The IPU3 CIO2 driver supports camera sensors connected to the CIO2
38MIPI CSI-2 interfaces through V4L2 sub-device sensor drivers.
39
40CIO2
41====
42
43The CIO2 is represented as a single V4L2 subdev, which provides a V4L2 subdev
44interface to the user space. There is a video node for each CSI-2 receiver,
45with a single media controller interface for the entire device.
46
47The CIO2 contains four independent capture channel, each with its own MIPI CSI-2
48receiver and DMA engine. Each channel is modelled as a V4L2 sub-device exposed
49to userspace as a V4L2 sub-device node and has two pads:
50
51.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
52
53.. flat-table::
54
55    * - pad
56      - direction
57      - purpose
58
59    * - 0
60      - sink
61      - MIPI CSI-2 input, connected to the sensor subdev
62
63    * - 1
64      - source
65      - Raw video capture, connected to the V4L2 video interface
66
67The V4L2 video interfaces model the DMA engines. They are exposed to userspace
68as V4L2 video device nodes.
69
70Capturing frames in raw Bayer format
71------------------------------------
72
73CIO2 MIPI CSI2 receiver is used to capture frames (in packed raw Bayer format)
74from the raw sensors connected to the CSI2 ports. The captured frames are used
75as input to the ImgU driver.
76
77Image processing using IPU3 ImgU requires tools such as raw2pnm [#f1]_, and
78yavta [#f2]_ due to the following unique requirements and / or features specific
79to IPU3.
80
81-- The IPU3 CSI2 receiver outputs the captured frames from the sensor in packed
82raw Bayer format that is specific to IPU3.
83
84-- Multiple video nodes have to be operated simultaneously.
85
86Let us take the example of ov5670 sensor connected to CSI2 port 0, for a
872592x1944 image capture.
88
89Using the media contorller APIs, the ov5670 sensor is configured to send
90frames in packed raw Bayer format to IPU3 CSI2 receiver.
91
92# This example assumes /dev/media0 as the CIO2 media device
93
94export MDEV=/dev/media0
95
96# and that ov5670 sensor is connected to i2c bus 10 with address 0x36
97
98export SDEV=$(media-ctl -d $MDEV -e "ov5670 10-0036")
99
100# Establish the link for the media devices using media-ctl [#f3]_
101media-ctl -d $MDEV -l "ov5670:0 -> ipu3-csi2 0:0[1]"
102
103# Set the format for the media devices
104media-ctl -d $MDEV -V "ov5670:0 [fmt:SGRBG10/2592x1944]"
105
106media-ctl -d $MDEV -V "ipu3-csi2 0:0 [fmt:SGRBG10/2592x1944]"
107
108media-ctl -d $MDEV -V "ipu3-csi2 0:1 [fmt:SGRBG10/2592x1944]"
109
110Once the media pipeline is configured, desired sensor specific settings
111(such as exposure and gain settings) can be set, using the yavta tool.
112
113e.g
114
115yavta -w 0x009e0903 444 $SDEV
116
117yavta -w 0x009e0913 1024 $SDEV
118
119yavta -w 0x009e0911 2046 $SDEV
120
121Once the desired sensor settings are set, frame captures can be done as below.
122
123e.g
124
125yavta --data-prefix -u -c10 -n5 -I -s2592x1944 --file=/tmp/frame-#.bin \
126      -f IPU3_SGRBG10 $(media-ctl -d $MDEV -e "ipu3-cio2 0")
127
128With the above command, 10 frames are captured at 2592x1944 resolution, with
129sGRBG10 format and output as IPU3_SGRBG10 format.
130
131The captured frames are available as /tmp/frame-#.bin files.
132
133ImgU
134====
135
136The ImgU is represented as two V4L2 subdevs, each of which provides a V4L2
137subdev interface to the user space.
138
139Each V4L2 subdev represents a pipe, which can support a maximum of 2 streams.
140This helps to support advanced camera features like Continuous View Finder (CVF)
141and Snapshot During Video(SDV).
142
143The ImgU contains two independent pipes, each modelled as a V4L2 sub-device
144exposed to userspace as a V4L2 sub-device node.
145
146Each pipe has two sink pads and three source pads for the following purpose:
147
148.. tabularcolumns:: |p{0.8cm}|p{4.0cm}|p{4.0cm}|
149
150.. flat-table::
151
152    * - pad
153      - direction
154      - purpose
155
156    * - 0
157      - sink
158      - Input raw video stream
159
160    * - 1
161      - sink
162      - Processing parameters
163
164    * - 2
165      - source
166      - Output processed video stream
167
168    * - 3
169      - source
170      - Output viewfinder video stream
171
172    * - 4
173      - source
174      - 3A statistics
175
176Each pad is connected to a corresponding V4L2 video interface, exposed to
177userspace as a V4L2 video device node.
178
179Device operation
180----------------
181
182With ImgU, once the input video node ("ipu3-imgu 0/1":0, in
183<entity>:<pad-number> format) is queued with buffer (in packed raw Bayer
184format), ImgU starts processing the buffer and produces the video output in YUV
185format and statistics output on respective output nodes. The driver is expected
186to have buffers ready for all of parameter, output and statistics nodes, when
187input video node is queued with buffer.
188
189At a minimum, all of input, main output, 3A statistics and viewfinder
190video nodes should be enabled for IPU3 to start image processing.
191
192Each ImgU V4L2 subdev has the following set of video nodes.
193
194input, output and viewfinder video nodes
195----------------------------------------
196
197The frames (in packed raw Bayer format specific to the IPU3) received by the
198input video node is processed by the IPU3 Imaging Unit and are output to 2 video
199nodes, with each targeting a different purpose (main output and viewfinder
200output).
201
202Details onand the Bayer format specific to the IPU3 can be found in
203:ref:`v4l2-pix-fmt-ipu3-sbggr10`.
204
205The driver supports V4L2 Video Capture Interface as defined at :ref:`devices`.
206
207Only the multi-planar API is supported. More details can be found at
208:ref:`planar-apis`.
209
210Parameters video node
211---------------------
212
213The parameters video node receives the ImgU algorithm parameters that are used
214to configure how the ImgU algorithms process the image.
215
216Details on processing parameters specific to the IPU3 can be found in
217:ref:`v4l2-meta-fmt-params`.
218
2193A statistics video node
220------------------------
221
2223A statistics video node is used by the ImgU driver to output the 3A (auto
223focus, auto exposure and auto white balance) statistics for the frames that are
224being processed by the ImgU to user space applications. User space applications
225can use this statistics data to compute the desired algorithm parameters for
226the ImgU.
227
228Configuring the Intel IPU3
229==========================
230
231The IPU3 ImgU pipelines can be configured using the Media Controller, defined at
232:ref:`media_controller`.
233
234Running mode and firmware binary selection
235------------------------------------------
236
237ImgU works based on firmware, currently the ImgU firmware support run 2 pipes in
238time-sharing with single input frame data. Each pipe can run at certain mode -
239"VIDEO" or "STILL", "VIDEO" mode is commonly used for video frames capture, and
240"STILL" is used for still frame capture. However, you can also select "VIDEO" to
241capture still frames if you want to capture images with less system load and
242power. For "STILL" mode, ImgU will try to use smaller BDS factor and output
243larger bayer frame for further YUV processing than "VIDEO" mode to get high
244quality images. Besides, "STILL" mode need XNR3 to do noise reduction, hence
245"STILL" mode will need more power and memory bandwidth than "VIDEO" mode. TNR
246will be enabled in "VIDEO" mode and bypassed by "STILL" mode. ImgU is running at
247“VIDEO” mode by default, the user can use v4l2 control V4L2_CID_INTEL_IPU3_MODE
248(currently defined in drivers/staging/media/ipu3/include/intel-ipu3.h) to query
249and set the running mode. For user, there is no difference for buffer queueing
250between the "VIDEO" and "STILL" mode, mandatory input and main output node
251should be enabled and buffers need be queued, the statistics and the view-finder
252queues are optional.
253
254The firmware binary will be selected according to current running mode, such log
255"using binary if_to_osys_striped " or "using binary if_to_osys_primary_striped"
256could be observed if you enable the ImgU dynamic debug, the binary
257if_to_osys_striped is selected for "VIDEO" and the binary
258"if_to_osys_primary_striped" is selected for "STILL".
259
260
261Processing the image in raw Bayer format
262----------------------------------------
263
264Configuring ImgU V4L2 subdev for image processing
265~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
266
267The ImgU V4L2 subdevs have to be configured with media controller APIs to have
268all the video nodes setup correctly.
269
270Let us take "ipu3-imgu 0" subdev as an example.
271
272media-ctl -d $MDEV -r
273
274media-ctl -d $MDEV -l "ipu3-imgu 0 input":0 -> "ipu3-imgu 0":0[1]
275
276media-ctl -d $MDEV -l "ipu3-imgu 0":2 -> "ipu3-imgu 0 output":0[1]
277
278media-ctl -d $MDEV -l "ipu3-imgu 0":3 -> "ipu3-imgu 0 viewfinder":0[1]
279
280media-ctl -d $MDEV -l "ipu3-imgu 0":4 -> "ipu3-imgu 0 3a stat":0[1]
281
282Also the pipe mode of the corresponding V4L2 subdev should be set as desired
283(e.g 0 for video mode or 1 for still mode) through the control id 0x009819a1 as
284below.
285
286yavta -w "0x009819A1 1" /dev/v4l-subdev7
287
288Certain hardware blocks in ImgU pipeline can change the frame resolution by
289cropping or scaling, these hardware blocks include Input Feeder(IF), Bayer Down
290Scaler (BDS) and Geometric Distortion Correction (GDC).
291There is also a block which can change the frame resolution - YUV Scaler, it is
292only applicable to the secondary output.
293
294RAW Bayer frames go through these ImgU pipeline hardware blocks and the final
295processed image output to the DDR memory.
296
297.. kernel-figure::  ipu3_rcb.svg
298   :alt: ipu3 resolution blocks image
299
300   IPU3 resolution change hardware blocks
301
302**Input Feeder**
303
304Input Feeder gets the Bayer frame data from the sensor, it can enable cropping
305of lines and columns from the frame and then store pixels into device's internal
306pixel buffer which are ready to readout by following blocks.
307
308**Bayer Down Scaler**
309
310Bayer Down Scaler is capable of performing image scaling in Bayer domain, the
311downscale factor can be configured from 1X to 1/4X in each axis with
312configuration steps of 0.03125 (1/32).
313
314**Geometric Distortion Correction**
315
316Geometric Distortion Correction is used to performe correction of distortions
317and image filtering. It needs some extra filter and envelop padding pixels to
318work, so the input resolution of GDC should be larger than the output
319resolution.
320
321**YUV Scaler**
322
323YUV Scaler which similar with BDS, but it is mainly do image down scaling in
324YUV domain, it can support up to 1/12X down scaling, but it can not be applied
325to the main output.
326
327The ImgU V4L2 subdev has to be configured with the supported resolutions in all
328the above hardware blocks, for a given input resolution.
329For a given supported resolution for an input frame, the Input Feeder, Bayer
330Down Scaler and GDC blocks should be configured with the supported resolutions
331as each hardware block has its own alignment requirement.
332
333You must configure the output resolution of the hardware blocks smartly to meet
334the hardware requirement along with keeping the maximum field of view. The
335intermediate resolutions can be generated by specific tool -
336
337https://github.com/intel/intel-ipu3-pipecfg
338
339This tool can be used to generate intermediate resolutions. More information can
340be obtained by looking at the following IPU3 ImgU configuration table.
341
342https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master
343
344Under baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss
345directory, graph_settings_ov5670.xml can be used as an example.
346
347The following steps prepare the ImgU pipeline for the image processing.
348
3491. The ImgU V4L2 subdev data format should be set by using the
350VIDIOC_SUBDEV_S_FMT on pad 0, using the GDC width and height obtained above.
351
3522. The ImgU V4L2 subdev cropping should be set by using the
353VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_CROP as the target,
354using the input feeder height and width.
355
3563. The ImgU V4L2 subdev composing should be set by using the
357VIDIOC_SUBDEV_S_SELECTION on pad 0, with V4L2_SEL_TGT_COMPOSE as the target,
358using the BDS height and width.
359
360For the ov5670 example, for an input frame with a resolution of 2592x1944
361(which is input to the ImgU subdev pad 0), the corresponding resolutions
362for input feeder, BDS and GDC are 2592x1944, 2592x1944 and 2560x1920
363respectively.
364
365Once this is done, the received raw Bayer frames can be input to the ImgU
366V4L2 subdev as below, using the open source application v4l2n [#f1]_.
367
368For an image captured with 2592x1944 [#f4]_ resolution, with desired output
369resolution as 2560x1920 and viewfinder resolution as 2560x1920, the following
370v4l2n command can be used. This helps process the raw Bayer frames and produces
371the desired results for the main output image and the viewfinder output, in NV12
372format.
373
374v4l2n --pipe=4 --load=/tmp/frame-#.bin --open=/dev/video4
375--fmt=type:VIDEO_OUTPUT_MPLANE,width=2592,height=1944,pixelformat=0X47337069
376--reqbufs=type:VIDEO_OUTPUT_MPLANE,count:1 --pipe=1 --output=/tmp/frames.out
377--open=/dev/video5
378--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
379--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=2 --output=/tmp/frames.vf
380--open=/dev/video6
381--fmt=type:VIDEO_CAPTURE_MPLANE,width=2560,height=1920,pixelformat=NV12
382--reqbufs=type:VIDEO_CAPTURE_MPLANE,count:1 --pipe=3 --open=/dev/video7
383--output=/tmp/frames.3A --fmt=type:META_CAPTURE,?
384--reqbufs=count:1,type:META_CAPTURE --pipe=1,2,3,4 --stream=5
385
386You can also use yavta [#f2]_ command to do same thing as above:
387
388.. code-block:: none
389
390	yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
391	--file=frame-#.out-f NV12 /dev/video5 & \
392	yavta --data-prefix -Bcapture-mplane -c10 -n5 -I -s2592x1944 \
393	--file=frame-#.vf -f NV12 /dev/video6 & \
394	yavta --data-prefix -Bmeta-capture -c10 -n5 -I \
395	--file=frame-#.3a /dev/video7 & \
396	yavta --data-prefix -Boutput-mplane -c10 -n5 -I -s2592x1944 \
397	--file=/tmp/frame-in.cio2 -f IPU3_SGRBG10 /dev/video4
398
399where /dev/video4, /dev/video5, /dev/video6 and /dev/video7 devices point to
400input, output, viewfinder and 3A statistics video nodes respectively.
401
402Converting the raw Bayer image into YUV domain
403----------------------------------------------
404
405The processed images after the above step, can be converted to YUV domain
406as below.
407
408Main output frames
409~~~~~~~~~~~~~~~~~~
410
411raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.out /tmp/frames.out.ppm
412
413where 2560x1920 is output resolution, NV12 is the video format, followed
414by input frame and output PNM file.
415
416Viewfinder output frames
417~~~~~~~~~~~~~~~~~~~~~~~~
418
419raw2pnm -x2560 -y1920 -fNV12 /tmp/frames.vf /tmp/frames.vf.ppm
420
421where 2560x1920 is output resolution, NV12 is the video format, followed
422by input frame and output PNM file.
423
424Example user space code for IPU3
425================================
426
427User space code that configures and uses IPU3 is available here.
428
429https://chromium.googlesource.com/chromiumos/platform/arc-camera/+/master/
430
431The source can be located under hal/intel directory.
432
433Overview of IPU3 pipeline
434=========================
435
436IPU3 pipeline has a number of image processing stages, each of which takes a
437set of parameters as input. The major stages of pipelines are shown here:
438
439.. kernel-render:: DOT
440   :alt: IPU3 ImgU Pipeline
441   :caption: IPU3 ImgU Pipeline Diagram
442
443   digraph "IPU3 ImgU" {
444       node [shape=box]
445       splines="ortho"
446       rankdir="LR"
447
448       a [label="Raw pixels"]
449       b [label="Bayer Downscaling"]
450       c [label="Optical Black Correction"]
451       d [label="Linearization"]
452       e [label="Lens Shading Correction"]
453       f [label="White Balance / Exposure / Focus Apply"]
454       g [label="Bayer Noise Reduction"]
455       h [label="ANR"]
456       i [label="Demosaicing"]
457       j [label="Color Correction Matrix"]
458       k [label="Gamma correction"]
459       l [label="Color Space Conversion"]
460       m [label="Chroma Down Scaling"]
461       n [label="Chromatic Noise Reduction"]
462       o [label="Total Color Correction"]
463       p [label="XNR3"]
464       q [label="TNR"]
465       r [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
466       s [label="YUV Downscaling"]
467       t [label="DDR", style=filled, fillcolor=yellow, shape=cylinder]
468
469       { rank=same; a -> b -> c -> d -> e -> f -> g -> h -> i }
470       { rank=same; j -> k -> l -> m -> n -> o -> p -> q -> s -> t}
471
472       a -> j [style=invis, weight=10]
473       i -> j
474       q -> r
475   }
476
477The table below presents a description of the above algorithms.
478
479======================== =======================================================
480Name			 Description
481======================== =======================================================
482Optical Black Correction Optical Black Correction block subtracts a pre-defined
483			 value from the respective pixel values to obtain better
484			 image quality.
485			 Defined in :c:type:`ipu3_uapi_obgrid_param`.
486Linearization		 This algo block uses linearization parameters to
487			 address non-linearity sensor effects. The Lookup table
488			 table is defined in
489			 :c:type:`ipu3_uapi_isp_lin_vmem_params`.
490SHD			 Lens shading correction is used to correct spatial
491			 non-uniformity of the pixel response due to optical
492			 lens shading. This is done by applying a different gain
493			 for each pixel. The gain, black level etc are
494			 configured in :c:type:`ipu3_uapi_shd_config_static`.
495BNR			 Bayer noise reduction block removes image noise by
496			 applying a bilateral filter.
497			 See :c:type:`ipu3_uapi_bnr_static_config` for details.
498ANR			 Advanced Noise Reduction is a block based algorithm
499			 that performs noise reduction in the Bayer domain. The
500			 convolution matrix etc can be found in
501			 :c:type:`ipu3_uapi_anr_config`.
502DM			 Demosaicing converts raw sensor data in Bayer format
503			 into RGB (Red, Green, Blue) presentation. Then add
504			 outputs of estimation of Y channel for following stream
505			 processing by Firmware. The struct is defined as
506			 :c:type:`ipu3_uapi_dm_config`.
507Color Correction	 Color Correction algo transforms sensor specific color
508			 space to the standard "sRGB" color space. This is done
509			 by applying 3x3 matrix defined in
510			 :c:type:`ipu3_uapi_ccm_mat_config`.
511Gamma correction	 Gamma correction :c:type:`ipu3_uapi_gamma_config` is a
512			 basic non-linear tone mapping correction that is
513			 applied per pixel for each pixel component.
514CSC			 Color space conversion transforms each pixel from the
515			 RGB primary presentation to YUV (Y: brightness,
516			 UV: Luminance) presentation. This is done by applying
517			 a 3x3 matrix defined in
518			 :c:type:`ipu3_uapi_csc_mat_config`
519CDS			 Chroma down sampling
520			 After the CSC is performed, the Chroma Down Sampling
521			 is applied for a UV plane down sampling by a factor
522			 of 2 in each direction for YUV 4:2:0 using a 4x2
523			 configurable filter :c:type:`ipu3_uapi_cds_params`.
524CHNR			 Chroma noise reduction
525			 This block processes only the chrominance pixels and
526			 performs noise reduction by cleaning the high
527			 frequency noise.
528			 See struct :c:type:`ipu3_uapi_yuvp1_chnr_config`.
529TCC			 Total color correction as defined in struct
530			 :c:type:`ipu3_uapi_yuvp2_tcc_static_config`.
531XNR3			 eXtreme Noise Reduction V3 is the third revision of
532			 noise reduction algorithm used to improve image
533			 quality. This removes the low frequency noise in the
534			 captured image. Two related structs are  being defined,
535			 :c:type:`ipu3_uapi_isp_xnr3_params` for ISP data memory
536			 and :c:type:`ipu3_uapi_isp_xnr3_vmem_params` for vector
537			 memory.
538TNR			 Temporal Noise Reduction block compares successive
539			 frames in time to remove anomalies / noise in pixel
540			 values. :c:type:`ipu3_uapi_isp_tnr3_vmem_params` and
541			 :c:type:`ipu3_uapi_isp_tnr3_params` are defined for ISP
542			 vector and data memory respectively.
543======================== =======================================================
544
545Other often encountered acronyms not listed in above table:
546
547	ACC
548		Accelerator cluster
549	AWB_FR
550		Auto white balance filter response statistics
551	BDS
552		Bayer downscaler parameters
553	CCM
554		Color correction matrix coefficients
555	IEFd
556		Image enhancement filter directed
557	Obgrid
558		Optical black level compensation
559	OSYS
560		Output system configuration
561	ROI
562		Region of interest
563	YDS
564		Y down sampling
565	YTM
566		Y-tone mapping
567
568A few stages of the pipeline will be executed by firmware running on the ISP
569processor, while many others will use a set of fixed hardware blocks also
570called accelerator cluster (ACC) to crunch pixel data and produce statistics.
571
572ACC parameters of individual algorithms, as defined by
573:c:type:`ipu3_uapi_acc_param`, can be chosen to be applied by the user
574space through struct :c:type:`ipu3_uapi_flags` embedded in
575:c:type:`ipu3_uapi_params` structure. For parameters that are configured as
576not enabled by the user space, the corresponding structs are ignored by the
577driver, in which case the existing configuration of the algorithm will be
578preserved.
579
580References
581==========
582
583.. [#f5] drivers/staging/media/ipu3/include/intel-ipu3.h
584
585.. [#f1] https://github.com/intel/nvt
586
587.. [#f2] http://git.ideasonboard.org/yavta.git
588
589.. [#f3] http://git.ideasonboard.org/?p=media-ctl.git;a=summary
590
591.. [#f4] ImgU limitation requires an additional 16x16 for all input resolutions
592