1.. SPDX-License-Identifier: GPL-2.0
2
3Media Controller devices
4------------------------
5
6Media Controller
7~~~~~~~~~~~~~~~~
8
9The media controller userspace API is documented in
10:ref:`the Media Controller uAPI book <media_controller>`. This document focus
11on the kernel-side implementation of the media framework.
12
13Abstract media device model
14^^^^^^^^^^^^^^^^^^^^^^^^^^^
15
16Discovering a device internal topology, and configuring it at runtime, is one
17of the goals of the media framework. To achieve this, hardware devices are
18modelled as an oriented graph of building blocks called entities connected
19through pads.
20
21An entity is a basic media hardware building block. It can correspond to
22a large variety of logical blocks such as physical hardware devices
23(CMOS sensor for instance), logical hardware devices (a building block
24in a System-on-Chip image processing pipeline), DMA channels or physical
25connectors.
26
27A pad is a connection endpoint through which an entity can interact with
28other entities. Data (not restricted to video) produced by an entity
29flows from the entity's output to one or more entity inputs. Pads should
30not be confused with physical pins at chip boundaries.
31
32A link is a point-to-point oriented connection between two pads, either
33on the same entity or on different entities. Data flows from a source
34pad to a sink pad.
35
36Media device
37^^^^^^^^^^^^
38
39A media device is represented by a struct media_device
40instance, defined in ``include/media/media-device.h``.
41Allocation of the structure is handled by the media device driver, usually by
42embedding the :c:type:`media_device` instance in a larger driver-specific
43structure.
44
45Drivers register media device instances by calling
46:c:func:`__media_device_register()` via the macro ``media_device_register()``
47and unregistered by calling :c:func:`media_device_unregister()`.
48
49Entities
50^^^^^^^^
51
52Entities are represented by a struct media_entity
53instance, defined in ``include/media/media-entity.h``. The structure is usually
54embedded into a higher-level structure, such as
55:c:type:`v4l2_subdev` or :c:type:`video_device`
56instances, although drivers can allocate entities directly.
57
58Drivers initialize entity pads by calling
59:c:func:`media_entity_pads_init()`.
60
61Drivers register entities with a media device by calling
62:c:func:`media_device_register_entity()`
63and unregistered by calling
64:c:func:`media_device_unregister_entity()`.
65
66Interfaces
67^^^^^^^^^^
68
69Interfaces are represented by a
70struct media_interface instance, defined in
71``include/media/media-entity.h``. Currently, only one type of interface is
72defined: a device node. Such interfaces are represented by a
73struct media_intf_devnode.
74
75Drivers initialize and create device node interfaces by calling
76:c:func:`media_devnode_create()`
77and remove them by calling:
78:c:func:`media_devnode_remove()`.
79
80Pads
81^^^^
82Pads are represented by a struct media_pad instance,
83defined in ``include/media/media-entity.h``. Each entity stores its pads in
84a pads array managed by the entity driver. Drivers usually embed the array in
85a driver-specific structure.
86
87Pads are identified by their entity and their 0-based index in the pads
88array.
89
90Both information are stored in the struct media_pad,
91making the struct media_pad pointer the canonical way
92to store and pass link references.
93
94Pads have flags that describe the pad capabilities and state.
95
96``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data.
97``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data.
98
99.. note::
100
101  One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must
102  be set for each pad.
103
104Links
105^^^^^
106
107Links are represented by a struct media_link instance,
108defined in ``include/media/media-entity.h``. There are two types of links:
109
110**1. pad to pad links**:
111
112Associate two entities via their PADs. Each entity has a list that points
113to all links originating at or targeting any of its pads.
114A given link is thus stored twice, once in the source entity and once in
115the target entity.
116
117Drivers create pad to pad links by calling:
118:c:func:`media_create_pad_link()` and remove with
119:c:func:`media_entity_remove_links()`.
120
121**2. interface to entity links**:
122
123Associate one interface to a Link.
124
125Drivers create interface to entity links by calling:
126:c:func:`media_create_intf_link()` and remove with
127:c:func:`media_remove_intf_links()`.
128
129.. note::
130
131   Links can only be created after having both ends already created.
132
133Links have flags that describe the link capabilities and state. The
134valid values are described at :c:func:`media_create_pad_link()` and
135:c:func:`media_create_intf_link()`.
136
137Graph traversal
138^^^^^^^^^^^^^^^
139
140The media framework provides APIs to iterate over entities in a graph.
141
142To iterate over all entities belonging to a media device, drivers can use
143the media_device_for_each_entity macro, defined in
144``include/media/media-device.h``.
145
146..  code-block:: c
147
148    struct media_entity *entity;
149
150    media_device_for_each_entity(entity, mdev) {
151    // entity will point to each entity in turn
152    ...
153    }
154
155Drivers might also need to iterate over all entities in a graph that can be
156reached only through enabled links starting at a given entity. The media
157framework provides a depth-first graph traversal API for that purpose.
158
159.. note::
160
161   Graphs with cycles (whether directed or undirected) are **NOT**
162   supported by the graph traversal API. To prevent infinite loops, the graph
163   traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``,
164   currently defined as 16.
165
166Drivers initiate a graph traversal by calling
167:c:func:`media_graph_walk_start()`
168
169The graph structure, provided by the caller, is initialized to start graph
170traversal at the given entity.
171
172Drivers can then retrieve the next entity by calling
173:c:func:`media_graph_walk_next()`
174
175When the graph traversal is complete the function will return ``NULL``.
176
177Graph traversal can be interrupted at any moment. No cleanup function call
178is required and the graph structure can be freed normally.
179
180Helper functions can be used to find a link between two given pads, or a pad
181connected to another pad through an enabled link
182:c:func:`media_entity_find_link()` and
183:c:func:`media_entity_remote_pad()`.
184
185Use count and power handling
186^^^^^^^^^^^^^^^^^^^^^^^^^^^^
187
188Due to the wide differences between drivers regarding power management
189needs, the media controller does not implement power management. However,
190the struct media_entity includes a ``use_count``
191field that media drivers
192can use to track the number of users of every entity for power management
193needs.
194
195The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by
196media drivers and must not be
197touched by entity drivers. Access to the field must be protected by the
198:c:type:`media_device`.\ ``graph_mutex`` lock.
199
200Links setup
201^^^^^^^^^^^
202
203Link properties can be modified at runtime by calling
204:c:func:`media_entity_setup_link()`.
205
206Pipelines and media streams
207^^^^^^^^^^^^^^^^^^^^^^^^^^^
208
209When starting streaming, drivers must notify all entities in the pipeline to
210prevent link states from being modified during streaming by calling
211:c:func:`media_pipeline_start()`.
212
213The function will mark all entities connected to the given entity through
214enabled links, either directly or indirectly, as streaming.
215
216The struct media_pipeline instance pointed to by
217the pipe argument will be stored in every entity in the pipeline.
218Drivers should embed the struct media_pipeline
219in higher-level pipeline structures and can then access the
220pipeline through the struct media_entity
221pipe field.
222
223Calls to :c:func:`media_pipeline_start()` can be nested.
224The pipeline pointer must be identical for all nested calls to the function.
225
226:c:func:`media_pipeline_start()` may return an error. In that case,
227it will clean up any of the changes it did by itself.
228
229When stopping the stream, drivers must notify the entities with
230:c:func:`media_pipeline_stop()`.
231
232If multiple calls to :c:func:`media_pipeline_start()` have been
233made the same number of :c:func:`media_pipeline_stop()` calls
234are required to stop streaming.
235The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last
236nested stop call.
237
238Link configuration will fail with ``-EBUSY`` by default if either end of the
239link is a streaming entity. Links that can be modified while streaming must
240be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag.
241
242If other operations need to be disallowed on streaming entities (such as
243changing entities configuration parameters) drivers can explicitly check the
244media_entity stream_count field to find out if an entity is streaming. This
245operation must be done with the media_device graph_mutex held.
246
247Link validation
248^^^^^^^^^^^^^^^
249
250Link validation is performed by :c:func:`media_pipeline_start()`
251for any entity which has sink pads in the pipeline. The
252:c:type:`media_entity`.\ ``link_validate()`` callback is used for that
253purpose. In ``link_validate()`` callback, entity driver should check
254that the properties of the source pad of the connected entity and its own
255sink pad match. It is up to the type of the entity (and in the end, the
256properties of the hardware) what matching actually means.
257
258Subsystems should facilitate link validation by providing subsystem specific
259helper functions to provide easy access for commonly needed information, and
260in the end provide a way to use driver-specific callbacks.
261
262Media Controller Device Allocator API
263^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
264
265When the media device belongs to more than one driver, the shared media
266device is allocated with the shared struct device as the key for look ups.
267
268The shared media device should stay in registered state until the last
269driver unregisters it. In addition, the media device should be released when
270all the references are released. Each driver gets a reference to the media
271device during probe, when it allocates the media device. If media device is
272already allocated, the allocate API bumps up the refcount and returns the
273existing media device. The driver puts the reference back in its disconnect
274routine when it calls :c:func:`media_device_delete()`.
275
276The media device is unregistered and cleaned up from the kref put handler to
277ensure that the media device stays in registered state until the last driver
278unregisters the media device.
279
280**Driver Usage**
281
282Drivers should use the appropriate media-core routines to manage the shared
283media device life-time handling the two states:
2841. allocate -> register -> delete
2852. get reference to already registered device -> delete
286
287call :c:func:`media_device_delete()` routine to make sure the shared media
288device delete is handled correctly.
289
290**driver probe:**
291Call :c:func:`media_device_usb_allocate()` to allocate or get a reference
292Call :c:func:`media_device_register()`, if media devnode isn't registered
293
294**driver disconnect:**
295Call :c:func:`media_device_delete()` to free the media_device. Freeing is
296handled by the kref put handler.
297
298API Definitions
299^^^^^^^^^^^^^^^
300
301.. kernel-doc:: include/media/media-device.h
302
303.. kernel-doc:: include/media/media-devnode.h
304
305.. kernel-doc:: include/media/media-entity.h
306
307.. kernel-doc:: include/media/media-request.h
308
309.. kernel-doc:: include/media/media-dev-allocator.h
310