1.. SPDX-License-Identifier: GPL-2.0
2
3Media Controller devices
4------------------------
5
6Media Controller
7~~~~~~~~~~~~~~~~
8
9The media controller userspace API is documented in
10:ref:`the Media Controller uAPI book <media_controller>`. This document focus
11on the kernel-side implementation of the media framework.
12
13Abstract media device model
14^^^^^^^^^^^^^^^^^^^^^^^^^^^
15
16Discovering a device internal topology, and configuring it at runtime, is one
17of the goals of the media framework. To achieve this, hardware devices are
18modelled as an oriented graph of building blocks called entities connected
19through pads.
20
21An entity is a basic media hardware building block. It can correspond to
22a large variety of logical blocks such as physical hardware devices
23(CMOS sensor for instance), logical hardware devices (a building block
24in a System-on-Chip image processing pipeline), DMA channels or physical
25connectors.
26
27A pad is a connection endpoint through which an entity can interact with
28other entities. Data (not restricted to video) produced by an entity
29flows from the entity's output to one or more entity inputs. Pads should
30not be confused with physical pins at chip boundaries.
31
32A link is a point-to-point oriented connection between two pads, either
33on the same entity or on different entities. Data flows from a source
34pad to a sink pad.
35
36Media device
37^^^^^^^^^^^^
38
39A media device is represented by a struct media_device
40instance, defined in ``include/media/media-device.h``.
41Allocation of the structure is handled by the media device driver, usually by
42embedding the :c:type:`media_device` instance in a larger driver-specific
43structure.
44
45Drivers initialise media device instances by calling
46:c:func:`media_device_init()`. After initialising a media device instance, it is
47registered by calling :c:func:`__media_device_register()` via the macro
48``media_device_register()`` and unregistered by calling
49:c:func:`media_device_unregister()`. An initialised media device must be
50eventually cleaned up by calling :c:func:`media_device_cleanup()`.
51
52Note that it is not allowed to unregister a media device instance that was not
53previously registered, or clean up a media device instance that was not
54previously initialised.
55
56Entities
57^^^^^^^^
58
59Entities are represented by a struct media_entity
60instance, defined in ``include/media/media-entity.h``. The structure is usually
61embedded into a higher-level structure, such as
62:c:type:`v4l2_subdev` or :c:type:`video_device`
63instances, although drivers can allocate entities directly.
64
65Drivers initialize entity pads by calling
66:c:func:`media_entity_pads_init()`.
67
68Drivers register entities with a media device by calling
69:c:func:`media_device_register_entity()`
70and unregistered by calling
71:c:func:`media_device_unregister_entity()`.
72
73Interfaces
74^^^^^^^^^^
75
76Interfaces are represented by a
77struct media_interface instance, defined in
78``include/media/media-entity.h``. Currently, only one type of interface is
79defined: a device node. Such interfaces are represented by a
80struct media_intf_devnode.
81
82Drivers initialize and create device node interfaces by calling
83:c:func:`media_devnode_create()`
84and remove them by calling:
85:c:func:`media_devnode_remove()`.
86
87Pads
88^^^^
89Pads are represented by a struct media_pad instance,
90defined in ``include/media/media-entity.h``. Each entity stores its pads in
91a pads array managed by the entity driver. Drivers usually embed the array in
92a driver-specific structure.
93
94Pads are identified by their entity and their 0-based index in the pads
95array.
96
97Both information are stored in the struct media_pad,
98making the struct media_pad pointer the canonical way
99to store and pass link references.
100
101Pads have flags that describe the pad capabilities and state.
102
103``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data.
104``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data.
105
106.. note::
107
108  One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must
109  be set for each pad.
110
111Links
112^^^^^
113
114Links are represented by a struct media_link instance,
115defined in ``include/media/media-entity.h``. There are two types of links:
116
117**1. pad to pad links**:
118
119Associate two entities via their PADs. Each entity has a list that points
120to all links originating at or targeting any of its pads.
121A given link is thus stored twice, once in the source entity and once in
122the target entity.
123
124Drivers create pad to pad links by calling:
125:c:func:`media_create_pad_link()` and remove with
126:c:func:`media_entity_remove_links()`.
127
128**2. interface to entity links**:
129
130Associate one interface to a Link.
131
132Drivers create interface to entity links by calling:
133:c:func:`media_create_intf_link()` and remove with
134:c:func:`media_remove_intf_links()`.
135
136.. note::
137
138   Links can only be created after having both ends already created.
139
140Links have flags that describe the link capabilities and state. The
141valid values are described at :c:func:`media_create_pad_link()` and
142:c:func:`media_create_intf_link()`.
143
144Graph traversal
145^^^^^^^^^^^^^^^
146
147The media framework provides APIs to iterate over entities in a graph.
148
149To iterate over all entities belonging to a media device, drivers can use
150the media_device_for_each_entity macro, defined in
151``include/media/media-device.h``.
152
153..  code-block:: c
154
155    struct media_entity *entity;
156
157    media_device_for_each_entity(entity, mdev) {
158    // entity will point to each entity in turn
159    ...
160    }
161
162Drivers might also need to iterate over all entities in a graph that can be
163reached only through enabled links starting at a given entity. The media
164framework provides a depth-first graph traversal API for that purpose.
165
166.. note::
167
168   Graphs with cycles (whether directed or undirected) are **NOT**
169   supported by the graph traversal API. To prevent infinite loops, the graph
170   traversal code limits the maximum depth to ``MEDIA_ENTITY_ENUM_MAX_DEPTH``,
171   currently defined as 16.
172
173Drivers initiate a graph traversal by calling
174:c:func:`media_graph_walk_start()`
175
176The graph structure, provided by the caller, is initialized to start graph
177traversal at the given entity.
178
179Drivers can then retrieve the next entity by calling
180:c:func:`media_graph_walk_next()`
181
182When the graph traversal is complete the function will return ``NULL``.
183
184Graph traversal can be interrupted at any moment. No cleanup function call
185is required and the graph structure can be freed normally.
186
187Helper functions can be used to find a link between two given pads, or a pad
188connected to another pad through an enabled link
189(:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`,
190:c:func:`media_entity_remote_source_pad_unique()` and
191:c:func:`media_pad_remote_pad_unique()`).
192
193Use count and power handling
194^^^^^^^^^^^^^^^^^^^^^^^^^^^^
195
196Due to the wide differences between drivers regarding power management
197needs, the media controller does not implement power management. However,
198the struct media_entity includes a ``use_count``
199field that media drivers
200can use to track the number of users of every entity for power management
201needs.
202
203The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by
204media drivers and must not be
205touched by entity drivers. Access to the field must be protected by the
206:c:type:`media_device`.\ ``graph_mutex`` lock.
207
208Links setup
209^^^^^^^^^^^
210
211Link properties can be modified at runtime by calling
212:c:func:`media_entity_setup_link()`.
213
214Pipelines and media streams
215^^^^^^^^^^^^^^^^^^^^^^^^^^^
216
217A media stream is a stream of pixels or metadata originating from one or more
218source devices (such as a sensors) and flowing through media entity pads
219towards the final sinks. The stream can be modified on the route by the
220devices (e.g. scaling or pixel format conversions), or it can be split into
221multiple branches, or multiple branches can be merged.
222
223A media pipeline is a set of media streams which are interdependent. This
224interdependency can be caused by the hardware (e.g. configuration of a second
225stream cannot be changed if the first stream has been enabled) or by the driver
226due to the software design. Most commonly a media pipeline consists of a single
227stream which does not branch.
228
229When starting streaming, drivers must notify all entities in the pipeline to
230prevent link states from being modified during streaming by calling
231:c:func:`media_pipeline_start()`.
232
233The function will mark all the pads which are part of the pipeline as streaming.
234
235The struct media_pipeline instance pointed to by the pipe argument will be
236stored in every pad in the pipeline. Drivers should embed the struct
237media_pipeline in higher-level pipeline structures and can then access the
238pipeline through the struct media_pad pipe field.
239
240Calls to :c:func:`media_pipeline_start()` can be nested.
241The pipeline pointer must be identical for all nested calls to the function.
242
243:c:func:`media_pipeline_start()` may return an error. In that case,
244it will clean up any of the changes it did by itself.
245
246When stopping the stream, drivers must notify the entities with
247:c:func:`media_pipeline_stop()`.
248
249If multiple calls to :c:func:`media_pipeline_start()` have been
250made the same number of :c:func:`media_pipeline_stop()` calls
251are required to stop streaming.
252The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last
253nested stop call.
254
255Link configuration will fail with ``-EBUSY`` by default if either end of the
256link is a streaming entity. Links that can be modified while streaming must
257be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag.
258
259If other operations need to be disallowed on streaming entities (such as
260changing entities configuration parameters) drivers can explicitly check the
261media_entity stream_count field to find out if an entity is streaming. This
262operation must be done with the media_device graph_mutex held.
263
264Link validation
265^^^^^^^^^^^^^^^
266
267Link validation is performed by :c:func:`media_pipeline_start()`
268for any entity which has sink pads in the pipeline. The
269:c:type:`media_entity`.\ ``link_validate()`` callback is used for that
270purpose. In ``link_validate()`` callback, entity driver should check
271that the properties of the source pad of the connected entity and its own
272sink pad match. It is up to the type of the entity (and in the end, the
273properties of the hardware) what matching actually means.
274
275Subsystems should facilitate link validation by providing subsystem specific
276helper functions to provide easy access for commonly needed information, and
277in the end provide a way to use driver-specific callbacks.
278
279Media Controller Device Allocator API
280^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
281
282When the media device belongs to more than one driver, the shared media
283device is allocated with the shared struct device as the key for look ups.
284
285The shared media device should stay in registered state until the last
286driver unregisters it. In addition, the media device should be released when
287all the references are released. Each driver gets a reference to the media
288device during probe, when it allocates the media device. If media device is
289already allocated, the allocate API bumps up the refcount and returns the
290existing media device. The driver puts the reference back in its disconnect
291routine when it calls :c:func:`media_device_delete()`.
292
293The media device is unregistered and cleaned up from the kref put handler to
294ensure that the media device stays in registered state until the last driver
295unregisters the media device.
296
297**Driver Usage**
298
299Drivers should use the appropriate media-core routines to manage the shared
300media device life-time handling the two states:
3011. allocate -> register -> delete
3022. get reference to already registered device -> delete
303
304call :c:func:`media_device_delete()` routine to make sure the shared media
305device delete is handled correctly.
306
307**driver probe:**
308Call :c:func:`media_device_usb_allocate()` to allocate or get a reference
309Call :c:func:`media_device_register()`, if media devnode isn't registered
310
311**driver disconnect:**
312Call :c:func:`media_device_delete()` to free the media_device. Freeing is
313handled by the kref put handler.
314
315API Definitions
316^^^^^^^^^^^^^^^
317
318.. kernel-doc:: include/media/media-device.h
319
320.. kernel-doc:: include/media/media-devnode.h
321
322.. kernel-doc:: include/media/media-entity.h
323
324.. kernel-doc:: include/media/media-request.h
325
326.. kernel-doc:: include/media/media-dev-allocator.h
327