1==================================
2DMAengine controller documentation
3==================================
4
5Hardware Introduction
6=====================
7
8Most of the Slave DMA controllers have the same general principles of
9operations.
10
11They have a given number of channels to use for the DMA transfers, and
12a given number of requests lines.
13
14Requests and channels are pretty much orthogonal. Channels can be used
15to serve several to any requests. To simplify, channels are the
16entities that will be doing the copy, and requests what endpoints are
17involved.
18
19The request lines actually correspond to physical lines going from the
20DMA-eligible devices to the controller itself. Whenever the device
21will want to start a transfer, it will assert a DMA request (DRQ) by
22asserting that request line.
23
24A very simple DMA controller would only take into account a single
25parameter: the transfer size. At each clock cycle, it would transfer a
26byte of data from one buffer to another, until the transfer size has
27been reached.
28
29That wouldn't work well in the real world, since slave devices might
30require a specific number of bits to be transferred in a single
31cycle. For example, we may want to transfer as much data as the
32physical bus allows to maximize performances when doing a simple
33memory copy operation, but our audio device could have a narrower FIFO
34that requires data to be written exactly 16 or 24 bits at a time. This
35is why most if not all of the DMA controllers can adjust this, using a
36parameter called the transfer width.
37
38Moreover, some DMA controllers, whenever the RAM is used as a source
39or destination, can group the reads or writes in memory into a buffer,
40so instead of having a lot of small memory accesses, which is not
41really efficient, you'll get several bigger transfers. This is done
42using a parameter called the burst size, that defines how many single
43reads/writes it's allowed to do without the controller splitting the
44transfer into smaller sub-transfers.
45
46Our theoretical DMA controller would then only be able to do transfers
47that involve a single contiguous block of data. However, some of the
48transfers we usually have are not, and want to copy data from
49non-contiguous buffers to a contiguous buffer, which is called
50scatter-gather.
51
52DMAEngine, at least for mem2dev transfers, require support for
53scatter-gather. So we're left with two cases here: either we have a
54quite simple DMA controller that doesn't support it, and we'll have to
55implement it in software, or we have a more advanced DMA controller,
56that implements in hardware scatter-gather.
57
58The latter are usually programmed using a collection of chunks to
59transfer, and whenever the transfer is started, the controller will go
60over that collection, doing whatever we programmed there.
61
62This collection is usually either a table or a linked list. You will
63then push either the address of the table and its number of elements,
64or the first item of the list to one channel of the DMA controller,
65and whenever a DRQ will be asserted, it will go through the collection
66to know where to fetch the data from.
67
68Either way, the format of this collection is completely dependent on
69your hardware. Each DMA controller will require a different structure,
70but all of them will require, for every chunk, at least the source and
71destination addresses, whether it should increment these addresses or
72not and the three parameters we saw earlier: the burst size, the
73transfer width and the transfer size.
74
75The one last thing is that usually, slave devices won't issue DRQ by
76default, and you have to enable this in your slave device driver first
77whenever you're willing to use DMA.
78
79These were just the general memory-to-memory (also called mem2mem) or
80memory-to-device (mem2dev) kind of transfers. Most devices often
81support other kind of transfers or memory operations that dmaengine
82support and will be detailed later in this document.
83
84DMA Support in Linux
85====================
86
87Historically, DMA controller drivers have been implemented using the
88async TX API, to offload operations such as memory copy, XOR,
89cryptography, etc., basically any memory to memory operation.
90
91Over time, the need for memory to device transfers arose, and
92dmaengine was extended. Nowadays, the async TX API is written as a
93layer on top of dmaengine, and acts as a client. Still, dmaengine
94accommodates that API in some cases, and made some design choices to
95ensure that it stayed compatible.
96
97For more information on the Async TX API, please look the relevant
98documentation file in Documentation/crypto/async-tx-api.rst.
99
100DMAEngine APIs
101==============
102
103``struct dma_device`` Initialization
104------------------------------------
105
106Just like any other kernel framework, the whole DMAEngine registration
107relies on the driver filling a structure and registering against the
108framework. In our case, that structure is dma_device.
109
110The first thing you need to do in your driver is to allocate this
111structure. Any of the usual memory allocators will do, but you'll also
112need to initialize a few fields in there:
113
114- ``channels``: should be initialized as a list using the
115  INIT_LIST_HEAD macro for example
116
117- ``src_addr_widths``:
118  should contain a bitmask of the supported source transfer width
119
120- ``dst_addr_widths``:
121  should contain a bitmask of the supported destination transfer width
122
123- ``directions``:
124  should contain a bitmask of the supported slave directions
125  (i.e. excluding mem2mem transfers)
126
127- ``residue_granularity``:
128  granularity of the transfer residue reported to dma_set_residue.
129  This can be either:
130
131  - Descriptor:
132    your device doesn't support any kind of residue
133    reporting. The framework will only know that a particular
134    transaction descriptor is done.
135
136  - Segment:
137    your device is able to report which chunks have been transferred
138
139  - Burst:
140    your device is able to report which burst have been transferred
141
142- ``dev``: should hold the pointer to the ``struct device`` associated
143  to your current driver instance.
144
145Supported transaction types
146---------------------------
147
148The next thing you need is to set which transaction types your device
149(and driver) supports.
150
151Our ``dma_device structure`` has a field called cap_mask that holds the
152various types of transaction supported, and you need to modify this
153mask using the dma_cap_set function, with various flags depending on
154transaction types you support as an argument.
155
156All those capabilities are defined in the ``dma_transaction_type enum``,
157in ``include/linux/dmaengine.h``
158
159Currently, the types available are:
160
161- DMA_MEMCPY
162
163  - The device is able to do memory to memory copies
164
165- DMA_XOR
166
167  - The device is able to perform XOR operations on memory areas
168
169  - Used to accelerate XOR intensive tasks, such as RAID5
170
171- DMA_XOR_VAL
172
173  - The device is able to perform parity check using the XOR
174    algorithm against a memory buffer.
175
176- DMA_PQ
177
178  - The device is able to perform RAID6 P+Q computations, P being a
179    simple XOR, and Q being a Reed-Solomon algorithm.
180
181- DMA_PQ_VAL
182
183  - The device is able to perform parity check using RAID6 P+Q
184    algorithm against a memory buffer.
185
186- DMA_INTERRUPT
187
188  - The device is able to trigger a dummy transfer that will
189    generate periodic interrupts
190
191  - Used by the client drivers to register a callback that will be
192    called on a regular basis through the DMA controller interrupt
193
194- DMA_PRIVATE
195
196  - The devices only supports slave transfers, and as such isn't
197    available for async transfers.
198
199- DMA_ASYNC_TX
200
201  - Must not be set by the device, and will be set by the framework
202    if needed
203
204  - TODO: What is it about?
205
206- DMA_SLAVE
207
208  - The device can handle device to memory transfers, including
209    scatter-gather transfers.
210
211  - While in the mem2mem case we were having two distinct types to
212    deal with a single chunk to copy or a collection of them, here,
213    we just have a single transaction type that is supposed to
214    handle both.
215
216  - If you want to transfer a single contiguous memory buffer,
217    simply build a scatter list with only one item.
218
219- DMA_CYCLIC
220
221  - The device can handle cyclic transfers.
222
223  - A cyclic transfer is a transfer where the chunk collection will
224    loop over itself, with the last item pointing to the first.
225
226  - It's usually used for audio transfers, where you want to operate
227    on a single ring buffer that you will fill with your audio data.
228
229- DMA_INTERLEAVE
230
231  - The device supports interleaved transfer.
232
233  - These transfers can transfer data from a non-contiguous buffer
234    to a non-contiguous buffer, opposed to DMA_SLAVE that can
235    transfer data from a non-contiguous data set to a continuous
236    destination buffer.
237
238  - It's usually used for 2d content transfers, in which case you
239    want to transfer a portion of uncompressed data directly to the
240    display to print it
241
242- DMA_COMPLETION_NO_ORDER
243
244  - The device does not support in order completion.
245
246  - The driver should return DMA_OUT_OF_ORDER for device_tx_status if
247    the device is setting this capability.
248
249  - All cookie tracking and checking API should be treated as invalid if
250    the device exports this capability.
251
252  - At this point, this is incompatible with polling option for dmatest.
253
254  - If this cap is set, the user is recommended to provide an unique
255    identifier for each descriptor sent to the DMA device in order to
256    properly track the completion.
257
258- DMA_REPEAT
259
260  - The device supports repeated transfers. A repeated transfer, indicated by
261    the DMA_PREP_REPEAT transfer flag, is similar to a cyclic transfer in that
262    it gets automatically repeated when it ends, but can additionally be
263    replaced by the client.
264
265  - This feature is limited to interleaved transfers, this flag should thus not
266    be set if the DMA_INTERLEAVE flag isn't set. This limitation is based on
267    the current needs of DMA clients, support for additional transfer types
268    should be added in the future if and when the need arises.
269
270- DMA_LOAD_EOT
271
272  - The device supports replacing repeated transfers at end of transfer (EOT)
273    by queuing a new transfer with the DMA_PREP_LOAD_EOT flag set.
274
275  - Support for replacing a currently running transfer at another point (such
276    as end of burst instead of end of transfer) will be added in the future
277    based on DMA clients needs, if and when the need arises.
278
279These various types will also affect how the source and destination
280addresses change over time.
281
282Addresses pointing to RAM are typically incremented (or decremented)
283after each transfer. In case of a ring buffer, they may loop
284(DMA_CYCLIC). Addresses pointing to a device's register (e.g. a FIFO)
285are typically fixed.
286
287Per descriptor metadata support
288-------------------------------
289Some data movement architecture (DMA controller and peripherals) uses metadata
290associated with a transaction. The DMA controller role is to transfer the
291payload and the metadata alongside.
292The metadata itself is not used by the DMA engine itself, but it contains
293parameters, keys, vectors, etc for peripheral or from the peripheral.
294
295The DMAengine framework provides a generic ways to facilitate the metadata for
296descriptors. Depending on the architecture the DMA driver can implement either
297or both of the methods and it is up to the client driver to choose which one
298to use.
299
300- DESC_METADATA_CLIENT
301
302  The metadata buffer is allocated/provided by the client driver and it is
303  attached (via the dmaengine_desc_attach_metadata() helper to the descriptor.
304
305  From the DMA driver the following is expected for this mode:
306
307  - DMA_MEM_TO_DEV / DEV_MEM_TO_MEM
308
309    The data from the provided metadata buffer should be prepared for the DMA
310    controller to be sent alongside of the payload data. Either by copying to a
311    hardware descriptor, or highly coupled packet.
312
313  - DMA_DEV_TO_MEM
314
315    On transfer completion the DMA driver must copy the metadata to the client
316    provided metadata buffer before notifying the client about the completion.
317    After the transfer completion, DMA drivers must not touch the metadata
318    buffer provided by the client.
319
320- DESC_METADATA_ENGINE
321
322  The metadata buffer is allocated/managed by the DMA driver. The client driver
323  can ask for the pointer, maximum size and the currently used size of the
324  metadata and can directly update or read it. dmaengine_desc_get_metadata_ptr()
325  and dmaengine_desc_set_metadata_len() is provided as helper functions.
326
327  From the DMA driver the following is expected for this mode:
328
329  - get_metadata_ptr()
330
331    Should return a pointer for the metadata buffer, the maximum size of the
332    metadata buffer and the currently used / valid (if any) bytes in the buffer.
333
334  - set_metadata_len()
335
336    It is called by the clients after it have placed the metadata to the buffer
337    to let the DMA driver know the number of valid bytes provided.
338
339  Note: since the client will ask for the metadata pointer in the completion
340  callback (in DMA_DEV_TO_MEM case) the DMA driver must ensure that the
341  descriptor is not freed up prior the callback is called.
342
343Device operations
344-----------------
345
346Our dma_device structure also requires a few function pointers in
347order to implement the actual logic, now that we described what
348operations we were able to perform.
349
350The functions that we have to fill in there, and hence have to
351implement, obviously depend on the transaction types you reported as
352supported.
353
354- ``device_alloc_chan_resources``
355
356- ``device_free_chan_resources``
357
358  - These functions will be called whenever a driver will call
359    ``dma_request_channel`` or ``dma_release_channel`` for the first/last
360    time on the channel associated to that driver.
361
362  - They are in charge of allocating/freeing all the needed
363    resources in order for that channel to be useful for your driver.
364
365  - These functions can sleep.
366
367- ``device_prep_dma_*``
368
369  - These functions are matching the capabilities you registered
370    previously.
371
372  - These functions all take the buffer or the scatterlist relevant
373    for the transfer being prepared, and should create a hardware
374    descriptor or a list of hardware descriptors from it
375
376  - These functions can be called from an interrupt context
377
378  - Any allocation you might do should be using the GFP_NOWAIT
379    flag, in order not to potentially sleep, but without depleting
380    the emergency pool either.
381
382  - Drivers should try to pre-allocate any memory they might need
383    during the transfer setup at probe time to avoid putting to
384    much pressure on the nowait allocator.
385
386  - It should return a unique instance of the
387    ``dma_async_tx_descriptor structure``, that further represents this
388    particular transfer.
389
390  - This structure can be initialized using the function
391    ``dma_async_tx_descriptor_init``.
392
393  - You'll also need to set two fields in this structure:
394
395    - flags:
396      TODO: Can it be modified by the driver itself, or
397      should it be always the flags passed in the arguments
398
399    - tx_submit: A pointer to a function you have to implement,
400      that is supposed to push the current transaction descriptor to a
401      pending queue, waiting for issue_pending to be called.
402
403  - In this structure the function pointer callback_result can be
404    initialized in order for the submitter to be notified that a
405    transaction has completed. In the earlier code the function pointer
406    callback has been used. However it does not provide any status to the
407    transaction and will be deprecated. The result structure defined as
408    ``dmaengine_result`` that is passed in to callback_result
409    has two fields:
410
411    - result: This provides the transfer result defined by
412      ``dmaengine_tx_result``. Either success or some error condition.
413
414    - residue: Provides the residue bytes of the transfer for those that
415      support residue.
416
417- ``device_issue_pending``
418
419  - Takes the first transaction descriptor in the pending queue,
420    and starts the transfer. Whenever that transfer is done, it
421    should move to the next transaction in the list.
422
423  - This function can be called in an interrupt context
424
425- ``device_tx_status``
426
427  - Should report the bytes left to go over on the given channel
428
429  - Should only care about the transaction descriptor passed as
430    argument, not the currently active one on a given channel
431
432  - The tx_state argument might be NULL
433
434  - Should use dma_set_residue to report it
435
436  - In the case of a cyclic transfer, it should only take into
437    account the current period.
438
439  - Should return DMA_OUT_OF_ORDER if the device does not support in order
440    completion and is completing the operation out of order.
441
442  - This function can be called in an interrupt context.
443
444- device_config
445
446  - Reconfigures the channel with the configuration given as argument
447
448  - This command should NOT perform synchronously, or on any
449    currently queued transfers, but only on subsequent ones
450
451  - In this case, the function will receive a ``dma_slave_config``
452    structure pointer as an argument, that will detail which
453    configuration to use.
454
455  - Even though that structure contains a direction field, this
456    field is deprecated in favor of the direction argument given to
457    the prep_* functions
458
459  - This call is mandatory for slave operations only. This should NOT be
460    set or expected to be set for memcpy operations.
461    If a driver support both, it should use this call for slave
462    operations only and not for memcpy ones.
463
464- device_pause
465
466  - Pauses a transfer on the channel
467
468  - This command should operate synchronously on the channel,
469    pausing right away the work of the given channel
470
471- device_resume
472
473  - Resumes a transfer on the channel
474
475  - This command should operate synchronously on the channel,
476    resuming right away the work of the given channel
477
478- device_terminate_all
479
480  - Aborts all the pending and ongoing transfers on the channel
481
482  - For aborted transfers the complete callback should not be called
483
484  - Can be called from atomic context or from within a complete
485    callback of a descriptor. Must not sleep. Drivers must be able
486    to handle this correctly.
487
488  - Termination may be asynchronous. The driver does not have to
489    wait until the currently active transfer has completely stopped.
490    See device_synchronize.
491
492- device_synchronize
493
494  - Must synchronize the termination of a channel to the current
495    context.
496
497  - Must make sure that memory for previously submitted
498    descriptors is no longer accessed by the DMA controller.
499
500  - Must make sure that all complete callbacks for previously
501    submitted descriptors have finished running and none are
502    scheduled to run.
503
504  - May sleep.
505
506
507Misc notes
508==========
509
510(stuff that should be documented, but don't really know
511where to put them)
512
513``dma_run_dependencies``
514
515- Should be called at the end of an async TX transfer, and can be
516  ignored in the slave transfers case.
517
518- Makes sure that dependent operations are run before marking it
519  as complete.
520
521dma_cookie_t
522
523- it's a DMA transaction ID that will increment over time.
524
525- Not really relevant any more since the introduction of ``virt-dma``
526  that abstracts it away.
527
528DMA_CTRL_ACK
529
530- If clear, the descriptor cannot be reused by provider until the
531  client acknowledges receipt, i.e. has a chance to establish any
532  dependency chains
533
534- This can be acked by invoking async_tx_ack()
535
536- If set, does not mean descriptor can be reused
537
538DMA_CTRL_REUSE
539
540- If set, the descriptor can be reused after being completed. It should
541  not be freed by provider if this flag is set.
542
543- The descriptor should be prepared for reuse by invoking
544  ``dmaengine_desc_set_reuse()`` which will set DMA_CTRL_REUSE.
545
546- ``dmaengine_desc_set_reuse()`` will succeed only when channel support
547  reusable descriptor as exhibited by capabilities
548
549- As a consequence, if a device driver wants to skip the
550  ``dma_map_sg()`` and ``dma_unmap_sg()`` in between 2 transfers,
551  because the DMA'd data wasn't used, it can resubmit the transfer right after
552  its completion.
553
554- Descriptor can be freed in few ways
555
556  - Clearing DMA_CTRL_REUSE by invoking
557    ``dmaengine_desc_clear_reuse()`` and submitting for last txn
558
559  - Explicitly invoking ``dmaengine_desc_free()``, this can succeed only
560    when DMA_CTRL_REUSE is already set
561
562  - Terminating the channel
563
564- DMA_PREP_CMD
565
566  - If set, the client driver tells DMA controller that passed data in DMA
567    API is command data.
568
569  - Interpretation of command data is DMA controller specific. It can be
570    used for issuing commands to other peripherals/register reads/register
571    writes for which the descriptor should be in different format from
572    normal data descriptors.
573
574- DMA_PREP_REPEAT
575
576  - If set, the transfer will be automatically repeated when it ends until a
577    new transfer is queued on the same channel with the DMA_PREP_LOAD_EOT flag.
578    If the next transfer to be queued on the channel does not have the
579    DMA_PREP_LOAD_EOT flag set, the current transfer will be repeated until the
580    client terminates all transfers.
581
582  - This flag is only supported if the channel reports the DMA_REPEAT
583    capability.
584
585- DMA_PREP_LOAD_EOT
586
587  - If set, the transfer will replace the transfer currently being executed at
588    the end of the transfer.
589
590  - This is the default behaviour for non-repeated transfers, specifying
591    DMA_PREP_LOAD_EOT for non-repeated transfers will thus make no difference.
592
593  - When using repeated transfers, DMA clients will usually need to set the
594    DMA_PREP_LOAD_EOT flag on all transfers, otherwise the channel will keep
595    repeating the last repeated transfer and ignore the new transfers being
596    queued. Failure to set DMA_PREP_LOAD_EOT will appear as if the channel was
597    stuck on the previous transfer.
598
599  - This flag is only supported if the channel reports the DMA_LOAD_EOT
600    capability.
601
602General Design Notes
603====================
604
605Most of the DMAEngine drivers you'll see are based on a similar design
606that handles the end of transfer interrupts in the handler, but defer
607most work to a tasklet, including the start of a new transfer whenever
608the previous transfer ended.
609
610This is a rather inefficient design though, because the inter-transfer
611latency will be not only the interrupt latency, but also the
612scheduling latency of the tasklet, which will leave the channel idle
613in between, which will slow down the global transfer rate.
614
615You should avoid this kind of practice, and instead of electing a new
616transfer in your tasklet, move that part to the interrupt handler in
617order to have a shorter idle window (that we can't really avoid
618anyway).
619
620Glossary
621========
622
623- Burst: A number of consecutive read or write operations that
624  can be queued to buffers before being flushed to memory.
625
626- Chunk: A contiguous collection of bursts
627
628- Transfer: A collection of chunks (be it contiguous or not)
629