1===============================
2Adjunct Processor (AP) facility
3===============================
4
5
6Introduction
7============
8The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
9of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
10The AP devices provide cryptographic functions to all CPUs assigned to a
11linux system running in an IBM Z system LPAR.
12
13The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
14is to make AP cards available to KVM guests using the VFIO mediated device
15framework. This implementation relies considerably on the s390 virtualization
16facilities which do most of the hard work of providing direct access to AP
17devices.
18
19AP Architectural Overview
20=========================
21To facilitate the comprehension of the design, let's start with some
22definitions:
23
24* AP adapter
25
26  An AP adapter is an IBM Z adapter card that can perform cryptographic
27  functions. There can be from 0 to 256 adapters assigned to an LPAR. Adapters
28  assigned to the LPAR in which a linux host is running will be available to
29  the linux host. Each adapter is identified by a number from 0 to 255; however,
30  the maximum adapter number is determined by machine model and/or adapter type.
31  When installed, an AP adapter is accessed by AP instructions executed by any
32  CPU.
33
34  The AP adapter cards are assigned to a given LPAR via the system's Activation
35  Profile which can be edited via the HMC. When the linux host system is IPL'd
36  in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
37  creates a sysfs device for each assigned adapter. For example, if AP adapters
38  4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
39  sysfs device entries::
40
41    /sys/devices/ap/card04
42    /sys/devices/ap/card0a
43
44  Symbolic links to these devices will also be created in the AP bus devices
45  sub-directory::
46
47    /sys/bus/ap/devices/[card04]
48    /sys/bus/ap/devices/[card04]
49
50* AP domain
51
52  An adapter is partitioned into domains. An adapter can hold up to 256 domains
53  depending upon the adapter type and hardware configuration. A domain is
54  identified by a number from 0 to 255; however, the maximum domain number is
55  determined by machine model and/or adapter type.. A domain can be thought of
56  as a set of hardware registers and memory used for processing AP commands. A
57  domain can be configured with a secure private key used for clear key
58  encryption. A domain is classified in one of two ways depending upon how it
59  may be accessed:
60
61    * Usage domains are domains that are targeted by an AP instruction to
62      process an AP command.
63
64    * Control domains are domains that are changed by an AP command sent to a
65      usage domain; for example, to set the secure private key for the control
66      domain.
67
68  The AP usage and control domains are assigned to a given LPAR via the system's
69  Activation Profile which can be edited via the HMC. When a linux host system
70  is IPL'd in the LPAR, the AP bus module detects the AP usage and control
71  domains assigned to the LPAR. The domain number of each usage domain and
72  adapter number of each AP adapter are combined to create AP queue devices
73  (see AP Queue section below). The domain number of each control domain will be
74  represented in a bitmask and stored in a sysfs file
75  /sys/bus/ap/ap_control_domain_mask. The bits in the mask, from most to least
76  significant bit, correspond to domains 0-255.
77
78* AP Queue
79
80  An AP queue is the means by which an AP command is sent to a usage domain
81  inside a specific adapter. An AP queue is identified by a tuple
82  comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
83  APQI corresponds to a given usage domain number within the adapter. This tuple
84  forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
85  instructions include a field containing the APQN to identify the AP queue to
86  which the AP command is to be sent for processing.
87
88  The AP bus will create a sysfs device for each APQN that can be derived from
89  the cross product of the AP adapter and usage domain numbers detected when the
90  AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
91  domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
92  following sysfs entries::
93
94    /sys/devices/ap/card04/04.0006
95    /sys/devices/ap/card04/04.0047
96    /sys/devices/ap/card0a/0a.0006
97    /sys/devices/ap/card0a/0a.0047
98
99  The following symbolic links to these devices will be created in the AP bus
100  devices subdirectory::
101
102    /sys/bus/ap/devices/[04.0006]
103    /sys/bus/ap/devices/[04.0047]
104    /sys/bus/ap/devices/[0a.0006]
105    /sys/bus/ap/devices/[0a.0047]
106
107* AP Instructions:
108
109  There are three AP instructions:
110
111  * NQAP: to enqueue an AP command-request message to a queue
112  * DQAP: to dequeue an AP command-reply message from a queue
113  * PQAP: to administer the queues
114
115  AP instructions identify the domain that is targeted to process the AP
116  command; this must be one of the usage domains. An AP command may modify a
117  domain that is not one of the usage domains, but the modified domain
118  must be one of the control domains.
119
120AP and SIE
121==========
122Let's now take a look at how AP instructions executed on a guest are interpreted
123by the hardware.
124
125A satellite control block called the Crypto Control Block (CRYCB) is attached to
126our main hardware virtualization control block. The CRYCB contains an AP Control
127Block (APCB) that has three fields to identify the adapters, usage domains and
128control domains assigned to the KVM guest:
129
130* The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
131  to the KVM guest. Each bit in the mask, from left to right, corresponds to
132  an APID from 0-255. If a bit is set, the corresponding adapter is valid for
133  use by the KVM guest.
134
135* The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
136  assigned to the KVM guest. Each bit in the mask, from left to right,
137  corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the
138  corresponding queue is valid for use by the KVM guest.
139
140* The AP Domain Mask field is a bit mask that identifies the AP control domains
141  assigned to the KVM guest. The ADM bit mask controls which domains can be
142  changed by an AP command-request message sent to a usage domain from the
143  guest. Each bit in the mask, from left to right, corresponds to a domain from
144  0-255. If a bit is set, the corresponding domain can be modified by an AP
145  command-request message sent to a usage domain.
146
147If you recall from the description of an AP Queue, AP instructions include
148an APQN to identify the AP queue to which an AP command-request message is to be
149sent (NQAP and PQAP instructions), or from which a command-reply message is to
150be received (DQAP instruction). The validity of an APQN is defined by the matrix
151calculated from the APM and AQM; it is the Cartesian product of all assigned
152adapter numbers (APM) with all assigned queue indexes (AQM). For example, if
153adapters 1 and 2 and usage domains 5 and 6 are assigned to a guest, the APQNs
154(1,5), (1,6), (2,5) and (2,6) will be valid for the guest.
155
156The APQNs can provide secure key functionality - i.e., a private key is stored
157on the adapter card for each of its domains - so each APQN must be assigned to
158at most one guest or to the linux host::
159
160   Example 1: Valid configuration:
161   ------------------------------
162   Guest1: adapters 1,2  domains 5,6
163   Guest2: adapter  1,2  domain 7
164
165   This is valid because both guests have a unique set of APQNs:
166      Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
167      Guest2 has APQNs (1,7), (2,7)
168
169   Example 2: Valid configuration:
170   ------------------------------
171   Guest1: adapters 1,2 domains 5,6
172   Guest2: adapters 3,4 domains 5,6
173
174   This is also valid because both guests have a unique set of APQNs:
175      Guest1 has APQNs (1,5), (1,6), (2,5), (2,6);
176      Guest2 has APQNs (3,5), (3,6), (4,5), (4,6)
177
178   Example 3: Invalid configuration:
179   --------------------------------
180   Guest1: adapters 1,2  domains 5,6
181   Guest2: adapter  1    domains 6,7
182
183   This is an invalid configuration because both guests have access to
184   APQN (1,6).
185
186The Design
187==========
188The design introduces three new objects:
189
1901. AP matrix device
1912. VFIO AP device driver (vfio_ap.ko)
1923. VFIO AP mediated pass-through device
193
194The VFIO AP device driver
195-------------------------
196The VFIO AP (vfio_ap) device driver serves the following purposes:
197
1981. Provides the interfaces to secure APQNs for exclusive use of KVM guests.
199
2002. Sets up the VFIO mediated device interfaces to manage a vfio_ap mediated
201   device and creates the sysfs interfaces for assigning adapters, usage
202   domains, and control domains comprising the matrix for a KVM guest.
203
2043. Configures the APM, AQM and ADM in the APCB contained in the CRYCB referenced
205   by a KVM guest's SIE state description to grant the guest access to a matrix
206   of AP devices
207
208Reserve APQNs for exclusive use of KVM guests
209---------------------------------------------
210The following block diagram illustrates the mechanism by which APQNs are
211reserved::
212
213				+------------------+
214		 7 remove       |                  |
215	   +--------------------> cex4queue driver |
216	   |                    |                  |
217	   |                    +------------------+
218	   |
219	   |
220	   |                    +------------------+          +----------------+
221	   |  5 register driver |                  | 3 create |                |
222	   |   +---------------->   Device core    +---------->  matrix device |
223	   |   |                |                  |          |                |
224	   |   |                +--------^---------+          +----------------+
225	   |   |                         |
226	   |   |                         +-------------------+
227	   |   | +-----------------------------------+       |
228	   |   | |      4 register AP driver         |       | 2 register device
229	   |   | |                                   |       |
230  +--------+---+-v---+                      +--------+-------+-+
231  |                  |                      |                  |
232  |      ap_bus      +--------------------- >  vfio_ap driver  |
233  |                  |       8 probe        |                  |
234  +--------^---------+                      +--^--^------------+
235  6 edit   |                                   |  |
236    apmask |     +-----------------------------+  | 11 mdev create
237    aqmask |     |           1 modprobe           |
238  +--------+-----+---+           +----------------+-+         +----------------+
239  |                  |           |                  |10 create|     mediated   |
240  |      admin       |           | VFIO device core |--------->     matrix     |
241  |                  +           |                  |         |     device     |
242  +------+-+---------+           +--------^---------+         +--------^-------+
243	 | |                              |                            |
244	 | | 9 create vfio_ap-passthrough |                            |
245	 | +------------------------------+                            |
246	 +-------------------------------------------------------------+
247		     12  assign adapter/domain/control domain
248
249The process for reserving an AP queue for use by a KVM guest is:
250
2511. The administrator loads the vfio_ap device driver
2522. The vfio-ap driver during its initialization will register a single 'matrix'
253   device with the device core. This will serve as the parent device for
254   all vfio_ap mediated devices used to configure an AP matrix for a guest.
2553. The /sys/devices/vfio_ap/matrix device is created by the device core
2564. The vfio_ap device driver will register with the AP bus for AP queue devices
257   of type 10 and higher (CEX4 and newer). The driver will provide the vfio_ap
258   driver's probe and remove callback interfaces. Devices older than CEX4 queues
259   are not supported to simplify the implementation by not needlessly
260   complicating the design by supporting older devices that will go out of
261   service in the relatively near future, and for which there are few older
262   systems around on which to test.
2635. The AP bus registers the vfio_ap device driver with the device core
2646. The administrator edits the AP adapter and queue masks to reserve AP queues
265   for use by the vfio_ap device driver.
2667. The AP bus removes the AP queues reserved for the vfio_ap driver from the
267   default zcrypt cex4queue driver.
2688. The AP bus probes the vfio_ap device driver to bind the queues reserved for
269   it.
2709. The administrator creates a passthrough type vfio_ap mediated device to be
271   used by a guest
27210. The administrator assigns the adapters, usage domains and control domains
273    to be exclusively used by a guest.
274
275Set up the VFIO mediated device interfaces
276------------------------------------------
277The VFIO AP device driver utilizes the common interfaces of the VFIO mediated
278device core driver to:
279
280* Register an AP mediated bus driver to add a vfio_ap mediated device to and
281  remove it from a VFIO group.
282* Create and destroy a vfio_ap mediated device
283* Add a vfio_ap mediated device to and remove it from the AP mediated bus driver
284* Add a vfio_ap mediated device to and remove it from an IOMMU group
285
286The following high-level block diagram shows the main components and interfaces
287of the VFIO AP mediated device driver::
288
289   +-------------+
290   |             |
291   | +---------+ | mdev_register_driver() +--------------+
292   | |  Mdev   | +<-----------------------+              |
293   | |  bus    | |                        | vfio_mdev.ko |
294   | | driver  | +----------------------->+              |<-> VFIO user
295   | +---------+ |    probe()/remove()    +--------------+    APIs
296   |             |
297   |  MDEV CORE  |
298   |   MODULE    |
299   |   mdev.ko   |
300   | +---------+ | mdev_register_parent() +--------------+
301   | |Physical | +<-----------------------+              |
302   | | device  | |                        |  vfio_ap.ko  |<-> matrix
303   | |interface| +----------------------->+              |    device
304   | +---------+ |       callback         +--------------+
305   +-------------+
306
307During initialization of the vfio_ap module, the matrix device is registered
308with an 'mdev_parent_ops' structure that provides the sysfs attribute
309structures, mdev functions and callback interfaces for managing the mediated
310matrix device.
311
312* sysfs attribute structures:
313
314  supported_type_groups
315    The VFIO mediated device framework supports creation of user-defined
316    mediated device types. These mediated device types are specified
317    via the 'supported_type_groups' structure when a device is registered
318    with the mediated device framework. The registration process creates the
319    sysfs structures for each mediated device type specified in the
320    'mdev_supported_types' sub-directory of the device being registered. Along
321    with the device type, the sysfs attributes of the mediated device type are
322    provided.
323
324    The VFIO AP device driver will register one mediated device type for
325    passthrough devices:
326
327      /sys/devices/vfio_ap/matrix/mdev_supported_types/vfio_ap-passthrough
328
329    Only the read-only attributes required by the VFIO mdev framework will
330    be provided::
331
332	... name
333	... device_api
334	... available_instances
335	... device_api
336
337    Where:
338
339	* name:
340	    specifies the name of the mediated device type
341	* device_api:
342	    the mediated device type's API
343	* available_instances:
344	    the number of vfio_ap mediated passthrough devices
345	    that can be created
346	* device_api:
347	    specifies the VFIO API
348  mdev_attr_groups
349    This attribute group identifies the user-defined sysfs attributes of the
350    mediated device. When a device is registered with the VFIO mediated device
351    framework, the sysfs attribute files identified in the 'mdev_attr_groups'
352    structure will be created in the vfio_ap mediated device's directory. The
353    sysfs attributes for a vfio_ap mediated device are:
354
355    assign_adapter / unassign_adapter:
356      Write-only attributes for assigning/unassigning an AP adapter to/from the
357      vfio_ap mediated device. To assign/unassign an adapter, the APID of the
358      adapter is echoed into the respective attribute file.
359    assign_domain / unassign_domain:
360      Write-only attributes for assigning/unassigning an AP usage domain to/from
361      the vfio_ap mediated device. To assign/unassign a domain, the domain
362      number of the usage domain is echoed into the respective attribute
363      file.
364    matrix:
365      A read-only file for displaying the APQNs derived from the Cartesian
366      product of the adapter and domain numbers assigned to the vfio_ap mediated
367      device.
368    guest_matrix:
369      A read-only file for displaying the APQNs derived from the Cartesian
370      product of the adapter and domain numbers assigned to the APM and AQM
371      fields respectively of the KVM guest's CRYCB. This may differ from the
372      the APQNs assigned to the vfio_ap mediated device if any APQN does not
373      reference a queue device bound to the vfio_ap device driver (i.e., the
374      queue is not in the host's AP configuration).
375    assign_control_domain / unassign_control_domain:
376      Write-only attributes for assigning/unassigning an AP control domain
377      to/from the vfio_ap mediated device. To assign/unassign a control domain,
378      the ID of the domain to be assigned/unassigned is echoed into the
379      respective attribute file.
380    control_domains:
381      A read-only file for displaying the control domain numbers assigned to the
382      vfio_ap mediated device.
383
384* functions:
385
386  create:
387    allocates the ap_matrix_mdev structure used by the vfio_ap driver to:
388
389    * Store the reference to the KVM structure for the guest using the mdev
390    * Store the AP matrix configuration for the adapters, domains, and control
391      domains assigned via the corresponding sysfs attributes files
392    * Store the AP matrix configuration for the adapters, domains and control
393      domains available to a guest. A guest may not be provided access to APQNs
394      referencing queue devices that do not exist, or are not bound to the
395      vfio_ap device driver.
396
397  remove:
398    deallocates the vfio_ap mediated device's ap_matrix_mdev structure.
399    This will be allowed only if a running guest is not using the mdev.
400
401* callback interfaces
402
403  open_device:
404    The vfio_ap driver uses this callback to register a
405    VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the matrix mdev
406    devices. The open_device callback is invoked by userspace to connect the
407    VFIO iommu group for the matrix mdev device to the MDEV bus. Access to the
408    KVM structure used to configure the KVM guest is provided via this callback.
409    The KVM structure, is used to configure the guest's access to the AP matrix
410    defined via the vfio_ap mediated device's sysfs attribute files.
411
412  close_device:
413    unregisters the VFIO_GROUP_NOTIFY_SET_KVM notifier callback function for the
414    matrix mdev device and deconfigures the guest's AP matrix.
415
416  ioctl:
417    this callback handles the VFIO_DEVICE_GET_INFO and VFIO_DEVICE_RESET ioctls
418    defined by the vfio framework.
419
420Configure the guest's AP resources
421----------------------------------
422Configuring the AP resources for a KVM guest will be performed when the
423VFIO_GROUP_NOTIFY_SET_KVM notifier callback is invoked. The notifier
424function is called when userspace connects to KVM. The guest's AP resources are
425configured via its APCB by:
426
427* Setting the bits in the APM corresponding to the APIDs assigned to the
428  vfio_ap mediated device via its 'assign_adapter' interface.
429* Setting the bits in the AQM corresponding to the domains assigned to the
430  vfio_ap mediated device via its 'assign_domain' interface.
431* Setting the bits in the ADM corresponding to the domain dIDs assigned to the
432  vfio_ap mediated device via its 'assign_control_domains' interface.
433
434The linux device model precludes passing a device through to a KVM guest that
435is not bound to the device driver facilitating its pass-through. Consequently,
436an APQN that does not reference a queue device bound to the vfio_ap device
437driver will not be assigned to a KVM guest's matrix. The AP architecture,
438however, does not provide a means to filter individual APQNs from the guest's
439matrix, so the adapters, domains and control domains assigned to vfio_ap
440mediated device via its sysfs 'assign_adapter', 'assign_domain' and
441'assign_control_domain' interfaces will be filtered before providing the AP
442configuration to a guest:
443
444* The APIDs of the adapters, the APQIs of the domains and the domain numbers of
445  the control domains assigned to the matrix mdev that are not also assigned to
446  the host's AP configuration will be filtered.
447
448* Each APQN derived from the Cartesian product of the APIDs and APQIs assigned
449  to the vfio_ap mdev is examined and if any one of them does not reference a
450  queue device bound to the vfio_ap device driver, the adapter will not be
451  plugged into the guest (i.e., the bit corresponding to its APID will not be
452  set in the APM of the guest's APCB).
453
454The CPU model features for AP
455-----------------------------
456The AP stack relies on the presence of the AP instructions as well as three
457facilities: The AP Facilities Test (APFT) facility; the AP Query
458Configuration Information (QCI) facility; and the AP Queue Interruption Control
459facility. These features/facilities are made available to a KVM guest via the
460following CPU model features:
461
4621. ap: Indicates whether the AP instructions are installed on the guest. This
463   feature will be enabled by KVM only if the AP instructions are installed
464   on the host.
465
4662. apft: Indicates the APFT facility is available on the guest. This facility
467   can be made available to the guest only if it is available on the host (i.e.,
468   facility bit 15 is set).
469
4703. apqci: Indicates the AP QCI facility is available on the guest. This facility
471   can be made available to the guest only if it is available on the host (i.e.,
472   facility bit 12 is set).
473
4744. apqi: Indicates AP Queue Interruption Control faclity is available on the
475   guest. This facility can be made available to the guest only if it is
476   available on the host (i.e., facility bit 65 is set).
477
478Note: If the user chooses to specify a CPU model different than the 'host'
479model to QEMU, the CPU model features and facilities need to be turned on
480explicitly; for example::
481
482     /usr/bin/qemu-system-s390x ... -cpu z13,ap=on,apqci=on,apft=on,apqi=on
483
484A guest can be precluded from using AP features/facilities by turning them off
485explicitly; for example::
486
487     /usr/bin/qemu-system-s390x ... -cpu host,ap=off,apqci=off,apft=off,apqi=off
488
489Note: If the APFT facility is turned off (apft=off) for the guest, the guest
490will not see any AP devices. The zcrypt device drivers on the guest that
491register for type 10 and newer AP devices - i.e., the cex4card and cex4queue
492device drivers - need the APFT facility to ascertain the facilities installed on
493a given AP device. If the APFT facility is not installed on the guest, then no
494adapter or domain devices will get created by the AP bus running on the
495guest because only type 10 and newer devices can be configured for guest use.
496
497Example
498=======
499Let's now provide an example to illustrate how KVM guests may be given
500access to AP facilities. For this example, we will show how to configure
501three guests such that executing the lszcrypt command on the guests would
502look like this:
503
504Guest1
505------
506=========== ===== ============
507CARD.DOMAIN TYPE  MODE
508=========== ===== ============
50905          CEX5C CCA-Coproc
51005.0004     CEX5C CCA-Coproc
51105.00ab     CEX5C CCA-Coproc
51206          CEX5A Accelerator
51306.0004     CEX5A Accelerator
51406.00ab     CEX5A Accelerator
515=========== ===== ============
516
517Guest2
518------
519=========== ===== ============
520CARD.DOMAIN TYPE  MODE
521=========== ===== ============
52205          CEX5C CCA-Coproc
52305.0047     CEX5C CCA-Coproc
52405.00ff     CEX5C CCA-Coproc
525=========== ===== ============
526
527Guest3
528------
529=========== ===== ============
530CARD.DOMAIN TYPE  MODE
531=========== ===== ============
53206          CEX5A Accelerator
53306.0047     CEX5A Accelerator
53406.00ff     CEX5A Accelerator
535=========== ===== ============
536
537These are the steps:
538
5391. Install the vfio_ap module on the linux host. The dependency chain for the
540   vfio_ap module is:
541   * iommu
542   * s390
543   * zcrypt
544   * vfio
545   * vfio_mdev
546   * vfio_mdev_device
547   * KVM
548
549   To build the vfio_ap module, the kernel build must be configured with the
550   following Kconfig elements selected:
551   * IOMMU_SUPPORT
552   * S390
553   * ZCRYPT
554   * VFIO
555   * KVM
556
557   If using make menuconfig select the following to build the vfio_ap module::
558
559     -> Device Drivers
560	-> IOMMU Hardware Support
561	   select S390 AP IOMMU Support
562	-> VFIO Non-Privileged userspace driver framework
563	   -> Mediated device driver frramework
564	      -> VFIO driver for Mediated devices
565     -> I/O subsystem
566	-> VFIO support for AP devices
567
5682. Secure the AP queues to be used by the three guests so that the host can not
569   access them. To secure them, there are two sysfs files that specify
570   bitmasks marking a subset of the APQN range as usable only by the default AP
571   queue device drivers. All remaining APQNs are available for use by
572   any other device driver. The vfio_ap device driver is currently the only
573   non-default device driver. The location of the sysfs files containing the
574   masks are::
575
576     /sys/bus/ap/apmask
577     /sys/bus/ap/aqmask
578
579   The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
580   (APID). Each bit in the mask, from left to right, corresponds to an APID from
581   0-255. If a bit is set, the APID belongs to the subset of APQNs marked as
582   available only to the default AP queue device drivers.
583
584   The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
585   (APQI). Each bit in the mask, from left to right, corresponds to an APQI from
586   0-255. If a bit is set, the APQI belongs to the subset of APQNs marked as
587   available only to the default AP queue device drivers.
588
589   The Cartesian product of the APIDs corresponding to the bits set in the
590   apmask and the APQIs corresponding to the bits set in the aqmask comprise
591   the subset of APQNs that can be used only by the host default device drivers.
592   All other APQNs are available to the non-default device drivers such as the
593   vfio_ap driver.
594
595   Take, for example, the following masks::
596
597      apmask:
598      0x7d00000000000000000000000000000000000000000000000000000000000000
599
600      aqmask:
601      0x8000000000000000000000000000000000000000000000000000000000000000
602
603   The masks indicate:
604
605   * Adapters 1, 2, 3, 4, 5, and 7 are available for use by the host default
606     device drivers.
607
608   * Domain 0 is available for use by the host default device drivers
609
610   * The subset of APQNs available for use only by the default host device
611     drivers are:
612
613     (1,0), (2,0), (3,0), (4.0), (5,0) and (7,0)
614
615   * All other APQNs are available for use by the non-default device drivers.
616
617   The APQN of each AP queue device assigned to the linux host is checked by the
618   AP bus against the set of APQNs derived from the Cartesian product of APIDs
619   and APQIs marked as available to the default AP queue device drivers. If a
620   match is detected,  only the default AP queue device drivers will be probed;
621   otherwise, the vfio_ap device driver will be probed.
622
623   By default, the two masks are set to reserve all APQNs for use by the default
624   AP queue device drivers. There are two ways the default masks can be changed:
625
626   1. The sysfs mask files can be edited by echoing a string into the
627      respective sysfs mask file in one of two formats:
628
629      * An absolute hex string starting with 0x - like "0x12345678" - sets
630	the mask. If the given string is shorter than the mask, it is padded
631	with 0s on the right; for example, specifying a mask value of 0x41 is
632	the same as specifying::
633
634	   0x4100000000000000000000000000000000000000000000000000000000000000
635
636	Keep in mind that the mask reads from left to right, so the mask
637	above identifies device numbers 1 and 7 (01000001).
638
639	If the string is longer than the mask, the operation is terminated with
640	an error (EINVAL).
641
642      * Individual bits in the mask can be switched on and off by specifying
643	each bit number to be switched in a comma separated list. Each bit
644	number string must be prepended with a ('+') or minus ('-') to indicate
645	the corresponding bit is to be switched on ('+') or off ('-'). Some
646	valid values are:
647
648	   - "+0"    switches bit 0 on
649	   - "-13"   switches bit 13 off
650	   - "+0x41" switches bit 65 on
651	   - "-0xff" switches bit 255 off
652
653	The following example:
654
655	      +0,-6,+0x47,-0xf0
656
657	Switches bits 0 and 71 (0x47) on
658
659	Switches bits 6 and 240 (0xf0) off
660
661	Note that the bits not specified in the list remain as they were before
662	the operation.
663
664   2. The masks can also be changed at boot time via parameters on the kernel
665      command line like this:
666
667	 ap.apmask=0xffff ap.aqmask=0x40
668
669	 This would create the following masks::
670
671	    apmask:
672	    0xffff000000000000000000000000000000000000000000000000000000000000
673
674	    aqmask:
675	    0x4000000000000000000000000000000000000000000000000000000000000000
676
677	 Resulting in these two pools::
678
679	    default drivers pool:    adapter 0-15, domain 1
680	    alternate drivers pool:  adapter 16-255, domains 0, 2-255
681
682   **Note:**
683   Changing a mask such that one or more APQNs will be taken from a vfio_ap
684   mediated device (see below) will fail with an error (EBUSY). A message
685   is logged to the kernel ring buffer which can be viewed with the 'dmesg'
686   command. The output identifies each APQN flagged as 'in use' and identifies
687   the vfio_ap mediated device to which it is assigned; for example:
688
689   Userspace may not re-assign queue 05.0054 already assigned to 62177883-f1bb-47f0-914d-32a22e3a8804
690   Userspace may not re-assign queue 04.0054 already assigned to cef03c3c-903d-4ecc-9a83-40694cb8aee4
691
692Securing the APQNs for our example
693----------------------------------
694   To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
695   06.00ab, and 06.00ff for use by the vfio_ap device driver, the corresponding
696   APQNs can be removed from the default masks using either of the following
697   commands::
698
699      echo -5,-6 > /sys/bus/ap/apmask
700
701      echo -4,-0x47,-0xab,-0xff > /sys/bus/ap/aqmask
702
703   Or the masks can be set as follows::
704
705      echo 0xf9ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff \
706      > apmask
707
708      echo 0xf7fffffffffffffffeffffffffffffffffffffffffeffffffffffffffffffffe \
709      > aqmask
710
711   This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
712   06.0047, 06.00ab, and 06.00ff getting bound to the vfio_ap device driver. The
713   sysfs directory for the vfio_ap device driver will now contain symbolic links
714   to the AP queue devices bound to it::
715
716     /sys/bus/ap
717     ... [drivers]
718     ...... [vfio_ap]
719     ......... [05.0004]
720     ......... [05.0047]
721     ......... [05.00ab]
722     ......... [05.00ff]
723     ......... [06.0004]
724     ......... [06.0047]
725     ......... [06.00ab]
726     ......... [06.00ff]
727
728   Keep in mind that only type 10 and newer adapters (i.e., CEX4 and later)
729   can be bound to the vfio_ap device driver. The reason for this is to
730   simplify the implementation by not needlessly complicating the design by
731   supporting older devices that will go out of service in the relatively near
732   future and for which there are few older systems on which to test.
733
734   The administrator, therefore, must take care to secure only AP queues that
735   can be bound to the vfio_ap device driver. The device type for a given AP
736   queue device can be read from the parent card's sysfs directory. For example,
737   to see the hardware type of the queue 05.0004:
738
739     cat /sys/bus/ap/devices/card05/hwtype
740
741   The hwtype must be 10 or higher (CEX4 or newer) in order to be bound to the
742   vfio_ap device driver.
743
7443. Create the mediated devices needed to configure the AP matrixes for the
745   three guests and to provide an interface to the vfio_ap driver for
746   use by the guests::
747
748     /sys/devices/vfio_ap/matrix/
749     --- [mdev_supported_types]
750     ------ [vfio_ap-passthrough] (passthrough vfio_ap mediated device type)
751     --------- create
752     --------- [devices]
753
754   To create the mediated devices for the three guests::
755
756	uuidgen > create
757	uuidgen > create
758	uuidgen > create
759
760	or
761
762	echo $uuid1 > create
763	echo $uuid2 > create
764	echo $uuid3 > create
765
766   This will create three mediated devices in the [devices] subdirectory named
767   after the UUID written to the create attribute file. We call them $uuid1,
768   $uuid2 and $uuid3 and this is the sysfs directory structure after creation::
769
770     /sys/devices/vfio_ap/matrix/
771     --- [mdev_supported_types]
772     ------ [vfio_ap-passthrough]
773     --------- [devices]
774     ------------ [$uuid1]
775     --------------- assign_adapter
776     --------------- assign_control_domain
777     --------------- assign_domain
778     --------------- matrix
779     --------------- unassign_adapter
780     --------------- unassign_control_domain
781     --------------- unassign_domain
782
783     ------------ [$uuid2]
784     --------------- assign_adapter
785     --------------- assign_control_domain
786     --------------- assign_domain
787     --------------- matrix
788     --------------- unassign_adapter
789     ----------------unassign_control_domain
790     ----------------unassign_domain
791
792     ------------ [$uuid3]
793     --------------- assign_adapter
794     --------------- assign_control_domain
795     --------------- assign_domain
796     --------------- matrix
797     --------------- unassign_adapter
798     ----------------unassign_control_domain
799     ----------------unassign_domain
800
801   Note *****: The vfio_ap mdevs do not persist across reboots unless the
802               mdevctl tool is used to create and persist them.
803
8044. The administrator now needs to configure the matrixes for the mediated
805   devices $uuid1 (for Guest1), $uuid2 (for Guest2) and $uuid3 (for Guest3).
806
807   This is how the matrix is configured for Guest1::
808
809      echo 5 > assign_adapter
810      echo 6 > assign_adapter
811      echo 4 > assign_domain
812      echo 0xab > assign_domain
813
814   Control domains can similarly be assigned using the assign_control_domain
815   sysfs file.
816
817   If a mistake is made configuring an adapter, domain or control domain,
818   you can use the unassign_xxx files to unassign the adapter, domain or
819   control domain.
820
821   To display the matrix configuration for Guest1::
822
823	 cat matrix
824
825   To display the matrix that is or will be assigned to Guest1::
826
827	 cat guest_matrix
828
829   This is how the matrix is configured for Guest2::
830
831      echo 5 > assign_adapter
832      echo 0x47 > assign_domain
833      echo 0xff > assign_domain
834
835   This is how the matrix is configured for Guest3::
836
837      echo 6 > assign_adapter
838      echo 0x47 > assign_domain
839      echo 0xff > assign_domain
840
841   In order to successfully assign an adapter:
842
843   * The adapter number specified must represent a value from 0 up to the
844     maximum adapter number configured for the system. If an adapter number
845     higher than the maximum is specified, the operation will terminate with
846     an error (ENODEV).
847
848     Note: The maximum adapter number can be obtained via the sysfs
849	   /sys/bus/ap/ap_max_adapter_id attribute file.
850
851   * Each APQN derived from the Cartesian product of the APID of the adapter
852     being assigned and the APQIs of the domains previously assigned:
853
854     - Must only be available to the vfio_ap device driver as specified in the
855       sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
856       one APQN is reserved for use by the host device driver, the operation
857       will terminate with an error (EADDRNOTAVAIL).
858
859     - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
860       is assigned to another vfio_ap mediated device, the operation will
861       terminate with an error (EBUSY).
862
863     - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
864       sys/bus/ap/aqmask attribute files are being edited or the operation may
865       terminate with an error (EBUSY).
866
867   In order to successfully assign a domain:
868
869   * The domain number specified must represent a value from 0 up to the
870     maximum domain number configured for the system. If a domain number
871     higher than the maximum is specified, the operation will terminate with
872     an error (ENODEV).
873
874     Note: The maximum domain number can be obtained via the sysfs
875	   /sys/bus/ap/ap_max_domain_id attribute file.
876
877    * Each APQN derived from the Cartesian product of the APQI of the domain
878      being assigned and the APIDs of the adapters previously assigned:
879
880     - Must only be available to the vfio_ap device driver as specified in the
881       sysfs /sys/bus/ap/apmask and /sys/bus/ap/aqmask attribute files. If even
882       one APQN is reserved for use by the host device driver, the operation
883       will terminate with an error (EADDRNOTAVAIL).
884
885     - Must NOT be assigned to another vfio_ap mediated device. If even one APQN
886       is assigned to another vfio_ap mediated device, the operation will
887       terminate with an error (EBUSY).
888
889     - Must NOT be assigned while the sysfs /sys/bus/ap/apmask and
890       sys/bus/ap/aqmask attribute files are being edited or the operation may
891       terminate with an error (EBUSY).
892
893   In order to successfully assign a control domain:
894
895   * The domain number specified must represent a value from 0 up to the maximum
896     domain number configured for the system. If a control domain number higher
897     than the maximum is specified, the operation will terminate with an
898     error (ENODEV).
899
9005. Start Guest1::
901
902     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
903	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid1 ...
904
9057. Start Guest2::
906
907     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
908	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid2 ...
909
9107. Start Guest3::
911
912     /usr/bin/qemu-system-s390x ... -cpu host,ap=on,apqci=on,apft=on,apqi=on \
913	-device vfio-ap,sysfsdev=/sys/devices/vfio_ap/matrix/$uuid3 ...
914
915When the guest is shut down, the vfio_ap mediated devices may be removed.
916
917Using our example again, to remove the vfio_ap mediated device $uuid1::
918
919   /sys/devices/vfio_ap/matrix/
920      --- [mdev_supported_types]
921      ------ [vfio_ap-passthrough]
922      --------- [devices]
923      ------------ [$uuid1]
924      --------------- remove
925
926::
927
928   echo 1 > remove
929
930This will remove all of the matrix mdev device's sysfs structures including
931the mdev device itself. To recreate and reconfigure the matrix mdev device,
932all of the steps starting with step 3 will have to be performed again. Note
933that the remove will fail if a guest using the vfio_ap mdev is still running.
934
935It is not necessary to remove a vfio_ap mdev, but one may want to
936remove it if no guest will use it during the remaining lifetime of the linux
937host. If the vfio_ap mdev is removed, one may want to also reconfigure
938the pool of adapters and queues reserved for use by the default drivers.
939
940Hot plug/unplug support:
941========================
942An adapter, domain or control domain may be hot plugged into a running KVM
943guest by assigning it to the vfio_ap mediated device being used by the guest if
944the following conditions are met:
945
946* The adapter, domain or control domain must also be assigned to the host's
947  AP configuration.
948
949* Each APQN derived from the Cartesian product comprised of the APID of the
950  adapter being assigned and the APQIs of the domains assigned must reference a
951  queue device bound to the vfio_ap device driver.
952
953* To hot plug a domain, each APQN derived from the Cartesian product
954  comprised of the APQI of the domain being assigned and the APIDs of the
955  adapters assigned must reference a queue device bound to the vfio_ap device
956  driver.
957
958An adapter, domain or control domain may be hot unplugged from a running KVM
959guest by unassigning it from the vfio_ap mediated device being used by the
960guest.
961
962Over-provisioning of AP queues for a KVM guest:
963===============================================
964Over-provisioning is defined herein as the assignment of adapters or domains to
965a vfio_ap mediated device that do not reference AP devices in the host's AP
966configuration. The idea here is that when the adapter or domain becomes
967available, it will be automatically hot-plugged into the KVM guest using
968the vfio_ap mediated device to which it is assigned as long as each new APQN
969resulting from plugging it in references a queue device bound to the vfio_ap
970device driver.
971
972Limitations
973===========
974Live guest migration is not supported for guests using AP devices without
975intervention by a system administrator. Before a KVM guest can be migrated,
976the vfio_ap mediated device must be removed. Unfortunately, it can not be
977removed manually (i.e., echo 1 > /sys/devices/vfio_ap/matrix/$UUID/remove) while
978the mdev is in use by a KVM guest. If the guest is being emulated by QEMU,
979its mdev can be hot unplugged from the guest in one of two ways:
980
9811. If the KVM guest was started with libvirt, you can hot unplug the mdev via
982   the following commands:
983
984      virsh detach-device <guestname> <path-to-device-xml>
985
986      For example, to hot unplug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 from
987      the guest named 'my-guest':
988
989         virsh detach-device my-guest ~/config/my-guest-hostdev.xml
990
991            The contents of my-guest-hostdev.xml:
992
993.. code-block:: xml
994
995            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
996              <source>
997                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
998              </source>
999            </hostdev>
1000
1001
1002      virsh qemu-monitor-command <guest-name> --hmp "device-del <device-id>"
1003
1004      For example, to hot unplug the vfio_ap mediated device identified on the
1005      qemu command line with 'id=hostdev0' from the guest named 'my-guest':
1006
1007.. code-block:: sh
1008
1009         virsh qemu-monitor-command my-guest --hmp "device_del hostdev0"
1010
10112. A vfio_ap mediated device can be hot unplugged by attaching the qemu monitor
1012   to the guest and using the following qemu monitor command:
1013
1014      (QEMU) device-del id=<device-id>
1015
1016      For example, to hot unplug the vfio_ap mediated device that was specified
1017      on the qemu command line with 'id=hostdev0' when the guest was started:
1018
1019         (QEMU) device-del id=hostdev0
1020
1021After live migration of the KVM guest completes, an AP configuration can be
1022restored to the KVM guest by hot plugging a vfio_ap mediated device on the target
1023system into the guest in one of two ways:
1024
10251. If the KVM guest was started with libvirt, you can hot plug a matrix mediated
1026   device into the guest via the following virsh commands:
1027
1028   virsh attach-device <guestname> <path-to-device-xml>
1029
1030      For example, to hot plug mdev 62177883-f1bb-47f0-914d-32a22e3a8804 into
1031      the guest named 'my-guest':
1032
1033         virsh attach-device my-guest ~/config/my-guest-hostdev.xml
1034
1035            The contents of my-guest-hostdev.xml:
1036
1037.. code-block:: xml
1038
1039            <hostdev mode='subsystem' type='mdev' managed='no' model='vfio-ap'>
1040              <source>
1041                <address uuid='62177883-f1bb-47f0-914d-32a22e3a8804'/>
1042              </source>
1043            </hostdev>
1044
1045
1046   virsh qemu-monitor-command <guest-name> --hmp \
1047   "device_add vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
1048
1049      For example, to hot plug the vfio_ap mediated device
1050      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest named 'my-guest' with
1051      device-id hostdev0:
1052
1053      virsh qemu-monitor-command my-guest --hmp \
1054      "device_add vfio-ap,\
1055      sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
1056      id=hostdev0"
1057
10582. A vfio_ap mediated device can be hot plugged by attaching the qemu monitor
1059   to the guest and using the following qemu monitor command:
1060
1061      (qemu) device_add "vfio-ap,sysfsdev=<path-to-mdev>,id=<device-id>"
1062
1063      For example, to plug the vfio_ap mediated device
1064      62177883-f1bb-47f0-914d-32a22e3a8804 into the guest with the device-id
1065      hostdev0:
1066
1067         (QEMU) device-add "vfio-ap,\
1068         sysfsdev=/sys/devices/vfio_ap/matrix/62177883-f1bb-47f0-914d-32a22e3a8804,\
1069         id=hostdev0"
1070