arch/s390/vfio-ap.rst

2 Adjunct Processor (AP) facility
8 The Adjunct Processor (AP) facility is an IBM Z cryptographic facility comprised
9 of three AP instructions and from 1 up to 256 PCIe cryptographic adapter cards.
10 The AP devices provide cryptographic functions to all CPUs assigned to a
13 The AP adapter cards are exposed via the AP bus. The motivation for vfio-ap
14 is to make AP cards available to KVM guests using the VFIO mediated device
16 facilities which do most of the hard work of providing direct access to AP
19 AP Architectural Overview
24 * AP adapter
26   An AP adapter is an IBM Z adapter card that can perform cryptographic
31   When installed, an AP adapter is accessed by AP instructions executed by any
34   The AP adapter cards are assigned to a given LPAR via the system's Activation
36   in the LPAR, the AP bus detects the AP adapter cards assigned to the LPAR and
37   creates a sysfs device for each assigned adapter. For example, if AP adapters
38   4 and 10 (0x0a) are assigned to the LPAR, the AP bus will create the following
44   Symbolic links to these devices will also be created in the AP bus devices
50 * AP domain
56   as a set of hardware registers and memory used for processing AP commands. A
61     * Usage domains are domains that are targeted by an AP instruction to
62       process an AP command.
64     * Control domains are domains that are changed by an AP command sent to a
68   The AP usage and control domains are assigned to a given LPAR via the system's
70   is IPL'd in the LPAR, the AP bus module detects the AP usage and control
72   adapter number of each AP adapter are combined to create AP queue devices
73   (see AP Queue section below). The domain number of each control domain will be
78 * AP Queue
80   An AP queue is the means by which an AP command is sent to a usage domain
81   inside a specific adapter. An AP queue is identified by a tuple
82   comprised of an AP adapter ID (APID) and an AP queue index (APQI). The
84   forms an AP Queue Number (APQN) uniquely identifying an AP queue. AP
85   instructions include a field containing the APQN to identify the AP queue to
86   which the AP command is to be sent for processing.
88   The AP bus will create a sysfs device for each APQN that can be derived from
89   the cross product of the AP adapter and usage domain numbers detected when the
90   AP bus module is loaded. For example, if adapters 4 and 10 (0x0a) and usage
91   domains 6 and 71 (0x47) are assigned to the LPAR, the AP bus will create the
99   The following symbolic links to these devices will be created in the AP bus
107 * AP Instructions:
109   There are three AP instructions:
111   * NQAP: to enqueue an AP command-request message to a queue
112   * DQAP: to dequeue an AP command-reply message from a queue
115   AP instructions identify the domain that is targeted to process the AP
116   command; this must be one of the usage domains. An AP command may modify a
120 AP and SIE
122 Let's now take a look at how AP instructions executed on a guest are interpreted
126 our main hardware virtualization control block. The CRYCB contains an AP Control
130 * The AP Mask (APM) field is a bit mask that identifies the AP adapters assigned
135 * The AP Queue Mask (AQM) field is a bit mask identifying the AP usage domains
137   corresponds to an AP queue index (APQI) from 0-255. If a bit is set, the
140 * The AP Domain Mask field is a bit mask that identifies the AP control domains
142   changed by an AP command-request message sent to a usage domain from the
144   0-255. If a bit is set, the corresponding domain can be modified by an AP
147 If you recall from the description of an AP Queue, AP instructions include
148 an APQN to identify the AP queue to which an AP command-request message is to be
190 1. AP matrix device
191 2. VFIO AP device driver (vfio_ap.ko)
192 3. VFIO AP mediated pass-through device
194 The VFIO AP device driver
196 The VFIO AP (vfio_ap) device driver serves the following purposes:
206    of AP devices
228 	   |   | |      4 register AP driver         |       | 2 register device
249 The process for reserving an AP queue for use by a KVM guest is:
254    all vfio_ap mediated devices used to configure an AP matrix for a guest.
256 4. The vfio_ap device driver will register with the AP bus for AP queue devices
263 5. The AP bus registers the vfio_ap device driver with the device core
264 6. The administrator edits the AP adapter and queue masks to reserve AP queues
266 7. The AP bus removes the AP queues reserved for the vfio_ap driver from the
268 8. The AP bus probes the vfio_ap device driver to bind the queues reserved for
277 The VFIO AP device driver utilizes the common interfaces of the VFIO mediated
280 * Register an AP mediated bus driver to add a vfio_ap mediated device to and
283 * Add a vfio_ap mediated device to and remove it from the AP mediated bus driver
287 of the VFIO AP mediated device driver::
324     The VFIO AP device driver will register one mediated device type for
356       Write-only attributes for assigning/unassigning an AP adapter to/from the
360       Write-only attributes for assigning/unassigning an AP usage domain to/from
374       queue is not in the host's AP configuration).
376       Write-only attributes for assigning/unassigning an AP control domain
390     * Store the AP matrix configuration for the adapters, domains, and control
392     * Store the AP matrix configuration for the adapters, domains and control
409     The KVM structure, is used to configure the guest's access to the AP matrix
414     matrix mdev device and deconfigures the guest's AP matrix.
420 Configure the guest's AP resources
422 Configuring the AP resources for a KVM guest will be performed when the
424 function is called when userspace connects to KVM. The guest's AP resources are
437 driver will not be assigned to a KVM guest's matrix. The AP architecture,
441 'assign_control_domain' interfaces will be filtered before providing the AP
446   the host's AP configuration will be filtered.
454 The CPU model features for AP
456 The AP stack relies on the presence of the AP instructions as well as three
457 facilities: The AP Facilities Test (APFT) facility; the AP Query
458 Configuration Information (QCI) facility; and the AP Queue Interruption Control
462 1. ap: Indicates whether the AP instructions are installed on the guest. This
463    feature will be enabled by KVM only if the AP instructions are installed
470 3. apqci: Indicates the AP QCI facility is available on the guest. This facility
474 4. apqi: Indicates AP Queue Interruption Control faclity is available on the
484 A guest can be precluded from using AP features/facilities by turning them off
490 will not see any AP devices. The zcrypt device drivers on the guest that
491 register for type 10 and newer AP devices - i.e., the cex4card and cex4queue
493 a given AP device. If the APFT facility is not installed on the guest, then no
494 adapter or domain devices will get created by the AP bus running on the
500 access to AP facilities. For this example, we will show how to configure
561 	   select S390 AP IOMMU Support
566 	-> VFIO support for AP devices
568 2. Secure the AP queues to be used by the three guests so that the host can not
570    bitmasks marking a subset of the APQN range as usable only by the default AP
579    The 'apmask' is a 256-bit mask that identifies a set of AP adapter IDs
582    available only to the default AP queue device drivers.
584    The 'aqmask' is a 256-bit mask that identifies a set of AP queue indexes
587    available only to the default AP queue device drivers.
617    The APQN of each AP queue device assigned to the linux host is checked by the
618    AP bus against the set of APQNs derived from the Cartesian product of APIDs
619    and APQIs marked as available to the default AP queue device drivers. If a
620    match is detected,  only the default AP queue device drivers will be probed;
624    AP queue device drivers. There are two ways the default masks can be changed:
694    To secure the AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004, 06.0047,
711    This will result in AP queues 05.0004, 05.0047, 05.00ab, 05.00ff, 06.0004,
714    to the AP queue devices bound to it::
734    The administrator, therefore, must take care to secure only AP queues that
735    can be bound to the vfio_ap device driver. The device type for a given AP
744 3. Create the mediated devices needed to configure the AP matrixes for the
947   AP configuration.
962 Over-provisioning of AP queues for a KVM guest:
965 a vfio_ap mediated device that do not reference AP devices in the host's AP
974 Live guest migration is not supported for guests using AP devices without
1021 After live migration of the KVM guest completes, an AP configuration can be