docs/devel/multi-process.rst

1 Multi-process QEMU
6   This is the design document for multi-process QEMU. It does not
31 -------------
34 VM control point, where VMs can be created, migrated, re-configured, and
40 A multi-process QEMU
43 A multi-process QEMU involves separating QEMU services into separate
51 A QEMU control process would remain, but in multi-process mode, will
53 provide the user interface to hot-plug devices or live migrate the VM.
55 A first step in creating a multi-process QEMU is to separate IO services
62 ----------------------
82 VM. (e.g., via the QEMU command line as *-device foo*)
85 parent object (such as "pci-device" for a PCI device) and QEMU will
104 mission-mode IO is performed by the application. The vhost user
106 domain socket.
108 vhost socket
115 the socket, which then can be used by the vhost user application in
132 socket to the vhost application. This path is accelerated by the QEMU
168 Much of the vhost model can be re-used by separated device emulation. In
169 particular, the ideas of using a socket between QEMU and the device
179 break vhost store acceleration since they are synchronous - guest
188 #### qemu-io model
190 ``qemu-io`` is a test harness used to test changes to the QEMU block backend
192 emulation). ``qemu-io`` is not a device emulation application per se, but it
198 -------------------------------------------
212 a tractable to re-implement both the object model and the many device
223     disk-proc -blockdev driver=file,node-name=file0,filename=disk-file0  \
224     -blockdev driver=qcow2,node-name=drive0,file=file0
226 would indicate process *disk-proc* uses a qcow2 emulated disk named
238 socket that connects with the Proxy object. This is a required argument.
242     disk-proc <socket number> <backend list>
248 itself. The QMP monitor socket is specified the same as for a QEMU
253     disk-proc -qmp unix:/tmp/disk-mon,server
255 can be monitored over the UNIX socket path */tmp/disk-mon*.
261 represented as a *-device* of type *pci-proxy-dev*. A socket
262 sub-option to this option specifies the Unix socket that connects
263 to the remote process. An *id* sub-option is required, and it should
268     qemu-system-x86_64 ... -device pci-proxy-dev,id=lsi0,socket=3
286 the remote process. It is also used to pass on device-agnostic commands
289 per-device channels
299 QEMU has an object model based on sub-classes inherited from the
300 "object" super-class. The sub-classes that are of interest here are the
301 "device" and "bus" sub-classes whose child sub-classes make up the
308 sub-class of the "pci-device" class, and will have the same PCI bus
322 - Parses the "socket" sub option and connects to the remote process
324 - Uses the "id" sub-option to connect to the emulated device on the
342 Other tasks will be device-specific. For example, PCI device objects
368 read-only, but certain registers (especially BAR and MSI-related ones)
375 "pci-device-proxy" class that can serve as the parent of a PCI device
376 proxy object. This class's parent would be "pci-device" and it would
385 socket to receive interrupt indications from the emulation process. An
412 must be backed by shared file-backed memory, for example, using
413 *-object memory-backend-file,share=on* and setting that memory backend
420 QEMU will need to create a socket for IOMMU requests from the emulation
427 over the IOMMU socket.
429 device hot-plug via QMP
434 *-device* command line option does. The remote process may either be one
435 started at QEMU startup, or be one added by the "add-process" QMP
445 the proxy will send the remote process a secondary socket file
449 will be extracted, and a secondary socket file descriptor will be sent
460 handle requests from the QEMU process, and route machine-level requests
471 -  address spaces
478 -  RAM
487 -  PCI
517 -  PCI pin interrupts
528 -  PCI MSI/X interrupts
531 CPU-specific PCI address. In QEMU on x86, a KVM APIC object receives
555 -  IOTLB cache
564 -  IOTLB purge
578 restore - the channel will be passed to ``qemu_loadvm_state()`` to
608 -  guest physical range
620 +--------+----------------------------+
624 +--------+----------------------------+
626 +--------+----------------------------+
628 +--------+----------------------------+
630 +--------+----------------------------+
632 -  MMIO request structure
640 +----------+------------------------+
644 +----------+------------------------+
646 +----------+------------------------+
648 +----------+------------------------+
650 +----------+------------------------+
652 +----------+------------------------+
654 +----------+------------------------+
656 -  MMIO request queues
664 -  scoreboard
669 wait queue and sequence number for the per-CPU threads, allowing them to
676 -  device shadow memory
678 Some MMIO loads do not have device side-effects. These MMIOs can be
687 side-effects (and can be completed immediately), and which require a
707 -  create
710 ``ioctl()`` on its per-VM file descriptor. It will allocate and
714 -  ioctl
742 -  destroy
755 -  read
764 -  write
769 removed, then the number of posted stores in the per-CPU scoreboard is
770 decremented. When the number is zero, and a non side-effect load was
773 -  ioctl
789 -  poll
795 -  mmap
799 memory, changes with no side-effects will be reflected in the shadow,
806 Each KVM per-CPU thread can handle MMIO operation on behalf of the guest
812 -  read
815 Loads with side-effects must be handled synchronously, with the KVM
817 process reply before re-starting the guest. Loads that do not have
818 side-effects may be optimized by satisfying them from the shadow image,
823 -  write
828 the per-CPU scoreboard, in order to implement the PCI ordering
845 irq file descriptor, a re-sampling file descriptor needs to be sent to
848 acknowledged by the guest, so they can re-trigger the interrupt if their
849 device has not de-asserted its interrupt.
855 ``using event_notifier_init()`` to create the irq and re-sampling
867 interrupt logic to change the route: de-assigning the existing irq
890 The guest may dynamically update several MSI-related tables in the
891 device's PCI config space. These include per-MSI interrupt enables and
897 --------------
900 ---------------------------
908 --------------------
946 types separate from the main QEMU process and non-disk emulation
950 and non-network emulation process, and only that type can access the