docs/devel/multi-process.rst

1 Multi-process QEMU
6   This is the design document for multi-process QEMU. It does not
31 -------------
34 VM control point, where VMs can be created, migrated, re-configured, and
40 A multi-process QEMU
43 A multi-process QEMU involves separating QEMU services into separate
51 A QEMU control process would remain, but in multi-process mode, will
53 provide the user interface to hot-plug devices or live migrate the VM.
55 A first step in creating a multi-process QEMU is to separate IO services
62 ----------------------
80 code to emulate a device named "foo" is always present in QEMU, but its
82 VM. (e.g., via the QEMU command line as *-device foo*)
85 parent object (such as "pci-device" for a PCI device) and QEMU will
104 mission-mode IO is performed by the application. The vhost user
168 Much of the vhost model can be re-used by separated device emulation. In
179 break vhost store acceleration since they are synchronous - guest
188 #### qemu-io model
190 ``qemu-io`` is a test harness used to test changes to the QEMU block backend
192 emulation). ``qemu-io`` is not a device emulation application per se, but it
198 -------------------------------------------
212 a tractable to re-implement both the object model and the many device
223     disk-proc -blockdev driver=file,node-name=file0,filename=disk-file0  \
224     -blockdev driver=qcow2,node-name=drive0,file=file0
226 would indicate process *disk-proc* uses a qcow2 emulated disk named
242     disk-proc <socket number> <backend list>
253     disk-proc -qmp unix:/tmp/disk-mon,server
255 can be monitored over the UNIX socket path */tmp/disk-mon*.
261 represented as a *-device* of type *pci-proxy-dev*. A socket
262 sub-option to this option specifies the Unix socket that connects
263 to the remote process. An *id* sub-option is required, and it should
268     qemu-system-x86_64 ... -device pci-proxy-dev,id=lsi0,socket=3
286 the remote process. It is also used to pass on device-agnostic commands
289 per-device channels
299 QEMU has an object model based on sub-classes inherited from the
300 "object" super-class. The sub-classes that are of interest here are the
301 "device" and "bus" sub-classes whose child sub-classes make up the
308 sub-class of the "pci-device" class, and will have the same PCI bus
322 - Parses the "socket" sub option and connects to the remote process
324 - Uses the "id" sub-option to connect to the emulated device on the
342 Other tasks will be device-specific. For example, PCI device objects
368 read-only, but certain registers (especially BAR and MSI-related ones)
375 "pci-device-proxy" class that can serve as the parent of a PCI device
376 proxy object. This class's parent would be "pci-device" and it would
412 must be backed by shared file-backed memory, for example, using
413 *-object memory-backend-file,share=on* and setting that memory backend
429 device hot-plug via QMP
434 *-device* command line option does. The remote process may either be one
435 started at QEMU startup, or be one added by the "add-process" QMP
460 handle requests from the QEMU process, and route machine-level requests
471 -  address spaces
478 -  RAM
487 -  PCI
517 -  PCI pin interrupts
528 -  PCI MSI/X interrupts
531 CPU-specific PCI address. In QEMU on x86, a KVM APIC object receives
555 -  IOTLB cache
564 -  IOTLB purge
578 restore - the channel will be passed to ``qemu_loadvm_state()`` to
608 -  guest physical range
620 +--------+----------------------------+
624 +--------+----------------------------+
626 +--------+----------------------------+
628 +--------+----------------------------+
630 +--------+----------------------------+
632 -  MMIO request structure
640 +----------+------------------------+
644 +----------+------------------------+
646 +----------+------------------------+
648 +----------+------------------------+
650 +----------+------------------------+
652 +----------+------------------------+
654 +----------+------------------------+
656 -  MMIO request queues
659 are two queues: pending queue is for MMIOs that haven't been read by the
660 emulation program, and the sent queue is for MMIOs that haven't been
661 acknowledged. The main use of the second queue is to validate MMIO
664 -  scoreboard
669 wait queue and sequence number for the per-CPU threads, allowing them to
676 -  device shadow memory
678 Some MMIO loads do not have device side-effects. These MMIOs can be
687 side-effects (and can be completed immediately), and which require a
707 -  create
710 ``ioctl()`` on its per-VM file descriptor. It will allocate and
714 -  ioctl
742 -  destroy
755 -  read
760 pending queue to the sent queue, and if there are threads waiting for
764 -  write
767 the MMIO requests in the sent queue. Matches are removed from the sent
768 queue, and any threads waiting for the reply are woken. If a store is
769 removed, then the number of posted stores in the per-CPU scoreboard is
770 decremented. When the number is zero, and a non side-effect load was
773 -  ioctl
789 -  poll
793 return if the pending MMIO request queue is not empty.
795 -  mmap
799 memory, changes with no side-effects will be reflected in the shadow,
806 Each KVM per-CPU thread can handle MMIO operation on behalf of the guest
812 -  read
815 Loads with side-effects must be handled synchronously, with the KVM
817 process reply before re-starting the guest. Loads that do not have
818 side-effects may be optimized by satisfying them from the shadow image,
823 -  write
826 queue is full. In this case, the QEMU thread must sleep waiting for
827 space in the queue. Stores will increment the number of posted stores in
828 the per-CPU scoreboard, in order to implement the PCI ordering
845 irq file descriptor, a re-sampling file descriptor needs to be sent to
848 acknowledged by the guest, so they can re-trigger the interrupt if their
849 device has not de-asserted its interrupt.
855 ``using event_notifier_init()`` to create the irq and re-sampling
867 interrupt logic to change the route: de-assigning the existing irq
890 The guest may dynamically update several MSI-related tables in the
891 device's PCI config space. These include per-MSI interrupt enables and
897 --------------
900 ---------------------------
908 --------------------
946 types separate from the main QEMU process and non-disk emulation
950 and non-network emulation process, and only that type can access the