docs/devel/replay.rst

13 execution. Execution recording writes a non-deterministic events log, which
16 non-deterministic events including external input, hardware clocks,
21 Devices' models that have non-deterministic input from external devices were
26 All non-deterministic events are coming from these devices. But to
33 …th Multi-target QEMU Simulator for Dynamic Analysis and Reverse Debugging <https://www.computer.or…
38  * wrappers for clock and time functions to save their return values in the log
43  * adding internal checkpoints for cpu and io synchronization
50 --------------------
52 QEMU should work in icount mode to use record/replay feature. icount was
54 of the virtual machine. We also use icount to control the occurrence of the
55 non-deterministic events. The number of instructions elapsed from the last event
60 ----------------------------------
68 another lock to keep things in lock-step. This role is now handled by
71 in a deterministic ping-pong between the two main threads.
82 -----------
85 non-determinism. These are inputs from clock and peripheral devices,
87 from timers, asynchronous input-output, and bottom halves.
89 Invocations of timers are coupled with clock reads and changing the state
90 of the virtual machine. Reads produce non-deterministic data taken from
91 host clock. And VM state changes should preserve their order. Their relative
93 To preserve this order we use checkpoints. When a specific clock is processed
103 Two other checkpoints govern the "warping" of the virtual clock.
104 While the virtual machine is idle, the virtual clock increments at
106 (called the warp timer) on the virtual real time clock, so that the
107 timer fires at the next deadline of the virtual clock; the virtual clock
108 is then incremented (which is called "warping" the virtual clock) as
113 starts accounting real time to virtual clock. ``icount_account_warp_timer``
115 and it warps the virtual clock by the amount of real time that has passed
127 ------
132  * Real time clock. Based on host time and used only for callbacks that
134    clock and timers does not affect deterministic replay at all.
135  * Virtual clock. These timers run only during the emulation. In icount
136    mode virtual clock value is calculated using executed instructions counter.
138  * Host clock. This clock is used by device models that simulate real time
139    sources (e.g. real time clock chip). Host clock is the one of the sources
140    of non-determinism. Host clock read operations should be logged to
142  * Virtual real time clock. This clock is similar to real time clock but
143    it is used only for increasing virtual clock while virtual machine is
144    sleeping. Due to its nature it is also non-deterministic as the host clock
147 All virtual devices should use virtual clock for timers that change the guest
148 state. Virtual clock is deterministic, therefore such timers are deterministic
151 Virtual devices can also use realtime clock for the events that do not change
152 the guest state directly. When the clock ticking should depend on VM execution
153 speed, use virtual clock with EXTERNAL attribute. It is not deterministic,
154 but its speed depends on the guest execution. This clock is used by
159 -------------
171 -------------
197 Sometimes the block layer uses asynchronous callbacks for its internal purposes
203 -----------------------------
214 the dependencies that may make restoring the VM state non-deterministic.
217 ---------------
228 events. The header includes 4-byte replay version id and 8-byte reserved
233 It includes all non-deterministic inputs of VM, synchronization marks and
241 Every event in the log includes 1-byte event id and optional arguments.
242 When argument is an array, it is stored as 4-byte array length
246  - EVENT_INSTRUCTION. Instructions executed since last event. Followed by:
248    - 4-byte number of executed instructions.
250  - EVENT_INTERRUPT. Used to synchronize interrupt processing.
251  - EVENT_EXCEPTION. Used to synchronize exception handling.
252  - EVENT_ASYNC. This is a group of events. When such an event is generated,
256      - REPLAY_ASYNC_EVENT_BH. Bottom-half callback. This event synchronizes
260         - 8-byte operation id.
262      - REPLAY_ASYNC_EVENT_INPUT. Input device event. Contains
266         - 9-16 bytes depending of input event.
268      - REPLAY_ASYNC_EVENT_INPUT_SYNC. Internal input synchronization event.
269      - REPLAY_ASYNC_EVENT_CHAR_READ. Character (e.g., serial port) device input
272         - 1-byte character device id.
273         - Array with bytes were read.
275      - REPLAY_ASYNC_EVENT_BLOCK. Block device operation. Used to synchronize
278         - 8-byte operation id.
280      - REPLAY_ASYNC_EVENT_NET. Incoming network packet. Followed by:
282         - 1-byte network adapter id.
283         - 4-byte packet flags.
284         - Array with packet bytes.
286  - EVENT_SHUTDOWN. Occurs when user sends shutdown event to qemu,
288  - EVENT_CHAR_WRITE. Used to synchronize character output operations. Followed by:
290     - 4-byte output function return value.
291     - 4-byte offset in the output array.
293  - EVENT_CHAR_READ_ALL. Used to synchronize character input operations,
296     - Array with bytes that were read.
298  - EVENT_CHAR_READ_ALL_ERROR. Unsuccessful character input operation,
301     - 4-byte error code.
303  - EVENT_CLOCK + clock_id. Group of events for host clock read operations. Followed by:
305     - 8-byte clock value.
307  - EVENT_CHECKPOINT + checkpoint_id. Checkpoint for synchronization of
308    CPU, internal threads, and asynchronous input events.
309  - EVENT_END. Last event in the log.