1.. 2 Copyright (C) 2017, Emilio G. Cota <cota@braap.org> 3 Copyright (c) 2019, Linaro Limited 4 Written by Emilio Cota and Alex Bennée 5 6================ 7QEMU TCG Plugins 8================ 9 10QEMU TCG plugins provide a way for users to run experiments taking 11advantage of the total system control emulation can have over a guest. 12It provides a mechanism for plugins to subscribe to events during 13translation and execution and optionally callback into the plugin 14during these events. TCG plugins are unable to change the system state 15only monitor it passively. However they can do this down to an 16individual instruction granularity including potentially subscribing 17to all load and store operations. 18 19API Stability 20============= 21 22This is a new feature for QEMU and it does allow people to develop 23out-of-tree plugins that can be dynamically linked into a running QEMU 24process. However the project reserves the right to change or break the 25API should it need to do so. The best way to avoid this is to submit 26your plugin upstream so they can be updated if/when the API changes. 27 28API versioning 29-------------- 30 31All plugins need to declare a symbol which exports the plugin API 32version they were built against. This can be done simply by:: 33 34 QEMU_PLUGIN_EXPORT int qemu_plugin_version = QEMU_PLUGIN_VERSION; 35 36The core code will refuse to load a plugin that doesn't export a 37``qemu_plugin_version`` symbol or if plugin version is outside of QEMU's 38supported range of API versions. 39 40Additionally the ``qemu_info_t`` structure which is passed to the 41``qemu_plugin_install`` method of a plugin will detail the minimum and 42current API versions supported by QEMU. The API version will be 43incremented if new APIs are added. The minimum API version will be 44incremented if existing APIs are changed or removed. 45 46Exposure of QEMU internals 47-------------------------- 48 49The plugin architecture actively avoids leaking implementation details 50about how QEMU's translation works to the plugins. While there are 51conceptions such as translation time and translation blocks the 52details are opaque to plugins. The plugin is able to query select 53details of instructions and system configuration only through the 54exported *qemu_plugin* functions. 55 56Query Handle Lifetime 57--------------------- 58 59Each callback provides an opaque anonymous information handle which 60can usually be further queried to find out information about a 61translation, instruction or operation. The handles themselves are only 62valid during the lifetime of the callback so it is important that any 63information that is needed is extracted during the callback and saved 64by the plugin. 65 66API 67=== 68 69.. kernel-doc:: include/qemu/qemu-plugin.h 70 71Usage 72===== 73 74Any QEMU binary with TCG support has plugins enabled by default. 75Earlier releases needed to be explicitly enabled with:: 76 77 configure --enable-plugins 78 79Once built a program can be run with multiple plugins loaded each with 80their own arguments:: 81 82 $QEMU $OTHER_QEMU_ARGS \ 83 -plugin tests/plugin/libhowvec.so,inline=on,count=hint \ 84 -plugin tests/plugin/libhotblocks.so 85 86Arguments are plugin specific and can be used to modify their 87behaviour. In this case the howvec plugin is being asked to use inline 88ops to count and break down the hint instructions by type. 89 90Plugin Life cycle 91================= 92 93First the plugin is loaded and the public qemu_plugin_install function 94is called. The plugin will then register callbacks for various plugin 95events. Generally plugins will register a handler for the *atexit* 96if they want to dump a summary of collected information once the 97program/system has finished running. 98 99When a registered event occurs the plugin callback is invoked. The 100callbacks may provide additional information. In the case of a 101translation event the plugin has an option to enumerate the 102instructions in a block of instructions and optionally register 103callbacks to some or all instructions when they are executed. 104 105There is also a facility to add an inline event where code to 106increment a counter can be directly inlined with the translation. 107Currently only a simple increment is supported. This is not atomic so 108can miss counts. If you want absolute precision you should use a 109callback which can then ensure atomicity itself. 110 111Finally when QEMU exits all the registered *atexit* callbacks are 112invoked. 113 114Internals 115========= 116 117Locking 118------- 119 120We have to ensure we cannot deadlock, particularly under MTTCG. For 121this we acquire a lock when called from plugin code. We also keep the 122list of callbacks under RCU so that we do not have to hold the lock 123when calling the callbacks. This is also for performance, since some 124callbacks (e.g. memory access callbacks) might be called very 125frequently. 126 127 * A consequence of this is that we keep our own list of CPUs, so that 128 we do not have to worry about locking order wrt cpu_list_lock. 129 * Use a recursive lock, since we can get registration calls from 130 callbacks. 131 132As a result registering/unregistering callbacks is "slow", since it 133takes a lock. But this is very infrequent; we want performance when 134calling (or not calling) callbacks, not when registering them. Using 135RCU is great for this. 136 137We support the uninstallation of a plugin at any time (e.g. from 138plugin callbacks). This allows plugins to remove themselves if they no 139longer want to instrument the code. This operation is asynchronous 140which means callbacks may still occur after the uninstall operation is 141requested. The plugin isn't completely uninstalled until the safe work 142has executed while all vCPUs are quiescent. 143 144Example Plugins 145=============== 146 147There are a number of plugins included with QEMU and you are 148encouraged to contribute your own plugins plugins upstream. There is a 149``contrib/plugins`` directory where they can go. 150 151- tests/plugins 152 153These are some basic plugins that are used to test and exercise the 154API during the ``make check-tcg`` target. 155 156- contrib/plugins/hotblocks.c 157 158The hotblocks plugin allows you to examine the where hot paths of 159execution are in your program. Once the program has finished you will 160get a sorted list of blocks reporting the starting PC, translation 161count, number of instructions and execution count. This will work best 162with linux-user execution as system emulation tends to generate 163re-translations as blocks from different programs get swapped in and 164out of system memory. 165 166If your program is single-threaded you can use the ``inline`` option for 167slightly faster (but not thread safe) counters. 168 169Example:: 170 171 ./aarch64-linux-user/qemu-aarch64 \ 172 -plugin contrib/plugins/libhotblocks.so -d plugin \ 173 ./tests/tcg/aarch64-linux-user/sha1 174 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 175 collected 903 entries in the hash table 176 pc, tcount, icount, ecount 177 0x0000000041ed10, 1, 5, 66087 178 0x000000004002b0, 1, 4, 66087 179 ... 180 181- contrib/plugins/hotpages.c 182 183Similar to hotblocks but this time tracks memory accesses:: 184 185 ./aarch64-linux-user/qemu-aarch64 \ 186 -plugin contrib/plugins/libhotpages.so -d plugin \ 187 ./tests/tcg/aarch64-linux-user/sha1 188 SHA1=15dd99a1991e0b3826fede3deffc1feba42278e6 189 Addr, RCPUs, Reads, WCPUs, Writes 190 0x000055007fe000, 0x0001, 31747952, 0x0001, 8835161 191 0x000055007ff000, 0x0001, 29001054, 0x0001, 8780625 192 0x00005500800000, 0x0001, 687465, 0x0001, 335857 193 0x0000000048b000, 0x0001, 130594, 0x0001, 355 194 0x0000000048a000, 0x0001, 1826, 0x0001, 11 195 196The hotpages plugin can be configured using the following arguments: 197 198 * sortby=reads|writes|address 199 200 Log the data sorted by either the number of reads, the number of writes, or 201 memory address. (Default: entries are sorted by the sum of reads and writes) 202 203 * io=on 204 205 Track IO addresses. Only relevant to full system emulation. (Default: off) 206 207 * pagesize=N 208 209 The page size used. (Default: N = 4096) 210 211- contrib/plugins/howvec.c 212 213This is an instruction classifier so can be used to count different 214types of instructions. It has a number of options to refine which get 215counted. You can give a value to the `count` argument for a class of 216instructions to break it down fully, so for example to see all the system 217registers accesses:: 218 219 ./aarch64-softmmu/qemu-system-aarch64 $(QEMU_ARGS) \ 220 -append "root=/dev/sda2 systemd.unit=benchmark.service" \ 221 -smp 4 -plugin ./contrib/plugins/libhowvec.so,count=sreg -d plugin 222 223which will lead to a sorted list after the class breakdown:: 224 225 Instruction Classes: 226 Class: UDEF not counted 227 Class: SVE (68 hits) 228 Class: PCrel addr (47789483 hits) 229 Class: Add/Sub (imm) (192817388 hits) 230 Class: Logical (imm) (93852565 hits) 231 Class: Move Wide (imm) (76398116 hits) 232 Class: Bitfield (44706084 hits) 233 Class: Extract (5499257 hits) 234 Class: Cond Branch (imm) (147202932 hits) 235 Class: Exception Gen (193581 hits) 236 Class: NOP not counted 237 Class: Hints (6652291 hits) 238 Class: Barriers (8001661 hits) 239 Class: PSTATE (1801695 hits) 240 Class: System Insn (6385349 hits) 241 Class: System Reg counted individually 242 Class: Branch (reg) (69497127 hits) 243 Class: Branch (imm) (84393665 hits) 244 Class: Cmp & Branch (110929659 hits) 245 Class: Tst & Branch (44681442 hits) 246 Class: AdvSimd ldstmult (736 hits) 247 Class: ldst excl (9098783 hits) 248 Class: Load Reg (lit) (87189424 hits) 249 Class: ldst noalloc pair (3264433 hits) 250 Class: ldst pair (412526434 hits) 251 Class: ldst reg (imm) (314734576 hits) 252 Class: Loads & Stores (2117774 hits) 253 Class: Data Proc Reg (223519077 hits) 254 Class: Scalar FP (31657954 hits) 255 Individual Instructions: 256 Instr: mrs x0, sp_el0 (2682661 hits) (op=0xd5384100/ System Reg) 257 Instr: mrs x1, tpidr_el2 (1789339 hits) (op=0xd53cd041/ System Reg) 258 Instr: mrs x2, tpidr_el2 (1513494 hits) (op=0xd53cd042/ System Reg) 259 Instr: mrs x0, tpidr_el2 (1490823 hits) (op=0xd53cd040/ System Reg) 260 Instr: mrs x1, sp_el0 (933793 hits) (op=0xd5384101/ System Reg) 261 Instr: mrs x2, sp_el0 (699516 hits) (op=0xd5384102/ System Reg) 262 Instr: mrs x4, tpidr_el2 (528437 hits) (op=0xd53cd044/ System Reg) 263 Instr: mrs x30, ttbr1_el1 (480776 hits) (op=0xd538203e/ System Reg) 264 Instr: msr ttbr1_el1, x30 (480713 hits) (op=0xd518203e/ System Reg) 265 Instr: msr vbar_el1, x30 (480671 hits) (op=0xd518c01e/ System Reg) 266 ... 267 268To find the argument shorthand for the class you need to examine the 269source code of the plugin at the moment, specifically the ``*opt`` 270argument in the InsnClassExecCount tables. 271 272- contrib/plugins/lockstep.c 273 274This is a debugging tool for developers who want to find out when and 275where execution diverges after a subtle change to TCG code generation. 276It is not an exact science and results are likely to be mixed once 277asynchronous events are introduced. While the use of -icount can 278introduce determinism to the execution flow it doesn't always follow 279the translation sequence will be exactly the same. Typically this is 280caused by a timer firing to service the GUI causing a block to end 281early. However in some cases it has proved to be useful in pointing 282people at roughly where execution diverges. The only argument you need 283for the plugin is a path for the socket the two instances will 284communicate over:: 285 286 287 ./sparc-softmmu/qemu-system-sparc -monitor none -parallel none \ 288 -net none -M SS-20 -m 256 -kernel day11/zImage.elf \ 289 -plugin ./contrib/plugins/liblockstep.so,sockpath=lockstep-sparc.sock \ 290 -d plugin,nochain 291 292which will eventually report:: 293 294 qemu-system-sparc: warning: nic lance.0 has no peer 295 @ 0x000000ffd06678 vs 0x000000ffd001e0 (2/1 since last) 296 @ 0x000000ffd07d9c vs 0x000000ffd06678 (3/1 since last) 297 Δ insn_count @ 0x000000ffd07d9c (809900609) vs 0x000000ffd06678 (809900612) 298 previously @ 0x000000ffd06678/10 (809900609 insns) 299 previously @ 0x000000ffd001e0/4 (809900599 insns) 300 previously @ 0x000000ffd080ac/2 (809900595 insns) 301 previously @ 0x000000ffd08098/5 (809900593 insns) 302 previously @ 0x000000ffd080c0/1 (809900588 insns) 303 304- contrib/plugins/hwprofile.c 305 306The hwprofile tool can only be used with system emulation and allows 307the user to see what hardware is accessed how often. It has a number of options: 308 309 * track=read or track=write 310 311 By default the plugin tracks both reads and writes. You can use one 312 of these options to limit the tracking to just one class of accesses. 313 314 * source 315 316 Will include a detailed break down of what the guest PC that made the 317 access was. Not compatible with the pattern option. Example output:: 318 319 cirrus-low-memory @ 0xfffffd00000a0000 320 pc:fffffc0000005cdc, 1, 256 321 pc:fffffc0000005ce8, 1, 256 322 pc:fffffc0000005cec, 1, 256 323 324 * pattern 325 326 Instead break down the accesses based on the offset into the HW 327 region. This can be useful for seeing the most used registers of a 328 device. Example output:: 329 330 pci0-conf @ 0xfffffd01fe000000 331 off:00000004, 1, 1 332 off:00000010, 1, 3 333 off:00000014, 1, 3 334 off:00000018, 1, 2 335 off:0000001c, 1, 2 336 off:00000020, 1, 2 337 ... 338 339- contrib/plugins/execlog.c 340 341The execlog tool traces executed instructions with memory access. It can be used 342for debugging and security analysis purposes. 343Please be aware that this will generate a lot of output. 344 345The plugin takes no argument:: 346 347 qemu-system-arm $(QEMU_ARGS) \ 348 -plugin ./contrib/plugins/libexeclog.so -d plugin 349 350which will output an execution trace following this structure:: 351 352 # vCPU, vAddr, opcode, disassembly[, load/store, memory addr, device]... 353 0, 0xa12, 0xf8012400, "movs r4, #0" 354 0, 0xa14, 0xf87f42b4, "cmp r4, r6" 355 0, 0xa16, 0xd206, "bhs #0xa26" 356 0, 0xa18, 0xfff94803, "ldr r0, [pc, #0xc]", load, 0x00010a28, RAM 357 0, 0xa1a, 0xf989f000, "bl #0xd30" 358 0, 0xd30, 0xfff9b510, "push {r4, lr}", store, 0x20003ee0, RAM, store, 0x20003ee4, RAM 359 0, 0xd32, 0xf9893014, "adds r0, #0x14" 360 0, 0xd34, 0xf9c8f000, "bl #0x10c8" 361 0, 0x10c8, 0xfff96c43, "ldr r3, [r0, #0x44]", load, 0x200000e4, RAM 362 363- contrib/plugins/cache.c 364 365Cache modelling plugin that measures the performance of a given cache 366configuration when a given working set is run:: 367 368 qemu-x86_64 -plugin ./contrib/plugins/libcache.so \ 369 -d plugin -D cache.log ./tests/tcg/x86_64-linux-user/float_convs 370 371will report the following:: 372 373 core #, data accesses, data misses, dmiss rate, insn accesses, insn misses, imiss rate 374 0 996695 508 0.0510% 2642799 18617 0.7044% 375 376 address, data misses, instruction 377 0x424f1e (_int_malloc), 109, movq %rax, 8(%rcx) 378 0x41f395 (_IO_default_xsputn), 49, movb %dl, (%rdi, %rax) 379 0x42584d (ptmalloc_init.part.0), 33, movaps %xmm0, (%rax) 380 0x454d48 (__tunables_init), 20, cmpb $0, (%r8) 381 ... 382 383 address, fetch misses, instruction 384 0x4160a0 (__vfprintf_internal), 744, movl $1, %ebx 385 0x41f0a0 (_IO_setb), 744, endbr64 386 0x415882 (__vfprintf_internal), 744, movq %r12, %rdi 387 0x4268a0 (__malloc), 696, andq $0xfffffffffffffff0, %rax 388 ... 389 390The plugin has a number of arguments, all of them are optional: 391 392 * limit=N 393 394 Print top N icache and dcache thrashing instructions along with their 395 address, number of misses, and its disassembly. (default: 32) 396 397 * icachesize=N 398 * iblksize=B 399 * iassoc=A 400 401 Instruction cache configuration arguments. They specify the cache size, block 402 size, and associativity of the instruction cache, respectively. 403 (default: N = 16384, B = 64, A = 8) 404 405 * dcachesize=N 406 * dblksize=B 407 * dassoc=A 408 409 Data cache configuration arguments. They specify the cache size, block size, 410 and associativity of the data cache, respectively. 411 (default: N = 16384, B = 64, A = 8) 412 413 * evict=POLICY 414 415 Sets the eviction policy to POLICY. Available policies are: :code:`lru`, 416 :code:`fifo`, and :code:`rand`. The plugin will use the specified policy for 417 both instruction and data caches. (default: POLICY = :code:`lru`) 418 419 * cores=N 420 421 Sets the number of cores for which we maintain separate icache and dcache. 422 (default: for linux-user, N = 1, for full system emulation: N = cores 423 available to guest) 424