1.. _qgraph: 2 3Qtest Driver Framework 4====================== 5 6In order to test a specific driver, plain libqos tests need to 7take care of booting QEMU with the right machine and devices. 8This makes each test "hardcoded" for a specific configuration, reducing 9the possible coverage that it can reach. 10 11For example, the sdhci device is supported on both x86_64 and ARM boards, 12therefore a generic sdhci test should test all machines and drivers that 13support that device. 14Using only libqos APIs, the test has to manually take care of 15covering all the setups, and build the correct command line. 16 17This also introduces backward compatibility issues: if a device/driver command 18line name is changed, all tests that use that will not work 19properly anymore and need to be adjusted. 20 21The aim of qgraph is to create a graph of drivers, machines and tests such that 22a test aimed to a certain driver does not have to care of 23booting the right QEMU machine, pick the right device, build the command line 24and so on. Instead, it only defines what type of device it is testing 25(interface in qgraph terms) and the framework takes care of 26covering all supported types of devices and machine architectures. 27 28Following the above example, an interface would be ``sdhci``, 29so the sdhci-test should only care of linking its qgraph node with 30that interface. In this way, if the command line of a sdhci driver 31is changed, only the respective qgraph driver node has to be adjusted. 32 33QGraph concepts 34--------------- 35 36The graph is composed by nodes that represent machines, drivers, tests 37and edges that define the relationships between them (``CONSUMES``, ``PRODUCES``, and 38``CONTAINS``). 39 40Nodes 41~~~~~ 42 43A node can be of four types: 44 45- **QNODE_MACHINE**: for example ``arm/raspi2b`` 46- **QNODE_DRIVER**: for example ``generic-sdhci`` 47- **QNODE_INTERFACE**: for example ``sdhci`` (interface for all ``-sdhci`` 48 drivers). 49 An interface is not explicitly created, it will be automatically 50 instantiated when a node consumes or produces it. 51 An interface is simply a struct that abstracts the various drivers 52 for the same type of device, and offers an API to the nodes that 53 use it ("consume" relation in qgraph terms) that is implemented/backed up by the drivers that implement it ("produce" relation in qgraph terms). 54- **QNODE_TEST**: for example ``sdhci-test``. A test consumes an interface 55 and tests the functions provided by it. 56 57Notes for the nodes: 58 59- QNODE_MACHINE: each machine struct must have a ``QGuestAllocator`` and 60 implement ``get_driver()`` to return the allocator mapped to the interface 61 "memory". The function can also return ``NULL`` if the allocator 62 is not set. 63- QNODE_DRIVER: driver names must be unique, and machines and nodes 64 planned to be "consumed" by other nodes must match QEMU 65 drivers name, otherwise they won't be discovered 66 67Edges 68~~~~~ 69 70An edge relation between two nodes (drivers or machines) ``X`` and ``Y`` can be: 71 72- ``X CONSUMES Y``: ``Y`` can be plugged into ``X`` 73- ``X PRODUCES Y``: ``X`` provides the interface ``Y`` 74- ``X CONTAINS Y``: ``Y`` is part of ``X`` component 75 76Execution steps 77~~~~~~~~~~~~~~~ 78 79The basic framework steps are the following: 80 81- All nodes and edges are created in their respective 82 machine/driver/test files 83- The framework starts QEMU and asks for a list of available devices 84 and machines (note that only machines and "consumed" nodes are mapped 85 1:1 with QEMU devices) 86- The framework walks the graph starting from the available machines and 87 performs a Depth First Search for tests 88- Once a test is found, the path is walked again and all drivers are 89 allocated accordingly and the final interface is passed to the test 90- The test is executed 91- Unused objects are cleaned and the path discovery is continued 92 93Depending on the QEMU binary used, only some drivers/machines will be 94available and only test that are reached by them will be executed. 95 96Command line 97~~~~~~~~~~~~ 98 99Command line is built by using node names and optional arguments 100passed by the user when building the edges. 101 102There are three types of command line arguments: 103 104- ``in node`` : created from the node name. For example, machines will 105 have ``-M <machine>`` to its command line, while devices 106 ``-device <device>``. It is automatically done by the framework. 107- ``after node`` : added as additional argument to the node name. 108 This argument is added optionally when creating edges, 109 by setting the parameter ``after_cmd_line`` and 110 ``extra_edge_opts`` in ``QOSGraphEdgeOptions``. 111 The framework automatically adds 112 a comma before ``extra_edge_opts``, 113 because it is going to add attributes 114 after the destination node pointed by 115 the edge containing these options, and automatically 116 adds a space before ``after_cmd_line``, because it 117 adds an additional device, not an attribute. 118- ``before node`` : added as additional argument to the node name. 119 This argument is added optionally when creating edges, 120 by setting the parameter ``before_cmd_line`` in 121 ``QOSGraphEdgeOptions``. This attribute 122 is going to add attributes before the destination node 123 pointed by the edge containing these options. It is 124 helpful to commands that are not node-representable, 125 such as ``-fdsev`` or ``-netdev``. 126 127While adding command line in edges is always used, not all nodes names are 128used in every path walk: this is because the contained or produced ones 129are already added by QEMU, so only nodes that "consumes" will be used to 130build the command line. Also, nodes that will have ``{ "abstract" : true }`` 131as QMP attribute will loose their command line, since they are not proper 132devices to be added in QEMU. 133 134Example:: 135 136 QOSGraphEdgeOptions opts = { 137 .before_cmd_line = "-drive id=drv0,if=none,file=null-co://," 138 "file.read-zeroes=on,format=raw", 139 .after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0", 140 141 opts.extra_device_opts = "id=vs0"; 142 }; 143 144 qos_node_create_driver("virtio-scsi-device", 145 virtio_scsi_device_create); 146 qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts); 147 148Will produce the following command line: 149``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0`` 150 151Troubleshooting unavailable tests 152~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 153 154If there is no path from an available machine to a test then that test will be 155unavailable and won't execute. This can happen if a test or driver did not set 156up its qgraph node correctly. It can also happen if the necessary machine type 157or device is missing from the QEMU binary because it was compiled out or 158otherwise. 159 160It is possible to troubleshoot unavailable tests by running:: 161 162 $ QTEST_QEMU_BINARY=build/qemu-system-x86_64 build/tests/qtest/qos-test --verbose 163 # ALL QGRAPH EDGES: { 164 # src='virtio-net' 165 # |-> dest='virtio-net-tests/vhost-user/multiqueue' type=2 (node=0x559142109e30) 166 # |-> dest='virtio-net-tests/vhost-user/migrate' type=2 (node=0x559142109d00) 167 # src='virtio-net-pci' 168 # |-> dest='virtio-net' type=1 (node=0x55914210d740) 169 # src='pci-bus' 170 # |-> dest='virtio-net-pci' type=2 (node=0x55914210d880) 171 # src='pci-bus-pc' 172 # |-> dest='pci-bus' type=1 (node=0x559142103f40) 173 # src='i440FX-pcihost' 174 # |-> dest='pci-bus-pc' type=0 (node=0x55914210ac70) 175 # src='x86_64/pc' 176 # |-> dest='i440FX-pcihost' type=0 (node=0x5591421117f0) 177 # src='' 178 # |-> dest='x86_64/pc' type=0 (node=0x559142111600) 179 # |-> dest='arm/raspi2b' type=0 (node=0x559142110740) 180 ... 181 # } 182 # ALL QGRAPH NODES: { 183 # name='virtio-net-tests/announce-self' type=3 cmd_line='(null)' [available] 184 # name='arm/raspi2b' type=0 cmd_line='-M raspi2b ' [UNAVAILABLE] 185 ... 186 # } 187 188The ``virtio-net-tests/announce-self`` test is listed as "available" in the 189"ALL QGRAPH NODES" output. This means the test will execute. We can follow the 190qgraph path in the "ALL QGRAPH EDGES" output as follows: '' -> 'x86_64/pc' -> 191'i440FX-pcihost' -> 'pci-bus-pc' -> 'pci-bus' -> 'virtio-net-pci' -> 192'virtio-net'. The root of the qgraph is '' and the depth first search begins 193there. 194 195The ``arm/raspi2b`` machine node is listed as "UNAVAILABLE". Although it is 196reachable from the root via '' -> 'arm/raspi2b' the node is unavailable because 197the QEMU binary did not list it when queried by the framework. This is expected 198because we used the ``qemu-system-x86_64`` binary which does not support ARM 199machine types. 200 201If a test is unexpectedly listed as "UNAVAILABLE", first check that the "ALL 202QGRAPH EDGES" output reports edge connectivity from the root ('') to the test. 203If there is no connectivity then the qgraph nodes were not set up correctly and 204the driver or test code is incorrect. If there is connectivity, check the 205availability of each node in the path in the "ALL QGRAPH NODES" output. The 206first unavailable node in the path is the reason why the test is unavailable. 207Typically this is because the QEMU binary lacks support for the necessary 208machine type or device. 209 210Creating a new driver and its interface 211--------------------------------------- 212 213Here we continue the ``sdhci`` use case, with the following scenario: 214 215- ``sdhci-test`` aims to test the ``read[q,w], writeq`` functions 216 offered by the ``sdhci`` drivers. 217- The current ``sdhci`` device is supported by both ``x86_64/pc`` and ``ARM`` 218 (in this example we focus on the ``arm-raspi2b``) machines. 219- QEMU offers 2 types of drivers: ``QSDHCI_MemoryMapped`` for ``ARM`` and 220 ``QSDHCI_PCI`` for ``x86_64/pc``. Both implement the 221 ``read[q,w], writeq`` functions. 222 223In order to implement such scenario in qgraph, the test developer needs to: 224 225- Create the ``x86_64/pc`` machine node. This machine uses the 226 ``pci-bus`` architecture so it ``contains`` a PCI driver, 227 ``pci-bus-pc``. The actual path is 228 229 ``x86_64/pc --contains--> 1440FX-pcihost --contains--> 230 pci-bus-pc --produces--> pci-bus``. 231 232 For the sake of this example, 233 we do not focus on the PCI interface implementation. 234- Create the ``sdhci-pci`` driver node, representing ``QSDHCI_PCI``. 235 The driver uses the PCI bus (and its API), 236 so it must ``consume`` the ``pci-bus`` generic interface (which abstracts 237 all the pci drivers available) 238 239 ``sdhci-pci --consumes--> pci-bus`` 240- Create an ``arm/raspi2b`` machine node. This machine ``contains`` 241 a ``generic-sdhci`` memory mapped ``sdhci`` driver node, representing 242 ``QSDHCI_MemoryMapped``. 243 244 ``arm/raspi2b --contains--> generic-sdhci`` 245- Create the ``sdhci`` interface node. This interface offers the 246 functions that are shared by all ``sdhci`` devices. 247 The interface is produced by ``sdhci-pci`` and ``generic-sdhci``, 248 the available architecture-specific drivers. 249 250 ``sdhci-pci --produces--> sdhci`` 251 252 ``generic-sdhci --produces--> sdhci`` 253- Create the ``sdhci-test`` test node. The test ``consumes`` the 254 ``sdhci`` interface, using its API. It doesn't need to look at 255 the supported machines or drivers. 256 257 ``sdhci-test --consumes--> sdhci`` 258 259``arm-raspi2b`` machine, simplified from 260``tests/qtest/libqos/arm-raspi2-machine.c``:: 261 262 #include "qgraph.h" 263 264 struct QRaspi2Machine { 265 QOSGraphObject obj; 266 QGuestAllocator alloc; 267 QSDHCI_MemoryMapped sdhci; 268 }; 269 270 static void *raspi2_get_driver(void *object, const char *interface) 271 { 272 QRaspi2Machine *machine = object; 273 if (!g_strcmp0(interface, "memory")) { 274 return &machine->alloc; 275 } 276 277 fprintf(stderr, "%s not present in arm/raspi2b\n", interface); 278 g_assert_not_reached(); 279 } 280 281 static QOSGraphObject *raspi2_get_device(void *obj, 282 const char *device) 283 { 284 QRaspi2Machine *machine = obj; 285 if (!g_strcmp0(device, "generic-sdhci")) { 286 return &machine->sdhci.obj; 287 } 288 289 fprintf(stderr, "%s not present in arm/raspi2b\n", device); 290 g_assert_not_reached(); 291 } 292 293 static void *qos_create_machine_arm_raspi2(QTestState *qts) 294 { 295 QRaspi2Machine *machine = g_new0(QRaspi2Machine, 1); 296 297 alloc_init(&machine->alloc, ...); 298 299 /* Get node(s) contained inside (CONTAINS) */ 300 machine->obj.get_device = raspi2_get_device; 301 302 /* Get node(s) produced (PRODUCES) */ 303 machine->obj.get_driver = raspi2_get_driver; 304 305 /* free the object */ 306 machine->obj.destructor = raspi2_destructor; 307 qos_init_sdhci_mm(&machine->sdhci, ...); 308 return &machine->obj; 309 } 310 311 static void raspi2_register_nodes(void) 312 { 313 /* arm/raspi2b --contains--> generic-sdhci */ 314 qos_node_create_machine("arm/raspi2b", 315 qos_create_machine_arm_raspi2); 316 qos_node_contains("arm/raspi2b", "generic-sdhci", NULL); 317 } 318 319 libqos_init(raspi2_register_nodes); 320 321``x86_64/pc`` machine, simplified from 322``tests/qtest/libqos/x86_64_pc-machine.c``:: 323 324 #include "qgraph.h" 325 326 struct i440FX_pcihost { 327 QOSGraphObject obj; 328 QPCIBusPC pci; 329 }; 330 331 struct QX86PCMachine { 332 QOSGraphObject obj; 333 QGuestAllocator alloc; 334 i440FX_pcihost bridge; 335 }; 336 337 /* i440FX_pcihost */ 338 339 static QOSGraphObject *i440FX_host_get_device(void *obj, 340 const char *device) 341 { 342 i440FX_pcihost *host = obj; 343 if (!g_strcmp0(device, "pci-bus-pc")) { 344 return &host->pci.obj; 345 } 346 fprintf(stderr, "%s not present in i440FX-pcihost\n", device); 347 g_assert_not_reached(); 348 } 349 350 /* x86_64/pc machine */ 351 352 static void *pc_get_driver(void *object, const char *interface) 353 { 354 QX86PCMachine *machine = object; 355 if (!g_strcmp0(interface, "memory")) { 356 return &machine->alloc; 357 } 358 359 fprintf(stderr, "%s not present in x86_64/pc\n", interface); 360 g_assert_not_reached(); 361 } 362 363 static QOSGraphObject *pc_get_device(void *obj, const char *device) 364 { 365 QX86PCMachine *machine = obj; 366 if (!g_strcmp0(device, "i440FX-pcihost")) { 367 return &machine->bridge.obj; 368 } 369 370 fprintf(stderr, "%s not present in x86_64/pc\n", device); 371 g_assert_not_reached(); 372 } 373 374 static void *qos_create_machine_pc(QTestState *qts) 375 { 376 QX86PCMachine *machine = g_new0(QX86PCMachine, 1); 377 378 /* Get node(s) contained inside (CONTAINS) */ 379 machine->obj.get_device = pc_get_device; 380 381 /* Get node(s) produced (PRODUCES) */ 382 machine->obj.get_driver = pc_get_driver; 383 384 /* free the object */ 385 machine->obj.destructor = pc_destructor; 386 pc_alloc_init(&machine->alloc, qts, ALLOC_NO_FLAGS); 387 388 /* Get node(s) contained inside (CONTAINS) */ 389 machine->bridge.obj.get_device = i440FX_host_get_device; 390 391 return &machine->obj; 392 } 393 394 static void pc_machine_register_nodes(void) 395 { 396 /* x86_64/pc --contains--> 1440FX-pcihost --contains--> 397 * pci-bus-pc [--produces--> pci-bus (in pci.h)] */ 398 qos_node_create_machine("x86_64/pc", qos_create_machine_pc); 399 qos_node_contains("x86_64/pc", "i440FX-pcihost", NULL); 400 401 /* contained drivers don't need a constructor, 402 * they will be init by the parent */ 403 qos_node_create_driver("i440FX-pcihost", NULL); 404 qos_node_contains("i440FX-pcihost", "pci-bus-pc", NULL); 405 } 406 407 libqos_init(pc_machine_register_nodes); 408 409``sdhci`` taken from ``tests/qtest/libqos/sdhci.c``:: 410 411 /* Interface node, offers the sdhci API */ 412 struct QSDHCI { 413 uint16_t (*readw)(QSDHCI *s, uint32_t reg); 414 uint64_t (*readq)(QSDHCI *s, uint32_t reg); 415 void (*writeq)(QSDHCI *s, uint32_t reg, uint64_t val); 416 /* other fields */ 417 }; 418 419 /* Memory Mapped implementation of QSDHCI */ 420 struct QSDHCI_MemoryMapped { 421 QOSGraphObject obj; 422 QSDHCI sdhci; 423 /* other driver-specific fields */ 424 }; 425 426 /* PCI implementation of QSDHCI */ 427 struct QSDHCI_PCI { 428 QOSGraphObject obj; 429 QSDHCI sdhci; 430 /* other driver-specific fields */ 431 }; 432 433 /* Memory mapped implementation of QSDHCI */ 434 435 static void *sdhci_mm_get_driver(void *obj, const char *interface) 436 { 437 QSDHCI_MemoryMapped *smm = obj; 438 if (!g_strcmp0(interface, "sdhci")) { 439 return &smm->sdhci; 440 } 441 fprintf(stderr, "%s not present in generic-sdhci\n", interface); 442 g_assert_not_reached(); 443 } 444 445 void qos_init_sdhci_mm(QSDHCI_MemoryMapped *sdhci, QTestState *qts, 446 uint32_t addr, QSDHCIProperties *common) 447 { 448 /* Get node contained inside (CONTAINS) */ 449 sdhci->obj.get_driver = sdhci_mm_get_driver; 450 451 /* SDHCI interface API */ 452 sdhci->sdhci.readw = sdhci_mm_readw; 453 sdhci->sdhci.readq = sdhci_mm_readq; 454 sdhci->sdhci.writeq = sdhci_mm_writeq; 455 sdhci->qts = qts; 456 } 457 458 /* PCI implementation of QSDHCI */ 459 460 static void *sdhci_pci_get_driver(void *object, 461 const char *interface) 462 { 463 QSDHCI_PCI *spci = object; 464 if (!g_strcmp0(interface, "sdhci")) { 465 return &spci->sdhci; 466 } 467 468 fprintf(stderr, "%s not present in sdhci-pci\n", interface); 469 g_assert_not_reached(); 470 } 471 472 static void *sdhci_pci_create(void *pci_bus, 473 QGuestAllocator *alloc, 474 void *addr) 475 { 476 QSDHCI_PCI *spci = g_new0(QSDHCI_PCI, 1); 477 QPCIBus *bus = pci_bus; 478 uint64_t barsize; 479 480 qpci_device_init(&spci->dev, bus, addr); 481 482 /* SDHCI interface API */ 483 spci->sdhci.readw = sdhci_pci_readw; 484 spci->sdhci.readq = sdhci_pci_readq; 485 spci->sdhci.writeq = sdhci_pci_writeq; 486 487 /* Get node(s) produced (PRODUCES) */ 488 spci->obj.get_driver = sdhci_pci_get_driver; 489 490 spci->obj.start_hw = sdhci_pci_start_hw; 491 spci->obj.destructor = sdhci_destructor; 492 return &spci->obj; 493 } 494 495 static void qsdhci_register_nodes(void) 496 { 497 QOSGraphEdgeOptions opts = { 498 .extra_device_opts = "addr=04.0", 499 }; 500 501 /* generic-sdhci */ 502 /* generic-sdhci --produces--> sdhci */ 503 qos_node_create_driver("generic-sdhci", NULL); 504 qos_node_produces("generic-sdhci", "sdhci"); 505 506 /* sdhci-pci */ 507 /* sdhci-pci --produces--> sdhci 508 * sdhci-pci --consumes--> pci-bus */ 509 qos_node_create_driver("sdhci-pci", sdhci_pci_create); 510 qos_node_produces("sdhci-pci", "sdhci"); 511 qos_node_consumes("sdhci-pci", "pci-bus", &opts); 512 } 513 514 libqos_init(qsdhci_register_nodes); 515 516In the above example, all possible types of relations are created:: 517 518 x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc 519 | 520 sdhci-pci --consumes--> pci-bus <--produces--+ 521 | 522 +--produces--+ 523 | 524 v 525 sdhci 526 ^ 527 | 528 +--produces-- + 529 | 530 arm/raspi2b --contains--> generic-sdhci 531 532or inverting the consumes edge in consumed_by:: 533 534 x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc 535 | 536 sdhci-pci <--consumed by-- pci-bus <--produces--+ 537 | 538 +--produces--+ 539 | 540 v 541 sdhci 542 ^ 543 | 544 +--produces-- + 545 | 546 arm/raspi2b --contains--> generic-sdhci 547 548Adding a new test 549----------------- 550 551Given the above setup, adding a new test is very simple. 552``sdhci-test``, taken from ``tests/qtest/sdhci-test.c``:: 553 554 static void check_capab_sdma(QSDHCI *s, bool supported) 555 { 556 uint64_t capab, capab_sdma; 557 558 capab = s->readq(s, SDHC_CAPAB); 559 capab_sdma = FIELD_EX64(capab, SDHC_CAPAB, SDMA); 560 g_assert_cmpuint(capab_sdma, ==, supported); 561 } 562 563 static void test_registers(void *obj, void *data, 564 QGuestAllocator *alloc) 565 { 566 QSDHCI *s = obj; 567 568 /* example test */ 569 check_capab_sdma(s, s->props.capab.sdma); 570 } 571 572 static void register_sdhci_test(void) 573 { 574 /* sdhci-test --consumes--> sdhci */ 575 qos_add_test("registers", "sdhci", test_registers, NULL); 576 } 577 578 libqos_init(register_sdhci_test); 579 580Here a new test is created, consuming ``sdhci`` interface node 581and creating a valid path from both machines to a test. 582Final graph will be like this:: 583 584 x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc 585 | 586 sdhci-pci --consumes--> pci-bus <--produces--+ 587 | 588 +--produces--+ 589 | 590 v 591 sdhci <--consumes-- sdhci-test 592 ^ 593 | 594 +--produces-- + 595 | 596 arm/raspi2b --contains--> generic-sdhci 597 598or inverting the consumes edge in consumed_by:: 599 600 x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc 601 | 602 sdhci-pci <--consumed by-- pci-bus <--produces--+ 603 | 604 +--produces--+ 605 | 606 v 607 sdhci --consumed by--> sdhci-test 608 ^ 609 | 610 +--produces-- + 611 | 612 arm/raspi2b --contains--> generic-sdhci 613 614Assuming there the binary is 615``QTEST_QEMU_BINARY=./qemu-system-x86_64`` 616a valid test path will be: 617``/x86_64/pc/1440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pc/sdhci/sdhci-test`` 618 619and for the binary ``QTEST_QEMU_BINARY=./qemu-system-arm``: 620 621``/arm/raspi2b/generic-sdhci/sdhci/sdhci-test`` 622 623Additional examples are also in ``test-qgraph.c`` 624 625Qgraph API reference 626-------------------- 627 628.. kernel-doc:: tests/qtest/libqos/qgraph.h 629