1Backwards compatibility 2======================= 3 4How backwards compatibility works 5--------------------------------- 6 7When we do migration, we have two QEMU processes: the source and the 8target. There are two cases, they are the same version or they are 9different versions. The easy case is when they are the same version. 10The difficult one is when they are different versions. 11 12There are two things that are different, but they have very similar 13names and sometimes get confused: 14 15- QEMU version 16- machine type version 17 18Let's start with a practical example, we start with: 19 20- qemu-system-x86_64 (v5.2), from now on qemu-5.2. 21- qemu-system-x86_64 (v5.1), from now on qemu-5.1. 22 23Related to this are the "latest" machine types defined on each of 24them: 25 26- pc-q35-5.2 (newer one in qemu-5.2) from now on pc-5.2 27- pc-q35-5.1 (newer one in qemu-5.1) from now on pc-5.1 28 29First of all, migration is only supposed to work if you use the same 30machine type in both source and destination. The QEMU hardware 31configuration needs to be the same also on source and destination. 32Most aspects of the backend configuration can be changed at will, 33except for a few cases where the backend features influence frontend 34device feature exposure. But that is not relevant for this section. 35 36I am going to list the number of combinations that we can have. Let's 37start with the trivial ones, QEMU is the same on source and 38destination: 39 401 - qemu-5.2 -M pc-5.2 -> migrates to -> qemu-5.2 -M pc-5.2 41 42 This is the latest QEMU with the latest machine type. 43 This have to work, and if it doesn't work it is a bug. 44 452 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1 46 47 Exactly the same case than the previous one, but for 5.1. 48 Nothing to see here either. 49 50This are the easiest ones, we will not talk more about them in this 51section. 52 53Now we start with the more interesting cases. Consider the case where 54we have the same QEMU version in both sides (qemu-5.2) but we are using 55the latest machine type for that version (pc-5.2) but one of an older 56QEMU version, in this case pc-5.1. 57 583 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 59 60 It needs to use the definition of pc-5.1 and the devices as they 61 were configured on 5.1, but this should be easy in the sense that 62 both sides are the same QEMU and both sides have exactly the same 63 idea of what the pc-5.1 machine is. 64 654 - qemu-5.1 -M pc-5.2 -> migrates to -> qemu-5.1 -M pc-5.2 66 67 This combination is not possible as the qemu-5.1 doesn't understand 68 pc-5.2 machine type. So nothing to worry here. 69 70Now it comes the interesting ones, when both QEMU processes are 71different. Notice also that the machine type needs to be pc-5.1, 72because we have the limitation than qemu-5.1 doesn't know pc-5.2. So 73the possible cases are: 74 755 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1 76 77 This migration is known as newer to older. We need to make sure 78 when we are developing 5.2 we need to take care about not to break 79 migration to qemu-5.1. Notice that we can't make updates to 80 qemu-5.1 to understand whatever qemu-5.2 decides to change, so it is 81 in qemu-5.2 side to make the relevant changes. 82 836 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 84 85 This migration is known as older to newer. We need to make sure 86 than we are able to receive migrations from qemu-5.1. The problem is 87 similar to the previous one. 88 89If qemu-5.1 and qemu-5.2 were the same, there will not be any 90compatibility problems. But the reason that we create qemu-5.2 is to 91get new features, devices, defaults, etc. 92 93If we get a device that has a new feature, or change a default value, 94we have a problem when we try to migrate between different QEMU 95versions. 96 97So we need a way to tell qemu-5.2 that when we are using machine type 98pc-5.1, it needs to **not** use the feature, to be able to migrate to 99real qemu-5.1. 100 101And the equivalent part when migrating from qemu-5.1 to qemu-5.2. 102qemu-5.2 has to expect that it is not going to get data for the new 103feature, because qemu-5.1 doesn't know about it. 104 105How do we tell QEMU about these device feature changes? In 106hw/core/machine.c:hw_compat_X_Y arrays. 107 108If we change a default value, we need to put back the old value on 109that array. And the device, during initialization needs to look at 110that array to see what value it needs to get for that feature. And 111what are we going to put in that array, the value of a property. 112 113To create a property for a device, we need to use one of the 114DEFINE_PROP_*() macros. See include/hw/qdev-properties.h to find the 115macros that exist. With it, we set the default value for that 116property, and that is what it is going to get in the latest released 117version. But if we want a different value for a previous version, we 118can change that in the hw_compat_X_Y arrays. 119 120hw_compat_X_Y is an array of registers that have the format: 121 122- name_device 123- name_property 124- value 125 126Let's see a practical example. 127 128In qemu-5.2 virtio-blk-device got multi queue support. This is a 129change that is not backward compatible. In qemu-5.1 it has one 130queue. In qemu-5.2 it has the same number of queues as the number of 131cpus in the system. 132 133When we are doing migration, if we migrate from a device that has 4 134queues to a device that have only one queue, we don't know where to 135put the extra information for the other 3 queues, and we fail 136migration. 137 138Similar problem when we migrate from qemu-5.1 that has only one queue 139to qemu-5.2, we only sent information for one queue, but destination 140has 4, and we have 3 queues that are not properly initialized and 141anything can happen. 142 143So, how can we address this problem. Easy, just convince qemu-5.2 144that when it is running pc-5.1, it needs to set the number of queues 145for virtio-blk-devices to 1. 146 147That way we fix the cases 5 and 6. 148 1495 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.1 -M pc-5.1 150 151 qemu-5.2 -M pc-5.1 sets number of queues to be 1. 152 qemu-5.1 -M pc-5.1 expects number of queues to be 1. 153 154 correct. migration works. 155 1566 - qemu-5.1 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 157 158 qemu-5.1 -M pc-5.1 sets number of queues to be 1. 159 qemu-5.2 -M pc-5.1 expects number of queues to be 1. 160 161 correct. migration works. 162 163And now the other interesting case, case 3. In this case we have: 164 1653 - qemu-5.2 -M pc-5.1 -> migrates to -> qemu-5.2 -M pc-5.1 166 167 Here we have the same QEMU in both sides. So it doesn't matter a 168 lot if we have set the number of queues to 1 or not, because 169 they are the same. 170 171 WRONG! 172 173 Think what happens if we do one of this double migrations: 174 175 A -> migrates -> B -> migrates -> C 176 177 where: 178 179 A: qemu-5.1 -M pc-5.1 180 B: qemu-5.2 -M pc-5.1 181 C: qemu-5.2 -M pc-5.1 182 183 migration A -> B is case 6, so number of queues needs to be 1. 184 185 migration B -> C is case 3, so we don't care. But actually we 186 care because we haven't started the guest in qemu-5.2, it came 187 migrated from qemu-5.1. So to be in the safe place, we need to 188 always use number of queues 1 when we are using pc-5.1. 189 190Now, how was this done in reality? The following commit shows how it 191was done:: 192 193 commit 9445e1e15e66c19e42bea942ba810db28052cd05 194 Author: Stefan Hajnoczi <stefanha@redhat.com> 195 Date: Tue Aug 18 15:33:47 2020 +0100 196 197 virtio-blk-pci: default num_queues to -smp N 198 199The relevant parts for migration are:: 200 201 @@ -1281,7 +1284,8 @@ static Property virtio_blk_properties[] = { 202 #endif 203 DEFINE_PROP_BIT("request-merging", VirtIOBlock, conf.request_merging, 0, 204 true), 205 - DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 1), 206 + DEFINE_PROP_UINT16("num-queues", VirtIOBlock, conf.num_queues, 207 + VIRTIO_BLK_AUTO_NUM_QUEUES), 208 DEFINE_PROP_UINT16("queue-size", VirtIOBlock, conf.queue_size, 256), 209 210It changes the default value of num_queues. But it fishes it for old 211machine types to have the right value:: 212 213 @@ -31,6 +31,7 @@ 214 GlobalProperty hw_compat_5_1[] = { 215 ... 216 + { "virtio-blk-device", "num-queues", "1"}, 217 ... 218 }; 219 220A device with different features on both sides 221---------------------------------------------- 222 223Let's assume that we are using the same QEMU binary on both sides, 224just to make the things easier. But we have a device that has 225different features on both sides of the migration. That can be 226because the devices are different, because the kernel driver of both 227devices have different features, whatever. 228 229How can we get this to work with migration. The way to do that is 230"theoretically" easy. You have to get the features that the device 231has in the source of the migration. The features that the device has 232on the target of the migration, you get the intersection of the 233features of both sides, and that is the way that you should launch 234QEMU. 235 236Notice that this is not completely related to QEMU. The most 237important thing here is that this should be handled by the managing 238application that launches QEMU. If QEMU is configured correctly, the 239migration will succeed. 240 241That said, actually doing it is complicated. Almost all devices are 242bad at being able to be launched with only some features enabled. 243With one big exception: cpus. 244 245You can read the documentation for QEMU x86 cpu models here: 246 247https://qemu-project.gitlab.io/qemu/system/qemu-cpu-models.html 248 249See when they talk about migration they recommend that one chooses the 250newest cpu model that is supported for all cpus. 251 252Let's say that we have: 253 254Host A: 255 256Device X has the feature Y 257 258Host B: 259 260Device X has not the feature Y 261 262If we try to migrate without any care from host A to host B, it will 263fail because when migration tries to load the feature Y on 264destination, it will find that the hardware is not there. 265 266Doing this would be the equivalent of doing with cpus: 267 268Host A: 269 270$ qemu-system-x86_64 -cpu host 271 272Host B: 273 274$ qemu-system-x86_64 -cpu host 275 276When both hosts have different cpu features this is guaranteed to 277fail. Especially if Host B has less features than host A. If host A 278has less features than host B, sometimes it works. Important word of 279last sentence is "sometimes". 280 281So, forgetting about cpu models and continuing with the -cpu host 282example, let's see that the differences of the cpus is that Host A and 283B have the following features: 284 285Features: 'pcid' 'stibp' 'taa-no' 286Host A: X X 287Host B: X 288 289And we want to migrate between them, the way configure both QEMU cpu 290will be: 291 292Host A: 293 294$ qemu-system-x86_64 -cpu host,pcid=off,stibp=off 295 296Host B: 297 298$ qemu-system-x86_64 -cpu host,taa-no=off 299 300And you would be able to migrate between them. It is responsibility 301of the management application or of the user to make sure that the 302configuration is correct. QEMU doesn't know how to look at this kind 303of features in general. 304 305Notice that we don't recommend to use -cpu host for migration. It is 306used in this example because it makes the example simpler. 307 308Other devices have worse control about individual features. If they 309want to be able to migrate between hosts that show different features, 310the device needs a way to configure which ones it is going to use. 311 312In this section we have considered that we are using the same QEMU 313binary in both sides of the migration. If we use different QEMU 314versions process, then we need to have into account all other 315differences and the examples become even more complicated. 316 317How to mitigate when we have a backward compatibility error 318----------------------------------------------------------- 319 320We broke migration for old machine types continuously during 321development. But as soon as we find that there is a problem, we fix 322it. The problem is what happens when we detect after we have done a 323release that something has gone wrong. 324 325Let see how it worked with one example. 326 327After the release of qemu-8.0 we found a problem when doing migration 328of the machine type pc-7.2. 329 330- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2 331 332 This migration works 333 334- $ qemu-8.0 -M pc-7.2 -> qemu-8.0 -M pc-7.2 335 336 This migration works 337 338- $ qemu-8.0 -M pc-7.2 -> qemu-7.2 -M pc-7.2 339 340 This migration fails 341 342- $ qemu-7.2 -M pc-7.2 -> qemu-8.0 -M pc-7.2 343 344 This migration fails 345 346So clearly something fails when migration between qemu-7.2 and 347qemu-8.0 with machine type pc-7.2. The error messages, and git bisect 348pointed to this commit. 349 350In qemu-8.0 we got this commit:: 351 352 commit 010746ae1db7f52700cb2e2c46eb94f299cfa0d2 353 Author: Jonathan Cameron <Jonathan.Cameron@huawei.com> 354 Date: Thu Mar 2 13:37:02 2023 +0000 355 356 hw/pci/aer: Implement PCI_ERR_UNCOR_MASK register 357 358 359The relevant bits of the commit for our example are this ones:: 360 361 --- a/hw/pci/pcie_aer.c 362 +++ b/hw/pci/pcie_aer.c 363 @@ -112,6 +112,10 @@ int pcie_aer_init(PCIDevice *dev, 364 365 pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS, 366 PCI_ERR_UNC_SUPPORTED); 367 + pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK, 368 + PCI_ERR_UNC_MASK_DEFAULT); 369 + pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK, 370 + PCI_ERR_UNC_SUPPORTED); 371 372 pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER, 373 PCI_ERR_UNC_SEVERITY_DEFAULT); 374 375The patch changes how we configure PCI space for AER. But QEMU fails 376when the PCI space configuration is different between source and 377destination. 378 379The following commit shows how this got fixed:: 380 381 commit 5ed3dabe57dd9f4c007404345e5f5bf0e347317f 382 Author: Leonardo Bras <leobras@redhat.com> 383 Date: Tue May 2 21:27:02 2023 -0300 384 385 hw/pci: Disable PCI_ERR_UNCOR_MASK register for machine type < 8.0 386 387 [...] 388 389The relevant parts of the fix in QEMU are as follow: 390 391First, we create a new property for the device to be able to configure 392the old behaviour or the new behaviour:: 393 394 diff --git a/hw/pci/pci.c b/hw/pci/pci.c 395 index 8a87ccc8b0..5153ad63d6 100644 396 --- a/hw/pci/pci.c 397 +++ b/hw/pci/pci.c 398 @@ -79,6 +79,8 @@ static Property pci_props[] = { 399 DEFINE_PROP_STRING("failover_pair_id", PCIDevice, 400 failover_pair_id), 401 DEFINE_PROP_UINT32("acpi-index", PCIDevice, acpi_index, 0), 402 + DEFINE_PROP_BIT("x-pcie-err-unc-mask", PCIDevice, cap_present, 403 + QEMU_PCIE_ERR_UNC_MASK_BITNR, true), 404 DEFINE_PROP_END_OF_LIST() 405 }; 406 407Notice that we enable the feature for new machine types. 408 409Now we see how the fix is done. This is going to depend on what kind 410of breakage happens, but in this case it is quite simple:: 411 412 diff --git a/hw/pci/pcie_aer.c b/hw/pci/pcie_aer.c 413 index 103667c368..374d593ead 100644 414 --- a/hw/pci/pcie_aer.c 415 +++ b/hw/pci/pcie_aer.c 416 @@ -112,10 +112,13 @@ int pcie_aer_init(PCIDevice *dev, uint8_t cap_ver, 417 uint16_t offset, 418 419 pci_set_long(dev->w1cmask + offset + PCI_ERR_UNCOR_STATUS, 420 PCI_ERR_UNC_SUPPORTED); 421 - pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK, 422 - PCI_ERR_UNC_MASK_DEFAULT); 423 - pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK, 424 - PCI_ERR_UNC_SUPPORTED); 425 + 426 + if (dev->cap_present & QEMU_PCIE_ERR_UNC_MASK) { 427 + pci_set_long(dev->config + offset + PCI_ERR_UNCOR_MASK, 428 + PCI_ERR_UNC_MASK_DEFAULT); 429 + pci_set_long(dev->wmask + offset + PCI_ERR_UNCOR_MASK, 430 + PCI_ERR_UNC_SUPPORTED); 431 + } 432 433 pci_set_long(dev->config + offset + PCI_ERR_UNCOR_SEVER, 434 PCI_ERR_UNC_SEVERITY_DEFAULT); 435 436I.e. If the property bit is enabled, we configure it as we did for 437qemu-8.0. If the property bit is not set, we configure it as it was in 7.2. 438 439And now, everything that is missing is disabling the feature for old 440machine types:: 441 442 diff --git a/hw/core/machine.c b/hw/core/machine.c 443 index 47a34841a5..07f763eb2e 100644 444 --- a/hw/core/machine.c 445 +++ b/hw/core/machine.c 446 @@ -48,6 +48,7 @@ GlobalProperty hw_compat_7_2[] = { 447 { "e1000e", "migrate-timadj", "off" }, 448 { "virtio-mem", "x-early-migration", "false" }, 449 { "migration", "x-preempt-pre-7-2", "true" }, 450 + { TYPE_PCI_DEVICE, "x-pcie-err-unc-mask", "off" }, 451 }; 452 const size_t hw_compat_7_2_len = G_N_ELEMENTS(hw_compat_7_2); 453 454And now, when qemu-8.0.1 is released with this fix, all combinations 455are going to work as supposed. 456 457- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2 (works) 458- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 (works) 459- $ qemu-8.0.1 -M pc-7.2 -> qemu-7.2 -M pc-7.2 (works) 460- $ qemu-7.2 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 (works) 461 462So the normality has been restored and everything is ok, no? 463 464Not really, now our matrix is much bigger. We started with the easy 465cases, migration from the same version to the same version always 466works: 467 468- $ qemu-7.2 -M pc-7.2 -> qemu-7.2 -M pc-7.2 469- $ qemu-8.0 -M pc-7.2 -> qemu-8.0 -M pc-7.2 470- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 471 472Now the interesting ones. When the QEMU processes versions are 473different. For the 1st set, their fail and we can do nothing, both 474versions are released and we can't change anything. 475 476- $ qemu-7.2 -M pc-7.2 -> qemu-8.0 -M pc-7.2 477- $ qemu-8.0 -M pc-7.2 -> qemu-7.2 -M pc-7.2 478 479This two are the ones that work. The whole point of making the 480change in qemu-8.0.1 release was to fix this issue: 481 482- $ qemu-7.2 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 483- $ qemu-8.0.1 -M pc-7.2 -> qemu-7.2 -M pc-7.2 484 485But now we found that qemu-8.0 neither can migrate to qemu-7.2 not 486qemu-8.0.1. 487 488- $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 489- $ qemu-8.0.1 -M pc-7.2 -> qemu-8.0 -M pc-7.2 490 491So, if we start a pc-7.2 machine in qemu-8.0 we can't migrate it to 492anything except to qemu-8.0. 493 494Can we do better? 495 496Yeap. If we know that we are going to do this migration: 497 498- $ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 499 500We can launch the appropriate devices with:: 501 502 --device...,x-pci-e-err-unc-mask=on 503 504And now we can receive a migration from 8.0. And from now on, we can 505do that migration to new machine types if we remember to enable that 506property for pc-7.2. Notice that we need to remember, it is not 507enough to know that the source of the migration is qemu-8.0. Think of 508this example: 509 510$ qemu-8.0 -M pc-7.2 -> qemu-8.0.1 -M pc-7.2 -> qemu-8.2 -M pc-7.2 511 512In the second migration, the source is not qemu-8.0, but we still have 513that "problem" and have that property enabled. Notice that we need to 514continue having this mark/property until we have this machine 515rebooted. But it is not a normal reboot (that don't reload QEMU) we 516need the machine to poweroff/poweron on a fixed QEMU. And from now 517on we can use the proper real machine. 518