1*06320cedSSean V Kelley.. SPDX-License-Identifier: GPL-2.0 2*06320cedSSean V Kelley 3*06320cedSSean V Kelley=============== 4*06320cedSSean V KelleyBoot Interrupts 5*06320cedSSean V Kelley=============== 6*06320cedSSean V Kelley 7*06320cedSSean V Kelley:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com> 8*06320cedSSean V Kelley 9*06320cedSSean V KelleyOverview 10*06320cedSSean V Kelley======== 11*06320cedSSean V Kelley 12*06320cedSSean V KelleyOn PCI Express, interrupts are represented with either MSI or inbound 13*06320cedSSean V Kelleyinterrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a 14*06320cedSSean V Kelleygiven Core IO converts the legacy interrupt messages from PCI Express to 15*06320cedSSean V KelleyMSI interrupts. If the IO-APIC is disabled (via the mask bits in the 16*06320cedSSean V KelleyIO-APIC table entries), the messages are routed to the legacy PCH. This 17*06320cedSSean V Kelleyin-band interrupt mechanism was traditionally necessary for systems that 18*06320cedSSean V Kelleydid not support the IO-APIC and for boot. Intel in the past has used the 19*06320cedSSean V Kelleyterm "boot interrupts" to describe this mechanism. Further, the PCI Express 20*06320cedSSean V Kelleyprotocol describes this in-band legacy wire-interrupt INTx mechanism for 21*06320cedSSean V KelleyI/O devices to signal PCI-style level interrupts. The subsequent paragraphs 22*06320cedSSean V Kelleydescribe problems with the Core IO handling of INTx message routing to the 23*06320cedSSean V KelleyPCH and mitigation within BIOS and the OS. 24*06320cedSSean V Kelley 25*06320cedSSean V Kelley 26*06320cedSSean V KelleyIssue 27*06320cedSSean V Kelley===== 28*06320cedSSean V Kelley 29*06320cedSSean V KelleyWhen in-band legacy INTx messages are forwarded to the PCH, they in turn 30*06320cedSSean V Kelleytrigger a new interrupt for which the OS likely lacks a handler. When an 31*06320cedSSean V Kelleyinterrupt goes unhandled over time, they are tracked by the Linux kernel as 32*06320cedSSean V KelleySpurious Interrupts. The IRQ will be disabled by the Linux kernel after it 33*06320cedSSean V Kelleyreaches a specific count with the error "nobody cared". This disabled IRQ 34*06320cedSSean V Kelleynow prevents valid usage by an existing interrupt which may happen to share 35*06320cedSSean V Kelleythe IRQ line. 36*06320cedSSean V Kelley 37*06320cedSSean V Kelley irq 19: nobody cared (try booting with the "irqpoll" option) 38*06320cedSSean V Kelley CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1 39*06320cedSSean V Kelley Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020 40*06320cedSSean V Kelley Call Trace: 41*06320cedSSean V Kelley <IRQ> 42*06320cedSSean V Kelley ? dump_stack+0x46/0x5e 43*06320cedSSean V Kelley ? __report_bad_irq+0x2e/0xb0 44*06320cedSSean V Kelley ? note_interrupt+0x242/0x290 45*06320cedSSean V Kelley ? nNIKAL100_memoryRead16+0x8/0x10 [nikal] 46*06320cedSSean V Kelley ? handle_irq_event_percpu+0x55/0x70 47*06320cedSSean V Kelley ? handle_irq_event+0x4f/0x80 48*06320cedSSean V Kelley ? handle_fasteoi_irq+0x81/0x180 49*06320cedSSean V Kelley ? handle_irq+0x1c/0x30 50*06320cedSSean V Kelley ? do_IRQ+0x41/0xd0 51*06320cedSSean V Kelley ? common_interrupt+0x84/0x84 52*06320cedSSean V Kelley </IRQ> 53*06320cedSSean V Kelley 54*06320cedSSean V Kelley handlers: 55*06320cedSSean V Kelley irq_default_primary_handler threaded usb_hcd_irq 56*06320cedSSean V Kelley Disabling IRQ #19 57*06320cedSSean V Kelley 58*06320cedSSean V Kelley 59*06320cedSSean V KelleyConditions 60*06320cedSSean V Kelley========== 61*06320cedSSean V Kelley 62*06320cedSSean V KelleyThe use of threaded interrupts is the most likely condition to trigger 63*06320cedSSean V Kelleythis problem today. Threaded interrupts may not be reenabled after the IRQ 64*06320cedSSean V Kelleyhandler wakes. These "one shot" conditions mean that the threaded interrupt 65*06320cedSSean V Kelleyneeds to keep the interrupt line masked until the threaded handler has run. 66*06320cedSSean V KelleyEspecially when dealing with high data rate interrupts, the thread needs to 67*06320cedSSean V Kelleyrun to completion; otherwise some handlers will end up in stack overflows 68*06320cedSSean V Kelleysince the interrupt of the issuing device is still active. 69*06320cedSSean V Kelley 70*06320cedSSean V KelleyAffected Chipsets 71*06320cedSSean V Kelley================= 72*06320cedSSean V Kelley 73*06320cedSSean V KelleyThe legacy interrupt forwarding mechanism exists today in a number of 74*06320cedSSean V Kelleydevices including but not limited to chipsets from AMD/ATI, Broadcom, and 75*06320cedSSean V KelleyIntel. Changes made through the mitigations below have been applied to 76*06320cedSSean V Kelleydrivers/pci/quirks.c 77*06320cedSSean V Kelley 78*06320cedSSean V KelleyStarting with ICX there are no longer any IO-APICs in the Core IO's 79*06320cedSSean V Kelleydevices. IO-APIC is only in the PCH. Devices connected to the Core IO's 80*06320cedSSean V KelleyPCIe Root Ports will use native MSI/MSI-X mechanisms. 81*06320cedSSean V Kelley 82*06320cedSSean V KelleyMitigations 83*06320cedSSean V Kelley=========== 84*06320cedSSean V Kelley 85*06320cedSSean V KelleyThe mitigations take the form of PCI quirks. The preference has been to 86*06320cedSSean V Kelleyfirst identify and make use of a means to disable the routing to the PCH. 87*06320cedSSean V KelleyIn such a case a quirk to disable boot interrupt generation can be 88*06320cedSSean V Kelleyadded.[1] 89*06320cedSSean V Kelley 90*06320cedSSean V Kelley Intel® 6300ESB I/O Controller Hub 91*06320cedSSean V Kelley Alternate Base Address Register: 92*06320cedSSean V Kelley BIE: Boot Interrupt Enable 93*06320cedSSean V Kelley 0 = Boot interrupt is enabled. 94*06320cedSSean V Kelley 1 = Boot interrupt is disabled. 95*06320cedSSean V Kelley 96*06320cedSSean V Kelley Intel® Sandy Bridge through Sky Lake based Xeon servers: 97*06320cedSSean V Kelley Coherent Interface Protocol Interrupt Control 98*06320cedSSean V Kelley dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2: 99*06320cedSSean V Kelley When this bit is set. Local INTx messages received from the 100*06320cedSSean V Kelley Intel® Quick Data DMA/PCI Express ports are not routed to legacy 101*06320cedSSean V Kelley PCH - they are either converted into MSI via the integrated IO-APIC 102*06320cedSSean V Kelley (if the IO-APIC mask bit is clear in the appropriate entries) 103*06320cedSSean V Kelley or cause no further action (when mask bit is set) 104*06320cedSSean V Kelley 105*06320cedSSean V KelleyIn the absence of a way to directly disable the routing, another approach 106*06320cedSSean V Kelleyhas been to make use of PCI Interrupt pin to INTx routing tables for 107*06320cedSSean V Kelleypurposes of redirecting the interrupt handler to the rerouted interrupt 108*06320cedSSean V Kelleyline by default. Therefore, on chipsets where this INTx routing cannot be 109*06320cedSSean V Kelleydisabled, the Linux kernel will reroute the valid interrupt to its legacy 110*06320cedSSean V Kelleyinterrupt. This redirection of the handler will prevent the occurrence of 111*06320cedSSean V Kelleythe spurious interrupt detection which would ordinarily disable the IRQ 112*06320cedSSean V Kelleyline due to excessive unhandled counts.[2] 113*06320cedSSean V Kelley 114*06320cedSSean V KelleyThe config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or 115*06320cedSSean V Kelleydisable) the redirection of the interrupt handler to the PCH interrupt 116*06320cedSSean V Kelleyline. The option can be overridden by either pci=ioapicreroute or 117*06320cedSSean V Kelleypci=noioapicreroute.[3] 118*06320cedSSean V Kelley 119*06320cedSSean V Kelley 120*06320cedSSean V KelleyMore Documentation 121*06320cedSSean V Kelley================== 122*06320cedSSean V Kelley 123*06320cedSSean V KelleyThere is an overview of the legacy interrupt handling in several datasheets 124*06320cedSSean V Kelley(6300ESB and 6700PXH below). While largely the same, it provides insight 125*06320cedSSean V Kelleyinto the evolution of its handling with chipsets. 126*06320cedSSean V Kelley 127*06320cedSSean V KelleyExample of disabling of the boot interrupt 128*06320cedSSean V Kelley------------------------------------------ 129*06320cedSSean V Kelley 130*06320cedSSean V KelleyIntel® 6300ESB I/O Controller Hub (Document # 300641-004US) 131*06320cedSSean V Kelley 5.7.3 Boot Interrupt 132*06320cedSSean V Kelley https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf 133*06320cedSSean V Kelley 134*06320cedSSean V KelleyIntel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families 135*06320cedSSean V KelleyDatasheet - Volume 2: Registers (Document # 330784-003) 136*06320cedSSean V Kelley 6.6.41 cipintrc Coherent Interface Protocol Interrupt Control 137*06320cedSSean V Kelley https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf 138*06320cedSSean V Kelley 139*06320cedSSean V KelleyExample of handler rerouting 140*06320cedSSean V Kelley---------------------------- 141*06320cedSSean V Kelley 142*06320cedSSean V KelleyIntel® 6700PXH 64-bit PCI Hub (Document # 302628) 143*06320cedSSean V Kelley 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt 144*06320cedSSean V Kelley https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf 145*06320cedSSean V Kelley 146*06320cedSSean V Kelley 147*06320cedSSean V KelleyIf you have any legacy PCI interrupt questions that aren't answered, email me. 148*06320cedSSean V Kelley 149*06320cedSSean V KelleyCheers, 150*06320cedSSean V Kelley Sean V Kelley 151*06320cedSSean V Kelley sean.v.kelley@linux.intel.com 152*06320cedSSean V Kelley 153*06320cedSSean V Kelley[1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/ 154*06320cedSSean V Kelley[2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/ 155*06320cedSSean V Kelley[3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/ 156