106320cedSSean V Kelley.. SPDX-License-Identifier: GPL-2.0 206320cedSSean V Kelley 306320cedSSean V Kelley=============== 406320cedSSean V KelleyBoot Interrupts 506320cedSSean V Kelley=============== 606320cedSSean V Kelley 706320cedSSean V Kelley:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com> 806320cedSSean V Kelley 906320cedSSean V KelleyOverview 1006320cedSSean V Kelley======== 1106320cedSSean V Kelley 1206320cedSSean V KelleyOn PCI Express, interrupts are represented with either MSI or inbound 1306320cedSSean V Kelleyinterrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a 1406320cedSSean V Kelleygiven Core IO converts the legacy interrupt messages from PCI Express to 1506320cedSSean V KelleyMSI interrupts. If the IO-APIC is disabled (via the mask bits in the 1606320cedSSean V KelleyIO-APIC table entries), the messages are routed to the legacy PCH. This 1706320cedSSean V Kelleyin-band interrupt mechanism was traditionally necessary for systems that 1806320cedSSean V Kelleydid not support the IO-APIC and for boot. Intel in the past has used the 1906320cedSSean V Kelleyterm "boot interrupts" to describe this mechanism. Further, the PCI Express 2006320cedSSean V Kelleyprotocol describes this in-band legacy wire-interrupt INTx mechanism for 2106320cedSSean V KelleyI/O devices to signal PCI-style level interrupts. The subsequent paragraphs 2206320cedSSean V Kelleydescribe problems with the Core IO handling of INTx message routing to the 2306320cedSSean V KelleyPCH and mitigation within BIOS and the OS. 2406320cedSSean V Kelley 2506320cedSSean V Kelley 2606320cedSSean V KelleyIssue 2706320cedSSean V Kelley===== 2806320cedSSean V Kelley 2906320cedSSean V KelleyWhen in-band legacy INTx messages are forwarded to the PCH, they in turn 3006320cedSSean V Kelleytrigger a new interrupt for which the OS likely lacks a handler. When an 3106320cedSSean V Kelleyinterrupt goes unhandled over time, they are tracked by the Linux kernel as 3206320cedSSean V KelleySpurious Interrupts. The IRQ will be disabled by the Linux kernel after it 3306320cedSSean V Kelleyreaches a specific count with the error "nobody cared". This disabled IRQ 3406320cedSSean V Kelleynow prevents valid usage by an existing interrupt which may happen to share 35*a588332fSMauro Carvalho Chehabthe IRQ line:: 3606320cedSSean V Kelley 3706320cedSSean V Kelley irq 19: nobody cared (try booting with the "irqpoll" option) 3806320cedSSean V Kelley CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1 3906320cedSSean V Kelley Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020 4006320cedSSean V Kelley Call Trace: 41*a588332fSMauro Carvalho Chehab 4206320cedSSean V Kelley <IRQ> 4306320cedSSean V Kelley ? dump_stack+0x46/0x5e 4406320cedSSean V Kelley ? __report_bad_irq+0x2e/0xb0 4506320cedSSean V Kelley ? note_interrupt+0x242/0x290 4606320cedSSean V Kelley ? nNIKAL100_memoryRead16+0x8/0x10 [nikal] 4706320cedSSean V Kelley ? handle_irq_event_percpu+0x55/0x70 4806320cedSSean V Kelley ? handle_irq_event+0x4f/0x80 4906320cedSSean V Kelley ? handle_fasteoi_irq+0x81/0x180 5006320cedSSean V Kelley ? handle_irq+0x1c/0x30 5106320cedSSean V Kelley ? do_IRQ+0x41/0xd0 5206320cedSSean V Kelley ? common_interrupt+0x84/0x84 5306320cedSSean V Kelley </IRQ> 5406320cedSSean V Kelley 5506320cedSSean V Kelley handlers: 5606320cedSSean V Kelley irq_default_primary_handler threaded usb_hcd_irq 5706320cedSSean V Kelley Disabling IRQ #19 5806320cedSSean V Kelley 5906320cedSSean V Kelley 6006320cedSSean V KelleyConditions 6106320cedSSean V Kelley========== 6206320cedSSean V Kelley 6306320cedSSean V KelleyThe use of threaded interrupts is the most likely condition to trigger 6406320cedSSean V Kelleythis problem today. Threaded interrupts may not be reenabled after the IRQ 6506320cedSSean V Kelleyhandler wakes. These "one shot" conditions mean that the threaded interrupt 6606320cedSSean V Kelleyneeds to keep the interrupt line masked until the threaded handler has run. 6706320cedSSean V KelleyEspecially when dealing with high data rate interrupts, the thread needs to 6806320cedSSean V Kelleyrun to completion; otherwise some handlers will end up in stack overflows 6906320cedSSean V Kelleysince the interrupt of the issuing device is still active. 7006320cedSSean V Kelley 7106320cedSSean V KelleyAffected Chipsets 7206320cedSSean V Kelley================= 7306320cedSSean V Kelley 7406320cedSSean V KelleyThe legacy interrupt forwarding mechanism exists today in a number of 7506320cedSSean V Kelleydevices including but not limited to chipsets from AMD/ATI, Broadcom, and 7606320cedSSean V KelleyIntel. Changes made through the mitigations below have been applied to 7706320cedSSean V Kelleydrivers/pci/quirks.c 7806320cedSSean V Kelley 7906320cedSSean V KelleyStarting with ICX there are no longer any IO-APICs in the Core IO's 8006320cedSSean V Kelleydevices. IO-APIC is only in the PCH. Devices connected to the Core IO's 8106320cedSSean V KelleyPCIe Root Ports will use native MSI/MSI-X mechanisms. 8206320cedSSean V Kelley 8306320cedSSean V KelleyMitigations 8406320cedSSean V Kelley=========== 8506320cedSSean V Kelley 8606320cedSSean V KelleyThe mitigations take the form of PCI quirks. The preference has been to 8706320cedSSean V Kelleyfirst identify and make use of a means to disable the routing to the PCH. 8806320cedSSean V KelleyIn such a case a quirk to disable boot interrupt generation can be 89*a588332fSMauro Carvalho Chehabadded. [1]_ 9006320cedSSean V Kelley 9106320cedSSean V KelleyIntel® 6300ESB I/O Controller Hub 9206320cedSSean V Kelley Alternate Base Address Register: 9306320cedSSean V Kelley BIE: Boot Interrupt Enable 94*a588332fSMauro Carvalho Chehab 95*a588332fSMauro Carvalho Chehab == =========================== 96*a588332fSMauro Carvalho Chehab 0 Boot interrupt is enabled. 97*a588332fSMauro Carvalho Chehab 1 Boot interrupt is disabled. 98*a588332fSMauro Carvalho Chehab == =========================== 9906320cedSSean V Kelley 10006320cedSSean V KelleyIntel® Sandy Bridge through Sky Lake based Xeon servers: 10106320cedSSean V Kelley Coherent Interface Protocol Interrupt Control 10206320cedSSean V Kelley dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2: 10306320cedSSean V Kelley When this bit is set. Local INTx messages received from the 10406320cedSSean V Kelley Intel® Quick Data DMA/PCI Express ports are not routed to legacy 10506320cedSSean V Kelley PCH - they are either converted into MSI via the integrated IO-APIC 10606320cedSSean V Kelley (if the IO-APIC mask bit is clear in the appropriate entries) 10706320cedSSean V Kelley or cause no further action (when mask bit is set) 10806320cedSSean V Kelley 10906320cedSSean V KelleyIn the absence of a way to directly disable the routing, another approach 11006320cedSSean V Kelleyhas been to make use of PCI Interrupt pin to INTx routing tables for 11106320cedSSean V Kelleypurposes of redirecting the interrupt handler to the rerouted interrupt 11206320cedSSean V Kelleyline by default. Therefore, on chipsets where this INTx routing cannot be 11306320cedSSean V Kelleydisabled, the Linux kernel will reroute the valid interrupt to its legacy 11406320cedSSean V Kelleyinterrupt. This redirection of the handler will prevent the occurrence of 11506320cedSSean V Kelleythe spurious interrupt detection which would ordinarily disable the IRQ 116*a588332fSMauro Carvalho Chehabline due to excessive unhandled counts. [2]_ 11706320cedSSean V Kelley 11806320cedSSean V KelleyThe config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or 11906320cedSSean V Kelleydisable) the redirection of the interrupt handler to the PCH interrupt 12006320cedSSean V Kelleyline. The option can be overridden by either pci=ioapicreroute or 121*a588332fSMauro Carvalho Chehabpci=noioapicreroute. [3]_ 12206320cedSSean V Kelley 12306320cedSSean V Kelley 12406320cedSSean V KelleyMore Documentation 12506320cedSSean V Kelley================== 12606320cedSSean V Kelley 12706320cedSSean V KelleyThere is an overview of the legacy interrupt handling in several datasheets 12806320cedSSean V Kelley(6300ESB and 6700PXH below). While largely the same, it provides insight 12906320cedSSean V Kelleyinto the evolution of its handling with chipsets. 13006320cedSSean V Kelley 13106320cedSSean V KelleyExample of disabling of the boot interrupt 13206320cedSSean V Kelley------------------------------------------ 13306320cedSSean V Kelley 134*a588332fSMauro Carvalho Chehab - Intel® 6300ESB I/O Controller Hub (Document # 300641-004US) 13506320cedSSean V Kelley 5.7.3 Boot Interrupt 13606320cedSSean V Kelley https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf 13706320cedSSean V Kelley 138*a588332fSMauro Carvalho Chehab - Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families 13906320cedSSean V Kelley Datasheet - Volume 2: Registers (Document # 330784-003) 14006320cedSSean V Kelley 6.6.41 cipintrc Coherent Interface Protocol Interrupt Control 14106320cedSSean V Kelley https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf 14206320cedSSean V Kelley 14306320cedSSean V KelleyExample of handler rerouting 14406320cedSSean V Kelley---------------------------- 14506320cedSSean V Kelley 146*a588332fSMauro Carvalho Chehab - Intel® 6700PXH 64-bit PCI Hub (Document # 302628) 14706320cedSSean V Kelley 2.15.2 PCI Express Legacy INTx Support and Boot Interrupt 14806320cedSSean V Kelley https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf 14906320cedSSean V Kelley 15006320cedSSean V Kelley 15106320cedSSean V KelleyIf you have any legacy PCI interrupt questions that aren't answered, email me. 15206320cedSSean V Kelley 15306320cedSSean V KelleyCheers, 15406320cedSSean V Kelley Sean V Kelley 15506320cedSSean V Kelley sean.v.kelley@linux.intel.com 15606320cedSSean V Kelley 157*a588332fSMauro Carvalho Chehab.. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/ 158*a588332fSMauro Carvalho Chehab.. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/ 159*a588332fSMauro Carvalho Chehab.. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/ 160