xref: /openbmc/linux/Documentation/PCI/boot-interrupts.rst (revision 4b4193256c8d3bc3a5397b5cd9494c2ad386317d)
106320cedSSean V Kelley.. SPDX-License-Identifier: GPL-2.0
206320cedSSean V Kelley
306320cedSSean V Kelley===============
406320cedSSean V KelleyBoot Interrupts
506320cedSSean V Kelley===============
606320cedSSean V Kelley
706320cedSSean V Kelley:Author: - Sean V Kelley <sean.v.kelley@linux.intel.com>
806320cedSSean V Kelley
906320cedSSean V KelleyOverview
1006320cedSSean V Kelley========
1106320cedSSean V Kelley
1206320cedSSean V KelleyOn PCI Express, interrupts are represented with either MSI or inbound
1306320cedSSean V Kelleyinterrupt messages (Assert_INTx/Deassert_INTx). The integrated IO-APIC in a
1406320cedSSean V Kelleygiven Core IO converts the legacy interrupt messages from PCI Express to
1506320cedSSean V KelleyMSI interrupts.  If the IO-APIC is disabled (via the mask bits in the
1606320cedSSean V KelleyIO-APIC table entries), the messages are routed to the legacy PCH. This
1706320cedSSean V Kelleyin-band interrupt mechanism was traditionally necessary for systems that
1806320cedSSean V Kelleydid not support the IO-APIC and for boot. Intel in the past has used the
1906320cedSSean V Kelleyterm "boot interrupts" to describe this mechanism. Further, the PCI Express
2006320cedSSean V Kelleyprotocol describes this in-band legacy wire-interrupt INTx mechanism for
2106320cedSSean V KelleyI/O devices to signal PCI-style level interrupts. The subsequent paragraphs
2206320cedSSean V Kelleydescribe problems with the Core IO handling of INTx message routing to the
2306320cedSSean V KelleyPCH and mitigation within BIOS and the OS.
2406320cedSSean V Kelley
2506320cedSSean V Kelley
2606320cedSSean V KelleyIssue
2706320cedSSean V Kelley=====
2806320cedSSean V Kelley
2906320cedSSean V KelleyWhen in-band legacy INTx messages are forwarded to the PCH, they in turn
3006320cedSSean V Kelleytrigger a new interrupt for which the OS likely lacks a handler. When an
3106320cedSSean V Kelleyinterrupt goes unhandled over time, they are tracked by the Linux kernel as
3206320cedSSean V KelleySpurious Interrupts. The IRQ will be disabled by the Linux kernel after it
3306320cedSSean V Kelleyreaches a specific count with the error "nobody cared". This disabled IRQ
3406320cedSSean V Kelleynow prevents valid usage by an existing interrupt which may happen to share
35*a588332fSMauro Carvalho Chehabthe IRQ line::
3606320cedSSean V Kelley
3706320cedSSean V Kelley  irq 19: nobody cared (try booting with the "irqpoll" option)
3806320cedSSean V Kelley  CPU: 0 PID: 2988 Comm: irq/34-nipalk Tainted: 4.14.87-rt49-02410-g4a640ec-dirty #1
3906320cedSSean V Kelley  Hardware name: National Instruments NI PXIe-8880/NI PXIe-8880, BIOS 2.1.5f1 01/09/2020
4006320cedSSean V Kelley  Call Trace:
41*a588332fSMauro Carvalho Chehab
4206320cedSSean V Kelley  <IRQ>
4306320cedSSean V Kelley   ? dump_stack+0x46/0x5e
4406320cedSSean V Kelley   ? __report_bad_irq+0x2e/0xb0
4506320cedSSean V Kelley   ? note_interrupt+0x242/0x290
4606320cedSSean V Kelley   ? nNIKAL100_memoryRead16+0x8/0x10 [nikal]
4706320cedSSean V Kelley   ? handle_irq_event_percpu+0x55/0x70
4806320cedSSean V Kelley   ? handle_irq_event+0x4f/0x80
4906320cedSSean V Kelley   ? handle_fasteoi_irq+0x81/0x180
5006320cedSSean V Kelley   ? handle_irq+0x1c/0x30
5106320cedSSean V Kelley   ? do_IRQ+0x41/0xd0
5206320cedSSean V Kelley   ? common_interrupt+0x84/0x84
5306320cedSSean V Kelley  </IRQ>
5406320cedSSean V Kelley
5506320cedSSean V Kelley  handlers:
5606320cedSSean V Kelley  irq_default_primary_handler threaded usb_hcd_irq
5706320cedSSean V Kelley  Disabling IRQ #19
5806320cedSSean V Kelley
5906320cedSSean V Kelley
6006320cedSSean V KelleyConditions
6106320cedSSean V Kelley==========
6206320cedSSean V Kelley
6306320cedSSean V KelleyThe use of threaded interrupts is the most likely condition to trigger
6406320cedSSean V Kelleythis problem today. Threaded interrupts may not be reenabled after the IRQ
6506320cedSSean V Kelleyhandler wakes. These "one shot" conditions mean that the threaded interrupt
6606320cedSSean V Kelleyneeds to keep the interrupt line masked until the threaded handler has run.
6706320cedSSean V KelleyEspecially when dealing with high data rate interrupts, the thread needs to
6806320cedSSean V Kelleyrun to completion; otherwise some handlers will end up in stack overflows
6906320cedSSean V Kelleysince the interrupt of the issuing device is still active.
7006320cedSSean V Kelley
7106320cedSSean V KelleyAffected Chipsets
7206320cedSSean V Kelley=================
7306320cedSSean V Kelley
7406320cedSSean V KelleyThe legacy interrupt forwarding mechanism exists today in a number of
7506320cedSSean V Kelleydevices including but not limited to chipsets from AMD/ATI, Broadcom, and
7606320cedSSean V KelleyIntel. Changes made through the mitigations below have been applied to
7706320cedSSean V Kelleydrivers/pci/quirks.c
7806320cedSSean V Kelley
7906320cedSSean V KelleyStarting with ICX there are no longer any IO-APICs in the Core IO's
8006320cedSSean V Kelleydevices.  IO-APIC is only in the PCH.  Devices connected to the Core IO's
8106320cedSSean V KelleyPCIe Root Ports will use native MSI/MSI-X mechanisms.
8206320cedSSean V Kelley
8306320cedSSean V KelleyMitigations
8406320cedSSean V Kelley===========
8506320cedSSean V Kelley
8606320cedSSean V KelleyThe mitigations take the form of PCI quirks. The preference has been to
8706320cedSSean V Kelleyfirst identify and make use of a means to disable the routing to the PCH.
8806320cedSSean V KelleyIn such a case a quirk to disable boot interrupt generation can be
89*a588332fSMauro Carvalho Chehabadded. [1]_
9006320cedSSean V Kelley
9106320cedSSean V KelleyIntel® 6300ESB I/O Controller Hub
9206320cedSSean V Kelley  Alternate Base Address Register:
9306320cedSSean V Kelley   BIE: Boot Interrupt Enable
94*a588332fSMauro Carvalho Chehab
95*a588332fSMauro Carvalho Chehab	  ==  ===========================
96*a588332fSMauro Carvalho Chehab	  0   Boot interrupt is enabled.
97*a588332fSMauro Carvalho Chehab	  1   Boot interrupt is disabled.
98*a588332fSMauro Carvalho Chehab	  ==  ===========================
9906320cedSSean V Kelley
10006320cedSSean V KelleyIntel® Sandy Bridge through Sky Lake based Xeon servers:
10106320cedSSean V Kelley  Coherent Interface Protocol Interrupt Control
10206320cedSSean V Kelley   dis_intx_route2pch/dis_intx_route2ich/dis_intx_route2dmi2:
10306320cedSSean V Kelley	  When this bit is set. Local INTx messages received from the
10406320cedSSean V Kelley	  Intel® Quick Data DMA/PCI Express ports are not routed to legacy
10506320cedSSean V Kelley	  PCH - they are either converted into MSI via the integrated IO-APIC
10606320cedSSean V Kelley	  (if the IO-APIC mask bit is clear in the appropriate entries)
10706320cedSSean V Kelley	  or cause no further action (when mask bit is set)
10806320cedSSean V Kelley
10906320cedSSean V KelleyIn the absence of a way to directly disable the routing, another approach
11006320cedSSean V Kelleyhas been to make use of PCI Interrupt pin to INTx routing tables for
11106320cedSSean V Kelleypurposes of redirecting the interrupt handler to the rerouted interrupt
11206320cedSSean V Kelleyline by default.  Therefore, on chipsets where this INTx routing cannot be
11306320cedSSean V Kelleydisabled, the Linux kernel will reroute the valid interrupt to its legacy
11406320cedSSean V Kelleyinterrupt. This redirection of the handler will prevent the occurrence of
11506320cedSSean V Kelleythe spurious interrupt detection which would ordinarily disable the IRQ
116*a588332fSMauro Carvalho Chehabline due to excessive unhandled counts. [2]_
11706320cedSSean V Kelley
11806320cedSSean V KelleyThe config option X86_REROUTE_FOR_BROKEN_BOOT_IRQS exists to enable (or
11906320cedSSean V Kelleydisable) the redirection of the interrupt handler to the PCH interrupt
12006320cedSSean V Kelleyline. The option can be overridden by either pci=ioapicreroute or
121*a588332fSMauro Carvalho Chehabpci=noioapicreroute. [3]_
12206320cedSSean V Kelley
12306320cedSSean V Kelley
12406320cedSSean V KelleyMore Documentation
12506320cedSSean V Kelley==================
12606320cedSSean V Kelley
12706320cedSSean V KelleyThere is an overview of the legacy interrupt handling in several datasheets
12806320cedSSean V Kelley(6300ESB and 6700PXH below). While largely the same, it provides insight
12906320cedSSean V Kelleyinto the evolution of its handling with chipsets.
13006320cedSSean V Kelley
13106320cedSSean V KelleyExample of disabling of the boot interrupt
13206320cedSSean V Kelley------------------------------------------
13306320cedSSean V Kelley
134*a588332fSMauro Carvalho Chehab      - Intel® 6300ESB I/O Controller Hub (Document # 300641-004US)
13506320cedSSean V Kelley	5.7.3 Boot Interrupt
13606320cedSSean V Kelley	https://www.intel.com/content/dam/doc/datasheet/6300esb-io-controller-hub-datasheet.pdf
13706320cedSSean V Kelley
138*a588332fSMauro Carvalho Chehab      - Intel® Xeon® Processor E5-1600/2400/2600/4600 v3 Product Families
13906320cedSSean V Kelley	Datasheet - Volume 2: Registers (Document # 330784-003)
14006320cedSSean V Kelley	6.6.41 cipintrc Coherent Interface Protocol Interrupt Control
14106320cedSSean V Kelley	https://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xeon-e5-v3-datasheet-vol-2.pdf
14206320cedSSean V Kelley
14306320cedSSean V KelleyExample of handler rerouting
14406320cedSSean V Kelley----------------------------
14506320cedSSean V Kelley
146*a588332fSMauro Carvalho Chehab      - Intel® 6700PXH 64-bit PCI Hub (Document # 302628)
14706320cedSSean V Kelley	2.15.2 PCI Express Legacy INTx Support and Boot Interrupt
14806320cedSSean V Kelley	https://www.intel.com/content/dam/doc/datasheet/6700pxh-64-bit-pci-hub-datasheet.pdf
14906320cedSSean V Kelley
15006320cedSSean V Kelley
15106320cedSSean V KelleyIf you have any legacy PCI interrupt questions that aren't answered, email me.
15206320cedSSean V Kelley
15306320cedSSean V KelleyCheers,
15406320cedSSean V Kelley    Sean V Kelley
15506320cedSSean V Kelley    sean.v.kelley@linux.intel.com
15606320cedSSean V Kelley
157*a588332fSMauro Carvalho Chehab.. [1] https://lore.kernel.org/r/12131949181903-git-send-email-sassmann@suse.de/
158*a588332fSMauro Carvalho Chehab.. [2] https://lore.kernel.org/r/12131949182094-git-send-email-sassmann@suse.de/
159*a588332fSMauro Carvalho Chehab.. [3] https://lore.kernel.org/r/487C8EA7.6020205@suse.de/
160