Interrupt Remapping of MSIs behind a PCIe-to-PCI bridge

[cjeremi42]

After reading "Intel Virtualization Technology for Directed I/O Architecture Specification" Rev 1.1, I have some questions regarding Interrupt Remapping of Message Signalled Interrupts for PCI devices behind a PCIe-to-PCI bridge. Page 37 of the spec says:

For message-signaled interrupt requests from devices behind PCI Express-to-PCI/PCI-X
bridges, the requester identifier in those interrupt requests may be that of the interrupting
device or the requester-id with the bus number field equal to the bridges secondary
interfaces bus number and device and function number fields value of zero. Section 3.6.1.1
describes legacy behavior of these bridges. Due to this aliasing, interrupt-remapping
hardware does not isolate interrupts from individual devices behind such bridges.

Does "interrupt-remapping hardware does not isolate interrupts from individual devices behind such bridges" mean *all* PCIe-to-PCI bridges?

If I have multiple devices behind a PCIe-to-PCI bridge, and I am using the interrupt remapping feature of VT-d, all of the MSI devices behind the PCIe-to-PCI bridge will be assigned to the same domain. Do these devices receive the same HANDLE/SUBHANDLE; the same message address & data would be written to their MSI capability registers? In essence, their MSIs will no longer be assigned a unique interrupt vector -- they will all be sharing 1 MSI interrupt vector? How do you handle MSI-X?

It sound like VT-d interrupt remapping would require drivers for legacy PCI devices using MSI/MSI-X to check "is this mine" at primary interrupt time. This seems to invalidate the "MSIs are not shared" paradigm set by Windows Vista. Please reference the discussion by Jake Oshins and Davis Walker at the OSROnline discussion "MSIs: documentation discrepancies regarding" http://www.osronline.com/showThread.cfm?link=118548. To support VT-d's "MSIs behind PCIe-to-PCI bridges are shared" paradigm, drivers authored for Vista would require modification and would likely incur additional performance overhead at primary interrupt time to check "is this interrupt mine?".

I have a large install base of legacy PCI devices (CompactPCI and PXI devices in expansion chasses) that connect to newer, PCI Express-based systems through PCIe-to-PCI bridges. Perhaps we don't want to use VT-d with these devices? Customers would like partitioning, but generally not if the performance cost is to high.

Thank you,
Jeremiah

[Steven_T_Intel]

I believe you have mis-understood the VT-d interrupt-remapping limitation with respect to interrupts originating from behind PCIe-to-PCI* bridges.

If there are multiple interrupts sources behind a PCIe-to-PCI bridge, VT-d interrupt-remapping indeed can remap each such interrupt uniquely. No problems there, and hence no need for you to share these interrupts. The only limitation is with respect to the "interrupt isolation" capability VT-d provides. Specifically, using the interrupt-remapping table, you can filter/isolate which interrupt-source can reference a given interrupt-remap table-entry (using the SID/SVT/.... fields). This is needed in some security/isolation critical usages such as virtualization, where a hypervisor may want to isolate interrupts generated by devices assigned to different partitions/virtual-machines. VT-d hardware uses the PCI requester-id (bus/dev/func#) in the PCI transaction header to perform this isolation. Theonly limitation with interrupts sources behind PCIe-to-PCI* bridges is that, the PCIe-to-PCI bridges override the requester-id with the requester-id of the bridge when forwarding MSI/MSI-X requests (this is true even for interrupts from IOAPICs behind PCIe-to-pCI bridges), and due to this, VT-d cannot perform isolate interrupt to the interrupt source (and only to the bridge).

Obviously, it would be bad if VT-d cannot support existing interrupt/MSI/MSI-X usages from each/all interrupt sources !!

If you need more details, you could also referencethe Linux/Xen open source code that enables VT-d interrupt-remapping.

[zigic]

My problem is with mapping of interrupts behind PCIe to Local bus bridge (PLX, PEX8311AA bridge).
Host controller is Intel motherboard D945GNT. I am using legacy PCI software to access all resources on PCIe to Local bus bridge. There is no problem with accessing registers, everything works fine. There is no operating system because I am using direct access from protected mode (DJGPP compiler).

If I want to use INTx method instead of MSI, there is a problem.

In order to properly program APIC and other settings, I need two registers from PCIe space:

Interrupt Pin and Interrupt Line Routing

I am reading these values:

Interrupt Pin = 0x01
Interrupt Line = 0xff

Why is Interrupt Line = 0xff ?

In EEPROM for the configuration of PCIe bridge, it is clear where to set bits, so that during PCIe system configuration process (executed by Pentium on the host motherboard), system is configured to use INTx instead of MSI.

Is this problem with motherboard or chipset? (There is no setting in BIOS to control INTx or MSI).
Is it possible that this was intentionally done by Intel designers or there is something wrong in my understanding of INTx method on PCIe?

I would expect Interrupt Line register valid value to be 0x09, 0x0a, 0x0b etc. in order to use legacy PCI software to route hardware interrupts to CPU.

Thanks,

Slavisa Zigic

[David_O_Intel1]

Here is a comment I received from a fellow engineer:

"If I remember correctly, the value in Interrupt Line is only used for non-ACPI capable OSes to indicate the IRQ number. For ACPI capable OSs, this information is overridden by PRT method.

In some BIOSs, there is a setting for indicating the OS is ACPI capable. Im wondering if there is such setting in the BIOS on this system.

In any case, I dont think BIOS should set Interrupt Line to a valid value (non 0xff) so that the system can boot either non-APCI or ACPI capable OSs."

David Ott

[yuhong2]

Actually, ACPI is not the key setting here. The key setting here is Plug and Play OS. If this is disabled, BIOS configure resources for all PnP hardware, include setting the Interrupt Line register. If this is enabled, BIOS configures resources only for hardware needed at boot time, leaving the rest for the OS to configure, which is what is happening here. The non-ACPI way to do this is to use PCI BIOS or the $PIR table, and the ACPI way is to use the aformentioned _PRT method. In any case, the Interrupt Line register is only applicatable for PCI interrupt routing to the 8259 PICs. For APICs, you need to use the MPS tables instead for non-ACPI.