Accessing Local Resource of other KCN

[Reto_A_]

Hi,

Recall Figure 2-12 (Host and Intel® MIC Architecture Physical Memory Map) on in the Intel Xeon Phi Systems Software Developers Guide.

The low memory (0 - 512G) is divided into 8x64GB blocks where block 0 is the Xeon Phi's own GDDR / MMIO space. The blocks 1-7 are referred to as "Local Resources of Other KNC". I have multiple Xeon Phis in my machine and want to access the GDDR memory of the other card directly. I assume that this should be possible by accessing this memory range.

My questions:

Assume, I have 4 cards A, B, C and D. I start two applications one A and C. I want to have these two communicate with each other. How do they know which coprocessor (0,1,2,3) they have to access i.e. how do I know that coprocessor 2 on the figure is in fact C and not B or D? Is this consistent, i.e. if I am on coprocessor 2, then the other blocks will be coprocessor 0,1,4 ? How do I know my own ID?

I hope it's clear what I mean...

Further Information: I am not using Linux / MPSS for doing that. We have developed our own OS for the host / card.

Thanks,

 Reto

[TaylorIoTKidd]

Reto,

I'm looking for someone who can answer your questions.

Regards
--
Taylor

[Evan_P_Intel · Accepted solution]

Reto A. wrote:

Recall Figure 2-12 (Host and Intel® MIC Architecture Physical Memory Map) on in the Intel Xeon Phi Systems Software Developers Guide. ... The blocks 1-7 are referred to as "Local Resources of Other KNC".

I was among those who reviewed an early draft of the Intel® Xeon Phi™ Coprocessor System Software Developers Guide for correctness a few years ago. I remember pointing out that section 2.1.12 mentioned the "Local Resources of other KNC" address region and needed to be revised not to do that...evidently, the text was updated, but the reference within Figure 2-12 escaped notice.

The seven blocks in Figure 2-12 marked "Local Resources of other KNC" are the last vestiges of a hardware capability which was proposed and abandoned early in the design phase of KNC; the reference survived only because section 2.1.12 happens to be derived from an internal document written between those two points in time. Despite the discrepancy, it's the text immediately following the figure that is correct—the address range is in fact "reserved," and, as stated in the first full paragraph of the following page, the memories of "remote Intel® Xeon Phi™ coprocessor devices are also accessed through System addresses."

Reto A. wrote:

I ... want to access the GDDR memory of the other card directly.

Recall that system addresses (the "Host Resources" region of Figure 2-12) are between 512GiB and 1TiB, and that the SMPT controls the mapping between each 16GiB page of the system address region and a corresponding 16GiB region of Xeon Phi device address space. In turn, the host system's IOMMU (if it is enabled) controls the mapping between device addresses and physical addresses (otherwise they are identical).

So, when a program running on a Xeon Phi accesses another's memory:

• The virtual address used by the program is converted (via page table lookup by the coprocessor) into a coprocessor physical address falling between 512GiB and 1TiB,
• which is converted (via SMPT lookup) into a device address,
• which is converted (via page table lookup by the host's IOMMU) into a physical address falling within the other Xeon Phi's PCI MEMBAR0.
• After subtracting the MEMBAR0 base address, the resulting coprocessor physical address will lie in the local GDDR region between 0 and 32GiB.

Since only the host knows where in host physical address space the MEMBAR0s of any installed Xeon Phis are located, and only it can configure the IOMMU, direct access to another Xeon Phi's memory requires cooperation from the Xeon Phi device driver executing on the host.

If you're using the coprocessor's DMA engine to access the other Xeon Phi's memory instead of a thread, then the first conversion doesn't occur since the DMA engine operates directly on coprocessor physical addresses, not virtual addresses.

Reto A. wrote:

Further Information: I am not using Linux / MPSS for doing that. We have developed our own OS for the host / card.

There are a lot of details involved in setting up this scenario; I'd still recommend you peruse the Linux SCIF driver's source code—it's a working example, even if it's not directly applicable.

[Reto_A_]

Thanks a lot for your reply. It cleared out a lot of things. Being able to access the other KNC directly w/o wasting a SMPT entry would be a nice feature to have. However, I managed to access the other cards resources via the system memory range.

Hopefully the documentation gets revised and extended soon.