Poor performance with E810-CQDA2 with RDMA and NetworkDirect SPI problems

[Roberto_BR]

Hello,

I am testing RDMA performances between two Windows Server 2022 nodes. The network cards used for the RDMA connection are one Intel E810CQDA2BLK 100Gb/s and one Supermicro  AOC-S100GC-i2C 100Gb/s (based on Intel E810-CAM2). I installed in each node the latest Intel drivers following all the Intel instructions to have RDMA correctly working.

I tested the RDMA communication with the Test-Rdma powershell script from Microsoft (using diskspd) and with MPI Ping-Pong (HPC Pack 2019) obtaining throughput values around 4.5GB/s (while I would expect at least 10 GB/s with 12.5 GB/s maximum).

Another problem with this is that some of the NetworkDirect SPI examples from Microsoft (https://github.com/microsoft/NetworkDirect) do not seem to work (e.g. ndping, ndrping) while other work perfectly (ndpingpong).

Can you help me understanding what I am doing wrong or if I am overlooking something?

Regards,

Roberto

[Roberto_BR]

Hello,

we are now using the same Intel NIC on both nodes (E810CQDA2BLK 100Gb/s) with the same NVM version and drivers. The issues remain the same but, after further investigation, the performance may be so poor due to wrong fiber cable type, so we will order a new fiber cable and re-analyze the performance.

Despite the wrong cable, the problem with Network Direct seems to be of other type. Following more details.

ndrping

PS C:\Users\Administrator\Desktop\NetworkDirect> .\ndrping.exe -c 10.10.32.20

Beginning test: ndrping.exe

Test failed.Test Result: 0xc0000120, Expected Result:0, Unexpected completion status

Code debugging:

void NdTestBase::CreateMW(HRESULT expectedResult, const char* errorMessage)

{

    HRESULT hr = m_pAdapter->CreateMemoryWindow(

        IID_IND2MemoryWindow,

        reinterpret_cast<VOID**>(&m_pMw)

    );

    LogIfErrorExit(hr, expectedResult, errorMessage, __LINE__);

}

hr = 0xc00000bb : The request is not supported.

ndping

PS C:\Users\Administrator\Desktop\NetworkDirect> .\ndping.exe -c 10.10.32.20

Beginning test: ndping.exe

GetPrivateData failed

  Line: 213

Code debugging:

        // get peer queue depth

        m_peerQueueDepth = 0;

        ULONG len = 0;

        if (m_pConnector->GetPrivateData(nullptr, &len) != ND_BUFFER_OVERFLOW)

        {

            LOG_FAILURE_AND_EXIT(L"GetPrivateData failed\n", __LINE__);

        }

Regards,

Roberto

[Roberto_BR]

Hello,
we replaced the fiber cables with the correct models but the problems remained. If we do not solve these problems we will be forced to buy network cards from other manufacturers.

Regards,

Roberto