Is there a way to capture RoCEv2 packet? Intel E810 RNIC

[devansh_rastogi]

Hi.

I have 2 identical rack servers installed with Intel E810-CAM2 NIC. These 2 are connected directly with each other through 2 DAC 100G cable. There are no switches/routers involved in between their connection. Also note that both ports are connected with each other (node1's port A connected to node2's port A and similar setup for port B).

I am able to successfully setup RoCEv2 on both RNICs using irdma driver. I have even verified the use of RoCEv2 using 'ibv_devinfo' command. I am also able to successfully run benchmarks like 'ib_send_bw' and have been able to perform 'rping' tests successfully to each other on both sides.

Now, I wanted to tcpdump RoCEv2 packet. Going through irdma's README, the only listed way is to introduce 2 additional components: a switch and another node - this is not feasible for me at the moment.

Is there any other way in which I can capture and analyse RoCEv2 network packet? Anyone's help would be really appreciated.

[ManoranjanDas]

Hello devansh_rastogi,

Greetings!

Thank you for reaching out to the Intel Community. We sincerely apologize for the delay in our response. Your concern has been received, and assisting you is our top priority.

Could you kindly let us know if the issue has been resolved, or if you need further assistance from us? A prompt response would greatly assist us in diagnosing and addressing the issue as quickly as possible.

We truly appreciate your patience and understanding, and we look forward to hearing from you soon.

Best regards,

Manoranjan

Intel Customer Support Technician

[devansh_rastogi]

Hi Manoranjan!
Hope you are doing well.

Thanks for getting back on this issue.

I am still looking for a response in this regard. Your inputs would be valuable.

[ManoranjanDas]

Hello devansh_rastogi,

Greetings!

Thank you for your patience and understanding. We are currently coordinating with our internal team to address your query. As soon as we receive any updates, we will promptly inform you of the status.

We appreciate your cooperation and will keep you updated as soon as possible.

Regards,

Manoranjan

Intel Customer Support Technician

[ManoranjanDas]

Hello devansh_rastogi,

Greetings!

Thank you for your patience and understanding. 

Regarding your query, Since RDMA (Remote Direct Memory Access) traffic bypasses the standard network stack, meaning it doesn't pass through the operating system kernel, making it difficult to capture with standard monitoring tools. 

Additionally, To capture RDMA traffic, we recommend using techniques such as switch port mirroring or an inline protocol analyzer, which can directly monitor the traffic at the network interface level. Please let us know if you need any assistance with this.

If you are observing any additional issues, please share them with us. We are happy to assist you.

Thank you for your understanding.

Regards,

Manoranjan

Intel Customer Support Technician

[devansh_rastogi]

Thanks Manoranjan!

Well, I was expecting the presence of some sort of a tool/hardware entity on the Intel RNIC itself for this purpose. Tools like this can be found on Nvidia RNICs and unfortunately, I had to place an order for those.

Nonetheless, thanks for your support!

[ManoranjanDas · Accepted solution]

Hello devansh_rastogi,

Greetings for the day! 

I hope this message finds you well.

Thank you for your feedback. I understand you were expecting a tool or hardware feature on the Intel RNIC similar to Nvidia RNICs for monitoring. Unfortunately, this feature is not available on Intel RNICs, which led to your order for alternative hardware.

Since your query seems resolved, we will proceed with closing the case. If you need further assistance, please feel free to reply, and we can reopen or initiate a new case.

We appreciate your continued trust in Intel products and are here to assist you.

Best regards,

Manoranjan.

Intel Customer Support Technician