Issues with second physical interface on 2P X710 using SR-IOV

[PLamb5]

Hi,

I am seeing some issues with SR-IOV on 2P X710s.

When using the second Physical port, I am not seeing traffic being balanced across the 4 Tx/Rx queues when running NVM 6.80 or later.

This issue wasn't present on 4.53 to my recollection, however it isn't possible to downgrade to validate this. I am in the process of trying to locate some cards with an appropriate NVM for this, but in the mean time, has anyone experienced the same issues?

So all traffic on Port1 is balanced across the 4 queues, allowing me to set cpu affinity in my VM and handle much higher load than with Port2, which appears to put everything on the first queue.

I have tried various ESXi driver versions, from 1.7 to 1.9.5 and see the same behaviour regardless of driver version.

Looking for some guidance!

Thanks,

Paul

[Caguicla_Intel]

Hello Paul,

We hope that the latest iavf driver version was of help on this issue.

Please be informed that we will now proceed in closing this request since we haven't received any response from our previous follow up. In case you have any other assistance needed in the future, feel free to post a new question.

May you have a lovely day!

Best regards

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[PLamb5]

Crisselle, I didn’t see your previous update. Updating the driver isn’t something I can do as this is an appliance. Can you confirm that there is anything in that new driver to fix this problem, or is this just a ‘try this’ kind of approach? Have you undertaken any steps to reproduce the problem? Many thanks, Paul.

[Caguicla_Intel]

Hello Paul,

Thank you for the reply.

We have sent a private message for some personal information needed. Kindly check your inbox.

Awaiting to your response.

Best regards

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Paul,

Thank you for providing the requested information.

Please allow us to forward this to our engineering team. We will give you an update as soon as possible but no later than 3-5 business days.

Hoping for your patience.

Best regards

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Paul,

Good day!

We highly appreciate your effort in providing the requested details.

It is important to update the iavf driver of the Ethernet controller as it may enhance its performance as well as the other features. However, as you mentioned, we totally understand that updating the driver is something that you can’t do. With this, please be informed that our engineers are closely looking into this matter. Rest assured that we will give you an update as soon as we heard back from them but no later than 3-5 business days.

Thank you for your kind understanding.

Best regards

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Paul,

How are you today?

We have received feedback from our engineering team and would like to check if you are still unable to update the vf driver to its latest version. 

We look forward to your response.

We'll reach out after 3 business days in case we don't hear from you.

Best regards

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[PLamb5]

Crisselle, What feedback have you received from the engineering team? Have you been able to determine if there’s any change in the driver that will be contribute to a resolution to this issue? This is a Firewall Appliance, running an embedded operating system. It’s not just a case of changing the driver – it would require engineering time, QA, etc. Do you have release notes that document the changes and their impact on the issue reported? Many thanks, Paul.

[Caguicla_Intel]

Hello Paul,

Thank you for the prompt reply.

Our engineering team was checking if you were able to install the latest drivers. 

Since you have already your response, please allow us to forward it to them as well as your additional questions. We will get back to you within 3-5 business days. 

Thank you for your kind patience and understanding.

Best regards

Crisselle C

Intel Customer Support

A Contingent Worker at Intel

[Caguicla_Intel]

Hello Paul,

Good day!

Please see below details for the feedback from our engineering team.

We have informed that they were unable to pin point if this is a driver issue or an NVM issue. They believe this issue could be related to VF driver because ESX PF driver is not responsible for traffic processing within the VM. ESX PF driver is responsible for creating the VF and assigning the VF to a VM. Once ESX PF driver has accomplished this, it's not involved in traffic processing within the VM. For this reason, they are requesting to test with iavf driver version 3.7.61.20 that might address this issue.  

Looking forward to your response.

In case we don't hear from you, we'll follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[PLamb5]

Hi Crisselle. The issue I have with this, is that the tx/rx queues are being used in an imbalanced fashion ONLY when you upgrade the NVM beyond 4.53. The issue appears to be related to Firmware, and not drivers, and ONLY on the second physical port. Given the driver is unaware that these two ports are on the same physical 710, this would lead me to believe the issue is on the card/nvm, not in the driver code base at all.. Can you please ask your engineering team if there is any change in that newer driver that would change how the physical device adjusts the tx/rx queue implementation, that would affect only one of the physical interfaces on the card? Many thanks, Paul.

[Caguicla_Intel]

Hello Paul,

Thank you for the reply.

Please allow us some time to ask this to our engineering team per your request. Rest assured that we will get back to you no later than 2-4 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

Apologies for the delay on this matter.

We totally understand the reluctance to update the driver as requested. Though we still would like to get the driver updated to a current state, we have also engaged our NVM engineering team to have them examine this issue as well.

We sincerely appreciate your patience and will hopefully have some information/update on the case soon.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

Good day!

Please be advised that we still haven't received any update from our Firmware team. Rest assured that we made another follow up and we will get back to you on Thursday, March 5 to provide an update.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

Thank you for the patience on this matter.

Kindly provide the etrack ID of the working X710 and the non-working X710. This would help us in further investigating the issue.

To get the ETrackID

GUI — Navigate to the network adapter's properties, and click the Identify Adapter button on the Link Speed tab. The NVM version and ETrackID is displayed

Via PowerShell:

Get-IntelNetAdapter | Format-List -Property DriverVersion, ETrackID, NVMVersion

Using Ethtool:

Output of the ethtool -i shows the running version of the network driver, the firmware version, and

ethtool -i ethX

ETrackID, as follows:

driver: i40e

version: 1.4.25

firmware-version: 4.25 0x800014cc 0.0.0

bus-info: 0000:01:00.0

The last 4 hex characters in the firmware version denote the ETrackID.

We look forward to your response.

In case we don't hear from you, we'll follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[PLamb5]

Crisselle: The working version: Firmware Version: 4.53 0x80001dc0 1.1147.0 Non-working is literally any version on the Intel Website from this point on (5.05, 6.01, 6.80, 7.0, 7.1). I do not have the e-track ID as I don’t have a card updated to test with right now. The issue is also present in OEM NVM’s, for the working Dell Version: Firmware Version: 4.53 0x80001e5b 17.0.12 Again, any NVM version beyond 4.53 exhibits the behaviour described with the second physical port. Kind regards, Paul.

[Caguicla_Intel]

Hello Paul,

Thank you for providing the requested information.

Please allow us to continue to check on this. Due to complexity of the issue, we will get back to you within 3-6 business days to provide an update.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

How are you today?

We'd like to inform you that we are still checking this request with our firmware team. Rest assured that we will provide an update as soon as there is any but no later than 3-6 business days.

Thank you for your kind understanding.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

Good day!

We sincerely apologize for the delay as we are still investigating this request. We will give you an update as soon as possible but no later than 3-6 business days.

Hoping for your patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

How are you today?

Unfortunately, there may be a delay in validation and testing of this nature as employees at Intel have been asked to work from home until future notice as a result of the Coronavirus. This makes it difficult to perform the Lab validation and testing required.

We apologize for this inconvenience as we continue to work to resolve this case. 

Hoping for your kind understanding.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

Good day and I hope this message finds you well.

Please accept our sincerest apologies for the delay on this matter as it was impacted by COVID-19. We now have someone able to test this issue. 

Please check and provide the information below as requested by our firmware team.

We matched the eTrack to oldest and not working version of NVM (5.05): 80002892.

To localize the problem we would like to reproduce the issue locally and we have questions to clarify the HW configuration:

1.) Which modules and speed do you used?

2.) Is there any VM with specific OS under ESXi?

3.) Was the performance test done on the VM?

We look forward to your reply.

Should there be no response from you, I’ll make sure to reach out after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello Paul,

How are you today?

We'd like to follow up the additional information that we requested for us to further investigate the issue.

Awaiting to hear from you.

I’ll make sure to follow up after 3 business days in case we don't receive a reply.

Best regards,

Crisselle C

Intel® Customer Support