XL710 VF driver can reply to ARP request but cannot reply to ICMP's

[jliu85]

We try to write a XL710 VF driver simply,and test it by using ping command form PF to VF.

Now the VF driver can receive the ARP request and can reply it successfully, and the VF can also receive the ICMP request,and the reply data seems to put into the transmit buffer.

But in the PF, we cannot catch any reply data.

The ARP request can be reply that is to say the VF driver works normally, does it correctly? At least, it shows that the receive part and transmit part are correct.

Things that I want to know is what registers are relative to this issues, or any ideas about this issues?

Thanks & regards!

Animy.

[SYeo3]

Dear animy,

Thank you for contacting Intel.

We would like to request for the operating system and driver version that you are working on.

We look forward to your reply.

Sincerely,

Sandy

[jliu85]

Hi,Sandy

Thanks for your reply.

The driver i used is based on Linux-kernel 3.17.6 and this problem have been solved by ignoring the protocol differences in transmit.

And there is a new problem occurred that shows as follows:

1. after transmit be called 4 times that process 4 transmit packets. (solved. we have fixed it to one transmit,one process tx packet).

2. process 4 receive pakets only after 4 tx packets processing happened. (not solved)

i don't kown where or what register makes the 4 receive packets be processed together after processing 4 tx packets.

Could you give me a help?

Regards!

Animy

[SYeo3]

Hi Animy,

Thanks for providing the information.

I'll check on this internally and will get back to you with my findings.

Sincerely,

Sandy

[SYeo3]

Hi Animy,

In addition to the Linux kernel version, please provide the driver filename you used so we can understand your environment. Thanks.

Sincerely,

Sandy

[jliu85]

Hi Sandy,

Thanks for replying.

The driver i referred is i40evf in kernel 3.17.6. Now i used it in iPXE(single process),and the driver works in polling mode.

Now the VF diver can ping successfully, but the process of receive packets is always after 4 packets received,

and the the register is rx_desc->wb.qword1.status_error_len in i40evf code or DD in datasheet 8.3.2.2 Receive Descriptor - Write Back Format, section 8.3.2.2.1 16 Byte Receive Descriptors WB Format, or section 8.3.2.2.2 32 Byte Receive Descriptors WB Format (for 32 byte mode) correspondingly.

In linux code, we find if one packet received,the DD bit of rx_desc->wb.qword1.status_error_len is setted,but in our iPXE code, we find that only after 4 packets received, the register bit will be set.

The things i want to know is what factors make this register be setted (or need some configuration by the VF driver ? ), is it relative to the QRX_TAIL set or it is only done by hardware?

I look forward your reply sincerely.

Thanks,

Animy

[SYeo3]

Hi Animy,

Thanks for providing the details. We'll check on this and update you accordingly.

Sincerely,

Sandy

[jliu85]

Hi Sandy,

Thanks for replying. We look forward your further reply as soon as possible.

Best Regards,

Animy

[jliu85]

Hi Sandy,

How is going on?

I checked i40evf codes based on Linux 3.17.6, and i found that the value of

num_rx_qp and num_tx_qp be parsed to 1

in i40e_common.c i40e_vf_parse_hw_config

hw->dev_caps.num_rx_qp = msg->num_queue_pairs;

hw->dev_caps.num_tx_qp = msg->num_queue_pairs;

and i forced set these parameters to 1,and the adapter->vsi_res->num_queue_pairs is still 4.

then i the adapter->vsi_res->num_queue_pairs to 1,the divers could not work normally.(both in Linux 14.04. PF driver(work on real machine) and VF driver(work on QEMU) based on Linux-kernel-3.17.6.

I dont't konw the receive latency is relative to the number of queue pairs. If it is yes,how to fix this problem.

I look forword to your reply.

Best,

Animy

[SYeo3]

Hi animy,

Thanks for providing the additional details. I'll check this.

Sincerely,

Sandy

[jliu85]

Hi Sandy,

I'm so sorry. In former label line 3

animy 撰写:

Hi Sandy,

How is going on?

I checked i40evf codes based on Linux 3.17.6, and i found that the value of

num_rx_qp and num_tx_qp be parsed to 1

in i40e_common.c i40e_vf_parse_hw_config

hw->dev_caps.num_rx_qp = msg->num_queue_pairs;

hw->dev_caps.num_tx_qp = msg->num_queue_pairs;

and i forced set these parameters to 1,and the adapter->vsi_res->num_queue_pairs is still 4.

then i the adapter->vsi_res->num_queue_pairs to 1,the divers could not work normally.(both in Linux 14.04. PF driver(work on real machine) and VF driver(work on QEMU) based on Linux-kernel-3.17.6.

I dont't konw the receive latency is relative to the number of queue pairs. If it is yes,how to fix this problem.

I look forword to your reply.

Best,

Animy

the num_rx_qp and num_tx_qp be parsed to 4 from HW not to 1.

Best,

Animy

[SYeo3]

Hi Animy,

Thanks for the additional info. I'm still checking it.

I'll get back to you once I have any updates.

Sincerely,

Sandy

[SYeo3]

Hi Animy,

It seems the issue is with RX descriptor write back and why it only operates 4 at a time. This is default operation but can be changed. If you look at the DPDK driver on dpdk.org you can see how this can be done as that is poll mode driver.

Look at functions i40e_vsi_queues_bind_intr and i40e_vsi_enable_queues_intr.

Looking at the i40e and i40vf won't help as they don't work in poll mode.

Hope this helps.

Sincerely,

Sandy

[HXie3]

Hi Sandy,

My problem is same with Animy. And the PF used Linux driver, not dpdk PMD. So I think i40e_vsi_queues_bind_intr can not be called. because it is in DPDK pmd, not in Linux driver.

I check the code of vf, it will call I40E_VIRTCHNL_OP_CONFIG_VSI_QUEUES, when Linux driver receive this message from VF, it will do some configuration, in i40e_config_vsi_rx_queue function. but I don't know which option is that you mentioned above.

[jliu85]

Hi Sandy,

I'm sorry to reply so late.

1. I have already read the DPDK driver simply, and I found the DPDK driver set interval for irq in i40e_vsi_queues_bind_intr and i40e_vsi_enable_queues_intr. I do the same work in Linux-kernel 3.17.6 that is not effective.

# define I40E_GLINT_CTL

0x0003F800

in i40e_vsi_queues_bind_intr( DPDK ) is not clear to me, and there is no description in XL710 Datasheet.

3. i40e_vsi_queues_bind_intr and i40e_vsi_enable_queues_intr this two function are mentioned above both work in i40e PF,and our work only about XL710 driver. Is there any way to fix this problem only by modifying the XL710 VF driver?

Sincerely,

Animy

[SYeo3]

Thanks for your posts and updates, animy and GroundSea.

Let me check on this.

Sincerely,

Sandy

[jliu85]

Hi Sandy,

the detail information i use XL710 PF(Linux kernel 3.17.6 PF driver) ping XL710 VF (iPXE driver,polling mode) as follows:

PING 192.168.57.81 (192.168.57.81) 56(84) bytes of data.

64 bytes from 192.168.57.81: icmp_seq=4 ttl=64 time=3039 ms

64 bytes from 192.168.57.81: icmp_seq=5 ttl=64 time=2034 ms

64 bytes from 192.168.57.81: icmp_seq=6 ttl=64 time=1032 ms

64 bytes from 192.168.57.81: icmp_seq=7 ttl=64 time=35.0 ms

64 bytes from 192.168.57.81: icmp_seq=8 ttl=64 time=3045 ms

64 bytes from 192.168.57.81: icmp_seq=9 ttl=64 time=2039 ms

64 bytes from 192.168.57.81: icmp_seq=10 ttl=64 time=1038 ms

64 bytes from 192.168.57.81: icmp_seq=11 ttl=64 time=42.1 ms

64 bytes from 192.168.57.81: icmp_seq=12 ttl=64 time=3047 ms

64 bytes from 192.168.57.81: icmp_seq=13 ttl=64 time=2042 ms

64 bytes from 192.168.57.81: icmp_seq=14 ttl=64 time=1035 ms

64 bytes from 192.168.57.81: icmp_seq=15 ttl=64 time=36.7 ms

FYI

Best regards,

Animy

[SYeo3]

Hi GroundSea,

The flow in that section is done by the DPDK driver not the Linux driver. This might help explain what you are seeing:

See section 8.3.3.1.4.2 Receive descriptor reporting (descriptor write back):

http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/xl710-10-40-controller-datasheet.pdf Intel® Ethernet Controller XL710 Datasheet

Sincerely,

Sandy

[HXie3]

Hi Sandy,

Thank you for your information, this is a newer version datasheet, in the previous version, I didn't see such description.

I'll try based on its description. Thank you very much.

Regards,

Haifeng

[HXie3]

Hi Sandy,

The datasheet said "The receive descriptors could be reported instantly for each packet by setting the ITR_INDX to NoITR. Another option is enabling some coalescing of reported descriptors by using the ITR option. This option could be used for better PCIe and host memory bandwidth use at the expense of some possible latency of reporting the completed descriptor."

So I have modified the RTE_LIBRTE_I40E_ITR_INTERVAL macro to 6 in VF's code, then in the operation I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP, rxitr_idx equals to 3(NoITR). and I added some print in Linux PF driver( in function i40e_config_irq_link_list), it showed the itr_idx had already been set to 3. But the problem still exists. I don't know the reason. is there anything else neet to be set?

Thank you!

Regards,

Haifeng

[SYeo3]

Hi animy, Thank you for posting the ping results. Does it mean the VF is able to reply to ping already?

Hi Haifeng, I'll check on this.

Sincerely,

Sandy