Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4810 Discussions

i40e sr-iov packets latency

HXie3
Novice
3,782 Views

Hi all,

Currently, I'm using XL710 SR-IOV in VM. We are using DPDK 1.8 PMD driver. The VM can communicate with other equipment, but the packets have long latency. you can refer below picture.

I don't know reason, it seemed rte_eth_rx_burst didn't return the packets it received in time.

Does anyone meet this problem before? or I just need to modify some macro in the code?

BTW: I have tried modify RTE_LIBRTE_I40E_ITR_INTERVAL and RTE_LIBRTE_I40E_QUEUE_NUM_PER_VF, and no effect.

 

Any answer will be appreciated!

 

 

0 Kudos
16 Replies
st4
New Contributor III
1,259 Views

Hi Groundsea,

Let me check on this.

rgds,

wb

0 Kudos
HXie3
Novice
1,259 Views

Hi wb,

How is going on?

I use RHEL7(which kernel version is 3.10.0), the driver's version of i40e is as following:

driver: i40e

version: 1.2.37

firmware-version: f4.33.31377 a1.2 n4.42 e18a3

0 Kudos
st4
New Contributor III
1,259 Views

Hi GroundSea,

We are still checking on this.

rgds,

wb

0 Kudos
HXie3
Novice
1,259 Views

Hi wb,

I check the code, there is a macro RX_WTHRESH which was defined as 4, with the comment "/*< Default value of RX write-back threshold register. */". I don't know whether it is correlation with the problem. So I modify it to 0 in our code, but it doesn't take effect ether. then I check the code in i40e_rxtx.c, function i40e_dev_rx_queue_setup doesn't use this field(rx_conf.rx_thresh), so I want to know this register is correlation with this problem or not, if yes, how to set this register?

 

Thank you very much!

 

0 Kudos
st4
New Contributor III
1,259 Views

Hi GroundSea,

Thank you for the additional information. I will update you as soon there is any finding.

rgds,

wb

0 Kudos
st4
New Contributor III
1,259 Views

Hi GroundSea,

Please refer to the website for Intel® Data Plane Development Kit (Intel® DPDK) at http://www.intel.com/content/dam/www/public/us/en/documents/release-note http://www.intel.com/content/dam/www/public/us/en/documents/release-notes/intel-dpdk-release-notes.pdf

On page 57 Section 8.6, there is a specific section on Latency adjustments.

If the issue still persist, there is a note in the guide :

All Intel® DPDK questions and technical problems including those regarding the Ethernet* Controllers for the Poll Mode Driver should be reported through the Intel®

Premier Support site http://premier.intel.com/premier http://premier.intel.com/premier or access your IBL account and click the Intel® Premier Support link to enter issues under the Product

Name "Data Plane Development Kit (DPDK)",

Hope this info helps.

rgds,

wb

0 Kudos
HXie3
Novice
1,259 Views

Hi wb,

Thanks for your response. I have read section 8.6, and have try to modify the burst packets. but it is no effect.

There is another thread /message/293478# 293478 https://communities.intel.com/message/293478# 293478 , which mentioned the same question.

It said the problem may correlation with some features of PF driver. But we use VF PMD driver, can we set something in VF to solve this problem?

if VF driver cannot solve this problem, then how to modify it in the Linux PF driver?

Thank you for your support.

0 Kudos
st4
New Contributor III
1,259 Views

Hi GroundSea,

Thank you for sharing the additional info. I will further check on this.

rgds,

wb

0 Kudos
st4
New Contributor III
1,259 Views

Hi GroundSea,

You may contact Premier support for further assistance:

Visit http://premier.intel.com/premier http://premier.intel.com/premier or access your IBL account click the Intel® Premier Support link to enter issues

With regard to support on Intel Business Link, you may visit this website

https://kaveri.intel.com/ibl/help/support.htm https://kaveri.intel.com/ibl/help/support.htm

Thanks

wb

0 Kudos
HXie3
Novice
1,259 Views

Hi wb,

I got new datasheet from Sandy, I'll try first.

Thank you.

Regards,

Haifeng

0 Kudos
jliu85
Beginner
1,259 Views

Hi Haifeng,

Congratulations to solved this problem.

I met the similiar issues to you.

And I have set the DIS_AUTOMASK_N_MASK in I40E_GLINT_CTL in Linux driver( i40e_vsi_configure_msix of i40e_main.c), it have no effect on this problem.

Could you tell me the details that you modified, and does the latency still exist?

Regards,

Animy

0 Kudos
HXie3
Novice
1,259 Views

Hi Animy,

First, I modified the VF driver, set RTE_LIBRTE_I40E_ITR_INTERVAL equals to 6, so in function i40evf_config_irq_map (located in i40e_ethdev_vf.c of dpdk lib), it will send 3 in I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP message to PF. which means NoITR.

Second, in Linux PF driver, in function i40e_config_irq_link_list which process I40E_VIRTCHNL_OP_CONFIG_IRQ_MAP message,

I added following lines at last.

# ifndef I40E_GLINT_CTL

# define I40E_GLINT_CTL 0x0003F800

# define I40E_GLINT_CTL_DIS_AUTOMASK_N_MASK 0x4

# endif

/* Disable auto-mask on enabling of all none-zero interrupt */

wr32(hw, I40E_GLINT_CTL, I40E_GLINT_CTL_DIS_AUTOMASK_N_MASK);

 

In fact these lines are copied from dpdk pf driver, function i40e_vsi_queues_bind_intr in i40e_ethdev.c

after modifed, the latency disappeared, but I'm not sure is there any disadvantage exists.

Regards,

Haifeng

jliu85
Beginner
1,259 Views

Hi Haifeng,

Thanks for your detailed explanation.

I do the same setting as you in my iPXE VF driver and linux driver,but it doesn't work.

And I want to know the 3 you refered is PF or DPDK logs show or you calculate? (set RTE_LIBRTE_I40E_ITR_INTERVAL equals to 6)

And I think set ITR_INDX to NoITR that is bit3-4 in VFINT_DYN_CTLN or PFINT_DYN_CTLN register, I don't know how rxitr_idx reflect on it. Could you give me an explanation about it?

Thanks you very very much!

Best Regards,

Animy

0 Kudos
HXie3
Novice
1,259 Views

Hi Animy,

The ITR_INDX I used is bit 11-12 in QINT_RQCTL register. I added log in kernel driver and saw the value equals to 3.

 

The VF code like below:

 

The PF code like below:

 

0 Kudos
jliu85
Beginner
1,259 Views

Hi Haifeng,

Thanks for your further illustration.

Because i don't have correct test environment,could you help me to locate the following value?

1. irqmap_info->num_vectors at function i40e_vc_config_irq_map_msg in i40e_virchnl_pf.c

2. next_q in function i40e_config_irq_link_list in i40e_virchnl_pf.c

3. vecmap->txq_map in function i40e_config_irq_link_list in i40e_virchnl_pf.c

4. vecmap->txq_map at function i40e_config_irq_link_list in i40e_virchnl_pf.c

5. vsi_idx at function i40e_config_irq_link_list in i40e_virchnl_pf.c

and I will try to build a dpdk developing environment next.

Thanks!

0 Kudos
HXie3
Novice
1,259 Views

Hi Animy,

Following lines are my output in the kernel driver. I use 2 VF in the VM.

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========irqmap_info->num_vectors=1

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========vector_id=1

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========vsi_idx=9

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========txq_map=0, rxq_map=1

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========before loop next_q=0

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========in while loop next_q=32

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========itr_idx=3

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========irqmap_info->num_vectors=1

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========vector_id=1

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========vsi_idx=8

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========txq_map=0, rxq_map=1

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========before loop next_q=0

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========in while loop next_q=32

May 22 08:29:30 hfxie-RHEL7 kernel: i40e 0000:0a:00.1: ========hfxie========itr_idx=3

0 Kudos
Reply