SR-IOV VF-prioritization

[MindSpeed]

Is it possible to setup some prioritization between VF's on PF ?

Or all VF's on PF has the same priority always?

If yes - how this feature calls and how to use it (how to setup priority) ?

my case:

Two hosts with Intel NIC's:

• I'm creating two VF's on PF interface and bind the first VF to DPDK (on each Host-side) . DPDK applications (on each Host) starts and send RT-traffic to each other
• second VF's (on each Host) I'm setting up with IP addresses and run iperf traffic between them

I'm trying to get config where iperf traffic on the second pair of VF's will NOT impact primary RT-traffic between DPDK-applications.

I'm looking a way to config my first VF's (on each side) with highest priority then other VF's on the PF

Currently I'm using i350 NIC and also have x722, x710 NICs

Working on Linux CentOS 7, kernel-3.10.0-rt56

[Caguicla_Intel]

Hello MindSpeed,

Thank you for posting in Intel Ethernet Communities.

Since this question involves DPDK, please check with Intel Premier Support. See the following for access to Intel Premier Support. Click the "Access and Log In" to contact the DPDK support " https://www.intel.com/content/www/us/en/design/support/ips/training/welcome.html

Please be informed that we will be closing this request after 3 business days in case we don't hear from you. If you may need an assistance from Intel Ethernet team in the future, feel free to post a new question and we will be glad to be of help.

May you have an amazing day ahead!

Best regards,

Crisselle C

Intel® Customer Support

[MindSpeed]

Let's leave DPDK aside, focus on VF prioritization - my primary question

to make it easy we could forget about DPDK for a while and imagine two IPERF streams on two VF's. And we have to find a way to prioritize streams.

[Caguicla_Intel]

Hello MindSpeed,

Thank you for the reply.

Please allow us to check on this internally. We will give you an update as soon as there is any but no later than 3 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[MindSpeed]

Sure, please check on this internally,

I really need an answer

Regards

[Caguicla_Intel]

Hello MindSpeed,

Thank you for the patience on this matter.

Please refer to the details below for multiple ways to do VF prioritization from our engineering team. Let us know if you have additional questions and clarifications.

• VF daemon: VFd (the VF daemon) is a mechanism which can be used to configure features on a VF (SR-IOV Virtual Function) without direct access to the PF (SR-IOV Physical Function)

6.4.8. VF TC bandwidth allocation

Run a testpmd runtime command on the PF to set the TCs (traffic class) TX bandwidth allocation for a specific VF:

set vf tc tx min-bandwidth 0 0 (20,20,20,40)

The allocated bandwidth should be set for all the TCs. The allocated bandwidth is a relative value as a percentage. The sum of all the bandwidth should be 100.

6.4.9. VF TC max bandwidth

Run a testpmd runtime command on the PF to set the TCs TX maximum bandwidth for a specific VF:

set vf tc tx max-bandwidth 0 0 0 10000

The maximum bandwidth is an absolute value in Mbps.

6.4.10. TC strict priority scheduling

Run a testpmd runtime command on the PF to enable/disable several TCs TX strict priority scheduling:

set tx strict-link-priority 0 0x3

The 0 in the TC bitmap means disabling the strict priority scheduling for this TC. To enable use a value of 1.

More details on this here: https://doc.dpdk.org/guides-18.08/howto/vfd.html

• Rte_flow: I believe our Intel E810 series only supports Rte_flow

More details here on RTE_FLOW priority tests: https://doc.dpdk.org/dts/test_plans/rteflow_priority_test_plan.html

• Flow director:

22.7.2. Flow Director

The Flow Director works in receive mode to identify specific flows or sets of flows and route them to specific queues. The Flow Director filters can match the different fields for different type of packet: flow type, specific input set per flow type and the flexible payload.

The default input set of each flow type is:

ipv4-other : src_ip_address, dst_ip_address

ipv4-frag : src_ip_address, dst_ip_address

ipv4-tcp  : src_ip_address, dst_ip_address, src_port, dst_port

ipv4-udp  : src_ip_address, dst_ip_address, src_port, dst_port

ipv4-sctp : src_ip_address, dst_ip_address, src_port, dst_port,

            verification_tag

ipv6-other : src_ip_address, dst_ip_address

ipv6-frag : src_ip_address, dst_ip_address

ipv6-tcp  : src_ip_address, dst_ip_address, src_port, dst_port

ipv6-udp  : src_ip_address, dst_ip_address, src_port, dst_port

ipv6-sctp : src_ip_address, dst_ip_address, src_port, dst_port,

            verification_tag

l2_payload : ether_type

The flex payload is selected from offset 0 to 15 of packet’s payload by default, while it is masked out from matching.

Start testpmd with --disable-rss and --pkt-filter-mode=perfect:

./app/testpmd -l 0-15 -n 4 -- -i --disable-rss --pkt-filter-mode=perfect \

             --rxq=8 --txq=8 --nb-cores=8 --nb-ports=1

Add a rule to direct ipv4-udp packet whose dst_ip=2.2.2.5, src_ip=2.2.2.3, src_port=32, dst_port=32 to queue 1:

testpmd> flow_director_filter 0 mode IP add flow ipv4-udp \

        src 2.2.2.3 32 dst 2.2.2.5 32 vlan 0 flexbytes () \

        fwd pf queue 1 fd_id 1

Check the flow director status:

testpmd> show port fdir 0

######################## FDIR infos for port 0     ####################

 MODE:  PERFECT

 SUPPORTED FLOW TYPE: ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other

                       ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other

                       l2_payload

 FLEX PAYLOAD INFO:

 max_len:      16         payload_limit: 480

 payload_unit: 2          payload_seg:  3

 bitmask_unit: 2          bitmask_num:  2

 MASK:

   vlan_tci: 0x0000,

   src_ipv4: 0x00000000,

   dst_ipv4: 0x00000000,

   src_port: 0x0000,

   dst_port: 0x0000

   src_ipv6: 0x00000000,0x00000000,0x00000000,0x00000000,

   dst_ipv6: 0x00000000,0x00000000,0x00000000,0x00000000

 FLEX PAYLOAD SRC OFFSET:

   L2_PAYLOAD:   0     1     2     3     4     5     6 ...

   L3_PAYLOAD:   0     1     2     3     4     5     6 ...

   L4_PAYLOAD:   0     1     2     3     4     5     6 ...

 FLEX MASK CFG:

   ipv4-udp:   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv4-tcp:   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv4-sctp:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv4-other: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv4-frag:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv6-udp:   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv6-tcp:   00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv6-sctp:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv6-other: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   ipv6-frag:  00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

   l2_payload: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

 guarant_count: 1          best_count:   0

 guarant_space: 512        best_space:   7168

 collision:    0          free:         0

 maxhash:      0          maxlen:       0

 add:          0          remove:       0

 f_add:        0          f_remove:     0

Delete all flow director rules on a port:

testpmd> flush_flow_director 0

We look forward to your response.

In case we don't hear from you, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello MindSpeed,

Good day!

Please let us know if the previous post was of help on your query. In case you have additional questions and clarifications on this matter. please do not hesitate to ask.

We hope to hear from you soon.

In case we don't receive a reply, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[MindSpeed]

Hi Crisselle C !

Thanks for the information and your support.

Honestly we are still in investigation

We have tried to use testpmd application to setup a VF but seems like when we close the testpmd app VF-settings are not remain and we cannot starts two DPDK apps by some reasons.

And we decided to use kernel driver i40e, since it has debugfs interface and seems like it is possible to setup everything by "send aq_cmd"

(echo "send aq_cmd ...." > /sys/kernel/debug/i40e/0000\:b8\:00.0/command)

Now I'm learning this possibilities and if you have some "i40e Debugfs guide" with some setup examples - I will be highly appreciate your for it, because I'm doing the reverse to understand what exactly I should send (sniffing AdminQueue of testpmd and trying to send the same by debugfs to kernel i40e)

Really, I discovered i40e debugfs accidentally by studying i40e sources and found no mention to it in readme or even Google

So we are still in our way.

Will update you with details in case of success.

Thanks!

[Caguicla_Intel]

Hello MindSpeed,

Thank you for sharing your observation with us.

While we are forwarding your feedback with our engineers, we will be patiently waiting for your update regarding your investigation. If you have additional questions or clarifications, feel free to let us know.

Awaiting to hear form you.

In case we don't receive a reply, we will make a follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[MindSpeed]

Hi Crisselle C

how about "i40e Debugfs guide" ?

Have you read my last message to the end?

Regards

[Caguicla_Intel]

Hello MindSpeed,

Good day!

We are currently checking on this and we will give you an update as soon as there is any findings but no later than 2-4 business days.

Hoping for your kind patience.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello MindSpeed,

Thank you for the patience. Please see below information from our engineering team.

To run two instances of testpmd on the same system, you would have to run it on different sockets/cores and invoke a different testpmd prefix name in the application. For Example:

1st instance of testpmd:

root@nd-wolfpass-39:/usr/src/dpdk/dpdk/install/bin# ./testpmd -l 18-36 -n 4 -w 18:01.0 --file-prefix testpmd18010 --socket-mem=0,1024 --proc-type=auto -- --nb- cores=4 --rxq=4 --txq=4 -i --forward-mode=mac --eth-peer=0,68:05:ca:a6:0b:1c

2nd instance of testpmd:

root@nd-wolfpass-39:/usr/src/dpdk/dpdk/install/bin# ./testpmd -l 0-9 -n 4 -w 18:00.0 --file-prefix testpmd18000 --socket-mem=1024,0 --proc-type=auto -- --nb- cores=4 --rxq=4 --txq=4 -i --forward-mode=mac --eth-peer=0,68:05:ca:a6:0b:1d

Although debugfs is not well documented, it can be used just like sysfs (http://patchwork.ozlabs.org/project/netdev/patch/1377224142-25160-8-git-send-email-jeffrey.t.kirsher@intel.com/)

Yes, this cmd echo "send aq_cmd ...." > /sys/kernel/debug/i40e/0000\:b8\:00.0/command can be used to read/write registers, you will have to check those registers in datasheet.

Let us know if you have questions and clarifications.

Awaiting to you reply.

Should there be no response from you, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello MindSpeed,

Good day!

We'd like to check if you have received our previous message. In case you have additional questions or clarifications, please let us know and we will be glad to be of help.

Looking forward to hear from you.

In case we don't receive a reply, we will follow up after 3 business days.

Best regards,

Crisselle C

Intel® Customer Support

[Caguicla_Intel]

Hello MindSpeed,

How are you today?

Please be informed that we will now close this request since we haven't received any response from our previous follow ups. Just feel free to post a new question if you may have any other inquiry in the future.

Thank you for continuing to use of Intel products!

Best regards,

Crisselle C

Intel® Customer Support