Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
5292 Discussions

XL710 too many irq

AndriiV
Novice
‎03-26-2024 07:39 AM
6,767 Views

Hello!

I have a problem with Intel XL710,  after 22 Gbps of traffic some cores of processor loads 100% IRQ and traffic goes down.

 

atop.png

 

There are latast firmware and driver:

 

 

 

# ethtool -i ens2
driver: i40e
version: 2.24.6
firmware-version: 9.40 0x8000ecc0 1.3429.0
expansion-rom-version:
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

 

 

 

 

graphs.png

I read this guide

And made all possible configurations.  But nothing changes

 

 

 

382  set_irq_affinity local ens2
  384  set_irq_affinity all ens2
  387  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0
  389  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 5000 tx-usecs 20000
  391  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 10000 tx-usecs 20000
  393  ethtool -g ens2
  394  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 84 tx-usecs 84
  396  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 62 tx-usecs 62
  398   ethtool -S ens2 | grep drop
  399   ethtool -S ens2 | grep drop
  400   ethtool -S ens2 | grep drop
  401   ethtool -S ens2 | grep drop
  402  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 336 tx-usecs 84
  403   ethtool -S ens2 | grep drop
  404   ethtool -S ens2 | grep drop
  406   ethtool -S ens2 | grep drop
  407  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 672 tx-usecs 84
  408   ethtool -S ens2 | grep drop
  409   ethtool -S ens2 | grep drop
  411   ethtool -S ens2 | grep drop
  412   ethtool -S ens2 | grep drop
  425   ethtool -S ens2 | grep drop
  426   ethtool -S ens2 | grep drop
  427  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 8400 tx-usecs 840
  428  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 840
  430   ethtool -S ens2 | grep drop
  431   ethtool -S ens2 | grep drop
  432   ethtool -S ens2 | grep drop
  433   ethtool -S ens2 | grep drop
  434  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1680
  435   ethtool -S ens2 | grep drop
  436   ethtool -S ens2 | grep drop
  439  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 3200 tx-usecs 3200
  469  ethtool -a ens2
  472  ethtool ens2
  473  ethtool -i ens2
  475  ethtool -i ens2
  476  ethtool ens2
  482  ethtool -C ens2 adaptive-rx on
  484  ethtool -c ens2
  486  ethtool -C ens2 adaptive-tx on
  487  ethtool -c ens2
  492  history | grep ens2
  494  ethtool -m ens2
  499  ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 4200 tx-usecs 1600
  501  history | grep ens2

 

 

 

 

Server configuration:

80 cores  Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz

RAM 960 GB

8 x SAMSUNG MZQLB7T6HMLA-000AZ NVME disks

Ethernet Controller XL710 for 40GbE QSFP+ (rev 02)

 

What can be done with the settings of this network card to solve the problem?

 

On another server with a similar configuration, but a different network cards, everything is fine

atop_2.png

graphs_2.png

Server configuration:

88 cores Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz

RAM 512 GB

8 x SAMSUNG MZQLB7T6HMLA-00007 NVME disks

4 x 82599ES 10-Gigabit SFI/SFP+ Network Connection

 

 

0 Kudos

Link Copied

44 Replies
Simon-Intel
Employee
‎04-17-2024 08:14 AM
4,303 Views

Hi AndriiV,

 

Thank you for reaching out to the Intel community. We sincerely apologize for the delay in our response.

 

We acknowledge receipt of your concern and want to assure you that assisting you is our top priority. To effectively troubleshoot the issue you're facing, we require some additional information from your end.

 

Could you kindly provide the following details?

  1. System Model:
  2. Operating System:
  3. Is the Ethernet card embedded on the board?
  4. Was the Ethernet card working before? Have any changes been made recently?

 

Your prompt response with this information will greatly assist us in diagnosing and resolving the issue as quickly as possible.

 

We look forward to hearing from you soon.

 

Best Regards,

Simon

0 Kudos
Copy link
AndriiV
Novice
‎04-17-2024 09:58 AM
4,293 Views
In response to Simon-Intel

Hello Simon!

1.  Supermicro SuperServer 1029U-TRTP2

2.  Debian 10.13

# uname -a
Linux 15224 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux

3. No

4. No,  I am using XL710 card for a first time

In response to Simon-Intel
0 Kudos
Copy link
Simon-Intel
Employee
‎04-18-2024 06:41 AM
4,265 Views
In response to AndriiV

Hi AndriiV,

 

Thank you for sharing the requested details.

 

As you mentioned, there is another server with a similar configuration, but with different network cards. Could you please share the server motherboard and the network card details?

 

We look forward to hearing from you soon.

 

Best Regards,

Simon

In response to AndriiV
0 Kudos
Copy link
AndriiV
Novice
‎04-18-2024 01:12 PM
4,260 Views
In response to Simon-Intel

Hi Simon!

 

Supermicro SuperServer 1028U-TR4+

Motherboard: X10DRU-i+

Debian 10.13

# uname -a
Linux 157540 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08) x86_64 GNU/Linux

 4x10Gbps network cards:

87:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
87:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
88:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535

 

In response to Simon-Intel
0 Kudos
Copy link
Simon-Intel
Employee
‎04-19-2024 01:29 AM
4,238 Views
In response to AndriiV

Hi Andrii,

Good day!

Thank you for providing the requested details.

We are currently investigating this inquiry with our resources, and we will provide you with an update at the earliest opportunity.

Thank you for your patience and understanding.

Best regards,
Simon

In response to AndriiV
Copy link
Simon-Intel
Employee
‎04-24-2024 04:57 AM
4,135 Views

Hi Andrii,

 

Good Day!

 

Thank you for your patience as we work to resolve the issue you've encountered. We at Intel are fully committed to ensuring your satisfaction.

 

To address your concern, please refer to the following link for detailed instructions: Intel Ethernet 700 Series Linux Performance Tuning Guide - Section 4.1.

 

Your understanding and cooperation are greatly appreciated.

 

Best regards,

Simon


0 Kudos
Copy link
AndriiV
Novice
‎04-24-2024 12:22 PM
4,117 Views
In response to Simon-Intel

Hi Simon!

As you can see my first message in this topic. I already read this guide and made all tunnings. I tried

 

382  set_irq_affinity local ens2
  384  set_irq_affinity all ens2

 

 With "local" configuration 100% IRQ on some cores at 19Gbps yet. You can see atop screen after 

set_irq_affinity local ens2

atop_3.png

In response to Simon-Intel
0 Kudos
Copy link
AndriiV
Novice
‎04-25-2024 12:27 AM
4,088 Views

I think the problem is that the traffic is unevenly distributed across the queues

0 Kudos
Copy link
Simon-Intel
Employee
‎04-25-2024 05:18 AM
4,068 Views

Hi Andrii,


Good Day!


Firstly, I'd like to extend our apologies for any inconvenience caused by our previous communication. Your detailed explanation of the issue and server configuration was immensely helpful, and we appreciate your patience and cooperation.


I'm writing to inform you that we've escalated the matter to our next level team for further investigation and resolution. Rest assured, we're prioritizing this issue to ensure a swift resolution.


We will keep you updated on the progress and provide any necessary information as soon as it becomes available.


Once again, thank you for your understanding and cooperation throughout this process.


Best regards,

Simon


0 Kudos
Copy link
Azeem_Intel
Employee
‎05-16-2024 05:24 PM
3,764 Views

Hi AndriiV,

 

Greetings for the day!

 

 

Thank you for your patience. We attempted to replicate the issue in the lab with the latest drivers/NVM (FVL9.5) but were unable to do so.

 

Note: Results are the same whether I tune the interface level or not. This indicates that I can achieve 37.7Gbps throughput with or without any interface level tuning, and there are also no packet drops.

 

Results and tunings can be found in the attached document. Please have a look.

 

Please clarify below question

  1. According to the link you provided, there are two server configurations with different processor details. Does this mean that the issue persists with both configurations?
  2. The customer experimented with different Tx/Rx interrupt coalescing times. Did you observe consistent results each time?
  3. You also attempted to see if there were any packet drops. Are there any drops? We are not seeing any drops.
  4. Are you still experiencing a maximum throughput of 22 Gbps even after multiple tunings?
  5. Could you please share the OS and kernel details of the setup provided by you

 

Your configuration vs our configuration:

you used -> Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz

We used -> Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz

 

Both CPUs offer high performance , but differences between them may affect performance depending on the workload. The details are as follows::

  1. No.of cores and threads
  2. Clock speed
  3. TDP (Thermal Design Power)
  4. Cache size.

 

Could you please try the tunings recommended in the attached document?

 

Best Regards,

Azeem_Intel

Preview file
259 KB
0 Kudos
Copy link
AndriiV
Novice
‎05-17-2024 02:48 AM
3,736 Views
In response to Azeem_Intel

Hi Azeem!

Thanks for your job!

1. As you can see, in the second configuration there are completely different network cards that use a different driver. I provided the second configuration rather to show that these two servers are used in the same data center, in the same project, almost the same configuration (in particular disks), they are caching servers, the load is distributed between them the same.

 

2. Yes, I didn’t notice any special changes.

3. 14.05.2024 (3 days ago) I updated driver and firmware for the latest.

# ethtool -i ens2
driver: i40e
version: 2.25.7
firmware-version: 9.50 0x8000f163 1.3597.0
expansion-rom-version:
bus-info: 0000:d8:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Packet count for these 3 days:

# ethtool -S ens2
NIC statistics:
     rx_packets: 137882815105
     tx_packets: 274937740003
     rx_bytes: 84537446775570
     tx_bytes: 401450641546052
     rx_errors: 2022092
     tx_errors: 0
     rx_dropped: 229385
     tx_dropped: 0
     collisions: 0
     rx_length_errors: 0
     rx_crc_errors: 0
     rx_unicast: 137882060073
     tx_unicast: 274936560996
     rx_multicast: 0
     tx_multicast: 0
     rx_broadcast: 343515
     tx_broadcast: 0
     rx_unknown_protocol: 0
     tx_linearize: 2313
     tx_force_wb: 0
     tx_busy: 0
     tx_stopped: 5303
     rx_alloc_fail: 0
     rx_pg_alloc_fail: 0
     rx_cache_reuse: 368417417274
     tx-0.packets: 25837685
     tx-0.bytes: 33979734149
     rx-0.packets: 168279842
     rx-0.bytes: 170602932069
     rx-0.xdp.pass: 0
     rx-0.xdp.drop: 0
     rx-0.xdp.tx: 0
     rx-0.xdp.unknown: 0
     rx-0.xdp.redirect: 0
     rx-0.xdp.redirect_fail: 0
     tx-1.packets: 5355399756
     tx-1.bytes: 7812404286883
     rx-1.packets: 2516813387
     rx-1.bytes: 1774770706455
     rx-1.xdp.pass: 0
     rx-1.xdp.drop: 0
     rx-1.xdp.tx: 0
     rx-1.xdp.unknown: 0
     rx-1.xdp.redirect: 0
     rx-1.xdp.redirect_fail: 0
     tx-2.packets: 21682826
     tx-2.bytes: 28480770887
     rx-2.packets: 161801610
     rx-2.bytes: 164209951123
     rx-2.xdp.pass: 0
     rx-2.xdp.drop: 0
     rx-2.xdp.tx: 0
     rx-2.xdp.unknown: 0
     rx-2.xdp.redirect: 0
     rx-2.xdp.redirect_fail: 0
     tx-3.packets: 5301254825
     tx-3.bytes: 7734386600593
     rx-3.packets: 2494644834
     rx-3.bytes: 1756602843454
     rx-3.xdp.pass: 0
     rx-3.xdp.drop: 0
     rx-3.xdp.tx: 0
     rx-3.xdp.unknown: 0
     rx-3.xdp.redirect: 0
     rx-3.xdp.redirect_fail: 0
     tx-4.packets: 15224069
     tx-4.bytes: 19652026211
     rx-4.packets: 150379555
     rx-4.bytes: 152052109771
     rx-4.xdp.pass: 0
     rx-4.xdp.drop: 0
     rx-4.xdp.tx: 0
     rx-4.xdp.unknown: 0
     rx-4.xdp.redirect: 0
     rx-4.xdp.redirect_fail: 0
     tx-5.packets: 5254224120
     tx-5.bytes: 7666284532673
     rx-5.packets: 2478075581
     rx-5.bytes: 1743109229012
     rx-5.xdp.pass: 0
     rx-5.xdp.drop: 0
     rx-5.xdp.tx: 0
     rx-5.xdp.unknown: 0
     rx-5.xdp.redirect: 0
     rx-5.xdp.redirect_fail: 0
     tx-6.packets: 11111160
     tx-6.bytes: 14072825797
     rx-6.packets: 147614285
     rx-6.bytes: 151208273615
     rx-6.xdp.pass: 0
     rx-6.xdp.drop: 0
     rx-6.xdp.tx: 0
     rx-6.xdp.unknown: 0
     rx-6.xdp.redirect: 0
     rx-6.xdp.redirect_fail: 0
     tx-7.packets: 5231649268
     tx-7.bytes: 7633880383497
     rx-7.packets: 2467490966
     rx-7.bytes: 1727873025410
     rx-7.xdp.pass: 0
     rx-7.xdp.drop: 0
     rx-7.xdp.tx: 0
     rx-7.xdp.unknown: 0
     rx-7.xdp.redirect: 0
     rx-7.xdp.redirect_fail: 0
     tx-8.packets: 9928118
     tx-8.bytes: 12342185197
     rx-8.packets: 144579783
     rx-8.bytes: 147087173052
     rx-8.xdp.pass: 0
     rx-8.xdp.drop: 0
     rx-8.xdp.tx: 0
     rx-8.xdp.unknown: 0
     rx-8.xdp.redirect: 0
     rx-8.xdp.redirect_fail: 0
     tx-9.packets: 5204698936
     tx-9.bytes: 7595022219312
     rx-9.packets: 2449855836
     rx-9.bytes: 1707250231577
     rx-9.xdp.pass: 0
     rx-9.xdp.drop: 0
     rx-9.xdp.tx: 0
     rx-9.xdp.unknown: 0
     rx-9.xdp.redirect: 0
     rx-9.xdp.redirect_fail: 0
     tx-10.packets: 8175661
     tx-10.bytes: 10073920773
     rx-10.packets: 142968142
     rx-10.bytes: 146695366930
     rx-10.xdp.pass: 0
     rx-10.xdp.drop: 0
     rx-10.xdp.tx: 0
     rx-10.xdp.unknown: 0
     rx-10.xdp.redirect: 0
     rx-10.xdp.redirect_fail: 0
     tx-11.packets: 5157512915
     tx-11.bytes: 7526504135406
     rx-11.packets: 2427772204
     rx-11.bytes: 1684990668038
     rx-11.xdp.pass: 0
     rx-11.xdp.drop: 0
     rx-11.xdp.tx: 0
     rx-11.xdp.unknown: 0
     rx-11.xdp.redirect: 0
     rx-11.xdp.redirect_fail: 0
     tx-12.packets: 7585912
     tx-12.bytes: 9201941239
     rx-12.packets: 143219886
     rx-12.bytes: 146971694227
     rx-12.xdp.pass: 0
     rx-12.xdp.drop: 0
     rx-12.xdp.tx: 0
     rx-12.xdp.unknown: 0
     rx-12.xdp.redirect: 0
     rx-12.xdp.redirect_fail: 0
     tx-13.packets: 5153818605
     tx-13.bytes: 7521098070323
     rx-13.packets: 2433463509
     rx-13.bytes: 1692067903822
     rx-13.xdp.pass: 0
     rx-13.xdp.drop: 0
     rx-13.xdp.tx: 0
     rx-13.xdp.unknown: 0
     rx-13.xdp.redirect: 0
     rx-13.xdp.redirect_fail: 0
     tx-14.packets: 6733993
     tx-14.bytes: 8021367196
     rx-14.packets: 141110460
     rx-14.bytes: 144718927156
     rx-14.xdp.pass: 0
     rx-14.xdp.drop: 0
     rx-14.xdp.tx: 0
     rx-14.xdp.unknown: 0
     rx-14.xdp.redirect: 0
     rx-14.xdp.redirect_fail: 0
     tx-15.packets: 5146347509
     tx-15.bytes: 7510402462613
     rx-15.packets: 2429800226
     rx-15.bytes: 1686744528732
     rx-15.xdp.pass: 0
     rx-15.xdp.drop: 0
     rx-15.xdp.tx: 0
     rx-15.xdp.unknown: 0
     rx-15.xdp.redirect: 0
     rx-15.xdp.redirect_fail: 0
     tx-16.packets: 6288014
     tx-16.bytes: 7422482221
     rx-16.packets: 142737163
     rx-16.bytes: 147757208628
     rx-16.xdp.pass: 0
     rx-16.xdp.drop: 0
     rx-16.xdp.tx: 0
     rx-16.xdp.unknown: 0
     rx-16.xdp.redirect: 0
     rx-16.xdp.redirect_fail: 0
     tx-17.packets: 5153273892
     tx-17.bytes: 7520748979663
     rx-17.packets: 2431759357
     rx-17.bytes: 1682905350979
     rx-17.xdp.pass: 0
     rx-17.xdp.drop: 0
     rx-17.xdp.tx: 0
     rx-17.xdp.unknown: 0
     rx-17.xdp.redirect: 0
     rx-17.xdp.redirect_fail: 0
     tx-18.packets: 5917693
     tx-18.bytes: 6783176550
     rx-18.packets: 144656301
     rx-18.bytes: 149198188478
     rx-18.xdp.pass: 0
     rx-18.xdp.drop: 0
     rx-18.xdp.tx: 0
     rx-18.xdp.unknown: 0
     rx-18.xdp.redirect: 0
     rx-18.xdp.redirect_fail: 0
     tx-19.packets: 5133828421
     tx-19.bytes: 7492497965828
     rx-19.packets: 2426018808
     rx-19.bytes: 1680498388729
     rx-19.xdp.pass: 0
     rx-19.xdp.drop: 0
     rx-19.xdp.tx: 0
     rx-19.xdp.unknown: 0
     rx-19.xdp.redirect: 0
     rx-19.xdp.redirect_fail: 0
     tx-20.packets: 32021056
     tx-20.bytes: 42713819553
     rx-20.packets: 174510718
     rx-20.bytes: 172945330141
     rx-20.xdp.pass: 0
     rx-20.xdp.drop: 0
     rx-20.xdp.tx: 0
     rx-20.xdp.unknown: 0
     rx-20.xdp.redirect: 0
     rx-20.xdp.redirect_fail: 0
     tx-21.packets: 4908767776
     tx-21.bytes: 7155880834589
     rx-21.packets: 2262681036
     rx-21.bytes: 1694548329911
     rx-21.xdp.pass: 0
     rx-21.xdp.drop: 0
     rx-21.xdp.tx: 0
     rx-21.xdp.unknown: 0
     rx-21.xdp.redirect: 0
     rx-21.xdp.redirect_fail: 0
     tx-22.packets: 22942526
     tx-22.bytes: 30257850037
     rx-22.packets: 162688755
     rx-22.bytes: 163256711274
     rx-22.xdp.pass: 0
     rx-22.xdp.drop: 0
     rx-22.xdp.tx: 0
     rx-22.xdp.unknown: 0
     rx-22.xdp.redirect: 0
     rx-22.xdp.redirect_fail: 0
     tx-23.packets: 4823018783
     tx-23.bytes: 7031703420140
     rx-23.packets: 2227592449
     rx-23.bytes: 1666421909016
     rx-23.xdp.pass: 0
     rx-23.xdp.drop: 0
     rx-23.xdp.tx: 0
     rx-23.xdp.unknown: 0
     rx-23.xdp.redirect: 0
     rx-23.xdp.redirect_fail: 0
     tx-24.packets: 16041703
     tx-24.bytes: 20913960784
     rx-24.packets: 155601104
     rx-24.bytes: 157432173356
     rx-24.xdp.pass: 0
     rx-24.xdp.drop: 0
     rx-24.xdp.tx: 0
     rx-24.xdp.unknown: 0
     rx-24.xdp.redirect: 0
     rx-24.xdp.redirect_fail: 0
     tx-25.packets: 4764271626
     tx-25.bytes: 6946701442665
     rx-25.packets: 2205802501
     rx-25.bytes: 1643520544503
     rx-25.xdp.pass: 0
     rx-25.xdp.drop: 0
     rx-25.xdp.tx: 0
     rx-25.xdp.unknown: 0
     rx-25.xdp.redirect: 0
     rx-25.xdp.redirect_fail: 0
     tx-26.packets: 12405218
     tx-26.bytes: 15942960396
     rx-26.packets: 148659356
     rx-26.bytes: 150386448593
     rx-26.xdp.pass: 0
     rx-26.xdp.drop: 0
     rx-26.xdp.tx: 0
     rx-26.xdp.unknown: 0
     rx-26.xdp.redirect: 0
     rx-26.xdp.redirect_fail: 0
     tx-27.packets: 4756551502
     tx-27.bytes: 6936202019721
     rx-27.packets: 2197427464
     rx-27.bytes: 1630471167551
     rx-27.xdp.pass: 0
     rx-27.xdp.drop: 0
     rx-27.xdp.tx: 0
     rx-27.xdp.unknown: 0
     rx-27.xdp.redirect: 0
     rx-27.xdp.redirect_fail: 0
     tx-28.packets: 10151000
     tx-28.bytes: 12774643708
     rx-28.packets: 144960837
     rx-28.bytes: 148599813147
     rx-28.xdp.pass: 0
     rx-28.xdp.drop: 0
     rx-28.xdp.tx: 0
     rx-28.xdp.unknown: 0
     rx-28.xdp.redirect: 0
     rx-28.xdp.redirect_fail: 0
     tx-29.packets: 4732011900
     tx-29.bytes: 6900790435536
     rx-29.packets: 2189226865
     rx-29.bytes: 1620663612882
     rx-29.xdp.pass: 0
     rx-29.xdp.drop: 0
     rx-29.xdp.tx: 0
     rx-29.xdp.unknown: 0
     rx-29.xdp.redirect: 0
     rx-29.xdp.redirect_fail: 0
     tx-30.packets: 8358978
     tx-30.bytes: 10380924736
     rx-30.packets: 142710443
     rx-30.bytes: 145145089022
     rx-30.xdp.pass: 0
     rx-30.xdp.drop: 0
     rx-30.xdp.tx: 0
     rx-30.xdp.unknown: 0
     rx-30.xdp.redirect: 0
     rx-30.xdp.redirect_fail: 0
     tx-31.packets: 4723869582
     tx-31.bytes: 6889695080524
     rx-31.packets: 2185841250
     rx-31.bytes: 1614799018158
     rx-31.xdp.pass: 0
     rx-31.xdp.drop: 0
     rx-31.xdp.tx: 0
     rx-31.xdp.unknown: 0
     rx-31.xdp.redirect: 0
     rx-31.xdp.redirect_fail: 0
     tx-32.packets: 7424933
     tx-32.bytes: 9092723292
     rx-32.packets: 140971438
     rx-32.bytes: 144927121001
     rx-32.xdp.pass: 0
     rx-32.xdp.drop: 0
     rx-32.xdp.tx: 0
     rx-32.xdp.unknown: 0
     rx-32.xdp.redirect: 0
     rx-32.xdp.redirect_fail: 0
     tx-33.packets: 4713949797
     tx-33.bytes: 6875416042641
     rx-33.packets: 2180065654
     rx-33.bytes: 1606146744706
     rx-33.xdp.pass: 0
     rx-33.xdp.drop: 0
     rx-33.xdp.tx: 0
     rx-33.xdp.unknown: 0
     rx-33.xdp.redirect: 0
     rx-33.xdp.redirect_fail: 0
     tx-34.packets: 6645734
     tx-34.bytes: 8032100529
     rx-34.packets: 142889514
     rx-34.bytes: 146829744580
     rx-34.xdp.pass: 0
     rx-34.xdp.drop: 0
     rx-34.xdp.tx: 0
     rx-34.xdp.unknown: 0
     rx-34.xdp.redirect: 0
     rx-34.xdp.redirect_fail: 0
     tx-35.packets: 4705775181
     tx-35.bytes: 6863749652811
     rx-35.packets: 2179507232
     rx-35.bytes: 1602543181560
     rx-35.xdp.pass: 0
     rx-35.xdp.drop: 0
     rx-35.xdp.tx: 0
     rx-35.xdp.unknown: 0
     rx-35.xdp.redirect: 0
     rx-35.xdp.redirect_fail: 0
     tx-36.packets: 6110848
     tx-36.bytes: 7275990490
     rx-36.packets: 141626294
     rx-36.bytes: 146083863407
     rx-36.xdp.pass: 0
     rx-36.xdp.drop: 0
     rx-36.xdp.tx: 0
     rx-36.xdp.unknown: 0
     rx-36.xdp.redirect: 0
     rx-36.xdp.redirect_fail: 0
     tx-37.packets: 4697287850
     tx-37.bytes: 6851595622014
     rx-37.packets: 2173140049
     rx-37.bytes: 1595182400285
     rx-37.xdp.pass: 0
     rx-37.xdp.drop: 0
     rx-37.xdp.tx: 0
     rx-37.xdp.unknown: 0
     rx-37.xdp.redirect: 0
     rx-37.xdp.redirect_fail: 0
     tx-38.packets: 5680927
     tx-38.bytes: 6687879567
     rx-38.packets: 138503460
     rx-38.bytes: 140511672601
     rx-38.xdp.pass: 0
     rx-38.xdp.drop: 0
     rx-38.xdp.tx: 0
     rx-38.xdp.unknown: 0
     rx-38.xdp.redirect: 0
     rx-38.xdp.redirect_fail: 0
     tx-39.packets: 4670505028
     tx-39.bytes: 6812661284648
     rx-39.packets: 2163878145
     rx-39.bytes: 1587768451384
     rx-39.xdp.pass: 0
     rx-39.xdp.drop: 0
     rx-39.xdp.tx: 0
     rx-39.xdp.unknown: 0
     rx-39.xdp.redirect: 0
     rx-39.xdp.redirect_fail: 0
     tx-40.packets: 5582064
     tx-40.bytes: 6417514804
     rx-40.packets: 140648540
     rx-40.bytes: 144846915227
     rx-40.xdp.pass: 0
     rx-40.xdp.drop: 0
     rx-40.xdp.tx: 0
     rx-40.xdp.unknown: 0
     rx-40.xdp.redirect: 0
     rx-40.xdp.redirect_fail: 0
     tx-41.packets: 5202830816
     tx-41.bytes: 7593307389024
     rx-41.packets: 2453481355
     rx-41.bytes: 1695051504856
     rx-41.xdp.pass: 0
     rx-41.xdp.drop: 0
     rx-41.xdp.tx: 0
     rx-41.xdp.unknown: 0
     rx-41.xdp.redirect: 0
     rx-41.xdp.redirect_fail: 0
     tx-42.packets: 5320754
     tx-42.bytes: 6121974895
     rx-42.packets: 141213599
     rx-42.bytes: 146686782915
     rx-42.xdp.pass: 0
     rx-42.xdp.drop: 0
     rx-42.xdp.tx: 0
     rx-42.xdp.unknown: 0
     rx-42.xdp.redirect: 0
     rx-42.xdp.redirect_fail: 0
     tx-43.packets: 5209746697
     tx-43.bytes: 7603607833291
     rx-43.packets: 2457853475
     rx-43.bytes: 1696080937943
     rx-43.xdp.pass: 0
     rx-43.xdp.drop: 0
     rx-43.xdp.tx: 0
     rx-43.xdp.unknown: 0
     rx-43.xdp.redirect: 0
     rx-43.xdp.redirect_fail: 0
     tx-44.packets: 5053836
     tx-44.bytes: 5761081932
     rx-44.packets: 140662343
     rx-44.bytes: 144393302376
     rx-44.xdp.pass: 0
     rx-44.xdp.drop: 0
     rx-44.xdp.tx: 0
     rx-44.xdp.unknown: 0
     rx-44.xdp.redirect: 0
     rx-44.xdp.redirect_fail: 0
     tx-45.packets: 5214859333
     tx-45.bytes: 7611359948938
     rx-45.packets: 2461817439
     rx-45.bytes: 1697943876889
     rx-45.xdp.pass: 0
     rx-45.xdp.drop: 0
     rx-45.xdp.tx: 0
     rx-45.xdp.unknown: 0
     rx-45.xdp.redirect: 0
     rx-45.xdp.redirect_fail: 0
     tx-46.packets: 5051264
     tx-46.bytes: 5734376370
     rx-46.packets: 140476501
     rx-46.bytes: 143676407191
     rx-46.xdp.pass: 0
     rx-46.xdp.drop: 0
     rx-46.xdp.tx: 0
     rx-46.xdp.unknown: 0
     rx-46.xdp.redirect: 0
     rx-46.xdp.redirect_fail: 0
     tx-47.packets: 5220069975
     tx-47.bytes: 7618932224810
     rx-47.packets: 2462508791
     rx-47.bytes: 1697330708846
     rx-47.xdp.pass: 0
     rx-47.xdp.drop: 0
     rx-47.xdp.tx: 0
     rx-47.xdp.unknown: 0
     rx-47.xdp.redirect: 0
     rx-47.xdp.redirect_fail: 0
     tx-48.packets: 4937412
     tx-48.bytes: 5549726840
     rx-48.packets: 142152820
     rx-48.bytes: 145886059390
     rx-48.xdp.pass: 0
     rx-48.xdp.drop: 0
     rx-48.xdp.tx: 0
     rx-48.xdp.unknown: 0
     rx-48.xdp.redirect: 0
     rx-48.xdp.redirect_fail: 0
     tx-49.packets: 5215579828
     tx-49.bytes: 7612462870273
     rx-49.packets: 2461172863
     rx-49.bytes: 1695531339111
     rx-49.xdp.pass: 0
     rx-49.xdp.drop: 0
     rx-49.xdp.tx: 0
     rx-49.xdp.unknown: 0
     rx-49.xdp.redirect: 0
     rx-49.xdp.redirect_fail: 0
     tx-50.packets: 4915707
     tx-50.bytes: 5600595784
     rx-50.packets: 142573143
     rx-50.bytes: 146988083692
     rx-50.xdp.pass: 0
     rx-50.xdp.drop: 0
     rx-50.xdp.tx: 0
     rx-50.xdp.unknown: 0
     rx-50.xdp.redirect: 0
     rx-50.xdp.redirect_fail: 0
     tx-51.packets: 11237559696
     tx-51.bytes: 16437725489984
     rx-51.packets: 5810562202
     rx-51.bytes: 2739136792069
     rx-51.xdp.pass: 0
     rx-51.xdp.drop: 0
     rx-51.xdp.tx: 0
     rx-51.xdp.unknown: 0
     rx-51.xdp.redirect: 0
     rx-51.xdp.redirect_fail: 0
     tx-52.packets: 139105353
     tx-52.bytes: 196440985943
     rx-52.packets: 275597413
     rx-52.bytes: 279146045900
     rx-52.xdp.pass: 0
     rx-52.xdp.drop: 0
     rx-52.xdp.tx: 0
     rx-52.xdp.unknown: 0
     rx-52.xdp.redirect: 0
     rx-52.xdp.redirect_fail: 0
     tx-53.packets: 11150191691
     tx-53.bytes: 16307118398062
     rx-53.packets: 5744895513
     rx-53.bytes: 2729355235029
     rx-53.xdp.pass: 0
     rx-53.xdp.drop: 0
     rx-53.xdp.tx: 0
     rx-53.xdp.unknown: 0
     rx-53.xdp.redirect: 0
     rx-53.xdp.redirect_fail: 0
     tx-54.packets: 139861008
     tx-54.bytes: 197334666026
     rx-54.packets: 277073607
     rx-54.bytes: 281314521879
     rx-54.xdp.pass: 0
     rx-54.xdp.drop: 0
     rx-54.xdp.tx: 0
     rx-54.xdp.unknown: 0
     rx-54.xdp.redirect: 0
     rx-54.xdp.redirect_fail: 0
     tx-55.packets: 11100938191
     tx-55.bytes: 16232183354138
     rx-55.packets: 5712317173
     rx-55.bytes: 2732976654371
     rx-55.xdp.pass: 0
     rx-55.xdp.drop: 0
     rx-55.xdp.tx: 0
     rx-55.xdp.unknown: 0
     rx-55.xdp.redirect: 0
     rx-55.xdp.redirect_fail: 0
     tx-56.packets: 140880238
     tx-56.bytes: 198918939751
     rx-56.packets: 279225002
     rx-56.bytes: 283203726294
     rx-56.xdp.pass: 0
     rx-56.xdp.drop: 0
     rx-56.xdp.tx: 0
     rx-56.xdp.unknown: 0
     rx-56.xdp.redirect: 0
     rx-56.xdp.redirect_fail: 0
     tx-57.packets: 11144476094
     tx-57.bytes: 16292860797206
     rx-57.packets: 5734279270
     rx-57.bytes: 2752886930988
     rx-57.xdp.pass: 0
     rx-57.xdp.drop: 0
     rx-57.xdp.tx: 0
     rx-57.xdp.unknown: 0
     rx-57.xdp.redirect: 0
     rx-57.xdp.redirect_fail: 0
     tx-58.packets: 140597852
     tx-58.bytes: 198633473413
     rx-58.packets: 278895264
     rx-58.bytes: 282489488881
     rx-58.xdp.pass: 0
     rx-58.xdp.drop: 0
     rx-58.xdp.tx: 0
     rx-58.xdp.unknown: 0
     rx-58.xdp.redirect: 0
     rx-58.xdp.redirect_fail: 0
     tx-59.packets: 11051468184
     tx-59.bytes: 16154765733849
     rx-59.packets: 5688479502
     rx-59.bytes: 2752148601359
     rx-59.xdp.pass: 0
     rx-59.xdp.drop: 0
     rx-59.xdp.tx: 0
     rx-59.xdp.unknown: 0
     rx-59.xdp.redirect: 0
     rx-59.xdp.redirect_fail: 0
     tx-60.packets: 142356513
     tx-60.bytes: 201320868538
     rx-60.packets: 278013249
     rx-60.bytes: 281575258605
     rx-60.xdp.pass: 0
     rx-60.xdp.drop: 0
     rx-60.xdp.tx: 0
     rx-60.xdp.unknown: 0
     rx-60.xdp.redirect: 0
     rx-60.xdp.redirect_fail: 0
     tx-61.packets: 9336978756
     tx-61.bytes: 13645828265901
     rx-61.packets: 4736654625
     rx-61.bytes: 2545320363258
     rx-61.xdp.pass: 0
     rx-61.xdp.drop: 0
     rx-61.xdp.tx: 0
     rx-61.xdp.unknown: 0
     rx-61.xdp.redirect: 0
     rx-61.xdp.redirect_fail: 0
     tx-62.packets: 143438727
     tx-62.bytes: 202809194383
     rx-62.packets: 275486795
     rx-62.bytes: 275787966018
     rx-62.xdp.pass: 0
     rx-62.xdp.drop: 0
     rx-62.xdp.tx: 0
     rx-62.xdp.unknown: 0
     rx-62.xdp.redirect: 0
     rx-62.xdp.redirect_fail: 0
     tx-63.packets: 9288100048
     tx-63.bytes: 13571778097557
     rx-63.packets: 4701780623
     rx-63.bytes: 2553460122246
     rx-63.xdp.pass: 0
     rx-63.xdp.drop: 0
     rx-63.xdp.tx: 0
     rx-63.xdp.unknown: 0
     rx-63.xdp.redirect: 0
     rx-63.xdp.redirect_fail: 0
     tx-64.packets: 2172995
     tx-64.bytes: 3013872297
     rx-64.packets: 2338455
     rx-64.bytes: 1646611932
     rx-64.xdp.pass: 0
     rx-64.xdp.drop: 0
     rx-64.xdp.tx: 0
     rx-64.xdp.unknown: 0
     rx-64.xdp.redirect: 0
     rx-64.xdp.redirect_fail: 0
     tx-65.packets: 9201606819
     tx-65.bytes: 13451504319538
     rx-65.packets: 4420621152
     rx-65.bytes: 2259150329806
     rx-65.xdp.pass: 0
     rx-65.xdp.drop: 0
     rx-65.xdp.tx: 0
     rx-65.xdp.unknown: 0
     rx-65.xdp.redirect: 0
     rx-65.xdp.redirect_fail: 0
     tx-66.packets: 1610850
     tx-66.bytes: 2223576415
     rx-66.packets: 1855754
     rx-66.bytes: 1407107993
     rx-66.xdp.pass: 0
     rx-66.xdp.drop: 0
     rx-66.xdp.tx: 0
     rx-66.xdp.unknown: 0
     rx-66.xdp.redirect: 0
     rx-66.xdp.redirect_fail: 0
     tx-67.packets: 9231747566
     tx-67.bytes: 13495294161025
     rx-67.packets: 4429819735
     rx-67.bytes: 2270325037100
     rx-67.xdp.pass: 0
     rx-67.xdp.drop: 0
     rx-67.xdp.tx: 0
     rx-67.xdp.unknown: 0
     rx-67.xdp.redirect: 0
     rx-67.xdp.redirect_fail: 0
     tx-68.packets: 1393370
     tx-68.bytes: 1896504785
     rx-68.packets: 1709552
     rx-68.bytes: 1410020474
     rx-68.xdp.pass: 0
     rx-68.xdp.drop: 0
     rx-68.xdp.tx: 0
     rx-68.xdp.unknown: 0
     rx-68.xdp.redirect: 0
     rx-68.xdp.redirect_fail: 0
     tx-69.packets: 9180497539
     tx-69.bytes: 13418984742271
     rx-69.packets: 4406785411
     rx-69.bytes: 2270700790686
     rx-69.xdp.pass: 0
     rx-69.xdp.drop: 0
     rx-69.xdp.tx: 0
     rx-69.xdp.unknown: 0
     rx-69.xdp.redirect: 0
     rx-69.xdp.redirect_fail: 0
     tx-70.packets: 1205866
     tx-70.bytes: 1638544790
     rx-70.packets: 1546092
     rx-70.bytes: 1293311625
     rx-70.xdp.pass: 0
     rx-70.xdp.drop: 0
     rx-70.xdp.tx: 0
     rx-70.xdp.unknown: 0
     rx-70.xdp.redirect: 0
     rx-70.xdp.redirect_fail: 0
     tx-71.packets: 9235493303
     tx-71.bytes: 13498947850921
     rx-71.packets: 4432466509
     rx-71.bytes: 2286754563991
     rx-71.xdp.pass: 0
     rx-71.xdp.drop: 0
     rx-71.xdp.tx: 0
     rx-71.xdp.unknown: 0
     rx-71.xdp.redirect: 0
     rx-71.xdp.redirect_fail: 0
     tx-72.packets: 1020837
     tx-72.bytes: 1398334919
     rx-72.packets: 1290369
     rx-72.bytes: 1107113237
     rx-72.xdp.pass: 0
     rx-72.xdp.drop: 0
     rx-72.xdp.tx: 0
     rx-72.xdp.unknown: 0
     rx-72.xdp.redirect: 0
     rx-72.xdp.redirect_fail: 0
     tx-73.packets: 9231228324
     tx-73.bytes: 13491926606844
     rx-73.packets: 4428966870
     rx-73.bytes: 2289089162198
     rx-73.xdp.pass: 0
     rx-73.xdp.drop: 0
     rx-73.xdp.tx: 0
     rx-73.xdp.unknown: 0
     rx-73.xdp.redirect: 0
     rx-73.xdp.redirect_fail: 0
     tx-74.packets: 902171
     tx-74.bytes: 1178268786
     rx-74.packets: 1123631
     rx-74.bytes: 999317218
     rx-74.xdp.pass: 0
     rx-74.xdp.drop: 0
     rx-74.xdp.tx: 0
     rx-74.xdp.unknown: 0
     rx-74.xdp.redirect: 0
     rx-74.xdp.redirect_fail: 0
     tx-75.packets: 9240675397
     tx-75.bytes: 13505896095485
     rx-75.packets: 4429361136
     rx-75.bytes: 2288051533556
     rx-75.xdp.pass: 0
     rx-75.xdp.drop: 0
     rx-75.xdp.tx: 0
     rx-75.xdp.unknown: 0
     rx-75.xdp.redirect: 0
     rx-75.xdp.redirect_fail: 0
     tx-76.packets: 796770
     tx-76.bytes: 1051531227
     rx-76.packets: 1187761
     rx-76.bytes: 1155412577
     rx-76.xdp.pass: 0
     rx-76.xdp.drop: 0
     rx-76.xdp.tx: 0
     rx-76.xdp.unknown: 0
     rx-76.xdp.redirect: 0
     rx-76.xdp.redirect_fail: 0
     tx-77.packets: 9291563508
     tx-77.bytes: 13580117214277
     rx-77.packets: 4453232039
     rx-77.bytes: 2296535519245
     rx-77.xdp.pass: 0
     rx-77.xdp.drop: 0
     rx-77.xdp.tx: 0
     rx-77.xdp.unknown: 0
     rx-77.xdp.redirect: 0
     rx-77.xdp.redirect_fail: 0
     tx-78.packets: 723753
     tx-78.bytes: 955646562
     rx-78.packets: 1181701
     rx-78.bytes: 1198728585
     rx-78.xdp.pass: 0
     rx-78.xdp.drop: 0
     rx-78.xdp.tx: 0
     rx-78.xdp.unknown: 0
     rx-78.xdp.redirect: 0
     rx-78.xdp.redirect_fail: 0
     tx-79.packets: 9230915572
     tx-79.bytes: 13490311718806
     rx-79.packets: 4430181562
     rx-79.bytes: 2287906566917
     rx-79.xdp.pass: 0
     rx-79.xdp.drop: 0
     rx-79.xdp.tx: 0
     rx-79.xdp.unknown: 0
     rx-79.xdp.redirect: 0
     rx-79.xdp.redirect_fail: 0
     veb.rx_bytes: 0
     veb.tx_bytes: 0
     veb.rx_unicast: 0
     veb.tx_unicast: 0
     veb.rx_multicast: 0
     veb.tx_multicast: 0
     veb.rx_broadcast: 0
     veb.tx_broadcast: 0
     veb.rx_discards: 0
     veb.tx_discards: 0
     veb.tx_errors: 0
     veb.rx_unknown_protocol: 0
     veb.tc_0_tx_packets: 0
     veb.tc_0_tx_bytes: 0
     veb.tc_0_rx_packets: 0
     veb.tc_0_rx_bytes: 0
     veb.tc_1_tx_packets: 0
     veb.tc_1_tx_bytes: 0
     veb.tc_1_rx_packets: 0
     veb.tc_1_rx_bytes: 0
     veb.tc_2_tx_packets: 0
     veb.tc_2_tx_bytes: 0
     veb.tc_2_rx_packets: 0
     veb.tc_2_rx_bytes: 0
     veb.tc_3_tx_packets: 0
     veb.tc_3_tx_bytes: 0
     veb.tc_3_rx_packets: 0
     veb.tc_3_rx_bytes: 0
     veb.tc_4_tx_packets: 0
     veb.tc_4_tx_bytes: 0
     veb.tc_4_rx_packets: 0
     veb.tc_4_rx_bytes: 0
     veb.tc_5_tx_packets: 0
     veb.tc_5_tx_bytes: 0
     veb.tc_5_rx_packets: 0
     veb.tc_5_rx_bytes: 0
     veb.tc_6_tx_packets: 0
     veb.tc_6_tx_bytes: 0
     veb.tc_6_rx_packets: 0
     veb.tc_6_rx_bytes: 0
     veb.tc_7_tx_packets: 0
     veb.tc_7_tx_bytes: 0
     veb.tc_7_rx_packets: 0
     veb.tc_7_rx_bytes: 0
     port.rx_bytes: 85089469570155
     port.tx_bytes: 121076154453173
     port.rx_unicast: 137883084747
     port.tx_unicast: 274937794383
     port.rx_multicast: 9128
     port.tx_multicast: 8467
     port.rx_broadcast: 343517
     port.tx_broadcast: 1
     port.tx_errors: 0
     port.rx_dropped: 354515
     port.tx_dropped_link_down: 0
     port.rx_crc_errors: 0
     port.illegal_bytes: 0
     port.mac_local_faults: 0
     port.mac_remote_faults: 0
     port.tx_timeout: 0
     port.rx_csum_bad: 2022092
     port.rx_length_errors: 0
     port.link_xon_rx: 0
     port.link_xoff_rx: 0
     port.link_xon_tx: 0
     port.link_xoff_tx: 0
     port.rx_size_64: 21863233950
     port.rx_size_127: 64037694392
     port.rx_size_255: 17192125
     port.rx_size_511: 16389377
     port.rx_size_1023: 190328800
     port.rx_size_1522: 51758598761
     port.rx_size_big: 0
     port.tx_size_64: 116808966
     port.tx_size_127: 2546764083
     port.tx_size_255: 168710866
     port.tx_size_511: 519025380
     port.tx_size_1023: 1553551118
     port.tx_size_1522: 270032942462
     port.tx_size_big: 0
     port.rx_undersize: 0
     port.rx_fragments: 0
     port.rx_oversize: 0
     port.rx_jabber: 0
     port.VF_admin_queue_requests: 0
     port.arq_overflows: 0
     port.tx_hwtstamp_timeouts: 0
     port.rx_hwtstamp_cleared: 0
     port.tx_hwtstamp_skipped: 0
     port.fdir_flush_cnt: 9079
     port.fdir_atr_match: 129856552330
     port.fdir_atr_tunnel_match: 0
     port.fdir_atr_status: 1
     port.fdir_sb_match: 0
     port.fdir_sb_status: 1
     port.tx_lpi_status: 0
     port.rx_lpi_status: 0
     port.tx_lpi_count: 0
     port.rx_lpi_count: 0
     port.tx_lpi_duration: 0
     port.rx_lpi_duration: 0
     port.tx_priority_0_xon_tx: 0
     port.tx_priority_0_xoff_tx: 0
     port.rx_priority_0_xon_rx: 0
     port.rx_priority_0_xoff_rx: 0
     port.rx_priority_0_xon_2_xoff: 0
     port.tx_priority_1_xon_tx: 0
     port.tx_priority_1_xoff_tx: 0
     port.rx_priority_1_xon_rx: 0
     port.rx_priority_1_xoff_rx: 0
     port.rx_priority_1_xon_2_xoff: 0
     port.tx_priority_2_xon_tx: 0
     port.tx_priority_2_xoff_tx: 0
     port.rx_priority_2_xon_rx: 0
     port.rx_priority_2_xoff_rx: 0
     port.rx_priority_2_xon_2_xoff: 0
     port.tx_priority_3_xon_tx: 0
     port.tx_priority_3_xoff_tx: 0
     port.rx_priority_3_xon_rx: 0
     port.rx_priority_3_xoff_rx: 0
     port.rx_priority_3_xon_2_xoff: 0
     port.tx_priority_4_xon_tx: 0
     port.tx_priority_4_xoff_tx: 0
     port.rx_priority_4_xon_rx: 0
     port.rx_priority_4_xoff_rx: 0
     port.rx_priority_4_xon_2_xoff: 0
     port.tx_priority_5_xon_tx: 0
     port.tx_priority_5_xoff_tx: 0
     port.rx_priority_5_xon_rx: 0
     port.rx_priority_5_xoff_rx: 0
     port.rx_priority_5_xon_2_xoff: 0
     port.tx_priority_6_xon_tx: 0
     port.tx_priority_6_xoff_tx: 0
     port.rx_priority_6_xon_rx: 0
     port.rx_priority_6_xoff_rx: 0
     port.rx_priority_6_xon_2_xoff: 0
     port.tx_priority_7_xon_tx: 0
     port.tx_priority_7_xoff_tx: 0
     port.rx_priority_7_xon_rx: 0
     port.rx_priority_7_xoff_rx: 0
     port.rx_priority_7_xon_2_xoff: 0
     vf000.rx_bytes: 0
     vf000.rx_unicast: 0
     vf000.rx_multicast: 0
     vf000.rx_broadcast: 0
     vf000.rx_discards: 0
     vf000.rx_unknown_protocol: 0
     vf000.tx_bytes: 0
     vf000.tx_unicast: 0
     vf000.tx_multicast: 0
     vf000.tx_broadcast: 0
     vf000.tx_discards: 0
     vf000.tx_errors: 0
     vf001.rx_bytes: 0
     vf001.rx_unicast: 0
     vf001.rx_multicast: 0
     vf001.rx_broadcast: 0
     vf001.rx_discards: 0
     vf001.rx_unknown_protocol: 0
     vf001.tx_bytes: 0
     vf001.tx_unicast: 0
     vf001.tx_multicast: 0
     vf001.tx_broadcast: 0
     vf001.tx_discards: 0
     vf001.tx_errors: 0
     vf002.rx_bytes: 0
     vf002.rx_unicast: 0
     vf002.rx_multicast: 0
     vf002.rx_broadcast: 0
     vf002.rx_discards: 0
     vf002.rx_unknown_protocol: 0
     vf002.tx_bytes: 0
     vf002.tx_unicast: 0
     vf002.tx_multicast: 0
     vf002.tx_broadcast: 0
     vf002.tx_discards: 0
     vf002.tx_errors: 0
     vf003.rx_bytes: 0
     vf003.rx_unicast: 0
     vf003.rx_multicast: 0
     vf003.rx_broadcast: 0
     vf003.rx_discards: 0
     vf003.rx_unknown_protocol: 0
     vf003.tx_bytes: 0
     vf003.tx_unicast: 0
     vf003.tx_multicast: 0
     vf003.tx_broadcast: 0
     vf003.tx_discards: 0
     vf003.tx_errors: 0
     vf004.rx_bytes: 0
     vf004.rx_unicast: 0
     vf004.rx_multicast: 0
     vf004.rx_broadcast: 0
     vf004.rx_discards: 0
     vf004.rx_unknown_protocol: 0
     vf004.tx_bytes: 0
     vf004.tx_unicast: 0
     vf004.tx_multicast: 0
     vf004.tx_broadcast: 0
     vf004.tx_discards: 0
     vf004.tx_errors: 0
     vf005.rx_bytes: 0
     vf005.rx_unicast: 0
     vf005.rx_multicast: 0
     vf005.rx_broadcast: 0
     vf005.rx_discards: 0
     vf005.rx_unknown_protocol: 0
     vf005.tx_bytes: 0
     vf005.tx_unicast: 0
     vf005.tx_multicast: 0
     vf005.tx_broadcast: 0
     vf005.tx_discards: 0
     vf005.tx_errors: 0
     vf006.rx_bytes: 0
     vf006.rx_unicast: 0
     vf006.rx_multicast: 0
     vf006.rx_broadcast: 0
     vf006.rx_discards: 0
     vf006.rx_unknown_protocol: 0
     vf006.tx_bytes: 0
     vf006.tx_unicast: 0
     vf006.tx_multicast: 0
     vf006.tx_broadcast: 0
     vf006.tx_discards: 0
     vf006.tx_errors: 0
     vf007.rx_bytes: 0
     vf007.rx_unicast: 0
     vf007.rx_multicast: 0
     vf007.rx_broadcast: 0
     vf007.rx_discards: 0
     vf007.rx_unknown_protocol: 0
     vf007.tx_bytes: 0
     vf007.tx_unicast: 0
     vf007.tx_multicast: 0
     vf007.tx_broadcast: 0
     vf007.tx_discards: 0
     vf007.tx_errors: 0
     vf008.rx_bytes: 0
     vf008.rx_unicast: 0
     vf008.rx_multicast: 0
     vf008.rx_broadcast: 0
     vf008.rx_discards: 0
     vf008.rx_unknown_protocol: 0
     vf008.tx_bytes: 0
     vf008.tx_unicast: 0
     vf008.tx_multicast: 0
     vf008.tx_broadcast: 0
     vf008.tx_discards: 0
     vf008.tx_errors: 0
     vf009.rx_bytes: 0
     vf009.rx_unicast: 0
     vf009.rx_multicast: 0
     vf009.rx_broadcast: 0
     vf009.rx_discards: 0
     vf009.rx_unknown_protocol: 0
     vf009.tx_bytes: 0
     vf009.tx_unicast: 0
     vf009.tx_multicast: 0
     vf009.tx_broadcast: 0
     vf009.tx_discards: 0
     vf009.tx_errors: 0
     vf010.rx_bytes: 0
     vf010.rx_unicast: 0
     vf010.rx_multicast: 0
     vf010.rx_broadcast: 0
     vf010.rx_discards: 0
     vf010.rx_unknown_protocol: 0
     vf010.tx_bytes: 0
     vf010.tx_unicast: 0
     vf010.tx_multicast: 0
     vf010.tx_broadcast: 0
     vf010.tx_discards: 0
     vf010.tx_errors: 0
     vf011.rx_bytes: 0
     vf011.rx_unicast: 0
     vf011.rx_multicast: 0
     vf011.rx_broadcast: 0
     vf011.rx_discards: 0
     vf011.rx_unknown_protocol: 0
     vf011.tx_bytes: 0
     vf011.tx_unicast: 0
     vf011.tx_multicast: 0
     vf011.tx_broadcast: 0
     vf011.tx_discards: 0
     vf011.tx_errors: 0
     vf012.rx_bytes: 0
     vf012.rx_unicast: 0
     vf012.rx_multicast: 0
     vf012.rx_broadcast: 0
     vf012.rx_discards: 0
     vf012.rx_unknown_protocol: 0
     vf012.tx_bytes: 0
     vf012.tx_unicast: 0
     vf012.tx_multicast: 0
     vf012.tx_broadcast: 0
     vf012.tx_discards: 0
     vf012.tx_errors: 0
     vf013.rx_bytes: 0
     vf013.rx_unicast: 0
     vf013.rx_multicast: 0
     vf013.rx_broadcast: 0
     vf013.rx_discards: 0
     vf013.rx_unknown_protocol: 0
     vf013.tx_bytes: 0
     vf013.tx_unicast: 0
     vf013.tx_multicast: 0
     vf013.tx_broadcast: 0
     vf013.tx_discards: 0
     vf013.tx_errors: 0
     vf014.rx_bytes: 0
     vf014.rx_unicast: 0
     vf014.rx_multicast: 0
     vf014.rx_broadcast: 0
     vf014.rx_discards: 0
     vf014.rx_unknown_protocol: 0
     vf014.tx_bytes: 0
     vf014.tx_unicast: 0
     vf014.tx_multicast: 0
     vf014.tx_broadcast: 0
     vf014.tx_discards: 0
     vf014.tx_errors: 0
     vf015.rx_bytes: 0
     vf015.rx_unicast: 0
     vf015.rx_multicast: 0
     vf015.rx_broadcast: 0
     vf015.rx_discards: 0
     vf015.rx_unknown_protocol: 0
     vf015.tx_bytes: 0
     vf015.tx_unicast: 0
     vf015.tx_multicast: 0
     vf015.tx_broadcast: 0
     vf015.tx_discards: 0
     vf015.tx_errors: 0
     vf016.rx_bytes: 0
     vf016.rx_unicast: 0
     vf016.rx_multicast: 0
     vf016.rx_broadcast: 0
     vf016.rx_discards: 0
     vf016.rx_unknown_protocol: 0
     vf016.tx_bytes: 0
     vf016.tx_unicast: 0
     vf016.tx_multicast: 0
     vf016.tx_broadcast: 0
     vf016.tx_discards: 0
     vf016.tx_errors: 0
     vf017.rx_bytes: 0
     vf017.rx_unicast: 0
     vf017.rx_multicast: 0
     vf017.rx_broadcast: 0
     vf017.rx_discards: 0
     vf017.rx_unknown_protocol: 0
     vf017.tx_bytes: 0
     vf017.tx_unicast: 0
     vf017.tx_multicast: 0
     vf017.tx_broadcast: 0
     vf017.tx_discards: 0
     vf017.tx_errors: 0
     vf018.rx_bytes: 0
     vf018.rx_unicast: 0
     vf018.rx_multicast: 0
     vf018.rx_broadcast: 0
     vf018.rx_discards: 0
     vf018.rx_unknown_protocol: 0
     vf018.tx_bytes: 0
     vf018.tx_unicast: 0
     vf018.tx_multicast: 0
     vf018.tx_broadcast: 0
     vf018.tx_discards: 0
     vf018.tx_errors: 0
     vf019.rx_bytes: 0
     vf019.rx_unicast: 0
     vf019.rx_multicast: 0
     vf019.rx_broadcast: 0
     vf019.rx_discards: 0
     vf019.rx_unknown_protocol: 0
     vf019.tx_bytes: 0
     vf019.tx_unicast: 0
     vf019.tx_multicast: 0
     vf019.tx_broadcast: 0
     vf019.tx_discards: 0
     vf019.tx_errors: 0
     vf020.rx_bytes: 0
     vf020.rx_unicast: 0
     vf020.rx_multicast: 0
     vf020.rx_broadcast: 0
     vf020.rx_discards: 0
     vf020.rx_unknown_protocol: 0
     vf020.tx_bytes: 0
     vf020.tx_unicast: 0
     vf020.tx_multicast: 0
     vf020.tx_broadcast: 0
     vf020.tx_discards: 0
     vf020.tx_errors: 0
     vf021.rx_bytes: 0
     vf021.rx_unicast: 0
     vf021.rx_multicast: 0
     vf021.rx_broadcast: 0
     vf021.rx_discards: 0
     vf021.rx_unknown_protocol: 0
     vf021.tx_bytes: 0
     vf021.tx_unicast: 0
     vf021.tx_multicast: 0
     vf021.tx_broadcast: 0
     vf021.tx_discards: 0
     vf021.tx_errors: 0
     vf022.rx_bytes: 0
     vf022.rx_unicast: 0
     vf022.rx_multicast: 0
     vf022.rx_broadcast: 0
     vf022.rx_discards: 0
     vf022.rx_unknown_protocol: 0
     vf022.tx_bytes: 0
     vf022.tx_unicast: 0
     vf022.tx_multicast: 0
     vf022.tx_broadcast: 0
     vf022.tx_discards: 0
     vf022.tx_errors: 0
     vf023.rx_bytes: 0
     vf023.rx_unicast: 0
     vf023.rx_multicast: 0
     vf023.rx_broadcast: 0
     vf023.rx_discards: 0
     vf023.rx_unknown_protocol: 0
     vf023.tx_bytes: 0
     vf023.tx_unicast: 0
     vf023.tx_multicast: 0
     vf023.tx_broadcast: 0
     vf023.tx_discards: 0
     vf023.tx_errors: 0
     vf024.rx_bytes: 0
     vf024.rx_unicast: 0
     vf024.rx_multicast: 0
     vf024.rx_broadcast: 0
     vf024.rx_discards: 0
     vf024.rx_unknown_protocol: 0
     vf024.tx_bytes: 0
     vf024.tx_unicast: 0
     vf024.tx_multicast: 0
     vf024.tx_broadcast: 0
     vf024.tx_discards: 0
     vf024.tx_errors: 0
     vf025.rx_bytes: 0
     vf025.rx_unicast: 0
     vf025.rx_multicast: 0
     vf025.rx_broadcast: 0
     vf025.rx_discards: 0
     vf025.rx_unknown_protocol: 0
     vf025.tx_bytes: 0
     vf025.tx_unicast: 0
     vf025.tx_multicast: 0
     vf025.tx_broadcast: 0
     vf025.tx_discards: 0
     vf025.tx_errors: 0
     vf026.rx_bytes: 0
     vf026.rx_unicast: 0
     vf026.rx_multicast: 0
     vf026.rx_broadcast: 0
     vf026.rx_discards: 0
     vf026.rx_unknown_protocol: 0
     vf026.tx_bytes: 0
     vf026.tx_unicast: 0
     vf026.tx_multicast: 0
     vf026.tx_broadcast: 0
     vf026.tx_discards: 0
     vf026.tx_errors: 0
     vf027.rx_bytes: 0
     vf027.rx_unicast: 0
     vf027.rx_multicast: 0
     vf027.rx_broadcast: 0
     vf027.rx_discards: 0
     vf027.rx_unknown_protocol: 0
     vf027.tx_bytes: 0
     vf027.tx_unicast: 0
     vf027.tx_multicast: 0
     vf027.tx_broadcast: 0
     vf027.tx_discards: 0
     vf027.tx_errors: 0
     vf028.rx_bytes: 0
     vf028.rx_unicast: 0
     vf028.rx_multicast: 0
     vf028.rx_broadcast: 0
     vf028.rx_discards: 0
     vf028.rx_unknown_protocol: 0
     vf028.tx_bytes: 0
     vf028.tx_unicast: 0
     vf028.tx_multicast: 0
     vf028.tx_broadcast: 0
     vf028.tx_discards: 0
     vf028.tx_errors: 0
     vf029.rx_bytes: 0
     vf029.rx_unicast: 0
     vf029.rx_multicast: 0
     vf029.rx_broadcast: 0
     vf029.rx_discards: 0
     vf029.rx_unknown_protocol: 0
     vf029.tx_bytes: 0
     vf029.tx_unicast: 0
     vf029.tx_multicast: 0
     vf029.tx_broadcast: 0
     vf029.tx_discards: 0
     vf029.tx_errors: 0
     vf030.rx_bytes: 0
     vf030.rx_unicast: 0
     vf030.rx_multicast: 0
     vf030.rx_broadcast: 0
     vf030.rx_discards: 0
     vf030.rx_unknown_protocol: 0
     vf030.tx_bytes: 0
     vf030.tx_unicast: 0
     vf030.tx_multicast: 0
     vf030.tx_broadcast: 0
     vf030.tx_discards: 0
     vf030.tx_errors: 0
     vf031.rx_bytes: 0
     vf031.rx_unicast: 0
     vf031.rx_multicast: 0
     vf031.rx_broadcast: 0
     vf031.rx_discards: 0
     vf031.rx_unknown_protocol: 0
     vf031.tx_bytes: 0
     vf031.tx_unicast: 0
     vf031.tx_multicast: 0
     vf031.tx_broadcast: 0
     vf031.tx_discards: 0
     vf031.tx_errors: 0
     vf032.rx_bytes: 0
     vf032.rx_unicast: 0
     vf032.rx_multicast: 0
     vf032.rx_broadcast: 0
     vf032.rx_discards: 0
     vf032.rx_unknown_protocol: 0
     vf032.tx_bytes: 0
     vf032.tx_unicast: 0
     vf032.tx_multicast: 0
     vf032.tx_broadcast: 0
     vf032.tx_discards: 0
     vf032.tx_errors: 0
     vf033.rx_bytes: 0
     vf033.rx_unicast: 0
     vf033.rx_multicast: 0
     vf033.rx_broadcast: 0
     vf033.rx_discards: 0
     vf033.rx_unknown_protocol: 0
     vf033.tx_bytes: 0
     vf033.tx_unicast: 0
     vf033.tx_multicast: 0
     vf033.tx_broadcast: 0
     vf033.tx_discards: 0
     vf033.tx_errors: 0
     vf034.rx_bytes: 0
     vf034.rx_unicast: 0
     vf034.rx_multicast: 0
     vf034.rx_broadcast: 0
     vf034.rx_discards: 0
     vf034.rx_unknown_protocol: 0
     vf034.tx_bytes: 0
     vf034.tx_unicast: 0
     vf034.tx_multicast: 0
     vf034.tx_broadcast: 0
     vf034.tx_discards: 0
     vf034.tx_errors: 0
     vf035.rx_bytes: 0
     vf035.rx_unicast: 0
     vf035.rx_multicast: 0
     vf035.rx_broadcast: 0
     vf035.rx_discards: 0
     vf035.rx_unknown_protocol: 0
     vf035.tx_bytes: 0
     vf035.tx_unicast: 0
     vf035.tx_multicast: 0
     vf035.tx_broadcast: 0
     vf035.tx_discards: 0
     vf035.tx_errors: 0
     vf036.rx_bytes: 0
     vf036.rx_unicast: 0
     vf036.rx_multicast: 0
     vf036.rx_broadcast: 0
     vf036.rx_discards: 0
     vf036.rx_unknown_protocol: 0
     vf036.tx_bytes: 0
     vf036.tx_unicast: 0
     vf036.tx_multicast: 0
     vf036.tx_broadcast: 0
     vf036.tx_discards: 0
     vf036.tx_errors: 0
     vf037.rx_bytes: 0
     vf037.rx_unicast: 0
     vf037.rx_multicast: 0
     vf037.rx_broadcast: 0
     vf037.rx_discards: 0
     vf037.rx_unknown_protocol: 0
     vf037.tx_bytes: 0
     vf037.tx_unicast: 0
     vf037.tx_multicast: 0
     vf037.tx_broadcast: 0
     vf037.tx_discards: 0
     vf037.tx_errors: 0
     vf038.rx_bytes: 0
     vf038.rx_unicast: 0
     vf038.rx_multicast: 0
     vf038.rx_broadcast: 0
     vf038.rx_discards: 0
     vf038.rx_unknown_protocol: 0
     vf038.tx_bytes: 0
     vf038.tx_unicast: 0
     vf038.tx_multicast: 0
     vf038.tx_broadcast: 0
     vf038.tx_discards: 0
     vf038.tx_errors: 0
     vf039.rx_bytes: 0
     vf039.rx_unicast: 0
     vf039.rx_multicast: 0
     vf039.rx_broadcast: 0
     vf039.rx_discards: 0
     vf039.rx_unknown_protocol: 0
     vf039.tx_bytes: 0
     vf039.tx_unicast: 0
     vf039.tx_multicast: 0
     vf039.tx_broadcast: 0
     vf039.tx_discards: 0
     vf039.tx_errors: 0
     vf040.rx_bytes: 0
     vf040.rx_unicast: 0
     vf040.rx_multicast: 0
     vf040.rx_broadcast: 0
     vf040.rx_discards: 0
     vf040.rx_unknown_protocol: 0
     vf040.tx_bytes: 0
     vf040.tx_unicast: 0
     vf040.tx_multicast: 0
     vf040.tx_broadcast: 0
     vf040.tx_discards: 0
     vf040.tx_errors: 0
     vf041.rx_bytes: 0
     vf041.rx_unicast: 0
     vf041.rx_multicast: 0
     vf041.rx_broadcast: 0
     vf041.rx_discards: 0
     vf041.rx_unknown_protocol: 0
     vf041.tx_bytes: 0
     vf041.tx_unicast: 0
     vf041.tx_multicast: 0
     vf041.tx_broadcast: 0
     vf041.tx_discards: 0
     vf041.tx_errors: 0
     vf042.rx_bytes: 0
     vf042.rx_unicast: 0
     vf042.rx_multicast: 0
     vf042.rx_broadcast: 0
     vf042.rx_discards: 0
     vf042.rx_unknown_protocol: 0
     vf042.tx_bytes: 0
     vf042.tx_unicast: 0
     vf042.tx_multicast: 0
     vf042.tx_broadcast: 0
     vf042.tx_discards: 0
     vf042.tx_errors: 0
     vf043.rx_bytes: 0
     vf043.rx_unicast: 0
     vf043.rx_multicast: 0
     vf043.rx_broadcast: 0
     vf043.rx_discards: 0
     vf043.rx_unknown_protocol: 0
     vf043.tx_bytes: 0
     vf043.tx_unicast: 0
     vf043.tx_multicast: 0
     vf043.tx_broadcast: 0
     vf043.tx_discards: 0
     vf043.tx_errors: 0
     vf044.rx_bytes: 0
     vf044.rx_unicast: 0
     vf044.rx_multicast: 0
     vf044.rx_broadcast: 0
     vf044.rx_discards: 0
     vf044.rx_unknown_protocol: 0
     vf044.tx_bytes: 0
     vf044.tx_unicast: 0
     vf044.tx_multicast: 0
     vf044.tx_broadcast: 0
     vf044.tx_discards: 0
     vf044.tx_errors: 0
     vf045.rx_bytes: 0
     vf045.rx_unicast: 0
     vf045.rx_multicast: 0
     vf045.rx_broadcast: 0
     vf045.rx_discards: 0
     vf045.rx_unknown_protocol: 0
     vf045.tx_bytes: 0
     vf045.tx_unicast: 0
     vf045.tx_multicast: 0
     vf045.tx_broadcast: 0
     vf045.tx_discards: 0
     vf045.tx_errors: 0
     vf046.rx_bytes: 0
     vf046.rx_unicast: 0
     vf046.rx_multicast: 0
     vf046.rx_broadcast: 0
     vf046.rx_discards: 0
     vf046.rx_unknown_protocol: 0
     vf046.tx_bytes: 0
     vf046.tx_unicast: 0
     vf046.tx_multicast: 0
     vf046.tx_broadcast: 0
     vf046.tx_discards: 0
     vf046.tx_errors: 0
     vf047.rx_bytes: 0
     vf047.rx_unicast: 0
     vf047.rx_multicast: 0
     vf047.rx_broadcast: 0
     vf047.rx_discards: 0
     vf047.rx_unknown_protocol: 0
     vf047.tx_bytes: 0
     vf047.tx_unicast: 0
     vf047.tx_multicast: 0
     vf047.tx_broadcast: 0
     vf047.tx_discards: 0
     vf047.tx_errors: 0
     vf048.rx_bytes: 0
     vf048.rx_unicast: 0
     vf048.rx_multicast: 0
     vf048.rx_broadcast: 0
     vf048.rx_discards: 0
     vf048.rx_unknown_protocol: 0
     vf048.tx_bytes: 0
     vf048.tx_unicast: 0
     vf048.tx_multicast: 0
     vf048.tx_broadcast: 0
     vf048.tx_discards: 0
     vf048.tx_errors: 0
     vf049.rx_bytes: 0
     vf049.rx_unicast: 0
     vf049.rx_multicast: 0
     vf049.rx_broadcast: 0
     vf049.rx_discards: 0
     vf049.rx_unknown_protocol: 0
     vf049.tx_bytes: 0
     vf049.tx_unicast: 0
     vf049.tx_multicast: 0
     vf049.tx_broadcast: 0
     vf049.tx_discards: 0
     vf049.tx_errors: 0
     vf050.rx_bytes: 0
     vf050.rx_unicast: 0
     vf050.rx_multicast: 0
     vf050.rx_broadcast: 0
     vf050.rx_discards: 0
     vf050.rx_unknown_protocol: 0
     vf050.tx_bytes: 0
     vf050.tx_unicast: 0
     vf050.tx_multicast: 0
     vf050.tx_broadcast: 0
     vf050.tx_discards: 0
     vf050.tx_errors: 0
     vf051.rx_bytes: 0
     vf051.rx_unicast: 0
     vf051.rx_multicast: 0
     vf051.rx_broadcast: 0
     vf051.rx_discards: 0
     vf051.rx_unknown_protocol: 0
     vf051.tx_bytes: 0
     vf051.tx_unicast: 0
     vf051.tx_multicast: 0
     vf051.tx_broadcast: 0
     vf051.tx_discards: 0
     vf051.tx_errors: 0
     vf052.rx_bytes: 0
     vf052.rx_unicast: 0
     vf052.rx_multicast: 0
     vf052.rx_broadcast: 0
     vf052.rx_discards: 0
     vf052.rx_unknown_protocol: 0
     vf052.tx_bytes: 0
     vf052.tx_unicast: 0
     vf052.tx_multicast: 0
     vf052.tx_broadcast: 0
     vf052.tx_discards: 0
     vf052.tx_errors: 0
     vf053.rx_bytes: 0
     vf053.rx_unicast: 0
     vf053.rx_multicast: 0
     vf053.rx_broadcast: 0
     vf053.rx_discards: 0
     vf053.rx_unknown_protocol: 0
     vf053.tx_bytes: 0
     vf053.tx_unicast: 0
     vf053.tx_multicast: 0
     vf053.tx_broadcast: 0
     vf053.tx_discards: 0
     vf053.tx_errors: 0
     vf054.rx_bytes: 0
     vf054.rx_unicast: 0
     vf054.rx_multicast: 0
     vf054.rx_broadcast: 0
     vf054.rx_discards: 0
     vf054.rx_unknown_protocol: 0
     vf054.tx_bytes: 0
     vf054.tx_unicast: 0
     vf054.tx_multicast: 0
     vf054.tx_broadcast: 0
     vf054.tx_discards: 0
     vf054.tx_errors: 0
     vf055.rx_bytes: 0
     vf055.rx_unicast: 0
     vf055.rx_multicast: 0
     vf055.rx_broadcast: 0
     vf055.rx_discards: 0
     vf055.rx_unknown_protocol: 0
     vf055.tx_bytes: 0
     vf055.tx_unicast: 0
     vf055.tx_multicast: 0
     vf055.tx_broadcast: 0
     vf055.tx_discards: 0
     vf055.tx_errors: 0
     vf056.rx_bytes: 0
     vf056.rx_unicast: 0
     vf056.rx_multicast: 0
     vf056.rx_broadcast: 0
     vf056.rx_discards: 0
     vf056.rx_unknown_protocol: 0
     vf056.tx_bytes: 0
     vf056.tx_unicast: 0
     vf056.tx_multicast: 0
     vf056.tx_broadcast: 0
     vf056.tx_discards: 0
     vf056.tx_errors: 0
     vf057.rx_bytes: 0
     vf057.rx_unicast: 0
     vf057.rx_multicast: 0
     vf057.rx_broadcast: 0
     vf057.rx_discards: 0
     vf057.rx_unknown_protocol: 0
     vf057.tx_bytes: 0
     vf057.tx_unicast: 0
     vf057.tx_multicast: 0
     vf057.tx_broadcast: 0
     vf057.tx_discards: 0
     vf057.tx_errors: 0
     vf058.rx_bytes: 0
     vf058.rx_unicast: 0
     vf058.rx_multicast: 0
     vf058.rx_broadcast: 0
     vf058.rx_discards: 0
     vf058.rx_unknown_protocol: 0
     vf058.tx_bytes: 0
     vf058.tx_unicast: 0
     vf058.tx_multicast: 0
     vf058.tx_broadcast: 0
     vf058.tx_discards: 0
     vf058.tx_errors: 0
     vf059.rx_bytes: 0
     vf059.rx_unicast: 0
     vf059.rx_multicast: 0
     vf059.rx_broadcast: 0
     vf059.rx_discards: 0
     vf059.rx_unknown_protocol: 0
     vf059.tx_bytes: 0
     vf059.tx_unicast: 0
     vf059.tx_multicast: 0
     vf059.tx_broadcast: 0
     vf059.tx_discards: 0
     vf059.tx_errors: 0
     vf060.rx_bytes: 0
     vf060.rx_unicast: 0
     vf060.rx_multicast: 0
     vf060.rx_broadcast: 0
     vf060.rx_discards: 0
     vf060.rx_unknown_protocol: 0
     vf060.tx_bytes: 0
     vf060.tx_unicast: 0
     vf060.tx_multicast: 0
     vf060.tx_broadcast: 0
     vf060.tx_discards: 0
     vf060.tx_errors: 0
     vf061.rx_bytes: 0
     vf061.rx_unicast: 0
     vf061.rx_multicast: 0
     vf061.rx_broadcast: 0
     vf061.rx_discards: 0
     vf061.rx_unknown_protocol: 0
     vf061.tx_bytes: 0
     vf061.tx_unicast: 0
     vf061.tx_multicast: 0
     vf061.tx_broadcast: 0
     vf061.tx_discards: 0
     vf061.tx_errors: 0
     vf062.rx_bytes: 0
     vf062.rx_unicast: 0
     vf062.rx_multicast: 0
     vf062.rx_broadcast: 0
     vf062.rx_discards: 0
     vf062.rx_unknown_protocol: 0
     vf062.tx_bytes: 0
     vf062.tx_unicast: 0
     vf062.tx_multicast: 0
     vf062.tx_broadcast: 0
     vf062.tx_discards: 0
     vf062.tx_errors: 0
     vf063.rx_bytes: 0
     vf063.rx_unicast: 0
     vf063.rx_multicast: 0
     vf063.rx_broadcast: 0
     vf063.rx_discards: 0
     vf063.rx_unknown_protocol: 0
     vf063.tx_bytes: 0
     vf063.tx_unicast: 0
     vf063.tx_multicast: 0
     vf063.tx_broadcast: 0
     vf063.tx_discards: 0
     vf063.tx_errors: 0
     vf064.rx_bytes: 0
     vf064.rx_unicast: 0
     vf064.rx_multicast: 0
     vf064.rx_broadcast: 0
     vf064.rx_discards: 0
     vf064.rx_unknown_protocol: 0
     vf064.tx_bytes: 0
     vf064.tx_unicast: 0
     vf064.tx_multicast: 0
     vf064.tx_broadcast: 0
     vf064.tx_discards: 0
     vf064.tx_errors: 0
     vf065.rx_bytes: 0
     vf065.rx_unicast: 0
     vf065.rx_multicast: 0
     vf065.rx_broadcast: 0
     vf065.rx_discards: 0
     vf065.rx_unknown_protocol: 0
     vf065.tx_bytes: 0
     vf065.tx_unicast: 0
     vf065.tx_multicast: 0
     vf065.tx_broadcast: 0
     vf065.tx_discards: 0
     vf065.tx_errors: 0
     vf066.rx_bytes: 0
     vf066.rx_unicast: 0
     vf066.rx_multicast: 0
     vf066.rx_broadcast: 0
     vf066.rx_discards: 0
     vf066.rx_unknown_protocol: 0
     vf066.tx_bytes: 0
     vf066.tx_unicast: 0
     vf066.tx_multicast: 0
     vf066.tx_broadcast: 0
     vf066.tx_discards: 0
     vf066.tx_errors: 0
     vf067.rx_bytes: 0
     vf067.rx_unicast: 0
     vf067.rx_multicast: 0
     vf067.rx_broadcast: 0
     vf067.rx_discards: 0
     vf067.rx_unknown_protocol: 0
     vf067.tx_bytes: 0
     vf067.tx_unicast: 0
     vf067.tx_multicast: 0
     vf067.tx_broadcast: 0
     vf067.tx_discards: 0
     vf067.tx_errors: 0
     vf068.rx_bytes: 0
     vf068.rx_unicast: 0
     vf068.rx_multicast: 0
     vf068.rx_broadcast: 0
     vf068.rx_discards: 0
     vf068.rx_unknown_protocol: 0
     vf068.tx_bytes: 0
     vf068.tx_unicast: 0
     vf068.tx_multicast: 0
     vf068.tx_broadcast: 0
     vf068.tx_discards: 0
     vf068.tx_errors: 0
     vf069.rx_bytes: 0
     vf069.rx_unicast: 0
     vf069.rx_multicast: 0
     vf069.rx_broadcast: 0
     vf069.rx_discards: 0
     vf069.rx_unknown_protocol: 0
     vf069.tx_bytes: 0
     vf069.tx_unicast: 0
     vf069.tx_multicast: 0
     vf069.tx_broadcast: 0
     vf069.tx_discards: 0
     vf069.tx_errors: 0
     vf070.rx_bytes: 0
     vf070.rx_unicast: 0
     vf070.rx_multicast: 0
     vf070.rx_broadcast: 0
     vf070.rx_discards: 0
     vf070.rx_unknown_protocol: 0
     vf070.tx_bytes: 0
     vf070.tx_unicast: 0
     vf070.tx_multicast: 0
     vf070.tx_broadcast: 0
     vf070.tx_discards: 0
     vf070.tx_errors: 0
     vf071.rx_bytes: 0
     vf071.rx_unicast: 0
     vf071.rx_multicast: 0
     vf071.rx_broadcast: 0
     vf071.rx_discards: 0
     vf071.rx_unknown_protocol: 0
     vf071.tx_bytes: 0
     vf071.tx_unicast: 0
     vf071.tx_multicast: 0
     vf071.tx_broadcast: 0
     vf071.tx_discards: 0
     vf071.tx_errors: 0
     vf072.rx_bytes: 0
     vf072.rx_unicast: 0
     vf072.rx_multicast: 0
     vf072.rx_broadcast: 0
     vf072.rx_discards: 0
     vf072.rx_unknown_protocol: 0
     vf072.tx_bytes: 0
     vf072.tx_unicast: 0
     vf072.tx_multicast: 0
     vf072.tx_broadcast: 0
     vf072.tx_discards: 0
     vf072.tx_errors: 0
     vf073.rx_bytes: 0
     vf073.rx_unicast: 0
     vf073.rx_multicast: 0
     vf073.rx_broadcast: 0
     vf073.rx_discards: 0
     vf073.rx_unknown_protocol: 0
     vf073.tx_bytes: 0
     vf073.tx_unicast: 0
     vf073.tx_multicast: 0
     vf073.tx_broadcast: 0
     vf073.tx_discards: 0
     vf073.tx_errors: 0
     vf074.rx_bytes: 0
     vf074.rx_unicast: 0
     vf074.rx_multicast: 0
     vf074.rx_broadcast: 0
     vf074.rx_discards: 0
     vf074.rx_unknown_protocol: 0
     vf074.tx_bytes: 0
     vf074.tx_unicast: 0
     vf074.tx_multicast: 0
     vf074.tx_broadcast: 0
     vf074.tx_discards: 0
     vf074.tx_errors: 0
     vf075.rx_bytes: 0
     vf075.rx_unicast: 0
     vf075.rx_multicast: 0
     vf075.rx_broadcast: 0
     vf075.rx_discards: 0
     vf075.rx_unknown_protocol: 0
     vf075.tx_bytes: 0
     vf075.tx_unicast: 0
     vf075.tx_multicast: 0
     vf075.tx_broadcast: 0
     vf075.tx_discards: 0
     vf075.tx_errors: 0
     vf076.rx_bytes: 0
     vf076.rx_unicast: 0
     vf076.rx_multicast: 0
     vf076.rx_broadcast: 0
     vf076.rx_discards: 0
     vf076.rx_unknown_protocol: 0
     vf076.tx_bytes: 0
     vf076.tx_unicast: 0
     vf076.tx_multicast: 0
     vf076.tx_broadcast: 0
     vf076.tx_discards: 0
     vf076.tx_errors: 0
     vf077.rx_bytes: 0
     vf077.rx_unicast: 0
     vf077.rx_multicast: 0
     vf077.rx_broadcast: 0
     vf077.rx_discards: 0
     vf077.rx_unknown_protocol: 0
     vf077.tx_bytes: 0
     vf077.tx_unicast: 0
     vf077.tx_multicast: 0
     vf077.tx_broadcast: 0
     vf077.tx_discards: 0
     vf077.tx_errors: 0
     vf078.rx_bytes: 0
     vf078.rx_unicast: 0
     vf078.rx_multicast: 0
     vf078.rx_broadcast: 0
     vf078.rx_discards: 0
     vf078.rx_unknown_protocol: 0
     vf078.tx_bytes: 0
     vf078.tx_unicast: 0
     vf078.tx_multicast: 0
     vf078.tx_broadcast: 0
     vf078.tx_discards: 0
     vf078.tx_errors: 0
     vf079.rx_bytes: 0
     vf079.rx_unicast: 0
     vf079.rx_multicast: 0
     vf079.rx_broadcast: 0
     vf079.rx_discards: 0
     vf079.rx_unknown_protocol: 0
     vf079.tx_bytes: 0
     vf079.tx_unicast: 0
     vf079.tx_multicast: 0
     vf079.tx_broadcast: 0
     vf079.tx_discards: 0
     vf079.tx_errors: 0
     vf080.rx_bytes: 0
     vf080.rx_unicast: 0
     vf080.rx_multicast: 0
     vf080.rx_broadcast: 0
     vf080.rx_discards: 0
     vf080.rx_unknown_protocol: 0
     vf080.tx_bytes: 0
     vf080.tx_unicast: 0
     vf080.tx_multicast: 0
     vf080.tx_broadcast: 0
     vf080.tx_discards: 0
     vf080.tx_errors: 0
     vf081.rx_bytes: 0
     vf081.rx_unicast: 0
     vf081.rx_multicast: 0
     vf081.rx_broadcast: 0
     vf081.rx_discards: 0
     vf081.rx_unknown_protocol: 0
     vf081.tx_bytes: 0
     vf081.tx_unicast: 0
     vf081.tx_multicast: 0
     vf081.tx_broadcast: 0
     vf081.tx_discards: 0
     vf081.tx_errors: 0
     vf082.rx_bytes: 0
     vf082.rx_unicast: 0
     vf082.rx_multicast: 0
     vf082.rx_broadcast: 0
     vf082.rx_discards: 0
     vf082.rx_unknown_protocol: 0
     vf082.tx_bytes: 0
     vf082.tx_unicast: 0
     vf082.tx_multicast: 0
     vf082.tx_broadcast: 0
     vf082.tx_discards: 0
     vf082.tx_errors: 0
     vf083.rx_bytes: 0
     vf083.rx_unicast: 0
     vf083.rx_multicast: 0
     vf083.rx_broadcast: 0
     vf083.rx_discards: 0
     vf083.rx_unknown_protocol: 0
     vf083.tx_bytes: 0
     vf083.tx_unicast: 0
     vf083.tx_multicast: 0
     vf083.tx_broadcast: 0
     vf083.tx_discards: 0
     vf083.tx_errors: 0
     vf084.rx_bytes: 0
     vf084.rx_unicast: 0
     vf084.rx_multicast: 0
     vf084.rx_broadcast: 0
     vf084.rx_discards: 0
     vf084.rx_unknown_protocol: 0
     vf084.tx_bytes: 0
     vf084.tx_unicast: 0
     vf084.tx_multicast: 0
     vf084.tx_broadcast: 0
     vf084.tx_discards: 0
     vf084.tx_errors: 0
     vf085.rx_bytes: 0
     vf085.rx_unicast: 0
     vf085.rx_multicast: 0
     vf085.rx_broadcast: 0
     vf085.rx_discards: 0
     vf085.rx_unknown_protocol: 0
     vf085.tx_bytes: 0
     vf085.tx_unicast: 0
     vf085.tx_multicast: 0
     vf085.tx_broadcast: 0
     vf085.tx_discards: 0
     vf085.tx_errors: 0
     vf086.rx_bytes: 0
     vf086.rx_unicast: 0
     vf086.rx_multicast: 0
     vf086.rx_broadcast: 0
     vf086.rx_discards: 0
     vf086.rx_unknown_protocol: 0
     vf086.tx_bytes: 0
     vf086.tx_unicast: 0
     vf086.tx_multicast: 0
     vf086.tx_broadcast: 0
     vf086.tx_discards: 0
     vf086.tx_errors: 0
     vf087.rx_bytes: 0
     vf087.rx_unicast: 0
     vf087.rx_multicast: 0
     vf087.rx_broadcast: 0
     vf087.rx_discards: 0
     vf087.rx_unknown_protocol: 0
     vf087.tx_bytes: 0
     vf087.tx_unicast: 0
     vf087.tx_multicast: 0
     vf087.tx_broadcast: 0
     vf087.tx_discards: 0
     vf087.tx_errors: 0
     vf088.rx_bytes: 0
     vf088.rx_unicast: 0
     vf088.rx_multicast: 0
     vf088.rx_broadcast: 0
     vf088.rx_discards: 0
     vf088.rx_unknown_protocol: 0
     vf088.tx_bytes: 0
     vf088.tx_unicast: 0
     vf088.tx_multicast: 0
     vf088.tx_broadcast: 0
     vf088.tx_discards: 0
     vf088.tx_errors: 0
     vf089.rx_bytes: 0
     vf089.rx_unicast: 0
     vf089.rx_multicast: 0
     vf089.rx_broadcast: 0
     vf089.rx_discards: 0
     vf089.rx_unknown_protocol: 0
     vf089.tx_bytes: 0
     vf089.tx_unicast: 0
     vf089.tx_multicast: 0
     vf089.tx_broadcast: 0
     vf089.tx_discards: 0
     vf089.tx_errors: 0
     vf090.rx_bytes: 0
     vf090.rx_unicast: 0
     vf090.rx_multicast: 0
     vf090.rx_broadcast: 0
     vf090.rx_discards: 0
     vf090.rx_unknown_protocol: 0
     vf090.tx_bytes: 0
     vf090.tx_unicast: 0
     vf090.tx_multicast: 0
     vf090.tx_broadcast: 0
     vf090.tx_discards: 0
     vf090.tx_errors: 0
     vf091.rx_bytes: 0
     vf091.rx_unicast: 0
     vf091.rx_multicast: 0
     vf091.rx_broadcast: 0
     vf091.rx_discards: 0
     vf091.rx_unknown_protocol: 0
     vf091.tx_bytes: 0
     vf091.tx_unicast: 0
     vf091.tx_multicast: 0
     vf091.tx_broadcast: 0
     vf091.tx_discards: 0
     vf091.tx_errors: 0
     vf092.rx_bytes: 0
     vf092.rx_unicast: 0
     vf092.rx_multicast: 0
     vf092.rx_broadcast: 0
     vf092.rx_discards: 0
     vf092.rx_unknown_protocol: 0
     vf092.tx_bytes: 0
     vf092.tx_unicast: 0
     vf092.tx_multicast: 0
     vf092.tx_broadcast: 0
     vf092.tx_discards: 0
     vf092.tx_errors: 0
     vf093.rx_bytes: 0
     vf093.rx_unicast: 0
     vf093.rx_multicast: 0
     vf093.rx_broadcast: 0
     vf093.rx_discards: 0
     vf093.rx_unknown_protocol: 0
     vf093.tx_bytes: 0
     vf093.tx_unicast: 0
     vf093.tx_multicast: 0
     vf093.tx_broadcast: 0
     vf093.tx_discards: 0
     vf093.tx_errors: 0
     vf094.rx_bytes: 0
     vf094.rx_unicast: 0
     vf094.rx_multicast: 0
     vf094.rx_broadcast: 0
     vf094.rx_discards: 0
     vf094.rx_unknown_protocol: 0
     vf094.tx_bytes: 0
     vf094.tx_unicast: 0
     vf094.tx_multicast: 0
     vf094.tx_broadcast: 0
     vf094.tx_discards: 0
     vf094.tx_errors: 0
     vf095.rx_bytes: 0
     vf095.rx_unicast: 0
     vf095.rx_multicast: 0
     vf095.rx_broadcast: 0
     vf095.rx_discards: 0
     vf095.rx_unknown_protocol: 0
     vf095.tx_bytes: 0
     vf095.tx_unicast: 0
     vf095.tx_multicast: 0
     vf095.tx_broadcast: 0
     vf095.tx_discards: 0
     vf095.tx_errors: 0
     vf096.rx_bytes: 0
     vf096.rx_unicast: 0
     vf096.rx_multicast: 0
     vf096.rx_broadcast: 0
     vf096.rx_discards: 0
     vf096.rx_unknown_protocol: 0
     vf096.tx_bytes: 0
     vf096.tx_unicast: 0
     vf096.tx_multicast: 0
     vf096.tx_broadcast: 0
     vf096.tx_discards: 0
     vf096.tx_errors: 0
     vf097.rx_bytes: 0
     vf097.rx_unicast: 0
     vf097.rx_multicast: 0
     vf097.rx_broadcast: 0
     vf097.rx_discards: 0
     vf097.rx_unknown_protocol: 0
     vf097.tx_bytes: 0
     vf097.tx_unicast: 0
     vf097.tx_multicast: 0
     vf097.tx_broadcast: 0
     vf097.tx_discards: 0
     vf097.tx_errors: 0
     vf098.rx_bytes: 0
     vf098.rx_unicast: 0
     vf098.rx_multicast: 0
     vf098.rx_broadcast: 0
     vf098.rx_discards: 0
     vf098.rx_unknown_protocol: 0
     vf098.tx_bytes: 0
     vf098.tx_unicast: 0
     vf098.tx_multicast: 0
     vf098.tx_broadcast: 0
     vf098.tx_discards: 0
     vf098.tx_errors: 0
     vf099.rx_bytes: 0
     vf099.rx_unicast: 0
     vf099.rx_multicast: 0
     vf099.rx_broadcast: 0
     vf099.rx_discards: 0
     vf099.rx_unknown_protocol: 0
     vf099.tx_bytes: 0
     vf099.tx_unicast: 0
     vf099.tx_multicast: 0
     vf099.tx_broadcast: 0
     vf099.tx_discards: 0
     vf099.tx_errors: 0
     vf100.rx_bytes: 0
     vf100.rx_unicast: 0
     vf100.rx_multicast: 0
     vf100.rx_broadcast: 0
     vf100.rx_discards: 0
     vf100.rx_unknown_protocol: 0
     vf100.tx_bytes: 0
     vf100.tx_unicast: 0
     vf100.tx_multicast: 0
     vf100.tx_broadcast: 0
     vf100.tx_discards: 0
     vf100.tx_errors: 0
     vf101.rx_bytes: 0
     vf101.rx_unicast: 0
     vf101.rx_multicast: 0
     vf101.rx_broadcast: 0
     vf101.rx_discards: 0
     vf101.rx_unknown_protocol: 0
     vf101.tx_bytes: 0
     vf101.tx_unicast: 0
     vf101.tx_multicast: 0
     vf101.tx_broadcast: 0
     vf101.tx_discards: 0
     vf101.tx_errors: 0
     vf102.rx_bytes: 0
     vf102.rx_unicast: 0
     vf102.rx_multicast: 0
     vf102.rx_broadcast: 0
     vf102.rx_discards: 0
     vf102.rx_unknown_protocol: 0
     vf102.tx_bytes: 0
     vf102.tx_unicast: 0
     vf102.tx_multicast: 0
     vf102.tx_broadcast: 0
     vf102.tx_discards: 0
     vf102.tx_errors: 0
     vf103.rx_bytes: 0
     vf103.rx_unicast: 0
     vf103.rx_multicast: 0
     vf103.rx_broadcast: 0
     vf103.rx_discards: 0
     vf103.rx_unknown_protocol: 0
     vf103.tx_bytes: 0
     vf103.tx_unicast: 0
     vf103.tx_multicast: 0
     vf103.tx_broadcast: 0
     vf103.tx_discards: 0
     vf103.tx_errors: 0
     vf104.rx_bytes: 0
     vf104.rx_unicast: 0
     vf104.rx_multicast: 0
     vf104.rx_broadcast: 0
     vf104.rx_discards: 0
     vf104.rx_unknown_protocol: 0
     vf104.tx_bytes: 0
     vf104.tx_unicast: 0
     vf104.tx_multicast: 0
     vf104.tx_broadcast: 0
     vf104.tx_discards: 0
     vf104.tx_errors: 0
     vf105.rx_bytes: 0
     vf105.rx_unicast: 0
     vf105.rx_multicast: 0
     vf105.rx_broadcast: 0
     vf105.rx_discards: 0
     vf105.rx_unknown_protocol: 0
     vf105.tx_bytes: 0
     vf105.tx_unicast: 0
     vf105.tx_multicast: 0
     vf105.tx_broadcast: 0
     vf105.tx_discards: 0
     vf105.tx_errors: 0
     vf106.rx_bytes: 0
     vf106.rx_unicast: 0
     vf106.rx_multicast: 0
     vf106.rx_broadcast: 0
     vf106.rx_discards: 0
     vf106.rx_unknown_protocol: 0
     vf106.tx_bytes: 0
     vf106.tx_unicast: 0
     vf106.tx_multicast: 0
     vf106.tx_broadcast: 0
     vf106.tx_discards: 0
     vf106.tx_errors: 0
     vf107.rx_bytes: 0
     vf107.rx_unicast: 0
     vf107.rx_multicast: 0
     vf107.rx_broadcast: 0
     vf107.rx_discards: 0
     vf107.rx_unknown_protocol: 0
     vf107.tx_bytes: 0
     vf107.tx_unicast: 0
     vf107.tx_multicast: 0
     vf107.tx_broadcast: 0
     vf107.tx_discards: 0
     vf107.tx_errors: 0
     vf108.rx_bytes: 0
     vf108.rx_unicast: 0
     vf108.rx_multicast: 0
     vf108.rx_broadcast: 0
     vf108.rx_discards: 0
     vf108.rx_unknown_protocol: 0
     vf108.tx_bytes: 0
     vf108.tx_unicast: 0
     vf108.tx_multicast: 0
     vf108.tx_broadcast: 0
     vf108.tx_discards: 0
     vf108.tx_errors: 0
     vf109.rx_bytes: 0
     vf109.rx_unicast: 0
     vf109.rx_multicast: 0
     vf109.rx_broadcast: 0
     vf109.rx_discards: 0
     vf109.rx_unknown_protocol: 0
     vf109.tx_bytes: 0
     vf109.tx_unicast: 0
     vf109.tx_multicast: 0
     vf109.tx_broadcast: 0
     vf109.tx_discards: 0
     vf109.tx_errors: 0
     vf110.rx_bytes: 0
     vf110.rx_unicast: 0
     vf110.rx_multicast: 0
     vf110.rx_broadcast: 0
     vf110.rx_discards: 0
     vf110.rx_unknown_protocol: 0
     vf110.tx_bytes: 0
     vf110.tx_unicast: 0
     vf110.tx_multicast: 0
     vf110.tx_broadcast: 0
     vf110.tx_discards: 0
     vf110.tx_errors: 0
     vf111.rx_bytes: 0
     vf111.rx_unicast: 0
     vf111.rx_multicast: 0
     vf111.rx_broadcast: 0
     vf111.rx_discards: 0
     vf111.rx_unknown_protocol: 0
     vf111.tx_bytes: 0
     vf111.tx_unicast: 0
     vf111.tx_multicast: 0
     vf111.tx_broadcast: 0
     vf111.tx_discards: 0
     vf111.tx_errors: 0
     vf112.rx_bytes: 0
     vf112.rx_unicast: 0
     vf112.rx_multicast: 0
     vf112.rx_broadcast: 0
     vf112.rx_discards: 0
     vf112.rx_unknown_protocol: 0
     vf112.tx_bytes: 0
     vf112.tx_unicast: 0
     vf112.tx_multicast: 0
     vf112.tx_broadcast: 0
     vf112.tx_discards: 0
     vf112.tx_errors: 0
     vf113.rx_bytes: 0
     vf113.rx_unicast: 0
     vf113.rx_multicast: 0
     vf113.rx_broadcast: 0
     vf113.rx_discards: 0
     vf113.rx_unknown_protocol: 0
     vf113.tx_bytes: 0
     vf113.tx_unicast: 0
     vf113.tx_multicast: 0
     vf113.tx_broadcast: 0
     vf113.tx_discards: 0
     vf113.tx_errors: 0
     vf114.rx_bytes: 0
     vf114.rx_unicast: 0
     vf114.rx_multicast: 0
     vf114.rx_broadcast: 0
     vf114.rx_discards: 0
     vf114.rx_unknown_protocol: 0
     vf114.tx_bytes: 0
     vf114.tx_unicast: 0
     vf114.tx_multicast: 0
     vf114.tx_broadcast: 0
     vf114.tx_discards: 0
     vf114.tx_errors: 0
     vf115.rx_bytes: 0
     vf115.rx_unicast: 0
     vf115.rx_multicast: 0
     vf115.rx_broadcast: 0
     vf115.rx_discards: 0
     vf115.rx_unknown_protocol: 0
     vf115.tx_bytes: 0
     vf115.tx_unicast: 0
     vf115.tx_multicast: 0
     vf115.tx_broadcast: 0
     vf115.tx_discards: 0
     vf115.tx_errors: 0
     vf116.rx_bytes: 0
     vf116.rx_unicast: 0
     vf116.rx_multicast: 0
     vf116.rx_broadcast: 0
     vf116.rx_discards: 0
     vf116.rx_unknown_protocol: 0
     vf116.tx_bytes: 0
     vf116.tx_unicast: 0
     vf116.tx_multicast: 0
     vf116.tx_broadcast: 0
     vf116.tx_discards: 0
     vf116.tx_errors: 0
     vf117.rx_bytes: 0
     vf117.rx_unicast: 0
     vf117.rx_multicast: 0
     vf117.rx_broadcast: 0
     vf117.rx_discards: 0
     vf117.rx_unknown_protocol: 0
     vf117.tx_bytes: 0
     vf117.tx_unicast: 0
     vf117.tx_multicast: 0
     vf117.tx_broadcast: 0
     vf117.tx_discards: 0
     vf117.tx_errors: 0
     vf118.rx_bytes: 0
     vf118.rx_unicast: 0
     vf118.rx_multicast: 0
     vf118.rx_broadcast: 0
     vf118.rx_discards: 0
     vf118.rx_unknown_protocol: 0
     vf118.tx_bytes: 0
     vf118.tx_unicast: 0
     vf118.tx_multicast: 0
     vf118.tx_broadcast: 0
     vf118.tx_discards: 0
     vf118.tx_errors: 0
     vf119.rx_bytes: 0
     vf119.rx_unicast: 0
     vf119.rx_multicast: 0
     vf119.rx_broadcast: 0
     vf119.rx_discards: 0
     vf119.rx_unknown_protocol: 0
     vf119.tx_bytes: 0
     vf119.tx_unicast: 0
     vf119.tx_multicast: 0
     vf119.tx_broadcast: 0
     vf119.tx_discards: 0
     vf119.tx_errors: 0
     vf120.rx_bytes: 0
     vf120.rx_unicast: 0
     vf120.rx_multicast: 0
     vf120.rx_broadcast: 0
     vf120.rx_discards: 0
     vf120.rx_unknown_protocol: 0
     vf120.tx_bytes: 0
     vf120.tx_unicast: 0
     vf120.tx_multicast: 0
     vf120.tx_broadcast: 0
     vf120.tx_discards: 0
     vf120.tx_errors: 0
     vf121.rx_bytes: 0
     vf121.rx_unicast: 0
     vf121.rx_multicast: 0
     vf121.rx_broadcast: 0
     vf121.rx_discards: 0
     vf121.rx_unknown_protocol: 0
     vf121.tx_bytes: 0
     vf121.tx_unicast: 0
     vf121.tx_multicast: 0
     vf121.tx_broadcast: 0
     vf121.tx_discards: 0
     vf121.tx_errors: 0
     vf122.rx_bytes: 0
     vf122.rx_unicast: 0
     vf122.rx_multicast: 0
     vf122.rx_broadcast: 0
     vf122.rx_discards: 0
     vf122.rx_unknown_protocol: 0
     vf122.tx_bytes: 0
     vf122.tx_unicast: 0
     vf122.tx_multicast: 0
     vf122.tx_broadcast: 0
     vf122.tx_discards: 0
     vf122.tx_errors: 0
     vf123.rx_bytes: 0
     vf123.rx_unicast: 0
     vf123.rx_multicast: 0
     vf123.rx_broadcast: 0
     vf123.rx_discards: 0
     vf123.rx_unknown_protocol: 0
     vf123.tx_bytes: 0
     vf123.tx_unicast: 0
     vf123.tx_multicast: 0
     vf123.tx_broadcast: 0
     vf123.tx_discards: 0
     vf123.tx_errors: 0
     vf124.rx_bytes: 0
     vf124.rx_unicast: 0
     vf124.rx_multicast: 0
     vf124.rx_broadcast: 0
     vf124.rx_discards: 0
     vf124.rx_unknown_protocol: 0
     vf124.tx_bytes: 0
     vf124.tx_unicast: 0
     vf124.tx_multicast: 0
     vf124.tx_broadcast: 0
     vf124.tx_discards: 0
     vf124.tx_errors: 0
     vf125.rx_bytes: 0
     vf125.rx_unicast: 0
     vf125.rx_multicast: 0
     vf125.rx_broadcast: 0
     vf125.rx_discards: 0
     vf125.rx_unknown_protocol: 0
     vf125.tx_bytes: 0
     vf125.tx_unicast: 0
     vf125.tx_multicast: 0
     vf125.tx_broadcast: 0
     vf125.tx_discards: 0
     vf125.tx_errors: 0
     vf126.rx_bytes: 0
     vf126.rx_unicast: 0
     vf126.rx_multicast: 0
     vf126.rx_broadcast: 0
     vf126.rx_discards: 0
     vf126.rx_unknown_protocol: 0
     vf126.tx_bytes: 0
     vf126.tx_unicast: 0
     vf126.tx_multicast: 0
     vf126.tx_broadcast: 0
     vf126.tx_discards: 0
     vf126.tx_errors: 0
     vf127.rx_bytes: 0
     vf127.rx_unicast: 0
     vf127.rx_multicast: 0
     vf127.rx_broadcast: 0
     vf127.rx_discards: 0
     vf127.rx_unknown_protocol: 0
     vf127.tx_bytes: 0
     vf127.tx_unicast: 0
     vf127.tx_multicast: 0
     vf127.tx_broadcast: 0
     vf127.tx_discards: 0
     vf127.tx_errors: 0

As you can see, there are packet losses

4. Yes. Below are the graphs of these two servers for the last 2 days. As you can see, on the second server with 4x10G cards the traffic in peaks was 23 Gbit/s, and on the first with the XL710 card it was only 21 Gbit/s. Although in theory it should be the same.

graphs_3.png

graphs_4.png

 

5. Linux 15224 4.19.0-26-amd64 #1 SMP Debian 4.19.304-1 (2024-01-09) x86_64 GNU/Linux

You can download it from https://packages.debian.org/buster/linux-image-4.19.0-26-amd64

# cat /etc/debian_version
10.13

You can download it from https://cdimage.debian.org/cdimage/archive/10.13.0/amd64/iso-cd/debian-10.13.0-amd64-netinst.iso

 

Thank you for your document. I'll try the settings from it. I’m really not sure that it will be better with 32 queues.

In response to Azeem_Intel
0 Kudos
Copy link
Dineshbabu
Employee
‎05-17-2024 07:14 AM
3,693 Views

Hi AndriiV,


Please let us know the outcome once you tried tunings recommended in the attached document.


Regards,

Dinesh


Copy link
AndriiV
Novice
‎05-20-2024 03:45 AM
3,559 Views
In response to Dineshbabu

Hi, Dinesh! Hi, Azeem!

Yesterday was the day with the highest traffic for the whole week.

I made changes 17.05.2024 at the evening

1. disable irqbalance

 

systemctl disable irqbalance
systemctl stop irqbalance

 

was allready stoped and disabled

2. Set the number of hardware receive (RX) and transmit (TX) queues

 

ethtool -L ens2 combined 40

 

was

 

ethtool -L ens2 combined 80

 

I made 40 queues becase CPU has 40 kernels

3. Set the interrupt moderation and packet coalescing for a network interface

 

ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 100 tx-usecs 100 rx-usecs-high 100

 

was

 

ethtool -C ens2 adaptive-rx off adaptive-tx off rx-usecs 2048 tx-usecs 2048

 

 4. Set the receive (RX) and transmit (TX) ring parameters for a network interface

 

ethtool -G ens2 rx 2048 tx 2048

 

was

 

ethtool -G ens2 tx 4096 rx 4096

 

5. Disable GRO

 

ethtool -K ens2 gro off

 

was

 

ethtool -K ens2 tx on sg on tso on rx on

 

6. Disable auto-negotiation for RX and TX features

 

ethtool -A ens2 rx off tx off

 

already was disabled

7. Set the IRQ Affnity (so IRQ affinity to be local to each CPU)

 

./ set_irq_affinity -x local ens2

 

was

 

./ set_irq_affinity -x all ens2

 

8. Set the network socket buffer sizes to max

 

sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.wmem_max=16777216

 

already was in sysctl.conf

9. Set the tuning profile

 

tuned-adm profile network-throughput

 

allready was the same. Also I tried (long time ago)

 

tuned-adm profile throughput-performance

 

10. Disable Firewall

 

service firewalld stop; systemctl mask firewalld

 

There are no firewalld in my system. Iptables has some fail2ban rulles for sshd and pure-ftpd

 

iptables -L -n -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
2787K 7817M f2b-pure-ftpd  tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 21,20,990,989
3299K  381M f2b-sshd   tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            multiport dports 22

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain f2b-sshd (1 references)
 pkts bytes target     prot opt in     out     source               destination
1835K  276M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain f2b-pure-ftpd (1 references)
 pkts bytes target     prot opt in     out     source               destination
2739K 7814M RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

 

 

Results

 

 

ethtool -S ens2 | grep dropped | column
     rx_dropped: 10215                       tx_dropped: 0                           port.rx_dropped: 174490                 port.tx_dropped_link_down: 0

 

 I made atop screenshots of both servers  yeasterday at 23:23 GMT+2 and 22:27 GMT+1 (Almost at the same time)

XL710:

atop_4.png

 

4х 82599ES:

atop_5.png

 

As you can see, first server has 21 Gbit/s, the second has 26 Gbit/s

IRQ loading with XL710 from 47% to 79% at the same time IRQ loading at 4x82599ES from 0% to 42%

You might think that the interrupts are due to the disks, but these servers have the same disks.

Graphs as a whole with yesterday's peak (GMT+2):

XL710:

graphs_5.png

4х 82599ES:

graphs_6.png

You may also be interested in Nginx statistics for this time:

XL710:

nginx_1.png

4х 82599ES:

nginx_2.png

 

As you can see, at approximately 21:30 with 15k connections on the xl710 problems begin.

A synthetic test in iperf for 10 connections is of course good, but it is far from real conditions.

In response to Dineshbabu
0 Kudos
Copy link
Simon-Intel
Employee
‎05-20-2024 06:15 AM
3,544 Views


Hi AndriiV,


Thank you for your prompt response.


We appreciate you sharing your concern with us. Please rest assured that we understand the importance of your situation and are fully committed to assisting you.


Please allow us some time to check this with the internal team, and we will get back to you as soon as we have an update.


If you have any additional information or questions, please feel free to share them with us. We're here to support you every step of the way.


Best regards,

Simon


Copy link
AndriiV
Novice
‎05-23-2024 01:49 AM
3,440 Views

Hello!

If it helps in any way. We are using optical twinax cable:

# ethtool -m ens2
        Identifier                                : 0x0d (QSFP+)
        Extended identifier                       : 0x10
        Extended identifier description           : 1.5W max. Power consumption
        Extended identifier description           : No CDR in TX, No CDR in RX
        Extended identifier description           : High Power Class (> 3.5 W) not enabled
        Connector                                 : 0x23 (No separable connector)
        Transceiver codes                         : 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
        Encoding                                  : 0x05 (64B/66B)
        BR, Nominal                               : 10300Mbps
        Rate identifier                           : 0x00
        Length (SMF,km)                           : 0km
        Length (OM3 50um)                         : 0m
        Length (OM2 50um)                         : 0m
        Length (OM1 62.5um)                       : 0m
        Length (Copper or Active cable)           : 3m
        Transmitter technology                    : 0x00 (850 nm VCSEL)
        Laser wavelength                          : 0.000nm
        Laser wavelength tolerance                : 0.000nm
        Vendor name                               : CISCO-FINISAR
        Vendor OUI                                : 00:90:65
        Vendor PN                                 : FCBN410QE2C03-C2
        Vendor rev                                : A
        Vendor SN                                 : FIW231301GE-B
        Date code                                 : 190326
        Revision Compliance                       : Revision not specified
        Module temperature                        : 0.00 degrees C / 32.00 degrees F
        Module voltage                            : 0.0000 V

 

On other side is Juniper QFX5200-32C 

0 Kudos
Copy link
Azeem_Intel
Employee
‎05-23-2024 04:44 PM
3,409 Views

Hi AndriiV,


Greetings for the day!

 

You observed the problem in the setup with the "Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz" but not in the setup with the "Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz".

 

Since we did not have both processors available, we tested with an Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz and were unable to reproduce the issue.

 

Here is the small comparison b/w your setups:

 

Intel(R) Xeon(R) Gold 6230 CPU @ 2.10GHz: It is a part of the Skylake microarchitecture. It has 20 cores and 40 threads, a clock speed of 2.10 GHz, a cache of 27.5 MB, and a TDP (termal design power) of 125 watts.

 

Intel(R) Xeon(R) CPU E5-2699 v4 @ 2.20GHz: It is based on Broadwell microarchitecture. It has 22 cores and 44 threads, a clock speed of 2.20 GHz, a cache of 55 MB, and a TDP of 145 watts.

 

Conclusion : Despite the Xeon E5-2699 v4 having slightly higher speed and cache, the Xeon Gold 6230 might still perform well or even better in certain tasks due to its architectural improvements.

 

Please clarify the below question:

 

Is this the same pattern used for network traffic, where one setup has four 10-gigabit ports and the other has a 40-gigabit NIC (XL710) Did you initiate traffic on every port simultaneously in the 10-gigabit setup

 

Some tunings: Please try using these

 

   # Disable IRQ Balance

       systemctl stop irqbalance

 

   # Set IRQ Affinity

   ${pathtoi40epackage}/scripts/set_irq_affinity -X local <interface>

   Note: “-x all” to use all the cores,”-x local” to use only the local NUMA socket

 

   # Increase the maximum number of packets allowed to queue when a particular interface receives packets faster than the kernel can process them

     sysctl -w net.core.netdev_max_backlog=8192

 

   # Enable interrupt coalescing to aggregate and process multiple interrupts as a single event, reducing CPU overhead

     ethtool -C $NIC_INTERFACE adaptive-rx off adaptive-tx off rx-usecs 50 tx-usecs 50 rx-usecs-high 50

 

   # Enable Multi-Queue support on NIC, to distribute network traffic processing to across multiple CPU cores.

     ethtool -L ethX combined Y   (ex: ethtool -L ethX combined 32  this is to set 32 Tx/Rx queues)

 

   # set max Tx/RX buffer size

   sysctl -w net.core.rmem_max=16777216

   sysctl -w net.core.wmem_max=16777216

 

   #Set the tuning profile

   tuned-adm profile network-throughput

 

   #Disable Firewall

   service firewalld stop; systemctl mask firewalld

 

# Set the receive (RX) and transmit (TX) ring parameters for a network interface (These parameters are to optimize the buffer space allocated for handling incoming and outgoing network packets)

     ethtool -G $NIC_INTERFACE rx 2048 tx 2048

 

 


Best Regards,

Azeem_Intel


0 Kudos
Copy link
AndriiV
Novice
‎05-29-2024 02:06 PM
3,106 Views
In response to Azeem_Intel

Hi Azeem, hi Simon!

Is this the same pattern used for network traffic, where one setup has four 10-gigabit ports and the other has a 40-gigabit NIC (XL710) Did you initiate traffic on every port simultaneously in the 10-gigabit setup

As you can see in my second message in this topic, there are Linux software bonding

cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 200
Down Delay (ms): 200

802.3ad info
LACP rate: fast
Min links: 0
Aggregator selection policy (ad_select): stable
System priority: 65535

The atop screenshot shows that the traffic is divided evenly between all 4 ports.

Actually, that’s why they installed the second server with 10G network cards, because there were problems with 40G.

 

   # Disable IRQ Balance

       systemctl stop irqbalance

Alredy Stopped

 

  # Set IRQ Affinity

   ${pathtoi40epackage}/scripts/set_irq_affinity -X local <interface>

   Note: “-x all” to use all the cores,”-x local” to use only the local NUMA socket

Made it. Was  set_irq_affinity -x all ens2

   # Increase the maximum number of packets allowed to queue when a particular interface receives packets faster than the kernel can process them

     sysctl -w net.core.netdev_max_backlog=8192

Made it. Was  net.core.netdev_max_backlog = 300000

   # Enable interrupt coalescing to aggregate and process multiple interrupts as a single event, reducing CPU overhead

     ethtool -C $NIC_INTERFACE adaptive-rx off adaptive-tx off rx-usecs 50 tx-usecs 50 rx-usecs-high 50

Made it

   # Enable Multi-Queue support on NIC, to distribute network traffic processing to across multiple CPU cores.

     ethtool -L ethX combined Y   (ex: ethtool -L ethX combined 32  this is to set 32 Tx/Rx queues)

Made it before set_irq_affinity -x loacal ens2 I think it's better this way.

   # set max Tx/RX buffer size

   sysctl -w net.core.rmem_max=16777216

   sysctl -w net.core.wmem_max=16777216

Alredy was the same. I did it again just in case

 

   #Set the tuning profile

   tuned-adm profile network-throughput

Alredy was the same. I did it again just in case

   #Disable Firewall

   service firewalld stop; systemctl mask firewalld

# service firewalld stop; systemctl mask firewalld
Failed to stop firewalld.service: Unit firewalld.service not loaded.
Unit firewalld.service does not exist, proceeding anyway.
Created symlink /etc/systemd/system/firewalld.service → /dev/null.

 

# Set the receive (RX) and transmit (TX) ring parameters for a network interface (These parameters are to optimize the buffer space allocated for handling incoming and outgoing network packets)

     ethtool -G $NIC_INTERFACE rx 2048 tx 2048

Made it.

Now the picture is like this

atop_6.png
 

There are still a lot of IRQ compared to 10G network cards.

Peak day is still Sunday. On weekdays, traffic rarely exceeds 22 Gbps

In response to Azeem_Intel
0 Kudos
Copy link
AndriiV
Novice
‎06-03-2024 12:44 AM
2,797 Views
In response to Azeem_Intel

Hi Azeem! Yesterday was highest traffic of the week. Graphs below (after tunnings):

graphs_8.png
atop_7.png

In response to Azeem_Intel
0 Kudos
Copy link
Simon-Intel
Employee
‎05-27-2024 08:52 AM
3,207 Views

Hello AndriiV,

 

Thank you for choosing Intel.

 

This is the first follow-up regarding the issue you reported to us.

 

We wanted to inquire whether you had the opportunity to review the plan of action (POA) we provided. 

 

Feel free to reply to this message, and we'll be more than happy to assist you further.

 

Regards,

Simon


0 Kudos
Copy link
Simon-Intel
Employee
‎05-29-2024 12:55 AM
3,124 Views

Hello AndriiV,


Thank you for contacting Intel.


This is a follow-up regarding the issue you reported. We are committed to ensuring a swift resolution and would appreciate any updates or additional information you can provide.


Please respond to this message at your earliest convenience. If we do not receive any correspondence from you by tomorrow, we will need to close this case.


We look forward to your reply.


Best regards,

Simon


0 Kudos
Copy link
Reply

Community support is provided Monday to Friday. Other contact methods are available here.

Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.

For more complete information about compiler optimizations, see our Optimization Notice.