- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi all,
I've installed FreeBSD 10.2-STABLE on a server with 2*CPU E5-2643v3 (with HyperThreading on) and Intel XL710 4*10G SFP+ card.
At first I updated FreeBSD drivers to 1.4.0 (from download.intel.com).
I see next strange thing - Every acive ixl interface creates 24 queues (6 core *2 (HT) *2 CPU) but uses only 16-17 of them:
irq284: ixl0:q0 164383663 1941
irq285: ixl0:q1 371238730 4384
irq286: ixl0:q2 378286557 4468
irq287: ixl0:q3 365073427 4312
irq288: ixl0:q4 371116376 4383
irq289: ixl0:q5 372589584 4400
irq290: ixl0:q6 361879025 4274
irq291: ixl0:q7 354607200 4188
irq292: ixl0:q8 223602267 2641
irq293: ixl0:q9 199067474 2351
irq294: ixl0:q10 212598000 2511
irq295: ixl0:q11 202534854 2392
irq296: ixl0:q12 212050675 2504
irq297: ixl0:q13 209106917 2469
irq298: ixl0:q14 201452403 2379
irq299: ixl0:q15 203896634 2408
irq300: ixl0:q16 76328643 901
irq301: ixl0:q17 6030 0
irq302: ixl0:q18 5433 0
irq303: ixl0:q19 6804 0
irq304: ixl0:q20 6098 0
irq305: ixl0:q21 6603 0
irq306: ixl0:q22 6476 0
irq307: ixl0:q23 7141 0
irq309: ixl1:q0 161169757 1903
irq310: ixl1:q1 402042077 4748
irq311: ixl1:q2 399166615 4714
irq312: ixl1:q3 389702886 4602
irq313: ixl1:q4 383371508 4528
irq314: ixl1:q5 388621686 4590
irq315: ixl1:q6 385533771 4553
irq316: ixl1:q7 390478220 4612
irq317: ixl1:q8 232313544 2743
irq318: ixl1:q9 248387076 2933
irq319: ixl1:q10 233942388 2763
irq320: ixl1:q11 237794942 2808
irq321: ixl1:q12 227292626 2684
irq322: ixl1:q13 222151566 2623
irq323: ixl1:q14 234209020 2766
irq324: ixl1:q15 217878026 2573
irq325: ixl1:q16 80177041 947
irq326: ixl1:q17 83 0
irq327: ixl1:q18 74 0
irq328: ixl1:q19 201 0
irq329: ixl1:q20 98 0
irq330: ixl1:q21 95 0
irq331: ixl1:q22 91 0
irq332: ixl1:q23 87 0
# top -aSCHP
last pid: 28661; load averages: 7.06, 6.35, 6.23 up 0+23:35:07 17:15:12
391 processes: 31 running, 215 sleeping, 145 waiting
CPU 0: 0.0% user, 0.0% nice, 0.0% system, 39.4% interrupt, 60.6% idle
CPU 1: 0.0% user, 0.0% nice, 0.0% system, 48.8% interrupt, 51.2% idle
CPU 2: 0.0% user, 0.0% nice, 0.0% system, 42.1% interrupt, 57.9% idle
CPU 3: 0.0% user, 0.0% nice, 0.0% system, 40.2% interrupt, 59.8% idle
CPU 4: 0.0% user, 0.0% nice, 0.4% system, 41.3% interrupt, 58.3% idle
CPU 5: 0.0% user, 0.0% nice, 0.0% system, 37.0% interrupt, 63.0% idle
CPU 6: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% idle
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 39.0% interrupt, 61.0% idle
CPU 8: 0.0% user, 0.0% nice, 0.0% system, 22.0% interrupt, 78.0% idle
CPU 9: 0.0% user, 0.0% nice, 0.0% system, 26.0% interrupt, 74.0% idle
CPU 10: 0.0% user, 0.0% nice, 0.0% system, 17.7% interrupt, 82.3% idle
CPU 11: 0.0% user, 0.0% nice, 0.0% system, 19.3% interrupt, 80.7% idle
CPU 12: 0.0% user, 0.0% nice, 0.4% system, 25.2% interrupt, 74.4% idle
CPU 13: 0.0% user, 0.0% nice, 0.0% system, 23.6% interrupt, 76.4% idle
CPU 14: 0.0% user, 0.0% nice, 0.0% system, 22.4% interrupt, 77.6% idle
CPU 15: 0.0% user, 0.0% nice, 0.0% system, 26.8% interrupt, 73.2% idle
CPU 16: 0.0% user, 0.0% nice, 1.2% system, 1.6% interrupt, 97.2% idle
CPU 17: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle
CPU 18: 0.0% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.8% idle
CPU 19: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle
CPU 20: 0.0% user, 0.0% nice, 3.5% system, 0.0% interrupt, 96.5% idle
CPU 21: 0.0% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.4% idle
CPU 22: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle
CPU 23: 0.0% user, 0.0% nice, 2.0% system, 0.0% interrupt, 98.0% idle
# netstat -I ixl0 -w1 -h
input ixl0 output
packets errs idrops bytes packets errs bytes colls
235K 0 0 126M 300K 0 321M 0
233K 0 0 114M 297K 0 312M 0
232K 0 0 116M 300K 0 315M 0
227K 0 0 108M 297K 0 316M 0
Where is the source of problem that network adapter doesn't utilize all queues ?
Link Copied
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi John_obn,
Thank you for the post. We will check on this.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi John_obn,
Thank you for the ffup. I am still checking this, will update you once there is any finding.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evgeny,
Good day. Please help provide below information for further clarification:
Did you manually assigning the IRQs or was it automatically assigned?
How is the test being setup and how the traffic is being sent?
Thanks,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi wb,
1. I've tried manually IRQ assigning as well as automatically.
2. I've installed FreeBSD 10.2-STABLE. Installed the drivers from download.intel.com. Setup a routing, ipfw or pf rules for nat.
The traffic from 172.16.0.0/16 is received in ixl0 interface and forwarded (and NATed) to ixl1 interface. Returning traffic is forwarded vice versa. What details would you like to know?
Did you tried this schema on your machine with FreeBSD with the latest drivers?
Thanks,
Evgeny
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evgeny,
Thank you for the information. We will further check based on your setup, I will update you when there is any finding.
thanks,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evgeny,
Good day. Please try to send packets from 24 different IPs instead.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello wb,
Are you serious? This is a router PC. It forwards traffic from/to at least 5000 IPs.
Regards,
Evgeny
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evgeny,
Thank you for the clarification. Let me check on this.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi Evgeny,
Good day. Further checking we have already confirmed this issue and will be submitting a fix to FreeBSD.org. The fix will be available on the next
software release.
rgds,
wb
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi wb,
Thank you for good news. Which release did you mean? FreeBSD or Intel official driver for FreeBSD?
Is there any fix in 1.4.8 official driver on download.intel.com?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi John_obn,
We will submit the driver to Freebsd and they will post on the Freebsd.org website. With regards to the fix in 1.4.8 official driver, you may refer to the readme at
https://downloadmirror.intel.com/25160/eng/readmefirst.txt https://downloadmirror.intel.com/25160/eng/readmefirst.txt
This release includes the 40 gigabit FreeBSD Base Driver for Intel® Network Connection.
- ixl driver supports all XL710 and XL710-based 40 gigabit network connections.
- ixlv driver supports X710 and XL710-based virtual function devices that
can only be activated on kernels that support SR-IOV.
Hope the above information help.
rgds,
wb

- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page