Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4808 Discussions

FreeBSD and Intel XL710 10G

EKhor1
Beginner
3,213 Views

Hi all,

I've installed FreeBSD 10.2-STABLE on a server with 2*CPU E5-2643v3 (with HyperThreading on) and Intel XL710 4*10G SFP+ card.

At first I updated FreeBSD drivers to 1.4.0 (from download.intel.com).

I see next strange thing - Every acive ixl interface creates 24 queues (6 core *2 (HT) *2 CPU) but uses only 16-17 of them:

irq284: ixl0:q0 164383663 1941

irq285: ixl0:q1 371238730 4384

irq286: ixl0:q2 378286557 4468

irq287: ixl0:q3 365073427 4312

irq288: ixl0:q4 371116376 4383

irq289: ixl0:q5 372589584 4400

irq290: ixl0:q6 361879025 4274

irq291: ixl0:q7 354607200 4188

irq292: ixl0:q8 223602267 2641

irq293: ixl0:q9 199067474 2351

irq294: ixl0:q10 212598000 2511

irq295: ixl0:q11 202534854 2392

irq296: ixl0:q12 212050675 2504

irq297: ixl0:q13 209106917 2469

irq298: ixl0:q14 201452403 2379

irq299: ixl0:q15 203896634 2408

irq300: ixl0:q16 76328643 901

irq301: ixl0:q17 6030 0

irq302: ixl0:q18 5433 0

irq303: ixl0:q19 6804 0

irq304: ixl0:q20 6098 0

irq305: ixl0:q21 6603 0

irq306: ixl0:q22 6476 0

irq307: ixl0:q23 7141 0

irq309: ixl1:q0 161169757 1903

irq310: ixl1:q1 402042077 4748

irq311: ixl1:q2 399166615 4714

irq312: ixl1:q3 389702886 4602

irq313: ixl1:q4 383371508 4528

irq314: ixl1:q5 388621686 4590

irq315: ixl1:q6 385533771 4553

irq316: ixl1:q7 390478220 4612

irq317: ixl1:q8 232313544 2743

irq318: ixl1:q9 248387076 2933

irq319: ixl1:q10 233942388 2763

irq320: ixl1:q11 237794942 2808

irq321: ixl1:q12 227292626 2684

irq322: ixl1:q13 222151566 2623

irq323: ixl1:q14 234209020 2766

irq324: ixl1:q15 217878026 2573

irq325: ixl1:q16 80177041 947

irq326: ixl1:q17 83 0

irq327: ixl1:q18 74 0

irq328: ixl1:q19 201 0

irq329: ixl1:q20 98 0

irq330: ixl1:q21 95 0

irq331: ixl1:q22 91 0

irq332: ixl1:q23 87 0

# top -aSCHP

last pid: 28661; load averages: 7.06, 6.35, 6.23 up 0+23:35:07 17:15:12

391 processes: 31 running, 215 sleeping, 145 waiting

CPU 0: 0.0% user, 0.0% nice, 0.0% system, 39.4% interrupt, 60.6% idle

CPU 1: 0.0% user, 0.0% nice, 0.0% system, 48.8% interrupt, 51.2% idle

CPU 2: 0.0% user, 0.0% nice, 0.0% system, 42.1% interrupt, 57.9% idle

CPU 3: 0.0% user, 0.0% nice, 0.0% system, 40.2% interrupt, 59.8% idle

CPU 4: 0.0% user, 0.0% nice, 0.4% system, 41.3% interrupt, 58.3% idle

CPU 5: 0.0% user, 0.0% nice, 0.0% system, 37.0% interrupt, 63.0% idle

CPU 6: 0.0% user, 0.0% nice, 0.0% system, 35.8% interrupt, 64.2% idle

CPU 7: 0.0% user, 0.0% nice, 0.0% system, 39.0% interrupt, 61.0% idle

CPU 8: 0.0% user, 0.0% nice, 0.0% system, 22.0% interrupt, 78.0% idle

CPU 9: 0.0% user, 0.0% nice, 0.0% system, 26.0% interrupt, 74.0% idle

CPU 10: 0.0% user, 0.0% nice, 0.0% system, 17.7% interrupt, 82.3% idle

CPU 11: 0.0% user, 0.0% nice, 0.0% system, 19.3% interrupt, 80.7% idle

CPU 12: 0.0% user, 0.0% nice, 0.4% system, 25.2% interrupt, 74.4% idle

CPU 13: 0.0% user, 0.0% nice, 0.0% system, 23.6% interrupt, 76.4% idle

CPU 14: 0.0% user, 0.0% nice, 0.0% system, 22.4% interrupt, 77.6% idle

CPU 15: 0.0% user, 0.0% nice, 0.0% system, 26.8% interrupt, 73.2% idle

CPU 16: 0.0% user, 0.0% nice, 1.2% system, 1.6% interrupt, 97.2% idle

CPU 17: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle

CPU 18: 0.0% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.8% idle

CPU 19: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle

CPU 20: 0.0% user, 0.0% nice, 3.5% system, 0.0% interrupt, 96.5% idle

CPU 21: 0.0% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.4% idle

CPU 22: 0.0% user, 0.0% nice, 0.8% system, 0.0% interrupt, 99.2% idle

CPU 23: 0.0% user, 0.0% nice, 2.0% system, 0.0% interrupt, 98.0% idle

# netstat -I ixl0 -w1 -h

input ixl0 output

packets errs idrops bytes packets errs bytes colls

235K 0 0 126M 300K 0 321M 0

233K 0 0 114M 297K 0 312M 0

232K 0 0 116M 300K 0 315M 0

227K 0 0 108M 297K 0 316M 0

Where is the source of problem that network adapter doesn't utilize all queues ?

0 Kudos
13 Replies
st4
New Contributor III
1,038 Views

Hi John_obn,

Thank you for the post. We will check on this.

rgds,

wb

 

0 Kudos
EKhor1
Beginner
1,038 Views

Hi wb,

Did you have luck to check this issue?

Regards,

Evgeny

0 Kudos
st4
New Contributor III
1,038 Views

Hi John_obn,

Thank you for the ffup. I am still checking this, will update you once there is any finding.

rgds,

wb

 

0 Kudos
st4
New Contributor III
1,038 Views

Hi Evgeny,

Good day. Please help provide below information for further clarification:

Did you manually assigning the IRQs or was it automatically assigned?

 

How is the test being setup and how the traffic is being sent?

Thanks,

wb

0 Kudos
EKhor1
Beginner
1,038 Views

Hi wb,

1. I've tried manually IRQ assigning as well as automatically.

2. I've installed FreeBSD 10.2-STABLE. Installed the drivers from download.intel.com. Setup a routing, ipfw or pf rules for nat.

The traffic from 172.16.0.0/16 is received in ixl0 interface and forwarded (and NATed) to ixl1 interface. Returning traffic is forwarded vice versa. What details would you like to know?

Did you tried this schema on your machine with FreeBSD with the latest drivers?

Thanks,

Evgeny

0 Kudos
st4
New Contributor III
1,038 Views

Hi Evgeny,

Thank you for the information. We will further check based on your setup, I will update you when there is any finding.

thanks,

wb

 

0 Kudos
st4
New Contributor III
1,038 Views

Hi Evgeny,

Good day. Please try to send packets from 24 different IPs instead.

rgds,

wb

 

0 Kudos
EKhor1
Beginner
1,038 Views

Hello wb,

Are you serious? This is a router PC. It forwards traffic from/to at least 5000 IPs.

Regards,

Evgeny

0 Kudos
st4
New Contributor III
1,038 Views

Hi Evgeny,

Thank you for the clarification. Let me check on this.

rgds,

wb

 

0 Kudos
st4
New Contributor III
1,038 Views

Hi Evgeny,

Good day. Further checking we have already confirmed this issue and will be submitting a fix to FreeBSD.org. The fix will be available on the next

software release.

rgds,

wb

 

0 Kudos
EKhor1
Beginner
1,038 Views

Hi wb,

Thank you for good news. Which release did you mean? FreeBSD or Intel official driver for FreeBSD?

Is there any fix in 1.4.8 official driver on download.intel.com?

0 Kudos
st4
New Contributor III
1,038 Views

HI John_obn,

Let me check for you.

rgds,

wb

 

0 Kudos
st4
New Contributor III
1,038 Views

Hi John_obn,

We will submit the driver to Freebsd and they will post on the Freebsd.org website. With regards to the fix in 1.4.8 official driver, you may refer to the readme at

 

https://downloadmirror.intel.com/25160/eng/readmefirst.txt https://downloadmirror.intel.com/25160/eng/readmefirst.txt

 

This release includes the 40 gigabit FreeBSD Base Driver for Intel® Network Connection.

- ixl driver supports all XL710 and XL710-based 40 gigabit network connections.

 

- ixlv driver supports X710 and XL710-based virtual function devices that

 

can only be activated on kernels that support SR-IOV.

Hope the above information help.

rgds,

wb

 

0 Kudos
Reply