Ethernet Products
Determine ramifications of Intel® Ethernet products and technologies
4811 Discussions

IGB/E1000 : Watchdog bite/ crash while network performance measurement test.

msing16
Beginner
3,864 Views

Hi,

I am using network performance measurement tool nuttcp with IGB and E1000 cards on one of my development boards.

Issue: Observed crash/watchdog bite using testing with both IGB and E1000 cards.

Kernel: kernel_msm-3.18

Steps to reproduce.

1.) On device: ./nuttcp-8.1.4.arm -S

2.) On PC side run below command xxx.xx.xxx.xxx -> IP address of device.

./nuttcp-8.1.4.x86 -w2m -u -R 160M -i 1 -T 1m xxx.xx.xxx.xxx

3) After a couple of iterations we see the crash reported in the log.

IGB Log

------------------

Parsing debug information for MSM_DUMP_DATA_CPU_CTX. Version: 20 Magic: 42445953 Source:

Parsing CPU1 context start 171c8a800 end 171c8b000

Core 1 PC: arch_counter_get_cntvct+1c

Core 1 LR: arch_counter_get_cntvct+1c

[] arch_counter_get_cntvct+0x1c

[] __delay+0x24

[] __const_udelay+0x24

[] msm_trigger_wdog_bite+0xd0

[] spin_bug+0x94

[] do_raw_spin_lock+0x104

[] _raw_spin_lock+0x28

[] igb_get_stats64+0x30

[] dev_get_stats+0x4c

[] iface_stat_fmt_proc_show+0x98

[] seq_read+0x18c

[] proc_reg_read+0x8c

[] vfs_read+0xa0

[] SyS_read+0x58

[] el0_svc_naked+0x24

From the code, it looks like it could be stuck in igb_update_stats and so the unlock might not be happening in time.

Code file:/kernel_msm-3.18/kernel/drivers/net/ethernet/intel/igb/igb_main.c

5160static struct rtnl_link_stats64 *igb_get_stats64(struct net_device *netdev,

5161 struct rtnl_link_stats64 *stats)

5162{

5163 struct igb_adapter *adapter = netdev_priv(netdev);

5164

5165 spin_lock(&adapter->stats64_lock);

5166 igb_update_stats(adapter, &adapter->stats64);

5167 memcpy(stats, &adapter->stats64, sizeof(*stats));

5168 spin_unlock(&adapter->stats64_lock);

5169

5170 return stats;

E1000 Log

-------------

[ 100.703446] init: Service 'atfwd' (pid 765) exited with status 255

[ 100.708673] init: Service 'atfwd' (pid 765) killing any children in process group

[ 104.343224] init: Untracked pid 2636 exited with status 0

[ 133.202677] BUG: spinlock lockup suspected on CPU# 0, kworker/0:3/986

[ 133.208030] lock: iface_stat_list_lock+0x0/0x18, .magic: dead4ead, .owner: NetworkStats/1294, .owner_cpu: 1

[ 133.217936] Causing a watchdog bite!

[ 133.345512] Backtrace for cpu 1 (current):

[ 133.348758] CPU: 1 PID: 1294 Comm: NetworkStats Tainted: G W 3.18.31-g12d3836-dirty # 2

[ 133.357697] Hardware name: Qualcomm Technologies, Inc. APQ8096v3 + PMI8994 DragonBoard (DT)

[ 133.366028] Call trace:

[ 133.368473] [] dump_backtrace+0x0/0x278

[ 133.373843] [] show_stack+0x20/0x28

[ 133.378884] [] dump_stack+0x9c/0xd4

[ 133.383914] [] arch_trigger_all_cpu_backtrace+0x6c/0xdc

[ 133.390687] [] do_raw_spin_lock+0x108/0x160

[ 133.396415] [] _raw_spin_lock+0x28/0x34

[ 133.401800] [] e1000e_get_stats64+0x44/0x118

[ 133.407614] [] dev_get_stats+0x4c/0xac

[ 133.412907] [] iface_stat_fmt_proc_show+0x98/0x198

[ 133.419243] [] seq_read+0x18c/0x3b4

[ 133.424277] [] proc_reg_read+0x8c/0xb4

[ 133.429571] [] vfs_read+0xa0/0x14c

[ 133.434520] [] SyS_read+0x58/0x94

cheers,

mohit

0 Kudos
19 Replies
idata
Employee
2,199 Views

Hi Mps,

 

 

Thank you for posting in Wired Communities. Can you help us to identify what is the exact iGB and e1000 network adapter of ethernet controller used on your development board so that we can better assist. Thank you.

 

 

Regards,

 

Sharon T
0 Kudos
msing16
Beginner
2,199 Views

Hi Sharon,

Please find below the details regarding IGB and E1000

IGB:

[ 8.201552] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.2.15-k

[ 8.208280] igb: Copyright (c) 2007-2014 Intel Corporation.

[ 8.215295] PPP generic driver version 2.4.2

10.885029] igb 0001:01:00.0: enabling device (0000 -> 0002)

[ 10.928050] igb 0001:01:00.0: added PHC on eth0

[ 10.931583] igb 0001:01:00.0: Intel(R) Gigabit Ethernet Network Connection

[ 10.938405] igb 0001:01:00.0: eth0: (PCIe:2.5Gb/s:Width x1) 00:06:cf:0a:cf:e7

[ 10.945691] igb 0001:01:00.0: eth0: PBA No: 000300-000

[ 10.950694] igb 0001:01:00.0: Using MSI interrupts. 1 rx queue(s), 1 tx queue(s)

E1000:

[ 7.209517] e1000: Intel(R) PRO/1000 Network Driver - version 7.3.21-k8-NAPI

[ 7.209519] e1000: Copyright (c) 1999-2006 Intel Corporation.

[ 7.209597] e1000e: Intel(R) PRO/1000 Network Driver - 2.3.2-k

[ 7.209600] e1000e: Copyright(c) 1999 - 2014 Intel Corporation.

[ 7.209752] e1000e 0002:01:00.0: Disabling ASPM L1

[ 7.209824] pci 0002:00:00.0: enabling device (0006 -> 0007)

[ 7.209860] e1000e 0002:01:00.0: enabling device (0000 -> 0002)

[ 7.210415] e1000e 0002:01:00.0: Interrupt Throttling Rate (ints/sec) set to dynamic conservative mode

[ 7.210679] e1000e 0002:01:00.0 0002:01:00.0 (uninitialized): Failed to initialize MSI-X interrupts. Falling back to MSI interrupts.

[ 7.339019] e1000e 0002:01:00.0 eth0: registered PHC clock

[ 7.343617] e1000e 0002:01:00.0 eth0: (PCI Express:2.5GT/s:Width x1) 68:05:ca:2d:a0:2f

[ 7.351502] e1000e 0002:01:00.0 eth0: Intel(R) PRO/1000 Network Connection

[ 7.358265] e1000e 0002:01:00.0 eth0: MAC: 3, PHY: 8, PBA No: E46981-008

[ 7.365867] PPP generic driver version 2.4.2

[ 7.369365] PPP BSD Compression module registered

cheers,

Mohit

0 Kudos
idata
Employee
2,199 Views

Hi Mps,

 

 

Thank you for the information. The PBA # E46981-008 is the part number of CT desktop adapter, are you using a standalone CT desktop on your board? You may refer to the link of the NIC

 

https://ark.intel.com/products/50395/Intel-Gigabit-CT-Desktop-Adapter.

 

 

I will further check on this.

 

 

Regards,

 

Sharon T
0 Kudos
msing16
Beginner
2,199 Views

For E1000 I am using Intel GIGABIT CT DESKTOP ADAPTER.

cheers,

mohit

0 Kudos
idata
Employee
2,199 Views

Hi Mps,

Thank you for the confirmation. Just to double check if the IGB NIC is an onboard NIC? I can't find the NIC based on the PBA# . Let me double check on this.

Regards,

 

Sharon T
0 Kudos
idata
Employee
2,199 Views

Hi Mps,

 

 

Please try the latest e1000e and igb driver in sourceforge site at - https://sourceforge.net/projects/e1000/files/?source=navbar https://sourceforge.net/projects/e1000/files/?source=navbar

 

 

Just to double check if this NIC is an ethernet controller integrated by you on the board?

 

 

Please feel free to update us.

 

 

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel

 

 

0 Kudos
msing16
Beginner
2,199 Views

Hi Sharon,

>>Please try the latest e1000e and igb driver in sourceforge site at - https://sourceforge.net/projects/e1000/files/%3Fsource%3Dnavbar https://sourceforge.net/projects/e1000/files/?source=navbar

Which version of driver should I try ? As you can see in the previous messages we are on kernel/msm-3.18.

I ran into issues when I tried to use igb folder from https://sourceforge.net/projects/e1000/files/igb%20stable/3.2.9/ Intel Ethernet Drivers and Utilities - Browse /igb stable/3.2.9 at SourceForge.net

UPD include/config/kernel.release

Using /home/intrinsyc/Work/Th_7_1_1/kernel/msm-3.18 as source for kernel

/bin/sh: 1: [: -ge: unexpected operator

/home/intrinsyc/Work/Th_7_1_1/kernel/msm-3.18/drivers/net/ethernet/intel/igb/Makefile:181: *** *** Aborting the build. *** This driver is not supported on kernel versions older than 2.4.21, because this driver requires NAPI support.. Stop.

make[5]: *** [drivers/net/ethernet/intel/igb] Error 2

make[4]: *** [drivers/net/ethernet/intel] Error 2

make[3]: *** [drivers/net/ethernet] Error 2

make[2]: *** [drivers/net] Error 2

make[2]: *** Waiting for unfinished jobs....

>> Just to double check if this NIC is an ethernet controller integrated by you on the board?

In case of igb yes.

cheers,

mohit

0 Kudos
idata
Employee
2,199 Views

Hi Mps,

 

 

Thank you for the information. Please try use the latest driver at Sourceforge:

 

 

igb https://sourceforge.net/projects/e1000/files/igb%20stable/

 

e1000e https://sourceforge.net/projects/e1000/files/e1000e%20stable/

 

 

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
idata
Employee
2,199 Views

Hi Mps,

 

 

Thank you for the confirmation. Let me further check.

 

 

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
idata
Employee
2,199 Views

Hi Mps,

 

 

Please install the latest driver version posted on Sourceforge site.

 

 

https://sourceforge.net/projects/e1000/files/igb%20stable/ (latest version is 5.3.5.18 dated 4/27/18)

 

 

https://sourceforge.net/projects/e1000/files/e1000e%20stable/ (version 3.4.1.1 dated 5/16/18)

 

 

You can also post the issue in the NUTTCP bug report site for NUTTCP development team to look into the issue. ( http://nuttcp.net/nuttcp/nuttcp.html# BUGS). And if the kernel is too old, you will have to update the kernel first.

 

 

Hope the above information help.

 

 

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel

 

 

 

0 Kudos
idata
Employee
2,199 Views

HI Mbps,

 

 

Please feel free to update me if you have updated the driver and kernel (if needed). Thanks.

 

 

 

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
msing16
Beginner
2,199 Views

Hi Sharon,

Updating to new kernel is a huge effort.

At best we can backport IGB driver provided Intel let us know which patch/kernel version has fixed this issue.

cheers,

mohit

0 Kudos
idata
Employee
2,199 Views

Hi Mps,

Thanks for the reply and let me further check and have you also contacted NUTTCP support?

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
TLiu35
Beginner
2,199 Views

Hi Sharon,

We commented out the spinlock/unlock calls during function to retrieve stats from the i210 and it worked fine without crashed. Will the change below have any negative affect?

./8096_Open-Q_820_Android_BSP-N_v3.5/Source_Package/APQ8096_LA.UM.5.5.r1-06700-8x96.0_N_v3.5/kernel/msm-3.18/drivers/net/ethernet/intel/igb/igb_main.c

@@ -5158,10 +5158,10 @@

{

struct igb_adapter *adapter = netdev_priv(netdev);

- spin_lock(&adapter->stats64_lock);

+ /* spin_lock(&adapter->stats64_lock); */

igb_update_stats(adapter, &adapter->stats64);

memcpy(stats, &adapter->stats64, sizeof(*stats));

- spin_unlock(&adapter->stats64_lock);

+ /* spin_unlock(&adapter->stats64_lock); */

return stats;

}

Thanks,

Tony

0 Kudos
idata
Employee
2,199 Views

Hi TL_TLS,

Please be informed this may affect performance depending on what software is running. We cannot support this configuration using an older driver. Thanks.

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
idata
Employee
2,199 Views

Hi Mps,

Please feel free to update us if you still have other inquiries. Thank you.

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
msing16
Beginner
2,199 Views

We have fixed Intel IGB driver.

Cheers,

Mohit

0 Kudos
idata
Employee
2,199 Views

Hi Mps,

Thank you for the update and glad to know the driver is fixed. In case you have other inquiries, please feel free to contact us again.

Regards,

 

Sharon T

 

Intel Customer Support

 

Agent under contract to Intel
0 Kudos
Reply