FlexRAN crash issue observed in 23.07 in a long run followed by ACC100 PCie issue

[jeffinjoy]

Hello ,

We have been running Intel FlexRAN 23.07 in our setup for a long and we are facing one crash issue on the L1 side. While checking we found some PCIe errors from ACC100 following that L1 got crashed. 

kernel: [1402131.454101] {4}[Hardware Error]: Hardware error from APEI Generic Hardware Error Source: 0
kernel: [1402131.454104] {4}[Hardware Error]: It has been corrected by h/w and requires no further action
kernel: [1402131.454105] {4}[Hardware Error]: event severity: corrected
kernel: [1402131.454106] {4}[Hardware Error]: Error 0, type: corrected
l1.sh[299592]: #033[1;32mPDSCH / PUSCH Stats
kernel: [1402131.454107] {4}[Hardware Error]: section_type: PCIe error
kernel: [1402131.454107] {4}[Hardware Error]: port_type: 0, PCIe end point
kernel: [1402131.454109] {4}[Hardware Error]: version: 3.0
kernel: [1402131.454109] {4}[Hardware Error]: command: 0x0546, status: 0x0018
kernel: [1402131.454110] {4}[Hardware Error]: device_id: 0000:c3:00.0
kernel: [1402131.454111] {4}[Hardware Error]: slot: 0
kernel: [1402131.454112] {4}[Hardware Error]: secondary_bus: 0x00
kernel: [1402131.454112] {4}[Hardware Error]: vendor_id: 0x8086, device_id: 0x0d5c
kernel: [1402131.454113] {4}[Hardware Error]: class_code: 120001
kernel: [1402131.454145] igb_uio 0000:c3:00.0: AER: aer_status: 0x00000001, aer_mask: 0x00000000
kernel: [1402131.454147] igb_uio 0000:c3:00.0: [ 0] RxErr (First)
kernel: [1402131.454149] igb_uio 0000:c3:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
l1.sh[299592]: #033[0m----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
l1.sh[299592]: | MAC | RU | MAC-to-PHY Tput | PHY-to-MAC Tput | UL FEC CB Iterations | UL Packet Errors | |
l1.sh[299592]: Cell (MU / DL,UL MHz, CId) | Inst | Port | kbps Num CB | kbps UL BLER Num CB | Min Avg Max | PUSCH PUCCH PRACH SRS | SRS SNR |
l1.sh[299592]: -----------------------------|------|-------|------------------------|------------------------------------------------|-----------------------|-------------------------|------------|
l1.sh[299592]: 0 (MU 1 / 100,100, 21) | 0 | 0, 0 | 202,327 122,553 | 124,056 / 140,877 11.94% 85,102 | 1 2.49 12 | 40 39 0 0 | 0 Db |
l1.sh[299592]: 1 (MU 1 / 100,100, 31) | 0 | 1, 0 | 61 411 | 0 / 0 0.00% 0 | 0 0.00 0 | 0 0 0 0 | 0 Db |
l1.sh[299592]: 2 (MU 1 / 100,100, 41) | 0 | 2, 0 | 61 411 | 0 / 0 0.00% 0 | 0 0.00 0 | 0 0 0 0 | 0 Db |
l1.sh[299592]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
l1.sh[299592]: #033[1;32mProcessing Latency Stats
l1.sh[299592]: #033[0m -----------------|--------------------------------|------------------------------
l1.sh[299592]: | usecs | % of TTI
l1.sh[299592]: Latency (-OTA) | Min Avg Max | Min Avg Max
l1.sh[299592]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
l1.sh[299592]: #033[1;32mCore Utilization Stats [5 BBU core(s)]:
l1.sh[299592]: #033[0m Core Id : 16 17 18 19 20 Avg
l1.sh[299592]: Numa Node : 0 0 0 0 0
l1.sh[299592]: Core Type : ALL ALL ALL ALL ALL
l1.sh[299592]: Util % : 21.68 21.74 21.82 21.80 21.69 21.75
l1.sh[299592]: Intr % : 1.65 1.64 1.63 1.66 1.67 1.65
l1.sh[299592]: Spare % : 0.36 0.36 0.36 0.36 0.36 0.36
l1.sh[299592]: Sleep % : 76.29 76.24 76.17 76.16 76.27 76.23
l1.sh[299592]: Numa % : 0.00 0.00 0.00 0.00 0.00 0.00
l1.sh[299592]: TTI Cnt : 10001 10001 10001 10001 10001
l1.sh[299592]: TTI Min : 2 2 2 2 2
l1.sh[299592]: TTI Avg : 21 21 21 21 21
l1.sh[299592]: TTI Max : 91 99 99 98 99
l1.sh[299592]: Xran Cores: 21 22 Master Core Util: 65 %
l1.sh[299592]: ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Sep 29 17:47:39 ptp_all.sh[299519]: phc2sys[1402132.400]: CLOCK_REALTIME phc offset -8 s2 freq -278 delay 515
Sep 29 17:47:39 phc2sys: [1402132.400] CLOCK_REALTIME phc offset -8 s2 freq -278 delay 515

Sep 29 17:47:40 ptp_all.sh[299519]: phc2sys[1402133.400]: CLOCK_REALTIME phc offset -14 s2 freq -286 delay 513
Sep 29 17:47:40 phc2sys: [1402133.400] CLOCK_REALTIME phc offset -14 s2 freq -286 delay 513
Sep 29 17:47:40 kernel: [1402133.535477] fh_main_poll-21[299600]: segfault at 13540 ip 00000000014ed060 sp 00007fa1e3c70888 error 6 in l1app[645000+416d000]
Sep 29 17:47:40 kernel: [1402133.535488] Code: 4a c7 84 33 e0 00 00 00 00 00 00 00 ff 4b 14 44 89 f8 48 83 c4 38 5b 41 5c 41 5d 41 5e 41 5f 5d c3 66 0f 1f 84 00 00 00 00 00 <89> b7 40 35 01 00 c3 66 0f 1f 84 00 00 00 00 00 53 41 89 f1 48 8d

[Sazirah]

Hi jeffinjoy,

Thank you for posting in Intel Ethernet Communities.

We understand that you are having concerns regarding the FPGA product. Therefore we will move this forum thread to the respective FPGAs and Programmable Solutions to ensure you receive proper solution for this issue.

Please give them some time and the respective team will attend to your enquiry shortly.

Regards,

Sazzy_Intel