Not seeing any Rx packets while running Flexran xran sample-app on RU and DU

[Rajisha]

Hi Team,

We are trying to launch the xran sample-app on both DU and RU.

We could launch the application, but we don't see any Rx packets on both side.

We followed https://docs.o-ran-sc.org/projects/o-ran-sc-o-du-phy/en/latest/Setup-Configuration_fh.html to make the setup and did probing on VF and did dpdk binding as well.

xran_init: MTU 9600
total cores 144 c_mask 0x401 core 10 [id] system_core 0 [id] pkt_proc_core 0x0 [mask] pkt_aux_core 0 [id] timing_core 10 [id]
xran_ethdi_init_dpdk_io: Calling rte_eal_init:wls_1 -c 0x401 -n2 --iova-mode=pa --socket-mem=8192 --socket-limit=8192 --proc-type=auto --file-prefix wls_1 -w 0000:00:00.0
EAL: Detected 128 lcore(s)
EAL: Detected 4 NUMA nodes
EAL: Auto-detected process type: PRIMARY
EAL: Multi-process socket /var/run/dpdk/wls_1/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: ###TEST sample-app: rte_eth_dev_count: (0)

EAL: PCI device 0000:88:02.0 on NUMA socket 2
EAL: probe driver: 8086:154c net_i40e_vf
EAL: using IOMMU type 1 (Type 1)
EAL: PCI device 0000:88:02.0 on NUMA socket 2
EAL: Failed to attach device on primary process
EAL: ###TEST sample-app: rte_eth_dev_count: (1)
initializing port 0 for TX, drv=net_i40e_vf
Port 0 MAC: 00 11 22 33 44 55
Port 0: nb_rxd 4096 nb_txd 4096

EAL: ###TEST sample-app: rte_eth_dev_count: (1)
EAL: PCI device 0000:88:02.1 on NUMA socket 2
EAL: probe driver: 8086:154c net_i40e_vf
EAL: PCI device 0000:88:02.1 on NUMA socket 2
EAL: Failed to attach device on primary process
EAL: ###TEST sample-app: rte_eth_dev_count: (2)
initializing port 1 for TX, drv=net_i40e_vf
Port 1 MAC: 00 11 22 33 44 55
Port 1: nb_rxd 4096 nb_txd 4096

Checking link status portid [1] vf 0 local SRC MAC: 00 11 22 33 44 55
vf 0 remote DST MAC: 00 11 22 33 44 66
vf 1 local SRC MAC: 00 11 22 33 44 55
vf 1 remote DST MAC: 00 11 22 33 44 66
FFT Order 10
app_xran_cal_nrarfcn: nCenterFreq[2650140] nDeltaFglobal[5] nFoffs[0] nNoffs[0] nNRARFCN[530028]
DL center freq 2650140 DL NR-ARFCN 530028
app_xran_cal_nrarfcn: nCenterFreq[2530140] nDeltaFglobal[5] nFoffs[0] nNoffs[0] nNRARFCN[506028]
UL center freq 2530140 UL NR-ARFCN 506028
XRAN front haul xran_mm_init
xran_sector_get_instances [0]: CC 0 handle 0x1341da00
Handle: 0x118cc918 Instance: 0x1341da00
init_xran [0]: CC 0 handle 0x1341da00
Sucess xran_mm_init
nSectorNum 1
nSectorIndex[0] = 0
ru_0_cc_0_idx_0: [ handle 0x1341da00 0 0 ] [nPoolIndex 0] nNumberOfBuffers 2240 nBufferSize 15696
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 0] mb pool 0x2ea58de00
ru_0_cc_0_idx_1: [ handle 0x1341da00 0 0 ] [nPoolIndex 1] nNumberOfBuffers 35840 nBufferSize 24
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 1] mb pool 0x2e8168ac0
ru_0_cc_0_idx_2: [ handle 0x1341da00 0 0 ] [nPoolIndex 2] nNumberOfBuffers 2240 nBufferSize 48416
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 2] mb pool 0x2e6bb6480
ru_0_cc_0_idx_3: [ handle 0x1341da00 0 0 ] [nPoolIndex 3] nNumberOfBuffers 2240 nBufferSize 15696
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 3] mb pool 0x2e0189140
ru_0_cc_0_idx_4: [ handle 0x1341da00 0 0 ] [nPoolIndex 4] nNumberOfBuffers 35840 nBufferSize 24
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 4] mb pool 0x2ddd63e00
ru_0_cc_0_idx_5: [ handle 0x1341da00 0 0 ] [nPoolIndex 5] nNumberOfBuffers 2240 nBufferSize 48416
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 5] mb pool 0x2dc7b17c0
ru_0_cc_0_idx_6: [ handle 0x1341da00 0 0 ] [nPoolIndex 6] nNumberOfBuffers 2240 nBufferSize 8192
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 6] mb pool 0x2d5d84480
ru_0_cc_0_idx_7: [ handle 0x1341da00 0 0 ] [nPoolIndex 7] nNumberOfBuffers 2240 nBufferSize 15696
CC:[ handle 0x1341da00 ru 0 cc_idx 0 ] [nPoolIndex 7] mb pool 0x2d493ce40
init_xran_iq_content
xran_open: 5G NR Category A
xRAN open PRACH config: Numerology 0 ConfIdx 189, preambleFmrt 7 startsymb 0, numSymbol 2, occassionsInPrachSlot 7
PRACH: x 1 y[0] 0, y[1] 0 prach slot: 1 .. 6 ..
PRACH start symbol 0 lastsymbol 13
xran_open: interval_us=1000
XRAN_UP_VF: 0x0000
XRAN_CP_VF: 0x0001
----------------------------------------
MLog Info: virt=0x0000000000000000 size=0
----------------------------------------
Start XRAN traffic
Set debug stop 1, debug stop count 0
xran_timing_source_thread [CPU 10] [PID: 158188]
Start UL U-plane 160 us after OTA [offset in sym 3]
RU: thread_run start time: 12/08/21 12:24:31.000000020 UTC [1000]
interval_us 1000
+---------------------------------------+
| Press 1 to start 5G NR XRAN traffic |
| Press 2 reserved for future use |
| Press 3 to quit |
+---------------------------------------+
[o-ru][rx 0 pps 0 kbps 0][tx 262320 pps 262320 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.75 %
[o-ru][rx 0 pps 0 kbps 0][tx 329524 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 396728 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 463936 pps 67208 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 531140 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.75 %
[o-ru][rx 0 pps 0 kbps 0][tx 598344 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 665548 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 732752 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 799956 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 867160 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 934364 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1001568 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1068772 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1135976 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1203180 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1270384 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1337588 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1404796 pps 67208 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1472000 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1539204 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1606408 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1673612 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.77 %
[o-ru][rx 0 pps 0 kbps 0][tx 1740816 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1808020 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.75 %
[o-ru][rx 0 pps 0 kbps 0][tx 1875224 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
[o-ru][rx 0 pps 0 kbps 0][tx 1942428 pps 67204 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
STOP:[1638966300.000000083]
Closing timing source thread...
[o-ru][rx 0 pps 0 kbps 0][tx 1948796 pps 6368 kbps 1185945] [on_time 0 early 0 late 0 corrupt 0 pkt_dupl 0 Total 0] IO Util: 68.76 %
get_xran_iq_content
Closing l1 app... Ending all threads...
Failed at xran_mm_destroy, status -2
Dump IQs...
RX: Convert S16 I and S16 Q to cpu byte order from XRAN Ant: [0]
RX: Convert S16 I and S16 Q to cpu byte order from XRAN Ant: [1]
RX: Convert S16 I and S16 Q to cpu byte order from XRAN Ant: [2]
RX: Convert S16 I and S16 Q to cpu byte order from XRAN Ant: [3]
Storing UL FFT OUT IQ Samples in human readable format to file ./logs/o-ru-rx_log_ant0.txt: from addr (0x7fdc6cb7d010) size (1397760) IQ num (349440)
Storing UL FFT OUT IQ Samples in binary format to file ./logs/o-ru-rx_log_ant0.bin: from addr (0x7fdc6cb7d010) size (698880) bytes num (698880)
Storing UL FFT OUT IQ Samples in human readable format to file ./logs/o-ru-rx_log_ant1.txt: from addr (0x7fdc6ca27010) size (1397760) IQ num (349440)
Storing UL FFT OUT IQ Samples in binary format to file ./logs/o-ru-rx_log_ant1.bin: from addr (0x7fdc6ca27010) size (698880) bytes num (698880)
Storing UL FFT OUT IQ Samples in human readable format to file ./logs/o-ru-rx_log_ant2.txt: from addr (0x7fdc6c8d1010) size (1397760) IQ num (349440)
Storing UL FFT OUT IQ Samples in binary format to file ./logs/o-ru-rx_log_ant2.bin: from addr (0x7fdc6c8d1010) size (698880) bytes num (698880)
Storing UL FFT OUT IQ Samples in human readable format to file ./logs/o-ru-rx_log_ant3.txt: from addr (0x7fdc6c77b010) size (1397760) IQ num (349440)
Storing UL FFT OUT IQ Samples in binary format to file ./logs/o-ru-rx_log_ant3.bin: from addr (0x7fdc6c77b010) size (698880) bytes num (698880)

The received IQ samples shows zeros only.

Is there any way to check the received packets? or testing with traffic?

Are we missing any config changes?

Can anyone please help?

[root@localhost app]# uname -r
3.10.0-1160.42.2.rt56.1182.el7.x86_64

[root@localhost app]# cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

[root@localhost app]# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-1160.42.2.rt56.1182.el7.x86_64 root=/dev/mapper/centos-root ro crashkernel=auto spectre_v2=retpoline rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet processor.max_cstate=1 intel_idle.max_cstate=0 intel_pstate=disable idle=poll default_hugepagesz=1G hugepagesz=1G hugepages=32 intel_iommu=on iommu=pt selinux=0 enforcing=0 nmi_watchdog=0 audit=0 mce=off kthread_cpus=0 irqaffinity=0 skew_tick=1 isolcpus=1-17 intel_pstate=disable nosoftlockup nohz=on nohz_full=1-17 rcu_nocbs=1-17
[root@localhost app]#

./usertools/dpdk-devbind.py -s

Network devices using DPDK-compatible driver
============================================
0000:88:02.0 'Ethernet Virtual Function 700 Series 154c' drv=vfio-pci unused=iavf
0000:88:02.1 'Ethernet Virtual Function 700 Series 154c' drv=vfio-pci unused=iavf

Network devices using kernel driver
===================================
0000:01:00.0 'I350 Gigabit Network Connection 1521' if=em3 drv=igb unused=vfio-pci
0000:01:00.1 'I350 Gigabit Network Connection 1521' if=em4 drv=igb unused=vfio-pci *Active*
0000:17:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=em1 drv=i40e unused=vfio-pci *Active*
0000:17:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=em2 drv=i40e unused=vfio-pci
0000:88:00.0 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=p12p1 drv=i40e unused=vfio-pci *Active*
0000:88:00.1 'Ethernet Controller X710 for 10GbE SFP+ 1572' if=p12p2 drv=i40e unused=vfio-pci
0000:9b:00.0 'I350 Gigabit Network Connection 1521' if=p9p1 drv=igb unused=vfio-pci
0000:9b:00.1 'I350 Gigabit Network Connection 1521' if=p9p2 drv=igb unused=vfio-pci

No 'Baseband' devices detected
==============================

No 'Crypto' devices detected
============================

No 'Eventdev' devices detected
==============================

No 'Mempool' devices detected
=============================

No 'Compress' devices detected
==============================

No 'Misc (rawdev)' devices detected
===================================
[root@localhost dpdk-19.11]#

[AndrewG_Intel]

Hello @Rajisha

Thank you for posting on the Intel® communities.

We understand that you have some inquiries regarding Intel® FlexRAN™. Please allow us to review this request so we can route you to the proper channel of support.

Best regards,

Andrew G.

Intel Customer Support Technician

[Rajisha]

If we run the sample-app simultaneously on both servers (Du and RU), we are able to see the Rx counter also updating and Rx IQ sample is also matching.

Thank you. 

[AndrewG_Intel]

Hello Rajisha

Thank you for your patience in this matter.

For Intel® FlexRAN related requests, please kindly refer to our Resource & Documentation Center for more information and support options. Some of the information there it's public and some other information might need special rights, for instance, a Corporate Non-Disclosure Agreement (CNDA) with Intel.

It is required to register first and create an account in order to access the public resources available. Refer to the Sign-in link.

To contact the Resource & Documentation Center support team, please refer to the following link Get Help under "Contact Us", for instance: Priority Support or Intel® Premier Support.

We kindly suggest contacting your Intel account manager in case you don't have access yet.

Having said that, we will proceed to close this thread now. Thank you for your understanding.

Best regards,

Andrew G.

Intel Customer Support Technician