bf_shell inconsistencies when setting single/multi-program (SDE9.3.0)

Hi,

Based on bri_set_forwarding_pipeline example, we set 2 distinct programs on each pipe.

Everything is working well, both programs are running succesfully, but when I try to access the P4 program table via bf_shell we see the errors below and impossible to move forward: (i.e cannot inspect table object etc.)

bfshell> bfrt_python 
cwd : /root

Devices found :  [0]
We've found 2 p4 programs for device 0:
bf_router_a
bf_router_b
Creating tree for dev 0 and program bf_router_a

> /opt/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py(1022)_set_dynamic_method()
-> setattr(self, method_name, types.MethodType(d[method_name], self))
(Pdb)

When we start 2 programs and load them onto 2 pipes with run_switchd we can access the P4 program via bf_shell. Maybe there is another step to trigger via BRI. I read AP-UG33-001_Bf-Runtime-gRPC-Guide.pdf, in order to see if another message had to be sent but I could not find any additional step defined within BfRuntime protobuf message definition besides those that were included into bri_set_forwarding_pipeline example.

Thanks in advance for your insight,

Frederic

Hi !

Thanks for your reply. Please find the code of switchdctl.py which basically loads one program onto a device at address using grpc port, with device-id/client-id and p4-profile-name.

You can grab it in GÉANT RARE bitbucket.

This is very useful in our case as we have a P4 lab shared by researcher. The lab is time shared and instead of logging into the p4 switch and run_switchd -p <p4-program> , they can do all the compilation on a specific build machine and load their program on all p4 switches.

This is working very well except that console via bf_shell is not yet possible. But this has great potential.

It also help us to overcome p4 compiled code distribution problem.

Last but not least we can run different p4 programs on different pipes so we can split a P4 switch on a per pipe basic. Even if this cannot be hitless and requires synchronisation between the research group, these limitation acceptable as it gives one user the possibility tu run a P4 switch with 16x100GE ports …

I presume we can divide the switch even more with bf_knet usage. (but it won’t be a pipe separation as the p4 program will have to handle itself this virtual switchd IP tag. But this is conceptually/technically possible)

switchdctl.py will be adapted so that people can load one program on each pipe if they want.

After reading GRPC documentation guide, it seems the a “PIPE API” is underway but not yet implemented . Do you think this will be the case soon ? It will help us to plan switchdctl.py evolution.

Indeed, in order to be generic enough switchdctl.py needs to:

1- log into p4 switch

2- Ask him which TOFINO chip it is and based on the answer deduce the number of PIPE. In the output below we see that the WEDGE has a T32D (so 2 pipes IIRC your BA-102 training :wink: )

bf-sde.pipe_mgr> dev 0
-------------------------------------
Device|Type |#pipe|#stg|#prsr|#macblk
------|-----|-----|----|-----|-------
0     |T32D |2    |12  |18   |-1

3- needs to ask location of each program on a per pipe basis.

For now, published code is loading 1 program. (we tested with 2 programs against WEDGE100BF32X and BF2556X-1T) But the implementation of the above PIPE discovery steps would be of help as some of our member have also  WEDGE100BF65X and BF6064X that have 4 pipes.

usage: switchdctl.py [-h] [--platform PLATFORM]
                     [--bfruntime-address BFRUNTIME_ADDRESS]
                     [--p4-program-name P4_PROGRAM_NAME] [--p4-base P4_BASE]
                     [--device-id DEVICE_ID] [--client-id CLIENT_ID]
                     [--profile-name PROFILE_NAME]

BfRuntime controller

optional arguments:
  -h, --help            show this help message and exit
  --platform PLATFORM   Platform used: wedge100bf32x, bf2556x1t
  --bfruntime-address BFRUNTIME_ADDRESS
                        BfRuntime address
  --p4-program-name P4_PROGRAM_NAME
                        P4 program to load
  --p4-base P4_BASE     P4 program base directory
  --device-id DEVICE_ID
                        p4 switch device-id
  --client-id CLIENT_ID
                        grpc client-id
  --profile-name PROFILE_NAME
                        TOFINO profile name

Thanks !

Happy hacking

Frederic

Glad you liked it and that it worked for you !

My single program has the same behaviour as my multi-program so it should also work for you.

I’ll commit a new version of switchdctl.py adding multi-program capability and add a NPU parameter so we can offload pipe number guess to the user. :slightly_smiling_face:

I’ll push it ASAP. Will advise once done.

Hi Vladimir,

I’ve a committed a new version of switchdctl.py that you can grab here. This version has the capability:

  • to run one single p4 program on all pipes (pipe_scope [0,1,2,3] default)
  • to run multiple p4 program on dual or quad pipes
    • pipe_scope [0],pipe_scope [1]
    • pipe_scope [0],pipe_scope [1],pipe_scope [2],pipe_scope [3]

To use it:

  • single program on all pipes
p4@UMU-Stordis-bf2556x-1t-P4:~/tmp/rare/bfrt_python$ ./switchdctl.py --single --p4-program bf_router

single pipe output:

SDE=/home/p4/BF/bf-sde-9.3.0
switchdctl.py running on NPU (single pipe mode)
GRPC_ADDRESS: localhost:50052
P4_BASE: /home/p4/BF/bf-sde-9.3.0/install/share/tofinopd/
P4_NAME_LIST: ['bf_router']
DEVICE_ID: 0
CLIENT_ID: 0
PROFILE_NAME: pipe
Binding client[0] to device[0]@localhost:50052 with MASTER role
Subscribe attempt #1
Subscribe response received 0
Setting config for: bf_router
p4_base = /home/p4/BF/bf-sde-9.3.0/install/share/tofinopd/
Sending VERIFY, VERIFY_AND_WARM_INIT_BEGIN and WARM_INIT_END
WARM_INIT_FINISHED received
Config sent for ['bf_router'] !
Tearing down gracefully grpc session with localhost:50052
  • 2 p4 program on dual pipes
./switchdctl.py --multi dual --p4-p0 bf_router_a --p4-p1 bf_router_b

dual pipe output:

SDE=/home/p4/BF/bf-sde-9.3.0
switchdctl.py running on NPU (dual pipe mode)
GRPC_ADDRESS: localhost:50052
P4_BASE: /home/p4/BF/bf-sde-9.3.0/install/share/tofinopd/
P4_NAME_LIST: ['bf_router_a', 'bf_router_b']
DEVICE_ID: 0
CLIENT_ID: 0
PROFILE_NAME: pipe
Binding client[0] to device[0]@localhost:50052 with MASTER role
Subscribe attempt #1
Subscribe response received 0
Setting config for: bf_router_a
Setting config for: bf_router_b
Sending VERIFY, VERIFY_AND_WARM_INIT_BEGIN and WARM_INIT_END
WARM_INIT_FINISHED received
Config sent for ['bf_router_a', 'bf_router_b'] !
Tearing down gracefully grpc session with localhost:50052

If by any chance you could test this script on a quad pipe TOFINO switch, I would be grateful.

However, this does not correct my bf_shell problem. I must miss something in my python 2/3 environment …

(I’m using genuine ubuntu 18.04  / debian 10 install and p4studio_build.py script …) It seems something related to python environment … which environment OS are you using ?

bfshell> bfrt_python 
cwd : /home/p4

Devices found :  [0]
We've found 2 p4 programs for device 0:
bf_router_a
bf_router_b
Creating tree for dev 0 and program bf_router_a

> /home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py(1022)_set_dynamic_method()
-> setattr(self, method_name, types.MethodType(d[method_name], self))
(Pdb) exit
Traceback (most recent call last):
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1905, in start_bfrt
    sts = load_bfrt(dev_id_list)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1870, in load_bfrt
    sts = populate_bfrt(dev_id_list)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1820, in populate_bfrt
    make_deep_tree(p4_name, bf_rt_info, dev_node, cintf)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1759, in make_deep_tree
    BFLeaf(name=prefs[-1], c_tbl=tbl_obj, cintf=cintf, parent_node=parent_node)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 573, in __init__
    self._init_table_methods()
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 986, in _init_table_methods
    self._create_get(key_fields)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1459, in _create_get
bf_rt cli python call failed
    get_method = self._set_dynamic_method(code, method_name)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1022, in _set_dynamic_method
    setattr(self, method_name, types.MethodType(d[method_name], self))
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bfrtcli.py", line 1022, in _set_dynamic_method
    setattr(self, method_name, types.MethodType(d[method_name], self))
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bdb.py", line 48, in trace_dispatch
    return self.dispatch_line(frame)
  File "/home/p4/BF/bf-sde-9.3.0/install/lib/python3.4/bdb.py", line 67, in dispatch_line
    if self.quitting: raise BdbQuit
bdb.BdbQuit
bfrt_cli_cmd:198 could not initialize bf_rt for the cli. err: 1

Hi Vladimir,

It took a bit of time trying to narrow the issue and it seems that the inconsistencies observed are linked to the program you compile. When for example I compile all of your lab program using SDE 9.3.0 and then load the program in one or multiple pipes, it works.

  • simple_l3
p4@bf2556x-1t:~/labs/01-simple_l3/p4src$ p4_build.sh simple_l3.p4 
Using SDE          /home/p4/BF/bf-sde-9.3.0
Using SDE_INSTALL /home/p4/BF/bf-sde-9.3.0/install
Using SDE version bf-sde-9.3.0

OS Name: Ubuntu 18.04.5 LTS
This system has 32GB of RAM and 16 CPU(s)
Parallelization:  Recommended: -j16   Actual: -j16

Compiling for p4_16/tna
P4 compiler path:    /home/p4/BF/bf-sde-9.3.0/install/bin/bf-p4c
P4 compiler version: 9.3.0 (SHA: 242f356) (p4c-based)
Build Dir: /home/p4/BF/bf-sde-9.3.0/build/p4-build/simple_l3
 Logs Dir: /home/p4/BF/bf-sde-9.3.0/logs/p4-build/simple_l3

  Building simple_l3        CLEAR CONFIGURE MAKE INSTALL ... DONE
  • simple_l3_lag_ecmp
p4@bf2556x-1t:~/labs/09-simple_l3_lag_ecmp/solution/p4src$ p4_build.sh simple_l3_lag_ecmp.p4 
Using SDE          /home/p4/BF/bf-sde-9.3.0
Using SDE_INSTALL /home/p4/BF/bf-sde-9.3.0/install
Using SDE version bf-sde-9.3.0

OS Name: Ubuntu 18.04.5 LTS
This system has 32GB of RAM and 16 CPU(s)
Parallelization:  Recommended: -j16   Actual: -j16

Compiling for p4_16/tna
P4 compiler path:    /home/p4/BF/bf-sde-9.3.0/install/bin/bf-p4c
P4 compiler version: 9.3.0 (SHA: 242f356) (p4c-based)
Build Dir: /home/p4/BF/bf-sde-9.3.0/build/p4-build/simple_l3_lag_ecmp
 Logs Dir: /home/p4/BF/bf-sde-9.3.0/logs/p4-build/simple_l3_lag_ecmp

  Building simple_l3_lag_ecmp        CLEAR CONFIGURE MAKE INSTALL ... DONE

When i want to load them:

  • single program on all pipes (simple_l3)
p4@bf2556x-1t:~/rare/bfrt_python$ switchdctl.py --single --p4-program simple_l3 --bfruntime-address 193.49.159.237:50052
SDE=/home/p4/BF/bf-sde-9.3.0
switchdctl.py running on NPU (single pipe mode)
GRPC_ADDRESS: 193.49.159.237:50052
P4_BASE: /home/p4/BF/bf-sde-9.3.0/install/share/tofinopd/
P4_NAME_LIST: ['simple_l3']
DEVICE_ID: 0
CLIENT_ID: 0
PROFILE_NAME: pipe
Binding client[0] to device[0]@193.49.159.237:50052 with MASTER role
Subscribe attempt #1
Subscribe response received 0
Setting config for: simple_l3
p4_base = /home/p4/BF/bf-sde-9.3.0/install/share/tofinopd/
Sending VERIFY, VERIFY_AND_WARM_INIT_BEGIN and WARM_INIT_END
WARM_INIT_FINISHED received
Config sent for ['simple_l3'] !
Tearing down gracefully grpc session with 193.49.159.237:50052

Perform warm reboot and device_add in kernel module pkt path

bfshell> 
bfshell> bfrt_python 
cwd : /home/p4

Devices found :  [0]
We've found 1 p4 programs for device 0:
simple_l3
Creating tree for dev 0 and program simple_l3

Python 3.4.8+ (default, Oct 14 2020, 02:44:53) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0.dev -- An enhanced Interactive Python. Type '?' for help.

bfrt_root>bfrt
--------> bfrt()
Available symbols:
dump                 - Command
info                 - Command
mirror               - Node
pktgen               - Node
port                 - Node
pre                  - Node
simple_l3            - Node
tm                   - Node

bfrt> simple_l3
----> simple_l3()
Available symbols:
dump                 - Command
info                 - Command
pipe                 - Node

bfrt.simple_l3> pipe
--------------> pipe()
Available symbols:
Ingress              - Node
IngressParser        - Node
dump                 - Command
info                 - Command
snapshot_egress      - Table
snapshot_egress_liveness - Table
snapshot_ingress     - Table
snapshot_ingress_liveness - Table

bfrt.simple_l3.pipe>
  • multiple programs on 2 pipes (simple_l3 and simple_l3_lag_ecmp)

p4@bf2556x-1t:~/rare/bfrt_python$ switchdctl.py --multi dual --p4-p0 simple_l3 --p4-p1 simple_l3_lag_ecmp --bfruntime-address 193.49.159.237:50052
SDE=/home/p4/BF/bf-sde-9.3.0
switchdctl.py running on NPU (dual pipe mode)
GRPC_ADDRESS: 193.49.159.237:50052
P4_BASE: /home/p4/BF/bf-sde-9.3.0/install/share/tofinopd/
P4_NAME_LIST: ['simple_l3', 'simple_l3_lag_ecmp']
DEVICE_ID: 0
CLIENT_ID: 0
PROFILE_NAME: pipe
Binding client[0] to device[0]@193.49.159.237:50052 with MASTER role
Subscribe attempt #1
Subscribe response received 0
Setting config for: simple_l3
Setting config for: simple_l3_lag_ecmp
Sending VERIFY, VERIFY_AND_WARM_INIT_BEGIN and WARM_INIT_END
WARM_INIT_FINISHED received
Config sent for ['simple_l3', 'simple_l3_lag_ecmp'] !
Tearing down gracefully grpc session with 193.49.159.237:50052

Perform warm reboot and device_add in kernel module pkt path

bfshell> bfrt_python 
cwd : /home/p4

Devices found :  [0]
We've found 2 p4 programs for device 0:
simple_l3
simple_l3_lag_ecmp
Creating tree for dev 0 and program simple_l3_lag_ecmp

Creating tree for dev 0 and program simple_l3

Python 3.4.8+ (default, Oct 14 2020, 02:44:53) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0.dev -- An enhanced Interactive Python. Type '?' for help.

bfrt_root>

  • using now bf_router P4 program (you can git clone here)
Perform warm reboot and device_add in kernel module pkt path

bfshell> bfrt_python 
cwd : /home/p4

Devices found :  [0]
We've found 1 p4 programs for device 0:
bf_router
Creating tree for dev 0 and program bf_router

> /nix/store/n4k3npid6yn2clz0kqn0ps0rziwpimyb-bf-sde-9.3.0/lib/python3.4/bfrtcli.py(1022)_set_dynamic_method()
-> setattr(self, method_name, types.MethodType(d[method_name], self))
(Pdb)

So there is an issue  related to artefact generated by bf-p4c that is not correctly interpreted into bf_shell.

Again this does not prevent the program to run properly.

Hi Vladimir,

First thanks for spending some times on his problem. I tried your bfrtcli.py and it has solved the problem.

  • I can run a single program now.
bf_switchd: server started - listening on port 9999
bfruntime gRPC server started on 0.0.0.0:50052

        ********************************************
        *      WARNING: Authorised Access Only     *
        ********************************************

bfshell> bfrt_python 
cwd : /home/p4

Devices found :  [0]
We've found 1 p4 programs for device 0:
bf_router
Creating tree for dev 0 and program bf_router

WARNING: BFLeaf._set_dynamic_method(snapshot_ingress, ..., get): more than 255 arguments (<string>, line 2)
WARNING: Skipping setting thre method
WARNING: BFLeaf._set_dynamic_method(snapshot_ingress, ..., get_handle): more than 255 arguments (<string>, line 2)
WARNING: Skipping setting thre method
WARNING: BFLeaf._set_dynamic_method(snapshot_ingress, ..., delete): more than 255 arguments (<string>, line 2)
WARNING: Skipping setting thre method
WARNING: BFLeaf._set_dynamic_method(snapshot_ingress, ..., entry): more than 255 arguments (<string>, line 1)
WARNING: Skipping setting thre method
WARNING: BFLeaf._set_dynamic_method(snapshot_ingress, ..., add): more than 255 arguments (<string>, line 2)
WARNING: Skipping setting thre method

Python 3.4.8+ (default, Dec 14 2020, 17:04:19) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0.dev -- An enhanced Interactive Python. Type '?' for help.
  • Also switchdctl.py could now send p4 programs whether we want to use all the pipe or if we want to use the pipes independently.
bfshell> bf_switchd: starting warm init for dev_id 0 mode 1 serdes_upgrade 0 
Removing Thrift service for bf-platforms from server
bf_switchd: thrift deinitialized for agent : 0
2020-12-17 09:03:46.560128 BF_PLTFM ERROR - pltfm_mgr: Error while de-initializing pltfm mgr
bf_switchd: agent[0] library unloaded for dev_id 0
bf_switchd: library /home/p4/BF/bf-sde-9.3.0/install/lib/libpltfm_mgr.so loaded
bf_switchd: agent[0] initialized
Tcl server started..
Tcl server: listen socket created
Tcl server: bind done on port 8008, listening...
Tcl server: waiting for incoming connections...
sh: 1: onie-syseeprom: not found
Health monitor started 
Adding Thrift service for bf-platforms to server
bf_switchd: thrift initialized for agent : 0
-Perform warm reboot and device_add in kernel module pkt path

bfshell> bfrt_python 
cwd : /home/p4

Devices found :  [0]
We've found 2 p4 programs for device 0:
simple_l3
simple_l3_lag_ecmp
Creating tree for dev 0 and program simple_l3

Creating tree for dev 0 and program simple_l3_lag_ecmp

Python 3.4.8+ (default, Dec 14 2020, 17:04:19) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0.dev -- An enhanced Interactive Python. Type '?' for help.

bfrt_root>

Strangely enough for simple_l3 and simple_l3_lag_ecmp does not seem to pass through the same part of bfrtcli.py. (The warning observed are raised only for bf_router p4 program)

Thanks !

Let’s wait the patch correction for the next SDE version !

Hi,

As expected from you, the problem has indeed gone … :slightly_smiling_face:

I confirm that your bfrtTable.py fix cleared the problem.

bfruntime gRPC server started on 0.0.0.0:50052

        ********************************************
        *      WARNING: Authorised Access Only     *
        ********************************************

bfshell> bfrt_python 
cwd : /home/p4/sde-9.3.0

Devices found :  [0]
We've found 1 p4 programs for device 0:
bf_router
Creating tree for dev 0 and program bf_router

Python 3.4.8+ (default, Dec 14 2020, 17:04:19) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0.dev -- An enhanced Interactive Python. Type '?' for help.

bfrt_root>

Cherry on the cake switchdctl.py can now set a program on all pipes and also multiple program in each pipe without crashing.

-Perform warm reboot and device_add in kernel module pkt path
bfshell> bfrt_python
cwd : /home/p4/sde-9.3.0

Devices found :  [0]
We've found 2 p4 programs for device 0:
simple_l3
simple_l3_lag_ecmp
Creating tree for dev 0 and program simple_l3

Creating tree for dev 0 and program simple_l3_lag_ecmp

Python 3.4.8+ (default, Dec 14 2020, 17:04:19) 
Type 'copyright', 'credits' or 'license' for more information
IPython 6.5.0.dev -- An enhanced Interactive Python. Type '?' for help.

bfrt_root>

I tested it on T32D NPU, do you have (or anyone else) having access to a machine with T32Q or T64Q NPU can confirm that we can load 4 program on each pipe ?

The syntax would be:

switchdctl.py --multi quad --p4-p0 simple_l3 --p4-p1 simple_l3_lag_ecmp --p4-p2 simple_l3_nexthop --p4-p3 simple_l3_mpls --bfruntime-address 0.0.0.0:50052

I don’t have access to a P4 switch with 4 pipes…

Waiting eagerly to buy a TOFINO2 switch !

PS: As a side note, we see already reference to TOFINO 3 in the SDE 9.3.0… :slightly_smiling_face: