issue on running video_profiling tool in CSL .

[Jacky_Gupta]

Hi ,

I am able to run profiling tool in EIS (without having orchestration).

But when i follow the same steps on CSL system(with orchestration) in client system i get below error.

*********************************************************************************************************

edge:~/edge_insights_industrial/Edge_Insights_for_Industrial_2.3.2/IEdgeInsights/tools/VideoProfiler$ python3.6 video_profiler.py
2021-02-01 16:20:27 ies-edge eis.etcd_client[797] ERROR Exception raised in _setEnv with error:
Exception in thread Thread-2:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 263, in get_response
metadata=self.metadata
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1612176627.748427384","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4134,"referenced_errors":[{"created":"@1612176627.748425233","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "video_profiler.py", line 407, in thread_runner
config_dict)
File "video_profiler.py", line 150, in __init__
self.config_client = cfg_mgr.get_config_client("etcd", conf)
File "/usr/local/lib/python3.6/dist-packages/eis/config_manager.py", line 39, in get_config_client
return EtcdCli(config)
File "/usr/local/lib/python3.6/dist-packages/eis/etcd_client.py", line 84, in __init__
self._setEnv()
File "/usr/local/lib/python3.6/dist-packages/eis/etcd_client.py", line 103, in _setEnv
raise e
File "/usr/local/lib/python3.6/dist-packages/eis/etcd_client.py", line 92, in _setEnv
value = self.etcd.get(global_env_prfix)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 285, in get
range_response = self.get_response(key, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError

2021-02-01 16:20:27 ies-edge eis.etcd_client[797] ERROR Exception raised in _setEnv with error:
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 46, in handler
return f(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 263, in get_response
metadata=self.metadata
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 826, in __call__
return _end_unary_response_blocking(state, call, False, None)
File "/usr/local/lib/python3.6/dist-packages/grpc/_channel.py", line 729, in _end_unary_response_blocking
raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "failed to connect to all addresses"
debug_error_string = "{"created":"@1612176627.749969729","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":4134,"referenced_errors":[{"created":"@1612176627.749967571","description":"failed to connect to all addresses","file":"src/core/ext/filters/client_channel/lb_policy/pick_first/pick_first.cc","file_line":398,"grpc_status":14}]}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/lib/python3.6/threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "video_profiler.py", line 407, in thread_runner
config_dict)
File "video_profiler.py", line 150, in __init__
self.config_client = cfg_mgr.get_config_client("etcd", conf)
File "/usr/local/lib/python3.6/dist-packages/eis/config_manager.py", line 39, in get_config_client
return EtcdCli(config)
File "/usr/local/lib/python3.6/dist-packages/eis/etcd_client.py", line 84, in __init__
self._setEnv()
File "/usr/local/lib/python3.6/dist-packages/eis/etcd_client.py", line 103, in _setEnv
raise e
File "/usr/local/lib/python3.6/dist-packages/eis/etcd_client.py", line 92, in _setEnv
value = self.etcd.get(global_env_prfix)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 285, in get
range_response = self.get_response(key, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 48, in handler
_translate_exception(exc)
File "/usr/local/lib/python3.6/dist-packages/etcd3-0.10.0-py3.6.egg/etcd3/client.py", line 32, in _translate_exception
raise exception
etcd3.exceptions.ConnectionFailedError

2021-02-01 16:20:27 ies-edge __main__[797] INFO Average FPS for each topic {}
2021-02-01 16:20:27 ies-edge __main__[797] INFO Total FPS for 10 frames 0.0
ieisw@ies-edge:~/edge_insights_industrial/Edge_Insights_for_Industrial_2.3.2/IEdgeInsights/tools/VideoProfiler$

****************************************************************************************************************

Is there any other changes to be done on CSL

[Murni_J_Intel]

Hi, 

May I know if you are running the this tool on the same node that VI and VA container is running? Can you help to share the configuration for your VideoProfiler and the current running module on the CSL manager UI?

[Jacky_Gupta]

Hi @Murni_J_Intel ,

I am only using VI container and running this tool on the same client node where VI is running.

currently i am trying just for FPS mode with dev mode false(in both .env files).

I have attached config file .

EIs version :2.3.2.

[Murni_J_Intel]

Hi @Jacky_Gupta 

Let me take a look at this issue and let you know the update.

[Murni_J_Intel]

Hi @Jacky_Gupta 

There are few extra steps that need to be done in order to run the VideoProfiler tools with CSL. In your case mentioned, you need to check for the port and container IP of your Video Ingestion.

1. On the node system that Video Ingestion is running, run $ docker ps command to get the container ID of VideoIngestion. 

2. On the same system, run $ docker inspect <container_ID>.  Change the <container_ID> accordingly.

3. Check for the VideoIngestion container IP and port. Below is the example of VideoIngestion IP and Port

4.  update the config.json file for VideoProfiler. Update the IP, Port, and certificates accordingly. For CSL, you can replace the cert as below 

"certFile" : "../../build/provision/Certificates/root/root_client_certificate.pem",
"keyFile" : "../../build/provision/Certificates/root/root_client_key.pem",
"trustFile": "../../build/provision/Certificates/ca/ca_certificate.pem"

5.  Update the ETCD configuration in env.sh file. You can get the ETCD configuration details at [WORK_DIR]/IEdeInsights/build/ in env.sh file. You will also need to add "export" at the beginning of each parameter as below. 

export PYTHONPATH=$PYTHONPATH:../../:../../common/libs/EISMessageBus:../../common
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib

# For running VideoProfiler on a separate node or with CSL, set these variables to
# point to the etcd on different node
export ETCD_HOST=192.168.0.10
export ETCD_CLIENT_PORT=2379
# ETCD prefix for the key
# Eg. ETCD_PREFIX=/csl/apps/EIS
export ETCD_PREFIX=/csl/apps/EIS
export no_proxy=localhost,127.0.0.1,$ETCD_HOST

if you are running behind a proxy, ensure the etcd_host IP is being included in no_proxy seeting.

6. source the env file and run the videoprofiler tool again. 

Let me know if you are able to get the expected result from your end. 

[Jacky_Gupta]

Hello @Murni_J_Intel ,

i have followed all the below steps but got error.

After running step 1 & 2 ,i got port no :29170.

This was updated in config.json as per step 4

certificates is by default same as provided by you.

In step 5 env.sh is updated as mentioned by you.

i think only issue i might have faced was with container Ip.

can you please let me know how to get that and where to update it.

all the relevant files are attached.

kindly let me know how to proceed .

[Murni_J_Intel]

Hi @Jacky_Gupta 

Can you help to verify if your VideoIngestion2 container is running on the CSL Manager UI? if yes, when you run command $docker inspect <container_id> you should be able to see as below example to get the IP of that container. 

please note that for env.sh file, you need to update the ETCD configuration based on your ETCD configuration at [WORK_DIR]/IEdgeInsights/build env.sh file, previously I only provided you the example.

[Jacky_Gupta]

Hi @Murni_J_Intel ,

I have updated all the files as mentioned by you but still getting error.

I have send all the configuration files along with  error logs.

can you pls check the configurations.

[Murni_J_Intel]

Hi @Jacky_Gupta 

The error seems caused by the certificate issue. As previously mentioned, you need to change the certificates to use the one that is generated for CSL and change the 127.0.0.1 IP to your VideoIngestion container IP

[Jacky_Gupta]

Hello @Murni_J_Intel ,

Thanks Murni i have updated IP from 127.0.0.1 to 10.105.38.118, but i could not find the certificates that is generated for CSL.

Can you pls specify the where i can get this.

I looked into below path

[WORK_DIR]/IEdgeInsights/build/provision/Certificates/root 

[WORK_DIR]/IEdgeInsights/build/provision/Certificates/ca and it contained the same file as mentioned in config.json as shown below.

"certFile" : "../../build/provision/Certificates/root/root_client_certificate.pem",
"keyFile" : "../../build/provision/Certificates/root/root_client_key.pem",
"trustFile": "../../build/provision/Certificates/ca/ca_certificate.pem"

[Murni_J_Intel]

Hi @Jacky_Gupta 

This system is CLS node system that you did the provisioning for CLS right? if that is the case, you can find the certificates on below directory.

../../build/provision/Certificates/csl

[Jacky_Gupta]

Hi @Murni_J_Intel ,

I have done all the steps, but after i run 

"python3.6 video_profiler.py" cmd it doesnot show any output.

I have shared the screen shot of it.

Is there any thing to be changed for below steps as per read.md

5. If using Video Profiler in PROD mode, make sure to set required permissions to certificates.

```sh

sudo chmod -R 755 ../../build/provision/Certificates/ca

sudo chmod -R 755 ../../build/provision/Certificates/root

[Murni_J_Intel]

Hi, 

By default it in the config.json it has been to 1000 frame, just to ensure everything can be running as expected you can change it to 10 first and see if you are getting the expected output. For the permission on the certificate, yes you will need to pass the correct permission as below.

sudo chmod -R 755 ../../build/provision/Certificates/csl

[Jacky_Gupta]

Hi @Murni_J_Intel ,

Thanks i could run video profiling tool in CSL setup.But I could run that only in 'FPS' mode.I tried to run in "Monitor" mode by changing parameter in config.json from 'FPS' to 'Monitor'

Making profiler mode to 'true'  in .env of build directory.Also updating the same in csl app-spec.json file.

All other condition and configuration remains same as per my previous communication.But i get error .I have attached it's snapshot

can you please let me know ,if other configuration changes is required for it.

[Murni_J_Intel]

Hi @Jacky_Gupta 

If you want to run the videoprofiler using "monitor" mode. You can follow below steps

1. Change in the config.json from "fps" to monitor

2. Update VI and VA container in csl_app_spec.json to set the profiling mode=true.

3. Delete your current running application in CSL manager UI.

4. Submit the new application will the updated csl_app_spec.json

5. Check for the new VI container IP, it might be changed now as you are running a new application.

6. update the VI container IP accordingly in the config.json file

7. source the env.sh and run again the VideoProfiler.

8. All other configuration should remain unchanged

let me know if this is working on your side