2 video ingestion containers using orchestrator

[Smrithi]

I am trying to run two video ingestion containers simultaneously. I have made all the necessary changes in docker compose and eis_config.json files, provisioned it and built it. I was also able to update container image using smr in the csl server. When I tried to add 2 ingestors in module spec file, I got an endpoint error while pasting the module spec in the submit new app window in csl manager dashboard. I am also unabe to view the logs through csl manager dashboard as it says error retreiving logs. Despite having multiple video ingestion modules will there only be one docker image of video ingestion after executing docker compose build?

Kindly help.

[Smrithi]

I have also attached the SS of the module spec file and the subsequent error message for reference.

[Murni_J_Intel]

Hi @Smrithi 

In order to run multiple container, it will be good if using the eis_builder.py to help generated all of the configuration file including the csl_app_spec.json instead of manually editing all of that file to avoid any human error.

you can follow the below step to generate all configuration files using eis_builder.py for multiple container for VideoIngestion.

1. Go to [WORK_DIR]/IEdeInsights/VideoIngestion

2. edit the docker-compose.yml and config.json file accordingly to run multiple instances. You can refer  to [WORK_DIR]/IEdgeInsights/build/sample/multi_cam_sample for the example on running multi instances.

3. Go to [WORK_DIR]/IEdgeInsights/build and run python3 eis_builder.py

after successfully running that,  all of the configuration files including csl_app_spec.json will be created to run multiple VideoIngestion containers accordingly.  After done, you can proceed to provision, register again the app_spec and do the deployment. 

Let me know if this helps to solve your issue. 

[Smrithi]

Hi,

I made all the necessary changes as suggested in the docker-compose.yml and config.json file inside  [WORK_DIR]/IEdeInsights/VideoIngestion. But when I ran the python3 eis_builder.py to update the changes in the csl_app_spec.json, it did not reflect the changes made by me, instead reverted back to the very initial configuration it had. I am attaching the SS of generated  csl_app_spec.json and docker-compose.yml and config.json of video ingestion module that I edited, for your reference.

[Murni_J_Intel]

Hi @Smrithi 

Sorry for the inconvenience.  These features will actually improve on the next PV release. In the meantime, I have attached below the csl_app_spec and VideoIngestion_module_spec files for your reference. You can use the attached file and change the naming accordingly, below file indicate to create 2 containers for VideoIngestion. 

Let me know if this is working on your side.

Note: please convert the txt file to JSON format before use it.

[Murni_J_Intel]

Hi @Smrithi 

for the csl_app_spec.json file that I have shared just now, you need to changes a few parameters on the ExecutionEnv to make it work with the current EIIv2.3 

VideoIngestion 1

 "AppName": "VideoIngestion1",
"ETCD_ENDPOINT": "${datastore.endpoint}",
"CONFIGMGR_CACERT": "${databucket.cacert}",
"CONFIGMGR_CERT": "${databucket.cert}",
"CONFIGMGR_KEY": "${databucket.key}",
"ETCD_PREFIX": "",
"DEV_MODE": "false",
"Clients": "VideoAnalytics",
"PROFILING_MODE": "false",
"ZMQ_RECV_HWM": "50",
"PubTopics": "camera1_stream",
"camera1_stream_cfg": "zmq_tcp,0.0.0.0:${ep.output-vi.localport}",
"Server": "zmq_tcp,0.0.0.0:${ep.output-vi-sw-tr.localport}"

VideoIngestion2 

 "AppName": "VideoIngestion1",
"ETCD_ENDPOINT": "${datastore.endpoint}",
"CONFIGMGR_CACERT": "${databucket.cacert}",
"CONFIGMGR_CERT": "${databucket.cert}",
"CONFIGMGR_KEY": "${databucket.key}",
"ETCD_PREFIX": "",
"DEV_MODE": "false",
"Clients": "VideoAnalytics",
"PROFILING_MODE": "false",
"ZMQ_RECV_HWM": "50",
"PubTopics": "camera2_stream",
"camera1_stream_cfg": "zmq_tcp,0.0.0.0:${ep.output-vi.localport}",
"Server": "zmq_tcp,0.0.0.0:${ep.output-vi-sw-tr.localport}"

and maintain the Endpoints configuration as the new file that I shared. 

[Murni_J_Intel]

Hi @Smrithi 

Attached is the generated csl_app_spec.json for EIIv2.3 for multi-container, you can directly use this with the attached file. 

[Smrithi]

Hi,

I did all the changes as suggested in the shared csl_app_spec.json and VideoIngestion_module_spec.json. But when I submit the csl_app_spec.json file in the csl manager dashboard both the video ingestion containers are not running simultaneously. If video ingestion1 is running video ingestion 2 is pending and vice-versa. Also I cannot see any video in the web visualizer and it is shown as disconnected. I have attaches the SS for your reference.

[WengWai_C_Intel]

Hi @Smrithi ,

For multi Instance of video ingestion we will use the same module spec for all instances, which means you no need to register the module spec multiple times. This is because down the layer it use same image & configurations. You just need to register the video ingestion module spec once into SMR. Here in attachment is the module spec sample for 2 video ingestion instances you can use. 

Note: please change the attachment of module spec file extension from .txt back to .json before you use it.

[Smrithi]

I registered the updated video ingestion module spec shared here, into smr only once. And did the corresponding changes in docker-compose.yml and eis-config.json and built it. But the docker images did not update after building it freshly. And when I submitted the csl_app_spec.json, with 2 video ingestion modules given by @Murni_J_Intel , only one video ingestion module is shown when I do docker ps and both the video ingestion modules are always shown as pending in the dashboard. I have attached the SS of the manager dashboard also for your reference. 

[WengWai_C_Intel]

@Smrithi 

After reviewing your csl_app_spec.json file, if you are not using any accelerators for the VideoAnalytics 1 and 2 containers, please remove the "Constraints" section to exclude the inferencing accelerators hardware. Then use this csl_app_spec.json file to submit as new app in the CSL manager UI.

Before you submit the new app in CSL manager UI, please make sure you follow the User Guide doc section " EII Master Node Provisioning in Castle Lake Client Node" to provision your CSL client node with the new docker-compose.yml file you have that both 2nd VideoIngestion and VideoAnalytics being defined to ensure the ETCD store has all the key for these new added containers.

[Smrithi]

I am not using any VideoAnalytics container in eis orchestrator. Before submitting the csl_app_spec.json file, I have provisioned and built to incorporate the new changes made in the docker-compose.yml and eis_config.json files . Both the provisioning and build was successful. But after submitting  csl_app_spec.json to CSL manager UI, when I run docker ps in client node to view the containers it just shows videoingestion1, when actually there are 2 videoingestion modules mentioned in the docker-compose.yml and eis_config.json files. I have attached the SS of that for reference. Should there be 2 separate video ingestion container image for this or one common ia_video_ingestion container image?

[WengWai_C_Intel]

@Smrithi 

If this is the case, you can remove the VideoAnalytics and VideoAnalytics2 section from your csl_app_spec.json file and update the "Endpoints" section for Visualizer and WebVisualier, also update the  "Links" section as well.

For VideoIngestion container, it should require only one container image from the SMR as both will use the same base image. You can see both the VideoIngestion and VideoIngestion2 in the csl_app_spec.json file has defined it as below.

"ManifestFile": "${idx:videoingestion:2.3}"

[Smrithi]

I have already done all these mentioned changes in the csl_app_spec.json and single video ingestion was working fine. But when I extend this to 2 video ingestion containers by making all the necessary changes in csl_app_spec.json both the video ingestion containers are not running and I have also attached the csl_app_spec.json with the necessary changes that I am updating to manager UI for your reference. They are just shown as pending. I do not understand where the problem lies.

[WengWai_C_Intel]

@Smrithi 

May i know what is your hardware configuration on your current setup, e.g how many CPU cores and memory RAM? You can check this by using 'lscpu' and 'free -m' command.

In your csl_app_spec.json file, each of the VideoIngestion workload has been defined to use 4 CPU cores. In this case, you will need min 8 CPU cores for VideoIngestion alone. You may reduce it from '4000' to '1000' for both VideoIngestion configuration in csl_app_spec.json file and redeploy it.

[Smrithi]

Actually there are 8 cpu cores and 16 gb RAM for the machine that I am using. Reducing it from '4000' to '1000' for both VideoIngestion configuration in csl_app_spec.json file and redeploying  it solved the issue. But suppose that I have to run and view 34 video ingestion containers simultaneously, there will definitely be further resource crunch. How to go about solving that, would increasing the capacity of the machines solve it? Also what is the basis of these resource distribution?

[WengWai_C_Intel · Accepted solution]

@Smrithi 

It is good to root caused the issue in the setup now. The app_spec file that we shared earlier for your reference already reduced the CPU allocation for VideoIngestion to 1000 (1 cpu), you should be able to use it and just update according to the containers that you are running.

For respective use case, you can refer to modules.pdf file under the orchestrator/docs directory for resources configuration that suite to own use cases. Since orchestrator is mainly targeting to setup distributed nodes, you can add additional client node as per setup guide. Having only 1 client node as per current setup, may not make full use of orchestrator functionality.