Error retrieving the logs from csl manager dashboard

[Smrithi]

I am having trouble retrieving logs from the csl manager dashboard. Whenever I click on retrieve logs there pops up a message saying error retrieving logs. I have attached the SS also. How to solve this issue?

[Smrithi]

As the logs in dashboard were still not available we have seen them in the terminal. In those logs we could observe socker related error. Because of this error the notification are not coming in our custom built UI.

Invalid session HLImtQkQy9FsL4-uAAAO (further occurrences of this error will be logged with level INFO)
ERROR:engineio.server:Invalid session HLImtQkQy9FsL4-uAAAO (further occurrences of this error will be logged with level INFO)
INFO:werkzeug:10.32.0.1 - - [28/Jan/2021 09:12:00] GET /socket.io/?EIO=4&transport=polling&t=NT8ELCU&sid=HLImtQkQy9FsL4-uAAAO HTTP/1.1 400 -

[Smrithi]

We are getting time out error in the UI as shown in the attached SS.

[WengWai_C_Intel]

@Smrithi 

Firstly, can you help us understand the scenario that you are seeing on the setup you have currently? Is that the web_visualizer error log is not able to be displayed only, or all other containers also having similar issue? If all containers also having similar issue not able to view the error log from the web UI, can you try reboot the CSL manager/server system and confirm if still same issue? By the way, can you also share what setup/changes has been made since the earlier working setup was done with all containers are running? These info might help to further understand what potentially changes to the setup that may cause error log not appear from the UI.

Thanks!

[Snehal_Yesugade]

Hello

We have done 2 changes -

1. The application was running for 2 ingestions which we increased to 10 now

2. Movidius card is inserted on worker node 2

"The error retrieving logs" is seen on both the nodes worker node 1 and worker node 2

[WengWai_C_Intel]

@Smrithi 

As per suggestion earlier, have you try reboot the CSL manager/server system and confirm if still same issue that cannot retrieve error log? Alternatively, can you reset the weave interface for the CSL manager/server system, since both CSL worker node 1 and 2 having similar scenario?

[Snehal_Yesugade]

Hello

Today we could observe that -

On worker node 2 still the issue persists - no logs are seen

On worker node 1 we could see logs, but when we are trying to connect to UI it shows can not connect

Also we we saw logs on terminal for worker node 2. For etcd container we are getting errors, logs are as attached

This is a strange behaviour, the webvisualizer is unable to load UI ! on both the nodes

[WengWai_C_Intel]

@Smrithi and @Snehal_Yesugade ,

Since you can see the log now for node 1 from the CSL mgr UI, can you share the latest error log message and for node 2 share the error log from terminal? 

As you have done changes to increase the ingestion from 2 to 10, can you firstly verify by running the previous configuration (2 ingestions) to see if the previous setup still can be up and running to rule out issues due to configuration? If the previous configuration is up running as normal, can you share us the files that you have made changes so we can help to check on your setup? Thanks!