All webrtc peer connections disconnected.

[N__Aji]

We have hosted intel 4.3.1 server in AWS Ec2 instances.  The webrtc server is working fine. But sometimes all webrtc connections in the server disconnects automatically. We got the subscription error and publication error during that time. All webrtc connections reconnects immediately after the disconnection with our own re-connection scenario.
 
Our application structure will look like this : Around 356 plus concurrent peer connections during that time. 60 plus streams to be combined in to one stream and the combined stream to be subscribed by 60 plus users.    
 
We are using two AWS server  for hosting the intel 4.3.1 server.  An 8 core CPU and 16 GB RAM with Ubuntu 18.04 server is used for video and audio agent. All other agents (Management API, Portal, Conference agent, and Webrtc agent) are running in another Ubuntu 18.04 server with 4 core 8 GB  RAM. 

The maximum resource usage of these servers are fine during the issue occurred time.

What can be the reasons for this disconnections?

Appreciating your opinions and guidelines in advance.

[Jijo-Jayaprakash]

You need to configure the TURN server to avoid disconnection.https://ourcodeworld.com/articles/read/1175/how-to-create-and-configure-your-own-stun-turn-server-with-coturn-in-ubuntu-18-04

[N__Aji]

Thank you for your valuable support Jijo-Jayaprakash and sorry for the late reply.  We had already configured a Turn server and integrated with our current media server cluster.  We don't believe this issue is related to turn server, because our clients machines are not behind in NAT and Firewalls.  

We strongly believe that the webrtc child nodes are ended unexpectedly. Please check out the webrtc and conference agent logs.  

Conference agent logs.

2022-12-18 04:38:45.901 - INFO: Conference - streamControl failed TypeError: Cannot read property 'media' of undefined

at /root/Release-v4.3.1/conference_agent/conference.js:1:21754

at <anonymous>

at process._tickCallback (internal/process/next_tick.js:189:7)

2022-12-18 04:38:55.887 - INFO: Conference - streamControl failed TypeError: Cannot read property 'media' of undefined

at /root/Release-v4.3.1/conference_agent/conference.js:1:21754

at process._tickCallback (internal/process/next_tick.js:189:7)

2022-12-18 04:39:33.294 - INFO: AccessController - Fault detected on node: { agent: 'webrtc-ba09dcc77c07cc89835b@0.0.12.0',

node: 'webrtc-ba09dcc77c07cc89835b@_1731' }

2022-12-18 04:39:33.295 - INFO: AccessController - Fault detected on node: { agent: 'webrtc-ba09dcc77c07cc89835b@0.0.12.0',

node: 'webrtc-ba09dcc77c07cc89835b@_1731' }

2022-12-18 04:39:33.296 - INFO: AccessController - Fault detected on node: { agent: 'webrtc-ba09dcc77c07cc89835b@0.0.12.0',

node: 'webrtc-ba09dcc77c07cc89835b@_1731' }

2022-12-18 04:39:33.296 - INFO: AccessController - Fault detected on node: { agent: 'webrtc-ba09dcc77c07cc89835b@0.0.12.0',

node: 'webrtc-ba09dcc77c07cc89835b@_1731' }

2022-12-18 04:39:33.297 - INFO: AccessController - Fault detected on node: { agent: 'webrtc-ba09dcc77c07cc89835b@0.0.12.0',

node: 'webrtc-ba09dcc77c07cc89835b@_1731' }

Please see the below logs of the webrtc worker webrtc-ba09dcc77c07cc89835b@_1731 on the same time.

2022-12-18 04:33:11,322 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 8

2022-12-18 04:33:12,322 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 8

2022-12-18 04:33:13,322 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 7

2022-12-18 04:33:23,799 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:34:33,743 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:34:37,995 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:34:41.580 - INFO: Connections - Connection does NOT exist:829826927633566100

2022-12-18 04:34:41.581 - INFO: Connections - Connection does NOT exist:134666530254979870

2022-12-18 04:36:54,980 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:36:55,980 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 13

2022-12-18 04:36:57,028 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:36:57,028 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:36:58,029 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 13

2022-12-18 04:37:33,233 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

2022-12-18 04:38:16,776 - DEBUG: TimeoutChecker - Handling dtls timeout, checks left: 14

Remaining no more information are available in that file. 

Appreciating your opinions and guidelines in advance.