How to handle video conference more than 10 participants using Intel CS for WebRTC framework?

[Chandramouli_P]

Hello Team,

Good evening and hope all are doing well. With the great support and help of you all, we developed our video conferencing product using Intel CS for WebRTC framework and now it is in live testing by our existing client base. As part of this, we are getting some issues, which are related to memory. Please find the below information:

Existing Infrastructure & Configuration:
1) Hosted platform: Amazon Web Services (AWS)
2) MCU (v. 4.2.1): CentOS v.7.6.1810 (x86_64), Kernel v.3.10.0-957.1.3.el7.x86_64, 16 vCPU, 32 GB RAM
3) RabbitMQ (v. 21.2.3): CentOS v.7.6.1810 (x86_64), Kernel v.3.10.0-957.1.3.el7.x86_64, 8 Core, 64 GB RAM
4) MongoDB (v. 4.0.6): CentOS v.7.6.1810 (x86_64), Kernel v.3.10.0-957.1.3.el7.x86_64, 4 Core, 32 GB RAM
5) ICE (STUN/TURN): CentOS v.7.6.1810 (x86_64), Kernel v.3.10.0-957.1.3.el7.x86_64, 8 vCPU, 16 GB RAM

MCU Configuration: Configured with default values in all .toml files and with out enabling hardware acceleration
Codecs that are using: Using VP8, H.264, and OPUS codecs, SFU mode

First, what I noticed is many users facing same or similar issues. When we are trying to have video conference with more than 8 members, call is not continuing smoothly. We are facing many issues like subscription error, blank screens, unable to see video/hear audio, disconnections, etc. As you noticed, we are using the servers with high configuration. If we noticed and observed correctly based on the old threads in this forum, many users are reported that workers are consuming more memory. The same thing is happening in our case also. But, at the same time, our overall CPU consumption is not more than 8% and overall RAM consumption is not more than 16% in the MCU server and screenshots are attached here.

1) Is there any memory limitations for webrtc or video agents or any other agents?
2) If there are memory limitations, what we have to do to increase the memory limitations?
3) Do we need to change the value of:
    #The max CPU/GPU load under which this worker can take new tasks.
    max_load = 0.85 #default: 0.85
4) Do we need to change the values of any other parameters in .toml files or anywhere?

I don't think that Intel CS for WebRTC framework can handle only maximum of 10 particpants per meeting, moreover we are using high-configured servers.

Please find the below error log and particularly attached screen shots for your reference.

Errors in one call:
INFO: Conference - And is recovering the previous subscription: {"id":"52942343341376250","locality":{"agent":"webrtc-fd127a5545c6e5c339f1@10.1.0.97","node":"webrtc-fd127a5545c6e5c339f1@10.1.0.97_80"},"media":{"audio":{"from":"91235689200","format":{"codec":"opus","sampleRate":48000,"channelNum":2},"status":"active"},"video":{"from":"91235689200","format":{"codec":"vp8"},"parameters":{"framerate":15,"resolution":{}},"status":"active"}},"info":{"type":"webrtc","owner":"shFwwKOxUWuDhaYhAAR1"}}
2020-04-14 22:34:25.223  - ERROR: RoomController - subscribe failed, reason: Failed in scheduling video worker, reason: No worker available, all in full load.
2020-04-14 22:34:25.223  - INFO: Conference - roomController.subscribe failed, reason: Failed in scheduling video worker, reason: No worker available, all in full load.
2020-04-14 22:34:25.299  - ERROR: RoomController - subscribe failed, reason: Failed in scheduling video worker, reason: No worker available, all in full load.
2020-04-14 22:34:25.299  - INFO: Conference - roomController.subscribe failed, reason: Failed in scheduling video worker, reason: No worker available, all in full load.
2020-04-14 22:34:25.300  - INFO: Conference - Update subscription failed: roomController.subscribe failed, reason: Failed in scheduling video worker, reason: No worker available, all in full load.

Errors in second call:
main-es2015.91f0c7d8c5949159a690.js:1 ERROR Error: Uncaught (in promise): Object: {"name":"Error","message":"Failed in scheduling conference worker, reason: No worker available, all in full load."}
    at P (polyfills-es2015.320e13241f9346356285.js:1)
    at P (polyfills-es2015.320e13241f9346356285.js:1)
    at polyfills-es2015.320e13241f9346356285.js:1
    at a.invokeTask (polyfills-es2015.320e13241f9346356285.js:1)
    at Object.onInvokeTask (main-es2015.91f0c7d8c5949159a690.js:1)
    at a.invokeTask (polyfills-es2015.320e13241f9346356285.js:1)
    at s.runTask (polyfills-es2015.320e13241f9346356285.js:1)
    at m (polyfills-es2015.320e13241f9346356285.js:1)
    at c.invokeTask [as invoke] (polyfills-es2015.320e13241f9346356285.js:1)
    at y (polyfills-es2015.320e13241f9346356285.js:1)
yt @ main-es2015.91f0c7d8c5949159a690.js:1
main-es2015.91f0c7d8c5949159a690.js:1 ERROR TypeError: Cannot read property 'subscription' of undefined
    at h.mobileLayoutmute (3-es2015.5a404112eea69c8376d5.js:1)
    at m.layoutMute (3-es2015.5a404112eea69c8376d5.js:1)
    at s.onMuteEvent (21-es2015.2f04b7e44fb4ea77c073.js:1)
    at Object.handleEvent (21-es2015.2f04b7e44fb4ea77c073.js:1)
    at Object.handleEvent (main-es2015.91f0c7d8c5949159a690.js:1)
    at Object.handleEvent (main-es2015.91f0c7d8c5949159a690.js:1)
    at ls (main-es2015.91f0c7d8c5949159a690.js:1)
    at main-es2015.91f0c7d8c5949159a690.js:1
    at u.r [as _next] (main-es2015.91f0c7d8c5949159a690.js:1)
    at u.__tryOrUnsub (main-es2015.91f0c7d8c5949159a690.js:1)
yt @ main-es2015.91f0c7d8c5949159a690.js:1

If you need any other information, please let me know. Any help would be appreciated.

Thank you.

Best Regards,
Chandramouli.

[Naresh_R_1]

Hi Team,

Yes this is happening with us too. there is memory heap issues happening with management API. 

Dear Chandramouli,

Do u have any experience with Mediasoup version 3, it has got simulcast and SVC and seems to work fine we have tested with more than 12 users in single core machine with 50 percent usage of CPU. 

Please let me know you have any experience on call quality and stability.

Thanks

Naresh

[Chandramouli_P]

Hello Naresh,

Good morning. Yes, I had experience with Mediasoup v3. But, that is not suitable to fulfill our requirement.

Thank you.

Best Regards,
Chandramouli.

[Naresh_R_1]

Hi,

How many rooms does the MCU has, as per my observation if MCU has more rooms more than 100 has some problem.

Thanks

Naresh

[Chandramouli_P]

Hello Intel team,

Good evening and any update would be appreciated.

Thank you.

Best Regards,
Chandramouli.

[Chandramouli_P]

Hello Intel team,

Good evening and any help would be appreciated.

Thank you.

Best Regards,
Chandramouli.