Intel webrtc version 4 stream subscription failed above 128 subscribers

[Nithin_P_]

Hi

                  Today i have done a load test on intel mcu version 4. two presenter published and viewers subscribing mixed stream.when the viewers count reach above 128 then no new users can subscribe the stream. when new users joined then showing an error in the console. i have attached the error.but this error not affecting existing subscribers but unable to subscribe for new users.I think this is a very serious issue for large scale applications

I have attached the console error. please review 

invermcu version :  4.0

server : centos7

used cluster with multiple webrtc agents

browser used : google chrome latest version

Thank you,

[Kiran_Raj_RA]

i am also facing the same issue when using cluster

Uncaught (in promise) Error: roomController.subscribe failed, reason: Failed in adding output to audio-engine
    at new t (ics.js:1497)
    at t.value (ics.js:1176)
    at t.value (ics.js:834)
    at O (ics.js:1311)
    at ics.js:1384
    at ics.js:148
    at Array.map (<anonymous>)
    at t.dispatchEvent (ics.js:147)
    at r.<anonymous> (ics.js:1719)
    at r.emit (bundle.js?t=3543453:8916)
    at r.onevent (bundle.js?t=3543453:8916)
    at r.onpacket (bundle.js?t=3543453:8916)
    at r.<anonymous> (bundle.js?t=3543453:8916)
    at r.emit (bundle.js?t=3543453:8916)
    at r.ondecoded (bundle.js?t=3543453:8916)
    at s.<anonymous> (bundle.js?t=3543453:8916)

[Nithin_P_]

Hi,

            I have tested again , about 175 to 180 users this issue happening. during this range the cpu utlization of the server which running audio agent become very high.I think this issue is not present in previous versions. may be this is due to the new update - audio mixing feature 

Thank you,

[Nithin_P_]

Hi,

This issue happening freequently. when we restart audio agent then the issue resolved and new users can subscribe 

[Kiran_Raj_RA]

2018-05-04 13:09:08,569  - WARN: mcu.media.AcmmFrameMixer - No free Id, max participants reached(128)!
2018-05-04 13:09:08,569  - WARN: mcu.media.AcmmFrameMixer - Can not add output participant

[Nithin_P_]

Hi,

            Again tested 

On  128 th subscriber following errors logged in intel 4.0 server

Error logged In Audio agent

WARN: mcu.media.AcmmFrameMixer - No free Id, max participants reached(128)!
WARN: mcu.media.AcmmFrameMixer - Can not add output participant

Error logged In Conference agent

2018-05-04 15:08:58.546  - INFO: Conference - roomController.subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:08:58.546  - INFO: Conference - Exception: roomController.subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:09:00.144  - ERROR: RoomController - subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:09:00.144  - INFO: Conference - roomController.subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:09:00.144  - INFO: Conference - Exception: roomController.subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:09:00.790  - ERROR: RoomController - subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:09:00.790  - INFO: Conference - roomController.subscribe failed, reason: Failed in adding output to audio-engine
2018-05-04 15:09:00.791  - INFO: Conference - Exception: roomController.subscribe failed, reason: Failed in adding output to audio-engine

Error logged  browser console

Uncaught (in promise) Error: roomController.subscribe failed, reason: Failed in adding output to audio-engine
    at new t (ics.js:1497)
    at t.value (ics.js:1176)
    at t.value (ics.js:834)
    at O (ics.js:1311)
    at ics.js:1384
    at ics.js:148
    at Array.map (<anonymous>)
    at t.dispatchEvent (ics.js:147)
    at r.<anonymous> (ics.js:1719)
    at r.emit (bundle.js?t=3543453:8916)
    at r.onevent (bundle.js?t=3543453:8916)
    at r.onpacket (bundle.js?t=3543453:8916)
    at r.<anonymous> (bundle.js?t=3543453:8916)
    at r.emit (bundle.js?t=3543453:8916)
    at r.ondecoded (bundle.js?t=3543453:8916)
    at s.<anonymous> (bundle.js?t=3543453:8916)

Final result is no subscribers after 128 th user can subscribe stream from the room.existing subscribers have no issues

intel webrtc version : 4.0

server : centos 7.2

browser : chrome latest version

Thank you,

[Qiujiao_W_Intel]

Thanks Nithin and Kiran for pointing this issue, we can reproduce it locally and we are working on this issue, will get back to you when it's fixed.

[Kiran_Raj_RA]

Hi Qiujiao W,

Thanks for the reply, We deployed conference server cluster with the below cluster strategy

we have 2 webrtc,audio,video agents  with cluster strategy least-used 

1. In webrtc agent the load is eventually distributing and least-used working perfect.
2. In audio and video agent least-used not working and the load is coming to a single server only.

[Qiujiao_W_Intel]

Hi, Kiran, we cannot reproduce least-used policy not working for audio/video agent issue, could you provide more detail info on your scenario and testing environment? and is there any error log in MCU logs?

[Nithin_P_]

Hi Qiujiao,

                 When we can expect the fix. we are absolutely stuck. All our events have min 1000 to 2000 subscribers so that we cant upgrade the version. 128 user limit is a very small figure for a webinar application.

Thank you,

[Qiujiao_W_Intel]

Nithin, we are working on this issue now and will get back to you ASAP. BTW, did you successfully subscribe 1000 to 2000 mix stream with previous version?

[Nithin_P_]

HI,

No, We have checked maximum up to 500 subscribers and 3 publishers  in a single room ( mix = true ) on version 3.4.1, 

Limitation we faced in previous versions 

1. due to ports not releasing issue in webrtc agent(webrtc stability issue) 
sometimes the users cant enter into the room so we used 5 to 10 webrtc agents and round-robin 
cluster stratergy for distributing webrtc hits so they can enter in the room(sometimes it fail) in version 3.4.1

2. The major challenge is when some one start a screen share the cpu hits a huge value in webrtc agent because screenshare video is not mixed so each user will subscribe it.

3. audio video sync in mixed stream as well as recorded video is another issue in 

Currently we are in beta version. when move to live there will be 1000 to 2000 participant subscribe mixed stream ( based on the current flash version running on live ). but publisher count will be three or four.

[Nithin_P_]

Hi Qiujiao,

Did you fix the issue. Do we need to wait for next version ?

[Qiujiao_W_Intel]

Yes, we will have a minor version release which will fix this issue soon.

[Kiran_Raj_RA]

Hi, Qiujiao W,

When can we get the next release that fix 128 user issue, or is it possible to provide any other solution to add more users.

[Kiran_Raj_RA]

Hi, Qiujiao W,

We have done alive event with 5 publishers (audio only) mixed mode and 150 subscribers (mcu v3.4.1). the audio feels like slow motion and missing, so what will be if we use 4.0 is the same slow we feel or is it work better. 

[Nithin_P_]

Hi Qiujiao,

Any fix on 128 maximum user limit issue ?

[Qiujiao_W_Intel]

Hi, Nithin, we will have a 4.0.1 release early next week which will fix 128 limit issue, stay tuned

[Qiujiao_W_Intel]

HI, Kiran, audio missing issue in your scenario will also be fixed in 4.0.1.

[Qiujiao_W_Intel]

Hi, Kiran and Nithin, our 4.0.1 package is available now, please have a try.

[Qiujiao_W_Intel]

Please use 'viewer' role for subscribe-only user so that they will not trigger any publish related pipeline