Cluster video has a black screen

[Lawrence1]

Hi Team,

Test scenario as below

Server1 : Is running: rabbitmq mongodb management-api cluster-manager portal  webrtc-agent conference-agent audio-agent streaming-agent recording-agent sip-agent management-console sip-portal app

(Comment out video agent start script in start-all.sh)

Server2: Is running: video-agent

(Comment out all other components start scripts except video agent in start-all.sh)

the video agent is configured with the correct rabbit-mq server and cluster ip_address.

I can start all the components correctly on all the nodes. The firewall is turned off and all ports are open

black screen:The problem is that there is no streaming video when running the video agent on Server2.

Logs follow：

Server1：logs/webrtc-xxx

2019-08-07 17:03:38.838  - INFO: WorkingNode - pid: 8548
2019-08-07 17:03:38.841  - INFO: WorkingNode - Connecting to rabbitMQ server...
2019-08-07 17:03:38.855  - INFO: AmqpClient - Connecting to rabbitMQ server OK, options: { host: '172.31.28.159', port: 5672 }
2019-08-07 17:03:38.882  - INFO: InternalConnectionFactory - QUIC is not enabled for internal IO
2019-08-07 17:03:38.892  - INFO: WorkingNode - webrtc-b2b8837850a94ff8ec5c@172.31.28.159_0 as rpc server ready
2019-08-07 17:03:38.895  - INFO: WorkingNode - webrtc-b2b8837850a94ff8ec5c@172.31.28.159_0 as monitor ready
2019-08-07 17:04:16,709  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:04:16,723  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:05:54,351  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:05:54,362  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:21,728  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:21,738  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:22,060  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:22,074  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:48,179  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:48,181  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:48,214  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:48,230  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:48,235  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:48,249  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:52,183  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:52,207  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:52,929  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:52,941  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:54,898  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:54,949  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:56,159  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:56,171  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:56,314  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:06:56,352  - ERROR: owt.RawTransport - Error establishing the TCP connection: Connection refused
2019-08-07 17:07:18.650  - WARN: WorkingNode - Exiting on SIGTERM

Server2: logs/video-xxx

2019-08-07 17:03:47.668  - INFO: WorkingNode - pid: 11337
2019-08-07 17:03:47.671  - INFO: WorkingNode - Connecting to rabbitMQ server...
2019-08-07 17:03:47.682  - INFO: AmqpClient - Connecting to rabbitMQ server OK, options: { host: '172.31.28.159', port: 5672 }
2019-08-07 17:03:47.694  - INFO: InternalConnectionFactory - QUIC is not enabled for internal IO
2019-08-07 17:03:47.704  - INFO: WorkingNode - video-cbd446b5e5aed9c3dbf9@172.31.28.162_0 as rpc server ready
2019-08-07 17:03:47.708  - INFO: WorkingNode - video-cbd446b5e5aed9c3dbf9@172.31.28.162_0 as monitor ready
2019-08-07 17:04:10,629  - INFO: mcu.media.VideoMixer - Init maxInput(16), rootSize(640, 480), bgColor(16, 128, 128)
2019-08-07 17:07:48.494  - WARN: WorkingNode - Exiting on SIGTERM

Thanks

[Li_C_Intel]

Hi, Can Server1 reach Server2 in the network?

[Lawrence1]

Hi,

I turned off the firewall.

Server1 can reach Server2 in the network.

Ping IP:

Ping 172.31.28.162
PING 172.31.28.162 (172.31.28.162) 56 (84) bytes of data.
64 bytes from 172.31.28.162: icmp_seq = 1 ttl = 64 times = 3.08 ms
64 bytes from 172.31.28.162: icmp_seq = 2 ttl = 64 times = 4.16 ms
64 bytes from 172.31.28.162: icmp_seq = 3 ttl = 64 time = 4.71 Ms.

Ping 172.31.28.159
PING 172.31.28.159 (172.31.28.159) 56 (84) bytes of data.
64 bytes from 172.31.28.159: icmp_seq = 1 ttl = 64 times = 0.191 ms
64 bytes from 172.31.28.159: icmp_seq = 2 ttl = 64 times = 0.257 ms
64 bytes from 172.31.28.159: icmp_seq = 3 ttl = 64 time = 0.175
Ms 64 bytes from 172.31.28.159: icmp_seq = 4 ttl = 64 times = 0.198 ms

The cluster configuration is as follows

[Lawrence1]

The cluster configuration is as follows.

Is there an error in the configuration?

Server1:

/management_api/management_api.toml

[server]
keystorePath = "./cert/certificate.pfx"
superserviceID = "5d41639bdf3e3812bae053ba"
ssl = true #if use HTTPS
port = 3000 #default: 3000, port of server
numberOfProcess = 4 #default: 1

[cluster]
name = "owt-cluster"

[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[mongo]
dataBaseURL = "172.31.28.159/owtdb" #default: "localhost/owtdb"

/cluster_manager/cluster_manager.toml

[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[manager]
name = "owt-cluster"

#The time for cluster manager getting ready to handle the incoming 'schedule' requirements.
initial_time = 6000 #ms

#The interval of checking the attendences of all cluster workers.
check_alive_interval = 1000 #ms

#The threshold count of consecutive absences after which the corresponding cluster workers will be deleted.
check_alive_count = 3

#The default reservation time if not specified in the scheduling requirement.
schedule_reserve_time = 60000 #ms

[strategy]
general = "last-used"
portal = "last-used"
conference = "last-used"
webrtc = "last-used"
sip = "round-robin"
streaming = "round-robin"
recording = "randomly-pick"
audio = "most-used"
video = "least-used"
analytics = "least-used"
 

/portal/portal.toml

[portal]
keystorePath = "./cert/certificate.pfx"
hostname = "" #default: ""
ip_address = "47.52.141.92" #default: ""
# Port that the socket.io server listens at.
port = 8080 #default: 8080
ssl = true #default: true
force_tls_v12 = false #default: false, tls1 and tls1.1 will not be allowed if set to true
#The network inferface the socket.io will be bound to in case portal.ip_address (the 'ip_address' item above) is not specified or equal to "".
# networkInterface = "eth1"
# How many seconds before Socket.IO client sending a new ping packet. This value might be overwrote by client side.
# ping_interval = 25 #default: undefined
# How many seconds without a pong packet to consider the Socket.IO connection closed. This value might be overwrote by client side.
# ping_timeout = 60 #default: undefined
# Life time for reconnection ticket. Unit: second.
reconnection_ticket_lifetime = 600 #default: 600
# Client will leave conference if it cannot reconnect to Socket.IO server after |reconnection_timeout| seconds.
reconnection_timeout = 60 #default: 60

[cluster]
name = "owt-cluster"

#The number of times to retry joining if the first try fails.
join_retry = 60 #default: 60

#The interval of reporting the work load
report_load_interval = 1000 #default: 1000, unit: millisecond

#The max CPU load under which this worker can taks new tasks.
max_load = 0.85 #default: 0.85

[capacity]
#The ISP list this portal is able to handle
#If the ISP list is set to be non-empty, only the creating token requests with preference.ips being matched with one element of this list will be scheduled to this portal.
isps = [] #default: [], which means all ISPs.
#The region list this portal prefers to handle
#If the region list is set to be non-empty, the creating token requests with preference.region being matched with one element of this list will be priorly scheduled to this portal.
regions = [] #default: [], which means all regions.

[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[mongo]
dataBaseURL = "172.31.28.159/owtdb" #default: "localhost/owtdb"
 

/webrtc_agent/agent.toml

[agent]
#Max processes that agent can run
maxProcesses = 13 #default: 13

#Number of precesses that agent runs when it starts. 1 <= prerunProcesses <= maxProcesses.
prerunProcesses = 2 #default: 2

[cluster]
name = "owt-cluster"

#The number of times to retry joining if the first try fails.
join_retry = 60 #default: 60

#The interval of reporting the work load
report_load_interval = 1000 #default: 1000, unit: millisecond

#The max network load under which this worker can take new tasks.
max_load = 0.85 #default: 0.85

#The bandwidth of network-interface used for WebRTC peerconnections. 
network_max_scale = 1000 #unit: Mbps
[capacity]
#The ISP list this agent is able to handle.
#If the ISP list is set to be non-empty, only the creating token requests with preference.ips being matched with one element of this list will be scheduled to this agent.
isps = [] #default: [], which means all ISPs.
#The region list this agent prefers to handle
#If the region list is set to be non-empty, the creating token requests with preference.region being matched with one element of this list will be priorly scheduled to this agent.
regions = [] #default: [], which means all regions.
[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[internal]
#The IP address used for internal-cluster media spreading. Will use the IP got from the 'network_interface' item if 'ip_address' is not specified or equal to "".
ip_address = "172.31.28.159" #default: ""

#The network interface used for internal-cluster media spreading. The first enumerated network interface in the system will be adopted if this item is not specified.
network_interface = "eth0" # default: undefined

# The internal listening port range, only works for TCP now
maxport = 20000 #default: 0
minport = 10000 #default: 0

#########################################################################################
[webrtc]
#The network inferface all peer-connections will be established through. All network interfaces in the system will be adopted if this item is not specified or specified with an empty array.
##Valid value is an array of objects that has two properties: name and replaced_ip_address. name is the name of network interface that will be used, replaced_ip_address is the IP address used for replacing the internal IP address in locally generated SDP's and ICE candidates during establishing the peer-connection(useful when behind NATs). replaced_ip_address is optional, if it is not specified, IP address from corresponding network interface will not be replaced.
#Example of valid value: [{name = "eth1"}, {name = "eth2", replaced_ip_address = "192.0.2.2"}].
network_interfaces = [{name = "eth0", replaced_ip_address = "47.52.141.92"}]  # default: []

keystorePath = "./cert/certificate.pfx"
#note, this won't work with all versions of libnice. With 0 all the available ports are used

# The webrtc port range
maxport = 9000 #default: 0
minport = 2000 #default: 0

#STUN server IP address and port to be used by the server.
#if "" is used, the address is discovered locally
stunport = 0 #default: 0
stunserver = "" #default: ""

#ThreadPool worker numbers for peer connection
num_workers = 24 #default: 24

/conference_agent/agent.toml

[agent]
#Max processes that agent can run
maxProcesses = -1 #default: -1, unlimited

#Number of precesses that agent runs when it starts. 1 <= prerunProcesses <= maxProcesses.
prerunProcesses = 2 #default: 2

#Roles defined in services have been moved to database

[cluster]
name = "owt-cluster"

#The number of times to retry joining if the first try fails.
join_retry = 60 #default: 60

#The interval of reporting the work load
report_load_interval = 1000 #default: 1000, unit: millisecond

#The max CPU load under which this worker can take new tasks.
max_load = 0.85 #default: 0.85
[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[mongo]
dataBaseURL = "172.31.28.159/owtdb" #default: "localhost/owtdb"

[internal]
# tcp/sctp available, tcp is default
protocol = "tcp"

/audio_agent/agent.toml

[agent]
#Max processes that agent can run
maxProcesses = -1 #default: -1, unlimited

#Number of precesses that agent runs when it starts. 1 <= prerunProcesses <= maxProcesses.
prerunProcesses = 2 #default: 2

[cluster]
name = "owt-cluster"

#The number of times to retry joining if the first try fails.
join_retry = 60 #default: 60

#The interval of reporting the work load
report_load_interval = 1000 #default: 1000, unit: millisecond

#The max CPU load under which this worker can take new tasks.
max_load = 0.85 #default: 0.85

[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[internal]
#The IP address used for internal-cluster media spreading. Will use the IP got from the 'network_interface' item if 'ip_address' is not specified or equal to "".
ip_address = "172.31.28.159" #default: ""

#The network interface used for internal-cluster media spreading. The first enumerated network interface in the system will be adopted if this item is not specified.
network_interface = "eth0" # default: undefined

# The internal listening port range, only works for TCP now
maxport = 20000 #default: 0
minport = 10000 #default: 0
 

Server2

/video_agent/agent.toml

[agent]
#Max processes that agent can run
maxProcesses = -1 #default: -1, unlimited

#Number of precesses that agent runs when it starts. 1 <= prerunProcesses <= maxProcesses.
prerunProcesses = 2 #default: 2

[cluster]
name = "owt-cluster"

#The number of times to retry joining if the first try fails.
join_retry = 60 #default: 60

#The interval of reporting the work load
report_load_interval = 1000 #default: 1000, unit: millisecond

#The max CPU/GPU load under which this worker can take new tasks.
max_load = 0.85 #default: 0.85

[rabbit]
host = "172.31.28.159" #default: "localhost"
port = 5672 #default: 5672

[internal]
#The IP address used for internal-cluster media spreading. Will use the IP got from the 'network_interface' item if 'ip_address' is not specified or equal to "".
ip_address = "172.31.28.159" #default: ""

#The network interface used for internal-cluster media spreading. The first enumerated network interface in the system will be adopted if this item is not specified.
network_interface = "eth0" # default: undefined

# The internal listening port range, only works for TCP now
maxport = 20000 #default: 0
minport = 10000 #default: 0

#########################################################################################
[video]
#If true and the machine has the capability, the mixer will be accelerated by hardware graphic chips
hardwareAccelerated = false

#If true and hardwareAccleration is enabled, setting this to true allows GACC HEVC encoder plugin to be used
#for better video quality.
#Warning: enabling this will introduce extra CPU resource consumption.
enableBetterHEVCQuality = false

#Multi Frame Encoding
#timeout[0, 100] in millisecond, setting to "0" disables this feature
MFE_timeout = 0 #default: 0

[avatar]
#widthxheight between the two dot ("180x180" between the "avatar." and ".yuv" in the default) in the location indicates the image size
location = "avatars/avatar_blue.180x180.yuv"
 

[Lawrence1]

Hello Team,

Any update would be appreciated.

Thank you.
 

[YanbinZ_Intel]

We can't reproduce this issue .  Which version do you used ?  And could your please provide more detail information e.g test devices, test steps ?

Thanks

Yanbin

[Naresh_R_1]

Hi Team,

We have same issue in cluster servers. Mix stream subcribe stream is black. but forward stream always work.Any resolution ?