About agent behavior in redundant configuration of EMA server

[SKubo]

Hello,

When we have two EMA servers in a redundant configuration, and one of them is stopped and an instruction is sent to the agent, the agent does not respond.
I assume that the agent is probably communicating with the stopped EMA server and automatically switches to the other server, but what is the trigger for switching to the other server when the EMA server stops? Is it?
Will it switch automatically if we wait?
If it does switch automatically, how long will it take?

Best regards

[Suneesh]

Hello Satoshi,

Good day.

Please share us your environment details for us to troubleshoot further on the issue.

• OS version of the Server:
• SQL version:
• Location of both: (physical, virtual)
• Will they be on the same server machine?
• Authentication mode: Local, Azure AD, or Windows AD
• Intel® EMA software version:
• Number of endpoints to be provisioned:
• Will remote access to BIOS be necessary?
• Location of endpoints: local or remote.
• In which mode you are using EMA (ACM or CCM)

Also please share the us the EMA logs from the Server

Default Path:[System Drive]\Program File(x86)\Intel\Platform Manager\EmaLogs

Please send me the files without the date called:

EMAlog-Webserver.txt

EMAlog-Swarmserver.txt

EMAlog-Ajaxserver.txt

EMAlog-Recoveryserver.txt

EMAlog-Manageabilityserver.txt

EMA log from the endpoint:

Default Path:[System drive]\Program Files\Intel\EMA Agent\EMAagentlog

Looking forward to your response.

Best regards,

Suneesh

[SKubo]

Hello, Suneesh

I will answer the questions you asked below.

・OS version of the Server:
　⇒　Windows Server 2019 Datacenter
・SQL version:
　⇒　Azure SQL
・Location of both: (physical, virtual)
　⇒　virtual（Azure VM）
・Will they be on the same server machine?
　⇒　There were two servers, one was deployed from an Azure template and the other was built manually.
・Authentication mode: Local, Azure AD, or Windows AD
　⇒　AzureAD（EntraID）
　　　…　Command-based migration from Windows AD authentication to Azure AD authentication
・Intel® EMA software version:
　⇒　Intel® EMA v1.12.2.0
・Number of endpoints to be provisioned:
　⇒　1（For verification）
・Will remote access to BIOS be necessary?
　⇒　Yes
・Location of endpoints: local or remote.
　⇒　Remote
・In which mode you are using EMA (ACM or CCM)
　⇒　Both

We will send the log separately.

Regards,

Skubo

[SKubo]

Since I could not attach the files, I will write the contents of each log below.

〇ENA#1
[EMALog-AJAXServer]
2024-08-05 18:25:53.1302|INFO||3800|9|.ctor - MeshAjaxServer.CentralAjaxServer, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, 1
2024-08-05 18:25:53.5052|INFO||3800|9|AttemptServerLaunch - MeshAjaxServer.MainForm, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Server Started, ProcessID=3800, DB Schema=v39862272

[EMALog-ManageabilityServer]
2024-08-05 18:25:53.7395|INFO||5000|8|StartRedirectionSessionManager - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [0] - Information: Created RedirectionSession with throttling rate 0
2024-08-05 18:25:53.9583|INFO||5000|8|Start - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [0] - Message:Starting Manageability Server.
2024-08-05 18:25:53.9583|INFO||5000|8|Start - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, 0

[EMALog-RecoveryServer]
2024-08-05 18:25:50.9895|INFO||3788|7|TestDbConnection - MeshServersCommon.code.DB.ServerCommonDb, EMAServersCommon, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: Information, Database ok
2024-08-05 18:25:52.7708|INFO||3788|7|Start - EmaRecoveryServer.code.EmaCentralRecoveryServer, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - MaxDbConnections: 4
2024-08-05 18:25:52.7864|INFO||3788|7|Start - EmaRecoveryServer.code.EmaCentralRecoveryServer, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, Recovery Server starting.
2024-08-05 18:25:52.9739|INFO||3788|7|StartHttpListener - EmaRecoveryServer.code.EmaCentralRecoveryServer, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Recovery Enabled setting is True. Starting HTTP Listener for Recovery port.
2024-08-05 18:25:52.9739|INFO||3788|7|ServerLaunch - EmaRecoveryServer.code.EmaServerLoader, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Server Started, ProcessID=3788

[EMALog-SwarmServer]
2024-08-05 18:25:53.5520|INFO||5052|9|UpdateAgentStore - MeshServer.AgentVersionControl, EMASwarmServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Pulled Agent: Type=2, Version=11200, Time=7/10/2024 12:28:16 AM, size=4217536
2024-08-05 18:25:53.8177|INFO||5052|9|UpdateAgentStore - MeshServer.AgentVersionControl, EMASwarmServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Pulled Agent: Type=4, Version=11200, Time=7/10/2024 12:28:20 AM, size=4403904
2024-08-05 18:25:54.0677|INFO||5052|9|.ctor - MeshServer.CentralServer, EMASwarmServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, 1

[EMALog-WebServer]
2024-08-05 18:26:00.1302|INFO||5164|1|SetupBackendLogger - MeshWebCore.WebApi.WebApiConfig, EMAWebCore, Version=1.12.2.0, Culture=neutral, PublicKeyToken=null - EVENT: Information, Web API server is starting up.
2024-08-05 18:26:00.7864|INFO||5164|1|Initialize - MeshServersCommon.code.SiteService.WebConfigurationWrapper, EMAServersCommon, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: Information, Loading web config file at C:\inetpub\wwwroot\web.config.

〇ENA#2
[EMALog-AJAXServer]
2024-08-05 18:12:15.9754|INFO||5080|8|.ctor - MeshAjaxServer.CentralAjaxServer, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, 2
2024-08-05 18:12:16.2722|INFO||5080|8|AttemptServerLaunch - MeshAjaxServer.MainForm, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Server Started, ProcessID=5080, DB Schema=v39862272
2024-08-05 18:22:58.5673|INFO||5080|3|.ctor - MeshAjaxServer.AjaxWebSocketSession, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - WEBSocket login: satoshi.kubo.bp@bxoedu.onmicrosoft.com, expires in 59 minutes
2024-08-05 18:23:24.3932|INFO||5080|44|RunInternal - System.Threading.ExecutionContext, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - IIS WebSocket Closed: WsState: CloseReceived
2024-08-05 18:24:33.6900|INFO||5080|42|.ctor - MeshAjaxServer.AjaxWebSocketSession, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - WEBSocket login: satoshi.kubo.bp@bxoedu.onmicrosoft.com, expires in 60 minutes
2024-08-05 18:24:43.9391|INFO||5080|42|RunInternal - System.Threading.ExecutionContext, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - IIS WebSocket Closed: WsState: CloseReceived
2024-08-05 18:24:44.9252|INFO||5080|41|.ctor - MeshAjaxServer.AjaxWebSocketSession, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - WEBSocket login: satoshi.kubo.bp@bxoedu.onmicrosoft.com, expires in 60 minutes
2024-08-05 18:24:54.2218|INFO||5080|41|RunInternal - System.Threading.ExecutionContext, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - IIS WebSocket Closed: WsState: CloseReceived
2024-08-05 18:24:54.8343|ERROR||5080|3|Complete - System.Net.LazyAsyncResult, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - Error:Cannot connect SwarmServer
Exception Level(0):A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.11:8089
StackTrace Level(0): at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at MeshAjaxServer.LocalSwarmServer.OnConnect(IAsyncResult asyncResult)

2024-08-05 18:24:55.7871|INFO||5080|3|.ctor - MeshAjaxServer.AjaxWebSocketSession, EMAAjaxServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - WEBSocket login: satoshi.kubo.bp@bxoedu.onmicrosoft.com, expires in 60 minutes
2024-08-05 18:25:03.4685|INFO||5080|42|RunInternal - System.Threading.ExecutionContext, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - IIS WebSocket Closed: WsState: CloseReceived
2024-08-05 18:25:05.9816|ERROR||5080|3|Complete - System.Net.LazyAsyncResult, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - Error:Cannot connect SwarmServer
Exception Level(0):A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.11:8089
StackTrace Level(0): at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at MeshAjaxServer.LocalSwarmServer.OnConnect(IAsyncResult asyncResult)

2024-08-05 18:25:16.8390|ERROR||5080|3|Complete - System.Net.LazyAsyncResult, System, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089 - Error:Cannot connect SwarmServer
Exception Level(0):A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.0.0.11:8089
StackTrace Level(0): at System.Net.Sockets.Socket.InternalEndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.Sockets.TcpClient.EndConnect(IAsyncResult asyncResult)
at MeshAjaxServer.LocalSwarmServer.OnConnect(IAsyncResult asyncResult)

[EMALog-ManageabilityServer]
2024-08-05 18:12:16.5066|INFO||4836|8|StartRedirectionSessionManager - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [0] - Information: Created RedirectionSession with throttling rate 0
2024-08-05 18:12:16.7097|ERROR||4836|8|Start - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [0] - Manageability Server does not have access to USB-R directory, USB-R feature may not work properly.
2024-08-05 18:12:16.7097|INFO||4836|8|Start - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - [0] - Message:Starting Manageability Server.
2024-08-05 18:12:16.7097|INFO||4836|8|Start - MeshManageabilityServer.CentralManageabilityServer, EMAManageabilityServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, 0

[EMALog-RecoveryServer]
2024-08-05 18:12:14.3191|INFO||5020|8|TestDbConnection - MeshServersCommon.code.DB.ServerCommonDb, EMAServersCommon, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: Information, Database ok
2024-08-05 18:12:15.6785|INFO||5020|8|Start - EmaRecoveryServer.code.EmaCentralRecoveryServer, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - MaxDbConnections: 4
2024-08-05 18:12:15.6941|INFO||5020|8|Start - EmaRecoveryServer.code.EmaCentralRecoveryServer, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, Recovery Server starting.
2024-08-05 18:12:15.9285|INFO||5020|8|StartHttpListener - EmaRecoveryServer.code.EmaCentralRecoveryServer, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Recovery Enabled setting is True. Starting HTTP Listener for Recovery port.
2024-08-05 18:12:15.9285|INFO||5020|8|ServerLaunch - EmaRecoveryServer.code.EmaServerLoader, EmaRecoveryServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Server Started, ProcessID=5020

[EMALog-SwarmServer]
2024-08-05 18:12:16.4129|INFO||4940|9|UpdateAgentStore - MeshServer.AgentVersionControl, EMASwarmServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Pulled Agent: Type=2, Version=11200, Time=7/10/2024 12:28:16 AM, size=4217536
2024-08-05 18:12:16.6629|INFO||4940|9|UpdateAgentStore - MeshServer.AgentVersionControl, EMASwarmServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Pulled Agent: Type=4, Version=11200, Time=7/10/2024 12:28:20 AM, size=4403904
2024-08-05 18:12:16.9597|INFO||4940|9|.ctor - MeshServer.CentralServer, EMASwarmServer, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: ServerStart, 2
2024-08-05 18:18:25.7397|INFO||4940|15|<RunReceivedMessageProcess>b__65_0 - MeshServersCommon.code.TcpStack.MessageManager, EMAServersCommon, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - Received ServerState for server type 1 id 1 with state AGENT_START. Endpoint count is 1. Endpoint id list is 45F94BD07148E074BAFD3F66459F6712D3F86B448656029E63A03AAAED236FFD.

[EMALog-WebServer]
2024-08-05 18:17:01.1608|INFO||7352|1|SetupBackendLogger - MeshWebCore.WebApi.WebApiConfig, EMAWebCore, Version=1.12.2.0, Culture=neutral, PublicKeyToken=null - EVENT: Information, Web API server is starting up.
2024-08-05 18:17:01.4889|INFO||7352|1|Initialize - MeshServersCommon.code.SiteService.WebConfigurationWrapper, EMAServersCommon, Version=1.12.2.0, Culture=neutral, PublicKeyToken=57d11e903ea1ca2c - EVENT: Information, Loading web config file at C:\inetpub\wwwroot\web.config.

[Suneesh]

Hello Satoshi,

Greetings of the day.

Thanks for sharing the logs.

Based on the analysis, please check the following:

1. Log into EMA using Global Admin. Adjust the Endpoint "Agent Update Interval (Seconds)" to a smaller value for faster failover.

2. Review the load balancer settings to determine how quickly a server outage is detected and traffic is redirected to the failover server.

Regards,

Suneesh