All those calls to enable/disable/delete/... basically go
try {
AgentClient agentClient = agentManager.getAgentClient(agent);
ResourceFactoryAgentService resourceFactoryAgentService = agentClient.getResourceFactoryAgentService();
resourceFactoryAgentService.deleteResource(request);
return persistedHistory;
} catch (CannotConnectException e) {
so we try to get the AgentClient, create a remote-POJO and then use the remoted agent service to do the work.
And this is (unless told differently) waiting up to org.rhq.enterprise.communications.command.client.ClientRemotePojoFactory#m_timeoutMillis
I did not yet find out what this value really is, but just trying to open the socket may take some time, that gets worse for group removals where the resources are on different agents.
Perhaps we can shortcut the logic by checking if the agent is currently marked as available or not and don't try to send when it is no up.
Currently when I have shut down the agent "snert" (and the server knows about it ) I see this in the log.
19:18:16,307 INFO [org.rhq.enterprise.server.core.AgentManagerBean] (http-/0.0.0.0:7080-1) Agent with name [snert] just went down
19:18:40,443 ERROR [org.rhq.enterprise.communications.command.client.ClientCommandSenderTask] (http-/0.0.0.0:7080-5) {ClientCommandSenderTask.send-failed}Failed to send command [Command: type=[remotepojo]; cmd-in-response=[false]; config=[{rhq.send-throttle=true}]; params=[{invocation=NameBasedInvocation[uninventoryResource], targetInterfaceName=org.rhq.core.clientapi.agent.discovery.DiscoveryAgentService}]]. Cause: org.jboss.remoting.CannotConnectException:Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://172.31.7.7:16163/?backlog=200&clientMaxPoolSize=304&enableTcpNoDelay=true&maxPoolSize=303&numAcceptThreads=1&rhq.communications.connector.rhqtype=agent&socketTimeout=60000] -> java.net.ConnectException:Connection refused. Cause: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://172.31.7.7:16163/?backlog=200&clientMaxPoolSize=304&enableTcpNoDelay=true&maxPoolSize=303&numAcceptThreads=1&rhq.communications.connector.rhqtype=agent&socketTimeout=60000]
19:18:40,444 WARN [org.rhq.enterprise.server.resource.ResourceManagerBean] (http-/0.0.0.0:7080-5) Unable to inform agent of inventory removal for resource [10019]: org.jboss.remoting.CannotConnectException: Can not get connection to server. Problem establishing socket connection for InvokerLocator [socket://172.31.7.7:16163/?backlog=200&clientMaxPoolSize=304&enableTcpNoDelay=true&maxPoolSize=303&numAcceptThreads=1&rhq.communications.connector.rhqtype=agent&socketTimeout=60000]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:838) [jboss-remoting-2.5.4.SP4.jar:]
at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:169) [jboss-remoting-2.5.4.SP4.jar:]
at org.jboss.remoting.Client.invoke(Client.java:2084) [jboss-remoting-2.5.4.SP4.jar:]
at org.jboss.remoting.Client.invoke(Client.java:879) [jboss-remoting-2.5.4.SP4.jar:]
at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.rawSend(JBossRemotingRemoteCommunicator.java:514) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.sendWithoutCallbacks(JBossRemotingRemoteCommunicator.java:456) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.sendWithoutInitializeCallback(JBossRemotingRemoteCommunicator.java:475) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.JBossRemotingRemoteCommunicator.send(JBossRemotingRemoteCommunicator.java:496) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.AbstractCommandClient.invoke(AbstractCommandClient.java:143) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.ClientCommandSender.send(ClientCommandSender.java:1091) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.send(ClientCommandSenderTask.java:229) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:107) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at org.rhq.enterprise.communications.command.client.ClientCommandSenderTask.call(ClientCommandSenderTask.java:55) [rhq-enterprise-comm-4.9.0-SNAPSHOT.jar:4.9.0-SNAPSHOT]
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) [rt.jar:1.7.0_25]
at java.util.concurrent.FutureTask.run(FutureTask.java:166) [rt.jar:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_25]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_25]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_25]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_25]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_25]
at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_25]
at org.jboss.remoting.transport.socket.SocketClientInvoker$1.run(SocketClientInvoker.java:303) [jboss-remoting-2.5.4.SP4.jar:]
at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.7.0_25]
at org.jboss.remoting.transport.socket.SocketClientInvoker.connect(SocketClientInvoker.java:299) [jboss-remoting-2.5.4.SP4.jar:]
at org.jboss.remoting.transport.socket.SocketClientInvoker.createSocket(SocketClientInvoker.java:210) [jboss-remoting-2.5.4.SP4.jar:]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.getConnection(MicroSocketClientInvoker.java:1172) [jboss-remoting-2.5.4.SP4.jar:]
at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:816) [jboss-remoting-2.5.4.SP4.jar:]
... 17 more