10 Replies Latest reply on Aug 10, 2012 6:19 PM by fecsedy

    Using EJBs as agents

    fecsedy

      Our target platform is AS 7.1.1.Final and we are trying to do the following:

       

      We have multiple homogeneous systems across our network. Each system has a standalone 7.1.1.Final server on it that can perform command and control on that server only. We would like all of our systems to run identical war files with each system being differentiated by a custom standalone.xml file. Each system can share its current state with the other systems over JMS. A user can control the system via a UI on the server located closest to them geographically. I want to use the "agent" concept to have commands performed at the server that can take the action. It may be the same server that the user is connected to but it may not. So, basically it involves server to server remote invocation of stateless EJBs. I have read http://docs.jboss.org/author/display/AS71/EJB+invocations+from+a+remote+server+instance and tried that approach. The answer is always "No EJB receiver available for handling ...". I have tried every combination of app name, module name, distinct name with the same result. I can confirm that the servers make remote connections over 4447 to each other but the "ejb:" lookup does not work. Plus, I have the added problem of needing to use local-outbound-connection for connecting to myself and there does not seem to be any examples floating around. And the fact that 7.1.1.Final does not allow multiple remote-outbound-connections without blowing up (fixed in 7.1.3, supposedly). So, that leaves the following:

       

          Properties jndiProperties = new Properties();
          jndiProperties.put("jboss.naming.client.ejb.context", true);
          jndiProperties.put(InitialContext.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");
          jndiProperties.put(InitialContext.PROVIDER_URL, remoteSystemUrl);
          jndiProperties.put(InitialContext.SECURITY_PRINCIPAL,username);
          jndiProperties.put(InitialContext.SECURITY_CREDENTIALS,password);
          InitialContext context = new InitialContext(jndiProperties);

       

      Sadly, this tries to do its duty but:

       

      javax.naming.NamingException: Failed to create remoting connection [Root exception is java.lang.RuntimeException: Failed to setup EJB remote context]

      ...

      Caused by: java.lang.LinkageError: loader constraint violation: loader (instance of org/jboss/modules/ModuleClassLoader) previously initiated loading for a different type with name "org/jboss/remoting3/Connection"

       

      which I can sort of understand means that there is already a class loaded that implements the org.jboss.remoting3.Connection interface and I am trying to load a different implementation. Also, I am not sure this works if the remoteSystemUrl is something like "remote://localhost:4447". For us clustering is not an option. Any ideas on how to make this work?

       

      Thanks.

        • 1. Re: Using EJBs as agents
          jaikiran

          Let's get back to the original issue:

           

           

          Frank Ecsedy wrote:

           

          I have read http://docs.jboss.org/author/display/AS71/EJB+invocations+from+a+remote+server+instance and tried that approach. The answer is always "No EJB receiver available for handling ...". I have tried every combination of app name, module name, distinct name with the same result. I can confirm that the servers make remote connections over 4447 to each other but the "ejb:" lookup does not work.

          Please post the entire details including the relevant jboss-ejb-client.xml and the exception stacktrace and even your lookup code.

           

           

          Frank Ecsedy wrote:

           

          And the fact that 7.1.1.Final does not allow multiple remote-outbound-connections without blowing up (fixed in 7.1.3, supposedly).

          Do you have a link to a JIRA which tells us what this issue is?

          • 2. Re: Using EJBs as agents
            fecsedy

            standalone-server1.xml:

             

            <system-properties>

                <property name="ourhost" value="server1"/>

                <property name="jndi.remote.agent" value="ejb:app/module/%s/Agent!com.example.RemoteAgent"/>

            </system-properties>

             

            <management>
                <security-realms>
                    <security-realm name="ejb-security-realm">
                        <server-identities>
                            <secret value="base64encodedstring"/>
                        </server-identities>
                    </security-realm>
                </security-realms>
            </management>

             

            <subsystem xmlns="urn:jboss:domain:remoting:1.1">
                <connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>
                <outbound-connections>
                    <local-outbound-connection name="server1-remoting" outbound-socket-binding-ref="server1-remoting-socket">
                        <properties>
                            <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
                            <property name="SSL_ENABLED" value="false"/>
                        </properties>
                    </local-outbound-connection>
                    <remote-outbound-connection name="server2-remoting" outbound-socket-binding-ref="server2-remoting-socket" username="username" security-realm="ejb-security-realm">
                        <properties>
                            <property name="SASL_POLICY_NOANONYMOUS" value="false"/>
                            <property name="SSL_ENABLED" value="false"/>
                        </properties>
                   </remote-outbound-connection>

                </outbound-connections>

            </subsystem>

             

            <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
                <outbound-socket-binding name="server1-remoting-socket">
                    <remote-destination host="localhost" port="4447"/>
                </outbound-socket-binding>
                <outbound-socket-binding name="server2-remoting-socket">
                    <remote-destination host="server2" port="4447"/>
                </outbound-socket-binding>
            </socket-binding-group>

             

            standalone-server2.xml is identical to server1's with server1 and server2 reversed. In WAR file:

             

            WEB-INF/jboss-web.xml:

             

            <jboss-web>

                <distinct-name>${ourhost}</distinct-name>

            </jboss-web>

             

            WEB-INF/jboss-ejb-client.xml:

             

            <jboss-ejb-client xmlns="urn:jboss:ejb-client:1.0">

                <client-context>

                    <ejb-receivers>

                        <remoting-ejb-receiver outbound-connection-ref="server1-remoting"/>

                        <remoting-ejb-receiver outbound-connection-ref="server2-remoting"/>

                    </ejb-receivers>

                </client-context>

            </jboss-ejb-client>

             

            Java code:

             

            private RemoteAgent getAgentForSystem(SystemEnum system) throws NamingException
            {

             

                Properties jndiProperties = new Properties();
                jndiProperties.put("jboss.naming.client.ejb.context", true);
                jndiProperties.put(Context.URL_PKG_PREFIXES, "org.jboss.ejb.client.naming");
                InitialContext context = new InitialContext(jndiProperties);

             

                try
                {

             

                    String lookupName = String.format(System.getProperty("jndi.remote.agent"), system.name().toLowerCase());
                    ServerUtilities.getLogger().log(Level.FINEST, String.format("getAgentForSystem: looking up %s", lookupName));
                    return (RemoteAgent)context.lookup(lookupName);

             

                }
                finally { context.close(); }

             

            }

             

            SystemEnum is SERVER1, SERVER2, etc. So the lookup name is "ejb:/app/module/server1/Agent!com.example.RemoteAgent".

             

            server.log:

             

            20:00:51,536 WARNING [com.example.webservices.Service] (http--0.0.0.0-8443-1) getData: java.lang.IllegalStateException: No EJB receiver available for handling [appName:app,

            modulename:module,distinctname:server2] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@12be9fbe

                    at org.jboss.ejb.client.EJBClientContext.requireEJBReceiver(EJBClientContext.java:584) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                    at org.jboss.ejb.client.ReceiverInterceptor.handleInvocation(ReceiverInterceptor.java:119) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                    at org.jboss.ejb.client.EJBClientInvocationContext.sendRequest(EJBClientInvocationContext.java:181) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:136) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                    at org.jboss.ejb.client.EJBInvocationHandler.doInvoke(EJBInvocationHandler.java:121) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                    at org.jboss.ejb.client.EJBInvocationHandler.invoke(EJBInvocationHandler.java:104) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                    at $Proxy135.getData(Unknown Source)  at com.example.webservices.Service.getData(Service.java:635) [BusinessLogic.jar:]

                    at com.example.webservices.Service$Proxy$_$$_WeldClientProxy.getData(Service$Proxy$_$$_WeldClientProxy.java) [BusinessLogic.jar:]

                    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_04]

                    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.7.0_04]

                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [rt.jar:1.7.0_04]

                    at java.lang.reflect.Method.invoke(Unknown Source) [rt.jar:1.7.0_04]

                    at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:155) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.core.ResourceMethod.invokeOnTarget(ResourceMethod.java:257) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:222) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.core.ResourceMethod.invoke(ResourceMethod.java:211) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.core.SynchronousDispatcher.getResponse(SynchronousDispatcher.java:525) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:502) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:119) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:208) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:55) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:50) [resteasy-jaxrs-2.3.2.Final.jar:]

                    at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec-1.0.0.Final.jar:1.0.0.Final]

                    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:329) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

                    at org.jboss.weld.servlet.ConversationPropagationFilter.doFilter(ConversationPropagationFilter.java:62) [weld-core-1.1.5.AS71.Final.jar:2012-02-10 15:31]

                    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

                    at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:259) [isomorphic-core-rpc.jar:]

                    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:280) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:248) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:397) [jbossweb-7.0.13.Final.jar:]

                    at org.jboss.as.jpa.interceptor.WebNonTxEmCloserValve.invoke(WebNonTxEmCloserValve.java:50) [jboss-as-jpa-7.1.1.Final.jar:7.1.1.Final]

                    at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:153) [jboss-as-web-7.1.1.Final.jar:7.1.1.Final]

                    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:155) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:368) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:877) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:671) [jbossweb-7.0.13.Final.jar:]

                    at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:930) [jbossweb-7.0.13.Final.jar:]

                    at java.lang.Thread.run(Unknown Source) [rt.jar:1.7.0_04]

             

            It have tried all 8 combinations of app, module and distinct name with the same result as above. The JIRA issue for multiple remote-outbound-connections is http://issues.jboss.org/browse/AS7-4834

             

            Thanks

            • 3. Re: Using EJBs as agents
              feder

              Hi Frank,

               

              The error message is ambiguous. It appears if a remoting request on jndi cannot be mapped to an EJB receiver. Thus, a wrong jndi resolution (e.g. jboss-ejb-client.properties in your ACC-Client, jboss-ejb-client.xml on your client-server). You know that by know.

               

              Try the ejb lookup without a leading front slash. Thus, ejb:app/module/server1/Agent!com.example.RemoteAgent and NOT ejb:/app/module/server1/Agent!com.example.RemoteAgent

              It never sorted out on my remoting invocations with JBoss AS 7.1.1 using the leading slash.

              • 4. Re: Using EJBs as agents
                fecsedy

                My understanding is that jboss-ejb-client.properties is for stand-alone client apps connecting to a server. I am going server to server so I use the jboss-ejb-client.xml instead.

                If you look at the property shown above: <property name="jndi.remote.agent" value="ejb:app/module/%s/Agent!com.example.RemoteAgent"/> you will see there is no leading slash. The ejb:/app was a typo on my part, sorry about the confusion. So it is ejb:app/module/distinct-name/Agent!com.example.RemoteAgent and its variants.

                 

                Thanks

                • 5. Re: Using EJBs as agents
                  jaikiran

                  Few things that you should check:

                   

                  1) Make sure the jboss-ejb-client.xml is indeed being picked up.

                  2) Is your second server (destination server) up when the first server (client server) deploys your application? We had a bug in 7.1.1.Final where if the destination server wasn't up then it wouldn't try reconnecting to it once it's up.

                  3) Do you see any other error logs on the client server when the application is being deployed?

                  4) Can you try this against the latest nightly build and see if it works there https://community.jboss.org/thread/167590. This will rule out any bugs that were fixed after 7.1.1.Final was released.

                  • 6. Re: Using EJBs as agents
                    fecsedy

                    1) Set org.jboss.as.ee to TRACE. First line of the ee.log file: 16:31:12,967 DEBUG [org.jboss.as.ee.structure.EJBClientDescriptorParsingProcessor] (MSC service thread 1-25) Successfully parsed jboss-ejb-client.xml for deployment unit deployment ...

                    2) Yes.

                    3) No other errors in the log. The EJB invocation exception is the first one.

                    4) I will see what I can do.

                     

                    Is there any way to do the following:

                     

                    1) Get JBoss AS to show you the distinct-name for a deployment.

                    2) Get JBoss AS to dump all of the ejb: URIs that it recognizes.

                     

                    It would be helpful to have a dump of the ejb: nam space directory.

                     

                    Thanks

                    • 7. Re: Using EJBs as agents
                      fecsedy

                      Completed 4). Downloaded jboss-as-7.2.0.Alpha1-SNAPSHOT and ran our app on it. No exception in the server log but the client got this:

                       

                      EJBCLIENT000025: No EJB receiver available for handling [appName:app, moduleName:module, distinctName:server2] combination for invocation context org.jboss.ejb.client.EJBClientInvocationContext@9eb65cd

                       

                      One weird thing: logging to our custom app log does not work. The log file is there but is 0 bytes in length.

                      • 8. Re: Using EJBs as agents
                        fecsedy

                        OK. Since there has been no movement on this issue I have abandoned the the remoting endpoint approach since it does not work. Also, it is not very flexible because it requires a load of static configuration in both the jboss-ejb-client.xml file we place in the war file and the standalone files we place on the servers. So, it's back to using remote-naming, which I do understand is deprecated/unstable/causes acne. Here is the latest failed attempt:

                         

                                InetSocketAddress address = OTHER_SYSTEMS.get(system);

                                String url = String.format("remote://%s:%d", address.getAddress().getHostAddress(), address.getPort());

                               

                                Properties jndiProperties = new Properties();

                                jndiProperties.put("jboss.naming.client.ejb.context", true);

                                jndiProperties.put(InitialContext.INITIAL_CONTEXT_FACTORY, org.jboss.naming.remote.client.InitialContextFactory.class.getName());

                                jndiProperties.put(InitialContext.PROVIDER_URL, url);

                                jndiProperties.put(InitialContext.SECURITY_PRINCIPAL, USERNAME);

                                jndiProperties.put(InitialContext.SECURITY_CREDENTIALS, PASSWORD);

                                InitialContext context = new InitialContext(jndiProperties);

                         

                                try

                                {

                         

                                    String lookupName = System.getProperty("jndi.remote.agent");

                                    ServerUtilities.getLogger().log(Level.FINEST, String.format("Service.getAgentForSystem: looking up %s using url %s", lookupName, url));

                                    return (RemoteAgent)context.lookup(lookupName);

                         

                                }

                                finally { context.close(); }

                         

                        We get:

                         

                        14:37:20,993 INFO  [stdout] (http--0.0.0.0-8443-1) Caused by: java.lang.ClassNotFoundException: org.jboss.naming.remote.client.InitialContextFactory from [Module "____________" from Service Module Loader]

                         

                        The remote-naming module looks like it should be activated via org.jboss.as.naming but I must be missing something. One other question I had: It there a way to use <bindings> in the standalone.xml to create a local binding, java:global/blah/Agent to refer to a remote EJB? I thought I saw something like this when googling and now I can't find it.

                         

                        Thanks

                        • 9. Re: Using EJBs as agents
                          fecsedy

                          We added the following to our jboss-deployment-structure.xml:

                           

                                  <module name="org.jboss.remote-naming" export="true"/>

                           

                          We added the following to our modules/org/jboss/remote-naming/main/module.xml:

                           

                              <module name="org.jboss.ejb-client"/>

                           

                          Now the InitialContextFactory issue is in the past. Our new error is:

                           

                          Caused by: java.lang.SecurityException: EJB client context selector may not be changed

                                  at org.jboss.ejb.client.EJBClientContext.setSelector(EJBClientContext.java:181) [jboss-ejb-client-1.0.5.Final.jar:1.0.5.Final]

                                  at org.jboss.naming.remote.client.ejb.RemoteNamingEjbClientContextSelector.setupSelector(RemoteNamingEjbClientContextSelector.java:18) [jboss-remote-naming-1.0.2.Final.jar:1.0.2.Final]

                                  ... 44 more

                          • 10. Re: Using EJBs as agents
                            fecsedy

                            I decided to go back and try the jboss-ejb-client.xml approach one more time and it worked. The difference was that I deployed an ear file that had the war inside of it rather than just deploying the war on its own. Now looking up ejb:app/war/distinct-name/Agent!... works and routes the RMI invocation to the correct server. I don't know what should be different (other than a lack of a module name) when deploying straight from a war instead of a war in an ear.

                             

                            There is still the problem of a lack of flexibility. Is there a way to create the EJB receivers dynamically instead of through xml?

                             

                            Thanks.