1 2 Previous Next 22 Replies Latest reply on Nov 5, 2012 2:35 AM by jskovjyskebankdk

    Alpha2 sporadic deployment problems

    jskovjyskebankdk

      We see sporadic deployment problems with Alpha2 on JunoSR1 (on XP), deploying to EAP5.1.1.

       

      Nothing in the logs that appears related. Deployments just stop working.

       

       

      We'll be reverting to Alpha1 where the problem did not appear.

      (we only used Alpha1 shortly, so switching to use it may just reveal it to also be broken.)

       

      I'll post more information when I get some feedback from the developers.

        • 1. Re: Alpha2 sporadic deployment problems
          maxandersen

          ...are these different from the cases you see at https://issues.jboss.org/browse/JBIDE-11215  ?

          • 2. Re: Alpha2 sporadic deployment problems
            jskovjyskebankdk

            I think so. There is no output in the log at all.

             

            The workaround for JBIDE-11215 (setting 'unauthenticatedIdentity') is still in use here, so *that* issue may still be present in Alpha1.

             

             

            I just checked with the developers, and they also see the problem with Alpha1 But not as prone to happen as with Alpha2.

            Restarting the server twice fixes the problem...


            • 3. Re: Alpha2 sporadic deployment problems
              maxandersen

              ...im....what?

               

              There must be something going on here we are missing - something your setup is super-good at triggering...

               

              Any chance we can get to chat with those developers/see them demo it/interact with their setup somehow to digg deeper ?

              • 4. Re: Alpha2 sporadic deployment problems
                jskovjyskebankdk

                We're looking at a Windows7 upgrade in the next couple of months. And we sort of hope that this will solve the problem....

                 

                I don't think we'll want to waste more time looking for ghosts until after that switch; we know that our platform is fairly old and sucky.

                 

                We'll keep the majority of the developers on Eclipse 3.7sr1 + JbossTools 3.3.0M4 until we know if Windows7 will fix the problems with later releases.

                The few developers using Alpha1 are OK with limping along until the switch.

                 

                If the problem persists after the Windows7 switch, I'll come running back for support

                • 5. Re: Alpha2 sporadic deployment problems
                  maxandersen

                  well, windowx XP is the second biggest platform for devs so it is important and I dont expect to windows 7 magically fix it either.

                  • 6. Re: Alpha2 sporadic deployment problems
                    jskovjyskebankdk

                    I hear you.

                     

                    But we're not just running XP. We're running XP loaded with all sorts of old crappy applications, banking infrastructure and whatnot - which will mostly disappear in the switch to Windows7.

                     

                    Given the trouble with reproducing the other issue elsewhere, I think it reasonable to suspect (at least in part) our XP+crud platform as the culprit.

                     

                    And since we're leaving it soon, I'd rather spend time looking for the problem on a leaner Windows7 platform, should it persist.

                    • 7. Re: Alpha2 sporadic deployment problems
                      jskovjyskebankdk

                      I've been granted early access to the Windows7 platform in something like 2 weeks.

                       

                      I'll focus 100% on the problem then.

                      • 8. Re: Alpha2 sporadic deployment problems
                        maxandersen

                        cool!

                        • 9. Re: Alpha2 sporadic deployment problems
                          jskovjyskebankdk

                          We just noticed something; when deployment fails, it is because the VFS entry for Eclipse is deleted in JBoss.

                           

                          When things are working, I have an entry like this:

                           

                          file:/D:/udvikler/ws/dev2/.metadata/.plugins/org.jboss.ide.eclipse.as.core/JBoss_5.1_EAP_Runtime_Server1350643918395/deploy

                           

                           

                          It disappears after some time. Not sure what triggers this, but I've started poking it with server restarts and full deployments to see if I can trigger it.

                           

                          Is it possible for us to enable some debugging that will show a trace of when this happens - so we can find the (likely) cause?

                          At present, the developers have no hints in the log about this event.

                           

                          I'm trying this on Juno SR1 with JBossTools Alpha2 release. As noted in another thread, we only have a partial JBossTools installation, namely the features: org.jboss.ide.eclipse.as.feature, org.jboss.tools.cdi.feature, org.jboss.tools.jsf.feature, org.jboss.tools.ws.feature, org.hibernate.eclipse.feature

                          • 10. Re: Alpha2 sporadic deployment problems
                            jskovjyskebankdk

                            I think we may have some clues. There are two issues - I'll post them separately.

                             

                             

                            The first is a crash in your project scanner - see below.

                             

                            The folder mentioned is always added to our project's deployment assembly list.

                            But the folder only actually exists if the project has certain dependencies (deployable artifacts from IVY dependencies).

                             

                            Everything else in eclipse appears to cope fine with this, but the JBossTools scanner bombs.

                             

                            I wonder if this terminated thread may be somehow releated to deployment problem?

                             

                            I've taken steps to ensure that the folder always exists, but I think JBossTools should probably ignore the folder if non-existant instead of exploding.

                             

                            I'll let you know if it makes a difference.

                             

                            FAOD: I had seen this log entry   I'm not that blind, but I attributed it to the KB scanner and unreleated (which it may still be, of course).

                             

                             

                            !ENTRY org.jboss.ide.eclipse.archives.core 2 0 2012-10-18 10:41:55.012
                             !MESSAGE basedir D:\udvikler\ws\ws2\test\build\gen-lib-deployable is not a directory
                             !STACK 0
                             java.lang.IllegalStateException: basedir D:\udvikler\ws\ws2\test\build\gen-lib-deployable is not a directory
                                 at org.jboss.ide.eclipse.archives.core.asf.DirectoryScanner.scanPrepare(DirectoryScanner.java:552)
                                 at org.jboss.ide.eclipse.archives.core.asf.DirectoryScanner.scan(DirectoryScanner.java:588)
                                 at org.jboss.ide.eclipse.archives.core.model.DirectoryScannerFactory.createDirectoryScanner(DirectoryScannerFactory.java:66)
                                 at org.jboss.ide.eclipse.archives.webtools.filesets.vcf.WorkspaceFilesetVirtualComponent$WorkspaceFilter.<init>(WorkspaceFilesetVirtualComponent.java:100)
                                 at org.jboss.ide.eclipse.archives.webtools.filesets.vcf.WorkspaceFilesetVirtualComponent.getRootFolder(WorkspaceFilesetVirtualComponent.java:53)
                                 at org.eclipse.wst.common.componentcore.internal.flat.FlatVirtualComponent.consumeComponent(FlatVirtualComponent.java:203)
                                 at org.eclipse.wst.common.componentcore.internal.flat.FlatVirtualComponent.addConsumedReferences(FlatVirtualComponent.java:196)
                                 at org.eclipse.wst.common.componentcore.internal.flat.FlatVirtualComponent.treeWalk(FlatVirtualComponent.java:167)
                                 at org.eclipse.wst.common.componentcore.internal.flat.FlatVirtualComponent.cacheResources(FlatVirtualComponent.java:121)
                                 at org.eclipse.wst.common.componentcore.internal.flat.FlatVirtualComponent.getChildModules(FlatVirtualComponent.java:111)
                                 at org.eclipse.wst.web.internal.deployables.FlatComponentDeployable.getExportModelChildren(FlatComponentDeployable.java:252)
                                 at org.eclipse.wst.web.internal.deployables.FlatComponentDeployable.getModules(FlatComponentDeployable.java:263)
                                 at org.jboss.ide.eclipse.as.wtp.core.util.ServerModelUtilities.getChildModules(ServerModelUtilities.java:82)
                                 at org.jboss.ide.eclipse.as.wtp.core.util.ServerModelUtilities.getChildModules(ServerModelUtilities.java:74)
                                 at org.jboss.ide.eclipse.as.core.server.internal.DeployableServer.getChildModules(DeployableServer.java:82)
                                 at org.eclipse.wst.server.core.internal.Server.getChildModules(Server.java:2573)
                                 at org.eclipse.wst.server.core.internal.Server.visitModule(Server.java:2994)
                                 at org.eclipse.wst.server.core.internal.Server.isModuleDeployed(Server.java:942)
                                 at org.eclipse.wst.server.core.internal.Server.handleModuleProjectChange(Server.java:894)
                                 at org.eclipse.wst.server.core.internal.ResourceManager.publishHandleProjectChange(ResourceManager.java:1093)
                                 at org.eclipse.wst.server.core.internal.ResourceManager$ServerResourceChangeListener$1.visit(ResourceManager.java:125)
                                 at org.eclipse.core.internal.events.ResourceDelta.accept(ResourceDelta.java:69)
                                 at org.eclipse.core.internal.events.ResourceDelta.accept(ResourceDelta.java:80)
                                 at org.eclipse.core.internal.events.ResourceDelta.accept(ResourceDelta.java:49)
                                 at org.eclipse.wst.server.core.internal.ResourceManager$ServerResourceChangeListener.resourceChanged(ResourceManager.java:119)
                                 at org.eclipse.core.internal.events.NotificationManager$1.run(NotificationManager.java:291)
                                 at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42)
                                 at org.eclipse.core.internal.events.NotificationManager.notify(NotificationManager.java:285)
                                 at org.eclipse.core.internal.events.NotificationManager.broadcastChanges(NotificationManager.java:149)
                                 at org.eclipse.core.internal.resources.Workspace.broadcastBuildEvent(Workspace.java:381)
                                 at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:146)
                                 at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:241)
                                 at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53)
                               
                            
                            • 11. Re: Alpha2 sporadic deployment problems
                              maxandersen

                              Not sure about debug options for EAP on this - but if this is a problem caused by external deployment have you tried enabling deployment *inside* EAP instead ?

                               

                              You can configure this under the Deployment tab of the server.

                              • 12. Re: Alpha2 sporadic deployment problems
                                jskovjyskebankdk

                                The other issue is related to the start/stop of the server.

                                 

                                We had problems starting EAP511 with the JBossTools4 defaults, so we changed (both) state detectors to JMX.

                                 

                                The server would start fine, but a stop always causes this:

                                 

                                 

                                Exception in thread "main" javax.naming.CommunicationException: Could not obtain connection to any of these urls: 0.0.0.0:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused: connect]]]
                                    at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763)
                                    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
                                    at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
                                    at javax.naming.InitialContext.lookup(Unknown Source)
                                    at org.jboss.Shutdown.main(Shutdown.java:219)
                                Caused by: javax.naming.CommunicationException: Failed to connect to server /0.0.0.0:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused: connect]]
                                    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335)
                                    at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
                                    ... 4 more
                                Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server /0.0.0.0:1099 [Root exception is java.net.ConnectException: Connection refused: connect]
                                    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305)
                                    ... 5 more
                                Caused by: java.net.ConnectException: Connection refused: connect
                                    at java.net.PlainSocketImpl.socketConnect(Native Method)
                                    at java.net.PlainSocketImpl.doConnect(Unknown Source)
                                    at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
                                    at java.net.PlainSocketImpl.connect(Unknown Source)
                                    at java.net.SocksSocketImpl.connect(Unknown Source)
                                    at java.net.Socket.connect(Unknown Source)
                                    at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:97)
                                    at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:82)
                                    at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301)
                                    ... 5 more
                                

                                 

                                 

                                And looking in one of the logs, we saw the below (note third entry):

                                 

                                 

                                !ENTRY org.jboss.ide.eclipse.as.core 4 16973826 2012-10-18 10:44:30.447
                                 !MESSAGE The server was shutdown forcefully. All processes were terminated.
                                
                                !ENTRY org.jboss.ide.eclipse.as.core 4 16842768 2012-10-18 10:44:30.525
                                 !MESSAGE Server Shutdown Failed
                                
                                !ENTRY org.jboss.ide.eclipse.as.core 4 33816577 2012-10-18 10:44:32.447
                                 !MESSAGE Error adding deployment folder to deployment scanner
                                
                                !STACK 0
                                 javax.naming.CommunicationException: Could not obtain connection to any of these urls: localhost:1099 and discovery failed with error: java.lang.ClassCastException: java.lang.Boolean cannot be cast to java.lang.String [Root exception is javax.naming.CommunicationException: Failed to connect to server localhost/127.0.0.1:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server localhost/127.0.0.1:1099 [Root exception is java.net.ConnectException: Connection refused: connect]]]
                                     at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1763)
                                     at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:693)
                                     at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
                                     at javax.naming.InitialContext.lookup(Unknown Source)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JBossServerConnection.createConnection(JBossServerConnection.java:159)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JBossServerConnection.run(JBossServerConnection.java:138)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JBossServerConnection.run(JBossServerConnection.java:122)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JBossServerConnection.run(JBossServerConnection.java:105)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JBossJMXConnectionProviderModel.run(JBossJMXConnectionProviderModel.java:79)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JMXServerLifecycleListener.ensureScannersAdded(JMXServerLifecycleListener.java:65)
                                     at org.jboss.ide.eclipse.as.jmx.integration.JMXServerLifecycleListener.modifyDeploymentScanners(JMXServerLifecycleListener.java:54)
                                     at org.jboss.ide.eclipse.as.core.server.internal.v7.LocalJBoss7DeploymentScannerAdditions$1.run(LocalJBoss7DeploymentScannerAdditions.java:127)
                                     at org.eclipse.core.internal.jobs.Worker.run(Worker.java:53)
                                Caused by: javax.naming.CommunicationException: Failed to connect to server localhost/127.0.0.1:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server localhost/127.0.0.1:1099 [Root exception is java.net.ConnectException: Connection refused: connect]]
                                     at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:335)
                                     at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1734)
                                     ... 12 more
                                
                                Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server localhost/127.0.0.1:1099 [Root exception is java.net.ConnectException: Connection refused: connect]
                                     at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:305)
                                     ... 13 more
                                
                                Caused by: java.net.ConnectException: Connection refused: connect
                                     at java.net.PlainSocketImpl.socketConnect(Native Method)
                                     at java.net.PlainSocketImpl.doConnect(Unknown Source)
                                     at java.net.PlainSocketImpl.connectToAddress(Unknown Source)
                                     at java.net.PlainSocketImpl.connect(Unknown Source)
                                     at java.net.SocksSocketImpl.connect(Unknown Source)
                                     at java.net.Socket.connect(Unknown Source)
                                     at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:97)
                                     at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:82)
                                     at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:301)
                                     ... 13 more
                                
                                

                                 

                                 

                                 

                                So this happened as part of a server restart.

                                 

                                It may not be a full explanation, since some of the other developers don't restart unless they already see deployment problems....

                                 

                                 

                                I've asked them all to switch Shutdown Poller to 'Process Terminated' which I suspect will fix the problem when caused as a side effect of a bad shutdown.

                                • 13. Re: Alpha2 sporadic deployment problems
                                  jskovjyskebankdk

                                  Not sure about debug options for EAP on this - but if this is a problem caused by external deployment have you tried enabling deployment *inside* EAP instead ?

                                   

                                  You can configure this under the Deployment tab of the server.

                                  One of the developers uses this, and he does not have as many problems as the others.

                                   

                                  So there may be something to this.

                                  • 14. Re: Alpha2 sporadic deployment problems
                                    maxandersen

                                    Jesper - these errors or at least the one about scanner being disabled matches what is/was found at https://issues.jboss.org/browse/JBIDE-11215 (verified bug in EAP 5.x family).

                                     

                                    The other two weird errors here is the string/boolean cast - that should not happen and we should get that investigated.

                                     

                                    Unfortunately just starting today AS tooling master is on PTO so it might be a week before we can digg deep.

                                    1 2 Previous Next