JBoss AS 7.1.0 Final fails to start and server hangs : Due to : ClassCastException: __redirected.__XMLInputFactory cannot be cast to javax.xml.stream.XMLInputFactory

[subashsdm]

Hi,

I am trying to start JBoss 7.1.0 Final version having deployed a web app (which is also work in OSGi environment).

NOTE: My war needs some jars which were earlier put inside endorsed dir in JBoss 5.X . Seems JBoss picking some other XML libraries over mine. For more clarification, following are the jars that I expected the JBoss to pick at runtime. 

geronimo-jaxws_2.2_spec-1.0.jar 

geronimo-stax-api_1.0_spec-1.0.1.wso2v1.jar

xalan-2.7.0.wso2v1.jar

xercesImpl-2.8.1.wso2v2.jar

xml-apis-1.3.04.wso2v1.ja

But for some reason I am getting following error log on JBoss and server hangs there. No error logs in normal mode. But got error logs on debug mode.

Please find the attachment where I have attached the error logs.

It will be great if I can get to know any reason for this issue.

Thanks

[jason.greene]

Are you sure the server hangs, or is it just that your application is not available? (For example does the welcome page display still?)

Looking at your logs there is custom classloading in your application going on here with some thirdparty frameworks you are using (looks like it is launching an osgi container of some kind?). Those classloaders are attempting to override the stax apis which are part of the JDK with the ones in your deployment. Normally we ignore JAXP apis in your deployment but yet honor the implementation (so you can still have your own XML parser). In any case the problem is that the framework is overriding the API yet still honoring the environment's impl, which is tied to the Java APIs (hence CCE). Overriding JAXP apis is usually not a good idea and needs to be done with care, because you will efectively be unable to communicate with anything else in the JVM that takes a reference to a JAXP factory.

I would recommend removing

geronimo-stax-api_1.0_spec-1.0.1.wso2v1.jar

[subashsdm]

Hi Jason,

Yes, the webapp I am using does its custom class loading. I am using JBoss 7.1.0 and there I cant endorse my 5 jars which cause the class cast exceptions on JAXP apis. So what I did was created a JBoss global module

and put the jars inside, which needs to endorsed but it is not picking them.(which means even i didn't create a global module with those jars, this exeptions should occur, because the problem is those jars not get picked ).  Is there a way for me to make those jars endorsed with JBoss 7.1 ? I also tried by ,manually adding those jars inside JAVA_HOME/lib/endorsed and still not picking ? Can you suggest a solution please ?

Thanks

[subashsdm]

FYI, I even couldn't access the JBoss welcome page.(https://localhost:8443/). I am getting a SSL connection error.

Here is what I have changed in my standalone.xml . 

<subsystem xmlns="urn:jboss:domain:web:1.1" native="false" default-virtual-server="default-host">

    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https">

        <ssl name="ssl" password="wso2carbon" certificate-key-file="/home/subash/greg-repo/repository/resources/security/wso2carbon.jks"

              protocol="TLS" verify-client="true" ca-certificate-file="/home/subash/greg-repo/repository/resources/security/wso2carbon.jks"/>

    </connector>

    <virtual-server name="default-host" enable-welcome-root="true">

        <alias name="localhost"/>

        <alias name="example.com"/>

    </virtual-server>

</subsystem>

[subashsdm]

Hi Jason,

I just saw the system properties set by JBoss.

And those causing factors are also there.

javax.xml.parsers.SAXParserFactory=__redirected.__SAXParserFactory

javax.xml.stream.XMLEventFactory=__redirected.__XMLEventFactory

javax.xml.stream.XMLInputFactory=__redirected.__XMLInputFactory

javax.xml.stream.XMLOutputFactory=__redirected.__XMLOutputFactory

So my app fails because of these system properties. Is there a mechanism from JBoss to prevent adding some of these sys properties / or how to remove them manually ?

[lcsn]

HI.

I had a similar problem. The JBoss does not finshed the startup in eclipse because some

background operation were still in progress. After a while eclipse comes up with

an error message telling me there was a timeout.

There is a workaround for the jbossas7.1b which helped me.

https://community.jboss.org/en/tools/blog/2011/11/24/jboss-as-71-beta1-with-jboss-tools--use-web-port-poller

I have the same log entries which Subash Chaturanga described above, but i am having no further problems with these at the moment.

[subashsdm]

Hi Jason Greene,

I hope those couple of comments I made would not make you confuse. I hope you got my problem. Can we change those system parameters(above mentioned) when my web app gets deployed so that

it will not load those javax.xml classes and load the classes which were in my custom module ?

[jason.greene]

Hi Subash,

You can still use endorsed but you need to add the "-Djava.endorsed.dirs=" JVM option and specify a directory which contains your jars. In standalone mode this means editing standalone.conf. Keep in mind that doing so can have unintended consequences on other applications or other areas of the server, so it bares some risk, and certainly requires some testing. In other words "Beware of dragons"

[subashsdm]

Hi Jason Greene,

I created a folder JBOSS_HOME/standalone/lib/endorsed and put those jars and set the above mentioned parameter .Add the following entry in standalone.sh.

JAVA_OPTS="-Djava.endorsed.dirs=$JBOSS_HOME/standalone/lib/endorsed/" . Still I am getting those class cast exceptions

Here is the last error log as same as earlier.

BTW I thought endorsing is completely restricted from JBoss 7 (according to your initial explanation ). If we can do so by this mechanism this should work. Any idea ?

00:12:00,552 DEBUG [org.wso2.carbon.identity.authenticator.token.ui.internal.Activator] (Start Level Event Dispatcher) Started the SSO Authenticator

00:12:00,566 DEBUG [org.wso2.carbon.jaxwsservices.internal.JAXWSServiceComponent] (Component Resolve Thread (Bundle 71)) JAXWS bundle is activated

00:12:00,669 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) About to create XMLInputFactory implementation with classloader=org.eclipse.core.runtime.internal.adaptor.ContextFinder@409a6317

00:12:00,669 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) The classloader for javax.xml.stream.XMLInputFactory is: org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@7e19c472

00:12:00,671 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) Failed creation of XMLInputFactory implementation with classloader=org.eclipse.core.runtime.internal.adaptor.ContextFinder@409a6317

00:12:00,671 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) Exception is=java.lang.ClassCastException: __redirected.__XMLInputFactory cannot be cast to javax.xml.stream.XMLInputFactory

00:12:00,672 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) Attempting with classloader: org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@7e19c472

00:12:00,777 DEBUG [org.wso2.carbon.securevault.internal.SecretManagerInitializerComponent] (Component Resolve Thread (Bundle 71)) Starting 'SecretManagerInitializerComponent'

00:12:00,780 DEBUG [org.wso2.securevault.commons.MiscellaneousUtil] (Component Resolve Thread (Bundle 71)) The name with ' secret.manager.conf ' cannot be found. Using default value secret-manager.properties

00:12:00,781 DEBUG [org.wso2.securevault.secret.SecretManager] (Component Resolve Thread (Bundle 71)) Configuration properties can not be loaded form : secret-manager.properties Will use synapse properties

00:12:00,781 DEBUG [org.wso2.securevault.secret.SecretManager] (Component Resolve Thread (Bundle 71)) No secret repositories have been configured

00:12:00,919 INFO  [org.apache.qpid.wso2.internal.QpidBundleActivator] (Start Level Event Dispatcher) Setting BundleContext in PluginManager

00:12:00,929 DEBUG [org.wso2.carbon.policy.internal.Activator] (Start Level Event Dispatcher) ********* Security secpolicy ****

00:12:00,952 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) About to create XMLInputFactory implementation with classloader=org.eclipse.core.runtime.internal.adaptor.ContextFinder@409a6317

00:12:00,952 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) The classloader for javax.xml.stream.XMLInputFactory is: org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@7e19c472

00:12:00,953 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) Failed creation of XMLInputFactory implementation with classloader=org.eclipse.core.runtime.internal.adaptor.ContextFinder@409a6317

00:12:00,954 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) Exception is=java.lang.ClassCastException: __redirected.__XMLInputFactory cannot be cast to javax.xml.stream.XMLInputFactory

00:12:00,954 DEBUG [org.apache.axiom.om.util.StAXUtils] (Start Level Event Dispatcher) Attempting with classloader: org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@7e19c472

00:12:00,962 DEBUG [org.apache.jasper.compiler.JspRuntimeContext] (MSC service thread 1-15) Parent class loader is: ModuleClassLoader for Module "deployment.wso2greg.war:main" from Service Module Loader

00:12:00,96

[jason.greene]

Did you remember to remove the API jars from your deployment after adding them to your endorsed directory?

[subashsdm]

Hi Jason Greene,

Yes I put them inside JAVA_HOME/lib/endorsed and point that without the API jars. But still the same error. JBoss loading those classes by looking at system properties. So when we give our endorsed dir in JAVA_OPTS,

does they avoid looking at system properties ? Any reason you see ? It will be great if you can provide any clude for to get rid from this 

I also saw some debug error logs i the early part of the server log now.  Is this a causing factor or can be ignored ?

08:33:32,228 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) JBAS015003: Found wso2greg.war in deployment directory. To trigger deployment create a file called wso2greg.war.dodeploy

08:33:32,230 DEBUG [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) Deployment scan of [/home/subash/greg_on_jboss/jboss-as-7.1.0.Final/standalone/deployments] found update action [{

    "operation" => "composite",

    "address" => undefined,

    "steps" => [

        {

            "operation" => "add",

            "address" => [("deployment" => "wso2greg.war")],

            "content" => [{

                "path" => "deployments/wso2greg.war",

                "relative-to" => "jboss.server.base.dir",

                "archive" => false

            }],

            "persistent" => false

        },

        {

            "operation" => "deploy",

            "address" => [("deployment" => "wso2greg.war")]

        }

    ]

}]

08:33:32,234 INFO  [org.jboss.as.remoting] (MSC service thread 1-15) JBAS017100: Listening on /127.0.0.1:9999

08:33:32,236 INFO  [org.jboss.as.remoting] (MSC service thread 1-11) JBAS017100: Listening on /127.0.0.1:4447

08:33:32,252 CONFIG [com.sun.net.httpserver] (MSC service thread 1-4) HttpServer timer1 enabled period in ms:  1000

08:33:32,253 CONFIG [com.sun.net.httpserver] (MSC service thread 1-4) MAX_REQ_TIME:  15000

08:33:32,253 CONFIG [com.sun.net.httpserver] (MSC service thread 1-4) MAX_RSP_TIME:  -1

08:33:32,254 CONFIG [com.sun.net.httpserver] (MSC service thread 1-4) HttpServer created http /127.0.0.1:9990

08:33:32,257 DEBUG [org.jboss.modules] (MSC service thread 1-4) Module org.jboss.as.console:main defined by local module loader @5464ea66 (roots: /home/subash/greg_on_jboss/jboss-as-7.1.0.Final/modules)

08:33:32,262 DEBUG [org.jboss.tm.TransactionManagerLocator] (MSC service thread 1-16) Unable to lookup: java:/TransactionManager: javax.naming.NameNotFoundException: Error looking up TransactionManager, service service jboss.naming.context.java.TransactionManager is not started

  at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:126) [jboss-as-naming-7.1.0.Final.jar:7.1.0.Final]

  at org.jboss.as.naming.ServiceBasedNamingStore.lookup(ServiceBasedNamingStore.java:74) [jboss-as-naming-7.1.0.Final.jar:7.1.0.Final]

  at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:177) [jboss-as-naming-7.1.0.Final.jar:7.1.0.Final]

  at org.jboss.as.naming.InitialContext.lookup(InitialContext.java:113) [jboss-as-naming-7.1.0.Final.jar:7.1.0.Final]

  at org.jboss.as.naming.NamingContext.lookup(NamingContext.java:213) [jboss-as-naming-7.1.0.Final.jar:7.1.0.Final]

  at javax.naming.InitialContext.lookup(InitialContext.java:392) [rt.jar:1.6.0_26]

  at org.jboss.tm.TransactionManagerLocator.tryJNDI(TransactionManagerLocator.java:150) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.TransactionManagerLocator.locate(TransactionManagerLocator.java:131) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.TransactionManagerLocator.locateTransactionManager(TransactionManagerLocator.java:94) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.<init>(ServerVMClientUserTransaction.java:93) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.<clinit>(ServerVMClientUserTransaction.java:60) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.as.txn.service.ArjunaTransactionManagerService.start(ArjunaTransactionManagerService.java:115) [jboss-as-transactions-7.1.0.Final.jar:7.1.0.Final]

  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_26]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_26]

  at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_26]

08:33:32,268 DEBUG [org.jboss.modules] (MSC service thread 1-4) Module org.jboss.as.domain-http-error-context:main defined by local module loader @5464ea66 (roots: /home/subash/greg_on_jboss/jboss-as-7.1.0.Final/modules)

08:33:32,269 DEBUG [org.jboss.tm.TransactionManagerLocator] (MSC service thread 1-16) Unable to instantiate legacy transaction manager: java.lang.ClassNotFoundException: org.jboss.tm.TxManager from [Module "org.jboss.jboss-transaction-spi:main" from local module loader @5464ea66 (roots: /home/subash/greg_on_jboss/jboss-as-7.1.0.Final/modules)]

  at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:190)

  at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:468)

  at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:456)

  at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:423)

  at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:398)

  at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:120)

  at java.lang.Class.forName0(Native Method) [rt.jar:1.6.0_26]

  at java.lang.Class.forName(Class.java:169) [rt.jar:1.6.0_26]

  at org.jboss.tm.TransactionManagerLocator.usePrivateAPI(TransactionManagerLocator.java:172) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.TransactionManagerLocator.locate(TransactionManagerLocator.java:133) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.TransactionManagerLocator.locateTransactionManager(TransactionManagerLocator.java:94) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.<init>(ServerVMClientUserTransaction.java:93) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.tm.usertx.client.ServerVMClientUserTransaction.<clinit>(ServerVMClientUserTransaction.java:60) [jboss-transaction-spi-7.0.0.Final.jar:7.0.0.Final]

  at org.jboss.as.txn.service.ArjunaTransactionManagerService.start(ArjunaTransactionManagerService.java:115) [jboss-as-transactions-7.1.0.Final.jar:7.1.0.Final]

  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.2.GA.jar:1.0.2.GA]

  at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [rt.jar:1.6.0_26]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [rt.jar:1.6.0_26]

  at java.lang.Thread.run(Thread.java:662) [rt.jar:1.6.0_26]

08:33:32,271 DEBUG [org.apache.tomcat.util.net.jsse.JSSESocketFactory] (MSC service thread 1-3) Truststore = /home/subash/greg-repo/repository/resources/security/wso2carbon.jks

08:33:32,271 DEBUG [org.apache.tomcat.util.net.jsse.JSSESocketFactory] (MSC service thread 1-3) TrustPass = null

08:33:32,271 CONFIG [com.sun.net.httpserver] (MSC service thread 1-4) context created: /

08:33:32,272

[ronhan]

I have run into the same exception. Our war file needs to support multiple web servers, and we load an OSGi constainer inside our web app. We have a StAX bundle that exports javax.xml.stream and the woodstox implementation, and JBoss 7 is giving this same exception. It is not an option for us to take out our StAX bundle and use JBoss's because we do support other web servers and obviously do not want to build custom war files for each. Here is the 2-step solution I found:

1. __redirected.__XML?????Factory registers itself by setting Java system properties "javax.xml.stream.XML?????Factory" to its FQCN. So I programmatically set those properties again to the class from the woodstox implementation:

javax.xml.stream.XMLEventFactory=com.ctc.wstx.stax.WstxEventFactory

javax.xml.stream.XMLInputFactory=com.ctc.wstx.stax.WstxInputFactory

javax.xml.stream.XMLOutputFactory=com.ctc.wstx.stax.WstxOutputFactory

This is able to work because JBoss 7 also ship with the woodstox implementation. So our OSGi bundles will be able to use our StAX bundle, and JBoss's own code will ultimately uses its own woodstox implementation. Unless you do other server configuration, it is what __redirected.__XML?????Factory redirects to anyway. When I do this, JBoss still uses __redirected.__XML?????Factory until my application code runs and overwrites those properties.

2. After my application starts (i.e. properties have been reset), one of JBoss’s server threads would complain about not being able to find com.ctc.wstx.stax.WstxOutputFactory. This can be resolved by adding a dependency between the offending module to JBoss 7's woodstox module. For me, <module name="org.codehaus.woodstox" export="true"/> needs to be added to \jboss-as-7.1.1.Final\modules\org\jboss\as\controller\main\module.xml

For example:

<module xmlns="urn:jboss:module:1.1" name="org.jboss.as.controller">

<properties>

        <property name="jboss.api" value="private"/>

</properties>

<resources>

<resource-root path="jboss-as-controller-7.1.1.Final.jar"/>

        <!-- Insert resources here -->

</resources>

<dependencies>

<module name="javax.api"/>

<module name="org.codehaus.woodstox" export="true"/>

<module name="org.jboss.as.controller-client" export="true"/>

<module name="org.jboss.as.protocol"/>

<module name="org.jboss.dmr" export="true"/>

<module name="org.jboss.logging"/>

<module name="org.jboss.modules"/>

<module name="org.jboss.msc"/>

<module name="org.jboss.remoting3"/>

<module name="org.jboss.staxmapper"/>

<module name="org.jboss.threads"/>

</dependencies>

</module>

Afterwards, everything seems to work and I get no exceptions in my log file.

[mcr701]

With JBoss 7.1.1.Final, I managed to get rid of this problem by removing geronimo-stax-api,

like Jason recommended, and by adding this into OSGi subsystem configuration:

<property name="org.osgi.framework.system.packages">

     javax.xml.stream;version=1.0.0

</property>

I know this can be a different thing being on OSGi and not on web contained, but this worked for me

Not sure but you might need to add this also

<property name="org.jboss.osgi.system.modules.extra">

     javax.api

</property>

[sanjay05222]

For Some other people who are facing the same issue , probably they can try this as well. Probably it will solve their issue as well.

Error Which I was getting .

13:19:36,758 INFO [com.caiso.mds.soa.parser.Stax2Parser] (http--0.0.0.0-55080-3) Decompress Payload Attachment ended @ Thu Dec 10 13:19:36 PST 2015

13:19:36,844 INFO [com.caiso.mds.soa.parser.Stax2Parser] (http--0.0.0.0-55080-3) Time taken in ms to decompress the payload 155

13:19:36,857 ERROR [com.caiso.mds.soa.parser.Stax2Parser] (http--0.0.0.0-55080-3) Excepton :: java.lang.ClassCastException: __redirected.__XMLInputFactory cannot be cast

to org.codehaus.stax2.XMLInputFactory2

        at com.caiso.mds.soa.parser.Stax2Parser.process(Stax2Parser.java:87) [classes:]

        at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:98) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:105) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:67) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:103) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:71) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) [camel-core-2.14.1.jar:2.14.

1]

        at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:120) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:72) [camel-core-2.14.1.jar:2.14.1]

        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:416) [camel-core-2.14.1.jar:2.14.1]

Solution

Don't include a jar with this class into your application archive, but use MANIFEST.MF Dependencies declaration or jboss-deployment-structure.xml to create a dependency on 'javax.xml.stream.api'

Thanks

Sanjay Gautam