Intermittent JULLogDelegate error
Hi guys.
This has been a problem that has been paguing the development of our system with Hornet Q.
After hours of working correctly our java application will no long be able to connect to a queue. This state will remain for some time before it again accepts connections. Trying to find a guranteed replicable senario has so far evaded me.
This is the error text that gets reported:
2013-05-01 17:11:03,188 ERROR No filename pattern provided for FileAppender with name FileLogger
2013-05-01 17:11:03,188 ERROR Null object returned for RollingFile in appenders
2013-05-01 17:11:03,198 ERROR Unable to locate appender FileLogger for logger
Path: java.io.BufferedInputStream@4b9237a
[queue/test].[destination]=[jms/queue/test]
[queue/test].[url]=[remote://192.168.2.108:4447]
[queue/logging].[user]=[remote]
[queue/test].[user]=[remote]
[queue/logging].[password]=[remote123]
[queue/test].[context]=[org.jboss.naming.remote.client.InitialContextFactory]
[queue/logging].[destination]=[jms/queue/logging]
[queue/test].[password]=[remote123]
[queue/logging].[url]=[remote://192.168.2.108:4447]
[queue/logging].[context]=[org.jboss.naming.remote.client.InitialContextFactory]
May 01, 2013 5:11:03 PM org.xnio.Xnio <clinit>
INFO: XNIO Version 3.0.3.GA
May 01, 2013 5:11:03 PM org.xnio.nio.NioXnio <clinit>
INFO: XNIO NIO Implementation Version 3.0.3.GA
May 01, 2013 5:11:03 PM org.jboss.remoting3.EndpointImpl <clinit>
INFO: JBoss Remoting version 3.2.3.GA
May 01, 2013 5:11:03 PM org.hornetq.core.logging.impl.JULLogDelegate error
SEVERE: Failed to create netty connection
java.net.UnknownHostException: JBoss-Server.gpslsolutions.com
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.connect(OioClientSocketPipelineSink.java:114)
at org.jboss.netty.channel.socket.oio.OioClientSocketPipelineSink.eventSunk(OioClientSocketPipelineSink.java:74)
at org.jboss.netty.channel.Channels.connect(Channels.java:541)
at org.jboss.netty.channel.AbstractChannel.connect(AbstractChannel.java:210)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:227)
at org.jboss.netty.bootstrap.ClientBootstrap.connect(ClientBootstrap.java:188)
at org.hornetq.core.remoting.impl.netty.NettyConnector.createConnection(NettyConnector.java:473)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnection(ClientSessionFactoryImpl.java:1143)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.getConnectionWithRetry(ClientSessionFactoryImpl.java:993)
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.connect(ClientSessionFactoryImpl.java:224)
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:747)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
at com.gpsl.messaging.core.Broker.getConnection(Broker.java:41)
at com.gpsl.jndi.Jndi.connectToServer(Jndi.java:55)
at com.gpsl.messaging.core.SynchronousConsumer.<init>(SynchronousConsumer.java:37)
at com.gpsl.messaging.examples.SimpleLoggingService.<init>(SimpleLoggingService.java:40)
at com.gpsl.messaging.examples.SimpleLoggingService.main(SimpleLoggingService.java:176)
Exception in thread "main" javax.jms.JMSException: Failed to create session factory
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:605)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnection(HornetQConnectionFactory.java:119)
at com.gpsl.messaging.core.Broker.getConnection(Broker.java:41)
at com.gpsl.jndi.Jndi.connectToServer(Jndi.java:55)
at com.gpsl.messaging.core.SynchronousConsumer.<init>(SynchronousConsumer.java:37)
at com.gpsl.messaging.examples.SimpleLoggingService.<init>(SimpleLoggingService.java:40)
at com.gpsl.messaging.examples.SimpleLoggingService.main(SimpleLoggingService.java:176)
Caused by: HornetQException[errorCode=2 message=Cannot connect to server(s). Tried with all available servers.]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:769)
at org.hornetq.jms.client.HornetQConnectionFactory.createConnectionInternal(HornetQConnectionFactory.java:601)
... 6 more
This is the standalone-full.xml file that is being used to launch JBoss AS 7.1 (and hence, Hornet Q):
<?xml version='1.0' encoding='UTF-8'?>
<server xmlns="urn:jboss:domain:1.2">
<extensions>
<extension module="org.jboss.as.clustering.infinispan"/>
<extension module="org.jboss.as.cmp"/>
<extension module="org.jboss.as.configadmin"/>
<extension module="org.jboss.as.connector"/>
<extension module="org.jboss.as.deployment-scanner"/>
<extension module="org.jboss.as.ee"/>
<extension module="org.jboss.as.ejb3"/>
<extension module="org.jboss.as.jacorb"/>
<extension module="org.jboss.as.jaxr"/>
<extension module="org.jboss.as.jaxrs"/>
<extension module="org.jboss.as.jdr"/>
<extension module="org.jboss.as.jmx"/>
<extension module="org.jboss.as.jpa"/>
<extension module="org.jboss.as.jsr77"/>
<extension module="org.jboss.as.logging"/>
<extension module="org.jboss.as.mail"/>
<extension module="org.jboss.as.messaging"/>
<extension module="org.jboss.as.naming"/>
<extension module="org.jboss.as.osgi"/>
<extension module="org.jboss.as.pojo"/>
<extension module="org.jboss.as.remoting"/>
<extension module="org.jboss.as.sar"/>
<extension module="org.jboss.as.security"/>
<extension module="org.jboss.as.threads"/>
<extension module="org.jboss.as.transactions"/>
<extension module="org.jboss.as.web"/>
<extension module="org.jboss.as.webservices"/>
<extension module="org.jboss.as.weld"/>
</extensions>
<system-properties>
<property name="catalina.home" value="C:\Program Files (x86)\jboss-as-7.1.1.Final\standalone"/>
</system-properties>
<management>
<security-realms>
<security-realm name="ManagementRealm">
<authentication>
<properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
<security-realm name="ApplicationRealm">
<authentication>
<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
</authentication>
</security-realm>
</security-realms>
<management-interfaces>
<native-interface security-realm="ManagementRealm">
<socket-binding native="management-native"/>
</native-interface>
<http-interface security-realm="ManagementRealm">
<socket-binding http="management-http"/>
</http-interface>
</management-interfaces>
</management>
<profile>
<subsystem xmlns="urn:jboss:domain:logging:1.1">
<console-handler name="CONSOLE">
<level name="INFO"/>
<formatter>
<pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
</formatter>
</console-handler>
<periodic-rotating-file-handler name="FILE">
<formatter>
<pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/>
</formatter>
<file relative-to="jboss.server.log.dir" path="server.log"/>
<suffix value=".yyyy-MM-dd"/>
<append value="true"/>
</periodic-rotating-file-handler>
<logger category="com.arjuna">
<level name="WARN"/>
</logger>
<logger category="org.apache.tomcat.util.modeler">
<level name="WARN"/>
</logger>
<logger category="sun.rmi">
<level name="WARN"/>
</logger>
<logger category="jacorb">
<level name="WARN"/>
</logger>
<logger category="jacorb.config">
<level name="ERROR"/>
</logger>
<root-logger>
<level name="INFO"/>
<handlers>
<handler name="CONSOLE"/>
<handler name="FILE"/>
</handlers>
</root-logger>
</subsystem>
<subsystem xmlns="urn:jboss:domain:cmp:1.0"/>
<subsystem xmlns="urn:jboss:domain:configadmin:1.0"/>
<subsystem xmlns="urn:jboss:domain:datasources:1.0">
<datasources>
<datasource jndi-name="java:jboss/datasources/ExampleDS" pool-name="ExampleDS" enabled="true" use-java-context="true">
<connection-url>jdbc:h2:mem:test;DB_CLOSE_DELAY=-1</connection-url>
<driver>h2</driver>
<security>
<user-name>sa</user-name>
<password>sa</password>
</security>
</datasource>
<datasource jndi-name="java:/saas-authentication-database" pool-name="AuthDS" enabled="true" use-java-context="true">
<connection-url>jdbc:mysql://localhost:3306/saas_authentication</connection-url>
<driver>mysql</driver>
<security>
<user-name>authentication</user-name>
<password>gpsl123saas</password>
</security>
</datasource>
<datasource jta="false" jndi-name="java:/guvnor-repo" pool-name="MySqlDS1" enabled="true" use-java-context="true" use-ccm="false">
<connection-url>jdbc:mysql://localhost:3306/guvnor-repo</connection-url>
<driver>mysql</driver>
<security>
<user-name>guvnor</user-name>
<password>gpsl123saas</password>
</security>
</datasource>
<drivers>
<driver name="h2" module="com.h2database.h2">
<xa-datasource-class>org.h2.jdbcx.JdbcDataSource</xa-datasource-class>
</driver>
<driver name="mysql" module="com.mysql.jdbc">
<xa-datasource-class>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</xa-datasource-class>
</driver>
</drivers>
</datasources>
</subsystem>
<subsystem xmlns="urn:jboss:domain:deployment-scanner:1.1">
<deployment-scanner path="deployments" relative-to="jboss.server.base.dir" scan-interval="5000"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:ee:1.0"/>
<subsystem xmlns="urn:jboss:domain:ejb3:1.2">
<session-bean>
<stateless>
<bean-instance-pool-ref pool-name="slsb-strict-max-pool"/>
</stateless>
<stateful default-access-timeout="5000" cache-ref="simple"/>
<singleton default-access-timeout="5000"/>
</session-bean>
<mdb>
<resource-adapter-ref resource-adapter-name="hornetq-ra"/>
<bean-instance-pool-ref pool-name="mdb-strict-max-pool"/>
</mdb>
<pools>
<bean-instance-pools>
<strict-max-pool name="slsb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
<strict-max-pool name="mdb-strict-max-pool" max-pool-size="20" instance-acquisition-timeout="5" instance-acquisition-timeout-unit="MINUTES"/>
</bean-instance-pools>
</pools>
<caches>
<cache name="simple" aliases="NoPassivationCache"/>
<cache name="passivating" passivation-store-ref="file" aliases="SimpleStatefulCache"/>
</caches>
<passivation-stores>
<file-passivation-store name="file"/>
</passivation-stores>
<async thread-pool-name="default"/>
<timer-service thread-pool-name="default">
<data-store path="timer-service-data" relative-to="jboss.server.data.dir"/>
</timer-service>
<remote connector-ref="remoting-connector" thread-pool-name="default"/>
<thread-pools>
<thread-pool name="default">
<max-threads count="10"/>
<keepalive-time time="100" unit="milliseconds"/>
</thread-pool>
</thread-pools>
<iiop enable-by-default="false" use-qualified-name="false"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:infinispan:1.2" default-cache-container="hibernate">
<cache-container name="hibernate" default-cache="local-query">
<local-cache name="entity">
<transaction mode="NON_XA"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="local-query">
<transaction mode="NONE"/>
<eviction strategy="LRU" max-entries="10000"/>
<expiration max-idle="100000"/>
</local-cache>
<local-cache name="timestamps">
<transaction mode="NONE"/>
<eviction strategy="NONE"/>
</local-cache>
</cache-container>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jacorb:1.1">
<orb>
<initializers security="on" transactions="spec"/>
</orb>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jaxr:1.1">
<connection-factory jndi-name="java:jboss/jaxr/ConnectionFactory"/>
<properties/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jaxrs:1.0"/>
<subsystem xmlns="urn:jboss:domain:jca:1.1">
<archive-validation enabled="true" fail-on-error="true" fail-on-warn="false"/>
<bean-validation enabled="true"/>
<default-workmanager>
<short-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</short-running-threads>
<long-running-threads>
<core-threads count="50"/>
<queue-length count="50"/>
<max-threads count="50"/>
<keepalive-time time="10" unit="seconds"/>
</long-running-threads>
</default-workmanager>
<cached-connection-manager/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jdr:1.0"/>
<subsystem xmlns="urn:jboss:domain:jmx:1.1">
<show-model value="true"/>
<remoting-connector/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jpa:1.0">
<jpa default-datasource=""/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:jsr77:1.0"/>
<subsystem xmlns="urn:jboss:domain:mail:1.0">
<mail-session jndi-name="java:jboss/mail/Default">
<smtp-server outbound-socket-binding-ref="mail-smtp"/>
</mail-session>
</subsystem>
<subsystem xmlns="urn:jboss:domain:messaging:1.1">
<hornetq-server>
<persistence-enabled>true</persistence-enabled>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
<connectors>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<netty-acceptor name="netty" socket-binding="messaging"/>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="RemoteConnectionFactory"/>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
<jms-destinations>
<jms-queue name="testQueue">
<entry name="queue/test"/>
<entry name="java:jboss/exported/jms/queue/test"/>
</jms-queue>
<jms-queue name="loggingQueue">
<entry name="queue/logging"/>
<entry name="java:jboss/exported/jms/queue/logging"/>
</jms-queue>
<jms-queue name="steveQueue">
<entry name="queue/steve"/>
<entry name="java:jboss/exported/jms/queue/steve"/>
</jms-queue>
<jms-queue name="pdf.composition">
<entry name="queue/pdf.composition"/>
<entry name="java:jboss/exported/jms/queue/pdf.composition"/>
</jms-queue>
<jms-queue name="wordToODTQueue">
<entry name="queue/wordToODT"/>
<entry name="java:jboss/exported/jms/queue/wordToODT"/>
</jms-queue>
<jms-queue name="sectionaliserQueue">
<entry name="queue/sectionaliser"/>
<entry name="java:jboss/exported/jms/queue/sectionaliser"/>
</jms-queue>
<jms-topic name="testTopic">
<entry name="topic/test"/>
<entry name="java:jboss/exported/jms/topic/test"/>
</jms-topic>
</jms-destinations>
</hornetq-server>
</subsystem>
<subsystem xmlns="urn:jboss:domain:naming:1.1"/>
<subsystem xmlns="urn:jboss:domain:osgi:1.2" activation="lazy">
<properties>
<property name="org.osgi.framework.startlevel.beginning">
1
</property>
</properties>
<capabilities>
<capability name="javax.servlet.api:v25"/>
<capability name="javax.transaction.api"/>
<capability name="org.apache.felix.log" startlevel="1"/>
<capability name="org.jboss.osgi.logging" startlevel="1"/>
<capability name="org.apache.felix.configadmin" startlevel="1"/>
<capability name="org.jboss.as.osgi.configadmin" startlevel="1"/>
</capabilities>
</subsystem>
<subsystem xmlns="urn:jboss:domain:pojo:1.0"/>
<subsystem xmlns="urn:jboss:domain:remoting:1.1">
<connector name="remoting-connector" socket-binding="remoting" security-realm="ApplicationRealm"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:resource-adapters:1.0"/>
<subsystem xmlns="urn:jboss:domain:sar:1.0"/>
<subsystem xmlns="urn:jboss:domain:security:1.1">
<security-domains>
<security-domain name="other" cache-type="default">
<authentication>
<login-module code="Remoting" flag="optional">
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
<login-module code="RealmUsersRoles" flag="required">
<module-option name="usersProperties" value="${jboss.server.config.dir}/application-users.properties"/>
<module-option name="rolesProperties" value="${jboss.server.config.dir}/application-roles.properties"/>
<module-option name="realm" value="ApplicationRealm"/>
<module-option name="password-stacking" value="useFirstPass"/>
</login-module>
</authentication>
</security-domain>
<security-domain name="jboss-web-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
<security-domain name="jboss-ejb-policy" cache-type="default">
<authorization>
<policy-module code="Delegating" flag="required"/>
</authorization>
</security-domain>
<security-domain name="saas-authentication-policy" cache-type="default">
<authentication>
<login-module code="org.jboss.security.auth.spi.DatabaseServerLoginModule" flag="required">
<module-option name="unauthenticatedIdentity" value="guest"/>
<module-option name="dsJndiName" value="java:/saas-authentication-database"/>
<module-option name="principalsQuery" value="SELECT password FROM users WHERE username=?"/>
<module-option name="rolesQuery" value="SELECT role, 'Roles' FROM user_role WHERE username=?"/>
</login-module>
</authentication>
</security-domain>
</security-domains>
</subsystem>
<subsystem xmlns="urn:jboss:domain:threads:1.1"/>
<subsystem xmlns="urn:jboss:domain:transactions:1.1">
<core-environment>
<process-id>
<uuid/>
</process-id>
</core-environment>
<recovery-environment socket-binding="txn-recovery-environment" status-socket-binding="txn-status-manager"/>
<coordinator-environment default-timeout="300"/>
</subsystem>
<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host" native="false">
<connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http"/>
<virtual-server name="default-host" enable-welcome-root="true">
<alias name="JBoss-Server.gpslsolutions.com"/>
<alias name="example.com"/>
</virtual-server>
</subsystem>
<subsystem xmlns="urn:jboss:domain:webservices:1.1">
<modify-wsdl-address>true</modify-wsdl-address>
<wsdl-host>${jboss.bind.address:JBoss-Server.gpslsolutions.com}</wsdl-host>
<endpoint-config name="Standard-Endpoint-Config"/>
<endpoint-config name="Recording-Endpoint-Config">
<pre-handler-chain name="recording-handlers" protocol-bindings="##SOAP11_HTTP ##SOAP11_HTTP_MTOM ##SOAP12_HTTP ##SOAP12_HTTP_MTOM">
<handler name="RecordingHandler" class="org.jboss.ws.common.invocation.RecordingServerHandler"/>
</pre-handler-chain>
</endpoint-config>
</subsystem>
<subsystem xmlns="urn:jboss:domain:weld:1.0"/>
</profile>
<interfaces>
<interface name="management">
<inet-address value="${jboss.bind.address.management:JBoss-Server.gpslsolutions.com}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:JBoss-Server.gpslsolutions.com}"/>
</interface>
<interface name="unsecure">
<inet-address value="${jboss.bind.address.unsecure:JBoss-Server.gpslsolutions.com}"/>
</interface>
</interfaces>
<socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}">
<socket-binding name="management-native" interface="management" port="${jboss.management.native.port:9999}"/>
<socket-binding name="management-http" interface="management" port="${jboss.management.http.port:9990}"/>
<socket-binding name="management-https" interface="management" port="${jboss.management.https.port:9443}"/>
<socket-binding name="ajp" port="8009"/>
<socket-binding name="http" port="8080"/>
<socket-binding name="https" port="8443"/>
<socket-binding name="jacorb" interface="unsecure" port="3528"/>
<socket-binding name="jacorb-ssl" interface="unsecure" port="3529"/>
<socket-binding name="messaging" port="5445"/>
<socket-binding name="messaging-throughput" port="5455"/>
<socket-binding name="osgi-http" interface="management" port="8090"/>
<socket-binding name="remoting" port="4447"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="JBoss-Server.gpslsolutions.com" port="25"/>
</outbound-socket-binding>
</socket-binding-group>
</server>
Any pointers you can provide would be most helpful, I have been tearing my hair out over this for a week now.
Thanks a lot
Rob
