Resource adapter JMS lookups failing| JBoss.org Content Archive (Read Only)

15. Re: Resource adapter JMS lookups failing

jewellgm Jun 4, 2015 7:47 AM (in response to jaikiran)

Jaikiran,

In another thread, you described how to set up a deployment dependency in a web.xml. Is there a similar capability for ejb or connector modules? Since the ears that I'm deploying don't contain any wars, I don't have a web.xml. If nothing similar exists, could I put a empty war into the ear with the dependencies defined in web.xml, and then specify that the war has to deploy first in the application.xml?

Re: Wildfly.9.0.0CR1 Issues error for infinispan config

Thanks,

Greg

16. Re: Resource adapter JMS lookups failing

jaikiran Jun 4, 2015 7:56 AM (in response to jewellgm)

You can add similar dependencies through ejb-jar.xml. The resource-ref, resource-env-ref and other similar constructs are available across all Java EE components so you can use them in ejb-jar.xml as well. Take a look at the ejb-jar xsd (3.0 or higher) which should give you an idea. If you need more details, let us know, there should be a testcase within the WildFly testsuite which should have an example which we can point you to.

P.S: In this specific case of yours I would expect a dependency on the right set of JNDI names to be implicitly added by the server when it notices that it has to deploy an MDB listening on a topic at a particular JNDI name, so you don't necessarily need an explicitly dependency to be added. However, I haven't completely grasped the details of this thread yet to understand what issue you are running into.

17. Re: Resource adapter JMS lookups failing

jewellgm Jun 4, 2015 9:12 AM (in response to jaikiran)

Thank you again Jaikiran. I've added an ejb-all.xml file to one of the ears, but I'm still getting error messages in the log indicating that the message bean failed in some manner.

Here's the entire contents of the ejb-all.xml file that is contained in the ear's META-INF directory:

<ejb-jar xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/ejb-jar_3_2.xsd" version="3.2">

<display-name>TestMessageBean</display-name>

<enterprise-beans>

<message-driven>

<description>A message bean that listens on the UpdateTopic</description>

<ejb-name>TestMessageBean</ejb-name>

<ejb-class>com.test.TestMessageBean</ejb-class>

<resource-env-ref>

<resource-env-ref-name>topic/UpdateTopic</resource-env-ref-name>

<resource-env-ref-type>javax.jms.Topic</resource-env-ref-type>

</resource-env-ref>

</message-driven>

</enterprise-beans>

</ejb-jar>

Here's the declaration of the message bean itself:

@MessageDriven(activationConfig = {

@ActivationConfigProperty(

propertyName = "destinationType",

propertyValue = "javax.jms.Topic"),

@ActivationConfigProperty(

propertyName = "destination",

propertyValue = "topic/UpdateTopic"),

@ActivationConfigProperty(

propertyName = "subscriptionDurability",

propertyValue = "NonDurable"),

@ActivationConfigProperty(

propertyName = "messageSelector",

propertyValue = "ClassName LIKE 'TestRequest'")

})

public class TestMessageBean implements MessageListener { ... }

And the following is what I see in the server.log when wildfly is starting up:

2015-06-04 09:02:36,729 WARN [org.hornetq.ra] (default-threads - 6) HQ152005: Failure in HornetQ activation org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@3c6f8375 connectionFactoryLookup=null destination=topic/UpdateTopic destinationType=javax.jms.Topic selector=ClassName LIKE 'TestRequest' ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15): java.lang.NullPointerException

at org.hornetq.ra.inflow.HornetQActivation.getAddress(HornetQActivation.java:598) [hornetq-ra-2.4.7.Final.jar:]

at org.hornetq.ra.inflow.HornetQMessageHandler.setup(HornetQMessageHandler.java:165) [hornetq-ra-2.4.7.Final.jar:]

at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:340) [hornetq-ra-2.4.7.Final.jar:]

at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:819) [hornetq-ra-2.4.7.Final.jar:]

at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:226)

at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)

at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:808)

at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)

at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:828)

at java.lang.Thread.run(Unknown Source) [rt.jar:1.8.0_45]

at org.jboss.threads.JBossThread.run(JBossThread.java:320)

This message bean is the only one that filters the "TestRequest" messages, so I'm confident that the error relates to this message bean, and not another. I have not placed an ejb-all.xml file into any of the other ears at this point.

18. Re: Resource adapter JMS lookups failing

jaikiran Jun 4, 2015 9:29 AM (in response to jewellgm)

Are there any changes in your standalone/domain configuration xml that you are using on this instance? Or is this NPE against a clean instance of a the 9.0.0.CR1 instance? Either way, that NPE is definitely a bug. What remains to be seen is how is that triggered. Looking at the HornetQ RA code, I don't see how that destination can end up being null unless of course it's skipping a code path for some reason.

If you can confirm there's no changes in the domain/standalone files or if there are any post those here, then maybe one of us can find an hint in there.

19. Re: Resource adapter JMS lookups failing

jaikiran Jun 4, 2015 9:37 AM (in response to jewellgm)

I read through this thread again.

Our resource adapters are being used to communicate to other external systems through a variety of methods. We're not intending to write our own JMS adapter. What we're doing, perhaps incorrectly, is using our resource adapter to drop messages on the JMS bus that are received from these external systems. We're trying to look up the HornetQ InVM connection factory by manually doing JNDI lookups (rather than resource injection). It appears that our resource adapters are being deployed prior to the JMS adapter provided by WildFly, thereby causing the issue we're seeing.

Could you post relevant configurations and code that's involved here? It seems that this is what is the missing piece which seems to be triggering that NPE.

20. Re: Resource adapter JMS lookups failing

jewellgm Jun 4, 2015 10:15 AM (in response to jaikiran)

jaikiran pai wrote:

I read through this thread again.

Our resource adapters are being used to communicate to other external systems through a variety of methods. We're not intending to write our own JMS adapter. What we're doing, perhaps incorrectly, is using our resource adapter to drop messages on the JMS bus that are received from these external systems. We're trying to look up the HornetQ InVM connection factory by manually doing JNDI lookups (rather than resource injection). It appears that our resource adapters are being deployed prior to the JMS adapter provided by WildFly, thereby causing the issue we're seeing.

Could you post relevant configurations and code that's involved here? It seems that this is what is the missing piece which seems to be triggering that NPE.

It appears that there are two separate issues:

1.) The resource adapters were trying to look up the JMS connection factory when they were started. They were failing to deploy because the lookup failed, causing the initialization to bomb out. I delayed the lookup of the connection factory until after a message is received from a remote system, and that seems to have resolved that issue.

2.) Message-driven beans are throwing the exception that I pasted above. These MDBs are not contained within the resource adapter, but are in another (ejb) module within the same ear.

Both of these issues only occur when I'm deploying several of these ears when WildFly starts. If I deploy only a few, everything appears to work as expected. After WildFly is completely started, I can then manually deploy each of the remaining ears without issue.

Having said that, I'll post code snippets (and configuration changes) as I'm able. Unfortunately, my employer won't allow me to post anything that would be considered proprietary, so I have to scrub things down.

There are many changes that are made to standalone.xml, but here are the relevant pieces in regards to messaging and JMS:

<hornetq-server>

<persistence-enabled>true</persistence-enabled>

<journal-file-size>102400</journal-file-size>

<journal-min-files>2</journal-min-files>

<http-connector name="http-connector" socket-binding="http">

</http-connector>

<http-connector name="http-connector-throughput" socket-binding="http">

</http-connector>

<in-vm-connector name="in-vm" server-id="0"/>

</connectors>

<http-acceptor http-listener="default" name="http-acceptor"/>

<http-acceptor http-listener="default" name="http-acceptor-throughput">

</http-acceptor>

<in-vm-acceptor name="in-vm" server-id="0"/>

</acceptors>

<security-settings>

<security-setting match="#">

</security-setting>

</security-settings>

<address-settings>

<address-setting match="#">

<dead-letter-address>jms.queue.DLQ</dead-letter-address>

<expiry-address>jms.queue.ExpiryQueue</expiry-address>

<redelivery-delay>0</redelivery-delay>

<max-size-bytes>10485760</max-size-bytes>

<address-full-policy>BLOCK</address-full-policy>

<message-counter-history-day-limit>10</message-counter-history-day-limit>

</address-setting>

</address-settings>

<jms-connection-factories>

<connection-factory name="InVmConnectionFactory">

<connector-ref connector-name="in-vm"/>

</connectors>

</entries>

</connection-factory>

<connection-factory name="RemoteConnectionFactory">

<connector-ref connector-name="http-connector"/>

</connectors>

</entries>

</connection-factory>

<pooled-connection-factory name="hornetq-ra">

<connector-ref connector-name="in-vm"/>

</connectors>

</entries>

</pooled-connection-factory>

</jms-connection-factories>

<jms-destinations>

<jms-topic name="UpdateTopic">

</jms-topic>

<jms-topic name="AnotherUpdateTopic">

</jms-topic>

</jms-destinations>

</hornetq-server>

</subsystem>

For our resource adapters, the following code is invoked after we receive a message from the external system. This used to be executed from within the start(BootstrapContext bsc) of the resource adapter, but the lookup of the connection factory would fail. Note that a given resource adapter may drop message on more than one topic, so this code may be called a few times with different topic names.

	private void initializeJMS(String topicName) throws NamingException, JMSException {
	Context ctx = null;
	try {
	// Retrieve the JNDI Context.
	ctx = new InitialContext();

	// Establish a JMS Connection.
	TopicConnectionFactory factory = (TopicConnectionFactory)
	ctx.lookup("ConnectionFactory");
	conn = factory.createConnection();
	conn.start();

	// Create a JMS Session.
	session = conn.createSession(false, Session.AUTO_ACKNOWLEDGE);

	// Create a JMS Message Producer.
	Destination dest = (Destination) ctx.lookup(topicName);
	producer = session.createProducer(dest);
	producer.setDeliveryMode(DeliveryMode.NON_PERSISTENT);
	} finally {
	ctx.close();
	}
	}

21. Re: Resource adapter JMS lookups failing

jaikiran Jun 4, 2015 10:09 PM (in response to jewellgm)

Having looked at the WildFly and HornetQ RA code, I think I know what's going on. It ultimately boils down to the proper dependency not being set which results in a JNDI lookup at the "wrong" time. However, it's a bit weird that even after the deployment descriptor change you did, it still shows up. Unless of course the deployment descriptor change wasn't picked up correctly (I would have expected the deployment to fail if that was the case, but anyway).

Edit: (after reading your post about the ejb-jar.xml) - Never mind, I see that the ejb-jar.xml content is invalid (it's missing the lookup-name element in the resource-env-ref). That should have failed the deployment, but then it looks like you place that file in a wrong location:

Here's the entire contents of the ejb-all.xml file that is contained in the ear's META-INF directory

As a result, that file was completely ignored by the server. So it effectively means, nothing changed

So here's what you could do to "workaround" this issue:

- Place the ejb-jar.xml into the META-INF folder of the EJB jar file which contains your MDB.

- Make sure that the content of the ejb-jar.xml is this:

<ejb-jar xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/ejb-jar_3_2.xsd" version="3.2">
    <display-name>TestMessageBean</display-name>
    <enterprise-beans>
        <message-driven>
            <description>A message bean that listens on the UpdateTopic</description>
            <ejb-name>TestMessageBean</ejb-name>
            <ejb-class>com.test.TestMessageBean</ejb-class>
            <resource-env-ref>
                <resource-env-ref-name>jms/topic/UpdateTopic</resource-env-ref-name>
                <resource-env-ref-type>javax.jms.Topic</resource-env-ref-type>
                <lookup-name>java:jboss/topic/UpdateTopic<lookup-name>
            </resource-env-ref>
        </message-driven>
    </enterprise-beans>
</ejb-jar>

Make sure that the ejb-name matches the simple class name of the MDB and the ejb-class matches the fully qualified classname of the MDB.

OR

You can just get rid of the ejb-jar.xml and just use annotation to setup similar dependency:

@MessageDriven(activationConfig = {

        @ActivationConfigProperty(

                propertyName = "destinationType",

                propertyValue = "javax.jms.Topic"),

        @ActivationConfigProperty(

                propertyName = "destination",

                propertyValue = "topic/UpdateTopic"),

        @ActivationConfigProperty(

                propertyName = "subscriptionDurability",

                propertyValue = "NonDurable"),

        @ActivationConfigProperty(

                propertyName = "messageSelector",

                propertyValue = "ClassName LIKE 'TestRequest'")

})
@Resource(name="jms/topic/UpdateTopic", type="javax.jms.Topic", lookup="java:jboss/topic/UpdateTopic")
public class TestMessageBean implements MessageListener { ... }

Notice the use of @Resource. In the context of your testing this issue, the annotation way is "easier" since you don't have to worry about checking if the ejb-jar.xml you packaged was placed in the right path or not.

If this works and you no longer see the NPE, then it confirms a bug within WildFly (which I can explain later) and you can stick with this workaround till the time it is fixed.

P.S: You might see compile error in your IDE with the @Resource annotation, depending on your IDE and how it's setup. If you do see it, just follow this http://jaitechwriteups.blogspot.in/2011/02/resource-and-new-lookup-attribute-how.html

22. Re: Resource adapter JMS lookups failing

jewellgm Jun 5, 2015 10:02 AM (in response to jaikiran)

Jaikiran,

Thank you for looking into this more deeply. I annotated all of my message driven beans as you suggested, but it didn't seem to make any difference. I am still getting the same exceptions that I did before.

Having said that, I had to modify the annotation you suggested slightly. The "type" parameter has to be a Class<T> rather than a String, and the lookup value was incorrect. With the wrong lookup value, everything would roll back. Here's the annotation that I ended up using:

@Resource(name="jms/topic/UpdateTopic", type=javax.jms.Topic.class, lookup="java:jboss/exported/jms/topic/UpdateTopic")

It appears that the exceptions I'm getting for the message beans are not fatal. Later in the log, I see messages like the following:

HQ151001: Attempting to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@2231b9b8 connectionFactoryLookup=null destination=topic/UpdateTopic destinationType=javax.jms.Topic ack=Auto-acknowledge durable=false clientID=null user=null maxSession=15)

When I see these messages, I'll either get the same stack trace, or it will appear to successfully connect. Limited testing makes it appear that messages are in fact being received by the MDB even in the scenario where these exceptions appear.

In the cases where my resource adapters are creating message producers, I still have to delay the instantiation. The attempted lookup for the connection factory occurs only once (of course), and failing to find the connection factory during the start of the resource adapter was fatal, causing everything to roll back.

At this point, I'm not going to actively pursue this thread any further. It appears that what I have now is sufficient, although not optimal. If you (or anybody else) have any more suggestions, I would be happy to try them.

23. Re: Resource adapter JMS lookups failing

jaikiran Jun 5, 2015 10:13 AM (in response to jewellgm)

Can you post use the console logs which shows what JNDI names the topics actually get bound to? You could even attach teh entire server.log (after removing any company specific details).

24. Re: Resource adapter JMS lookups failing

jewellgm Jun 5, 2015 11:07 AM (in response to jaikiran)

The only thing that I see in the log with regards to actual deployment of the topic is this:

[org.hornetq.core.server] (ServerService Thread Pool -- 70) HQ221003: trying to deploy queue jms.topic.UpdateTopic

Setting TRACE level logging on org.hornetq.core.server adds the following:

2015-06-05 10:46:21,794 DEBUG [org.hornetq.core.server] (ServerService Thread Pool -- 72) registered address org.hornetq:module=Core,type=Address,name="jms.topic.UpdateTopic"

2015-06-05 10:46:21,794 DEBUG [org.hornetq.core.server] (ServerService Thread Pool -- 72) registered queue org.hornetq:module=Core,type=Queue,address="jms.topic.UpdateTopic",name="jms.topic.UpdateTopic"

2015-06-05 10:46:21,794 TRACE [org.hornetq.core.server] (ServerService Thread Pool -- 72) Sending Notification = Notification[uid=null, type=TOPIC_CREATED, properties=TypedProperties[message=UpdateTopic]], notificationEnabled=true messagingServerControl=org.hornetq.core.management.impl.HornetQServerControlImpl@5415f16f

2015-06-05 10:46:21,794 DEBUG [org.hornetq.core.server] (ServerService Thread Pool -- 72) Couldn't find any bindings for address=hornetq.notifications on message=ServerMessage[messageID=24,durable=true,userID=null,priority=0, bodySize=512,expiration=0, durable=true, address=hornetq.notifications,properties=TypedProperties[_HQ_NotifType=TOPIC_CREATED,_HQ_NotifTimestamp=1433515581794,message=UpdateTopic]]@278679795

2015-06-05 10:46:21,794 TRACE [org.hornetq.core.server] (ServerService Thread Pool -- 72) Message after routed=ServerMessage[messageID=24,durable=true,userID=null,priority=0, bodySize=512,expiration=0, durable=true, address=hornetq.notifications,properties=TypedProperties[_HQ_NotifType=TOPIC_CREATED,_HQ_NotifTimestamp=1433515581794,message=UpdateTopic]]@278679795

2015-06-05 10:46:21,795 DEBUG [org.hornetq.core.server] (ServerService Thread Pool -- 72) Message ServerMessage[messageID=24,durable=true,userID=null,priority=0, bodySize=512,expiration=0, durable=true, address=hornetq.notifications,properties=TypedProperties[_HQ_NotifType=TOPIC_CREATED,_HQ_NotifTimestamp=1433515581794,message=UpdateTopic]]@278679795 is not going anywhere as it didn't have a binding on address:hornetq.notifications

I don't see anything that shows what the actual JNDI lookup for the topic is, except what is specified in the standalone.xml. If the lookup value in the @Resource annotation didn't match what was in my standalone.xml, I'd get an error message like this:

"jboss.naming.context.java.comp.Test.test.TestMessageBean.env.jms.topic.UpdateTopic is missing [jboss.naming.context.java.jboss.topic.UpdateTopic]",

Once this occurred, everything would roll back.

At any point, does the limitation of only being able to post once per hour go away? I've been delayed in responding several times due to th

25. Re: Resource adapter JMS lookups failing

jaikiran Jun 5, 2015 12:19 PM (in response to jewellgm)

You should see something like this at INFO level in console and server.log:

20:54:12,145 INFO [org.hornetq.core.server] (ServerService Thread Pool -- 58) HQ221003: trying to deploy queue jms.queue.ExpiryQueue

....

20:54:12,294 INFO [org.jboss.as.messaging] (ServerService Thread Pool -- 58) JBAS011601: Bound messaging object to jndi name java:/jms/queue/ExpiryQueue

I suspect you will see that in your case it gets bound to:

[org.jboss.as.messaging] (ServerService Thread Pool -- 60) JBAS011601: Bound messaging object to jndi name /topic/UpdateTopic

So I provided you the wrong snippet to use earlier on. Should have paid more attention. If you are still willing to give this a try, could you try changing the MDB to use this:

@MessageDriven(activationConfig = { 

        @ActivationConfigProperty( 

                propertyName = "destinationType", 

                propertyValue = "javax.jms.Topic"), 

        @ActivationConfigProperty( 

                propertyName = "destination", 

                propertyValue = "java:jboss/jms/topic/UpdateTopic"), 

        @ActivationConfigProperty( 

                propertyName = "subscriptionDurability", 

                propertyValue = "NonDurable"), 

        @ActivationConfigProperty( 

                propertyName = "messageSelector", 

                propertyValue = "ClassName LIKE 'TestRequest'") 

}) 
@Resource(name="jms/topic/UpdateTopic", type="javax.jms.Topic", lookup="java:jboss/jms/topic/UpdateTopic") 
public class TestMessageBean implements MessageListener { ... }

Notice I've changed the lookup attribute of @Resource as well as changed the destination @ActivationvationConfigProperty value.

Furthermore, change your standalone/domain config for that topic to:

<jms-topic name="UpdateTopic">
                        <entry name="java:jboss/jms/topic/UpdateTopic"/>
                        <entry name="java:jboss/exported/jms/topic/UpdateTopic"/>
                    </jms-topic>

At any point, does the limitation of only being able to post once per hour go away? I've been delayed in responding several times due to th

Clearly no one cares Current state of forums - Neglect, spam and blocking genuine participation, can someone from .org team help?. The reasons stated in that thread and a few other things have made these forums almost useless and it's no longer the place where there used to be lively discussions. I no longer visit here as frequently as I used to and there are only a few regulars left here now. Some projects have even moved to other places for discussions/forums.

26. Re: Resource adapter JMS lookups failing

jewellgm Jun 5, 2015 12:33 PM (in response to jaikiran)

I'll do this, but it will take some time to go in and change all of the lookups. Incidentally, there is no place in my log that explicitly says what JNDI name a topic is bound to, even with your suggestion of changing the name to "java:jboss/jms/topic/UpdateTopic". The *only* thing that the log shows is explicitly bound to a name (with regards to JMS), are the various connection factories. All of the topics simply say "HQ221003: trying to deploy queue jms.topic.UpdateTopic", and that's it.

Once I update all of the code with the modifications, I'll post another update.

27. Re: Resource adapter JMS lookups failing

jewellgm Jun 5, 2015 2:06 PM (in response to jaikiran)

Even after with the changes suggested, I'm still getting the same exception. As I mentioned before, there's nothing in the WF9 log that tells me explicitly what JNDI name a given topic is bound to. In addition to the exceptions I'm getting on my MDBs, I also see the following stack error on producers:

Caused by: java.lang.UnsupportedOperationException: Cannot specify destination if producer has a default destination

at org.hornetq.jms.client.HornetQMessageProducer.checkDefaultDestination(HornetQMessageProducer.java:322)

at org.hornetq.jms.client.HornetQMessageProducer.send(HornetQMessageProducer.java:186)

I don't have any idea what this error message means -- I create producer from the session from the ConnectionFactory, a producer (with a specific destination) from the session, and then send a message with the producer. I don't know why it says I'm trying to send a message to something other than the default. It's possible that several threads are hitting the code that sends messages at once, but everything is a local variable, so there shouldn't be any crossing of wires.

These aren't new messages with the changes I just made. It's an intermittent issue with WF9 that occurs while the system is coming up. Once everything is deployed, the errors don't occur anymore.

28. Re: Resource adapter JMS lookups failing

jbertram Jun 5, 2015 3:25 PM (in response to jewellgm)

Got a reproducer yet I can take a look at?

29. Re: Resource adapter JMS lookups failing

jewellgm Jun 5, 2015 7:48 PM (in response to jbertram)

Justin,

Unfortunately, I don't. I had been trying to build something from the ground up to reproduce the problem, but was not having any luck. Maybe if I go in reverse, and just try to pare everything down to a minimum in our existing software, I can do it. I have to really strip all data types and logic out, otherwise I wouldn't be allowed to post it. I'll try when I'm back in the office next week.