5 Replies Latest reply on Oct 30, 2009 9:14 PM by kkoster

lookupLocaly failing

kkoster Oct 27, 2009 1:55 PM

I am attempting to deploy an EAR that contains annotated EJB3 beans. I have already done this on several instances of JBoss, but this one is causing me problems. I am getting the following exception.

javax.naming.NamingException: Could not dereference object [Root exception is java.lang.RuntimeException: Exception while trying to locate proxy factory in JNDI, at key ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR]

I have checked the JNDI tree in the web-console using JNDIView and can see the name that I am looking up (i.e. kdr/remote/UserContextManagement) registered with what appear to the the appropriate bindings.

I turned up the logging to trace on org.jboss.ha.jndi and got the following output:

2009-10-27 13:38:56,522 TRACE [org.jboss.ha.jndi.HANamingService] (JBoss System Threads(1)-4) Accepted bootstrap client: Socket[addr=/192.168.11.3,port=51953,localport=1100]
2009-10-27 13:38:57,086 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(4)-192.168.11.3) lookup, name=kdr/remote/UserContextManagement
2009-10-27 13:38:57,087 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) lookupLocally, name=kdr/remote/UserContextManagement
2009-10-27 13:38:57,124 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(4)-192.168.11.3) lookup, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
2009-10-27 13:38:57,125 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) lookupLocally, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
2009-10-27 13:38:57,125 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) lookupLocally failed, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
javax.naming.NameNotFoundException: ProxyFactory not bound
 at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
 at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
 at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
 at org.jnp.server.NamingServer.lookup(NamingServer.java:396)
 at org.jboss.ha.jndi.HAJNDI.lookupLocally(HAJNDI.java:131)
 at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:202)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.ha.framework.server.HARMIServerImpl.invoke(HARMIServerImpl.java:209)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
 at sun.rmi.transport.Transport$1.run(Transport.java:159)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)
2009-10-27 13:38:57,125 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) calling lookupLocally(ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR) on HAJNDI cluster
2009-10-27 13:38:57,132 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(4)-192.168.11.3) Returned results size: 0

It looks like an internal caching issue of the proxy. This is going a bit deeper than I can spend time looking at right now. If anyone has any ideas what is going on here, I would be grateful for some suggestions.

Karl

1. Re: lookupLocaly failing

kkoster Oct 27, 2009 5:53 PM (in response to kkoster)

Sorry, forgot to mention this is on JBoss 5.1.0.GA
Actions

2. Re: lookupLocaly failing

kkoster Oct 27, 2009 7:54 PM (in response to kkoster)

Just to add more detail. I turned up the logging on org.jnp.server to TRACE (it produces a lot of output) and found this in the log:

2009-10-27 19:09:23,284 TRACE [org.jnp.server.NamingServer] (main) bind UserContextManagement=Reference Class Name: Proxy for: org.kdr.security.user.UserContextManagementRemote
Type: ProxyFactoryKey
Content: ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
Type: EJB Container Name
Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3
Type: Proxy Factory is Local
Content: false
Type: Remoting Host URL
Content: socket://GFL1:3873/?
Type: Remote Business Interface
Content: org.kdr.security.user.UserContextManagementRemote
Type: Cluster Partition Name
Content: KDR
Type: Cluster Proxy Factory Load Balance Policy
Content: org.jboss.ha.client.loadbalance.RoundRobin
Type: Cluster Family Name
Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3socketKDR
Type: Cluster Target Remoting Host URL
Content: socket://GFL1:3873/?
, Proxy for: org.kdr.security.user.UserContextManagementRemote

2009-10-27 19:09:23,285 TRACE [org.jnp.server.NamingServer] (main) org.jnp.server.NamingServer@66bc8644, setBinding: name=UserContextManagement, obj=Reference Class Name: Proxy for: org.kdr.security.user.UserContextManagementRemote
Type: ProxyFactoryKey
Content: ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
Type: EJB Container Name
Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3
Type: Proxy Factory is Local
Content: false
Type: Remoting Host URL
Content: socket://GFL1:3873/?
Type: Remote Business Interface
Content: org.kdr.security.user.UserContextManagementRemote
Type: Cluster Partition Name
Content: KDR
Type: Cluster Proxy Factory Load Balance Policy
Content: org.jboss.ha.client.loadbalance.RoundRobin
Type: Cluster Family Name
Content: jboss.j2ee:ear=risk-server.ear,jar=risk-ejb.jar,name=UserContextManagementBean,service=EJB3socketKDR
Type: Cluster Target Remoting Host URL
Content: socket://GFL1:3873/?
, className=Proxy for: org.kdr.security.user.UserContextManagementRemote

This looks like the been and the relevant interfaces/classes/names are being parsed out correctly. I later see them in what looks like the existing bindings with the name server when new artifacts are being bound. Example:

2009-10-27 19:09:23,469 TRACE [org.jnp.server.NamingServer] (main) org.jnp.server.NamingServer@7841514b, No binding for: BusinessCalendarManagementBean in context risk-server, bindings:
EnvironmentObjectSourceManagementBean=org.jnp.server.NamingServer@422f4736
ParameterConstructorManagementBean=org.jnp.server.NamingServer@5711297a
EnvironmentObjectFactoryContextManagementBean=org.jnp.server.NamingServer@550c45fe
InstrumentEnvironmentMatcherManagementBean=org.jnp.server.NamingServer@3fbd15da
TradeManagementBean=org.jnp.server.NamingServer@69d7135a
RoleContextManagementBean=org.jnp.server.NamingServer@ebab1b6
PerturbationRuleManagementBean=org.jnp.server.NamingServer@1d89e611
TranslationSourceManagementBean=org.jnp.server.NamingServer@553d70ce
SpotCurveHistoryStateManagementBean=org.jnp.server.NamingServer@780bfb81
TradeMatchStateManagementBean=org.jnp.server.NamingServer@27f7ee30
TradeMatchContextManagementBean=org.jnp.server.NamingServer@565de293
ObjectStateManagementBean=org.jnp.server.NamingServer@3fa472bf
AggregatorNameManagementBean=org.jnp.server.NamingServer@3822ee3c
TradeSettlementApplyRuleManagementBean=org.jnp.server.NamingServer@435f7db5
EnvironmentObjectFactoryConfigurationManagementBean=org.jnp.server.NamingServer@3f834f84
TradeReaderBuilderManagementBean=org.jnp.server.NamingServer@37f0d3ce
ObjectTypeManagementBean=org.jnp.server.NamingServer@577c76cd
DateRangeToDayRuleManagementBean=org.jnp.server.NamingServer@423425c1
TradeLinkTypeManagementBean=org.jnp.server.NamingServer@221c02f5
TradeMatchRuleManagementBean=org.jnp.server.NamingServer@163542ef
ValuatorManagementBean=org.jnp.server.NamingServer@3e6df2b0
TradeSettlementRuleManagementBean=org.jnp.server.NamingServer@1b9f1a8b
UnitManagementBean=org.jnp.server.NamingServer@20e38957
EnvironmentObjectFactoryManagementBean=org.jnp.server.NamingServer@4fc74c82
ModelTypeManagementBean=org.jnp.server.NamingServer@7a70c042
EnvironmentObjectTypeManagementBean=org.jnp.server.NamingServer@7d065976
ValuationAggregatorSpecificationManagementBean=org.jnp.server.NamingServer@6b091424
PerturbationContextManagementBean=org.jnp.server.NamingServer@5622ea37
InstrumentTypeManagementBean=org.jnp.server.NamingServer@44b89dd0
CashTypeManagementBean=org.jnp.server.NamingServer@6811c805
TradeFeeRuleContextManagementBean=org.jnp.server.NamingServer@361227f
RoleManagementBean=org.jnp.server.NamingServer@1884bb5
ObjectTypeStateManagementBean=org.jnp.server.NamingServer@dd0ca4f
TradeMatchManagementBean=org.jnp.server.NamingServer@3a135e2a
CurveStrategyContextManagementBean=org.jnp.server.NamingServer@60648df2
SynchWorkerManagementBean=org.jnp.server.NamingServer@61df79fe
TradeLinkStateManagementBean=org.jnp.server.NamingServer@1523db0
TradeFeeApplyRuleManagementBean=org.jnp.server.NamingServer@793044e
TradeFeeRuleManagementBean=org.jnp.server.NamingServer@e5ac92f
LocationManagementBean=org.jnp.server.NamingServer@104c2c06
TradeFeeSuppressionRuleManagementBean=org.jnp.server.NamingServer@188d7d8b
JMSEmitterBean=org.jnp.server.NamingServer@840509b
ValuatorContextManagementBean=org.jnp.server.NamingServer@6a562541
TradeFeeContextManagementBean=org.jnp.server.NamingServer@4dd2c7f5
ValuationJobSpecificationManagementBean=org.jnp.server.NamingServer@3ae6f61c
BucketerManagementBean=org.jnp.server.NamingServer@3ea6da02
TradeSettlementRuleContextManagementBean=org.jnp.server.NamingServer@6ba964d4
ValueTypeManagementBean=org.jnp.server.NamingServer@6dc4a8dd
ScenarioManagementBean=org.jnp.server.NamingServer@19dad138
UnitConvertManagementBean=org.jnp.server.NamingServer@2cafed2f
PerturbationRuleContextManagementBean=org.jnp.server.NamingServer@1d836fe
TradeSourceManagementBean=org.jnp.server.NamingServer@cf72664
HolidayTableManagementBean=org.jnp.server.NamingServer@77df9558
TranslationManagementBean=org.jnp.server.NamingServer@2a9d704c
PerturbationApplyRuleManagementBean=org.jnp.server.NamingServer@2486e1cd
InstrumentTemplateManagementBean=org.jnp.server.NamingServer@5c31d43
AggregatorExtractorManagementBean=org.jnp.server.NamingServer@18979865
DayCountManagementBean=org.jnp.server.NamingServer@44130dba
CurveStrategyManagementBean=org.jnp.server.NamingServer@204d57a2
TranslationTypeManagementBean=org.jnp.server.NamingServer@260f204b
PortfolioManagementBean=org.jnp.server.NamingServer@7bc08345
KeywordRendererManagementBean=org.jnp.server.NamingServer@6c9de34c
TradeSettlementContextManagementBean=org.jnp.server.NamingServer@4807f3e2
TimeContextManagementBean=org.jnp.server.NamingServer@54e5ebc5
DateRangeToDateRangeRuleManagementBean=org.jnp.server.NamingServer@744f439d
UserContextManagementBean=org.jnp.server.NamingServer@65821e23
GroupContextManagementBean=org.jnp.server.NamingServer@78d96b4b
ProductManagementBean=org.jnp.server.NamingServer@5b19393d
ShareClassManagementBean=org.jnp.server.NamingServer@40b96b99

But when the remote application makes a JNDI lookup the following details emerge:

2009-10-27 19:11:50,589 TRACE [org.jboss.ha.jndi.HANamingService] (JBoss System Threads(1)-4) Accepted bootstrap client: Socket[addr=/192.168.11.3,port=57934,localport=1100]
2009-10-27 19:11:51,619 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(3)-192.168.11.3) lookup, name=kdr/remote/UserContextManagement
2009-10-27 19:11:51,620 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookupLocally, name=kdr/remote/UserContextManagement
2009-10-27 19:11:51,786 TRACE [org.jboss.ha.jndi.impl.jbc.JBossCacheDistributedTreeManager] (RMI TCP Connection(3)-192.168.11.3) lookup, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
2009-10-27 19:11:51,786 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookupLocally, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
2009-10-27 19:11:51,786 TRACE [org.jnp.server.NamingServer] (RMI TCP Connection(3)-192.168.11.3) org.jnp.server.NamingServer@7e628e42, No binding for: ProxyFactory in context , bindings:
UserTransactionSessionFactory=org.jnp.interfaces.MarshalledValuePair@1951dea9
risk-server=org.jnp.server.NamingServer@7841514b
UUIDKeyGeneratorFactory=org.jnp.interfaces.MarshalledValuePair@36c2b1d3
HAPartition=org.jnp.server.NamingServer@1206cda0
SecureManagementView=org.jnp.server.NamingServer@2113cfd1
SecureDeploymentManager=org.jnp.server.NamingServer@1624bdb0
HiLoKeyGeneratorFactory=org.jnp.interfaces.MarshalledValuePair@435b92dd
XAConnectionFactory=org.jnp.interfaces.MarshalledValuePair@72381307
topic=org.jnp.server.NamingServer@32ae8db4
kdr=org.jnp.server.NamingServer@4f840e24
ClusteredConnectionFactory=org.jnp.interfaces.MarshalledValuePair@7cb9682a
ProfileService=org.jnp.interfaces.MarshalledValuePair@10f1abd1
HASessionState=org.jnp.server.NamingServer@1d7275cd
SecureProfileService=org.jnp.server.NamingServer@4127e183
queue=org.jnp.server.NamingServer@56d6d6c6
ClusteredXAConnectionFactory=org.jnp.interfaces.MarshalledValuePair@7065dd5e
UserTransaction=Reference Class Name: org.jboss.tm.usertx.client.ClientUserTransaction

invokers=org.jnp.server.NamingServer@5e0fae9
ConnectionFactory=org.jnp.interfaces.MarshalledValuePair@536e5b7b
jmx=org.jnp.server.NamingServer@755392d5
JAXR=org.jnp.interfaces.MarshalledValuePair@5edbde1
persistence.unit:unitName=risk-server.ear=org.jnp.server.NamingServer@14dd3d
TomcatAuthenticators=org.jnp.interfaces.MarshalledValuePair@40b29eda
console=org.jnp.server.NamingServer@20550f84

2009-10-27 19:11:51,787 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookupLocally failed, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR
javax.naming.NameNotFoundException: ProxyFactory not bound
 at org.jnp.server.NamingServer.getBinding(NamingServer.java:771)
 at org.jnp.server.NamingServer.getBinding(NamingServer.java:779)
 at org.jnp.server.NamingServer.getObject(NamingServer.java:785)
 at org.jnp.server.NamingServer.lookup(NamingServer.java:396)
 at org.jboss.ha.jndi.HAJNDI.lookupLocally(HAJNDI.java:131)
 at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:202)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.jboss.ha.framework.server.HARMIServerImpl.invoke(HARMIServerImpl.java:209)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305)
 at sun.rmi.transport.Transport$1.run(Transport.java:159)
 at java.security.AccessController.doPrivileged(Native Method)
 at sun.rmi.transport.Transport.serviceCall(Transport.java:155)
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)
2009-10-27 19:11:51,787 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) calling lookupLocally(ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManagement/KDR) on HAJNDI cluster
2009-10-27 19:11:51,791 TRACE [org.jboss.ha.framework.interfaces.HAPartition.KDR] (RMI TCP Connection(3)-192.168.11.3) callMethodOnCluster(true), objName=HAJNDI, methodName=lookupLocally, members=[]
2009-10-27 19:11:51,791 TRACE [org.jboss.ha.framework.server.ClusterPartition$RpcHandler] (RMI TCP Connection(3)-192.168.11.3) destination list of HAJNDI.lookupLocally() is empty: no need to send message
2009-10-27 19:11:51,791 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) Returned results size: 0

It looks like the only bindings are the root of the name tree. I have this application running without any problems on Windows but when I moved to Linux, this problem arose. The annotations on this bean are:

@Stateless
@Clustered(loadBalancePolicy = "org.jboss.ha.client.loadbalance.RandomRobin", partition = "KDR")
@Local( { UserContextManagementLocal.class })
@LocalBinding(jndiBinding = "kdr/local/UserContextManagement")
@Remote( { UserContextManagementRemote.class })
@RemoteBinding(jndiBinding = "kdr/remote/UserContextManagement")
public class UserContextManagementBean implements UserContextManagementLocal,
 UserContextManagementRemote, Serializable {

I am stumped as to what is going on.

3. Re: lookupLocaly failing

jaikiran Oct 28, 2009 3:20 AM (in response to kkoster)

at org.jboss.ha.jndi.HAJNDI.lookupLocally(HAJNDI.java:131)
at org.jboss.ha.jndi.HAJNDI.lookup(HAJNDI.java:202)

Are you running the "all" server configuration?

javax.naming.NamingException: Could not dereference object [Root exception is java.lang.RuntimeExcep
tion: Exception while trying to locate proxy factory in JNDI, at key ProxyFactory/risk-server/UserCo
ntextManagementBean/kdr/remote/UserContextManagement/KDR]

Please post this entire exception stacktrace.
Actions
4. Re: lookupLocaly failing

kkoster Oct 30, 2009 9:10 PM (in response to kkoster)

I have copied all to a different name and am running that. I don't like to touch the baseline configurations so I have something to compare to if something goes wrong.
Actions
5. Re: lookupLocaly failing

kkoster Oct 30, 2009 9:14 PM (in response to kkoster)
The entire stack trace of the exception thrown is the next to last code posting and starts with
2009-10-27 19:11:51,787 TRACE [org.jboss.ha.jndi.HAJNDI] (RMI TCP Connection(3)-192.168.11.3) lookup Locally failed, name=ProxyFactory/risk-server/UserContextManagementBean/kdr/remote/UserContextManage ment/KDR javax.naming.NameNotFoundException: ProxyFactory not bound

I have additional TRACE level logging coming from the server but is larger than 1MB by the the JNDI lookup fails. I can send it, but I believe it will be too large to post.
Actions

Go to original post