JBoss Shutdown Problems - Communication Exception

[alexires]

I know this has been done to death, but I've read all the discussions mentioning the problem, and none of them have helped.

I'm having a problem shutting down my jboss install (OpenKM). The VM runs fine and is started via init.d which runs run.sh -b 0.0.0.0. When I use the provided shutdown.sh (using ./shutdown.sh -s localhost:1099 -S), I get this error:

[code]

Exception in thread "main" javax.naming.CommunicationException: Could not obtain connection to any of these urls: localhost:1099 [Root exception is javax.naming.CommunicationException: Failed to connect to server localhost:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server localhost:1099 [Root exception is java.net.SocketException: Network is unreachable]]]

        at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1562)

        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:634)

        at org.jnp.interfaces.NamingContext.lookup(NamingContext.java:627)

        at javax.naming.InitialContext.lookup(InitialContext.java:392)

        at org.jboss.Shutdown.main(Shutdown.java:214)

Caused by: javax.naming.CommunicationException: Failed to connect to server localhost:1099 [Root exception is javax.naming.ServiceUnavailableException: Failed to connect to server localhost:1099 [Root exception is java.net.SocketException: Network is unreachable]]

        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:274)

        at org.jnp.interfaces.NamingContext.checkRef(NamingContext.java:1533)

        ... 4 more

Caused by: javax.naming.ServiceUnavailableException: Failed to connect to server localhost:1099 [Root exception is java.net.SocketException: Network is unreachable]

        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:248)

        ... 5 more

Caused by: java.net.SocketException: Network is unreachable

        at java.net.PlainSocketImpl.socketConnect(Native Method)

        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)

        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)

        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)

        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)

        at java.net.Socket.connect(Socket.java:529)

        at java.net.Socket.connect(Socket.java:478)

        at java.net.Socket.<init>(Socket.java:375)

        at java.net.Socket.<init>(Socket.java:276)

        at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:84)

        at org.jnp.interfaces.TimedSocketFactory.createSocket(TimedSocketFactory.java:77)

        at org.jnp.interfaces.NamingContext.getServer(NamingContext.java:244)

        ... 5 more[/code]

I've checked various files and the jmx console and made sure 1099 is the right port. Running lsof -i shows java as running 1098 BUT NOT 1099. netstat -an shows 0.0.0.0:1099 listening to 0.0.0.0:* (the same for 1098). jmx-console Naming service gives the port as 1099.

On jboss 4.2.3.GA, Ubuntu 10.10 x64.

[wdfink]

Sounds as a network issue.

Do you can ping localhost successful?

[alexires]

PING localhost.localdomain (127.0.0.1) 56(84) bytes of data.

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=1 ttl=64 time=0.026 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=2 ttl=64 time=0.030 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=3 ttl=64 time=0.041 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=4 ttl=64 time=0.031 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=5 ttl=64 time=0.040 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=6 ttl=64 time=0.028 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=7 ttl=64 time=0.034 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=8 ttl=64 time=0.040 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=9 ttl=64 time=0.036 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=10 ttl=64 time=0.032 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=11 ttl=64 time=0.029 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=12 ttl=64 time=0.031 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=13 ttl=64 time=0.035 ms

64 bytes from localhost.localdomain (127.0.0.1): icmp_req=14 ttl=64 time=0.027 ms

--- localhost.localdomain ping statistics ---

14 packets transmitted, 14 received, 0% packet loss, time 13002ms

rtt min/avg/max/mdev = 0.026/0.032/0.041/0.008 ms

Ping seems to work fine. That is from the terminal, and I can connect to the server from outside (I'm running OpenKM on jboss as we speak).

[wdfink]

does the script work without using "-s" oprion or use 127.0.0.1 or the real ip address?

[alexires]

./shutdown.sh -S does not work (same error), nor does ./shutdown.sh -s 127.0.0.1:1099 -S or ./shutdown.sh -s REALIP:1099 -S

All of those give the same error (just the reported server differs obviously).

[peterj]

Please post the results from running:

sudo netstat -anp | grep java

Also, try running this an post the response:

telnet localhost 1099

See this prior post on using telnet to debug hostname errors: https://community.jboss.org/thread/120995?tstart=0#409029

[alexires]

tcp	0	0 0.0.0.0:3873	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:48360	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:8009	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:1098	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:1099	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:8080	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:8083	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:60980	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:44667	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:4444	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:8093	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:4445	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 0.0.0.0:4446	0.0.0.0:*	LISTEN	5782/java
tcp	0	0 127.0.0.1:55279	127.0.0.1:2002	ESTABLISHED 5782/java
tcp	0	0 127.0.0.1:8009	127.0.0.1:48718	ESTABLISHED 5782/java
tcp	0	0 127.0.0.1:8009	127.0.0.1:54597	ESTABLISHED 5782/java
tcp	0	0 127.0.0.1:8009	127.0.0.1:36501	ESTABLISHED 5782/java
tcp	0	0 127.0.0.1:8009	127.0.0.1:36825	ESTABLISHED 5782/java
tcp	0	0 127.0.0.1:8009	127.0.0.1:44485	ESTABLISHED 5782/java
tcp	0	0 127.0.0.1:8009	127.0.0.1:42740	ESTABLISHED 5782/java
unix  2	[ ]	STREAM	CONNECTED	872326002 5782/java

telnet localhost 1099 -

Trying ::1...

Trying 127.0.0.1...

Connected to localhost.

Escape character is '^]'.

¬ísrjava.rmi.MarshalledObject|½ícü>IhashlocBytest[BobjBytesq~xpöuª¥ur[B¬óTàxp'¬íthttp://dplibrary:8083/q~q~uq~À¬ísr org.jnp.server.NamingServer_Stubxrjava.rmi.server.RemoteStubéþÜÉáexrjava.rmi.server.RemoteObjectÓa´

a3xpw4

      UnicastRef2       dplibraryJ®¬ã3,´&8!8äÍxConnection closed by foreign host.

I've checked that thread (dplibrary is the hostname of my server) and my /etc/hosts file looks very similar to the one you posted on 01NOV2006 -

::1 localhost.localdomain localhost

127.0.0.1 localhost.localdomain localhost

# Auto-generated hostname. Please do not remove this comment.

REALIP dplibrary

[peterj]

Things I would try and look into:

Try running: telnet dplibrary 1099

If that gets a connection error, then look at your firewall settings.

Are you running SELinux? If so, try disabling it and try again. If it works after disabling SELinux, then you need to figure out which SE:Linux setting is preventing the connection.

I have found this blog post to be helpful on configuring the hostname: http://jblevins.org/log/hostname

[alexires]

telnet dplibrary 1099 yields the same response as the previous post (garbled junk). I'm pretty sure I'm not running SELinux (it isn't marked as installed in aptitude).

[peterj]

Well, I'm out of ideas. Looks like you are going to have to run shutdown within a debugger to get to the root cause. You'll need the sources for the JVM runtime since that is where the error is happening. If you downloaded the Oracle JDK, it came with the sources (look for src.zip). If you installed the JDK from Ubuntu package manager, you'll have to dig to find out if it came with the sources or if that is a separate package.

[alexires]

Hmmm, alrighty then. Thanks for all your help anyway guys; I'll let you know how that goes once I work out what the hell I'm doing (linux newbie).

[jayashree.a2010]

Did you get any solution to this?