[dimitris]

It seems like a JDK issue, so can you try switching to an older/different jdk5, or even jdk1.4 and see if you still get this problem.

[rlgoldman]

Uninstalled Sun jdk1.5.0_09, and installed the _08 version.

I recompiled .tar.gz source (from SourceForge) under _08.

Same problems, as described (i.e. shutdowns from cluster-capable configs via "run -c all" show error msg for both "plain" and "ejb" versions obtained from the build/output directory).

I see that "ntdll.dll" is mentioned in the error message, so I acknowledge it could be something with my machine.

Forgive the long output, but here's the full message that the VM stored in a .log file:

#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
# EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x7c918fea, pid=2204, tid=3604
#
# Java VM: Java HotSpot(TM) Client VM (1.5.0_08-b03 mixed mode)
# Problematic frame:
# C [ntdll.dll+0x18fea]
#

--------------- T H R E A D ---------------

Current thread (0x270448c8): JavaThread "UDP.UcastReceiverThread" daemon [_thread_in_native, id=3604]

siginfo: ExceptionCode=0xc0000005, writing address 0x00000010

Registers:
EAX=0x00000000, EBX=0x00000000, ECX=0x00001034, EDX=0x27d464ec
ESP=0x28e6f044, EBP=0x28e6f0b8, ESI=0x27d464dc, EDI=0x00000000
EIP=0x7c918fea, EFLAGS=0x00010246

Top of Stack: (sp=0x28e6f044)
0x28e6f044: 27d13f20 00000000 00000000 00000000
0x28e6f054: 00001160 000a8ab8 000a8ab8 28e6f0d4
0x28e6f064: 71a61186 71a5334e 28e6f170 28e6f17c
0x28e6f074: 00000001 28e6f158 00000001 00000000
0x28e6f084: 00000020 28e6f994 28e6f9b8 28e6f118
0x28e6f094: 28e6f164 00000000 00000001 000a8ab8
0x28e6f0a4: 00001160 00000000 28e6f098 00000000
0x28e6f0b4: 00001034 28e6f158 7c90104b 00d464dc

Instructions: (pc=0x7c918fea)
0x7c918fda: 8b 46 10 3b c3 89 45 fc 0f 84 9e 00 00 00 8b 06
0x7c918fea: ff 40 10 8b 45 fc 83 e0 01 89 45 e8 8b 06 ff 40


Stack: [0x28e30000,0x28e70000), sp=0x28e6f044, free space=252k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C [ntdll.dll+0x18fea]
C [ntdll.dll+0x104b]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.net.PlainDatagramSocketImpl.receive0(Ljava/net/DatagramPacket;)V+0
j java.net.PlainDatagramSocketImpl.receive(Ljava/net/DatagramPacket;)V+2
j java.net.DatagramSocket.receive(Ljava/net/DatagramPacket;)V+243
j org.jgroups.protocols.UDP$UcastReceiver.run()V+56
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub

--------------- P R O C E S S ---------------

Java Threads: ( => current thread )
0x2799aab0 JavaThread "TimeScheduler.Thread" daemon [_thread_blocked, id=2800]
0x279bab70 JavaThread "IdleRemover" daemon [_thread_blocked, id=2572]
0x279d3df0 JavaThread "JBoss Shutdown Hook" daemon [_thread_blocked, id=3952]
0x0099fe70 JavaThread "SIGINT handler" daemon [_thread_blocked, id=3552]
0x278b28c8 JavaThread "TP-Processor4" daemon [_thread_blocked, id=3620]
0x278e0e80 JavaThread "http-0.0.0.0-8080-1" daemon [_thread_blocked, id=3568]
0x278dcdc8 JavaThread "http-0.0.0.0-8080" daemon [_thread_blocked, id=3544]
0x27a8ea18 JavaThread "JBossLifeThread" [_thread_blocked, id=3412]
0x27a656f8 JavaThread "JCA PoolFiller" [_thread_blocked, id=1784]
0x27573c08 JavaThread "Timer-4" daemon [_thread_blocked, id=2560]
0x279b6e80 JavaThread "HSQLDB Timer @1ceebfa" daemon [_thread_blocked, id=1976]
0x26e93380 JavaThread "PooledInvokerAcceptor#0-4445" [_thread_in_native, id=3756]
0x270ab9f0 JavaThread "RMI TCP Accept-4444" daemon [_thread_in_native, id=2792]
0x26ea4d00 JavaThread "Timer-3" [_thread_blocked, id=1352]
0x26ea4290 JavaThread "MERGE2.FindSubgroups thread" daemon [_thread_blocked, id=1268]
0x26e93df0 JavaThread "UDP mcast receiver" daemon [_thread_in_native, id=3700]
0x0098e638 JavaThread "UDP.UcastReceiverThread" daemon [_thread_in_native, id=3692]
0x00984e68 JavaThread "TimeScheduler.Thread" daemon [_thread_blocked, id=3644]
0x2757c808 JavaThread "PooledInvokerAcceptor#0-4446" [_thread_in_native, id=2960]
0x0097fe08 JavaThread "UDP.OutgoingPacketHandler thread" daemon [_thread_blocked, id=480]
=>0x270448c8 JavaThread "UDP.UcastReceiverThread" daemon [_thread_in_native, id=3604]
0x26ed8e30 JavaThread "RMI TCP Accept-4447" daemon [_thread_in_native, id=2464]
0x26ec08a0 JavaThread "JBoss System Threads(1)-4" daemon [_thread_in_native, id=2968]
0x26e7fa40 JavaThread "JBoss System Threads(1)-3" daemon [_thread_blocked, id=1920]
0x27045ad8 JavaThread "RMI TCP Accept-1101" daemon [_thread_in_native, id=252]
0x270efe30 JavaThread "MERGE2.FindSubgroups thread" daemon [_thread_blocked, id=4036]
0x27075958 JavaThread "AsynchViewChangeHandler Thread" [_thread_blocked, id=3320]
0x270753d0 JavaThread "AsynchKeyChangeHandler Thread" [_thread_blocked, id=2136]
0x270639e8 JavaThread "UDP mcast receiver" daemon [_thread_in_native, id=2948]
0x270633a8 JavaThread "UDP.UcastReceiverThread" daemon [_thread_in_native, id=2904]
0x270628e0 JavaThread "TimeScheduler.Thread" daemon [_thread_blocked, id=3800]
0x270626a8 JavaThread "SubscriptionWatchDog" [_thread_blocked, id=2148]
0x27064e28 JavaThread "JBoss System Threads(1)-2" daemon [_thread_in_native, id=4008]
0x27052940 JavaThread "GC Daemon" daemon [_thread_blocked, id=2556]
0x270546f8 JavaThread "RMI Reaper" [_thread_blocked, id=3608]
0x2705d170 JavaThread "Timer-2" daemon [_thread_blocked, id=3996]
0x27054480 JavaThread "RMI TCP Accept-1098" daemon [_thread_in_native, id=1112]
0x2705a6f8 JavaThread "MessageDispatcher up processing thread" daemon [_thread_blocked, id=4068]
0x27058f10 JavaThread "UpHandler (GMS)" daemon [_thread_blocked, id=1560]
0x27058988 JavaThread "DownHandler (GMS)" daemon [_thread_blocked, id=4084]
0x2704d708 JavaThread "UpHandler (UNICAST)" daemon [_thread_blocked, id=1988]
0x2704d580 JavaThread "UpHandler (FD)" daemon [_thread_blocked, id=3512]
0x270576a8 JavaThread "DownHandler (FD)" daemon [_thread_blocked, id=3516]
0x270512b8 JavaThread "DownHandler (UDP)" daemon [_thread_blocked, id=2344]
0x27056088 JavaThread "MessageDispatcher up processing thread" daemon [_thread_blocked, id=736]
0x27055700 JavaThread "MessageDispatcher up processing thread" daemon [_thread_blocked, id=2728]
0x26ffb2f8 JavaThread "UpHandler (STATE_TRANSFER)" daemon [_thread_blocked, id=444]
0x26ffa760 JavaThread "DownHandler (STATE_TRANSFER)" daemon [_thread_blocked, id=2992]
0x26ff9bc8 JavaThread "UpHandler (GMS)" daemon [_thread_blocked, id=2668]
0x26ff9030 JavaThread "DownHandler (GMS)" daemon [_thread_blocked, id=2716]
0x26ff84c0 JavaThread "UpHandler (FRAG)" daemon [_thread_blocked, id=3016]
0x26ff7970 JavaThread "DownHandler (FRAG)" daemon [_thread_blocked, id=3164]
0x26ff6e58 JavaThread "UpHandler (STABLE)" daemon [_thread_blocked, id=3168]
0x26ff6358 JavaThread "DownHandler (STABLE)" daemon [_thread_blocked, id=4056]
0x26ff5858 JavaThread "UpHandler (UNICAST)" daemon [_thread_blocked, id=3948]
0x26ff4db8 JavaThread "DownHandler (UNICAST)" daemon [_thread_blocked, id=2224]
0x26ff43a0 JavaThread "UpHandler (NAKACK)" daemon [_thread_blocked, id=612]
0x26ff36d8 JavaThread "DownHandler (NAKACK)" daemon [_thread_blocked, id=2196]
0x26ff2bd8 JavaThread "UpHandler (VERIFY_SUSPECT)" daemon [_thread_blocked, id=2212]
0x26ff20d8 JavaThread "DownHandler (VERIFY_SUSPECT)" daemon [_thread_blocked, id=1992]
0x26ff15d8 JavaThread "UpHandler (FD)" daemon [_thread_blocked, id=220]
0x26ff0ad8 JavaThread "DownHandler (FD)" daemon [_thread_blocked, id=2160]
0x26ff0150 JavaThread "UpHandler (PING)" daemon [_thread_blocked, id=2220]
0x26fe3e90 JavaThread "DownHandler (PING)" daemon [_thread_blocked, id=2084]
0x26fe1570 JavaThread "DownHandler (UDP)" daemon [_thread_blocked, id=2396]
0x26fc8290 JavaThread "JBossMQ Cache Reference Softner" daemon [_thread_blocked, id=2896]
0x26f58018 JavaThread "Timer-1" [_thread_blocked, id=3296]
0x26f59d00 JavaThread "MERGE2.FindSubgroups thread" daemon [_thread_blocked, id=1372]
0x26f2ab10 JavaThread "UDP mcast receiver" daemon [_thread_in_native, id=2468]
0x26f2a988 JavaThread "UDP.UcastReceiverThread" daemon [_thread_in_native, id=2988]
0x26f311d0 JavaThread "TimeScheduler.Thread" daemon [_thread_blocked, id=3084]
0x26f31048 JavaThread "MessageDispatcher up processing thread" daemon [_thread_blocked, id=2452]
0x26f34dc0 JavaThread "UpHandler (GMS)" daemon [_thread_blocked, id=3424]
0x26f02c70 JavaThread "DownHandler (GMS)" daemon [_thread_blocked, id=3380]
0x26f02ae8 JavaThread "UpHandler (UNICAST)" daemon [_thread_blocked, id=2776]
0x26f18618 JavaThread "UpHandler (FD)" daemon [_thread_blocked, id=3864]
0x26f18490 JavaThread "DownHandler (FD)" daemon [_thread_blocked, id=3404]
0x26f0e3d0 JavaThread "DownHandler (UDP)" daemon [_thread_blocked, id=2804]
0x27691b98 JavaThread "TimeoutFactory-0" daemon [_thread_blocked, id=4040]
0x27755670 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=3368]
0x2758c120 JavaThread "SocketServerInvoker#0-3873" [_thread_in_native, id=912]
0x273dce10 JavaThread "Timer-0" daemon [_thread_blocked, id=2856]
0x00037710 JavaThread "DestroyJavaVM" [_thread_blocked, id=3096]
0x009780b8 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3040]
0x00976b48 JavaThread "CompilerThread0" daemon [_thread_blocked, id=3060]
0x00975e98 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3080]
0x0096cf80 JavaThread "Finalizer" daemon [_thread_blocked, id=4048]
0x0096bb18 JavaThread "Reference Handler" daemon [_thread_blocked, id=2736]

Other Threads:
0x00969068 VMThread [id=3088]
0x00979298 WatcherThread [id=3580]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
def new generation total 9216K, used 3776K [0x02a20000, 0x03420000, 0x05180000)
eden space 8192K, 46% used [0x02a20000, 0x02dd00b8, 0x03220000)
from space 1024K, 0% used [0x03220000, 0x03220000, 0x03320000)
to space 1024K, 0% used [0x03320000, 0x03320000, 0x03420000)
tenured generation total 121024K, used 23034K [0x05180000, 0x0c7b0000, 0x22a20000)
the space 121024K, 19% used [0x05180000, 0x067fe8e0, 0x067fea00, 0x0c7b0000)
compacting perm gen total 22016K, used 21795K [0x22a20000, 0x23fa0000, 0x26a20000)
the space 22016K, 98% used [0x22a20000, 0x23f68f20, 0x23f69000, 0x23fa0000)
No shared spaces configured.

Dynamic libraries:
0x00400000 - 0x0040d000 C:\Java\jdk1.5.0_08\bin\java.exe
0x7c900000 - 0x7c9b0000 C:\WINDOWS\system32\ntdll.dll
0x7c800000 - 0x7c8f4000 C:\WINDOWS\system32\kernel32.dll
0x77dd0000 - 0x77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
0x77e70000 - 0x77f01000 C:\WINDOWS\system32\RPCRT4.dll
0x77c10000 - 0x77c68000 C:\WINDOWS\system32\MSVCRT.dll
0x6d730000 - 0x6d8cb000 C:\Java\jdk1.5.0_08\jre\bin\client\jvm.dll
0x77d40000 - 0x77dd0000 C:\WINDOWS\system32\USER32.dll
0x77f10000 - 0x77f57000 C:\WINDOWS\system32\GDI32.dll
0x76b40000 - 0x76b6d000 C:\WINDOWS\system32\WINMM.dll
0x6d2f0000 - 0x6d2f8000 C:\Java\jdk1.5.0_08\jre\bin\hpi.dll
0x76bf0000 - 0x76bfb000 C:\WINDOWS\system32\PSAPI.DLL
0x6d700000 - 0x6d70c000 C:\Java\jdk1.5.0_08\jre\bin\verify.dll
0x6d370000 - 0x6d38d000 C:\Java\jdk1.5.0_08\jre\bin\java.dll
0x6d720000 - 0x6d72f000 C:\Java\jdk1.5.0_08\jre\bin\zip.dll
0x6d530000 - 0x6d543000 C:\Java\jdk1.5.0_08\jre\bin\net.dll
0x71ab0000 - 0x71ac7000 C:\WINDOWS\system32\WS2_32.dll
0x71aa0000 - 0x71aa8000 C:\WINDOWS\system32\WS2HELP.dll
0x71a50000 - 0x71a8f000 C:\WINDOWS\System32\mswsock.dll
0x76f20000 - 0x76f47000 C:\WINDOWS\system32\DNSAPI.dll
0x76fb0000 - 0x76fb8000 C:\WINDOWS\System32\winrnr.dll
0x76f60000 - 0x76f8c000 C:\WINDOWS\system32\WLDAP32.dll
0x76fc0000 - 0x76fc6000 C:\WINDOWS\system32\rasadhlp.dll
0x6d520000 - 0x6d528000 C:\Java\jdk1.5.0_08\jre\bin\management.dll
0x27cc0000 - 0x27d06000 C:\WINDOWS\system32\imon.dll
0x71ad0000 - 0x71ad9000 C:\WINDOWS\system32\WSOCK32.dll
0x77690000 - 0x776b1000 C:\WINDOWS\system32\NTMARTA.DLL
0x774e0000 - 0x7761d000 C:\WINDOWS\system32\ole32.dll
0x71bf0000 - 0x71c03000 C:\WINDOWS\system32\SAMLIB.dll
0x662b0000 - 0x66308000 C:\WINDOWS\system32\hnetcfg.dll
0x71a90000 - 0x71a98000 C:\WINDOWS\System32\wshtcpip.dll
0x289c0000 - 0x289e8000 C:\WINDOWS\system32\rsaenh.dll
0x769c0000 - 0x76a73000 C:\WINDOWS\system32\USERENV.dll
0x5b860000 - 0x5b8b4000 C:\WINDOWS\system32\netapi32.dll
0x6d550000 - 0x6d559000 C:\Java\jdk1.5.0_08\jre\bin\nio.dll

VM Arguments:
jvm_args: -Dprogram.name=run.bat -Xms128m -Xmx512m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Djava.endorsed.dirs=C:\ent\container\test\jboss-4.0.4.GA\bin\\..\lib\endorsed
java_command: org.jboss.Main -c all
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=C:\Java\jdk1.5.0_08
JRE_HOME=C:\Java\jre1.5.0_08
CLASSPATH=.;C:\Java\jdk1.5.0_8\lib\tools.jar
PATH=C:\ent\tools\Perl\bin\;C:\ent\rdbms\OracleXE\app\oracle\product\10.2.0\server\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Microsoft SQL Server\80\Tools\Binn\;C:\ent\ide\apache-ant-1.6.5\bin;C:\ent\container\jboss-4.0.4.GA\bin;C:\Java\jdk1.5.0_08\bin;C:\Program Files\Common Files\GTK\2.0\bin;C:\Program Files\Subversion\bin;c:\Program Files\Microsoft SQL Server\90\Tools\binn\;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Common Files\Roxio Shared\DLLShared;C:\ent\ide\netbeans-5.5rc2\uml3\modules\DoorsIntegrationFiles\modules\bin;C:\ent\tools\UnixTools;C:\Program Files\VMFX\OpenLibraries\lib\release\;C:\Program Files\VMFX\OpenLibraries\lib\debug\;C:\ent\tools\OpenSSL\bin;;C:\PROGRA~1\COMMON~1\MUVEET~1\030625
USERNAME=Richard
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 9 Stepping 5, GenuineIntel



--------------- S Y S T E M ---------------

OS: Windows XP Build 2600 Service Pack 2

CPU:total 1 (cores per cpu 1, threads per core 1) family 6 model 9 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2

Memory: 4k page, physical 2096360k(1465704k free), swap 4052576k(3560892k free)

vm_info: Java HotSpot(TM) Client VM (1.5.0_08-b03) for windows-x86, built on Jul 26 2006 01:10:50 by "java_re" with MS VC++ 6.0

[dimitris]

The error seems to be in the JDK code PlainDatagramSocketImpl code anyway. I don't know if there is something to do there, other than report the bug to Sun.

I noticed also this looks like a jboss 4.0.4.GA installation?

[rlgoldman]

Yes, it was from 4.0.4.GA, after fresh .jar installation for ejb3-clustered config, which I did out of curiousity.

The 4.0.4.GA shutdown "crash" only happened once (it came down cleanly on a subsequent shutdown). 4.0.5.GA has given me consistent end-of-shutdown failures (when stopping a clustered config).

Thanks, Dimitris.