Apache Crash / 1.2.0 Final / jboss 7.1.0
black_dev Jun 4, 2012 8:36 AMOk, my story is:
I have apache with mod_cluster 1.2.0 Final and two jboss AS 7.1.0 in cluster with sticky-session
I started all servers.
Apache have five proces thread ( 0.0 % cpu busy per proces)
Jboss about 2-5% cpu busy.
After some time ( one time more , one time less ) under production load apache spawn new thread but:
- this tread not serve a content ( nothing in logs )
- have 100% cpu busy
- I think this proces was zombie/or not function.
like This:
apache 10912 0.0 0.1 186288 3524 ? | Sl 11:28 0:00 /usr/sbin/httpd |
apache 10917 0.0 0.1 186284 3556 ? | Sl 11:28 0:00 /usr/sbin/httpd |
apache 10921 99.7 0.1 186148 3196 ? | Rl 11:29 17:24 /usr/sbin/httpd |
apache 10953 0.0 0.1 251820 3564 ? | Sl 11:30 0:00 /usr/sbin/httpd |
apache 10981 0.0 0.1 186284 3520 ? | Sl 11:33 0:00 /usr/sbin/httpd |
apache 10995 98.9 0.1 186148 3196 ? | Rl 11:44 2:27 /usr/sbin/httpd |
ok I make some debug thinks:
pstack <pid>
Thread 2 (Thread 0x7f7b5a835700 (LWP 15404)):
#0 0x00007f7b638928b3 in select () from /lib64/libc.so.6
#1 0x00007f7b63d83155 in apr_sleep () from /usr/lib64/libapr-1.so.0
#2 0x00007f7b5ad4e6fd in proxy_cluster_watchdog_func () from /etc/httpd/modules/mod_proxy_cluster.so
#3 0x00007f7b63b4b7f1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7b63899ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f7b652657e0 (LWP 15403)):
#0 0x00007f7b5ad5024e in remove_session_route () from /etc/httpd/modules/mod_proxy_cluster.so
#1 0x00007f7b5ad5086f in proxy_cluster_pre_request () from /etc/httpd/modules/mod_proxy_cluster.so
#2 0x00007f7b5c1a20d6 in proxy_run_pre_request () from /etc/httpd/modules/mod_proxy.so
#3 0x00007f7b5c1a9026 in ap_proxy_pre_request () from /etc/httpd/modules/mod_proxy.so
#4 0x00007f7b5c1a62c2 in ?? () from /etc/httpd/modules/mod_proxy.so
#5 0x00007f7b6529fb00 in ap_run_handler ()
#6 0x00007f7b652a33be in ap_invoke_handler ()
#7 0x00007f7b652aea30 in ap_process_request ()
#8 0x00007f7b652ab8f8 in ?? ()
#9 0x00007f7b652a7608 in ap_run_process_connection ()
#10 0x00007f7b652b3807 in ?? ()
#11 0x00007f7b652b3b1a in ?? ()
#12 0x00007f7b652b479c in ap_mpm_run ()
#13 0x00007f7b6528b900 in main ()
gdb -q -p <pid>
Attaching to process 14970
Reading symbols from /usr/sbin/httpd...(no debugging symbols found)...done.
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libpcre.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libpcre.so.0
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /usr/lib64/libaprutil-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libaprutil-1.so.0
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libexpat.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libexpat.so.1
Reading symbols from /lib64/libdb-4.7.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdb-4.7.so
Reading symbols from /usr/lib64/libapr-1.so.0...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libapr-1.so.0
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f7b5a835700 (LWP 14996)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libuuid.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libuuid.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
Reading symbols from /lib64/libldap-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libldap-2.4.so.2
Reading symbols from /lib64/liblber-2.4.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/liblber-2.4.so.2
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /usr/lib64/libssl3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl3.so
Reading symbols from /usr/lib64/libsmime3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsmime3.so
Reading symbols from /usr/lib64/libnss3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnss3.so
Reading symbols from /usr/lib64/libnssutil3.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libnssutil3.so
Reading symbols from /lib64/libplds4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplds4.so
Reading symbols from /lib64/libplc4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libplc4.so
Reading symbols from /lib64/libnspr4.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnspr4.so
Reading symbols from /usr/lib64/libsasl2.so.2...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libsasl2.so.2
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
...
...
...
Loaded symbols for /etc/httpd/modules/mod_substitute.so
Reading symbols from /etc/httpd/modules/mod_rewrite.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_rewrite.so
Reading symbols from /etc/httpd/modules/mod_proxy.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy.so
Reading symbols from /etc/httpd/modules/mod_proxy_ftp.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ftp.so
Reading symbols from /etc/httpd/modules/mod_proxy_http.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_http.so
Reading symbols from /etc/httpd/modules/mod_proxy_ajp.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_ajp.so
Reading symbols from /etc/httpd/modules/mod_proxy_connect.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_connect.so
Reading symbols from /etc/httpd/modules/mod_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cache.so
Reading symbols from /etc/httpd/modules/mod_suexec.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_suexec.so
Reading symbols from /etc/httpd/modules/mod_disk_cache.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_disk_cache.so
Reading symbols from /etc/httpd/modules/mod_cgi.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_cgi.so
Reading symbols from /etc/httpd/modules/mod_version.so...(no debugging symbols found)...done.
Loaded symbols for /etc/httpd/modules/mod_version.so
Reading symbols from /etc/httpd/modules/mod_slotmem.so...done.
Loaded symbols for /etc/httpd/modules/mod_slotmem.so
Reading symbols from /etc/httpd/modules/mod_manager.so...done.
Loaded symbols for /etc/httpd/modules/mod_manager.so
Reading symbols from /etc/httpd/modules/mod_proxy_cluster.so...done.
Loaded symbols for /etc/httpd/modules/mod_proxy_cluster.so
Reading symbols from /etc/httpd/modules/mod_advertise.so...done.
Loaded symbols for /etc/httpd/modules/mod_advertise.so
0x00007f7b5ad50250 in remove_session_route (r=0x7f7b67280cc8, name=0x7f7b67285f63 "jsessionid") at mod_proxy_cluster.c:2472
in mod_proxy_cluster.c
backtrace
Reading symbols from /etc/httpd/modules/mod_advertise.so...done.
Loaded symbols for /etc/httpd/modules/mod_advertise.so
Core was generated by `/usr/sbin/httpd'.
#0 0x00007f7b638928b3 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install httpd-2.2.15-15.el6.centos.1.x86_64
(gdb) backtrace
#0 0x00007f7b638928b3 in select () from /lib64/libc.so.6
#1 0x00007f7b63d83155 in apr_sleep () from /usr/lib64/libapr-1.so.0
#2 0x00007f7b5ad4e6fd in proxy_cluster_watchdog_func (thd=0x7f7b67271080, data=0x0) at mod_proxy_cluster.c:1783
#3 0x00007f7b63b4b7f1 in start_thread () from /lib64/libpthread.so.0
#4 0x00007f7b63899ccd in clone () from /lib64/libc.so.6
my system is:
uname -a
Linux jb-web-04 2.6.32-220.7.1.el6.x86_64 #1 SMP Wed Mar 7 00:52:02 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux
our apache + we add mod_cluster modul
httpd -V
Server version: Apache/2.2.15 (Unix)
Server built: Feb 13 2012 22:31:42
Server's Module Magic Number: 20051115:24
Server loaded: APR 1.3.9, APR-Util 1.3.9
Compiled using: APR 1.3.9, APR-Util 1.3.9
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
before crash - nothing bad in log files.
pls help... meybe somethink tips how to debug... or... change in system.