High CPU usage Jboss 7.1.1-Final

[manishdevraj]

I have deployed application on Jboss 7.1.1-Final and since couple days I am experiencing heavy CPU load upto 180%. This usage is higher both when idle and when application is running and getting accessed. The load on system is not higher.

Application is deployed in standalone non-cluster mode. It is running behind Apache mod_jk cluster (SSL). Highlights of functionality are -

1. Hbase connection

2. Infinispan cache

3. HornetQ

4. Kafka Producer and Consumer

5. OSGI bundles

6. Spring modules

7. RPC calls

Attached is thread dump to look at

cat /proc/cpuinfo

processor          : 0

vendor_id          : GenuineIntel

cpu family          : 6

model                    : 44

model name          : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz

stepping          : 2

cpu MHz                    : 2659.999

cache size          : 12288 KB

fpu                    : yes

fpu_exception          : yes

cpuid level          : 11

wp                    : yes

flags                    : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat dts

bogomips          : 5319.99

clflush size          : 64

cache_alignment          : 64

address sizes          : 40 bits physical, 48 bits virtual

power management:

processor          : 1

vendor_id          : GenuineIntel

cpu family          : 6

model                    : 44

model name          : Intel(R) Xeon(R) CPU           X5650  @ 2.67GHz

stepping          : 2

cpu MHz                    : 2659.999

cache size          : 12288 KB

fpu                    : yes

fpu_exception          : yes

cpuid level          : 11

wp                    : yes

flags                    : fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm ida arat dts

bogomips          : 5319.99

clflush size          : 64

cache_alignment          : 64

address sizes          : 40 bits physical, 48 bits virtual

power management:

cat /proc/meminfo

MemTotal:        3925164 kB

MemFree:           99040 kB

Buffers:             280 kB

Cached:            12152 kB

SwapCached:       503896 kB

Active:          1221652 kB

Inactive:         465000 kB

Active(anon):    1216184 kB

Inactive(anon):   457988 kB

Active(file):       5468 kB

Inactive(file):     7012 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:       6160376 kB

SwapFree:        4444484 kB

Dirty:                16 kB

Writeback:            64 kB

AnonPages:       1632520 kB

Mapped:             9036 kB

Shmem:                 8 kB

Slab:              77684 kB

SReclaimable:      18288 kB

SUnreclaim:        59396 kB

KernelStack:        3784 kB

PageTables:        23212 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:     8122956 kB

Committed_AS:    3580484 kB

VmallocTotal:   34359738367 kB

VmallocUsed:      282780 kB

VmallocChunk:   34359445628 kB

HardwareCorrupted:     0 kB

AnonHugePages:     14336 kB

HugePages_Total:       0

HugePages_Free:        0

HugePages_Rsvd:        0

HugePages_Surp:        0

Hugepagesize:       2048 kB

DirectMap4k:       10240 kB

DirectMap2M:     4184064 kB