6 Replies Latest reply on Sep 25, 2003 8:43 AM by mudblood

    Kernel parameters? Or configuration? -

    mudblood Newbie

      Hi all, if someone could help me out with this problem I'm having (below) I swear I'll send you a thankyou note with lots of smiley faces attached :)

      The Problem:
      Here's the errors we get every few hours or so which result in jboss unable to accept new connections. The system load is always below 1 with the uptime command, and we never run out of memory:

      14:06:37,187 ERROR [STDERR] java.lang.OutOfMemoryError: unable to create new native thread
      14:06:37,188 ERROR [STDERR] at java.lang.Thread.start(Native Method)
      14:06:37,188 ERROR [STDERR] at org.mortbay.util.ThreadPool.newThread(ThreadPool.java:495)
      14:06:37,188 ERROR [STDERR] at org.mortbay.util.ThreadPool.access$700(ThreadPool.java:35)
      14:06:37,188 ERROR [STDERR] at org.mortbay.util.ThreadPool$JobRunner.run(ThreadPool.java:709)
      14:06:37,188 ERROR [STDERR] at java.lang.Thread.run(Thread.java:536)

      Here is my system info:
      Dual 2.4 gig Pentium4s with Hyper-threading
      2.4.20-18.9bigmem #1 SMP
      4 gigs of memory
      Redhat 8.0

      Here is my startup script's Java options:

      JAVA_OPTS="$JAVA_OPTS -Dprogram.name=$PROGNAME -Xmx1536m -Xms1536m -Xss1024k"
      (note - we've tried lowering Xss to 128k and ended up with stack overflows)

      Here are my kernel parameters:

      abi.fake_utsname = 0
      abi.trace = 0
      abi.defhandler_libcso = 68157441
      abi.defhandler_lcall7 = 68157441
      abi.defhandler_elf = 0
      abi.defhandler_coff = 117440515
      dev.raid.speed_limit_max = 10000
      dev.raid.speed_limit_min = 100
      dev.rtc.max-user-freq = 64
      debug.rpmarch =
      debug.kerneltype =
      net.unix.max_dgram_qlen = 10
      net.token-ring.rif_timeout = 60000
      net.ipv4.conf.eth1.arp_filter = 0
      net.ipv4.conf.eth1.tag = 0
      net.ipv4.conf.eth1.log_martians = 0
      net.ipv4.conf.eth1.bootp_relay = 0
      net.ipv4.conf.eth1.medium_id = 0
      net.ipv4.conf.eth1.proxy_arp = 0
      net.ipv4.conf.eth1.accept_source_route = 1
      net.ipv4.conf.eth1.send_redirects = 1
      net.ipv4.conf.eth1.rp_filter = 1
      net.ipv4.conf.eth1.shared_media = 1
      net.ipv4.conf.eth1.secure_redirects = 1
      net.ipv4.conf.eth1.accept_redirects = 1
      net.ipv4.conf.eth1.mc_forwarding = 0
      net.ipv4.conf.eth1.forwarding = 1
      net.ipv4.conf.eth0.arp_filter = 0
      net.ipv4.conf.eth0.tag = 0
      net.ipv4.conf.eth0.log_martians = 0
      net.ipv4.conf.eth0.bootp_relay = 0
      net.ipv4.conf.eth0.medium_id = 0
      net.ipv4.conf.eth0.proxy_arp = 0
      net.ipv4.conf.eth0.accept_source_route = 1
      net.ipv4.conf.eth0.send_redirects = 1
      net.ipv4.conf.eth0.rp_filter = 1
      net.ipv4.conf.eth0.shared_media = 1
      net.ipv4.conf.eth0.secure_redirects = 1
      net.ipv4.conf.eth0.accept_redirects = 1
      net.ipv4.conf.eth0.mc_forwarding = 0
      net.ipv4.conf.eth0.forwarding = 1
      net.ipv4.conf.lo.arp_filter = 0
      net.ipv4.conf.lo.tag = 0
      net.ipv4.conf.lo.log_martians = 0
      net.ipv4.conf.lo.bootp_relay = 0
      net.ipv4.conf.lo.medium_id = 0
      net.ipv4.conf.lo.proxy_arp = 0
      net.ipv4.conf.lo.accept_source_route = 1
      net.ipv4.conf.lo.send_redirects = 1
      net.ipv4.conf.lo.rp_filter = 1
      net.ipv4.conf.lo.shared_media = 1
      net.ipv4.conf.lo.secure_redirects = 1
      net.ipv4.conf.lo.accept_redirects = 1
      net.ipv4.conf.lo.mc_forwarding = 0
      net.ipv4.conf.lo.forwarding = 1
      net.ipv4.conf.default.arp_filter = 0
      net.ipv4.conf.default.tag = 0
      net.ipv4.conf.default.log_martians = 0
      net.ipv4.conf.default.bootp_relay = 0
      net.ipv4.conf.default.medium_id = 0
      net.ipv4.conf.default.proxy_arp = 0
      net.ipv4.conf.default.accept_source_route = 1
      net.ipv4.conf.default.send_redirects = 1
      net.ipv4.conf.default.rp_filter = 1
      net.ipv4.conf.default.shared_media = 1
      net.ipv4.conf.default.secure_redirects = 1
      net.ipv4.conf.default.accept_redirects = 1
      net.ipv4.conf.default.mc_forwarding = 0
      net.ipv4.conf.default.forwarding = 1
      net.ipv4.conf.all.arp_filter = 0
      net.ipv4.conf.all.tag = 0
      net.ipv4.conf.all.log_martians = 0
      net.ipv4.conf.all.bootp_relay = 0
      net.ipv4.conf.all.medium_id = 0
      net.ipv4.conf.all.proxy_arp = 0
      net.ipv4.conf.all.accept_source_route = 0
      net.ipv4.conf.all.send_redirects = 1
      net.ipv4.conf.all.rp_filter = 0
      net.ipv4.conf.all.shared_media = 1
      net.ipv4.conf.all.secure_redirects = 1
      net.ipv4.conf.all.accept_redirects = 0
      net.ipv4.conf.all.mc_forwarding = 0
      net.ipv4.conf.all.forwarding = 1
      net.ipv4.neigh.eth1.locktime = 100
      net.ipv4.neigh.eth1.proxy_delay = 80
      net.ipv4.neigh.eth1.anycast_delay = 100
      net.ipv4.neigh.eth1.proxy_qlen = 64
      net.ipv4.neigh.eth1.unres_qlen = 3
      net.ipv4.neigh.eth1.gc_stale_time = 60
      net.ipv4.neigh.eth1.delay_first_probe_time = 5
      net.ipv4.neigh.eth1.base_reachable_time = 30
      net.ipv4.neigh.eth1.retrans_time = 100
      net.ipv4.neigh.eth1.app_solicit = 0
      net.ipv4.neigh.eth1.ucast_solicit = 3
      net.ipv4.neigh.eth1.mcast_solicit = 3
      net.ipv4.neigh.eth0.locktime = 100
      net.ipv4.neigh.eth0.proxy_delay = 80
      net.ipv4.neigh.eth0.anycast_delay = 100
      net.ipv4.neigh.eth0.proxy_qlen = 64
      net.ipv4.neigh.eth0.unres_qlen = 3
      net.ipv4.neigh.eth0.gc_stale_time = 60
      net.ipv4.neigh.eth0.delay_first_probe_time = 5
      net.ipv4.neigh.eth0.base_reachable_time = 30
      net.ipv4.neigh.eth0.retrans_time = 100
      net.ipv4.neigh.eth0.app_solicit = 0
      net.ipv4.neigh.eth0.ucast_solicit = 3
      net.ipv4.neigh.eth0.mcast_solicit = 3
      net.ipv4.neigh.lo.locktime = 100
      net.ipv4.neigh.lo.proxy_delay = 80
      net.ipv4.neigh.lo.anycast_delay = 100
      net.ipv4.neigh.lo.proxy_qlen = 64
      net.ipv4.neigh.lo.unres_qlen = 3
      net.ipv4.neigh.lo.gc_stale_time = 60
      net.ipv4.neigh.lo.delay_first_probe_time = 5
      net.ipv4.neigh.lo.base_reachable_time = 30
      net.ipv4.neigh.lo.retrans_time = 100
      net.ipv4.neigh.lo.app_solicit = 0
      net.ipv4.neigh.lo.ucast_solicit = 3
      net.ipv4.neigh.lo.mcast_solicit = 3
      net.ipv4.neigh.default.gc_thresh3 = 1024
      net.ipv4.neigh.default.gc_thresh2 = 512
      net.ipv4.neigh.default.gc_thresh1 = 128
      net.ipv4.neigh.default.gc_interval = 30
      net.ipv4.neigh.default.locktime = 100
      net.ipv4.neigh.default.proxy_delay = 80
      net.ipv4.neigh.default.anycast_delay = 100
      net.ipv4.neigh.default.proxy_qlen = 64
      net.ipv4.neigh.default.unres_qlen = 3
      net.ipv4.neigh.default.gc_stale_time = 60
      net.ipv4.neigh.default.delay_first_probe_time = 5
      net.ipv4.neigh.default.base_reachable_time = 30
      net.ipv4.neigh.default.retrans_time = 100
      net.ipv4.neigh.default.app_solicit = 0
      net.ipv4.neigh.default.ucast_solicit = 3
      net.ipv4.neigh.default.mcast_solicit = 3
      net.ipv4.ipfrag_secret_interval = 600
      net.ipv4.tcp_low_latency = 0
      net.ipv4.tcp_frto = 0
      net.ipv4.tcp_tw_reuse = 0
      net.ipv4.icmp_ratemask = 6168
      net.ipv4.icmp_ratelimit = 100
      net.ipv4.tcp_adv_win_scale = 2
      net.ipv4.tcp_app_win = 31
      net.ipv4.tcp_rmem = 4096 87380 174760
      net.ipv4.tcp_wmem = 4096 16384 131072
      net.ipv4.tcp_mem = 195584 196096 196608
      net.ipv4.tcp_dsack = 1
      net.ipv4.tcp_ecn = 0
      net.ipv4.tcp_reordering = 3
      net.ipv4.tcp_fack = 1
      net.ipv4.tcp_orphan_retries = 0
      net.ipv4.inet_peer_gc_maxtime = 120
      net.ipv4.inet_peer_gc_mintime = 10
      net.ipv4.inet_peer_maxttl = 600
      net.ipv4.inet_peer_minttl = 120
      net.ipv4.inet_peer_threshold = 65664
      net.ipv4.igmp_max_memberships = 20
      net.ipv4.route.secret_interval = 600
      net.ipv4.route.min_adv_mss = 256
      net.ipv4.route.min_pmtu = 552
      net.ipv4.route.mtu_expires = 600
      net.ipv4.route.gc_elasticity = 8
      net.ipv4.route.error_burst = 500
      net.ipv4.route.error_cost = 100
      net.ipv4.route.redirect_silence = 2048
      net.ipv4.route.redirect_number = 9
      net.ipv4.route.redirect_load = 2
      net.ipv4.route.gc_interval = 60
      net.ipv4.route.gc_timeout = 300
      net.ipv4.route.gc_min_interval = 5
      net.ipv4.route.max_size = 1048576
      net.ipv4.route.gc_thresh = 65536
      net.ipv4.route.max_delay = 10
      net.ipv4.route.min_delay = 2
      net.ipv4.icmp_ignore_bogus_error_responses = 0
      net.ipv4.icmp_echo_ignore_broadcasts = 0
      net.ipv4.icmp_echo_ignore_all = 0
      net.ipv4.ip_local_port_range = 32768 61000
      net.ipv4.tcp_max_syn_backlog = 1024
      net.ipv4.tcp_rfc1337 = 0
      net.ipv4.tcp_stdurg = 0
      net.ipv4.tcp_abort_on_overflow = 0
      net.ipv4.tcp_tw_recycle = 0
      net.ipv4.tcp_syncookies = 0
      net.ipv4.tcp_fin_timeout = 60
      net.ipv4.tcp_retries2 = 15
      net.ipv4.tcp_retries1 = 3
      net.ipv4.tcp_keepalive_intvl = 75
      net.ipv4.tcp_keepalive_probes = 9
      net.ipv4.tcp_keepalive_time = 7200
      net.ipv4.ipfrag_time = 30
      net.ipv4.ip_dynaddr = 0
      net.ipv4.ipfrag_low_thresh = 196608
      net.ipv4.ipfrag_high_thresh = 262144
      net.ipv4.tcp_max_tw_buckets = 180000
      net.ipv4.tcp_max_orphans = 32768
      net.ipv4.tcp_synack_retries = 5
      net.ipv4.tcp_syn_retries = 5
      net.ipv4.ip_nonlocal_bind = 0
      net.ipv4.ip_no_pmtu_disc = 0
      net.ipv4.ip_autoconfig = 0
      net.ipv4.ip_default_ttl = 64
      net.ipv4.ip_forward = 1
      net.ipv4.tcp_retrans_collapse = 1
      net.ipv4.tcp_sack = 1
      net.ipv4.tcp_window_scaling = 1
      net.ipv4.tcp_timestamps = 1
      net.core.divert_version = 0.46
      net.core.hot_list_length = 128
      net.core.optmem_max = 10240
      net.core.message_burst = 50
      net.core.message_cost = 5
      net.core.mod_cong = 290
      net.core.lo_cong = 100
      net.core.no_cong = 20
      net.core.no_cong_thresh = 20
      net.core.netdev_max_backlog = 300
      net.core.dev_weight = 64
      net.core.rmem_default = 65535
      net.core.wmem_default = 65535
      net.core.rmem_max = 131071
      net.core.wmem_max = 131071
      vm.max_map_count = 65536
      vm.max-readahead = 31
      vm.min-readahead = 3
      vm.page-cluster = 3
      vm.pagetable_cache = 25 50
      vm.kswapd = 512 32 8
      vm.overcommit_memory = 0
      vm.bdflush = 30 500 0 0 500 3000 60 20 0
      kernel.overflowgid = 65534
      kernel.overflowuid = 65534
      kernel.random.uuid = 902f6c8d-6983-45a6-8c18-1bcfe473619a
      kernel.random.boot_id = 8f59d695-0ad3-46a1-b744-36b4ff3278fb
      kernel.random.write_wakeup_threshold = 128
      kernel.random.read_wakeup_threshold = 8
      kernel.random.entropy_avail = 4096
      kernel.random.poolsize = 512
      kernel.pid_max = 32768
      kernel.threads-max = 250000
      kernel.cad_pid = 1
      kernel.sysrq = 0
      kernel.sem = 256 32000 128 128
      kernel.msgmnb = 16384
      kernel.msgmni = 16
      kernel.msgmax = 8192
      kernel.shmmni = 4096
      kernel.shmall = 1073741824
      kernel.shmmax = 1073741824
      kernel.rtsig-max = 1024
      kernel.rtsig-nr = 1
      kernel.acct = 4 2 30
      kernel.hotplug = /sbin/hotplug
      kernel.modprobe = /sbin/modprobe
      kernel.printk = 6 4 1 7
      kernel.ctrl-alt-del = 0
      kernel.real-root-dev = 256
      kernel.cap-bound = -257
      kernel.tainted = 0
      kernel.core_pattern = core
      kernel.core_uses_pid = 1
      kernel.print_fatal_signals = 0
      kernel.panic = 0
      kernel.domainname = (none)
      kernel.hostname = nrsweb
      kernel.version = #1 SMP Thu May 29 06:41:01 EDT 2003
      kernel.osrelease = 2.4.20-18.9bigmem
      kernel.ostype = Linux
      fs.quota.syncs = 8
      fs.quota.free_dquots = 0
      fs.quota.allocated_dquots = 0
      fs.quota.cache_hits = 0
      fs.quota.writes = 0
      fs.quota.reads = 0
      fs.quota.drops = 0
      fs.quota.lookups = 0
      fs.lease-break-time = 45
      fs.dir-notify-enable = 1
      fs.leases-enable = 1
      fs.overflowgid = 65534
      fs.overflowuid = 65534
      fs.dentry-state = 94865 83143 45 0 0 0
      fs.file-max = 471859
      fs.file-nr = 1361 815 471859
      fs.inode-state = 88801 1260 0 0 0 0 0

        • 1. Re: Kernel parameters? Or configuration? -
          Juha Lindfors Master

          I've seen this before, try searching the forums or mailing lists archives if you can find an answer. I've forgotten the exact details.

          If my memory serves, you're running out of some resources at the OS level (unable to create a new thread) and the OutOfMemory is misleading (filedescriptors??).

          Again, check if you can find anything on google or the list archives or here in the forums.

          -- Juha

          • 2. Re: Kernel parameters? Or configuration? -
            Juha Lindfors Master

            I've seen this before, try searching the forums or mailing lists archives if you can find an answer. I've forgotten the exact details.

            If my memory serves, you're running out of some resources at the OS level (unable to create a new thread) and the OutOfMemory is misleading (filedescriptors??).

            Again, check if you can find anything on google or the list archives or here in the forums.

            -- Juha

            • 3. Re: Kernel parameters? Or configuration? -
              Itai Maoz Newbie

              Hi,

              I have the same problem, have you solved it?

              Thanks,
              Itai

              • 4. Re: Kernel parameters? Or configuration? -
                Juha Lindfors Master

                Try using ulimit to set a higher number of file descriptors on your system.

                -- Juha

                • 5. Re: Kernel parameters? Or configuration? -
                  mudblood Newbie

                  Not sure if I solved it, but I "did things" and the problem "went away" :) What I did was to up the ulimit on my system for both root and jboss (the processes run as root, but there's a su - jboss involved in the startup script which I wasn't sure about so I just upped the ulimit for jboss too). I upped the number of open files and the maximum number of user processes. On linux, you'll have to set these in /etc/security/limits.conf (if you wanted to set jboss's) as well as root's profile (.bash_profile or .bashrc - also for the jboss user, if you wanted to set its too). I also created an /etc/initscript and put the values in there so that there was no question that the rc-scripts would start with the right values. At some point I'd tried a number of different stack sizes, and I've finally settled on 1024k.

                  Give it a try and see how it goes - and please let me know if it fixes your problem so that I can know that "yep, that's what did it" for sure.

                  • 6. Re: Kernel parameters? Or configuration? -
                    mudblood Newbie

                    Heh, Juha you rule.