7 Replies Latest reply on Feb 26, 2016 6:00 AM by grgrzybek

Jboss Fuse 6.1 Active threads problems

antonio.giambanco Feb 23, 2016 1:21 PM

Hi all,

anybody ever had problems with active thread in jboss fuse?

from cpu and memory monitor seems everything correct, from active threads monitor they grows forever.

To cut them I need to restart the container.

I have a frontend that calls some passthrough proxies that call backend service. The slowest backend service answers no more than 15 seconds

many thanks in advice

1. Re: Jboss Fuse 6.1 Active threads problems

grgrzybek Feb 24, 2016 2:32 AM (in response to antonio.giambanco)

Hello Antonio

What version of Fuse 6.1 do you use? Any patches installed?
Could you try running `jstack -l PID-OF-FUSE` to check which threads are those "extra" threads?

regards
Grzegorz Grzybek
Actions
2. Re: Jboss Fuse 6.1 Active threads problems

antonio.giambanco Feb 24, 2016 8:56 AM (in response to grgrzybek)

Hi Grzegorz,
thanks for your answer I foundout we have this version 2.3.0.redhat-611412 without any patches

they are cutted, just few rows

Line 4: "CamelJettyClient(0x6878954a)-341136" daemon prio=10 tid=0x00007f68f5de3000 nid=0x3fb5 waiting on condition [0x00007f68c9a5a000]
Line 19: "CamelJettyClient(0x91cae9a)-341135" daemon prio=10 tid=0x00007f68f64a9000 nid=0x3fb4 waiting on condition [0x00007f68bf0b1000]
Line 34: "CamelJettyClient(0x21a2cb07)-341134" daemon prio=10 tid=0x00007f68f5920800 nid=0x3fb3 waiting on condition [0x00007f68c31f2000]
Line 49: "CamelJettyClient(0x7752fb3a)-341133" daemon prio=10 tid=0x00007f68f5a99000 nid=0x3fb0 waiting on condition [0x00007f68d15d5000]
Line 64: "CamelJettyClient(0x5b09da6b)-341132" daemon prio=10 tid=0x00007f68f4fe7000 nid=0x3faf waiting on condition [0x00007f68bd99a000]
Line 79: "CamelJettyClient(0x1663b94b)-341131" daemon prio=10 tid=0x00007f68f6319000 nid=0x3fae waiting on condition [0x00007f68c32f3000]
Line 94: "CamelJettyClient(0x372bc286)-341130" daemon prio=10 tid=0x00007f68f45cb000 nid=0x3fab waiting on condition [0x00007f68b9556000]
Line 109: "CamelJettyClient(0x2973162e)-341129" daemon prio=10 tid=0x00007f68f5c64000 nid=0x3faa waiting on condition [0x00007f68ccd8d000]
Line 124: "CamelJettyClient(0x554659db)-341128" daemon prio=10 tid=0x00007f68f58f0000 nid=0x3fa9 waiting on condition [0x00007f68c23e4000]
Line 139: "CamelJettyClient(0x2d4f71e4)-341127" daemon prio=10 tid=0x00007f68f6007800 nid=0x3fa7 waiting on condition [0x00007f68bb475000]
Line 154: "CamelJettyClient(0x8e61520)-341124" daemon prio=10 tid=0x00007f68f641f000 nid=0x3fa1 waiting on condition [0x00007f68b591a000]
Line 169: "CamelJettyClient(0x34e20b68)-341123" daemon prio=10 tid=0x00007f68f5a05000 nid=0x3fa0 waiting on condition [0x00007f68ce4a4000]
Line 184: "CamelJettyClient(0x7d8f550a)-341122" daemon prio=10 tid=0x00007f68f4c65800 nid=0x3f9f waiting on condition [0x00007f68be4a5000]
Line 199: "CamelJettyClient(0x7face696)-341121" daemon prio=10 tid=0x00007f68f40bf800 nid=0x3f9e waiting on condition [0x00007f68bf5b6000]
Line 214: "CamelJettyClient(0x1777cee6)-341120" daemon prio=10 tid=0x00007f68f612b000 nid=0x3f9d waiting on condition [0x00007f68c7f3f000]
Line 229: "CamelJettyClient(0x5684988e)-341117" daemon prio=10 tid=0x00007f68f5ce9800 nid=0x3f96 waiting on condition [0x00007f68c38f9000]
Line 244: "CamelJettyClient(0x4b89e2b3)-341116" daemon prio=10 tid=0x00007f68f5b85000 nid=0x3f95 waiting on condition [0x00007f68d7635000]
Line 259: "CamelJettyClient(0x1fcbc58a)-341115" daemon prio=10 tid=0x00007f68f631b800 nid=0x3f94 waiting on condition [0x00007f68bde9f000]
Line 274: "CamelJettyClient(0x4418adb9)-341114" daemon prio=10 tid=0x00007f68f636e000 nid=0x3f93 waiting on condition [0x00007f68d2ded000]
Line 289: "CamelJettyClient(0x6dd634d7)-341113" daemon prio=10 tid=0x00007f68f6260800 nid=0x3f90 waiting on condition [0x00007f68baa6b000]
Line 304: "CamelJettyClient(0x28ca4458)-341112" daemon prio=10 tid=0x00007f68f4e39000 nid=0x3f8f waiting on condition [0x00007f68bb071000]
Line 319: "CamelJettyClient(0x6bcece7d)-341111" daemon prio=10 tid=0x00007f68f5be5800 nid=0x3f8c waiting on condition [0x00007f68d9a59000]
Line 334: "CamelJettyClient(0x4b148adc)-341108" daemon prio=10 tid=0x00007f68f6522800 nid=0x3f85 waiting on condition [0x00007f68b9859000]
Line 349: "CamelJettyClient(0x601e1e2f)-341107" daemon prio=10 tid=0x00007f68f6010800 nid=0x3f84 waiting on condition [0x00007f68cfcbc000]
Line 364: "CamelJettyClient(0x77f4ca55)-341106" daemon prio=10 tid=0x00007f68f6172000 nid=0x3f83 waiting on condition [0x00007f68c5212000]
Line 379: "CamelJettyClient(0x1bd4858a)-341105" daemon prio=10 tid=0x00007f68fc537800 nid=0x3f82 waiting on condition [0x00007f68b7a3b000]
Line 394: "CamelJettyClient(0x67e7e763)-341104" daemon prio=10 tid=0x00007f68f40e5800 nid=0x3f81 waiting on condition [0x00007f68b7334000]
Line 409: "CamelJettyClient(0x26314e46)-341103" daemon prio=10 tid=0x00007f68f5f0d800 nid=0x3f80 waiting on condition [0x00007f68bf8b9000]
Line 424: "CamelJettyClient(0x78c7bf46)-341102" daemon prio=10 tid=0x00007f68f59da800 nid=0x3f7f waiting on condition [0x00007f68d08c8000]
Line 439: "CamelJettyClient(0x436db1ba)-341099" daemon prio=10 tid=0x00007f68f6008800 nid=0x3f7a waiting on condition [0x00007f68bd495000]
Line 454: "CamelJettyClient(0x3a577c46)-341098" daemon prio=10 tid=0x00007f68f641c800 nid=0x3f79 waiting on condition [0x00007f68c8949000]
Line 469: "CamelJettyClient(0x8fed18a)-341097" daemon prio=10 tid=0x00007f68f55c5800 nid=0x3f78 waiting on condition [0x00007f68c00c1000]
Line 484: "CamelJettyClient(0x74e0bbb7)-341096" daemon prio=10 tid=0x00007f68f54d6000 nid=0x3f73 waiting on condition [0x00007f68b6021000]
Line 499: "CamelJettyClient(0x6f156d33)-341095" daemon prio=10 tid=0x00007f68f491e800 nid=0x3f72 waiting on condition [0x00007f68bedae000]
Line 514: "CamelJettyClient(0x2570b77c)-341094" daemon prio=10 tid=0x00007f68f6542800 nid=0x3f71 waiting on condition [0x00007f68bc586000]
Line 529: "CamelJettyClient(0x4490abb3)-341093" daemon prio=10 tid=0x00007f68f5d4b800 nid=0x3f70 waiting on condition [0x00007f68b5314000]
Line 544: "CamelJettyClient(0x8f9e078)-341092" daemon prio=10 tid=0x00007f68f63a9800 nid=0x3f6f waiting on condition [0x00007f68bdb9c000]
Line 559: "CamelJettyClient(0x1abcd3ec)-341091" daemon prio=10 tid=0x00007f68f45d0800 nid=0x3f6e waiting on condition [0x00007f68b6b2c000]
Line 574: "CamelJettyClient(0x26091c44)-341088" daemon prio=10 tid=0x00007f68f5f27000 nid=0x3f67 waiting on condition [0x00007f68bffc0000]
Line 589: "CamelJettyClient(0x57b583db)-341087" daemon prio=10 tid=0x00007f68fc6a0000 nid=0x3f64 waiting on condition [0x00007f68ba667000]
Line 604: "CamelJettyClient(0x19c97146)-341086" daemon prio=10 tid=0x00007f68f5f2f800 nid=0x3f63 waiting on condition [0x00007f68bb273000]
Line 619: "CamelJettyClient(0x7ef15d31)-341085" daemon prio=10 tid=0x00007f68f636d000 nid=0x3f60 waiting on condition [0x00007f68be5a6000]
Line 634: "CamelJettyClient(0x9b7a383)-341083" daemon prio=10 tid=0x00007f68f5cdf800 nid=0x3f5e waiting on condition [0x00007f68c6020000]
Line 649: "CamelJettyClient(0x41f2b008)-341081" daemon prio=10 tid=0x00007f68f64fc000 nid=0x3f5a waiting on condition [0x00007f6909f6b000]
Line 664: "CamelJettyClient(0x4d26660)-341080" daemon prio=10 tid=0x00007f68f4edc000 nid=0x3f57 waiting on condition [0x00007f68b9051000]
Line 679: "CamelJettyClient(0x58ff004e)-341077" daemon prio=10 tid=0x00007f68f5e6e000 nid=0x3f44 waiting on condition [0x00007f68bcb8c000]
Line 694: "CamelJettyClient(0x284154c5)-341076" daemon prio=10 tid=0x00007f68f4efb800 nid=0x3f43 waiting on condition [0x00007f68c0ecf000]
Actions
3. Re: Jboss Fuse 6.1 Active threads problems

grgrzybek Feb 24, 2016 6:36 AM (in response to antonio.giambanco)

HI Antonio!

Could you please grep the log by 'tid=' and edit the post? I've checked and the suspicious thread pattern is:
"CamelJettyClient(0x5b09da6b)-9886 Selector0" daemon prio=10 tid=0x00007f68f5403800 nid=0x4b8b runnable [0x00007f68d4908000]
So probably some camel-jetty thing. It may be related to the application you deploy. I'll check possible camel bugs, fixed in later patches.
regards
Grzegorz
Actions
4. Re: Jboss Fuse 6.1 Active threads problems

grgrzybek Feb 24, 2016 8:08 AM (in response to antonio.giambanco)

@Antonio, there's a chance that your application has lot of bundles with Camel routes having `to("jetty:")`. You need to tweak the jetty client pool which you configure on the component level. See "httpClientMaxThreads" property on Apache Camel: Jetty.

regards
Grzegorz
Actions
5. Re: Jboss Fuse 6.1 Active threads problems

antonio.giambanco Feb 24, 2016 9:06 AM (in response to grgrzybek)

HI grgrzybek
thanks so muche for you advice,
unfortunatly I couldn't paste the grepped log becouse my browser crashes, I posted just a subset.
Also I already configured the route as follow
<to uri="jetty:{{baseUrl}}{{uriDestination}}?httpClientMinThreads={{jetty.minThreads}}&httpClientMaxThreads={{jetty.maxThreads}}" />

Do you have orther advices?
Should we change component insted of jetty? In the past we used standard http component but we had to change becouse of cutted requestes.

Thanks again
Actions
6. Re: Jboss Fuse 6.1 Active threads problems

antonio.giambanco Feb 26, 2016 5:52 AM (in response to antonio.giambanco)

Hi Grzegorz
I tried to configure timeout on http component but still not working.

any update?

thanks
Actions
7. Re: Jboss Fuse 6.1 Active threads problems

grgrzybek Feb 26, 2016 6:00 AM (in response to antonio.giambanco)

Hi Antonio

Unfortunately I have no Fuse-related advices.... About those "cutted requestes" - try http requests, maybe some problems were resolved at Camel side.

regards
Grzegorz
Actions

Go to original post