-
1. Re: Domain deployment slow
brian.stansberry Apr 10, 2012 8:04 PM (in response to mackerman)Do you know how long it takes if there's only a single host in the domain (besides the local master)?
Do you know how long it takes to deploy to a standalone server using the CLI?
-
2. Re: Domain deployment slow
mackerman Apr 11, 2012 12:29 AM (in response to brian.stansberry)For one server (other than the master) it appears to take the same length of time as for 4 servers, i.e., about 15 mins
Can't say that I have tried using the CLI, i'll give that a shot tomorrow.
thanks, Mitchell
-
3. Re: Domain deployment slow
mackerman Apr 11, 2012 12:31 AM (in response to brian.stansberry)I also forgot to add that we are currently using JBoss 7.0.2. Have there been any improvements to deployment speeds in 7.1.1?
-
4. Re: Domain deployment slow
emuckenhuber Apr 11, 2012 3:51 AM (in response to mackerman)We did not look at deployment speed in particular, but there have been a couple of other related changes and improvements.
So it's hard to say if this is still an issue. While we start investing this a bit, do you have an easy way to test this scenario with 7.1.1?
-
5. Re: Domain deployment slow
safetytrick Apr 11, 2012 12:25 PM (in response to emuckenhuber)I'm having a similar problem with very slow upload speeds when using the domain web interface and also when using the jboss-cli. When using the web interface I'm running into this timeout issue: https://issues.jboss.org/browse/AS7-4234 when i scp the file to the same server the upload completes in a few seconds. I tried scping the file to the same server as the domain controller and using the cli command "deploy /home/somefile.ear" but even locally the deploy command never completes.
The only error message I've seen comes from the web interface when the file upload times out.
I have a second domain controller setup that isn't experiencing these issues so i'm not sure what's wrong?
I'm using JBoss 7.1.1 on linux. Any pointers on how to figure this out?
Michael
-
6. Re: Domain deployment slow
emuckenhuber Apr 11, 2012 1:23 PM (in response to safetytrick)In general you cannot compare deploying a deployment using the mgmt API to scp or a simple copy. The deploy operation will wait until the deployment is completely deployed on all servers in the specified groups before returning. So it also depends on the size of the server group and the deployment time of the application.
Can you check the logs and once you see a log message like: JBAS015876: Starting deployment of "deployment.war" would mean that the content was copied successfully. Perhaps the issues are when trying to deploy the actual deployment?
-
7. Re: Domain deployment slow
safetytrick Apr 12, 2012 4:56 PM (in response to emuckenhuber)This turned out to be a combination of issues for me. One the web upload was timing out after 15 seconds, Two using the jboss-cli remotely is freezing, deploy is never copying any data (haven't figured out why yet), and Three the jboss-cli.sh script uses significantly more memory than the size of the artifact being deployed, I eventually set -Xmx=1024m (512m was not enough) to load a 160mb ear into the domain controller.
I'm still looking for answers on the remote jboss-cli.sh deploy issue.
Michael
-
8. Re: Domain deployment slow
safetytrick Apr 13, 2012 12:58 AM (in response to emuckenhuber)One interesting bottleneck seems to be ProcessControllerConnectionService.THREAD_POOL_MAX_SIZE = 4; copying ears out to 6 hosts runs 4 at a time. Would it make sense to increase this constant?
-
9. Re: Domain deployment slow
emuckenhuber Apr 13, 2012 7:21 AM (in response to safetytrick)One the web upload was timing out after 15 seconds
Yeah, that should be fixed already in upstream and available in a nightly snapshot build.
Two using the jboss-cli remotely is freezing, deploy is never copying any data (haven't figured out why yet)
When that happens can you take a thread-dump and also check the memory usage of the domain-controller?
Three the jboss-cli.sh script uses significantly more memory than the size of the artifact being deployed, I eventually set -Xmx=1024m (512m was not enough) to load a 160mb ear into the domain controller.
Hmm, good point we should improve that.
-
10. Re: Domain deployment slow
mackerman May 16, 2012 12:46 PM (in response to emuckenhuber)Sorry it's taken me so long to try this scenario with 7.1.1. We just upgraded our servers to 7.1.1.Final, and I can unfortunately report that the upload speed is no better. Our war file has shrunk a little, but to upload to the servers still takes in excess of 11 mins, compared to < 2 for filezilla.
-
11. Re: Domain deployment slow
emuckenhuber May 16, 2012 3:07 PM (in response to mackerman)1 of 1 people found this helpfulHi, interresting timing - since i've just had some time in the last couple of days to investigate this. It seems to be a problem in the configuration of our remoting protocol, using a too small chunk size. So i am currently working on a patch for AS to speed things up.
-
12. Re: Domain deployment slow
mackerman May 16, 2012 3:49 PM (in response to emuckenhuber)great! thanks
-
13. Re: Domain deployment slow
emuckenhuber May 21, 2012 2:22 PM (in response to mackerman)FYI we also found an issue where the slave host-controller was synchronizing deployment content althought it was already available locally. This should be fixed upstream soon as well.
-
14. Re: Domain deployment slow
ymartin Sep 5, 2013 8:34 AM (in response to mackerman)I got the same issue with 7.1.1 even using
export JAVA_OPTS=-Xmx1024m
to run jboss-cli.sh
The host controller has logged
JBAS014900: Content added at location /opt/jboss/domain/data/content/f5/88b80e83b11449a016e695e638d0dfdc6b0d90/content
...but everything remains stuck without any explanation.
So I decided to delete everything and get back to a clean JBoss, run my deployment script and it ends then successfully. Probably some internal state was messed up.