6 Replies Latest reply on Oct 10, 2016 9:06 PM by wcomnisky

Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

peterbecker Jan 24, 2016 10:39 PM

Hello,

I'm trying to start Hawkular on my machine which runs Ubuntu 14.04.3. I tried the ZIP version of hawkular-1.0.0.Alpha9, as well as building it myself from git. Both result in the same issue with ActiveMQ failing to start up:

13:16:20,939 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221000: live Message Broker is starting with configuration Broker Configuration (clustered=false,journalDirectory=/home/pbecker/projects/hawkular/dist/target/hawkular-1.0.0.Alpha10-SNAPSHOT/standalone/data/activemq/journal,bindingsDirectory=/home/pbecker/projects/hawkular/dist/target/hawkular-1.0.0.Alpha10-SNAPSHOT/standalone/data/activemq/bindings,largeMessagesDirectory=/home/pbecker/projects/hawkular/dist/target/hawkular-1.0.0.Alpha10-SNAPSHOT/standalone/data/activemq/largemessages,pagingDirectory=/home/pbecker/projects/hawkular/dist/target/hawkular-1.0.0.Alpha10-SNAPSHOT/standalone/data/activemq/paging)
13:16:20,941 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-7) WFLYDS0013: Started FileSystemDeploymentService for directory /home/pbecker/projects/hawkular/dist/target/hawkular-1.0.0.Alpha10-SNAPSHOT/standalone/deployments
13:16:20,942 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "hawkular-inventory-dist.war" (runtime-name: "hawkular-inventory-dist.war")
13:16:20,943 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-7) WFLYSRV0027: Starting deployment of "hawkular-alerts-actions-email.war" (runtime-name: "hawkular-alerts-actions-email.war")
13:16:21,032 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221012: Using AIO Journal
13:16:21,092 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-8) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.0.Final
13:16:21,092 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-3) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.0.Final
13:16:21,098 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.0.Final
13:16:21,092 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-2) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.0.Final
13:16:21,092 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-5) ISPN000128: Infinispan version: Infinispan 'Mahou' 8.1.0.Final
13:16:21,534 INFO  [org.jboss.ws.common.management] (MSC service thread 1-6) JBWS022052: Starting JBossWS 5.1.2.Final (Apache CXF 3.1.4) 
13:16:21,778 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221043: Protocol module found: [artemis-server]. Adding protocol support for: CORE
13:16:21,850 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221043: Protocol module found: [artemis-amqp-protocol]. Adding protocol support for: AMQP
13:16:22,194 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221043: Protocol module found: [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ
13:16:22,196 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221043: Protocol module found: [artemis-stomp-protocol]. Adding protocol support for: STOMP
13:16:22,691 WARN  [org.apache.activemq.artemis.journal] (ServerService Thread Pool -- 68) AMQ142018: Temporary files were left unnatended after a crash on journal directory, deleting invalid files now
13:16:22,692 WARN  [org.apache.activemq.artemis.journal] (ServerService Thread Pool -- 68) AMQ142019: Deleting orphaned file activemq-data-1.amq.tmp
13:16:22,693 WARN  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ222010: Critical IO Error, shutting down the server. file=AIOSequentialFile:/home/pbecker/projects/hawkular/dist/target/hawkular-1.0.0.Alpha10-SNAPSHOT/standalone/data/activemq/journal/activemq-data-1.amq.tmp, message=Cannot open file:Invalid argument: java.io.IOException: Cannot open file:Invalid argument
  at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
  at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:290)
  at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:273)
  at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
  at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:577)
  at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:530)
  at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:205)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
  at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1246)
  at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
  at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
  at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
  at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
  at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
  at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:194)
  at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
  at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  at org.jboss.threads.JBossThread.run(JBossThread.java:320)


13:16:22,717 ERROR [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ224000: Failure in initialisation: ActiveMQNativeIOError[errorType=NATIVE_ERROR_CANT_INITIALIZE_AIO message=Cannot open file:Invalid argument]
  at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:137)
  at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile0(JournalFilesRepository.java:577)
  at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.createFile(JournalFilesRepository.java:530)
  at org.apache.activemq.artemis.core.journal.impl.JournalFilesRepository.ensureMinFiles(JournalFilesRepository.java:205)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.setUpCurrentFile(JournalImpl.java:2652)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1728)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1126)
  at org.apache.activemq.artemis.core.journal.impl.JournalImpl.load(JournalImpl.java:1110)
  at org.apache.activemq.artemis.core.persistence.impl.journal.JournalStorageManager.loadMessageJournal(JournalStorageManager.java:1246)
  at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.loadJournals(ActiveMQServerImpl.java:1701)
  at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart2(ActiveMQServerImpl.java:1595)
  at org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:60)
  at org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:393)
  at org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:381)
  at org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:194)
  at org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:63)
  at org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:97)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)
  at org.jboss.threads.JBossThread.run(JBossThread.java:320)
Caused by: java.io.IOException: Cannot open file:Invalid argument
  at org.apache.activemq.artemis.jlibaio.LibaioContext.open(Native Method)
  at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:290)
  at org.apache.activemq.artemis.jlibaio.LibaioContext.openFile(LibaioContext.java:273)
  at org.apache.activemq.artemis.core.io.aio.AIOSequentialFile.open(AIOSequentialFile.java:133)
  ... 22 more


13:16:22,722 INFO  [org.apache.activemq.artemis.core.server] (ServerService Thread Pool -- 68) AMQ221001: Apache ActiveMQ Artemis Message Broker version 1.1.0.wildfly-010 [nodeID=6a4e735e-c30f-11e5-a7ac-893713ec38aa] 
13:16:22,830 INFO  [org.apache.activemq.artemis.core.server] (pool-4-thread-1) AMQ221002: Apache ActiveMQ Artemis Message Broker version 1.1.0.wildfly-010 [6a4e735e-c30f-11e5-a7ac-893713ec38aa] stopped

I've tried to fix this by adding {{HAWKULAR_BASE_DIR}}/modules/system/layers/base/org/apache/activemq/artemis/main/lib/linux-x86_64 to $LD_LIBRARY_PATH as well as to -Djava.library.path, but neither seems to change anything.

Any advice on how to tackle this one? At the moment I'm only evaluating - is there a way to disable AIO?

Cheers,

Peter

1. Re: Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

tsegismont Jan 26, 2016 3:17 AM (in response to peterbecker)

Hi,
This issue looks a lot like [JBEAP-1887] Artemis fails to load journal with libAIO on tmpfs file system - JBoss Issue Tracker Except that you don't seem to be using a tmpfs filesystem... are you?
Before anything else, can you make sure there is enough disk space?

Then can you try to temporarily uninstall {{libaio}} from your system with {{apt-get remove libaio1}} or similar ? That should force Artemis to fallback to standard Java NIO. Or configure the messages journal to always Java NIO.

Regards,
Thomas
Actions
2. Re: Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

peterbecker Jan 31, 2016 8:19 PM (in response to tsegismont)

Hi Thomas,

first of all sorry for the delay - other things took priority.

I had seen that JIRA issue, but no: I'm not running tmpfs. However: it is an ecryptfs, which makes me wonder if that could cause it. A quick google seems to indicate that this combination has issues as well: [TORQUE-972] Torquebox will fail if installed in an ecryptfs forder and libaio is installed - JBoss Issue Tracker

I can't really try to remove libaio1, since I'm on my dev machine and MySQL depends on it (and I need MySQL). I have switched to using the docker image - I'm only evaluating at this time anyway. That seems to work fine, and at the moment it looks like it's not going to be a fit for our needs anyway. If there's anything else you want me to try I'm happy to do so, but otherwise I'll be moving on.

Cheers,
Peter
Actions
3. Re: Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

tsegismont Feb 1, 2016 3:38 AM (in response to peterbecker)

Hi Peter,
Thanks for the TORQUEBOX issue pointer, that could help us in the future. There's nothing else I'd like you try, but may I ask why it does not fit?
Best regards,
Thomas
Actions
4. Re: Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

peterbecker Feb 1, 2016 10:00 PM (in response to tsegismont)
Hi Thomas,

we are looking for something to monitor various Java stacks and their underlying infrastructure:
core OS metrics such as CPU load, memory use, processes running/waiting, disk/network IO, disk space used/available
standard JVM metrics such as memory spaces, threads, etc.
stack/application specific metrics e.g. for:
Dropwizard (standalone)
Spring Boot (standalone)
ActiveMQ
Mule ESB
WebLogic (JMS / WS)
HTTP request/response patterns (e.g. Dropwizard style healthchecks: Health Checks | Metrics )

All that needs graphing and alerting.

My impression is that Hawkular doesn't support that range all that well (yet?). While we would be in a position to put some effort in ourselves, we won't get too much of a budget for that.

I'd be happy to stand corrected :-)

Regards,
Peter
Actions
5. Re: Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

mnovak Mar 21, 2016 7:12 AM (in response to peterbecker)

Hi Peter, do you still see the problem with loading AIO journal? Does fail on your disk with ext4 or xfs file system?

Thanks,
Mirek
Actions
6. Re: Hawkular does not start due to ActiveMQNativeIOError: NATIVE_ERROR_CANT_INITIALIZE_AIO (Ubuntu 14.04)

wcomnisky Oct 10, 2016 9:06 PM (in response to peterbecker)

Topic quite old but as it is still not answered I'll give my 10cents.

Check the AIO issue on WildFly 10 section at the following link:
AIO - NIO for messaging journal - WildFly 10 - Project Documentation Editor

cheers
Actions

Go to original post