Recently, my EAP app on OpenShift stopped working. It wouldn't start. ctl_app -t start  (-t is for trace log) shown:

[web-ozcz.rhcloud.com ...]\> ctl_app -t start

Starting gear...
MySQL already running
Starting PHPMyAdmin cartridge
httpd (pid 279843) already running
Starting jbosseap cartridge
jbosseap process failed to start
An error occurred executing 'gear start' (exit code: 2)
Error message: CLIENT_ERROR: Failed to execute: 'control start' for /var/lib/openshift/.../jbosseap

/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1313:in `block in do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1114:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1276:in `do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1136:in `do_control'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1498:in `start_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1437:in `block in start_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:80:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1120:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1118:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1118:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1432:in `start_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/v2_cart_model.rb:1441:in `start_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.21.6/lib/openshift-origin-node/model/application_container.rb:446:in `start_gear'
/usr/bin/gear:479:in `block (3 levels) in <main>'
/usr/bin/gear:65:in `do_command'
/usr/bin/gear:474:in `block (2 levels) in <main>'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:74:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'

There was no log I could use, and the message above was a dead end.

I've tried to start EAP on my own. Didn't work, saying:

org.jboss.modules.ModuleNotFoundException: org.jboss.as.standalone:main
        at org.jboss.modules.ModuleLoader.loadModule(ModuleLoader.java:240)
        at org.jboss.modules.Main.main(Main.java:385)

So I started going through the labyrinth of links in the OpenShift infrastructure, and found that there's a link vs. actual dirs mismatch:

/etc/alternatives/jbosseap-6.0/modules (this is a link)
vs.
/etc/alternatives/jbosseap-6/ (this is what exists on the system)

After fixing the link manually, EAP started working again.

I don't remember changing the link before. Did it happend during some infrastructure changes? Was I the only one?

Shortly before this happened, I was struggling with insufficient disk space and EAP crashed because of that. Was that related? ...

Hope this helps someone.