Wildfly will fail when java keystore storepasswd or keypasswd contains special characters

[eryabies]

Hi!

I have issue with Wildfly 10.1.0-Final and JAVA keystore key password/storapassword which contains special characters (chars like: .&%,/ etc.).

JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true


13:06:10,469 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
13:06:10,684 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
13:06:10,774 INFO  [org.jboss.as] (MSC service thread 1-3) WFLYSRV0049: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final) starting
13:06:12,112 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0015: Re-attempting failed deployment mariadb-java-client.jar
13:06:12,147 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 3) WFLYDR0001: Content added at location /opt/local/wildfly/standalone/data/content/63/9be502c0d191e1cc21e4e86d388486358fddf8/content
13:06:12,163 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
13:06:12,178 INFO  [org.xnio] (MSC service thread 1-3) XNIO version 3.4.0.Final
13:06:12,185 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.4.0.Final
13:06:12,240 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 4.0.21.Final
13:06:12,298 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 38) WFLYCLINF0001: Activating Infinispan subsystem.
13:06:12,336 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 37) WFLYIO001: Worker 'default' has auto-configured to 4 core threads with 32 task threads based on your 2 available processors
13:06:12,346 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 46) WFLYNAM0001: Activating Naming Subsystem
13:06:12,356 INFO  [org.jboss.as.connector] (MSC service thread 1-3) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
13:06:12,356 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 44) WFLYJSF0007: Activated the following JSF Implementations: [main]
13:06:12,413 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 54) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
13:06:12,450 INFO  [org.jboss.as.naming] (MSC service thread 1-3) WFLYNAM0003: Starting Naming Service
13:06:12,451 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
13:06:12,458 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 53) WFLYSEC0002: Activating Security Subsystem
13:06:12,476 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 56) WFLYWS0002: Activating WebServices Extension
13:06:12,479 INFO  [org.jboss.as.security] (MSC service thread 1-3) WFLYSEC0001: Current PicketBox version=4.9.6.Final
13:06:12,554 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0003: Undertow 1.4.0.Final starting
13:06:12,708 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 8 (per class), which is derived from the number of CPUs on this host.
13:06:12,738 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 32 (per class), which is derived from thread worker pool sizing.
13:06:12,765 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
13:06:12,866 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow HTTP listener remoting listening on 127.0.0.1:4447
13:06:12,867 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener http listening on 0.0.0.0:8080
13:06:13,308 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Chakra' 8.2.4.Final
13:06:13,373 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
13:06:13,377 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
13:06:13,378 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/local/wildfly/standalone/deployments
13:06:13,395 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "mariadb-java-client.jar" (runtime-name: "mariadb-java-client.jar")
13:06:13,398 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
13:06:13,399 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
13:06:13,399 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
13:06:13,400 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
13:06:13,444 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.server.controller.management.security_realm.SSLRealm.trust-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.SSLRealm.trust-manager: WFLYDM0018: Unable to start service
at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:153)
at org.jboss.as.domain.management.security.FileTrustManagerService.start(FileTrustManagerService.java:140)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
at java.security.KeyStore.load(KeyStore.java:1445)
at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:112)
... 6 more
Caused by: java.security.UnrecoverableKeyException: Password verification failed
at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
... 11 more

    <management>
        <security-realms>
            <security-realm name="ManagementRealm">
                <authentication>
                    <local default-user="$local" skip-group-loading="true"/>
                    <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
                <authorization map-groups-to-roles="false">
                    <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                </authorization>
            </security-realm>
            <security-realm name="ApplicationRealm">
                <server-identities>
                    <ssl>
                        <keystore path="${jboss.server.config.dir}/keystore/keystore.jks" keystore-password="8complexpasswd!," alias="myhost" key-password="8complexpasswd!,"/>
                    </ssl>
                </server-identities>
                <authentication>
                    <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                    <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                </authentication>
                <authorization>
                    <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                </authorization>
            </security-realm>
            <security-realm name="SSLRealm">
                <server-identities>
                    <ssl>
                        <keystore path="${jboss.server.config.dir}/keystore/keystore.jks" keystore-password="8complexpasswd!," alias="myhost"/>
                    </ssl>
                </server-identities>
                <authentication>
                    <truststore path="${jboss.server.config.dir}/keystore/truststore.jks" keystore-password="8complexpasswd!,"/>
                </authentication>
            </security-realm>
        </security-realms>

When I use very simple passwords which do not contains any special characters then it works as expected.

NB! Same time special characters in the <datasource block works!

Any suggetsions how to fix this issue?

Thanks

Ery

[mchoma]

I have tried with latest wildfly and it works OK for me. Could you post your keytool command?

keytool -genkeypair  -keystore keystore.jks  -alias myhost  -keyalg RSA  -keysize 2048  -validity 36500  -storepass '8complexpasswd!,'  -keypass '8complexpasswd!,' -dname "CN=test"

	<security-realm name="ApplicationRealm">
	<server-identities>
	<ssl>
	<keystore path="${jboss.server.config.dir}/keystore.jks" keystore-password="8complexpasswd!," alias="myhost" key-password="8complexpasswd!,"/>
	</ssl>
	</server-identities>
	<authentication>
	<local default-user="$local" allowed-users="*" skip-group-loading="true"/>
	<properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
	<truststore path="${jboss.server.config.dir}/keystore.jks" keystore-password="8complexpasswd!,"/>
	</authentication>
	<authorization>
	<properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
	</authorization>
	</security-realm>

[eryabies]

I'm using Java(TM) SE Runtime Environment (build 1.8.0_131-b11) and wildfly-10.1.0.Final.

I just changed my key password to O7q10FC05!09 with keytool -keypasswd -keystore keystore.jks -alias myhost.

It was test123 and with this test123 it works, but now with this new key password I got error:

           <security-realm name="ApplicationRealm">
                <server-identities>
                    <ssl>
                        <keystore path="${jboss.server.config.dir}/keystore/keystore.jks" keystore-password="changeit" alias="myhost" key-password="O7q10FC05!09"/>
                    </ssl>
                </server-identities>

10:13:03,609 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/local/software/wildfly/standalone/deployments
10:13:03,625 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.server.controller.management.security_realm.SSLRealm.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.SSLRealm.key-manager: Failed to start service
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: org.jboss.msc.service.StartException in anonymous service: WFLYDM0018: Unable to start service
at org.jboss.as.domain.management.security.FileKeyManagerService.loadKeyStore(FileKeyManagerService.java:193)
at org.jboss.as.domain.management.security.AbstractKeyManagerService.createKeyManagers(AbstractKeyManagerService.java:125)
at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:83)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
... 3 more
Caused by: org.jboss.msc.service.StartException in anonymous service: WFLYDM0018: Unable to start service
at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:155)
at org.jboss.as.domain.management.security.FileKeyManagerService.loadKeyStore(FileKeyManagerService.java:189)
... 7 more
Caused by: java.security.UnrecoverableKeyException: Cannot recover key
at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:146)
at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:56)
at java.security.KeyStoreSpi.engineGetEntry(KeyStoreSpi.java:473)
at sun.security.provider.KeyStoreDelegator.engineGetEntry(KeyStoreDelegator.java:172)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetEntry(JavaKeyStore.java:70)
at java.security.KeyStore.getEntry(KeyStore.java:1521)
at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:134)
... 8 more

And the password for the private key is correct one because I can import my private key from the keystore.jks without any problems.

keytool -importkeystore -srckeystore keystore.jks -destkeystore /tmp/keystore.p12 -deststoretype PKCS12 -srcalias myhost -destkeypass changeit
Enter destination keystore password: -> p12 password is changeit
Re-enter new password: -> p12 password is changeit
Enter source keystore password: -> storepassword is changeit
Enter key password for <myhost> -> this is the password from my standalone.xml file and it is O7q10FC05!09

Checking private key which I just imported from my keystore.jks with the openssl command:

openssl pkcs12 -in /tmp/keystore.p12 -nodes
Enter Import Password:
MAC verified OK
Bag Attributes
    friendlyName: myhost
    localKeyID: 54 69 6D 65 20 31 35 30 35 33 38 34 36 30 31 34 39 34
Key Attributes: <No Attributes>
-----BEGIN PRIVATE KEY-----

Thanks

[gir489]

We have a bang in our Dev SSL keystore passwords and it works fine for me, too.

[jaikiran]

Please post the exact command you used for the initial keystore generation (not for password change). What storetype did you specify (if any)? Is the JRE that is being used to generate the keytool different from the JRE that is running the WildFly server? Are there any locale differences between the runtimes where the keystore is being generated and the one running the WildFly server?

[eryabies]

Hi!

I use external CA for issuing certificate for my Wildfly instance.

This is how I created Java keystore.

openssl pkcs12 -export -out keystore.p12 -inkey myhost.key.pem -i myhost.crt.pem -certfile ca.crt.pem
keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -srcalias 1 -destkeystore keystore.jks -deststoretype JKS -destalias myhost

NB! Both above commands were executed @Wildfly host

NB! ca.crt.pem contains the RootCA and IntermediateCA and the private key and certificate are in the PEM format.

[jaikiran]

I actually have run out of ideas on what could be causing this and I can't reproduce it on my *nix system where the server boots up fine with such passwords for keystore/key password. The only thing I can think of, at this point is that, the console where you are using the keytool might be playing some kind of role in interpreting those characters when you are setting the password. Are you copy/pasting password when prompted by that tool? Either way, one way to rule that out is to have the password input fed from a file instead of typing/pasting it on the console and redirecting stdin to that file. You will have to experiment a bit to make sure the content of the file, that you are redirecting stdin to, matches what's being prompted for, but really I can't think of anything else that will play a role in something as straightforward as this thing.

[eryabies]

Ok, seems that I have some unknown issue with my standalone.xml file. After I deleted all SSL parts from my config and added them back via jboss cli special characters are allow:D

Strange issue indeed.

Thanks

[jaikiran]

Ery Abies wrote:

Ok, seems that I have some unknown issue with my standalone.xml file. After I deleted all SSL parts from my config and added them back via jboss cli special characters are allow:D

If you copy pasted it previously from some editor (for example, Microsoft Word), it's possible that some character issue might have caused it.