8 Replies Latest reply on Sep 15, 2017 11:11 AM by jaikiran pai

    Wildfly will fail when java keystore storepasswd or keypasswd contains special characters

    Ery Abies Newbie

      Hi!

       

      I have issue with Wildfly 10.1.0-Final and JAVA keystore key password/storapassword which contains special characters (chars like: .&%,/ etc.).

      JAVA_OPTS:  -server -Xms64m -Xmx512m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true -Dfile.encoding=UTF-8 -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
      
      
      13:06:10,469 INFO  [org.jboss.modules] (main) JBoss Modules version 1.5.2.Final
      13:06:10,684 INFO  [org.jboss.msc] (main) JBoss MSC version 1.2.6.Final
      13:06:10,774 INFO  [org.jboss.as] (MSC service thread 1-3) WFLYSRV0049: WildFly Full 10.1.0.Final (WildFly Core 2.2.0.Final) starting
      13:06:12,112 INFO  [org.jboss.as.server.deployment.scanner] (DeploymentScanner-threads - 1) WFLYDS0015: Re-attempting failed deployment mariadb-java-client.jar
      13:06:12,147 INFO  [org.jboss.as.repository] (ServerService Thread Pool -- 3) WFLYDR0001: Content added at location /opt/local/wildfly/standalone/data/content/63/9be502c0d191e1cc21e4e86d388486358fddf8/content
      13:06:12,163 INFO  [org.jboss.as.server] (Controller Boot Thread) WFLYSRV0039: Creating http management service using socket-binding (management-http)
      13:06:12,178 INFO  [org.xnio] (MSC service thread 1-3) XNIO version 3.4.0.Final
      13:06:12,185 INFO  [org.xnio.nio] (MSC service thread 1-3) XNIO NIO Implementation Version 3.4.0.Final
      13:06:12,240 INFO  [org.jboss.remoting] (MSC service thread 1-3) JBoss Remoting version 4.0.21.Final
      13:06:12,298 INFO  [org.jboss.as.clustering.infinispan] (ServerService Thread Pool -- 38) WFLYCLINF0001: Activating Infinispan subsystem.
      13:06:12,336 INFO  [org.wildfly.extension.io] (ServerService Thread Pool -- 37) WFLYIO001: Worker 'default' has auto-configured to 4 core threads with 32 task threads based on your 2 available processors
      13:06:12,346 INFO  [org.jboss.as.naming] (ServerService Thread Pool -- 46) WFLYNAM0001: Activating Naming Subsystem
      13:06:12,356 INFO  [org.jboss.as.connector] (MSC service thread 1-3) WFLYJCA0009: Starting JCA Subsystem (WildFly/IronJacamar 1.3.4.Final)
      13:06:12,356 INFO  [org.jboss.as.jsf] (ServerService Thread Pool -- 44) WFLYJSF0007: Activated the following JSF Implementations: [main]
      13:06:12,413 WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 54) WFLYTX0013: Node identifier property is set to the default value. Please make sure it is unique.
      13:06:12,450 INFO  [org.jboss.as.naming] (MSC service thread 1-3) WFLYNAM0003: Starting Naming Service
      13:06:12,451 INFO  [org.jboss.as.mail.extension] (MSC service thread 1-3) WFLYMAIL0001: Bound mail session [java:jboss/mail/Default]
      13:06:12,458 INFO  [org.jboss.as.security] (ServerService Thread Pool -- 53) WFLYSEC0002: Activating Security Subsystem
      13:06:12,476 INFO  [org.jboss.as.webservices] (ServerService Thread Pool -- 56) WFLYWS0002: Activating WebServices Extension
      13:06:12,479 INFO  [org.jboss.as.security] (MSC service thread 1-3) WFLYSEC0001: Current PicketBox version=4.9.6.Final
      13:06:12,554 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0003: Undertow 1.4.0.Final starting
      13:06:12,708 INFO  [org.jboss.as.ejb3] (MSC service thread 1-2) WFLYEJB0482: Strict pool mdb-strict-max-pool is using a max instance size of 8 (per class), which is derived from the number of CPUs on this host.
      13:06:12,738 INFO  [org.jboss.as.ejb3] (MSC service thread 1-4) WFLYEJB0481: Strict pool slsb-strict-max-pool is using a max instance size of 32 (per class), which is derived from thread worker pool sizing.
      13:06:12,765 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-1) WFLYUT0012: Started server default-server.
      13:06:12,866 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-2) WFLYUT0006: Undertow HTTP listener remoting listening on 127.0.0.1:4447
      13:06:12,867 INFO  [org.wildfly.extension.undertow] (MSC service thread 1-4) WFLYUT0006: Undertow HTTP listener http listening on 0.0.0.0:8080
      13:06:13,308 INFO  [org.infinispan.factories.GlobalComponentRegistry] (MSC service thread 1-1) ISPN000128: Infinispan version: Infinispan 'Chakra' 8.2.4.Final
      13:06:13,373 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      13:06:13,377 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 59) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      13:06:13,378 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-2) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/local/wildfly/standalone/deployments
      13:06:13,395 INFO  [org.jboss.as.server.deployment] (MSC service thread 1-2) WFLYSRV0027: Starting deployment of "mariadb-java-client.jar" (runtime-name: "mariadb-java-client.jar")
      13:06:13,398 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      13:06:13,399 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      13:06:13,399 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      13:06:13,400 INFO  [org.infinispan.configuration.cache.EvictionConfigurationBuilder] (ServerService Thread Pool -- 58) ISPN000152: Passivation configured without an eviction policy being selected. Only manually evicted entities will be passivated.
      13:06:13,444 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-4) MSC000001: Failed to start service jboss.server.controller.management.security_realm.SSLRealm.trust-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.SSLRealm.trust-manager: WFLYDM0018: Unable to start service
      at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:153)
      at org.jboss.as.domain.management.security.FileTrustManagerService.start(FileTrustManagerService.java:140)
      at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
      at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
      at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:780)
      at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:56)
      at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
      at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:70)
      at java.security.KeyStore.load(KeyStore.java:1445)
      at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:112)
      ... 6 more
      Caused by: java.security.UnrecoverableKeyException: Password verification failed
      at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:778)
      ... 11 more
      

       

          <management>
              <security-realms>
                  <security-realm name="ManagementRealm">
                      <authentication>
                          <local default-user="$local" skip-group-loading="true"/>
                          <properties path="mgmt-users.properties" relative-to="jboss.server.config.dir"/>
                      </authentication>
                      <authorization map-groups-to-roles="false">
                          <properties path="mgmt-groups.properties" relative-to="jboss.server.config.dir"/>
                      </authorization>
                  </security-realm>
                  <security-realm name="ApplicationRealm">
                      <server-identities>
                          <ssl>
                              <keystore path="${jboss.server.config.dir}/keystore/keystore.jks" keystore-password="8complexpasswd!," alias="myhost" key-password="8complexpasswd!,"/>
                          </ssl>
                      </server-identities>
                      <authentication>
                          <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                          <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                      </authentication>
                      <authorization>
                          <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
                      </authorization>
                  </security-realm>
                  <security-realm name="SSLRealm">
                      <server-identities>
                          <ssl>
                              <keystore path="${jboss.server.config.dir}/keystore/keystore.jks" keystore-password="8complexpasswd!," alias="myhost"/>
                          </ssl>
                      </server-identities>
                      <authentication>
                          <truststore path="${jboss.server.config.dir}/keystore/truststore.jks" keystore-password="8complexpasswd!,"/>
                      </authentication>
                  </security-realm>
              </security-realms>
      

       

      When I use very simple passwords which do not contains any special characters then it works as expected.

       

      NB! Same time special characters in the <datasource block works!

       

      Any suggetsions how to fix this issue?

       

      Thanks

      Ery

        • 1. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
          Martin Choma Expert

          I have tried with latest wildfly and it works OK for me. Could you post your keytool command?

           

          keytool -genkeypair  -keystore keystore.jks  -alias myhost  -keyalg RSA  -keysize 2048  -validity 36500  -storepass '8complexpasswd!,'  -keypass '8complexpasswd!,' -dname "CN=test"
          

           

           

          <security-realm name="ApplicationRealm">
              <server-identities>
                  <ssl>
                      <keystore path="${jboss.server.config.dir}/keystore.jks" keystore-password="8complexpasswd!," alias="myhost" key-password="8complexpasswd!,"/>
                  </ssl>
              </server-identities>
              <authentication>
                  <local default-user="$local" allowed-users="*" skip-group-loading="true"/>
                  <properties path="application-users.properties" relative-to="jboss.server.config.dir"/>
                  <truststore path="${jboss.server.config.dir}/keystore.jks" keystore-password="8complexpasswd!,"/>
              </authentication>
              <authorization>
                  <properties path="application-roles.properties" relative-to="jboss.server.config.dir"/>
              </authorization>
          </security-realm>
          • 2. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
            Ery Abies Newbie

            I'm using Java(TM) SE Runtime Environment (build 1.8.0_131-b11) and wildfly-10.1.0.Final.

             

            I just changed my key password to O7q10FC05!09 with keytool -keypasswd -keystore keystore.jks -alias myhost.

             

            It was test123 and with this test123 it works, but now with this new key password I got error:

                       <security-realm name="ApplicationRealm">
                            <server-identities>
                                <ssl>
                                    <keystore path="${jboss.server.config.dir}/keystore/keystore.jks" keystore-password="changeit" alias="myhost" key-password="O7q10FC05!09"/>
                                </ssl>
                            </server-identities>
            

             

             

            10:13:03,609 INFO  [org.jboss.as.server.deployment.scanner] (MSC service thread 1-1) WFLYDS0013: Started FileSystemDeploymentService for directory /opt/local/software/wildfly/standalone/deployments
            10:13:03,625 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-2) MSC000001: Failed to start service jboss.server.controller.management.security_realm.SSLRealm.key-manager: org.jboss.msc.service.StartException in service jboss.server.controller.management.security_realm.SSLRealm.key-manager: Failed to start service
            at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1904)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:748)
            Caused by: java.lang.IllegalStateException: org.jboss.msc.service.StartException in anonymous service: WFLYDM0018: Unable to start service
            at org.jboss.as.domain.management.security.FileKeyManagerService.loadKeyStore(FileKeyManagerService.java:193)
            at org.jboss.as.domain.management.security.AbstractKeyManagerService.createKeyManagers(AbstractKeyManagerService.java:125)
            at org.jboss.as.domain.management.security.AbstractKeyManagerService.start(AbstractKeyManagerService.java:83)
            at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1948)
            at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1881)
            ... 3 more
            Caused by: org.jboss.msc.service.StartException in anonymous service: WFLYDM0018: Unable to start service
            at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:155)
            at org.jboss.as.domain.management.security.FileKeyManagerService.loadKeyStore(FileKeyManagerService.java:189)
            ... 7 more
            Caused by: java.security.UnrecoverableKeyException: Cannot recover key
            at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
            at sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:146)
            at sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:56)
            at java.security.KeyStoreSpi.engineGetEntry(KeyStoreSpi.java:473)
            at sun.security.provider.KeyStoreDelegator.engineGetEntry(KeyStoreDelegator.java:172)
            at sun.security.provider.JavaKeyStore$DualFormatJKS.engineGetEntry(JavaKeyStore.java:70)
            at java.security.KeyStore.getEntry(KeyStore.java:1521)
            at org.jboss.as.domain.management.security.FileKeystore.load(FileKeystore.java:134)
            ... 8 more
            

             

            And the password for the private key is correct one because I can import my private key from the keystore.jks without any problems.

            keytool -importkeystore -srckeystore keystore.jks -destkeystore /tmp/keystore.p12 -deststoretype PKCS12 -srcalias myhost -destkeypass changeit
            Enter destination keystore password: -> p12 password is changeit
            Re-enter new password: -> p12 password is changeit
            Enter source keystore password: -> storepassword is changeit
            Enter key password for <myhost> -> this is the password from my standalone.xml file and it is O7q10FC05!09
            

             

            Checking private key which I just imported from my keystore.jks with the openssl command:

            openssl pkcs12 -in /tmp/keystore.p12 -nodes
            Enter Import Password:
            MAC verified OK
            Bag Attributes
                friendlyName: myhost
                localKeyID: 54 69 6D 65 20 31 35 30 35 33 38 34 36 30 31 34 39 34
            Key Attributes: <No Attributes>
            -----BEGIN PRIVATE KEY-----
            

             

            Thanks

            • 3. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
              gir489 Newbie

              We have a bang in our Dev SSL keystore passwords and it works fine for me, too.

              • 4. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
                jaikiran pai Master

                Please post the exact command you used for the initial keystore generation (not for password change). What storetype did you specify (if any)? Is the JRE that is being used to generate the keytool different from the JRE that is running the WildFly server? Are there any locale differences between the runtimes where the keystore is being generated and the one running the WildFly server?

                • 5. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
                  Ery Abies Newbie

                  Hi!

                   

                  I use external CA for issuing certificate for my Wildfly instance.

                  This is how I created Java keystore.

                  openssl pkcs12 -export -out keystore.p12 -inkey myhost.key.pem -i myhost.crt.pem -certfile ca.crt.pem
                  keytool -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -srcalias 1 -destkeystore keystore.jks -deststoretype JKS -destalias myhost
                  

                   

                  NB! Both above commands were executed @Wildfly host

                   

                  NB! ca.crt.pem contains the RootCA and IntermediateCA and the private key and certificate are in the PEM format.

                  • 6. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
                    jaikiran pai Master

                    I actually have run out of ideas on what could be causing this and I can't reproduce it on my *nix system where the server boots up fine with such passwords for keystore/key password. The only thing I can think of, at this point is that, the console where you are using the keytool might be playing some kind of role in interpreting those characters when you are setting the password. Are you copy/pasting password when prompted by that tool? Either way, one way to rule that out is to have the password input fed from a file instead of typing/pasting it on the console and redirecting stdin to that file. You will have to experiment a bit to make sure the content of the file, that you are redirecting stdin to, matches what's being prompted for, but really I can't think of anything else that will play a role in something as straightforward as this thing.

                    • 7. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
                      Ery Abies Newbie

                      Ok, seems that I have some unknown issue with my standalone.xml file. After I deleted all SSL parts from my config and added them back via jboss cli special characters are allow:D

                      Strange issue indeed.

                       

                      Thanks

                      • 8. Re: Wildfly will fail when java keystore storepasswd or keypasswd contains special characters
                        jaikiran pai Master

                        Ery Abies wrote:

                         

                        Ok, seems that I have some unknown issue with my standalone.xml file. After I deleted all SSL parts from my config and added them back via jboss cli special characters are allow:D

                        If you copy pasted it previously from some editor (for example, Microsoft Word), it's possible that some character issue might have caused it.