Error while retrieving variable from vault - domain mode

[pplopez]

Hi guys, I'm having trouble with JBoss' vault. I've just created the keystore, the vault, configured JBoss to use the vault, and actually I can add a system property with its value referencing the vault, but when I start the server it fails, throwing this exception:

ERROR [org.jboss.as.controller.management-operation] (Host Controller Service Threads - 58) JBAS014612: Operation ("start") failed - address: ([

("host" => "HOST_XXX"),

("server-config" => "SERVER_XXX")

java.lang.SecurityException: JBAS013311: Security Exception

at org.jboss.as.security.vault.RuntimeVaultReader.retrieveFromVault(RuntimeVaultReader.java:115) [jboss-as-security-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2]

at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:45) [jboss-as-server-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2]

at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:319) [jboss-as-controller-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2]

....more exception trace

Caused by: org.jboss.security.vault.SecurityVaultException: java.lang.IllegalArgumentException: Null input buffer

at org.picketbox.plugins.vault.PicketBoxSecurityVault.retrieve(PicketBoxSecurityVault.java:297) [picketbox-4.1.2.Final-redhat-1.jar:4.1.2.Final-redhat-1]

at org.jboss.as.security.vault.RuntimeVaultReader.getValue(RuntimeVaultReader.java:141) [jboss-as-security-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2]

at org.jboss.as.security.vault.RuntimeVaultReader.getValueAsString(RuntimeVaultReader.java:123) [jboss-as-security-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2]

at org.jboss.as.security.vault.RuntimeVaultReader.retrieveFromVault(RuntimeVaultReader.java:113) [jboss-as-security-7.5.6.Final-redhat-2.jar:7.5.6.Final-redhat-2]

... 38 more

[Host Controller] [31mCaused by: java.lang.IllegalArgumentException: Null input buffer

at javax.crypto.Cipher.doFinal(Cipher.java:2161) [jce.jar:1.8.0_20]

at org.picketbox.util.EncryptionUtil.decrypt(EncryptionUtil.java:134) [picketbox-4.1.2.Final-redhat-1.jar:4.1.2.Final-redhat-1]

at org.picketbox.plugins.vault.PicketBoxSecurityVault.retrieve(PicketBoxSecurityVault.java:293) [picketbox-4.1.2.Final-redhat-1.jar:4.1.2.Final-redhat-1]

... 41 more

(Didn't show the full stack trace, in case you need it just let me know)

Next, the steps I've followed:

Step one:

keytool -genseckey -alias jbossvault -storetype jceks -keyalg AES -keysize 256 -keystore /some/path/jbvault.ks  (same value for both keystore password and alias password)

Step two:

./vault.sh

Please enter a Digit::  0: Start Interactive Session  1: Remove Interactive Session  2: Exit

0

Starting an interactive session

Enter directory to store encrypted files:/some/path/

Enter Keystore URL:/some/path/jbvault.ks

Enter Keystore password: {password, just alphanumeric characters}

Enter Keystore password again: {password, just alphanumeric characters}

Values match

Enter 8 character salt:12345678

Enter iteration count as a number (Eg: 44):25

Enter Keystore Alias:jbossvault

Initializing Vault

Aug 30, 2016 2:45:34 PM org.picketbox.plugins.vault.PicketBoxSecurityVault init

INFO: PBOX000361: Default Security Vault Implementation Initialized and Ready

Step three:

/host=MY_HC_NAME/core-service=vault:add(vault-options=[("KEYSTORE_URL" => "/some/path/jbvault.ks"), ("KEYSTORE_PASSWORD" => "MASKED_VALUE"), ("KEYSTORE_ALIAS" => "jbossvault"), ("SALT" => "12345678"),("ITERATION_COUNT" => "25"), ("ENC_FILE_DIR" => "/some/path/")])

Step four:

Add to host.xml and host-slave.xml the next <vault> element:

<vault>

  <vault-option name="KEYSTORE_URL" value="/some/path/jbvault.ks"/>

  <vault-option name="KEYSTORE_PASSWORD" value="MASKED_VALUE"/>

  <vault-option name="KEYSTORE_ALIAS" value="jbossvault"/>

  <vault-option name="SALT" value="12345678"/>

  <vault-option name="ITERATION_COUNT" value="25"/>

  <vault-option name="ENC_FILE_DIR" value="/some/path/ "/>

</vault>

Then I run again ./vault.sh, enter the same data as Step two, and in the menu it shows (Please enter a Digit::  0: Store a secured attribute  1: Check whether a secured attribute exists  2: Remove secured attribute  3: Exit), I choose 0, enter the required values, and that's all with the vault (of course, taking note of the values below "Configuration should be done as follows:").

For the system property, I stop the server, put the value specified by the vault.sh script inside ${}, and when I start the servers I get the "Security Exception".

Can anyone give me a hint about what could be the reason of the error?

PS: if I select the option "Check whether a secured attribute exists" and enter the values, it founds the variable in the vault.

[ken.wallis]

Presumably, Pedro, you've uncovered an answer.  However, I wanted to provide information we've uncovered in our deployments.

Step 1: You reference using the AES algorithm with a 256 keysize.  It's my understanding that the AES 128 is actually more secure.  For JB EAP 6, the 256 wasn't even an option.  Also, you don't mention a "validity" assignment; you might want to specify rather than allow a default.

You might execute the 'keytool --list' to verify the keystore can be read.

Step 2: You mention executing the "vault.sh" interactively; this will only affect your ability to use env-var references in the values.  Within a script, it most likely doesn't have full access to all possible env-var definitions only those specifically exported; you should not expect shell env-var references to be expanded to their corresponding values at input prompts.  So, if you execute it interactively, you should enter values manually not by \$\{\} reference.  If you execute it command-line, the \$\{\} env-var references work perfectly because the shell expands them prior to sending to the script.  However, use caution and quotations for any that expand to multi-word values because the script will treat the expanded values separately.

In this step, you do not specify entering a "vault-block" value.  Realize, this value acts as a configuration grouping for the vault entries.  It will be identified in each of the vault value references later in the configuration files.  You really should provide a specific definition so that you're confident of the correct reference value later.

Step 3: You don't mention what process was executed but presumably it was the "bin/jboss-cli.sh".  Also, you don't specify "standalone" or "domain" but from your Step 4 references, I'll infer that you're deploying a "domain".  The Step 2 execution should have provided the service initialization definition for both types of deployment.  You should be able to copy and paste the correct variation into the prompt.  However, for this step, you need to ensure the EAP is currently active and that you "connect" prior to entering the initialization command.  Once entered, you should receive a JSON-like response object indicating outcome is successful.

You do not mention whether you were executing initially on the master or a slave.  I strongly recommend the master.  Also, I recommend entering the configuration separately with a host-name reference for every slave being managed by this master. For these, your entry with the MY_HC_NAME replaced by each host-target in succession should suffice.

Step 4: It's quite probable that Step 3 actually insert the Vault configurations into the appropriate runtime configuration file for the current server.  In our case, we keep that file in source control. So, you have to checkout, modify, and commit the targets or the changes are overwritten.  If you keep the host.xml and all host-slave*.xml variants together, you can easily push these, along with the latest domain.xml to all servers in the domain.

Depending on your actual domain organization, you may have multiple EAP server instances defined in differing 'server-group' definitions.  We've found that configuration properties need to be insert within each of these server subsections directly.  This causes some duplication.  This would also be the most likely place to enter configuration property definitions that reference system env-vars. 

One caution is to absolutely confirm that these system env-vars are truly inherited into the JBoss EAP runtime env.  Since multiple scripts are involved in the server startups and since some of these deliberately generate the runtime env of the EAP instance, you may be lossing the values before the EAP instance initializes.  Check whether or not the startup runtime needs to have -Ddefs added to push these references thru.

Thru these steps, all appears and should have proceeded well.  Just be sure all servers have their EAP restarted to integrate your very latest configuration updates. Also, never try to manually modify any of the configuration files while the server is active; it re-writes files on shutdown and your changes will likely be lost.  Further, if the Step 2 execution properly insert the Vault configuration entries into the file, do not rearrange their location; the XSD dictates where within the file they belong and moving can cause issues (ex: domain's host*.xml places vault entries at the very head before 'management' section.  The "vault.sh" output should indicate proper placement.

Beyond your specified steps, it appears you're attempting to re-execute the vault.sh to add your actual key-value string pairs.  Your statements regarding use of env-vars can be misleading/misinterpreted easily.  So, repeating my earlier statement.  If you execute vault.sh by command-line, these will be expanded by the shell; if you execute interactively, expect to manually expand the references at appropriate prompts.  Doing this, you should be able to execute the command readily.

A Step 5 that you imply:  editting the domain host* configuration file to insert Vault value references.  You do not specify the vault value references nor their syntax.  Also, you previously failed to mention entering a "vault-block" value. So, I presume the values entered were not resolvable and caused the server to encounter startup issues.

The proper reference entry would take the form:  \$\{VAULT::\$\{vault.block\}::\$\{vault.key.name.foo\}::1\}  where your values are substituted within this syntax in place of vault.block and vault.key.name.foo.

ex:  ...<password>\$\{VAULT::ex::\$\{db.pwd\}::1\}  where the "db.pwd" key has been insert in the vault using vault-block "ex"

Another Step 6 that you do not mention:  Since you're deploying a domain, you need to push the vault to each of the slaves.  This actually consists of the keystore identified in Step 1 and another file in the same folder with a similar basename and a ".dat" (ex:  jboss/vault/vault.keystore and jboss/vault/VAULT.dat).  Push these as a pair to the slaves!  Also, ensure they reside in the correct vault-folder on each slave; I strongly recommend making them all consistent.  If you don't push both files, the slaves will generate their own but will encounter runtime issues.

Thereafter, any changes to the vault should always be made on the master and pushed to the slaves.