Tag: hashicorp

Hashicorp Vault docker installation and client testing

Posted on by Bian Xi Leave a Comment on Hashicorp Vault docker installation and client testing

Hashicorp Vault docker installation and client testing

Vault Server Installation

Create one folder with 3 subfolders

mkdir -p vault/{config,file,logs}

Create vault configuration file

Create vault/config/vault.json

{
  "backend": {
    "file": {
      "path": "/vault/file"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Create docker-compose.yml

Create file vault/docker-compose.yml

version: '3.7'
services:
  vault:
    image: vault:latest
    container_name: vault
    ports:
      - "8200:8200"
    restart: unless-stopped
    volumes:
      -  ./logs:/vault/logs
      -  ./file:/vault/file
      -  ./config:/vault/config
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -config=/vault/config/vault.json

Create container

Run docker-compose command in vault folder

cd vault
docker-compose up -d

Access WebUI

Access http://localhost:8200/ from browser

  • Select 5 as Key shares, and 3 as Key threshold, and Initialize
  • Download keys into a Json file
  • Use 3 keys to unseal vault
  • Use root token to login

Client installation

Ubuntu x86

  • Add the HashiCorp GPG key
# curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
  • Add the official HashiCorp Linux repository
# apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
  • Install vault
# apt-get install vault
  • Verify
# vault

Connect to vault

  • Set environment
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ export VAULT_TOKEN="<token>"
  • Check status
$ vault status
...
Sealed          false
...

Secrets operations

Subcommand kv v1 kv v2 Description
delete x x Delete versions of secrets stored in K/V
destroy x Permanently remove one or more versions of secrets
enable-versioning x Turns on versioning for an existing K/V v1 store
get x x Retrieve data
list x x List data or secrets
metadata x Interact with Vault\'s Key-Value storage
patch x Update secrets without overwriting existing secrets
put x x Sets or update secrets (this replaces existing secrets)
rollback x Rolls back to a previous version of secrets
undelete x Restore the deleted version of secrets

Example:

vault-getting-started:~# vault login root
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                root
token_accessor       rSn3h08ikdez4zch5ghr4wYY
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
vault-getting-started:~# vault kv put secret/hello foo=world
Key              Value
---              -----
created_time     2021-11-25T06:15:45.332182013Z
deletion_time    n/a
destroyed        false
version          1
vault-getting-started:~# vault kv put secret/hello foo=world excited=yes
Key              Value
---              -----
created_time     2021-11-25T06:15:48.808651794Z
deletion_time    n/a
destroyed        false
version          2
vault-getting-started:~# vault kv get secret/hello
====== Metadata ======
Key              Value
---              -----
created_time     2021-11-25T06:15:48.808651794Z
deletion_time    n/a
destroyed        false
version          2

===== Data =====
Key        Value
---        -----
excited    yes
foo        world
vault-getting-started:~# vault kv get -field=excited secret/hello
yes
vault-getting-started:~# vault kv get -format=json secret/hello | jq -r .data.data.excited
yes
vault-getting-started:~# vault kv delete secret/hello
Success! Data deleted (if it existed) at: secret/hello
vault-getting-started:~#

Secret Engine

The driver to save secret in different way, type of secret.

List

Every path has it's own secret type

$ vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_78189996    per-token private secret storage
identity/     identity     identity_ac07951e     identity store
kv/           kv           kv_15087625           n/a
secret/       kv           kv_4b990c45           key/value secret storage
sys/          system       system_adff0898       system endpoints used for control, policy and debugging

Enable

Set one path to specific secret type

$ vault secrets enable -path=kv kv

Success! Enabled the kv secrets engine at: kv/

or

$ vault secrets enable kv

Create secret

$ vault kv put kv/hello target=world
Success! Data written to: kv/hello

Get secret

$ vault kv get kv/hello

===== Data =====
Key       Value
---       -----
target    world

Delete secret

$ vault kv delete kv/hello
Success! Data deleted (if it existed) at: kv/hello

List

$ vault kv list kv/

Keys
----
hello

Disable

$ vault secrets disable kv/

Success! Disabled the secrets engine (if it existed) at: kv/

Dynamic Secrets

When using secret engine such as aws engine.

$ vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

More Info: Dynamic Secrets

Authentication

Token

  • Create token
$ vault token create
Key                  Value
---                  -----
token                s.iyNUhq8Ov4hIAx6snw5mB2nL
token_accessor       maMfHsZfwLB6fi18Zenj3qh6
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
  • Login
$ vault login s.iyNUhq8Ov4hIAx6snw5mB2nL

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.iyNUhq8Ov4hIAx6snw5mB2nL
token_accessor       maMfHsZfwLB6fi18Zenj3qh6
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
  • Revoke
$ vault token revoke s.iyNUhq8Ov4hIAx6snw5mB2nL

Success! Revoked token (if it existed)

GitHub

  • Enable
$ vault auth enable github

Success! Enabled github auth method at: github/
  • Set organization
$ vault write auth/github/config organization=hashicorp

Success! Data written to: auth/github/config
  • Configure the GitHub engineering team authentication to be granted the default and applications policies
$ vault write auth/github/map/teams/engineering value=default,applications

Success! Data written to: auth/github/map/teams/engineering
  • List
$ vault auth list

Path       Type      Description
----       ----      -----------
github/    github    n/a
token/     token     token based credentials
  • Set login method
$ vault login -method=github

GitHub Personal Access Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.DNtKCjVQ1TxAzgMqtDuwjjC2
token_accessor         e7zLJuPg2tLpav66ZSu5AyDC
token_duration         768h
token_renewable        true
token_policies         [default applications]
token_meta_org         hashicorp
token_meta_username    my-user
  • Login
$ vault login root
  • Revoke all tokens generated the github auth method
$ vault token revoke -mode path auth/github
  • Disable the github auth method
$ vault auth disable github

Success! Disabled the auth method (if it existed) at: github/

Policy

  • Policy for token

The policy path secret/data/* is related to all secret path secret/*.
The policy path secret/data/foo is related to secret path secret/foo.

  • Policy for approle

The policy path secret/approle/* is related to role_id + secret_id authentication.

Policy for token

  • Create
$ vault policy write my-policy - << EOF
# Dev servers have version 2 of KV secrets engine mounted by default, so will
# need these paths to grant permissions:
path "secret/data/*" {
  capabilities = ["create", "update"]
}

path "secret/data/foo" {
  capabilities = ["read"]
}
EOF
  • List
$ vault policy list

default
my-policy
root
  • Show
$ vault policy read my-policy

# Dev servers have version 2 of KV secrets engine mounted by default, so will
# need these paths to grant permissions:
path "secret/data/*" {
  capabilities = ["create", "update"]
}

path "secret/data/foo" {
  capabilities = ["read"]
}
  • Create token
$ export VAULT_TOKEN="$(vault token create -field token -policy=my-policy)"
  • Check policy
$ vault token lookup | grep policies
policies            [default my-policy]
  • Write success
$ vault kv put secret/creds password="my-long-password"

Key              Value
---              -----
created_time     2018-05-22T18:05:42.537496856Z
deletion_time    n/a
destroyed        false
version          1
  • Write failed
$ vault kv put secret/foo robot=beepboop

Error writing data to secret/data/foo: Error making API request.

URL: PUT http://localhost:8200/v1/secret/data/foo
Code: 403. Errors:

* 1 error occurred:
  * permission denied

Policy for approle

  • Enable
$ vault auth enable approle
Success! Enabled approle auth method at: approle/
  • Create my-role link to my-policy
$ vault write auth/approle/role/my-role \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=my-policy
Success! Data written to: auth/approle/role/my-role
  • Create role_id
$ export ROLE_ID="$(vault read -field=role_id auth/approle/role/my-role/role-id)"
  • Create secret_id
$ export SECRET_ID="$(vault write -f -field=secret_id auth/approle/role/my-role/secret-id)"
  • Login
$ vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"
Key                     Value
---                     -----
token                   s.Sh9h1wZ9ycATeSaASoOQvovr
token_accessor          xCgUIu6WWLM9opkEkAiNLsRc
token_duration          20m
token_renewable         true
token_policies          ["default" "my-policy"]
identity_policies       []
policies                ["default" "my-policy"]
token_meta_role_name    my-role

References

vault-docker
Install Vault

Troubleshooting Hashicorp Vault SSH Certificate Login

Posted on by Bian Xi Leave a Comment on Troubleshooting Hashicorp Vault SSH Certificate Login

Troubleshooting Hashicorp Vault SSH Certificate Login

If can not login via SSH, normally can use -vvv as SSH option in SSH client command to verify, but it could be very long to read.

Another way is use systemctl status sshd command at server side to check the error.

For example, the output of systemctl status sshd got following lines,

Dec 12 00:40:37 example-host systemd[1]: Started OpenBSD Secure Shell server.
Dec 12 00:40:54 example-host sshd[22712]: error: Certificate invalid: expired
Dec 12 00:40:54 example-host sshd[22712]: Connection closed by authenticating user ubuntu 101.78.78.154 port 53369 [preauth]
Dec 12 00:41:12 example-host sshd[22716]: error: Certificate invalid: name is not a listed principal
Dec 12 00:41:12 example-host sshd[22716]: Connection closed by authenticating user ubuntu 101.78.78.154 port 53372 [preauth]

The first error shows vault signed certificate at client side had expired, need to rerun vault command to regenerate signed certificate.

The second error shows the user name was used in client is not listed in vault signed certificate, so need to use correct user name or configure a new role in the vault.

Admin Token for AppRole in Hashicorp Vault

Posted on by Bian Xi Leave a Comment on Admin Token for AppRole in Hashicorp Vault

Admin Token for AppRole in Hashicorp Vault

As suggested, root token should not be used, and it should be revoked immediately after used.

Root token

Follow the steps in page below to create a new root token and revoke it after used.

Generate a new root token for Hashicorp Vault

Admin token

For example, SSH secret engine, following admin policy can be created

vault policy write ssh-admin-policy - << EOF
# SSH secret engine
path "ssh-client-signer/sign/*" {
  capabilities = ["create", "read", "update", "delete", "sudo", "list" ]
}

# Mount the AppRole auth method
path "sys/auth/approle" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Configure the AppRole auth method
path "sys/auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# Create and manage roles
path "auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

##### Add other requirement if required. For example
# Write test data
# Set the path to "secret/data/mysql/*" if you are running `kv-v2`
path "secret/mysql/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
EOF

Then create token under this policy

vault token create -field token -policy=ssh-admin-policy

The using this token follow the steps in page below:

Signed SSH Certificates using Hashicorp Vault in Practice

  • Generate role_id and secret_id
  • Login using role_id and secret_id
  • Generate SSH policy token
  • Use SSH policy token to generate signed public key
  • Use the signed public key and private key to login to remote system

Renew token itself

To get renew token before expired, run following command

vault token renew

The expire time can be view using following command

vault token lookup

References

Tokens
AppRole Pull Authentication

Generate a new root token for Hashicorp Vault

Posted on by Bian Xi Leave a Comment on Generate a new root token for Hashicorp Vault

Generate a new root token for Hashicorp Vault

To generate a new root token without old token.

Steps

  • run shell in vault docker
$ docker exec -it vault sh
  • Unseal if haven't
$ vault operator unseal
  • Get Nonce and OTP
$ vault operator generate-root -init
Nonce         15565c79-cc9e-5e64b986-8506e7bd1918
...
OTP           mOXx7iVimjE6LXQ2Zna6NA==
...
  • Provide unseal key to retrieve Encoded Token

Note: Beware of last -.

echo $UNSEAL_KEY | vault operator generate-root -nonce=f67f4da3... -

Note: run vault operator generate-root only, will show nonce key.

The last person will get Encoded Token

Encoded Token    IxJpyqxn3YafOGhqhvP6cQ==
  • Get root token
vault operator generate-root \  -decode=IxJpyqxn3YafOGhqhvP6cQ== \  -otp=mOXx7iVimjE6LXQ2Zna6NA==

Revoke token

Note: The root token can be used to revoke itself.

Revoke a token and all the token's children:

$ vault token revoke 96ddf4bc-d217-f3ba-f9bd-017055595017Success! Revoked token (if it existed)

Revoke a token leaving the token's children:

$ vault token revoke -mode=orphan 96ddf4bc-d217-f3ba-f9bd-017055595017Success! Revoked token (if it existed)

Revoke a token by accessor:

$ vault token revoke -accessor 9793c9b3-e04a-46f3-e7b8-748d7da248daSuccess! Revoked token (if it existed)

References

Generate Root Tokens Using Unseal Keys
token revoke

Signed SSH Certificates using Hashicorp Vault in Practice

Posted on by Bian Xi Leave a Comment on Signed SSH Certificates using Hashicorp Vault in Practice

Signed SSH Certificates using Hashicorp Vault in Practice

In previous setup, root token is used all the time, which should not be the case in real situation.

Steps

Preparation of Vault and SSH Server

The steps can follow Signed SSH Certificates using Hashicorp Vault

Using root token

export VAULT_TOKEN=<root_token>
export VAULT_ADDR='http://host:8200'

Create Policy

$ vault policy write my-policy - << EOF
path "ssh-client-signer/sign/my-role" {
  capabilities = ["create", "update", "read"]
}
EOF

Get Policy token

$ vault token create -field token -policy=my-policy

Test Policy in SSH client machine

  • Using token in SSH client
$ export VAULT_TOKEN="<policy_token>"
$ vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
  • SSH into target system using signed public key and private key in SSH client
ssh -i signed-cert.pub -i ~/.ssh/id_rsa ubuntu@host

Enable approle

vault auth enable approle

Create role

vault write auth/approle/role/my-role \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=my-policy

Get role_id and secret_id

vault read -field=role_id auth/approle/role/my-role/role-id
vault write -f -field=secret_id auth/approle/role/my-role/secret-id

Login from SSH client machine

export VAULT_ADDR='http://host:8200'
export ROLE_ID=<role_id>
export SECRET_ID=<secret_id>
vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

Result contains token

Key                     Value
---                     -----
token                   s.Sh9h1wZ9ycATeSaASoOQvovr
...

Check the token info

vault token lookup | grep policies
policies            [default my-policy]

Test Policy in SSH client machine

  • Using token in SSH client
$ export VAULT_TOKEN="<policy_token>"
$ vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
  • SSH into target system using signed public key and private key in SSH client
ssh -i signed-cert.pub -i ~/.ssh/id_rsa ubuntu@host

Consideration

Disable password authentication

If tested finished, password authentication can be set to no:

PasswordAuthentication no

Time valid for SECRET_ID

If the secret_id expires so fast, then needs to have master token to generate secret_id again, because the policy token is generated using role_id and secret_id pair.

If don't use approle, then all SSH clients can use policy token. Still thinking the advantage of using role_id and secret_id.

Use admin token instead

The root token should not be used as recommended. Will try admin token later.

References

Policies

Signed SSH Certificates using Hashicorp Vault

Posted on by Bian Xi Leave a Comment on Signed SSH Certificates using Hashicorp Vault

Signed SSH Certificates using Hashicorp Vault

The idear of signed SSH certificates verification is to use valid (signed) SSH certificate to be verified by SSH server or by SSH client, or by both.

Mechanism

Vaildated by SSH server

Client retrieves signed public key which issued by the CA key in Vault. This key has short expiry date.

Server uses the CA public key configured in SSH configuration, validates the client public key issued by Vault.

Validated by SSH client

This is to validate server public whether signed by Vault by place public key in .ssh/known_hosts file. This key should have long expiry date.

Steps

Vault Server preparation

  • Login into Vault
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="<token>"
  • Enable SSH secret engine
$ vault secrets enable -path=ssh-client-signer ssh
Successfully mounted 'ssh' at 'ssh-client-signer'!
  • Configure CA
$ vault write ssh-client-signer/config/ca generate_signing_key=true
Key             Value
---             -----
public_key      ssh-rsa AAAAB3NzaC1yc2EA...
  • Create Role

Beware of *allowed_users" and "default_user", they must be set correctly.

$ vault write ssh-client-signer/roles/my-role -<<"EOH"
{
  "allow_user_certificates": true,
  "allowed_users": "*",
  "allowed_extensions": "permit-pty,permit-port-forwarding",
  "default_extensions": [
    {
      "permit-pty": ""
    }
  ],
  "key_type": "ca",
  "default_user": "ubuntu",
  "ttl": "30m0s"
}
EOH

SSH Server Setup

  • Login to Vault
export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN="<token>"
  • Save CA key
# vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem
  • Configure SSHD

Add following lines in /etc/ssh/sshd_config

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
  • Restart SSHD
# systemctl restart sshd

SSH Client

  • Generate SSH key pair if haven't done
$ ssh-keygen -t rsa -C "user@example.com"

This will generate a pair of files, .ssh/id_rsa and .ssh/id_rsa.pub.

  • Generate and save signed public key using client public key
$ vault write -field=signed_key ssh-client-signer/sign/my-role \
    public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
  • Verify signed key (optional)

This can verify the valid period and user

$ ssh-keygen -Lf ~/.ssh/signed-cert.pub
...
        Valid: from 2021-11-27T17:51:29 to 2021-11-27T18:21:59
        Principals: 
                ubuntu
...
  • Login to server using both signed key and private key
$ ssh -i signed-cert.pub -i ~/.ssh/id_rsa username@10.0.23.5

Note: Add following configure in /etc/ssh/sshd_config if got error __userauth_pubkey: certificate signature algorithm ssh-rsa: signature algorithm not supported [preauth]__

CASignatureAlgorithms ^ssh-rsa

References

Signed SSH Certificates
Leveraging Signed SSH for Remote Access with Vault