Category: vault

Commands for Signed SSH Certificates using Hashicorp Vault

Commands for Signed SSH Certificates using Hashicorp Vault

List down the commands required.

Client

Generate SSH Admin token (One time)

export VAULT_ADDR='https://vault.bx.net:8200'
export VAULT_TOKEN="<ROOT_TOKEN>"

vault token create -field token -policy=ssh-admin-policy

Renew Admin token

export VAULT_TOKEN="<SSH_ADMIN_TOKEN>"
vault token renew

Generate signed certificate

export VAULT_TOKEN="<SSH_ADMIN_TOKEN>"
vault token lookup
vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > ~/.ssh/signed-cert.pub

SSH using signed certificate

ssh -i ~/.ssh/signed-cert.pub -i ~/.ssh/id_rsa <host>

Server

Save CA key

export VAULT_ADDR='https://vault.bx.net:8200'
export VAULT_TOKEN="<ROOT_TOKEN>"

vault read -field=public_key ssh-client-signer/config/ca > /etc/ssh/trusted-user-ca-keys.pem

Configure /etc/ssh/sshd_config

Add following lines in /etc/ssh/sshd_config

TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem
CASignatureAlgorithms ^ssh-rsa

Note: Comment out last line if SSH got error

Troubleshooting

Server SSL cert

The SSL cert in vault server needs to be trusted by local client, otherwise, following server occurred.

Error writing data to ssh-client-signer/sign/my-role: Put "<role_name>": x509: certificate signed by unknown authority

References

Bitwarden Docker Installation

Bitwarden Docker Installation

Update: Bitwarden could not detect new and update password in browsers in most of cases. No matter how easy it can be used, without this auto detection feature, it is useless.

Bitwarden is an opensource password manager, can be self-hosted, and can be installed as docker container. It supports many browsers and OSes.

Steps

Create docker-compose.yaml

Create docker-compose.yaml, and make sure

  • SIGNUPS_ALLOWED is 'true'
# docker-compose.yml
version: '3'

services:
  bitwarden:
    image: bitwardenrs/server
    restart: always
    ports:
      - 8000:80
    volumes:
      - ./bw-data:/data
    environment:
      WEBSOCKET_ENABLED: 'true' # Required to use websockets
      SIGNUPS_ALLOWED: 'true'   # set to false to disable signups

Create

Run following command, in the directory contains docker-compose.yaml

docker-compose up -d

Configure HTTPS in NGINX

Without HTTPS, bitwarden doesn't allow new user registration.

Add following statements in server location definition.

server {
    server_name  nginx_host;
    listen 443 ssl;

    ...

    location /bw {
        rewrite /bw(.*) /$1 break;
        proxy_pass  'http://192.168.1.222:8000';
        proxy_redirect     off;
        proxy_set_header   Host $host;
    }
    ...
}

Then the URL of bitwarden will be https://nginx_host/bw/

Register

Access https://nginx_host/bw/, and register email and password.

Disable new user creation

  • Destory old bitwarden instance
docker-compose down
  • Update docker-compose.yaml file
SIGNUPS_ALLOWED: 'false';
  • Recreate instance
docker-compose up -d

Trun on 2FA

In website, go to Settings => Two-step Login ...

Install clients

Go to bitwarden website to install.

References

Run Bitwarden Password Manager in Docker Container
Install and Deploy - Linux
The Best Password Managers to Secure Your Digital Life
Host your own FREE Password Manager with your Synology NAS
Install and Sync All of Your Devices
Connect Clients to your Instance
Request Hosting Installation Id & Key

Hashicorp Vault docker installation and client testing

Hashicorp Vault docker installation and client testing

Vault Server Installation

Create one folder with 3 subfolders

mkdir -p vault/{config,file,logs}

Create vault configuration file

Create vault/config/vault.json

{
  "backend": {
    "file": {
      "path": "/vault/file"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Create docker-compose.yml

Create file vault/docker-compose.yml

version: '3.7'
services:
  vault:
    image: vault:latest
    container_name: vault
    ports:
      - "8200:8200"
    restart: unless-stopped
    volumes:
      -  ./logs:/vault/logs
      -  ./file:/vault/file
      -  ./config:/vault/config
    cap_add:
      - IPC_LOCK
    entrypoint: vault server -config=/vault/config/vault.json

Create container

Run docker-compose command in vault folder

cd vault
docker-compose up -d

Access WebUI

Access http://localhost:8200/ from browser

  • Select 5 as Key shares, and 3 as Key threshold, and Initialize
  • Download keys into a Json file
  • Use 3 keys to unseal vault
  • Use root token to login

Client installation

Ubuntu x86

  • Add the HashiCorp GPG key
# curl -fsSL https://apt.releases.hashicorp.com/gpg | apt-key add -
  • Add the official HashiCorp Linux repository
# apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"
  • Install vault
# apt-get install vault
  • Verify
# vault

Connect to vault

  • Set environment
$ export VAULT_ADDR='http://127.0.0.1:8200'
$ export VAULT_TOKEN="<token>"
  • Check status
$ vault status
...
Sealed          false
...

Secrets operations

Subcommand kv v1 kv v2 Description
delete x x Delete versions of secrets stored in K/V
destroy x Permanently remove one or more versions of secrets
enable-versioning x Turns on versioning for an existing K/V v1 store
get x x Retrieve data
list x x List data or secrets
metadata x Interact with Vault\'s Key-Value storage
patch x Update secrets without overwriting existing secrets
put x x Sets or update secrets (this replaces existing secrets)
rollback x Rolls back to a previous version of secrets
undelete x Restore the deleted version of secrets

Example:

vault-getting-started:~# vault login root
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                root
token_accessor       rSn3h08ikdez4zch5ghr4wYY
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
vault-getting-started:~# vault kv put secret/hello foo=world
Key              Value
---              -----
created_time     2021-11-25T06:15:45.332182013Z
deletion_time    n/a
destroyed        false
version          1
vault-getting-started:~# vault kv put secret/hello foo=world excited=yes
Key              Value
---              -----
created_time     2021-11-25T06:15:48.808651794Z
deletion_time    n/a
destroyed        false
version          2
vault-getting-started:~# vault kv get secret/hello
====== Metadata ======
Key              Value
---              -----
created_time     2021-11-25T06:15:48.808651794Z
deletion_time    n/a
destroyed        false
version          2

===== Data =====
Key        Value
---        -----
excited    yes
foo        world
vault-getting-started:~# vault kv get -field=excited secret/hello
yes
vault-getting-started:~# vault kv get -format=json secret/hello | jq -r .data.data.excited
yes
vault-getting-started:~# vault kv delete secret/hello
Success! Data deleted (if it existed) at: secret/hello
vault-getting-started:~#

Secret Engine

The driver to save secret in different way, type of secret.

List

Every path has it's own secret type

$ vault secrets list

Path          Type         Accessor              Description
----          ----         --------              -----------
cubbyhole/    cubbyhole    cubbyhole_78189996    per-token private secret storage
identity/     identity     identity_ac07951e     identity store
kv/           kv           kv_15087625           n/a
secret/       kv           kv_4b990c45           key/value secret storage
sys/          system       system_adff0898       system endpoints used for control, policy and debugging

Enable

Set one path to specific secret type

$ vault secrets enable -path=kv kv

Success! Enabled the kv secrets engine at: kv/

or

$ vault secrets enable kv

Create secret

$ vault kv put kv/hello target=world
Success! Data written to: kv/hello

Get secret

$ vault kv get kv/hello

===== Data =====
Key       Value
---       -----
target    world

Delete secret

$ vault kv delete kv/hello
Success! Data deleted (if it existed) at: kv/hello

List

$ vault kv list kv/

Keys
----
hello

Disable

$ vault secrets disable kv/

Success! Disabled the secrets engine (if it existed) at: kv/

Dynamic Secrets

When using secret engine such as aws engine.

$ vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

More Info: Dynamic Secrets

Authentication

Token

  • Create token
$ vault token create
Key                  Value
---                  -----
token                s.iyNUhq8Ov4hIAx6snw5mB2nL
token_accessor       maMfHsZfwLB6fi18Zenj3qh6
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
  • Login
$ vault login s.iyNUhq8Ov4hIAx6snw5mB2nL

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.iyNUhq8Ov4hIAx6snw5mB2nL
token_accessor       maMfHsZfwLB6fi18Zenj3qh6
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]
  • Revoke
$ vault token revoke s.iyNUhq8Ov4hIAx6snw5mB2nL

Success! Revoked token (if it existed)

GitHub

  • Enable
$ vault auth enable github

Success! Enabled github auth method at: github/
  • Set organization
$ vault write auth/github/config organization=hashicorp

Success! Data written to: auth/github/config
  • Configure the GitHub engineering team authentication to be granted the default and applications policies
$ vault write auth/github/map/teams/engineering value=default,applications

Success! Data written to: auth/github/map/teams/engineering
  • List
$ vault auth list

Path       Type      Description
----       ----      -----------
github/    github    n/a
token/     token     token based credentials
  • Set login method
$ vault login -method=github

GitHub Personal Access Token (will be hidden):
Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                    Value
---                    -----
token                  s.DNtKCjVQ1TxAzgMqtDuwjjC2
token_accessor         e7zLJuPg2tLpav66ZSu5AyDC
token_duration         768h
token_renewable        true
token_policies         [default applications]
token_meta_org         hashicorp
token_meta_username    my-user
  • Login
$ vault login root
  • Revoke all tokens generated the github auth method
$ vault token revoke -mode path auth/github
  • Disable the github auth method
$ vault auth disable github

Success! Disabled the auth method (if it existed) at: github/

Policy

  • Policy for token

The policy path secret/data/* is related to all secret path secret/*.
The policy path secret/data/foo is related to secret path secret/foo.

  • Policy for approle

The policy path secret/approle/* is related to role_id + secret_id authentication.

Policy for token

  • Create
$ vault policy write my-policy - << EOF
# Dev servers have version 2 of KV secrets engine mounted by default, so will
# need these paths to grant permissions:
path "secret/data/*" {
  capabilities = ["create", "update"]
}

path "secret/data/foo" {
  capabilities = ["read"]
}
EOF
  • List
$ vault policy list

default
my-policy
root
  • Show
$ vault policy read my-policy

# Dev servers have version 2 of KV secrets engine mounted by default, so will
# need these paths to grant permissions:
path "secret/data/*" {
  capabilities = ["create", "update"]
}

path "secret/data/foo" {
  capabilities = ["read"]
}
  • Create token
$ export VAULT_TOKEN="$(vault token create -field token -policy=my-policy)"
  • Check policy
$ vault token lookup | grep policies
policies            [default my-policy]
  • Write success
$ vault kv put secret/creds password="my-long-password"

Key              Value
---              -----
created_time     2018-05-22T18:05:42.537496856Z
deletion_time    n/a
destroyed        false
version          1
  • Write failed
$ vault kv put secret/foo robot=beepboop

Error writing data to secret/data/foo: Error making API request.

URL: PUT http://localhost:8200/v1/secret/data/foo
Code: 403. Errors:

* 1 error occurred:
  * permission denied

Policy for approle

  • Enable
$ vault auth enable approle
Success! Enabled approle auth method at: approle/
  • Create my-role link to my-policy
$ vault write auth/approle/role/my-role \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=my-policy
Success! Data written to: auth/approle/role/my-role
  • Create role_id
$ export ROLE_ID="$(vault read -field=role_id auth/approle/role/my-role/role-id)"
  • Create secret_id
$ export SECRET_ID="$(vault write -f -field=secret_id auth/approle/role/my-role/secret-id)"
  • Login
$ vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"
Key                     Value
---                     -----
token                   s.Sh9h1wZ9ycATeSaASoOQvovr
token_accessor          xCgUIu6WWLM9opkEkAiNLsRc
token_duration          20m
token_renewable         true
token_policies          ["default" "my-policy"]
identity_policies       []
policies                ["default" "my-policy"]
token_meta_role_name    my-role

References

vault-docker
Install Vault

Troubleshooting Hashicorp Vault SSH Certificate Login

Troubleshooting Hashicorp Vault SSH Certificate Login

If can not login via SSH, normally can use -vvv as SSH option in SSH client command to verify, but it could be very long to read.

Another way is use systemctl status sshd command at server side to check the error.

For example, the output of systemctl status sshd got following lines,

Dec 12 00:40:37 example-host systemd[1]: Started OpenBSD Secure Shell server.
Dec 12 00:40:54 example-host sshd[22712]: error: Certificate invalid: expired
Dec 12 00:40:54 example-host sshd[22712]: Connection closed by authenticating user ubuntu 101.78.78.154 port 53369 [preauth]
Dec 12 00:41:12 example-host sshd[22716]: error: Certificate invalid: name is not a listed principal
Dec 12 00:41:12 example-host sshd[22716]: Connection closed by authenticating user ubuntu 101.78.78.154 port 53372 [preauth]

The first error shows vault signed certificate at client side had expired, need to rerun vault command to regenerate signed certificate.

The second error shows the user name was used in client is not listed in vault signed certificate, so need to use correct user name or configure a new role in the vault.

Operation Model using Hashicorp Vault

Operational Model using Hashicorp Vault

Steps

Preprepation

This is to create an operational task to pass it to operator. For example, SSH to host.

  • Vault Admin creates AppRole (role_id), pass role_id to Operator as operational task reference id

  • Vault Admin creates Admin Token (admin_token), pass it to App Token Admin

Now, Operator has a operational task reference id, role_id.

Change request

  • Task Requester submit request to Operator

  • Operator submit the request to App Token Admin

  • App Token Admin uses Admin Token against AppRole to create Secret ID (secret_id), pass it to Operator

  • Operator use role_id and secret_id login to retrieve App token, and retrieve credential, such as signed public key in SSH case

  • Operator pass credential to Task Performer

  • Then complete change task.

Roles

  • Vault Admin - can access vault to generate root token
  • App Token Admin - manage App operations
  • Operator - manage and issue AppRole credentials
  • Task Requester - Change requester
  • Task Performer - Change implemenator

Token or Keys

Root Token - Manage Vault
App Token - Manage App, for example, SSH App as whole
Role ID - Identify AppRole, for example, Project or Host
Secret ID - Retrieve Task Token
Task Token - Retrieve credential

Root Token should be revoken after used
App Token should be securely managed
Secret ID and Task Token should have short life

Other consideration

Secret ID and Task Token should be held by operator or task performer, this can be decided by how AppRole managed. If AppRole cannot restrict the task to be performed, then only can pass credential to task performer.

In order to identify the host, the Host Key Signing mentioned in following page should be considered.

Signed SSH Certificates

Cons

There is no clear info on the machines managed.

References

Admin Token for AppRole in Hashicorp Vault

Enable SSL for Hashicorp Vault

Enable SSL for Hashicorp Vault

Update configuration

vault.json

cat vault.json
{
  "backend": {
    "file": {
      "path": "/vault/file"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_enable": 1,
      "tls_cert_file": "/vault/config/cert.pem",
      "tls_key_file": "/vault/config/privkey.pem"
    }
  },
  "ui": true
}

Copy certificate files

Copy into /vault/config folder

Restart vault container

Unseal

References

Hashicorp - SSL/TLS Question #212

Admin Token for AppRole in Hashicorp Vault

Admin Token for AppRole in Hashicorp Vault

As suggested, root token should not be used, and it should be revoked immediately after used.

Root token

Follow the steps in page below to create a new root token and revoke it after used.

Generate a new root token for Hashicorp Vault

Admin token

For example, SSH secret engine, following admin policy can be created

vault policy write ssh-admin-policy - << EOF
# SSH secret engine
path "ssh-client-signer/sign/*" {
  capabilities = ["create", "read", "update", "delete", "sudo", "list" ]
}

# Mount the AppRole auth method
path "sys/auth/approle" {
  capabilities = [ "create", "read", "update", "delete", "sudo" ]
}

# Configure the AppRole auth method
path "sys/auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete" ]
}

# Create and manage roles
path "auth/approle/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

# Write ACL policies
path "sys/policies/acl/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}

##### Add other requirement if required. For example
# Write test data
# Set the path to "secret/data/mysql/*" if you are running `kv-v2`
path "secret/mysql/*" {
  capabilities = [ "create", "read", "update", "delete", "list" ]
}
EOF

Then create token under this policy

vault token create -field token -policy=ssh-admin-policy

The using this token follow the steps in page below:

Signed SSH Certificates using Hashicorp Vault in Practice

  • Generate role_id and secret_id
  • Login using role_id and secret_id
  • Generate SSH policy token
  • Use SSH policy token to generate signed public key
  • Use the signed public key and private key to login to remote system

Renew token itself

To get renew token before expired, run following command

vault token renew

The expire time can be view using following command

vault token lookup

References

Tokens
AppRole Pull Authentication

Generate a new root token for Hashicorp Vault

Generate a new root token for Hashicorp Vault

To generate a new root token without old token.

Steps

  • run shell in vault docker
$ docker exec -it vault sh
  • Unseal if haven't
$ vault operator unseal
  • Get Nonce and OTP
$ vault operator generate-root -init
Nonce         15565c79-cc9e-5e64b986-8506e7bd1918
...
OTP           mOXx7iVimjE6LXQ2Zna6NA==
...
  • Provide unseal key to retrieve Encoded Token

Note: Beware of last -.

echo $UNSEAL_KEY | vault operator generate-root -nonce=f67f4da3... -

Note: run vault operator generate-root only, will show nonce key.

The last person will get Encoded Token

Encoded Token    IxJpyqxn3YafOGhqhvP6cQ==
  • Get root token
vault operator generate-root \  -decode=IxJpyqxn3YafOGhqhvP6cQ== \  -otp=mOXx7iVimjE6LXQ2Zna6NA==

Revoke token

Note: The root token can be used to revoke itself.

Revoke a token and all the token's children:

$ vault token revoke 96ddf4bc-d217-f3ba-f9bd-017055595017Success! Revoked token (if it existed)

Revoke a token leaving the token's children:

$ vault token revoke -mode=orphan 96ddf4bc-d217-f3ba-f9bd-017055595017Success! Revoked token (if it existed)

Revoke a token by accessor:

$ vault token revoke -accessor 9793c9b3-e04a-46f3-e7b8-748d7da248daSuccess! Revoked token (if it existed)

References

Generate Root Tokens Using Unseal Keys
token revoke

Signed SSH Certificates using Hashicorp Vault in Practice

Signed SSH Certificates using Hashicorp Vault in Practice

In previous setup, root token is used all the time, which should not be the case in real situation.

Steps

Preparation of Vault and SSH Server

The steps can follow Signed SSH Certificates using Hashicorp Vault

Using root token

export VAULT_TOKEN=<root_token>
export VAULT_ADDR='http://host:8200'

Create Policy

$ vault policy write my-policy - << EOF
path "ssh-client-signer/sign/my-role" {
  capabilities = ["create", "update", "read"]
}
EOF

Get Policy token

$ vault token create -field token -policy=my-policy

Test Policy in SSH client machine

  • Using token in SSH client
$ export VAULT_TOKEN="<policy_token>"
$ vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
  • SSH into target system using signed public key and private key in SSH client
ssh -i signed-cert.pub -i ~/.ssh/id_rsa ubuntu@host

Enable approle

vault auth enable approle

Create role

vault write auth/approle/role/my-role \
    secret_id_ttl=10m \
    token_num_uses=10 \
    token_ttl=20m \
    token_max_ttl=30m \
    secret_id_num_uses=40 \
    token_policies=my-policy

Get role_id and secret_id

vault read -field=role_id auth/approle/role/my-role/role-id
vault write -f -field=secret_id auth/approle/role/my-role/secret-id

Login from SSH client machine

export VAULT_ADDR='http://host:8200'
export ROLE_ID=<role_id>
export SECRET_ID=<secret_id>
vault write auth/approle/login role_id="$ROLE_ID" secret_id="$SECRET_ID"

Result contains token

Key                     Value
---                     -----
token                   s.Sh9h1wZ9ycATeSaASoOQvovr
...

Check the token info

vault token lookup | grep policies
policies            [default my-policy]

Test Policy in SSH client machine

  • Using token in SSH client
$ export VAULT_TOKEN="<policy_token>"
$ vault write -field=signed_key ssh-client-signer/sign/my-role public_key=@$HOME/.ssh/id_rsa.pub > signed-cert.pub
  • SSH into target system using signed public key and private key in SSH client
ssh -i signed-cert.pub -i ~/.ssh/id_rsa ubuntu@host

Consideration

Disable password authentication

If tested finished, password authentication can be set to no:

PasswordAuthentication no

Time valid for SECRET_ID

If the secret_id expires so fast, then needs to have master token to generate secret_id again, because the policy token is generated using role_id and secret_id pair.

If don't use approle, then all SSH clients can use policy token. Still thinking the advantage of using role_id and secret_id.

Use admin token instead

The root token should not be used as recommended. Will try admin token later.

References

Policies