Dynamic Secrets: HashiCorp Vault, PostgreSQL and Python

It is standard security practice to isolate secrets from code, and developers should not concern themselves with the origin of these secrets. This is where HashiCorp Vault comes in to centralize those secrets. However, if one or multiple of these secrets become compromised, you’ll need to revoke them and generate new ones to mitigate risks with minimal impact on the systems utilizing them.

In this article, we’ll cover how you can achieve that using HashiCorp Vault to secure a PostgreSQL database with the help of Dynamic Secrets. We will end by demonstrating how we can use these secrets with Python.

What’s Dynamic Secrets?

From HashiCorp Vault website:

A dynamic secret is generated on demand and is unique to a client, instead of a static secret, which is defined ahead of time and shared. Vault associates each dynamic secret with a lease and automatically destroys the credentials when the lease expires.

Why Dynamic Secrets?

Here are some leaks that led us to consider using Dynamic Secrets instead of Static Secrets.

• Applications frequently log configurations, leaving them in log files or centralized logging systems like Elasticsearch, where unauthorized individuals can access your credentials.
• Often, secrets are captured in exception tracebacks while attempting to access the database, for instance. These crash reports are sent to external monitoring systems, or they may be leaked via debugging endpoints and diagnostic pages after encountering an error.
• It’s common within organizations for multiple applications to share the same credentials to access other systems, such as a database. This means that rotating those passwords will impact all applications using those credentials. Now, imagine one of these applications getting compromised. It will require you to rotate these credentials across all your applications.

All these challenges drive us to use a secret that we can generate and revoke on demand with less impact.

PostgreSQL Database

In this section, we assume that you already have a PostgreSQL database server running. Alternatively, if you’re a Kubernetes and Helm user, you can set up a PostgreSQL database server by executing the commands below.

helm install postgres oci://registry-1.docker.io/bitnamicharts/postgresql
export POSTGRES_PASSWORD=$(kubectl get secret --namespace default postgres-postgresql -o jsonpath="{.data.postgres-password}" | base64 -d)

The PostgreSQL server that we are using throughout this tutorial is accessed using the configurations below.

Please note that these configurations are temporary and local, and you cannot use them in your environment.

• User: postgres
• Port: 5432
• Password: FAKE_PWD
• Host: postgres-postgresql

Create a Database

• Connect to the database

psql --host localhost -U postgres -d postgres -p 5432

• Create a database named dbtest

CREATE DATABASE dbtest;

Above, we establish a connection to the database using the psql tool (alternatively, you can use any other PostgreSQL client tool). Subsequently, we execute the create database command to set up our test database.

HashiCorp Vault

Generating secrets dynamically for a PostgreSQL database involves two steps: configuring the plugin and creating a role.

Set up Vault Server (Optional)

Below, we are going to set up a Vault Server using Kubernetes. The required tools to follow the same procedure are Helm and kubectl. However, you may consider using other alternatives like Docker or local installation.

The code below sets up the Vault Server in development mode, which means that you should not use it in production. The default root token to log in to the server is root.

helm repo add hashicorp https://helm.releases.hashicorp.com
helm search repo hashicorp/vault
helm search repo hashicorp/vault --versions
helm install vault hashicorp/vault --set "server.dev.enabled=true"

Forward the port:

kubectl port-forward svc/vault 8200:8200

Configure the plugin

First, you need to log in to Vault. If you have not already enabled the database Secrets engine, you need to do so by running the commands below.

Note that by default, the secrets engine will be enabled at the name of the engine. To enable the secrets engine at a different path, use the -path argument. In this tutorial, we’ll keep the default path.

vault login
vault secrets enable database

The success message confirms the enabling of the database Secrets engine.

Now, with the database Secrets engine enabled, we need to enable the postgresql-database-plugin. This plugin allows the dynamic generation of database credentials based on configured roles for the PostgreSQL database.

vault write database/config/my-postgresql-database \
    plugin_name="postgresql-database-plugin" \
    allowed_roles="test-role" \
    connection_url="postgresql://{{username}}:{{password}}@my-postgresql-hl:5432/dbtest" \
    username="postgres" \
    password="FAKE_PWD" \
    password_authentication="scram-sha-256"

Create a Role

Here, we create a role called test-role and then map it to an SQL statement that will create a new database user with select permission granted for all tables inside the public schema.

 vault write database/roles/test-role \
    db_name="my-postgresql-database" \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
        GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1h" \
    max_ttl="24h" 

To confirm that the plugin was correctly configured and the role was mapped, run the command below:

vault read database/creds/test-role

The result should output a username and password.

Python Script

In this section, we’ll demonstrate how you can use the hvac and requests libraries to generate and retrieve the username and password of the database created.

Install required dependencies

To access Vault you need to install the dependencies, hvac, and requests. It’s a best practice in Python to create a virtual environment to install dependencies. Below we create a virtual environment named venv and then we install dependencies using pip.

python -m venv venv
source venv/bin/activate
pip install hvac requests

Import Required Packages and Set Variables

Below, we import the required packages to access HashiCorp Vault. Then, we set up the address of the Vault server, a token to access it, and the path from which to generate secrets for the PostgreSQL database created above.

import hvac
import requests

VAULT_HOST = 'http://127.0.0.1:8200'
VAULT_TOKEN = 'root'
PATH = 'database/creds/my-role'

Read Dynamic Secrets using Requests

The code below demonstrates how to retrieve dynamic secrets for a PostgreSQL database using the requests library.

response = requests.get(
    f'{VAULT_HOST}/v1/{PATH}',
    params={'q': 'requests+language:python'},
    headers={'X-Vault-Token': VAULT_TOKEN},
)
if response:
    json_response = response.json()
    print(json_response)
    user = json_response['data']['username']
    password = json_response['data']['password']
    print(f"User: {user}, Password:  {password}")

Read Dynamic Secrets using Vault Client

Here, we’re retrieving the secrets using the Vault client. First, you need to create a client by providing the host URL and the token. Then, you read the secrets from the path provided above.

client = hvac.Client(
    url=VAULT_HOST,
    token=VAULT_TOKEN,
)
response = client.read(PATH)
if response and 'data' in response:
    secret_data = response['data']
    user = secret_data['username']
    password = secret_data['password']
    print(f"User: {user}, Password:  {password}")

Complete Source Code

Here is the complete source code for the methods discussed above, and you can also find it in my GitHub repository.

import hvac
import requests


VAULT_HOST = 'http://127.0.0.1:8200'
VAULT_TOKEN = 'root'
PATH = 'database/creds/test-role'


def read_dynamic_pwd_request(): 
    """Read dynamic database password using get request"""
    response = requests.get(
        f'{VAULT_HOST}/v1/{PATH}',
        params={'q': 'requests+language:python'},
        headers={'X-Vault-Token': VAULT_TOKEN},
    )
    if response:
        json_response = response.json()
        print(json_response)
        user = json_response['data']['username']
        password = json_response['data']['password']
        print(f"User: {user}, Password:  {password}")


def read_dynamic_pwd_with_hvac(): 
    """Read dynamic database password using vault python client"""
    client = hvac.Client(
        url=VAULT_HOST,
        token=VAULT_TOKEN,
    )
    response = client.read(PATH)
    if response and 'data' in response:
        secret_data = response['data']
        user = secret_data['username']
        password = secret_data['password']
        print(f"User: {user}, Password:  {password}")


if __name__ == '__main__':

    read_dynamic_pwd_request()
    read_dynamic_pwd_with_hvac()

Conclusion

To sum up, note that this was an introduction to Dynamic Secrets with HashiCorp Vault, a powerful tool that can help secure various types of secrets, certificates, and more (I’m not advertising; it’s just that I appreciate this technology :).

Thanks! I hope you found it helpful.