Accessing HashiCorp Vault Secrets from Tanzu Application Platform Workloads
⚠️ This article was automatically translated by OpenAI (gpt-4o). It may be edited eventually, but please be aware that it may contain incorrect information at this time.
In Tanzu Application Platform, it is common to configure sensitive information for Workloads by setting K8s Secrets via Service Binding. However, if you do not want to place sensitive information in K8s Secrets, you might want to use HashiCorp Vault.
To access Vault from K8s, you can use:
- Vault Agent Injector
- Container Storage Interface (CSI) Volume
- External Secrets Operator
However,
- It is difficult to configure CSI Volume for TAP Workloads without hacks.
- External Secrets syncs Vault Secrets to K8s Secrets, which does not meet the need to avoid placing sensitive information in K8s Secrets.
Therefore, we will use Vault Agent Injector.
External Secret is experimentally supported in Tanzu Application Platform 1.4.
https://docs.vmware.com/en/VMware-Tanzu-Application-Platform/1.4/tap/external-secrets-about-external-secrets-operator.html
This time, we will deploy a Workload on TAP on Kind, created in this article, and refer to the Secret set on Vault.
In TAP, a Multi Cluster configuration is common, and Vault is likely to run in a different location from the Run Cluster where the Workload resides. Therefore, we will access an existing Vault from the Workload. We referred to the following tutorial.
https://developer.hashicorp.com/vault/tutorials/kubernetes/kubernetes-external-vault
Table of Contents
Starting Vault
Start Vault outside the K8s cluster. This time, we will start it in dev mode.
vault server -dev -dev-root-token-id root -dev-listen-address 0.0.0.0:8200
Installing Vault Agent Injector
Install Vault Agent Injector on the cluster where TAP is installed (Run Cluster in a Multi Cluster configuration).
Use the Helm Chart.
helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update
Generate the manifest with helm template. Specify the Vault URL as seen from the K8s cluster in injector.externalVaultAddr.
helm template vault hashicorp/vault -n vault --set "injector.externalVaultAddr=http://host.docker.internal:8200" > vault-agent-injector.yaml
kubectl create ns vault
kubectl apply -f vault-agent-injector.yaml -n vault
Check the Pod.
$ kubectl get pod -n vault
NAME READY STATUS RESTARTS AGE
vault-agent-injector-547d8fc8db-kxtd7 1/1 Running 0 14s
Enabling Kubernetes Auth Method
Enable the Kubernetes Auth Method.
vault auth enable kubernetes
Create a token for the Service Account.
kubectl apply -n vault -f -<<EOF
apiVersion: v1
kind: Secret
metadata:
name: vault-token
namespace: vault
annotations:
kubernetes.io/service-account.name: vault
type: kubernetes.io/service-account-token
EOF
Set the K8s information to access Vault.
TOKEN_REVIEW_JWT=$(kubectl get secret -n vault vault-token -otemplate='{{index .data "token" | base64decode}}')
KUBE_CA_CERT=$(kubectl get secret -n vault vault-token -otemplate='{{index .data "ca.crt"| base64decode}}')
KUBE_HOST=$(kubectl config view --raw --minify --flatten --output='jsonpath={.clusters[].cluster.server}')
vault write auth/kubernetes/config \
token_reviewer_jwt="$TOKEN_REVIEW_JWT" \
kubernetes_host="$KUBE_HOST" \
kubernetes_ca_cert="$KUBE_CA_CERT" \
issuer="https://kubernetes.default.svc.cluster.local"
Registering Secrets for the Application
We will use https://github.com/making/vehicle-api for the application. This app is a simple Spring Boot application that accesses PostgreSQL and is completely independent of Vault.
We will use the free plan of ElephantSQL for the PostgreSQL to be accessed.
Create an instance on ElephantSQL and register the instance information as a Secret for vehicle-api.
vault kv put secret/vehicle-api/config \
host='floppy.db.elephantsql.com' \
username='ixyepwbw' \
password='QIDpi7cBKioNGyp7JeN8ZMTL-rIWL_9B' \
database='ixyepwbw'
Check the registered information.
$ vault read secret/data/vehicle-api/config
Key Value
--- -----
data map[database:ixyepwbw host:floppy.db.elephantsql.com password:QIDpi7cBKioNGyp7JeN8ZMTL-rIWL_9B username:ixyepwbw]
metadata map[created_time:2023-01-11T08:46:51.471224Z custom_metadata:<nil> deletion_time: destroyed:false version:1]
Create a read-only Policy for this Secret.
vault policy write vehicle-api - <<EOF
path "secret/data/vehicle-api/config" {
capabilities = ["read"]
}
EOF
Deploying the Workload (Adding application.properties)
Create a Role for the default Service Account in the demo namespace to deploy the Workload and attach the previously created Policy.
vault write auth/kubernetes/role/vehicle-api \
bound_service_account_names=default \
bound_service_account_namespaces=demo \
policies=vehicle-api \
ttl=24h
Set annotations for the Workload to access this Secret.
Write the contents of the Secret to /vault/secrets/application.properties and set the environment variable SPRING_CONFIG_IMPORT to read this file from the app.
cat <<EOF > workload.yaml
apiVersion: carto.run/v1alpha1
kind: Workload
metadata:
labels:
app.kubernetes.io/part-of: vehicle-api
apps.tanzu.vmware.com/workload-type: web
name: vehicle-api
spec:
params:
- name: annotations
value:
autoscaling.knative.dev/minScale: '1'
vault.hashicorp.com/agent-inject: 'true'
vault.hashicorp.com/role: vehicle-api
vault.hashicorp.com/agent-inject-secret-application.properties: secret/data/vehicle-api/config
vault.hashicorp.com/agent-inject-template-application.properties: |
{{- with secret "secret/data/vehicle-api/config" -}}
spring.datasource.url=jdbc:postgresql://{{ .Data.data.host }}/{{ .Data.data.database }}
spring.datasource.username={{ .Data.data.username }}
spring.datasource.password={{ .Data.data.password }}
{{- end -}}
env:
- name: SPRING_CONFIG_IMPORT
value: /vault/secrets/application.properties
build:
env:
- name: BP_JVM_VERSION
value: "17"
source:
git:
url: https://github.com/making/vehicle-api
ref:
branch: main
EOF
tanzu apps workload apply -f workload.yaml -n demo
# or kubectl apply -f workload.yaml -n demo
vault.hashicorp.com/agent-inject-template-...could not be set with thetanzuCLI--annotation.
If creating the Workload with CLI only, the following parameter specification is required.tanzu apps workload apply vehicle-api2 \ -n demo\ --git-repo https://github.com/making/vehicle-api \ --git-branch main \ --type web \ --app vehicle-api \ --env SPRING_CONFIG_IMPORT=/vault/secrets/application.properties \ --build-env BP_JVM_VERSION=17 \ --param-yaml annotation='{"autoscaling.knative.dev/minScale":"1","vault.hashicorp.com/agent-inject":"true","vault.hashicorp.com/agent-inject-secret-application.properties":"secret/data/vehicle-api/config","vault.hashicorp.com/agent-inject-template-application.properties":"{{- with secret \"secret/data/vehicle-api/config\" -}}\nspring.datasource.url=jdbc:postgresql://{{ .Data.data.host }}/{{ .Data.data.database }}\nspring.datasource.username={{ .Data.data.username }}\nspring.datasource.password={{ .Data.data.password }}\n{{- end -}}","vault.hashicorp.com/role":"vehicle-api"}'
After the app is deployed, check the Workload. The Pod will include a vault agent sidecar, so the number of containers will be 3/3.
$ tanzu apps workload get -n demo vehicle-api
📡 Overview
name: vehicle-api
type: web
💾 Source
type: git
url: https://github.com/making/vehicle-api
branch: main
📦 Supply Chain
name: source-to-url
RESOURCE READY HEALTHY TIME OUTPUT
source-provider True True 134m GitRepository/vehicle-api
image-provider True True 131m Image/vehicle-api
config-provider True True 131m PodIntent/vehicle-api
app-config True True 131m ConfigMap/vehicle-api
service-bindings True True 131m ConfigMap/vehicle-api-with-claims
api-descriptors True True 131m ConfigMap/vehicle-api-with-api-descriptors
config-writer True True 131m Runnable/vehicle-api-config-writer
🚚 Delivery
name: delivery-basic
RESOURCE READY HEALTHY TIME OUTPUT
source-provider True True 130m ImageRepository/vehicle-api-delivery
deployer True True 130m App/vehicle-api
💬 Messages
No messages found.
🛶 Pods
NAME READY STATUS RESTARTS AGE
vehicle-api-00001-deployment-7b6d9f7f49-rl25z 3/3 Running 0 30m
vehicle-api-build-1-build-pod 0/1 Completed 0 34m
vehicle-api-config-writer-trr8r-pod 0/1 Completed 0 31m
🚢 Knative Services
NAME READY URL
vehicle-api Ready https://vehicle-api-demo.127-0-0-1.sslip.io
To see logs: "tanzu apps workload tail vehicle-api --namespace demo"
Access the app and confirm that data can be retrieved from the database.
$ curl -sk https://vehicle-api-demo.127-0-0-1.sslip.io/vehicles
[{"id":1,"name":"Avalon"},{"id":2,"name":"Corolla"},{"id":3,"name":"Crown"},{"id":4,"name":"Levin"},{"id":5,"name":"Yaris"},{"id":6,"name":"Vios"},{"id":7,"name":"Glanza"},{"id":8,"name":"Aygo"}]
Confirm that the Secret obtained from Vault is written to the file.
POD_NAME=$(kubectl get pod -n demo -l serving.knative.dev/service=vehicle-api -ojsonpath='{.items[0].metadata.name}')
$ kubectl exec -it ${POD_NAME} -n demo -c workload -- ls -la /vault/secrets/
total 8
drwxrwxrwt 2 root root 60 Jan 11 09:14 .
drwxr-xr-x 3 root root 4096 Jan 11 09:14 ..
-rw-r--r-- 1 _apt cnb 170 Jan 11 09:14 application.properties
$ kubectl exec -it ${POD_NAME} -n demo -c workload -- cat /vault/secrets/application.properties
spring.datasource.url=jdbc:postgresql://floppy.db.elephantsql.com/ixyepwbw
spring.datasource.username=ixyepwbw
spring.datasource.password=QIDpi7cBKioNGyp7JeN8ZMTL-rIWL_9
Access the Actuator env endpoint and confirm that the settings are actually read from this file.
kubectl port-forward ${POD_NAME} -n demo 8081:8081
$ curl -s localhost:8081/actuator/env | jq .
{
"activeProfiles": [],
"propertySources": [
{
"name": "server.ports",
"properties": {
...
}
},
{
"name": "servletContextInitParams",
"properties": {}
},
{
"name": "systemProperties",
"properties": {
...
}
},
{
"name": "systemEnvironment",
"properties": {
...
}
},
{
"name": "Config resource 'file [/vault/secrets/application.properties]' via location '/vault/secrets/application.properties'",
"properties": {
"spring.datasource.url": {
"value": "jdbc:postgresql://floppy.db.elephantsql.com/ixyepwbw",
"origin": "URL [file:/vault/secrets/application.properties] - 1:23"
},
"spring.datasource.username": {
"value": "ixyepwbw",
"origin": "URL [file:/vault/secrets/application.properties] - 2:28"
},
"spring.datasource.password": {
"value": "******",
"origin": "URL [file:/vault/secrets/application.properties] - 3:28"
}
}
},
{
"name": "Config resource 'class path resource [application.properties]' via location 'optional:classpath:/'",
"properties": {
...
}
}
]
}
We successfully accessed Vault from TAP. It was confirmed that sensitive information could be retrieved from TAP without using K8s Secrets.