Install Tanzu Application Platform with GitOps by kapp-controller

️⚠️ The content of this article is not supported by VMware. Any issues arising from the content of this article are your responsibility and please do not contact VMware Support.

Install TAP with GitOps. GitOps allows you to install TAP itself declaratively, manage configuration files, and prevent cluster operation errors. There are several ways to achieve GitOps in Kubernetes, here we use kapp-controller. kapp-controller is a prerequisite for installing TAP, so you don't need to install any additional software for GitOps.

I talked abount kapp-controller at Cloud Native Operator Days Tokyo 2022, so please refer to the slides (in Japanese) or video (in Japanese) for details.

This time it is like the GitOps version of installing TAP manually in this article.

table of contents

Create a kind cluster
Install kapp-controller
Create a Service Account
Create directories for GitOps
Define App CRs
Push manifests to a git repository
Creatr a Secret for credentials
Install TAP
Uninstall TAP

Create a kind cluster

Use kind as the environment to install TAP .

Allocate at least 4 CPUs and 4 GB memory to Docker.

The following version is used. K8s 1.24 node didn't start on the old kind cluster.

$ kind version
kind v0.17.0 go1.19.2 darwin/amd64

Use k8s 1.24.

cat <<EOF > kind-expose-port.yaml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
 - role: control-plane
   extraPortMappings:
   - containerPort: 31443 # expose port 31443 of the node to port 80 on the host for use later by Contour ingress (envoy)
     hostPort: 443
   - containerPort: 31080 # expose port 31080 of the node to port 80 on the host for use later by Contour ingress (envoy)
     hostPort: 80
EOF

kind create cluster --config kind-expose-port.yaml --image kindest/node:v1.24.7

Install kapp-controller

The TAP document installs kapp-controller with Cluster Essentials for VMware Tanzu, but here we use the manifest on Github to install kapp-controller with the minimum number of steps.

kubectl apply -f https://github.com/vmware-tanzu/carvel-kapp-controller/releases/download/v0.44.1/release.yml

Create a Service Account

Create a Service Account for use by the kapp controller and set the ClusterRoleBinding. Here we bind the cluster-admin, but you can also use a restricted ClusterRole.

NAMESPACE=kapp
kubectl create ns ${NAMESPACE}
kubectl create -n ${NAMESPACE} sa kapp
kubectl create clusterrolebinding kapp-cluster-admin-${NAMESPACE} --clusterrole cluster-admin --serviceaccount=${NAMESPACE}:kapp

Create directories for GitOps

kapp directory contains the kapp-controller's App CR definition, and config directory contains the k8s resource definition to be deployed. The files are placed per cluster that creates it. This time, we will create kind-iterate directory to manage the iterator cluster deployed on kind. These will be pushed to a git repo later.

mkdir -p tap-install-gitops/kind-iterate/kapp
mkdir -p tap-install-gitops/kind-iterate/config

The file hierarchy will look like bellow:

$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    `-- kapp

Manifests will be stored in https://github.com/making/tap-install-gitops .

Define App CRs

Define the kapp-controller's App CR and the resources deployed from the App CR.

App CR for secretgen-controller installation

First , define the installation of secretgen-controller , which is another prerequisite for TAP, in App CR. Here we will deploy the secretgen-controller from the manifest on Github in stead of Cluster Essentials.

cat <<EOF > tap-install-gitops/kind-iterate/kapp/secretgen-controller.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: secretgen-controller
  namespace: kapp
  annotations:
    kapp.k14s.io/change-group: "{name}"
spec:
  serviceAccountName: kapp
  fetch:
  - http:
      url: https://github.com/vmware-tanzu/carvel-secretgen-controller/releases/download/v0.13.0/release.yml
  syncPeriod: 6h
  template:
  - ytt: { }
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=true
      - --diff-mask=true
EOF

The file hierarchy will look like bellow:

$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    `-- kapp
        `-- secretgen-controller.yaml

App CR for TAP's PackageRepository

Define PackageRepository and Namespace for TAP and the Secret required to use it in App CRs.

mkdir -p tap-install-gitops/kind-iterate/config/tap-repository

Namespace definition

cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: tap-install
EOF

Definition of Secrets. Registry information is externalized and passed from the outside later.

cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/secret.yaml
#@ load("@ytt:data", "data")
apiVersion: v1
kind: Secret
metadata:
  name: tap-registry
  namespace: tap-install
type: kubernetes.io/dockerconfigjson
stringData:
  #@yaml/text-templated-strings
  .dockerconfigjson: |-
    {
      "auths": {
        "(@= data.values.tap_registry.server @)": {
          "username": "(@= data.values.tap_registry.username @)",
          "password": "(@= data.values.tap_registry.password @)"
        }
      }
    }
EOF

Definition of SecretExport to allow the above Secret to be exported to other Namespaces

cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/secret-export.yaml
apiVersion: secretgen.carvel.dev/v1alpha1
kind: SecretExport
metadata:
  name: tap-registry
  namespace: tap-install
spec:
  toNamespaces:
  - '*'
EOF

PackageRepository definition (for TAP 1.3.3)

cat <<EOF > tap-install-gitops/kind-iterate/config/tap-repository/pkgr.yaml
#@ load("@ytt:data", "data")
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
  name: tanzu-tap-repository-1.3.3
  namespace: tap-install
spec:
  fetch:
    imgpkgBundle:
      image: #@ "{}/tanzu-application-platform/tap-packages:1.3.3".format(data.values.tap_registry.server)
EOF

Definition of App CR for managing resources prepared so far with GitOps. Set it to be installed after secretgen-controller. It will be reconciled every 6 hours.

cat <<EOF > tap-install-gitops/kind-iterate/kapp/tap-repository.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: tap-repository
  namespace: kapp
  annotations:
    kapp.k14s.io/change-group: "{name}"
    kapp.k14s.io/change-rule.create-order.0: "upsert after upserting secretgen-controller"
    kapp.k14s.io/change-rule.delete-order.0: "delete before deleting secretgen-controller"
spec:
  syncPeriod: 6h
  serviceAccountName: kapp
  fetch:
  - git:
      url: https://github.com/making/tap-install-gitops.git
      ref: origin/main
      subPath: kind-iterate/config
  template:
  - ytt:
      paths:
      - tap-repository
      valuesFrom:
      - secretRef:
          name: tap-install-gitops
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=false
      - --diff-mask=true
EOF

The file hierarchy will look like bellow:

$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    |   `-- tap-repository
    |       |-- namespace.yaml
    |       |-- pkgr.yaml
    |       |-- secret-export.yaml
    |       `-- secret.yaml
    `-- kapp
        |-- secretgen-controller.yaml
        `-- tap-repository.yaml

App CR for TAP's PackageInstall

Define TAP's PackageInstall and Secrets that store tap-values.yaml and overlays in App CRs.

Create a CA certificate to issue self-signed TLS certificates.

mkdir -p certs
rm -f certs/*
docker run --rm -v ${PWD}/certs:/certs hitch openssl req -new -nodes -out /certs/ca.csr -keyout /certs/ca.key -subj "/CN=default-ca/O=TAP/C=JP"
chmod og-rwx ca.key
docker run --rm -v ${PWD}/certs:/certs hitch openssl x509 -req -in /certs/ca.csr -days 3650 -extfile /etc/ssl/openssl.cnf -extensions v3_ca -signkey /certs/ca.key -out /certs/ca.crt

mkdir -p tap-install-gitops/kind-iterate/config/tap/overlays

Here the base domain name is 127-0-0-1.sslip.io for Kind. This domain and subdomains are all resolved to 127.0.0.1.

export BASE_DOMAIN=127-0-0-1.sslip.io

First, define the overlay secret used for TAP installation.

Overlay for issuing the default TLS certificate

cat <<EOF > tap-install-gitops/kind-iterate/config/tap/overlays/contour-default-tls.yaml
#@ load("@ytt:data", "data")
apiVersion: v1
kind: Secret
metadata:
  name: contour-default-tls
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-overlays"
type: Opaque
stringData:
  #@yaml/text-templated-strings
  contour-default-tls.yaml: |
    #@ load("@ytt:data", "data")
    #@ load("@ytt:overlay", "overlay")
    #@ namespace = data.values.namespace
    ---
    apiVersion: v1
    kind: Secret
    metadata:
      name: default-ca
      namespace: #@ namespace
    type: kubernetes.io/tls
    stringData:
      tls.crt: "(@= data.values.default_ca.crt.replace('\n', '\\n') @)"
      tls.key: "(@= data.values.default_ca.key.replace('\n', '\\n') @)"
    ---
    apiVersion: cert-manager.io/v1
    kind: Issuer
    metadata:
      name: default-ca-issuer
      namespace: #@ namespace
    spec:
      ca:
        secretName: default-ca
    ---
    apiVersion: cert-manager.io/v1
    kind: Certificate
    metadata:
      name: tap-default-tls
      namespace: #@ namespace
    spec:
      dnsNames:
      - #@ "*.${BASE_DOMAIN}"
      issuerRef:
        kind: Issuer
        name: default-ca-issuer
      secretName: tap-default-tls
    ---
    apiVersion: projectcontour.io/v1
    kind: TLSCertificateDelegation
    metadata:
      name: contour-delegation
      namespace: #@ namespace
    spec:
      delegations:
      - secretName: tap-default-tls
        targetNamespaces:
        - "*"
EOF

Overlay for default HTTPS in Knative

cat <<EOF > tap-install-gitops/kind-iterate/config/tap/overlays/cnrs-https.yaml
apiVersion: v1
kind: Secret
metadata:
  name: cnrs-https
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-overlays"
type: Opaque
stringData:
  cnrs-https.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"metadata":{"name":"config-network"}, "kind": "ConfigMap"})
    ---
    data:
      #@overlay/match missing_ok=True
      default-external-scheme: https
      #@overlay/match missing_ok=True
      http-protocol: redirected      
EOF

Overlay for deleting Telemetry Pods (:P)

cat <<EOF > tap-install-gitops/kind-iterate/config/tap/overlays/tap-telemetry-remove.yaml
apiVersion: v1
kind: Secret
metadata:
  name: tap-telemetry-remove
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-overlays"
type: Opaque
stringData:
  tap-telemetry-remove.yaml: |
    #@ load("@ytt:overlay", "overlay")
    #@overlay/match by=overlay.subset({"metadata":{"namespace":"tap-telemetry"}}), expects="1+"
    #@overlay/remove
    ---     
EOF

Then, define the resources that are equivalent to tanzu package install tap -n tap-install -f tap-values.yaml.

Please refer to this article for the difference between installing a Package with tanzu CLI and installing a Package by creating a PackageInstall.

Secret definition to store tap-values.yaml

cat <<EOF > tap-install-gitops/kind-iterate/config/tap/values.yaml
#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")

#@ def tap_values():
shared:
  ingress_domain: ${BASE_DOMAIN}
  image_registry:
    project_path: #@ data.values.image_registry.project_path
    username: #@ data.values.image_registry.username
    password: #@ data.values.image_registry.password
  ca_cert_data: #@ data.values.default_ca.crt

ceip_policy_disclosed: true
profile: iterate

supply_chain: basic

contour:
  contour:
    replicas: 1
  envoy:
    service:
      type: NodePort
      nodePorts:
        http: 31080
        https: 31443
    hostPorts:
      enable: true

cnrs:
  domain_template: "{{.Name}}-{{.Namespace}}.{{.Domain}}"
  default_tls_secret: tanzu-system-ingress/tap-default-tls
  provider: local

package_overlays:
- name: contour
  secrets:
  - name: contour-default-tls
- name: cnrs
  secrets:
  - name: cnrs-https
- name: tap-telemetry
  secrets:
  - name: tap-telemetry-remove

excluded_packages:
- policy.apps.tanzu.vmware.com
- image-policy-webhook.signing.apps.tanzu.vmware.com
- eventing.tanzu.vmware.com
- sso.apps.tanzu.vmware.com
#@ end

apiVersion: v1
kind: Secret
metadata:
  name: tap-tap-install-values
  namespace: tap-install
  annotations:
    kapp.k14s.io/change-group: "tap-install-values"
type: Opaque
stringData:
  tap-values.yaml: #@ yaml.encode(tap_values())
EOF

Defining RBAC for installing packages

cat <<EOF > tap-install-gitops/kind-iterate/config/tap/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: tap-tap-install
    kapp.k14s.io/change-group: "tap-rbac"
  name: tap-tap-install-cluster-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: tap-tap-install
    kapp.k14s.io/change-group: "tap-rbac"
  name: tap-tap-install-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tap-tap-install-cluster-role
subjects:
- kind: ServiceAccount
  name: tap-tap-install-sa
  namespace: tap-install
---
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    tkg.tanzu.vmware.com/tanzu-package: tap-tap-install
    kapp.k14s.io/change-group: "tap-rbac"
  name: tap-tap-install-sa
  namespace: tap-install 
EOF

Definition of PackageInstall

cat <<EOF > tap-install-gitops/kind-iterate/config/tap/pkgi.yaml
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: tap
  namespace: tap-install
  annotations:
    tkg.tanzu.vmware.com/tanzu-package-ClusterRole: tap-tap-install-cluster-role
    tkg.tanzu.vmware.com/tanzu-package-ClusterRoleBinding: tap-tap-install-cluster-rolebinding
    tkg.tanzu.vmware.com/tanzu-package-Secret: tap-tap-install-values
    tkg.tanzu.vmware.com/tanzu-package-ServiceAccount: tap-tap-install-sa
    kapp.k14s.io/change-group: "{name}"
    kapp.k14s.io/change-rule.create-order.1: "upsert after upserting tap-rbac"
    kapp.k14s.io/change-rule.delete-order.1: "delete before deleting tap-rbac"
    kapp.k14s.io/change-rule.create-order.2: "upsert after upserting tap-values"
    kapp.k14s.io/change-rule.delete-order.2: "delete before deleting tap-values"
    kapp.k14s.io/change-rule.create-order.3: "upsert after upserting tap-overlays"
    kapp.k14s.io/change-rule.delete-order.3: "delete before deleting tap-overlays"
spec:
  syncPeriod: 3h
  serviceAccountName: tap-tap-install-sa
  packageRef:
    refName: tap.tanzu.vmware.com
    versionSelection:
      constraints: 1.3.3
      prereleases: { }
  values:
  - secretRef:
      name: tap-tap-install-values 
EOF

Definition of App CR for managing resources prepared so far with GitOps. Set it to be installed after PackageRepository. It will be reconciled every hour.

cat <<EOF > tap-install-gitops/kind-iterate/kapp/tap.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: tap
  namespace: kapp
  annotations:
    kapp.k14s.io/change-group: "{name}"
    kapp.k14s.io/change-rule.create-order.0: "upsert after upserting tap-repository"
    kapp.k14s.io/change-rule.delete-order.0: "delete before deleting tap-repository"
spec:
  syncPeriod: 1h
  serviceAccountName: kapp
  fetch:
  - git:
      url: https://github.com/making/tap-install-gitops.git
      ref: origin/main
      subPath: kind-iterate/config
  template:
  - ytt:
      paths:
      - tap
      valuesFrom:
      - secretRef:
          name: tap-install-gitops
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=false
      - --diff-mask=true
EOF

The file hierarchy will look like bellow:

$ tree tap-install-gitops 
tap-install-gitops
`-- kind-iterate
    |-- config
    |   |-- tap
    |   |   |-- overlays
    |   |   |   |-- cnrs-https.yaml
    |   |   |   |-- contour-default-tls.yaml
    |   |   |   `-- tap-telemetry-remove.yaml
    |   |   |-- pkgi.yaml
    |   |   |-- rbac.yaml
    |   |   `-- values.yaml
    |   `-- tap-repository
    |       |-- namespace.yaml
    |       |-- pkgr.yaml
    |       |-- secret-export.yaml
    |       `-- secret.yaml
    `-- kapp
        |-- secretgen-controller.yaml
        |-- tap-repository.yaml
        `-- tap.yaml

A parent App CR that collectively manages a group of App CRs

App CR definition for managing the App CR group defined so far with GitOps. It will be reconciled every 10 minutes.

☝️ It follows app of apps pattern

cat <<EOF > tap-install-gitops/kind-iterate/apps.yaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  name: apps
  namespace: kapp
spec:
  serviceAccountName: kapp
  fetch:
  - git:
      url: https://github.com/making/tap-install-gitops.git
      ref: origin/main
      subPath: kind-iterate/kapp
  syncPeriod: 10m
  template:
  - ytt:
      paths:
      - '.'
  deploy:
  - kapp:
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=true
      - --diff-mask=false
EOF

The file hierarchy will look like bellow:

$ tree tap-install-gitops                              
tap-install-gitops
`-- kind-iterate
    |-- apps.yaml
    |-- config
    |   |-- tap
    |   |   |-- overlays
    |   |   |   |-- cnrs-https.yaml
    |   |   |   |-- contour-default-tls.yaml
    |   |   |   `-- tap-telemetry-remove.yaml
    |   |   |-- pkgi.yaml
    |   |   |-- rbac.yaml
    |   |   `-- values.yaml
    |   `-- tap-repository
    |       |-- namespace.yaml
    |       |-- pkgr.yaml
    |       |-- secret-export.yaml
    |       `-- secret.yaml
    `-- kapp
        |-- secretgen-controller.yaml
        |-- tap-repository.yaml
        `-- tap.yaml

Push manifests to a git repository

Push the created manifest to https://github.com/making/tap-install-gitops .

GITHUB_USERNAME=...

cd tap-install-gitops 
git init
git add -A
git commit -m "first commit"
git branch -M main
git remote add origin git@github.com:${GITHUB_USERNAME}/tap-install-gitops.git
git push -u origin main
cd ..

Creatr a Secret for credentials

Git-managed manifests shoule not include credentials, but reference externalized parameters. Externalized parameters are set to be referenced from Secret. This Secret is outside Git management and is created as follows:

TANZUNET_USERNAME=...
TANZUNET_PASSWORD=...
GITHUB_USERNAME=...
GITHUB_API_TOKEN=...

kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: tap-install-gitops
  namespace: kapp
type: Opaque
stringData:
  credentials.yaml: |
    tap_registry:
      server: registry.tanzu.vmware.com
      username: ${TANZUNET_USERNAME}
      password: ${TANZUNET_PASSWORD}
    image_registry:
      project_path: ghcr.io/${GITHUB_USERNAME}
      username: ${GITHUB_USERNAME}
      password: ${GITHUB_API_TOKEN}
    default_ca:
      crt: |
$(cat certs/ca.crt | sed 's/^/        /g')
      key: |
$(cat certs/ca.key | sed 's/^/        /g')
EOF

You can also include "encrypted" credentials in the git repo instead of using Secret out of the git repository. kapp-controller supports sops. See https://carvel.dev/kapp-controller/docs/v0.43.2/sops/ for how to do this.

This blog is managed using GitOps using sops.
The file structure is slightly different from this article, but https://github.com/categolj/k8s-manifests will be helpful

Install TAP

Now that we have a Git repository and externalized parameters, it's time to install TAP with App CR. Just create the parent App CR and everything will be installed.

kubectl apply -f tap-install-gitops/kind-iterate/apps.yaml

Check the progress until it completes with the following command:

while [ "$(kubectl -n kapp get app apps -o=jsonpath='{.status.friendlyDescription}')" != "Reconcile succeeded" ];do
  date
  kubectl get app -A
  echo "---------------------------------------------------------------------"
  sleep 10
done
echo "✅ Install succeeded"

After a successful install, each app will reconcile at syncPeriod intervals. If you can't wait for the syncPeriod, use the bellow command to trigger the reconciliation forcely:

kctrl app kick -n kapp -a <App name>

You can download the kctrl CLI from the kapp-controller Releases page.

Uninstall TAP

Simply deleting the parent App CR will uninstall everything.

kubectl delete -f tap-install-gitops/kind-iterate/apps.yaml

I introduced how to GitOps TAP with kapp-controller.

Using this method, to install TAP, you just need to

Install kapp-controller
Create Service Account
Register Secret of credentials (or Register GPG Private Key when using sops)
Create an apps App CR

No operation by tanzu CLI is required.

Software other than TAP (Prometheus, etc.) can be installed at the same time, making it easier to automate environment construction. It will also be useful if you want to repeatedly build and destroy the TAP testing environment.

IK.AM