IK.AM

@making's tech note


Tanzu Kubernetes Grid on vSphereを1.3.1-patch1 -> 1.4.0にアップデートするメモ

🗃 {Dev/CaaS/Kubernetes/TKG/vSphere}
🏷 Cluster API 🏷 Kubernetes 🏷 TKG 🏷 Tanzu 🏷 vSphere 
🗓 Updated at 2021-09-14T01:05:20Z  🗓 Created at 2021-09-10T03:13:55Z {✒️️ Edit  ⏰ History  🗑 Delete}  🌎 Translation by Google

⚠️ 本記事の内容はVMwareによってサポートされていません。 記事の内容で生じた問題については自己責任で対応し、 VMwareサポート窓口には問い合わせないでください

"Tanzu Kubernetes Grid on vSphereを1.3.0 -> 1.3.1-patch1にアップデートするメモ"で作成した環境をアップデートします。

アップデートの手順はこちら

Release Noteはこちら

https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/rn/VMware-Tanzu-Kubernetes-Grid-14-Release-Notes.html

Table of contents

CLIのインストール

https://customerconnect.vmware.com/en/downloads/details?downloadGroup=TKG-140&productId=988&rPId=73652

から

  • VMware Tanzu CLI for Mac
tar xvf tanzu-cli-bundle-darwin-amd64.tar

install cli/core/v1.4.0/tanzu-core-darwin_amd64 /usr/local/bin/tanzu

次のバージョンで動作確認します。

$ tanzu version
version: v1.3.1
buildDate: 2021-05-07
sha: e5c37c4

初回コマンド実行時に、

!  Configuration is now stored in /Users/toshiaki/.config/tanzu. Legacy configuration directory /Users/toshiaki/.tanzu is deprecated and will be removed in a future release.
!  To complete migration, please remove legacy configuration directory /Users/toshiaki/.tanzu and adjust your script(s), if any, to point to the new location.

というようなログが出力されます。TKG 1.3.1の時に使用していた$HOME/.tanzu/tkg/providersはバックアップフォルダにrenameされ、 1.4.0用のファイルが$HOME/.config/tanzu/tkg/providersに用意されます。 この時、$HOME/.tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yamlのようなクラスタ構築時にカスタマイズしたytt overlayファイルは1.3.1に引き継がれないので、 1.3.1->1.4.0においても引き続き利用したいOverlayファイルは改めて$HOME/.config/tanzu/tkg/providersへ設定する必要があります。

cp -r $HOME/.config/tanzu/tkg/providers-20210910004336-y622uivb/infrastructure-vsphere/ytt/vsphere-overlay.yaml $HOME/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml
tanzu plugin clean
tanzu plugin install --local cli all

プラグイン一覧を確認します。

$ tanzu plugin list
  NAME                LATEST VERSION  DESCRIPTION                                                        REPOSITORY  VERSION  STATUS         
  alpha               v1.3.1          Alpha CLI commands                                                 core                 not installed  
  cluster             v1.3.1          Kubernetes cluster operations                                      core        v1.4.0   installed      
  kubernetes-release  v1.3.1          Kubernetes release operations                                      core        v1.4.0   installed      
  login               v1.3.1          Login to the platform                                              core        v1.4.0   installed      
  management-cluster  v1.3.1          Kubernetes management cluster operations                           core        v1.4.0   installed      
  package                             Tanzu package management                                                       v1.4.0   installed      
  pinniped-auth       v1.3.1          Pinniped authentication operations (usually not directly invoked)  core        v1.4.0   installed  

旧バージョンからアップデートした場合は、プラグインのLATEST VERSIONが1.3.1になるようです。 プラグインを配布するbucket名が変わったようです。

~/.config/tanzu/config.yamlgcpPluginRepositorycorebucketNametanzu-cli -> tanzu-cli-tkgに変更してください。

apiVersion: config.tanzu.vmware.com/v1alpha1
clientOptions:
  cli:
    repositories:
    - gcpPluginRepository:
        bucketName: tanzu-cli-tkg
        name: core

これで正しく表示されます。

$ tanzu plugin list
  NAME                LATEST VERSION  DESCRIPTION                                                        REPOSITORY  VERSION  STATUS         
  alpha               v1.4.0          Alpha CLI commands                                                 core                 not installed  
  cluster             v1.4.0          Kubernetes cluster operations                                      core        v1.4.0   installed      
  kubernetes-release  v1.4.0          Kubernetes release operations                                      core        v1.4.0   installed      
  login               v1.4.0          Login to the platform                                              core        v1.4.0   installed      
  management-cluster  v1.4.0          Kubernetes management cluster operations                           core        v1.4.0   installed      
  package             v1.4.0          Tanzu package management                                           core        v1.4.0   installed      
  pinniped-auth       v1.4.0          Pinniped authentication operations (usually not directly invoked)  core        v1.4.0   installed 

vCenterにOVAファイルをアップロード

https://my.vmware.com/en/web/vmware/downloads/details?downloadGroup=TKG-131&productId=988&rPId=65946

から

  • Photon v3 Kubernetes v1.21.2 OVA

をダウンロードし、~/Downloadsに保存します。

TKGをインストールする環境の情報を次の例のように設定してください。

cat <<EOF > tkg-env.sh
export GOVC_URL=vcsa.v.maki.lol
export GOVC_USERNAME=administrator@vsphere.local
export GOVC_PASSWORD=VMware1!
export GOVC_DATACENTER=Datacenter
export GOVC_NETWORK="VM Network"
export GOVC_DATASTORE=datastore1
export GOVC_RESOURCE_POOL=/Datacenter/host/Cluster/Resources/tkg
export GOVC_INSECURE=1
export TEMPLATE_FOLDER=/Datacenter/vm/tkg
EOF

次のコマンドでOVAファイルをvCenterにアップロードします。

source tkg-env.sh
govc import.ova -folder $TEMPLATE_FOLDER ~/Downloads/photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova
govc vm.markastemplate $TEMPLATE_FOLDER/photon-3-kube-v1.21.2

Management Clusterをアップデート

次のコマンドを実行し、アップデート対象のManagement Clusterにログインします。

tanzu login

アップデート前のクラスタ一覧を確認します。Kubernetesのバージョンはv1.20.4+vmware.1です。

$ tanzu cluster list --include-management-cluster
  NAME    NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES       PLAN  
  apple   default     running  1/1           2/2      v1.20.5+vmware.2  <none>      dev   
  grape   default     running  1/1           2/2      v1.20.4+vmware.1  <none>      prod  
  lemon   default     running  1/1           1/1      v1.20.5+vmware.2  <none>      dev   
  orange  default     running  1/1           2/2      v1.20.5+vmware.2  <none>      dev   
  carrot  tkg-system  running  1/1           1/1      v1.20.5+vmware.2  management  dev  

次のコマンドでManagement Clusterをアップデートします。

tanzu management-cluster upgrade -v9 -y

次のようなログが出力されます。

compatibility file (/Users/toshiaki/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /Users/toshiaki/.config/tanzu/tkg/bom already exists, skipping download
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading management cluster providers...
clusterctl upgrade apply options: {Kubeconfig:{Path:/Users/toshiaki/.kube-tkg/config Context:carrot-admin@carrot} ManagementGroup:capi-system/cluster-api Contract: CoreProvider:capi-system/cluster-api:v0.3.23 BootstrapProviders:[capi-kubeadm-bootstrap-system/kubeadm:v0.3.23] ControlPlaneProviders:[capi-kubeadm-control-plane-system/kubeadm:v0.3.23] InfrastructureProviders:[capv-system/vsphere:v0.7.10]}
Checking cert-manager version...
Deleting cert-manager Version="v0.16.1"
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Performing upgrade...
Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system"
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Waiting for provider infrastructure-vsphere
Waiting for provider bootstrap-kubeadm
Waiting for provider cluster-api
Waiting for provider control-plane-kubeadm
Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-kubeadm-bootstrap-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-kubeadm-bootstrap-controller-manager of type *v1.Deployment to be up and running
pods are not yet running for deployment 'capi-kubeadm-control-plane-controller-manager' in namespace 'capi-kubeadm-control-plane-system', retrying
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
Passed waiting on provider cluster-api after 341.338078ms
Passed waiting on provider bootstrap-kubeadm after 340.817975ms
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running
Passed waiting on provider control-plane-kubeadm after 5.455206185s
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running
Passed waiting on provider infrastructure-vsphere after 20.246601997s
Success waiting on all providers.
Management cluster providers upgraded successfully...
Upgrading management cluster kubernetes version...
Creating management cluster client...
Verifying kubernetes version...
Unable to detect current OS for the cluster. Using name:photon version:3 arch:amd64
Using OS options, name:photon version:3 arch:amd64
Retrieving configuration for upgrade cluster...
unable to find azure-image config in user configuration file, using it from BoM files
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Applying KubeadmControlPlane Patch: {
        "spec": {
          "version": "v1.21.2+vmware.1",
          "infrastructureTemplate": {
            "name": "carrot-control-plane-v1-21-2-vmware-1-dl61v",
            "namespace": "tkg-system"
          },
          "kubeadmConfigSpec": {
            "clusterConfiguration": {
              "imageRepository" : "projects.registry.vmware.com/tkg",
              "dns": {
                "imageRepository": "projects.registry.vmware.com/tkg",
                "imageTag": "v1.8.0_vmware.5"
              },
              "etcd": {
                "local": {
                  "imageRepository": "projects.registry.vmware.com/tkg",
                  "imageTag": "v3.4.13_vmware.15"
                }
              }
            }
          }
        }
      }
Applying patch to resource carrot-control-plane of type *v1alpha3.KubeadmControlPlane ...
patch cluster object with operation status: 
    {
        "metadata": {
            "annotations": {
                "TKGOperationInfo" : "{\"Operation\":\"Upgrade\",\"OperationStartTimestamp\":\"2021-09-09 16:10:45.498847 +0000 UTC\",\"OperationTimeout\":900}",
                "TKGOperationLastObservedTimestamp" : "2021-09-09 16:10:45.498847 +0000 UTC"
            }
        }
    }
Waiting for kubernetes version to be updated for control plane nodes
...
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Applying MachineDeployment Patch: {
            "spec": {
              "template": {
                "spec": {
                  "version": "v1.21.2+vmware.1",
                  "infrastructureRef": {
                    "name": "carrot-worker-v1-19-3-vmware-1-muxc1-v1-20-4-vmware-1-5mlj6-v1-20-5-vmware-2-4xqok-v1-21-2-vmware-1-fgltw",
                    "namespace": "tkg-system"
                  }
                }
              }
            }
          }
Applying patch to resource carrot-md-0 of type *v1alpha3.MachineDeployment ...
Waiting for kubernetes version to be updated for worker nodes...
...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
updating additional components: 'addons-management/tanzu-addons-manager' ...
updating additional components: 'tkr/tkr-controller' ...
updating additional components: 'addons-management/core-package-repo' ...
...
Waiting for packages to be up and running...
Waiting for package: antrea
Waiting for package: metrics-server
Waiting for package: pinniped
Waiting for package: tanzu-addons-manager
Waiting for package: vsphere-cpi
Waiting for package: vsphere-csi
Waiting for resource pinniped of type *v1alpha1.PackageInstall to be up and running
Waiting for resource vsphere-csi of type *v1alpha1.PackageInstall to be up and running
Waiting for resource antrea of type *v1alpha1.PackageInstall to be up and running
Waiting for resource tanzu-addons-manager of type *v1alpha1.PackageInstall to be up and running
Waiting for resource vsphere-cpi of type *v1alpha1.PackageInstall to be up and running
Waiting for resource metrics-server of type *v1alpha1.PackageInstall to be up and running
...
waiting for 'pinniped' Package to be installed, retrying
Successfully reconciled package: pinniped
Management cluster 'carrot' successfully upgraded to TKG version 'v1.4.0' with kubernetes version 'v1.21.2+vmware.1'

次のコマンドでManagement ClusterのみKubernetes v1.20.5+vmware.2にバージョンアップされたことがわかります。

$ tanzu cluster list --include-management-cluster
  NAME    NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES       PLAN  
  apple   default     running  1/1           2/2      v1.20.5+vmware.2  <none>      dev   
  grape   default     running  1/1           2/2      v1.20.4+vmware.1  <none>      prod  
  lemon   default     running  1/1           1/1      v1.20.5+vmware.2  <none>      dev   
  orange  default     running  1/1           2/2      v1.20.5+vmware.2  <none>      dev   
  carrot  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management  dev   

Management Clusterにkubectlコマンドでアクセスしてみます。

$ kubectl get node -o wide
NAME                           STATUS   ROLES                  AGE   VERSION            INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                 KERNEL-VERSION   CONTAINER-RUNTIME
carrot-control-plane-cw9v5     Ready    control-plane,master   50m   v1.21.2+vmware.1   192.168.11.98   192.168.11.98   VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6
carrot-md-0-697b79849d-mghbl   Ready    <none>                 27m   v1.21.2+vmware.1   192.168.11.38   192.168.11.38   VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6

アップデート後はControlPlaneのDHCP予約の設定を再度行う必要があります。

Workload Clusterをアップデート

次にWorkload Clusterをアップデートします。

次のコマンドでWorkload Clusterを作成します。

tanzu cluster upgrade lemon -v9 -y 

次のようなログが出力されます。

compatibility file (/Users/toshiaki/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /Users/toshiaki/.config/tanzu/tkg/bom already exists, skipping download
Creating management cluster client...
Validating configuration...
Creating workload cluster client...
Retrieving credentials for workload cluster lemon 
Getting secret for cluster
Waiting for resource lemon-kubeconfig of type *v1.Secret to be up and running
Merging credentials into kubeconfig file /Users/toshiaki/.kube-tkg/config 
Verifying kubernetes version...
Unable to detect current OS for the cluster. Using name:photon version:3 arch:amd64
Using OS options, name:photon version:3 arch:amd64
Retrieving configuration for upgrade cluster...
unable to find azure-image config in user configuration file, using it from BoM files
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Applying KubeadmControlPlane Patch: {
        "spec": {
          "version": "v1.21.2+vmware.1",
          "infrastructureTemplate": {
            "name": "lemon-control-plane-v1-21-2-vmware-1-7tsq5",
            "namespace": "default"
          },
          "kubeadmConfigSpec": {
            "clusterConfiguration": {
              "imageRepository" : "projects.registry.vmware.com/tkg",
              "dns": {
                "imageRepository": "projects.registry.vmware.com/tkg",
                "imageTag": "v1.8.0_vmware.5"
              },
              "etcd": {
                "local": {
                  "imageRepository": "projects.registry.vmware.com/tkg",
                  "imageTag": "v3.4.13_vmware.15"
                }
              }
            }
          }
        }
      }
Applying patch to resource lemon-control-plane of type *v1alpha3.KubeadmControlPlane ...
patch cluster object with operation status: 
    {
        "metadata": {
            "annotations": {
                "TKGOperationInfo" : "{\"Operation\":\"Upgrade\",\"OperationStartTimestamp\":\"2021-09-10 01:55:37.431052 +0000 UTC\",\"OperationTimeout\":900}",
                "TKGOperationLastObservedTimestamp" : "2021-09-10 01:55:37.431052 +0000 UTC"
            }
        }
    }
Waiting for kubernetes version to be updated for control plane nodes
...
control-plane is still being upgraded, reason:'ScalingDown', message:'Scaling down control plane to 1 replicas (actual 2)' , retrying
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Applying MachineDeployment Patch: {
            "spec": {
              "template": {
                "spec": {
                  "version": "v1.21.2+vmware.1",
                  "infrastructureRef": {
                    "name": "lemon-worker-8g-v1-21-2-vmware-1-ecyq4",
                    "namespace": "default"
                  }
                }
              }
            }
          }
Applying patch to resource lemon-md-0 of type *v1alpha3.MachineDeployment ...
Waiting for kubernetes version to be updated for worker nodes...
...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
cluster autoscaler is not enabled for cluster lemon
Waiting for packages to be up and running...
Waiting for package: antrea
Waiting for package: metrics-server
Waiting for package: vsphere-cpi
Waiting for package: vsphere-csi
...
Successfully reconciled package: antrea
waiting for 'vsphere-csi' Package to be installed, retrying
Successfully reconciled package: vsphere-csi
Cluster 'lemon' successfully upgraded to kubernetes version 'v1.21.2+vmware.1'

tanzu cluster listでWorkload Clusterを確認できます。lemonのみKubernetes 1.21.2にバージョンアップされたことがわかります

$ tanzu cluster list --include-management-cluster
  NAME    NAMESPACE   STATUS   CONTROLPLANE  WORKERS  KUBERNETES        ROLES       PLAN  
  apple   default     running  1/1           2/2      v1.20.5+vmware.2  <none>      dev   
  grape   default     running  1/1           2/2      v1.20.4+vmware.1  <none>      prod  
  lemon   default     running  1/1           1/1      v1.21.2+vmware.1  <none>      dev   
  orange  default     running  1/1           2/2      v1.20.5+vmware.2  <none>      dev   
  carrot  tkg-system  running  1/1           1/1      v1.21.2+vmware.1  management  dev 
kubectl config use-context lemon-admin@lemon 
$ kubectl get node -owide
NAME                          STATUS   ROLES                  AGE   VERSION            INTERNAL-IP     EXTERNAL-IP     OS-IMAGE                 KERNEL-VERSION   CONTAINER-RUNTIME
lemon-control-plane-jd4gc     Ready    control-plane,master   65m   v1.21.2+vmware.1   192.168.11.39   192.168.11.39   VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6
lemon-md-0-7fb4f9f5dc-wswp6   Ready    <none>                 42m   v1.21.2+vmware.1   192.168.11.61   192.168.11.61   VMware Photon OS/Linux   4.19.198-1.ph3   containerd://1.4.6

アップデート後はControlPlaneのDHCP予約の設定を再度行う必要があります。

ここまででKubernetesのアップデートは完了しました。

Packageの確認

TKG 1.4ではAdd-onやExtensionは新しくCarvelのPackaging APIで管理されるようになりました。

Add-on -> Core Packageは自動でアップデートされます。

Management Cluster

$ kubectl get package,packageinstall,app -n tkg-system --context carrot-admin@carrot 
NAME                                                                                                        PACKAGEMETADATA NAME                                 VERSION                 AGE
package.data.packaging.carvel.dev/addons-manager.tanzu.vmware.com.1.4.0+vmware.1-tkg.1                      addons-manager.tanzu.vmware.com                      1.4.0+vmware.1-tkg.1    10h26m31s
package.data.packaging.carvel.dev/ako-operator.tanzu.vmware.com.1.4.0+vmware.1-tkg.1                        ako-operator.tanzu.vmware.com                        1.4.0+vmware.1-tkg.1    10h26m31s
package.data.packaging.carvel.dev/antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1                             antrea.tanzu.vmware.com                              0.13.3+vmware.1-tkg.1   10h26m31s
package.data.packaging.carvel.dev/calico.tanzu.vmware.com.3.11.3+vmware.1-tkg.1                             calico.tanzu.vmware.com                              3.11.3+vmware.1-tkg.1   10h26m31s
package.data.packaging.carvel.dev/kapp-controller.tanzu.vmware.com.0.23.0+vmware.1-tkg.1                    kapp-controller.tanzu.vmware.com                     0.23.0+vmware.1-tkg.1   10h26m30s
package.data.packaging.carvel.dev/load-balancer-and-ingress-service.tanzu.vmware.com.1.4.3+vmware.1-tkg.1   load-balancer-and-ingress-service.tanzu.vmware.com   1.4.3+vmware.1-tkg.1    10h26m30s
package.data.packaging.carvel.dev/metrics-server.tanzu.vmware.com.0.4.0+vmware.1-tkg.1                      metrics-server.tanzu.vmware.com                      0.4.0+vmware.1-tkg.1    10h26m30s
package.data.packaging.carvel.dev/pinniped.tanzu.vmware.com.0.4.4+vmware.1-tkg.1                            pinniped.tanzu.vmware.com                            0.4.4+vmware.1-tkg.1    10h26m30s
package.data.packaging.carvel.dev/vsphere-cpi.tanzu.vmware.com.1.21.0+vmware.1-tkg.1                        vsphere-cpi.tanzu.vmware.com                         1.21.0+vmware.1-tkg.1   10h26m30s
package.data.packaging.carvel.dev/vsphere-csi.tanzu.vmware.com.2.3.0+vmware.1-tkg.2                         vsphere-csi.tanzu.vmware.com                         2.3.0+vmware.1-tkg.2    10h26m30s

NAME                                                       PACKAGE NAME                      PACKAGE VERSION         DESCRIPTION           AGE
packageinstall.packaging.carvel.dev/antrea                 antrea.tanzu.vmware.com           0.13.3+vmware.1-tkg.1   Reconcile succeeded   10h
packageinstall.packaging.carvel.dev/metrics-server         metrics-server.tanzu.vmware.com   0.4.0+vmware.1-tkg.1    Reconcile succeeded   10h
packageinstall.packaging.carvel.dev/pinniped               pinniped.tanzu.vmware.com         0.4.4+vmware.1-tkg.1    Reconcile succeeded   10h
packageinstall.packaging.carvel.dev/tanzu-addons-manager   addons-manager.tanzu.vmware.com   1.4.0+vmware.1-tkg.1    Reconcile succeeded   10h
packageinstall.packaging.carvel.dev/vsphere-cpi            vsphere-cpi.tanzu.vmware.com      1.21.0+vmware.1-tkg.1   Reconcile succeeded   10h
packageinstall.packaging.carvel.dev/vsphere-csi            vsphere-csi.tanzu.vmware.com      2.3.0+vmware.1-tkg.2    Reconcile succeeded   10h

NAME                                        DESCRIPTION           SINCE-DEPLOY   AGE
app.kappctrl.k14s.io/antrea                 Reconcile succeeded   96s            166d
app.kappctrl.k14s.io/metrics-server         Reconcile succeeded   114s           166d
app.kappctrl.k14s.io/pinniped               Reconcile succeeded   4m12s          56d
app.kappctrl.k14s.io/tanzu-addons-manager   Reconcile succeeded   5m10s          167d
app.kappctrl.k14s.io/vsphere-cpi            Reconcile succeeded   116s           143d
app.kappctrl.k14s.io/vsphere-csi            Reconcile succeeded   83s            166d

Workload Cluster

$ kubectl get package,packageinstall,app -n tkg-system --context lemon-admin@lemon  
NAME                                                                                                        PACKAGEMETADATA NAME                                 VERSION                 AGE
package.data.packaging.carvel.dev/cert-manager.tanzu.vmware.com.1.1.0+vmware.1-tkg.2                        cert-manager.tanzu.vmware.com                        1.1.0+vmware.1-tkg.2    42m16s
package.data.packaging.carvel.dev/contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1                            contour.tanzu.vmware.com                             1.17.1+vmware.1-tkg.1   42m16s
package.data.packaging.carvel.dev/external-dns.tanzu.vmware.com.0.8.0+vmware.1-tkg.1                        external-dns.tanzu.vmware.com                        0.8.0+vmware.1-tkg.1    42m16s
package.data.packaging.carvel.dev/fluent-bit.tanzu.vmware.com.1.7.5+vmware.1-tkg.1                          fluent-bit.tanzu.vmware.com                          1.7.5+vmware.1-tkg.1    42m16s
package.data.packaging.carvel.dev/grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1                             grafana.tanzu.vmware.com                             7.5.7+vmware.1-tkg.1    42m16s
package.data.packaging.carvel.dev/harbor.tanzu.vmware.com.2.2.3+vmware.1-tkg.1                              harbor.tanzu.vmware.com                              2.2.3+vmware.1-tkg.1    42m16s
package.data.packaging.carvel.dev/multus-cni.tanzu.vmware.com.3.7.1+vmware.1-tkg.1                          multus-cni.tanzu.vmware.com                          3.7.1+vmware.1-tkg.1    42m16s
package.data.packaging.carvel.dev/prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1                         prometheus.tanzu.vmware.com                          2.27.0+vmware.1-tkg.1   42m16s
package.data.packaging.carvel.dev/addons-manager.tanzu.vmware.com.1.4.0+vmware.1-tkg.1                      addons-manager.tanzu.vmware.com                      1.4.0+vmware.1-tkg.1    40m56s
package.data.packaging.carvel.dev/ako-operator.tanzu.vmware.com.1.4.0+vmware.1-tkg.1                        ako-operator.tanzu.vmware.com                        1.4.0+vmware.1-tkg.1    40m56s
package.data.packaging.carvel.dev/antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1                             antrea.tanzu.vmware.com                              0.13.3+vmware.1-tkg.1   40m56s
package.data.packaging.carvel.dev/calico.tanzu.vmware.com.3.11.3+vmware.1-tkg.1                             calico.tanzu.vmware.com                              3.11.3+vmware.1-tkg.1   40m56s
package.data.packaging.carvel.dev/kapp-controller.tanzu.vmware.com.0.23.0+vmware.1-tkg.1                    kapp-controller.tanzu.vmware.com                     0.23.0+vmware.1-tkg.1   40m56s
package.data.packaging.carvel.dev/load-balancer-and-ingress-service.tanzu.vmware.com.1.4.3+vmware.1-tkg.1   load-balancer-and-ingress-service.tanzu.vmware.com   1.4.3+vmware.1-tkg.1    40m56s
package.data.packaging.carvel.dev/metrics-server.tanzu.vmware.com.0.4.0+vmware.1-tkg.1                      metrics-server.tanzu.vmware.com                      0.4.0+vmware.1-tkg.1    40m55s
package.data.packaging.carvel.dev/pinniped.tanzu.vmware.com.0.4.4+vmware.1-tkg.1                            pinniped.tanzu.vmware.com                            0.4.4+vmware.1-tkg.1    40m56s
package.data.packaging.carvel.dev/vsphere-cpi.tanzu.vmware.com.1.21.0+vmware.1-tkg.1                        vsphere-cpi.tanzu.vmware.com                         1.21.0+vmware.1-tkg.1   40m56s
package.data.packaging.carvel.dev/vsphere-csi.tanzu.vmware.com.2.3.0+vmware.1-tkg.2                         vsphere-csi.tanzu.vmware.com                         2.3.0+vmware.1-tkg.2    40m55s

NAME                                                 PACKAGE NAME                      PACKAGE VERSION         DESCRIPTION           AGE
packageinstall.packaging.carvel.dev/antrea           antrea.tanzu.vmware.com           0.13.3+vmware.1-tkg.1   Reconcile succeeded   45m
packageinstall.packaging.carvel.dev/metrics-server   metrics-server.tanzu.vmware.com   0.4.0+vmware.1-tkg.1    Reconcile succeeded   45m
packageinstall.packaging.carvel.dev/vsphere-cpi      vsphere-cpi.tanzu.vmware.com      1.21.0+vmware.1-tkg.1   Reconcile succeeded   45m
packageinstall.packaging.carvel.dev/vsphere-csi      vsphere-csi.tanzu.vmware.com      2.3.0+vmware.1-tkg.2    Reconcile succeeded   45m

NAME                                  DESCRIPTION           SINCE-DEPLOY   AGE
app.kappctrl.k14s.io/antrea           Reconcile succeeded   113s           166d
app.kappctrl.k14s.io/metrics-server   Reconcile succeeded   72s            166d
app.kappctrl.k14s.io/vsphere-cpi      Reconcile succeeded   2m10s          158d
app.kappctrl.k14s.io/vsphere-csi      Reconcile succeeded   86s            166d

全てReconcile succeededになっていればOKです。

tanzu CLIで確認することもできます。

$ tanzu package available list
/ Retrieving available packages... 
  NAME                           DISPLAY-NAME  SHORT-DESCRIPTION                                                                                           
  cert-manager.tanzu.vmware.com  cert-manager  Certificate management                                                                                      
  contour.tanzu.vmware.com       Contour       An ingress controller                                                                                       
  external-dns.tanzu.vmware.com  external-dns  This package provides DNS synchronization functionality.                                                    
  fluent-bit.tanzu.vmware.com    fluent-bit    Fluent Bit is a fast Log Processor and Forwarder                                                            
  grafana.tanzu.vmware.com       grafana       Visualization and analytics software                                                                        
  harbor.tanzu.vmware.com        Harbor        OCI Registry                                                                                                
  multus-cni.tanzu.vmware.com    multus-cni    This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes  
  prometheus.tanzu.vmware.com    prometheus    A time series database for your metrics  

$ tanzu package installed list -A
\ Retrieving installed packages... 
  NAME            PACKAGE-NAME                     PACKAGE-VERSION  STATUS               NAMESPACE   
  antrea          antrea.tanzu.vmware.com                           Reconcile succeeded  tkg-system  
  metrics-server  metrics-server.tanzu.vmware.com                   Reconcile succeeded  tkg-system  
  vsphere-cpi     vsphere-cpi.tanzu.vmware.com                      Reconcile succeeded  tkg-system  
  vsphere-csi     vsphere-csi.tanzu.vmware.com                      Reconcile succeeded  tkg-system 

TKG Extensionのアップデート

TKG Extensionを使用している場合はこちらもアップデートが必要です。 Addonと同じくkapp controllerのPackageへの移行が必要です。

ドキュメントを参照してください。 https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-extensions.html

Cert Manager Packageのインストール

Cert Managerのインストール方法をメモしておきます。1.3.1からのアップデートの場合も同じです。Packageのインストールはtanzu package installコマンドを使用します。

tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --namespace cert-manager --version 1.1.0+vmware.1-tkg.2 --create-namespace

次のようなログが出力されます。

/ Installing package 'cert-manager.tanzu.vmware.com' 
| Creating namespace 'cert-manager' 
| Getting package metadata for 'cert-manager.tanzu.vmware.com' 
| Creating service account 'cert-manager-cert-manager-sa' 
| Creating cluster admin role 'cert-manager-cert-manager-cluster-role' 
| Creating cluster role binding 'cert-manager-cert-manager-cluster-rolebinding' 
/ Creating package resource 
| Package install status: Reconciling 


 Added installed package 'cert-manager' in namespace 'cert-manager'

イントールされたパッケージを確認します。

$ kubectl get app,packageinstall -n cert-manager
NAME                                DESCRIPTION           SINCE-DEPLOY   AGE
app.kappctrl.k14s.io/cert-manager   Reconcile succeeded   54s            66s

NAME                                               PACKAGE NAME                    PACKAGE VERSION        DESCRIPTION           AGE
packageinstall.packaging.carvel.dev/cert-manager   cert-manager.tanzu.vmware.com   1.1.0+vmware.1-tkg.2   Reconcile succeeded   66s

次のコマンドでも確認できます。

$ tanzu package installed list -n cert-manager
- Retrieving installed packages... 
  NAME          PACKAGE-NAME                   PACKAGE-VERSION       STATUS               
  cert-manager  cert-manager.tanzu.vmware.com  1.1.0+vmware.1-tkg.2  Reconcile succeeded  

Contour Packageのインストール

次にContourのインストール方法をメモしておきます。1.3.1からのアップデートの場合も同じです。

設定項目をvalues.yamlに定義します。例えばMetalLBを使ってEnvoyのService TypeをLoadBalancerに変更したい場合は次のような設定になります。 1.3までとは異なり、このvalues.yaml#@から始まる行を書かないでください。含めるとデプロイがエラーになります。

---
infrastructure_provider: "vsphere"
envoy:
  service:
    type: LoadBalancer

1.3からアップデートする場合は、次のコマンドでvalues.yamlを出力してください。

kubectl get secret -n tanzu-system-ingress contour-data-values -otemplate='{{index .data "values.yaml" | base64decode}}' > values.yaml 

このvalues.yamlから#@から始まる行を削除してください。

values.yamlが準備できたら次のコマンドでインストールします。

tanzu package install contour --package-name contour.tanzu.vmware.com --namespace tanzu-system-ingress --version 1.17.1+vmware.1-tkg.1 -f values.yml --create-namespace

values.yamlを適用した結果どのようなManifestがデプロイされるのか知りたい場合は次のようにして確認できます。

まずはImage PackageのBundleイメージ名を取得します。

$ kubectl get package contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 -otemplate='{{(index .spec.template.spec.fetch 0).imgpkgBundle.image}}'
projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4

このイメージをimgpkgコマンドでダウンロードします。

imgpkg pull -b projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4 -o /tmp/contour

マニフェストが展開されます。

$ find /tmp/contour/config

/tmp/contour/config
/tmp/contour/config/contour.star
/tmp/contour/config/_ytt_lib
/tmp/contour/config/_ytt_lib/bundle
/tmp/contour/config/_ytt_lib/bundle/config
/tmp/contour/config/_ytt_lib/bundle/config/overlays
/tmp/contour/config/_ytt_lib/bundle/config/overlays/change-namespace.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/remove-certgen-job.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/update-contour-configmap.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/update-crds.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/update-envoy-daemonset.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/add-certmanager-certs.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/update-envoy-service.yaml
/tmp/contour/config/_ytt_lib/bundle/config/overlays/update-contour-deployment.yaml
/tmp/contour/config/_ytt_lib/bundle/config/certificates.lib.yaml
/tmp/contour/config/_ytt_lib/bundle/config/values.yaml
/tmp/contour/config/_ytt_lib/bundle/config/upstream
/tmp/contour/config/_ytt_lib/bundle/config/upstream/contour.yaml
/tmp/contour/config/contour.yaml
/tmp/contour/config/overlays
/tmp/contour/config/overlays/update-role-envoy-psp.yaml
/tmp/contour/config/overlays/update-role-contour-psp.yaml
/tmp/contour/config/overlays/add-envoy-rbac.yaml
/tmp/contour/config/values.yaml

このディレクトリと作成したvalues.yamlyttに食わせることでこのPackageでデプロイされるYAMLを確認できます。

ytt -f /tmp/contour/config --data-values-file values.yaml 

Packageに対してOverlayを使用したい場合

TKG ExtensionをyttのOverlayでカスタマイズしているかもしてません。

tanzu package installコマンドではoverlayを設定できません。代わりにPackageInstallリソースに対してアノテーションを設定する必要があります。

次のドキュメントを参照してください。


TKG 1.3.1-patch1 -> 1.4.0へのアップデートを行いました。

設定ファイルの場所が~/.tanzuから~/.config/tanzuに変わったこと以外は、 TKG 1.2 -> 1.3に比べれば簡単で特にハマりポイントはありませんでした。

TKG Extensionのアップデートは注意が必要です。