--- title: Tanzu Kubernetes Grid on vSphereを1.3.1-patch1 -> 1.4.0にアップデートするメモ tags: ["Kubernetes", "vSphere", "TKG", "Tanzu", "Cluster API"] categories: ["Dev", "CaaS", "Kubernetes", "TKG", "vSphere"] date: 2021-09-10T03:13:55Z updated: 2021-09-14T01:05:20Z --- "[Tanzu Kubernetes Grid on vSphereを1.3.0 -> 1.3.1-patch1にアップデートするメモ](/entries/652)"で作成した環境をアップデートします。 アップデートの手順はこちら * https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-index.html * https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-management-cluster.html * https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-workload-clusters.html Release Noteはこちら https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/rn/VMware-Tanzu-Kubernetes-Grid-14-Release-Notes.html **Table of contents** ### CLIのインストール [https://customerconnect.vmware.com/en/downloads/details?downloadGroup=TKG-140&productId=988&rPId=73652](https://customerconnect.vmware.com/en/downloads/details?downloadGroup=TKG-140&productId=988&rPId=73652) から - VMware Tanzu CLI for Mac ``` tar xvf tanzu-cli-bundle-darwin-amd64.tar install cli/core/v1.4.0/tanzu-core-darwin_amd64 /usr/local/bin/tanzu ``` 次のバージョンで動作確認します。 ``` $ tanzu version version: v1.3.1 buildDate: 2021-05-07 sha: e5c37c4 ``` 初回コマンド実行時に、 ``` ! Configuration is now stored in /Users/toshiaki/.config/tanzu. Legacy configuration directory /Users/toshiaki/.tanzu is deprecated and will be removed in a future release. ! To complete migration, please remove legacy configuration directory /Users/toshiaki/.tanzu and adjust your script(s), if any, to point to the new location. ``` というようなログが出力されます。TKG 1.3.1の時に使用していた`$HOME/.tanzu/tkg/providers`はバックアップフォルダにrenameされ、 1.4.0用のファイルが`$HOME/.config/tanzu/tkg/providers`に用意されます。 この時、`$HOME/.tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml`のようなクラスタ構築時にカスタマイズしたytt overlayファイルは1.3.1に引き継がれないので、 1.3.1->1.4.0においても引き続き利用したいOverlayファイルは改めて`$HOME/.config/tanzu/tkg/providers`へ設定する必要があります。 ``` cp -r $HOME/.config/tanzu/tkg/providers-20210910004336-y622uivb/infrastructure-vsphere/ytt/vsphere-overlay.yaml $HOME/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml ``` ``` tanzu plugin clean tanzu plugin install --local cli all ``` プラグイン一覧を確認します。 ``` $ tanzu plugin list NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS alpha v1.3.1 Alpha CLI commands core not installed cluster v1.3.1 Kubernetes cluster operations core v1.4.0 installed kubernetes-release v1.3.1 Kubernetes release operations core v1.4.0 installed login v1.3.1 Login to the platform core v1.4.0 installed management-cluster v1.3.1 Kubernetes management cluster operations core v1.4.0 installed package Tanzu package management v1.4.0 installed pinniped-auth v1.3.1 Pinniped authentication operations (usually not directly invoked) core v1.4.0 installed ``` 旧バージョンからアップデートした場合は、プラグインのLATEST VERSIONが1.3.1になるようです。 プラグインを配布するbucket名が変わったようです。 `~/.config/tanzu/config.yaml`の`gcpPluginRepository`の`core`の`bucketName`を`tanzu-cli` -> `tanzu-cli-tkg`に変更してください。 ```yaml apiVersion: config.tanzu.vmware.com/v1alpha1 clientOptions: cli: repositories: - gcpPluginRepository: bucketName: tanzu-cli-tkg name: core ``` これで正しく表示されます。 ``` $ tanzu plugin list NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS alpha v1.4.0 Alpha CLI commands core not installed cluster v1.4.0 Kubernetes cluster operations core v1.4.0 installed kubernetes-release v1.4.0 Kubernetes release operations core v1.4.0 installed login v1.4.0 Login to the platform core v1.4.0 installed management-cluster v1.4.0 Kubernetes management cluster operations core v1.4.0 installed package v1.4.0 Tanzu package management core v1.4.0 installed pinniped-auth v1.4.0 Pinniped authentication operations (usually not directly invoked) core v1.4.0 installed ``` ### vCenterにOVAファイルをアップロード [https://my.vmware.com/en/web/vmware/downloads/details?downloadGroup=TKG-131&productId=988&rPId=65946](https://my.vmware.com/en/web/vmware/downloads/details?downloadGroup=TKG-131&productId=988&rPId=65946) から - Photon v3 Kubernetes v1.21.2 OVA をダウンロードし、`~/Downloads`に保存します。 TKGをインストールする環境の情報を次の例のように設定してください。 ``` cat < tkg-env.sh export GOVC_URL=vcsa.v.maki.lol export GOVC_USERNAME=administrator@vsphere.local export GOVC_PASSWORD=VMware1! export GOVC_DATACENTER=Datacenter export GOVC_NETWORK="VM Network" export GOVC_DATASTORE=datastore1 export GOVC_RESOURCE_POOL=/Datacenter/host/Cluster/Resources/tkg export GOVC_INSECURE=1 export TEMPLATE_FOLDER=/Datacenter/vm/tkg EOF ``` 次のコマンドでOVAファイルをvCenterにアップロードします。 ``` source tkg-env.sh govc import.ova -folder $TEMPLATE_FOLDER ~/Downloads/photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova govc vm.markastemplate $TEMPLATE_FOLDER/photon-3-kube-v1.21.2 ``` ### Management Clusterをアップデート 次のコマンドを実行し、アップデート対象のManagement Clusterにログインします。 ``` tanzu login ``` アップデート前のクラスタ一覧を確認します。Kubernetesのバージョンは`v1.20.4+vmware.1`です。 ``` $ tanzu cluster list --include-management-cluster NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN apple default running 1/1 2/2 v1.20.5+vmware.2 dev grape default running 1/1 2/2 v1.20.4+vmware.1 prod lemon default running 1/1 1/1 v1.20.5+vmware.2 dev orange default running 1/1 2/2 v1.20.5+vmware.2 dev carrot tkg-system running 1/1 1/1 v1.20.5+vmware.2 management dev ``` 次のコマンドでManagement Clusterをアップデートします。 ``` tanzu management-cluster upgrade -v9 -y ``` 次のようなログが出力されます。 ``` compatibility file (/Users/toshiaki/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download BOM files inside /Users/toshiaki/.config/tanzu/tkg/bom already exists, skipping download cluster specific secret is not present, fallback on bootstrap credential secret Upgrading management cluster providers... clusterctl upgrade apply options: {Kubeconfig:{Path:/Users/toshiaki/.kube-tkg/config Context:carrot-admin@carrot} ManagementGroup:capi-system/cluster-api Contract: CoreProvider:capi-system/cluster-api:v0.3.23 BootstrapProviders:[capi-kubeadm-bootstrap-system/kubeadm:v0.3.23] ControlPlaneProviders:[capi-kubeadm-control-plane-system/kubeadm:v0.3.23] InfrastructureProviders:[capv-system/vsphere:v0.7.10]} Checking cert-manager version... Deleting cert-manager Version="v0.16.1" Installing cert-manager Version="v1.1.0" Waiting for cert-manager to be available... Performing upgrade... Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system" Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system" Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system" Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system" Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system" Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system" Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system" Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system" Waiting for provider infrastructure-vsphere Waiting for provider bootstrap-kubeadm Waiting for provider cluster-api Waiting for provider control-plane-kubeadm Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running Waiting for resource capi-kubeadm-bootstrap-controller-manager of type *v1.Deployment to be up and running Waiting for resource capi-controller-manager of type *v1.Deployment to be up and running Waiting for resource capi-controller-manager of type *v1.Deployment to be up and running Waiting for resource capi-kubeadm-bootstrap-controller-manager of type *v1.Deployment to be up and running pods are not yet running for deployment 'capi-kubeadm-control-plane-controller-manager' in namespace 'capi-kubeadm-control-plane-system', retrying pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying Passed waiting on provider cluster-api after 341.338078ms Passed waiting on provider bootstrap-kubeadm after 340.817975ms pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running Passed waiting on provider control-plane-kubeadm after 5.455206185s pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running Passed waiting on provider infrastructure-vsphere after 20.246601997s Success waiting on all providers. Management cluster providers upgraded successfully... Upgrading management cluster kubernetes version... Creating management cluster client... Verifying kubernetes version... Unable to detect current OS for the cluster. Using name:photon version:3 arch:amd64 Using OS options, name:photon version:3 arch:amd64 Retrieving configuration for upgrade cluster... unable to find azure-image config in user configuration file, using it from BoM files Create InfrastructureTemplate for upgrade... cluster specific secret is not present, fallback on bootstrap credential secret Upgrading control plane nodes... Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1... Applying KubeadmControlPlane Patch: { "spec": { "version": "v1.21.2+vmware.1", "infrastructureTemplate": { "name": "carrot-control-plane-v1-21-2-vmware-1-dl61v", "namespace": "tkg-system" }, "kubeadmConfigSpec": { "clusterConfiguration": { "imageRepository" : "projects.registry.vmware.com/tkg", "dns": { "imageRepository": "projects.registry.vmware.com/tkg", "imageTag": "v1.8.0_vmware.5" }, "etcd": { "local": { "imageRepository": "projects.registry.vmware.com/tkg", "imageTag": "v3.4.13_vmware.15" } } } } } } Applying patch to resource carrot-control-plane of type *v1alpha3.KubeadmControlPlane ... patch cluster object with operation status: { "metadata": { "annotations": { "TKGOperationInfo" : "{\"Operation\":\"Upgrade\",\"OperationStartTimestamp\":\"2021-09-09 16:10:45.498847 +0000 UTC\",\"OperationTimeout\":900}", "TKGOperationLastObservedTimestamp" : "2021-09-09 16:10:45.498847 +0000 UTC" } } } Waiting for kubernetes version to be updated for control plane nodes ... Upgrading worker nodes... Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1... Applying MachineDeployment Patch: { "spec": { "template": { "spec": { "version": "v1.21.2+vmware.1", "infrastructureRef": { "name": "carrot-worker-v1-19-3-vmware-1-muxc1-v1-20-4-vmware-1-5mlj6-v1-20-5-vmware-2-4xqok-v1-21-2-vmware-1-fgltw", "namespace": "tkg-system" } } } } } Applying patch to resource carrot-md-0 of type *v1alpha3.MachineDeployment ... Waiting for kubernetes version to be updated for worker nodes... ... updating additional components: 'metadata/tkg' ... updating additional components: 'addons-management/kapp-controller' ... updating additional components: 'addons-management/standard-package-repo' ... updating additional components: 'addons-management/tanzu-addons-manager' ... updating additional components: 'tkr/tkr-controller' ... updating additional components: 'addons-management/core-package-repo' ... ... Waiting for packages to be up and running... Waiting for package: antrea Waiting for package: metrics-server Waiting for package: pinniped Waiting for package: tanzu-addons-manager Waiting for package: vsphere-cpi Waiting for package: vsphere-csi Waiting for resource pinniped of type *v1alpha1.PackageInstall to be up and running Waiting for resource vsphere-csi of type *v1alpha1.PackageInstall to be up and running Waiting for resource antrea of type *v1alpha1.PackageInstall to be up and running Waiting for resource tanzu-addons-manager of type *v1alpha1.PackageInstall to be up and running Waiting for resource vsphere-cpi of type *v1alpha1.PackageInstall to be up and running Waiting for resource metrics-server of type *v1alpha1.PackageInstall to be up and running ... waiting for 'pinniped' Package to be installed, retrying Successfully reconciled package: pinniped Management cluster 'carrot' successfully upgraded to TKG version 'v1.4.0' with kubernetes version 'v1.21.2+vmware.1' ``` 次のコマンドでManagement ClusterのみKubernetes `v1.20.5+vmware.2`にバージョンアップされたことがわかります。 ``` $ tanzu cluster list --include-management-cluster NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN apple default running 1/1 2/2 v1.20.5+vmware.2 dev grape default running 1/1 2/2 v1.20.4+vmware.1 prod lemon default running 1/1 1/1 v1.20.5+vmware.2 dev orange default running 1/1 2/2 v1.20.5+vmware.2 dev carrot tkg-system running 1/1 1/1 v1.21.2+vmware.1 management dev ``` Management Clusterに`kubectl`コマンドでアクセスしてみます。 ``` $ kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME carrot-control-plane-cw9v5 Ready control-plane,master 50m v1.21.2+vmware.1 192.168.11.98 192.168.11.98 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6 carrot-md-0-697b79849d-mghbl Ready 27m v1.21.2+vmware.1 192.168.11.38 192.168.11.38 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6 ``` アップデート後はControlPlaneのDHCP予約の設定を再度行う必要があります。 ### Workload Clusterをアップデート 次にWorkload Clusterをアップデートします。 次のコマンドでWorkload Clusterを作成します。 ``` tanzu cluster upgrade lemon -v9 -y ``` 次のようなログが出力されます。 ``` compatibility file (/Users/toshiaki/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download BOM files inside /Users/toshiaki/.config/tanzu/tkg/bom already exists, skipping download Creating management cluster client... Validating configuration... Creating workload cluster client... Retrieving credentials for workload cluster lemon Getting secret for cluster Waiting for resource lemon-kubeconfig of type *v1.Secret to be up and running Merging credentials into kubeconfig file /Users/toshiaki/.kube-tkg/config Verifying kubernetes version... Unable to detect current OS for the cluster. Using name:photon version:3 arch:amd64 Using OS options, name:photon version:3 arch:amd64 Retrieving configuration for upgrade cluster... unable to find azure-image config in user configuration file, using it from BoM files Create InfrastructureTemplate for upgrade... cluster specific secret is not present, fallback on bootstrap credential secret Upgrading control plane nodes... Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1... Applying KubeadmControlPlane Patch: { "spec": { "version": "v1.21.2+vmware.1", "infrastructureTemplate": { "name": "lemon-control-plane-v1-21-2-vmware-1-7tsq5", "namespace": "default" }, "kubeadmConfigSpec": { "clusterConfiguration": { "imageRepository" : "projects.registry.vmware.com/tkg", "dns": { "imageRepository": "projects.registry.vmware.com/tkg", "imageTag": "v1.8.0_vmware.5" }, "etcd": { "local": { "imageRepository": "projects.registry.vmware.com/tkg", "imageTag": "v3.4.13_vmware.15" } } } } } } Applying patch to resource lemon-control-plane of type *v1alpha3.KubeadmControlPlane ... patch cluster object with operation status: { "metadata": { "annotations": { "TKGOperationInfo" : "{\"Operation\":\"Upgrade\",\"OperationStartTimestamp\":\"2021-09-10 01:55:37.431052 +0000 UTC\",\"OperationTimeout\":900}", "TKGOperationLastObservedTimestamp" : "2021-09-10 01:55:37.431052 +0000 UTC" } } } Waiting for kubernetes version to be updated for control plane nodes ... control-plane is still being upgraded, reason:'ScalingDown', message:'Scaling down control plane to 1 replicas (actual 2)' , retrying Upgrading worker nodes... Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1... Applying MachineDeployment Patch: { "spec": { "template": { "spec": { "version": "v1.21.2+vmware.1", "infrastructureRef": { "name": "lemon-worker-8g-v1-21-2-vmware-1-ecyq4", "namespace": "default" } } } } } Applying patch to resource lemon-md-0 of type *v1alpha3.MachineDeployment ... Waiting for kubernetes version to be updated for worker nodes... ... updating additional components: 'metadata/tkg' ... updating additional components: 'addons-management/kapp-controller' ... updating additional components: 'addons-management/standard-package-repo' ... cluster autoscaler is not enabled for cluster lemon Waiting for packages to be up and running... Waiting for package: antrea Waiting for package: metrics-server Waiting for package: vsphere-cpi Waiting for package: vsphere-csi ... Successfully reconciled package: antrea waiting for 'vsphere-csi' Package to be installed, retrying Successfully reconciled package: vsphere-csi Cluster 'lemon' successfully upgraded to kubernetes version 'v1.21.2+vmware.1' ``` `tanzu cluster list`でWorkload Clusterを確認できます。`lemon`のみKubernetes 1.21.2にバージョンアップされたことがわかります ``` $ tanzu cluster list --include-management-cluster NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN apple default running 1/1 2/2 v1.20.5+vmware.2 dev grape default running 1/1 2/2 v1.20.4+vmware.1 prod lemon default running 1/1 1/1 v1.21.2+vmware.1 dev orange default running 1/1 2/2 v1.20.5+vmware.2 dev carrot tkg-system running 1/1 1/1 v1.21.2+vmware.1 management dev ``` ``` kubectl config use-context lemon-admin@lemon ``` ``` $ kubectl get node -owide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME lemon-control-plane-jd4gc Ready control-plane,master 65m v1.21.2+vmware.1 192.168.11.39 192.168.11.39 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6 lemon-md-0-7fb4f9f5dc-wswp6 Ready 42m v1.21.2+vmware.1 192.168.11.61 192.168.11.61 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6 ``` アップデート後はControlPlaneのDHCP予約の設定を再度行う必要があります。 ここまででKubernetesのアップデートは完了しました。 ### Packageの確認 TKG 1.4ではAdd-onやExtensionは新しくCarvelの[Packaging API](https://carvel.dev/kapp-controller/docs/latest/packaging/)で管理されるようになりました。 * Add-on -> [Core Pakcage](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-packages-core-index.html) * Extension -> [User-Managed Packages](https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-packages-user-managed-index.html) Add-on -> Core Packageは自動でアップデートされます。 Management Cluster ``` $ kubectl get package,packageinstall,app -n tkg-system --context carrot-admin@carrot NAME PACKAGEMETADATA NAME VERSION AGE package.data.packaging.carvel.dev/addons-manager.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 addons-manager.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 10h26m31s package.data.packaging.carvel.dev/ako-operator.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 ako-operator.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 10h26m31s package.data.packaging.carvel.dev/antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1 antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 10h26m31s package.data.packaging.carvel.dev/calico.tanzu.vmware.com.3.11.3+vmware.1-tkg.1 calico.tanzu.vmware.com 3.11.3+vmware.1-tkg.1 10h26m31s package.data.packaging.carvel.dev/kapp-controller.tanzu.vmware.com.0.23.0+vmware.1-tkg.1 kapp-controller.tanzu.vmware.com 0.23.0+vmware.1-tkg.1 10h26m30s package.data.packaging.carvel.dev/load-balancer-and-ingress-service.tanzu.vmware.com.1.4.3+vmware.1-tkg.1 load-balancer-and-ingress-service.tanzu.vmware.com 1.4.3+vmware.1-tkg.1 10h26m30s package.data.packaging.carvel.dev/metrics-server.tanzu.vmware.com.0.4.0+vmware.1-tkg.1 metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 10h26m30s package.data.packaging.carvel.dev/pinniped.tanzu.vmware.com.0.4.4+vmware.1-tkg.1 pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 10h26m30s package.data.packaging.carvel.dev/vsphere-cpi.tanzu.vmware.com.1.21.0+vmware.1-tkg.1 vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 10h26m30s package.data.packaging.carvel.dev/vsphere-csi.tanzu.vmware.com.2.3.0+vmware.1-tkg.2 vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 10h26m30s NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE packageinstall.packaging.carvel.dev/antrea antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 Reconcile succeeded 10h packageinstall.packaging.carvel.dev/metrics-server metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 Reconcile succeeded 10h packageinstall.packaging.carvel.dev/pinniped pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 Reconcile succeeded 10h packageinstall.packaging.carvel.dev/tanzu-addons-manager addons-manager.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 Reconcile succeeded 10h packageinstall.packaging.carvel.dev/vsphere-cpi vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 Reconcile succeeded 10h packageinstall.packaging.carvel.dev/vsphere-csi vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 Reconcile succeeded 10h NAME DESCRIPTION SINCE-DEPLOY AGE app.kappctrl.k14s.io/antrea Reconcile succeeded 96s 166d app.kappctrl.k14s.io/metrics-server Reconcile succeeded 114s 166d app.kappctrl.k14s.io/pinniped Reconcile succeeded 4m12s 56d app.kappctrl.k14s.io/tanzu-addons-manager Reconcile succeeded 5m10s 167d app.kappctrl.k14s.io/vsphere-cpi Reconcile succeeded 116s 143d app.kappctrl.k14s.io/vsphere-csi Reconcile succeeded 83s 166d ``` Workload Cluster ``` $ kubectl get package,packageinstall,app -n tkg-system --context lemon-admin@lemon NAME PACKAGEMETADATA NAME VERSION AGE package.data.packaging.carvel.dev/cert-manager.tanzu.vmware.com.1.1.0+vmware.1-tkg.2 cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 42m16s package.data.packaging.carvel.dev/contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/external-dns.tanzu.vmware.com.0.8.0+vmware.1-tkg.1 external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/fluent-bit.tanzu.vmware.com.1.7.5+vmware.1-tkg.1 fluent-bit.tanzu.vmware.com 1.7.5+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1 grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/harbor.tanzu.vmware.com.2.2.3+vmware.1-tkg.1 harbor.tanzu.vmware.com 2.2.3+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/multus-cni.tanzu.vmware.com.3.7.1+vmware.1-tkg.1 multus-cni.tanzu.vmware.com 3.7.1+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1 prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 42m16s package.data.packaging.carvel.dev/addons-manager.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 addons-manager.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/ako-operator.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 ako-operator.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1 antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/calico.tanzu.vmware.com.3.11.3+vmware.1-tkg.1 calico.tanzu.vmware.com 3.11.3+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/kapp-controller.tanzu.vmware.com.0.23.0+vmware.1-tkg.1 kapp-controller.tanzu.vmware.com 0.23.0+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/load-balancer-and-ingress-service.tanzu.vmware.com.1.4.3+vmware.1-tkg.1 load-balancer-and-ingress-service.tanzu.vmware.com 1.4.3+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/metrics-server.tanzu.vmware.com.0.4.0+vmware.1-tkg.1 metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 40m55s package.data.packaging.carvel.dev/pinniped.tanzu.vmware.com.0.4.4+vmware.1-tkg.1 pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/vsphere-cpi.tanzu.vmware.com.1.21.0+vmware.1-tkg.1 vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 40m56s package.data.packaging.carvel.dev/vsphere-csi.tanzu.vmware.com.2.3.0+vmware.1-tkg.2 vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 40m55s NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE packageinstall.packaging.carvel.dev/antrea antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 Reconcile succeeded 45m packageinstall.packaging.carvel.dev/metrics-server metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 Reconcile succeeded 45m packageinstall.packaging.carvel.dev/vsphere-cpi vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 Reconcile succeeded 45m packageinstall.packaging.carvel.dev/vsphere-csi vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 Reconcile succeeded 45m NAME DESCRIPTION SINCE-DEPLOY AGE app.kappctrl.k14s.io/antrea Reconcile succeeded 113s 166d app.kappctrl.k14s.io/metrics-server Reconcile succeeded 72s 166d app.kappctrl.k14s.io/vsphere-cpi Reconcile succeeded 2m10s 158d app.kappctrl.k14s.io/vsphere-csi Reconcile succeeded 86s 166d ``` 全て`Reconcile succeeded`になっていればOKです。 `tanzu` CLIで確認することもできます。 ``` $ tanzu package available list / Retrieving available packages... NAME DISPLAY-NAME SHORT-DESCRIPTION cert-manager.tanzu.vmware.com cert-manager Certificate management contour.tanzu.vmware.com Contour An ingress controller external-dns.tanzu.vmware.com external-dns This package provides DNS synchronization functionality. fluent-bit.tanzu.vmware.com fluent-bit Fluent Bit is a fast Log Processor and Forwarder grafana.tanzu.vmware.com grafana Visualization and analytics software harbor.tanzu.vmware.com Harbor OCI Registry multus-cni.tanzu.vmware.com multus-cni This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes prometheus.tanzu.vmware.com prometheus A time series database for your metrics $ tanzu package installed list -A \ Retrieving installed packages... NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-system ``` ### TKG Extensionのアップデート TKG Extensionを使用している場合はこちらもアップデートが必要です。 Addonと同じくkapp controllerのPackageへの移行が必要です。 ドキュメントを参照してください。 https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-extensions.html #### Cert Manager Packageのインストール Cert Managerのインストール方法をメモしておきます。1.3.1からのアップデートの場合も同じです。Packageのインストールは`tanzu package install`コマンドを使用します。 ``` tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --namespace cert-manager --version 1.1.0+vmware.1-tkg.2 --create-namespace ``` 次のようなログが出力されます。 ``` / Installing package 'cert-manager.tanzu.vmware.com' | Creating namespace 'cert-manager' | Getting package metadata for 'cert-manager.tanzu.vmware.com' | Creating service account 'cert-manager-cert-manager-sa' | Creating cluster admin role 'cert-manager-cert-manager-cluster-role' | Creating cluster role binding 'cert-manager-cert-manager-cluster-rolebinding' / Creating package resource | Package install status: Reconciling Added installed package 'cert-manager' in namespace 'cert-manager' ``` イントールされたパッケージを確認します。 ``` $ kubectl get app,packageinstall -n cert-manager NAME DESCRIPTION SINCE-DEPLOY AGE app.kappctrl.k14s.io/cert-manager Reconcile succeeded 54s 66s NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE packageinstall.packaging.carvel.dev/cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded 66s ``` 次のコマンドでも確認できます。 ``` $ tanzu package installed list -n cert-manager - Retrieving installed packages... NAME PACKAGE-NAME PACKAGE-VERSION STATUS cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded ``` #### Contour Packageのインストール 次にContourのインストール方法をメモしておきます。1.3.1からのアップデートの場合も同じです。 設定項目を`values.yaml`に定義します。例えばMetalLBを使ってEnvoyのService TypeをLoadBalancerに変更したい場合は次のような設定になります。 1.3までとは異なり、この`values.yaml`に`#@`から始まる行を書かないでください。含めるとデプロイがエラーになります。 ```yaml --- infrastructure_provider: "vsphere" envoy: service: type: LoadBalancer ``` 1.3からアップデートする場合は、次のコマンドで`values.yaml`を出力してください。 ``` kubectl get secret -n tanzu-system-ingress contour-data-values -otemplate='{{index .data "values.yaml" | base64decode}}' > values.yaml ``` この`values.yaml`から`#@`から始まる行を削除してください。 `values.yaml`が準備できたら次のコマンドでインストールします。 ``` tanzu package install contour --package-name contour.tanzu.vmware.com --namespace tanzu-system-ingress --version 1.17.1+vmware.1-tkg.1 -f values.yml --create-namespace ``` > `values.yaml`を適用した結果どのようなManifestがデプロイされるのか知りたい場合は次のようにして確認できます。 > > まずはImage PackageのBundleイメージ名を取得します。 > ``` > $ kubectl get package contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 -otemplate='{{(index .spec.template.spec.fetch 0).imgpkgBundle.image}}' > projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4 > ``` > > このイメージを`imgpkg`コマンドでダウンロードします。 > > ``` > imgpkg pull -b projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4 -o /tmp/contour > ``` > > マニフェストが展開されます。 > > ``` > $ find /tmp/contour/config > > /tmp/contour/config > /tmp/contour/config/contour.star > /tmp/contour/config/_ytt_lib > /tmp/contour/config/_ytt_lib/bundle > /tmp/contour/config/_ytt_lib/bundle/config > /tmp/contour/config/_ytt_lib/bundle/config/overlays > /tmp/contour/config/_ytt_lib/bundle/config/overlays/change-namespace.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/remove-certgen-job.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-contour-configmap.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-crds.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-envoy-daemonset.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/add-certmanager-certs.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-envoy-service.yaml > /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-contour-deployment.yaml > /tmp/contour/config/_ytt_lib/bundle/config/certificates.lib.yaml > /tmp/contour/config/_ytt_lib/bundle/config/values.yaml > /tmp/contour/config/_ytt_lib/bundle/config/upstream > /tmp/contour/config/_ytt_lib/bundle/config/upstream/contour.yaml > /tmp/contour/config/contour.yaml > /tmp/contour/config/overlays > /tmp/contour/config/overlays/update-role-envoy-psp.yaml > /tmp/contour/config/overlays/update-role-contour-psp.yaml > /tmp/contour/config/overlays/add-envoy-rbac.yaml > /tmp/contour/config/values.yaml > ``` > > このディレクトリと作成した`values.yaml`を`ytt`に食わせることでこのPackageでデプロイされるYAMLを確認できます。 > > ``` > ytt -f /tmp/contour/config --data-values-file values.yaml > ``` #### Packageに対してOverlayを使用したい場合 TKG ExtensionをyttのOverlayでカスタマイズしているかもしてません。 `tanzu package install`コマンドではoverlayを設定できません。代わりにPackageInstallリソースに対してアノテーションを設定する必要があります。 次のドキュメントを参照してください。 * https://carvel.dev/kapp-controller/docs/latest/package-install-exentensions/#adding-paths-to-ytt-overlays * https://github.com/vmware-tanzu/carvel-kapp-controller/issues/123#issuecomment-848084192 --- TKG 1.3.1-patch1 -> 1.4.0へのアップデートを行いました。 設定ファイルの場所が`~/.tanzu`から`~/.config/tanzu`に変わったこと以外は、 TKG 1.2 -> 1.3に比べれば簡単で特にハマりポイントはありませんでした。 TKG Extensionのアップデートは注意が必要です。