"Tanzu Kubernetes Grid on vSphereを1.3.0 -> 1.3.1-patch1にアップデートするメモ"で作成した環境をアップデートします。
アップデートの手順はこちら
- https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-index.html
- https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-management-cluster.html
- https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-workload-clusters.html
Release Noteはこちら
Table of contents
- CLIのインストール
- vCenterにOVAファイルをアップロード
- Management Clusterをアップデート
- Workload Clusterをアップデート
- Packageの確認
- TKG Extensionのアップデート
CLIのインストール
から
- VMware Tanzu CLI for Mac
tar xvf tanzu-cli-bundle-darwin-amd64.tar
install cli/core/v1.4.0/tanzu-core-darwin_amd64 /usr/local/bin/tanzu
次のバージョンで動作確認します。
$ tanzu version
version: v1.3.1
buildDate: 2021-05-07
sha: e5c37c4
初回コマンド実行時に、
! Configuration is now stored in /Users/toshiaki/.config/tanzu. Legacy configuration directory /Users/toshiaki/.tanzu is deprecated and will be removed in a future release.
! To complete migration, please remove legacy configuration directory /Users/toshiaki/.tanzu and adjust your script(s), if any, to point to the new location.
というようなログが出力されます。TKG 1.3.1の時に使用していた$HOME/.tanzu/tkg/providersはバックアップフォルダにrenameされ、 1.4.0用のファイルが$HOME/.config/tanzu/tkg/providersに用意されます。
この時、$HOME/.tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yamlのようなクラスタ構築時にカスタマイズしたytt overlayファイルは1.3.1に引き継がれないので、
1.3.1->1.4.0においても引き続き利用したいOverlayファイルは改めて$HOME/.config/tanzu/tkg/providersへ設定する必要があります。
cp -r $HOME/.config/tanzu/tkg/providers-20210910004336-y622uivb/infrastructure-vsphere/ytt/vsphere-overlay.yaml $HOME/.config/tanzu/tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml
tanzu plugin clean
tanzu plugin install --local cli all
プラグイン一覧を確認します。
$ tanzu plugin list
NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS
alpha v1.3.1 Alpha CLI commands core not installed
cluster v1.3.1 Kubernetes cluster operations core v1.4.0 installed
kubernetes-release v1.3.1 Kubernetes release operations core v1.4.0 installed
login v1.3.1 Login to the platform core v1.4.0 installed
management-cluster v1.3.1 Kubernetes management cluster operations core v1.4.0 installed
package Tanzu package management v1.4.0 installed
pinniped-auth v1.3.1 Pinniped authentication operations (usually not directly invoked) core v1.4.0 installed
旧バージョンからアップデートした場合は、プラグインのLATEST VERSIONが1.3.1になるようです。 プラグインを配布するbucket名が変わったようです。
~/.config/tanzu/config.yamlのgcpPluginRepositoryのcoreのbucketNameをtanzu-cli -> tanzu-cli-tkgに変更してください。
apiVersion: config.tanzu.vmware.com/v1alpha1
clientOptions:
cli:
repositories:
- gcpPluginRepository:
bucketName: tanzu-cli-tkg
name: core
これで正しく表示されます。
$ tanzu plugin list
NAME LATEST VERSION DESCRIPTION REPOSITORY VERSION STATUS
alpha v1.4.0 Alpha CLI commands core not installed
cluster v1.4.0 Kubernetes cluster operations core v1.4.0 installed
kubernetes-release v1.4.0 Kubernetes release operations core v1.4.0 installed
login v1.4.0 Login to the platform core v1.4.0 installed
management-cluster v1.4.0 Kubernetes management cluster operations core v1.4.0 installed
package v1.4.0 Tanzu package management core v1.4.0 installed
pinniped-auth v1.4.0 Pinniped authentication operations (usually not directly invoked) core v1.4.0 installed
vCenterにOVAファイルをアップロード
https://my.vmware.com/en/web/vmware/downloads/details?downloadGroup=TKG-131&productId=988&rPId=65946
から
- Photon v3 Kubernetes v1.21.2 OVA
をダウンロードし、~/Downloadsに保存します。
TKGをインストールする環境の情報を次の例のように設定してください。
cat <<EOF > tkg-env.sh
export GOVC_URL=vcsa.v.maki.lol
export GOVC_USERNAME=administrator@vsphere.local
export GOVC_PASSWORD=VMware1!
export GOVC_DATACENTER=Datacenter
export GOVC_NETWORK="VM Network"
export GOVC_DATASTORE=datastore1
export GOVC_RESOURCE_POOL=/Datacenter/host/Cluster/Resources/tkg
export GOVC_INSECURE=1
export TEMPLATE_FOLDER=/Datacenter/vm/tkg
EOF
次のコマンドでOVAファイルをvCenterにアップロードします。
source tkg-env.sh
govc import.ova -folder $TEMPLATE_FOLDER ~/Downloads/photon-3-kube-v1.21.2+vmware.1-tkg.2-12816990095845873721.ova
govc vm.markastemplate $TEMPLATE_FOLDER/photon-3-kube-v1.21.2
Management Clusterをアップデート
次のコマンドを実行し、アップデート対象のManagement Clusterにログインします。
tanzu login
アップデート前のクラスタ一覧を確認します。Kubernetesのバージョンはv1.20.4+vmware.1です。
$ tanzu cluster list --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
apple default running 1/1 2/2 v1.20.5+vmware.2 <none> dev
grape default running 1/1 2/2 v1.20.4+vmware.1 <none> prod
lemon default running 1/1 1/1 v1.20.5+vmware.2 <none> dev
orange default running 1/1 2/2 v1.20.5+vmware.2 <none> dev
carrot tkg-system running 1/1 1/1 v1.20.5+vmware.2 management dev
次のコマンドでManagement Clusterをアップデートします。
tanzu management-cluster upgrade -v9 -y
次のようなログが出力されます。
compatibility file (/Users/toshiaki/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /Users/toshiaki/.config/tanzu/tkg/bom already exists, skipping download
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading management cluster providers...
clusterctl upgrade apply options: {Kubeconfig:{Path:/Users/toshiaki/.kube-tkg/config Context:carrot-admin@carrot} ManagementGroup:capi-system/cluster-api Contract: CoreProvider:capi-system/cluster-api:v0.3.23 BootstrapProviders:[capi-kubeadm-bootstrap-system/kubeadm:v0.3.23] ControlPlaneProviders:[capi-kubeadm-control-plane-system/kubeadm:v0.3.23] InfrastructureProviders:[capv-system/vsphere:v0.7.10]}
Checking cert-manager version...
Deleting cert-manager Version="v0.16.1"
Installing cert-manager Version="v1.1.0"
Waiting for cert-manager to be available...
Performing upgrade...
Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system"
Installing Provider="cluster-api" Version="v0.3.23" TargetNamespace="capi-system"
Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-bootstrap-system"
Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.23" TargetNamespace="capi-kubeadm-control-plane-system"
Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.10" TargetNamespace="capv-system"
Waiting for provider infrastructure-vsphere
Waiting for provider bootstrap-kubeadm
Waiting for provider cluster-api
Waiting for provider control-plane-kubeadm
Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-kubeadm-bootstrap-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-controller-manager of type *v1.Deployment to be up and running
Waiting for resource capi-kubeadm-bootstrap-controller-manager of type *v1.Deployment to be up and running
pods are not yet running for deployment 'capi-kubeadm-control-plane-controller-manager' in namespace 'capi-kubeadm-control-plane-system', retrying
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
Passed waiting on provider cluster-api after 341.338078ms
Passed waiting on provider bootstrap-kubeadm after 340.817975ms
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
Waiting for resource capi-kubeadm-control-plane-controller-manager of type *v1.Deployment to be up and running
Passed waiting on provider control-plane-kubeadm after 5.455206185s
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
pods are not yet running for deployment 'capv-controller-manager' in namespace 'capv-system', retrying
Waiting for resource capv-controller-manager of type *v1.Deployment to be up and running
Passed waiting on provider infrastructure-vsphere after 20.246601997s
Success waiting on all providers.
Management cluster providers upgraded successfully...
Upgrading management cluster kubernetes version...
Creating management cluster client...
Verifying kubernetes version...
Unable to detect current OS for the cluster. Using name:photon version:3 arch:amd64
Using OS options, name:photon version:3 arch:amd64
Retrieving configuration for upgrade cluster...
unable to find azure-image config in user configuration file, using it from BoM files
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Applying KubeadmControlPlane Patch: {
"spec": {
"version": "v1.21.2+vmware.1",
"infrastructureTemplate": {
"name": "carrot-control-plane-v1-21-2-vmware-1-dl61v",
"namespace": "tkg-system"
},
"kubeadmConfigSpec": {
"clusterConfiguration": {
"imageRepository" : "projects.registry.vmware.com/tkg",
"dns": {
"imageRepository": "projects.registry.vmware.com/tkg",
"imageTag": "v1.8.0_vmware.5"
},
"etcd": {
"local": {
"imageRepository": "projects.registry.vmware.com/tkg",
"imageTag": "v3.4.13_vmware.15"
}
}
}
}
}
}
Applying patch to resource carrot-control-plane of type *v1alpha3.KubeadmControlPlane ...
patch cluster object with operation status:
{
"metadata": {
"annotations": {
"TKGOperationInfo" : "{\"Operation\":\"Upgrade\",\"OperationStartTimestamp\":\"2021-09-09 16:10:45.498847 +0000 UTC\",\"OperationTimeout\":900}",
"TKGOperationLastObservedTimestamp" : "2021-09-09 16:10:45.498847 +0000 UTC"
}
}
}
Waiting for kubernetes version to be updated for control plane nodes
...
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Applying MachineDeployment Patch: {
"spec": {
"template": {
"spec": {
"version": "v1.21.2+vmware.1",
"infrastructureRef": {
"name": "carrot-worker-v1-19-3-vmware-1-muxc1-v1-20-4-vmware-1-5mlj6-v1-20-5-vmware-2-4xqok-v1-21-2-vmware-1-fgltw",
"namespace": "tkg-system"
}
}
}
}
}
Applying patch to resource carrot-md-0 of type *v1alpha3.MachineDeployment ...
Waiting for kubernetes version to be updated for worker nodes...
...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
updating additional components: 'addons-management/tanzu-addons-manager' ...
updating additional components: 'tkr/tkr-controller' ...
updating additional components: 'addons-management/core-package-repo' ...
...
Waiting for packages to be up and running...
Waiting for package: antrea
Waiting for package: metrics-server
Waiting for package: pinniped
Waiting for package: tanzu-addons-manager
Waiting for package: vsphere-cpi
Waiting for package: vsphere-csi
Waiting for resource pinniped of type *v1alpha1.PackageInstall to be up and running
Waiting for resource vsphere-csi of type *v1alpha1.PackageInstall to be up and running
Waiting for resource antrea of type *v1alpha1.PackageInstall to be up and running
Waiting for resource tanzu-addons-manager of type *v1alpha1.PackageInstall to be up and running
Waiting for resource vsphere-cpi of type *v1alpha1.PackageInstall to be up and running
Waiting for resource metrics-server of type *v1alpha1.PackageInstall to be up and running
...
waiting for 'pinniped' Package to be installed, retrying
Successfully reconciled package: pinniped
Management cluster 'carrot' successfully upgraded to TKG version 'v1.4.0' with kubernetes version 'v1.21.2+vmware.1'
次のコマンドでManagement ClusterのみKubernetes v1.20.5+vmware.2にバージョンアップされたことがわかります。
$ tanzu cluster list --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
apple default running 1/1 2/2 v1.20.5+vmware.2 <none> dev
grape default running 1/1 2/2 v1.20.4+vmware.1 <none> prod
lemon default running 1/1 1/1 v1.20.5+vmware.2 <none> dev
orange default running 1/1 2/2 v1.20.5+vmware.2 <none> dev
carrot tkg-system running 1/1 1/1 v1.21.2+vmware.1 management dev
Management Clusterにkubectlコマンドでアクセスしてみます。
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
carrot-control-plane-cw9v5 Ready control-plane,master 50m v1.21.2+vmware.1 192.168.11.98 192.168.11.98 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6
carrot-md-0-697b79849d-mghbl Ready <none> 27m v1.21.2+vmware.1 192.168.11.38 192.168.11.38 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6
アップデート後はControlPlaneのDHCP予約の設定を再度行う必要があります。
Workload Clusterをアップデート
次にWorkload Clusterをアップデートします。
次のコマンドでWorkload Clusterを作成します。
tanzu cluster upgrade lemon -v9 -y
次のようなログが出力されます。
compatibility file (/Users/toshiaki/.config/tanzu/tkg/compatibility/tkg-compatibility.yaml) already exists, skipping download
BOM files inside /Users/toshiaki/.config/tanzu/tkg/bom already exists, skipping download
Creating management cluster client...
Validating configuration...
Creating workload cluster client...
Retrieving credentials for workload cluster lemon
Getting secret for cluster
Waiting for resource lemon-kubeconfig of type *v1.Secret to be up and running
Merging credentials into kubeconfig file /Users/toshiaki/.kube-tkg/config
Verifying kubernetes version...
Unable to detect current OS for the cluster. Using name:photon version:3 arch:amd64
Using OS options, name:photon version:3 arch:amd64
Retrieving configuration for upgrade cluster...
unable to find azure-image config in user configuration file, using it from BoM files
Create InfrastructureTemplate for upgrade...
cluster specific secret is not present, fallback on bootstrap credential secret
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.21.2+vmware.1...
Applying KubeadmControlPlane Patch: {
"spec": {
"version": "v1.21.2+vmware.1",
"infrastructureTemplate": {
"name": "lemon-control-plane-v1-21-2-vmware-1-7tsq5",
"namespace": "default"
},
"kubeadmConfigSpec": {
"clusterConfiguration": {
"imageRepository" : "projects.registry.vmware.com/tkg",
"dns": {
"imageRepository": "projects.registry.vmware.com/tkg",
"imageTag": "v1.8.0_vmware.5"
},
"etcd": {
"local": {
"imageRepository": "projects.registry.vmware.com/tkg",
"imageTag": "v3.4.13_vmware.15"
}
}
}
}
}
}
Applying patch to resource lemon-control-plane of type *v1alpha3.KubeadmControlPlane ...
patch cluster object with operation status:
{
"metadata": {
"annotations": {
"TKGOperationInfo" : "{\"Operation\":\"Upgrade\",\"OperationStartTimestamp\":\"2021-09-10 01:55:37.431052 +0000 UTC\",\"OperationTimeout\":900}",
"TKGOperationLastObservedTimestamp" : "2021-09-10 01:55:37.431052 +0000 UTC"
}
}
}
Waiting for kubernetes version to be updated for control plane nodes
...
control-plane is still being upgraded, reason:'ScalingDown', message:'Scaling down control plane to 1 replicas (actual 2)' , retrying
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.21.2+vmware.1...
Applying MachineDeployment Patch: {
"spec": {
"template": {
"spec": {
"version": "v1.21.2+vmware.1",
"infrastructureRef": {
"name": "lemon-worker-8g-v1-21-2-vmware-1-ecyq4",
"namespace": "default"
}
}
}
}
}
Applying patch to resource lemon-md-0 of type *v1alpha3.MachineDeployment ...
Waiting for kubernetes version to be updated for worker nodes...
...
updating additional components: 'metadata/tkg' ...
updating additional components: 'addons-management/kapp-controller' ...
updating additional components: 'addons-management/standard-package-repo' ...
cluster autoscaler is not enabled for cluster lemon
Waiting for packages to be up and running...
Waiting for package: antrea
Waiting for package: metrics-server
Waiting for package: vsphere-cpi
Waiting for package: vsphere-csi
...
Successfully reconciled package: antrea
waiting for 'vsphere-csi' Package to be installed, retrying
Successfully reconciled package: vsphere-csi
Cluster 'lemon' successfully upgraded to kubernetes version 'v1.21.2+vmware.1'
tanzu cluster listでWorkload Clusterを確認できます。lemonのみKubernetes 1.21.2にバージョンアップされたことがわかります
$ tanzu cluster list --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES PLAN
apple default running 1/1 2/2 v1.20.5+vmware.2 <none> dev
grape default running 1/1 2/2 v1.20.4+vmware.1 <none> prod
lemon default running 1/1 1/1 v1.21.2+vmware.1 <none> dev
orange default running 1/1 2/2 v1.20.5+vmware.2 <none> dev
carrot tkg-system running 1/1 1/1 v1.21.2+vmware.1 management dev
kubectl config use-context lemon-admin@lemon
$ kubectl get node -owide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
lemon-control-plane-jd4gc Ready control-plane,master 65m v1.21.2+vmware.1 192.168.11.39 192.168.11.39 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6
lemon-md-0-7fb4f9f5dc-wswp6 Ready <none> 42m v1.21.2+vmware.1 192.168.11.61 192.168.11.61 VMware Photon OS/Linux 4.19.198-1.ph3 containerd://1.4.6
アップデート後はControlPlaneのDHCP予約の設定を再度行う必要があります。
ここまででKubernetesのアップデートは完了しました。
Packageの確認
TKG 1.4ではAdd-onやExtensionは新しくCarvelのPackaging APIで管理されるようになりました。
- Add-on -> Core Pakcage
- Extension -> User-Managed Packages
Add-on -> Core Packageは自動でアップデートされます。
Management Cluster
$ kubectl get package,packageinstall,app -n tkg-system --context carrot-admin@carrot
NAME PACKAGEMETADATA NAME VERSION AGE
package.data.packaging.carvel.dev/addons-manager.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 addons-manager.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 10h26m31s
package.data.packaging.carvel.dev/ako-operator.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 ako-operator.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 10h26m31s
package.data.packaging.carvel.dev/antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1 antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 10h26m31s
package.data.packaging.carvel.dev/calico.tanzu.vmware.com.3.11.3+vmware.1-tkg.1 calico.tanzu.vmware.com 3.11.3+vmware.1-tkg.1 10h26m31s
package.data.packaging.carvel.dev/kapp-controller.tanzu.vmware.com.0.23.0+vmware.1-tkg.1 kapp-controller.tanzu.vmware.com 0.23.0+vmware.1-tkg.1 10h26m30s
package.data.packaging.carvel.dev/load-balancer-and-ingress-service.tanzu.vmware.com.1.4.3+vmware.1-tkg.1 load-balancer-and-ingress-service.tanzu.vmware.com 1.4.3+vmware.1-tkg.1 10h26m30s
package.data.packaging.carvel.dev/metrics-server.tanzu.vmware.com.0.4.0+vmware.1-tkg.1 metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 10h26m30s
package.data.packaging.carvel.dev/pinniped.tanzu.vmware.com.0.4.4+vmware.1-tkg.1 pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 10h26m30s
package.data.packaging.carvel.dev/vsphere-cpi.tanzu.vmware.com.1.21.0+vmware.1-tkg.1 vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 10h26m30s
package.data.packaging.carvel.dev/vsphere-csi.tanzu.vmware.com.2.3.0+vmware.1-tkg.2 vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 10h26m30s
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/antrea antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 Reconcile succeeded 10h
packageinstall.packaging.carvel.dev/metrics-server metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 Reconcile succeeded 10h
packageinstall.packaging.carvel.dev/pinniped pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 Reconcile succeeded 10h
packageinstall.packaging.carvel.dev/tanzu-addons-manager addons-manager.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 Reconcile succeeded 10h
packageinstall.packaging.carvel.dev/vsphere-cpi vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 Reconcile succeeded 10h
packageinstall.packaging.carvel.dev/vsphere-csi vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 Reconcile succeeded 10h
NAME DESCRIPTION SINCE-DEPLOY AGE
app.kappctrl.k14s.io/antrea Reconcile succeeded 96s 166d
app.kappctrl.k14s.io/metrics-server Reconcile succeeded 114s 166d
app.kappctrl.k14s.io/pinniped Reconcile succeeded 4m12s 56d
app.kappctrl.k14s.io/tanzu-addons-manager Reconcile succeeded 5m10s 167d
app.kappctrl.k14s.io/vsphere-cpi Reconcile succeeded 116s 143d
app.kappctrl.k14s.io/vsphere-csi Reconcile succeeded 83s 166d
Workload Cluster
$ kubectl get package,packageinstall,app -n tkg-system --context lemon-admin@lemon
NAME PACKAGEMETADATA NAME VERSION AGE
package.data.packaging.carvel.dev/cert-manager.tanzu.vmware.com.1.1.0+vmware.1-tkg.2 cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 42m16s
package.data.packaging.carvel.dev/contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 contour.tanzu.vmware.com 1.17.1+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/external-dns.tanzu.vmware.com.0.8.0+vmware.1-tkg.1 external-dns.tanzu.vmware.com 0.8.0+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/fluent-bit.tanzu.vmware.com.1.7.5+vmware.1-tkg.1 fluent-bit.tanzu.vmware.com 1.7.5+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/grafana.tanzu.vmware.com.7.5.7+vmware.1-tkg.1 grafana.tanzu.vmware.com 7.5.7+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/harbor.tanzu.vmware.com.2.2.3+vmware.1-tkg.1 harbor.tanzu.vmware.com 2.2.3+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/multus-cni.tanzu.vmware.com.3.7.1+vmware.1-tkg.1 multus-cni.tanzu.vmware.com 3.7.1+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/prometheus.tanzu.vmware.com.2.27.0+vmware.1-tkg.1 prometheus.tanzu.vmware.com 2.27.0+vmware.1-tkg.1 42m16s
package.data.packaging.carvel.dev/addons-manager.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 addons-manager.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/ako-operator.tanzu.vmware.com.1.4.0+vmware.1-tkg.1 ako-operator.tanzu.vmware.com 1.4.0+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/antrea.tanzu.vmware.com.0.13.3+vmware.1-tkg.1 antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/calico.tanzu.vmware.com.3.11.3+vmware.1-tkg.1 calico.tanzu.vmware.com 3.11.3+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/kapp-controller.tanzu.vmware.com.0.23.0+vmware.1-tkg.1 kapp-controller.tanzu.vmware.com 0.23.0+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/load-balancer-and-ingress-service.tanzu.vmware.com.1.4.3+vmware.1-tkg.1 load-balancer-and-ingress-service.tanzu.vmware.com 1.4.3+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/metrics-server.tanzu.vmware.com.0.4.0+vmware.1-tkg.1 metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 40m55s
package.data.packaging.carvel.dev/pinniped.tanzu.vmware.com.0.4.4+vmware.1-tkg.1 pinniped.tanzu.vmware.com 0.4.4+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/vsphere-cpi.tanzu.vmware.com.1.21.0+vmware.1-tkg.1 vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 40m56s
package.data.packaging.carvel.dev/vsphere-csi.tanzu.vmware.com.2.3.0+vmware.1-tkg.2 vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 40m55s
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/antrea antrea.tanzu.vmware.com 0.13.3+vmware.1-tkg.1 Reconcile succeeded 45m
packageinstall.packaging.carvel.dev/metrics-server metrics-server.tanzu.vmware.com 0.4.0+vmware.1-tkg.1 Reconcile succeeded 45m
packageinstall.packaging.carvel.dev/vsphere-cpi vsphere-cpi.tanzu.vmware.com 1.21.0+vmware.1-tkg.1 Reconcile succeeded 45m
packageinstall.packaging.carvel.dev/vsphere-csi vsphere-csi.tanzu.vmware.com 2.3.0+vmware.1-tkg.2 Reconcile succeeded 45m
NAME DESCRIPTION SINCE-DEPLOY AGE
app.kappctrl.k14s.io/antrea Reconcile succeeded 113s 166d
app.kappctrl.k14s.io/metrics-server Reconcile succeeded 72s 166d
app.kappctrl.k14s.io/vsphere-cpi Reconcile succeeded 2m10s 158d
app.kappctrl.k14s.io/vsphere-csi Reconcile succeeded 86s 166d
全てReconcile succeededになっていればOKです。
tanzu CLIで確認することもできます。
$ tanzu package available list
/ Retrieving available packages...
NAME DISPLAY-NAME SHORT-DESCRIPTION
cert-manager.tanzu.vmware.com cert-manager Certificate management
contour.tanzu.vmware.com Contour An ingress controller
external-dns.tanzu.vmware.com external-dns This package provides DNS synchronization functionality.
fluent-bit.tanzu.vmware.com fluent-bit Fluent Bit is a fast Log Processor and Forwarder
grafana.tanzu.vmware.com grafana Visualization and analytics software
harbor.tanzu.vmware.com Harbor OCI Registry
multus-cni.tanzu.vmware.com multus-cni This package provides the ability for enabling attaching multiple network interfaces to pods in Kubernetes
prometheus.tanzu.vmware.com prometheus A time series database for your metrics
$ tanzu package installed list -A
\ Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS NAMESPACE
antrea antrea.tanzu.vmware.com Reconcile succeeded tkg-system
metrics-server metrics-server.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-cpi vsphere-cpi.tanzu.vmware.com Reconcile succeeded tkg-system
vsphere-csi vsphere-csi.tanzu.vmware.com Reconcile succeeded tkg-system
TKG Extensionのアップデート
TKG Extensionを使用している場合はこちらもアップデートが必要です。 Addonと同じくkapp controllerのPackageへの移行が必要です。
ドキュメントを参照してください。 https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.4/vmware-tanzu-kubernetes-grid-14/GUID-upgrade-tkg-extensions.html
Cert Manager Packageのインストール
Cert Managerのインストール方法をメモしておきます。1.3.1からのアップデートの場合も同じです。Packageのインストールはtanzu package installコマンドを使用します。
tanzu package install cert-manager --package-name cert-manager.tanzu.vmware.com --namespace cert-manager --version 1.1.0+vmware.1-tkg.2 --create-namespace
次のようなログが出力されます。
/ Installing package 'cert-manager.tanzu.vmware.com'
| Creating namespace 'cert-manager'
| Getting package metadata for 'cert-manager.tanzu.vmware.com'
| Creating service account 'cert-manager-cert-manager-sa'
| Creating cluster admin role 'cert-manager-cert-manager-cluster-role'
| Creating cluster role binding 'cert-manager-cert-manager-cluster-rolebinding'
/ Creating package resource
| Package install status: Reconciling
Added installed package 'cert-manager' in namespace 'cert-manager'
イントールされたパッケージを確認します。
$ kubectl get app,packageinstall -n cert-manager
NAME DESCRIPTION SINCE-DEPLOY AGE
app.kappctrl.k14s.io/cert-manager Reconcile succeeded 54s 66s
NAME PACKAGE NAME PACKAGE VERSION DESCRIPTION AGE
packageinstall.packaging.carvel.dev/cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded 66s
次のコマンドでも確認できます。
$ tanzu package installed list -n cert-manager
- Retrieving installed packages...
NAME PACKAGE-NAME PACKAGE-VERSION STATUS
cert-manager cert-manager.tanzu.vmware.com 1.1.0+vmware.1-tkg.2 Reconcile succeeded
Contour Packageのインストール
次にContourのインストール方法をメモしておきます。1.3.1からのアップデートの場合も同じです。
設定項目をvalues.yamlに定義します。例えばMetalLBを使ってEnvoyのService TypeをLoadBalancerに変更したい場合は次のような設定になります。
1.3までとは異なり、このvalues.yamlに#@から始まる行を書かないでください。含めるとデプロイがエラーになります。
---
infrastructure_provider: "vsphere"
envoy:
service:
type: LoadBalancer
1.3からアップデートする場合は、次のコマンドでvalues.yamlを出力してください。
kubectl get secret -n tanzu-system-ingress contour-data-values -otemplate='{{index .data "values.yaml" | base64decode}}' > values.yaml
このvalues.yamlから#@から始まる行を削除してください。
values.yamlが準備できたら次のコマンドでインストールします。
tanzu package install contour --package-name contour.tanzu.vmware.com --namespace tanzu-system-ingress --version 1.17.1+vmware.1-tkg.1 -f values.yml --create-namespace
values.yamlを適用した結果どのようなManifestがデプロイされるのか知りたい場合は次のようにして確認できます。まずはImage PackageのBundleイメージ名を取得します。
$ kubectl get package contour.tanzu.vmware.com.1.17.1+vmware.1-tkg.1 -otemplate='{{(index .spec.template.spec.fetch 0).imgpkgBundle.image}}' projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4このイメージを
imgpkgコマンドでダウンロードします。imgpkg pull -b projects.registry.vmware.com/tkg/packages/standard/contour@sha256:73dc13131e6c1cfa8d3b56aeacd97734447acdf1ab8c0862e936623ca744e7c4 -o /tmp/contourマニフェストが展開されます。
$ find /tmp/contour/config /tmp/contour/config /tmp/contour/config/contour.star /tmp/contour/config/_ytt_lib /tmp/contour/config/_ytt_lib/bundle /tmp/contour/config/_ytt_lib/bundle/config /tmp/contour/config/_ytt_lib/bundle/config/overlays /tmp/contour/config/_ytt_lib/bundle/config/overlays/change-namespace.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/remove-certgen-job.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-contour-configmap.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-crds.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-envoy-daemonset.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/add-certmanager-certs.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-envoy-service.yaml /tmp/contour/config/_ytt_lib/bundle/config/overlays/update-contour-deployment.yaml /tmp/contour/config/_ytt_lib/bundle/config/certificates.lib.yaml /tmp/contour/config/_ytt_lib/bundle/config/values.yaml /tmp/contour/config/_ytt_lib/bundle/config/upstream /tmp/contour/config/_ytt_lib/bundle/config/upstream/contour.yaml /tmp/contour/config/contour.yaml /tmp/contour/config/overlays /tmp/contour/config/overlays/update-role-envoy-psp.yaml /tmp/contour/config/overlays/update-role-contour-psp.yaml /tmp/contour/config/overlays/add-envoy-rbac.yaml /tmp/contour/config/values.yamlこのディレクトリと作成した
values.yamlをyttに食わせることでこのPackageでデプロイされるYAMLを確認できます。ytt -f /tmp/contour/config --data-values-file values.yaml
Packageに対してOverlayを使用したい場合
TKG ExtensionをyttのOverlayでカスタマイズしているかもしてません。
tanzu package installコマンドではoverlayを設定できません。代わりにPackageInstallリソースに対してアノテーションを設定する必要があります。
次のドキュメントを参照してください。
- https://carvel.dev/kapp-controller/docs/latest/package-install-exentensions/#adding-paths-to-ytt-overlays
- https://github.com/vmware-tanzu/carvel-kapp-controller/issues/123#issuecomment-848084192
TKG 1.3.1-patch1 -> 1.4.0へのアップデートを行いました。
設定ファイルの場所が~/.tanzuから~/.config/tanzuに変わったこと以外は、
TKG 1.2 -> 1.3に比べれば簡単で特にハマりポイントはありませんでした。
TKG Extensionのアップデートは注意が必要です。