"Tanzu Kubernetes Grid 1.2をvSphere 7にインストールするメモ"で作成した環境をアップデートします。
現在TKG 1.2.0で次のようなクラスタを管理している状態で1.2.1にバージョンアップします。
$ tkg get clusters --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
grape default running 1/1 3/3 v1.19.1+vmware.2 <none>
lemon default running 1/1 3/3 v1.19.1+vmware.2 <none>
carrot tkg-system running 1/1 1/1 v1.19.1+vmware.2 management
念のため、~/.tkgディレクトリをバックアップしておきます。
cp -r ~/.tkg ~/.tkg-20201212
アップデートの手順はこちら https://docs.vmware.com/en/VMware-Tanzu-Kubernetes-Grid/1.2/vmware-tanzu-kubernetes-grid-12/GUID-upgrade-tkg-index.html
Table of contents
CLIのインストール
https://my.vmware.com/group/vmware/downloads/details?downloadGroup=TKG-121&productId=988&rPId=56777
から
- VMware Tanzu Kubernetes Grid CLI for Mac
をダウンロードして次のようにインストールしてください。
tar xzvf tkg-darwin-amd64-v1.2.1-vmware.1.tar.gz
mv tkg/tkg-darwin-amd64-v1.2.1+vmware.1 /usr/local/bin/tkg
chmod +x /usr/local/bin/tkg
次のバージョンで動作確認しています。
$ tkg version
Client:
Version: v1.2.1
Git commit: 9d15a485f2ccc462622f8df6a81e5fa831c51895
次のコマンドを実行すると1.2.1用の設定ファイルが追加されます。
$ tkg get management-cluster
It seems that the TKG settings on this system are out-of-date. Proceeding on this command will cause them to be backed up and overwritten by the latest settings.
Do you want to continue? [y/N]: y
the old providers folder /Users/toshiaki/.tkg/providers is backed up to /Users/toshiaki/.tkg/providers-20201212002337-ml4ooek8
The old bom folder /Users/toshiaki/.tkg/bom is backed up to /Users/toshiaki/.tkg/bom-20201212002337-j81lmhje
MANAGEMENT-CLUSTER-NAME CONTEXT-NAME STATUS
carrot * carrot-admin@carrot Success
次のようなフォルダ構成になります。
$ find ~/.tkg | grep -v 20201212002337
/Users/toshiaki/.tkg
/Users/toshiaki/.tkg/bom
/Users/toshiaki/.tkg/bom/bom-1.1.2+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.18.8+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.18.10+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-tkg-1.0.0.yaml
/Users/toshiaki/.tkg/bom/bom-1.2.0+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.17.13+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.17.6+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.2.1+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.17.11+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.1.0+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.17.9+vmware.1.yaml
/Users/toshiaki/.tkg/bom/bom-1.1.3+vmware.1.yaml
/Users/toshiaki/.tkg/features.json
/Users/toshiaki/.tkg/providers
/Users/toshiaki/.tkg/providers/infrastructure-azure
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/infrastructure-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/ytt
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-azure/v0.4.8/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-azure/ytt
/Users/toshiaki/.tkg/providers/infrastructure-azure/ytt/azure-overlay.yaml
/Users/toshiaki/.tkg/providers/control-plane-kubeadm
/Users/toshiaki/.tkg/providers/control-plane-kubeadm/v0.3.11
/Users/toshiaki/.tkg/providers/control-plane-kubeadm/v0.3.11/control-plane-components.yaml
/Users/toshiaki/.tkg/providers/providers.md5sum
/Users/toshiaki/.tkg/providers/infrastructure-vsphere
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/infrastructure-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/add_csi.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/csi-vsphere.lib.txt
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/csi.lib.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/v0.7.1/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/ytt
/Users/toshiaki/.tkg/providers/infrastructure-vsphere/ytt/vsphere-overlay.yaml
/Users/toshiaki/.tkg/providers/cluster-api
/Users/toshiaki/.tkg/providers/cluster-api/v0.3.11
/Users/toshiaki/.tkg/providers/cluster-api/v0.3.11/core-components.yaml
/Users/toshiaki/.tkg/providers/config.yaml
/Users/toshiaki/.tkg/providers/bootstrap-kubeadm
/Users/toshiaki/.tkg/providers/bootstrap-kubeadm/v0.3.11
/Users/toshiaki/.tkg/providers/bootstrap-kubeadm/v0.3.11/bootstrap-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/infrastructure-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/ytt
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.5.5/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/infrastructure-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/metadata.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/ytt
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.3/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/infrastructure-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/metadata.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/ytt
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/v0.6.1/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-aws/ytt
/Users/toshiaki/.tkg/providers/infrastructure-aws/ytt/aws-overlay.yaml
/Users/toshiaki/.tkg/providers/config_default.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/infrastructure-components.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/metadata.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/ytt
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker/v0.3.10/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-docker/ytt
/Users/toshiaki/.tkg/providers/infrastructure-docker/ytt/docker-overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/cluster-template-definition-prod.yaml
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/cluster-template-definition-dev.yaml
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/ytt
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/ytt/overlay.yaml
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/v1.0.0/ytt/base-template.yaml
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/ytt
/Users/toshiaki/.tkg/providers/infrastructure-tkg-service-vsphere/ytt/tkg-service-vsphere-overlay.yaml
/Users/toshiaki/.tkg/providers/ytt
/Users/toshiaki/.tkg/providers/ytt/02_addons
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/antrea
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/antrea/antrea.lib.yaml
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/antrea/antrea_overlay.lib.yaml
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/add_cni.yaml
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/calico
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/calico/calico_overlay.lib.yaml
/Users/toshiaki/.tkg/providers/ytt/02_addons/cni/calico/calico.lib.yaml
/Users/toshiaki/.tkg/providers/ytt/02_addons/metadata
/Users/toshiaki/.tkg/providers/ytt/02_addons/metadata/add_cluster_metadata.yaml
/Users/toshiaki/.tkg/providers/ytt/01_plans
/Users/toshiaki/.tkg/providers/ytt/01_plans/dev.yaml
/Users/toshiaki/.tkg/providers/ytt/01_plans/prod.yaml
/Users/toshiaki/.tkg/providers/ytt/01_plans/oidc.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations
/Users/toshiaki/.tkg/providers/ytt/03_customizations/registry_ca_cert.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations/filter.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations/add_mhc.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations/add_sc.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations/registry_skip_tls_verify.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations/autoscaler
/Users/toshiaki/.tkg/providers/ytt/03_customizations/autoscaler/enable_autoscaler.yaml
/Users/toshiaki/.tkg/providers/ytt/03_customizations/autoscaler/autoscaler_overlay.yaml
/Users/toshiaki/.tkg/providers/ytt/lib
/Users/toshiaki/.tkg/providers/ytt/lib/validate.star
/Users/toshiaki/.tkg/providers/ytt/lib/helpers.star
/Users/toshiaki/.tkg/.providers.zip.lock
/Users/toshiaki/.tkg/config.yaml
/Users/toshiaki/.tkg/.boms.lock
サポートされるKubernetesのバージョンは次の通りです。
$ tkg get kubernetesversions
VERSIONS
v1.17.11+vmware.1
v1.17.13+vmware.1
v1.17.3+vmware.2
v1.17.6+vmware.1
v1.17.9+vmware.1
v1.18.10+vmware.1
v1.18.2+vmware.1
v1.18.3+vmware.1
v1.18.6+vmware.1
v1.18.8+vmware.1
v1.19.1+vmware.2
v1.19.3+vmware.1
vCenterにOVAファイルをアップロード
から
- Photon v3 Kubernetes v1.19.3 OVA
をダウンロードし、~/Downloadsに保存します。1.18.10を使いたい場合も以下同様です。
TKGをインストールする環境の情報を次の例のように設定してください。インストール先のネットワークはDHCPが有効になっている必要があります。
cat <<EOF > tkg-env.sh
export GOVC_URL=vcsa.v.maki.lol
export GOVC_USERNAME=administrator@vsphere.local
export GOVC_PASSWORD=VMware1!
export GOVC_DATACENTER=Datacenter
export GOVC_NETWORK="VM Network"
export GOVC_DATASTORE=datastore1
export GOVC_RESOURCE_POOL=/Datacenter/host/Cluster/Resources/tkg
export GOVC_INSECURE=1
export TEMPLATE_FOLDER=/Datacenter/vm/tkg
EOF
次のコマンドでOVAファイルをvCenterにアップロードします。
source tkg-env.sh
govc import.ova -folder $TEMPLATE_FOLDER ~/Downloads/photon-3-kube-v1.19.3-vmware.1.ova
govc vm.markastemplate $TEMPLATE_FOLDER/photon-3-kube-v1.19.3
Management Clusterをアップデート
次のコマンドでManagement Clusterをアップデートします。
tkg upgrade management-cluster carrot
次のようなログが出力されます。
Logs of the command execution can also be found at: /var/folders/76/vg4pyy253pbgwzncmx2mb1gh0000gq/T/tkg-20201212T004546901230328.log
Upgrading management cluster providers...
Checking cert-manager version...
Cert-manager is already up to date
Performing upgrade...
Deleting Provider="cluster-api" Version="" TargetNamespace="capi-system"
Installing Provider="cluster-api" Version="v0.3.11" TargetNamespace="capi-system"
Deleting Provider="bootstrap-kubeadm" Version="" TargetNamespace="capi-kubeadm-bootstrap-system"
Installing Provider="bootstrap-kubeadm" Version="v0.3.11" TargetNamespace="capi-kubeadm-bootstrap-system"
Deleting Provider="control-plane-kubeadm" Version="" TargetNamespace="capi-kubeadm-control-plane-system"
Installing Provider="control-plane-kubeadm" Version="v0.3.11" TargetNamespace="capi-kubeadm-control-plane-system"
Deleting Provider="infrastructure-vsphere" Version="" TargetNamespace="capv-system"
Installing Provider="infrastructure-vsphere" Version="v0.7.1" TargetNamespace="capv-system"
Management cluster providers upgraded successfully...
Upgrading management cluster kubernetes version...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
consuming Azure VM image information from BOM
Create InfrastructureTemplate for upgrade...
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.19.3+vmware.1...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.19.3+vmware.1...
Waiting for kubernetes version to be updated for worker nodes...
updating 'metadata/tkg' add-on...
Management cluster 'carrot' successfully upgraded to TKG version 'v1.2.1' with kubernetes version 'v1.19.3+vmware.1'
...
次のコマンドでManagment ClusterのみKubernetes 1.19.3にバージョンアップされたことがわかります。
$ tkg get cluster --include-management-cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
grape default running 1/1 3/3 v1.19.1+vmware.2 <none>
lemon default running 1/1 3/3 v1.19.1+vmware.2 <none>
carrot tkg-system running 1/1 1/1 v1.19.3+vmware.1 management
Management Clusterにkubectlコマンドでアクセスしてみます。
$ kubectl cluster-info
Kubernetes master is running at https://192.168.11.110:6443
KubeDNS is running at https://192.168.11.110:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
carrot-control-plane-9bbxq Ready master 11m v1.19.3+vmware.1 192.168.11.97 192.168.11.97 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1
carrot-md-0-647c46f976-9cnsb Ready <none> 5m41s v1.19.3+vmware.1 192.168.11.98 192.168.11.98 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1
Workload Clusterをアップデート
対象のManagement Clusterをtkg set management-clusterで指定します。1つしかなければ不要です。
$ tkg set management-cluster carrot
The current management cluster context is switched to carrot
対象となっているManagement Clusterをtkg set management-clusterで確認できます。*がついているクラスタが対象です。
tkg update clusterでWorkload Clusterを作成します。
tkg upgrade cluster grape --yes
次のようなログが出力されます。
Logs of the command execution can also be found at: /var/folders/76/vg4pyy253pbgwzncmx2mb1gh0000gq/T/tkg-20201212T010537187656254.log
Validating configuration...
Verifying kubernetes version...
Retrieving configuration for upgrade cluster...
consuming Azure VM image information from BOM
Create InfrastructureTemplate for upgrade...
Upgrading control plane nodes...
Patching KubeadmControlPlane with the kubernetes version v1.19.3+vmware.1...
Waiting for kubernetes version to be updated for control plane nodes
Upgrading worker nodes...
Patching MachineDeployment with the kubernetes version v1.19.3+vmware.1...
Waiting for kubernetes version to be updated for worker nodes...
updating 'metadata/tkg' add-on...
Cluster 'grape' successfully upgraded to kubernetes version 'v1.19.3+vmware.1'
tkg get clusterでWorkload Clusterを確認できます。grapeのみKubernetes 1.19.3にバージョンアップされたことがわかります
$ tkg get cluster
NAME NAMESPACE STATUS CONTROLPLANE WORKERS KUBERNETES ROLES
grape default running 1/1 3/3 v1.19.3+vmware.1 <none>
lemon default running 1/1 3/3 v1.19.1+vmware.2 <none>
Workload Cluster grapeを確認します。
$ kubectl config use-context grape-admin@grape
$ kubectl cluster-info
Kubernetes master is running at https://192.168.11.113:6443
KubeDNS is running at https://192.168.11.113:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
$ kubectl get node -o wide
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
grape-control-plane-st8sh Ready master 11m v1.19.3+vmware.1 192.168.11.30 192.168.11.30 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1
grape-md-0-7486cf9b49-ft96f Ready <none> 6m3s v1.19.3+vmware.1 192.168.11.48 192.168.11.48 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1
grape-md-0-7486cf9b49-hrsn8 Ready <none> 4m45s v1.19.3+vmware.1 192.168.11.51 192.168.11.51 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1
grape-md-0-7486cf9b49-shfqn Ready <none> 7m15s v1.19.3+vmware.1 192.168.11.31 192.168.11.31 VMware Photon OS/Linux 4.19.150-1.ph3 containerd://1.4.1
他のクラスタも同じ手順でアップデートできます。
TKG 1.2.0を1.2.1にアップデートしました。アップデートも簡単でした。