TUNA-JP Advent Calendar 2021 その2の19日目のエントリです。
Tanzu Community Editionで提供されているPackage Repository は次のコマンドで登録できます。
tanzu package repository add tce-repo \
--url projects.registry.vmware.com/tce/main:0.9.1 \
--namespace tanzu-package-repo-global
このRepositoryに含まれるPackageのいくつかはDockerHub上のコンテナイメージをmanifestに持っています。 DockerHubはrate limitがあるため、環境によってはPackageをインストールしようとすると409エラーでImagePullBackOffになるかもしれません。 あるいはTCEをAir-gappedな環境にインストールしたい場合もパブリックなコンテナイメージレジストリからは取得できません。
CarvelのPackage Repositoryはレポジトリに含まれるPackage及び、そのPackageで使用される全てのコンテナイメージを別のレジストリにrelocateすることができます。ドキュメントは次の通りです。
https://carvel.dev/kapp-controller/docs/latest/air-gapped-workflow/#copy-packagerepository-bundle-to-new-location
今回はTCEのRepositoryを こちらの記事 で利用したHarborにrelocateしてみます。
次のコマンドで一括でイメージをrelocateできます。
imgpkg copy -b projects.registry.vmware.com/tce/main:0.9.1 --to-repo ${HARBOR_HOST}/library/tce/main --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem
次のようなログが出力されます。合計75個のイメージがrelocateされたことが分かります。
copy | exporting 75 images...
copy | will export gcr.io/knative-releases/knative.dev/net-certmanager/cmd/controller@sha256:cab4e2f78c14d200191bef8873b7a37fed7cc7747f767268b30a6dbcdb2f27c0
copy | will export gcr.io/knative-releases/knative.dev/net-certmanager/cmd/webhook@sha256:2ae59e8006ec196fae3ff8b4adce9e12d00076217b0b34431adc134d5de61deb
copy | will export gcr.io/knative-releases/knative.dev/net-contour/cmd/controller@sha256:705ac18c11352a874ed35d7404eed1792d599bbd5fcdaed7b945484b1d567bd9
copy | will export gcr.io/knative-releases/knative.dev/serving/cmd/activator@sha256:91e67a579378fa39d7c941e379db183464c3add3d53b4617f65d9cbc2f0c770a
copy | will export gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler@sha256:761dc36210e69ebef3a64ce72ad9f54f8172e4aed6b97e8a706e3128956ec54d
copy | will export gcr.io/knative-releases/knative.dev/serving/cmd/controller@sha256:d772809059033e437d6e98248a334ded37b6f430c2ca23377875cc2459a3b73e
copy | will export gcr.io/knative-releases/knative.dev/serving/cmd/default-domain@sha256:ee79c1b6c9e07d6bbdf8ff6a53a30b7c288140c1a253e4e545251fe2e8911d44
copy | will export gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:6cd0c234bfbf88ac75df5243c2f9213dcc9def610414c506d418f9388187b771
copy | will export gcr.io/knative-releases/knative.dev/serving/cmd/webhook@sha256:268bd1383b56ba7b9acf391c681f7a63780c22dcd4555c2f4a7b61ec6da81cf4
copy | will export index.docker.io/bitnami/external-dns@sha256:1a26ba96760f5f488be15a08070254de9705893bbddd4ec51f40cc6f2cc1ecfd
copy | will export index.docker.io/envoyproxy/envoy@sha256:55d35e368436519136dbd978fa0682c49d8ab99e4d768413510f226762b30b07
copy | will export index.docker.io/envoyproxy/envoy@sha256:ac6a29af5bee160a1b4425d7c7a41a4d8a08a7f9dd7f225f21b5375f6439457a
copy | will export index.docker.io/envoyproxy/envoy@sha256:e5c2bb2870d0e59ce917a5100311813b4ede96ce4eb0c6bfa879e3fbe3e83935
copy | will export index.docker.io/fluent/fluent-bit@sha256:63745ff3b3ced72611f31b9998435721405573a3239d79974f0e3a60e4903d02
copy | will export index.docker.io/goharbor/harbor-core@sha256:b4703fd32a9dc4de208649405ac88de36126122f5d30ea0e7e2eff5626f28397
copy | will export index.docker.io/goharbor/harbor-db@sha256:4fac4598a1066611a7aaca000a56f81b951b5d047d6cefc8643fc4ecc638da67
copy | will export index.docker.io/goharbor/harbor-exporter@sha256:17b33dc8816db4fa214ce1765d5cc9e6ea6e2f7b80f0f64182aa84c9575bc3d2
copy | will export index.docker.io/goharbor/harbor-jobservice@sha256:e555c6736db6383e8147c847c733a36a16400f292e78ee4040c3396bae486da5
copy | will export index.docker.io/goharbor/harbor-portal@sha256:bd5103391fc37ecfa84cb07a4a7ff0dbed88b72f02d92e38d730da7141518b6c
copy | will export index.docker.io/goharbor/harbor-registryctl@sha256:f6764ff0ee618c916568d7bfbf144a9f9ec70c23589985efa451614aa735a371
copy | will export index.docker.io/goharbor/notary-server-photon@sha256:b1f41cfc903ecdf1e5b4efe7e640fe858f70a1c5cd0d3c51d4cc4ab34eb568fc
copy | will export index.docker.io/goharbor/notary-signer-photon@sha256:a0b375fa2e7c27de1cf41e23800e97878adcdeb236ecbf94dea16d03ed4449ef
copy | will export index.docker.io/goharbor/redis-photon@sha256:dc34c1f49c0bffe5fff540d1902bd45f3b6231f08d7abc244885744066085781
copy | will export index.docker.io/goharbor/registry-photon@sha256:c5c65ec7c78ca3965f75536d83d8252780191819b45e8ab5a7e9ff07f9e11046
copy | will export index.docker.io/goharbor/trivy-adapter-photon@sha256:42649438283db6db7e2086058d690be776266a25bea7255b67d24be0352b85e7
copy | will export index.docker.io/jimmidyson/configmap-reload@sha256:904d08e9f701d3d8178cb61651dbe8edc5d08dd5895b56bdcac9e5805ea82b52
copy | will export index.docker.io/library/busybox@sha256:b37dd066f59a4961024cf4bed74cae5e68ac26b48807292bd12198afa3ecb778
copy | will export index.docker.io/nfvpe/multus@sha256:31a0dc9ae164819c86f3ae49af8a01a34d2b1e0ced9340d1fb9fdb0743e3b599
copy | will export index.docker.io/nfvpe/multus@sha256:43d6316cc38d1922d26662d750cc85dd2e1233ded10f64c4ca63dde29d79493c
copy | will export index.docker.io/nfvpe/multus@sha256:78af60d4083dabf6ee6ec762ec0facaff8a05ea58621d45b607b99236105ceb7
copy | will export index.docker.io/nfvpe/multus@sha256:aa849206c510add3e3e1c6a843036bbcab0378312aa9b3e664b5c5139638ecdc
copy | will export index.docker.io/openpolicyagent/gatekeeper@sha256:9cd6e864dcc37d55970e592cc885c129cd7d13e59e3988a337abd77a41852879
copy | will export index.docker.io/projectcontour/contour@sha256:31c376f80f5b80f2ac5558c1c50711438796b22b742285789ea21cd2bba244e2
copy | will export index.docker.io/projectcontour/contour@sha256:3e2eb259697ad21415902436ba7d6544a2da007a8cb6d0a4abd46fa956a95d49
copy | will export index.docker.io/projectcontour/contour@sha256:4ff3e4b23770d80e25915e1450fed692d8490aa491b04f32124908931dd7d5f8
copy | will export index.docker.io/prom/alertmanager@sha256:624c1a5063c7c80635081a504c3e1b020d89809651978eb5d0b652a394f3022d
copy | will export index.docker.io/prom/prometheus@sha256:d1a9a86b9a3e60a9ea3cde141bdc936847456acc497e0affe7e288234383efa5
copy | will export index.docker.io/prom/pushgateway@sha256:ca32c7864bb2573bf27ff6628a03d17b37b1aa3dc367b5d86831e6c0f0761376
copy | will export projects.registry.vmware.com/tce/cert-manager@sha256:31cdadff5b576fb04622b15c9c8fc22006bde0a4da21b9fce0dbf0b99254e2bb
copy | will export projects.registry.vmware.com/tce/cert-manager@sha256:c087c02ac3957486baeb91f9e0713a2d672dd7532c739ed836e9795e4160aeb6
copy | will export projects.registry.vmware.com/tce/cert-manager@sha256:fcd2ea27f3d2c86ef56da7bb92c231537b12824d7c1bc3d7e5cea7d2aed4a9b9
copy | will export projects.registry.vmware.com/tce/contour@sha256:0c3d0f33c171e437268e57bdbe0d83feb1606362d8235f1b656556da8c944e18
copy | will export projects.registry.vmware.com/tce/contour@sha256:7f031ada007ab3ba53a8a71b55a7f2123343a7f180263de13ac72d7de97a16b0
copy | will export projects.registry.vmware.com/tce/contour@sha256:e485ed740ad65c6831ce821d09f419fe92b13f00e1abac40f20f19f5bde9696a
copy | will export projects.registry.vmware.com/tce/external-dns@sha256:8e797800abde577a732d571947d70e1866764c57e00bb587b756911852dd41c3
copy | will export projects.registry.vmware.com/tce/fluent-bit@sha256:e8c0873ef8e1ea04f32b88afbab64fc4338d0c0536ef2f91b9dd54bee2c59ed7
copy | will export projects.registry.vmware.com/tce/gatekeeper@sha256:b7a2102712f4de53f1d46532d468b5af5f1bfba60d5b9c6d1cffe59967b9077d
copy | will export projects.registry.vmware.com/tce/grafana@sha256:53d41d9ac1534fc381efa5bb181aa4cac1ec26fc77c7ffadb34550930112e193
copy | will export projects.registry.vmware.com/tce/harbor@sha256:51c9728301e64763ff872378e86b17d4cedc9cb64305519faa4e90c1ce257e9a
copy | will export projects.registry.vmware.com/tce/knative-serving@sha256:e99da49f3a7515e787ca5278a5d99600e9ee6eaa863e5283e0f6477a7c3d8bc1
copy | will export projects.registry.vmware.com/tce/local-path-provisioner@sha256:6434e827349036958783c1f81b01838b5d7316c1275a25ba0f76ea7a89455231
copy | will export projects.registry.vmware.com/tce/local-path-provisioner@sha256:a2d213121067f0f3df26ec813eb752b4e4d5e9f590b51d0a922c4ecd71541d65
copy | will export projects.registry.vmware.com/tce/local-path-storage@sha256:336f49fb455dae12a03507c1874fdbee38240f4243c512539a2bb1835580462f
copy | will export projects.registry.vmware.com/tce/local-path-storage@sha256:e0db08cc6e83efb1f772ab9714d78900b5634146c266954abc805461a005beb1
copy | will export projects.registry.vmware.com/tce/main@sha256:7e2e49ae8ffec158cf0b9f33701c31d5f70f4ac3a9688fc28523a95c988d0813
copy | will export projects.registry.vmware.com/tce/multus-cni@sha256:c2f76e193746de752a7387f6068293af649a5ddf9492d80b66f46f266af8c770
copy | will export projects.registry.vmware.com/tce/prometheus@sha256:b6d81e04a8ba1d6b5a33fb3f781f8238a26fa9a2add1df301b9e189d58f6682b
copy | will export projects.registry.vmware.com/tce/velero/velero-plugin-for-aws@sha256:b2b82dc47cec1f3d86595b07251efddc870133a4fc89b837502cb8e73e8187e9
copy | will export projects.registry.vmware.com/tce/velero/velero-plugin-for-microsoft-azure@sha256:84e07ce2bf61bed674c32c7f981094c0c3c2f82981ccc459ef33eccd00483c7d
copy | will export projects.registry.vmware.com/tce/velero/velero@sha256:a99d463670667fbef3363785181317e1dd87c44af808a775fc09046b80c3643f
copy | will export projects.registry.vmware.com/tce/velero@sha256:d1334e22e7c3e4e3e8e1b9b7a728f5f4b00c885f5963a74e20eac95484d82313
copy | will export projects.registry.vmware.com/tkg/grafana/grafana@sha256:df8f25cc9ee43d6ea4c22f9c6c46644e2b9a485562dd0dafe831b5b582ac0a71
copy | will export projects.registry.vmware.com/tkg/grafana/k8s-sidecar@sha256:9f1ad1e5e404bc43f9591b1189c187f535d6f61769468c49b4fc97add803d7b9
copy | will export projects.registry.vmware.com/tkg/prometheus/cadvisor@sha256:b4cd4cc0ef05630f70d621420ad1316f631f35cef21edb7a62fff7bd787bbfd3
copy | will export quay.io/coreos/kube-state-metrics@sha256:ace842fc85031688d06c4aa000b5b1e58ba3b9dd13d26e7c8f2547f7ee0bcc84
copy | will export quay.io/jetstack/cert-manager-cainjector@sha256:51c0df411b66aa175e9fc6840f3135d55b52c3781d0b3d4aa10862066d460193
copy | will export quay.io/jetstack/cert-manager-cainjector@sha256:a24ab35640bb9ddd58a0801e8021bec68e0ab247d62ea10c69d86506097e9eaa
copy | will export quay.io/jetstack/cert-manager-cainjector@sha256:de02e3f445cfe7c035f2a9939b948c4d043011713389d9437311a62740f20bef
copy | will export quay.io/jetstack/cert-manager-controller@sha256:22543d32793abc0069680f80ee5be348dcbb3c74c85ba55835b4cf6c76fe18da
copy | will export quay.io/jetstack/cert-manager-controller@sha256:7b039d469ed739a652f3bb8a1ddc122942b66cceeb85bac315449724ee64287f
copy | will export quay.io/jetstack/cert-manager-controller@sha256:85f03d2ffdfd53f4dab53147b3192a5322a3300b5368f60416e7b838e65f42d1
copy | will export quay.io/jetstack/cert-manager-webhook@sha256:41eacd93a30b566b780a6ae525b2547d2a87f1ec5f067fc02840a220aeb0c3f7
copy | will export quay.io/jetstack/cert-manager-webhook@sha256:ed6354190d259524d32ae74471f93bf46bfdcf4df6f73629eedf576cd87e10b8
copy | will export quay.io/jetstack/cert-manager-webhook@sha256:f91867a7769436d72e6f4bb68e4e3d240d93d5bc8cd59742298a2a2b3ccf11b7
copy | will export quay.io/prometheus/node-exporter@sha256:22fbde17ab647ddf89841e5e464464eece111402b7d599882c2a3393bc0d2810
copy | exported 75 images
copy | importing 75 images...
4.05 GiB / 4.05 GiB [==================================================================================================================================================================================================================================================] 99.94% 3.79 MiB/s 18m15s
copy | done uploading images
Succeeded
HarborのUIを見ると沢山のイメージが登録されたことが分かります。
relocation先がair-gapped環境の場合は次のようにtarファイルを経由することができます。
imgpkg copy -b projects.registry.vmware.com/tce/main:0.9.1 --to-tar repo.tar --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem
imgpkg copy --tar repo.tar --to-repo ${HARBOR_HOST}/library/tce/main --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem
relocate後のPackageRepositoryの登録は、次のようにURLを変えるだけです。
tanzu package repository add tce-repo \
--url ${HARBOR_HOST}/library/tce/main:0.9.1 \
--namespace tanzu-package-repo-global
次のコマンドでPackageRepositoryを確認できます。
$ kubectl get packagerepository -n tanzu-package-repo-global tce-repo -oyaml
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageRepository
metadata:
creationTimestamp: "2021-12-18T14:48:39Z"
finalizers:
- finalizers.packagerepository.packaging.carvel.dev/delete
generation: 1
name: tce-repo
namespace: tanzu-package-repo-global
resourceVersion: "2198551"
uid: c9d30610-2d0a-478c-aceb-849740efffc1
spec:
fetch:
imgpkgBundle:
image: harbor-10-213-232-22.sslip.io/library/tce/main:0.9.1
status:
conditions:
- status: "True"
type: ReconcileSucceeded
consecutiveReconcileSuccesses: 1
deploy:
exitCode: 0
finished: true
startedAt: "2021-12-18T14:49:13Z"
stdout: |-
Target cluster 'https://100.64.0.1:443'
Changes
Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri
tanzu-package-repo-global cert-manager.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ cert-manager.community.tanzu.vmware.com.1.3.3 Package - - create - - - -
^ cert-manager.community.tanzu.vmware.com.1.4.4 Package - - create - - - -
^ cert-manager.community.tanzu.vmware.com.1.5.3 Package - - create - - - -
^ contour.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ contour.community.tanzu.vmware.com.1.17.1 Package - - create - - - -
^ contour.community.tanzu.vmware.com.1.17.2 Package - - create - - - -
^ contour.community.tanzu.vmware.com.1.18.1 Package - - create - - - -
^ external-dns.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ external-dns.community.tanzu.vmware.com.0.8.0 Package - - create - - - -
^ fluent-bit.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ fluent-bit.community.tanzu.vmware.com.1.7.5 Package - - create - - - -
^ gatekeeper.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ gatekeeper.community.tanzu.vmware.com.1.0.0 Package - - create - - - -
^ grafana.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ grafana.community.tanzu.vmware.com.7.5.7 Package - - create - - - -
^ harbor.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ harbor.community.tanzu.vmware.com.2.2.3 Package - - create - - - -
^ knative-serving.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ knative-serving.community.tanzu.vmware.com.0.22.0 Package - - create - - - -
^ local-path-storage.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ local-path-storage.community.tanzu.vmware.com.0.0.19 Package - - create - - - -
^ local-path-storage.community.tanzu.vmware.com.0.0.20 Package - - create - - - -
^ multus-cni.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ multus-cni.community.tanzu.vmware.com.3.7.1 Package - - create - - - -
^ prometheus.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ prometheus.community.tanzu.vmware.com.2.27.0 Package - - create - - - -
^ velero.community.tanzu.vmware.com PackageMetadata - - create - - - -
^ velero.community.tanzu.vmware.com.1.6.3 Package - - create - - - -
Op: 29 create, 0 delete, 0 update, 0 noop
Wait to: 0 reconcile, 0 delete, 29 noop
Pod watching error: pods is forbidden: User "system:serviceaccount:tkg-system:kapp-controller-sa" cannot list resource "pods" in API group "" in the namespace "tanzu-package-repo-global"
2:49:58PM: ---- applying 29 changes [0/29 done] ----
2:49:58PM: create package/cert-manager.community.tanzu.vmware.com.1.4.4 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create package/velero.community.tanzu.vmware.com.1.6.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create package/cert-manager.community.tanzu.vmware.com.1.3.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create packagemetadata/cert-manager.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create package/cert-manager.community.tanzu.vmware.com.1.5.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create package/contour.community.tanzu.vmware.com.1.17.1 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create packagemetadata/contour.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create package/contour.community.tanzu.vmware.com.1.18.1 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:58PM: create package/contour.community.tanzu.vmware.com.1.17.2 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/external-dns.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/external-dns.community.tanzu.vmware.com.0.8.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/gatekeeper.community.tanzu.vmware.com.1.0.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/fluent-bit.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/fluent-bit.community.tanzu.vmware.com.1.7.5 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/gatekeeper.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/local-path-storage.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/grafana.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/grafana.community.tanzu.vmware.com.7.5.7 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/harbor.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/harbor.community.tanzu.vmware.com.2.2.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/knative-serving.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/knative-serving.community.tanzu.vmware.com.0.22.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/multus-cni.community.tanzu.vmware.com.3.7.1 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/local-path-storage.community.tanzu.vmware.com.0.0.19 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/local-path-storage.community.tanzu.vmware.com.0.0.20 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/multus-cni.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create package/prometheus.community.tanzu.vmware.com.2.27.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/prometheus.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: create packagemetadata/velero.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ---- waiting on 29 changes [0/29 done] ----
2:49:59PM: ok: noop packagemetadata/velero.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/gatekeeper.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/fluent-bit.community.tanzu.vmware.com.1.7.5 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/cert-manager.community.tanzu.vmware.com.1.4.4 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/local-path-storage.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/grafana.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/grafana.community.tanzu.vmware.com.7.5.7 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/harbor.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/velero.community.tanzu.vmware.com.1.6.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/cert-manager.community.tanzu.vmware.com.1.3.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/harbor.community.tanzu.vmware.com.2.2.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/knative-serving.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/cert-manager.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/cert-manager.community.tanzu.vmware.com.1.5.3 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/contour.community.tanzu.vmware.com.1.17.1 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/contour.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/contour.community.tanzu.vmware.com.1.18.1 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/contour.community.tanzu.vmware.com.1.17.2 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/knative-serving.community.tanzu.vmware.com.0.22.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/external-dns.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/external-dns.community.tanzu.vmware.com.0.8.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/gatekeeper.community.tanzu.vmware.com.1.0.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/fluent-bit.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/local-path-storage.community.tanzu.vmware.com.0.0.20 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/multus-cni.community.tanzu.vmware.com.3.7.1 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/local-path-storage.community.tanzu.vmware.com.0.0.19 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop package/prometheus.community.tanzu.vmware.com.2.27.0 (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/multus-cni.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ok: noop packagemetadata/prometheus.community.tanzu.vmware.com (data.packaging.carvel.dev/v1alpha1) namespace: tanzu-package-repo-global
2:49:59PM: ---- applying complete [29/29 done] ----
2:49:59PM: ---- waiting complete [29/29 done] ----
Succeeded
updatedAt: "2021-12-18T14:49:59Z"
fetch:
exitCode: 0
startedAt: "2021-12-18T14:49:11Z"
stdout: |
apiVersion: vendir.k14s.io/v1alpha1
directories:
- contents:
- imgpkgBundle:
image: harbor-10-213-232-22.sslip.io/library/tce/main@sha256:7e2e49ae8ffec158cf0b9f33701c31d5f70f4ac3a9688fc28523a95c988d0813
path: .
path: "0"
kind: LockConfig
updatedAt: "2021-12-18T14:49:13Z"
friendlyDescription: Reconcile succeeded
observedGeneration: 1
template:
exitCode: 0
stderr: |
resolve | final: projects.registry.vmware.com/tce/cert-manager@sha256:31cdadff5b576fb04622b15c9c8fc22006bde0a4da21b9fce0dbf0b99254e2bb -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:31cdadff5b576fb04622b15c9c8fc22006bde0a4da21b9fce0dbf0b99254e2bb
resolve | final: projects.registry.vmware.com/tce/cert-manager@sha256:c087c02ac3957486baeb91f9e0713a2d672dd7532c739ed836e9795e4160aeb6 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:c087c02ac3957486baeb91f9e0713a2d672dd7532c739ed836e9795e4160aeb6
resolve | final: projects.registry.vmware.com/tce/cert-manager@sha256:fcd2ea27f3d2c86ef56da7bb92c231537b12824d7c1bc3d7e5cea7d2aed4a9b9 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:fcd2ea27f3d2c86ef56da7bb92c231537b12824d7c1bc3d7e5cea7d2aed4a9b9
resolve | final: projects.registry.vmware.com/tce/contour@sha256:0c3d0f33c171e437268e57bdbe0d83feb1606362d8235f1b656556da8c944e18 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:0c3d0f33c171e437268e57bdbe0d83feb1606362d8235f1b656556da8c944e18
resolve | final: projects.registry.vmware.com/tce/contour@sha256:7f031ada007ab3ba53a8a71b55a7f2123343a7f180263de13ac72d7de97a16b0 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:7f031ada007ab3ba53a8a71b55a7f2123343a7f180263de13ac72d7de97a16b0
resolve | final: projects.registry.vmware.com/tce/contour@sha256:e485ed740ad65c6831ce821d09f419fe92b13f00e1abac40f20f19f5bde9696a -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:e485ed740ad65c6831ce821d09f419fe92b13f00e1abac40f20f19f5bde9696a
resolve | final: projects.registry.vmware.com/tce/external-dns@sha256:8e797800abde577a732d571947d70e1866764c57e00bb587b756911852dd41c3 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:8e797800abde577a732d571947d70e1866764c57e00bb587b756911852dd41c3
resolve | final: projects.registry.vmware.com/tce/fluent-bit@sha256:e8c0873ef8e1ea04f32b88afbab64fc4338d0c0536ef2f91b9dd54bee2c59ed7 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:e8c0873ef8e1ea04f32b88afbab64fc4338d0c0536ef2f91b9dd54bee2c59ed7
resolve | final: projects.registry.vmware.com/tce/gatekeeper@sha256:b7a2102712f4de53f1d46532d468b5af5f1bfba60d5b9c6d1cffe59967b9077d -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:b7a2102712f4de53f1d46532d468b5af5f1bfba60d5b9c6d1cffe59967b9077d
resolve | final: projects.registry.vmware.com/tce/grafana@sha256:53d41d9ac1534fc381efa5bb181aa4cac1ec26fc77c7ffadb34550930112e193 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:53d41d9ac1534fc381efa5bb181aa4cac1ec26fc77c7ffadb34550930112e193
resolve | final: projects.registry.vmware.com/tce/harbor@sha256:51c9728301e64763ff872378e86b17d4cedc9cb64305519faa4e90c1ce257e9a -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:51c9728301e64763ff872378e86b17d4cedc9cb64305519faa4e90c1ce257e9a
resolve | final: projects.registry.vmware.com/tce/knative-serving@sha256:e99da49f3a7515e787ca5278a5d99600e9ee6eaa863e5283e0f6477a7c3d8bc1 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:e99da49f3a7515e787ca5278a5d99600e9ee6eaa863e5283e0f6477a7c3d8bc1
resolve | final: projects.registry.vmware.com/tce/local-path-storage@sha256:336f49fb455dae12a03507c1874fdbee38240f4243c512539a2bb1835580462f -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:336f49fb455dae12a03507c1874fdbee38240f4243c512539a2bb1835580462f
resolve | final: projects.registry.vmware.com/tce/local-path-storage@sha256:e0db08cc6e83efb1f772ab9714d78900b5634146c266954abc805461a005beb1 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:e0db08cc6e83efb1f772ab9714d78900b5634146c266954abc805461a005beb1
resolve | final: projects.registry.vmware.com/tce/multus-cni@sha256:c2f76e193746de752a7387f6068293af649a5ddf9492d80b66f46f266af8c770 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:c2f76e193746de752a7387f6068293af649a5ddf9492d80b66f46f266af8c770
resolve | final: projects.registry.vmware.com/tce/prometheus@sha256:b6d81e04a8ba1d6b5a33fb3f781f8238a26fa9a2add1df301b9e189d58f6682b -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:b6d81e04a8ba1d6b5a33fb3f781f8238a26fa9a2add1df301b9e189d58f6682b
resolve | final: projects.registry.vmware.com/tce/velero@sha256:d1334e22e7c3e4e3e8e1b9b7a728f5f4b00c885f5963a74e20eac95484d82313 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:d1334e22e7c3e4e3e8e1b9b7a728f5f4b00c885f5963a74e20eac95484d82313
updatedAt: "2021-12-18T14:49:13Z"
spec.fetch.imgpkgBundle.image
やstatus.template.stderr
を見るとイメージがHarborにrelocateされていることが分かります。
Packageのインストールもしてみます。PackageRepositoryが登録されてしまえば、インストール方法はrelocation関係なく行えます。
次のコマンドでPrometheus Packageをインストールします。
cat <<EOF > prometheus-data-values.yaml
prometheus:
service:
type: LoadBalancer
EOF
tanzu package install prometheus --package-name prometheus.community.tanzu.vmware.com --version 2.27.0 --namespace default --values-file prometheus-data-values.yaml
インストールが完了したら、次のコマンドで作成されたApp CRを確認します。
$ kubectl get app prometheus -oyaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
creationTimestamp: "2021-12-18T14:55:28Z"
finalizers:
- finalizers.kapp-ctrl.k14s.io/delete
generation: 1
name: prometheus
namespace: default
ownerReferences:
- apiVersion: packaging.carvel.dev/v1alpha1
blockOwnerDeletion: true
controller: true
kind: PackageInstall
name: prometheus
uid: 6791d34c-c136-4f48-a80d-12f5f349ed94
resourceVersion: "2200691"
uid: 947b27a4-67a1-417c-a3df-b8865e301936
spec:
deploy:
- kapp: {}
fetch:
- imgpkgBundle:
image: harbor-10-213-232-22.sslip.io/library/tce/main@sha256:b6d81e04a8ba1d6b5a33fb3f781f8238a26fa9a2add1df301b9e189d58f6682b
serviceAccountName: prometheus-default-sa
template:
- ytt:
paths:
- config/
valuesFrom:
- secretRef:
name: prometheus-default-values
- kbld:
paths:
- '-'
- .imgpkg/images.yml
status:
conditions:
- status: "True"
type: ReconcileSucceeded
consecutiveReconcileSuccesses: 4
deploy:
exitCode: 0
finished: true
startedAt: "2021-12-18T14:58:37Z"
stdout: |-
Target cluster 'https://100.64.0.1:443' (nodes: cheetah-control-plane-xtwcj, 1+)
02:58:37PM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreaclusternetworkpolicystats"}
02:58:37PM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"networkpolicystats"}
02:58:37PM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreanetworkpolicystats"}
Changes
Namespace Name Kind Conds. Age Op Op st. Wait to Rs Ri
Op: 0 create, 0 delete, 0 update, 0 noop
Wait to: 0 reconcile, 0 delete, 0 noop
Succeeded
updatedAt: "2021-12-18T14:58:39Z"
fetch:
exitCode: 0
startedAt: "2021-12-18T14:58:35Z"
stdout: |
apiVersion: vendir.k14s.io/v1alpha1
directories:
- contents:
- imgpkgBundle:
image: harbor-10-213-232-22.sslip.io/library/tce/main@sha256:b6d81e04a8ba1d6b5a33fb3f781f8238a26fa9a2add1df301b9e189d58f6682b
path: .
path: "0"
kind: LockConfig
updatedAt: "2021-12-18T14:58:36Z"
friendlyDescription: Reconcile succeeded
inspect:
exitCode: 0
stdout: |-
Target cluster 'https://100.64.0.1:443' (nodes: cheetah-control-plane-xtwcj, 1+)
02:58:39PM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"networkpolicystats"}
02:58:39PM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreaclusternetworkpolicystats"}
02:58:39PM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreanetworkpolicystats"}
Resources in app 'prometheus-ctrl'
Namespace Name Kind Owner Conds. Rs Ri Age
(cluster) alertmanager ClusterRole kapp - ok - 3m
^ alertmanager ClusterRoleBinding kapp - ok - 3m
^ prometheus Namespace kapp - ok - 3m
^ prometheus-cadvisor ClusterRole kapp - ok - 3m
^ prometheus-cadvisor ClusterRoleBinding kapp - ok - 3m
^ prometheus-kube-state-metrics ClusterRole kapp - ok - 3m
^ prometheus-kube-state-metrics ClusterRoleBinding kapp - ok - 3m
^ prometheus-node-exporter ClusterRole kapp - ok - 3m
^ prometheus-node-exporter ClusterRoleBinding kapp - ok - 3m
^ prometheus-node-exporter PodSecurityPolicy kapp - ok - 3m
^ prometheus-pushgateway ClusterRole kapp - ok - 3m
^ prometheus-pushgateway ClusterRoleBinding kapp - ok - 3m
^ prometheus-server ClusterRole kapp - ok - 3m
^ prometheus-server ClusterRoleBinding kapp - ok - 3m
prometheus alertmanager Deployment kapp 2/2 t ok - 3m
^ alertmanager Endpoints cluster - ok - 3m
^ alertmanager PersistentVolumeClaim kapp - ok - 3m
^ alertmanager Secret kapp - ok - 3m
^ alertmanager Service kapp - ok - 3m
^ alertmanager-548f4ccd66 ReplicaSet cluster - ok - 3m
^ alertmanager-548f4ccd66-lkhmk Pod cluster 4/4 t ok - 3m
^ alertmanager-qlpw9 EndpointSlice cluster - ok - 3m
^ alertmanager-sa ServiceAccount kapp - ok - 3m
^ prometheus-cadvisor DaemonSet kapp - ok - 3m
^ prometheus-cadvisor ServiceAccount kapp - ok - 3m
^ prometheus-cadvisor-6bdf8755f ControllerRevision cluster - ok - 3m
^ prometheus-cadvisor-s5k5r Pod cluster 4/4 t ok - 3m
^ prometheus-kube-state-metrics Deployment kapp 2/2 t ok - 3m
^ prometheus-kube-state-metrics Endpoints cluster - ok - 3m
^ prometheus-kube-state-metrics Service kapp - ok - 3m
^ prometheus-kube-state-metrics ServiceAccount kapp - ok - 3m
^ prometheus-kube-state-metrics-6b8dd89dcb ReplicaSet cluster - ok - 3m
^ prometheus-kube-state-metrics-6b8dd89dcb-cdrlf Pod cluster 4/4 t ok - 3m
^ prometheus-kube-state-metrics-vtspg EndpointSlice cluster - ok - 3m
^ prometheus-node-exporter DaemonSet kapp - ok - 3m
^ prometheus-node-exporter Endpoints cluster - ok - 3m
^ prometheus-node-exporter Service kapp - ok - 3m
^ prometheus-node-exporter-74f6d656b6 ControllerRevision cluster - ok - 3m
^ prometheus-node-exporter-crc4f Pod cluster 4/4 t ok - 3m
^ prometheus-node-exporter-k7c88 EndpointSlice cluster - ok - 3m
^ prometheus-node-exporter-sa ServiceAccount kapp - ok - 3m
^ prometheus-node-exporter-snkqc Pod cluster 4/4 t ok - 3m
^ prometheus-pushgateway Deployment kapp 2/2 t ok - 3m
^ prometheus-pushgateway Endpoints cluster - ok - 3m
^ prometheus-pushgateway Service kapp - ok - 3m
^ prometheus-pushgateway ServiceAccount kapp - ok - 3m
^ prometheus-pushgateway-65bb85d997 ReplicaSet cluster - ok - 3m
^ prometheus-pushgateway-65bb85d997-dz462 Pod cluster 4/4 t ok - 3m
^ prometheus-pushgateway-7wph9 EndpointSlice cluster - ok - 3m
^ prometheus-server ConfigMap kapp - ok - 3m
^ prometheus-server Deployment kapp 2/2 t ok - 3m
^ prometheus-server Endpoints cluster - ok - 3m
^ prometheus-server PersistentVolumeClaim kapp - ok - 3m
^ prometheus-server Service kapp - ok - 3m
^ prometheus-server-5dffddf9f8 ReplicaSet cluster - ok - 3m
^ prometheus-server-5dffddf9f8-qjldb Pod cluster 4/4 t ok - 3m
^ prometheus-server-sa ServiceAccount kapp - ok - 3m
^ prometheus-server-xlrbc EndpointSlice cluster - ok - 3m
Rs: Reconcile state
Ri: Reconcile information
58 resources
Succeeded
updatedAt: "2021-12-18T14:58:40Z"
observedGeneration: 1
template:
exitCode: 0
stderr: |
resolve | final: gcr.io/cadvisor/cadvisor:v0.39.1 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:b4cd4cc0ef05630f70d621420ad1316f631f35cef21edb7a62fff7bd787bbfd3
resolve | final: jimmidyson/configmap-reload:v0.5.0 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:904d08e9f701d3d8178cb61651dbe8edc5d08dd5895b56bdcac9e5805ea82b52
resolve | final: prom/alertmanager:v0.22.2 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:624c1a5063c7c80635081a504c3e1b020d89809651978eb5d0b652a394f3022d
resolve | final: prom/prometheus:v2.27.0 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:d1a9a86b9a3e60a9ea3cde141bdc936847456acc497e0affe7e288234383efa5
resolve | final: prom/pushgateway:v1.4.0 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:ca32c7864bb2573bf27ff6628a03d17b37b1aa3dc367b5d86831e6c0f0761376
resolve | final: quay.io/coreos/kube-state-metrics:v1.9.8 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:ace842fc85031688d06c4aa000b5b1e58ba3b9dd13d26e7c8f2547f7ee0bcc84
resolve | final: quay.io/prometheus/node-exporter:v1.1.2 -> harbor-10-213-232-22.sslip.io/library/tce/main@sha256:22fbde17ab647ddf89841e5e464464eece111402b7d599882c2a3393bc0d2810
updatedAt: "2021-12-18T14:58:37Z"
spec.fetch[0].imgpkgBundle.image
やstatus.template.stderr
を見るとイメージがHarborにrelocateされていることが分かります。
imgpkgでPackageRepositoryをrelocateしただけで、全てのPackageとイメージが一括でrelocateできるので、とても便利です。