IK.AM

@making's tech note


Helm Chartをimgpkg bundle化してair-gapped環境にインストールできるようにする

🗃 {Dev/Carvel/imgpkg}
🏷 Kubernetes 🏷 Carvle 🏷 air-gapped 🏷 kbld 🏷 imgpkg 🏷 Helm 
🗓 Updated at 2021-12-15T00:00:00+09:00  🗓 Created at 2021-12-15T00:00:00+09:00 {✒️️ Edit  ⏰ History  🗑 Delete}

TUNA-JP Advent Calendar 2021 その2の15日目のエントリです。

前の記事でimgpkgでコンテナイメージをrelocateする方法を説明しましたが、 今回はhelm chartをimgpkg bundle化してrelocateしてみます。

目次

Helm chartのimgpkg bundle化

Helm chartのimgpkg bundle化します。サンプルchartとして bitnami/nginx を使用します。

まずはHelm chartを次のコマンドでダウンロードします。

mkdir -p /tmp/chart
cd /tmp/chart
helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/nginx --untar
cd nginx

まずは次のコマンドでchartをレンダリングしてみましょう。

helm template nginx ./

次のYAMLが出力されます。

---
# Source: nginx/templates/server-block-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-server-block
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
data:
  server-blocks-paths.conf: |-
    include  "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
    include  "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
---
# Source: nginx/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
spec:
  type: LoadBalancer
  externalTrafficPolicy: "Cluster"
  ports:
    - name: http
      port: 80
      targetPort: http
  selector:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: nginx
---
# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx
      app.kubernetes.io/instance: nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx
        helm.sh/chart: nginx-9.5.7
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/managed-by: Helm
    spec:
      
      automountServiceAccountToken: false
      shareProcessNamespace: false
      serviceAccountName: default
      affinity:
        podAffinity:
          
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: nginx
                    app.kubernetes.io/instance: nginx
                namespaces:
                  - "default"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:
          
      containers:
        - name: nginx
          image: docker.io/bitnami/nginx:1.21.3-debian-10-r29
          imagePullPolicy: "IfNotPresent"
          env:
            - name: BITNAMI_DEBUG
              value: "false"
          ports:
            - name: http
              containerPort: 8080
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            tcpSocket:
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            successThreshold: 1
            failureThreshold: 3
          resources:
            limits: {}
            requests: {}
          volumeMounts:
      volumes:
        - name: nginx-server-block-paths
          configMap:
            name: nginx-server-block
            items:
              - key: server-blocks-paths.conf
                path: server-blocks-paths.conf

docker.io/bitnami/nginx:1.21.3-debian-10-r29がrelocation対象のイメージです。 ここで注意なのですが、helm chartによってはパラメータを指定したときのみ現れるコンテナ定義があります。 以下のコマンドで、テンプレートに何種類のimageが設定されているか確認します。

$ find templates -type f | xargs grep 'image: ' | uniq
templates/deployment.yaml:          image: {{ include "nginx.cloneStaticSiteFromGit.image" . }}
templates/deployment.yaml:          image: {{ include "nginx.image" . }}
templates/deployment.yaml:          image: {{ include "nginx.ldapDaemon.image" . }}
templates/deployment.yaml:          image: {{ include "nginx.metrics.image" . }}

このChartには実はイメージが4つあるようです。これらのイメージもrelocation対象としたい場合は、これらを有効にするパラメータを指定した上て helm chartをレンダリングする必要があります。レンダリングにimage:さえ出力できればkbldでlockファイルを作成できるので、他のパラメータはダミー値で構いません。

4つのimage:が出力されるように次のパラメータを指定してYAMLをレンダリングします。

helm template nginx ./ \
  --set metrics.enabled=true \
  --set ldapDaemon.enabled=true \
  --set cloneStaticSiteFromGit.enabled=true \
  --set cloneStaticSiteFromGit.repository=example.com \
  --set cloneStaticSiteFromGit.branch=main

次のYAMLが出力されます。

---
# Source: nginx/templates/ldap-daemon-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: nginx-ldap-daemon
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  ldap-daemon-ldap-bind-password: ""
  ldap_nginx.conf: |-
    c2VydmVyIHsKbGlzdGVuIDAuMC4wLjA6ODA4MDsKCiMgWW91IGNhbiBwcm92aWRlIGEgc3BlY2lhbCBzdWJQYXRoIG9yIHRoZSByb290CmxvY2F0aW9uID0gLyB7CiAgICBhdXRoX3JlcXVlc3QgL2F1dGgtcHJveHk7Cn0KCmxvY2F0aW9uID0gL2F1dGgtcHJveHkgewogICAgaW50ZXJuYWw7CgogICAgcHJveHlfcGFzcyBodHRwOi8vMTI3LjAuMC4xOjg4ODg7CgogICAgIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjCiAgICAjIFlPVSBTSE9VTEQgQ0hBTkdFIFRIRSBGT0xMT1dJTkcgVE8gWU9VUiBMREFQIENPTkZJR1VSQVRJT04gICMKICAgICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKICAgICMgVVJMIGFuZCBwb3J0IGZvciBjb25uZWN0aW5nIHRvIHRoZSBMREFQIHNlcnZlcgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1VUkwgImxkYXA6Ly9ZT1VSX0xEQVBfU0VSVkVSX0lQOllPVVJfTERBUF9TRVJWRVJfUE9SVCI7CgogICAgIyBCYXNlIEROCiAgICAjIHByb3h5X3NldF9oZWFkZXIgWC1MZGFwLUJhc2VETiAiZGM9ZXhhbXBsZSxkYz1vcmciOwoKICAgICMgQmluZCBETgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1CaW5kRE4gImNuPWFkbWluLGRjPWV4YW1wbGUsZGM9b3JnIjsKCiAgICAjIEJpbmQgcGFzc3dvcmQKICAgICMgcHJveHlfc2V0X2hlYWRlciBYLUxkYXAtQmluZFBhc3MgImFkbWlucGFzc3dvcmQiOwp9Cn0=
---
# Source: nginx/templates/server-block-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-server-block
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
data:
  server-blocks-paths.conf: |-
    include  "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
    include  "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
---
# Source: nginx/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
  annotations:
    prometheus.io/port: '9113'
    prometheus.io/scrape: "true"
spec:
  type: LoadBalancer
  externalTrafficPolicy: "Cluster"
  ports:
    - name: http
      port: 80
      targetPort: http
    - name: metrics
      port: 9113
      targetPort: metrics
  selector:
    app.kubernetes.io/name: nginx
    app.kubernetes.io/instance: nginx
---
# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: nginx
      app.kubernetes.io/instance: nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: nginx
        helm.sh/chart: nginx-9.5.7
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/managed-by: Helm
    spec:
      
      automountServiceAccountToken: false
      shareProcessNamespace: false
      serviceAccountName: default
      affinity:
        podAffinity:
          
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app.kubernetes.io/name: nginx
                    app.kubernetes.io/instance: nginx
                namespaces:
                  - "default"
                topologyKey: kubernetes.io/hostname
              weight: 1
        nodeAffinity:
          
      initContainers:
        - name: git-clone-repository
          image: docker.io/bitnami/git:2.33.0-debian-10-r53
          imagePullPolicy: "IfNotPresent"
          command:
            - /bin/bash
            - -ec
            - |
              [[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
              git clone example.com --branch main /app
          volumeMounts:
            - name: staticsite
              mountPath: /app
      containers:
        - name: git-repo-syncer
          image: docker.io/bitnami/git:2.33.0-debian-10-r53
          imagePullPolicy: "IfNotPresent"
          command:
            - /bin/bash
            - -ec
            - |
              [[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
              while true; do
                  cd /app && git pull origin main
                  sleep 60
              done
          volumeMounts:
            - name: staticsite
              mountPath: /app
        - name: nginx
          image: docker.io/bitnami/nginx:1.21.3-debian-10-r29
          imagePullPolicy: "IfNotPresent"
          env:
            - name: BITNAMI_DEBUG
              value: "false"
          ports:
            - name: http
              containerPort: 8080
          livenessProbe:
            tcpSocket:
              port: http
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            tcpSocket:
              port: http
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            successThreshold: 1
            failureThreshold: 3
          resources:
            limits: {}
            requests: {}
          volumeMounts:
            - name: nginx-server-block-paths
              mountPath: /opt/bitnami/nginx/conf/server_blocks
            - name: nginx-server-block-ldap
              mountPath: /opt/bitnami/nginx/conf/server_blocks/ldap
            - name: staticsite
              mountPath: /app
        - name: ldap-daemon
          image: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
          imagePullPolicy: "IfNotPresent"
          env:
            - name: NGINXLDAP_PORT_NUMBER
              value: "8888"
            - name: NGINXLDAP_LDAP_URI
              value: ""
            - name: NGINXLDAP_LDAP_BASE_DN
              value: ""
            - name: NGINXLDAP_LDAP_BIND_DN
              value: ""
            - name: NGINXLDAP_LDAP_BIND_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: nginx-ldap-daemon
                  key: ldap-daemon-ldap-bind-password
            - name: NGINXLDAP_LDAP_FILTER
              value: ""
            - name: NGINXLDAP_HTTP_REALM
              value: ""
            - name: NGINXLDAP_HTTP_COOKIE_NAME
              value: ""
          ports:
            - name: ldap-daemon
              containerPort: 8888
          livenessProbe:
            tcpSocket:
              port: ldap-daemon
            periodSeconds: 10
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 6
          readinessProbe:
            tcpSocket:
              port: ldap-daemon
            initialDelaySeconds: 5
            periodSeconds: 5
            timeoutSeconds: 3
            successThreshold: 1
            failureThreshold: 3
        - name: metrics
          image: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
          imagePullPolicy: "IfNotPresent"
          command: ['/usr/bin/exporter', '-nginx.scrape-uri', 'http://127.0.0.1:8080/status']
          ports:
            - name: metrics
              containerPort: 9113
          livenessProbe:
            httpGet:
              path: /metrics
              port: metrics
            initialDelaySeconds: 15
            timeoutSeconds: 5
          readinessProbe:
            httpGet:
              path: /metrics
              port: metrics
            initialDelaySeconds: 5
            timeoutSeconds: 1
          resources:
            limits: {}
            requests: {}
      volumes:
        - name: nginx-server-block-paths
          configMap:
            name: nginx-server-block
            items:
              - key: server-blocks-paths.conf
                path: server-blocks-paths.conf
        - name: staticsite
          
          emptyDir: {}
        - name: nginx-server-block-ldap
          secret:
            secretName: nginx-ldap-daemon

次のコマンドにより、このYAMLに対して、kbldを使ってlockファイルを作成します。

mkdir -p .imgpkg
helm template nginx ./ \
  --set metrics.enabled=true \
  --set ldapDaemon.enabled=true \
  --set cloneStaticSiteFromGit.enabled=true \
  --set cloneStaticSiteFromGit.repository=example.com \
  --set cloneStaticSiteFromGit.branch=main \
  | kbld -f - --imgpkg-lock-output .imgpkg/images.yml

生成された.imgpkg/images.ymlは次の通りです。

---
apiVersion: imgpkg.carvel.dev/v1alpha1
images:
- annotations:
    kbld.carvel.dev/id: docker.io/bitnami/git:2.33.0-debian-10-r53
    kbld.carvel.dev/origins: |
      - resolved:
          tag: 2.33.0-debian-10-r53
          url: docker.io/bitnami/git:2.33.0-debian-10-r53
  image: index.docker.io/bitnami/git@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
- annotations:
    kbld.carvel.dev/id: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
    kbld.carvel.dev/origins: |
      - resolved:
          tag: 0.9.0-debian-10-r182
          url: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
  image: index.docker.io/bitnami/nginx-exporter@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
- annotations:
    kbld.carvel.dev/id: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
    kbld.carvel.dev/origins: |
      - resolved:
          tag: 0.20200116.0-debian-10-r475
          url: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
  image: index.docker.io/bitnami/nginx-ldap-auth-daemon@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
- annotations:
    kbld.carvel.dev/id: docker.io/bitnami/nginx:1.21.3-debian-10-r29
    kbld.carvel.dev/origins: |
      - resolved:
          tag: 1.21.3-debian-10-r29
          url: docker.io/bitnami/nginx:1.21.3-debian-10-r29
  image: index.docker.io/bitnami/nginx@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
kind: ImagesLock

この.imgpkg/images.ymlを使って次のコマンドでimgpkg bundleを作成します。

CHART_VERSION=$(grep '^version:' Chart.yaml | awk '{print $2}')
imgpkg push -b ghcr.io/making/nginx-chart-bundle:${CHART_VERSION} -f .

次のログが出力されます。

dir: .
file: .helmignore
dir: .imgpkg
file: .imgpkg/images.yml
file: Chart.lock
file: Chart.yaml
file: README.md
dir: charts
dir: charts/common
file: charts/common/.helmignore
file: charts/common/Chart.yaml
file: charts/common/README.md
dir: charts/common/templates
file: charts/common/templates/_affinities.tpl
file: charts/common/templates/_capabilities.tpl
file: charts/common/templates/_errors.tpl
file: charts/common/templates/_images.tpl
file: charts/common/templates/_ingress.tpl
file: charts/common/templates/_labels.tpl
file: charts/common/templates/_names.tpl
file: charts/common/templates/_secrets.tpl
file: charts/common/templates/_storage.tpl
file: charts/common/templates/_tplvalues.tpl
file: charts/common/templates/_utils.tpl
file: charts/common/templates/_warnings.tpl
dir: charts/common/templates/validations
file: charts/common/templates/validations/_cassandra.tpl
file: charts/common/templates/validations/_mariadb.tpl
file: charts/common/templates/validations/_mongodb.tpl
file: charts/common/templates/validations/_postgresql.tpl
file: charts/common/templates/validations/_redis.tpl
file: charts/common/templates/validations/_validations.tpl
file: charts/common/values.yaml
dir: ci
file: ci/ct-values.yaml
file: ci/values-with-ingress-metrics-and-serverblock.yaml
dir: templates
file: templates/NOTES.txt
file: templates/_helpers.tpl
file: templates/deployment.yaml
file: templates/extra-list.yaml
file: templates/health-ingress.yaml
file: templates/hpa.yaml
file: templates/ingress.yaml
file: templates/ldap-daemon-secrets.yaml
file: templates/pdb.yaml
file: templates/server-block-configmap.yaml
file: templates/serviceaccount.yaml
file: templates/servicemonitor.yaml
file: templates/svc.yaml
file: templates/tls-secrets.yaml
file: values.schema.json
file: values.yaml
Pushed 'ghcr.io/making/nginx-chart-bundle@sha256:d9ef393cf39246e623e833581f7fb9dbf8547b610152da958d80221a23899a84'
Succeeded

このバンドルは ghcr.io/making/nginx-chart-bundle:9.5.7 で公開されています。

imgpkg bundleのrelocation

air-gapped環境へのインストールのため、次のコマンドでバンドルをtarに保存します。

imgpkg copy -b ghcr.io/making/nginx-chart-bundle:9.5.7 --to-tar nginx-chart-bundle.tar 

次のようなログが出力されます。

copy | exporting 5 images...
copy | will export ghcr.io/making/nginx-chart-bundle@sha256:d9ef393cf39246e623e833581f7fb9dbf8547b610152da958d80221a23899a84
copy | will export index.docker.io/bitnami/git@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
copy | will export index.docker.io/bitnami/nginx-exporter@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
copy | will export index.docker.io/bitnami/nginx-ldap-auth-daemon@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
copy | will export index.docker.io/bitnami/nginx@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
copy | exported 5 images
copy | writing layers...
copy | done: file 'manifest.json' (26.339µs)
copy | done: file 'sha256-d8cf0c803abf51d8e4a20ee4a17bf48b2f267396af4a45ea83be8af7ab188710.tar.gz' (1.112166955s)
copy | done: file 'sha256-35d1145925efafa29ff297ba25bf01387453922d714a871203a96966167e0d7f.tar.gz' (696.029043ms)
copy | done: file 'sha256-2e46367a9b5f947342898ef2845225822b277b367b05dc2367ed54c527ab1617.tar.gz' (44.279µs)
copy | done: file 'sha256-9dcc5c84c412c071744339f7dac5d245c125d474051ae941d7a33be6d1fb4576.tar.gz' (848.960806ms)
copy | done: file 'sha256-a8bb333e4d158e8ac7fe39ee6e9deb8054b1802d2be908159e56cde1b686dc6d.tar.gz' (78.474µs)
copy | done: file 'sha256-f31847b3e450bbf963c5af86373a0a834014d8642cd0069423427f9cb82c17f5.tar.gz' (55.975662ms)
copy | done: file 'sha256-c655512c34e4cd081998903c3662d39b888a180cb6d5a18d541593062929b22d.tar.gz' (34.12045ms)
copy | done: file 'sha256-557f5b495dcfa57a1dd8414fcc7f06e5f22a00221148ba6bb2cb4523431f4836.tar.gz' (57.326279ms)
copy | done: file 'sha256-d40f2b5421d688b4c3445a153dbe52941dfd51dd6a9db2a1a922183cd38d14e9.tar.gz' (1.611628034s)
copy | done: file 'sha256-71458009e090af3fc0d58c16ba8ceac1e55382f3b61d2eb9c2a626c441bd59a6.tar.gz' (47.613176ms)
copy | done: file 'sha256-1cb99cb3992aa72bc20e1d2f8292e19a7797a01d1987b636a454981a5778766e.tar.gz' (6.421488ms)
copy | done: file 'sha256-99f997e0805ac95c7f19d5694eb6a84ba66f63ccb0b33fe7bd97056552654196.tar.gz' (93.511804ms)
copy | done: file 'sha256-2c5c8e67f415847a39b4cda3b27d427ff24b83f858646c642f0b12acdb00ed15.tar.gz' (118.049µs)
copy | done: file 'sha256-4947c07ba2f7025c35fc4efc6545c33ceb54511a9a5a3616210131762a196f8e.tar.gz' (359.888µs)
copy | done: file 'sha256-ecba2195845da9aeaab33f1e7818c7eefd5be889f76eb3072ebc63b6a726d99b.tar.gz' (167.752475ms)
copy | done: file 'sha256-dfab7ee9ac0134ec1e93f4c126fc9f96a00e70b4e31e6818184a7d02116d8532.tar.gz' (115.502054ms)
copy | done: file 'sha256-8eacba373b59a1c6de1ce4af7f8c462765794e41f4759286b479da65bbe29dae.tar.gz' (300.924µs)
copy | done: file 'sha256-b895b9104086c79969b39d5716f55e5b0da17435a41e37ded64a085f2c38fd27.tar.gz' (49.023µs)
copy | done: file 'sha256-bc37c30461b64a1ebd46c5050db7b17c78599745a3c0ac59ad3ed8abf4518398.tar.gz' (141.655µs)
copy | done: file 'sha256-2289c50711e5a0106e029b15ae20ffef19c12f94cd14e4174a7281d7a7c7790a.tar.gz' (72.897µs)
copy | done: file 'sha256-e83dbfdc23800540058be45dbfd06580360f4988285e9d5256799547d8c1ba0a.tar.gz' (51.124µs)
copy | done: file 'sha256-b81a06bcef4c86affd0cd697b47cc630f047fd7b88e134812d197672e085d0e7.tar.gz' (51.122µs)
copy | done: file 'sha256-834e94929751ffe44b730a9dedffac7b329965207950dbe40a4e88d932f7d1e5.tar.gz' (30.931µs)
copy | done: file 'sha256-d96867e2f0eb39a15f355e8402c7c449448b8eb7afc28ce6eb4514c48cdb1524.tar.gz' (237.624181ms)
copy | done: file 'sha256-c39783cd65b065727e4cb3c5ab185fe4b1781b6c8c38287f8b07dfc15c345e1f.tar.gz' (48.784µs)
copy | done: file 'sha256-c024618adaddf7ecc4e5f7b6f7aa69417b129439631eea68c482472a0bd9fad2.tar.gz' (46.795µs)
copy | done: file 'sha256-e2db0a794ff24e562a7d3a6cdbe86786c49e6dad00a30517a84163813dc53eed.tar.gz' (33.333µs)
copy | done: file 'sha256-a0c4c037dae8485940eb4c92b2961a9d0d923f07717e37d62e977b416950bdf3.tar.gz' (56.272µs)
copy | done: file 'sha256-a79ca765cd5134e21cc64e0ac8ec6cc97eb0c23c662b0b9127e2afbdb254b5fa.tar.gz' (1.924933675s)
copy | done: file 'sha256-0bd67e46db6f6f43ec0dbce401f351fc8e9be20705d68ddd094f5d44c2a86358.tar.gz' (1.823683654s)

Succeeded

ファイルサイズを確認します。

$ ls -lh nginx-chart-bundle.tar 
-rw-r--r--  1 toshiaki  staff   113M Dec 14 12:14 nginx-chart-bundle.tar

次のコマンドで、このbundleを こちらの記事 で作成したHarborにrelocateします。

imgpkg copy --tar nginx-chart-bundle.tar --to-repo ${HARBOR_HOST}/library/nginx-chart-bundle --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem 

次のログが出力されます。

copy | importing 5 images...

 113.00 MiB / 113.25 MiB [=================================================================================================================================================================================================================================================]  99.78% 3.14 MiB/s 35s

copy | done uploading images

Succeeded

HarborのUIでイメージがpushされていることを確認できます。

image

relocateされたHelm Chartのインストール

次のコマンドでrelocateされたbundleをpullして展開します。

imgpkg pull -b ${HARBOR_HOST}/library/nginx-chart-bundle:${CHART_VERSION} -o /tmp/nginx-chart-bundle --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem

次のログが出力されます。

Pulling bundle 'harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:d9ef393cf39246e623e833581f7fb9dbf8547b610152da958d80221a23899a84'
Extracting layer 'sha256:1cb99cb3992aa72bc20e1d2f8292e19a7797a01d1987b636a454981a5778766e' (1/1)

Locating image lock file images...
The bundle repo (harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml)

Succeeded

次のコマンドでrelocateされたイメージを使ったYAMLを生成します。

helm template nginx /tmp/nginx-chart-bundle | kbld -f - -f /tmp/nginx-chart-bundle/.imgpkg/images.yml

次のYAMLが出力されます。image:が変わっていることが確認できます。

---
apiVersion: v1
data:
  server-blocks-paths.conf: |-
    include  "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
    include  "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx-server-block
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: nginx
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kbld.k14s.io/images: |
      - origins:
        - resolved:
            tag: 1.21.3-debian-10-r29
            url: docker.io/bitnami/nginx:1.21.3-debian-10-r29
        - preresolved:
            url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
        url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: nginx
      app.kubernetes.io/name: nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: nginx
        helm.sh/chart: nginx-9.5.7
    spec:
      affinity:
        nodeAffinity: null
        podAffinity: null
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: nginx
                  app.kubernetes.io/name: nginx
              namespaces:
              - default
              topologyKey: kubernetes.io/hostname
            weight: 1
      automountServiceAccountToken: false
      containers:
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 5
        name: nginx
        ports:
        - containerPort: 8080
          name: http
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 3
        resources:
          limits: {}
          requests: {}
        volumeMounts: null
      serviceAccountName: default
      shareProcessNamespace: false
      volumes:
      - configMap:
          items:
          - key: server-blocks-paths.conf
            path: server-blocks-paths.conf
          name: nginx-server-block
        name: nginx-server-block-paths

次のコマンドでこのYAMLを適用すれば、air-gapped環境にhelm chartのインストールができます。

helm template nginx /tmp/nginx-chart-bundle | kbld -f - -f /tmp/nginx-chart-bundle/.imgpkg/images.yml | kubectl apply -f -

次のコマンドで、Podが起動して、relocateされたイメージが使われていることを確認できます。

$ kubectl get pod -l app.kubernetes.io/name=nginx
NAME                    READY   STATUS    RESTARTS   AGE
nginx-7648796b9-5tdrk   1/1     Running   0          2m36s

$ kubectl get pod -l app.kubernetes.io/name=nginx -o jsonpath='{.items[0].spec.containers[0].image}' 
harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747

なお、次コマンドで4つのイメージを全て使うパターンも試します。

helm template nginx /tmp/nginx-chart-bundle \
  --set metrics.enabled=true \
  --set ldapDaemon.enabled=true \
  --set cloneStaticSiteFromGit.enabled=true \
  --set cloneStaticSiteFromGit.repository=example.com \
  --set cloneStaticSiteFromGit.branch=main \
 | kbld -f - -f /tmp/nginx-chart-bundle/.imgpkg/images.yml  

次のYAMLが出力されます。image:が全てHarbor上のものに置き換わっていることを確認できます。

---
apiVersion: v1
data:
  ldap-daemon-ldap-bind-password: ""
  ldap_nginx.conf: c2VydmVyIHsKbGlzdGVuIDAuMC4wLjA6ODA4MDsKCiMgWW91IGNhbiBwcm92aWRlIGEgc3BlY2lhbCBzdWJQYXRoIG9yIHRoZSByb290CmxvY2F0aW9uID0gLyB7CiAgICBhdXRoX3JlcXVlc3QgL2F1dGgtcHJveHk7Cn0KCmxvY2F0aW9uID0gL2F1dGgtcHJveHkgewogICAgaW50ZXJuYWw7CgogICAgcHJveHlfcGFzcyBodHRwOi8vMTI3LjAuMC4xOjg4ODg7CgogICAgIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjCiAgICAjIFlPVSBTSE9VTEQgQ0hBTkdFIFRIRSBGT0xMT1dJTkcgVE8gWU9VUiBMREFQIENPTkZJR1VSQVRJT04gICMKICAgICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKICAgICMgVVJMIGFuZCBwb3J0IGZvciBjb25uZWN0aW5nIHRvIHRoZSBMREFQIHNlcnZlcgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1VUkwgImxkYXA6Ly9ZT1VSX0xEQVBfU0VSVkVSX0lQOllPVVJfTERBUF9TRVJWRVJfUE9SVCI7CgogICAgIyBCYXNlIEROCiAgICAjIHByb3h5X3NldF9oZWFkZXIgWC1MZGFwLUJhc2VETiAiZGM9ZXhhbXBsZSxkYz1vcmciOwoKICAgICMgQmluZCBETgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1CaW5kRE4gImNuPWFkbWluLGRjPWV4YW1wbGUsZGM9b3JnIjsKCiAgICAjIEJpbmQgcGFzc3dvcmQKICAgICMgcHJveHlfc2V0X2hlYWRlciBYLUxkYXAtQmluZFBhc3MgImFkbWlucGFzc3dvcmQiOwp9Cn0=
kind: Secret
metadata:
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx-ldap-daemon
type: Opaque
---
apiVersion: v1
data:
  server-blocks-paths.conf: |-
    include  "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
    include  "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx-server-block
---
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9113"
    prometheus.io/scrape: "true"
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx
spec:
  externalTrafficPolicy: Cluster
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: metrics
    port: 9113
    targetPort: metrics
  selector:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/name: nginx
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kbld.k14s.io/images: |
      - origins:
        - resolved:
            tag: 0.20200116.0-debian-10-r475
            url: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
        - preresolved:
            url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
        url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
      - origins:
        - resolved:
            tag: 0.9.0-debian-10-r182
            url: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
        - preresolved:
            url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
        url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
      - origins:
        - resolved:
            tag: 2.33.0-debian-10-r53
            url: docker.io/bitnami/git:2.33.0-debian-10-r53
        - preresolved:
            url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
        url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
      - origins:
        - resolved:
            tag: 1.21.3-debian-10-r29
            url: docker.io/bitnami/nginx:1.21.3-debian-10-r29
        - preresolved:
            url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
        url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
  labels:
    app.kubernetes.io/instance: nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: nginx
    helm.sh/chart: nginx-9.5.7
  name: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: nginx
      app.kubernetes.io/name: nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: nginx
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: nginx
        helm.sh/chart: nginx-9.5.7
    spec:
      affinity:
        nodeAffinity: null
        podAffinity: null
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/instance: nginx
                  app.kubernetes.io/name: nginx
              namespaces:
              - default
              topologyKey: kubernetes.io/hostname
            weight: 1
      automountServiceAccountToken: false
      containers:
      - command:
        - /bin/bash
        - -ec
        - |
          [[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
          while true; do
              cd /app && git pull origin main
              sleep 60
          done
        image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
        imagePullPolicy: IfNotPresent
        name: git-repo-syncer
        volumeMounts:
        - mountPath: /app
          name: staticsite
      - env:
        - name: BITNAMI_DEBUG
          value: "false"
        image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 5
        name: nginx
        ports:
        - containerPort: 8080
          name: http
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          tcpSocket:
            port: http
          timeoutSeconds: 3
        resources:
          limits: {}
          requests: {}
        volumeMounts:
        - mountPath: /opt/bitnami/nginx/conf/server_blocks
          name: nginx-server-block-paths
        - mountPath: /opt/bitnami/nginx/conf/server_blocks/ldap
          name: nginx-server-block-ldap
        - mountPath: /app
          name: staticsite
      - env:
        - name: NGINXLDAP_PORT_NUMBER
          value: "8888"
        - name: NGINXLDAP_LDAP_URI
          value: ""
        - name: NGINXLDAP_LDAP_BASE_DN
          value: ""
        - name: NGINXLDAP_LDAP_BIND_DN
          value: ""
        - name: NGINXLDAP_LDAP_BIND_PASSWORD
          valueFrom:
            secretKeyRef:
              key: ldap-daemon-ldap-bind-password
              name: nginx-ldap-daemon
        - name: NGINXLDAP_LDAP_FILTER
          value: ""
        - name: NGINXLDAP_HTTP_REALM
          value: ""
        - name: NGINXLDAP_HTTP_COOKIE_NAME
          value: ""
        image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 6
          periodSeconds: 10
          successThreshold: 1
          tcpSocket:
            port: ldap-daemon
          timeoutSeconds: 5
        name: ldap-daemon
        ports:
        - containerPort: 8888
          name: ldap-daemon
        readinessProbe:
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 5
          successThreshold: 1
          tcpSocket:
            port: ldap-daemon
          timeoutSeconds: 3
      - command:
        - /usr/bin/exporter
        - -nginx.scrape-uri
        - http://127.0.0.1:8080/status
        image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /metrics
            port: metrics
          initialDelaySeconds: 15
          timeoutSeconds: 5
        name: metrics
        ports:
        - containerPort: 9113
          name: metrics
        readinessProbe:
          httpGet:
            path: /metrics
            port: metrics
          initialDelaySeconds: 5
          timeoutSeconds: 1
        resources:
          limits: {}
          requests: {}
      initContainers:
      - command:
        - /bin/bash
        - -ec
        - |
          [[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
          git clone example.com --branch main /app
        image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
        imagePullPolicy: IfNotPresent
        name: git-clone-repository
        volumeMounts:
        - mountPath: /app
          name: staticsite
      serviceAccountName: default
      shareProcessNamespace: false
      volumes:
      - configMap:
          items:
          - key: server-blocks-paths.conf
            path: server-blocks-paths.conf
          name: nginx-server-block
        name: nginx-server-block-paths
      - emptyDir: {}
        name: staticsite
      - name: nginx-server-block-ldap
        secret:
          secretName: nginx-ldap-daemon

imgpkg / kbldを使ってHelm Chartのimgpkg bundle化、及びair-gapped環境へのインストールができることを確認しました。 次の記事 では、今回作成したimgpkg bundleを使ってHelm ChartのCarvel Packageを作成します。