TUNA-JP Advent Calendar 2021 その2の15日目のエントリです。
前の記事でimgpkgでコンテナイメージをrelocateする方法を説明しましたが、 今回はhelm chartをimgpkg bundle化してrelocateしてみます。
目次
Helm chartのimgpkg bundle化
Helm chartのimgpkg bundle化します。サンプルchartとして bitnami/nginx を使用します。
まずはHelm chartを次のコマンドでダウンロードします。
mkdir -p /tmp/chart
cd /tmp/chart
helm repo add bitnami https://charts.bitnami.com/bitnami
helm pull bitnami/nginx --untar
cd nginx
まずは次のコマンドでchartをレンダリングしてみましょう。
helm template nginx ./
次のYAMLが出力されます。
---
# Source: nginx/templates/server-block-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-server-block
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
data:
server-blocks-paths.conf: |-
include "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
include "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
---
# Source: nginx/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
spec:
type: LoadBalancer
externalTrafficPolicy: "Cluster"
ports:
- name: http
port: 80
targetPort: http
selector:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
---
# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
template:
metadata:
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
spec:
automountServiceAccountToken: false
shareProcessNamespace: false
serviceAccountName: default
affinity:
podAffinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
namespaces:
- "default"
topologyKey: kubernetes.io/hostname
weight: 1
nodeAffinity:
containers:
- name: nginx
image: docker.io/bitnami/nginx:1.21.3-debian-10-r29
imagePullPolicy: "IfNotPresent"
env:
- name: BITNAMI_DEBUG
value: "false"
ports:
- name: http
containerPort: 8080
livenessProbe:
tcpSocket:
port: http
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
readinessProbe:
tcpSocket:
port: http
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
resources:
limits: {}
requests: {}
volumeMounts:
volumes:
- name: nginx-server-block-paths
configMap:
name: nginx-server-block
items:
- key: server-blocks-paths.conf
path: server-blocks-paths.conf
docker.io/bitnami/nginx:1.21.3-debian-10-r29がrelocation対象のイメージです。
ここで注意なのですが、helm chartによってはパラメータを指定したときのみ現れるコンテナ定義があります。
以下のコマンドで、テンプレートに何種類のimageが設定されているか確認します。
$ find templates -type f | xargs grep 'image: ' | uniq
templates/deployment.yaml: image: {{ include "nginx.cloneStaticSiteFromGit.image" . }}
templates/deployment.yaml: image: {{ include "nginx.image" . }}
templates/deployment.yaml: image: {{ include "nginx.ldapDaemon.image" . }}
templates/deployment.yaml: image: {{ include "nginx.metrics.image" . }}
このChartには実はイメージが4つあるようです。これらのイメージもrelocation対象としたい場合は、これらを有効にするパラメータを指定した上て
helm chartをレンダリングする必要があります。レンダリングにimage:さえ出力できればkbldでlockファイルを作成できるので、他のパラメータはダミー値で構いません。
4つのimage:が出力されるように次のパラメータを指定してYAMLをレンダリングします。
helm template nginx ./ \
--set metrics.enabled=true \
--set ldapDaemon.enabled=true \
--set cloneStaticSiteFromGit.enabled=true \
--set cloneStaticSiteFromGit.repository=example.com \
--set cloneStaticSiteFromGit.branch=main
次のYAMLが出力されます。
---
# Source: nginx/templates/ldap-daemon-secrets.yaml
apiVersion: v1
kind: Secret
metadata:
name: nginx-ldap-daemon
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
type: Opaque
data:
ldap-daemon-ldap-bind-password: ""
ldap_nginx.conf: |-
c2VydmVyIHsKbGlzdGVuIDAuMC4wLjA6ODA4MDsKCiMgWW91IGNhbiBwcm92aWRlIGEgc3BlY2lhbCBzdWJQYXRoIG9yIHRoZSByb290CmxvY2F0aW9uID0gLyB7CiAgICBhdXRoX3JlcXVlc3QgL2F1dGgtcHJveHk7Cn0KCmxvY2F0aW9uID0gL2F1dGgtcHJveHkgewogICAgaW50ZXJuYWw7CgogICAgcHJveHlfcGFzcyBodHRwOi8vMTI3LjAuMC4xOjg4ODg7CgogICAgIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjCiAgICAjIFlPVSBTSE9VTEQgQ0hBTkdFIFRIRSBGT0xMT1dJTkcgVE8gWU9VUiBMREFQIENPTkZJR1VSQVRJT04gICMKICAgICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKICAgICMgVVJMIGFuZCBwb3J0IGZvciBjb25uZWN0aW5nIHRvIHRoZSBMREFQIHNlcnZlcgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1VUkwgImxkYXA6Ly9ZT1VSX0xEQVBfU0VSVkVSX0lQOllPVVJfTERBUF9TRVJWRVJfUE9SVCI7CgogICAgIyBCYXNlIEROCiAgICAjIHByb3h5X3NldF9oZWFkZXIgWC1MZGFwLUJhc2VETiAiZGM9ZXhhbXBsZSxkYz1vcmciOwoKICAgICMgQmluZCBETgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1CaW5kRE4gImNuPWFkbWluLGRjPWV4YW1wbGUsZGM9b3JnIjsKCiAgICAjIEJpbmQgcGFzc3dvcmQKICAgICMgcHJveHlfc2V0X2hlYWRlciBYLUxkYXAtQmluZFBhc3MgImFkbWlucGFzc3dvcmQiOwp9Cn0=
---
# Source: nginx/templates/server-block-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-server-block
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
data:
server-blocks-paths.conf: |-
include "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
include "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
---
# Source: nginx/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
annotations:
prometheus.io/port: '9113'
prometheus.io/scrape: "true"
spec:
type: LoadBalancer
externalTrafficPolicy: "Cluster"
ports:
- name: http
port: 80
targetPort: http
- name: metrics
port: 9113
targetPort: metrics
selector:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
---
# Source: nginx/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
template:
metadata:
labels:
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
spec:
automountServiceAccountToken: false
shareProcessNamespace: false
serviceAccountName: default
affinity:
podAffinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/name: nginx
app.kubernetes.io/instance: nginx
namespaces:
- "default"
topologyKey: kubernetes.io/hostname
weight: 1
nodeAffinity:
initContainers:
- name: git-clone-repository
image: docker.io/bitnami/git:2.33.0-debian-10-r53
imagePullPolicy: "IfNotPresent"
command:
- /bin/bash
- -ec
- |
[[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
git clone example.com --branch main /app
volumeMounts:
- name: staticsite
mountPath: /app
containers:
- name: git-repo-syncer
image: docker.io/bitnami/git:2.33.0-debian-10-r53
imagePullPolicy: "IfNotPresent"
command:
- /bin/bash
- -ec
- |
[[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
while true; do
cd /app && git pull origin main
sleep 60
done
volumeMounts:
- name: staticsite
mountPath: /app
- name: nginx
image: docker.io/bitnami/nginx:1.21.3-debian-10-r29
imagePullPolicy: "IfNotPresent"
env:
- name: BITNAMI_DEBUG
value: "false"
ports:
- name: http
containerPort: 8080
livenessProbe:
tcpSocket:
port: http
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
readinessProbe:
tcpSocket:
port: http
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
resources:
limits: {}
requests: {}
volumeMounts:
- name: nginx-server-block-paths
mountPath: /opt/bitnami/nginx/conf/server_blocks
- name: nginx-server-block-ldap
mountPath: /opt/bitnami/nginx/conf/server_blocks/ldap
- name: staticsite
mountPath: /app
- name: ldap-daemon
image: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
imagePullPolicy: "IfNotPresent"
env:
- name: NGINXLDAP_PORT_NUMBER
value: "8888"
- name: NGINXLDAP_LDAP_URI
value: ""
- name: NGINXLDAP_LDAP_BASE_DN
value: ""
- name: NGINXLDAP_LDAP_BIND_DN
value: ""
- name: NGINXLDAP_LDAP_BIND_PASSWORD
valueFrom:
secretKeyRef:
name: nginx-ldap-daemon
key: ldap-daemon-ldap-bind-password
- name: NGINXLDAP_LDAP_FILTER
value: ""
- name: NGINXLDAP_HTTP_REALM
value: ""
- name: NGINXLDAP_HTTP_COOKIE_NAME
value: ""
ports:
- name: ldap-daemon
containerPort: 8888
livenessProbe:
tcpSocket:
port: ldap-daemon
periodSeconds: 10
timeoutSeconds: 5
successThreshold: 1
failureThreshold: 6
readinessProbe:
tcpSocket:
port: ldap-daemon
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 3
successThreshold: 1
failureThreshold: 3
- name: metrics
image: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
imagePullPolicy: "IfNotPresent"
command: ['/usr/bin/exporter', '-nginx.scrape-uri', 'http://127.0.0.1:8080/status']
ports:
- name: metrics
containerPort: 9113
livenessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 15
timeoutSeconds: 5
readinessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 5
timeoutSeconds: 1
resources:
limits: {}
requests: {}
volumes:
- name: nginx-server-block-paths
configMap:
name: nginx-server-block
items:
- key: server-blocks-paths.conf
path: server-blocks-paths.conf
- name: staticsite
emptyDir: {}
- name: nginx-server-block-ldap
secret:
secretName: nginx-ldap-daemon
次のコマンドにより、このYAMLに対して、kbldを使ってlockファイルを作成します。
mkdir -p .imgpkg
helm template nginx ./ \
--set metrics.enabled=true \
--set ldapDaemon.enabled=true \
--set cloneStaticSiteFromGit.enabled=true \
--set cloneStaticSiteFromGit.repository=example.com \
--set cloneStaticSiteFromGit.branch=main \
| kbld -f - --imgpkg-lock-output .imgpkg/images.yml
生成された.imgpkg/images.ymlは次の通りです。
---
apiVersion: imgpkg.carvel.dev/v1alpha1
images:
- annotations:
kbld.carvel.dev/id: docker.io/bitnami/git:2.33.0-debian-10-r53
kbld.carvel.dev/origins: |
- resolved:
tag: 2.33.0-debian-10-r53
url: docker.io/bitnami/git:2.33.0-debian-10-r53
image: index.docker.io/bitnami/git@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
- annotations:
kbld.carvel.dev/id: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
kbld.carvel.dev/origins: |
- resolved:
tag: 0.9.0-debian-10-r182
url: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
image: index.docker.io/bitnami/nginx-exporter@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
- annotations:
kbld.carvel.dev/id: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
kbld.carvel.dev/origins: |
- resolved:
tag: 0.20200116.0-debian-10-r475
url: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
image: index.docker.io/bitnami/nginx-ldap-auth-daemon@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
- annotations:
kbld.carvel.dev/id: docker.io/bitnami/nginx:1.21.3-debian-10-r29
kbld.carvel.dev/origins: |
- resolved:
tag: 1.21.3-debian-10-r29
url: docker.io/bitnami/nginx:1.21.3-debian-10-r29
image: index.docker.io/bitnami/nginx@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
kind: ImagesLock
この.imgpkg/images.ymlを使って次のコマンドでimgpkg bundleを作成します。
CHART_VERSION=$(grep '^version:' Chart.yaml | awk '{print $2}')
imgpkg push -b ghcr.io/making/nginx-chart-bundle:${CHART_VERSION} -f .
次のログが出力されます。
dir: .
file: .helmignore
dir: .imgpkg
file: .imgpkg/images.yml
file: Chart.lock
file: Chart.yaml
file: README.md
dir: charts
dir: charts/common
file: charts/common/.helmignore
file: charts/common/Chart.yaml
file: charts/common/README.md
dir: charts/common/templates
file: charts/common/templates/_affinities.tpl
file: charts/common/templates/_capabilities.tpl
file: charts/common/templates/_errors.tpl
file: charts/common/templates/_images.tpl
file: charts/common/templates/_ingress.tpl
file: charts/common/templates/_labels.tpl
file: charts/common/templates/_names.tpl
file: charts/common/templates/_secrets.tpl
file: charts/common/templates/_storage.tpl
file: charts/common/templates/_tplvalues.tpl
file: charts/common/templates/_utils.tpl
file: charts/common/templates/_warnings.tpl
dir: charts/common/templates/validations
file: charts/common/templates/validations/_cassandra.tpl
file: charts/common/templates/validations/_mariadb.tpl
file: charts/common/templates/validations/_mongodb.tpl
file: charts/common/templates/validations/_postgresql.tpl
file: charts/common/templates/validations/_redis.tpl
file: charts/common/templates/validations/_validations.tpl
file: charts/common/values.yaml
dir: ci
file: ci/ct-values.yaml
file: ci/values-with-ingress-metrics-and-serverblock.yaml
dir: templates
file: templates/NOTES.txt
file: templates/_helpers.tpl
file: templates/deployment.yaml
file: templates/extra-list.yaml
file: templates/health-ingress.yaml
file: templates/hpa.yaml
file: templates/ingress.yaml
file: templates/ldap-daemon-secrets.yaml
file: templates/pdb.yaml
file: templates/server-block-configmap.yaml
file: templates/serviceaccount.yaml
file: templates/servicemonitor.yaml
file: templates/svc.yaml
file: templates/tls-secrets.yaml
file: values.schema.json
file: values.yaml
Pushed 'ghcr.io/making/nginx-chart-bundle@sha256:d9ef393cf39246e623e833581f7fb9dbf8547b610152da958d80221a23899a84'
Succeeded
このバンドルは ghcr.io/making/nginx-chart-bundle:9.5.7 で公開されています。
imgpkg bundleのrelocation
air-gapped環境へのインストールのため、次のコマンドでバンドルをtarに保存します。
imgpkg copy -b ghcr.io/making/nginx-chart-bundle:9.5.7 --to-tar nginx-chart-bundle.tar
次のようなログが出力されます。
copy | exporting 5 images...
copy | will export ghcr.io/making/nginx-chart-bundle@sha256:d9ef393cf39246e623e833581f7fb9dbf8547b610152da958d80221a23899a84
copy | will export index.docker.io/bitnami/git@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
copy | will export index.docker.io/bitnami/nginx-exporter@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
copy | will export index.docker.io/bitnami/nginx-ldap-auth-daemon@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
copy | will export index.docker.io/bitnami/nginx@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
copy | exported 5 images
copy | writing layers...
copy | done: file 'manifest.json' (26.339µs)
copy | done: file 'sha256-d8cf0c803abf51d8e4a20ee4a17bf48b2f267396af4a45ea83be8af7ab188710.tar.gz' (1.112166955s)
copy | done: file 'sha256-35d1145925efafa29ff297ba25bf01387453922d714a871203a96966167e0d7f.tar.gz' (696.029043ms)
copy | done: file 'sha256-2e46367a9b5f947342898ef2845225822b277b367b05dc2367ed54c527ab1617.tar.gz' (44.279µs)
copy | done: file 'sha256-9dcc5c84c412c071744339f7dac5d245c125d474051ae941d7a33be6d1fb4576.tar.gz' (848.960806ms)
copy | done: file 'sha256-a8bb333e4d158e8ac7fe39ee6e9deb8054b1802d2be908159e56cde1b686dc6d.tar.gz' (78.474µs)
copy | done: file 'sha256-f31847b3e450bbf963c5af86373a0a834014d8642cd0069423427f9cb82c17f5.tar.gz' (55.975662ms)
copy | done: file 'sha256-c655512c34e4cd081998903c3662d39b888a180cb6d5a18d541593062929b22d.tar.gz' (34.12045ms)
copy | done: file 'sha256-557f5b495dcfa57a1dd8414fcc7f06e5f22a00221148ba6bb2cb4523431f4836.tar.gz' (57.326279ms)
copy | done: file 'sha256-d40f2b5421d688b4c3445a153dbe52941dfd51dd6a9db2a1a922183cd38d14e9.tar.gz' (1.611628034s)
copy | done: file 'sha256-71458009e090af3fc0d58c16ba8ceac1e55382f3b61d2eb9c2a626c441bd59a6.tar.gz' (47.613176ms)
copy | done: file 'sha256-1cb99cb3992aa72bc20e1d2f8292e19a7797a01d1987b636a454981a5778766e.tar.gz' (6.421488ms)
copy | done: file 'sha256-99f997e0805ac95c7f19d5694eb6a84ba66f63ccb0b33fe7bd97056552654196.tar.gz' (93.511804ms)
copy | done: file 'sha256-2c5c8e67f415847a39b4cda3b27d427ff24b83f858646c642f0b12acdb00ed15.tar.gz' (118.049µs)
copy | done: file 'sha256-4947c07ba2f7025c35fc4efc6545c33ceb54511a9a5a3616210131762a196f8e.tar.gz' (359.888µs)
copy | done: file 'sha256-ecba2195845da9aeaab33f1e7818c7eefd5be889f76eb3072ebc63b6a726d99b.tar.gz' (167.752475ms)
copy | done: file 'sha256-dfab7ee9ac0134ec1e93f4c126fc9f96a00e70b4e31e6818184a7d02116d8532.tar.gz' (115.502054ms)
copy | done: file 'sha256-8eacba373b59a1c6de1ce4af7f8c462765794e41f4759286b479da65bbe29dae.tar.gz' (300.924µs)
copy | done: file 'sha256-b895b9104086c79969b39d5716f55e5b0da17435a41e37ded64a085f2c38fd27.tar.gz' (49.023µs)
copy | done: file 'sha256-bc37c30461b64a1ebd46c5050db7b17c78599745a3c0ac59ad3ed8abf4518398.tar.gz' (141.655µs)
copy | done: file 'sha256-2289c50711e5a0106e029b15ae20ffef19c12f94cd14e4174a7281d7a7c7790a.tar.gz' (72.897µs)
copy | done: file 'sha256-e83dbfdc23800540058be45dbfd06580360f4988285e9d5256799547d8c1ba0a.tar.gz' (51.124µs)
copy | done: file 'sha256-b81a06bcef4c86affd0cd697b47cc630f047fd7b88e134812d197672e085d0e7.tar.gz' (51.122µs)
copy | done: file 'sha256-834e94929751ffe44b730a9dedffac7b329965207950dbe40a4e88d932f7d1e5.tar.gz' (30.931µs)
copy | done: file 'sha256-d96867e2f0eb39a15f355e8402c7c449448b8eb7afc28ce6eb4514c48cdb1524.tar.gz' (237.624181ms)
copy | done: file 'sha256-c39783cd65b065727e4cb3c5ab185fe4b1781b6c8c38287f8b07dfc15c345e1f.tar.gz' (48.784µs)
copy | done: file 'sha256-c024618adaddf7ecc4e5f7b6f7aa69417b129439631eea68c482472a0bd9fad2.tar.gz' (46.795µs)
copy | done: file 'sha256-e2db0a794ff24e562a7d3a6cdbe86786c49e6dad00a30517a84163813dc53eed.tar.gz' (33.333µs)
copy | done: file 'sha256-a0c4c037dae8485940eb4c92b2961a9d0d923f07717e37d62e977b416950bdf3.tar.gz' (56.272µs)
copy | done: file 'sha256-a79ca765cd5134e21cc64e0ac8ec6cc97eb0c23c662b0b9127e2afbdb254b5fa.tar.gz' (1.924933675s)
copy | done: file 'sha256-0bd67e46db6f6f43ec0dbce401f351fc8e9be20705d68ddd094f5d44c2a86358.tar.gz' (1.823683654s)
Succeeded
ファイルサイズを確認します。
$ ls -lh nginx-chart-bundle.tar
-rw-r--r-- 1 toshiaki staff 113M Dec 14 12:14 nginx-chart-bundle.tar
次のコマンドで、このbundleを こちらの記事 で作成したHarborにrelocateします。
imgpkg copy --tar nginx-chart-bundle.tar --to-repo ${HARBOR_HOST}/library/nginx-chart-bundle --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem
次のログが出力されます。
copy | importing 5 images...
113.00 MiB / 113.25 MiB [=================================================================================================================================================================================================================================================] 99.78% 3.14 MiB/s 35s
copy | done uploading images
Succeeded
HarborのUIでイメージがpushされていることを確認できます。
relocateされたHelm Chartのインストール
次のコマンドでrelocateされたbundleをpullして展開します。
imgpkg pull -b ${HARBOR_HOST}/library/nginx-chart-bundle:${CHART_VERSION} -o /tmp/nginx-chart-bundle --registry-ca-cert-path $HOME/.config/tanzu/tkg/providers/ytt/03_customizations/harbor-ca.pem
次のログが出力されます。
Pulling bundle 'harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:d9ef393cf39246e623e833581f7fb9dbf8547b610152da958d80221a23899a84'
Extracting layer 'sha256:1cb99cb3992aa72bc20e1d2f8292e19a7797a01d1987b636a454981a5778766e' (1/1)
Locating image lock file images...
The bundle repo (harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle) is hosting every image specified in the bundle's Images Lock file (.imgpkg/images.yml)
Succeeded
次のコマンドでrelocateされたイメージを使ったYAMLを生成します。
helm template nginx /tmp/nginx-chart-bundle | kbld -f - -f /tmp/nginx-chart-bundle/.imgpkg/images.yml
次のYAMLが出力されます。image:が変わっていることが確認できます。
---
apiVersion: v1
data:
server-blocks-paths.conf: |-
include "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
include "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx-server-block
---
apiVersion: v1
kind: Service
metadata:
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx
spec:
externalTrafficPolicy: Cluster
ports:
- name: http
port: 80
targetPort: http
selector:
app.kubernetes.io/instance: nginx
app.kubernetes.io/name: nginx
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kbld.k14s.io/images: |
- origins:
- resolved:
tag: 1.21.3-debian-10-r29
url: docker.io/bitnami/nginx:1.21.3-debian-10-r29
- preresolved:
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/name: nginx
template:
metadata:
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
spec:
affinity:
nodeAffinity: null
podAffinity: null
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/name: nginx
namespaces:
- default
topologyKey: kubernetes.io/hostname
weight: 1
automountServiceAccountToken: false
containers:
- env:
- name: BITNAMI_DEBUG
value: "false"
image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 6
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: http
timeoutSeconds: 5
name: nginx
ports:
- containerPort: 8080
name: http
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
tcpSocket:
port: http
timeoutSeconds: 3
resources:
limits: {}
requests: {}
volumeMounts: null
serviceAccountName: default
shareProcessNamespace: false
volumes:
- configMap:
items:
- key: server-blocks-paths.conf
path: server-blocks-paths.conf
name: nginx-server-block
name: nginx-server-block-paths
次のコマンドでこのYAMLを適用すれば、air-gapped環境にhelm chartのインストールができます。
helm template nginx /tmp/nginx-chart-bundle | kbld -f - -f /tmp/nginx-chart-bundle/.imgpkg/images.yml | kubectl apply -f -
次のコマンドで、Podが起動して、relocateされたイメージが使われていることを確認できます。
$ kubectl get pod -l app.kubernetes.io/name=nginx
NAME READY STATUS RESTARTS AGE
nginx-7648796b9-5tdrk 1/1 Running 0 2m36s
$ kubectl get pod -l app.kubernetes.io/name=nginx -o jsonpath='{.items[0].spec.containers[0].image}'
harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
なお、次コマンドで4つのイメージを全て使うパターンも試します。
helm template nginx /tmp/nginx-chart-bundle \
--set metrics.enabled=true \
--set ldapDaemon.enabled=true \
--set cloneStaticSiteFromGit.enabled=true \
--set cloneStaticSiteFromGit.repository=example.com \
--set cloneStaticSiteFromGit.branch=main \
| kbld -f - -f /tmp/nginx-chart-bundle/.imgpkg/images.yml
次のYAMLが出力されます。image:が全てHarbor上のものに置き換わっていることを確認できます。
---
apiVersion: v1
data:
ldap-daemon-ldap-bind-password: ""
ldap_nginx.conf: c2VydmVyIHsKbGlzdGVuIDAuMC4wLjA6ODA4MDsKCiMgWW91IGNhbiBwcm92aWRlIGEgc3BlY2lhbCBzdWJQYXRoIG9yIHRoZSByb290CmxvY2F0aW9uID0gLyB7CiAgICBhdXRoX3JlcXVlc3QgL2F1dGgtcHJveHk7Cn0KCmxvY2F0aW9uID0gL2F1dGgtcHJveHkgewogICAgaW50ZXJuYWw7CgogICAgcHJveHlfcGFzcyBodHRwOi8vMTI3LjAuMC4xOjg4ODg7CgogICAgIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjCiAgICAjIFlPVSBTSE9VTEQgQ0hBTkdFIFRIRSBGT0xMT1dJTkcgVE8gWU9VUiBMREFQIENPTkZJR1VSQVRJT04gICMKICAgICMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIwoKICAgICMgVVJMIGFuZCBwb3J0IGZvciBjb25uZWN0aW5nIHRvIHRoZSBMREFQIHNlcnZlcgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1VUkwgImxkYXA6Ly9ZT1VSX0xEQVBfU0VSVkVSX0lQOllPVVJfTERBUF9TRVJWRVJfUE9SVCI7CgogICAgIyBCYXNlIEROCiAgICAjIHByb3h5X3NldF9oZWFkZXIgWC1MZGFwLUJhc2VETiAiZGM9ZXhhbXBsZSxkYz1vcmciOwoKICAgICMgQmluZCBETgogICAgIyBwcm94eV9zZXRfaGVhZGVyIFgtTGRhcC1CaW5kRE4gImNuPWFkbWluLGRjPWV4YW1wbGUsZGM9b3JnIjsKCiAgICAjIEJpbmQgcGFzc3dvcmQKICAgICMgcHJveHlfc2V0X2hlYWRlciBYLUxkYXAtQmluZFBhc3MgImFkbWlucGFzc3dvcmQiOwp9Cn0=
kind: Secret
metadata:
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx-ldap-daemon
type: Opaque
---
apiVersion: v1
data:
server-blocks-paths.conf: |-
include "/opt/bitnami/nginx/conf/server_blocks/ldap/*.conf";
include "/opt/bitnami/nginx/conf/server_blocks/common/*.conf";
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx-server-block
---
apiVersion: v1
kind: Service
metadata:
annotations:
prometheus.io/port: "9113"
prometheus.io/scrape: "true"
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx
spec:
externalTrafficPolicy: Cluster
ports:
- name: http
port: 80
targetPort: http
- name: metrics
port: 9113
targetPort: metrics
selector:
app.kubernetes.io/instance: nginx
app.kubernetes.io/name: nginx
type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
kbld.k14s.io/images: |
- origins:
- resolved:
tag: 0.20200116.0-debian-10-r475
url: docker.io/bitnami/nginx-ldap-auth-daemon:0.20200116.0-debian-10-r475
- preresolved:
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
- origins:
- resolved:
tag: 0.9.0-debian-10-r182
url: docker.io/bitnami/nginx-exporter:0.9.0-debian-10-r182
- preresolved:
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
- origins:
- resolved:
tag: 2.33.0-debian-10-r53
url: docker.io/bitnami/git:2.33.0-debian-10-r53
- preresolved:
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
- origins:
- resolved:
tag: 1.21.3-debian-10-r29
url: docker.io/bitnami/nginx:1.21.3-debian-10-r29
- preresolved:
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
url: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/name: nginx
template:
metadata:
labels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx
helm.sh/chart: nginx-9.5.7
spec:
affinity:
nodeAffinity: null
podAffinity: null
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- podAffinityTerm:
labelSelector:
matchLabels:
app.kubernetes.io/instance: nginx
app.kubernetes.io/name: nginx
namespaces:
- default
topologyKey: kubernetes.io/hostname
weight: 1
automountServiceAccountToken: false
containers:
- command:
- /bin/bash
- -ec
- |
[[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
while true; do
cd /app && git pull origin main
sleep 60
done
image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
imagePullPolicy: IfNotPresent
name: git-repo-syncer
volumeMounts:
- mountPath: /app
name: staticsite
- env:
- name: BITNAMI_DEBUG
value: "false"
image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:f21acb4cd83d4b55930d8e290ebd3064fb0e019b0f06cd10ebc297cd67901747
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 6
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: http
timeoutSeconds: 5
name: nginx
ports:
- containerPort: 8080
name: http
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
tcpSocket:
port: http
timeoutSeconds: 3
resources:
limits: {}
requests: {}
volumeMounts:
- mountPath: /opt/bitnami/nginx/conf/server_blocks
name: nginx-server-block-paths
- mountPath: /opt/bitnami/nginx/conf/server_blocks/ldap
name: nginx-server-block-ldap
- mountPath: /app
name: staticsite
- env:
- name: NGINXLDAP_PORT_NUMBER
value: "8888"
- name: NGINXLDAP_LDAP_URI
value: ""
- name: NGINXLDAP_LDAP_BASE_DN
value: ""
- name: NGINXLDAP_LDAP_BIND_DN
value: ""
- name: NGINXLDAP_LDAP_BIND_PASSWORD
valueFrom:
secretKeyRef:
key: ldap-daemon-ldap-bind-password
name: nginx-ldap-daemon
- name: NGINXLDAP_LDAP_FILTER
value: ""
- name: NGINXLDAP_HTTP_REALM
value: ""
- name: NGINXLDAP_HTTP_COOKIE_NAME
value: ""
image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:1b1440a80097ffbae0a67367b0d9936594cfed1b3a2a5c95bfde1e00ead8541b
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 6
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: ldap-daemon
timeoutSeconds: 5
name: ldap-daemon
ports:
- containerPort: 8888
name: ldap-daemon
readinessProbe:
failureThreshold: 3
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
tcpSocket:
port: ldap-daemon
timeoutSeconds: 3
- command:
- /usr/bin/exporter
- -nginx.scrape-uri
- http://127.0.0.1:8080/status
image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:7e263a720dd6783b10472268b98dfd7d9278a1093a778c695efb5cc859b214b6
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 15
timeoutSeconds: 5
name: metrics
ports:
- containerPort: 9113
name: metrics
readinessProbe:
httpGet:
path: /metrics
port: metrics
initialDelaySeconds: 5
timeoutSeconds: 1
resources:
limits: {}
requests: {}
initContainers:
- command:
- /bin/bash
- -ec
- |
[[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
git clone example.com --branch main /app
image: harbor-10-213-232-22.sslip.io/library/nginx-chart-bundle@sha256:e4d55e309b37e7c703da622982684f581c234c83d26fe76e52d302825651a695
imagePullPolicy: IfNotPresent
name: git-clone-repository
volumeMounts:
- mountPath: /app
name: staticsite
serviceAccountName: default
shareProcessNamespace: false
volumes:
- configMap:
items:
- key: server-blocks-paths.conf
path: server-blocks-paths.conf
name: nginx-server-block
name: nginx-server-block-paths
- emptyDir: {}
name: staticsite
- name: nginx-server-block-ldap
secret:
secretName: nginx-ldap-daemon
imgpkg / kbldを使ってHelm Chartのimgpkg bundle化、及びair-gapped環境へのインストールができることを確認しました。 次の記事 では、今回作成したimgpkg bundleを使ってHelm ChartのCarvel Packageを作成します。