IK.AM

@making's tech note


imgpkg bundle化したk8s manifests + ytt overlayをCarvel Package化する

🗃 {Dev/Carvel/kapp-controller}
🏷 Kubernetes 🏷 Carvle 🏷 air-gapped 🏷 kbld 🏷 imgpkg 🏷 ytt 🏷 kapp-controller 🏷 Jenkins 
🗓 Updated at 2021-12-18T00:00:00+09:00  🗓 Created at 2021-12-18T00:00:00+09:00 {✒️️ Edit  ⏰ History  🗑 Delete}

TUNA-JP Advent Calendar 2021 その2の18日目のエントリです。

前の記事で、imgpkg bundle化したHelm ChartをCarvel Package化しましたが、 Carvel Packageを作るときはテンプレート処理にhelm templateを使うより同じCarvelのyttを使う方が一般的なので、 今回は普通のk8s manifest + ytt overlay構成をimgpkg bundle化し、Carvel Packageを作成します。

今回はJenkinsをCarvel Package化してみます。

よく使われるPackageのフォルダ構成を次のように作成します。

export PACKAGE_NAME=jenkins
export PACKAGE_VERSION=0.0.1

mkdir -p /tmp/addons/packages/${PACKAGE_NAME}/${PACKAGE_VERSION}/bundle/config/upstream
mkdir -p /tmp/addons/packages/${PACKAGE_NAME}/${PACKAGE_VERSION}/bundle/config/overlay
mkdir -p /tmp/addons/packages/${PACKAGE_NAME}/${PACKAGE_VERSION}/bundle/config/values.yaml
cd /tmp/addons/packages/${PACKAGE_NAME}/${PACKAGE_VERSION}

目次

ベースのk8sマニフェストを作成

ベースとなるk8s manifest自体は次のように事前にhelm templateで作成します。 ここでは最低限の設定のみをパラメータで設定し、その他の可変部分は後にytt overlayで用意します。

helm repo add bitnami https://charts.bitnami.com/bitnami
helm template jenkins bitnami/jenkins \
  --set jenkinsPassword=CHANGEME \
  --set updateStrategy.type=Recreate \
  --set ingress.enabled=true \
  --set service.type=ClusterIP \
  > bundle/config/upstream/jenkins.yaml

生成されたmanifestは次の通りです。これをimgpkg bundle化します。

---
# Source: jenkins/templates/secrets.yaml
apiVersion: v1
kind: Secret
metadata:
  name: jenkins
  namespace: "default"
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
type: Opaque
data:
  jenkins-password: "Q0hBTkdFTUU="
---
# Source: jenkins/templates/pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins
  namespace: "default"
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
  annotations:
    volume.alpha.kubernetes.io/storage-class: default
spec:
  accessModes:
  - "ReadWriteOnce"
  resources:
    requests:
      storage: "8Gi"
---
# Source: jenkins/templates/svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: "default"
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: null
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
    nodePort: null
  selector:
    app.kubernetes.io/name: jenkins
    app.kubernetes.io/instance: jenkins
---
# Source: jenkins/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: "default"
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: jenkins
      app.kubernetes.io/instance: jenkins
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: jenkins
        helm.sh/chart: jenkins-8.0.14
        app.kubernetes.io/instance: jenkins
        app.kubernetes.io/managed-by: Helm
      annotations:
        checksum/secrets: ecab545c66cdd482057d3f3e5e8ea8c093a44452e3462f99b7e347a9bba48f07
    spec:

      affinity:
        podAffinity:

        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: jenkins
                  app.kubernetes.io/instance: jenkins
              namespaces:
              - "default"
              topologyKey: kubernetes.io/hostname
            weight: 1
        nodeAffinity:

      securityContext:
        fsGroup: 1001
      containers:
      - name: jenkins
        image: docker.io/bitnami/jenkins:2.303.1-debian-10-r38
        imagePullPolicy: "IfNotPresent"
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        env:
        - name: JENKINS_USERNAME
          value: "user"
        - name: JENKINS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: jenkins
              key: jenkins-password
        - name: JENKINS_HOME
          value: "/bitnami/jenkins/home"
        - name: DISABLE_JENKINS_INITIALIZATION
          value: "no"
        - name: JENKINS_HOST
          value: "jenkins.local"
        - name: JENKINS_EXTERNAL_HTTP_PORT_NUMBER
          value: "80"
        - name: JENKINS_EXTERNAL_HTTPS_PORT_NUMBER
          value: "443"
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: https
          containerPort: 8443
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /login
            port: http
          initialDelaySeconds: 180
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 6
          successThreshold: 1
        readinessProbe:
          httpGet:
            path: /login
            port: http
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
          successThreshold: 1
        resources:
          limits: {}
          requests:
            cpu: 300m
            memory: 512Mi
        volumeMounts:
        - name: jenkins-data
          mountPath: /bitnami/jenkins
      volumes:
      - name: jenkins-data
        persistentVolumeClaim:
          claimName: jenkins
---
# Source: jenkins/templates/ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: "default"
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  rules:
  - host: jenkins.local
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: jenkins
            port:
              name: http

ytt overlayファイルを作成

まずCarvel PackageをインストールするタイミングでJenkinsをインストールするnamespaceを決められるように次のoverlayを作成します。

cat <<EOF > bundle/config/overlay/namespace.yaml
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@ if data.values.create_namespace:
apiVersion: v1
kind: Namespace
metadata:
  name: #@ data.values.namespace
#@ end

#@ for kind in [ "Secret", "PersistentVolumeClaim", "Service", "Deployment", "Ingress" ]:
#@overlay/match by=overlay.subset({"kind":kind})
---
metadata:
  #@overlay/match missing_ok=True
  namespace: #@ data.values.namespace
#@ end

#@overlay/match by=overlay.subset({"kind":"Deployment","metadata":{"name":"jenkins"}})
---
spec:
  template:
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          #@overlay/match by=overlay.index(0)
          - podAffinityTerm:
              namespaces:
              #@overlay/match by=overlay.index(0)
              - #@ data.values.namespace
EOF

次にユーザー名とパスワードを変更できるように次のoverlayを作成します。

cat <<EOF > bundle/config/overlay/change-user.yaml
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"Deployment","metadata":{"name":"jenkins"}})
---
spec:
  template:
    spec:
      containers:
      #@overlay/match by="name"
      - name: jenkins
        env:
        #@overlay/match by="name"
        - name: JENKINS_USERNAME
          value: #@ data.values.jenkins.username

#@overlay/match by=overlay.subset({"kind":"Secret","metadata":{"name":"jenkins"}})
---
#@overlay/remove
data:
#@overlay/match missing_ok=True
stringData:
  jenkins-password: #@ data.values.jenkins.password
EOF

最後にJenkinsのホスト名を変更できるように次のoverlayを作成します。

cat <<EOF > bundle/config/overlay/change-hostname.yaml
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

#@overlay/match by=overlay.subset({"kind":"Deployment","metadata":{"name":"jenkins"}})
---
spec:
  template:
    spec:
      containers:
      #@overlay/match by="name"
      - name: jenkins
        env:
        #@overlay/match by="name"
        - name: JENKINS_HOST
          value: #@ data.values.jenkins.host

#@overlay/match by=overlay.subset({"kind":"Ingress","metadata":{"name":"jenkins"}})
---
spec:
  rules:
  #@overlay/match by=overlay.index(0)
  - host: #@ data.values.jenkins.host
EOF

インストール時に設定可能な値のデフォルト値を作成します。

cat <<EOF > bundle/config/values.yaml
#@data/values
---
namespace: demo
create_namespace: True
jenkins:
  host: jenkins.example.com
  username: admin
  password: password
EOF

最後に次のコマンドでyamlに含まれるimage:のイメージ名をlockファイルに書き出します。

mkdir -p bundle/.imgpkg 
kbld -f bundle/config --imgpkg-lock-output bundle/.imgpkg/images.yml

ファイル構成は次のようになります。

$ tree -a bundle
bundle
|-- .imgpkg
|   `-- images.yml
`-- config
    |-- overlay
    |   |-- change-hostname.yaml
    |   |-- change-user.yaml
    |   `-- namespace.yaml
    |-- upstream
    |   `-- jenkins.yaml
    `-- values.yaml

4 directories, 6 files

overlayが期待通りに動作するか次のコマンドで確認します。

ytt -f bundle/config  

期待通り、次のYAMLが出力されます。

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: v1
kind: Secret
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
type: Opaque
stringData:
  jenkins-password: password
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
  annotations:
    volume.alpha.kubernetes.io/storage-class: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: null
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
    nodePort: null
  selector:
    app.kubernetes.io/name: jenkins
    app.kubernetes.io/instance: jenkins
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: jenkins
      app.kubernetes.io/instance: jenkins
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: jenkins
        helm.sh/chart: jenkins-8.0.14
        app.kubernetes.io/instance: jenkins
        app.kubernetes.io/managed-by: Helm
      annotations:
        checksum/secrets: ecab545c66cdd482057d3f3e5e8ea8c093a44452e3462f99b7e347a9bba48f07
    spec:
      affinity:
        podAffinity: null
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: jenkins
                  app.kubernetes.io/instance: jenkins
              namespaces:
              - demo
              topologyKey: kubernetes.io/hostname
            weight: 1
        nodeAffinity: null
      securityContext:
        fsGroup: 1001
      containers:
      - name: jenkins
        image: docker.io/bitnami/jenkins:2.303.1-debian-10-r38
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        env:
        - name: JENKINS_USERNAME
          value: admin
        - name: JENKINS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: jenkins
              key: jenkins-password
        - name: JENKINS_HOME
          value: /bitnami/jenkins/home
        - name: DISABLE_JENKINS_INITIALIZATION
          value: "no"
        - name: JENKINS_HOST
          value: jenkins.example.com
        - name: JENKINS_EXTERNAL_HTTP_PORT_NUMBER
          value: "80"
        - name: JENKINS_EXTERNAL_HTTPS_PORT_NUMBER
          value: "443"
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: https
          containerPort: 8443
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /login
            port: http
          initialDelaySeconds: 180
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 6
          successThreshold: 1
        readinessProbe:
          httpGet:
            path: /login
            port: http
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
          successThreshold: 1
        resources:
          limits: {}
          requests:
            cpu: 300m
            memory: 512Mi
        volumeMounts:
        - name: jenkins-data
          mountPath: /bitnami/jenkins
      volumes:
      - name: jenkins-data
        persistentVolumeClaim:
          claimName: jenkins
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  rules:
  - host: jenkins.example.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: jenkins
            port:
              name: http

imgpkg bundleの作成

ここまでできたのでimgpkg bundleを作成します。

imgpkg push -f bundle -b ghcr.io/making/jenkins-bundle:0.0.1

次のログが出力されます。

dir: .
dir: .imgpkg
file: .imgpkg/images.yml
dir: config
dir: config/overlay
file: config/overlay/change-hostname.yaml
file: config/overlay/change-user.yaml
file: config/overlay/namespace.yaml
dir: config/upstream
file: config/upstream/jenkins.yaml
dir: config/values.yaml
Pushed 'ghcr.io/making/jenkins-bundle@sha256:565c465885d1f6f7fd6294d4f42d9f769297dc41e1b485b668f067546d99f056'
Succeeded

Package Metadataの作成

次にCarvel Package化します。 今回はJenkins Packageをpkg-install namespaceに作成します。

kubectl create ns pkg-install

まずはMeatadataを作成します。

cat <<EOF > /tmp/addons/packages/${PACKAGE_NAME}/metadata.yaml
apiVersion: data.packaging.carvel.dev/v1alpha1
kind: PackageMetadata
metadata:
  name: jenkins.pkg.maki.lol
  namespace: pkg-install
spec:
  displayName: "Jenkins"
  longDescription: "Jenkins"
  shortDescription: "Jenkins"
  categories:
  - jenkins
  - bitnami
  - ci/cd
EOF

Package version 0.0.1を作成

初版のPackageを作成します。

cat <<EOF > /tmp/addons/packages/${PACKAGE_NAME}/${PACKAGE_VERSION}/package.yml
apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: jenkins.pkg.maki.lol.${PACKAGE_VERSION}
  namespace: pkg-install
spec:
  refName: jenkins.pkg.maki.lol
  version: ${PACKAGE_VERSION}
  releaseNotes: |
    initial release
  template:
    spec:
      fetch:
      - imgpkgBundle:
          image: ghcr.io/making/jenkins-bundle:${PACKAGE_VERSION}
      template:
      - ytt: {}
      - kbld:
          paths:
          - "-"
          - ".imgpkg/images.yml"
      deploy:
      - kapp:
          rawOptions:
          - --diff-changes=true
          inspect:
            rawOptions:
            - --tree=true          
EOF

kapprawOptionsinspect.rawOptionsを設定しておくと後にkubectl get app <app name> -oyamlで出力される結果が見やすいです。

次のコマンドでpacakgematadataとpackageを作成します。

kubectl apply -f /tmp/addons/packages/${PACKAGE_NAME}/metadata.yaml
kubectl apply -f /tmp/addons/packages/${PACKAGE_NAME}/${PACKAGE_VERSION}/package.yml

RBACの設定

次のコマンドでkapp controllerがpackageをinstallするのに必要なService AccountとRole Bindingの設定を行います。 作成するリソースの名前はtanzu package installコマンドで作成される場合の命名規則に合わせました。

NAMESPACE=pkg-install
PACKAGE_INSTALL_NAME=jenkins

cat <<EOF | kubectl apply -f-
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-sa
  namespace: ${NAMESPACE} 
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-cluster-role
rules:
- apiGroups:
  - '*'
  resources:
  - '*'
  verbs:
  - '*'
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-cluster-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-cluster-role
subjects:
- kind: ServiceAccount
  name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-sa
  namespace: ${NAMESPACE}
EOF

PackageInstallを作成してPackageをインストール

次のPackageInstall CRを作成してJenkins Packageをインストールします。 設定項目はSecretに設定できます。必要に応じて変更してください。

cat <<EOF | kubectl apply -f-
apiVersion: packaging.carvel.dev/v1alpha1
kind: PackageInstall
metadata:
  name: ${PACKAGE_INSTALL_NAME}
  namespace: ${NAMESPACE}
  annotations:
    tkg.tanzu.vmware.com/tanzu-package-ClusterRole: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-cluster-role
    tkg.tanzu.vmware.com/tanzu-package-ClusterRoleBinding: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-cluster-rolebinding
    tkg.tanzu.vmware.com/tanzu-package-Secret: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-values
    tkg.tanzu.vmware.com/tanzu-package-ServiceAccount: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-sa
spec:
  serviceAccountName: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-sa
  packageRef:
    refName: jenkins.pkg.maki.lol
    versionSelection:
      constraints: 0.0.1
  values:
  - secretRef:
      name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-values
---
apiVersion: v1
kind: Secret
metadata:
  name: ${PACKAGE_INSTALL_NAME}-${NAMESPACE}-values
  namespace: ${NAMESPACE}
stringData:
  values.yaml: |
    namespace: demo
    jenkins:
      host: jenkins.demo.tiger.maki.lol
      username: admin
      password: password
  EOF

annotationsについているtkg.tanzu.vmware.com/tanzu-package-*は必須ではありませんが、 PackageInstall CRをtanzu package installed delete コマンドで削除するときに、一緒にClusterRole, ClusterRoleBinding, Secret, ServiceAccountを削除してくれるようになるので便利です。

インストールの結果は次のコマンドで確認できます。 Reconcile succeededになっているのでインストール成功です。

$ kubectl get packageinstall -n pkg-install
NAME      PACKAGE NAME           PACKAGE VERSION   DESCRIPTION           AGE
jenkins   jenkins.pkg.maki.lol   0.0.2             Reconcile succeeded   18m

インストールの詳細は次のコマンドで確認できます。

$ kubectl get app -n pkg-install jenkins -oyaml                                  
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  creationTimestamp: "2021-12-16T02:27:22Z"
  finalizers:
  - finalizers.kapp-ctrl.k14s.io/delete
  generation: 1
  name: jenkins
  namespace: pkg-install
  ownerReferences:
  - apiVersion: packaging.carvel.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PackageInstall
    name: jenkins
    uid: b4a29810-ae24-43e3-81c2-d0daa3432964
  resourceVersion: "17394696"
  uid: 31dca596-60a9-4cfc-b098-881d9f6a4eb0
spec:
  deploy:
  - kapp:
      inspect:
        rawOptions:
        - --tree=true
      rawOptions:
      - --wait-timeout=5m
      - --diff-changes=true
  fetch:
  - imgpkgBundle:
      image: ghcr.io/making/jenkins-bundle:0.0.1
  serviceAccountName: jenkins-pkg-install-sa
  template:
  - ytt:
      valuesFrom:
      - secretRef:
          name: jenkins-pkg-install-values
  - kbld:
      paths:
      - '-'
      - .imgpkg/images.yml
status:
  conditions:
  - status: "True"
    type: ReconcileSucceeded
  consecutiveReconcileSuccesses: 7
  deploy:
    exitCode: 0
    finished: true
    startedAt: "2021-12-16T02:33:13Z"
    stdout: |-
      Target cluster 'https://100.64.0.1:443' (nodes: tiger-control-plane-k9ns7, 1+)
      02:33:13AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreaclusternetworkpolicystats"}
      02:33:13AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreanetworkpolicystats"}
      02:33:13AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"networkpolicystats"}
      Changes
      Namespace  Name  Kind  Conds.  Age  Op  Op st.  Wait to  Rs  Ri
      Op:      0 create, 0 delete, 0 update, 0 noop
      Wait to: 0 reconcile, 0 delete, 0 noop
      Succeeded
    updatedAt: "2021-12-16T02:33:14Z"
  fetch:
    exitCode: 0
    startedAt: "2021-12-16T02:33:11Z"
    stdout: |
      apiVersion: vendir.k14s.io/v1alpha1
      directories:
      - contents:
        - imgpkgBundle:
            image: ghcr.io/making/jenkins-bundle@sha256:25eb08343d1c290b5dd89c0e91c91aeba2cc5117c8601c303b2f7db6a6f66b6c
          path: .
        path: "0"
      kind: LockConfig
    updatedAt: "2021-12-16T02:33:13Z"
  friendlyDescription: Reconcile succeeded
  inspect:
    exitCode: 0
    stdout: |-
      Target cluster 'https://100.64.0.1:443' (nodes: tiger-control-plane-k9ns7, 1+)
      02:33:15AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreaclusternetworkpolicystats"}
      02:33:15AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"networkpolicystats"}
      02:33:15AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreanetworkpolicystats"}
      Resources in app 'jenkins-ctrl'
      Namespace  Name                           Kind                   Owner    Conds.  Rs  Ri  Age
      demo       jenkins                        Service                kapp     -       ok  -   5m
      demo        L jenkins-2pm54               EndpointSlice          cluster  -       ok  -   5m
      demo        L jenkins                     Endpoints              cluster  -       ok  -   5m
      demo       jenkins                        Deployment             kapp     2/2 t   ok  -   5m
      demo        L jenkins-5cd988d48b          ReplicaSet             cluster  -       ok  -   5m
      demo        L.. jenkins-5cd988d48b-mlqxf  Pod                    cluster  4/4 t   ok  -   5m
      demo       jenkins                        Secret                 kapp     -       ok  -   5m
      (cluster)  demo                           Namespace              kapp     -       ok  -   5m
      demo       jenkins                        PersistentVolumeClaim  kapp     -       ok  -   5m
      demo       jenkins                        Ingress                kapp     -       ok  -   5m
      Rs: Reconcile state
      Ri: Reconcile information
      10 resources
      Succeeded
    updatedAt: "2021-12-16T02:33:15Z"
  observedGeneration: 1
  template:
    exitCode: 0
    stderr: |
      resolve | final: docker.io/bitnami/jenkins:2.303.1-debian-10-r38 -> index.docker.io/bitnami/jenkins@sha256:50fc193ac61f9be76d8aaf1f46a40b0433af1fd3308a5a5ec7fadd89b859f868
    updatedAt: "2021-12-16T02:33:13Z"

jenkins.hostに設定したホスト名にアクセスし、jenkins.username / jenkins.passwordのアカウントでログインできることを確認します。

image

image

tanzu package installコマンドでもインストールできることを確認したいので、いったん作成したリソースを削除します。

kubectl delete packageinstal -n pkg-install jenkins
kubectl delete clusterrolebinding jenkins-pkg-install-cluster-rolebinding
kubectl delete clusterrole jenkins-pkg-install-cluster-role
kubectl delete sa -n pkg-install jenkins-pkg-install-sa

なお、PackageInstall CRにannotationsを設定したので、次のコマンドでもまとめて削除できます。

tanzu package installed delete -n pkg-install jenkins -y

tanzu CLIでPackageをインストール

次のコマンドでtanzu CLIでJenkinsがインストール可能なことを確認します。

$ tanzu package available list -n pkg-install jenkins.pkg.maki.lol
- Retrieving package versions for jenkins.pkg.maki.lol...
  NAME                  VERSION  RELEASED-AT                    
  jenkins.pkg.maki.lol  0.0.1    0001-01-01 00:00:00 +0000 UTC

次のコマンドでJenkins Packageをインストールします。

cat <<EOF > jenkins-data-values.yaml
---
namespace: demo
jenkins:
  host: jenkins.demo.tiger.maki.lol
  username: admin
  password: password
EOF

tanzu package install jenkins -p jenkins.pkg.maki.lol -v 0.0.1 -n pkg-install -f jenkins-data-values.yaml

次のコマンドでインストール結果を確認できます。

$ tanzu package installed list -n pkg-install
/ Retrieving installed packages...
NAME     PACKAGE-NAME          PACKAGE-VERSION  STATUS               
jenkins  jenkins.pkg.maki.lol  0.0.1            Reconcile succeeded

詳細の確認や動作確認は前と同じです。

Package version 0.0.2の作成

version 0.0.1では最小限の機能のみ提供したので、0.0.2ではTLS対応を行います。

0.0.1用のディレクトリを0.0.2用へコピーします。

cp -r /tmp/addons/packages/jenkins/0.0.1 /tmp/addons/packages/jenkins/0.0.2
cd /tmp/addons/packages/jenkins/0.0.2

cert-mangerのIssuer作成とIngressへの設定を変更するoverlayを次のように作成します。

cat <<EOF > bundle/config/overlay/use-cert-manager.yaml
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:data", "data")

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: jenkins-selfsigned-issuer
  namespace: #@ data.values.namespace
spec:
  selfSigned: { }
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: jenkins-ca
  namespace: #@ data.values.namespace
spec:
  commonName: jenkins-ca
  isCA: true
  issuerRef:
    kind: Issuer
    name: jenkins-selfsigned-issuer
  secretName: jenkins-ca
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: jenkins-ca-issuer
  namespace: #@ data.values.namespace
spec:
  ca:
    secretName: jenkins-ca
#@overlay/match by=overlay.subset({"kind":"Ingress","metadata":{"name":"jenkins"}})
---
metadata:
  #@overlay/match missing_ok=True
  #@overlay/match-child-defaults missing_ok=True  
  annotations:
    cert-manager.io/issuer: "jenkins-ca-issuer"
spec:
  #@overlay/match missing_ok=True
  tls:
  - hosts:
    - #@ data.values.jenkins.host
    secretName: jenkins-tls
EOF

overlayが期待通りに動作するか次のコマンドで確認します。

ytt -f bundle/config  

次のYAMLが出力されることを確認します。

apiVersion: v1
kind: Namespace
metadata:
  name: demo
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: jenkins-selfsigned-issuer
  namespace: demo
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: jenkins-ca
  namespace: demo
spec:
  commonName: jenkins-ca
  isCA: true
  issuerRef:
    kind: Issuer
    name: jenkins-selfsigned-issuer
  secretName: jenkins-ca
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: jenkins-ca-issuer
  namespace: demo
spec:
  ca:
    secretName: jenkins-ca
---
apiVersion: v1
kind: Secret
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
type: Opaque
stringData:
  jenkins-password: password
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
  annotations:
    volume.alpha.kubernetes.io/storage-class: default
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  type: ClusterIP
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: http
    nodePort: null
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
    nodePort: null
  selector:
    app.kubernetes.io/name: jenkins
    app.kubernetes.io/instance: jenkins
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: jenkins
      app.kubernetes.io/instance: jenkins
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app.kubernetes.io/name: jenkins
        helm.sh/chart: jenkins-8.0.14
        app.kubernetes.io/instance: jenkins
        app.kubernetes.io/managed-by: Helm
      annotations:
        checksum/secrets: ecab545c66cdd482057d3f3e5e8ea8c093a44452e3462f99b7e347a9bba48f07
    spec:
      affinity:
        podAffinity: null
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: jenkins
                  app.kubernetes.io/instance: jenkins
              namespaces:
              - demo
              topologyKey: kubernetes.io/hostname
            weight: 1
        nodeAffinity: null
      securityContext:
        fsGroup: 1001
      containers:
      - name: jenkins
        image: docker.io/bitnami/jenkins:2.303.1-debian-10-r38
        imagePullPolicy: IfNotPresent
        securityContext:
          runAsNonRoot: true
          runAsUser: 1001
        env:
        - name: JENKINS_USERNAME
          value: admin
        - name: JENKINS_PASSWORD
          valueFrom:
            secretKeyRef:
              name: jenkins
              key: jenkins-password
        - name: JENKINS_HOME
          value: /bitnami/jenkins/home
        - name: DISABLE_JENKINS_INITIALIZATION
          value: "no"
        - name: JENKINS_HOST
          value: jenkins.example.com
        - name: JENKINS_EXTERNAL_HTTP_PORT_NUMBER
          value: "80"
        - name: JENKINS_EXTERNAL_HTTPS_PORT_NUMBER
          value: "443"
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        - name: https
          containerPort: 8443
          protocol: TCP
        livenessProbe:
          httpGet:
            path: /login
            port: http
          initialDelaySeconds: 180
          periodSeconds: 10
          timeoutSeconds: 5
          failureThreshold: 6
          successThreshold: 1
        readinessProbe:
          httpGet:
            path: /login
            port: http
          initialDelaySeconds: 30
          periodSeconds: 5
          timeoutSeconds: 3
          failureThreshold: 3
          successThreshold: 1
        resources:
          limits: {}
          requests:
            cpu: 300m
            memory: 512Mi
        volumeMounts:
        - name: jenkins-data
          mountPath: /bitnami/jenkins
      volumes:
      - name: jenkins-data
        persistentVolumeClaim:
          claimName: jenkins
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins
  namespace: demo
  labels:
    app.kubernetes.io/name: jenkins
    helm.sh/chart: jenkins-8.0.14
    app.kubernetes.io/instance: jenkins
    app.kubernetes.io/managed-by: Helm
  annotations:
    cert-manager.io/issuer: jenkins-ca-issuer
spec:
  rules:
  - host: jenkins.example.com
    http:
      paths:
      - path: /
        pathType: ImplementationSpecific
        backend:
          service:
            name: jenkins
            port:
              name: http
  tls:
  - hosts:
    - jenkins.example.com
    secretName: jenkins-tls

今回は必要ないですが、image:の追加や変更があった場合は次のコマンドでlockファイルを更新します。

kbld -f bundle/config --imgpkg-lock-output bundle/.imgpkg/images.yml

imgpkg bundleを作成します。

imgpkg push -f bundle -b ghcr.io/making/jenkins-bundle:0.0.2

次のログを出力します。

dir: .
dir: .imgpkg
file: .imgpkg/images.yml
dir: config
dir: config/overlay
file: config/overlay/change-hostname.yaml
file: config/overlay/change-user.yaml
file: config/overlay/namespace.yaml
file: config/overlay/use-cert-manager.yaml
dir: config/upstream
file: config/upstream/jenkins.yaml
file: config/values.yaml
Pushed 'ghcr.io/making/jenkins-bundle@sha256:1262f1680dd02425461d13b70e5b75101c942529395d02024a040780a89153fa'
Succeeded

0.0.2用のPackageを作成します。

cat <<EOF > /tmp/addons/packages/jenkins/0.0.2/package.yml
apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: jenkins.pkg.maki.lol.0.0.2
  namespace: pkg-install
spec:
  refName: jenkins.pkg.maki.lol
  version: 0.0.2
  releaseNotes: |
    Support cert-manager
  template:
    spec:
      fetch:
      - imgpkgBundle:
          image: ghcr.io/making/jenkins-bundle:0.0.2
      template:
      - ytt: {}
      - kbld:
          paths:
          - "-"
          - ".imgpkg/images.yml"
      deploy:
      - kapp:
          rawOptions:
          - --diff-changes=true
          inspect:
            rawOptions:
            - --tree=true          
EOF
kubectl apply -f /tmp/addons/packages/jenkins/0.0.2/package.yml

次のコマンド0.0.2が利用可能になったことを確認できます。

$ tanzu package available list -n pkg-install jenkins.pkg.maki.lol
- Retrieving package versions for jenkins.pkg.maki.lol... 
  NAME                  VERSION  RELEASED-AT                    
  jenkins.pkg.maki.lol  0.0.1    0001-01-01 00:00:00 +0000 UTC  
  jenkins.pkg.maki.lol  0.0.2    0001-01-01 00:00:00 +0000 UTC 

次のコマンドで既にインストール済みのJenkins Pacakge 0.0.1を0.0.2にアップデートします。

tanzu package installed update -n pkg-install jenkins -v 0.0.2

次のログが出力されます。

| Updating installed package 'jenkins' 
/ Getting package install for 'jenkins' 
- Getting package metadata for 'jenkins.pkg.maki.lol' 
/ Updating package install for 'jenkins' 
\ Waiting for 'PackageInstall' reconciliation for 'jenkins' 
/ 'PackageInstall' resource install status: Reconciling 


Updated installed package 'jenkins' in namespace 'pkg-install'

インストールの詳細は次のコマンドで確認できます。Package CRでkappの設定に--diff-changes=trueを設定したので、アップデートによる変更差分が見れます。

$ kubectl get app -n pkg-install jenkins -oyaml
apiVersion: kappctrl.k14s.io/v1alpha1
kind: App
metadata:
  creationTimestamp: "2021-12-16T02:27:22Z"
  finalizers:
  - finalizers.kapp-ctrl.k14s.io/delete
  generation: 2
  name: jenkins
  namespace: pkg-install
  ownerReferences:
  - apiVersion: packaging.carvel.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: PackageInstall
    name: jenkins
    uid: b4a29810-ae24-43e3-81c2-d0daa3432964
  resourceVersion: "17403186"
  uid: 31dca596-60a9-4cfc-b098-881d9f6a4eb0
spec:
  deploy:
  - kapp:
      inspect:
        rawOptions:
        - --tree=true
      rawOptions:
      - --diff-changes=true
  fetch:
  - imgpkgBundle:
      image: ghcr.io/making/jenkins-bundle:0.0.2
  serviceAccountName: jenkins-pkg-install-sa
  template:
  - ytt:
      valuesFrom:
      - secretRef:
          name: jenkins-pkg-install-values
  - kbld:
      paths:
      - '-'
      - .imgpkg/images.yml
status:
  conditions:
  - status: "True"
    type: ReconcileSucceeded
  consecutiveReconcileSuccesses: 64
  deploy:
    exitCode: 0
    finished: true
    startedAt: "2021-12-16T03:08:32Z"
    stdout: |-
      Target cluster 'https://100.64.0.1:443' (nodes: tiger-control-plane-k9ns7, 1+)
      03:08:32AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreaclusternetworkpolicystats"}
      03:08:32AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreanetworkpolicystats"}
      03:08:32AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"networkpolicystats"}
      @@ create issuer/jenkins-selfsigned-issuer (cert-manager.io/v1) namespace: demo @@
            0 + apiVersion: cert-manager.io/v1
            1 + kind: Issuer
            2 + metadata:
            3 +   labels:
            4 +     kapp.k14s.io/app: "1639621645860660653"
            5 +     kapp.k14s.io/association: v1.96f4e0c9cd1a447f316b95e62dfbc4e7
            6 +   name: jenkins-selfsigned-issuer
            7 +   namespace: demo
            8 + spec:
            9 +   selfSigned: {}
           10 +
      @@ create certificate/jenkins-ca (cert-manager.io/v1) namespace: demo @@
            0 + apiVersion: cert-manager.io/v1
            1 + kind: Certificate
            2 + metadata:
            3 +   labels:
            4 +     kapp.k14s.io/app: "1639621645860660653"
            5 +     kapp.k14s.io/association: v1.dca00bd36fa8d29a87d51b5263d4aec1
            6 +   name: jenkins-ca
            7 +   namespace: demo
            8 + spec:
            9 +   commonName: jenkins-ca
           10 +   isCA: true
           11 +   issuerRef:
           12 +     kind: Issuer
           13 +     name: jenkins-selfsigned-issuer
           14 +   secretName: jenkins-ca
           15 +
      @@ create issuer/jenkins-ca-issuer (cert-manager.io/v1) namespace: demo @@
            0 + apiVersion: cert-manager.io/v1
            1 + kind: Issuer
            2 + metadata:
            3 +   labels:
            4 +     kapp.k14s.io/app: "1639621645860660653"
            5 +     kapp.k14s.io/association: v1.496b486e1a077b8c00761f87b7d26e74
            6 +   name: jenkins-ca-issuer
            7 +   namespace: demo
            8 + spec:
            9 +   ca:
           10 +     secretName: jenkins-ca
           11 +
      @@ update ingress/jenkins (networking.k8s.io/v1) namespace: demo @@
        ...
        2,  2   metadata:
            3 +   annotations:
            4 +     cert-manager.io/issuer: jenkins-ca-issuer
        3,  5     creationTimestamp: "2021-12-16T02:27:27Z"
        4,  6     generation: 2
        ...
       59, 61           pathType: ImplementationSpecific
           62 +   tls:
           63 +   - hosts:
           64 +     - jenkins.demo.tiger.maki.lol
           65 +     secretName: jenkins-tls
       60, 66   status:
       61, 67     loadBalancer:
      Changes
      Namespace  Name                       Kind         Conds.  Age  Op      Op st.  Wait to    Rs  Ri
      demo       jenkins                    Ingress      -       41m  update  -       reconcile  ok  -
      ^          jenkins-ca                 Certificate  -       -    create  -       reconcile  -   -
      ^          jenkins-ca-issuer          Issuer       -       -    create  -       reconcile  -   -
      ^          jenkins-selfsigned-issuer  Issuer       -       -    create  -       reconcile  -   -
      Op:      3 create, 0 delete, 1 update, 0 noop
      Wait to: 4 reconcile, 0 delete, 0 noop
      3:08:33AM: ---- applying 4 changes [0/4 done] ----
      3:08:34AM: update ingress/jenkins (networking.k8s.io/v1) namespace: demo
      3:08:34AM: create issuer/jenkins-selfsigned-issuer (cert-manager.io/v1) namespace: demo
      3:08:34AM: create certificate/jenkins-ca (cert-manager.io/v1) namespace: demo
      3:08:35AM: create issuer/jenkins-ca-issuer (cert-manager.io/v1) namespace: demo
      3:08:35AM: ---- waiting on 4 changes [0/4 done] ----
      3:08:35AM: ok: reconcile issuer/jenkins-ca-issuer (cert-manager.io/v1) namespace: demo
      3:08:35AM: ok: reconcile certificate/jenkins-ca (cert-manager.io/v1) namespace: demo
      3:08:35AM: ok: reconcile issuer/jenkins-selfsigned-issuer (cert-manager.io/v1) namespace: demo
      3:08:35AM: ok: reconcile ingress/jenkins (networking.k8s.io/v1) namespace: demo
      3:08:35AM: ---- applying complete [4/4 done] ----
      3:08:35AM: ---- waiting complete [4/4 done] ----
      Succeeded
    updatedAt: "2021-12-16T03:08:35Z"
  fetch:
    exitCode: 0
    startedAt: "2021-12-16T03:08:28Z"
    stdout: |
      apiVersion: vendir.k14s.io/v1alpha1
      directories:
      - contents:
        - imgpkgBundle:
            image: ghcr.io/making/jenkins-bundle@sha256:1262f1680dd02425461d13b70e5b75101c942529395d02024a040780a89153fa
          path: .
        path: "0"
      kind: LockConfig
    updatedAt: "2021-12-16T03:08:32Z"
  friendlyDescription: Reconcile succeeded
  inspect:
    exitCode: 0
    stdout: |-
      Target cluster 'https://100.64.0.1:443' (nodes: tiger-control-plane-k9ns7, 1+)
      03:08:36AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"networkpolicystats"}
      03:08:36AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreaclusternetworkpolicystats"}
      03:08:36AM: info: Resources: Ignoring group version: schema.GroupVersionResource{Group:"stats.antrea.tanzu.vmware.com", Version:"v1alpha1", Resource:"antreanetworkpolicystats"}
      Resources in app 'jenkins-ctrl'
      Namespace  Name                         Kind                   Owner    Conds.  Rs  Ri  Age
      demo       jenkins                      Service                kapp     -       ok  -   41m
      demo        L jenkins-2pm54             EndpointSlice          cluster  -       ok  -   41m
      demo        L jenkins                   Endpoints              cluster  -       ok  -   41m
      demo       jenkins                      Deployment             kapp     2/2 t   ok  -   41m
      demo        L jenkins-5cd988d48b        ReplicaSet             cluster  -       ok  -   41m
      demo        L jenkins-97d968b8          ReplicaSet             cluster  -       ok  -   33m
      demo        L.. jenkins-97d968b8-jkdpw  Pod                    cluster  4/4 t   ok  -   33m
      demo       jenkins                      Secret                 kapp     -       ok  -   41m
      demo       jenkins-ca-issuer            Issuer                 kapp     1/1 t   ok  -   2s
      (cluster)  demo                         Namespace              kapp     -       ok  -   41m
      demo       jenkins-selfsigned-issuer    Issuer                 kapp     1/1 t   ok  -   2s
      demo       jenkins                      PersistentVolumeClaim  kapp     -       ok  -   41m
      demo       jenkins-ca                   Certificate            kapp     1/1 t   ok  -   2s
      demo        L jenkins-ca-9xsg2          CertificateRequest     cluster  1/1 t   ok  -   1s
      demo       jenkins                      Ingress                kapp     -       ok  -   41m
      demo        L jenkins-tls               Certificate            cluster  1/1 t   ok  -   2s
      demo        L.. jenkins-tls-lt4qm       CertificateRequest     cluster  1/1 t   ok  -   1s
      Rs: Reconcile state
      Ri: Reconcile information
      17 resources
      Succeeded
    updatedAt: "2021-12-16T03:08:36Z"
  observedGeneration: 2
  template:
    exitCode: 0
    stderr: |
      resolve | final: docker.io/bitnami/jenkins:2.303.1-debian-10-r38 -> index.docker.io/bitnami/jenkins@sha256:50fc193ac61f9be76d8aaf1f46a40b0433af1fd3308a5a5ec7fadd89b859f868
    updatedAt: "2021-12-16T03:08:32Z"

JenkinsにHTTPSでアクセスできることを確認します。

image

次のコマンドでJenkinsパッケージをアンインストールできます。

$ tanzu package installed delete -n pkg-install jenkins -y
| Uninstalling package 'jenkins' from namespace 'pkg-install' 
| Getting package install for 'jenkins' 
- Deleting package install 'jenkins' from namespace 'pkg-install' 
/ 'PackageInstall' resource deletion status: Deleting 
/ Deleting admin role 'jenkins-pkg-install-cluster-role' 
/ Deleting role binding 'jenkins-pkg-install-cluster-rolebinding' 
/ Deleting service account 'jenkins-pkg-install-sa' 
Uninstalled package 'jenkins' from namespace 'pkg-install'

Values Schemaの設定

Carvel Packageに設定な可能なvaluesのschemaをPackage CRに定義することができます。この定義がある場合は次のようなコマンドでどそのPackageに対してどのような設定が可能なのかを確認できます。

tanzu package available get -n pkg-install jenkins.pkg.maki.lol/0.0.2 --values-schema

今回はschemaの定義をしていなかったので、次のようなエラーが返ります。

Error: data value schema is empty
Error: exit status 1

✖  exit status 1

Package CRにschemaを定義します。schemaはOpenAPI Formatで記述します。

次のコマンドを実行し、Jenkin Package 0.0.2を更新します。

cat <<EOF > /tmp/addons/packages/jenkins/0.0.2/package.yml
apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: jenkins.pkg.maki.lol.0.0.2
  namespace: pkg-install
spec:
  refName: jenkins.pkg.maki.lol
  version: 0.0.2
  releaseNotes: |
    Support cert-manager
  template:
    spec:
      fetch:
      - imgpkgBundle:
          image: ghcr.io/making/jenkins-bundle:0.0.2
      template:
      - ytt: {}
      - kbld:
          paths:
          - "-"
          - ".imgpkg/images.yml"
      deploy:
      - kapp:
          rawOptions:
          - --diff-changes=true
          inspect:
            rawOptions:
            - --tree=true
  valuesSchema:
    openAPIv3:
      type: object
      additionalProperties: false
      properties:
        namespace:
          type: string
          default: demo
          description: Namespace to install the Jenkins
        create_namespace:
          type: boolean
          default: true
          description: Whether to create the namespace
        jenkins:
          type: object
          additionalProperties: false
          properties:
            host:
              type: string
              default: jenkins.example.com
              description: Jenkins hostname
            username:
              type: string
              default: admin
              description: Jenkins username
            password:
              type: string
              default: password
              description: Jenkins password            
EOF
kubectl apply -f /tmp/addons/packages/jenkins/0.0.2/package.yml

tanzu package available getコマンドでvalues schemaが出力されることを確認します。

$ tanzu package available get -n pkg-install jenkins.pkg.maki.lol/0.0.2 --values-schema                                                            
| Retrieving package details for jenkins.pkg.maki.lol/0.0.2... 
  KEY               DEFAULT              TYPE     DESCRIPTION                       
  create_namespace  true                 boolean  Whether to create the namespace   
  jenkins.host      jenkins.example.com  string   Jenkins hostname                  
  jenkins.password  password             string   Jenkins password                  
  jenkins.username  admin                string   Jenkins username                  
  namespace         demo                 string   Namespace to install the Jenkins  

Values Schemaの自動生成

bundle内のvalues.yamlにデフォルト値を定義しつつ、同じ内容をPackage CRのspec.valuesSchema.openAPIv3にも定義するのは二度手間だと感じるかもしれません。

ytt 0.38.0にてvalues.yamlから OpenAPI Formatをエクスポートする機能 がサポートされました。 この機能を使うと二度手間をなくせます。

values.yamlを次のように更新します。各valueのdescriptionはvalues.yamlに記述します。

cat <<EOF > bundle/config/values.yaml
#@data/values-schema
---
#@schema/desc "Namespace to install the Jenkins"
namespace: demo
#@schema/desc "Whether to create the namespace"
create_namespace: True
jenkins:
  #@schema/desc "Jenkins hostname"
  host: jenkins.example.com
  #@schema/desc "Jenkins username"
  username: admin
  #@schema/desc "Jenkins password"
  password: password
EOF

次のコマンドでOpenAPI Formatを生成します。

ytt -f bundle/config/values.yaml --data-values-schema-inspect -o openapi-v3 > /tmp/schema-openapi.yml

次のYAMLが生成されます。

openapi: 3.0.0
info:
  version: 0.1.0
  title: Schema for data values, generated by ytt
paths: {}
components:
  schemas:
    dataValues:
      type: object
      additionalProperties: false
      properties:
        namespace:
          type: string
          default: demo
          description: Namespace to install the Jenkins
        create_namespace:
          type: boolean
          default: true
          description: Whether to create the namespace
        jenkins:
          type: object
          additionalProperties: false
          properties:
            host:
              type: string
              default: jenkins.example.com
              description: Jenkins hostname
            username:
              type: string
              default: admin
              description: Jenkins username
            password:
              type: string
              default: password
              description: Jenkins password

生成されたOpen API FormatのYAMLのcomponents.schemas.dataValuesをPackage CRの定義に使えるように、 次のテンプレートを作成します。

cat <<EOF > /tmp/addons/packages/jenkins/package-template.yaml
#@ load("@ytt:data", "data")
#@ load("@ytt:yaml", "yaml")
apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: #@ "jenkins.pkg.maki.lol.{}".format(data.values.version)
  namespace: pkg-install
spec:
  refName: jenkins.pkg.maki.lol
  version: #@ data.values.version
  releaseNotes: |
    Support cert-manager
  template:
    spec:
      fetch:
      - imgpkgBundle:
          image: #@ "ghcr.io/making/jenkins-bundle:{}".format(data.values.version)
      template:
      - ytt: {}
      - kbld:
          paths:
          - "-"
          - ".imgpkg/images.yml"
      deploy:
      - kapp:
          rawOptions:
          - --diff-changes=true
          inspect:
            rawOptions:
            - --tree=true
  valuesSchema:
    openAPIv3: #@ yaml.decode(data.values.openapi)["components"]["schemas"]["dataValues"]                     
EOF

このTemplateを使用して、version 0.0.2用のPackageを改めて作成します。

PACKAGE_VERSION=0.0.2
ytt -f ../package-template.yaml --data-value-file openapi=/tmp/schema-openapi.yml -v version=${PACKAGE_VERSION} > package.yml

次のようなYAMLが生成されます。

apiVersion: data.packaging.carvel.dev/v1alpha1
kind: Package
metadata:
  name: jenkins.pkg.maki.lol.0.0.2
  namespace: pkg-install
spec:
  refName: jenkins.pkg.maki.lol
  version: 0.0.2
  releaseNotes: |
    Support cert-manager
  template:
    spec:
      fetch:
      - imgpkgBundle:
          image: ghcr.io/making/jenkins-bundle:0.0.2
      template:
      - ytt: {}
      - kbld:
          paths:
          - '-'
          - .imgpkg/images.yml
      deploy:
      - kapp:
          rawOptions:
          - --diff-changes=true
          inspect:
            rawOptions:
            - --tree=true
  valuesSchema:
    openAPIv3:
      type: object
      additionalProperties: false
      properties:
        namespace:
          type: string
          default: demo
          description: Namespace to install the Jenkins
        create_namespace:
          type: boolean
          default: true
          description: Whether to create the namespace
        jenkins:
          type: object
          additionalProperties: false
          properties:
            host:
              type: string
              default: jenkins.example.com
              description: Jenkins hostname
            username:
              type: string
              default: admin
              description: Jenkins username
            password:
              type: string
              default: password
              description: Jenkins password

このPackageを登録します。

kubectl apply -f package.yml

tanzu package available getコマンドでvalues schemaが出力されることを確認します。

$ tanzu package available get -n pkg-install jenkins.pkg.maki.lol/0.0.2 --values-schema                                        
| Retrieving package details for jenkins.pkg.maki.lol/0.0.2... 
  KEY               DEFAULT              TYPE     DESCRIPTION                       
  create_namespace  true                 boolean  Whether to create the namespace   
  jenkins.host      jenkins.example.com  string   Jenkins hostname                  
  jenkins.password  password             string   Jenkins password                  
  jenkins.username  admin                string   Jenkins username                  
  namespace         demo                 string   Namespace to install the Jenkins