Tanzu Application PlatformにWorkloadをデプロイする際にGatekeeperでresourcesを設定を強制するメモ

️⚠️ 本記事の内容はVMwareによってサポートされていません。記事の内容で生じた問題については自己責任で対応し、 VMwareサポート窓口には問い合わせないでください。

TAPにWorkloadをデプロイする際に、resourcesを指定しないとNodeのcpu, memoryを限界以上にアプリがデプロイされる可能性があるので、 resourcesの設定は推奨。これを強制するためにGatekeeperを使用する。

Gatekeeperのインストール
ConstraintTemplateの作成
Constraintの作成

Gatekeeperのインストール

ドキュメントの通りにインストール

kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml

Podの確認

$ kubectl get pod -n gatekeeper-system
NAME                                             READY   STATUS    RESTARTS   AGE
gatekeeper-audit-84b788f7c5-54m2r                1/1     Running   0          13s
gatekeeper-controller-manager-76dc9cbc64-2q26t   1/1     Running   0          13s
gatekeeper-controller-manager-76dc9cbc64-fx5m8   1/1     Running   0          13s
gatekeeper-controller-manager-76dc9cbc64-qb52w   1/1     Running   0          13s

ConstraintTemplateの作成

とりあえず spec.resources.{limits,requests}.memory の設定を必須とするConstraintTemplate。cpuも必須にしたい場合はコメントを外す

cat <<EOF > requiredresources.yaml
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: requiredresources
spec:
  crd:
    spec:
      names:
        kind: RequiredResources
  targets:
  - target: admission.k8s.gatekeeper.sh
    rego: |
      package requiredresources

      missing(obj, field) = true {
        not obj[field]
      }

      missing(obj, field) = true {
        obj[field] == ""
      }
      violation[{"msg": msg, "details": {}}] {
          missing(input.review.object.spec, "resources")
          msg := "spec.resources is missing."
      }
      violation[{"msg": msg, "details": {}}] {
          missing(input.review.object.spec.resources, "limits")
          msg := "spec.resources.limits is missing."
      }
      violation[{"msg": msg, "details": {}}] {
          missing(input.review.object.spec.resources, "requests")
          msg := "spec.resources.requests is missing."
      }
      # violation[{"msg": msg, "details": {}}] {
      #     missing(input.review.object.spec.resources.limits, "cpu")
      #     msg := "spec.resources.limits.cpu is missing."
      # }
      violation[{"msg": msg, "details": {}}] {
          missing(input.review.object.spec.resources.limits, "memory")
          msg := "spec.resources.limits.memory is missing."
      }
      # violation[{"msg": msg, "details": {}}] {
      #     missing(input.review.object.spec.resources.requests, "cpu")
      #     msg := "spec.resources.requests.cpu is missing."
      # }
      violation[{"msg": msg, "details": {}}] {
          missing(input.review.object.spec.resources.requests, "memory")
          msg := "spec.resources.requests.memory is missing."
      }
EOF

kubectl apply -f requiredresources.yaml

Constraintの作成

まずはdryrun modeで試す

cat <<EOF > requiredresources-workload.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequiredResources
metadata:
  name: requiredresources-workload
spec:
  enforcementAction: dryrun
  match:
    kinds:
    - apiGroups:
      - carto.run
      kinds:
      - Workload
EOF

kubectl apply -f requiredresources-workload.yaml

この状態だとまだresourcesの設定なしでもWorkloadを作成できる

$ tanzu apps workload apply hello-nodejs \
  --app hello-nodejs \
  --git-repo https://github.com/making/hello-nodejs \
  --git-branch master \
  --type web \
  -n demo \
  -y
Create workload:
      1 + |---
      2 + |apiVersion: carto.run/v1alpha1
      3 + |kind: Workload
      4 + |metadata:
      5 + |  labels:
      6 + |    app.kubernetes.io/part-of: hello-nodejs
      7 + |    apps.tanzu.vmware.com/workload-type: web
      8 + |  name: hello-nodejs
      9 + |  namespace: demo
     10 + |spec:
     11 + |  source:
     12 + |    git:
     13 + |      ref:
     14 + |        branch: master
     15 + |      url: https://github.com/making/hello-nodejs

Created workload "hello-nodejs"

To see logs:   "tanzu apps workload tail hello-nodejs --namespace demo"
To get status: "tanzu apps workload get hello-nodejs --namespace demo"

制約違反があるかどうかは次のようにチェックできる。

$ kubectl get requiredresources requiredresources-workload -ojsonpath='{.status.violations}' | jq .
[
  {
    "enforcementAction": "dryrun",
    "group": "carto.run",
    "kind": "Workload",
    "message": "spec.resources is missing.",
    "name": "hello-nodejs",
    "namespace": "demo",
    "version": "v1alpha1"
  }

dryrunをやめて強制モードに変更

cat <<EOF > requiredresources-workload.yaml
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: RequiredResources
metadata:
  name: requiredresources-workload
spec:
  enforcementAction: deny
  match:
    kinds:
    - apiGroups:
      - carto.run
      kinds:
      - Workload
EOF

kubectl apply -f requiredresources-workload.yaml

Workloadを一度削除する

tanzu apps workload delete -n demo hello-nodejs -y

再びWorkloadを作成しようとするとエラーになる

$ tanzu apps workload apply hello-nodejs \
  --app hello-nodejs \
  --git-repo https://github.com/making/hello-nodejs \
  --git-branch master \
  --type web \
  -n demo \
  -y
Create workload:
      1 + |---
      2 + |apiVersion: carto.run/v1alpha1
      3 + |kind: Workload
      4 + |metadata:
      5 + |  labels:
      6 + |    app.kubernetes.io/part-of: hello-nodejs
      7 + |    apps.tanzu.vmware.com/workload-type: web
      8 + |  name: hello-nodejs
      9 + |  namespace: demo
     10 + |spec:
     11 + |  source:
     12 + |    git:
     13 + |      ref:
     14 + |        branch: master
     15 + |      url: https://github.com/making/hello-nodejs

Error: Unable to complete command, you do not have permissions for this resource: admission webhook "validation.gatekeeper.sh" denied the request: [requiredresources-workload] spec.resources is missing.
Error: exit status 1

✖  exit status 1

--request-memoryだけ指定してもエラーになる

$ tanzu apps workload apply hello-nodejs \
  --app hello-nodejs \
  --git-repo https://github.com/making/hello-nodejs \
  --git-branch master \
  --type web \
  --request-memory 64Mi \
  -n demo \
  -y
Create workload:
      1 + |---
      2 + |apiVersion: carto.run/v1alpha1
      3 + |kind: Workload
      4 + |metadata:
      5 + |  labels:
      6 + |    app.kubernetes.io/part-of: hello-nodejs
      7 + |    apps.tanzu.vmware.com/workload-type: web
      8 + |  name: hello-nodejs
      9 + |  namespace: demo
     10 + |spec:
     11 + |  resources:
     12 + |    requests:
     13 + |      memory: 64Mi
     14 + |  source:
     15 + |    git:
     16 + |      ref:
     17 + |        branch: master
     18 + |      url: https://github.com/making/hello-nodejs

Error: Unable to complete command, you do not have permissions for this resource: admission webhook "validation.gatekeeper.sh" denied the request: [requiredresources-workload] spec.resources.limits is missing.
Error: exit status 1

✖  exit status 1

--limit-memoryも指定するとWorkloadは作成できる

$ tanzu apps workload apply hello-nodejs \
  --app hello-nodejs \
  --git-repo https://github.com/making/hello-nodejs \
  --git-branch master \
  --type web \
  --request-memory 64Mi \
  --limit-memory 64Mi \
  -n demo \
  -y
Create workload:
      1 + |---
      2 + |apiVersion: carto.run/v1alpha1
      3 + |kind: Workload
      4 + |metadata:
      5 + |  labels:
      6 + |    app.kubernetes.io/part-of: hello-nodejs
      7 + |    apps.tanzu.vmware.com/workload-type: web
      8 + |  name: hello-nodejs
      9 + |  namespace: demo
     10 + |spec:
     11 + |  resources:
     12 + |    limits:
     13 + |      memory: 64Mi
     14 + |    requests:
     15 + |      memory: 64Mi
     16 + |  source:
     17 + |    git:
     18 + |      ref:
     19 + |        branch: master
     20 + |      url: https://github.com/making/hello-nodejs

Created workload "hello-nodejs"

To see logs:   "tanzu apps workload tail hello-nodejs --namespace demo"
To get status: "tanzu apps workload get hello-nodejs --namespace demo"

IK.AM