--- title: Tanzu Application PlatformにWorkloadをデプロイする際にGatekeeperでresourcesを設定を強制するメモ tags: ["Kubernetes", "Tanzu", "TAP", "Gatekeeper", "Open Policy Agent"] categories: ["Dev", "CaaS", "Kubernetes", "TAP"] date: 2022-12-14T01:42:10Z updated: 2022-12-14T01:45:45Z --- TAPにWorkloadをデプロイする際に、resourcesを指定しないとNodeのcpu, memoryを限界以上にアプリがデプロイされる可能性があるので、 resourcesの設定は推奨。これを強制するために[Gatekeeper](https://open-policy-agent.github.io/gatekeeper/website/docs/)を使用する。 ### Gatekeeperのインストール [ドキュメント](https://open-policy-agent.github.io/gatekeeper/website/docs/install#installation)の通りにインストール ``` kubectl apply -f https://raw.githubusercontent.com/open-policy-agent/gatekeeper/master/deploy/gatekeeper.yaml ``` Podの確認 ``` $ kubectl get pod -n gatekeeper-system NAME READY STATUS RESTARTS AGE gatekeeper-audit-84b788f7c5-54m2r 1/1 Running 0 13s gatekeeper-controller-manager-76dc9cbc64-2q26t 1/1 Running 0 13s gatekeeper-controller-manager-76dc9cbc64-fx5m8 1/1 Running 0 13s gatekeeper-controller-manager-76dc9cbc64-qb52w 1/1 Running 0 13s ``` ### ConstraintTemplateの作成 とりあえず `spec.resources.{limits,requests}.memory` の設定を必須とするConstraintTemplate。cpuも必須にしたい場合はコメントを外す ```yaml cat < requiredresources.yaml apiVersion: templates.gatekeeper.sh/v1beta1 kind: ConstraintTemplate metadata: name: requiredresources spec: crd: spec: names: kind: RequiredResources targets: - target: admission.k8s.gatekeeper.sh rego: | package requiredresources missing(obj, field) = true { not obj[field] } missing(obj, field) = true { obj[field] == "" } violation[{"msg": msg, "details": {}}] { missing(input.review.object.spec, "resources") msg := "spec.resources is missing." } violation[{"msg": msg, "details": {}}] { missing(input.review.object.spec.resources, "limits") msg := "spec.resources.limits is missing." } violation[{"msg": msg, "details": {}}] { missing(input.review.object.spec.resources, "requests") msg := "spec.resources.requests is missing." } # violation[{"msg": msg, "details": {}}] { # missing(input.review.object.spec.resources.limits, "cpu") # msg := "spec.resources.limits.cpu is missing." # } violation[{"msg": msg, "details": {}}] { missing(input.review.object.spec.resources.limits, "memory") msg := "spec.resources.limits.memory is missing." } # violation[{"msg": msg, "details": {}}] { # missing(input.review.object.spec.resources.requests, "cpu") # msg := "spec.resources.requests.cpu is missing." # } violation[{"msg": msg, "details": {}}] { missing(input.review.object.spec.resources.requests, "memory") msg := "spec.resources.requests.memory is missing." } EOF kubectl apply -f requiredresources.yaml ``` ### Constraintの作成 まずはdryrun modeで試す ```yaml cat < requiredresources-workload.yaml apiVersion: constraints.gatekeeper.sh/v1beta1 kind: RequiredResources metadata: name: requiredresources-workload spec: enforcementAction: dryrun match: kinds: - apiGroups: - carto.run kinds: - Workload EOF kubectl apply -f requiredresources-workload.yaml ``` この状態だとまだresourcesの設定なしでもWorkloadを作成できる ``` $ tanzu apps workload apply hello-nodejs \ --app hello-nodejs \ --git-repo https://github.com/making/hello-nodejs \ --git-branch master \ --type web \ -n demo \ -y Create workload: 1 + |--- 2 + |apiVersion: carto.run/v1alpha1 3 + |kind: Workload 4 + |metadata: 5 + | labels: 6 + | app.kubernetes.io/part-of: hello-nodejs 7 + | apps.tanzu.vmware.com/workload-type: web 8 + | name: hello-nodejs 9 + | namespace: demo 10 + |spec: 11 + | source: 12 + | git: 13 + | ref: 14 + | branch: master 15 + | url: https://github.com/making/hello-nodejs Created workload "hello-nodejs" To see logs: "tanzu apps workload tail hello-nodejs --namespace demo" To get status: "tanzu apps workload get hello-nodejs --namespace demo" ``` 制約違反があるかどうかは次のようにチェックできる。 ``` $ kubectl get requiredresources requiredresources-workload -ojsonpath='{.status.violations}' | jq . [ { "enforcementAction": "dryrun", "group": "carto.run", "kind": "Workload", "message": "spec.resources is missing.", "name": "hello-nodejs", "namespace": "demo", "version": "v1alpha1" } ``` dryrunをやめて強制モードに変更 ```yaml cat < requiredresources-workload.yaml apiVersion: constraints.gatekeeper.sh/v1beta1 kind: RequiredResources metadata: name: requiredresources-workload spec: enforcementAction: deny match: kinds: - apiGroups: - carto.run kinds: - Workload EOF kubectl apply -f requiredresources-workload.yaml ``` Workloadを一度削除する ``` tanzu apps workload delete -n demo hello-nodejs -y ``` 再びWorkloadを作成しようとするとエラーになる ``` $ tanzu apps workload apply hello-nodejs \ --app hello-nodejs \ --git-repo https://github.com/making/hello-nodejs \ --git-branch master \ --type web \ -n demo \ -y Create workload: 1 + |--- 2 + |apiVersion: carto.run/v1alpha1 3 + |kind: Workload 4 + |metadata: 5 + | labels: 6 + | app.kubernetes.io/part-of: hello-nodejs 7 + | apps.tanzu.vmware.com/workload-type: web 8 + | name: hello-nodejs 9 + | namespace: demo 10 + |spec: 11 + | source: 12 + | git: 13 + | ref: 14 + | branch: master 15 + | url: https://github.com/making/hello-nodejs Error: Unable to complete command, you do not have permissions for this resource: admission webhook "validation.gatekeeper.sh" denied the request: [requiredresources-workload] spec.resources is missing. Error: exit status 1 ✖ exit status 1 ``` `--request-memory`だけ指定してもエラーになる ``` $ tanzu apps workload apply hello-nodejs \ --app hello-nodejs \ --git-repo https://github.com/making/hello-nodejs \ --git-branch master \ --type web \ --request-memory 64Mi \ -n demo \ -y Create workload: 1 + |--- 2 + |apiVersion: carto.run/v1alpha1 3 + |kind: Workload 4 + |metadata: 5 + | labels: 6 + | app.kubernetes.io/part-of: hello-nodejs 7 + | apps.tanzu.vmware.com/workload-type: web 8 + | name: hello-nodejs 9 + | namespace: demo 10 + |spec: 11 + | resources: 12 + | requests: 13 + | memory: 64Mi 14 + | source: 15 + | git: 16 + | ref: 17 + | branch: master 18 + | url: https://github.com/making/hello-nodejs Error: Unable to complete command, you do not have permissions for this resource: admission webhook "validation.gatekeeper.sh" denied the request: [requiredresources-workload] spec.resources.limits is missing. Error: exit status 1 ✖ exit status 1 ``` `--limit-memory`も指定するとWorkloadは作成できる ``` $ tanzu apps workload apply hello-nodejs \ --app hello-nodejs \ --git-repo https://github.com/making/hello-nodejs \ --git-branch master \ --type web \ --request-memory 64Mi \ --limit-memory 64Mi \ -n demo \ -y Create workload: 1 + |--- 2 + |apiVersion: carto.run/v1alpha1 3 + |kind: Workload 4 + |metadata: 5 + | labels: 6 + | app.kubernetes.io/part-of: hello-nodejs 7 + | apps.tanzu.vmware.com/workload-type: web 8 + | name: hello-nodejs 9 + | namespace: demo 10 + |spec: 11 + | resources: 12 + | limits: 13 + | memory: 64Mi 14 + | requests: 15 + | memory: 64Mi 16 + | source: 17 + | git: 18 + | ref: 19 + | branch: master 20 + | url: https://github.com/making/hello-nodejs Created workload "hello-nodejs" To see logs: "tanzu apps workload tail hello-nodejs --namespace demo" To get status: "tanzu apps workload get hello-nodejs --namespace demo" ```