---
title: Prometheusをubuntu上にインストールしremote writeを受け付けるメモ
tags: ["Prometheus", "Ubuntu", "Vultr"]
categories: ["Observability", "Prometheus"]
date: 2024-01-24T12:34:59Z
updated: 2024-07-25T04:00:49Z
---

PrometheusのManaged Serviceはコスパが悪いので、Self Hostすることにする。監視対象のKubernetes上にPrometheusを置きたくないので、PrometheusはUbuntu VMで運用する。
Kubernetesのクラスタからはremote writeでこのSelf HostのPrometheusにメトリクスを送信する。

Prometheusをubuntu上にインストールする。[Vultr](https://vultr.com)上でVMを作成する。1 vCPU、1GB RAMのインスタンスなら$5から始められる。

<img width="1502" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/0f070290-b1b6-4086-a312-5ae9e6f30c81">

<img width="1513" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/a37593ae-1151-42e0-b864-4b593363734c">

<img width="1506" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/fd5688c7-9eae-4ad2-9c48-b77ba69b9deb">

<img width="1477" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/766eee3f-e45c-4b1d-9839-84a7591cdda3">

<img width="1402" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/930fd4b3-2390-4a3c-a29a-efb8ecd6735e">

<img width="1171" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/6c36b09f-d579-4854-b597-109a375d2b33">

<img width="1510" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/004424fb-967e-46eb-96a5-eb8e63fd693c">

<img width="1486" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/02e5ac89-c55c-4a2c-80c1-7e2ef5a1d86d">

<img width="1491" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/38b6904e-f1bd-4318-8af5-a908177ac5d7">

sshでアクセス

```
ssh root@41.65.21.12
```

neofecthしてみる。

```
sudo apt-get update
sudo apt-get install neofetch -y
```

<img width="689" src="https://github.com/making/blog.ik.am/assets/106908/dfd0a834-8c9c-4a49-b9fe-83052f8ee191">

prometheusユーザー作成

```
sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus
```

apt-getでもPrometheusはインストールできるが、最新版を使いたかったのでバイナリをダウンロードする。

Prometheusのダウンロードと配置

```
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v2.49.1/prometheus-2.49.1.linux-amd64.tar.gz
tar xzvf prometheus*.tar.gz
cd prometheus*/

sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo mv prometheus /usr/local/bin
sudo mv promtool /usr/local/bin
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
```

```
sudo mv consoles /etc/prometheus
sudo mv console_libraries /etc/prometheus
sudo mv prometheus.yml /etc/prometheus

sudo chown prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/consoles
sudo chown -R prometheus:prometheus /etc/prometheus/console_libraries
sudo chown -R prometheus:prometheus /var/lib/prometheus
```

Basic認証を設定する

```
htpasswd -nB admin
```

```
cat <<'EOF' > /etc/prometheus/web.yml
tls_server_config:
  cert_file: tls.crt
  key_file: tls.key
basic_auth_users:
  admin: $2y$05$****************************
EOF
```

自己署名TLS証明書を作成する

```
cd /tmp

cat <<'EOC' > gen_tls_certs.sh
#!/bin/bash

set -e

ROOT_DOMAIN=$1

SSL_FILE=sslconf-${ROOT_DOMAIN}.conf

#Generate SSL Config with SANs
if [ ! -f $SSL_FILE ]; then
cat > $SSL_FILE <<EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
[req_distinguished_name]
countryName_default = JP
stateOrProvinceName_default = Tokyo
localityName_default = Minato
organizationalUnitName_default = IK.AM
[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[alt_names]
DNS.1 = *.${ROOT_DOMAIN}
EOF
fi

openssl genrsa -out ${ROOT_DOMAIN}.key 2048
openssl req -new -out ${ROOT_DOMAIN}.csr -subj "/CN=*.${ROOT_DOMAIN}/O=IK.AM/C=JP" -key ${ROOT_DOMAIN}.key -config ${SSL_FILE}
openssl req -text -noout -in ${ROOT_DOMAIN}.csr
openssl x509 -req -days 3650 -in ${ROOT_DOMAIN}.csr -signkey ${ROOT_DOMAIN}.key -out ${ROOT_DOMAIN}.crt -extensions v3_req -extfile ${SSL_FILE}
openssl x509 -in ${ROOT_DOMAIN}.crt -text -noout
rm ${ROOT_DOMAIN}.csr
EOC

chmod +x ./gen_tls_certs.sh
./gen_tls_certs.sh sslip.io

sudo mv sslip.io.crt /etc/prometheus/tls.crt
sudo mv sslip.io.key /etc/prometheus/tls.key
sudo chown prometheus:prometheus /etc/prometheus/tls.*
```


scrape_configsを修正

```yaml
scrape_configs:
- job_name: "prometheus"
  static_configs:
  - targets: ["localhost:9090"]
  scheme: https
  tls_config:
    insecure_skip_verify: true
  basic_auth:
    username: admin
    password: "**************"
```

systmctlで起動するための設定ファイル作成

```
cat <<'EOF' > /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --web.config.file=/etc/prometheus/web.yml \
    --storage.tsdb.path=/var/lib/prometheus/ \
    --storage.tsdb.retention.time=42d \
    --storage.tsdb.retention.size=15GB \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.enable-lifecycle \
    --enable-feature=exemplar-storage \
    --enable-feature=otlp-write-receiver \
    --web.enable-remote-write-receiver

[Install]
WantedBy=multi-user.target
EOF
```

```
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
```

```
sudo systemctl status prometheus
```

```
sudo ufw allow 9090/tcp
```

`https://41-65-21-12.sslip.io:9090`にアクセス

<img width="1675" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/22b3f3ae-21d5-4cae-8f7e-8d1e11d2bcf3">

<img width="1670" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/693b24b0-7d0b-49d8-8748-730fa36842ef">


Node Exporterをインストール。こっちは最新版じゃなくてもいいのでapt-getでインストール。

```
sudo apt-get install prometheus-node-exporter -y
```

scrape_configsを追加

```yaml
- job_name: node-exporter
  static_configs:
  - targets:
    - localhost:9100
```

prometheusをリスタート

```
sudo systemctl restart prometheus
```

<img width="1683" alt="image" src="https://github.com/making/blog.ik.am/assets/106908/ee7b0853-f501-448d-8094-f1b2e0591233">


このPrometheusにRemote Writeしたい時は

```yaml
remote_write:
- url: https://41-65-21-12.sslip.io:9090/api/v1/write
  remote_timeout: 30s
  send_exemplars: true
  basic_auth:
    username: admin
    password: "**************"
  tls_config:
    insecure_skip_verify: true
```