-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Monitoring
If you use the below config to set "METRICS_ENABLED" to 'true' inside of docker-compose.override.yml, you will need to edit env/netbox.env and comment out the METRICS_ENABLED line there. This is (I think) because the environment file is sourced after the docker-compose override. I think this line should probably be removed from env/netbox.env to reduce confusion, since it's the default.
The prometheus and grafana docker images are set to use the "latest" version. You might want to pin these to specific versions in your own deployment.
Add the following to your docker-compose.override.yml
(or create that file if you haven't done so):
services:
# netbox
netbox:
environment:
METRICS_ENABLED: 'true'
prometheus_multiproc_dir: /tmp/metrics
volumes:
- type: tmpfs
target: /tmp/metrics
read_only: false
tmpfs:
mode: 0o01777
# postgres
postgres-exporter:
image: quay.io/prometheuscommunity/postgres-exporter:latest
depends_on:
- postgres
env_file: env/pg_exporter.env
environment:
DATA_SOURCE_URI: postgres?sslmode=disable
# redis
redis-worker-exporter:
image: oliver006/redis_exporter
depends_on:
- redis
env_file: env/redis.env
environment:
REDIS_ADDR: redis://redis:6379
redis-cache-exporter:
image: oliver006/redis_exporter
depends_on:
- redis-cache
env_file: env/redis-cache.env
environment:
REDIS_ADDR: redis://redis-cache:6379
# prometheus
prometheus:
image: prom/prometheus:latest
depends_on:
- postgres-exporter
- redis-cache-exporter
- redis-worker-exporter
- netbox
ports:
- '9090:9090'
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- prometheus-data:/prometheus/data
# grafana
grafana:
image: grafana/grafana:latest
depends_on:
- prometheus
env_file: env/grafana.env
environment:
GF_SECURITY_ADMIN_USER: admin
GF_METRICS_ENABLED: 'true'
ports:
- '3000:3000'
volumes:
#- ./monitoring/grafana/plugins/:/var/lib/grafana/plugins/:z,ro
#- ./monitoring/grafana/provisioning/:/etc/grafana/provisioning/:z,ro
#- ./monitoring/grafana/dashboards/:/etc/grafana/dashboards/:z,ro
- grafana-data:/var/lib/grafana
volumes:
prometheus-data:
driver: local
grafana-data:
driver: local
Then create the new file prometheus.yml
:
scrape_configs:
- job_name: prometheus
static_configs:
- targets: ['localhost:9090']
- job_name: netbox
static_configs:
- targets: ['netbox:8080']
labels:
app: 'netbox'
- job_name: postgresql
static_configs:
- targets: ['postgres-exporter:9187']
- job_name: redis
static_configs:
- targets: ['redis-worker-exporter:9121', 'redis-cache-exporter:9121']
- job_name: grafana
static_configs:
- targets: ['grafana:3000']
Finally, create env/grafana.env and env/pg_exporter.env. You will need to know the GF_SECURITY_ADMIN_PASSWORD to login to grafana, but it will be changed on first login.
source env/postgres.env
echo "DATA_SOURCE_USER=$POSTGRES_USER" > env/pg_exporter.env
echo "DATA_SOURCE_PASS=$POSTGRES_PASSWORD" >> env/pg_exporter.env
echo "GF_SECURITY_SECRET_KEY=$(openssl rand -base64 32)" > env/grafana.env
echo "GF_SECURITY_ADMIN_PASSWORD=$(openssl rand -hex 5)" | tee -a env/grafana.env
Then run docker compose up
.
Prometheus grabs the data once every minute from each service (NetBox, Nginx, PostgreSQL, 2x Redis, Prometheus itself) and stores it in a TSDB. You can access Prometheus on port 9090.
Grafana provides visualization of the data, which can be composed into dashboard.
To fetch the data it talks directly to the Prometheus service.
You can access Grafana on port 3000.
The default credentials are admin
and what was configured in the GF_SECURITY_ADMIN_PASSWORD environment variable in env/grafana.env.
You will find several user-made dashboards on the Grafana page which are easy to import using the Grafana Dashboard ID. There are already dashboard for Nginx, PostgreSQL, Redis, Prometheus and also Django (which is the framework on which NetBox is built). For more information see the importing a dashboard and discover dashboards on grafana.com articles.
https://grafana.com/grafana/dashboards/17658-django/ works well with django-prometheus. It needs "app: 'netbox'" assigned in the labels which I have added to the example prometheus.yml above.
The "Cache Hit Ratio" does not work because netbox is not exposing it's cache information through django. According to (https://github.com/netbox-community/netbox/issues/13349) nothing is really cached directly, so they don't plan to add these.