Add SMART Monitoring with dash and alerts #251
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
technowhizz
force-pushed
the
feature/smartmon
branch
from
8f37c1c to
6de74ee
Compare
dougszumski
approved these changes
Nov 25, 2022
There was a problem hiding this comment.
This has been tested on sms-lab and can be demoed there. Good job @technowhizz
markgoddard
reviewed
Nov 28, 2022
dougszumski
previously requested changes
Nov 28, 2022
There was a problem hiding this comment.
Just noticed we are missing the change to globals.yml, @technowhizz, please could you add it?
Eg. this bit:
# Additional command line flags for node exporter to enable texfile collector for disk metrics and create textfile docker volume
prometheus_node_exporter_extra_volumes: "textfile:/var/lib/node_exporter/textfile_collector"
prometheus_node_exporter_cmdline_extras: "--collector.textfile.directory=/var/lib/node_exporter/textfile_collector"
|
Ah ok. I guess that answers my question to Mark about including the changes in this PR or not. :) Thank you! |
Enabled Textfile collector in node exporter in kolla/globals.yml Added smartmon script as is from the prometheus-community github and then removed NVME support from this script in favour of using the nvme-cli script, which has also been added in. This is because the nvme-cli script provides better metrics than the smartmon script does. The script also adds the serial number of the disk as a label to all SMART metrics. Added a Kayobe custom playbook to easily deploy the script and associated cron job. This playbook installs smartmontool and nvmecli then copies these over to the hosts and sets up a cronjob which runs the scripts and stores the metrics in the docker volume for node exporter. The playbook changes the way the metrics are saved to a file by making use of the mv command as it is atomic. This was needed as at times prometheus would read a partially completed file. Added a prometheus alert to alert when a drive is reported as not healthy for more than 10 minutes. Added a Grafana dashboard to display the number of healthy and unhealthy drives reported in prometheus.
technowhizz
force-pushed
the
feature/smartmon
branch
from
6de74ee to
d83ecde
Compare
|
@dougszumski @markgoddard Added in the changes to globals.yml |
|
docs now in stackhpc/yoga |
…e-config into feature/smartmon
jovial
reviewed
Nov 29, 2022
markgoddard
reviewed
Nov 30, 2022
markgoddard
reviewed
Nov 30, 2022
Fix kayobe command Co-authored-by: Will Szumski <[email protected]>
Fix Spelling Co-authored-by: Will Szumski <[email protected]>
markgoddard
suggested changes
Nov 30, 2022
technowhizz
dismissed
dougszumski’s stale review
force push means commit is gone and the only way to resolve this comment is to dismiss review. Sorry! :)
dougszumski
approved these changes
Nov 30, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added smartmon script as is from the prometheus-community github and then removed NVME support from this script in favour of using the nvme-cli script, which has also been added in. This is because the nvme-cli script provides better metrics than the smartmon script does. The script also adds the serial number of the disk as a label to all SMART metrics.
Added a Kayobe custom playbook to easily deploy the script and associated cron job. This playbook installs smartmontool and nvmecli then copies these over to the hosts and sets up a cronjob which runs the scripts and stores the metrics in the docker volume for node exporter. The playbook changes the way the metrics are saved to a file by making use of the mv command as it is atomic. This was needed as at times prometheus would read a partially completed file.
Added a prometheus alert to alert when a drive is reported as not healthy for more than 10 minutes.
Added a Grafana dashboard to display the number of healthy and unhealthy drives reported in prometheus.