mrtg script for monitoring temperature

[epixoip]

so this post got me thinking... this would actually be a great solution for remotely monitoring compute nodes in a vcl cluster. especially since ADL of cousre does not work with VCL.

so i whipped up this little shell script:

gpu_query

Code:

#!/bin/sh DISPLAY=:0 attr="$1" dev="$2" case "$attr" in temp) amdconfig --adapter=$dev --odgt | awk '/Temperature/ { printf("%d\n", $5); }' ;; core) amdconfig --adapter=$dev --odgc | awk '/Current\ Clocks/ { printf("%d\n", $4); }' ;; mem ) amdconfig --adapter=$dev --odgc | awk '/Current\ Clocks/ { printf("%d\n", $5); }' ;; load) amdconfig --adapter=$dev --odgc | awk '/GPU load/ { printf("%d\n", $4); }' ;; fan ) DISPLAY=:0.$(($2 - 1)) amdconfig --pplib-cmd "get fanspeed 0" | awk '/Result/ { printf("%d\n", $4); }' ;; * ) echo "$1: unknown option"; exit 2 ;; esac

then added a few extends to my snmpd.conf:

Code:

extend gpu_temp_1 /usr/local/bin/gpu_query temp 1 extend gpu_temp_2 /usr/local/bin/gpu_query temp 2 extend gpu_temp_3 /usr/local/bin/gpu_query temp 3 extend gpu_fan_1 /usr/local/bin/gpu_query fan 1 extend gpu_fan_2 /usr/local/bin/gpu_query fan 2 extend gpu_fan_3 /usr/local/bin/gpu_query fan 3 extend gpu_load_1 /usr/local/bin/gpu_query load 1 extend gpu_load_2 /usr/local/bin/gpu_query load 2 extend gpu_load_3 /usr/local/bin/gpu_query load 3 extend gpu_core_1 /usr/local/bin/gpu_query core 1 extend gpu_core_2 /usr/local/bin/gpu_query core 2 extend gpu_core_3 /usr/local/bin/gpu_query core 3 extend gpu_mem_1 /usr/local/bin/gpu_query mem 1 extend gpu_mem_2 /usr/local/bin/gpu_query mem 2 extend gpu_mem_3 /usr/local/bin/gpu_query mem 3

repeat on each compute node. install mrtg/cacti/whatever on the cluster controller, and voila.

Code:

epixoip@token:~$ snmpwalk -v 3 -u cacti -l authpriv -x AES -X cactisnmp -a SHA -A cactisnmp butters NET-SNMP-EXTEND-MIB::nsExtendOutput1Line NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_fan_1" = STRING: 100 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_fan_2" = STRING: 100 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_fan_3" = STRING: 100 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_mem_1" = STRING: 1630 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_mem_2" = STRING: 1630 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_mem_3" = STRING: 1630 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_core_1" = STRING: 925 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_core_2" = STRING: 925 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_core_3" = STRING: 925 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_load_1" = STRING: 99 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_load_2" = STRING: 99 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_load_3" = STRING: 73 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_temp_1" = STRING: 66 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_temp_2" = STRING: 74 NET-SNMP-EXTEND-MIB::nsExtendOutput1Line."gpu_temp_3" = STRING: 67