Event Monitoring

My recent problem with the Raspberry Pi is an event monitoring and notification problem. It’s one, that over my career, I have spent a lot of time considering, but today, I only have one server. I don’t think I need or want to afford something like Zabbix, which comes with a series of event monitors and reporting and dashboard functionality. I have however adopted the habit of attempting to dovetail with host OS’s logging and reporting schemes. At the moment, I have written two event monitors which report to the desk top. Can I do better? Here are my notes.

This is a companion piece to Raspberry Pi and messages

Monitors

I have two monitors.

  1. Whether the apt needs an upgrade/update and
  2. if the root file system is >94% full.

Raspberry Pi and messages talks about how to notify that events have taken place. This talks about how to integrate my alerts into the Linux logging system. They currently report to the desktop, but are not running with any regularity. They need to be installed into the cron daemon.

Historically, we have relied on syslog, as the event collector, and once we have multiple systems rsyslogd. The program logger does the heavy lifting.  Here is a “how to” from networkworld.com. I think logger can also write to the systemd job journals. Writing to the log and then extracting the event from the logs seems to be a faff for two events.

So I can run my detectors using cron, the package updater might be best reporting when upgrade has not been run for a given period.

Hysteresis control

This should be easier but think I’ll create a status file containing critical facts as at the last time run and so we can measure if the bad state is new or old. It might be good to be able to escalate from warning to error. But this is more difficult. (If this could be done, then maybe warn at 85% and Error at 95% for the filesystem). The apt control might be better escalated by size of the upgradable repo.

Genericisation and message code.

Now I have two maybe I should rewrite the agents as a general event manager and use parameters for the event properties, metrics and error messages.  Probaly not for two!

Hasn’t anyone done this in a more light weight manner than Zabbix?

The answer is yes

  1. https://opensource.com/article/19/4/log-analysis-tools
  2. https://geekflare.com/best-open-source-monitoring-software/ by CHnadan KUmare, which among other things puts https://prometheus.io/ in its 4th category, although most of these are aimed at managing networks, not single systems.

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.