Event Monitoring

My recent problem with the Raspberry Pi is an event monitoring and notification problem. It’s one, that over my career, I have spent a lot of time considering, but today, I only have one server. I don’t think I need or want to afford something like Zabbix, which comes with a series of event monitors and reporting and dashboard functionality. I have however adopted the habit of attempting to dovetail with host OS’s logging and reporting schemes. At the moment, I have written two event monitors which report to the desk top. Can I do better? Here are my notes.

This is a companion piece to Raspberry Pi and messages

I found simple alerts which describes an assembled architecture using unix tools, and a couple of 3rd party products.

Event Notifiers

I have three monitors.

1. Whether the apt needs an upgrade/update and
2. if the root file system is >94% full.
3. if the number of files is >94% of quota

Raspberry Pi and messages talks about how to notify that events have taken place. This talks about how to integrate my alerts into the Linux logging system. They currently report to the desktop, but are not running with any regularity. They need to be installed into the cron daemon. This article holds links to how to make RSS files and use IFTTT.

Historically, we have relied on syslog, as the event collector, and once we have multiple systems rsyslogd. The program logger does the heavy lifting.  Here is a “how to” from networkworld.com. I think logger can also write to the systemd job journals. Writing to the log and then extracting the event from the logs seems to be a faff for two events.

So I can run my detectors using cron, the package updater might be best reporting when upgrade has not been run for a given period.

IONOS

I do not have admin privilege of this system. WRT to the FD problem, I have written some scripts, as of today, I have a monitor, a trigger and an action. The first, writes the number of files into a state and history file, the second reads the state file and decides if to call the 3rd program. I plan to write an RSS item into a web available file, and raise an alert on my phone using IFTTT. See also running wordpress.

I have finished this for space and file descriptors for davelevy,info, the code is in my ubuntu-tools repo, I have documented them on that wiki. It does not have the log writer coded; that will need to wait until I install it on the PI.

Hysteresis control

This should be easier but think I’ll create a status file containing critical facts as at the last time run and so we can measure if the bad state is new or old. It might be good to be able to escalate from warning to error. But this is more difficult. (If this could be done, then maybe warn at 85% and Error at 95% for the filesystem). The apt control might be better escalated by size of the upgradable repo.

SNMP

I talk about snmp and tools at Raspberry Pi and messages

Logger

Here, is a link on using Logger, and wikipedia documents the error and severity levels in an article called syslog, so man syslog., and man logger. NB There is an RFC for syslog.

Genericisation and message code.

Now I have two maybe I should rewrite the agents as a general event manager and use parameters for the event properties, metrics and error messages.  Probaly not for two!

Hasn’t anyone done this in a more light weight manner than Zabbix?

The answer is yes

1. https://opensource.com/article/19/4/log-analysis-tools
2. https://geekflare.com/best-open-source-monitoring-software/ by CHnadan KUmare, which among other things puts https://prometheus.io/ in its 4th category, although most of these are aimed at managing networks, not single systems.

Raspbian

I was looking at how to monitor my pi. Here are some links,

1. Zabbix: Tabbix for Zabbix, it seems this is developed by a Turkish speaker or speakers, they have posted further instructions on a blogger page. The huge disadvantage with this is that you need zabbix on the PI, although there’s plenty of help putting it onto a cloud provider. In my case, Cloud, spoils the economics of doing it on a PI in my bedroom, and when I say disadvantage of Zabbix, it requires a database and you are reliant on their support offerings. (It’s too complex for my needs today.). pimylifeup explain how to put Zabbix on a Pi. 😐
2. SNMP: Enabling SNMP on a Raspberry Pi, a how to install article, on the PI, while manageengine do a free android app to consume the feeds.  12 Best SNMP Monitoring Tools + Complete SNMP Guide, from DNS Stuff catalogues alternatives but does not concentrate on android.
3. Syslog: Paper Trail, have a free plan which might do, to pick up syslogs. It’s a cloud offering and so usable via web browser one assumes. There must be more. This, Best free syslog servers for linux and windows points once again at Solarwinds and Nagios, a name I had forgotten.
4. Console/Dashboard: Eight apps to monitor raspberry pi is mostly phone apps but “pi healthcheck” looks interesting. I am on my third monitor on android, the previous two have given up on me. I am now on RaspController, it requires Python 3 on the server, which it will install, and they have a comprehensive web site, with a specific product page. a list of packages used and instructions on making custom user widgets, in python 😢 but it might be what I need. Its advertising space takes up more screen estate than I want, so if this is what I want I may have to pay for it.

These links were found when looking to report usage stats to a log, I started looking for sar, but found,Working with sysstat and sar, the package is called sysstat, note the double “s”, not ß. I wrote https://davelevy.info/wiki/linux-performance-monitoring/ to consolidate my notes on sar.