View source for MISC-TN-029 — A monitoring system for IoT devices with Glances, InfluxDB, and Grafana

{{InfoBoxTop}}
{{AppliesToSBCSPG}}
{{AppliesTo SBC Lynx TN}}
{{AppliesToIoT}}
{{AppliesTo IoT TN}}
{{InfoBoxBottom}}
{{WarningMessage|text=This technical note was validated against specific versions of hardware and software. It may not work with other versions.}}
[[Category:MISC-AN-TN]]
[[Category:MISC-TN]]
[[Category:IoT]]

__FORCETOC__
== History ==
{| class="wikitable" border="1"
!Version
!Date
!Notes
|-
|1.0.0
|September 2023
|First public release
|-
|}

==Introduction==
When dealing with IoT fleets, one of the major concerns is to monitor the devices' health status. The "health status" is a generic expression that may include tons of parameters, ranging from the system-on-chip's temperature to the number of processes running in Linux's user space.

This Technical Note (TN for short) illustrates how to combine some well-known, open-source software to implement a monitoring system for such devices. It is based on:
* [https://grafana.com/ Grafana]: ''visualization and analytics software''.
* [https://www.influxdata.com/ InfluxDB 2.0]: ''a time series database''.
* [https://nicolargo.github.io/glances/ Glances]: ''a cross-platform system monitoring tool written in Python''.

In essence, the example described implements the following scheme.

[[File:MISC-TN-029-BD.png|center|thumb|600px]]


The IoT device used for this TN is an [[MISC-TN-004:_Running_Debian_(armbian)_on_SBCSPG|SBCSPG IIoT gateway]] running Debian [[MISC-TN-026:_Automated_test_equipment_(ATE)_monitoring_with_SBCSPG_gateway,_ThingsBoard_IoT,_and_Fledge|Bullseye]] GNU/Linux distribution. An instance of Glances runs on SBCSPG for collecting data (CPU load, memory usage, network traffic, etc.). These data are pushed into an InfluxDB database. Grafana retrieves these data from the database and shows them in a beautiful dashboard.

For the sake of simplicity, Grafana and InfluxDB were installed on the same host running Ubuntu 20.04, but they can indeed run on different machines.

==Installing InfluxDB 2.0 and Grafana==
Please follow the instructions available [https://medium.com/@nanditasahu031/integration-of-influxdb2-with-grafana-28b4aebb3368 here], but refer to [https://repos.influxdata.com/ these page] for the InfluxDB GPG key to use.

For convenience, a ready-to-use dashboard was installed, which is available [https://grafana.com/grafana/dashboards/2387-glances-for-flux/ here].

For accessing the bucket from Grafana, use the API token provided by InfluxDB. It can be copied from the page shown in the following picture.


[[File:Sbcspg-glances-grafana-2.png|center|thumb|600px|Getting the API token from Grafana]]


==Installing Glances on the IoT device==
For convenience, a Python 3 virtual environment was created on the device first. Then, all the necessary packages were installed as illustrated in the following dump.

<pre class="board-terminal">
armbian@sbcspg:~/devel/glances$ python3 -m venv ~/devel/glances/venv/
armbian@sbcspg:~/devel/glances$ source venv/bin/activate
(venv) armbian@sbcspg:~/devel/glances$ pip install glances bottle influxdb
Collecting glances
  Downloading Glances-3.4.0.3-py3-none-any.whl (697 kB)
     |████████████████████████████████| 697 kB 1.4 MB/s 
Collecting bottle
  Downloading bottle-0.12.25-py3-none-any.whl (90 kB)
     |████████████████████████████████| 90 kB 800 kB/s 
Collecting influxdb
  Downloading influxdb-5.3.1-py2.py3-none-any.whl (77 kB)
     |████████████████████████████████| 77 kB 426 kB/s 
Collecting ujson>=5.4.0
  Downloading ujson-5.8.0.tar.gz (7.2 MB)
     |████████████████████████████████| 7.2 MB 79 kB/s 
  Installing build dependencies ... done
  WARNING: Missing build requirements in pyproject.toml for ujson>=5.4.0 from https://files.pythonhosted.org/packages/15/16/ff0a051f9a6e122f07630ed1e9cbe0e0b769273e123673f0d2aa17fe3a36/ujson-5.8.0.tar.gz#sha256=78e318def4ade898a461b3d92a79f9441e7e0e4d2ad5419abed4336d702c7425 (from glances).
  WARNING: The project does not specify a build backend, and pip cannot fall back to setuptools without 'wheel'.
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting defusedxml
  Downloading defusedxml-0.7.1-py2.py3-none-any.whl (25 kB)
Collecting psutil>=5.6.7
  Downloading psutil-5.9.5.tar.gz (493 kB)
     |████████████████████████████████| 493 kB 989 kB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting packaging
  Using cached packaging-23.1-py3-none-any.whl (48 kB)
Collecting msgpack
  Downloading msgpack-1.0.6.tar.gz (166 kB)
     |████████████████████████████████| 166 kB 1.9 MB/s 
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... done
    Preparing wheel metadata ... done
Collecting requests>=2.17.0
  Downloading requests-2.31.0-py3-none-any.whl (62 kB)
     |████████████████████████████████| 62 kB 102 kB/s 
Collecting python-dateutil>=2.6.0
  Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
     |████████████████████████████████| 247 kB 1.2 MB/s 
Collecting six>=1.10.0
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting pytz
  Downloading pytz-2023.3.post1-py2.py3-none-any.whl (502 kB)
     |████████████████████████████████| 502 kB 1.7 MB/s 
Collecting idna<4,>=2.5
  Downloading idna-3.4-py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 15 kB/s 
Collecting urllib3<3,>=1.21.1
  Downloading urllib3-2.0.5-py3-none-any.whl (123 kB)
     |████████████████████████████████| 123 kB 1.1 MB/s 
Collecting charset-normalizer<4,>=2
  Downloading charset_normalizer-3.2.0-py3-none-any.whl (46 kB)
     |████████████████████████████████| 46 kB 440 kB/s 
Collecting certifi>=2017.4.17
  Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
     |████████████████████████████████| 158 kB 1.9 MB/s 
Building wheels for collected packages: psutil, ujson, msgpack
  Building wheel for psutil (PEP 517) ... done
  Created wheel for psutil: filename=psutil-5.9.5-cp39-abi3-linux_armv7l.whl size=275912 sha256=d0b38cd2abba96499a39acfc923ec66a9ada40e3264035e2d0739d60a500ac1a
  Stored in directory: /home/armbian/.cache/pip/wheels/74/d1/7d/d9ae7d9aea0f1cebed73f37868df7b5f3333e7f30163b3e558
  Building wheel for ujson (PEP 517) ... done
  Created wheel for ujson: filename=ujson-5.8.0-cp39-cp39-linux_armv7l.whl size=43721 sha256=e75aa525edb1accecce46ffb68467622bd9f6976712e7d651340ddb380ba6335
  Stored in directory: /home/armbian/.cache/pip/wheels/fa/b8/d0/86f0f82c834d098fca594ab0fb3ca6d4b9d726b386d1e02885
  Building wheel for msgpack (PEP 517) ... done
  Created wheel for msgpack: filename=msgpack-1.0.6-cp39-cp39-linux_armv7l.whl size=487508 sha256=9e70c81642dc8f1cebb7dfdf407a41f076e8b46d717085560729105821d0ff9e
  Stored in directory: /home/armbian/.cache/pip/wheels/37/4e/d3/fa9d02a88faaceddfc21a594ad8673ba36dd2766c21bee55b6
Successfully built psutil ujson msgpack
Installing collected packages: urllib3, six, idna, charset-normalizer, certifi, ujson, requests, pytz, python-dateutil, psutil, packaging, msgpack, defusedxml, influxdb, glances, bottle
Successfully installed bottle-0.12.25 certifi-2023.7.22 charset-normalizer-3.2.0 defusedxml-0.7.1 glances-3.4.0.3 idna-3.4 influxdb-5.3.1 msgpack-1.0.6 packaging-23.1 psutil-5.9.5 python-dateutil-2.8.2 pytz-2023.3.post1 requests-2.31.0 six-1.16.0 ujson-5.8.0 urllib3-2.0.5
(venv) armbian@sbcspg:~/devel/glances$ pip install 'influxdb-client[ciso]'
Collecting influxdb-client[ciso]
  Downloading influxdb_client-1.37.0-py3-none-any.whl (743 kB)
     |████████████████████████████████| 743 kB 498 kB/s 
Requirement already satisfied: certifi>=14.05.14 in ./venv/lib/python3.9/site-packages (from influxdb-client[ciso]) (2023.7.22)
Requirement already satisfied: setuptools>=21.0.0 in ./venv/lib/python3.9/site-packages (from influxdb-client[ciso]) (44.1.1)
Requirement already satisfied: urllib3>=1.26.0 in ./venv/lib/python3.9/site-packages (from influxdb-client[ciso]) (2.0.5)
Requirement already satisfied: python-dateutil>=2.5.3 in ./venv/lib/python3.9/site-packages (from influxdb-client[ciso]) (2.8.2)
Collecting reactivex>=4.0.4
  Downloading reactivex-4.0.4-py3-none-any.whl (217 kB)
     |████████████████████████████████| 217 kB 1.1 MB/s 
Collecting ciso8601>=2.1.1
  Downloading ciso8601-2.3.0.tar.gz (26 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Requirement already satisfied: six>=1.5 in ./venv/lib/python3.9/site-packages (from python-dateutil>=2.5.3->influxdb-client[ciso]) (1.16.0)
Collecting typing-extensions<5.0.0,>=4.1.1
  Using cached typing_extensions-4.8.0-py3-none-any.whl (31 kB)
Building wheels for collected packages: ciso8601
  Building wheel for ciso8601 (PEP 517) ... done
  Created wheel for ciso8601: filename=ciso8601-2.3.0-cp39-cp39-linux_armv7l.whl size=35221 sha256=ad5df00eaede09786d81ef5a668ffd6e822101ae12d7b57006f4e6ce3c9f524b
  Stored in directory: /home/armbian/.cache/pip/wheels/27/60/c3/7db442d55c9a26ebfbe9d05ce281f045005d5c40296b1fa332
Successfully built ciso8601
Installing collected packages: typing-extensions, reactivex, influxdb-client, ciso8601
Successfully installed ciso8601-2.3.0 influxdb-client-1.37.0 reactivex-4.0.4 typing-extensions-4.8.0
</pre>

The next steps consists of creating a configuration file suitable for our setup.
<pre class="board-terminal">
(venv) armbian@sbcspg:~/.config/glances$ cp -v ~/devel/glances/venv/share/doc/glances/glances.conf  .
'/home/armbian/devel/glances/venv/share/doc/glances/glances.conf' -> './glances.conf'
</pre>

In particular, make sure the the <code>influxdb2</code> section is edited properly for matching your setup:
<pre class="board-terminal">
...
[influxdb2]
# Configuration for the --export influxdb2 option
# https://influxdb.com/
host=192.168.0.21
port=8086
protocol=http
org=nicolargo
bucket=glances
token=ZWZfgKt6O_GS38xzRZIAEFwXfeCh1pyYIDfBXFaeGig9eJvzdi1pFKseXVq8n-rkJ1MD-7mPS>
# Set the interval between two exports (in seconds)
# If the interval is set to 0, the Glances refresh time is used (default behavo>
#interval=0
# Prefix will be added for all measurement name
# Ex: prefix=foo
#     => foo.cpu
#     => foo.mem
# You can also use dynamic values
#prefix=foo
# Following tags will be added for all measurements
# You can also use dynamic values.
# Note: hostname is always added as a tag
#tags=foo:bar,spam:eggs,domain:`domainname`
...
</pre>

Once the configuration file is in place, Glances can be started in quiet mode as follows:
<pre class="board-terminal">
(venv) armbian@sbcspg:~/.config/glances$ glances -q --export influxdb2
</pre>

If everything works as expected, collected data are displayed live in the Grafana's dashboard:
[[File:Sbcspg-glances-grafana-1.png|center|thumb|600px|Grafana dashboard showing collected data]]