Systemd: Use erlang-systemd to notify systemd of service status

[dumbbell]

Until now, we were using socat(1). It was added as a dependency to our Debian and RPM packages. Unfortunately, the availability of this package has not always been good. We also supported erlang-sd_notify which relied on a UDP socket. None of them had journald support.

Now, thanks to erlang-systemd, we can use the Unix socket and provide more fine-grained status updates if we want.

Journald support is also provided by erlang-systemd. It was added to RabbitMQ in commit 91583a0 (#2940).

[dumbbell]

Valid systemd statuses are described in sd_notify(3):
https://www.freedesktop.org/software/systemd/man/sd_notify.html#Description

Currently, it is not possible to clear a status description with erlang-systemd. I'm discussing the possible root cause with the author. See hauleth/erlang-systemd#26. I would like to find a solution to that before submitting the patch for review.

Update 1: I submitted a pull request to keep empty values, allowing us to clear the status (hauleth/erlang-systemd#27).

Update 2: The pull request was accepted and merged. I'm updating this patch to pin the latest commit of erlang-systemd in rabbit/Makefile.

[dumbbell]

Here is how I tested it on a Debian Buster VM:

1. Create the source archive from the rabbitmq-server Git working copy root directory:

   make source-dist PROJECT_VERSION=3.9.0 PLUGINS=
   

   I set PLUGINS= to remove all plugins from the source archive & Debian package to save time.

2. Create the Debian package from a rabbitmq-packaging repository clone:

   make -C debs/Debian \
     SOURCE_DIST_FILE=/path/to/rabbitmq-server/clone/PACKAGES/rabbitmq-server-3.9.0.tar.xz \
     all
   

3. Install the package:

   sudo dpkg -i ./rabbitmq-server_3.9.0-1_all.deb
   

   The service is automatically started.

4. Control the service status in systemd:

   sudo systemctl status rabbitmq-server
   

   ● rabbitmq-server.service - RabbitMQ broker
      Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
      Active: active (running) since Thu 2021-04-08 13:18:29 UTC; 20s ago
    Main PID: 9526 (beam.smp)
       Tasks: 23 (limit: 4646)
      Memory: 74.7M
      CGroup: /system.slice/rabbitmq-server.service
              ├─9526 /usr/lib/erlang/erts-11.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 5
              ├─9537 erl_child_setup 32768
              ├─9586 inet_gethost 4
              └─9587 inet_gethost 4
   

5. (optional) As a bonus, we can play with erlang-systemd and statuses:

   sudo rabbitmqctl eval 'systemd:notify({status, "Playing with service status"}).'
   sudo systemctl status rabbitmq-server
   

   ● rabbitmq-server.service - RabbitMQ broker
      Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
      Active: active (running) since Thu 2021-04-08 13:18:29 UTC; 30s ago
    Main PID: 9526 (beam.smp)
      Status: "Playing with service status"
       Tasks: 23 (limit: 4646)
      Memory: 74.7M
      CGroup: /system.slice/rabbitmq-server.service
              ├─9526 /usr/lib/erlang/erts-11.2/bin/beam.smp -W w -MBas ageffcbf -MHas ageffcbf -MBlmbcs 5
              ├─9537 erl_child_setup 32768
              ├─9586 inet_gethost 4
              └─9587 inet_gethost 4
   

[michaelklishin]

@dumbbell this works as advertised but still required me to install socat before I could build the package. Is the socat dependency going to be dropped in rabbitmq-packaging in a separate PR? I currently don't see any.

[michaelklishin]

Let's backport this after 3.8.15 ships (if we want to backport this at all).

[dumbbell]

@dumbbell this works as advertised but still required me to install socat before I could build the package. Is the socat dependency going to be dropped in rabbitmq-packaging in a separate PR? I currently don't see any.

Yes, the dependency will be dropped from both the Debian and RPM packages in a separate patch.

[dumbbell]

Let's backport this after 3.8.15 ships (if we want to backport this at all).

No need to backport IMHO. It doesn't fix anything, just removes code mostly (now that master has this new dependency for journald logging).