Spring boot, systemd, and syslog

I recently created a dead simple twitter bot using Spring Boot and Twitter4J. Instead of the usual running of the application via the command line and piping the output from the console to a log, or using SLF4J to create log files in a desired location, I wanted to achieve 4 main goals:

• Stick to the 12 Factor mantra by letting the environment control operational concerns – not the application
• Be able to start and stop the application on Ubuntu via “service” commands
• Ensure that if the host restarted, the application would run on boot
• Write logs to both the syslog and to /var/log (writing to the syslog with a unique “ident” allows me to filter properly in my elasticsearch/kibana setup)

Here is how you can make it work:

Step 1: Adjust your POM

In Spring Boot (using Maven), you can adjust your POM file to ensure that when you run a “mvn clean package”, it will generate a runnable JAR file that is packaged correctly for starting via systemd. Find the build section of your POM and make sure that the executable section is included.

<build>
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <executable>true</executable>
            </configuration>
        </plugin>
    </plugins>
</build>

Step 2: Create a unit file for systemd

Systemd (the “replacement” for inid.d) on modern linux systems such as Ubuntu uses what are called “unit files” to control service startup and shutdown.

If you want to create a new service called “twitterbot”, create a file in /etc/systemd/system called “twitterbot.service”

In this file, add the following lines:

[Unit]
Description=Twitterbot
After=syslog.target

[Service]
User=root
ExecStart=<FULL PATH TO THE JAR FILE IN THE TARGET DIRECTORY HERE>/twitterbot.jar
SuccessExitStatus=143
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=twitterbot


[Install]
WantedBy=multi-user.target

Once this is in place, you can run the following to get your system to recognize the new service:

systemctl enable <servicename>
systemctl start <servicename>

By using this simple configuration of systemd, your linux host will automatically keep track of the PID that was started, and redirect output of stderr and stdout to the syslog, and automatically tags the syslog entries as “twitterbot”. This is perfect for my Kibana setup.

Step 3: Write a local log

For my case, in addition to writing logs to syslog, I also wanted a local log file that I could check should I need it. In order to do this, I was able to use rsyslog. Rsyslog is already installed and working on most ubuntu machines, do it is just a matter of a quick configuration file.

Add a file in /etc/rsyslog.d named servicename.conf. In my example is was twitterbot.conf. In this file add:

if $programname == 'twitterbot' then /var/log/twitterbot.log
& stop

(obviously change the names of the file and service to suit your needs)

Rsyslog uses a matching algorithm, so you need to set the $programme == ‘servicename’ to the same value that you used in the unit file in step 2 above in the SyslogIdentifier field.

Restart rsyslog and you should now see local logs in /var/log.

Step 4: logrotate (optional)

If you are going to be generating a large amount of logs, you are going to want to use logrotate. Logrotate is the linux standard way of removing old logs, gzipping them, and generally keeping your log files from growing out of control. I am not going to cover all of the options of logrotate (as there are many), but you can read about them here.

For me, I wanted to keep it simple.. I started by adding a file named twitterbot in /etc/logrotate.d.

Here is the configuration that I used:

/var/log/twitterbot.log
{
    rotate 7
    daily
    missingok
    notifempty
    delaycompress
    compress
    postrotate
}