UNDERSTANDING THE SUBJECT MATTER
Every activity, including services/application Logs are generated every time a system is powered on or shut down. Following the Linux filesystem structure, these logs are stored in the “/var/log” directory.
Have you ever wondered how these logs by default never get to fill up the “/var/log” filesystem? Well, this can only happen by a mechanism called log rotation with the use of logrotate tool.
Log rotation is a process whereby logs are rotated away from the system automatically and systematically. The systematic rotation of logs depends on the argument that is defined in the logrotate configuration file.
With logrotate configuration file, Log rotation can be configured in many ways. For example, log rotation can be configured to happen daily, weekly, monthly or yearly.
More so, Logs can be set to be rotated when it gets to a particular threshold or size, if a user wishes, logs can be zipped when it logged, it can even be configured to be sent to specific email addresses.
All these settings will depend on your environment. An environment that generates many log files will need to pay attention to the “/var/log” filesystem not to get filled up when setting up log rotation, especially if a log server is not configured for such environment.
I remember one of my customers running SAP, sometimes, I get calls and emails that a particular SAP log files have filled up the filesystem and the application will not start.
With a proper setup of log rotation and configuration of a log server, this problem can easily be solved/averted but due to reasons I won’t like to mention, all I do when such happens is to always extend the filesystem.
Moving forward, let’s look at what the logrotate configuration file looks like and how log rotation can be configured.
The logrotate configuration files can be found in two locations. One of the locations is the “/etc/logrotate.conf”, while the second is “/etc/logrotate.d”.
The “/etc/logrotate.conf” file contains logrotate generic information for log rotation, while the “/etc/logrotate.d” is the directory a user can define log rotatation scripts for services. Though, users can also define their log rotation scripts in “/etc/logrotate.conf”.
However, if a configuration of the same services is done in both “/etc/logrotate.d and “/etc/logrotate.conf “, the one in “/etc/logrotate.d” will override the one in “/etc/logrotate.conf” as it has a higher preference over “/etc/logrotate.conf.”
Let us look at “/etc/logrotate.conf” file.
[root@HQEBPRD ~]# cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
#compress
# RPM packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.
[root@HQEBPRD ~]#
These arguments are the default for RHEL 8.1 system, and you can see that it is self-explanatory.
Let us look at their meanings one after the other. By the side, let’s open the “/var/log” directory as well. Yours may be different depending on the services you are running and the number of periods your system has been running
[root@HQEBPRD log]# ls /var/log
anaconda hawkey.log-20200329 speech-dispatcher
audit hawkey.log-20200405 spooler
boot.log hawkey.log-20200407 spooler-20200322
boot.log-20200328 httpd spooler-20200329
boot.log-20200329 insights-client spooler-20200405
boot.log-20200401 lastlog spooler-20200407
boot.log-20200404 libvirt sssd
boot.log-20200405 maillog swtpm
boot.log-20200406 maillog-20200322 tekneed.log
boot.log-20200407 maillog-20200329 tekneed.log-20200407
btmp maillog-20200405 tuned
btmp-20200407 maillog-20200407 vmware-network.1.log
chrony messages vmware-network.2.log
cron messages-20200322 vmware-network.3.log
cron-20200322 messages-20200329 vmware-network.4.log
cron-20200329 messages-20200405 vmware-network.5.log
cron-20200405 messages-20200407 vmware-network.6.log1. It means that log files should be rotated weekly. This argument, weekly can be changed to daily, monthly or yearly depending on what a user wants to achieve
2. It means 4 weeks worth of backlogs should be kept. After the fourth week, the fourth log file will be deleted automatically and a new log file will be created. This argument can be changed to 2, 3, 5, 6, etc.
3. This means that a new empty log file should be created after rotating the old ones.
4. This means that the log files should be named by using the date it was created as the suffix.
something like this, as seen in “/var/log”
messages-20200315
messages-20200322From the dates, these log files were created on 15th of march,2020, and 22nd of march, 2020
5. It means that you can either make the log files be zipped or not when they are created. if you uncomment that line, the log files will be zipped, and you would need to unzip it before you can read the logs. This can be useful for services that generate a lot of logs.
6. It means, allow all other services, rpm packages to be able to define their log rotation settings in the “/etc/logrotate” directory. And include configuration from “/etc/logrotate.d”
There are a lot of arguments that can be defined in a logrotate script, you can check the manual page for logrotate to see a lot of arguments that can be used. However, let’s look at some of the arguments in other logrotate configuration files for different services.
The logrotate configuration files for services can be found and configured in the “/etc/logrotate.d” directory like I said above.
Let’s have a look at this directory.
[root@HQEBPRD ~]# cd /etc/logrotate.d/
[root@HQEBPRD logrotate.d]# ls
bootlog cups iscsiuiolog numad sssd up2date
btmp dnf libvirtd psacct subscription-manager wpa_supplicant
chrony httpd libvirtd.qemu samba syslog wtmp
[root@HQEBPRD logrotate.d]#
you can see the logrotate files named after their services. you can see the likes of httpd, dnf, libvirtd, samba, etc. We will also set one up in the “ACTION TIME” section.
Let’s take a look at the libvirtd file for example. The libvirtd service is responsible for managing KVM virtualization.
[root@HQEBPRD logrotate.d]# cat libvirtd
/var/log/libvirt/libvirtd.log {
weekly
missingok
rotate 4
compress
delaycompress
copytruncate
minsize 100k
}
[root@HQEBPRD logrotate.d]#
This script is also explanatory enough, I have explained some of them earlier when we opened the “/etc/logrotate.conf” file, but I will explain the ones I haven’t explained.
What does “missingok” mean? This means, it is okay for the log files to be missing, and if it is missing, don’t give an error, just go on to the next file.
It is also possible to set the value to “nomissingok”
What does “delaycompress” mean? This means that the compression of the previous log file should be delayed until the next rotation. Hence, without the compress value being defined in the script, this value is not useful.
What does “copytruncate” mean? This means that the original log file will be truncated to zero after a copy has been created.
What does minsize mean? This means that the log files shouldn’t be rotated until they grow bigger than the size bytes. However, if a time is specified, it won’t be rotated before the time.
Let’s also take a look at wtmp service as well, wtmp keeps events of system logins.
[root@HQEBPRD logrotate.d]# cat wtmp
# no packages own wtmp -- we'll rotate it here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
The logrotate utility on some Linux distribution and version is installed by default. However, you can verify if it is installed or not.
use the command,
[root@HQEBPRD ~]# rpm -q logrotate
logrotate-3.14.0-3.el8.x86_64
use the command,
[root@HQEBPRD logrotate.d]# yum install logrotate
Updating Subscription Management repositories.
Last metadata expiration check: 0:20:11 ago on Mon 06 Apr 2020 04:57:51 PM WAT.
Package logrotate-3.14.0-3.el8.x86_64 is already installed.
Dependencies resolved.[root@HQEBPRD logrotate.d]# logrotate --version
logrotate 3.14.0
Default mail command: /bin/mail
Default compress command: /bin/gzip
Default uncompress command: /bin/gunzip
Default compress extension: .gz
Default state file path: /var/lib/logrotate/logrotate.status
ACL support: yes
SELinux support: yes
[root@HQEBPRD logrotate.d]#logrotate is not a daemon, so it doesn’t run as a service, it runs as a cron job. The crontab file can be found in /”etc/cron.daily directory.”
[root@HQEBPRD ~]# cd /etc/cron.daily/
[root@HQEBPRD cron.daily]# ls
logrotate rhsmd
[root@HQEBPRD cron.daily]# cat logrotate
#!/bin/sh
/usr/sbin/logrotate /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
/usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit $EXITVALUE
[root@HQEBPRD cron.daily]#
However, you can verify if logrotate is working.
The last time when rotation happens can be verified from the file
“/var/lib/logrotate/logrotate.status”
To Verify, use the command,
[root@HQPRD2 ~]# cat /var/lib/logrotate/logrotate.status
logrotate state -- version 2
"/var/log/nginx/error.log" 2020-2-20-3:25:1
"/var/log/rabbitmq/rabbit@HQPRD2_upgrade.log" 2020-3-4-3:32:1
"/var/opt/remi/php72/log/php-fpm/*log" 2020-2-20-3:0:0
"/var/opt/rh/rh-redis5/log/redis/*.log" 2020-2-28-5:0:0
"/var/log/yum.log" 2020-1-2-6:31:2
"/var/named/data/named.run" 2020-4-5-4:13:1
"/var/log/boot.log" 2020-4-8-15:33:1
"/var/log/up2date" 2019-7-4-15:0:0
"/var/log/httpd/*log" 2019-7-13-14:0:0
"/var/log/httpd/error_log" 2020-2-27-7:32:2
Having understood what logrotate is and how log logrotation works, let’s create a logrotate script for a service.
ACTION TIME
How To Setup Log Rotation For A Service Using logrotate In Linux With Examples
Assuming the name of my service (application) is tekneed, just as there are sshd and httpd log rotation configuration files named after their services in “/var/log/.
To set up log rotation, follow the steps below.
1. Create a log rotation configuration file for the service in the directory, “/etc/logrotate.d”
[root@HQEBPRD ~]# vi /etc/logrotate.d/tekneed/var/log/tekneed.log {
weekly
missingok
rotate 4
compress
delaycompress
copytruncate
minsize 100k
}These arguments have already been explained in “UNDERSTANDING THE SUBJECT MATTER” section.
2. For the purpose of this activity, create a log file in the directory “/var/log” assuming that’s the location tekneed application logs its event.
[root@HQEBPRD ~]# vi /var/log/tekneed.logmy first log file2. For the purpose of this activity, allow logrotate to rotate all logs.
[root@HQEBPRD ~]# logrotate -f /etc/logrotate.conf
3. verify the logs in the “/var/log” directory
[root@HQEBPRD log]# ls /var/log
anaconda hawkey.log-20200329 speech-dispatcher
audit hawkey.log-20200405 spooler
boot.log hawkey.log-20200407 spooler-20200322
boot.log-20200328 httpd spooler-20200329
boot.log-20200329 insights-client spooler-20200405
boot.log-20200401 lastlog spooler-20200407
boot.log-20200404 libvirt sssd
boot.log-20200405 maillog swtpm
boot.log-20200406 maillog-20200322 tekneed.log
boot.log-20200407 maillog-20200329 tekneed.log-20200407
btmp maillog-20200405 tuned
btmp-20200407 maillog-20200407 vmware-network.1.log
chrony messages vmware-network.2.log
cron messages-20200322 vmware-network.3.log
cron-20200322 messages-20200329 vmware-network.4.log
cron-20200329 messages-20200405 vmware-network.5.log
cron-20200405 messages-20200407 vmware-network.6.log
cron-20200407 private vmware-network.7.log
cups qemu-ga vmware-network.8.log
dnf.librepo.log README vmware-network.9.log
dnf.log rhsm vmware-network.log
dnf.rpm.log samba vmware-vgauthsvc.log.0
firewalld secure vmware-vmsvc.log
gdm secure-20200322 vmware-vmusr.log
glusterfs secure-20200329 wtmp
hawkey.log secure-20200405 wtmp-20200407
hawkey.log-20200322 secure-20200407 Xorg.9.log
you can see the logs already created in the “/var/log” directory.
Just for Verification purposes, try rotating the logs again.
[root@HQEBPRD log]# logrotate -d /etc/logrotate.d/tekneed
WARNING: logrotate in debug mode does nothing except printing debug messages! Consider using verbose mode (-v) instead if this is not what you want.
reading config file /etc/logrotate.d/tekneed
Reading state from file: /var/lib/logrotate/logrotate.status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
Handling 1 logs
rotating pattern: /var/log/tekneed.log weekly (4 rotations)
empty log files are rotated, only log files >= 102400 bytes are rotated, old logs are removed
considering log /var/log/tekneed.log
Now: 2020-04-07 11:40
Last rotated at 2020-04-07 11:29
log does not need rotating (log has been already rotated)
From the screenshot, you can see that all the conditions that were defined in the log rotation configuration file were met.
logrotate In Linux is very easy to manage, all you need to do is to understand how it works and do a proper planning of your environment before setting it up.
Your feedback is welcomed. If you love others, you will share with others
Leave a Reply