Rotating logs for virtual servers

WARNING it is not over, if we leave the configuration as the files error.log and access.log for virtual servers will grow infinitely. Result after 6 months or 1 year the error or access file will make several Gigs result that will be hard to deal with. It is important to put a system of rotation of the logs with a conservation of definite time.

GNU / Linux has the Logrotate system available which is perfect for our needs, it is possible to define a rule for the rotation of your logs. The set of distributions offers rotation mechanism, if we take our machine there is already the set rotation method for the default Apache files is /var/log/apache2/error.log and / var /log/apache2/access.log. Let’s look at the file already in place available: /etc/logrotate.d/apache2

$ cat /etc/logrotate.d/apache2 /var/log/apache2/*.log { weekly missingok rotate 52 compress delaycompress notifempty create 640 root adm sharedscripts postrotate if /etc/init.d/apache2 status > /dev/null ; then /etc/init.d/apache2 reload > /dev/null; fi; endscript prerotate if [ -d /etc/logrotate.d/httpd-prerotate ]; then run-parts /etc/logrotate.d/httpd-prerotate; fi; endscript }

Let’s take a few minutes to analyze this file because it is composed by experienced people and it is a good source of reference we will be able to inspire for our need.

• /var/log/apache2/*.log: This defines the files that will be processed, it is important not to define / var / log / apache2 / * otherwise the files that have been processed in the past will be processed again and this endlessly !

• weekly: The rotation here is executed once a week, it is of course possible to set a daily configuration or a month as needed.

• missingok: The rotation system will not generate an error if there is no file

• rotate 52: The system will keep the files processed for 52 times, as we have defined a weekly rotation this indicates that the apache log files will be kept for 1 year.

• compress: Once the log file is processed the latter will be compressed by default with gzip, this saves disk space, because the text compresses it very well

• delaycompress: Compression processing can be slow especially if the files are large. The goal of setting up a delay for compression is to make sure that all the files are processed relatively at the same time and that they are compressed later. Result if the access log file is processed first then the error file they will be done at the same time, it will be easier to do a correlation of access / error than if the information is in separate file treat in a different time.

• notifyempty: If the file is present but empty an email will be sent to root.

• create 640 root adm: The files created by the rotation system will have the permissions User root and group adm with the permissions in octal 640.

• sharedscripts: This indicates that there are scripts that must be executed during the rotation, in our case postrotate and prerotate.

• postrotate: This defines an instruction that must be executed AFTER the rotation of all files. Let’s read the configuration the valid script that apache is running /etc/init.d/apache2 status if it is running it reloads the apache configuration /etc/init.d/apache2 reload. But why ?!?! When apache starts, it opens a File Descriptor on the file, this is a pointer to the file, when rotating the system logrotate realizes a move of the file. If we do not reload the Apache configuration this would result in apache continuing to write in the original file so that processed by logrotate. The process of reloading the configuration forces apache to redo the connection to the file thus to support the new file created by logrotate.

• prerotate: This defines an instruction that must be executed BEFORE the files are rotated, usually we use this mechanism to process the access and error files with a script provided by our statistics processing software. Since our log files are compressed once the rotation is done, it is easier to read the file before compression: D. In this case, the script validates the presence of the /etc/logrotate.d/httpd-prerotate directory and runs all the scripts in the run-parts /etc/logrotate.d/httpd-prerotate directory.

As we can see this gives a very good starting point to start. Now let’s realize our configuration by inspiring ourselves. We will create the file /etc/logrotate.d/vhosts_apache

/data/vhosts/*/logs/*.conf { daily missingok rotate 365 compress delaycompress create 644 root adm sharedscripts postrotate if /etc/init.d/apache2 status > /dev/null ; then /etc/init.d/apache2 reload > /dev/null; fi; endscript }

I will only highlight the changes made to the configurations, compared to the original apache2 file.

• /data/vhosts/*/logs/*.conf: I use a wildcard (*) to cover all websites under the / data / vhosts directory, this allows me to set up the rotation regardless of the number of site present and so I am sure not to forget the configuration. Just as for the default configuration I consider in rotation that * .conf files.

• daily: Personally, I prefer a daily rotation, especially if you have a lot of traffic.

• rotate 365: it’s a lot of files I wanted to keep the logs 1 year, free to you to judge the relevance of the operation.

• notifyempty: I disabled this feature because I do not want to know it: P.

• create 644 root adm: Here I modify the file permissions to allow ALL to read the content, however ONLY root can write / delete the content. It’s important because I want to give access to the person to access the information, but I do not want anyone to be able to remove information.

• prerotate: At this stage of the deployment I did not set up a file processing so I disabled it, if you want to set up a statistical software there is of course google analytic, but the world of free you have piwik of available which is very good.

So now we have a perennial log solution!