Using syslogd and logrotate to manage the server's log files

First note that on our machines the newer syslogd-ng is used as a more powerful replacement for the traditional syslogd. See man syslog-ng and man syslog for all details.

Facilities and severity levels

syslogd uses selectors to identify the program (called the facility) that is sending a log message and the message's severity level. So a selector has the general form facility.level. For example, the selector mail.warning specifies that a program that somehow deals with mail (a mail reader, a mail transfer agent, a random mail producer, ...) wants to log a warning message. Both facility names and levels must be chosen from a short list of predefined values. No, you cannot invent your own facilities or levels. You have to use the predefined levels, and you have to group your programs into the predefined facility groups. (But syslog-ng allows you to filter messages by regular expression matches, which you could use to filter out all messages that exclusively belong to a program called "therollingstones1965" by prepending this unique string to all messages of this and only this program.)

Facilities are defined for the kernel, for common groups of utilities, and for locally written programs. Everything else is classified under the generic facility user. The following facilities exist as defined in /usr/include/sys/syslog.h:

/* facility codes */
#define LOG_KERN        (0<<3)  /* kernel messages */
#define LOG_USER        (1<<3)  /* random user-level messages */
#define LOG_MAIL        (2<<3)  /* mail system */
#define LOG_DAEMON      (3<<3)  /* system daemons */
#define LOG_AUTH        (4<<3)  /* security/authorization messages */
#define LOG_SYSLOG      (5<<3)  /* messages generated internally by syslogd */
#define LOG_LPR         (6<<3)  /* line printer subsystem */
#define LOG_NEWS        (7<<3)  /* network news subsystem */
#define LOG_UUCP        (8<<3)  /* UUCP subsystem */
#define LOG_CRON        (9<<3)  /* clock daemon */
#define LOG_AUTHPRIV    (10<<3) /* security/authorization messages (private) */
#define LOG_FTP         (11<<3) /* ftp daemon */
#define LOG_LOCAL0      (16<<3) /* reserved for local use */
#define LOG_LOCAL1      (17<<3) /* reserved for local use */
#define LOG_LOCAL2      (18<<3) /* reserved for local use */
#define LOG_LOCAL3      (19<<3) /* reserved for local use */
#define LOG_LOCAL4      (20<<3) /* reserved for local use */
#define LOG_LOCAL5      (21<<3) /* reserved for local use */
#define LOG_LOCAL6      (22<<3) /* reserved for local use */
#define LOG_LOCAL7      (23<<3) /* reserved for local use */

(So the names of the facilities are kern, user, and so on.)

The following severity levels exist, in descending order of severity:

#define LOG_EMERG       0       /* system is unusable */
#define LOG_ALERT       1       /* action must be taken immediately */
#define LOG_CRIT        2       /* critical conditions */
#define LOG_ERR         3       /* error conditions */
#define LOG_WARNING     4       /* warning conditions */
#define LOG_NOTICE      5       /* normal but significant condition */
#define LOG_INFO        6       /* informational */
#define LOG_DEBUG       7       /* debug-level messages */

(So the names of the levels are emerg, alert, and so on.)

The severity level of a message specifies its importance. In the syslog-ng.conf file, levels indicate the minimum importance that a message must have in order to be logged upon a call to the syslog() function. For example, if syslog-ng.conf specifies that mail.info messages be logged to a file, then mail.warning messages will go there also unless there is a separate rule for these higher level messages.

Configuring syslogd-ng

The configuraton file for syslogd-ng is /etc/sylog-ng/syslog-ng.conf. You must be root to edit this file.

Suppose that we want to use the syslogd-ng to log messages for our own server, no matter what their severity level is. We decide to choose the (predefined!) local0 facility for our server program. (This is tantamount to say that our server belongs to a group of programs called local0.) We have to specify a destination file for the log, set up a filter that filters out all messages that originate from local0 programs, and connect the source of all log messages via the filter to the destination:

# file /etc/sylog-ng/syslog-ng.conf
destination myserverlog { file("/var/log/myserverlog" 
                          owner("root") 
                          group("adm") 
                          perm(0640)); };  

filter f_myserver { facility(local0); };  # all priorities match!

log { source(src); filter(f_myserver);  destination(myserverlog); };

Of course you will ask now what the source is. According to one of the first lines in /etc/sylog-ng/syslog-ng.conf:

# This is the default behavior of sysklogd package
# Logs may come from unix stream, but not from another machine.
#
source src { unix-stream("/dev/log"); internal(); file("/proc/kmsg" log_prefix("kernel: ")); };

(As a user, you do not have worry about the source. The syslog() function will write its messages to the /dev/log stream.)

Note that with syslog-ng you can write very complex filter rules to filter out only the messages that you really want. These rules even allow regular expression matching. A common use is not only to filter by facility, but also by level.

Now restart syslogd-ng:

# kill -HUP `/bin/cat /var/run/syslog-ng.pid`

Logging from the command line

You can test the newly created logging selector by the logger program. You do not have to be root to do this! (Of course, any user level program should be able to log its messages.) We log a demo message from a local0 facility with the info level:

$ logger -p local0.info "This is a test line"

# tail -f /var/log/myserverlog
Nov 18 20:00:55 mpino1301 logger: This is a test line

Note that there is no information about the facility and the level in the log file. (It is implicitly clear by the rules above that myserverlog only contains local0 messages.) If we only wanted to log info messages, we could create a separate destination file myserverlog.info together with a corresponding filter rule to filter out info messages of local0 facilities.

Logging from C programs

Here is a short C program (syslogdemo.c) that opens the log for the local0 facility, writes a log messages with the warning level into the log, then writes a log messages with the err level into the log, and finally closes the log again.

#include <syslog.h>

int main(int argc, char**argv)
{
  openlog(argv[0],            /* string constant prepended to every message */
                LOG_PID,        /* option: include PID with each message */
                LOG_LOCAL0 /* facility to log */);

  syslog(LOG_WARNING, "Just logged %s", "this warning.");

  syslog(LOG_ERR, "There were %d floating point errors.", 4711);

  closelog();
}

The manual page of syslog will give you all the details (but there are not many). The log file now ends by the follwing lines:

Nov 18 20:30:14 mpino1301 ./syslogdemo[21388]: Just logged this warning.
Nov 18 20:30:14 mpino1301 ./syslogdemo[21388]: There were 4711 floating point errors.

Note that these lines contain the name of the program (syslogdemo) and its PID (21388).

Also note that neither the facility name nor the severity level are contained in these lines. By the rules in syslog-ng.conf above, this information is implicitly contained in the file name. If we only wanted to log warning messages, we could introduce a separate destination file myserverlog.warning together with a corresponding filter:

# file /etc/sylog-ng/syslog-ng.conf
destination myserverlogwarn { file("/var/log/myserverlog.warning" 
                                                 ... };  
filter f_myserverwarn { facility(local0) and level(warning); };  
log { source(src); filter(f_myserverwarn);  destination(myserverlogwarn); };

With these additional lines in /etc/sylog-ng/syslog-ng.conf, both files myserverlog and myserverlog.warning would be filled by the above C program!

Using logrotate to keep the log file small

At some point in time, the server's log file may become too large. Then it is time to use the logrotate utility.

Edit the file /etc/logrotate.conf and insert the following lines. This configuration rotates /var/log/myserverlog once every week, keeping five versions of the file (one current and four older ones). All noncurrent versions of the log file are compressed. The log file is not rotated if it is empty. Error notifications are sent to bast@mpi-inf.mpg.de.

/var/log/myserverlog {
  weekly
  rotate 5
  compress
  notifempty
  errors bast@mpi-inf.mpg.de
}

logrotate is normally run out of cron once a day.

man logrotate gives you all the details.

CompleteSearch: completesearch/SyslogdAndLogrotate (last edited 2007-11-19 09:25:37 by mpiat1403)