syslog

What is syslog?

syslog is the standard logging system used on Unix-like operating systems that helps you easily log messages to a centralised location and later inspect them.

What problems does syslog solve?

On a Unix system, lots of things want to write logs:

• The kernel
• System services (cron, ssh, systemd)
• Applications (web servers, databases, shell scripts)
• Network devices (routers, firewalls)

Instead of every program inventing its own logging mechanism, they can all send messages to syslog which can decide:

• Where the log goes (file, console, remote server)
• What gets logged
• How it’s formatted

Core syslog components

1. Log producers (clients)

Programs can send log messages in different ways:

• Using the syslog() system call (C / native)
• Using libraries in other languages (PHP, Python, Go, Java, etc)
• Using the logger CLI tool

Here’s an example using the logger CLI tool:

logger "A user just signed up!"

2. syslog daemon

This is the background process that receives and processes logs.

Common implementations include:

• syslogd
• rsyslog (common on Linux)
• syslog-ng
• journald (systemd‘s logging system which often feeds into syslog)

The daemon listens on Unix sockets/ports/files, applies rules and writes the logs to particular destinations.

3. Log destinations

Logs can go to:

• Files: /var/log/syslog, /var/log/messages, /var/log/auth.log
• Console / TTY
• Remote servers for centralized logging
• Databases or other log processers (ELK, Loki, etc).

How is a syslog message structured?

Each message has metadata that gives more details about the log. Here’s an example syslog entry:

Facility: indicates who sent it

• kern – kernel
• auth / authpriv – authentication
• cron – scheduled jobs
• daemon – system services
• user – user processes
• local0-local7 – custom app usage

Severity: indicates importance

Level	Keyword
0	emergency
1	elert
2	Critical
3	error
4	warning
5	notice
6	info
7	debug

How to write a log message using the logger utility

Unix system provide a logger command-line tool that you can use to generate syslog messages.

A simple log message

logger "Hello there"

By default, if you just pass a message like above, the logger utility will use the user facility and the notice severity.

Specify facility and severity (-p)

logger "hello there" -p local0.info "Hello there"

Specify tag (-t)

You typically want to tag your message using the name of your script so you can then search your logs using the script name:

logger -t myscript -p local0.info "Hello there"

Specify process ID (-i)

logger -i myscript "Hello there"

Specifying the process ID is useful when you want to distinguish between log messages sent by different instances of your program.

Send message to the screen and to the logging system (-s)

logger -s "Hello there"

Example of a custom logger function

function log() {
  logger -t $0 -i -is $1
}

How to read logs

On a typical Ubuntu machine, logs are located in /var/log. So if you want to inspect auth logs, you can run:

sudo tail -f /var/logs/auth.log

If you want to inspect user logs, you can run:

sudo tail -f /var/logs/user.log