Categories
Life hack

How to remotely collect server events using syslog

The Netop Security Server is a service that provides centralized authentication and authorization in a LAN/WAN environment of users that are to access remote devices where privileged access management is required. Apart from managing remote access the Netop Security Server also acts as a centralized service for log events. These events are sent by Host and/or Guest and are always stored in the Security Server’s database for review and post-processing, but in some cases you would want these events also to be sent to your Syslog server.

In order for the Netop Security Server to be able to send log events to a Syslog server you need the following:

A Syslog Agent that grabs log event entries as they appear in a directory;

Configure the Netop Security Server to send log events to the defined directory;

If you already have a Syslog Server in operation you probably also have a Syslog Agent that you can use for the Security Server. And you would also know how to configure it. If not take a look at KIWI Syslog Server, which is available in a freeware version. And a free Syslog agent is provided by Datagram.

The configuration of the Netop Security Server is as follows:

  1. Create the directory where you want the log events to go;
  2. Open the text file C:\Windows\netop.ini for edit;
  3. Find the section [NSS]
    Add the following line:
    NETOPLOG_DIRECTORY=”PATH”

The default format of the log entry is YYYY-MM-DD HH:MM:SS, HOSTNAME, EVENTTYPE , DESCRIPTION, SERIAL, DTLERR, ERROR
This format will be used unless you define a custom format. The default format definition would be defined like this:

NETOP_LOG_FORMAT=%4.4d-%2.2d-%2.2d %2.2d:%2.2d:00, %s, %s, %s, %d, %d, %d

A sample event for Syslog using the default format may look like this:

2020-10-15 08:53:00, DK-GS , *CFGWUCHK , 0, 26, 0, 0

If you want to use a custom definition it must exist in the same section as the definition under [NSS].

Every enterprise needs to collect and monitor log data from devices across their network to ensure security, troubleshoot operational issues, and conduct forensic analysis on security incidents. For this they might rely on either a log management tool or a SIEM solution. Irrespective of the tool used, collecting logs to a centralized location is more difficult than it appears. From configuring the devices to send log data to central server to ensure security of logs in transit, log collection is as important and tough as any other log management processes.

Predominantly, there are two methods to collect log data—agent-based and agentless. Agent-based log collection requires installation of an agent in every machine that collects and forwards the log data from the device to the central server. When collecting log data from a secured network, agent-based log collection is employed. In other circumstances, this method is not preferred as it is difficult to administer. So, enterprises prefer native log forwarding and at times remote log collection.

When it comes to network devices, Linux/Unix machines, the syslog data can be captured using the log forwarding feature available in the native platform. However, for remotely collecting Windows event logs, the procedure is slightly different.

This page explains the steps needed to remotely collect syslog data using a Syslog server.

How to collect logs remotely using Syslog server?

Collecting syslogs remotely is a fairly simple process involving two steps—configuring the remote server that will centrally collect all log data, and configuring the devices to send log data to the remote server.

Step 1: Configuring the remote server

To configure a syslog server to collect logs remotely,

    Append the following to the /etc/rsyslog.conf file, in the /var/log folder of the server.

$ModLoad imtcp.so

$InputTCPServerRun 514

Here 514 refers to the TCP port number through which the syslog server receives the log data.

Create a variable template to ensure that the logs collected from different hosts do not get mixed up. Add the following to /etc/rsyslog.conf file:

$template

DynamicFile,”/var/log/loghost/%HOSTNAME%/%syslogfacility-text%.log”

*.* -?DynamicFile

/var/log/loghost/*/*.log

# systemctl restart rsyslog

# firewall-cmd –add-port=514/udp –permanent

# firewall-cmd –add-port=514/tcp –permanent

# firewall-cmd –reload

Step 2: Configure the syslog devices

    Add the following to the rules section in/var/rsyslog.conffile

*.* @@ :514

Here, refers to the static IP address of your Syslog server and 514 is the TCP port number through which the log data will be sent.

How to remotely collect Windows event logs?

There are multiple ways to remotely access and collect windows event logs.

  • Using API calls that utilize EvtOpenSession to establish a remote connection and call event log functions.
  • Establishing remote sessions through WMI and run WMI tasks for collecting event logs.
  • Collecting and accessing event logs through Event Viewer UI on an Active Directory account with permissions to read event logs.

Pre-requisites to remotely collect Windows event log:

To access and collect event logs using Event Viewer UI you need an Active Directory service account with specific permissions to access Windows event logs. These permissions can be granted through Local security policy or Group policy object (GPO) in the domain.

Below are the pre-requisite steps that you need to follow to remotely access and collect Windows event logs.

Creating service accounts and providing the required permissions

  • Create a service account and configure it on the remote collector. Another option is to have the account on the collector machine that has proper access so that you can use integrated AD authentication for log collection.
  • Add the account to the following built-in domain groups:
    • Event log readers
    • Distributed COM users
  • Give the service account ‘Manage auditing and security logs’ privilege. This can be done by creating a GPO or using the local security policy.
    • Providing privileges using ‘local security policy’
      • Navigate through the following: Computer configuration >> Windows Settings >> Security settings >> Local policies >> User rights assignment
      • Under User rights assignment, navigate to Manage auditing and security logs and add the service account to the list.
  • If you wish to remotely collect logs through WMI protocol, give this account WMI access by following the below steps:
    • Open ‘wmimgmt’ and right click -> Properties > Security -> Advanced.
    • Allow the service account to “Execute Methods”, “Provider Write”, “Enable Account”, “Remote Enable”.
  • Give registry permissions for this account.
    • Open Regedit -> Local machine ->
      System\CurrentControlSet\Services\eventlog\Security -> right click -> permissions and add the service account.
  • Assign DCOM rights and grant permissions on c:\windows\system32\winevt for the service account.

The service account is now able to read all the logs from any part of the domain through Event Viewer UI. Just a few more steps now.

  1. Enabling connectivity: Edit the Windows firewall rules on the machine on which the service account resides
    • Navigate to Inbound rules and enable Remote event log management (RPC)
    • Ensure that the protocol and profile are respectively specified as ‘TCP’ and ‘Domain’ respectively
  2. Enabling windows collector service: You need to enable the collector service on the remote server for it to receive the log files. To do that, login to the remote server as a local or domain admin and execute the following command in cmd. exe.

You are using an outdated browser. Please upgrade your browser to improve your experience.

–> share-line

–> –> –> –> plus

After you enable remote streaming, vCenter Server starts streaming and only the newly generated events are streamed to the remote syslog server.

All syslog messages begin with a specific prefix. You can distinguish the vCenter Server events from other syslog messages by their Event prefix.

The syslog protocol limits the length of syslog messages to 1024 characters. Messages that are longer than 1024 characters split into multiple syslog messages.

Item Description
syslog-prefix Displays the syslog prefix. The syslog-prefix > is determined by the remote syslog server configuration.
eventId Displays the unique ID of the event message. The default value is Event.
partInfo Displays whether the message is split into parts.
createdTime Displays the time when the event was generated.
eventType Displays the event type.
severity Displays whether the event is a piece information, a warning, or an error.
user Displays the name of the user who generated the event.
target Displays the object the event refers to.
chainId Displays information about the parent or the group ID.
desc Displays the description of the event.

Split of Long Event Message into Multiple Syslog Messages

The X stands for the number of the event message parts.

  • Config
  • windows
  • syslog-ng

How to remotely collect server events using syslog

With the release of syslog-ng Premium Edition 7.0.6, you can collect Windows event logs without installing any third party application on your Windows-based computer.

The benefits are obvious:

  • You don’t need to install any additional application (reducing administrative overhead and possible security risk).
  • You can manage the subscription from Windows Group Policy (permission, certificates, destination).

How does it work?

First of all, install syslog-ng Premium Edition 7.0.6 or newer on your Linux-based computer. In my case, I installed it on Ubuntu Xenial. The event logs will come from a server running Windows Server 2016.

syslog-ng will use the Windows Event Collector (WEC) tool of syslog-ng to collect logs from Windows. This tool is shipping with the syslog-ng installer. WEC uses the native Windows Event Forwarding protocol via subscription to collect the events.

You only need to set up a few things on Windows:

  • Generate and install the certificates because the communication uses HTTPS.
  • configure the required permissions to allow NETWORK SERVICE to access the installed certificate and to read the event logs (by adding it to Event Log Readers group).
  • Set up the forwarder (it’s single-line setting).

syslog-ng:

  • Generate and install the certificates (the same way as on Windows).
  • Set the configuration file.
  • Enable WEC to run as a service.

After you installed syslog-ng Premium Edition, you will find the WEC tool next to syslog-ng, in /opt/syslog-ng/sbin (if you installed syslog-ng using the default path).

The configuration of my WEC is at the end of this blog. This config will allow any computer to send event logs to this WEC (if it passed the certificate check), but will collect only login and logout events from the security container. This is a debug configuration, so don’t forget to set the debug level to info when you are ready.

Just replace the cert and key files with yours then start it as a foreground process.

You can see that WEC is listening on port 5986 and tries to connect to a socket that does not exist (because I haven’t yet started syslog-ng).

And there is another important entry: the connection is established with my Windows 2016 (10.140.1.11 is the IP address of this Windows).

How does the communication happen between WEC and syslog-ng?

  • When WEC reads the event logs from Windows, it writes the logs to the given socket in XML form.
  • syslog-ng reads this socket and parses the incoming XML log. After parsing is done, you can for example forward it to your SIEM via TCP+SSL.

Now, start syslog-ng with the configuration shown at the end of this post. This configuration overrides the default prefix of the windowsevent source, and will store every bit of the XML log in SDATA (this can be useful if you want to forward them for example to a syslog-ng Store Box).

In the WEC log, you will see the following changes:

It could connect to the datagram socket (because syslog-ng has already been started and listening on this socket), Windows began to forward the events to WEC and WEC forwards them to syslog-ng.

in the output file of syslog-ng, you will find the event:

A Windows event log can be quite big, so this is just a little part of the full log.

So now that we have a Windows that forwards the events to the WEC tool that is running on Linux next to syslog-ng, and that WEC tool forwards the logs to syslog-ng also running on Linux. We did not have to install any extra application on Windows.

Only one thing is left, enabling WEC to run as a service (because you don’t want to start it manually):

systemctl enable syslog-ng-wec (to allow to start at boot)
systemctl start syslog-ng-wec

For more details, please read the Windows Event Collector section of the Administrator’s Guide.

The Remote Logging options under Status > System Logs on the Settings tab allow syslog to copy log entries to a remote server.

The logs kept by pfSenseВ® software on the firewall itself are of a finite size. Copying these entries to a syslog server can aid troubleshooting and allow for long-term monitoring. Having a remote copy can also help diagnose events that occur before a firewall restarts or after they would have otherwise been lost due to clearing of the logs or when older entries are cycled out of the log, and in cases when local storage has failed but the network remains active.

Corporate or local legislative policies may dictate the length of time logs must be retained from firewalls and similar devices. If an organization requires long-term log retention for their own or government purposes, a remote syslog server is required to receive and retain these logs.

Logs sent using this method are delivered in the clear (not encrypted) unless the logs are sent through a VPN or using a mechanism such as Stunnel package . As an alternative, consider using the syslog-ng package which supports encrypted syslog.

The following options are available for remote logging:

Controls where the syslog daemon binds for sending out messages. In most cases, the default (Any) is the best option, so the firewall will use the address nearest the target. If the destination server is across a tunnel mode IPsec VPN, however, choosing an interface or Virtual IP address inside the local Phase 2 network will allow the log messages to flow properly over a tunnel.

When choosing an interface for the Source Address, this option gives the syslog daemon a preference for either using IPv4 or IPv6, depending on which is available. If there is no matching address for the selected type, the other type is used instead.

Remote Log Servers

Enter up to three remote servers using the boxes contained in this section. Each remote server can use either an IP address or hostname, and an optional UDP port number. If the port is not specified, the default syslogd port, 514 , is assumed.

A syslog server is typically a server that is directly reachable from the firewall on a local interface. Logging can also be sent to a server across a VPN.

Do not send log data directly across any WAN connection or unencrypted site-to-site link, as it is plain text and could contain sensitive information.

The syslog daemon only supports sending messages over UDP. To send syslog messages over TCP, consider using the syslog-ng package.

The options in this section control which log messages will be sent to the remote log server.

When set, all log messages from all areas are sent to the server.

Main system log messages that do not fall into other categories.

Firewall log messages in raw format. The format of the raw log is covered in Raw Filter Log Format .

Messages from the DNS Resolver ( unbound ), DNS Forwarder ( dnsmasq ), and from the filterdns daemon which periodically resolves hostnames in aliases.

Messages from the IPv4 and IPv6 DHCP daemons, relay agents, and clients.

Messages from PPP WAN clients (PPPoE, L2TP, PPTP)

General Authentication Events

Log messages about authentication events, such as for the GUI or certain types of VPNs.

Captive Portal Events

Messages from the Captive Portal system, typically authentication messages and errors.

Messages from VPN daemons such as IPsec and OpenVPN, as well as the L2TP server and PPPoE server.

Gateway Monitor Events

Messages from the gateway monitoring daemon, dpinger

Routing Daemon Events

Routing-related messages such as UPnP/NAT-PMP, IPv6 routing advertisements, and routing daemons from packages like OSPF, BGP, and RIP.

Network Time Protocol Events

Messages from the NTP daemon and client.

Messages from the Wireless AP daemon, hostapd .

To start logging remotely:

Navigate to Status > System Logs on the Settings tab

Check Send log messages to remote syslog server

Configure the options as described above

Click Save to store the changes.

If a syslog server is not already available, it is fairly easy to set one up. Almost any UNIX or UNIX-like system can be used as a syslog server. FreeBSD is described in the following section, but others may be similar.

Syslog configuration for remote logservers for syslog-ng and rsyslog, both client and server

❗ This post is over two years old. It may no longer be up to date. Opinions may have changed.

Table of Contents

  • Server: rsyslog
    • Logrotate
  • Client: rsyslog (Ubuntu)
  • Server: syslog-ng
  • Client: syslog-ng
  • What about systemd / journald?

How to remotely collect server events using syslog

A Teletype ASR-33 printing system output

Syslog is the protocol, format (and software) linux and most networking devices use to log messages. All kinds of messages, system, authentication, login and applications. There are multiple implementations of syslog, like syslog-ng and rsyslog. Syslog has the option to log to a remote server and to act as a remote logserver (that receives logs). With a remote logging server you can archive your logs and keep them secure (when a machine gets hacked, if root is compromised the logs on the machine are no longer trustworthy). This tutorial shows how to set up a syslog server with rsyslog and syslog-ng and shows how to setup servers as a syslog client (that log to a remote server) with syslog-ng and rsyslog.

Server: rsyslog

rsyslog is the default syslog service on Ubuntu, Debian, OpenSUSE and CentOS (next to systemd’s journald). The configuration syntax is simpler than syslog- ng’s, but complex configuration is more clear in syslog-ng. Bottom line they both work just as well. The below steps are to be taken to setup rsyslog as a syslog service to receive syslogs.

Edit the following file:

Add the following:

This will allow all hosts in the subnet 192.0.2.0/24 to log to this machine.

Restart rsyslog to make the changes active:

The files will be placed in /var/log/remote , sorted on hostname. For example:

When you just configured a client, it will take some time (a few minutes) before the logs and folder appear under /var/log/remote .

Logrotate

As you can see logging can take up some space, I recommend to setup logrotate for this remote folder. You can do so on Ubuntu by creating the following logrotate config file:

This will compress and rotate logs every day and keep them for 90 days (3 months). To test your config, use the following command:

(that will rotate all your logs, don’t CTRL+C it otherwise your log folder will be messed up)

You don’t have to restart a service since logrotate is ran via cron ( /etc/cron.daily/logrotate ).

Client: rsyslog (Ubuntu)

On Ubuntu or any rsyslog server, to log to a remote syslogserver, add the following to rsyslog.conf :

(Replace 192.0.2.10 with the IP or hostname of your syslog server)

The file can be either:

  • /etc/rsyslog.conf
  • /etc/rsyslog.d/99-remote.conf

Restart rsyslog to make the changes active:

Server: syslog-ng

syslog-ng is the default on older versions of SUSE Enterprise Linux and OpenSUSE next to systemd’s journald and on HP-UX. Most older distro’s use it as well, Debian, Fedora and Arch all had it as their default years ago.

To set up syslog-ng as a remote log server that can receive logs, edit the following file:

This file can also be in /etc/syslog-ng/conf.d/ under a different name. Restart syslog-ng to make the changes active:

This will place the logfiles in /var/log/remote . As far as I could find in the documentation, there is no option to limit on subnet like rsyslog has in the above example. Use the firewall to allow access from different networks.

With syslog-ng it is also recommended to setup logrotate and compression. See the rsyslog server section on how to do that.

Client: syslog-ng

The setup for sending logs to a remote syslog server is simple. Edit the syslog-ng.conf file:

Add or edit the following:

Older versions do not support the network() syntax, you need to use the older tcp() or udp() syntax:

In both cases, replace 192.0.2.10 with your logserver’s IP. (Unless you are using TEST-NET-1 of course).

This file can also be in /etc/syslog-ng/conf.d/ under a different name. Restart syslog-ng to make the changes active:

What about systemd / journald?

Systemd and journald are taking over every part of your linux system including logging. Most distro’s supply a syslog service which journald (systemd’s binary logging component) forwards logs to. If your system is not set up like that, you need to install either rsyslog or syslog-ng and tell journald to forward the logs to syslog:

If your syslog-ng or rsyslog version is recent enough, all journald logs will now appear in syslog as well.

Syslog is an event logging protocol that is common to Linux. Applications will send messages that may be stored on the local machine or delivered to a Syslog collector. When the Log Analytics agent for Linux is installed, it configures the local Syslog daemon to forward messages to the agent. The agent then sends the message to Azure Monitor where a corresponding record is created.

This article covers collecting Syslog events with the Log Analytics agent which is one of the agents used by Azure Monitor. Other agents collect different data and are configured differently. See Overview of Azure Monitor agents for a list of the available agents and the data they can collect.

Azure Monitor supports collection of messages sent by rsyslog or syslog-ng, where rsyslog is the default daemon. The default syslog daemon on version 5 of Red Hat Enterprise Linux, CentOS, and Oracle Linux version (sysklog) is not supported for syslog event collection. To collect syslog data from this version of these distributions, the rsyslog daemon should be installed and configured to replace sysklog.

How to remotely collect server events using syslog

The following facilities are supported with the Syslog collector:

  • kern
  • user
  • mail
  • daemon
  • auth
  • syslog
  • lpr
  • news
  • uucp
  • cron
  • authpriv
  • ftp
  • local0-local7

For any other facility, configure a Custom Logs data source in Azure Monitor.

Configuring Syslog

The Log Analytics agent for Linux will only collect events with the facilities and severities that are specified in its configuration. You can configure Syslog through the Azure portal or by managing configuration files on your Linux agents.

Configure Syslog in the Azure portal

Configure Syslog from the Agent configuration menu for the Log Analytics workspace. This configuration is delivered to the configuration file on each Linux agent.

You can add a new facility by clicking Add facility. For each facility, only messages with the selected severities will be collected. Check the severities for the particular facility that you want to collect. You cannot provide any additional criteria to filter messages.

How to remotely collect server events using syslog

By default, all configuration changes are automatically pushed to all agents. If you want to configure Syslog manually on each Linux agent, then uncheck the box Apply below configuration to my machines.

Configure Syslog on Linux agent

When the Log Analytics agent is installed on a Linux client, it installs a default syslog configuration file that defines the facility and severity of the messages that are collected. You can modify this file to change the configuration. The configuration file is different depending on the Syslog daemon that the client has installed.

If you edit the syslog configuration, you must restart the syslog daemon for the changes to take effect.

rsyslog

The configuration file for rsyslog is located at /etc/rsyslog.d/95-omsagent.conf. Its default contents are shown below. This collects syslog messages sent from the local agent for all facilities with a level of warning or higher.

You can remove a facility by removing its section of the configuration file. You can limit the severities that are collected for a particular facility by modifying that facility’s entry. For example, to limit the user facility to messages with a severity of error or higher you would modify that line of the configuration file to the following:

syslog-ng

The configuration file for syslog-ng is location at /etc/syslog-ng/syslog-ng.conf. Its default contents are shown below. This collects syslog messages sent from the local agent for all facilities and all severities.

You can remove a facility by removing its section of the configuration file. You can limit the severities that are collected for a particular facility by removing them from its list. For example, to limit the user facility to just alert and critical messages, you would modify that section of the configuration file to the following:

Collecting data from additional Syslog ports

The Log Analytics agent listens for Syslog messages on the local client on port 25224. When the agent is installed, a default syslog configuration is applied and found in the following location:

  • Rsyslog: /etc/rsyslog.d/95-omsagent.conf
  • Syslog-ng: /etc/syslog-ng/syslog-ng.conf

You can change the port number by creating two configuration files: a FluentD config file and a rsyslog-or-syslog-ng file depending on the Syslog daemon you have installed.

The FluentD config file should be a new file located in: /etc/opt/microsoft/omsagent/conf/omsagent.d and replace the value in the port entry with your custom port number.

For rsyslog, you should create a new configuration file located in: /etc/rsyslog.d/ and replace the value %SYSLOG_PORT% with your custom port number.

If you modify this value in the configuration file 95-omsagent.conf , it will be overwritten when the agent applies a default configuration.

The syslog-ng config should be modified by copying the example configuration shown below and adding the custom modified settings to the end of the syslog-ng.conf configuration file located in /etc/syslog-ng/ . Do not use the default label %WORKSPACE_ID%_oms or %WORKSPACE_ID_OMS, define a custom label to help distinguish your changes.

If you modify the default values in the configuration file, they will be overwritten when the agent applies a default configuration.

After completing the changes, the Syslog and the Log Analytics agent service needs to be restarted to ensure the configuration changes take effect.

Syslog record properties

Syslog records have a type of Syslog and have the properties in the following table.

Property Description
Computer Computer that the event was collected from.
Facility Defines the part of the system that generated the message.
HostIP IP address of the system sending the message.
HostName Name of the system sending the message.
SeverityLevel Severity level of the event.
SyslogMessage Text of the message.
ProcessID ID of the process that generated the message.
EventTime Date and time that the event was generated.

Log queries with Syslog records

The following table provides different examples of log queries that retrieve Syslog records.

Remote logging

Standard system log management configuration rotates log files every week and retains them for four rotations. It is often desirable to maintain logs longer than the four-week default, especially when establishing system performance trends related to tasks, such as month-end financial closings, which are executed just once a month. By sending log messages to a remote log host with dedicated mass storage, administrators can maintain large archives of system logs for their systems without changing the default log rotation configuration, which is intended to keep logs from overconsuming disk storage.

Central collection of system log messages can also be very useful for monitoring the state of systems and for quickly identifying problems. It also provides a backup location for log messages in case a system suffers a catastrophic hard drive failure or other problems, which cause the local logs to no longer be available. In these situations, the copy of the log messages which reside on the central log host can be used to help diagnose the issue that caused the problem.

Standardized system logging is implemented in Red Hat Enterprise Linux 7 by the rsyslog service. System programs can send syslog messages to the local rsyslogd service, which will then redirect those messages to files in /var/log, remote log servers, or other databases based on the settings in its configuration file, /etc/rsyslog.conf.

Log messages have two characteristics that are used to categorize them. The facility of a log message indicates the type of message it is. The priority, on the other hand, indicates the importance of the event logged in the message.

Syslog Priority Levels

Priority Meaning
emerg System is unusable
alert Immediate action required
crit Critical condition
err Error condition
warning Warning condition
notice Normal but significant condition
info Informational messages
debug Debugging messages

Configuring a central log host

The implementation of a central log host requires the configuration of the rsyslog service on two types of systems: the remote systems where the log messages originate from and the central log host receiving the messages. On the central log host, the rsyslog service needs to be configured so that log messages from remote hosts are accepted.

To configure the rsyslog service on the central log host to accept remote logs, uncomment either the TCP or UDP reception lines in the modules section in the /etc/rsyslog.conf file.

For UDP reception:

for TCP reception:

TCP provides more reliable delivery of remote log messages, but UDP is supported by a wider variety of operating systems and networking devices.

The rules contained in /etc/rsyslog.conf are configured by default to accommodate the logging of messages on a single host. Therefore, it sorts and bundles messages by the facility. For example, mail messages are funneled into /var/log/maillog while messages generated by
the crond daemon are consolidated into /var/log/cron to facilitate locating each type of message.

While sorting of messages by the facility is ideal on a single host, it produces an undesirable result on a central log host since it causes messages from different remote hosts to be mixed with each other. On a central log host, it is usually more optimal for log messages from remote systems to remain separate from each other. This separation can be achieved by defining dynamic log file names using the template function of rsyslog.

Templates are defined in /etc/rsyslog.conf and can be used to generate rules with dynamic log file names. A template definition consists of the $template directive, followed by a template name, and then a string representing the template text. The template text can be made dynamic by making use of values substituted from the properties of a log message. For example, to direct cron syslog messages from different systems to different files on a central log host, use the following template to generate dynamic log file names based on the HOSTNAME property of each message:

The dynamic file name created using the template definition can then be referenced by the template name in a rule as follows:

On systems performing extremely verbose logging, it may be desirable to turn off syncing of the log file after each writes operation in order to improve performance. The syncing of a log file after every logging can be omitted by prefixing the log file name with the minus (-) sign in a logging rule. However, the trade-off of improved performance does create the possibility of log data loss if the system crashes immediately after a write attempt.

The following is another example of the use of templates to generate dynamic log file names. In this example, remote log messages will be sorted by their host name and facility values by referencing the HOSTNAME and syslogfacility-test properties. Log messages will be written to the dynamically generated log file names and no syncing will be performed after the write operation.

Once syslog reception has been activated and the desired rules for log separation by host has been created, restart the rsyslog service for the configuration changes to take effect. In addition, add the necessary UDP and/or TCP firewall rules to allow incoming syslog traffic and then reload firewalld.

When new log files are created, they may not be included by the log host’s existing log rotation schedule. This should be remedied to ensure that the new log files do not grow to unmanageable sizes. For instance, to include the new log files from the previous examples in log rotation, add the following entry to the list of log files in the /etc/logrotate.d/syslog configuration file.

Redirecting logging to central log host

Once the central log host is configured to accept remote logging, the rsyslog service can be configured on remote systems to send logs to the central log host. To configure a machine to send logs to a remote rsyslog server, add a line to the rules section in the /etc/rsyslog.conf file. In place of the file name, use the IP address of the remote rsyslog server. To use UDP, prefix the IP address with a single @ sign. To use TCP, prefix it with two @ signs (@@).

For instance, to have all messages with info or higher priority sent to loghost.example.com via UDP, use the following line:

To have all messages sent to loghost.example.com via TCP, use the following line:

Optionally, the log hostname can be appended with :PORT, where PORT is the port that the remote rsyslog server is using. If no port is given, it assumes the default port 514.

After adding the rule(s), restart the rsyslog service and send a test message using the logger command:

Check the logs on the remote server to ensure the message was received.

  • Config
  • windows
  • syslog-ng

How to remotely collect server events using syslog

With the release of syslog-ng Premium Edition 7.0.6, you can collect Windows event logs without installing any third party application on your Windows-based computer.

The benefits are obvious:

  • You don’t need to install any additional application (reducing administrative overhead and possible security risk).
  • You can manage the subscription from Windows Group Policy (permission, certificates, destination).

How does it work?

First of all, install syslog-ng Premium Edition 7.0.6 or newer on your Linux-based computer. In my case, I installed it on Ubuntu Xenial. The event logs will come from a server running Windows Server 2016.

syslog-ng will use the Windows Event Collector (WEC) tool of syslog-ng to collect logs from Windows. This tool is shipping with the syslog-ng installer. WEC uses the native Windows Event Forwarding protocol via subscription to collect the events.

You only need to set up a few things on Windows:

  • Generate and install the certificates because the communication uses HTTPS.
  • configure the required permissions to allow NETWORK SERVICE to access the installed certificate and to read the event logs (by adding it to Event Log Readers group).
  • Set up the forwarder (it’s single-line setting).

syslog-ng:

  • Generate and install the certificates (the same way as on Windows).
  • Set the configuration file.
  • Enable WEC to run as a service.

After you installed syslog-ng Premium Edition, you will find the WEC tool next to syslog-ng, in /opt/syslog-ng/sbin (if you installed syslog-ng using the default path).

The configuration of my WEC is at the end of this blog. This config will allow any computer to send event logs to this WEC (if it passed the certificate check), but will collect only login and logout events from the security container. This is a debug configuration, so don’t forget to set the debug level to info when you are ready.

Just replace the cert and key files with yours then start it as a foreground process.

You can see that WEC is listening on port 5986 and tries to connect to a socket that does not exist (because I haven’t yet started syslog-ng).

And there is another important entry: the connection is established with my Windows 2016 (10.140.1.11 is the IP address of this Windows).

How does the communication happen between WEC and syslog-ng?

  • When WEC reads the event logs from Windows, it writes the logs to the given socket in XML form.
  • syslog-ng reads this socket and parses the incoming XML log. After parsing is done, you can for example forward it to your SIEM via TCP+SSL.

Now, start syslog-ng with the configuration shown at the end of this post. This configuration overrides the default prefix of the windowsevent source, and will store every bit of the XML log in SDATA (this can be useful if you want to forward them for example to a syslog-ng Store Box).

In the WEC log, you will see the following changes:

It could connect to the datagram socket (because syslog-ng has already been started and listening on this socket), Windows began to forward the events to WEC and WEC forwards them to syslog-ng.

in the output file of syslog-ng, you will find the event:

A Windows event log can be quite big, so this is just a little part of the full log.

So now that we have a Windows that forwards the events to the WEC tool that is running on Linux next to syslog-ng, and that WEC tool forwards the logs to syslog-ng also running on Linux. We did not have to install any extra application on Windows.

Only one thing is left, enabling WEC to run as a service (because you don’t want to start it manually):

systemctl enable syslog-ng-wec (to allow to start at boot)
systemctl start syslog-ng-wec

For more details, please read the Windows Event Collector section of the Administrator’s Guide.