Closing the Gaps in Linux Auditing & Detection Strategies

Linux systems are often overlooked when setting up security auditing and threat detection strategies. The main reason is that Linux auditing is far less explored by the security community than Windows auditing.

A survey done by W3Tech shows that among the websites whose OS is known to survey editors, 55.7% of them use Linux OS, opposed to 11.7% using Windows. 

Although it doesn’t tell us anything about the overall percentage of hosts running Linux OS in an average organization, it shows us that Linux is a lot more common than we might think.

From the perspective of Cyber Security, according to Elastic’s 2023 Global Threat Report, Linux is the most attacked server environment. Over half of all cloud-based malware infections target Linux systems, opposed to 39% percent of malware infections targeting Windows OS.

With this in mind, in this article we will break down how Linux operates and how it differs from how Windows operates. We review how data is generated on Linux using auditd, and how to leverage this data to detect suspicious activity.

Linux vs. Windows, from the Ground Up

Philosophical Differences: Core Principals

From its foundations in UNIX, Linux’s philosophy differs greatly from Windows. According to Peter H. Salus, the philosophy of UNIX is based on three principles:

• Write programs that do one thing and do it well.
  • This principle differs from Windows, which prioritizes ease of use and graphical interfaces, making it more accessible for everyday users. Linux follows UNIX principles, prioritizing efficiency, flexibility, and scriptability, which makes it more appealing for developers and power users.
• Write programs to work together.
  • Linux follows the UNIX philosophy of composability and interoperability, where programs can seamlessly work together using text streams and pipes (|). Windows, in contrast, was originally designed for individual applications to run independently, primarily through graphical interfaces. However, modern Windows PowerShell introduces object-based pipelines, allowing a similar but structured approach
• Write programs to handle text streams, because they’re universal interfaces.
  • UNIX systems are designed to handle text streams (in the form of stdin, or standard input), process data, and then write to stdout (standard output). Windows, on the other hand, was built primarily around a Graphical User Interface (GUI) philosophy, meaning many system operations are performed through graphical tools instead of command-line utilities.

Tactical Differences: Shell Commands and Log Structures

One key difference that significantly affects their respective auditing and threat detection workflows is how each OS processes shell commands.

In Linux, the shell itself handles redirections and process streams before forwarding commands to the kernel for execution. This means that security monitoring tools like auditd track syscalls but may only capture the final executed command, not the entire pipeline of redirections and transformations. In contrast, PowerShell processes commands after execution, capturing the entire sequence of execution—including what happens within pipes and filters.

Another key difference is their respective log structures. We will dive deep into the auditd log structure later but for now know this: Linux logs will not present all information needed to act on a threat in a single log, whereas Windows does. The security engineer will need to piece together the information necessary to see the full picture.

Understanding these differences puts detection engineers, especially those more familiar with Windows, on the right track to create a detection strategy suitable for Linux systems. 

Native Linux System Auditing

Understanding the Auditing System

Linux, like many other systems, comes with a pre-installed auditing system. This system has the ability to log data such as network traffic events, authentication events, process creation events and much more (check out this documentation for more detail). This auditing system has a Daemon, or a process running in the background that is not interactive. It takes events captured by auditing systems and writes them into the log files. The Daemon’s name is auditd (D for daemon). The audit daemon is instructed by rules that reside in “audit.rules” file.

The basic working principle of the auditing system is built upon two parts: the user-space applications and the kernel. The user-space application (for instance, Bash) sends system calls to the kernel, which in turn filters the calls through one of the three filters: user, task, or exit. Those filters send the record of the system calls to the fourth filter, exclude, that enlists the help of the rules file to decide if this call should be logged or excluded. If the rules file does decide the call should be logged, it is passed on to the audit daemon.

Introduction to Audit Rules

As described in the previous section, the audit daemon relies on the rules configured in the “audit.rules” file to determine which actions should be written to the log file (and which shouldn’t). To all the Windows specialists, you can think of those audit rules as the audit policy that is configured through the group policy.

This file is located in “/etc/audit/” and generated automatically by combining the rule files that are located in “/etc/audit/rules.d”. The reasoning behind this structure is as follows:

Imagine a large organization where a Linux server is serving several departments. Each department needs to audit different data, paths, and calls relevant to their function. For example, the sales department needs to audit access to the directory “/home/salesadmin/leads” because it contains PII. Meanwhile, the marketing department needs to audit access to the directory “/home/marketadmin/vendors” because it contains sensitive business data and vendor details. In this case, each department can maintain its own version of the .rules file, which will in turn be combined to a single file named “audit.rules” upon booting the system.

Rules Syntax

Now let’s break down the rule syntax. We will look at the two most effective rule types and break them down, Syscalls and Filesystem rules (If you want to read the full documentation of the audit.rules file, click here).

Filesystem Rules

Filesystem rules are rules that specify which files and access types to audit, following this pattern:

-w {path-to-file} -p {permissions} -k {keyname}

• The “-w” flag signifies this is filesystem rule, which will always starts with “-w”.
• Next, {path-to-file} specifies the file’s full path, starting at the root directory (for example “/etc/passwd”).
• The next flag is “-p” which stands for “permissions.” This flag is followed by a combination of one of the following options:
  • “r” – Read
  • “w” – Write
  • “x” – Execute
  • “a” – Changes in attribute
• The last (optional) part of the rule is the -k flag. This flag is followed by a string that the user chooses to add a unique marking to the rule.

Let’s look at a real rule to demonstrate what we’ve learned:

-w /etc/default/grub -p wa -k grub_modification

This rule will audit write actions and attribute modifications made to the file “/etc/default/grub.” Its key name is “grub_modification”.

After running the command “nano /etc/default/grub” (with some slight changes), this is the log that was generated:

type=SYSCALL msg=audit(1741173887.548:548698): arch=c000003e syscall=257 success=yes exit=3 a0=ffffff9c a1=5c4e846714b0 a2=241 a3=1b6 items=2 ppid=683566 pid=683724 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=3661 comm=”nano” exe=”/usr/bin/nano” subj=unconfined key=”grub_modification” ARCH=x86_64 SYSCALL=openat AUID=”azureuser” UID=”root” GID=”root” EUID=”root” SUID=”root” FSUID=”root” EGID=”root” SGID=”root” FSGID=”root”

If you look closely, you can see that the field “key” has the value “grub_modification”, just like the key in the rule. The “comm” field (command) holds the value of “nano”, from the editor used to edit this file.

Syscall Rules

Syscall rules are rules that specify what system calls should be audited and the criteria it should be audited by. It follows this pattern:

-a {action},{filter} -S {system_call} -F {field=value} -k {key_name}

• The -a at the beginning signifies that this is a syscall rule, which will always start with -a.
• The “action” specifies when a certain event will be logged. The two existent options are “always” that tells the system to always create an event when the terms of the rule are met, or “never” that will work as an exclusion by never auditing the event in which all the rule’s terms are met.
• The “filter” specifies which kernel filter will deliver the event to the auditd. The filter options are just like those discussed in the system overview above: task, exit, user and exclude. The most common combination is “always,exit,” which always logs the system call upon exit.
• The -S flag stands for system call and lets you specify which system call you want to audit. You can simply use “all” to audit all the system calls or choose as many system calls as you like from the 449 options available. To view the available system calls you can execute the command “sudo ausyscall –dump”.
• The -F flag will be followed by a “field=value” pair. This flag is used to further filter the system calls, for example, all the system calls made by a certain user.
• And the last (optional) flag is -k, which follows the same logic as the filesystem rules to add a unique marker to the rule.

Here’s an example of a real-life rule:

-a always,exit -F arch=b64 -S execve

In this article we will not go through every option of auditing, because they are pretty much endless. If you want some good references for predefined audit.rules, you can find them here, or here or here.

Now for some disclaimers: please remember, what you should audit is a very hard question to answer. Each organization varies greatly in its business operations, sensitive locations, priorities, and what’s important for auditing. Also note that the auditing system is a resource extensive process. We will not go through how to make it the most efficient, but we do recommend doing a good share of research on how to write your rules efficiently.

Closing the Gap with an In-Depth Example

Now that we’ve covered auditing processes and the syntax to configure rules, let’s close the gap by reviewing an event type we can audit in Windows and showcase how to audit it in Linux. For this we’ll use one of the most relevant rules for cybersecurity auditing: the Command Execution/Process creation event (in Windows, event ID 4688).

The rule that is needed for this event to be logged is the same from our previous example:

-a always,exit -F arch=b64 -S execve

This rule will create several different events upon a single command execution. Each event will have the following fields:

• type= – This field specifies the type of the event.
• msg= – This field contains multiple values. The first is the system used to audit this event. The second is the timestamp of the event, and the third is the log ID. The second and third values are placed inside parentheses and separated by colon. Logs that are generated for a single event will have a single log ID.

Event Types

Below are the event types that will be generated upon a single command execution:

type=SYSCALL

This event indicates that a syscall was successfully (or not successfully) executed. As simple as it sounds, it carries a lot of important data such as:

• arch – The architecture of the process executed.
• syscall – The type of system call. To view all the types and their meaning, execute the command “ausyscall –dump”.
• success – This field will indicate if the call was successful or not.
• exit – This field holds the value of the exit code returned by the system call. You can interpret those codes by running the command “ausearch –interpret –exit –{exit number}”.
•  a0-a3 – Encoded in hexadecimal, these are the first 4 arguments of the system call.
• items – This field contains the number of path records in the event.
• ppid – The numerical ID of the parent process.
• pid – The numerical ID of the running process.
• auid – The audit user ID. This very important field holds the id of the user that initially logged in, as opposed to the user who executed the command. Meaning that if a user “test” logged in, executed “sudo su” to elevate privileges and ran the command in question, the auid will hold the ID of the user “test” and not root.
• uid – The ID of the user who executed this syscall.
• gid – The group ID of the user who executed this syscall.
• euid – The Effective User ID that determines the user permissions the process is currently using.
• suid – The Saved User ID that stores the effective user ID when a setuid program is executed.
• fsuid – The File System User ID.
• egid – The Effective Group ID that determines the current group permissions being used by the process.
• sgid – Similar to suid but for group permissions.
• fsgid – The File System Group ID.
• tty – This records the terminal from which the syscall was made.
• ses – This holds the session ID from which the syscall was made.
• comm – This holds the name of the command that was used to run this syscall.
• exe – This holds the name of the executable that was used to run this syscall.
• key – This is where you will see the key that was given to the rule that created those logs.

type=CWD

This event type stands for Current Working Directory and contains the path from which the command was initiated. Except for the global fields, it has only 1 field:

• cwd – This is where the current working directory will be shown.

type=PATH

This record will be generated for every path that is passed to the system call as an argument. It contains several useful fields that describe the file executed. We won’t go through all the fields in this article. A full explanation can be found here, but some of the most interesting fields are:

• name – contains the path of the file that was passed as an argument to the syscall.
• mode – Contains the records the file or directory permissions, encoded in numerical notation.
• ouid – contains the ID of the object’s owner.
• ogid – Contains the group ID of the object’s owner.

 type=PROCTITLE

This event contains the full command that was executed, encoded in hexadecimal encoding to preserve any special characters that may appear in the command upon transit. It contains only one field:

• proctitle – The encoded command that was executed.

type=EXECVE

This event also contains the full command that was executed but in a more readable and reliable manner. The event will contain the command broken down into arguments with the following fields:

• argc – This field represents the argument count that will be presented in the event.
• a* – The fields named “a” followed by a number are the arguments, starting with a0, which will in most cases contain the executable that started the action. The arguments that were passed to it will follow as a1, a2, etc.

After understanding the events and the data we are expecting upon each command execution, let’s put this all together to create detection rules that will retrieve us the same data and more than we would expect from a Windows event (4688).

Data Manipulation

The first step in writing effective audit rules is to gather all the data from every event generated by a single command execution. Once you have the complete set of related events, you can filter for specific values that match the criteria of your overall query. For example, let’s take a simple command execution event. We will be looking for persistence via GRUB update, the commands for updating grub configuration are: “grub-mkconfig”, “grub2-mkconfig”, “update-grub”.

The first event we will want to look at is EXECVE, which will show us the command that was originally executed. We can use parsing or regex to reconstruct the full command to a single line named commandline. This will make the command field resemble the Windows systems’ command line. Also, it’s helpful to parse the event type and the log ID as they will be relevant later. 

Now, we create a query that matches the strings indicating GRUB update from the commandline in EXECVE events. Unfortunately, this event does not hold enough data to really act on the threat we’re trying to detect, so we can combine the matched event with a SYSCALL event of the same Log ID, then retrieve the fields: “success”, “ppid”, “pid”, “auid”, “uid” , “gid”, “ses” and “key”. Another event to combine into the detection is the CWD. 

Now let’s look at what data we will be getting by matching and combining all the mentioned events and fields:

• The host on which the command was executed.
• The command that was executed.
• Whether the command was successful.
• The process ID of the process that was initiated by the command.
• The parent process ID of the process that was initiated by the command.
• The User ID that the process executor used to log in initially to the host.
• The user ID that was used to execute the command.
• The group ID that the user who executed the command was part of.
• The session number of the user.
• The key of the rule that matched the action.
• The working directory from which the user had executed the command.

With all this information in place, the analyst or IR personnel investigating this activity will have a lot more contextualized data to drive more effective response and remediation workflows. Actually, if we look at the data we got from these events combined, we can see that we actually have more data to act on than we would get from Windows event ID 4688.

Another example can be with a rule that detects the same thing by auditing edits of a file related to GRUB like  “/etc/default/grub.d/*”. If an adversary executed the command “nano  “/etc/default/grub.d/{filename}” we can detect this easily by looking for commandlines that contain the word “nano” and the path “/etc/default/grub.d/”. 

But what about a scenario where the attacker is already in “/etc/” and the command he executes is “nano /default/grub.d/{filename}“? In that case we can reconstruct the full path of the file that the attacker has accessed by combining CWD with the EXECVE events that were related to the same command execution, again by using the Log ID field and detect this event the same way as before.

This should show us that data manipulation is invariably the first step of detecting the full action we would like to match in the query.

Event Correlation

Another important principle for writing rules is correlating data from several events that may seem unrelated. Previously in this article we mentioned that there is a difference in the way the command shell processes the command in Linux opposed to Windows. In Linux if we chain commands or redirect data from commands for example with the command “cat /etc/passwd > /dev/tcp/127.0.0.1/22”, we will get two separate logs. One for “cat /etc/passwd” and the other event for the execution of the SSHD.

This enables us to write higher fidelity rules that are based on a chain of seemingly unrelated events, so we can know the extent of the full sequence of actions that the adversary executed. These sequential logs can be correlated based on their nearly identical timestamps, as the command shell typically executes such chained commands in rapid succession with minimal time gaps.

Closing Thoughts

In this article we did not go through the entire process of threat detection for Linux because if we would, this article would be a lot longer, but hopefully the information curated in this article can help you to get on the right track to audit linux events like it initially helped me.

About the Author

Michael Vilshin is a cybersecurity researcher at CardinalOps focused on novel detection strategies for often overlooked infrastructure components (like Linux!). Prior to CardinalOps, Michael worked in a wide range of roles spanning SIEM Engineer, Tier 3 SOC Analyst, and Incident Responder for Experis Cyber.