Skip to content

Using Metaevent Filters

Investigations collects all of the metaevents for any tables you've turned on and writes them to a blob storage location of your choice. This can generate a lot of data, particularly for certain workloads, such as a microservice that outputs data to Kafka or a static webserver serving files from a single directory. In these instances, it may be helpful to remove any connection events that are writing data out on port 9092 or file events from the directory on the webserver.

This is where filters come in. Filters let you specify a set of rules that determine whether or not to generate a metaevent. Doing this reduces duplicate events, allowing faster query times, less bandwidth usage, and better sensor performance.

Syntax

Metaevent filters are specified on a per-table basis and use the same rule syntax as custom detections. The first step in writing a filter is determining whether it should default match or default ignore. You must specify a default action before adding custom rules.

  • default match means that the sensor will match all incoming metaevents by default, and your rules would then ignore metaevents that shouldn't be saved. An example of this would be a Kafka worker node matching all connection metaevents and ignoring those making a connection to port 9092.
  • default ignore means that the sensor will ignore all metaevents by default. Your rules would then match specific cases that need to be output. An example would be ignoring all file events but matching on file creation and modification for file integrity monitoring.

The next step to creating a filter is determining which fields to filter on. Each metaevent type has a set of valid fields that you can filter, ranging from container id to file path to connection port. You can find a full reference at the end of this document. These event types can then be compared to relevant values and either matched or ignored when they meet the condition. For example, to filter all traffic to port 9092, the rule would look like this: ignore outboundPort == 9092

Once you have your rules, you can construct them into a filter. A filter is a set of rules organized into a list. If a filter has multiple rules, all rules are evaluated in the order shown. The filter for the prior Kafka example would look like this:

filter: 
  - ignore outboundPort == 9092
  - default match

That filter when applied in a config file would look like this:

investigations:
  metaevent_dispatcher_enabled: true
  enable_incremental_flush: false
  reporting_interval: 5s
  sinks:
    - name: <a_folder_name>
      backend: aws
      automated: true
      type: parquet
  flight_recorder:
    enabled: true
    tables:
      ... all other tables ...
      - name: "connection_events"
        rows: 1000
        enabled: true
        filter:
          - ignore outboundPort == 9092
          - default match

Another example that uses multiple rules is file integrity monitoring for a static web server. In this case, only file creation and modifications are saved as we would expect to see file reads:

filter: 
  - match operation == "create"
  - match operation == "modify"
  - default ignore

When applied in a config file, that filter would look like the following example:

investigations: 
  reporting_interval: 5s 
  sinks: 
    - name: <a_folder_name>
      backend: file
      automated: true
      type: json
  flight_recorder:
    enabled: true 
    tables:
      ... all other tables ... 
      - name: "file_events"
        rows: 1000 
        enabled: true 
        filter: 
          - match eventType == "FILE_OPERATION_CREATE"
          - match eventType == "FILE_OPERATION_MODIFY"
          - default ignore

Success and Errors:

Once a filter has been added to a sensor config and the sensor has been restarted, there should be a log entry like the following example:

2021-04-30T13:00:00.337Z INFO Program filtering active

That line means that the filters were syntactically correct and are applied. If something is wrong with the filter, such as a type or an incorrect field name, there will be a log entry like the following example:

2021-04-30T12:55:43.382Z WARN Investigations event filters could not be applied: "evntType" is not a valid field for this strategy. Valid fields are [ancestorProgramName commandLine containerId
containerIsPrivileged containerLifetime containerName currentWorkingDirectory fileContainerId fileHash filePath gid group hasIncidentID hasNetworkServiceAncestor imageId imageName lineage loginGid loginG
roup loginUid loginUser namespace operation parentProgramArguments parentProgramName podName programArguments programDepth programHash programLifetime programName sourceFilePath uid userName] (in policy "
file_operation_metaevent_filter", rule "match evntType == \"FILE_OPERATION_CREATE\""). unexpected $end in rule: match eventType == "FILE_OPERATION_CREATE" Invalid expression in rule: match eventType == "
FILE_OPERATION_CREATE" 

In this case, an "e" was missed in eventType.

Reference

All of the fields that be used by filters:

Field Event type Description
containerId all The Container Id that the process is running in
containerName all The container name
containerLifetime all Length of time that the container has been running when the event occurred.
containerIsPrivileged all The container is running with cap_sys_admin privileges
imageId all Image id of the container in which the event occurred
imageName all Image name of the container in which the event occurred
namespace all Name of the kubernetes namespace in which the program's container resides
podName all Name of the kubernetes pod in which the program's container resides
programName all Name of the program which generated the event
parentProgramName all Name of the parent of the program which generated the event
programArguments all Arguments passed to the program which generated the event
parentProgramArguments all Arguments passed to the parent of the program which generated the event
programLifetime all Lifetime of the program as duration.
programDepth all Exec depth of process
ancestorProgramName all Any matching ancestor of the program which generated the event
lineage all Program lineage (currently based on program path without arguments) for the event
programHash all SHA256 hash of the program in which the event occurred
hasNetworkServiceAncestor all The program is descended from a network service
currentWorkingDirectory all Current working directory of the program which generated the event
hasIncidentID all The program or one of its ancestors has produced an alert
commandLine all Command line input as observed by a shell program
userName all Username of the program which generated the event
group all Group name of the program which generated the event
uid all UID of the program which generated the event
gid all GID of the program which generated the event
loginUser all Username of the login which generated the event
loginGroup all Group name of the login which generated the event
loginUid all UID of the login which generated the event
loginGid all GID of the login which generated the event
eventType all The type of action that generated the event. Reference below.
filePath file_events File path for a file event. For file events that reference both a source and a target, refers to the target path.
operation file_events Operation type for a file event. Possible values are: open, create, link, modify, append, rename, delete, setattr, or getattr.
sourceFilePath file_events Source path for a file event
fileContainerId file_events Id of the container in which the file resides
fileHash file_events SHA256 hash of the file path on which the event occurred
remoteHost connections Host IP address for a connection event
outboundPort connections Outbound port for a connection event
interfaceIP connections IP address of network service event
inboundPort connections Inbound port for a connection event
creatorProgramName process_events Name of the program which last modified the new file
creatorParentProgramName process_events Name of the parent of the program which last modified the new file
creatorAncestorProgramName process_events Any matching ancestor of the program which modified the file
creatorLineage process_events Lineage of the program which modified the file
creatorProgramArguments process_events Arguments passed to the program which modified the new file
creatorParentProgramArguments process_events Arguments passed to the parent of the program which modified the new file
creatorUserName process_events Username of the program which modified the new file
creatorGroup process_events Group name of the program which modified the new file
creatorUid process_events UID of the program which modified the new file
creatorGid process_events GID of the program which modified the new file
creatorLoginUser process_events Username of the login which modified the new file
creatorLoginGroup process_events Group name of the login which modified the new file
creatorLoginUid process_events UID of the login which modified the new file
creatorLoginGid process_events GID of the login which modified the new file
targetUid process_events The program is attempting to change its UID to the targetUid
targetEuid process_events The program is attempting to change its EUID to the targetEuid
targetSuid process_events The program is attempting to change its SUID to the targetSuid
targetFsuid process_events The program is attempting to change its FSUID to the targetFsuid
targetGid process_events The program is attempting to change its GID to the targetGid
targetEgid process_events The program is attempting to change its EGID to the targetEgid
targetSgid process_events The program is attempting to change its SGID to the targetSgid
targetFsgid process_events The program is attempting to change its FSGID to the targetFsgid

eventType Possible Values:

  • CONTAINER_EVENT_START
  • CONTAINER_EVENT_TYPE_CREATED
  • CONTAINER_EVENT_TYPE_RUNNING
  • CONTAINER_EVENT_TYPE_EXITED
  • CONTAINER_EVENT_TYPE_DESTROYED
  • CONTAINER_EVENT_TYPE_UPDATED
  • CONTAINER_EVENT_END
  • PROCESS_EVENT_START
  • PROCESS_EVENT_TYPE_FORK
  • PROCESS_EVENT_TYPE_EXEC
  • PROCESS_EVENT_TYPE_EXIT
  • PROCESS_EVENT_TYPE_UPDATE
  • PROCESS_EVENT_TYPE_UPDATE_CONTAINER
  • PROCESS_EVENT_TYPE_UPDATE_CREDENTIALS
  • PROCESS_EVENT_END
  • FILE_EVENT_START
  • FILE_EVENT_TYPE_OPEN_READ_ONLY
  • FILE_EVENT_TYPE_CREATE
  • FILE_EVENT_TYPE_DELETE
  • FILE_EVENT_TYPE_HARD_LINK
  • FILE_EVENT_TYPE_SYM_LINK
  • FILE_EVENT_TYPE_MODIFY
  • FILE_EVENT_TYPE_RENAME
  • FILE_EVENT_TYPE_OPEN_FOR_WRITE
  • FILE_EVENT_TYPE_CLOSE_FOR_WRITE
  • FILE_EVENT_TYPE_ATTRIBUTE_CHANGE
  • FILE_EVENT_TYPE_GET_ATTRIBUTES
  • FILE_EVENT_END
  • NETWORK_EVENT_START
  • SYSCALL_EVENT_ACCEPT
  • SYSCALL_ACCEPT_ENTER_EVENT
  • SYSCALL_ACCEPT_EXIT_EVENT
  • SYSCALL_EVENT_BIND
  • SYSCALL_BIND_ENTER_EVENT
  • SYSCALL_BIND_EXIT_EVENT
  • SYSCALL_EVENT_CONNECT
  • SYSCALL_CONNECT_ENTER_EVENT
  • SYSCALL_CONNECT_EXIT_EVENT
  • SYSCALL_EVENT_LISTEN
  • SYSCALL_LISTEN_ENTER_EVENT
  • SYSCALL_LISTEN_EXIT_EVENT
  • SYSCALL_EVENT_RECVFROM
  • SYSCALL_RECVFROM_ENTER_EVENT
  • SYSCALL_RECVFROM_EXIT_EVENT
  • SYSCALL_EVENT_SENDTO
  • SYSCALL_SENDTO_ENTER_EVENT
  • SYSCALL_SENDTO_EXIT_EVENT
  • NETWORK_EVENT_CONNECT
  • NETWORK_EVENT_END
  • NETWORKBOUND_IO_EVENT
  • INTERACTIVE_SHELL_EVENT
  • NETWORK_SERVICE_CREATED_EVENT
  • FILE_EVENT_TYPE_WRITE_BATCH_RAW
  • FILE_EVENT_TYPE_WRITE_BATCH
  • SHELL_COMMAND_EVENT
  • NEW_FILE_EXEC_EVENT
  • SET_CAPABILITIES_EVENT
  • COMMAND_LINE_ACTIVITY_EVENT
Back to top