Using Metaevent Filters

Investigations collects all of the metaevents for any tables you've turned on and writes them to a blob storage location of your choice. This can generate a lot of data, particularly for certain workloads, such as a microservice that outputs data to Kafka or a static webserver serving files from a single directory. In these instances, it may be helpful to remove any connection events that are writing data out on port 9092 or file events from the directory on the webserver.

This is where filters come in. Filters let you specify a set of rules that determine whether or not to generate a metaevent. Doing this reduces duplicate events, allowing faster query times, less bandwidth usage, and better sensor performance.

Syntax

Metaevent filters are specified on a per-table basis and use the same rule syntax as custom detections. The first step in writing a filter is determining whether it should default match or default ignore. You must specify a default action before adding custom rules.

• default match means that the sensor will match all incoming metaevents by default, and your rules would then ignore metaevents that shouldn't be saved. An example of this would be a Kafka worker node matching all connection metaevents and ignoring those making a connection to port 9092.
• default ignore means that the sensor will ignore all metaevents by default. Your rules would then match specific cases that need to be output. An example would be ignoring all file events but matching on file creation and modification for file integrity monitoring.

The next step to creating a filter is determining which fields to filter on. Each metaevent type has a set of valid fields that you can filter, ranging from container id to file path to connection port. You can find a full reference at the end of this document. These event types can then be compared to relevant values and either matched or ignored when they meet the condition. For example, to filter all traffic to port 9092, the rule would look like this: ignore outboundPort == 9092

Once you have your rules, you can construct them into a filter. A filter is a set of rules organized into a list. If a filter has multiple rules, all rules are evaluated in the order shown. The filter for the prior Kafka example would look like this:

filter: 
  - ignore outboundPort == 9092
  - default match

That filter when applied in a config file would look like this:

investigations:
  metaevent_dispatcher_enabled: true
  enable_incremental_flush: false
  reporting_interval: 5s
  sinks:
    - name: <a_folder_name>
      backend: aws
      automated: true
      type: parquet
  flight_recorder:
    enabled: true
    tables:
      ... all other tables ...
      - name: "connection_events"
        rows: 1000
        enabled: true
        filter:
          - ignore outboundPort == 9092
          - default match

Another example that uses multiple rules is file integrity monitoring for a static web server. In this case, only file creation and modifications are saved as we would expect to see file reads:

filter: 
  - match operation == "create"
  - match operation == "modify"
  - default ignore

When applied in a config file, that filter would look like the following example:

investigations: 
  reporting_interval: 5s 
  sinks: 
    - name: <a_folder_name>
      backend: file
      automated: true
      type: json
  flight_recorder:
    enabled: true 
    tables:
      ... all other tables ... 
      - name: "file_events"
        rows: 1000 
        enabled: true 
        filter: 
          - match eventType == "FILE_OPERATION_CREATE"
          - match eventType == "FILE_OPERATION_MODIFY"
          - default ignore

Success and Errors:

Once a filter has been added to a sensor config and the sensor has been restarted, there should be a log entry like the following example:

2021-04-30T13:00:00.337Z INFO Program filtering active

That line means that the filters were syntactically correct and are applied. If something is wrong with the filter, such as a type or an incorrect field name, there will be a log entry like the following example:

2021-04-30T12:55:43.382Z WARN Investigations event filters could not be applied: "evntType" is not a valid field for this strategy. Valid fields are [ancestorProgramName commandLine containerId
containerIsPrivileged containerLifetime containerName currentWorkingDirectory fileContainerId fileHash filePath gid group hasIncidentID hasNetworkServiceAncestor imageId imageName lineage loginGid loginG
roup loginUid loginUser namespace operation parentProgramArguments parentProgramName podName programArguments programDepth programHash programLifetime programName sourceFilePath uid userName] (in policy "
file_operation_metaevent_filter", rule "match evntType == \"FILE_OPERATION_CREATE\""). unexpected $end in rule: match eventType == "FILE_OPERATION_CREATE" Invalid expression in rule: match eventType == "
FILE_OPERATION_CREATE" 

In this case, an "e" was missed in eventType.

Reference

All of the fields that be used by filters:

eventType Possible Values:

• CONTAINER_EVENT_START
• CONTAINER_EVENT_TYPE_CREATED
• CONTAINER_EVENT_TYPE_RUNNING
• CONTAINER_EVENT_TYPE_EXITED
• CONTAINER_EVENT_TYPE_DESTROYED
• CONTAINER_EVENT_TYPE_UPDATED
• CONTAINER_EVENT_END
• PROCESS_EVENT_START
• PROCESS_EVENT_TYPE_FORK
• PROCESS_EVENT_TYPE_EXEC
• PROCESS_EVENT_TYPE_EXIT
• PROCESS_EVENT_TYPE_UPDATE
• PROCESS_EVENT_TYPE_UPDATE_CONTAINER
• PROCESS_EVENT_TYPE_UPDATE_CREDENTIALS
• PROCESS_EVENT_END
• FILE_EVENT_START
• FILE_EVENT_TYPE_OPEN_READ_ONLY
• FILE_EVENT_TYPE_CREATE
• FILE_EVENT_TYPE_DELETE
• FILE_EVENT_TYPE_HARD_LINK
• FILE_EVENT_TYPE_SYM_LINK
• FILE_EVENT_TYPE_MODIFY
• FILE_EVENT_TYPE_RENAME
• FILE_EVENT_TYPE_OPEN_FOR_WRITE
• FILE_EVENT_TYPE_CLOSE_FOR_WRITE
• FILE_EVENT_TYPE_ATTRIBUTE_CHANGE
• FILE_EVENT_TYPE_GET_ATTRIBUTES
• FILE_EVENT_END
• NETWORK_EVENT_START
• SYSCALL_EVENT_ACCEPT
• SYSCALL_ACCEPT_ENTER_EVENT
• SYSCALL_ACCEPT_EXIT_EVENT
• SYSCALL_EVENT_BIND
• SYSCALL_BIND_ENTER_EVENT
• SYSCALL_BIND_EXIT_EVENT
• SYSCALL_EVENT_CONNECT
• SYSCALL_CONNECT_ENTER_EVENT
• SYSCALL_CONNECT_EXIT_EVENT
• SYSCALL_EVENT_LISTEN
• SYSCALL_LISTEN_ENTER_EVENT
• SYSCALL_LISTEN_EXIT_EVENT
• SYSCALL_EVENT_RECVFROM
• SYSCALL_RECVFROM_ENTER_EVENT
• SYSCALL_RECVFROM_EXIT_EVENT
• SYSCALL_EVENT_SENDTO
• SYSCALL_SENDTO_ENTER_EVENT
• SYSCALL_SENDTO_EXIT_EVENT
• NETWORK_EVENT_CONNECT
• NETWORK_EVENT_END
• NETWORKBOUND_IO_EVENT
• INTERACTIVE_SHELL_EVENT
• NETWORK_SERVICE_CREATED_EVENT
• FILE_EVENT_TYPE_WRITE_BATCH_RAW
• FILE_EVENT_TYPE_WRITE_BATCH
• SHELL_COMMAND_EVENT
• NEW_FILE_EXEC_EVENT
• SET_CAPABILITIES_EVENT
• COMMAND_LINE_ACTIVITY_EVENT