Skip to content
Last update: 2022-06-08

Getting Investigations data via GCP/BigQuery

GCP

One additional cloud storage bucket is needed to store Sophos Linux Sensor (SLS) Investigations data for querying via BigQuery.

Sensor

You can configure sensors to emit data to support Investigations by editing /etc/sophos/runtimedetections-rules.yaml to include the following:

cloud_meta: auto

blob_storage_create_buckets_enabled: true
# blob_storage_create_buckets_enabled adds the ability for the sensor to create buckets
# if the bucket doesn't exist. By default this field is set to false.

investigations:

    enable_incremental_flush: true
    # enable_incremental_flush adds the ability to flush row groups rather than 
    # writing files during each flush event. Enabling this will result in larger files
    # being created. By default this field is set to false.
    #
    # Minimum chunksizes for an incremental flush:
    # GCP: 256KB
    # AWS: 5MB
    # AZURE: 1MB

  reporting_interval: 10s
  # reporting_interval sets a time interval for forced flushes.
  # timeout set the amount of time allowed for investigations data to be written
  # to a sink. By default the timeout is 1/3 of the reporting_interval duration.
  # timeout: 90s
  # sinks are a list of destinations where investigations data should be sent.

  sinks:

    - name: <investigations-metadata-bucket-name>

      backend: gcp
      # backend is the type of sink.
      # Supported backends:
      # "gcp": google cloud provider buckets
      # "aws": amazon S3 buckets
      # "azure": azure storage buckets

      automated: true

      type: parquet
            partition_format: "hostname_partition={{.Hostname}}/date_partition={{.Time.Format \"2006-01-02\"}}"
      # type is the file format. By default this field is set to parquet. 
      # Supported types:
      # "parquet": writes data to the apache parquet format
      # "json": writes data to the json format

    flight_recorder:
        enabled: true
        tables:
        - name: "shell_commands"
          enabled: true
        - name: "tty_data"
          enabled: true
        - name: "sensors"
          enabled: true
        - name: "sensor_metadata"
          enabled: true
        - name: "connections"
          enabled: true
        - name: "process_events"
          enabled: true
        - name: "container_events"
          enabled: true

IAM

The permissions needed for SLS to write to GCS are:

resourcemanager.projects.get
storage.buckets.create
storage.hmacKeys.create
storage.objects.create

Credentials can be passed to sensor in two ways:

  • A service account file pointed to by the GOOGLE_APPLICATION_CREDENTIALS environment variable
  • Compute Engine instance service account
Back to top