Setting up Investigations with Hadoop
Overview
Hadoop is an open-source framework capable of processing applications executed in a distributed computing environment. Using Sophos Linux Sensor (SLS), you can set up investigations by following the procedure explained below:
Requirements
- Hadoop Cluster
- SLS running in your environment
Configuring the Sensor:
Edit the configuration file /etc/sophos/runtimedetections-rules.yaml
.
Add a sink for HDFS, allow the sensor to create directories, and turn on the flight recorder:
Here's an example:
cloud_meta: auto
blob_storage_create_buckets_enabled: true
investigations:
reporting_interval: 30s
sinks:
- name: "[namenode hostname/ip]:9000/runtimedetections-investigations/"
backend: hdfs
automated: true
type: parquet
partition_format: "hostname_partition={{.Hostname}}/date_partition={{.Time.Format \"2006-01-02\"}}"
credentials:
blob_storage_hdfs_user: hadoop
flight_recorder:
enabled: true
tables:
- name: "shell_commands"
rows: 1000
enabled: true
- name: "tty_data"
rows: 1000
enabled: true
- name: "connections"
rows: 2000
enabled: true
- name: "sensor_metadata"
rows: 500
enabled: true
- name: "alerts"
rows: 100
enabled: true
- name: "sensors"
rows: 10
enabled: true
- name: "process_events"
rows: 4000
enabled: true
- name: "container_events"
rows: 300
enabled: true
Restart the sensor and verify it worked with the following command:
hdfs dfs -ls /runtimedetections-investigations/
This should list all of the tables that were enabled in the config:
drwxr--r-- - root supergroup 0 2020-10-27 18:33 /runtimedetections-investigations/alerts drwxr--r-- - root supergroup 0 2020-10-27 18:33 /runtimedetections-investigations/connections drwxr--r-- - root supergroup 0 2020-10-27 18:33 /runtimedetections-investigations/container_events drwxr--r-- - root supergroup 0 2020-10-27 18:33 /runtimedetections-investigations/process_events drwxr--r-- - root supergroup 0 2020-10-27 18:33 /runtimedetections-investigations/sensor_metadata drwxr--r-- - root supergroup 0 2020-10-27 18:33 /runtimedetections-investigations/sensors
Editing SLS config:
After you confirm that SLS is properly configured, edit the reporting interval to a more reasonable time.
Here's an example:
cloud_meta: auto
blob_storage_create_buckets_enabled: true
investigations:
reporting_interval: 5m
#...