Skip to content

Managing Sophos Linux Sensor health

Overview

All Sophos Linux Sensor (SLS) components report their health status via a /healthz (aka healthz) HTTP endpoint on a monitoring port. An application is healthy if its healthz endpoint returns either an HTTP status code with a value of 200 and an HTTP body consisting of a JSON object with a status value stating "ok". It's unhealthy if it returns any non-200 HTTP status code. For example a healthy response would be:

HTTP/1.0 200 OK
Content-Type: application/json
Date: Mon, 04 Jun 2018 13:45:02 GMT
Content-Length: 15

{"status":"ok"}

An application in an unhealthy state requires reconfiguration, more resources, or a restart. You should restart SLS components when unhealthy. Service discovery systems such as consul, can be used to automate health checking.

Example

In this example we demonstrate how to test the monitoring by connecting directly to a /healthz HTTP endpoint using the curl command.

This example uses curl to test the health of a sensor on the local machine via the server's default monitoring port:

[root@localhost ~]# curl -vvvs http://localhost:9010/healthz
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9010 (#0)
> GET /healthz HTTP/1.1
> Host: localhost:9010
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Wed, 14 Nov 2018 13:25:00 GMT
< Content-Length: 18
<
{"status": "ok"}
* Connection #0 to host localhost left intact

SLS health checks

SLS's health check reports that it's receiving telemetry, processing analytics, and that control messages are being received. In the event that SLS is unable to instrument telemetry or that control messages stop due to a network partition or other occurrence, the health check will report that it's unhealthy.

By default, the SLS monitoring port is 9010, but you can change this by setting monitor_port in /etc/sophos/runtimedetections-rules.yaml

Additionally, you can query internal service metrics via the /metrics endpoint in Prometheus/OpenMetrics format.

SLS Metrics

Introduction

SLS publishes operational metrics in Prometheus or OpenMetrics format via an HTTP endpoint.

By default, the endpoint is at http://localhost:9010/metrics but the port can be reconfigured by setting monitor_port in /etc/sophos/runtimedetections-rules.yaml or through the RUNTIMEDETECTIONS_MONITOR_PORT environment variable.

Push Gateway Support

Optionally it's possible to forward the metrics to a push gateway. This can be done by setting:

prometheus_push_gateway: <pushgateway address:port>
prometheus_push_interval: <interval you want to push metrics at>
prometheus_push_group_name: <any push group name>
prometheus_push_group_value: <any push group value>

Available Metrics

Commonly useful metrics are indicated in bold.

Analytics

Metric Description
analytics_alert_drop_count Shows a total count of alerts dropped due to buffer overflow or failure to dispatch
analytics_alert_drop_count_1m Shows a total count of alerts dropped due to buffer overflow or failure to dispatch estimated during the last minute
analytics_alert_drop_count_5m Shows a total count of alerts dropped due to buffer overflow or failure to dispatch estimated during the last five minutes
analytics_alert_output_1_authentication_failure_count Shows a total count of alerts that failed to be dispatched because the remote server returned an authentication failure on this output
analytics_alert_output_1_authentication_failure_count_1m Shows a total count of alerts that failed to be dispatched because the remote server returned an authentication failure on this output estimated during the last minute
analytics_alert_output_1_authentication_failure_count_5m Shows a total count of alerts that failed to be dispatched because the remote server returned an authentication failure on this output estimated during the last five minute
analytics_alert_output_1_connection_failure_count Shows a total count of alerts that failed to be dispatched because the connection failed on this output
analytics_alert_output_1_connection_failure_count_1m Shows a total count of alerts that failed to be dispatched because the connection failed on this output estimated during the last minute
analytics_alert_output_1_connection_failure_count_5m Shows a total count of alerts that failed to be dispatched because the connection failed on this output estimated during the last five minutes
analytics_alert_output_1_drop_count Shows a total count of alerts dropped due to buffer overflow or failure to dispatch on this output
analytics_alert_output_1_drop_count_1m Shows a total count of alerts dropped due to buffer overflow or failure to dispatch on this output estimated during the last minute
analytics_alert_output_1_drop_count_5m Shows a total count of alerts dropped due to buffer overflow or failure to dispatch on this output estimated during the last five minutes
analytics_alert_output_1_internal_server_error_count Shows a total count of alerts that failed to be dispatched because the remote server returned an internal error on this output
analytics_alert_output_1_internal_server_error_count_1m Shows a total count of alerts that failed to be dispatched because the remote server returned an internal error on this output estimated during the last minute
analytics_alert_output_1_internal_server_error_count_5m Shows a total count of alerts that failed to be dispatched because the remote server returned an internal error on this output estimated during the last five minute
analytics_alert_output_1_local_rate_limit_count Shows a total count of alerts that failed to be dispatched because they triggered the local rate limit on this output
analytics_alert_output_1_local_rate_limit_count_1m Shows a total count of alerts that failed to be dispatched because they triggered the local rate limit on this output estimated during the last minute
analytics_alert_output_1_local_rate_limit_count_5m Shows a total count of alerts that failed to be dispatched because they triggered the local rate limit on this output estimated during the last five minutes
analytics_alert_output_1_remote_rate_limit_count Shows a total count of alerts that failed to be dispatched because the remote server rate limited the request on this output
analytics_alert_output_1_remote_rate_limit_count_1m Shows a total count of alerts that failed to be dispatched because the remote server rate limited the request on this output estimated during the last minute
analytics_alert_output_1_remote_rate_limit_count_5m Shows a total count of alerts that failed to be dispatched because the remote server rate limited the request on this output estimated during the last five minutes
analytics_alert_output_1_sent_count Shows a total count of alerts successfully dispatched on this output
analytics_alert_output_1_sent_count_1m Shows a total count of alerts successfully dispatched on this output estimated during the last minute
analytics_alert_output_1_sent_count_5m Shows a total count of alerts successfully dispatched on this output estimated during the last five minutes
analytics_alert_output_1_timeout_count Shows a total count of alerts that failed to be dispatched because the remote server returned a timeout on this output
analytics_alert_output_1_timeout_count_1m Shows a total count of alerts that failed to be dispatched because the remote server returned a timeout on this output estimated during the last minute
analytics_alert_output_1_timeout_count_5m Shows a total count of alerts that failed to be dispatched because the remote server returned a timeout on this output estimated during the last five minutes
analytics_alert_output_1_unknown_failure_count Shows a total count of alerts that failed to be dispatched for unknown reasons on this output
analytics_alert_output_1_unknown_failure_count_1m Shows a total count of alerts that failed to be dispatched for unknown reasons on this output estimated during the last minute
analytics_alert_output_1_unknown_failure_count_5m Shows a total count of alerts that failed to be dispatched for unknown reasons on this output estimated during the last five minutes
analytics_alert_sent_count Shows a total count of alerts successfully dispatched
analytics_alert_sent_count_1m Shows a total count of alerts successfully dispatched estimated during the last minute
analytics_alert_sent_count_5m Shows a total count of alerts successfully dispatched estimated during the last five minutes
analytics_alert_total_count Shows a total count of alerts generated by analytics
analytics_alert_total_count_1m Shows a total count of alerts generated by analytics estimated during the last minute
analytics_alert_total_count_5m Shows a total count of alerts generated by analytics estimated during the last five minutes
analytics_container_active_count Shows a count of containers currently in the process tree
analytics_event_average_processing_delay_ econds
analytics_event_average_processing_delay_seconds_1m Shows the average processing delay of events processed by analytics in seconds estimated during the last minute
analytics_event_average_processing_delay_seconds_5m Shows the average processing delay of events processed by analytics in seconds estimated during the last five minutes
analytics_event_average_processing_seconds Shows the average processing time of events processed by analytics in seconds
analytics_event_average_processing_seconds_1m Shows the average processing time of events processed by analytics in seconds estimated during the last minute
analytics_event_average_processing_seconds_5m Shows the average processing time of events processed by analytics in seconds estimated during the last five minutes
analytics_event_total_count Shows a total count of events processed by analytics
analytics_event_total_count_1m Shows a total
analytics_event_total_count_5m Shows a total count of events processed by analytics estimated during the last five minutes
analytics_event_type_container_event_end_count Shows a total count of CONTAINER_EVENT_END events processed by analytics
analytics_event_type_container_event_start_count Shows a total count of CONTAINER_EVENT_START events processed by analytics
analytics_event_type_container_event_type_created_count Shows a total count of CONTAINER_EVENT_TYPE_CREATED events processed by analytics
analytics_event_type_container_event_type_destroyed_count Shows a total count of CONTAINER_EVENT_TYPE_DESTROYED events processed by analytics
analytics_event_type_container_event_type_exited_count Shows a total count of CONTAINER_EVENT_TYPE_EXITED events processed by analytics
analytics_event_type_container_event_type_running_count Shows a total count of CONTAINER_EVENT_TYPE_RUNNING events processed by analytics
analytics_event_type_container_event_type_updated_count Shows a total count of CONTAINER_EVENT_TYPE_UPDATED events processed by analytics
analytics_event_type_event_type_unknown_count Shows a total count of EVENT_TYPE_UNKNOWN events processed by analytics
analytics_event_type_file_event_end_count Shows a total count of FILE_EVENT_END events processed by analytics
analytics_event_type_file_event_start_count Shows a total count of FILE_EVENT_START events processed by analytics
analytics_event_type_file_event_type_attribute_change_count Shows a total count of FILE_EVENT_TYPE_ATTRIBUTE_CHANGE events processed by analytics
analytics_event_type_file_event_type_close_for_write_count Shows a total count of FILE_EVENT_TYPE_CLOSE_FOR_WRITE events processed by analytics
analytics_event_type_file_event_type_create_count Shows a total count of FILE_EVENT_TYPE_CREATE events processed by analytics
analytics_event_type_file_event_type_delete_count Shows a total count of FILE_EVENT_TYPE_DELETE events processed by analytics
analytics_event_type_file_event_type_get_attributes_count Shows a total count of FILE_EVENT_TYPE_GET_ATTRIBUTES events processed by analytics
analytics_event_type_file_event_type_hard_link_count Shows a total count of FILE_EVENT_TYPE_HARD_LINK events processed by analytics
analytics_event_type_file_event_type_modify_count Shows a total count of FILE_EVENT_TYPE_MODIFY events processed by analytics
analytics_event_type_file_event_type_open_for_write_count Shows a total count of FILE_EVENT_TYPE_OPEN_FOR_WRITE events processed by analytics
analytics_event_type_file_event_type_open_read_only_count Shows a total count of FILE_EVENT_TYPE_OPEN_READ_ONLY events processed by analytics
analytics_event_type_file_event_type_rename_count Shows a total count of FILE_EVENT_TYPE_RENAME events processed by analytics
analytics_event_type_file_event_type_sym_link_count Shows a total count of FILE_EVENT_TYPE_SYM_LINK events processed by analytics
analytics_event_type_file_event_type_write_batch_count Shows a total count of FILE_EVENT_TYPE_WRITE_BATCH events processed by analytics
analytics_event_type_file_event_type_write_batch_raw_count Shows a total count of FILE_EVENT_TYPE_WRITE_BATCH_RAW events processed by analytics
analytics_event_type_interactive_shell_event_count Shows a total count of INTERACTIVE_SHELL_EVENT events processed by analytics
analytics_event_type_kprobe_event_app_armor_type_count Shows a total count of KPROBE_EVENT_APP_ARMOR_TYPE events processed by analytics
analytics_event_type_kprobe_event_chmod_type_count Shows a total count of KPROBE_EVENT_CHMOD_TYPE events processed by analytics
analytics_event_type_kprobe_event_commit_creds_type_count Shows a total count of KPROBE_EVENT_COMMIT_CREDS_TYPE events processed by analytics
analytics_event_type_kprobe_event_connect_aws_count Shows a total count of KPROBE_EVENT_CONNECT_AWS events processed by analytics
analytics_event_type_kprobe_event_copy_fs_struct_count Shows a total count of KPROBE_EVENT_COPY_FS_STRUCT events processed by analytics
analytics_event_type_kprobe_event_cr4_pinning_type_count Shows a total count of KPROBE_EVENT_CR4_PINNING_TYPE events processed by analytics
analytics_event_type_kprobe_event_custom_count Shows a total count of KPROBE_EVENT_CUSTOM events processed by analytics
analytics_event_type_kprobe_event_end_count Shows a total count of KPROBE_EVENT_END events processed by analytics
analytics_event_type_kprobe_event_fchmodat_type_count Shows a total count of KPROBE_EVENT_FCHMODAT_TYPE events processed by analytics
analytics_event_type_kprobe_event_load_kernel_module_type_count Shows a total count of KPROBE_EVENT_LOAD_KERNEL_MODULE_TYPE events processed by analytics
analytics_event_type_kprobe_event_mm_access_fs_count Shows a total count of KPROBE_EVENT_MM_ACCESS_FS events processed by analytics
analytics_event_type_kprobe_event_prepare_creds_type_count Shows a total count of KPROBE_EVENT_PREPARE_CREDS_TYPE events processed by analytics
analytics_event_type_kprobe_event_prepare_kernel_creds_type_count Shows a total count of KPROBE_EVENT_PREPARE_KERNEL_CREDS_TYPE events processed by analytics
analytics_event_type_kprobe_event_rlimit_count Shows a total count of KPROBE_EVENT_RLIMIT events processed by analytics
analytics_event_type_kprobe_event_security_bprm_committed_creds_count Shows a total count of KPROBE_EVENT_SECURITY_BPRM_COMMITTED_CREDS events processed by analytics
analytics_event_type_kprobe_event_security_bprm_committing_creds_count Shows a total count of KPROBE_EVENT_SECURITY_BPRM_COMMITTING_CREDS events processed by analytics
analytics_event_type_kprobe_event_selinux_state_type_count Shows a total count of KPROBE_EVENT_SELINUX_STATE_TYPE events processed by analytics
analytics_event_type_kprobe_event_selinux_type_count Shows a total count of KPROBE_EVENT_SELINUX_TYPE events processed by analytics
analytics_event_type_kprobe_event_smep_smap_type_count Shows a total count of KPROBE_EVENT_SMEP_SMAP_TYPE events processed by analytics
analytics_event_type_kprobe_event_stack_create_count Shows a total count of KPROBE_EVENT_STACK_CREATE events processed by analytics
analytics_event_type_kprobe_event_stack_expand_count Shows a total count of KPROBE_EVENT_STACK_EXPAND events processed by analytics
analytics_event_type_kprobe_event_start_count Shows a total count of KPROBE_EVENT_START events processed by analytics
analytics_event_type_kprobe_event_switch_task_namespaces_count Shows a total count of KPROBE_EVENT_SWITCH_TASK_NAMESPACES events processed by analytics
analytics_event_type_kprobe_event_syscall_execve_sa_count Shows a total count of KPROBE_EVENT_SYSCALL_EXECVE_SA events processed by analytics
analytics_event_type_kprobe_event_syscall_mprotect_sa_count Shows a total count of KPROBE_EVENT_SYSCALL_MPROTECT_SA events processed by analytics
analytics_event_type_kprobe_event_tty_write_count Shows a total count of KPROBE_EVENT_TTY_WRITE events processed by analytics
analytics_event_type_lost_record_event_type_count Shows a total count of LOST_RECORD_EVENT_TYPE events processed by analytics
analytics_event_type_lost_record_start_type_count Shows a total count of LOST_RECORD_START_TYPE events processed by analytics
analytics_event_type_network_event_connect_count Shows a total count of NETWORK_EVENT_CONNECT events processed by analytics
analytics_event_type_network_event_end_count Shows a total count of NETWORK_EVENT_END events processed by analytics
analytics_event_type_network_event_start_count Shows a total count of NETWORK_EVENT_START events processed by analytics
analytics_event_type_network_service_created_event_count Shows a total count of NETWORK_SERVICE_CREATED_EVENT events processed by analytics
analytics_event_type_networkbound_io_event_count Shows a total count of NETWORKBOUND_IO_EVENT events processed by analytics
analytics_event_type_new_file_exec_event_count Shows a total count of NEW_FILE_EXEC_EVENT events processed by analytics
analytics_event_type_performance_event_type_create_count Shows a total count of PERFORMANCE_EVENT_TYPE_CREATE events processed by analytics
analytics_event_type_process_event_end_count Shows a total count of PROCESS_EVENT_END events processed by analytics
analytics_event_type_process_event_start_count Shows a total count of PROCESS_EVENT_START events processed by analytics
analytics_event_type_process_event_type_exec_count Shows a total count of PROCESS_EVENT_TYPE_EXEC events processed by analytics
analytics_event_type_process_event_type_exit_count Shows a total count of PROCESS_EVENT_TYPE_EXIT events processed by analytics
analytics_event_type_process_event_type_fork_count Shows a total count of PROCESS_EVENT_TYPE_FORK events processed by analytics
analytics_event_type_process_event_type_update_container_count Shows a total count of PROCESS_EVENT_TYPE_UPDATE_CONTAINER events processed by analytics
analytics_event_type_process_event_type_update_count Shows a total count of PROCESS_EVENT_TYPE_UPDATE events processed by analytics
analytics_event_type_process_event_type_update_credentials_count Shows a total count of PROCESS_EVENT_TYPE_UPDATE_CREDENTIALS events processed by analytics
analytics_event_type_shell_command_event_count Shows a total count of SHELL_COMMAND_EVENT events processed by analytics
analytics_event_type_strategy_mprotect_violation_count Shows a total count of STRATEGY_MPROTECT_VIOLATION events processed by analytics
analytics_event_type_syscall_accept_enter_event_count Shows a total count of SYSCALL_ACCEPT_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_accept_exit_event_count Shows a total count of SYSCALL_ACCEPT_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_bind_enter_event_count Shows a total count of SYSCALL_BIND_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_bind_exit_event_count Shows a total count of SYSCALL_BIND_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_bpf_enter_count Shows a total count of SYSCALL_BPF_ENTER events processed by analytics
analytics_event_type_syscall_brk_count Shows a total count of SYSCALL_BRK events processed by analytics
analytics_event_type_syscall_brk_enter_count Shows a total count of SYSCALL_BRK_ENTER events processed by analytics
analytics_event_type_syscall_brk_exit_count Shows a total count of SYSCALL_BRK_EXIT events processed by analytics
analytics_event_type_syscall_connect_enter_event_count Shows a total count of SYSCALL_CONNECT_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_connect_exit_event_count Shows a total count of SYSCALL_CONNECT_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_dup2_enter_event_count Shows a total count of SYSCALL_DUP2_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_dup3_enter_event_count Shows a total count of SYSCALL_DUP3_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_dup_enter_event_count Shows a total count of SYSCALL_DUP_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_dup_exit_event_count Shows a total count of SYSCALL_DUP_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_event_accept_count Shows a total count of SYSCALL_EVENT_ACCEPT events processed by analytics
analytics_event_type_syscall_event_bind_count Shows a total count of SYSCALL_EVENT_BIND events processed by analytics
analytics_event_type_syscall_event_close_count Shows a total count of SYSCALL_EVENT_CLOSE events processed by analytics
analytics_event_type_syscall_event_connect_count Shows a total count of SYSCALL_EVENT_CONNECT events processed by analytics
analytics_event_type_syscall_event_dup_count Shows a total count of SYSCALL_EVENT_DUP events processed by analytics
analytics_event_type_syscall_event_end_count Shows a total count of SYSCALL_EVENT_END events processed by analytics
analytics_event_type_syscall_event_listen_count Shows a total count of SYSCALL_EVENT_LISTEN events processed by analytics
analytics_event_type_syscall_event_recvfrom_count Shows a total count of SYSCALL_EVENT_RECVFROM events processed by analytics
analytics_event_type_syscall_event_sendto_count Shows a total count of SYSCALL_EVENT_SENDTO events processed by analytics
analytics_event_type_syscall_event_start_count Shows a total count of SYSCALL_EVENT_START events processed by analytics
analytics_event_type_syscall_fcntl_enter_event_count Shows a total count of SYSCALL_FCNTL_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_fcntl_event_count Shows a total count of SYSCALL_FCNTL_EVENT events processed by analytics
analytics_event_type_syscall_fcntl_exit_event_count Shows a total count of SYSCALL_FCNTL_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_ioctl_enter_count Shows a total count of SYSCALL_IOCTL_ENTER events processed by analytics
analytics_event_type_syscall_listen_enter_event_count Shows a total count of SYSCALL_LISTEN_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_listen_exit_event_count Shows a total count of SYSCALL_LISTEN_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_mmap_count Shows a total count of SYSCALL_MMAP events processed by analytics
analytics_event_type_syscall_mmap_enter_count Shows a total count of SYSCALL_MMAP_ENTER events processed by analytics
analytics_event_type_syscall_mmap_exit_count Shows a total count of SYSCALL_MMAP_EXIT events processed by analytics
analytics_event_type_syscall_mprotect_x_enter_count Shows a total count of SYSCALL_MPROTECT_X_ENTER events processed by analytics
analytics_event_type_syscall_privilege_event_count Shows a total count of SYSCALL_PRIVILEGE_EVENT events processed by analytics
analytics_event_type_syscall_process_vm_readv_event_count Shows a total count of SYSCALL_PROCESS_VM_READV_EVENT events processed by analytics
analytics_event_type_syscall_process_vm_writev_event_count Shows a total count of SYSCALL_PROCESS_VM_WRITEV_EVENT events processed by analytics
analytics_event_type_syscall_ptrace_enter_event_count Shows a total count of SYSCALL_PTRACE_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_ptrace_event_count Shows a total count of SYSCALL_PTRACE_EVENT events processed by analytics
analytics_event_type_syscall_ptrace_exit_event_count Shows a total count of SYSCALL_PTRACE_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_recvfrom_enter_event_count Shows a total count of SYSCALL_RECVFROM_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_recvfrom_exit_event_count Shows a total count of SYSCALL_RECVFROM_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_sendto_enter_event_count Shows a total count of SYSCALL_SENDTO_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_sendto_exit_event_count Shows a total count of SYSCALL_SENDTO_EXIT_EVENT events processed by analytics
analytics_event_type_syscall_setfsgid_enter_count Shows a total count of SYSCALL_SETFSGID_ENTER events processed by analytics
analytics_event_type_syscall_setfsgid_exit_count Shows a total count of SYSCALL_SETFSGID_EXIT events processed by analytics
analytics_event_type_syscall_setfsuid_enter_count Shows a total count of SYSCALL_SETFSUID_ENTER events processed by analytics
analytics_event_type_syscall_setfsuid_exit_count Shows a total count of SYSCALL_SETFSUID_EXIT events processed by analytics
analytics_event_type_syscall_setgid_enter_count Shows a total count of SYSCALL_SETGID_ENTER events processed by analytics
analytics_event_type_syscall_setgid_exit_count Shows a total count of SYSCALL_SETGID_EXIT events processed by analytics
analytics_event_type_syscall_setregid_enter_count Shows a total count of SYSCALL_SETREGID_ENTER events processed by analytics
analytics_event_type_syscall_setregid_exit_count Shows a total count of SYSCALL_SETREGID_EXIT events processed by analytics
analytics_event_type_syscall_setresgid_enter_count Shows a total count of SYSCALL_SETRESGID_ENTER events processed by analytics
analytics_event_type_syscall_setresgid_exit_count Shows a total count of SYSCALL_SETRESGID_EXIT events processed by analytics
analytics_event_type_syscall_setresuid_enter_count Shows a total count of SYSCALL_SETRESUID_ENTER events processed by analytics
analytics_event_type_syscall_setresuid_exit_count Shows a total count of SYSCALL_SETRESUID_EXIT events processed by analytics
analytics_event_type_syscall_setreuid_enter_count Shows a total count of SYSCALL_SETREUID_ENTER events processed by analytics
analytics_event_type_syscall_setreuid_exit_count Shows a total count of SYSCALL_SETREUID_EXIT events processed by analytics
analytics_event_type_syscall_setuid_enter_count Shows a total count of SYSCALL_SETUID_ENTER events processed by analytics
analytics_event_type_syscall_setuid_exit_count Shows a total count of SYSCALL_SETUID_EXIT events processed by analytics
analytics_event_type_syscall_sigaltstack_enter_event_count Shows a total count of SYSCALL_SIGALTSTACK_ENTER_EVENT events processed by analytics
analytics_event_type_syscall_userfaultfd_enter_count Shows a total count of SYSCALL_USERFAULTFD_ENTER events processed by analytics
analytics_event_type_tag_event_count Shows a total count of TAG_EVENT events processed by analytics
analytics_event_type_ticker_count Shows a total count of TICKER events processed by analytics
analytics_event_type_trigger_event_type_co nt
analytics_event_type_uprobe_event_custom_count Shows a total count of UPROBE_EVENT_CUSTOM events processed by analytics
analytics_event_type_uprobe_event_end_count Shows a total count of UPROBE_EVENT_END events processed by analytics
analytics_event_type_uprobe_event_start_count Shows a total count of UPROBE_EVENT_START events processed by analytics
analytics_event_type_uprobe_readline_event_count Shows a total count of UPROBE_READLINE_EVENT events processed by analytics
analytics_lost_event_type_lostrecordtypealertbuffer_count Shows a total count of LostRecordTypeAlertBuffer events lost by analytics
analytics_lost_event_type_lostrecordtypealertdispatch_count Shows a total count of LostRecordTypeAlertDispatch events lost by analytics
analytics_lost_event_type_lostrecordtypecontainer_count Shows a total count of LostRecordTypeContainer events lost by analytics
analytics_lost_event_type_lostrecordtypefilemodify_count Shows a total count of LostRecordTypeFileModify events lost by analytics
analytics_lost_event_type_lostrecordtypeflush_count Shows a total count of LostRecordTypeFlush events lost by analytics
analytics_lost_event_type_lostrecordtypemounts_count Shows a total count of LostRecordTypeMounts events lost by analytics
analytics_lost_event_type_lostrecordtypeprocess_count Shows a total count of LostRecordTypeProcess events lost by analytics
analytics_lost_event_type_lostrecordtypesubscription_count Shows a total count of LostRecordTypeSubscription events lost by analytics
analytics_lost_event_type_lostrecordtypeunknown_count Shows a total count of LostRecordTypeUnknown events lost by analytics
analytics_metaevent_total_count Shows a total count of metaevents generated by analytics
analytics_metaevent_total_count_1m Shows a total count of metaevents generated by analytics estimated during the last minute
analytics_metaevent_total_count_5m Shows a total count of metaevents generated by analytics estimated during the last five minutes
analytics_process_active_count Shows a count of processes currently in the process tree
analytics_reorder_late_count Shows a count of events that came in too late to be reordered correctly
analytics_reorder_late_count_1m Shows a count of events that came in too late to be reordered correctly estimated during the last minute
analytics_reorder_late_count_5m Shows a count of events that came in too late to be reordered correctly estimated during the last five minutes
analytics_task_active_count Shows a count of tasks currently in the process tree
analytics_task_evict_count Shows a count of
analytics_task_evict_count_1m Shows a count of tasks evicted from the process tree estimated during the last minute
analytics_task_evict_count_5m Shows a count of tasks evicted from the process tree estimated during the last five minutes
analytics_task_resurrect_count Shows a count of task resurrection operations in the process tree
analytics_task_resurrect_count_1m Shows a count of task resurrection operations in the process tree estimated during the last minute
analytics_task_resurrect_count_5m Shows a count of task resurrection operations in the process tree estimated during the last five minutes

Go

Metric Description
go_gc_duration_seconds A summary of the GC invocation durations.
go_goroutines Number of goroutines that currently exist.
go_info Information about the Go environment.
go_memstats_alloc_bytes Number of bytes allocated and still in use.
go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
go_memstats_frees_total Total number of frees.
go_memstats_gc_cpu_fraction The fraction of this program's available CPU time used by the GC since the program started.
go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
go_memstats_heap_objects Number of allocated objects.
go_memstats_heap_released_bytes Number of heap bytes released to OS.
go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
go_memstats_lookups_total Total number of pointer lookups.
go_memstats_mallocs_total Total number of mallocs.
go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
go_memstats_other_sys_bytes Number of bytes used for other system allocations.
go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
go_memstats_sys_bytes Number of bytes obtained from system.
go_threads Number of OS threads created.

Process

Metric Description
process_cpu_seconds_total Total user and system CPU time spent in seconds.
process_max_fds Maximum number of open file descriptors.
process_open_fds Number of open file descriptors.
process_resident_memory_bytes Resident memory size in bytes.
process_start_time_seconds Start time of the process since unix epoch in seconds.
process_virtual_memory_bytes Virtual memory size in bytes.
process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.

Prometheus

Metric Description
promhttp_metric_handler_requests_in_flight Current number of scrapes being served.
promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.

Sensor

Metric Description
sensor_active_subscriptions Amount of active sensor subscriptions
sensor_buffer_memory_usage Shows the current buffer memory usage by SLS in bytes
sensor_kernel_samples_lost Shows a count of kernel samples lost
sensor_perf_record_lost_count Shows the number of PERF_RECORD_LOST samples processed
sensor_perf_record_lost_total_count Shows the sum total of PERF_RECORD_LOST lost fields
sensor_perf_record_sample_count Shows the number of PERF_RECORD_SAMPLE samples processed
sensor_perf_record_sample_counter_count Shows the number of PERF_RECORD_SAMPLE samples having counter information
sensor_perf_record_sample_trace_count Shows the number of PERF_RECORD_SAMPLE samples having trace information
sensor_total_event_count Shows a count of events currently processed by the perf sensor