Managing Sophos Linux Sensor health
Overview
All Sophos Linux Sensor (SLS) components report their health status via a /healthz
(aka healthz) HTTP endpoint on a monitoring port. An application is healthy if its healthz endpoint returns either an HTTP status code with a value of 200 and an HTTP body consisting of a JSON object with a status
value stating "ok"
. It's unhealthy if it returns any non-200 HTTP status code. For example a healthy response would be:
HTTP/1.0 200 OK
Content-Type: application/json
Date: Mon, 04 Jun 2018 13:45:02 GMT
Content-Length: 15
{"status":"ok"}
An application in an unhealthy state requires reconfiguration, more resources, or a restart. You should restart SLS components when unhealthy. Service discovery systems such as consul
, can be used to automate health checking.
Example
In this example we demonstrate how to test the monitoring by connecting directly to a /healthz
HTTP endpoint using the curl
command.
This example uses curl
to test the health of a sensor on the local machine via the server's default monitoring port:
[root@localhost ~]# curl -vvvs http://localhost:9010/healthz
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 9010 (#0)
> GET /healthz HTTP/1.1
> Host: localhost:9010
> User-Agent: curl/7.54.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Wed, 14 Nov 2018 13:25:00 GMT
< Content-Length: 18
<
{"status": "ok"}
* Connection #0 to host localhost left intact
SLS health checks
SLS's health check reports that it's receiving telemetry, processing analytics, and that control messages are being received. In the event that SLS is unable to instrument telemetry or that control messages stop due to a network partition or other occurrence, the health check will report that it's unhealthy.
By default, the SLS monitoring port is 9010, but you can change this by setting monitor_port
in /etc/sophos/runtimedetections-rules.yaml
Additionally, you can query internal service metrics via the /metrics
endpoint in Prometheus/OpenMetrics format.
SLS Metrics
Introduction
SLS publishes operational metrics in Prometheus or OpenMetrics format via an HTTP endpoint.
By default, the endpoint is at http://localhost:9010/metrics
but the port can be reconfigured by setting monitor_port
in /etc/sophos/runtimedetections-rules.yaml
or through the RUNTIMEDETECTIONS_MONITOR_PORT
environment variable.
Push Gateway Support
Optionally it's possible to forward the metrics to a push gateway. This can be done by setting:
prometheus_push_gateway: <pushgateway address:port>
prometheus_push_interval: <interval you want to push metrics at>
prometheus_push_group_name: <any push group name>
prometheus_push_group_value: <any push group value>
Available Metrics
Commonly useful metrics are indicated in bold.
Analytics
Metric | Description |
---|---|
analytics_alert_drop_count | Shows a total count of alerts dropped due to buffer overflow or failure to dispatch |
analytics_alert_drop_count_1m | Shows a total count of alerts dropped due to buffer overflow or failure to dispatch estimated during the last minute |
analytics_alert_drop_count_5m | Shows a total count of alerts dropped due to buffer overflow or failure to dispatch estimated during the last five minutes |
analytics_alert_output_1_authentication_failure_count | Shows a total count of alerts that failed to be dispatched because the remote server returned an authentication failure on this output |
analytics_alert_output_1_authentication_failure_count_1m | Shows a total count of alerts that failed to be dispatched because the remote server returned an authentication failure on this output estimated during the last minute |
analytics_alert_output_1_authentication_failure_count_5m | Shows a total count of alerts that failed to be dispatched because the remote server returned an authentication failure on this output estimated during the last five minute |
analytics_alert_output_1_connection_failure_count | Shows a total count of alerts that failed to be dispatched because the connection failed on this output |
analytics_alert_output_1_connection_failure_count_1m | Shows a total count of alerts that failed to be dispatched because the connection failed on this output estimated during the last minute |
analytics_alert_output_1_connection_failure_count_5m | Shows a total count of alerts that failed to be dispatched because the connection failed on this output estimated during the last five minutes |
analytics_alert_output_1_drop_count | Shows a total count of alerts dropped due to buffer overflow or failure to dispatch on this output |
analytics_alert_output_1_drop_count_1m | Shows a total count of alerts dropped due to buffer overflow or failure to dispatch on this output estimated during the last minute |
analytics_alert_output_1_drop_count_5m | Shows a total count of alerts dropped due to buffer overflow or failure to dispatch on this output estimated during the last five minutes |
analytics_alert_output_1_internal_server_error_count | Shows a total count of alerts that failed to be dispatched because the remote server returned an internal error on this output |
analytics_alert_output_1_internal_server_error_count_1m | Shows a total count of alerts that failed to be dispatched because the remote server returned an internal error on this output estimated during the last minute |
analytics_alert_output_1_internal_server_error_count_5m | Shows a total count of alerts that failed to be dispatched because the remote server returned an internal error on this output estimated during the last five minute |
analytics_alert_output_1_local_rate_limit_count | Shows a total count of alerts that failed to be dispatched because they triggered the local rate limit on this output |
analytics_alert_output_1_local_rate_limit_count_1m | Shows a total count of alerts that failed to be dispatched because they triggered the local rate limit on this output estimated during the last minute |
analytics_alert_output_1_local_rate_limit_count_5m | Shows a total count of alerts that failed to be dispatched because they triggered the local rate limit on this output estimated during the last five minutes |
analytics_alert_output_1_remote_rate_limit_count | Shows a total count of alerts that failed to be dispatched because the remote server rate limited the request on this output |
analytics_alert_output_1_remote_rate_limit_count_1m | Shows a total count of alerts that failed to be dispatched because the remote server rate limited the request on this output estimated during the last minute |
analytics_alert_output_1_remote_rate_limit_count_5m | Shows a total count of alerts that failed to be dispatched because the remote server rate limited the request on this output estimated during the last five minutes |
analytics_alert_output_1_sent_count | Shows a total count of alerts successfully dispatched on this output |
analytics_alert_output_1_sent_count_1m | Shows a total count of alerts successfully dispatched on this output estimated during the last minute |
analytics_alert_output_1_sent_count_5m | Shows a total count of alerts successfully dispatched on this output estimated during the last five minutes |
analytics_alert_output_1_timeout_count | Shows a total count of alerts that failed to be dispatched because the remote server returned a timeout on this output |
analytics_alert_output_1_timeout_count_1m | Shows a total count of alerts that failed to be dispatched because the remote server returned a timeout on this output estimated during the last minute |
analytics_alert_output_1_timeout_count_5m | Shows a total count of alerts that failed to be dispatched because the remote server returned a timeout on this output estimated during the last five minutes |
analytics_alert_output_1_unknown_failure_count | Shows a total count of alerts that failed to be dispatched for unknown reasons on this output |
analytics_alert_output_1_unknown_failure_count_1m | Shows a total count of alerts that failed to be dispatched for unknown reasons on this output estimated during the last minute |
analytics_alert_output_1_unknown_failure_count_5m | Shows a total count of alerts that failed to be dispatched for unknown reasons on this output estimated during the last five minutes |
analytics_alert_sent_count | Shows a total count of alerts successfully dispatched |
analytics_alert_sent_count_1m | Shows a total count of alerts successfully dispatched estimated during the last minute |
analytics_alert_sent_count_5m | Shows a total count of alerts successfully dispatched estimated during the last five minutes |
analytics_alert_total_count | Shows a total count of alerts generated by analytics |
analytics_alert_total_count_1m | Shows a total count of alerts generated by analytics estimated during the last minute |
analytics_alert_total_count_5m | Shows a total count of alerts generated by analytics estimated during the last five minutes |
analytics_container_active_count | Shows a count of containers currently in the process tree |
analytics_event_average_processing_delay_ | econds |
analytics_event_average_processing_delay_seconds_1m | Shows the average processing delay of events processed by analytics in seconds estimated during the last minute |
analytics_event_average_processing_delay_seconds_5m | Shows the average processing delay of events processed by analytics in seconds estimated during the last five minutes |
analytics_event_average_processing_seconds | Shows the average processing time of events processed by analytics in seconds |
analytics_event_average_processing_seconds_1m | Shows the average processing time of events processed by analytics in seconds estimated during the last minute |
analytics_event_average_processing_seconds_5m | Shows the average processing time of events processed by analytics in seconds estimated during the last five minutes |
analytics_event_total_count | Shows a total count of events processed by analytics |
analytics_event_total_count_1m | Shows a total |
analytics_event_total_count_5m | Shows a total count of events processed by analytics estimated during the last five minutes |
analytics_event_type_container_event_end_count | Shows a total count of CONTAINER_EVENT_END events processed by analytics |
analytics_event_type_container_event_start_count | Shows a total count of CONTAINER_EVENT_START events processed by analytics |
analytics_event_type_container_event_type_created_count | Shows a total count of CONTAINER_EVENT_TYPE_CREATED events processed by analytics |
analytics_event_type_container_event_type_destroyed_count | Shows a total count of CONTAINER_EVENT_TYPE_DESTROYED events processed by analytics |
analytics_event_type_container_event_type_exited_count | Shows a total count of CONTAINER_EVENT_TYPE_EXITED events processed by analytics |
analytics_event_type_container_event_type_running_count | Shows a total count of CONTAINER_EVENT_TYPE_RUNNING events processed by analytics |
analytics_event_type_container_event_type_updated_count | Shows a total count of CONTAINER_EVENT_TYPE_UPDATED events processed by analytics |
analytics_event_type_event_type_unknown_count | Shows a total count of EVENT_TYPE_UNKNOWN events processed by analytics |
analytics_event_type_file_event_end_count | Shows a total count of FILE_EVENT_END events processed by analytics |
analytics_event_type_file_event_start_count | Shows a total count of FILE_EVENT_START events processed by analytics |
analytics_event_type_file_event_type_attribute_change_count | Shows a total count of FILE_EVENT_TYPE_ATTRIBUTE_CHANGE events processed by analytics |
analytics_event_type_file_event_type_close_for_write_count | Shows a total count of FILE_EVENT_TYPE_CLOSE_FOR_WRITE events processed by analytics |
analytics_event_type_file_event_type_create_count | Shows a total count of FILE_EVENT_TYPE_CREATE events processed by analytics |
analytics_event_type_file_event_type_delete_count | Shows a total count of FILE_EVENT_TYPE_DELETE events processed by analytics |
analytics_event_type_file_event_type_get_attributes_count | Shows a total count of FILE_EVENT_TYPE_GET_ATTRIBUTES events processed by analytics |
analytics_event_type_file_event_type_hard_link_count | Shows a total count of FILE_EVENT_TYPE_HARD_LINK events processed by analytics |
analytics_event_type_file_event_type_modify_count | Shows a total count of FILE_EVENT_TYPE_MODIFY events processed by analytics |
analytics_event_type_file_event_type_open_for_write_count | Shows a total count of FILE_EVENT_TYPE_OPEN_FOR_WRITE events processed by analytics |
analytics_event_type_file_event_type_open_read_only_count | Shows a total count of FILE_EVENT_TYPE_OPEN_READ_ONLY events processed by analytics |
analytics_event_type_file_event_type_rename_count | Shows a total count of FILE_EVENT_TYPE_RENAME events processed by analytics |
analytics_event_type_file_event_type_sym_link_count | Shows a total count of FILE_EVENT_TYPE_SYM_LINK events processed by analytics |
analytics_event_type_file_event_type_write_batch_count | Shows a total count of FILE_EVENT_TYPE_WRITE_BATCH events processed by analytics |
analytics_event_type_file_event_type_write_batch_raw_count | Shows a total count of FILE_EVENT_TYPE_WRITE_BATCH_RAW events processed by analytics |
analytics_event_type_interactive_shell_event_count | Shows a total count of INTERACTIVE_SHELL_EVENT events processed by analytics |
analytics_event_type_kprobe_event_app_armor_type_count | Shows a total count of KPROBE_EVENT_APP_ARMOR_TYPE events processed by analytics |
analytics_event_type_kprobe_event_chmod_type_count | Shows a total count of KPROBE_EVENT_CHMOD_TYPE events processed by analytics |
analytics_event_type_kprobe_event_commit_creds_type_count | Shows a total count of KPROBE_EVENT_COMMIT_CREDS_TYPE events processed by analytics |
analytics_event_type_kprobe_event_connect_aws_count | Shows a total count of KPROBE_EVENT_CONNECT_AWS events processed by analytics |
analytics_event_type_kprobe_event_copy_fs_struct_count | Shows a total count of KPROBE_EVENT_COPY_FS_STRUCT events processed by analytics |
analytics_event_type_kprobe_event_cr4_pinning_type_count | Shows a total count of KPROBE_EVENT_CR4_PINNING_TYPE events processed by analytics |
analytics_event_type_kprobe_event_custom_count | Shows a total count of KPROBE_EVENT_CUSTOM events processed by analytics |
analytics_event_type_kprobe_event_end_count | Shows a total count of KPROBE_EVENT_END events processed by analytics |
analytics_event_type_kprobe_event_fchmodat_type_count | Shows a total count of KPROBE_EVENT_FCHMODAT_TYPE events processed by analytics |
analytics_event_type_kprobe_event_load_kernel_module_type_count | Shows a total count of KPROBE_EVENT_LOAD_KERNEL_MODULE_TYPE events processed by analytics |
analytics_event_type_kprobe_event_mm_access_fs_count | Shows a total count of KPROBE_EVENT_MM_ACCESS_FS events processed by analytics |
analytics_event_type_kprobe_event_prepare_creds_type_count | Shows a total count of KPROBE_EVENT_PREPARE_CREDS_TYPE events processed by analytics |
analytics_event_type_kprobe_event_prepare_kernel_creds_type_count | Shows a total count of KPROBE_EVENT_PREPARE_KERNEL_CREDS_TYPE events processed by analytics |
analytics_event_type_kprobe_event_rlimit_count | Shows a total count of KPROBE_EVENT_RLIMIT events processed by analytics |
analytics_event_type_kprobe_event_security_bprm_committed_creds_count | Shows a total count of KPROBE_EVENT_SECURITY_BPRM_COMMITTED_CREDS events processed by analytics |
analytics_event_type_kprobe_event_security_bprm_committing_creds_count | Shows a total count of KPROBE_EVENT_SECURITY_BPRM_COMMITTING_CREDS events processed by analytics |
analytics_event_type_kprobe_event_selinux_state_type_count | Shows a total count of KPROBE_EVENT_SELINUX_STATE_TYPE events processed by analytics |
analytics_event_type_kprobe_event_selinux_type_count | Shows a total count of KPROBE_EVENT_SELINUX_TYPE events processed by analytics |
analytics_event_type_kprobe_event_smep_smap_type_count | Shows a total count of KPROBE_EVENT_SMEP_SMAP_TYPE events processed by analytics |
analytics_event_type_kprobe_event_stack_create_count | Shows a total count of KPROBE_EVENT_STACK_CREATE events processed by analytics |
analytics_event_type_kprobe_event_stack_expand_count | Shows a total count of KPROBE_EVENT_STACK_EXPAND events processed by analytics |
analytics_event_type_kprobe_event_start_count | Shows a total count of KPROBE_EVENT_START events processed by analytics |
analytics_event_type_kprobe_event_switch_task_namespaces_count | Shows a total count of KPROBE_EVENT_SWITCH_TASK_NAMESPACES events processed by analytics |
analytics_event_type_kprobe_event_syscall_execve_sa_count | Shows a total count of KPROBE_EVENT_SYSCALL_EXECVE_SA events processed by analytics |
analytics_event_type_kprobe_event_syscall_mprotect_sa_count | Shows a total count of KPROBE_EVENT_SYSCALL_MPROTECT_SA events processed by analytics |
analytics_event_type_kprobe_event_tty_write_count | Shows a total count of KPROBE_EVENT_TTY_WRITE events processed by analytics |
analytics_event_type_lost_record_event_type_count | Shows a total count of LOST_RECORD_EVENT_TYPE events processed by analytics |
analytics_event_type_lost_record_start_type_count | Shows a total count of LOST_RECORD_START_TYPE events processed by analytics |
analytics_event_type_network_event_connect_count | Shows a total count of NETWORK_EVENT_CONNECT events processed by analytics |
analytics_event_type_network_event_end_count | Shows a total count of NETWORK_EVENT_END events processed by analytics |
analytics_event_type_network_event_start_count | Shows a total count of NETWORK_EVENT_START events processed by analytics |
analytics_event_type_network_service_created_event_count | Shows a total count of NETWORK_SERVICE_CREATED_EVENT events processed by analytics |
analytics_event_type_networkbound_io_event_count | Shows a total count of NETWORKBOUND_IO_EVENT events processed by analytics |
analytics_event_type_new_file_exec_event_count | Shows a total count of NEW_FILE_EXEC_EVENT events processed by analytics |
analytics_event_type_performance_event_type_create_count | Shows a total count of PERFORMANCE_EVENT_TYPE_CREATE events processed by analytics |
analytics_event_type_process_event_end_count | Shows a total count of PROCESS_EVENT_END events processed by analytics |
analytics_event_type_process_event_start_count | Shows a total count of PROCESS_EVENT_START events processed by analytics |
analytics_event_type_process_event_type_exec_count | Shows a total count of PROCESS_EVENT_TYPE_EXEC events processed by analytics |
analytics_event_type_process_event_type_exit_count | Shows a total count of PROCESS_EVENT_TYPE_EXIT events processed by analytics |
analytics_event_type_process_event_type_fork_count | Shows a total count of PROCESS_EVENT_TYPE_FORK events processed by analytics |
analytics_event_type_process_event_type_update_container_count | Shows a total count of PROCESS_EVENT_TYPE_UPDATE_CONTAINER events processed by analytics |
analytics_event_type_process_event_type_update_count | Shows a total count of PROCESS_EVENT_TYPE_UPDATE events processed by analytics |
analytics_event_type_process_event_type_update_credentials_count | Shows a total count of PROCESS_EVENT_TYPE_UPDATE_CREDENTIALS events processed by analytics |
analytics_event_type_shell_command_event_count | Shows a total count of SHELL_COMMAND_EVENT events processed by analytics |
analytics_event_type_strategy_mprotect_violation_count | Shows a total count of STRATEGY_MPROTECT_VIOLATION events processed by analytics |
analytics_event_type_syscall_accept_enter_event_count | Shows a total count of SYSCALL_ACCEPT_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_accept_exit_event_count | Shows a total count of SYSCALL_ACCEPT_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_bind_enter_event_count | Shows a total count of SYSCALL_BIND_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_bind_exit_event_count | Shows a total count of SYSCALL_BIND_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_bpf_enter_count | Shows a total count of SYSCALL_BPF_ENTER events processed by analytics |
analytics_event_type_syscall_brk_count | Shows a total count of SYSCALL_BRK events processed by analytics |
analytics_event_type_syscall_brk_enter_count | Shows a total count of SYSCALL_BRK_ENTER events processed by analytics |
analytics_event_type_syscall_brk_exit_count | Shows a total count of SYSCALL_BRK_EXIT events processed by analytics |
analytics_event_type_syscall_connect_enter_event_count | Shows a total count of SYSCALL_CONNECT_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_connect_exit_event_count | Shows a total count of SYSCALL_CONNECT_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_dup2_enter_event_count | Shows a total count of SYSCALL_DUP2_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_dup3_enter_event_count | Shows a total count of SYSCALL_DUP3_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_dup_enter_event_count | Shows a total count of SYSCALL_DUP_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_dup_exit_event_count | Shows a total count of SYSCALL_DUP_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_event_accept_count | Shows a total count of SYSCALL_EVENT_ACCEPT events processed by analytics |
analytics_event_type_syscall_event_bind_count | Shows a total count of SYSCALL_EVENT_BIND events processed by analytics |
analytics_event_type_syscall_event_close_count | Shows a total count of SYSCALL_EVENT_CLOSE events processed by analytics |
analytics_event_type_syscall_event_connect_count | Shows a total count of SYSCALL_EVENT_CONNECT events processed by analytics |
analytics_event_type_syscall_event_dup_count | Shows a total count of SYSCALL_EVENT_DUP events processed by analytics |
analytics_event_type_syscall_event_end_count | Shows a total count of SYSCALL_EVENT_END events processed by analytics |
analytics_event_type_syscall_event_listen_count | Shows a total count of SYSCALL_EVENT_LISTEN events processed by analytics |
analytics_event_type_syscall_event_recvfrom_count | Shows a total count of SYSCALL_EVENT_RECVFROM events processed by analytics |
analytics_event_type_syscall_event_sendto_count | Shows a total count of SYSCALL_EVENT_SENDTO events processed by analytics |
analytics_event_type_syscall_event_start_count | Shows a total count of SYSCALL_EVENT_START events processed by analytics |
analytics_event_type_syscall_fcntl_enter_event_count | Shows a total count of SYSCALL_FCNTL_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_fcntl_event_count | Shows a total count of SYSCALL_FCNTL_EVENT events processed by analytics |
analytics_event_type_syscall_fcntl_exit_event_count | Shows a total count of SYSCALL_FCNTL_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_ioctl_enter_count | Shows a total count of SYSCALL_IOCTL_ENTER events processed by analytics |
analytics_event_type_syscall_listen_enter_event_count | Shows a total count of SYSCALL_LISTEN_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_listen_exit_event_count | Shows a total count of SYSCALL_LISTEN_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_mmap_count | Shows a total count of SYSCALL_MMAP events processed by analytics |
analytics_event_type_syscall_mmap_enter_count | Shows a total count of SYSCALL_MMAP_ENTER events processed by analytics |
analytics_event_type_syscall_mmap_exit_count | Shows a total count of SYSCALL_MMAP_EXIT events processed by analytics |
analytics_event_type_syscall_mprotect_x_enter_count | Shows a total count of SYSCALL_MPROTECT_X_ENTER events processed by analytics |
analytics_event_type_syscall_privilege_event_count | Shows a total count of SYSCALL_PRIVILEGE_EVENT events processed by analytics |
analytics_event_type_syscall_process_vm_readv_event_count | Shows a total count of SYSCALL_PROCESS_VM_READV_EVENT events processed by analytics |
analytics_event_type_syscall_process_vm_writev_event_count | Shows a total count of SYSCALL_PROCESS_VM_WRITEV_EVENT events processed by analytics |
analytics_event_type_syscall_ptrace_enter_event_count | Shows a total count of SYSCALL_PTRACE_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_ptrace_event_count | Shows a total count of SYSCALL_PTRACE_EVENT events processed by analytics |
analytics_event_type_syscall_ptrace_exit_event_count | Shows a total count of SYSCALL_PTRACE_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_recvfrom_enter_event_count | Shows a total count of SYSCALL_RECVFROM_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_recvfrom_exit_event_count | Shows a total count of SYSCALL_RECVFROM_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_sendto_enter_event_count | Shows a total count of SYSCALL_SENDTO_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_sendto_exit_event_count | Shows a total count of SYSCALL_SENDTO_EXIT_EVENT events processed by analytics |
analytics_event_type_syscall_setfsgid_enter_count | Shows a total count of SYSCALL_SETFSGID_ENTER events processed by analytics |
analytics_event_type_syscall_setfsgid_exit_count | Shows a total count of SYSCALL_SETFSGID_EXIT events processed by analytics |
analytics_event_type_syscall_setfsuid_enter_count | Shows a total count of SYSCALL_SETFSUID_ENTER events processed by analytics |
analytics_event_type_syscall_setfsuid_exit_count | Shows a total count of SYSCALL_SETFSUID_EXIT events processed by analytics |
analytics_event_type_syscall_setgid_enter_count | Shows a total count of SYSCALL_SETGID_ENTER events processed by analytics |
analytics_event_type_syscall_setgid_exit_count | Shows a total count of SYSCALL_SETGID_EXIT events processed by analytics |
analytics_event_type_syscall_setregid_enter_count | Shows a total count of SYSCALL_SETREGID_ENTER events processed by analytics |
analytics_event_type_syscall_setregid_exit_count | Shows a total count of SYSCALL_SETREGID_EXIT events processed by analytics |
analytics_event_type_syscall_setresgid_enter_count | Shows a total count of SYSCALL_SETRESGID_ENTER events processed by analytics |
analytics_event_type_syscall_setresgid_exit_count | Shows a total count of SYSCALL_SETRESGID_EXIT events processed by analytics |
analytics_event_type_syscall_setresuid_enter_count | Shows a total count of SYSCALL_SETRESUID_ENTER events processed by analytics |
analytics_event_type_syscall_setresuid_exit_count | Shows a total count of SYSCALL_SETRESUID_EXIT events processed by analytics |
analytics_event_type_syscall_setreuid_enter_count | Shows a total count of SYSCALL_SETREUID_ENTER events processed by analytics |
analytics_event_type_syscall_setreuid_exit_count | Shows a total count of SYSCALL_SETREUID_EXIT events processed by analytics |
analytics_event_type_syscall_setuid_enter_count | Shows a total count of SYSCALL_SETUID_ENTER events processed by analytics |
analytics_event_type_syscall_setuid_exit_count | Shows a total count of SYSCALL_SETUID_EXIT events processed by analytics |
analytics_event_type_syscall_sigaltstack_enter_event_count | Shows a total count of SYSCALL_SIGALTSTACK_ENTER_EVENT events processed by analytics |
analytics_event_type_syscall_userfaultfd_enter_count | Shows a total count of SYSCALL_USERFAULTFD_ENTER events processed by analytics |
analytics_event_type_tag_event_count | Shows a total count of TAG_EVENT events processed by analytics |
analytics_event_type_ticker_count | Shows a total count of TICKER events processed by analytics |
analytics_event_type_trigger_event_type_co | nt |
analytics_event_type_uprobe_event_custom_count | Shows a total count of UPROBE_EVENT_CUSTOM events processed by analytics |
analytics_event_type_uprobe_event_end_count | Shows a total count of UPROBE_EVENT_END events processed by analytics |
analytics_event_type_uprobe_event_start_count | Shows a total count of UPROBE_EVENT_START events processed by analytics |
analytics_event_type_uprobe_readline_event_count | Shows a total count of UPROBE_READLINE_EVENT events processed by analytics |
analytics_lost_event_type_lostrecordtypealertbuffer_count | Shows a total count of LostRecordTypeAlertBuffer events lost by analytics |
analytics_lost_event_type_lostrecordtypealertdispatch_count | Shows a total count of LostRecordTypeAlertDispatch events lost by analytics |
analytics_lost_event_type_lostrecordtypecontainer_count | Shows a total count of LostRecordTypeContainer events lost by analytics |
analytics_lost_event_type_lostrecordtypefilemodify_count | Shows a total count of LostRecordTypeFileModify events lost by analytics |
analytics_lost_event_type_lostrecordtypeflush_count | Shows a total count of LostRecordTypeFlush events lost by analytics |
analytics_lost_event_type_lostrecordtypemounts_count | Shows a total count of LostRecordTypeMounts events lost by analytics |
analytics_lost_event_type_lostrecordtypeprocess_count | Shows a total count of LostRecordTypeProcess events lost by analytics |
analytics_lost_event_type_lostrecordtypesubscription_count | Shows a total count of LostRecordTypeSubscription events lost by analytics |
analytics_lost_event_type_lostrecordtypeunknown_count | Shows a total count of LostRecordTypeUnknown events lost by analytics |
analytics_metaevent_total_count | Shows a total count of metaevents generated by analytics |
analytics_metaevent_total_count_1m | Shows a total count of metaevents generated by analytics estimated during the last minute |
analytics_metaevent_total_count_5m | Shows a total count of metaevents generated by analytics estimated during the last five minutes |
analytics_process_active_count | Shows a count of processes currently in the process tree |
analytics_reorder_late_count | Shows a count of events that came in too late to be reordered correctly |
analytics_reorder_late_count_1m | Shows a count of events that came in too late to be reordered correctly estimated during the last minute |
analytics_reorder_late_count_5m | Shows a count of events that came in too late to be reordered correctly estimated during the last five minutes |
analytics_task_active_count | Shows a count of tasks currently in the process tree |
analytics_task_evict_count | Shows a count of |
analytics_task_evict_count_1m | Shows a count of tasks evicted from the process tree estimated during the last minute |
analytics_task_evict_count_5m | Shows a count of tasks evicted from the process tree estimated during the last five minutes |
analytics_task_resurrect_count | Shows a count of task resurrection operations in the process tree |
analytics_task_resurrect_count_1m | Shows a count of task resurrection operations in the process tree estimated during the last minute |
analytics_task_resurrect_count_5m | Shows a count of task resurrection operations in the process tree estimated during the last five minutes |
Go
Metric | Description |
---|---|
go_gc_duration_seconds | A summary of the GC invocation durations. |
go_goroutines | Number of goroutines that currently exist. |
go_info | Information about the Go environment. |
go_memstats_alloc_bytes | Number of bytes allocated and still in use. |
go_memstats_alloc_bytes_total | Total number of bytes allocated, even if freed. |
go_memstats_buck_hash_sys_bytes | Number of bytes used by the profiling bucket hash table. |
go_memstats_frees_total | Total number of frees. |
go_memstats_gc_cpu_fraction | The fraction of this program's available CPU time used by the GC since the program started. |
go_memstats_gc_sys_bytes | Number of bytes used for garbage collection system metadata. |
go_memstats_heap_alloc_bytes | Number of heap bytes allocated and still in use. |
go_memstats_heap_idle_bytes | Number of heap bytes waiting to be used. |
go_memstats_heap_inuse_bytes | Number of heap bytes that are in use. |
go_memstats_heap_objects | Number of allocated objects. |
go_memstats_heap_released_bytes | Number of heap bytes released to OS. |
go_memstats_heap_sys_bytes | Number of heap bytes obtained from system. |
go_memstats_last_gc_time_seconds | Number of seconds since 1970 of last garbage collection. |
go_memstats_lookups_total | Total number of pointer lookups. |
go_memstats_mallocs_total | Total number of mallocs. |
go_memstats_mcache_inuse_bytes | Number of bytes in use by mcache structures. |
go_memstats_mcache_sys_bytes | Number of bytes used for mcache structures obtained from system. |
go_memstats_mspan_inuse_bytes | Number of bytes in use by mspan structures. |
go_memstats_mspan_sys_bytes | Number of bytes used for mspan structures obtained from system. |
go_memstats_next_gc_bytes | Number of heap bytes when next garbage collection will take place. |
go_memstats_other_sys_bytes | Number of bytes used for other system allocations. |
go_memstats_stack_inuse_bytes | Number of bytes in use by the stack allocator. |
go_memstats_stack_sys_bytes | Number of bytes obtained from system for stack allocator. |
go_memstats_sys_bytes | Number of bytes obtained from system. |
go_threads | Number of OS threads created. |
Process
Metric | Description |
---|---|
process_cpu_seconds_total | Total user and system CPU time spent in seconds. |
process_max_fds | Maximum number of open file descriptors. |
process_open_fds | Number of open file descriptors. |
process_resident_memory_bytes | Resident memory size in bytes. |
process_start_time_seconds | Start time of the process since unix epoch in seconds. |
process_virtual_memory_bytes | Virtual memory size in bytes. |
process_virtual_memory_max_bytes | Maximum amount of virtual memory available in bytes. |
Prometheus
Metric | Description |
---|---|
promhttp_metric_handler_requests_in_flight | Current number of scrapes being served. |
promhttp_metric_handler_requests_total | Total number of scrapes by HTTP status code. |
Sensor
Metric | Description |
---|---|
sensor_active_subscriptions | Amount of active sensor subscriptions |
sensor_buffer_memory_usage | Shows the current buffer memory usage by SLS in bytes |
sensor_kernel_samples_lost | Shows a count of kernel samples lost |
sensor_perf_record_lost_count | Shows the number of PERF_RECORD_LOST samples processed |
sensor_perf_record_lost_total_count | Shows the sum total of PERF_RECORD_LOST lost fields |
sensor_perf_record_sample_count | Shows the number of PERF_RECORD_SAMPLE samples processed |
sensor_perf_record_sample_counter_count | Shows the number of PERF_RECORD_SAMPLE samples having counter information |
sensor_perf_record_sample_trace_count | Shows the number of PERF_RECORD_SAMPLE samples having trace information |
sensor_total_event_count | Shows a count of events currently processed by the perf sensor |