Skip to content

Data Lake storage limits

There are limits on how much data you can store in the Sophos Data Lake.

For devices we set the limits as follows:

  • A daily limit for one device.
  • A 90-day limit for all your devices.

For cloud assets we set limits as described in the Sophos Cloud Optix section.

Daily limit for one device

Each device is limited to uploading a maximum of 2 GB of data per day.

When a device reaches the limit, it stops uploading data until the limit resets. Data that the device doesn’t upload to the Data Lake during this time won’t be sent later. You can only query that data directly on the device.

For Windows devices, the limit resets at midnight local time.

For Linux devices, the limit resets every 24 hours after the XDR agent has started.

All other communications between your devices and Sophos, including threat alerts, continue as usual.

If your devices are uploading more data than you expect, you can find out which of your queries are generating the most data. You can then use this information to investigate your data uploads. See Troubleshoot daily limit breaches.

90-day limit for all your devices

The Data Lake stores data for up to 90 days. The total storage allowed during that time is based on the number of XDR licenses.

There are separate storage pools for endpoints and servers:

  • The endpoint pool can have 20 MB per license per day (1.8 GB per license per 90 days).
  • The server pool can have 40 MB per license per day (3.6 GB per license per 90 days).

If devices exceed these limits, the Data Lake won't be able to make the full 90 days of data available to query.

How device limits work

Suppose you have the following licenses.

  • 10 Intercept X Advanced with XDR
  • 10 Intercept X Advanced for Server with XDR

During a 90-day period, you can store the following amounts of data.

  • Up to 18 GB of endpoint data (10 licenses x 20 MB x 90 days)
  • Up to 36 GB of server data (10 licenses x 40 MB x 90 days)

If you exceed a storage limit, we remove the oldest data until your data is under the limit.

For example, if your endpoints upload 40 MB per day (twice the average allowed across the month), they reach the limit after only 45 days. We then start to remove the oldest data, so you can't query the full 90 days of historic data.

However, if your devices upload less than the maximum amount allowed, we still only store their data for 90 days.

Sophos Cloud Optix storage limits

You can set up Sophos Cloud Optix to send data from your cloud environments to the Data Lake. If you do, you have Data Lake storage limits for Sophos Cloud Optix data.

Data from Sophos Cloud Optix is stored in the Data Lake for 90 days, or until you reach your storage limit, whichever comes first. Your storage limit is determined by the number of cloud assets you have licensed.

Over the 90-day period, you can store up to 1 MB per cloud asset per day. For example, if you have 100 cloud assets, you can store up to 9 GB data from Sophos Cloud Optix in the Data Lake (100 licenses x 1 MB x 90 days). If you exceed the storage limit, we remove the oldest data until your data is under the limit and the Data Lake can't make the full 90 days of data available to query.

A limit also applies to the amount of data you can upload from Sophos Cloud Optix to the Data Lake in a single day. Your daily upload limit is determined by the number of cloud assets that you have licensed. You can upload up to 1.25 MB per cloud asset per day. For example if you have 100 cloud assets, you can upload up to 125 MB data each day (100 licenses x 1.25 MB).

If you reach your daily upload limit, Sophos Cloud Optix stops uploading data until the limit resets at 00:00 UTC. Sophos Cloud Optix then automatically resumes uploading data from the point where it stopped.