Data Lake storage limits
There are limits on how much data you can store in the Sophos Data Lake.
For devices we set the limits as follows:
- A daily limit for one device.
- A monthly limit for all your devices.
For cloud assets we set limits as described in the Sophos Cloud Optix section.
Daily limit for one device
Devices can upload no more than 250MB of data per day.
When a device reaches the limit, it stops uploading data until the limit resets. Data that the device doesn’t upload to the Data Lake during this time won’t be sent later. You can only query that data directly on the device.
For Windows devices, the limit resets at midnight local time.
For Linux devices, the limit resets every 24 hours after the XDR agent has started.
All other communications between your devices and Sophos, including threat alerts, continue as usual.
If your devices are uploading more data than you expect, you can find out which of your queries are generating the most data. You can then use this information to investigate your data uploads. See Troubleshoot daily limit breaches.
Monthly limit for all your devices
The Data Lake stores data for up to 30 days. The total storage allowed during that time is based on the number of XDR licenses.
There are separate storage pools for endpoints and servers:
- The endpoint pool can have 20 MB per license per day (600 MB per license per month).
- The server pool can have 40 MB per license per day (1200 MB per license per month).
If devices exceed these limits, the Data Lake won't be able to make the full 30 days of data available to query.
How device limits work
Suppose you have the following licenses.
- 10 Intercept X Advanced with XDR
- 10 Intercept X Advanced for Server with XDR
During a 30-day period, you can store the following amounts of data.
- Up to 6 GB of endpoint data (10 licenses x 20MB x 30 days)
- Up to 12 GB of server data (10 licenses x 40MB x 30 days)
If you exceed a storage limit, we remove the oldest data until your data is under the limit.
For example, if your endpoints upload 40 MB per day (twice the average allowed across the month), they reach the limit after only 15 days. We then start to remove the oldest data, so you can't query the full 30 days of historic data.
However, if your devices upload less than the maximum amount allowed, we still only store their data for 30 days.
Sophos Cloud Optix storage limits
You can set up Sophos Cloud Optix to send data from your cloud environments to the Data Lake. If you do, you have Data Lake storage limits for Sophos Cloud Optix data.
Data from Sophos Cloud Optix is stored in the Data Lake for 30 days, or until you reach your storage limit, whichever comes first. Your storage limit is determined by the number of cloud assets you have licensed.
You can store up to 1MB per cloud asset per day. For example, if you have 100 cloud assets, you can store up to 3GB data from Sophos Cloud Optix in the Data Lake (100 licenses x 1MB x 30 days). If you exceed the storage limit, we remove the oldest data until your data is under the limit and the Data Lake can't make the full 30 days of data available to query.
A limit also applies to the amount of data you can upload from Sophos Cloud Optix to the Data Lake each day. Your daily upload limit is determined by the number of cloud assets that you have licensed. You can upload up to 1.25MB per cloud asset per day. For example if you have 100 cloud assets, you can upload up to 125MB data each day (100 licenses x 1.25MB).
If you reach your daily upload limit, Sophos Cloud Optix stops uploading data until the limit resets at 00:00 UTC. Sophos Cloud Optix then automatically resumes uploading data from the point where it stopped.