Skip to content

Page permalink

Always use the following permalink when referencing this page. It will remain unchanged in future help versions.

https://docs.sophos.com/fusion/customer/help/en-us/index.html?contextId=xdr-integration-custom-parsers-repeating-fields

Add data to repeating fields

A Sophos Next-Gen SIEM subscription is required to use custom parsers. See Sophos Next-Gen SIEM overview.

Use these examples to populate repeated schema fields in custom parsers. Repeated fields can contain multiple values or objects, such as indicators, attachments, metadata records, and other collections.

The appropriate method depends on the schema field and the structure of the data being normalized.

You can populate repeated fields in the following ways:

  • Use ADDFIELD() to create objects from arrays of values.
  • Create objects individually and add them to an array.
  • Populate KeyValuePairsIndexed fields such as event_metadata.

The following examples show how to create and populate an array of ThreatIntelligenceIndicators schema objects.

Use the ADDFIELD function

Use ADDFIELD() when you need to populate a repeated field from one or more arrays of values.

This function adds a field to an array of objects. The values of the field for each object are specified by fieldValues (also an array). The name of the new field is specified by fieldName. If collection is NULL, a new array of objects is created, each with a single field (fieldName) with the provided values.

In this example, each file hash becomes a ThreatIntelligenceIndicators object with a type and value field.

Example

{"id":"abc123_yessss","azureTenantId":"11111111-cccc-4444-9999-bbbbbbbbbbbb","azureSubscriptionId":null,"riskScore":null,"tags":[],"activityGroupName":null,"assignedTo":null,"category":"DefenseEvasion","closedDateTime":null,"comments":[],"confidence":null,"createdDateTime":"2022-11-03T14:58:39.8189203Z","description":"A Windows executable file that has been renamed was launched. Attackers might be attempting to use the executable to perform malicious activities, such as running malicious code or connecting to an attacker-controlled network address. By changing the path or file name of a system executable, attackers can avoid certain behavior-based detection technologies that rely on default paths and file names.","detectionIds":[],"eventDateTime":"2022-11-03T14:51:19.3637988Z","feedback":null,"incidentIds":[],"lastEventDateTime":null,"lastModifiedDateTime":"2022-11-03T14:59:15.9766667Z","recommendedActions":[],"severity":"medium","sourceMaterials":["https://security.microsoft.com/alerts/abc123_yessss","https://security.microsoft.com/incidents/12345"],"status":"newAlert","title":"System executable renamed and launched","vendorInformation":{"provider":"Microsoft Defender ATP","providerVersion":null,"subProvider":"MicrosoftDefenderATP","vendor":"Microsoft"},"alertDetections":[],"cloudAppStates":[],"fileStates":[{"name":"weirdFile.exe","path":"C:\\Users\\jdoe\\AppData\\Local\\Temp","riskScore":null,"fileHash":{"hashType":"sha1","hashValue":"8d7f701jk1234567jiff78q9e482hm41ef9a0h12k6k"}},{"name":"mammals.bat","path":"D:\\Destination","riskScore":null,"fileHash":{"hashType":"sha1","hashValue":"aaaaaabbbbbbbcccccdddddd1111122223333444"}}],"hostStates":[{"fqdn":"le_host.acme.com","isAzureAdJoined":true,"isAzureAdRegistered":null,"isHybridAzureDomainJoined":null,"netBiosName":null,"os":"Windows10","privateIpAddress":"10.220.100.100","publicIpAddress":"8.8.8.8","riskScore":"high"}],"historyStates":[],"investigationSecurityStates":[],"messageSecurityStates":[],"networkConnections":[],"processes":[],"registryKeyStates":[],"securityResources":[],"triggers":[],"userStates":[{"aadUserId":"777aaaaa-5555-4444-bbbb-777555555777","accountName":"jdoe","domainName":"HQ","emailRole":"unknown","isVpn":null,"logonDateTime":null,"logonId":null,"logonIp":null,"logonLocation":null,"logonType":null,"onPremisesSecurityIdentifier":null,"riskScore":null,"userAccountType":null,"userPrincipalName":"jane.doe@acme.com"}],"uriClickSecurityStates":[],"vulnerabilityStates":[]}

threatIntelligenceIndicators$ = ADDFIELD(NULL, "type$", json["$.fileStates[*].fileHash.hashType"])
threatIntelligenceIndicators$ = ADDFIELD(threatIntelligenceIndicators$, "value$", json["$.fileStates[*].fileHash.hashValue"])

Create key-value pairs and add to an object

Use this method when each object requires separate processing or conditional logic before being added to a repeated field.

Example

{"id":"abc123_yessss","azureTenantId":"11111111-cccc-4444-9999-bbbbbbbbbbbb","azureSubscriptionId":null,"riskScore":null,"tags":[],"activityGroupName":null,"assignedTo":null,"category":"DefenseEvasion","closedDateTime":null,"comments":[],"confidence":null,"createdDateTime":"2022-11-03T14:58:39.8189203Z","description":"A Windows executable file that has been renamed was launched. Attackers might be attempting to use the executable to perform malicious activities, such as running malicious code or connecting to an attacker-controlled network address. By changing the path or file name of a system executable, attackers can avoid certain behavior-based detection technologies that rely on default paths and file names.","detectionIds":[],"eventDateTime":"2022-11-03T14:51:19.3637988Z","feedback":null,"incidentIds":[],"lastEventDateTime":null,"lastModifiedDateTime":"2022-11-03T14:59:15.9766667Z","recommendedActions":[],"severity":"medium","sourceMaterials":["https://security.microsoft.com/alerts/abc123_yessss","https://security.microsoft.com/incidents/54321"],"status":"newAlert","title":"System executable renamed and launched","vendorInformation":{"provider":"Microsoft Defender ATP","providerVersion":null,"subProvider":"MicrosoftDefenderATP","vendor":"Microsoft"},"alertDetections":[],"cloudAppStates":[],"fileStates":[{"name":"weirdFile.exe","path":"C:\\Users\\jdoe\\AppData\\Local\\Temp","riskScore":null,"fileHash":{"hashType":"sha1","hashValue":"8f7f1jk1234567jiff78q9e482hm41ef9a0h12k6k"}},{"name":"mammals.bat","path":"D:\\Destination","riskScore":null,"fileHash":{"hashType":"sha1","hashValue":"aaaaaabbbbbbbcccccdddddd1111122223333444"}}],"hostStates":[{"fqdn":"le_host.acme.com","isAzureAdJoined":true,"isAzureAdRegistered":null,"isHybridAzureDomainJoined":null,"netBiosName":null,"os":"Windows10","privateIpAddress":"10.220.100.100","publicIpAddress":"8.8.8.8","riskScore":"high"}],"historyStates":[],"investigationSecurityStates":[],"messageSecurityStates":[],"networkConnections":[],"processes":[],"registryKeyStates":[],"securityResources":[],"triggers":[],"userStates":[{"aadUserId":"777aaaaa-5555-4444-bbbb-777555555777","accountName":"jdoe","domainName":"HQ","emailRole":"unknown","isVpn":null,"logonDateTime":null,"logonId":null,"logonIp":null,"logonLocation":null,"logonType":null,"onPremisesSecurityIdentifier":null,"riskScore":null,"userAccountType":null,"userPrincipalName":"jane.doe@acme.com"}],"uriClickSecurityStates":[],"vulnerabilityStates":[]}

fileInfo0.type$ = json["$.fileStates[0].fileHash.hashType"]
fileInfo0.value$ = json["$.fileStates[0].fileHash.hashValue"]
fileInfo1.type$ = json["$.fileStates[1].fileHash.hashType"]
fileInfo1.value$ = json["$.fileStates[1].fileHash.hashValue"]

threatIntelligenceIndicators$ = IF malwareInfo.value$ THEN [malwareInfo] ELSE []
threatIntelligenceIndicators$ = IF fileInfo0.value$ THEN threatIntelligenceIndicators$ + [fileInfo0] ELSE threatIntelligenceIndicators$
threatIntelligenceIndicators$ = IF fileInfo1.value$ THEN threatIntelligenceIndicators$ + [fileInfo1] ELSE threatIntelligenceIndicators$

Add key-value pairs to event_metadata and other KeyValuePairsIndexed fields

Use this approach when storing supplemental information that doesn't map directly to dedicated schema fields.

The following example shows how to populate the event_metadata schema field with source data that doesn't map to dedicated schema fields.

Example

Jul 29 19:03:55 10.12.133.192 1 2021-07-29T19:03:55.527482+00:00 epp.deeparmor.io Taegis NGAV - - - DANOTIFY : Event:"Action" EventTime:"2021-07-29T19:00:33.368744+00:00" AlertGuid:"ffg5bfc9-61b6-4cf6-dd34-33c613bb6edf" Account:"df45d555-5abd-4b33-9bb3-c6a43ee2208d" Partner:"SECUREWORKS" FileName:"macho_abnormal_file" FilePath:"C:\Users\taegisnextgenav\Downloads\macho_abnormal_file" FileSize:"335872" SHA1:"5A1F8BC4873B036AF13A908C5140FCBD0E550D81" FileType:"macho" AlertType:"SUSPICIOUS" Score:"0.93905586" ActionTaken:"QUARANTINED" DetectionMethod:"RTFM" First:"2021-07-23T17:19:16.929711+00:00" DACloud:"CONNECTED" Network:"CONNECTED" ThreatCategory:"Trojan" ThreatName:"macOS.Trojan" ThreatLevel:"5" RunningApps:"ApplicationFrameHost, csrss, ctfmon, dllhost, dwm, explorer, fontdrvhost, LogonUI, Microsoft.Photos, mmc, msedge, OneDrive, rdpclip, RuntimeBroker, SearchApp, SecurityHealthSystray, ShellExperienceHost, sihost, StartMenuExperienceHost, svchost, taskhostw, TextInputHost, winlogon, YourPhone" DeviceName:"NextGenAVTest" DeviceUsername:"(Not Logged In)" DeviceGuid:"36425b3b-c7b0-450d-9373-96a261607ebf" DeviceOS:"Microsoft Windows 10 Pro 10.0.19042" DeviceAgent:"2021.6.0" DeviceIP:"10.0.0.4" DeviceIPV4:"10.12.128.8" DeviceMac:"00113718B4F2" DeviceFqdn:"NextGenAVTest" DeviceGroup:"Test-Lab-DA" DevicePolicy:"Detection and Protection" DeviceScore:"95" DeviceCreated:"2021-07-23T13:48:08.340082+00:00"

deviceFqdn.key$ = "DeviceFqdn"
deviceFqdn.value$ = vals["DeviceFqdn"]
deviceGroup.key$ = "DeviceGroup"
deviceGroup.value$ = vals["DeviceGroup"]
deviceScore.key$ = "DeviceScore"
deviceScore.value$ = vals["DeviceScore"]
eventMetadata$.record$ = eventMetadata$.record$ + [deviceFqdn, deviceGroup, deviceScore]