Create custom parsers

Users with the Super Admin or Admin role can create custom parsers.

The key steps to creating custom parsers are as follows:

1. Create: Create the parser and select the message format. Name it using the recommended format. See Create a custom parser.
2. Write: Use the parser syntax and helper functions to extract and transform fields, and set a confirm string (a regular expression or expression) so the parser matches only its intended messages. See Write your custom parser.
3. Add sample messages: Provide example logs using original data, the copy-log method, or dummy sample data. See Sample messages.
4. Test: Run the parser against the sample messages. Repeat testing (12 to 15 runs is recommended) to catch unintended matches and reach high confidence. See Test a custom parser.
5. Save: Save the parser configuration. See Save a custom parser.
6. Deploy: Deploy the parser so it begins processing incoming messages. See Deploy a custom parser.

Known issues

Messages from unsupported data sources can occasionally match existing parsers and be normalized incorrectly.

To reduce this risk:

• Test sample messages multiple times.
• Use a variety of sample messages.
• Verify that the expected parser appears in the extractor path.

If another parser matches your data source, contact Sophos Support and ask for the parser specificity to be reviewed.

Create a custom parser

To create a new custom parser, do as follows:

1. Go to Security Operations > Integrations > Custom Parsers.
2. Click Add Custom Parser.

3. Name your custom parser. We recommend this format:

   Examples: FUNBEAM_SMART_TOASTER_CEF_PARENT and PAN_FW123456_JSON_CHILD_AUTHENTICATION.

4. Click Override Global Parser if the global parser behavior does not provide expected results for your data. See Override a global parser.

5. Click Set Global Parent in Parser Script to create extensions for new or unsupported event types of global parent parsers that are not available in the parent parser drop-down. See Extend a global parser.

6. Select whether it's a Parent, Child, or Standalone parser:

   • Parent parsers default to the generic event schema. Assume that you are creating child parsers to normalize incoming messages to multiple schemas.
   • Child parsers require a parent parser to be selected and should have their final destination schema selected.
   • A standalone parser is used for telemetry data sources that only have one message type and normalize to only one schema.

7. Select the source data format. This populates the generated parser boilerplate script with the correct helper function in order to split the incoming message.

8. Select the destination schema.

   Note

   The best practice for parent parsers is to set the destination schema to scwx.generic.

9. Click Create. The script editor loads a pre-formatted boilerplate to get you started writing your parser.

Write your custom parser

The following best practices can help with readability, maintenance, and troubleshooting for your custom parser:

• The top section of your custom parser should contain any required bang (!) declarations.
• Add an appropriate splitter function for the selected log message format.
• Add any variable declarations and content manipulations that are necessary to format the data for the destination schema or pass it to child parsers.
• protobuf declarations determine the data that gets passed into the event schema.

Set the !CONFIRMWITH and !CONFIRMSTRING declarations

Parent parsers are usually configured with a regex-based confirmation pattern by declaring !CONFIRMWITH=PATTERN. Write a regex pattern that matches a key part of all of the messages that you want to normalize with this custom parser. Typically this matches the vendor name and product as found in the log messages received from the data source.

!CONFIRMWITH=PATTERN
!CONFIRMSTRING=\|Imperva Inc.\|SecureSphere-MX

Nov 11 15:55:25 172.16.11.99 CEF:0|Imperva Inc.|SecureSphere-MX|13.0.1|Correlation|sql-failed-login|Medium|act=None dst=172.16.3.45 dpt=1433 duser=Multiple src=172.16.3.51 spt=47635 proto=TCP rt=Nov 11 2022 06:05:08 cat=Alert cs1=SQL Correlation Policy  cs1Label=Policy cs2=Group1 cs2Label=ServerGroup cs3=sql-service cs3Label=ServiceName cs4=Default SQL Application cs4Label=ApplicationName cs5=SQL Failed Login  cs5Label=Description

Child parsers are usually set to confirm using an expression that evaluates to TRUE that is derived from variables set in the parent parser, and passed to the child parser. This is done using normal comparison operators.

!CONFIRMWITH=EXPRESSION
!CONFIRMSTRING=eventType$=="AUTH"

We recommend that you test your parser at this point to see if the confirm string works. See Test a custom parser.

Setting variables and Protobuf fields

Custom parsers use two types of values:

• Variables: A variable can be declared at any point in the file. It can be used for operations including comparison and concatenation, being passed into a helper function, or any other operation a variable is valid for. A variable is only used within the parser script, and is not passed to the destination schema.
• Protobuf field: A Protobuf field ends in a dollar sign $. Protobuf fields are reserved for data passed into the destination schema. Declarations not matching the destination schema are ignored. For example, when normalizing to the authentication schema, encryption_type$ is a valid Protobuf field because it matches a field in the schema. my_snazzy_field$ is not a valid Protobuf field and is ignored by the normalizer.

Protobuf fields map 1:1 to schema fields, but the syntax is slightly different. Field names in the schema are in snake_case, while the equivalent Protobuf field is in camelCase and ends in a dollar sign. So event_time_usec as a schema field maps to eventTimeUsec$ as a Protobuf field.

Variables and Protobuf fields are passed from parent to child, so a variable or Protobuf field that is set in a parent parser is available for use in all of its descendants.

Common operations and functions

For common operations and functions to use in your parser, see Custom parser syntax.

Sample messages

Test your parser script frequently while editing. To test, you need at least one sample message.

Original data method

If you already have data being sent to Sophos XDR, the best option is to copy the original data from an event and paste it into the sample message field. Go to Data Lake Search and run the query FROM generic WHERE sensor_id='[my data source]', replacing [my data source] with the name of the product sending logs to Sophos XDR as defined in the product's logs.

Open the details of an event resulting from your search and go to the Original Data tab. Click Copy Original Data and then paste this data into the Sample Message field in your custom parser's edit screen. Be sure to use an event that has all of the data you'll want to parse.

Copy log method

If you haven't started sending data from your data source, you can still test your data, although we recommend starting by sending the data to help rule out connection issues while troubleshooting. Copy a single line log message, not a whole log file, into the sample message field. Sample messages are meant for single line logs, not an entire log file.

Dummy sample message data

If you don't have log data yet, you can still test a custom parser using the following log messages:

Nov 11 15:55:25 172.16.11.99 CEF:0|Imperva Inc.|SecureSphere-MX|13.0.1|Correlation|sql-failed-login|Medium|act=None dst=172.16.3.45 dpt=1433 duser=Multiple src=172.16.3.51 spt=47635 proto=TCP rt=Nov 11 2022 06:05:08 cat=Alert cs1=SQL Correlation Policy  cs1Label=Policy cs2=Group1 cs2Label=ServerGroup cs3=sql-service cs3Label=ServiceName cs4=Default SQL Application cs4Label=ApplicationName cs5=SQL Failed Login  cs5Label=Description

{"GUID":"8mr-Qf2IaEGl8ZrAgApMtUpJ1WE6MNw3","QID":"3m082vh0wa-1","ccAddresses":[],"cluster":"embdtech_production-vm","completelyRewritten":false,"fromAddress":["onedrive-storage-centers@post.com"],"headerFrom":"Portal Service \u003cOneDrive-storage-centers@post.com\u003e","headerReplyTo":"OneDrive-storage-centers@post.com","id":"17c704dd-32d7-7a7e-aabd-60bc25d6d7b8","impostorScore":0,"malwareScore":0,"messageID":"\u003cz2syHcM5R-KNfBRt164SIg@geopod-ismtpd-3-0\u003e","messageParts":[{"contentType":"text/plain","disposition":"inline","filename":"text.txt","md5":"02fb21a1047b2156d299dbbd5128eeeb","oContentType":"text/plain","sandboxStatus":null,"sha256":"46e52ba50776ff8b95baee20c8f7d2880653c02e60aeb5b7d2b9adce97d8eca2"},{"contentType":"text/html","disposition":"inline","filename":"text.html","md5":"cf84f8757abdbd98c0557b4fb595fc31","oContentType":"text/html","sandboxStatus":null,"sha256":"fbce44eb0a20aa14f8c4270a387ef8f7b86fa6df8227824555eac77286d6efbd"}],"messageSize":7320,"messageTime":"2022-11-22T16:08:29.000Z","modulesRun":["av","zerohour","dkimv","spf","spam","dmarc","pdr","urldefense"],"phishScore":100,"policyRoutes":["default_inbound"],"quarantineFolder":"Phish","quarantineRule":"inbound_passive_phish","recipient":["asmith@embdtech.com"],"replyToAddress":["onedrive-storage-centers@post.com"],"sender":"bounces+30215492-d119-asmith=embdtech.com@sendgrid.net","senderIP":"167.22.22.22","spamScore":0,"subject":"A notice sent to your mail-box asmith@embdtech.com just\r\n arrived","threatsInfoMap":[{"campaignID":null,"classification":"phish","threat":"bafybeidonkf2nkpftvrxzpjziloxhgjzdr4behzpikhktcsntarflqhkby.ipfs.w3s.link/","threatID":"3e453ec875e5d875d9a97bd2607e2b98e9915d3fbb01dd8c8e1b9314564c5c7b","threatStatus":"active","threatTime":"2022-11-22T15:38:23.000Z","threatType":"url","threatUrl":"https://threatinsight.proofpoint.com/44b3419d-b1e9-7db6-d77e-7a5cf27f7cf2/threat/email/3e453ec875e5d875d9a97bd2607e2b98e9915d3fbb01dd8c8e1b9314564c5c7b"}],"toAddresses":["asmith@embdtech.com"],"xmailer":null}

Test a custom parser

Once you have loaded a sample message, click Parse Message. If your message is well formed, the Parsed Source Message table shows one of the following:

• The key:value relationships between the fields for structured message formats like CEF, LEEF, and JSON
• An indexed breakdown of the fields for unstructured formats

Add a variety of sample messages to ensure that the range of data sent is parsed correctly. If you're using child parsers, do this at both the parent and child parser level. Once you've got a representative set of sample messages, click Run Test to see the event output for all of the sample messages.

Test Results

When you press the Parse Message button, the test results will show in the abbreviated tables below the message. From here you'll see whether your parser is generating any results at all. Once those tables are populating, you can expand them to see the full results. The Parsed Source Message table shows how the message is broken down into segments by the splitting function. The Source Destination Mapping table shows how transformed source data is assigned to event schema fields.

The extractor path above the Source Destination Mapping table shows exactly which parsers were used to parse the message. It is possible to create a situation where a message could be parsed by more than one available parser, and this is where you'll see if an unexpected parser is picking up your sample message. If your sample message is parsed by a parser other than the custom parser you're creating, a warning displays.

Click Run Test to test all sample messages and output the results similarly to an event search. It will also show what event schema the message is normalized to, as well as the extractor path.

Save a custom parser

The testing tools test whatever is currently loaded in the script editor interface. Testing does not save the script. This allows you to test changes to an active parser without changing its behavior in the production environment.

To save the parser, click Save and Continue.

Disabled parsers can be saved even if validation fails. Enabled parsers must pass validation before they can be saved.

Exit the edit interface

Click Exit to return to the Custom Parsers page.

Any unsaved changes are lost.