Hi Sentinel2Go team

Thank you for this very useful project, I appreciate it.

Two weeks ago I was able to successfully deploy the Win10-AD-WEC template
https://github.com/OTRF/Microsoft-Sentinel2Go/tree/master/grocery-list/Win10-AD-WEC

Today I have tried again and I get the following validation failure
{"code":"InvalidTemplate","message":"Deployment template validation failed: 'The template parameters 'Workspace Region' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'Workspace, WorkspaceRegion'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'."}

Am I doing something wrong? I'm using the same Azure account as a few weeks ago. Do you need any other information? Thanks

Jag

Hi

I am trying to deploy Win10-AD-WEC and I'm getting this error. I've tried adding to parameters but I'm not doing that correctly or the problem is elsewhere.

Error Details
ERROR TYPE
Deployment template validation failed: 'The value for the template parameter 'domainNetbiosName' at line '17' and column '30' is not provided. Please see https://aka.ms/arm-create-parameter-file for usage details.'. (Code: InvalidTemplate)

I have had success using s2g in the past and I am very grateful for your project. I haven't used Azure or Sentinel in a few years and I can't work out how to fix the error.

I've searched github and I find this but unfortunately I don't know how to piece it together to make it work
8778246#diff-fb4127e4c65adb83a36da85516f01088e0202a7dd0f7760eaf0c3ccd95e9e550

Thanks for reading

I've been having a look at the way you've created ARM templates for data connectors, and I saw that Windows Firewall and DNS (Preview) use the Microsoft.OperationsManagement/solutions type with the product property being "OMSGallery/WindowsFirewall" and "OMSGallery/DnsAnalytics" respectively. I was wondering how you find the correspondence between a Data Connector in the Sentinel GUI and the correct value to use for the product property in the template? Sorry if this is a noob question, but so far I've been unable to find much information about this.

Hey there,

First off, just wanna say I really appreciate your work Roberto and am looking forward to part 2 of your Sentinel2Go post.

The current deployment template for Azure Sentinel + Win10 + Domain Controller fails validation due to the fact that the enableWinEventProviders parameter doesn't exist in the deployAzureSentinel2Go template.

Changing enableWinEventProviders to collectWinEventProviders solves the issue.

{
  "deploymentStatusCode": -1,
  "stage": 6,
  "expected": true,
  "error": {
    "message": "Deployment template validation failed: 'The template parameters 'enableWinEventProviders' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'utcValue, workspaceName, pricingTier, dataRetention, immediatePurgeDataOn30Days, enableAdditionalLASolutions, enableDataConnectorsKind, enableLAFunctions, setSecurityCollectionTier, collectWinEventProviders, collectSyslogFacilities, postAnalyticRules, userAssignedIdentityName, _artifactsLocation, _artifactsLocationSasToken, location'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'."
  },
  "subscriptionId": "115ffef8-43d8-4b25-a900-XXXXXXXXX",
  "resourceGroupName": "VA-Lab",
  "resourceGroupLocation": "southeastasia",
  "deploymentName": "Microsoft.Template-20200917211020",
  "details": {
    "code": "InvalidTemplate",
    "message": "Deployment template validation failed: 'The template parameters 'enableWinEventProviders' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'utcValue, workspaceName, pricingTier, dataRetention, immediatePurgeDataOn30Days, enableAdditionalLASolutions, enableDataConnectorsKind, enableLAFunctions, setSecurityCollectionTier, collectWinEventProviders, collectSyslogFacilities, postAnalyticRules, userAssignedIdentityName, _artifactsLocation, _artifactsLocationSasToken, location'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.",
    "additionalInfo": [
      {
        "type": "TemplateViolation",
        "info": {
          "lineNumber": 0,
          "linePosition": 0,
          "path": ""
        }
      }
    ]
  }
}

Hey thanks for creating this, it's awesome.

I manually setup an ELK stack with Mordor data a year or so ago and it took me quite a few hours to get it working. This was up and running in a few minutes after reading the docs.

One area where I'm stuck is I'm getting different results than in the blog post here: https://techcommunity.microsoft.com/t5/microsoft-sentinel-blog/azure-sentinel-to-go-part1-a-lab-w-prerecorded-data-amp-a-custom/ba-p/1260191

I run the following command:
prerecorded_CL | extend m=parse_json(Message) | summarize count() by EventID=tostring(m.winlog.event_id),EventProvider=tostring(m.winlog.channel),Task=tostring(m.winlog.task)

And my results are as follows:

I can tell I have the mordor data present:

But for some reason it looks like parse_json() isn't working the same way on the Message field, or maybe I'm not executing the query properly. I'm used to Kibana KQL. I feel like I'm just doing something dumb or missing a step. Any ideas?

Edit: And I guess as a follow-up, when I imported these datasets into ELK I did some work to have them parsed and saved in Elastic Common Schema, ECS. I know you worked on OSSEM and It looks like MS has a similar set of common schema formats defined in ASIM here: https://docs.microsoft.com/en-us/azure/sentinel/normalization Have you used any parsers in the log ingest pipeline to get this into a common schema so that we don't need to worry about the 500 custom field limit or the parse_json() calls in the queries?

I got 404 when i try to deploy any of this
https://raw.githubusercontent.com/OTRF/Azure-Sentinel2Go/master/grocery-list/Win10/azuredeploy.json
https://raw.githubusercontent.com/OTRF/Azure-Sentinel2Go/master/grocery-list/Win10-AD/azuredeploy.json
https://raw.githubusercontent.com/OTRF/Azure-Sentinel2Go/master/grocery-list/Win10-AD-ADFS/azuredeploy.json
https://raw.githubusercontent.com/OTRF/Azure-Sentinel2Go/master/grocery-list/Win10-PAN-FW/azuredeploy.json
https://raw.githubusercontent.com/OTRF/Azure-Sentinel2Go/master/grocery-list/Linux/azuredeploy.json

I used https://raw.githubusercontent.com/OTRF/Azure-Sentinel2Go/master/grocery-list/Win10-AD/azuredeploy.json for some days ago but now 404.

Hi,

When I used logstash to send both large dataset and small dataset I have found the logstash showing in logs "resending the logs" How to avoid sending same logs again and again via logstash? how to ensure. My sentinel workspace populated with total of 1.3M

I have found many zeek logs instead of .json files in largedatasets. How to send such .log extention files to sentinel.
Do we have to use a seperate custom table?
How come the schema will be protected while sending?

Note: I am running a new system with logstash installed without docker setup and without your Template.

The below file is corrupted

• dataset-sample-small.tar.gz

root@Kali:~/scrip$ tar -xzvf dataset-sample-small.tar.gz

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Seven Days Ago (Today is May 10th, 2022), this broke things:
Azure/Azure-Sentinel@0702fcb

Same problem as this issue
Azure/Azure-Sentinel#4823

this was fixed for Windows with this PR. Now it needs to happen for Linux.

Azure/Azure-Sentinel@0eefa3e

The ASimNetworkSessionMicrosoftLinuxSysmon Template Parameters Changed
This template had a parameter named Workspace Region. However, it is now called WorkspaceRegion.

https://github.com/Azure/Azure-Sentinel/blob/master/Parsers/ASimNetworkSession/ARM/ASimNetworkSessionMicrosoftLinuxSysmon/ASimNetworkSessionMicrosoftLinuxSysmon.json#L11

Other Templates such as SysmonForLinuxFullDeployment Affected
Other templates call / deploy the ASimNetworkSessionMicrosoftLinuxSysmon template with the old parameter name Workspace Region.

https://github.com/Azure/Azure-Sentinel/blob/master/Parsers/ASim%20Sysmon%20for%20Linux/SysmonForLinuxFullDeployment.json#L108

Greetings Roberto..

Love your work! But currently having some trouble on deploying the Azure Sentinel + Win10 + Domain Controller Template.

Getting error on deployment phase deployWinAD fails:

Deployment template validation failed: 'The template variable 'ouPath' is not valid: The language expression property array index '1' is out of bounds.. Please see https://aka.ms/arm-template-expressions for usage details.'.

Hi - I tried to deploy the Sentinel 2 Go lab using the Custom Log Pipeline option and it appears that the nested Linux VM deployment script is failing due to the count values in the centOSVMs and redhatVMs Copy variables both being zero. The error I received was:

{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"BadRequest","message":"{\r\n \"error\": {\r\n \"code\": \"InvalidTemplate\",\r\n \"message\": \"Deployment template validation failed: 'The template 'copy' definition at line '283' and column '17' has an invalid copy count. The copy count must be a positive integer value and cannot exceed '800'. Please see https://aka.ms/arm-copy for usage details.'.\",\r\n \"additionalInfo\": [\r\n {\r\n \"type\": \"TemplateViolation\",\r\n \"info\": {\r\n \"lineNumber\": 283,\r\n \"linePosition\": 17,\r\n \"path\": \"variables.Copy\"\r\n }\r\n }\r\n ]\r\n }\r\n}"}]}

The input parameters (obfuscated where relevant) were:

{
    "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "workspaceName": {
            "value": "xxxxxxxxxxxxxx"
        },
        "pricingTier": {
            "value": "Free"
        },
        "dataRetention": {
            "value": 7
        },
        "immediatePurgeDataOn30Days": {
            "value": true
        },
        "adminUsername": {
            "value": "SecureUsername"
        },
        "authenticationType": {
            "value": "password"
        },
        "adminPasswordOrKey": {
            "value": "SecurePassword"
        },
        "vmSize": {
            "value": "Standard_B2s"
        },
        "allowedIPAddresses": {
            "value": "My.Home.IP.Address"
        },
        "addMordorDatasets": {
            "value": "mordor-large-apt29(2.0GB)"
        },
        "deployCustomLogsPipeline": {
            "value": "Logstash-EventHub"
        },
        "location": {
            "value": "uksouth"
        }
    }
}

Regards,
Andy

Really keen to get this set up for a more persistent research environment as a side project.

Have you come across any ways to get this up an running for non-academic researchers without having to commit to an Microsoft E5 license ?

https://github.com/OTRF/Azure-Sentinel2Go/blob/0746246df4a6a6fa899bb0be4efb221728cd1878/grocery-list/Win10-AD-ADFS/azuredeploy.json#L405

Resource deployWinADFS failed with the following reason. Looking at lines 404-406 in the above referenced ARM template, it looks like enableMonitoringAgent is not defined in https://github.com/OTRF/Blacksmith/blob/master/templates/azure/Win10-AD-ADFS/azuredeploy.json

{
    "status": "Failed",
    "error": {
        "code": "InvalidTemplate",
        "message": "Deployment template validation failed: 'The template parameters 'enableMonitoringAgent' in the parameters file are not valid; they are not present in the original template and can therefore not be provided at deployment time. The only supported parameters for this template are 'utcValue, adminUsername, adminPassword, adfsUsername, adfsPassword, domainUsers, remoteAccessMode, allowedIPAddresses, azureBastionHostName, azureBastionSubnetRange, domainFQDN, numberOfWorkstations, vmNamePrefix, windowsDesktopSKU, windowsDesktopVersion, windowsServerSKU, windowsServerVersion, vmSize, workspaceId, workspaceKey, virtualNetworkName, virtualNetworkAddressRange, subnetRange, subnetName, enableSysmon, enableAADConnect, _artifactsLocation, _artifactsLocationSasToken, pfxCertName, pfxCertPassword, certificateType, _pfxCertBlobSasUrl, location'. Please see https://aka.ms/arm-deploy/#parameter-file for usage details.'.",
        "additionalInfo": [
            {
                "type": "TemplateViolation",
                "info": {
                    "lineNumber": 0,
                    "linePosition": 0,
                    "path": ""
                }
            }
        ]
    }
}

You must update your templates because vimFileEventMicrosoftSysmonCreated and vimFileEventMicrosoftSysmonDeleted no longer available but now updated to vimFileEventMicrosoftSysmon.
{
"code": "MultipleErrorsOccurred",
"details": [
{
"code": "InvalidContentLink",
"message": "Unable to download deployment content from 'https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Parsers/ASimFileEvent/ARM/vimFileEventMicrosoftSysmonCreated/vimFileEventMicrosoftSysmonCreated.json'. The tracking Id is 'fa2e7104-634a-4526-b186-f9e7b0dc7930'. Please see https://aka.ms/arm-deploy-resources for usage details."
},
{
"code": "InvalidContentLink",
"message": "Unable to download deployment content from 'https://raw.githubusercontent.com/Azure/Azure-Sentinel/master/Parsers/ASimFileEvent/ARM/vimFileEventMicrosoftSysmonDeleted/vimFileEventMicrosoftSysmonDeleted.json'. The tracking Id is 'fa2e7104-634a-4526-b186-f9e7b0dc7930'. Please see https://aka.ms/arm-deploy-resources for usage details."
}
],
"message": "Multiple error occurred: BadRequest,BadRequest. Please see details."
}

Hi,

As checked the old module for injesting logs via logstash to azure log analytics is giving a lot of errors.

Solution:
The new module(microsoft-logstash-output-azure-loganalytics 1.0.0 version) installation:
/usr/share/logstash/bin/logstash-plugin install microsoft-logstash-output-azure-loganalytics

loganalytics-output.conf

output {
   microsoft-logstash-output-azure-loganalytics {
      workspace_id => "<workspace-ID"
      workspace_key => "sharedkey"
      custom_log_table_name => "prerecorded"
      plugin_flush_interval => 5
   }
   #stdout { codec => rubydebug }
}

Version is not supported.

Many thanks for all of your work. I really appreciate it!

I am trying to deploy the Log4Shell Research Lab but every attempt fails with the following error:

{
    "status": "Failed",
    "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.",
        "details": [
            {
                "code": "Conflict",
                "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource operation completed with terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"VMExtensionProvisioningError\",\r\n        \"message\": \"VM has reported a failure when processing extension 'LogAnalyticsAgent'. Error message: \\\"Enable failed with exit code 52 Couldn't create marker file\\\"\\r\\n\\r\\nMore information on troubleshooting is available at https://aka.ms/VMExtensionOMSAgentLinuxTroubleshoot \"\r\n      }\r\n    ]\r\n  }\r\n}"
            }
        ]
    }
}

I tried to use my existing Sentinel instance as well as creating a new one from scratch but the error stays the same. I also tried to use a larger VM size but that also didn't help. Maybe you can help me.

When I trigged the attack on UBUTU5, there is no Sentinel Alert or Incident.

Template: https://github.com/OTRF/Microsoft-Sentinel2Go/blob/master/grocery-list/Linux/demos/CVE-2021-44228-Log4Shell/azuredeploy.json

OMS Auditd Plugin: https://github.com/OTRF/Blacksmith/blob/master/templates/azure/Linux/azuredeploy.json#L630

Not all of the Pricing Tier options work with in my Subscription. Please consider updating these options. Thanks.

Deployment template validation failed: 'The template resource 'Microsoft.Resources/deployments/deployVnetPeering' reference to 'Microsoft.Resources/deployments/deployC2' requires an API version. Please see https://aka.ms/arm-template for usage details.'.