jonrau1 / syntheticsun Goto Github PK

Describe the bug
When running the gewalthaufen.py script in step 4 of stage 1, an error occurs due to endpoint_attachment() function being undefined

To Reproduce
Steps to reproduce the behavior:

1. In stage 1, step 4, populate the gewalthaufen.py with applicable values
2. Run the gewalthaufen.py script in step 4
3. See error:
   Traceback (most recent call last):
   File "gewalthaufen.py", line 256, in
   im_helping()
   File "gewalthaufen.py", line 248, in im_helping
   endpoint_attachment()
   NameError: name 'endpoint_attachment' is not defined

Expected behavior
Successful run

Story
As a user of SyntheticSun, I want to have a walkthrough and Jupyter notebook available to me so that I can train and deploy my own IP Insights model using my own datasets.

Definition of Done

• New appendix stage added to SyntheticSun
• Updated Readme and FAQ to point to this solution
• Jupyter notebook for IP Insights with pre-positioned data
• Script to prepare CloudTrail (IP Address + IAM Principal) data for training
• Script to prepare WAF logs (IP Address + user-agent) data for training

Nice to Have
N/A

Additional Information
https://github.com/awslabs/amazon-sagemaker-examples/blob/master/introduction_to_amazon_algorithms/ipinsights_login/ipinsights-tutorial.ipynb

Story
As a user of SyntheticSun, I want to refactor a majority of the solution for ease of deployment and dynamic onboarding for logging sources.

Definition of Done

• No more than 1 CFN template, if any at all
• Dynamic creation of all WAF and GuardDuty components (Threat Intel Sets, WAF IP Sets)
• Onboarding of FMS & baseline Policies
• Buffer all logging with SQS
• Dynamic onboarding of ALB and APIGW log sources
• Add CloudFront & Route53 to Scope
• New method to install Suricata on SSM-managed Instances
• Updated Readme

Nice to Have
N/A

Additional Information
In the current form, SyntheticSun is little more than a POC, it was not meant for ease of deployment and relied too heavily on CFN. A majority of the CFN templates should be removed, and new scripts take its place, this goes for the build automation pieces especially given that you would need to use a StackSet in most regions

Story
As a user of SyntheticSun, I want to have API Gateway Access Logs onboarded so that I can use the enrichment, analysis and anomaly detection of the solution for my APIs

Definition of Done

• ES Index added to gewalthaufen.py
• Add a HTTP APIGWv2 deployment to Stage 3 CFN
• Add new resources:
  • Lambda function to return values
  • Lambda function to parse logs
  • CloudWatch logs group for Access Logs
  • Lambda permission for APIGW to invoke the function
  • Lambda permissions & Log subscription for Access Logs
• Add ZIP lambda function to Stage-1/Artifacts
• Update both Stage 3 Architecture diagrams & readme steps
• Create helper script to add Access Logs in the correct format
• Integrate with WAF IP Insights

Nice to Have
N/A

Additional Information

• https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-apigatewayv2-api.html#aws-resource-apigatewayv2-api--examples
• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/apigatewayv2.html#ApiGatewayV2.Client.update_stage
• https://bl.ocks.org/magnetikonline/c314952045eee8e8375b82bc7ec68e88

Story
As a user of SyntheticSun I want findings for anomalous traffic or known threats that are allowed by my WAF populated in Security Hub so that I can triage, analyze and respond to them.

Definition of Done

• In-line BIF call added to the WAF Parser
• Dynamic ThreatIntelIndicators, Network and Resource ASFF population
• Updated architecture diagram
• Updated CFN with permissions to call BIF

Nice to Have
Explore expanding scope to ELB and API Gateway

Additional Information
N/A

When running step 4 in stage 1, The following error is generated:

botocore.exceptions.ClientError: An error occurred (InvalidParameter) when calling the CreateVpcEndpoint operation: 'vpc-endpoint-service' is not a valid taggable resource type for this operation.

Thanks,
David

Describe the bug
Hey man! I gave stage1 a shot today and I think I found a process issue that organizations are going to hit. I am unable to gain permissions to run the gewalthaufen.py script because I cannot run it as an AWS profile different than the default. can you add a flag to the script to specify the aws cli profile I want to run the script as?

To Reproduce
Steps to reproduce the behavior:

1. git clone project
2. run code
   cd - python3 gewalthaufen.py \ my-aws-region (us-east-1) \ vpc-id (vpc-123456) \ waf-arn (from CFN e.g. arn:aws:wafv2:us-east-1:12345678:regional/webacl/SyntheticSun-WACL/waf-id-goes-here) \ firehose-arn (from CFN e.g. arn:aws:firehose:us-east-1:12345678:deliverystream/aws-waf-logs-syntheticsun) \ elasticsearch-endpoint (e.g. https://my-domain-elasticsearch.com) \ misp-instance-id (i-123456789012)

Expected behavior
should be able to run with --profile blah flag

Screenshots
n/a

Logs
An error occurred (UnauthorizedOperation) when calling the DescribeRouteTables operation: You are not authorized to perform this operation. Traceback (most recent call last): File "gewalthaufen.py", line 390, in <module> im_helping() File "gewalthaufen.py", line 381, in im_helping endpoint_attachment() File "gewalthaufen.py", line 40, in endpoint_attachment response = ec2.describe_route_tables(Filters=[{'Name': 'vpc-id','Values': [vpcId]}],DryRun=False) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 276, in _api_call return self._make_api_call(operation_name, kwargs) File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 586, in _make_api_call raise error_class(parsed_response, operation_name) botocore.exceptions.ClientError: An error occurred (UnauthorizedOperation) when calling the DescribeRouteTables operation: You are not authorized to perform this operation.

Additional context
n/a

Documentation Notes
This is missing and might break the process if specifying a non-default profile setting.

This is for the following arg: --compatible-runtimes python3.7 python3.8

Story
As a user of SyntheticSun I want to have a URL filtering solution as an add-on so that I can utilized existing threat intel and automation infrastructure to provide protection against outbound communications to known malicious destinations.

Definition of Done

• Squid used for URL filtering, added in Appendix B
• Automation and configuration management scripts created to periodically update lists of malicious domains / URLs
• CloudFormation support for a new reference VPC using the solution
• Auto-scaled + load balanced Proxies
• Onboard Squid logs into SyntheticSun core solution
• Updated documentation

Nice to Have

• Fargate Services are used for Squid
• Multi-VPC egress pattern implemented via TGW

Additional Information

• https://hub.docker.com/r/sameersbn/squid/
• http://www.squid-cache.org/Doc/
• https://aws.amazon.com/blogs/security/how-to-add-dns-filtering-to-your-nat-instance-with-squid/
• https://aws.amazon.com/blogs/security/how-to-set-up-an-outbound-vpc-proxy-with-domain-whitelisting-and-content-filtering/
• https://aws.amazon.com/blogs/networking-and-content-delivery/how-to-use-aws-privatelink-to-secure-and-scale-web-filtering-using-explicit-proxy/

Describe the bug
S3 bucket access policy not created in cloudformation template and applied to MISP instance profile

To Reproduce
Steps to reproduce the behavior:

1. run cloudformation template
2. try and run aws s3 cp s3://<artifact-bucket-name-here>/suricata.yaml .
3. get error fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
4. cry

Expected behavior

1. Create new policy in cloudformation template
2. Attach new policy to MISP instance profile

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": [
"arn:aws:s3:::/",
"arn:aws:s3:::$syntheticsun-artifacts"
]
}
]
}

Jon,

I'm guessing this guy should be public but hitting the error that it doesn't exist when deploying.

Thanks,
David

In the readme-stage1 section, step 9, paragraph 2.

Story
As a maintainer of SyntheticSun, I need to add the GPLv3 lisc header to files that are in a ZIP because that's the right thing to do I suppose

Definition of Done

• GPLv3 header added to zipped artifacts

Nice to Have
N/A

Additional Information
N/A

In step 4 of the stage 1 process. when running the command, I receive an invalid length for parameter message: LoggingConfiguration.ResourceArn, value: 6, valid range: 20-inf. Here is a sanitized version of what is being ran:

python3 gewalthaufen.py
my-aws-region us-east-1
vpc-id vpc-09XXXXXXXXXe
waf-arn arn:aws:wafv2:us-east-1:%%%%%%%%%%:regional/webacl/SyntheticSun-WACL/00e^&7ce-0049-41%^-81c5-XXa1919893aa
firehose-arn arn:aws:firehose:us-east-1:44444444449:deliverystream/aws-waf-logs-syntheticsun
elasticsearch-endpoint https://search-syntheticsun-es-&&&&&&&&&&&-east-1.es.amazonaws.com
misp-instance-id i-059************

I'm running this from an Amazon Linux 2 instance using a virtual environment with Python 3.7.6.

BTW, It looks like GitHub is removing the trailing slash after each command, accept for the last command for the misp-instance argument.

Thanks,
David

Story
As a user of SyntheticSun, I want to create all IAM related resources and link them to other CFN created resources so that I do not have to depend on a python script

Definition of Done
All IAM policy-related resources are now created in the CFN and removed from the python script

related to merged branch

Describe the bug
When running the SyntheticSun_SETUP_CFN.yaml in stage 1, CloudFormation bombs when an invalid Elasticsearch data node is present in the template: t3.xlarge.elasticsearch

To Reproduce
Steps to reproduce the behavior:

1. Upload SyntheticSun_SETUP_CFN.yaml template into CloudFormation console
2. Provide parameters
3. Deploy template
4. See error:
   Invalid instance type: t3.xlarge.elasticsearch (Service: AWSElasticsearch; Status Code: 409; Error Code: InvalidTypeException; Request ID: bcab2b48-XXXXX-4d26-b2a4-XXXXXX)

Expected behavior
Deployment is successful

Screenshots
If applicable, add screenshots to help explain your problem.

Logs
Any stack traces, error reports, CloudTrail logs, etc.

Additional context
Add any other context about the problem here.

Story
As a user of SyntheticSun I want to utilize additional machine learning algorithms to detect anomalies in my CloudTrail logs based on different features so that I can reduce the residual risk of any unauthorized access

Definition of Done

• Updated architecture, maybe another Appendix
• Updated readme and permissions
• Expanded CT Training Grounds in Appendix A to include new Dockerfile, helper scripts and subdirectory
• ML that works?

Nice to Have

• Not being an idiot

Additional Information

• https://docs.aws.amazon.com/sagemaker/latest/dg/randomcutforest.html
• https://docs.aws.amazon.com/sagemaker/latest/dg/k-nearest-neighbors.html

Story
As a user of SyntheticSun, I want to do know what the exact name and ARN of resources created by the cloudformation template so that I can have full accountability of changes to my account as well as have a master list to quickly find resources.

Definition of Done
outputs tab of cloudformation console is populated with each unique resource name and ARN that was created or modified by the cloudformation template being run