widdix / aws-s3-virusscan Goto Github PK

Hi

Should I create my own VPC or it's created automatically ?
Should i also manually add the subnets ?

Thanks

I launch the template with vpc-2azs.yaml and template.yaml (https://s3-eu-west-1.amazonaws.com/widdix-aws-s3-virusscan/template.yaml).
(1) two templates launched successfully.
(2) EC2, SQS and S3 launched successfully.
(3) already config the s3 event for SQS.
(4) upload the image to s3, I can see the message on s3-virusscan-ScanQueue-EB8H5BNWNH2S, but ClamAV can't consume the sqs message.

I launched these two template 10 days ago, which can work successfully. But the same configuration and I launched again this morning, it can't work. could you pls help check?

@michaelwittig We are running into the same issue - initial deployment was months ago with everything working fine. As of the last autoscale, the new ec2 instance is throwing the same error: clamdscan: command not found with error code 127.

Is it possible to fix this on our current deployments/ec2 instances without updating our cloudformation template? Would we be able to do a yum update or config update for clam.d-conf for example?

What is the root cause of the new autoscaled instances not finding clamd?

Thanks in advance for your advice.

Is there a way to whitelist files to prevent them from being scanned? I know I can whitelist files in ClamAV. Is the best way to do this by modifying the template?

IS it possible can you provide me the complete workflow of this like how ec2 is scanning what file ec2 will use , how ec2 server doing tagging part etc

As I understand the new uploaded files are scanned automatically, so the files that haven't completed the scan yet are not available to get them from s3.

Do we know the approximate time that is needed for the virusscan to finish? (I guess it depends on the file size).

I would like to use your solution but was wondering if the files will take too much time until they are available?

ROLLBACK_IN_PROGRESS

I am getting a ROLLBACK_IN_PROGRESS when trying to install the template.yml from https://github.com/widdix/aws-s3-virusscan into my existing infrastructure.

Using:

• vpc-2azs.yaml
• vpc-ssh-bastion.yml
• vpc-nat-gateway.yml
• vpc-endpoints-s3.yml

Seems to be an issue with autoscaling. But I am unable to troubleshoot this. I was wondering if you know what could be the issue?

Hi,
We have recently enabled the FW and routed the ClamAV linux EC2 traffic through it. It needs outside connectivity to update the yum, clam AV OS and clam av db update.
However we also noticed some ips that are related to china, etc in FW log. I couldn't find which process on the CLAM AV ec2 is trying to connect to those IPs. Can you please provide any info if any other process needs outside connectivity from the server?

Hi,

I'm using the S3 VirusScan integration from the AWS Marketplace and have followed the instructions in the README to subscribe to emails to be alerted when a virus was found. It seems to however be sending alerts for both scans of BOTH clean and infected files. Was wondering if there was a way to set it up to only send when the file is infected? Could not seem to find any documentation on this.

Optionally, files that have been scanned but found to have no viruses should also be output to the SNS topic.

This is because many use cases will require to be notified positively that a file has been scanned and is clean, not just of dirty files.

For example consider where user-uploaded content is being accepted by an app and it must wait to virus scan that content before further processing the file. At the moment, we will only know when a file IS infected. The silence on the SNS topic about clean files doesn't give the app a signal it is safe to continue - silence may also indicate the file has not been scanned at all.

I would expect this to be an opt-in feature when setting up the CF template, much like the current "delete infected files from S3" functionality.

If the project maintainers are happy about this functionality I would be happy to submit a PR implementing it.

How do I downgrade from small to micro? I saw this but I do not know how to change a running implementation.

To use t2.micro (free tier) you can add this to the UserData as workaround:

#!/bin/bash -x
...
/usr/bin/fallocate -l 1G /swapfile && /bin/chmod 600 /swapfile && /sbin/mkswap /swapfile && /sbin/swapon /swapfile

More a comment than an issue:

There could be a need to prevent object access before they get scanned. One option is to restrict access only to objects that have a specific tag, e.g. antivirus: scanned-ok

The Ruby script could tag scanned objects if CLEAN_STATUS.

The matching policy would then be something like:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::yourbucket/*"
      ],
      "Condition": {
        "StringEquals": {
          "s3:ExistingObjectTag/antivirus": "scanned-ok"
        }
      }
    }
  ]
}

Hi,

Great project! Works a treat, thanks for sharing.

We're just auditing the setup so we can use it in production.

1. Is there a reason sendmail is running on the instances?
2. Could the security group used for the instances be locked down a little further? Seems it is open to all outbound traffic
3. We assume the AMI has been created from a base standard Amazon Linux AMI with some installs on top to set up the ClamAV worker and update regime. Do you have a script or similar to set this up so we could roll our own AMI?

Thanks again for the v. useful repo!

Chris

can you advise me after deleting a infected file i want to add another text file with same file name saying that file was infacted hence deleted

Hi,
I am using a "t3 small" Linux instance. when I uploaded a video of 1.47 GB to S3 bucket, I have received the following error in /var/log/messages. Is there a limit on the file size which clamav can scan up to?
If I need to scan files up to 5 GB, should I change my EC2 instance to t3 large or so ?

clamd[2463]: /tmp/586558b2-ee38-4c20-867b-d7556abdc03c: Can't allocate memory ERROR

Thanks,
Subhash.

Hi,

I deployed the solution earlier today. When I upload an EICAR test file, I can see the SQS message in the ScanQueue queue. However, it remains "in flight".

I checked /var/log/messages and I found:

Jan 16 15:02:52 ip-10-100-3-65 s3-virusscan[3065]: message failed: #<RuntimeError: s3://bucketnamehere/file3 could not be scanned, clamdscan exit status was 127, retry>

Any ideas?

Hi!

My solution already has a lambda function that is triggered by s3 "object created" event and performs some business logic.

What is the best option to utilize the "aws-s3-virusscan" solution in this scenario since i cannot directly subscribe an SQS queue to the s3 "object created" event?

Will i be able to use smth like the "fanout" pattern as described in https://aws.amazon.com/blogs/compute/fanout-s3-event-notifications-to-multiple-endpoints/ and subscribe both my lambda function and the SQS queue for the av scanner to an SNS topic?

Update 09 September 2019:
I have conducted an experiment and the "fanout" integration scheme worked just fine.
I believe, my question really comes down to the following:

How do i trigger the scan on-demand?

As far as i can see the s3 object created event message is quite verbose:

{
	"Records": [{
		"eventVersion": "2.1",
		"eventSource": "aws:s3",
		"awsRegion": "eu-west-1",
		"eventTime": "2019-09-11T15:31:18.840Z",
		"eventName": "ObjectCreated:Put",
		"userIdentity": {
			"principalId": "{principal id}"
		},
		"requestParameters": {
			"sourceIPAddress": "{source ip}"
		},
		"responseElements": {
			"x-amz-request-id": "{some id}",
			"x-amz-id-2": "{some id}"
		},
		"s3": {
			"s3SchemaVersion": "1.0",
			"configurationId": "{some id}",
			"bucket": {
				"name": "{bucket name}",
				"ownerIdentity": {
					"principalId": "{principal id}"
				},
				"arn": "arn:aws:s3:::{bucket name}"
			},
			"object": {
				"key": "req.txt",
				"size": 4306,
				"eTag": "495255f67ab57a4df5d63744640e93a4",
				"sequencer": "005D791346C7A9728A"
			}
		}
	}]
}

Should i send all the s3 object created event details in a sqs message in case i would like to trigger scanning on demand?

My experiment shows that this is the minimal amount of properties i should send:

{
	"Records": [{
		"eventVersion": "2.1",
		"eventSource": "aws:s3",
		"awsRegion": "eu-west-1",
		"eventTime": "2019-09-11T15:31:18.840Z",
		"eventName": "ObjectCreated:Put",		
		"s3": {
			"s3SchemaVersion": "1.0",			
			"bucket": {
				"name": "{bucket name}",				
				"arn": "arn:aws:s3:::{bucket name}"
			},
			"object": {
				"key": "req.txt",
				"size": 4306,
				"eTag": "495255f67ab57a4df5d63744640e93a4"			
			}
		}
	}]
}

Could you please advise:

1. What is the recommended way of triggering a scan on demand (from code)?
2. Is it ok if i do it by sending a short message provided above?

AWS Marketplace customers have access to support via email: [email protected]

Hello,

Tagging [clamav-status] is not creating for clean files, I am using below details, it created only for infected test file(s).

1. DeleteInfectedFiles --- > false
2. TagFiles --- > true

Thanks
Ratneswar

Hello ,
I've found this error in /var/log/cfn-init.log log file.

2019-03-15 05:54:21,606 [DEBUG] Checking to see if aws-sdk-s3-1.30.1 is already installed
2019-03-15 05:54:21,748 [INFO] Installing aws-sdk-s3 version 1.30.1 via gem
2019-03-15 05:54:28,616 [ERROR] Gem failed. Output: ERROR:  Error installing aws-sdk-s3:
        http-2 requires Ruby version >= 2.1.0.
Successfully installed jmespath-1.4.0
Successfully installed aws-partitions-1.144.0
Successfully installed aws-eventstream-1.0.2
Successfully installed aws-sigv4-1.1.0

2019-03-15 05:54:28,616 [ERROR] Error encountered during build of config: Failed to install gem: aws-sdk-s3-1.30.1 (return code 1)
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 229, in build
    changes['packages'][manager] = CloudFormationCarpenter._packageTools[manager]().apply(packages, self._auth_config)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/lang_package_tools.py", line 111, in apply
    if self._install_gem(pkg, ver):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/lang_package_tools.py", line 155, in _install_gem
    raise ToolError("Failed to install gem: %s-%s" % (pkg, ver), result.returncode)
ToolError: Failed to install gem: aws-sdk-s3-1.30.1 (return code 1)
2019-03-15 05:54:28,617 [ERROR] -----------------------BUILD FAILED!------------------------
2019-03-15 05:54:28,618 [ERROR] Unhandled exception during build: Failed to install gem: aws-sdk-s3-1.30.1 (return code 1)
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 129, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 530, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 542, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/construction.py", line 229, in build
    changes['packages'][manager] = CloudFormationCarpenter._packageTools[manager]().apply(packages, self._auth_config)
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/lang_package_tools.py", line 111, in apply
    if self._install_gem(pkg, ver):
  File "/usr/lib/python2.7/site-packages/cfnbootstrap/lang_package_tools.py", line 155, in _install_gem
    raise ToolError("Failed to install gem: %s-%s" % (pkg, ver), result.returncode)
ToolError: Failed to install gem: aws-sdk-s3-1.30.1 (return code 1)

Could you please help me with that?

Been trying to create new instance of this stack today and it consistanly failed with "unable to allocate memory" when starting the clam daemon on t2.small boxes

Moving to t2.medium solved the issue.

Perhaps the ClamAv db has become even larger now and this no longer works on t2.small either?

AWS Marketplace customers have access to support via email: [email protected]
Hi Michael,

We are planning to use aws-s3-virusscan in our environment, which does not have access to internet.
does your template works without public subnets configuration?

Thanks
Ratneswar

Hi,

I just set this up and it looks like I am having a similar issues to #26 except there are no errors being thrown that indicate that the service is not working.

I uploaded 3 test files which i downloaded from Objective-see at Dec 18, 2018 8:27:30 AM GMT-0800
BackTrack.zip
Adwind.zip
Clapzok.zip

I checked my CloudWatch metrics for my SQS queues, and it looks like there were two messages sent at 8:25 and 8:45, which doesn't really match up with my upload time. The metrics for my SNS topics indicate that no messages or alerts were sent in that time period.

var/log/messages:
se status OK.

08:02:35
Dec 18 16:02:35 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 112500ms.

08:04:27
Dec 18 16:04:27 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 117630ms.

08:06:25
Dec 18 16:06:25 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 123180ms.

08:08:28
Dec 18 16:08:28 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 123880ms.

08:10:32
Dec 18 16:10:32 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 125500ms.

08:11:58
Dec 18 16:11:58 ip-10-0-5-254 clamd[3050]: SelfCheck: Database status OK.

08:12:38
Dec 18 16:12:38 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 123670ms.

08:14:42
Dec 18 16:14:42 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 113160ms.

08:16:35
Dec 18 16:16:35 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 116520ms.

08:18:31
Dec 18 16:18:31 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 125380ms.

08:20:37
Dec 18 16:20:37 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 124380ms.

08:21:58
Dec 18 16:21:58 ip-10-0-5-254 clamd[3050]: SelfCheck: Database status OK.

08:22:41
Dec 18 16:22:41 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 115660ms.

08:23:07
Dec 18 16:23:07 ip-10-0-5-254 dhclient[2100]: DHCPREQUEST on eth0 to 10.0.0.1 port 67 (xid=0xfd9dc05)
Dec 18 16:23:07 ip-10-0-5-254 dhclient[2100]: DHCPREQUEST on eth0 to 10.0.0.1 port 67 (xid=0xfd9dc05)

08:23:07
Dec 18 16:23:07 ip-10-0-5-254 dhclient[2100]: DHCPACK from 10.0.0.1 (xid=0xfd9dc05)

08:23:07
Dec 18 16:23:07 ip-10-0-5-254 dhclient[2100]: bound to 10.0.5.254 -- renewal in 1787 seconds.

08:23:07
Dec 18 16:23:07 ip-10-0-5-254 ec2net: [get_meta] Trying to get http://169.254.169.254/latest/meta-data/network/interfaces/macs/02:22:0b:f4:e3:2e/local-ipv4s

08:23:07
Dec 18 16:23:07 ip-10-0-5-254 ec2net: [rewrite_aliases] Rewriting aliases of eth0

08:24:37
Dec 18 16:24:37 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 108870ms.

08:26:26
Dec 18 16:26:26 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 122480ms.

08:28:29
Dec 18 16:28:29 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 116510ms.

08:30:25
Dec 18 16:30:25 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 115960ms.

08:32:21
Dec 18 16:32:21 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 111920ms.

08:34:13
Dec 18 16:34:13 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 117450ms.

08:36:11
Dec 18 16:36:11 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 113310ms.

08:38:04
Dec 18 16:38:04 ip-10-0-5-254 dhclient[2213]: XMT: Solicit on eth0, interval 117040ms.


var/log/awslogs.log
:25:16,823 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150310000, 'start_position': 7715255L, 'end_position': 7715808L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150310000,
2018-12-18 16:25:16,823 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150310000, 'start_position': 7715255L, 'end_position': 7715808L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150310000, 'start_position': 7715255L, 'end_position': 7715808L}, 'source_id': 'fa61cbbd4f10193883ae90b9d76577fb', 'num_of_events': 1, 'batch_size_in_bytes': 578}

08:25:22
2018-12-18 16:25:22,874 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150316000, 'start_position': 7715808L, 'end_position': 7716360L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150316000,

08:25:28
2018-12-18 16:25:28,923 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150322000, 'start_position': 7716360L, 'end_position': 7716912L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150322000,

08:25:35
2018-12-18 16:25:35,422 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150328000, 'start_position': 7716912L, 'end_position': 7717464L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150328000,

08:25:41
2018-12-18 16:25:41,463 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150335000, 'start_position': 7717464L, 'end_position': 7718016L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150335000,

08:25:47
2018-12-18 16:25:47,504 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150341000, 'start_position': 7718016L, 'end_position': 7718568L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150341000,

08:25:53
2018-12-18 16:25:53,560 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150347000, 'start_position': 7718568L, 'end_position': 7719120L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150347000,

08:25:59
2018-12-18 16:25:59,891 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150353000, 'start_position': 7719120L, 'end_position': 7719672L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150353000,

08:26:05
2018-12-18 16:26:05,930 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150359000, 'start_position': 7719672L, 'end_position': 7720224L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150359000,

08:26:07
2018-12-18 16:26:07,603 - cwlogs.push.publisher - INFO - 2882 - Thread-9 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/cron, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150361000, 'start_position': 115254L, 'end_position': 115352L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150361000, 'start_po

08:26:11
2018-12-18 16:26:11,973 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150365000, 'start_position': 7720224L, 'end_position': 7720776L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150367000,

08:26:18
2018-12-18 16:26:18,035 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150371000, 'start_position': 7721317L, 'end_position': 7721870L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150371000,

08:26:24
2018-12-18 16:26:24,079 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150378000, 'start_position': 7721870L, 'end_position': 7722422L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150378000,

08:26:30
2018-12-18 16:26:30,125 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150384000, 'start_position': 7722422L, 'end_position': 7722974L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150384000,

08:26:32
2018-12-18 16:26:32,567 - cwlogs.push.publisher - INFO - 2882 - Thread-5 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/messages, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150386000, 'start_position': 127817L, 'end_position': 127904L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150386000, 'star

08:26:36
2018-12-18 16:26:36,169 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150390000, 'start_position': 7722974L, 'end_position': 7723526L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150392000,

08:26:42
2018-12-18 16:26:42,231 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150396000, 'start_position': 7724071L, 'end_position': 7724624L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150396000,

08:26:48
2018-12-18 16:26:48,295 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150402000, 'start_position': 7724624L, 'end_position': 7725176L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150402000,

08:26:54
2018-12-18 16:26:54,344 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150408000, 'start_position': 7725176L, 'end_position': 7725728L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150408000,

08:27:00
2018-12-18 16:27:00,179 - cwlogs.push.publisher - INFO - 2882 - Thread-21 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/cfn-hup.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150413000, 'start_position': 13429L, 'end_position': 13489L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150413000, 'st

08:27:00
2018-12-18 16:27:00,382 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150414000, 'start_position': 7725728L, 'end_position': 7726280L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150414000,

08:27:06
2018-12-18 16:27:06,439 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150420000, 'start_position': 7726280L, 'end_position': 7726824L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150420000,

08:27:07
2018-12-18 16:27:07,660 - cwlogs.push.publisher - INFO - 2882 - Thread-9 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/cron, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150421000, 'start_position': 115352L, 'end_position': 115450L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150421000, 'start_po

08:27:12
2018-12-18 16:27:12,473 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150426000, 'start_position': 7727376L, 'end_position': 7727929L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150427000,

08:27:18
2018-12-18 16:27:18,519 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150432000, 'start_position': 7728470L, 'end_position': 7729023L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150432000,

08:27:24
2018-12-18 16:27:24,566 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150438000, 'start_position': 7729023L, 'end_position': 7729575L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150438000,

08:27:30
2018-12-18 16:27:30,608 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150444000, 'start_position': 7729575L, 'end_position': 7730127L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150444000,

08:27:36
2018-12-18 16:27:36,644 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150450000, 'start_position': 7730127L, 'end_position': 7730679L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150450000,

08:27:42
2018-12-18 16:27:42,685 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150456000, 'start_position': 7730679L, 'end_position': 7731231L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150456000,

08:27:44
2018-12-18 16:27:44,279 - cwlogs.push.publisher - INFO - 2882 - Thread-7 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/secure, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150458000, 'start_position': 1049450L, 'end_position': 1049560L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150458000, 'star

08:27:48
2018-12-18 16:27:48,724 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150462000, 'start_position': 7731231L, 'end_position': 7731783L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150464000,

08:27:54
2018-12-18 16:27:54,764 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150468000, 'start_position': 7732330L, 'end_position': 7732883L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150468000,

08:28:00
2018-12-18 16:28:00,820 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150474000, 'start_position': 7732883L, 'end_position': 7733435L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150474000,

08:28:06
2018-12-18 16:28:06,885 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150480000, 'start_position': 7733435L, 'end_position': 7733987L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150480000,

08:28:07
2018-12-18 16:28:07,710 - cwlogs.push.publisher - INFO - 2882 - Thread-9 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/cron, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150481000, 'start_position': 115450L, 'end_position': 115548L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150481000, 'start_po

08:28:12
2018-12-18 16:28:12,931 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150486000, 'start_position': 7733987L, 'end_position': 7734539L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150487000,

08:28:18
2018-12-18 16:28:18,992 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150492000, 'start_position': 7735080L, 'end_position': 7735633L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150492000,

08:28:25
2018-12-18 16:28:25,032 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150498000, 'start_position': 7735633L, 'end_position': 7736185L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150498000,

08:28:31
2018-12-18 16:28:31,072 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150505000, 'start_position': 7736185L, 'end_position': 7736737L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150505000,

08:28:35
2018-12-18 16:28:35,640 - cwlogs.push.publisher - INFO - 2882 - Thread-5 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/messages, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150509000, 'start_position': 127904L, 'end_position': 127991L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150509000, 'star

08:28:37
2018-12-18 16:28:37,110 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150511000, 'start_position': 7736737L, 'end_position': 7737289L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150511000,

08:28:42
2018-12-18 16:28:42,150 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150515000, 'start_position': 7737289L, 'end_position': 7737834L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150517000,

08:28:48
2018-12-18 16:28:48,216 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150522000, 'start_position': 7738386L, 'end_position': 7738939L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150522000,

08:28:54
2018-12-18 16:28:54,279 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150528000, 'start_position': 7738939L, 'end_position': 7739491L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150528000,

08:29:00
2018-12-18 16:29:00,326 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150534000, 'start_position': 7739491L, 'end_position': 7740043L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150534000,

08:29:06
2018-12-18 16:29:06,759 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150540000, 'start_position': 7740043L, 'end_position': 7740595L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150540000,

08:29:07
2018-12-18 16:29:07,761 - cwlogs.push.publisher - INFO - 2882 - Thread-9 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/cron, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150541000, 'start_position': 115548L, 'end_position': 115646L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150541000, 'start_po

08:29:12
2018-12-18 16:29:12,806 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150546000, 'start_position': 7740595L, 'end_position': 7741147L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150547000,

08:29:18
2018-12-18 16:29:18,853 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150552000, 'start_position': 7741688L, 'end_position': 7742241L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150552000,

08:29:24
2018-12-18 16:29:24,907 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150558000, 'start_position': 7742241L, 'end_position': 7742793L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150558000,

08:29:30
2018-12-18 16:29:30,944 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150564000, 'start_position': 7742793L, 'end_position': 7743345L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150564000,

08:29:37
2018-12-18 16:29:37,009 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150570000, 'start_position': 7743345L, 'end_position': 7743897L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150570000,

08:29:43
2018-12-18 16:29:43,063 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150577000, 'start_position': 7743897L, 'end_position': 7744449L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150577000,

08:29:49
2018-12-18 16:29:49,245 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150583000, 'start_position': 7744449L, 'end_position': 7745001L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150583000,

08:29:55
2018-12-18 16:29:55,286 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150589000, 'start_position': 7745001L, 'end_position': 7745553L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150589000,

08:30:01
2018-12-18 16:30:01,353 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150595000, 'start_position': 7745553L, 'end_position': 7746105L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150595000,

08:30:07
2018-12-18 16:30:07,419 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150601000, 'start_position': 7746105L, 'end_position': 7746657L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150601000,

08:30:07
2018-12-18 16:30:07,820 - cwlogs.push.publisher - INFO - 2882 - Thread-9 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/cron, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150601000, 'start_position': 115646L, 'end_position': 115744L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150601000, 'start_po

08:30:13
2018-12-18 16:30:13,478 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150607000, 'start_position': 7746657L, 'end_position': 7747209L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150607000,

08:30:19
2018-12-18 16:30:19,542 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150613000, 'start_position': 7747750L, 'end_position': 7748303L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150613000,

08:30:25
2018-12-18 16:30:25,588 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150619000, 'start_position': 7748303L, 'end_position': 7748855L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150619000,

08:30:31
2018-12-18 16:30:31,644 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150625000, 'start_position': 7748855L, 'end_position': 7749407L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150625000,

08:30:31
2018-12-18 16:30:31,702 - cwlogs.push.publisher - INFO - 2882 - Thread-5 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/messages, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150625000, 'start_position': 127991L, 'end_position': 128078L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150625000, 'star

08:30:37
2018-12-18 16:30:37,685 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150631000, 'start_position': 7749407L, 'end_position': 7749959L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150631000,

08:30:43
2018-12-18 16:30:43,726 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150637000, 'start_position': 7750504L, 'end_position': 7751057L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150637000,

08:30:49
2018-12-18 16:30:49,800 - cwlogs.push.publisher - INFO - 2882 - Thread-2 - Log group: s3-virusscan-Logs-1156XNOE3R39Z, log stream: i-09870922583600324/var/log/awslogs.log, queue size: 0, Publish batch: {'skipped_events_count': 0, 'first_event': {'timestamp': 1545150643000, 'start_position': 7751057L, 'end_position': 7751609L}, 'fallback_events_count': 0, 'last_event': {'timestamp': 1545150643000,

08:30:55
2018-12-18 16

It does not appear that any files are being scanned

Can we make this solution work with other paid antivirus solutions? If yes, can you please recommend some and explain the process. Thanks in advance!

Hello, Any idea why EICAR file is scanned as malicious with txt extension but not when changing it in png format!
Thks

Hi,

I just set up this project and finally managed to get it working. I uploaded the eicar.com file, and it seems like it's either not getting scanned or it is not being deleted.

I've looked through the CloudWatch logs but cannot seem to find anything in them that the file has been scanned.

In SQS I can see the message come through,

{"Records":[{"eventVersion":"2.0","eventSource":"aws:s3","awsRegion":"us-west-2","eventTime":"2017-12-14T22:08:20.406Z","eventName":"ObjectCreated:Put","userIdentity":{"principalId":"----"},"requestParameters":{"sourceIPAddress":"----"},"responseElements":{"x-amz-request-id":"18F6891473E7350B","x-amz-id-2":"Xkh0HVRCQbSFvwB+RNYkOzDshXqNvgCaS1DqrmgXOvotAoDolagw+Sg30yg0ulZ3tevFiyBtLEA="},"s3":{"s3SchemaVersion":"1.0","configurationId":"AV","bucket":{"name":"---","ownerIdentity":{"principalId":"AWVVKMUPSHO6I"},"arn":"----"},"object":{"key":"eicar.com","size":68,"eTag":"44d88612fea8a8f36de82e1278abb02f","sequencer":"005A32F6545BC81647"}}}]}

We have been running the virus scanner for a few months now, but since some time last month it doesn't appear to be scanning files.
I have cleared the scan queue and verified that when new files are uploaded messages are pushed.
I updated our stack with the latest template link and redeployed.

Looking at the logs I found a few lines which lead me to believe something is going wrong, but I can't figure out exactly what it is or how to fix it.

• "cloud-init: Error occurred during build: Could not enable service amazon-ssm-agent (return code 1)"

• "cloud-init[3505]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]"

• "cloud-init[3505]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)"

Any advice you can give me is greatly appreciated.

Thank you for your hard work and time. I have been coding for a few years now and understand how much time and thought things take.

One thing, is there a way to move infected files to a specific folder?

hi,
do you know if it is possible to enable the s3 virusscan and getting an error response from the:
client.PutObject(request);
when trying to upload content from the dotnet awsdk to the s3 bucket (if virus check is positive) ?

thanks!

Can you help me out of the scripts which is in EC2 server , I mean from where exactly these scripting are coming either from user data or manually what is the process behind in background ?

This is required because of server harding

I'm having trouble launching this template, getting the following error consistently:

The following resource(s) failed to create: [ScanAutoScalingGroup]. . Rollback requested by user.
Received 0 SUCCESS signal(s) out of 1. Unable to satisfy 100% MinSuccessfulInstancesPercent requirement

I have a feeling it might be to do with the VPC settings perhaps? There's not a lot of info in the readme about the requirements of how to set up the VPC.

The VPC doesn't seem to be optional - if I omit these params completely I get a different error when trying to launch the stack complaining about the params being missing.

After working successfully for several months, aws-s3-virusscan stopped processing items from the ScanQueue on 9/3/2018 at around 5:30 UTC.

Hi,
thanks for this template, i appreciate it!
i would like to know what's the point behind "versioned buckets are not supported"?
Best regards,
Johannes

In worker.rb, there is a line
if system("clamdscan #{fileName}")

In the case of an error, clamdscan returns status 2. Since the code doesn't examine the status, the file is treated as infected and (may be) deleted. This is probably the wrong thing and perhaps an error status should be reported on the topic?

Hi,

I following this article to use ClamAV to scan my S3 bucket for viruses.

https://cloudonaut.io/s3-virusscan-aws-security/

I ran the vpc-*azs.yaml CF template to create the VPC, IGW, subnet. Then I created my S3 bucket and edit events and I don't see the SQS 'scanner2-ScanQueue' that is referenced in the article to choose from in the drop-box. I created my own SQS and pointed to that in my s3 buckets events but when I placed the EICAR test virus file I didn't see a SNS email or anything even though I do have a SNS topic tied to the SQS going to my email address. I tested SNS with a test email and it sent to me successfully.

I think the article above for widdix clamav s3 bucket is missing several steps like sqs, lambda, and clamav creatiosn. Is there another cloudformation template(s) I need to run to create my widdix ec2 instance(s) with clamav installed, sqs, and lambda?

Thanks,

Hi!

Could you please assist me in solving the following issue when trying to deploy the CF stack:

During creation of cloud formation stack... Its not displaying detail about ec2 instance on dashboard as well its not working properly...
Without vpc concept I wnt to start it for s3 scaning

Hello,
I'm trying to replicate the installation steps, but in my case the SQS name is not populated and whatever i try to type to create a new one i get invalid ARN message.
I feel like i'm missing a step .
Would appreciate any assistance how to get this step done.

Thank you so much for this. It is useful but we had some security concerns.

The cloud formation installs the s3-virusscan server (CLAMAV workers) with a security group which has SSH port open to 0.0.0.0/0 ? Is this necessary? Can we remove this?

Also, can we put on rules to block the CLAMAV workers from communicating from the outside world - 127.0.0.1/32

What are the security considerations to take and what are the risks?

i may have missed something.

i do not seem to be able to launch the stack.

the issue is 'No export named -SubnetAPublic found. Rollback requested by user.'

can anyone help?

many thanks.

Hi -

Having some issues creating the s3-virusscan stack.

I've followed the instructions you provided as best as possible.

1. Created vpc that you provided (vpc-2azs), which has A/B public subnet.
2. When creating the s3-virusscan stack, in the wizard I set parentvpc as vpc-2azs.

I'm still seeing this error:

Thank you

Hi

Should I manually create the workers and the SQS queue ?
If so, what AMI should I use for the workers ?

Thanks

Hi,

I am keep getting following warning in the log and also latest malware is not detected by scan while same file is flagged as infected on the other machine where I have ClamAV 0.100.1. Please help.

ClamAV update process started at Thu Aug 2 01:02:41 2018
Your ClamAV installation is OUTDATED!
Local version: 0.99.4 Recommended version: 0.100.1

We're using the aws-s3-virusscan tool and have found it to be very helpful. Thank you for putting it together, your work is much appreciated.

We received this alert today and don't know if it's something we should be doing something about or if it can be safely ignored. Where is the right place to look to get a list of alarms and what to do with them. Some of the alarms are well documented enough in the alarm to determine how to proceed. 'Add one instance', 'Remove one instance', etc. But ones like this 'Alarm if dead letter queue has messages' are harder to know what to do with.

If this is something I can help with let me know. I don't just want to bring problems to you without offering any assistance or help.

Thanks
Kevin

You are receiving this email because your Amazon CloudWatch Alarm "s3-virusscan-ScanQueueDLQAlarm-redacted" in the redacted region has entered the ALARM state, because "Threshold Crossed: 1 datapoint [42.0 (04/12/17 02:50:00)] was greater than or equal to the threshold (1.0)." at "Monday 04 December, 2017 02:51:16 UTC".

View this alarm in the AWS Management Console:
https://console.aws.amazon.com/cloudwatch/home?region=redacted

Alarm Details:

• Name: s3-virusscan-ScanQueueDLQAlarm-redacted
• Description: Alarm if dead letter queue has messages
• State Change: INSUFFICIENT_DATA -> ALARM
• Reason for State Change: Threshold Crossed: 1 datapoint [42.0 (04/12/17 02:50:00)] was greater than or equal to the threshold (1.0).
• Timestamp: Monday 04 December, 2017 02:51:16 UTC
• AWS Account: redacted

Threshold:

• The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0 for 60 seconds.

Monitored Metric:

• MetricNamespace: AWS/SQS
• MetricName: ApproximateNumberOfMessagesVisible
• Dimensions: [QueueName = s3-virusscan-ScanQueueDLQ-redacted]
• Period: 60 seconds
• Statistic: Sum
• Unit: not specified

State Change Actions:

• OK:
• ALARM: [arn:aws:sns:redacted:redacted:s3-virusscan-FindingsTopic-redacted]
• INSUFFICIENT_DATA:

Hi,

I have a use case where I need to run multiple things off the same S3 notification. For this I set the notification to go to an SNS topic, and then I subscribe various functions etc to that topic so they can do what they need to do.

When I subscribe the AV SQS queue to the SNS topic, the objects uploaded to S3 don't get scanned. Is it possible to use this AV solution in this manner, and if so, what configuration changes should I make?

Conor

Hi,

Is there a way to tell which bucket the infected file came from? I checked logs but didn't see any info on which bucket it came from.

Hi there,

I've successfully used this setup in the past, worked perfectly.

But now I am unable to create the stack, it fails with a message about
AWS::AutoScaling::AutoScalingGroup ScanAutoScalingGroup Received FAILURE signal with UniqueId i-00aeb0f212a7dd7ef

I disabled the rollback and debugged the EC2 instance, in /var/log/cfn-init.log I found the following error message:

2017-01-13 10:51:12,253 [DEBUG] Using service modifier: /sbin/chkconfig
2017-01-13 10:51:12,253 [DEBUG] Setting service clamd.scan to enabled
2017-01-13 10:51:12,256 [INFO] enabled service clamd.scan
2017-01-13 10:51:12,256 [DEBUG] Restarting clamd.scan due to change detected in dependency
2017-01-13 10:51:12,256 [DEBUG] Using service runner: /sbin/service
2017-01-13 10:51:22,683 [ERROR] Could not restart service clamd.scan; return code was 1
2017-01-13 10:51:22,683 [DEBUG] Service output: Stopping clamd.scan: [FAILED]^M
Starting clamd.scan: [FAILED]^M

2017-01-13 10:51:22,684 [ERROR] Error encountered during build of config: Could not restart clamd.scan
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 517, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 258, in build
    CloudFormationCarpenter._serviceTools[manager]().apply(services, changes)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 161, in apply
    self._restart_service(service)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 185, in _restart_service
    raise ToolError("Could not restart %s" % service)
ToolError: Could not restart clamd.scan
2017-01-13 10:51:22,686 [ERROR] -----------------------BUILD FAILED!------------------------
2017-01-13 10:51:22,687 [ERROR] Unhandled exception during build: Could not restart clamd.scan
Traceback (most recent call last):
  File "/opt/aws/bin/cfn-init", line 171, in <module>
    worklog.build(metadata, configSets)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 118, in build
    Contractor(metadata).build(configSets, self)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 505, in build
    self.run_config(config, worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 517, in run_config
    CloudFormationCarpenter(config, self._auth_config).build(worklog)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/construction.py", line 258, in build
    CloudFormationCarpenter._serviceTools[manager]().apply(services, changes)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 161, in apply
    self._restart_service(service)
  File "/usr/lib/python2.7/dist-packages/cfnbootstrap/service_tools.py", line 185, in _restart_service
    raise ToolError("Could not restart %s" % service)
ToolError: Could not restart clamd.scan
2017-01-13 10:51:22,979 [DEBUG] CloudFormation client initialized with endpoint https://cloudformation.eu-central-1.amazonaws.com
2017-01-13 10:51:22,979 [DEBUG] Signaling resource ScanAutoScalingGroup in stack s3-virusscan with unique ID i-00aeb0f212a7dd7ef and status FAILURE

The weird thing is that in the machine itself sudo service clamd.scan restart works fine and properly starts the clamd service (and shows it as stopped when first running). But the stack is of course still showing as CREATE_FAILED

Any help with further debugging, assistance or manually finishing the setup is greatly appreciated.

Recently I started use virus scan and found it is very convenient and good tool.

I get notification each time some virus is deleted. However it would be very useful to get also notification when new object are already scanned and no viruses found.

In other words, I want to know when object that was uploaded already scanned and no viruses found.
I understand that such functionality isn't implemented out of the box.
Could you please, advice me how can I implement it ?

Thanks in advance,
Yurii

Cloud formation build from this template fails.
No export named "ParentVPC" found or No export named "###"-SubnetBPublic found

I'm trying to test this project on an EC2 instance. I want to see how much time is it taking to scan files of different sizes. In the worker.rb, I see that it is logging start of scanning and end of scanning. But it is sending it to debug log. i am unable to find out the location of the debug.log
I've found the info log which is located at /var/log/messages