Forecast is a big data environment for understanding security anomalies using AWS services and Open Source projects. Forecast helps DevSecOps team to operate a single framework for Red and Blue Team activities to support faster feedback and security remediation. It can be used to ingest data from a reconnaissance library, logs and event feeds to support Continuous Delivery of software projects, security monitoring and incident response. And it is intended to utilize a catalog of rules for forecasting security issues as they progress through a Continuous Delivery pipeline.

Forecast is community driven and has a variety of sub-projects that are part of the Forecast Ecosystem. Additionally, Forecast pulls in the best of other projects to help reduce the amount of systems that need to be operated to support DevOps teams.

Intended Benefits:

• Supports both Red and Blue Team functions
• Reduce overhead and complexity of running lots of security tools
• Provides low cost storage options for all data sources used by Forecast
• Provides for retention and replay of data to support zero day evaluations
• Provides simplicity for software integrations using Rest Services and APIs
• Native format support for CSV, JSON and Yaml.
• Supports custom parser development to allow for extension.
• Provides for simple ingestion to reduce overhead of large data processing
• Provides support for sending alerts based on events and trends
• Supports integration with Pager Duty, Email, Slack, and Jira

Forecast is simple to install and can be run in a variety of modes using AWS as supporting infrastructure. You can choose to support your forecast environment using ELK or AWS EMR. We have chosen AWS EMR for Forecast because it gives us big data tools to work with without the overhead.

Data Feeds are a critical element of the Forecast Ecosystem and when organized well are highly useful in producing a scalable security information processing matched against a Continuous Delivery pipeline.

We are in the process of figuring out how to divide and conquer to make Forecast easier to work on to extend and improve it.