isabella232 / amazon-ray Goto Github PK

Ray provides a simple, universal API for building distributed applications. Please see the Latest Ray Docs.

This repository serves as a staging area for ongoing enhancements to Ray focused on improving its integration with AWS and other Amazon technologies. Associated large-scale refactoring, rearchitecture, or experimental changes to Ray will typically be released and vetted here prior to any subsequent contribution back to the Ray Project.

Latest Amazon Ray wheels:

All of the above wheels have passed unit tests. They can be installed via pip install -U [wheel URL].

Latest Ray-optimized EC2 AMIs with Amazon Ray wheels pre-installed:

All of the above AMIs have passed unit tests and EC2 cluster integration tests. To use any of the above AMIs, first ensure that you're launching your Ray EC2 cluster in the same region as the AMI, then specify the AMI ID to use with your cluster's head and worker nodes in your autoscaler config:

provider:
  type: aws
  region: us-east-1
  availability_zone: us-east-1a, us-east-1b, us-east-1c, us-east-1d, us-east-1f

head_node:
  InstanceType: r5n.xlarge
  ImageId: ami-0ec076d1d4cf24740

worker_nodes:
  InstanceType: r5n.2xlarge
  ImageId: ami-0ec076d1d4cf24740

# explicitly remove setup commands to keep the pre-installed version of ray
setup_commands: []

Each AMI comes with the Unified CloudWatch Agent pre-installed and configured. The agent will automatically collect additional system level metrics for each EC2 instance in your Ray cluster and write Ray system logs out to the CloudWatch log groups ray_logs_out and ray_logs_err.

Any logs written to /tmp/ray/user/var/output/logs/*.info.log will be automatically added to a corresponding ray_user_logs_info CloudWatch log group, while logs written to /tmp/ray/user/var/output/logs/*.debug.log will be added to a corresponding ray_user_logs_debug log group.

For log and metric publication to work you first need to ensure that the IAM role associated with your Ray cluster EC2 instances is authorized to publish them. This can be done by attaching the following policy to this IAM role, replacing {AWS_ACCOUNT_ID} with your 12-digit AWS account ID:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "cloudwatch:PutMetricData",
      "Resource": "*",
      "Condition": {
        "StringEquals": {
          "cloudwatch:namespace": "CWAgent"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:DescribeLogGroups"
      ],
      "Resource": [
        "arn:aws:logs:us-east-1:{AWS_ACCOUNT_ID}:log-group:ray_logs_out:*",
        "arn:aws:logs:us-east-1:{AWS_ACCOUNT_ID}:log-group:ray_logs_err:*",
        "arn:aws:logs:us-east-1:{AWS_ACCOUNT_ID}:log-group:ray_user_logs_debug:*",
        "arn:aws:logs:us-east-1:{AWS_ACCOUNT_ID}:log-group:ray_user_logs_info:*"
      ]
    }
  ]
}

A tail can be acquired on all logs written to a CloudWatch log group by ensuring that you have the AWS CLI V2+ installed and then running:

aws logs tail $log_group_name --follow

To disable the Unified CloudWatch Agent at cluster launch time, add the following setup command to your autoscaler config:

setup_commands:
  - sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -m ec2 -a stop

The AMIs below ship with the [Unified CloudWatch Agent] (https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/UseCloudWatchUnifiedAgent.html) pre-installed. They should be used with any Ray cluster declaring an agent configuration under the cloudwatch section of its AWS [cluster config file] (https://docs.ray.io/en/latest/cluster/config.html).

See CONTRIBUTING for more information about reporting potential security issues or vulnerabilities.

This project is licensed under the Apache-2.0 License.