cloudyr / aws.ec2 Goto Github PK

I've created two short functions that would be useful additions:

1. Get the aws instance id. Only run on an actual aws instance:

   # Only run on aws instance
   # Get the instance id
   get_instance_id = function(verbose = FALSE) {
    system(
      "curl http://instance-data/latest/meta-data/instance-id",
      intern = TRUE,
      ignore.stderr = !verbose
    )
   }
   

2. Check if we are running on an instance.

   is_instance = function() {
    hostname <- system("hostname -d", intern = TRUE)
    # Grep could be made better by checking for eu-west-1, etc.
    length(grep("*.compute.internal$", hostname)) > 1
   }
   

Example use case:

run = function() {
   if(is_instance()) {
       instance_id = get_instance_id()
       on.exit(terminate_instances(instance_id))
   }
   simulate_stuff()
}

As an aws.ec2 user, I'd like to programmatically terminate instances, so that I can use EC2 efficiently (for example, to run a computation and then die).

Working off the github version of clodyr ec2 API.

Based on the documentation the allocate_ip(domain="vpc") should produce an allocationId I can use to deallocate but it does not:

> ip.1 = aws.ec2::allocate_ip(domain="vpc")
> ip.1
> ip.1
$publicIp
[1] "11.22.33.444"

$domain
[1] "vpc"

attr(,"class")
[1] "ec2_ip"
attr(,"requestId")           #>>>>>>>>>>>>>>> NO ALLOCATION ID TO USE FOR RELEASE 
[1] "xxxxxxx-yyyy-aaaa-bbbb-cccccccccc"

The issue is easy to see from the code. Where it looks at publicIp field in the return code to determine whether the allocation is for vpc or standard. This does not seem correct based on the documentation. It seems you should look at the domain argument instead. I fixed the issue for myself, but I am not sure this fix is correct.

Also while debugging the code I was not able to allocate an IP for standard domain at all. Is it possible that my AWS account has some sort of restriction where I can only allocate EIPs for VPC domain but not for standard that makes the API behave this way? Here is the debug log:

debug(aws.ec2::allocate_ip)

> ip.1 = aws.ec2::allocate_ip("standard")
debugging in: aws.ec2::allocate_ip("standard")

Browse[2]> n
debug: query <- list(Action = "AllocateAddress")
Browse[2]>
debug: if (!missing(domain)) {
    domain <- match.arg(domain)
    query$Domain <- domain
}
Browse[2]>
debug: domain <- match.arg(domain)
Browse[2]>
debug: query$Domain <- domain
Browse[2]>
debug: r <- ec2HTTP(query = query, ...)
Browse[2]> n
Browse[2]> print(r)
$requestId
$requestId[[1]]
[1] "aaaaaa-xxxx-xxxx-xxxx-ccccccccccc"


$publicIp
$publicIp[[1]]
[1] "xx.xxx.xx.xxx"


$domain
$domain[[1]]
[1] "vpc"     # >>>>>>>>> STILL VPC EVEN THOUGH I REQUESTED STANDARD


$allocationId
$allocationId[[1]]
[1] "eipalloc-xxxxxx"


attr(,"xmlns")
[1] "http://ec2.amazonaws.com/doc/2015-10-01/"

I have a question about how to actually use the aws.ec2 package. I've reviewed the existing documentation as well as the code, but I still don't get how to do what I want--which is to get the private IP of an instance using its ID.

Here's what I have so far:

## load package
library("aws.ec2")

## code goes here
aws.signature::use_credentials()

aws.ec2::describe_instances("i-my-instance-id")

## session info 
sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-apple-darwin15.6.0 (64-bit)
Running under: macOS High Sierra 10.13.5

Matrix products: default
BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] aws.ec2_0.1.11 OrientR_0.2.8 

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.17        lubridate_1.7.4     tidyr_0.8.1         digest_0.6.12       mime_0.5            aws.signature_0.4.3
 [7] bitops_1.0-6        R6_2.2.2            jsonlite_1.5        magrittr_1.5        httr_1.3.1          pillar_1.2.3       
[13] rlang_0.2.1         stringi_1.2.2       curl_2.8.1          xml2_1.2.0          tools_3.4.1         stringr_1.3.1      
[19] glue_1.2.0          purrr_0.2.5         RCurl_1.95-4.10     compiler_3.4.1      base64enc_0.1-3     tibble_1.4.2       

The problem is that I get an empty list back, even though running the following using AWS CLI returns the private IP I'm looking for:

aws ec2 describe-instances --instance-id i-my-instance-id --query 'Reservations[].Instances[].PrivateIpAddress'

So the question is: how do I get aws.ec2::describe_instances("i-my-instance-id") to return my private IP?

Hi,

This is more of a feature request question but do you plan to support IAM profiles and block device mappings with run_instances()?

The function names are slightly misleading, since the input is an instance:

start_instances(instance, info, ...)
stop_instances(instance, force, ...)
terminate_instances(instance, ...)
reboot_instances(instance, ...)

Perhaps rename to

start_instance
stop_instance
#etc

https://cran.r-project.org/web/packages/remoter/vignettes/remoter.html

Is it possible to create a "spot" instances from run_instances ?

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-launch-templates.html

https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_CreateLaunchTemplate.html

Great work on this project, I have been looking for something like this.

The example shows how to start up an instance with Rstudio server, which I believe would be controlled from a web browser?

With this package, is it possible to use R to start up R on the AWS instance, and continue to send it commands? I believe this would have to be done through SSH.

I'm trying to follow your tutorial and getting the following error:

library(aws.ec2)
image <- "ami-b1b0c3c2"
describe_images(image)
No encoding supplied: defaulting to UTF-8.
Error in ec2HTTP(query = query, ...) : 

In case it helps, I configured my .Renviron with the following variables:

AWS_ACCESS_KEY_ID="My_Secret_Key_ID"
AWS_SECRET_ACCESS_KEY="My_Secret_Key"
AWS_DEFAULT_REGION="us-east-1"

What am I doing wrong?

Thanks,

Ignacio

PS: Please let me know if there is a better place to ask for help.

xml2 v1.2.0 changed the behavior of as_list(read_xml(tmp)) to add an additional layer of nesting to the list output. See https://github.com/r-lib/xml2/releases/tag/v1.2.0

Many functions pull items from ec2HTTP() using $ (ec. r$imagesSet in describe_images()) and all of those now return empty lists.

This may be the cause of #33, since error parsing in ec2HTTP() also uses output from as_list(read_xml(tmp)) and expects it to have an $Errors element.

http://stanke.co/r-on-aws-cloud/

Is it possible to ssh into the instance created in the README file?

i <- run_instances(image = image, 
                   type = "t2.micro", # <- you might want to change this
                   subnet = s[[1]], 
                   sgroup = g[[1]])

In the code for describe_sgroup(filter=list(description ='BLAH') the filter function doesn't work.

Because in the line , the function .makelist sets the names as ".Key" , when it should be .Name

 names(tmp) <- c(paste0(type, ".", 1:length(list), ".Key"),

This should be written as names(tmp) <- c(paste0(type, ".", 1:length(list), ".Name"),

This doesn't work according to the aws api, as used here in describe_sgroups.

query <- c(query, .makelist(filter, type = "Filter"))

http://docs.aws.amazon.com/ssm/latest/APIReference/aws-ssm-document.html

The session_token in ec2HTTP is not being read from locate_credentials or environment like it should, and default arg value is NULL which is causing temporary credentials to not work.

More details at cloudyr/aws.signature#19 (comment)

It does work in s3HTTP, since it is using session_token = Sys.getenv("AWS_SESSION_TOKEN") in the argument list.

Easy to reproduce:aws.ec2::describe_instances(c()) returns:

Error in names(instance) <- paste0("InstanceId.", 1:length(instance)) : 
  'names' attribute [2] must be the same length as the vector [0]

It is sort of an edge case, but may be common enough to handle instead of throwing an error.

I'd like to do a library that lets you call VMs easily from your local R session (working title, computeR) that could use this library and the Google compute engine library I'm going to work on next.

The API should be R-friendly and make it easy to launch servers such as RStudio, OpenCPU and Shiny, and also let you schedule scripts, run code on big-VMs then send the result to your local console, etc.

@MarkEdmondson1234 I'll take this one, too.

I found a multitude of problems when trying to use the package. Essentially none of the code examples worked for me. It's unclear whether this is because of a change in the AWS ec2 API.

I will submit a PR soon that provides fixes for all the code I tried.

If would be handy if create_keypair had an optional path argument for creating a correspond .pem file.

I'm thinking something like

create_keypair = function(keypair, path, ...) {
if(!is.null(path)) create file
...

I'm happy to make a PR if you think it's something useful.

As a developer of this package, I'd like to use Github Actions for continuous integration, so that we can easily determine whether everything is working.

Let's use styler with the (default) tidyverse style, and rebuild roxygen docs while we're at it.

I found this 2014 tutorial about running R jobs on an AWS EC2 cluster. Is it possible to use aws.ec2 to make this process easier? If so, is there an example somewhere?

As an aws.ec2 user, I'd like to create new EC2 instances, so that I can use EC2 efficiently.

Are you guys thinking of creating as.cluster() methods for this package like you did with googleComputeEngineR? It would be pretty sweet to fire up an EC2 instance (cluster?) and be able to send code up to it with future. In full transparency I haven't actually used your package yet but have been thinking a bit about how something like this might work, especially with furrr.

I'm unsure of the technicalities involved in the difference between firing up 1 powerful EC2 instance with multiple cores and sending code up to run in parallel there VS firing up multiple EC2 instances and having the code run in parallel across them, but being able to do one or the other could be useful.

The future docs actually give one example that you might could build from.

## Launching worker on Amazon AWS EC2 running one of the
## Amazon Machine Images (AMI) provided by RStudio
## (http://www.louisaslett.com/RStudio_AMI/)

public_ip <- "1.2.3.4"
ssh_private_key_file <- "~/.ssh/my-private-aws-key.pem"

cl <- makeClusterPSOCK(

## Public IP number of EC2 instance
public_ip,

## User name (always 'ubuntu')
user = "ubuntu",

## Use private SSH key registered with AWS
rshopts = c(
"-o", "StrictHostKeyChecking=no",
"-o", "IdentitiesOnly=yes",
"-i", ssh_private_key_file
),

## Set up .libPaths() for the 'ubuntu' user and
## install future package
rscript_args = c(
"-e", shQuote("local({
p <- Sys.getenv('R_LIBS_USER')
dir.create(p, recursive = TRUE, showWarnings = FALSE)
.libPaths(p)
})"),
"-e", shQuote("install.packages('future')")
),

dryrun = TRUE
)

Let me know which items are not ready yet so that I can contribute and accelerate development of this package. Thanks.

I'm putting together a checklist of what needs to be done before I consider this ready to return to CRAN:

• Update DESCRIPTION to add maintainer & clean up
• Verify ec2HTTP.R
• Update the behavior or dry_run in ec2HTTP.
• Make sure ec2HTTP behaves as desired. What do we want to do with the "version" parameter?
  I added a note that "version" isn't validated (for future-proofing), and I'll leave it at that. The only people calling ec2HTTP directly should be people who want to do something very specific.
• Consider moving the bit to make the list names syntactic into ec2HTTP. Will it be useful for all/most actions? Leaving it in the one function that needs it until I find a second case.
• describe_instances
• start_instances
• stop_instances
• Either remove tests/old_functions or find a permanent home for it.
• Update README.rmd
• devtools::release checks

Is it possible to specify a role with run_instances()? If found this documentation for the CLI but I don't see a role_arn argument in the run_instances function. I tried the following:

vm <- aws.ec2::run_instances(image = "ami-c3a2e8a2",
                             type = "t3.medium",
                             sgroup = "sg-cd0b5faa",
                             query_extra = list(role_arn = 'arn:aws-us-gov:iam::570108756302:user/imartinez_s3'))

Error: UnknownParameter : 
 - The parameter role_arn is not recognized

Thanks for the help!

Hi,

I saw your tweet @cloudyRproject, and I wanted to express my interest in maintaining this package (perhaps along with the aws.s3 package) in the future.

Thanks,
Alyssa

Tutorials to motivate vignettes:

http://www.slideshare.net/ianmcook/running-r-on-the-amazon-cloud-2013-0620
http://www.slideshare.net/talgalili/rstudio-in-aws-16-9
https://blogs.aws.amazon.com/bigdata/post/Tx3IJSB6BMHWZE5/Running-R-on-AWS
https://wbnicholson.wordpress.com/category/parallel-computing/
http://www.oliverelliott.org/article/computing/tut_bio_aws/
http://www.stat.yale.edu/~jay/EC2/CreateFromScratch.html

Shiny on EC2
https://github.com/chrisrzhou/RShiny-EC2Bootstrap
http://tylerhunt.co/2014/03/amazon-web-services-rstudio/

H20
https://github.com/navdeep-G/h2o-ec2/blob/master/h2o-cluster-launch-instances.py

describe_images source has avialableto instead of availableto (transposed "a" and "i") in several places within an if statement that runs if availableto argument is specified.

In the README, you have

# Launch the instance using appropriate settings
i <- run_instances(image = image, 
                   type = "t2.micro", # <- you might want to change this
                   subnet = s[[1]], 
                   sgroup = g[[1]])

# RStudio Server will be available at the "publicIp" address returned in `i`
# Note: the default security settings prohibit outbound traffic

However, i does not contain an element publicIp. I can manually log onto aws and view the public ip address .

It seems that the above API requires an AllocationIP based on the DisassociateAddress API doc request instead of allocation id. See this tentative PR for the changes I have made since to correct this.

The function describe_keypairs() returns a list of key pairs. It would be useful if the element name matched the keypair name.

Would you like a PR for this functionality?

Hi I am not sure if I am using the filter correctly but its not working for me.
describe_instances(filter=list("instance-state-name"="running"))
I first tried the above, but that gives me the error:

No encoding supplied: defaulting to UTF-8.
Error in ec2HTTP(query = query, ...) :
  UnknownParameter: The parameter Key is not recognized

Even goin by the Filter spec
and tried the following:

describe_instances(filter=c("instance-state-name"=c("running"))

What am I doing wrong?