mozillasecurity / laniakea Goto Github PK

Libs supporting DigitalOcean:
https://developers.digitalocean.com/libraries/
http://doto.readthedocs.io/en/latest/

$ python -m laniakea ec2 -create-spot -userdata default.sh -tags Name=test -max-spot-price 0.6 -region us-east-1 -zone us-east-1e -profile default
[Laniakea] 2018-08-24 10:25:49 INFO: Loading Laniakea configuration from /home/user/.config/laniakea/laniakea.json
[Laniakea] 2018-08-24 10:25:49 INFO: Reading user data script content from /home/user/src/m/laniakea/laniakea/userdata/ec2/default.sh
[Laniakea] 2018-08-24 10:25:49 INFO: Processing "import" of common.sh
[Laniakea] 2018-08-24 10:25:49 INFO: Using image definition "default" from /home/user/.config/laniakea/amazon.json
[Laniakea] 2018-08-24 10:25:49 INFO: Using Boto configuration profile "default"
[Laniakea] 2018-08-24 10:25:50 INFO: Waiting on fulfillment of requested spot instances.
$

The state of the spot request is 'failed' and status is 'bad-parameters' due to invalid availability zone (EC2 issue).

I would expect to see this logged as an error.

We should support boto3 and remove boto2. In particular, the spot request API was simplified significantly (same API for spot/demand instances), but we need to use boto3 to take advantage of that..

https://github.com/MozillaSecurity/laniakea/blob/master/laniakea/__init__.py#L16

A proper approach could be: @macro_name|default_value@
If |macro_name| was not provided with the -userdata-macros flag then fall back to |default_value|.

The Laniakea.images dictionary assumes one block device per machine and it is named /dev/sda1. This is not a universal name, and prevents laniakea from launching AMIs with other requirements. For example the official AMIs for CoreOS use /dev/xvda1.

Currently, launching a spot request for CoreOS-stable AMI with laniakea gives the following:

State: failed: Either snapshot ID or volume size must be provided for: /dev/sda1
Status: bad-parameters: Your Spot request failed due to bad parameters. 

GCE labels are closer to EC2 tags than GCE tags are.

https://pypi.python.org/pypi/packet-python
http://libcloud.readthedocs.io/en/latest/compute/drivers/packet.html

Story: Outsource common routines in UserData scripts to importable UserData scripts.
Suggestion: @import@(filepath)

Operations like create() and terminate() block execution until completed. There should be a way of polling for state changes as we do with EC2. Not sure if there is a precedent for this in libcloud or if we would have to abandon libcloud.

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please see Mozilla-GitHub-Standards or email [email protected].

(Message COC001)

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

On its way.

The following line throws if timeout is not specified:
https://github.com/MozillaSecurity/laniakea/blob/59536ad/core/manager.py#L158

Traceback (most recent call last):
  File "/Users/skywalker/trees/laniakea/laniakea.py", line 299, in <module>
    sys.exit(LaniakeaCommandLine().main())
  File "/Users/skywalker/trees/laniakea/laniakea.py", line 223, in main
    args.ebs_volume_type, args.ebs_volume_delete_on_termination)
  File "/Users/skywalker/trees/laniakea/core/manager.py", line 158, in create_spot
    elif time_exceeded:
UnboundLocalError: local variable 'time_exceeded' referenced before assignment

Laniakea is used in projects like FuzzManager which would benefit from being able to specify laniakea directly in the requirements.txt

Add a way to quickly add and kill instances.
Suggestion:
-scale-up N
-scale-down N
The scale-down function should probably kill instances which are the oldest ones.

Briefly discussed in #30

A possible idea would be:

Add a new directory called requirements and extract module dependencies out of the requirements.txt and put those into their own requirement files: i.e azure.txt, amazon.txt. A user then would run:

pip install -r requirements.txt -r azure.txt -r amazon.txt

Where requirements.txt would consist of the base requirements for Laniakea. Each module would then call sys.exit if the module is not available and point with an error message to the required requirements file.

This line located at

causes the returned list to be huge - it times out often, but when it succeeds, it gives a huge result.

We seem to be retrieving a list that is too big - some folks have had success switching to Python 3 - but Python 3 only worked once for me.

I worked around the issue by removing aws-marketplace to simplify the list retrieved, but it seems a solution might be to use paginator or some sort of filter as discussed.