nccgroup / opinel Goto Github PK
View Code? Open in Web Editor NEWPython code shared by Scout2 and AWS-Recipes
License: GNU General Public License v2.0
Python code shared by Scout2 and AWS-Recipes
License: GNU General Public License v2.0
To avoid hitting expired session errors, automatically prompt for an MFA code and re-initiate an STS session when a session token has expired
I get a ValueError: ('Unknown string format:', '2018-12-04 00:07:02 +0000 UTC')
exception in Opinel expiration = dateutil.parser.parse(credentials['Expiration'])
.
The problem is the string having UTC
at the end, if I modify the source to remove this from the string it seems to parse correctly.
I was trying to use Scout2 and was wondering why some of the newer regions weren't listed. It appears that boto-endpoints.json isn't quite up to date, and I had to update the list using http://docs.aws.amazon.com/general/latest/gr/rande.html
Just thought I'd give a heads up.
The file opinel/opinel/utils/credentials.py line 77, 78 and 79 is not looking for the correct variable to load the token code and serial number to be passed to the assumeRole Function.
This result in ScoutSuite failing to scan AWS environments when assume role and MFA are required.
The current code looks like this:
if 'mfa_serial' in credentials and 'mfa_code' in credentials: sts_args['TokenCode'] = credentials['mfa_code'] sts_args['SerialNumber'] = credentials['mfa_serial']
The correct code and solution for this problem(which I tested and found to be working properly)is to replace the code above with the following:
if 'TokenCode' in credentials and 'TokenCode' in credentials: sts_args['TokenCode'] = credentials['TokenCode'] sts_args['SerialNumber'] = credentials['TokenCode']
While using opinel with scout2 or ScoutSuite with crossaccount MFA access, STS assume role failing.
After investigation I found that the root cause is incorrect naming inside assume_role function credentials argument TokenCode and SerialNumber:
- if 'mfa_serial' in credentials and 'mfa_code' in credentials:
- sts_args['TokenCode'] = credentials['mfa_code']
- sts_args['SerialNumber'] = credentials['mfa_serial']
+ if 'SerialNumber' in credentials and 'TokenCode' in credentials:
+ sts_args['TokenCode'] = credentials['TokenCode']
+ sts_args['SerialNumber'] = credentials['SerialNumber']
Walking through the workflow, it seems somewhat troublesome.
Here's what is in what cred file, when. From a clean slate, start by aws configure
.
Now credentials
contains
[default]
aws_access_key_id = AKIA123121123123123123
aws_secret_access_key = 12312312312312312312312312312313123
Now, aws_iam_enable_mfa.py
, go through the prompts. credentials
is unchanged (still has the AKIA.... key) and credentials.no-mfa
has these contents:
[default]
aws_access_key_id = AKIA123121123123123123
aws_secret_access_key = 12312312312312312312312312312313123
aws_mfa_serial = arn:aws:iam::987654321:mfa/UserName
Now, we need to aws_recipes_init_sts_session.py
. After that, credentials.no-mfa
is unchanged, and credentials
contains:
[default]
aws_access_key_id = ASIA8768768768768
aws_secret_access_key = 876876876876876876876876
aws_session_token = 8768768768768.......................876876876876
Now we can use awscli
commands and it will use credentials
. But, consider the case where some organizations only require MFA for certain API actions. The original credentials have been sequestered away in credentials.no-mfa
, and to perform actions that they would ordinarily be allowed to do, they have to either re-MFA or else fetch them from credentials.no-mfa
somehow.
We could just keep all the bookkeeping in the same credentials
file, under a new default-MFA
profile. Under that model, the flow would go like this:
From a clean slate, aws configure
, then aws_iam_enable_mfa.py
. Now credentials
is:
[default]
aws_access_key_id = AKIA123121123123123123
aws_secret_access_key = 12312312312312312312312312312313123
aws_mfa_serial = arn:aws:iam::987654321:mfa/UserName
Then after aws_recipes_init_sts_session.py
:
[default]
aws_access_key_id = AKIA123121123123123123
aws_secret_access_key = 12312312312312312312312312312313123
aws_mfa_serial = arn:aws:iam::987654321:mfa/UserName
[default-MFA]
aws_access_key_id = ASIA8768768768768
aws_secret_access_key = 876876876876876876876876
aws_session_token = 8768768768768.......................876876876876
So then the user can use awscli
's --profile
option to pick whether they want to use their MFA/ASIA...
creds or their non-MFA/AKIA...
creds.
I just verified that aws configure
seems to pass through lines that it doesn't know about. I put a line
some_garbage = osidjfsoidjfoij
and it survived a re-run of aws configure
, as did the aws_mfa_serial
line belonging to the default
profile.
Alternatively, the final state of the credentials
file could be similar but with profiles default-NO-MFA
and default
(which contains the MFA creds). That's probably more likely to be useful, since people who are setting this up are probably going to want to prefer their MFA creds but be able to fall back to non-MFA creds.
If you think this is reasonable, I can take a stab at it.
I have really only been paying attention to the MFA-related things in AWSUtils, so if there are other reasons to use credentials.no-mfa
let me know and I'll see if I can think of a workaround.
@l01cd3v thanks for looking
assume_role function found in opinel/utils/credentials.py does not support DurationSeconds parameter, unlike the init_sts_session parameter. This creates some issues when using Scout2 to scan large AWS accounts, as the default duration for the credentials using assume_role is only 1 hour, which may not be sufficient for scanning of a large AWS account. Suggest that assume_role should, like init_sts_session, take in a parameter for duration with a large default value.
opinel 1.0x seems to have introduced a (roughly) 75-second delay to any anything making calls to the AWS SDK, whether running one of the AWS-recipes
Python scripts or running Scout2
.
For instance, when running aws_recipes_assume_role.py
, there's a 75-second delay before results are returned and the credentials are cached in the ~/.aws/cli/cache
directory.
When running aws_recipes_init_sts_session.py
, there's a 75-second delay before it prompts for the MFA code.
When running aws_recipes_create_ip_ranges.py
it says "Fetching public IP information for the 'xxxxxxx' environment..." and then pauses for 75 seconds before it starts iterating through the various regions and displaying "...in us-east-1: EC2 instances", etc.
After that initial 75-second delay, everything runs as expected. (For instance, there's not an additional 75-second delay for each step of the Scout2 report generation process, only at the very beginning.)
And running the scripts or Scout2 with the --debug
option returns no additional information. The scripts are succeeding; they just take 75 seconds longer than expected.
Currently, If you specify an mfa-code on the Scout2 command line, it will be ignored without any warnings or errors.
In Scout2, __main.py__
calls read_creds
from opinel/utils/credentials.py
In opinel/utils/credentials.py
, the read_creds
function only does an init_sts_session
with the mfa-code if the force_init
bool is set to True, but that never happens if your ~/.aws/credentials
file doesn't contain a SessionToken already. If the session token doesn't exist, it should probably set force_init
to True.
Spin off of nccgroup/Scout2#177
If no expiration information is available and no profile exists to renew the session, read_creds() fails currently
It would be great if the aws_config_dir variable could be overridden by an environmental variable. The awscli library supports reading from the AWS_CONFIG_FILE environmental variable for setups where the config file may not live in ~/.aws/config.
Similar issue applies for the temp file.
Currently opinel looks for the custom configuration variable aws_mfa_serial
in the ~/.aws/credentials file to get the arn of the MFA device.
For instance, my current ~/.aws/credentials
file might look like this:
[development]
aws_access_key_id=foo
aws_secret_access_key=bar
aws_mfa_serial=arn:aws:iam::XXXXXXXXXXXX:mfa/john.doe
However, the AWS/boto3 configuration variables have changed a bit since opinel was first introduced. When assuming roles, boto3 now looks for the arn of the MFA device via the mfa_serial
variable in the `~/.aws/config file.
It would be nice if opinel used the mfa_serial
variable in the ~/.aws/config file, too, both for assuming roles and other standard operations.
So a typical configuration might look something like this…
In ~/.aws/credentials
:
[development]
aws_access_key_id=foo
aws_secret_access_key=bar
In ~/.aws/config
:
[profile development]
mfa_serial=arn:aws:iam::XXXXXXXXXXXX:mfa/john.doe
region=us-east-1
output=json
[profile crossaccount]
source_profile=development
role_arn=arn:aws:iam::XXXXXXXXXXXX:role/whatever
mfa_serial=arn:aws:iam::XXXXXXXXXXXX:mfa/john.doe
region=us-east-1
output=json
Hi,
I'm using Scout2 from an AWS instance to scan both the local AWS account and other AWS accounts through the STS assumeRole API call. When setting the environment variables to access a remote AWS account it is still scanning the local account because of the search order in the read_creds() function in utils.py.
The current search order is: csv_file, instance_metadata and then environment variables. A more logical search order would be: csv_file, environment variables, instance_metadata.
Mike
Many of the scripts that rely on opinel are currently failing unless they are run from the directory where the scripts reside. Otherwise, you get something along the lines of…
Traceback (most recent call last):
File "/Users/xxxxxxx/Code/AWS-recipes/Python/awsrecipes_init_sts_session.py", line 40, in <module>
sys.exit(main())
File "/Users/xxxxxxx/Code/AWS-recipes/Python/awsrecipes_init_sts_session.py", line 31, in main
if not check_requirements(os.path.realpath(__file__)):
File "/usr/local/lib/python2.7/site-packages/opinel/utils/globals.py", line 35, in check_requirements
with open(requirements_file, 'rt') as f:
IOError: [Errno 2] No such file or directory: 'requirements.txt'
I think the reason is that opinel purports to look for the requirements file in two places:
Instead, opinel appears to be looking in:
My Python skills are rudimentary (at best), but I believe that line #34 in opinel/utils/globals.py
, which currently ends with:
...else 'requirements.txt'
...should end with:
...else os.path.join(script_dir, 'requirements.txt')
Hi,
The logic to get STS tokens if you're running in ECS is not quite the same as the logic that boto3 appears to use. In boto3, we don't notice the IAM creds used by ECS tasks to leverage the EC2 role arn, unless the ECS Task has no Role defined ( which we also don't do anymore, but we once did, and found this to be the case maybe 2ish years back ).
The condition by which ECS credentials are attempted to be gathered is dependent on EC2 credential data not being found.
We run with EC2 IAM Profiles/Roles, and ECS Task IAM Profiles/Roles. Scout2 works well for us, until we deployed it as a ECS task.
As a user, I would like for scout2 to be able to assume the role assigned to an ECS Task, while also being able to have a distinct role assigned to the underlying EC2 instance in my ECS Cluster.
Similar in spirit to #25, I would like to be able to pass in role_arn and external_id as environment variables.
The use case is calling scout2 from another process. It is a bit hacky to rewrite configs on the fly (which is how we have been doing it).
I plan to write the code and submit a PR myself if this is ok - are there any steps I need to take first? Or shall I just go ahead and submit the PR?
Reported in nccgroup/Scout2#136
As reported in nccgroup/Scout2#180
As reported in nccgroup/Scout2#145
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.