cloudgraphdev / cli Goto Github PK

CIS has updated GCP CIS to 1.3. Will this be added soon?

I updated the cli to the latest version and now im having this issue

:

Fresh install, failure when running cg launch.

To Reproduce
Steps to reproduce the behavior:

1. npm install -g @cloudgraph/cli
2. cg provider add aws
3. cg init
4. cg launch

cg launch
i Found config for cloudGraph, using...
i Dgraph host set as: http://localhost:8997
√ Docker found
√ No reusable instances found
√ Pulled Dgraph Docker image
× Failed starting Dgraph instance
× Error: Command failed: docker run -d -p 8995:5080 -p 8996:6080 -p 8997:8080 -p 8998:9080 -p 8999:8000 --label cloudgraph-cli-dgraph-standalone -v C:\Users\USER\AppData\Local\cloudgraph/dgraph:/dgraph --name dgraph dgraph/standalone:v21.03.1
docker: invalid reference format: repository name must be lowercase.
See 'docker run --help'.

Error: Dgraph was unable to start: Failed starting stopped Dgraph instance

Expected behavior

No failure.

Environment (please complete the following information):

OS Name: Microsoft Windows 10 Pro
OS Version: 10.0.19043 N/A Build 19043

$ aws --version
aws-cli/2.7.6 Python/3.9.11 Windows/10 exe/AMD64 prompt/off

$ cg -v
@cloudgraph/cli/0.22.0 win32-x64 node-v16.15.1

$ node -v
v16.15.1

Description
Support authentication with AWS SSO

To Reproduce
Steps to reproduce the behavior:

1. AWS credentials configured via aws sso configure
2. Run command CG_DEBUG=5 cg scan aws
3. Getting below error:

✔ accessKeyId: **************
✔ secretAccessKey: ******************************
⠏ SCANNING data for aws    InvalidClientTokenId: The security token included in the request is invalid.
    Code: InvalidClientTokenId

cg-debug.log

No valid credentials found for roleARN: arn:aws:sts::**********:assumed-role/****
AccessDenied: User: arn:aws:sts::**********:assumed-role/****
is not authorized to perform: sts:AssumeRole on resource: arn:aws:sts::**********:assumed-role/****
    at Request.extractError (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/opt/homebrew/Cellar/cg/0.21.4/libexec/node_modules/aws-sdk/lib/sequential_executor.js:116:18)

Expected behavior
Login with SSO credentials succeeds

Environment

❯ cg --version
@cloudgraph/cli/0.21.4 darwin-x64 node-v16.0.0

• Provider versions [email protected]
• Context: Local machine

The CLI breaks immediately when you try to follow the quickstart guide running:

npm install -g @cloudgraph/cli
cg init

you end up with this nice set of error messages:

$ cg init
ℹ Dgraph host set as: http://localhost:8997


                                          ╋╋╋╋╋╋╋╋╋╋╋╋╋┏┓╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋┏┓╋╋╋╋╋╋╋
                                          ╋┏┓┏┓┏┓╋┏━━┓╋┃┃╋╋┏━━┓╋┏━━┓╋┏┓┏┓╋┏━━┓╋╋╋╋┏┛┗┓╋┏━━┓╋
                                          ╋┃┗┛┗┛┃╋┃┃━┫╋┃┃╋╋┃┏━┛╋┃┏┓┃╋┃┗┛┃╋┃┃━┫╋╋╋╋┗┓┏┛╋┃┏┓┃╋
                                          ╋┗┓┏┓┏┛╋┃┃━┫╋┃┗┓╋┃┗━┓╋┃┗┛┃╋┃┃┃┃╋┃┃━┫╋╋╋╋╋┃┗┓╋┃┗┛┃╋
                                          ╋╋┗┛┗┛╋╋┗━━┛╋┗━┛╋┗━━┛╋┗━━┛╋┗┻┻┛╋┗━━┛╋╋╋╋╋┗━┛╋┗━━┛╋
                                          ╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋

                                         ╋╋╋╋╋╋┏┓╋╋╋╋╋╋╋╋╋╋╋╋╋╋┏┓╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋┏┓╋╋╋┏┓╋
                                         ╋┏━━┓╋┃┃╋╋┏━━┓╋┏┓┏┓╋┏━┛┃╋┏━━┓╋┏━┓╋┏━━┓╋┏━━┓╋┃┗━┓╋┃┃╋
                                         ╋┃┏━┛╋┃┃╋╋┃┏┓┃╋┃┃┃┃╋┃┏┓┃╋┃┏┓┃╋┃┏┛╋┃┏┓┃╋┃┏┓┃╋┃┏┓┃╋┃┃╋
                                         ╋┃┗━┓╋┃┗┓╋┃┗┛┃╋┃┗┛┃╋┃┗┛┃╋┃┗┛┃╋┃┃╋╋┃┏┓┃╋┃┗┛┃╋┃┃┃┃╋┗┛╋
                                         ╋┗━━┛╋┗━┛╋┗━━┛╋┗━━┛╋┗━━┛╋┗━┓┃╋┗┛╋╋┗┛┗┛╋┃┏━┛╋┗┛┗┛╋┏┓╋
                                         ╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋╋┗━━┛╋╋╋╋╋╋╋╋╋╋┗┛╋╋╋╋╋╋╋╋┗┛╋


                                                         ╓──────────────────╖
                                                         ║                  ║
                                                         ║   By AutoCloud   ║
                                                         ║                  ║
                                                         ╙──────────────────╜
(node:3130) [MODULE_NOT_FOUND] Error Plugin: @cloudgraph/cli: Cannot find module '@oclif/plugin-help/lib/command'
Require stack:
- /usr/local/lib/node_modules/@cloudgraph/cli/node_modules/@tiagonapoli/oclif-plugin-spaced-commands/lib/hooks/init.js
- /usr/local/lib/node_modules/@cloudgraph/cli/node_modules/@oclif/config/lib/config.js
- /usr/local/lib/node_modules/@cloudgraph/cli/node_modules/@oclif/config/lib/index.js
- /usr/local/lib/node_modules/@cloudgraph/cli/node_modules/@oclif/command/lib/command.js
- /usr/local/lib/node_modules/@cloudgraph/cli/node_modules/@oclif/command/lib/index.js
- /usr/local/lib/node_modules/@cloudgraph/cli/bin/run
module: @oclif/[email protected]
task: runHook init
plugin: @cloudgraph/cli
root: /usr/local/lib/node_modules/@cloudgraph/cli
See more details with DEBUG=*
(Use `node --trace-warnings ...` to show where the warning was created)
? Which cloud provider would you like to use? aws
ℹ Installing aws module version: latest
✖ Manager failed to install provider plugin for aws
✖ Error: provider aws module check FAILED, unable to find plugin
⚠ There was an error installing or requiring a plugin for aws, does one exist?
ℹ For more information on this error, please see https://github.com/cloudgraphdev/cli#common-errors
⚠ There was an issue initializing aws plugin, skipping...

Since the code in the master branch runs with the provided yarn.lock file I suspect that there's a conflict with some of your dependencies having changed in a breaking way.

getting above error while running CG scan, btw we use access and secret as exposed in the environment variable not from the ./.aws/creds or config

As Ankasoft we started to develop VMware Aria Automation 8.X Provider for CloudGraph.

Describe the bug

The CLI help instructions for cg provider currently look like this:

$ cg provider 
Commands to manage provider modules, run $ cg provider for more info.

USAGE
  $ cg provider:COMMAND

COMMANDS
  provider:add      Add new providers
  provider:install  Install providers based on the lock file
  provider:list     List currently installed providers and versions
  provider:remove   Remove currently installed provider
  provider:update   Update currently installed providers
$

One would think that you can e.g. run cg provider:list to get a list of all the providers however running this command will fail:

$ cg provider:list
 ›   Error: command provider:list not found
$

What does actually work is cg provider list:

$ cg provider list
ℹ Found config for cloudGraph, using...
ℹ Dgraph host set as: http://localhost:8997
✔ Provider [email protected] is installed
$

To Reproduce
Steps to reproduce the behavior:

1. Run e.g. cg provider:list

Expected behavior

Either make cg provider:list work or change the help text to show cg provider list instead :-)

Environment (please complete the following information):

• CLI version @cloudgraph/cli/0.15.5 darwin-x64 node-v16.13.1
• Context local machine

I have tried deleting this /root/.local/share/cloudgraph/cg/version-1, as well as running commands such as cg teardown and cg teardown --delete-image in order to fix the issue but the issue still remains. Kindly help me with this.

As Ankasoft we started to develop Oracle Cloud Provider for CloudGraph.

Is there any way to select all the aws accounts i have configures in the aws cli? because i hace 188 accounts, and adding one by one with "cg init" is a pain. Maybe by editing the configuration file and adding the accounts manually?

Thank you for filling out a bug report, we really appreciate any help in improving the CloudGraph CLI and providers!

Describe the bug
In the process of executing CG_DEBUG=5 cg scan aws, there is some error messages such as "unable to store data in Dgraph". These are from "alb, apiGatewayRestAPI, cloudwatchEventRule, kinesisFirehose, s3, securityHubStandardSubscription, vpc" services.
I check "Your data for aws has been saved to /root/.local/share/cloudgraph/cg/version-9" messages, for example in "kinesisFirehose" service there is 2 resources in "cg/version-9/aws_1697656492879", but unable to store this data in Dgraph. Understand? Help me.

To Reproduce
Steps to reproduce the behavior:

1. cg init
2. cg launch
3. CG_DEBUG=5 cg scan aws
4. and error

Please include the cg-debug.log file if applicable

Please solve this problem. Please able to store the data inDgraph.

Describe the bug
I've followed the instructions in README to install aws-cis-1.3.0 policy and execute the scans. The scan was successful but I'm unable to query the cis findings.

The output shows it identified some issues but I'm unable to query them

I can see some data inside dgraph container,

But when I try to query with graphql, the results are nil.

To Reproduce
Steps to reproduce the behavior:

1. Started docker instance for dgraph

   docker run -d -p 8995:5080 -p 8996:6080 -p 8997:8080 -p 8998:9080 -p 8999:8000 --label cloudgraph-cli-dgraph-standalone -v /Users/rewanthtammana/.local/share/cloudgraph/dgraph:/dgraph --name dgraph dgraph/standalone

2. Initialized cg, cg init
3. cg policy add aws-cis-1.3.0
4. cg scan aws
5. The scans are successful but unable to query cisbenchmarks from graphql database

Please include the cg-debug.log file if applicable

Expected behavior
queryawsCISFindings is expected to return the identified results.

Environment (please complete the following information):

• CLI version: @cloudgraph/cli/0.25.1 darwin-x64 node-v16.0.0
• Provider versions: [email protected] & aws-cis-1.3.0 module version: 0.4.0
• Context: Local machine

would like to be able to query for ec2 instances that had launch time prior to X date. with aws cli, i can do something similar to

aws ec2 describe-instances  --query 'sort_by(Reservations[].Instances[], &LaunchTime)[:-1].[InstanceId,PublicIpAddress,LaunchTime]'

to obtain launch time but it seems the cg scan doesn't pick this up yet.

cg init is run on an ec2 instance. The instance does not use any credential files instead uses roles.

Following is the output while running cg niti.
ℹ No lock file found for Cloud Graph, creating one...
Installing aws module version: latest
⠧ Installing aws plugin(node:12370) [DEP0148] DeprecationWarning: Use of deprecated folder mapping "./" in the "exports" field module resolution of the package at /home/ec2-user/.nvm/versions/node/v16.10.0/lib/node_modules/@cloudgraph/cli/node_modules/tslib/package.json.
Update this package.json to use a subpath pattern like "./*".
(Use node --trace-deprecation ... to show where the warning was created)
✔ aws plugin installed successfully!
ℹ aws version locked at: 0.28.2
✖ There was an error writing latest version to the lock file
⚠ Unable to read AWS shared credential file
? Select regions to scan us-east-1, us-east-2
✔ 🎊 AWS configuration successfully completed 🎊
TypeError: Cannot read properties of undefined (reading 'join')

Thank you for filling out a bug report, we really appreciate any help in improving the CloudGraph CLI and providers!

Describe the bug
When creating a new local provider within an organization and try to run the scan command the cli cannot locate the schema graphql file due to the organization subdirectory convention

To Reproduce

1. Create provider (e.g. @acme/cg-provider-oc)
2. Link it
3. Run yarn run:init @acme/oc (succesfully run)
4. Try to run yarn run:scan
   The command fails because it tries to write /Users/user/.local/share/cloudgraph/cg/version-6/@acme/oc_schema.graphql but cannot find the directory. I think the issue is in cli/src/utils/index.ts writeGraphqlSchemaToFile function

export function writeGraphqlSchemaToFile(
  dirPath: string,
  schema: string,
  provider?: string
): void {
  makeDirIfNotExists(dirPath)
  fs.writeFileSync(
    path.join(
      dirPath,
      provider ? `/${provider}_schema.graphql` : '/schema.graphql'
    ),
    schema
  )
}

➜  cli git:(master) yarn run:init @acme/oc
yarn run v1.22.17
$ cross-env NODE_ENV=development ./bin/run init @acme/oc
ℹ Found config for cloudGraph, using...
ℹ Dgraph host set as: http://localhost:8997
⚠ No required cli version found in provider module, assuming compatability
⚠ You are running CloudGraph in devMode. In devMode, CG will assume plugin modules are already installed. use $yarn link {pluginModule} to work with a local copy of a plugin module
✔ provider oc module check complete
ℹ Config for @acme/oc already exists
? Would you like to change @acme/oc's config Yes
? Which oc contexts would you like to scan? context-1
✔ 🎊 oc configuration successfully completed 🎊
ℹ Contexts configured: context-1
ℹ Resources configured: cronJob, deployment, ingress, job, namespace, networkPolicy, node, persistentVolume, persistentVolumeClaim, pod, role, secret, service, serviceAccount, storageClass
ℹ CloudGraph config found...
? Would you like to change CloudGraph config Yes
? Input your dgraph host url, if you are unsure, use the default by pressing ENTER http://localhost:8997
? Enter the maximum number of scanned versions of your cloud data that you would like to store 10
ℹ Note that none of your cloud's information is ever sent to or stored by CloudGraph or third parties
? What tool would you like to query your data with? GraphQL Playground
✔ Your config has been successfully stored at /Users/user/.config/cloudgraph/.cloud-graphrc.json
✔ Your data will be stored at /Users/user/.local/share/cloudgraph/cg
✨  Done in 18.81s.

➜  cli git:(master) yarn run:scan
yarn run v1.22.17
$ cross-env NODE_ENV=development ./bin/run scan
ℹ Found config for cloudGraph, using...
ℹ Dgraph host set as: http://localhost:8997
ℹ Beginning SCAN for @acme/oc
⚠ No required cli version found in provider module, assuming compatability
⚠ You are running CloudGraph in devMode. In devMode, CG will assume plugin modules are already installed. use $yarn link {pluginModule} to work with a local copy of a plugin module
✔ provider oc module check complete
✔ cronJob scan completed
✔ deployment scan completed
✔ ingress scan completed
✔ job scan completed
✔ namespace scan completed
✔ networkPolicy scan completed
✔ node scan completed
✔ persistentVolume scan completed
✔ persistentVolumeClaim scan completed
✔ pod scan completed
✔ role scan completed
✔ secret scan completed
✔ service scan completed
✔ serviceAccount scan completed
✔ storageClass scan completed
✔ Context: context-1 scan completed
✔ @acme/oc data scanned successfully
⠙ updating Schema for @acme/oc    Error: ENOENT: no such file or directory, open '/Users/user/.local/share/cloudgraph/cg/version-7/@acme/oc_schema.graphql'
    Code: ENOENT
error Command failed with exit code 1.

I followed the installation instructions on (from the GitHub README) to the letter, but when I do "sg init gcp" I get this:

$ cg init gcp
\u2139 Found config for cloudGraph, using...
\u2139 Dgraph host set as: http://localhost:8997
\u2139 No lock file found for Cloud Graph, creating one...
\u2139 Installing gcp module version: latest
\u2716 Manager failed to install provider plugin for gcp
\u2716 **Error: provider gcp module check FAILED, unable to find plugin**
\u26a0 There was an error installing or requiring a plugin for gcp, does one exist?
\u2139 For more information on this error, please see https://github.com/cloudgraphdev/cli#common-errors
\u26a0 There was an issue initializing gcp plugin, skipping...
\u2139 CloudGraph config found...

I tried "cg init aws" and get the same error.

Is there a way or is it possible to add a query for untagged resources?

Hi,

Am getting 'unable to store data in Dgraph' for many service resources of AWS in cloud graph. Screenshot attached below:

Could you please look into the same and help me to resolve it?

Any chance instead of providing a list of regions, you could support a wildcard of some sort "*" for scanning of all regions?

Thank you for filling out a bug report, we really appreciate any help in improving the CloudGraph CLI and providers!

Describe the bug
Scan completes printing the time took to scan each service and at the end, displays there are zero resources found for most services except for first few in the list and at the end displays the message in the below screenshot. Command returns 0 resources even though there are resources present in the subscription

To Reproduce
Steps to reproduce the behavior:

1. Run command 'cg init azure' and provide the required details for authentication
2. Run Command 'cg launch'
3. Run Command 'cg scan'

Please include the cg-debug.log file if applicable

Expected behavior
Get the number of resources available for each of the services.

Environment (please complete the following information):

• CLI version - @cloudgraph/cli/0.21.4 win32-x64 node-v14.17.6
• Provider versions - [email protected]
• Context Trying to setup it up on a Local Machine

Hey folks!

Im Yevgeny, Founder @ CloudQuery (which you might be familiar with :) ). We recently have number of security and cost vendor migrated to use our ELT engine under the hood so they can focus solely on the business, analysis and visualization logic on top.

I don't know if this is something relevant at this stage but if yes, we could look at adding DGraph to our destinations, which should fit your use-case with minimal schema changes hopefully.

Best,
Yevgeny

I'm getting a "couldn't rewrite mutation addawsDynamoDbTable because failed to rewrite mutation payload because duplicate XID found" error on a DynamoDB table.

I also get "couldn't rewrite mutation addawsTag because failed to rewrite mutation payload because duplicate XID found" on exactly the same arn.

Looking into the json I could find in the ~/.local/share/cloudgraph/cg/version-X/aws_someid.json, I suspect the issue is linked to the fact that specific dynamodb table has two global indexes which appear in the json file with the arn of the table.

That arn appears as is for 3 things: the table itself and the two global indexes.

Let me know if you need more info on the issue.