Proposed Changes

UPDATE:
   -up, -update                 update cloudlist to the latest version
   -duc, -disable-update-check  disable automatic update check

Reference

• Reference: https://github.com/projectdiscovery/tlsx/blob/8038ac06765d490210aa8bf753f44758a653b39f/cmd/tlsx/main.go#L125

Hey Team!

I'm Yevgeny, Founder @ CloudQuery. I stumbled upon this project and was wandering if it can be interesting for your to consider using CloudQuery under the hood as the ELT step so you can focus on the analysis layer.

If yes, will be happy to chat and exchange notes.

Best,
Yevgeny

Is your feature request related to a problem? Please describe.
In many cases when running cloudlist, I'm either already authenticated locally with GCP via gcloud or I'm running on a resource that has application default credentials configured (i.e. a GCE virtual machine with a service account that has the appropriate permissions attached).

If I run cloudlist with the following provider config:

- provider: gcp 
  id: gcp
  
  # No GCP service account key set
  # gcp_service_account_key: '{xxxxxxxxxxxxx}'

I receive the following output:

  _______             _____     __ 
 / ___/ /__  __ _____/ / (_)__ / /_
/ /__/ / _ \/ // / _  / / (_-</ __/
\___/_/\___/\_,_/\_,_/_/_/___/\__/ 

                projectdiscovery.io

[INF] Current cloudlist version 1.0.8 (latest)
[FTL] Could not create inventory: could not create provider gcp: [:RUNTIME] could not get API Key
exit status 1

Describe the solution you'd like
Instead of requiring gcp_service_account_key to be set, cloudlist should fall back on application default credentials.

When I run the cloudlist program, the Server Guard of my server issues an alarm, and the cloudlist process connects to this IP, 144.126.254.30
What caused this?

When you have a malformed config file (for example a typo in a provider) there isn't any feedback to the user that mentions that. For example I couldn't figure out why my Cloudflare provider wasn't being recognized, and then noticed the typo

Ideally we'd verify the config file and provider config file in a way that gives a meaningful error to the user.

Is your feature request related to a problem? Please describe.
Current cloud provider support does not contain AWS Lightsail specifically

Describe the solution you'd like
AWS Lightsail instance support - https://aws.amazon.com/free/compute/lightsail/

• -silent still prints the banner output
• -ep does not exclude private IPs. It only work with combination of -ip flag.

❯ ./cloudlist -p aws -ep 
10.0.0.113
65.1.245.250
172.31.6.162
172.31.30.33

Is your feature request related to a problem? Please describe.
Source - #15 by @rtcms

Describe the solution you'd like
Flag support to exclude local IP ranges from output, -exclude-IANA

Is your feature request related to a problem? Please describe.

I'd like to see support for Heroku added. Heroku official webpage: https://heroku.com.

Describe the solution you'd like

N/A

Please, let me know if anything else is needed from my side. Feel free to close this issue if you don't think this feature request is useful/needed.

Best regards,
Miguel

ref: projectdiscovery/notify#65

Is your feature request related to a problem? Please describe.
Support getting instances from Hetzner Cloud

Describe the solution you'd like
https://github.com/hetznercloud/hcloud-go

Add support for:

• Compute Instance IP #488
• Buckets
• Cloud Functions
• Cloud Run

As similar to other provider, add kubernetes as a provider.

Add ability to output:

• Service IP address from all the namespace
• Ingress rules from all the namespace

In the config file,

- # provider is the name of the provider
  provider: k8s OR kubernetes
  #  id is the name defined by user for filtering (optional)
  id: dev_cluster
  kubeconfig_file:  <location-of-kubeconfig>
  context: <name of context>

Is your feature request related to a problem? Please describe.
Some of the providers will show if an instance is active/up, or shutdown.

For example, for Digital Ocean retrieving droplet.Status will show active for running instances.

Describe the solution you'd like
Add a flag to exclude inactive instances from providers who support it.

Description

OpenStack is a free, open standard cloud computing platform. It is mostly deployed as infrastructure-as-a-service (IaaS) in both public and private clouds where virtual servers and other resources are made available to users.

Source : Wikipedia

Ability to display server names and ip adresses in specific or all tenants.

Describe the solution you'd like
Add Openstack as a new cloud provider to cloudlist. gophercloud seems to be an interesting library to implement Openstack API interactions.

I may be interested developing this feature myself if main contributors are open to the idea of adding this provider.

Hello,

Would it be possible to have a new release with new built-in binaries? Those on the release page are from Dec 2020 and it seems are missing some patches / improvements

Thank you!

Is your feature request related to a problem? Please describe.
The current format can be clunky when used in an enterprise setting, as a standard user has access to several accounts and Cloudlist doesn't have a way to filter your findings when it comes to Cloudflare on a per-zone level. Whilst Cloudflare API tokens would help, they aren't supported in this project.

Describe the solution you'd like
I would like for the project to support Cloudflare API tokens. This would allow me to finely scope API tokens on a per-zone, per-account basis to ensure it's only doing what it needs to.

Describe the bug
The sample configuration file located here is not valid.

Cloudlist version
git install: [INF] Current Version: 1.0.0

Screenshot of the error or bug

╰─$ ./cloudlist

   ________                _____      __
  / ____/ /___  __  ______/ / (_)____/ /_
 / /   / / __ \/ / / / __  / / / ___/ __/
/ /___/ / /_/ / /_/ / /_/ / / (__  ) /_
\____/_/\____/\__,_/\__,_/_/_/____/\__/  v1.0.0

		projectdiscovery.io

Use with caution. You are responsible for your actions
Developers assume no liability and are not responsible for any misuse or damage.
[FTL] Could not create runner: yaml: line 78: did not find expected '-' indicator

Yamllint:
79:2 error syntax error: expected <block end>, but found '<block sequence start>' (syntax)

As of now GCP provider fetches DNS name from GCP, it can be extended to fetch VM instances and other available public asset as well.

Describe the bug
When I try to use AWS temp session token, It is throwing the auth error(401). I have mentioned the below attributes in the config file.
1- aws_access_key:
2- aws_secret_key:
3- aws_session_token:

Screenshot of the error or bug
please add the screenshot showing bug or issue you are facing.

Is your feature request related to a problem? Please describe.
GKE provides DNS names that are often required to interact with services running in a given cluster. In these cases, the IP addresses of the nodes aren't as valuable as the DNS names that GKE has assigned to services and other items.

Describe the solution you'd like
Add GKE as a source from GCP

It would be great to have a provider for Azure related services to pull information for assets. The AWS/DO providers work great, just need support for Azure environments next.

Description

1. It is unclear where and from I find provider id from Azure and why its needed for?
2. subscription_id should be optional. Gathering all accessible subscription_id's user has access to should be possible.
3. Can this work even work if it is missing secret itself which walks with secret_id?

In verbose mode I don't get any errors or findings when not using provider id. I have specified subscription ID where at least one public IP is assigned.

- provider: azure # provider is the name of the provider
  # id is the name of the provider id
  id: staging
  # client_id is the client ID of registered application of the azure account (not requuired if using cli auth)
  client_id: xxxxxxxxxxxxxxxxxxxxxxxxx
  # client_secret is the secret ID of registered application of the zure account (not requuired if using cli uth)
  client_secret: xxxxxxxxxxxxxxxxxxxxx
  # tenant_id is the tenant ID of registered application of the azure account (not requuired if using cli auth)
  tenant_id: xxxxxxxxxxxxxxxxxxxxxxxxx
  #subscription_id is the azure subscription id
  subscription_id: xxxxxxxxxxxxxxxxxxx
  #use_cli_auth if set to true cloudlist will use azure cli auth
  use_cli_auth: true

cloudlist -version
1.0.1

Is your feature request related to a problem? Please describe.

Goflags:- https://github.com/projectdiscovery/goflags that comes with the following support

• Auto-generated Flag configuration file
• Long / Short formatted flag support
• Help menu grouping support

Describe the solution you'd like

• Moving current keys configuration file to $HOME/.config/cloudlist/provider.yaml
• Goflags integration with default flag config file located to $HOME/.config/cloudlist/config.yaml
• New flag for custom provider config file.

   -config string              path to the cloudlist flag configuration file (default "$HOME/.config/cloudlist/config.yaml")
   -pc, -provider-config string  custom provider configuration file to use (default "$HOME/.config/cloudlist/provider.yaml")

Describe the bug
The repository is missing sonarcloud integration

Discussed in #248

- provider: aws 
  id: staging
  aws_access_key: $ACCESS_KEY
  aws_secret_key: $SECRET_KEY

Note:

Probably support to add in https://github.com/projectdiscovery/goflags so all other PD projects can utilize this enhancement.

Describe the bug
According to the documentation:

The default provider config file should be located at $HOME/.config/cloudlist/provider-config.yaml and has the following contents as an example. 

However, the file is not created when installing via git.

Cloudlist version
Built from github: [INF] Current Version: 1.0.0

Expected Behavior
If installation method can't create a default config, provide one in the source tree to copy, or create one, during first run.

Screenshot of the error or bug

╰─$ ll $HOME/.config/cloudlist/provider-config.yaml
ls: /Users/sullo/.config/cloudlist/provider-config.yaml: No such file or directory

Use utils.

Speaking from personal experience, besides standard VM/NIC assigned IP's it would be so amazing if this could pull all the other endpoints that are publicly available and allow for misconfiguration. These are the examples that come to mind.

AWS:

• S3 bucket Endpoints #316
• EKS Management API Endpoints
• Redis Endpoints
• Any other PaaS Service endpoints

Azure:

• Blob Endpoints
• AKS Management API Endpoints
• Service Fabric Cluster Explorer Endpoints
• Traffic Manager Endpoints
• PaaS Database Endpoints(Redis, MySQL, Postgres, etc)
• APIM Endpoints (Developer portal URL & Gateway URL)
• Enterprise Applications (Homepage URL's)

We should consider adding the following assets to the AWS provider:

• ECS
• EKS (also in #5 )
• ELB
• ALB
• API Gateway
• Lambda
• Lightsail (See #493 )
• Cloudfront?
• Service option to filter services from provider

   -s, -service string[]  query and display results from given service (comma-separated)

Description

The list of targets visible/in scope for a particular account can be pulled also from the following bug bounty platform API (token/headless authentication):

• Hackerone
• TDB

original idea and reference implementation - https://github.com/sw33tLie/bbscope

Is your feature request related to a problem? Please describe.
To get a more complete list of live addresses, each provider should be investigated to the limits of the API or the user's selections. This would necessarily include each area a provider has, including:

• Load balancers (e.g., ALB/ELB in AWS)
• DNS
• Floating IPs (DO)
• Containers

And similar data points from each provider

Describe the solution you'd like
Pull as complete a record set as possible from each provider.

We should have more DNS provider integrations for getting complete list.

• Add Support for Azure DNS DNS provider
• Add Support for Linode DNS provider
• Add Support for Cloudflare DNS provider
• Add Support for Alibaba cloud DNS provider
• Add Support for Namecheap DNS provider
• Add Support for Active Directory DNS provider
• Add Support for AXFR+DDNS DNS provider
• Add Support for BIND DNS provider
• Add Support for ClouDNS DNS provider
• Add Support for Constellix (DNSMadeEasy) DNS provider
• Add Support for CSC Global DNS provider
• Add Support for deSEC DNS provider
• Add Support for DNSimple DNS provider
• Add Support for Exoscale DNS provider
• Add Support for Gandi DNS provider
• Add Support for GoDaddy DNS provider
• Add Support for Hetzner DNS provider
• Add Support for HEXONET DNS provider
• Add Support for Hurricane Electric DNS DNS provider
• Add Support for Internet.bs DNS provider
• Add Support for INWX DNS provider
• Add Support for Joker.com DNS provider
• Add Support for knot-dns DNS provider
• Add Support for Name.com DNS provider
• Add Support for NameSilo DNS provider
• Add Support for Netcup DNS provider
• Add Support for Netlify DNS provider
• Add Support for NS1 DNS provider
• Add Support for OctoDNS DNS provider
• Add Support for OpenSRS DNS provider
• Add Support for Oracle Cloud Infrastructure DNS provider
• Add Support for OVH DNS provider
• Add Support for PowerDNS DNS provider
• Add Support for RcodeZero DNS provider
• Add Support for SoftLayer DNS provider
• Add Support for Vultr DNS provider

Currently cloudlist doesn't let user to specify more than one provider using provider flag.

Fetching assets from consul or nomad for an organization.

Describe the bug
cloudlist -ip -silent gives me private IP's as well (I am not sure if this is inteneded)

Nuclei version
Please share the version of the nuclei you are running with dnsx -version
[INF] Current Version: 1.0.3

Screenshot of the error or bug

Expected Solution
This tool should essentially filter out all the Reserved IP's as mentioned by IANA

Describe the bug
I am not able to run Cloudlist as there is some kind of error.
I have tried the release, build it my self but still, the error is the same

Error

root@root:~/cloudlist/cmd/cloudlist# ./cloudlist -provider aws
v.0.0.1
Could not create runner: yaml: unmarshal errors:
line 21: cannot unmarshal !!map into schema.Options

Go Version
go version go1.15.5 linux/amd64