Git Product home page Git Product logo

steampipe-plugin-config's Introduction

image

Config Plugin for Steampipe

Use SQL to query data from various types of configuration files, e.g. INI, JSON, YML.

Quick start

Install the plugin with Steampipe:

steampipe plugin install config

Configure your config file for each supported file type to include directories with files to be parsed. For any of the paths arguments, if no directory is specified, the current working directory will be used.

Run steampipe:

steampipe query

Query all data in your INI files:

select
  path,
  section,
  key,
  value
from
  ini_key_value;
+----------------------------+----------+---------------+-------------------------------------------+
| path                       | section  | key           | value                                     |
+----------------------------+----------+---------------+-------------------------------------------+
| /Users/myuser/defaults.ini | Settings | DetailedLog   | 1                                         |
| /Users/myuser/defaults.ini | Status   | RunStatus     | 1                                         |
| /Users/myuser/defaults.ini | Status   | StatusRefresh | 10                                        |
| /Users/myuser/defaults.ini | Status   | StatusPort    | 6090                                      |
| /Users/myuser/logs.ini     | Server   | Archive       | 1                                         |
| /Users/myuser/logs.ini     | Server   | ServerName    | Unknown                                   |
| /Users/myuser/logs.ini     | Settings | LogFile       | /opt/ecs/mvuser/MV_IPTel/log/MV_IPTel.log |
| /Users/myuser/logs.ini     | Settings | Version       | 0.9 Build 4 Created July 11 2004 14:00    |
+----------------------------+----------+---------------+-------------------------------------------+

Query all data in your JSON files:

select
  path,
  jsonb_pretty(content) as file_content
from
  json_file;
+----------------------------+------------------------------------------------------------+
| path                       | file_content                                               |
+----------------------------+------------------------------------------------------------+
| /Users/myuser/invoice.json | {                                                          |
|                            |     "city": "East Centerville",                            |
|                            |     "date": "2012-08-06T00:00:00Z",                        |
|                            |     "items": [                                             |
|                            |         {                                                  |
|                            |             "price": 1.47,                                 |
|                            |             "part_no": "A4786",                            |
|                            |             "quantity": 4,                                 |
|                            |             "description": "Water Bucket (Filled)"         |
|                            |         },                                                 |
|                            |         {                                                  |
|                            |             "size": 8,                                     |
|                            |             "price": 133.7,                                |
|                            |             "part_no": "E1628",                            |
|                            |             "quantity": 1,                                 |
|                            |             "description": "High Heeled \"Ruby\" Slippers" |
|                            |         }                                                  |
|                            |     ],                                                     |
|                            |     "state": "KS",                                         |
|                            |     "street": "123 Tornado Alley\nSuite 16\n",             |
|                            |     "bill-to": null,                                       |
|                            |     "receipt": "Oz-Ware Purchase Invoice",                 |
|                            |     "ship-to": null,                                       |
|                            |     "customer": {                                          |
|                            |         "first_name": "Dorothy",                           |
|                            |         "family_name": "Gale"                              |
|                            |     },                                                     |
|                            | }                                                          |
| /Users/myuser/test.json    | {                                                          |
|                            |     "foo": "bar",                                          |
|                            |     "includes": [                                          |
|                            |         "common.json"                                      |
|                            |     ]                                                      |
|                            | }                                                          |
+----------------------------+------------------------------------------------------------+

Query all data in your YML files:

select
  path,
  jsonb_pretty(content) as file_content
from
  yml_file;
+---------------------------+------------------------------------------------------------+
| path                      | file_content                                               |
+---------------------------+------------------------------------------------------------+
| /Users/myuser/invoice.yml | {                                                          |
|                           |     "city": "East Centerville",                            |
|                           |     "date": "2012-08-06T00:00:00Z",                        |
|                           |     "items": [                                             |
|                           |         {                                                  |
|                           |             "price": 1.47,                                 |
|                           |             "part_no": "A4786",                            |
|                           |             "quantity": 4,                                 |
|                           |             "description": "Water Bucket (Filled)"         |
|                           |         },                                                 |
|                           |         {                                                  |
|                           |             "size": 8,                                     |
|                           |             "price": 133.7,                                |
|                           |             "part_no": "E1628",                            |
|                           |             "quantity": 1,                                 |
|                           |             "description": "High Heeled \"Ruby\" Slippers" |
|                           |         }                                                  |
|                           |     ],                                                     |
|                           |     "state": "KS",                                         |
|                           |     "street": "123 Tornado Alley\nSuite 16\n",             |
|                           |     "bill-to": null,                                       |
|                           |     "receipt": "Oz-Ware Purchase Invoice",                 |
|                           |     "ship-to": null,                                       |
|                           |     "customer": {                                          |
|                           |         "first_name": "Dorothy",                           |
|                           |         "family_name": "Gale"                              |
|                           |     },                                                     |
|                           | }                                                          |
| /Users/myuser/test.yaml   | {                                                          |
|                           |     "foo": "bar",                                          |
|                           |     "includes": [                                          |
|                           |         "common.yaml"                                      |
|                           |     ]                                                      |
|                           | }                                                          |
+---------------------------+------------------------------------------------------------+

Engines

This plugin is available for the following engines:

Engine Description
Steampipe The Steampipe CLI exposes APIs and services as a high-performance relational database, giving you the ability to write SQL-based queries to explore dynamic data. Mods extend Steampipe's capabilities with dashboards, reports, and controls built with simple HCL. The Steampipe CLI is a turnkey solution that includes its own Postgres database, plugin management, and mod support.
Postgres FDW Steampipe Postgres FDWs are native Postgres Foreign Data Wrappers that translate APIs to foreign tables. Unlike Steampipe CLI, which ships with its own Postgres server instance, the Steampipe Postgres FDWs can be installed in any supported Postgres database version.
SQLite Extension Steampipe SQLite Extensions provide SQLite virtual tables that translate your queries into API calls, transparently fetching information from your API or service as you request it.
Export Steampipe Plugin Exporters provide a flexible mechanism for exporting information from cloud services and APIs. Each exporter is a stand-alone binary that allows you to extract data using Steampipe plugins without a database.
Turbot Pipes Turbot Pipes is the only intelligence, automation & security platform built specifically for DevOps. Pipes provide hosted Steampipe database instances, shared dashboards, snapshots, and more.

Developing

Prerequisites:

Clone:

git clone https://github.com/turbot/steampipe-plugin-config.git
cd steampipe-plugin-config

Build, which automatically installs the new version to your ~/.steampipe/plugins directory:

make

Configure the plugin:

cp config/* ~/.steampipe/config
vi ~/.steampipe/config/config.spc

Try it!

steampipe query
> .inspect config

Further reading:

Open Source & Contributing

This repository is published under the Apache 2.0 (source code) and CC BY-NC-ND (docs) licenses. Please see our code of conduct. We look forward to collaborating with you!

Steampipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #steampipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

steampipe-plugin-config's People

Contributors

cbruno10 avatar dependabot[bot] avatar keilin-anz avatar misraved avatar pskrbasu avatar subhajit97 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Forkers

keilin-anz

steampipe-plugin-config's Issues

Add support for parsing `.ini` type files and represent it as table

Is your feature request related to a problem? Please describe.
Parse .ini files to represent them as table.

Describe the solution you'd like
Since the files structured with sections that contains one or more key-value pairs, table can be represent with each of key-value pair as a column?

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Add File Watcher support for paths argument in config

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Support for private git repositories

Is your feature request related to a problem? Please describe.

When wanting to parse and query data stored in config files in git repositories, this data is found in private repositories on GitLab or GitHub. Accessing those is currently not possible, in difference to accessing private S3 buckets.

Describe the solution you'd like
Both and other git hosting services provide token or SSH authentication possibilities. These were to leverage for implementing a "standard-compliant" way to allow git authentication.

Describe alternatives you've considered
This is a one-way street. There's no alternative to that, when wanting to support private git repositories.

Additional context
ArgoCD offers nice abstractions for managing git repository access & credentials.

Update plugin to use separate args for file types and remove implicit file extension matching

Is your feature request related to a problem? Please describe.
Currently the config file only have paths argument to provide required file paths. For example:

connection "config" {
  plugin = "config"
  
  paths = [ ... ]
}

Split the paths into different args for different file types. For example:

connection "config" {
  plugin = "config"
  
  ini_paths    = [ "*.ini" ]
  json_paths = [ "*.json" ]
  yml_paths  = [ "*.yml", "*.yaml" ]
}

Also currently plugin implicitly checks for required extension based on types of tables queried. For example:
If the config file contains paths *, and the query is based on yml_file table, plugin only matches files that ends with .yml or .yaml.

Remove that check to support files for which the files don't have to end in INI. For example, you have a directory of sample AWS config files that want to parse.

Remove prefix `config_` from table name

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Remove table `json_key_value` and `yml_key_value`

Remove table json_key_value and yml_key_value for now, since currently we won't have any stable steampipe release that supports Ltree.
We will add Ltree support in upcoming release, and will add these tables after that.

Support loading JSON arrays

Is your feature request related to a problem? Please describe.

Currently the config plugin will throw an error if any JSON files in the search path are JSON arrays, ie. the outermost container is an array instead of an object, eg.

This will fail:

[
  { "text": "I am valid json" },
]

🐛 Due to the way the plugin loads files, this actually prevents access to any files in the directory

Describe the solution you'd like

Allow loading json arrays into the json column, or fail more gracefully in their presence

Describe alternatives you've considered

Workaround is easy enough in smaller use cases, namely: process them into an object

eg. with jq:

cat <<EOF | jq '{ content: . }'
[ { "text": "I am valid json" } ]
EOF

outputs:

{
  "content": [
    {
      "text": "I am valid json"
    }
  ]
}

Additional context

Not much more to add!

Config plugin using 100% CPU on load due to rogue dynamic schema definition?

Describe the bug

When starting steampipe query with the config plugin installed in Steampipe v0.19.0-rc.3 I'm seeing the config plugin jump and stay at ~100% CPU.

I haven't even run a query at this stage:

image

image

I believe this is because we are declaring the schema as dynamic, when it's not actually?

Removing this line from the code fixed the 100% CPU issue:

https://github.com/turbot/steampipe-plugin-config/blob/main/tables/plugin.go#L18

Update `steampipe-plugin-sdk` to `v5.0.0`

Is your feature request related to a problem? Please describe.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.