turbot / steampipe-plugin-sdk Goto Github PK

Steampipe Plugin SDK is a simple abstraction layer to write a Steampipe plugin. Plugins automatically work across all engine types including the Steampipe CLI, Postgres FDW, SQLite extension and the export CLI.

Home Page: https://hub.steampipe.io/plugins

License: Apache License 2.0

Makefile 0.02% Go 99.28% HCL 0.66% Shell 0.04%

steampipe steampipe-plugin postgresql postgresql-fdw sql hacktoberfest etl sqlite sqlite-extension zero-etl

steampipe-plugin-sdk's Introduction

Steampipe Plugin SDK

This SDK makes it easy to write Steampipe plugins.

Go Documentation →

Getting Started

Compiling this SDK

To compile the SDK repo, you must install protoc.

brew install protoc
brew install protoc-gen-go-grpc

Open Source & Contributing

This repository is published under the Apache 2.0 license. Please see our code of conduct. We look forward to collaborating with you!

Steampipe is a product produced from this open source software, exclusively by Turbot HQ, Inc. It is distributed under our commercial terms. Others are allowed to make their own distribution of the software, but they cannot use any of the Turbot trademarks, cloud services, etc. You can learn more in our Open Source FAQ.

Get Involved

Join #steampipe on Slack →

Want to help but don't know where to start? Pick up one of the help wanted issues:

steampipe-plugin-sdk's People

Contributors

Stargazers

Watchers

Forkers

etienne-carriere daniel-garcia justlikeef aminvielledebatatbedrock blylei pdecat pskrbasu kaloob deepfence bradley39e aks-v-aks judell jonudell raw-labs antoine-conserto

steampipe-plugin-sdk's Issues

v0.2.4-rc.1

At present, we do not handle the list of qual values passed to a list call for an in clause, so if the list call has required key columns it fails

remove need to call StreamListItem for parent-child list calls

sdk should route the calls internally to call the appropriate stream function

v0.2.2

v0.2.3

Add mechanism to limit max hydrate concurrency

update connection config schema validation to validate the target struct

check all fields in the schema have corresponding tagged struct properties
check properties for optional fields are nullable

Update naming for concurrency properties

Update protobuf GetSchema contract to include key column information

Fix error for missing required quals

If no quals are provided for as get (or list with required quals), no error is being shown

for parent-child list call, store parent list call results in RowData ParentItem property

Tables with no list config defined hang indefinitely if you query without any qualifiers

Issue

If you have a table with just a get config (no list config) and do select * from <table>, the terminal will sit forever saying Loading results.

Expected Outcome

As the table has no list config defined, I would expect immediate failure with some kind of appropriate message.

Actual Outcome

The terminal will sit forever saying Loading results and you need to terminate the steampipe process from another session.

Env

CLI: 0.1.2
OS: macOS Big Sur 11.2
Terminal: iTerm 3.4.3 / zsh

make gets for 'in' asyncronous

currently if there is an 'in' clause we execute the get for each element syncronously

v0.2.1

Add support for multiregion

tidy up DisplayProfileData

		s := ""
		for _, b := range b.Bytes() {
			if b == '\n' {
				log.Printf("[WARN] %s\n", s)
				s = ""
			} else {
				s += string(b)
			}
		}```

Add support for logging env var STEAMPIPE_LOG

deprecate SP_LOG

Add LabelsToTagsMap and EnsureStringArray functions to sdk plugin

Only show concurrency summary if there is any summary data to show

Tidy logging for concurrency summary

Add transform to get a field value from two different paths

There are few AWS resources in which the output of GET and LIST API calls differ (The type of struct differs for them).

Can we have a transform through which we can mention 2 paths to get the value ("Path1", "Path2")
If the value is not found in Path1 the transform should try to retrieve it from Path2

For example in aws_dynamodb_backup table

{
	Name:        "backup_status",
	Description: "Current status of the backup. Backup can be in one of the following states: CREATING, ACTIVE, DELETED",
	Type:        proto.ColumnType_STRING,
	Transform:   transform.GetValue("BackupStatus", "BackupDetails.BackupStatus,"),
}

LIST CALL STRUCT `BackupSummary`

// Contains details for the backup.
type BackupSummary struct {
	_ struct{} `type:"structure"`
	BackupArn *string `min:"37" type:"string"`
	BackupCreationDateTime *time.Time `type:"timestamp"`
	BackupExpiryDateTime *time.Time `type:"timestamp"`
	BackupName *string `min:"3" type:"string"`
	BackupSizeBytes *int64 `type:"long"`
	BackupStatus *string `type:"string" enum:"BackupStatus"`
	BackupType *string `type:"string" enum:"BackupType"`
	TableArn *string `type:"string"`
	TableId *string `type:"string"`
	TableName *string `min:"3" type:"string"`
}

GET CALL STRUCT `BackupDescription`

type BackupDescription struct {
	_ struct{} `type:"structure"`

	// Contains the details of the backup created for the table.
	BackupDetails *BackupDetails `type:"structure"`

	// Contains the details of the table when the backup was created.
	SourceTableDetails *SourceTableDetails `type:"structure"`

	// Contains the details of the features enabled on the table when the backup
	// was created. For example, LSIs, GSIs, streams, TTL.
	SourceTableFeatureDetails *SourceTableFeatureDetails `type:"structure"`
}

// Contains the details of the backup created for the table.
type BackupDetails struct {
	_ struct{} `type:"structure"`
	BackupArn *string `min:"37" type:"string" required:"true"`
	BackupCreationDateTime *time.Time `type:"timestamp" required:"true"`
	BackupExpiryDateTime *time.Time `type:"timestamp"`
	BackupName *string `min:"3" type:"string" required:"true"`
	BackupSizeBytes *int64 `type:"long"`
	BackupStatus *string `type:"string" required:"true" enum:"BackupStatus"`
	BackupType *string `type:"string" required:"true" enum:"BackupType"`
}

// Contains the details of the table when the backup was created.
type SourceTableDetails struct {
	_ struct{} `type:"structure"`
	BillingMode *string `type:"string" enum:"BillingMode"`
	ItemCount *int64 `type:"long"`
	KeySchema []*KeySchemaElement `min:"1" type:"list" required:"true"`
	ProvisionedThroughput *ProvisionedThroughput `type:"structure" required:"true"`
	TableArn *string `type:"string"`
	TableCreationDateTime *time.Time `type:"timestamp" required:"true"`
	TableId *string `type:"string" required:"true"`
	TableName *string `min:"3" type:"string" required:"true"`
	TableSizeBytes *int64 `type:"long"`
}

return error if list call returns a value (it should ONLY stream items)

there is a return value for consistency with other hydrate functions but it should always be nil

Transform `ToLower` not converting string to lowercase

branch: issue-13

	{
		Name:        "resource_group",
		Description: ColumnDescriptionResourceGroup,
		Type:        proto.ColumnType_STRING,
		Transform:   transform.FromField("ResourceGroup").Transform(transform.ToLower),
	},

> select resource_group from azure.azure_storage_table;
+----------------+
| RESOURCE_GROUP |
+----------------+
| Lalit_test     |
+----------------+
>

Add a URI column type(?)

@e-gineer

e-gineer commented on 13 Nov 2020

I think we should consider having a specific column type for URIs.

It could be combined with https://github.com/petere/pguri

It would be used by things like github > organization > url

Handle cancellation of streams

when streaming results, we must handle cancellation

We need to consider the full cancellation journey - from steampipe > postgres > fdw > plugin

Refactor error streaming code to check error channel before row channel

this is to avoid potential timing errors when the row channel is closed but there is an error on the error channel

Support 'in' clause for list calls with required key columns

At present, we do not handle the list of qual values passed to a list call for an in clause, so if the list call has required key columns it fails

Legacy get calls failing

The hydrate item is not being populated for legacy get calls, causing them to fail when trying to cast the item

Provide dependency mechanism to allow steampipe to know if a plugin uses a newer sdk version

if the plugin is using a newer sdk version than steampipe, there may for example be a new column definition which stempipe/fdw does not know about

therefore if steampipe refreshes a connection for a plugin which uses a newer sdk version it should raise an error/warning

Add support for connection config

Add support for HCL format connection config - this will parsed by steampipe and sent to the plugin instance

The plugin will define a connection config schema which will be used to validate and parse the connection config

Rename ip_addr protobuf value to Inet

Ensure all public SDK functions recover from panics

update hcl to error conversion to include all hcl diagnostics

Add transform to convert Unix time to Steampipe timestamp

update list to handle key columns for in() queries

Add INET column type

We currently define ColumnType_CIDR and ColumnType_IPADDR which are validated to be a cidr and an IP address respectively. Add ColumnType_INET which may be either

Add support for plugin config

In azure.azure_firewall table seeing few keys are missing for json object when passing it directly using transform.FromField

Slack conversation thread
https://turbothq.slack.com/archives/C01AC8JQNHH/p1610017247038000

		{
			Name:        "ip_configurations",
			Description: "A collection of IP configuration of the Azure Firewall resource",
			Type:        proto.ColumnType_JSON,
			Transform:   transform.FromField("AzureFirewall.AzureFirewallPropertiesFormat.IPConfigurations"),
		},

Test column using transform

			{
				Name:        "test_column",
				Description: "A list of AKAs (also-known-as) that uniquely identify this resource",
				Type:        proto.ColumnType_JSON,
				Transform:   transform.From(testFunction),
			},

Here in field test_column we are getting all the properties but not in the field ip_configurations
missing keys for ip_configurations

    "PrivateIPAddress": "10.0.0.4",
    "ProvisioningState": "Succeeded",

steampipe query --output json "select name, test_column, ip_configurations from azure.azure_firewall"

[
 {
  "ip_configurations": [
   {
    "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/azureFirewalls/test_firewall/azureFirewallIpConfigurations/test_pi",
    "name": "test_pi",
    "properties": {
     "publicIPAddress": {
      "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/publicIPAddresses/test_pi"
     },
     "subnet": {
      "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/virtualNetworks/testvn/subnets/AzureFirewallSubnet"
     }
    }
   }
  ],
  "name": "test_firewall",
  "test_column": [
   {
    "PrivateIPAddress": "10.0.0.4",
    "ProvisioningState": "Succeeded",
    "PublicIPAddress": {
     "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/publicIPAddresses/test_pi"
    },
    "Subnet": {
     "id": "/subscriptions/d7245080-b4ae-4fe5-b6fa-2e71b3dae6c8/resourceGroups/turbot_rg/providers/Microsoft.Network/virtualNetworks/testvn/subnets/AzureFirewallSubnet"
    }
   }
  ]
 },
]

Test transform function

func testFunction(ctx context.Context, d *transform.TransformData) (interface{}, error) {
	data := d.HydrateItem.(firewallInfo)

	var output []map[string]interface{}
	for _, firewall := range *data.AzureFirewall.AzureFirewallPropertiesFormat.IPConfigurations {
		output = append(output, map[string]interface{}{
			"PrivateIPAddress":  firewall.AzureFirewallIPConfigurationPropertiesFormat.PrivateIPAddress,
			"Subnet":            firewall.AzureFirewallIPConfigurationPropertiesFormat.Subnet,
			"PublicIPAddress":   firewall.AzureFirewallIPConfigurationPropertiesFormat.PublicIPAddress,
			"ProvisioningState": firewall.AzureFirewallIPConfigurationPropertiesFormat.ProvisioningState,
		})
	}
	return output, nil
}

Struct for IPConfigurations

type AzureFirewallIPConfigurationPropertiesFormat struct {
	// PrivateIPAddress - READ-ONLY; The Firewall Internal Load Balancer IP to be used as the next hop in User Defined Routes.
	PrivateIPAddress *string `json:"privateIPAddress,omitempty"`
	// Subnet - Reference to the subnet resource. This resource must be named 'AzureFirewallSubnet' or 'AzureFirewallManagementSubnet'.
	Subnet *SubResource `json:"subnet,omitempty"`
	// PublicIPAddress - Reference to the PublicIP resource. This field is a mandatory input if subnet is not null.
	PublicIPAddress *SubResource `json:"publicIPAddress,omitempty"`
	// ProvisioningState - READ-ONLY; The provisioning state of the Azure firewall IP configuration resource. Possible values include: 'Succeeded', 'Updating', 'Deleting', 'Failed'
	ProvisioningState ProvisioningState `json:"provisioningState,omitempty"`
}

Branch name
azure-firewall-test on https://github.com/turbotio/steampipe-plugin-azure repo

 failed to refresh connections: plugin steampipe-plugin-aws does not define a connection config schema

Update to:

failed to refresh connections: connection config has been set for connection 'aws1', but plugin 'steampipe-plugin-aws' does 
not define connection config schema

Lines 89 to 90 in cdf6f8e

 // deprecated 

 ColumnType_DATETIME ColumnType = 5