tvarit-foggy / grafana-s3-plugin Goto Github PK

View Code? Open in Web Editor NEW

24.0 5.0 9.0 44.72 MB

Grafana Plugin for querying files on AWS S3 using S3 Select API

License: Apache License 2.0

JavaScript 0.84% TypeScript 64.02% Go 35.14%

aws-s3 grafana-plugin react-plugin grafana-datasource

grafana-s3-plugin's Introduction

AWS S3 Plugin

This plugin queries files on AWS S3 using S3 Select API

Installation

To install, download zip file from release page (for stable version) or download repository as zip file (for installing from git master)

grafana-cli --pluginUrl <path_to_zip_file> plugins install tvarit-s3-datasource

The plugin is not signed. To enable this plugin, change the following

[plugins]
...
# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
allow_loading_unsigned_plugins =

[plugins]
...
# Enter a comma-separated list of plugin identifiers to identify plugins that are allowed to be loaded even if they lack a valid signature.
allow_loading_unsigned_plugins = tvarit-s3-datasource

File Upload

To allow file upload from inbuilt file browser, add the following CORS policy in the bucket permissions (replace <<>> with the url in your browser)

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin><<<URL>>></AllowedOrigin>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <ExposeHeader>ETag</ExposeHeader>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
</CORSRule>
</CORSConfiguration>

Reference Links

S3 Select API Reference: https://docs.aws.amazon.com/AmazonS3/latest/API/API_SelectObjectContent.html
Query Reference: https://docs.aws.amazon.com/AmazonS3/latest/dev/s3-glacier-select-sql-reference.html

grafana-s3-plugin's People

Contributors

Stargazers

Watchers

Forkers

kamalgalrani galo burningalchemist starkblaze01 sumeshks29 shubhamkcloud lanphan kthwaite nenzyz

grafana-s3-plugin's Issues

Support for arm64 architecture

Any distribution for arm64 architecture?

msg="Failed to start plugin" logger=plugins.backend pluginId=tvarit-s3-datasource error="fork/exec /var/lib/grafana/plugins/tvarit-s3-datasource/dist/gpx_s3_plugin_linux_arm64: no such file or directory"

when i try to run query from plugin, it gives me nosuchkey error

The error : - NoSuchKey: The specified key does not exist. status code: 404, request id: E5AE50E2543C37CE, host id: mfcQYp9T1icc0AaF74jycp1mEua9DCT1qWdcH8MFThOFk4B6lCEogqg1m0zquD0rZkfeVPt8JuE=

However when added S3 as datasource i can see in s3 explorer in plugins all the files in s3 bucket. but when i query them it give me nosuchkey error as shown above.

can you pls share any screenshot where plugin has successfully queried a csv file from the plugin. just that i need to see what all options i want to fill

first option is "path" , not sure what to fill tried many options their.

Migrate from CircleCI to Jenkins

Have a look at workflows in .circleci folder
Migrate those workflows to Jenkinsfile
Setup CI/CD

When I try to run query from plugin, it gives me "Metric request error"

Here is the configuration of the request and the result

Is it necessary to use a specific identation for the csv? or what's wrong?

Thank you in advance

Add AWS SDK as an authorization method (similar to AWS cloudwatch data source)

I want to suggest this idea to access S3 buckets using AWS SDK authentication provider. Most of us use assumed roles to avoid dealing with access/secret keys for each user and machines.

An example of how Cloudwatch datasource does it is shown in the below screenshot:

Please help.

The plugin is not being installed properly

Hi there! I installed the plugin and it looks like is ok, but I'm not seeing it in the UI, I mean, I do not have any option for S3

bash-5.0$ grafana-cli plugins ls
installed plugins:
tvarit-s3-datasource @ 0.0.1

MinIO compatibility

Any roadmap to do this plugin compatible with MinIO, which also support S3 Select?

Using the plugin with grafana 7.5.4

I need to use this plugin with a docker installation.
I have tried to use the GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS env variable
BUT grafana is disabling the plugin.

Any workaround idea to use this plugin ?

Is it possible to make this plugin an "official grafana plugin" ?

Plugin cannot query JSON line format

Describe the bug
When querying JSON file from S3, the datasource tries to convert to CSV.

I guess we should be able to keep JSON format as output.

PS: The same query in AWS S3 select works when output is JSON

Please complete the following information:

OS: [linux]
Browser [chrome]
Grafana Version [7.5.6]

Additional context
Here is the error message in Grafana log :

t=2021-06-25T13:55:23+0000 lvl=info msg=S3Select logger=plugins.backend pluginId=tvarit-s3-datasource stats="map[BytesProcessed:131801 BytesReturned:2777 BytesScanned:196524]" ││ t=2021-06-25T13:55:23+0000 lvl=eror msg=S3Select logger=plugins.backend pluginId=tvarit-s3-datasource df.ToCSV="ToCSV (jsonRecordsToData: unknown type of map[])"

PS2: Maybe it cannot work on "complex JSON", only flat JSON can work :(

How to install this?

I'm interested in trying this out, but I was unable to figure out how I would install it. Can you provide a link or similar?

I'm using Grafana cloud starter edition.