santthosh / aws-es-kibana Goto Github PK
View Code? Open in Web Editor NEWAWS ElasticSearch Kibana Proxy
License: Apache License 2.0
AWS ElasticSearch Kibana Proxy
License: Apache License 2.0
it would be great to hack around it
I was using thius library for a while, but suddenly it starts to throw this error:
AWS ES cluster available at http://0.0.0.0:443
Kibana available at http://0.0.0.0:443/_plugin/kibana/
/usr/lib/node_modules/aws-es-kibana/node_modules/http-proxy/lib/http-proxy/index.js:120
throw err;
^
Error: socket hang up
at connResetException (internal/errors.js:604:14)
at TLSSocket.socketCloseListener (_http_client.js:400:25)
at TLSSocket.emit (events.js:333:22)
at TLSSocket.EventEmitter.emit (domain.js:482:12)
at net.js:668:12
at TCP.done (_tls_wrap.js:556:7) {
code: 'ECONNRESET'
Is there a fix for this error?
Hi,
I have created a co-ordinate map visualization in Kibana. However, the map continues to show white/blank background. On the developer console in chrome I am seeing example.com/v1/default/4/3/8.png:1 Failed to load resource: the server responded with a status of 404 ()
The network tab shows that the call is being initiated by kibana.bundle.js
I was wondering whether this could be because aws-es-kibana is acting as a proxy which is preventing kibana from accessing whatever cdn it uses.
Is there any way to resolve this?
Could your provide example of AWS_PROFILE setting? I mean the full string.
I use docker compose on my AWS instance and aws-es-kibana docker image.
Always get: "The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method."
would be nice if we could (easily) configure for larger payloads?
i'm not that familiar with kibana (yet), but maybe there is a logical value that should be synced?
maybe this helpful?? http://stackoverflow.com/questions/19917401/node-js-express-request-entity-too-large
Error: request entity too large
at readStream (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/raw-body/index.js:196:17)
at getRawBody (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/raw-body/index.js:106:12)
at read (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/body-parser/lib/read.js:76:3)
at rawParser (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/body-parser/lib/types/raw.js:81:5)
at Layer.handle [as handle_request] (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:312:13)
at /Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:280:7
at Function.process_params (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:330:12)
at next (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/express/lib/router/index.js:271:10)
at expressInit (/Users/user/.nvm/versions/node/v6.2.2/lib/node_modules/aws-es-kibana/node_modules/express/lib/middleware/init.js:33:5)
This only applies to the master branch.
Was investigating the feasibility of using the basic auth fix under #49 but ran into issues with building/running under docker.
Error log
AWS ES cluster available at http://0.0.0.0:9200
Kibana available at http://0.0.0.0:9200/_plugin/kibana/
fs.js:1384
throw error;
^
Error: watch /root/.aws/credentials ENOENT
at _errnoException (util.js:992:11)
at FSWatcher.start (fs.js:1382:19)
at Object.fs.watch (fs.js:1408:11)
at Object.<anonymous> (/app/index.js:219:4)
at Module._compile (module.js:652:30)
at Object.Module._extensions..js (module.js:663:10)
at Module.load (module.js:565:32)
at tryModuleLoad (module.js:505:12)
at Function.Module._load (module.js:497:3)
at Function.Module.runMain (module.js:693:10)
PR: #48
I'm getting the following error. Is there a way to increase the request size?
Error: request entity too large
at readStream (/app/node_modules/raw-body/index.js:196:17)
at getRawBody (/app/node_modules/raw-body/index.js:106:12)
at read (/app/node_modules/body-parser/lib/read.js:76:3)
at rawParser (/app/node_modules/body-parser/lib/types/raw.js:81:5)
at Layer.handle [as handle_request] (/app/node_modules/express/lib/router/layer.js:95:5)
at trim_prefix (/app/node_modules/express/lib/router/index.js:317:13)
at /app/node_modules/express/lib/router/index.js:284:7
at Function.process_params (/app/node_modules/express/lib/router/index.js:335:12)
at next (/app/node_modules/express/lib/router/index.js:275:10)
at compression (/app/node_modules/compression/index.js:220:5)
After configuring the access policy on aws,
I can access kibana via aws-es-kibana proxy.
But after deploying to heroku, I can see the log message like this:
2017-06-30T09:03:16.344356+00:00 app[web.1]: Kibana available at http://127.0.0.1:50970/_plugin/kibana/
2017-06-30T09:03:16.344258+00:00 app[web.1]: AWS ES cluster available at http://127.0.0.1:50970
But when I access to https://<MY_HEROKUAPP>.herokuapp.com
or https://<MY_HEROKUAPP>.herokuapp.com/_plugin/kibana
,
Errors suddenly show up:
2017-06-30T09:04:33.967079+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=<MY_HEROKUAPP>.herokuapp.com request_id=7b787b96-d988-4cab-9845-adf89a2d5bf4 fwd="IP" dyno= connect= service= status=503 bytes= protocol=https
2017-06-30T09:04:34.778436+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/_plugin/kibana" host=<MY_HEROKUAPP>.herokuapp.com request_id=2cc7a8ee-e679-47a2-b599-766e9325579c fwd="IP" dyno= connect= service= status=503 bytes= protocol=http
The weird thing is I don't get any other error message,
and the local version went just fine.
Hello,
First of all, congrats for this great Kibana proxy, it helps a lot!
When trying to use aws-es-kibana on a laptop connecting to AWS Kibana through an SSH tunnel, we get an [ERR_TLS_CERT_ALTNAME_INVALID]
error due to http-proxy.
However there is an option on line 129 of index.js that is hardcoded to secure: true
that would allow the use case to work.
Would it be possible to have a command-line argument to put this option as false?
Here is the command I'm trying to run on a brand new EC2 instance running Amazon Linux:
sudo docker run -d -e LIMIT=1000kb -e AWS_ACCESS_KEY_ID=someID -e AWS_SECRET_ACCESS_KEY=secret -p 80:9200 aws-es-kibana -b 0.0.0.0 myURL
It starts without a problem, but if I run wget localhost
I get the following.
--2017-08-10 21:00:58-- http://localhost/
Resolving localhost (localhost)... 127.0.0.1
Connecting to localhost (localhost)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2017-08-10 21:00:58 ERROR 403: Forbidden.
Because I'm running this command from localhost there shouldn't be any security issues getting in the way.
Any ideas?
On node version 10. I get the following error:
_http_outgoing.js:481
throw new ERR_HTTP_HEADERS_SENT('set');
^
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
at ClientRequest.setHeader (_http_outgoing.js:481:11)
at ProxyServer.<anonymous> (/usr/local/lib/node_modules/aws-es-kibana/index.js:175:14)
at ProxyServer.emit (/usr/local/lib/node_modules/aws-es-kibana/node_modules/eventemitter3/index.js:184:35)
at ClientRequest.<anonymous> (/usr/local/lib/node_modules/aws-es-kibana/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:132:27)
at ClientRequest.emit (events.js:187:15)
at ClientRequest.EventEmitter.emit (domain.js:442:20)
at tickOnSocket (_http_client.js:639:7)
at onSocketNT (_http_client.js:655:5)
at process._tickCallback (internal/process/next_tick.js:174:19)
On node 9.11.1 it works.
> n 9
> node --version
v9.11.1
Is there a way we can access kibana endpoint over HTTPS? And proxy forward request to ES endpoint over HTTPS?
Hello,
I am using aws-es-kibana plugin as proxy for AWS kibana. It works fine for some time but then suddenly I am getting socket hang up error. Any idea?
POST/ PUT requests fail when using aws-es-kibana
:
_http_outgoing.js:346
throw new Error('Can\'t set headers after they are sent.');
^
Error: Can't set headers after they are sent.
at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:346:11)
at ProxyServer.<anonymous> (/usr/local/lib/node_modules/aws-es-kibana/index.js:112:14)
at ProxyServer.emit (/usr/local/lib/node_modules/aws-es-kibana/node_modules/eventemitter3/index.js:119:35)
at ClientRequest.<anonymous> (/usr/local/lib/node_modules/aws-es-kibana/node_modules/http-proxy/lib/http-proxy/passes/web-incoming.js:115:27)
at emitOne (events.js:101:20)
at ClientRequest.emit (events.js:188:7)
at tickOnSocket (_http_client.js:541:7)
at onSocketNT (_http_client.js:553:5)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickDomainCallback (internal/process/next_tick.js:122:9)
triggered by PUT
ting a mapping with curl
It would be nice to expose proxy server's timeout as an env var
$ aws-es-kibana my-elastic-cluster-in-china.es.amazonaws.com.cn
region cannot be parsed from endpoint address, either the endpoint must end in .<region>.es.amazonaws.com or --region should be provided as an argument
..... <truncated>.....
When sending a large (>100kb) body the proxy fails:
Assuming the proxy is running on localhost:
printf '==========%.0s' {1..100000} > dummy.json
curl -X POST http://localhost:8200/index1/typ1 -d @dummy.json
=> body parser error:
Error: request entity too large
However increasing the bodyparser limit causes other issues:
app.use(bodyParser.raw({
limit: '1mb',
type: function() { return true; }}));
=> Error: Can't set headers after they are sent.
at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:357:11)
Is there a workaround for this issue?
When starting this up on heroku, aws-es-kibana binds to 127.0.0.1 (local loopback), which heroku obviously isn't looking at, so 60 seconds later, we get this:
Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch
Workaround: set the BIND_ADDRESS config (env) to 0.0.0.0, which for sure will bind to the address heroku can see. Confirmed this fixed the issue for us.
Suggestion: Bind to 0.0.0.0 by default (less secure in general, but fixes the problem on heroku)
looks like the Host var that you put the ENDPOINT in is now being parsed in ES.(on host it is https).
i fixed by:
npm install url -save
require('url') in index.js
add line to index.js
var TARGETHOST = url.parse(TARGET).hostname;
Use that for populating Host header.
Hey guys,
im not really sure if this is intended or not but aws-es-kibana doesnt seem to work with CLI profiles which includes roles like the following construct.
[xxx-logon]
aws_access_key_id = xxx
aws_secret_access_key = xxx
[es-test]
role_arn = arn:aws:iam::xxxxx:role/es_test
source_profile = xxx-logon
When i use AWS_PROFILE=es-test the webserver is giving me the following return:
{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.\n\nThe Canonical String for this request should have been\n'GET\n/\n\nhost:https\nx-amz-date:20170725T142058Z\nx-amz-security-................
Is anyone one of you working with roles instead of local users?
Best regards,
Markus
npm install -g aws-es-kibana
aws-es-kibana XXXX.us-west-2.es.amazonaws.com
/usr/local/lib/node_modules/aws-es-kibana/index.js:160
app.use(async function (req, res) {
^^^^^
SyntaxError: missing ) after argument list
at createScript (vm.js:56:10)
at Object.runInThisContext (vm.js:97:10)
at Module._compile (module.js:542:28)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:393:7)
at startup (bootstrap_node.js:150:9)
Hi ,
i am getting error as security key invalid when we try to access using AWS STS credentials. But using that profile credentials i am able to access all aws services. I am using that same role in other programs for signing it works fine. When i try to use the same credentials in aws-es-kibana getting security key invalid error. Kindly some one help me to solve this issue.
Mohan
I think this commit completely disabled http authantication.
This leaves ES completely open to the public, even when deployed on heroku.
Am I missing something?
Amir
I tried using this, and got the following error:
aws-es-kibana https://xxxxxxxxxx.us-east-1.es.amazonaws.com
__________ _________ _________________ ________ ______
___ |_ | / /_ ___/ ___ ____/_ ___/ ___ __ \________________ ______ ____ /
__ /| |_ | /| / /_____ \ __ __/ _____ \ __ /_/ /_ ___/ __ \_ |/_/_ / / /_ /
_ ___ |_ |/ |/ / ____/ / _ /___ ____/ / _ ____/_ / / /_/ /_> < _ /_/ / /_/
/_/ |_|___/|__/ /____/ /_____/ /____/ /_/ /_/ \____//_/|_| _\__, / (_)
/____/
AWS ES cluster available at http://127.0.0.1:9200
Kibana available at http://127.0.0.1:9200/_plugin/kibana/
/usr/local/lib/node_modules/aws-es-kibana/index.js:29
if (err) throw err;
^
TimeoutError: Connection timed out after 1000ms
at ClientRequest.<anonymous> (/usr/local/lib/node_modules/aws-es-kibana/node_modules/aws-sdk/lib/http/node.js:56:34)
at ClientRequest.g (events.js:260:16)
at emitNone (events.js:67:13)
at ClientRequest.emit (events.js:166:7)
at Socket.emitTimeout (_http_client.js:542:10)
at Socket.g (events.js:260:16)
at emitNone (events.js:67:13)
at Socket.emit (events.js:166:7)
at Socket._onTimeout (net.js:318:8)
at _runOnTimeout (timers.js:524:11)
Any ideas?
(I noticed that you just posted this today-- imagine the chance that I was searching for this very thing, and it appeared today!)
I am leveraging a proxy solution upstream of this service. Unfortunately my proxy inserts some rather large cookies and headers. I believe express is not happy about the size of the request header and is returning a 413 entity too large. Is there any option we can use to tune the request body or headers maximum size? Sorry if this has been asked already, I am not too familiar with NodeJS.
When --user xxx --password xxx, the arguments are ignored and values are read from the environment instead (AUTH_USER/AUTH_PASSWORD or USER/PASSWORD).
Hi,
It would be usefull to push regulary aws-es-kibana on your own Docker hub.
Indeed, I use aws-es-kibana
with Jenkins. I would like to have automaticaly fresh aws-es-kibana
without manual action.
To day, I build docker aws-es-kibana
on my computer and push it on my own docker hub. Jenkins use this image to push on kibana. It's not very clean.
Thanks for advance
The url schema for the dual stack endpoints is different from the old IPv4 only endpoints
Domain endpoint (IPv4)
https://{{domainId}}.{{region}}.es.amazonaws.com
Domain endpoint v2 (dual stack)
https://{{domainId}}.aos.{{region}}.on.aws
Using the new dual stack endpoint currently does not work:
region cannot be parsed from endpoint address, either the endpoint must end in .<region>.es.amazonaws.com or --region should be provided as an argument
usage: aws-es-kibana [options] <aws-es-cluster-endpoint>
Step#1: sudo npm install -g aws-es-kibana
Step#2: export the accessKey and secretKey
Step#3: aws-es-kibana cluster-address
internal/fs/watchers.js:226
throw error;
^
Error: ENOENT: no such file or directory, watch '/Users/mehrajuddin/.aws/credentials'
at FSWatcher. (internal/fs/watchers.js:218:26)
at Object.watch (fs.js:1582:34)
at Object. (/usr/local/lib/node_modules/aws-es-kibana/index.js:211:4)
at Module._compile (internal/modules/cjs/loader.js:1085:14)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
at Module.load (internal/modules/cjs/loader.js:950:32)
at Function.Module._load (internal/modules/cjs/loader.js:790:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:76:12)
at internal/main/run_main_module.js:17:47 {
errno: -2,
syscall: 'watch',
code: 'ENOENT',
path: '/Users/mehrajuddin/.aws/credentials',
filename: '/Users/mehrajuddin/.aws/credentials'
USER is set in most shells, including in Cloud Foundry and Heroku and will override AUTH_USER if set which can be confusing.
$ USER=vcap AUTH_USER=kibana AUTH_PASSWORD=secret aws-es-kibana --region us-gov-west-1 https://localhost:51409
Try to login with kibana/secret and you'll get a 401.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.