lukas-vlcek / bigdesk Goto Github PK

Make better use of the screen space. Some charts are not that important and useful - make then smaller. For some characteristic consider using of http://omnipotent.net/jquery.sparkline/ for some really small charts. For charts display also actual (the most recent) values.

Filter out unused data from indices status REST endpoint response.

My ElasticSearch cluster is listening private IP address and accepting connections only from frontend server.

How can I use big desk to monitor it in the case when ES server is not visible from the public network?

Hi,

When I try to install the Bigdesk plugin in elasticsearch (0.18.6) I get the following error. Guessing the location changed? Although I see the zipball at /lukas-vlcek/bigdesk/zipball/1.0.0, ES attempts to download from /lukas-vlcek/bigdesk/zipball/v1.0.0 .

/lukas-vlcek/bigdesk/downloads returns "There aren't any downloads yet. But don't worry! You can download the source code as a zip or tarball above".

# sh -x bin/plugin -install lukas-vlcek/bigdesk/1.0.0
+ CDPATH=
+ SCRIPT=bin/plugin
+ [ -h bin/plugin ]
+ dirname bin/plugin
+ ES_HOME=bin/..
+ cd bin/..
+ pwd
+ ES_HOME=/data/elasticsearch
+ [ -x /opt/java/sdk/current/bin/java ]
+ JAVA=/opt/java/sdk/current/bin/java
+ exec /opt/java/sdk/current/bin/java -Delasticsearch -Des.path.home=/data/elasticsearch -cp /data/elasticsearch/lib/* org.elasticsearch.plugins.PluginManager -install lukas-vlcek/bigdesk/1.0.0
-> Installing lukas-vlcek/bigdesk/1.0.0...
Trying https://github.com/downloads/lukas-vlcek/bigdesk/bigdesk-1.0.0.zip...
Trying https://github.com/lukas-vlcek/bigdesk/zipball/v1.0.0...
Failed to install lukas-vlcek/bigdesk/1.0.0, reason: failed to download

I am using ES 1.0.0 and bigdesk cloned with specific tag (v2.4.0) and Chrome browser (33.0.1750.117). After upgrade es to 1.0.0, bigdesk can't show detailed statistic info of each node.

Google Javascript console shows some messages like this:
Failed to load resource: the server responded with a status of 404 (Not Found)

The specific URI is as follows:
http://10.133.1.14:9200/_cluster/nodes/oIsi7nqjRi-aeG6luIz3pQ?all=true&callback=jQuery1710721426458330825_1393185588987&_=1393185609602

The returned message is:
jQuery1710721426458330825_1393185588987({"error":"IndexMissingException[[_cluster] missing]","status":404});

I can verify this request (same returned message) with curl command in the terminal. And if I change the above request URI to:
http://10.133.1.14:9200/_nodes/oIsi7nqjRi-aeG6luIz3pQ?all=true&callback=jQuery1710721426458330825_1393185588987&_=1393185609602
I can get lots of information returned though.

Hi

We have upgraded our ES Clusters to use ElasticSearch 0.90.0.RC1. We have noticed that BigDesk cannot connect to the cluster. Is there an update of BigDesk for this version?

Hello,

first of all thank you very much for the nice interface.

Would it be possible to add the elasticsearch version on the node's information?

Cheers,

Massimo

Hi!

I have a Found cluster with Bigdesk, I can see search, get & other queries but not any scan/scroll queries. Is that expected? We do mostly this type of queries and it would be great to be able to monitor them as well.

Thanks!

I get utc timezoned times. This should be my local ones (+2).

Tried running BigDesk against ES 1.7 and it does not seem to work...

Would like to check if 1.7 is supported, or if anyone was successful running BigDesk against their ES 1.7 cluster before i start hacking mine.

Thanks!

Hi,

Love bigdesk, by the way, but I upgraded to ES 0.90.4 (and also tried .5) and previously (with 0.90.3), bigdesk showed me the stats for my ES node but also for my 4 Logstash procs (nodes) that are running on the same server. This was really handy as I could see the heap usage for ES and logstash.

However now, (and I have latest bigdesk) it shows the ES and logstash nodes but the logstash ones are blank and show no info at all. It has the correct node name but what I notice is it shows the same PID and ID for all (the ES PID/ID).

I know bigdesk is primarily for ES so this is probably not an issue/priority for some but it was a really nice-to-have. I have attached a couple of images to show what I mean.

Is this something that could be fixed? I can see them with the kopf plugin but I preferred it in one place with bigdesk. Thanks.

just visit bigdesk like :
http://domain/context_path/_plugin/bigdesk

and then i can't get the cluster info,because bigdesk send ajax like
http://domain/_plugin/bigdesk

I would like to add explicit authentication as I did for elasticsearch-head - see mobz/elasticsearch-head#57 and Asquera/elasticsearch-http-basic#6

I know it is a stupid work around for ElasticSearch inflexible header customization, but I only need ElasticSearch and two plugins (head+bigdesk). Now I cannot make it working for bigdesk. How can I add something to every request with backbone? The ajaxSetup what I was using seems to be "unavailable" for backbone and using beforeSend would need to change every request/fetch, right?

Also this trick to intercept sync does not add the header to the request for me.

Thanks for awesome Bigdesk. I want to know if i can setup any authentication to access Bigdesk app. Thanks in advance for help.

I installed bigdesk 2.2.0 and i keep getting errors after i select a node from the cluster.

I altered the js to stop throwing errors but im not sure what it was suppose to do.

So in the file SelectedClusterNodeView.js i added the following check at line 611:
if(stats_the_latest.node.indices.cache){
}

In the file bigdesk_charts.js i added similar checks (line 816 for example):
value: snapshot.node.indices.cache ? snapshot.node.indices.cache.filter_size_in_bytes : ""

Also had to make a find and replace to replace all "+snapshot" to "snapshot". Not sure what the "+" was for.

See elastic/elasticsearch#7916 (or here: https://groups.google.com/d/msg/elasticsearch/jf_c1nyyeAc/cUcSyo58lDkJ)

Hi
I seem to only be able to see the cluster pack diagram if the cluster is named elastic search, if it is named anything else it just loads the node names on top of each other:

I'm using BigDesk 2.5.0 and ElasticSearch 1.5.0.

Is this a bug or could it be something to do with my config? when removing the "cluster.name" from elasticsearch.yml the diagram shows up fine

We pull a lot of data from REST endpoints. This can cause a serious performance issues when connected to large clusters. We do not make use of all the data anyway for now. Especially cluster state metadata can be large.

Hello,

I have installed latest ES and latest version of bigdesk (as of today).
I also use Apache as proxy layer.

When I connect to the url
http://127.0.0.1:8080/_plugin/bigdesk/#nodes

I see only page basic page with connection and a message:
"No cluster connected."

After debugging I found out that one request is not created correctly for me:

Last two requests in debug are:
/cluster/health?callback=jQuery17106085601344238967_1406811659076&=1406811659388
and then following:
?callback=jQuery17106085601344238967_1406811659077&_=1406811659504

The first one will get response correctly, but the second will not provide info and will cause an error. It's missing "_cluster/health"

When I add /_cluster/health? before the second request, it is served and response is OK.

Is this issue for someone else?

I tried FF 31.0
and Chrome 35.0.1916.153

Do you have any suggestions?

Thanks,
Rado

Getting a warning complaining of version mismatch. Bigdesk won't seem to let me connect after acknowledging the warning.

Installed ES 0.20.5, and lukas-vlcek/bigdesk/2.2.0

Here is the warning displayed in the popup:

Bigdesk may not work correctly!
Found ES node version: 0.20.5
Requires ES node version: 0.20.x

Using Elasticsearch 0.90.5 and the current master version of bigdesk, I cannot seem to get detailed statistics on the nodes.

After loading the bigdesk dashboard, I click one of the nodes and nothing appears to happen. Checking the Javascript log I find:

Uncaught TypeError: <template>:1
 >> CPU vendor: {{#cpu.vendor}}{{cpu.vendor}}{{/cpu.vendor}}{{^cpu.vendor}}n/a{{/cpu.vendor}}<br>CPU model: {{#cpu.model}}{{cpu.model}} ({{cpu.mhz}} MHz){{/cpu.model}}{{^cpu.model}}n/a{{/cpu.model}}<br>CPU total logical cores: {{#cpu.total_cores}}{{cpu.total_cores}}{{/cpu.total_cores}}{{^cpu.total_cores}}n/a{{/cpu.total_cores}}<br>CPU cache: {{#cpu.cache_size}}{{cpu.cache_size}}{{/cpu.cache_size}}{{^cpu.cache_size}}n/a{{/cpu.cache_size}}

This is then followed by a different explanation for the two browsers I use:
Chrome:

Cannot read property 'cpu' of undefined

Safari:

'undefined' is not an object (evaluating 'context[names[j++]]')

I assume it is the /stats API that is used so I have created a gist of that response from one of the nodes:
https://gist.github.com/martinskans/8705555

Information that could also be useful is
that the Elasticsearch nodes themselves run on Ubuntu on virtual Amazon instances. The node the call is made through is a "no data" and "no master" node which is just used as an access point to the cluster.

I will be happy to help with any more information I can provide!

Pre ES 2.x see elastic/elasticsearch#4378
Post ES 2.x see elastic/elasticsearch#12408

Using Chrome on Windows (will verify on other OSes soon).

While editing the default "http://localhost:9200" text in the "ES node REST endpoint" textbox, pressing Enter causes the page to refresh and the textbox revert to their default. The behavior should be execute "Connect".

Happy to see BigDesk on 0.19+

I get this for all OS charts; #2,#3,#4 of the PROCESS charts and the FILESYSTEM charts. Could it be because the host is FreeBSD?

Thanks.

When starting with 1hour limit no graphs were shown. Switching back to 15min (and then to 1hour again) I can see the JVM graph.

my system graph stays empty, maybe this is a jvm issue as e.g. the cpu usage is not shown even in jvisualvm (although the mem usage is shown there)

It'd be great if the refresh time could be saved in a cookie so that a browser/tab restart doesn't reset the timing down to 2 seconds, as that's not really a sane default when running bigdesk over a WAN link due to the amount of data it transfers.

It would be useful to have an option to store host and port values in URL so that users can bookmark particular settings and clusters and open BigDesk quickly for them.

Hi,

After upgrading ES to 0.20.2 i'm missing a few JVM graphs.
All 4 OS graphs and the 3 last Process graphs are grayed out.
Also the Filesystem information and graphs are missing.

Some of the index info is there along with the correct node name and cluster name.

I do not see any data for system, jvm, file descriptors and cache size graph...

The node info is fetched using /_nodes in RC1 and not /_cluster/nodes as in previous versions. This appears to only be used in NodeInfo.js line 25

I was wondering, is there a way to apply some sort of authentication mechanism for accessing the bigdesk interface? Currently, anyone who knows the address will be able to access it.

Thanks

I was doing some load testing and noticed in the Elasticsearch logs that i was getting No SPACE left on device. But this is something i couldn't see on the GRAPHS in Bigdesk,
-->Is there a way i can see this error on Big desk?

Also the error reported
java.io.FileNotFoundException: /var/elasticsearch/elasticsearch-0.90.3/data/elasticsearch5/nodes/0/indices/dw/3/index/_dzg.si (Too many open files)
--> Is there a way to determine this error in big desk?

Thanks

When pointing BigDesk (current master as of a few minutes ago) against a ES cluster I get this error when clicking on Nodes:

Uncaught TypeError: Cannot read property 'getMasterNodeId' of null bigdeskApp.js:372
Backbone.Router.extend.nodes_master bigdeskApp.js:372
(anonymous function) backbone-min.js:25
(anonymous function) backbone-min.js:30
b.some.b.any underscore-min.js:14
f.extend.loadUrl backbone-min.js:30
f.extend.checkUrl backbone-min.js:30
f.event.dispatch jquery-1.7.1.min.js:3
h.handle.i jquery-1.7.1.min.js:3

With Cluster I get:

Uncaught TypeError: Cannot read property 'get' of undefined ClusterHealthView.js:24
Backbone.View.extend.initialize ClusterHealthView.js:24
g.View backbone-min.js:32
d backbone-min.js:36
clusterView.render bigdeskApp.js:67
Backbone.Router.extend.cluster bigdeskApp.js:360
(anonymous function) backbone-min.js:25
(anonymous function) backbone-min.js:30
b.some.b.any underscore-min.js:14
f.extend.loadUrl backbone-min.js:30
f.extend.checkUrl backbone-min.js:30
f.event.dispatch jquery-1.7.1.min.js:3
h.handle.i jquery-1.7.1.min.js:3

The one query I see it do in Chrome's network trace is:

http://test:9200/_cluster/health?callback=jQuery171004769798368215561_1406574553380&_=1406574561869

which when called directly returns

{
  "cluster_name": "test",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 4,
  "number_of_data_nodes": 4,
  "active_primary_shards": 9,
  "active_shards": 18,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0
}

Make sure Bigdesk try to use AJAX (with CORS) if JSONP not allowed.
See: elastic/elasticsearch#6795

Bigdesk does not support Elasticsearch 2.x.
We need to fix the version check on startup (or better we can add an option to completely suppress it because it is doing more harm than good).

You used to show the IP address of the node, which was useful.

Please can we see that somewhere again?

hi
im getting JS error:
Error: Invalid value for attribute r="NaN" d3.v2.min.js:2131
ES version: 0.20.6
bigdesk version: well master, dunno where to find your versioning
installed ES and bigdesk out of the box, no data indexed yet.

For more details see elastic/elasticsearch#1113

If bigdesk remains open for long periods, memory keeps growing. For example, in Chomium browser, the tab used big-desk consumed about 5Gb of RAM in about 5 hours.
To free the memory, I have to reopen and reconnect bigdesk to ElasticSearch.

I installed the plugin just fine but I can't find anywhere the url to access the pretty graphs and such I see. It'd be nice to add that to the front page somewhere.

For more details see elastic/elasticsearch#1115

Hi!

In Issue #21 you said you would release a ZipFile for bigdesk. This - as it appears - hasn't happened yet. I want to use bigdesk for monitoring my ES instances on CentOS. For that I want to create an RPM. I'm normally used to Debian systems. But circumstances only allow me to use CentOS. Here is a Spec file. It is a Modification of the hadoop plugin package found on @tavisto's account under the elasticsearch-rpms repository. I would like to have it reviewed by you. In addition I would like to have a 1.0.0 release Zip on the repo. This would make the rpm building much simpler. Anyway here is the spec:

%define debug_package %{nil}
%define base_install_dir %{_javadir}/%{name}
%define site_install_dir %{base_install_dir}/plugins/bigdesk/_site
# Avoid running brp-java-repack-jars
%define __os_install_post %{nil}

Name:           elasticsearch-plugin-bigdesk
Version:        1.0.0
Release:        1%{?dist}
Summary:        ElasticSearch plugin to use BigDesk

Group:          System Environment/Daemons
License:        ASL 2.0
URL:            https://github.com/elasticsearch/elasticsearch-bigdesk

Source0:        https://github.com/lukas-vlcek/bigdesk/zipball/%{version}
BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
BuildArch:      noarch

Requires:       elasticsearch >= 0.19

%description
Live charts and statistics for ElasticSearch cluster.

%prep
rm -fR %{name}-%{version}
%{__mkdir} -p %{name}-%{version}
cd %{name}-%{version}
%{__mkdir} -p plugins
unzip %{SOURCE0} -d plugins/bigdesk

%build
true

%install
rm -rf $RPM_BUILD_ROOT
cd %{name}-%{version}
%{__mkdir} -p %{buildroot}/%{site_install_dir}
%{__mkdir} -p %{buildroot}/%{site_install_dir}/css
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js/lib
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js/lib/mustache
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js/lib/highcharts
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js/jQueryMustache
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js/jquery
%{__mkdir} -p %{buildroot}/%{site_install_dir}/js/parseUri/
%{__mkdir} -p %{buildroot}/%{site_install_dir}/images
%{__install} -D -m 755 plugins/bigdesk/index.html -t %{buildroot}/%{site_install_dir}/
%{__install} -D -m 755 plugins/bigdesk/README.textile -t %{buildroot}/%{site_install_dir}/
%{__install} -D -m 755 plugins/bigdesk/screenshot -t %{buildroot}/%{site_install_dir}/
%{__install} -D -m 755 plugins/bigdesk/css/bigdesk.css -t %{buildroot}/%{site_install_dir}/css/
%{__install} -D -m 755 plugins/bigdesk/css/reset.css -t %{buildroot}/%{site_install_dir}/css/
%{__install} -D -m 755 plugins/bigdesk/images/1license.txt -t %{buildroot}/%{site_install_dir}/images/
%{__install} -D -m 755 plugins/bigdesk/images/award_star_gold.png -t %{buildroot}/%{site_install_dir}/images/
%{__install} -D -m 755 plugins/bigdesk/images/es-logo.png -t %{buildroot}/%{site_install_dir}/images/
%{__install} -D -m 755 plugins/bigdesk/js/bigdesk.js -t %{buildroot}/%{site_install_dir}/js/
%{__install} -D -m 755 plugins/bigdesk/js/charts.js -t %{buildroot}/%{site_install_dir}/js/
%{__install} -D -m 755 plugins/bigdesk/js/lib/highcharts/highcharts.js -t %{buildroot}/%{site_install_dir}/js/lib/highcharts/
%{__install} -D -m 755 plugins/bigdesk/js/lib/jQueryMustache/License.txt -t %{buildroot}/%{site_install_dir}/js/jQueryMustache/
%{__install} -D -m 755 plugins/bigdesk/js/lib/jQueryMustache/jQueryMustache.js -t %{buildroot}/%{site_install_dir}/js/jQueryMustache/
%{__install} -D -m 755 plugins/bigdesk/js/lib/jquery/jquery-1.4.4.min.js -t %{buildroot}/%{site_install_dir}/js/lib/jquery/
%{__install} -D -m 755 plugins/bigdesk/js/lib/mustache/LICENSE -t %{buildroot}/%{site_install_dir}/js/lib/mustache/
%{__install} -D -m 755 plugins/bigdesk/js/lib/mustache/mustache.js -t %{buildroot}/%{site_install_dir}/js/lib/mustache/
%{__install} -D -m 755 plugins/bigdesk/js/lib/parseUri/parseuri.js -t %{buildroot}/%{site_install_dir}/js/lib/parseUri/

%files
%defattr(-,root,root,-)
%dir %{base_install_dir}/plugins/bigdesk
%{base_install_dir}/plugins/bigdesk/*   

%changelog
* Tue Aug 14 2012 Andreas Marschke 1.0.0-0
- Initial package

Thanks and happy Hacking!

Currently, if user switches between nodes then the history of the node is lost and new node values are loaded from scratch. It would be better to keep stats values in memory of all available nodes to allow smooth switching between nodes in UI.

Hi Lukas,

Seems 2.4.0 doesn't support es 1.2.1 ? Or am I missing something?

Spancer

Current version seems to be running fine on 1.4.x but this version is missing from the Support matrix. Worth updating.

I'm running BigDesk as an ES plugin, and I am forwarding port 9200 through port 80 using nginx. Ie. http://localhost:9200/ is reachable from the outside as http://ip-number/elasticsearch/

It would be nice if the default suggested endpoint is the part of the URL prior to /_plugin/bigdesk/.

Ie. if the BigDesk URL is http://ip-number/elasticsearch/_plugin/bigdesk/ then the suggested ES REST endpoint should be: http://ip-number/elasticsearch/

Some admin REST API has been changed.

I don't see 1.4 listed in the support matrix, so I wanted to file this bug as a tracker for 1.4 support. Thanks!

Hi loving the plugin! We started using it about a month ago and it has allowed my team to get a real time view on what our nodes are doing. Hoping I can help to fix an issue we have with using it on more sizable clusters.

For any of our larger clusters we have many customers, and many indexes (~100). Calling "_cluster/state" is a guaranteed way to kill a browser. it is currently over 11 mega bytes of data. Chrome kills the page, other browsers even crash.

It maybe possible to use the filters on cluster state to reduce the size while still maintaining functionality?

http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/cluster-state.html

If I switch on no metadata filter the size drops down to 665Kb. It's much faster and possibly would scale to clusters much larger than ours. Not sure that's doable.

I don't want to preemptively fork and code. If you think this is something that warrants attention let me know. I will be extremely happy to add my coding efforts.