rfarley3 / kibana Goto Github PK
View Code? Open in Web Editor NEWKibana 4 configuration index interaction (.kibana index control, import and export of config, dashboards, visualizations, searches)
License: MIT License
Kibana 4 configuration index interaction (.kibana index control, import and export of config, dashboards, visualizations, searches)
License: MIT License
Hello,
Good job on this. Is it possible for you to implement a way we can use SSL (basic auth) hosts?
Thanks
Readme says
"Use without installing: python -m kibana
For any example that follows, replace dotkibana with python -m kibana"
However I'm seeing the following errors if I try to do that:
python -m kibana --help
Traceback (most recent call last):
File "/Users/dubauski/anaconda/lib/python2.7/runpy.py", line 163, in _run_module_as_main
mod_name, _Error)
File "/Users/dubauski/anaconda/lib/python2.7/runpy.py", line 111, in _get_module_details
import(mod_name) # Do not catch exceptions initializing package
File "kibana/init.py", line 1, in
from .dotkibana import DotKibana # noqa
File "kibana/dotkibana.py", line 5, in
from .manager import KibanaManager
File "kibana/manager.py", line 4, in
from elasticsearch import Elasticsearch, RequestError
ImportError: No module named elasticsearch
Thank you for any help with this!
Hello
I unable to start the module.
I have installed Python 2.7.11
Do you think that I need to downgrade the python version?
root@ip-10-13-166-157:/tmp/Kibana# python setup.py install
running install
running bdist_egg
running egg_info
creating kibana.egg-info
writing requirements to kibana.egg-info/requires.txt
writing kibana.egg-info/PKG-INFO
writing top-level names to kibana.egg-info/top_level.txt
writing dependency_links to kibana.egg-info/dependency_links.txt
writing entry points to kibana.egg-info/entry_points.txt
writing manifest file 'kibana.egg-info/SOURCES.txt'
reading manifest file 'kibana.egg-info/SOURCES.txt'
writing manifest file 'kibana.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_py
creating build
creating build/lib
creating build/lib/kibana
copying kibana/__main__.py -> build/lib/kibana
copying kibana/dotkibana.py -> build/lib/kibana
copying kibana/mapping.py -> build/lib/kibana
copying kibana/__init__.py -> build/lib/kibana
copying kibana/manager.py -> build/lib/kibana
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/kibana
copying build/lib/kibana/__main__.py -> build/bdist.linux-x86_64/egg/kibana
copying build/lib/kibana/dotkibana.py -> build/bdist.linux-x86_64/egg/kibana
copying build/lib/kibana/mapping.py -> build/bdist.linux-x86_64/egg/kibana
copying build/lib/kibana/__init__.py -> build/bdist.linux-x86_64/egg/kibana
copying build/lib/kibana/manager.py -> build/bdist.linux-x86_64/egg/kibana
byte-compiling build/bdist.linux-x86_64/egg/kibana/__main__.py to __main__.pyc
byte-compiling build/bdist.linux-x86_64/egg/kibana/dotkibana.py to dotkibana.pyc
byte-compiling build/bdist.linux-x86_64/egg/kibana/mapping.py to mapping.pyc
byte-compiling build/bdist.linux-x86_64/egg/kibana/__init__.py to __init__.pyc
byte-compiling build/bdist.linux-x86_64/egg/kibana/manager.py to manager.pyc
File "build/bdist.linux-x86_64/egg/kibana/manager.py", line 107
print e.error
^
SyntaxError: invalid syntax
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying kibana.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kibana.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kibana.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kibana.egg-info/entry_points.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kibana.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kibana.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/kibana-0.1-py2.7.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing kibana-0.1-py2.7.egg
Copying kibana-0.1-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding kibana 0.1 to easy-install.pth file
Installing dotkibana script to /usr/local/bin
Installed /usr/local/lib/python2.7/site-packages/kibana-0.1-py2.7.egg
Processing dependencies for kibana==0.1
Searching for requests
Reading https://pypi.python.org/simple/requests/
Best match: requests 2.9.1
Downloading https://pypi.python.org/packages/source/r/requests/requests-2.9.1.tar.gz#md5=0b7f480d19012ec52bab78292efd976d
Processing requests-2.9.1.tar.gz
Writing /tmp/easy_install-gfc97i/requests-2.9.1/setup.cfg
Running requests-2.9.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-gfc97i/requests-2.9.1/egg-dist-tmp-KzO04g
creating /usr/local/lib/python2.7/site-packages/requests-2.9.1-py2.7.egg
Extracting requests-2.9.1-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding requests 2.9.1 to easy-install.pth file
Installed /usr/local/lib/python2.7/site-packages/requests-2.9.1-py2.7.egg
Searching for argparse
Reading https://pypi.python.org/simple/argparse/
Best match: argparse 1.4.0
Downloading https://pypi.python.org/packages/source/a/argparse/argparse-1.4.0.tar.gz#md5=08062d2ceb6596fcbc5a7e725b53746f
Processing argparse-1.4.0.tar.gz
Writing /tmp/easy_install-4_d9kj/argparse-1.4.0/setup.cfg
Running argparse-1.4.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-4_d9kj/argparse-1.4.0/egg-dist-tmp-RQl1dv
warning: no previously-included files matching '*.pyc' found anywhere in distribution
warning: no previously-included files matching '*.pyo' found anywhere in distribution
warning: no previously-included files matching '*.orig' found anywhere in distribution
warning: no previously-included files matching '*.rej' found anywhere in distribution
no previously-included directories found matching 'doc/_build'
no previously-included directories found matching 'env24'
no previously-included directories found matching 'env25'
no previously-included directories found matching 'env26'
no previously-included directories found matching 'env27'
zip_safe flag not set; analyzing archive contents...
Moving argparse-1.4.0-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding argparse 1.4.0 to easy-install.pth file
Installed /usr/local/lib/python2.7/site-packages/argparse-1.4.0-py2.7.egg
Searching for elasticsearch
Reading https://pypi.python.org/simple/elasticsearch/
Best match: elasticsearch 2.3.0
Downloading https://pypi.python.org/packages/source/e/elasticsearch/elasticsearch-2.3.0.tar.gz#md5=2550f3b51629cf1ef9636608af92c340
Processing elasticsearch-2.3.0.tar.gz
Writing /tmp/easy_install-eUtR8U/elasticsearch-2.3.0/setup.cfg
Running elasticsearch-2.3.0/setup.py -q bdist_egg --dist-dir /tmp/easy_install-eUtR8U/elasticsearch-2.3.0/egg-dist-tmp-HwLbWG
warning: no files found matching 'elasticsearch/connection/esthrift/Rest-remote'
no previously-included directories found matching 'docs/_build'
no previously-included directories found matching 'test_elasticsearch'
warning: no previously-included files matching '__pycache__' found under directory '*'
warning: no previously-included files matching '*.py[co]' found under directory '*'
zip_safe flag not set; analyzing archive contents...
Moving elasticsearch-2.3.0-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding elasticsearch 2.3.0 to easy-install.pth file
Installed /usr/local/lib/python2.7/site-packages/elasticsearch-2.3.0-py2.7.egg
Searching for urllib3<2.0,>=1.8
Reading https://pypi.python.org/simple/urllib3/
Best match: urllib3 1.14
Downloading https://pypi.python.org/packages/source/u/urllib3/urllib3-1.14.tar.gz#md5=5e1407428ac33b521c71a7ac273b3847
Processing urllib3-1.14.tar.gz
Writing /tmp/easy_install-d_g9UC/urllib3-1.14/setup.cfg
Running urllib3-1.14/setup.py -q bdist_egg --dist-dir /tmp/easy_install-d_g9UC/urllib3-1.14/egg-dist-tmp-tHnS2F
warning: no previously-included files matching '*' found under directory 'docs/_build'
zip_safe flag not set; analyzing archive contents...
Moving urllib3-1.14-py2.7.egg to /usr/local/lib/python2.7/site-packages
Adding urllib3 1.14 to easy-install.pth file
Installed /usr/local/lib/python2.7/site-packages/urllib3-1.14-py2.7.egg
Finished processing dependencies for kibana==0.1
root@ip-10-13-166-157:/tmp/Kibana# python -m kibana --help
Traceback (most recent call last):
File "/usr/local/lib/python2.7/runpy.py", line 151, in _run_module_as_main
mod_name, loader, code, fname = _get_module_details(mod_name)
File "/usr/local/lib/python2.7/runpy.py", line 109, in _get_module_details
return _get_module_details(pkg_main_name)
File "/usr/local/lib/python2.7/runpy.py", line 101, in _get_module_details
loader = get_loader(mod_name)
File "/usr/local/lib/python2.7/pkgutil.py", line 464, in get_loader
return find_loader(fullname)
File "/usr/local/lib/python2.7/pkgutil.py", line 474, in find_loader
for importer in iter_importers(fullname):
File "/usr/local/lib/python2.7/pkgutil.py", line 430, in iter_importers
__import__(pkg)
File "kibana/__init__.py", line 1, in <module>
from .dotkibana import DotKibana # noqa
File "kibana/dotkibana.py", line 5, in <module>
from .manager import KibanaManager
File "kibana/manager.py", line 107
print e.error
^
SyntaxError: invalid syntax
root@ip-10-13-166-157:/tmp/Kibana#
Thank you :)
Any thoughts about setting the output message level from the CLI? I can put together a quick PR if that helps.
After refreshing the mappings for my logstash-* index, I get the following error in Kibana: 'Zero or negative time interval not supported'. This because after refreshing the mapping, the timeFieldName has been removed from the index-pattern by this tool:
before:
{
"_index" : ".kibana",
"_type" : "index-pattern",
"_id" : "logstash-*",
"_version" : 4,
"found" : true,
"_source" : {
"fields" : " << FIELDS REMOVED BECAUSE TOO LONG >>"
"customFormats" : "{}",
"timeFieldName" : "@timestamp",
"title" : "logstash-*"
}
}
after:
{
"_index" : ".kibana",
"_type" : "index-pattern",
"_id" : "logstash-*",
"_version" : 4,
"found" : true,
"_source" : {
"fields" : " << FIELDS REMOVED BECAUSE TOO LONG >>"
"customFormats" : "{}",
"title" : "logstash-*"
}
}
If this timeFieldName param is present, it should be retained somehow. Wrote a quick workaround for mapping.py, but this will only work for instances where the timefieldname is @timestamp:
def field_cache_to_index_pattern(self, field_cache):
"""Return a .kibana index-pattern doc_type"""
mapping_dict = {}
mapping_dict['customFormats'] = "{}"
mapping_dict['title'] = self.index_pattern
mapping_dict['timeFieldName'] = '@timestamp'
# now post the data into .kibana
mapping_dict['fields'] = json.dumps(field_cache, separators=(',', ':'))
# in order to post, we need to create the post string
mapping_str = json.dumps(mapping_dict, separators=(',', ':'))
return mapping_str
Folks in manager to connect to elastic search, we should have an option to supply a transport class which help to solve the custom authentication.
I can pick this if you want.
when I generate a new index pattern there is 2 attributes
geoip.longitude and geoip.latitude wich are "float"
kibana want type="number" and format="string"
So, even if I never code in python, I found in mapping.py
https://github.com/rfarley3/Kibana/blob/master/kibana/mapping.py#L257
if key in self.mappings:
if (key == 'type' and
(val == "long" or
val == "integer" or
val == "double")):
val = "number"
I've change it
if key in self.mappings:
if (key == 'type' and
(val == "long" or
val == "float" or
val == "integer" or
val == "double")):
val = "number"
and it's works fine
(even if format still empty and not "sprint" as request)
Question of beginner :
Why export config, viz, search and board but not index-pattern ?
index-pattern is needed to a correct import.
Hi,
We are using ES for indexing cloudtrail data and it hitting 11 million events in a day. While refreshing the index pattern we are getting an error like "[INF] Mapping Mapping is incomplete, doing update". Could you please let us know how can we fix this issue?
When exporting dashboards, saved searches that were only referenced as a visualization dependency (panel[id]['_source']['savedSearchId']
) were exported.
This may already be done, but verify with some tests.
Goal is to facilitate git revisioning of dashboards. Also include sorting of the json.dumps data within some of the object types.
json.dumps(d, sort_keys=True)
As proposed in #9 , I've check dotkibana --export all --pkg
but entries where _type:index-pattern is not in the exported json.
I see data I expected to found in export with this query
GET /.kibana/_search?q=_type:index-pattern
When I export the kibana config, I can't import them in a new kibana because search need to found the index-pattern
So I need to manually create/define the index-pattern.
My need is to prepare the board in my local vagrant, and then share it via export.
Other developers need to import the config into their local installation.
It must also be able to import new board / visually / search in Kibana production and staging.
export kibana settings is great. Importing even greater. So great that I want to use your tool in a containerized, automatic way.
But I do not care about the filename. I do not want the timestamp within the export name, just a simple name. Timestamping/diffing is later done by a VC.
Hello,
Thank you for solving the install issue. My environment uses elasticsearch 2.2 and kibana 4.4.
I want to use your module to update automatically the "logstash-*' index pattern.
root@ip-10-123-208-126:/tmp/Kibana# python -m kibana --refresh 'logstash-*'
Mimicking Kibana GUI refreshFields
Error, couldn't find subkey doc_type[_type]['mapping'][_type]
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/tmp/Kibana/kibana/__main__.py", line 98, in <module>
sys.exit(main())
File "/tmp/Kibana/kibana/__main__.py", line 89, in main
return handle_mapping(dotk, map_cmd)
File "/tmp/Kibana/kibana/__main__.py", line 13, in handle_mapping
return dotk.do_mapping_refresh()
File "kibana/dotkibana.py", line 36, in do_mapping_refresh
return self.mapping.do_refresh()
File "kibana/mapping.py", line 245, in do_refresh
es_cache = self.get_field_cache('es')
File "kibana/mapping.py", line 89, in get_field_cache
self.get_index_mappings(es_mappings[index_name]['mappings'])
TypeError: 'NoneType' object is not iterable
root@ip-10-123-208-126:/tmp/Kibana# python -m kibana --help
Thank you,
Catalin
I started the following project a couple of months ago: https://github.com/jpparis-orange/kibtool. Although I searched github before starting my project, I stumbled upon rfarley3/Kibana right now.
As both projects seem to share common goals, would it be a good way to "merge" the efforts?
urllib.error.HTTPError: HTTP Error 401: Unauthorized
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.