netflix / inviso Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Everything in the UI is just empty. I tried changing settings.py and then running jes,py, but it dies with 'connection refused'.
I'm following the instructions for the quick start and I get to the point where I run the jes.py
It fails with the following
python jes.py
ERROR:inviso.jes:[Errno 32] Broken pipe
Traceback (most recent call last):
File "jes.py", line 35, in main
monitor.run()
File "/home/liamm/invisio/inviso/jes/inviso/monitor.py", line 295, in run
for f in listing:
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/client.py", line 139, in ls
recurse=recurse):
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/client.py", line 1072, in _find_items
fileinfo = self._get_file_info(path)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/client.py", line 1202, in _get_file_info
return self.service.getFileInfo(request)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/service.py", line 35, in <lambda>
rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.__dict__[method], request)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/service.py", line 41, in call
return method(self.service, controller, request)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 267, in <lambda>
self._StubMethod(inst, method, rpc_controller, request, callback))
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod
method_descriptor.output_type._concrete_class, callback)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 411, in CallMethod
self.send_rpc_message(method, request)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 309, in send_rpc_message
self.write_delimited(rpc_request_header)
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 238, in write_delimited
self.write(encoder._VarintBytes(len(data)))
File "/home/liamm/invisio/venv/lib/python2.6/site-packages/snakebite/channel.py", line 235, in write
self.sock.send(data)
error: [Errno 32] Broken pipe
I am able to follow the instructions and get all the way to the "python jes.py" part in step 6 but that blows up with the error below. I am trying to get this working with a locally installed cluster and not aws. I also don't have genie. what do I need to do to get this to work?
(venv)[skhehra@testvm jes]$ python jes.py
ERROR:inviso.jes:[Errno 32] Broken pipe
Traceback (most recent call last):
File "jes.py", line 35, in main
monitor.run()
File "/home/skhehra/inviso/jes/inviso/monitor.py", line 295, in run
for f in listing:
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/client.py", line 139, in ls
recurse=recurse):
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/client.py", line 1072, in _find_items
fileinfo = self._get_file_info(path)
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/client.py", line 1202, in _get_file_info
return self.service.getFileInfo(request)
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/service.py", line 35, in
rpc = lambda request, service=self, method=method.name: service.call(service_stub_class.dict[method], request)
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/service.py", line 41, in call
return method(self.service, controller, request)
File "/home/skhehra/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 267, in
self._StubMethod(inst, method, rpc_controller, request, callback))
File "/home/skhehra/venv/lib/python2.6/site-packages/google/protobuf/service_reflection.py", line 284, in _StubMethod
method_descriptor.output_type._concrete_class, callback)
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/channel.py", line 411, in CallMethod
self.send_rpc_message(method, request)
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/channel.py", line 309, in send_rpc_message
self.write_delimited(rpc_request_header)
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/channel.py", line 238, in write_delimited
self.write(encoder._VarintBytes(len(data)))
File "/home/skhehra/venv/lib/python2.6/site-packages/snakebite/channel.py", line 235, in write
self.sock.send(data)
error: [Errno 32] Broken pipe
Strangely nothing is shown on the web app. In particular there is nothing to select under the Cluster drop-down menu and it just shows Nothing selected
. So probably the webapp isn't correctly configured yet. I followed the guide and also changed settings.py to the following:
clusters = [
Cluster(id='clustername', name='clustername', host=socket.getfqdn())
]
and
clusters = [
Cluster(id='clustername', name='clustername', host='hostname')
]
to no avail. Any ideas?
I'm looking at using Inviso for our clusters, but since it's been about 2 years since there was any notable changes there are some obvious concerns about the status of the project.
It would be useful if you could document (in the README.md file?) what the roadmap and status of the project is (have there been further changes at Netflix that are yet to be released? Has there literally been no changes in this time?) and what the project's stance towards contributions is.
Hello folks,
I'm trying to run inviso with elastic search 2.3.2.
While trying to run this command:
curl -XPUT http://localhost:9200/inviso -d @inviso/elasticsearch/mappings/config-settings.json
I get the following error:
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"Mapping definition for [_timestamp] has unsupported parameters: [store : true]"}],"type":"mapper_parsing_exception","reason":"Failed to parse mapping [default]: Mapping definition for [_timestamp] has unsupported parameters: [store : true]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [_timestamp] has unsupported parameters: [store : true]"}},"status":400}
How can we adjust inviso in order to have it compatible with elastic search 2.3.2 ?
I search a user's job information, but the "Stop", "Duration", "Links", "Workflow ID" and "Genie Name" information is not showing up. Moreover, the "Job Status" shows "UNKNOWN".
Dose it have any connection with my hadoop configuration?
Which file finishes the work of gathering jobs' information so I can check up?
This may be related to gradle/maven but I cannot seem to get this build for hadoop 2.6.0. Changing the versions in trace-mr2/build.gradle to 2.6.0 crashes the build process, stating
/srv/invisio/inviso/trace-mr2/src/main/java/com/netflix/bdp/inviso/fs/S3DelegateFS.java:29: error: package org.apache.hadoop.fs.s3native does not exist
Is there any other place I have to change the hadoop version? Or something else I am missing?
Hi,
I've configured both python scripts and they work (I see new data in elastic search), tabs 'Search' and 'Profiler' work, but 'Cluster' doesn't.
When opening Cluster tab, there is only one request to ElasticSearch:
POST /inviso-cluster/metrics/_search?size=0
with form data:
{"query":{"bool":{"must":[{"range":{"timestamp":{"gte":1416559166153,"lte":1417768766154}}}]}},"aggs":{"clusters":{"terms":{"field":"cluster","size":50}}}}:
and response:
{"took":5,"timed_out":false,"_shards":{"total":6,"successful":6,"failed":0},"hits":{"total":1077,"max_score":0.0,"hits":[]},"aggregations":{"clusters":{"buckets":[{"key":"hefajstos","doc_count":1077}]}}}
What could be wrong? I don't see any errors neither in tomcat log or elasticsearch log.
I'm following the instructions for the QuickStart.
When it comes to step 6 with the command " python index_cluster_stats.py ", it fails with the following. I am confused with the port 9026, it just shows up suddenly, I do not know when it get configured. Should I change the port?
ERROR:inviso.cluster:Error processing: cluster_1
ERROR:inviso.cluster:HTTPConnectionPool(host='taoran1', port=9026): Max retries exceeded with url: /ws/v1/cluster/apps?state=RUNNING (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
Traceback (most recent call last):
File "index_cluster_stats.py", line 90, in index_stats
index_apps(es, cluster, info)
File "index_cluster_stats.py", line 17, in index_apps
apps = requests.get('http://%s:%s/ws/v1/cluster/apps?state=RUNNING' % (cluster.host, '9026'), headers = {'ACCEPT':'application/json'}).json().get('apps')
File "/home/venv/lib/python2.6/site-packages/requests/api.py", line 55, in get
return request('get', url, *_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, *_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/sessions.py", line 456, in request
resp = self.send(prep, *_send_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/sessions.py", line 559, in send
r = adapter.send(request, *_kwargs)
File "/home/venv/lib/python2.6/site-packages/requests/adapters.py", line 375, in send
raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='taoran1', port=9026): Max retries exceeded with url: /ws/v1/cluster/apps?state=RUNNING (Caused by <class 'socket.error'>: [Errno 111] Connection refused)
Also, when conducted the command "python jes.py", it came up with the followings, I do not know whether it matters for the Error above.
INFO:inviso-monitor:Publishing event: (cluster_1) job_1413958950234_0008 2014-10-23T02:37:35+00:00
INFO:inviso-handler:Processing 1 events
WARNING:inviso-handler:No trace info available for hdfs://taoran1:9000/tmp/hadoop-yarn/staging/history/done/2014/10/23/000000/job_1413958950234_0008-1414031808636-hadoop-QuasiMonteCarlo-1414031855566-4-1-SUCCEEDED-default-1414031818339.jhist
INFO:inviso-handler:Indexing 1 documents
INFO:inviso-handler:Events complete.
When clicking on "Open Logs" link on a task's information on the profile page, if I click on open logs I end up with "HTTP Status 500 - java.lang.reflect.InvocationTargetException". The root cause is
java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
If I change the url in the address bar at this point to NamenodeHostname:8020 it works ok. i.e. the following works. i have confirmed that the node inviso is running has the correct hadoop client conf and is able to handle hdfs://nameservice1 type of urls. Any ideas how i can get inviso to be able to as well?
replacing
?fs=hdfs://nameservice1&root
with
?fs=hdfs://NamenodeHostname:8020&root
How to use search and profiler ? I just get blank page? Can someone provide brief example ?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.