Comments (5)
The location is taken from the yarn-site.xml properties set in the job config. Can you check to see if the location of your yarn log aggregation directory has the correct name?
from inviso.
You should be able to check the following property in the job config of any job: 'yarn.nodemanager.remote-app-log-dir'.
It will be something like: hdfs:///path/to/log/agg/dir
If is missing or incorrect, it might have problems locating the file.
from inviso.
Its not that, that location referenced in the url is otherwise accessible just fine. Its just within inviso the nameservice name used in namenode HA configs doesnt seem to work correctly. I have to replace it with the currently active namenode for it to work. Seems like inviso assumes that the part after hdfs:// is a hostname. In namenode HA cases, its going to be a service name that hdfs client knows how to handle correctly but evidently inviso does not.
from inviso.
This might be an issue with how inviso expects the path.
Under the covers, inviso just takes the yarn log aggregation base path and uses the log reader to construct a path. The code for how it computes the path is below (LogService.java). I don't have an HA configuration so it's a little hard to see how it's different, but you might be able to see where this breaks down in your case.
Path logRoot = new Path(conf.get(YarnConfiguration.NM_REMOTE_APP_LOG_DIR, YarnConfiguration.DEFAULT_NM_REMOTE_APP_LOG_DIR));
if(root != null) {
logRoot = new Path(root);
}
Path logPath = LogAggregationUtils.getRemoteNodeLogFileForApp(
logRoot,
ConverterUtils.toApplicationId(appId),
owner,
ConverterUtils.toNodeId(nodeId),
LogAggregationUtils.getRemoteNodeLogDirSuffix(conf)
);
AggregatedLogFormat.LogReader reader = new AggregatedLogFormat.LogReader(conf, logPath);
from inviso.
So the problem is that you're creating a new empty Configuration object and it's missing the necessary options for the mapping to the actual hosts/ports. Good overview of the necessary Configuration options here:
from inviso.
Related Issues (13)
- Error Walking through the README quick start HOT 3
- Documentation for Search and Profiler HOT 5
- Can't run inviso with elastic search 2.3.2 HOT 2
- Document project roadmap and contribution stance HOT 3
- too many data points HOT 1
- How does this work with the docker container? HOT 1
- Error with index_cluster_stats.py HOT 22
- does inviso require genie and aws? HOT 1
- Missing jobs' information HOT 1
- Cluster charts doesn't work HOT 4
- cannot compile with 2.6.0 HOT 2
- Cluster not listed at tab #cluster HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from inviso.