Comments (4)
Whoah that's quite a lot of Regionservers. I am aware that Hannibal is currently not usable for such large installations. However, you came pretty far, I would have guessed Hannibal crashes a lot earlier ;-)
A colleague and me just looked at the request and response and think it can be improved quite a lot:
- Enable GZIP. Unfortunately it doesn't seem to come out of the box with Playframwork
- Add specific API for each graph to reduce payload. As you said: too much logic is done at the view layer. This is a major refactoring though.
However, I fear that this won't be enough. Adjustments have to be done at the UI to be usable for such huge amounts of data. Another bottleneck could be the communication between Hannibal and the Regionservers. And we should consider to make the recording of metric/compactions optional altogether or configurable on a per-region basis.
If someone wants to start working on some of those issues, let me know if I can assist you.
from hannibal.
Sorry for the late response, I started working on it a bit. The main thing I noticed when looking at the server side code was that for each graph we make calls out to every region server for data points. When I created a cache that would use a background thread to update at an interval of every 5 to 10 minutes for example this helped a great deal in response times in loading the graphs. Still the view layer was somewhat of an issue, but the region cache improved performance quite a bit.
A crude patch would be located here:
https://github.com/churrodog/hannibal/commit/13231aa4eb7d14016ea854b74d90cca329a66546
please forgive my scala code, I know its pretty horrible - just a POC.
The view layer refactoring seems like a daunting task as I would probably screw things up, but I would be up for helping create API's for each graph thus the rendering layer refactoring could be done incrementally.
from hannibal.
Thanks very much for the commit, this looks like a great improvement. I also like that you cleaned up the model a bit :-)
However I think we'll have to reduce the intervall of 30 minutes per default and introduce a configuration value for that. Also I have to think about how we'll sync this up with the regioninfo metrics as it doesn't make any sense to record the same cashed values over and over again. Maybe we should change the update of the regioninfo metrics so that they are recorded just after the cache gets updated.
I think we should introduce the following configuration values (dunno wether the defaults are good values though):
regions.update-interval
: default 120s : determines how often the cache get's updated and also how often metrics will get recordedcompactions.update-interval
: default 300s: determines how often the logfiles get fetched and compactions are recorded. 0 means recording the compactions is disabled altogether.
I hope I can implement it soon.
from hannibal.
I added most of your code and added the new configuration values (names differ a bit to the previous proposed ones).
It's available in the next
branch and I will merge it back to master
soon.
from hannibal.
Related Issues (20)
- Easy way to deploy hannibal HOT 4
- [RFE] Show "Region History" for whole cluster HOT 2
- can not load page: MasterNotRunningException HOT 26
- Region view doesn't show correct region when start/end keys are non-printable characters HOT 3
- Value too long for column HOT 2
- Make retention period for metrics configurable ... HOT 1
- Installation instructions in README.markdown links to old version HOT 3
- compaction history chart duration HOT 2
- HBase 0.96 support HOT 5
- Update Compaction Patterns for HBase 0.96 ...
- Compaction View should be disabled when there is no data
- table list panel should be draggable in compaction view HOT 1
- Configurable timezone in Rickshaw chart HOT 1
- Display number of requests per region
- JdbcSQLException: Timeout trying to lock table "METRIC" when visiting /compactions
- Handle different log file format across region servers
- Hannibal with HBase 1.0.0 version HOT 12
- bind port failed, port 9000 is in used already HOT 1
- Link is broken
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hannibal.