Exception occurred while waiting for cluster table to show up about hannibal HOT 8 CLOSED

Thank you for the bug-report, I guess that it takes too long to fetch the logfiles from your cluster.

It may help if you increase the timeout for fetching the compaction log-files by increasing the value of 'compactions.logfile-fetch-timeout-in-seconds' in conf/application.conf.

I will try to add a more helpful error message and to increase the default timeout.

from hannibal.

I changed the default timeout to 30 seconds and hope that solves your issues. But I have no idea about your browser showing "'done with errors' and a blank page". Can you write back whether that still happens?

from hannibal.

increased the timeout. the exception still appears. btw - it could be that the browser errors are unrelated. do you want me to open a new, separate issue?

Webpage error details

User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)
Timestamp: Fri, 14 Dec 2012 17:34:57 UTC

Message: Object doesn't support this property or method
Line: 3078
Char: 3
Code: 0
URI: http://localhost:9000/assets/javascripts/d3.v2.js

Message: Object doesn't support this property or method
Line: 396
Char: 2
Code: 0
URI: http://localhost:9000/assets/javascripts/rickshaw.js

from hannibal.

It seems from the last comments that the timeout on fetching the region server logs is gone but I'm still commenting here to give some additional background from my experience with this.

The 5 seconds default timeout was very aggressive for us since our log max file size is 200 MB. Getting 200 MB under 5 seconds from the host I was running hannibal on was just not feasible. The real problem though is that such huge files will always take a very long time to download. It's also a problem with the memory consomption since the content of the entire log file is kept in memory before parsing. Streaming the content would really help with that but there was a lower-hanging fruit which is just being a bit smarter in fetching logs.

Pull request #7 is my solution to this problem. The log file is retrieved with a range to start from where the last fetch ended. The initial fetch going back 1024 KB (configurable) prior to the current end of the log. With this code, I've been able to remove my heap size setting of 2048 MB and keep the default value without OutOfMemory errors. Also, the 5 seconds timeout was sufficient for me since in most cases, a fetch would be at most 1024 KB (unless logs produce more than 1 MB per 5 minutes).

from hannibal.

That's true and I am very happy with your solution - maybe we can also reduce the timeout back to 5 seconds.

from hannibal.

I'm actually having second thoughts and I'm not entirely sure if my testing on Friday was done using 5 or 30 seconds. In any case, each log fetch felt like less than 5 seconds.

I will check again tomorrow though but I think it would actually make more sense to have the timeout as lenient as it can be to be able to fetch logs for all region servers inside a refresh interval (which I made more agressive in the pull request #7).

from hannibal.

The fetching of logs is definitely faster now. I'm running this against a 20 nodes cluster and the compaction metrics update takes 8700 ms, well under 5 seconds for each region server.

from hannibal.

Ranlo It seams that you are using Internet Explorer 8, which doesn't support SVG. The graphs from Hannibal are rendered with SVG. Could you try another browser?

from hannibal.