Use username in compactions.logfile-url-pattern about hannibal HOT 10 CLOSED

If the username can't be queried easily, we could always just hit http://%hostname%:%infoport%/logs/ on init and figure out what the file pattern is from the list of logfiles.

from hannibal.

Great idea! This way the user would not even need to configure the logfile pattern.
However it could be a bit tricky to find out the pattern when there are multiple services logging to the same directory (like hbase 'master' 'regionsserver' 'rest' etc) or when there is logrotation enabled.

from hannibal.

I'm going to make a first attempt using the following pattern:
"(/logs/.*regionserver.*[.]log)"
On init, I'm going to find the matching log file on one of the region servers and then replace the regionserver's hostname and port values by the matching token to build the pattern then used for all region servers.

I'm also wondering if we can get rid of the compactions.loglevel-url-pattern since it looks to me like it's unlikely to vary very much. If we still want to keep this configurable, I would vote to rename the configs with the .override suffix and have them be null by default. Only when defined would they be used over anything that's hardcoded/automatically discovered.

Thoughts?

from hannibal.

Yes, that pattern should work for the most cases. Maybe we could remove the compactions.logfile-url-pattern completly and replace it with a new option for the new logfile-name-pattern with default value of "(/logs/.*regionserver.*[.]log)" ?

You are correct about the compactions.loglevel-url-pattern - I guess most people don't use the functionality of letting HBase LogLevels be changed by Hannibal. And even when they do, that pattern is very unlikely to change. However most values of 'application.conf' are unlikely to change - or don't even make sense for Hannibal (take application.langs for example).
I like your idea of an override conf, but I don't know wether it's worth the effort.

from hannibal.

I'll just focus on the logfile-url-pattern then. That pattern will be the new configuration. This configuration is going to be used to resolve an actual "pattern with tokens" so that each log fetch doesn't require a list prior to the operation.

I have some code already but I need to test it when I have a few minutes to spare.

from hannibal.

My lack of experience with sbt/ivy/play is slowing me down here. I tried using vagrant and the hbase092 VM to test my changes but for some strange reason, it can't find the hadoop-common, hbase dependencies. It's definitely looking up the right paths in the cloudera repository:

[warn] ==== local: tried
[warn]   /home/vagrant/play-2.0.4/framework/../repository/local/org.apache.hbase/hbase/0.92.1-cdh4.1.2/ivys/ivy.xml
[warn] ==== Typesafe Releases Repository: tried
[warn]   http://repo.typesafe.com/typesafe/releases/org/apache/hbase/hbase/0.92.1-cdh4.1.2/hbase-0.92.1-cdh4.1.2.pom
[warn] ==== Typesafe Snapshots Repository: tried
[warn]   http://repo.typesafe.com/typesafe/snapshots/org/apache/hbase/hbase/0.92.1-cdh4.1.2/hbase-0.92.1-cdh4.1.2.pom
[warn] ==== Cloudera Public Repository: tried
[warn]   http://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hbase/hbase/0.92.1-cdh4.1.2/hbase-0.92.1-cdh4.1.2.pom
[warn] ==== Mockito Core: tried
[warn]   http://repo2.maven.org/maven2/org/mockito/mockito-core/org/apache/hbase/hbase/0.92.1-cdh4.1.2/hbase-0.92.1-cdh4.1.2.pom
[warn] ==== Mockito All: tried
[warn]   http://repo2.maven.org/maven2/org/mockito/mockito-all/org/apache/hbase/hbase/0.92.1-cdh4.1.2/hbase-0.92.1-cdh4.1.2.pom
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/org/apache/hbase/hbase/0.92.1-cdh4.1.2/hbase-0.92.1-cdh4.1.2.pom
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::                       
[warn]  ::          UNRESOLVED DEPENDENCIES         ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.apache.hadoop#hadoop-common;2.0.0-cdh4.1.2: not found
[warn]  :: org.apache.hbase#hbase;0.92.1-cdh4.1.2: not found
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::

The only way I can get this to run is to locally do play run outside of the VM but my hbase config is setup correctly to use the VM's hbase. It's probably going to be faster if I test this against a real cluster tomorrow.

from hannibal.

The changes are available in the 'next' branch. I will merge them back to the master soon.

It works pretty well, the only thing I noticed is that when the pattern doesn't match, an NPE is raised. I would like to give the user a hint that he should review his pattern in that case. I try to do that on my own.

Regarding the Ivy-Problems: That's weird, did you check that the network is working correclty from inside the VM? Try to 'ping google.com' or something like that.

from hannibal.

I can clean that up too. I would have the init either fail/pass in a consequence of being able to discover the logFileUrlPattern. On init failure, I'll log an error and NOT schedule the compaction update.

How does that sound?

As for the dependencies problem, I did have network from inside the VM. I do wonder though if I had exported the hbase version prior to trying to run it from the VM. I'll check again this evening.

from hannibal.

I made the changes proposed this morning.

I also checked on the ivy in the VM issue and I can confirm that I had network and that I had exported the hbase version correctly. Not sure what the problem but I'm ok doing the testing with the VM for now.

from hannibal.

fix is now in the 'master'-branch

from hannibal.