Issues running priam on AWS instances: Error ListenerStart about priam HOT 14 CLOSED

Can you look at tomcat.log to find any errors?

from priam.

Looks like Priam war was not deployed can u check for permission issues if any? And also Step 3 looks wrong to me we should copy the jar file to the lib directory inside cassandra home.

from priam.

There are no errors in the tomcat.log file. When I start the server, I get the following error in catalina.out.
"SEVERE: Error listenerStart
Feb 29, 2012 1:56:09 PM org.apache.catalina.core.StandardContext start
SEVERE: Context [/Priam] startup failed due to previous errors"

I had put the Priam.jar in the apache-cassandra/lib folder. Still couldn't bring the Priam application up. I used cassandra version 0.8.9 and tomcat 6 on amazon linux ami.

Can you please let us know if I am missing anything.

from priam.

Have you created simpledb domain as specified here: https://github.com/Netflix/Priam/wiki/Setup? I suspect it might be a configuration thing/cred thing. I have added some logging and default configurations. Can you use the latest source code and look for activity in your logs?

from priam.

Also, as Vijay mentioned is tomcat able to successfully deploy Priam.war without perm issues? Also set the logger to debug and provide any logs in tomcat.log.

from priam.

I got the following error in the localhost.log when I deployed the Priam.war in the tomcat server.:
Mar 2, 2012 1:45:08 PM org.apache.catalina.core.StandardContext listenerStart
SEVERE: Exception sending context initialized event to listener instance of class com.netflix.priam.defaultimpl.InjectedWebListener
java.lang.RuntimeException
at com.netflix.priam.defaultimpl.InjectedWebListener.getInjector(InjectedWebListener.java:35)
at com.google.inject.servlet.GuiceServletContextListener.contextInitialized(GuiceServletContextListener.java:45)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4206)
at org.apache.catalina.core.StandardContext.start(StandardContext.java:4705)
at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:799)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:779)
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:601)
at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:943)
at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:778)
at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:504)
at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1317)
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:324)
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1065)
at org.apache.catalina.core.StandardHost.start(StandardHost.java:840)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1057)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463)
at org.apache.catalina.core.StandardService.start(StandardService.java:525)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:754)
at org.apache.catalina.startup.Catalina.start(Catalina.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414)
Caused by: java.lang.NullPointerException
at java.net.URLEncoder.encode(URLEncoder.java:205)
at com.amazonaws.util.HttpUtils.urlEncode(HttpUtils.java:37)
at com.amazonaws.auth.AbstractAWSSigner.getCanonicalizedQueryString(AbstractAWSSigner.java:141)
at com.amazonaws.auth.QueryStringSigner.calculateStringToSignV2(QueryStringSigner.java:128)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:78)
at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:5569)
at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:2493)
at com.netflix.priam.defaultimpl.PriamConfiguration.populateASGName(PriamConfiguration.java:176)
at com.netflix.priam.defaultimpl.PriamConfiguration.setupEnvVars(PriamConfiguration.java:163)
at com.netflix.priam.defaultimpl.PriamConfiguration.intialize(PriamConfiguration.java:144)
at com.netflix.priam.defaultimpl.InjectedWebListener.getInjector(InjectedWebListener.java:30)
... 25 more
Mar 2, 2012 1:45:09 PM org.apache.catalina.core.ApplicationContext log
INFO: ContextListener: contextInitialized().

Steps I had followed:

1. Cconfigured the PriamProperties in the Simple DB. Following is the xml I got when I queried it:
   Item1priam.jmx.port7199priam.thrift.port9160 priam.storage.port7000priam.multiregion.enablefalsepriam.endpoint_snitchorg.apache.cassandra.locator.Ec2Snitch priam.clusternameTestPriampriam.memory.compaction.limit128 MB priam.zones.availableus-east-1apriam.direct.memory.size50Gpriam.compaction.throughput8 MB/sec priam.heap.newgen.size2Gpriam.heap.size8G priam.data.location/var/lib/cassandra/data priam.cass.startscript/etc/init.d/cassandra start priam.cache.location/var/lib/cassandra/saved_caches priam.cass.stopscript/etc/init.d/cassandra stop priam.commitlog.location/var/lib/cassandra/commitlog priam.cass.home/opt/cassandra/apache-cassandra-0.8.10priam.backup.hour12priam.restore.threads8 priam.restore.closesttokenfalsepriam.restore.prefixprefixpriam.upload.throttleMax INT priam.backup.chunksizemb10 MBpriam.backup.incremental.enabletrue priam.s3.base_dirbackup priam.backup.threads2priam.restore.keyspacesnullpriam.s3.bucketcassandra-archive priam.restore.snapshotnull2a57daeb-c7be-60b5-986a-2893f5f7575d0.0000228616

Not sure if the following values are correct:
priam.restore.prefix = prefix
priam.upload.throttle = "Max INT"(or the value equivalent to it)
priam.restore.snapshot = null

Where are we specifying the instance type as m1.large?
2. Created domain InstanceIdentity in SimpleDB. But did not add any attributes to it(as it is not mentioned in the set up)
3. Added awscredential.properties in /etc.
4. Updated Cassandra start script for invoking classname com.netflix.priam.cassandra.NFThinCassandraDaemon instead of org.apache.cassandra.thrift.CassandraDaemon.

The deployment failed with the error mentioned earlier. I narrowed it down to :

   at com.netflix.priam.defaultimpl.PriamConfiguration.populateASGName(PriamConfiguration.java:176)
    at com.netflix.priam.defaultimpl.PriamConfiguration.setupEnvVars(PriamConfiguration.java:163)

So added a few loggers in PriamConfiguration. But did not get it in the logs. So tried setting ASGName and Region Name in the environment variable after looking at the code. But then also it did not succeed. If we'll have to configure ASG Name, what should be the value for it?

Please let us know if I have followed the right steps.

Thanks.

from priam.

Default properties values need not be set unless you plan to modify them (also set the fields as property, value and appId). Your ASG name is the name of the ASG where you spin your instances. SInce you have not provided the error message of the stack trace, I can only speculate the ASG describe call is failing because the region is not set correctly.

from priam.

We tried to debug the code by doing some hard codings. Following are the clarifications:

1. I have set the ASG_NAME and Region as env variables. But when I printed it in PriamCOnfiguration, it was always coming as null. Do we need to set it up some where else also?
2. Hard coded the values in the code. But it failed in populateASGName - Null pointer exception in AmazonEC2.describeInstances. I have given the region as 'us-east-1'. Let me know if this is a valid input.
3. When we did further hard coding, it failed in setDefaultRACList function at AmazonEC2.describeAvailabilityZones() function. The aws-java-sdk-1.3.2.jar is present in the tomcat lib folder. Still I tried adding it to the CLASSPATH and tried again. But it failed. Guess this is also problem with the region name?
4. Failed in PopulateProps function at ASG_NAME.substring(0, ASG_NAME.lastIndexOf('-').
   Caused by: java.lang.StringIndexOutOfBoundsException: String index out of range: -1
   at java.lang.String.substring(String.java:1949)
   at com.netflix.priam.defaultimpl.PriamConfiguration.populateProps(PriamConfiguration.java:251)
   at com.netflix.priam.defaultimpl.PriamConfiguration.intialize(PriamConfiguration.java:155)
   at com.netflix.priam.defaultimpl.InjectedWebListener.getInjector(InjectedWebListener.java:30)
   The ASG_NAME I gave is "test". Is the code expecting teh ASG_NAME is some specific format?

Thanks.

from priam.

Both ASG name and region will be picked automcatically without the env variables. PriamConfiguration was assuming a certain naming convention for ASG name. I removed this restriction. Also added doc on ASG naming when using multiple ASGs. Your appid is your ASG name by default. us-east-1 is a valid region.

from priam.

I took the latest code as suggested, it still gave the same error - failed at all the function calls to the AWS API. So I wrote a stand alone code which calls the AmazonEC2.describeAvailabilityZones() to test the AWS API. Initially it gave errors for missing jars for class

1. org.apache.http.protocol.ImmutableHttpProcessor
2. org.apache.http.HttpEntityEnclosingRequest
   When I added the httpcore-4.0.1.jar and httpcore-4.1-alpha1.jar, it ran successfully.

So I would like to know the list of jars needed for executing the AWS API used by Priam. looks like the maven did not download all the required jars.

Thanks.

from priam.

With the stand alone code that we developed using PriamConfiguration.java, we could get the ASGName also. But when we deploy the Priam application in tomcat server.
I found that maven did not download the jar httpcore-4.0.1.jar. So modified the pom.xml and got it downloaded during build. But it is not using it during compilation and it is not present in the war that is built. We are unable to find out from where maven gets the list of jars for compilation and building the war. Please let us know where to modify to use this jar for compilation.

Thanks.

from priam.

The current pom does depend on httpcore-4.1. By default it will pull from mvn central repo(http://maven.apache.org/guides/introduction/introduction-to-repositories.html).

from priam.

Pom has entry for httpore-4.1 but it does not have the entry for httpcore-4.1-alpha1.jar and i meant this one. Pleas e check and let me know if this jar is needed or not. In our test code I had to include both the jar and it worked.

In priam I changed the pom to use the httpcore-4.1-alpha1.jar but maven on downloaded but did not use it for build and the error. Please check.

thanks

from priam.

We tested on the instance created with the Auto scaling group. Hence it was failing at the populateProps since ASG name was coming as null. When deployed on instances created using ASG, it worked successfully.

Thanks for the support.

from priam.