Priam fails to hook into Cassandra about priam HOT 22 CLOSED

Looks like your awscredentials are either incorrect or dont have permissions to use AWS services.

from priam.

I granted it full permissions. It was an IAM keypair, but I can't imagine that would matter. Is that deny error about SDB or something else?

from priam.

It should be SDB. Can you provide the stack after the retries?

from priam.

I killed that cluster. I didn't get to it today. I will try to get it back up by EOW and post the results.

from priam.

I did a query in in the SDB for isntances and got nothing. Here is the entire stack

INFO 17:17:11,318 Calling URL API: http://169.254.169.254/latest/meta-data/placement/availability-zone returns: us-west-2b
INFO 17:17:11,321 Calling URL API: http://169.254.169.254/latest/meta-data/public-hostname returns: ec2-50-112-64-10.us-west-2.compute.amazonaws.com
INFO 17:17:11,323 Calling URL API: http://169.254.169.254/latest/meta-data/public-ipv4 returns: 50.112.64.10
INFO 17:17:11,323 Calling URL API: http://169.254.169.254/latest/meta-data/instance-id returns: i-fa080bca
INFO 17:17:11,324 Calling URL API: http://169.254.169.254/latest/meta-data/instance-type returns: m1.large
INFO 17:17:12,506 REGION set to us-west-2, ASG Name set to CassPriamTest
INFO 17:17:13,215 Job execution threads will use class loader of thread: main
INFO 17:17:13,239 Initialized Scheduler Signaller of type: class org.quartz.core.SchedulerSignalerImpl
INFO 17:17:13,240 Quartz Scheduler v.1.7.3 created.
INFO 17:17:13,244 RAMJobStore initialized.
INFO 17:17:13,244 Quartz scheduler 'DefaultQuartzScheduler' initialized from default resource file in Quartz package: 'quartz.properties'
INFO 17:17:13,245 Quartz scheduler version: 1.7.3
INFO 17:17:13,245 JobFactory set to: com.netflix.priam.scheduler.GuiceJobFactory@752d7d02
INFO 17:17:13,373 New update(s) found: 1.8.5 [http://www.terracotta.org/kit/reflector?kitID=default&pageID=QuartzChangeLog]
INFO 17:17:14,341 My token: 4253529586511730793292182594864475536
INFO 17:17:14,342 Scheduler DefaultQuartzScheduler_$_NON_CLUSTERED started.
INFO 17:17:14,589 cluster_name: cass_cluster
initial_token: null
hinted_handoff_enabled: true
max_hint_window_in_ms: 8
hinted_handoff_throttle_delay_in_ms: 1
authenticator: org.apache.cassandra.auth.AllowAllAuthenticator
authority: org.apache.cassandra.auth.AllowAllAuthority
partitioner: org.apache.cassandra.dht.RandomPartitioner
data_file_directories:

• /var/lib/cassandra/data
  commitlog_directory: /var/lib/cassandra/commitlog
  key_cache_size_in_mb: null
  key_cache_save_period: 14400
  row_cache_size_in_mb: 0
  row_cache_save_period: 0
  row_cache_provider: SerializingCacheProvider
  saved_caches_directory: /var/lib/cassandra/saved_caches
  commitlog_sync: periodic
  commitlog_sync_period_in_ms: 10000
  seed_provider:

• class_name: com.netflix.priam.cassandra.NFSeedProvider
  parameters:

  • seeds: 127.0.0.1
    flush_largest_memtables_at: 0.75
    reduce_cache_sizes_at: 0.85
    reduce_cache_capacity_to: 0.6
    concurrent_reads: 32
    concurrent_writes: 32
    memtable_flush_queue_size: 4
    trickle_fsync: false
    trickle_fsync_interval_in_kb: 10240
    storage_port: 7000
    ssl_storage_port: 7001
    listen_address: null
    rpc_address: null
    rpc_port: 9160
    rpc_keepalive: true
    rpc_server_type: sync
    thrift_framed_transport_size_in_mb: 15
    thrift_max_message_length_in_mb: 16
    incremental_backups: true
    snapshot_before_compaction: false
    auto_snapshot: true
    column_index_size_in_kb: 64
    in_memory_compaction_limit_in_mb: 128
    multithreaded_compaction: false
    compaction_throughput_mb_per_sec: 8
    compaction_preheat_key_cache: true
    rpc_timeout_in_ms: 10000
    endpoint_snitch: org.apache.cassandra.locator.Ec2Snitch
    dynamic_snitch_update_interval_in_ms: 100
    dynamic_snitch_reset_interval_in_ms: 600000
    dynamic_snitch_badness_threshold: 0.1
    request_scheduler: org.apache.cassandra.scheduler.NoScheduler
    index_interval: 128
    encryption_options:
    internode_encryption: none
    keystore: conf/.keystore
    keystore_password: cassandra
    truststore: conf/.truststore
    truststore_password: cassandra
    auto_bootstrap: true

  INFO 17:17:14,606 Starting cassandra server ....Join ring=true
  INFO 17:17:14,609 Starting cassandra server ....
  ERROR 17:17:14,680 Couldnt execute the task because of null
  java.lang.NullPointerException
  at com.netflix.priam.backup.IncrementalBackup.execute(IncrementalBackup.java:36)
  at com.netflix.priam.scheduler.Task.execute(Task.java:78)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
  Apr 26, 2012 5:17:14 PM com.sun.jersey.api.core.PackagesResourceConfig init
  INFO: Scanning for root resource and provider classes in the packages:
  com.netflix.priam.resources
  Apr 26, 2012 5:17:14 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
  INFO: Root resource classes found:
  class com.netflix.priam.resources.CassandraConfig
  class com.netflix.priam.resources.BackupServlet
  class com.netflix.priam.resources.CassandraAdmin
  Apr 26, 2012 5:17:14 PM com.sun.jersey.api.core.ScanningResourceConfig init
  INFO: No provider classes found.
  Apr 26, 2012 5:17:15 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
  INFO: Initiating Jersey application, version 'Jersey: 1.9.1 09/14/2011 02:36 PM'
  ERROR 17:17:16,102 Retry #1 for: Access Denied
  Apr 26, 2012 5:17:16 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
  INFO: Binding com.netflix.priam.resources.CassandraConfig to GuiceInstantiatedComponentProvider
  Apr 26, 2012 5:17:16 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
  INFO: Binding com.netflix.priam.resources.BackupServlet to GuiceInstantiatedComponentProvider
  ERROR 17:17:16,785 Retry #2 for: Access Denied
  Apr 26, 2012 5:17:16 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider
  INFO: Binding com.netflix.priam.resources.CassandraAdmin to GuiceInstantiatedComponentProvider
  Apr 26, 2012 5:17:16 PM org.apache.coyote.http11.Http11Protocol start
  INFO: Starting Coyote HTTP/1.1 on http-8080
  Apr 26, 2012 5:17:16 PM org.apache.catalina.startup.Catalina start
  INFO: Server startup in 7460 ms
  ERROR 17:17:17,408 Retry #3 for: Access Denied
  ERROR 17:17:18,045 Retry #4 for: Access Denied
  ERROR 17:17:18,661 Retry #5 for: Access Denied
  ERROR 17:17:19,283 Retry #6 for: Access Denied
  ERROR 17:17:19,910 Retry #7 for: Access Denied
  ERROR 17:17:20,525 Retry #8 for: Access Denied
  ERROR 17:17:21,162 Retry #9 for: Access Denied
  ERROR 17:17:21,787 Retry #10 for: Access Denied
  ERROR 17:17:22,412 Retry #11 for: Access Denied
  ERROR 17:17:23,035 Retry #12 for: Access Denied
  ERROR 17:17:23,662 Retry #13 for: Access Denied
  ERROR 17:17:24,290 Retry #14 for: Access Denied
  ERROR 17:17:24,634 Couldnt execute the task because of null
  java.lang.NullPointerException
  at com.netflix.priam.backup.IncrementalBackup.execute(IncrementalBackup.java:36)
  at com.netflix.priam.scheduler.Task.execute(Task.java:78)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
  ERROR 17:17:24,923 Couldnt execute the task because of Access Denied
  Status Code: 403, AWS Service: Amazon S3, AWS Request ID: F16A5999423495E8, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: ndQ6Q4u21SBo2Fymms2FsNMXLjRUXeapqpqibiRD7XMhn2RIG7+8j4PzUlurYj1o
  at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:548)
  at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:288)
  at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2619)
  at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:2591)
  at com.amazonaws.services.s3.AmazonS3Client.getBucketLifecycleConfiguration(AmazonS3Client.java:1494)
  at com.netflix.priam.aws.S3FileSystem.cleanup(S3FileSystem.java:183)
  at com.netflix.priam.aws.UpdateCleanupPolicy$1.retriableCall(UpdateCleanupPolicy.java:40)
  at com.netflix.priam.aws.UpdateCleanupPolicy$1.retriableCall(UpdateCleanupPolicy.java:36)
  at com.netflix.priam.utils.RetryableCallable.call(RetryableCallable.java:42)
  at com.netflix.priam.aws.UpdateCleanupPolicy.execute(UpdateCleanupPolicy.java:35)
  at com.netflix.priam.scheduler.Task.execute(Task.java:78)
  at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
  at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)

from priam.

Looks like ACL issue when accessing bucket configuration.

http://docs.amazonwebservices.com/AmazonS3/latest/dev/UsingAuthAccess.html

from priam.

So I am able to use the keys from my machine and touch the bucket. Here is my guess. We are running this test in us-west-2, but the system it trying to connect to SDB in us-east-1 instead of us-west-2?

I was able to confirm this. I see InstanceIdentify populated using the sdb.amazonaws.com endpoint instead of being region specific. I see some problems with this. Does that mean, since sdb is defaulted to the us-east-1 region, if the building catches fire, Priam will fail to function?

I also think I found the root cause of the nullpointer: "initial_token: null" is in my cassandra.yaml. I also see null for listen_address and rpc_address.

I am still also seeing the access denied, but like I said I confirmed that the keys are good. Are special chars in the keys not allowed by the underlying library?

Here is my test InstanceIdentity DB

itemName() appId availabilityZone elasticIP hostname id instanceId location token updateTimestamp
cass_cluster1967372894 cass_cluster us-west-2b 50.112.64.10 ec2-50-112-64-10.us-west-2.compute.amazonaws.com 1967372894 i-fa080bca us-west-2 4253529586511730793292182594864475536 1335458892342
cass_cluster-dead1967372893 cass_cluster-dead us-west-2a 50.112.34.58 ec2-50-112-34-58.us-west-2.compute.amazonaws.com 1967372893 i-6a08095a us-west-2 1967372893 1335530106917
cass_cluster1967372893 cass_cluster us-west-2a 50.112.6.169 ec2-50-112-6-169.us-west-2.compute.amazonaws.com 1967372893 i-60595b50 us-west-2 1967372893 1335530108102

Here is my configuration in PriamProperties:

itemName() appId property value
cass_clusterpriam.s3.bucket cass_cluster priam.s3.bucket REDADCTED-cass-archive

from priam.

Hi,

The access denied error is with S3 and not SimpleDB.

On the simpledb question, the instanceidentity domain is used by Priam to discover other nodes in the ring and identify itself during bootup. It default to 1 region to help with multi-regional clusters.

The "initial_token: null" in cassandra.yaml is fine. Cassandra will get token from Priam instead of yaml.

Ensure you are able ro read/write and access bucket life cycle config for the bucket REDADCTED-cass-archive

from priam.

The user was an IAM user (special chars were in the secret key). I was able to access using those specific keys and drop files via cyberduck into the bucket. I'm wondering if it is not picking up the PriamProperty. IS there any debugging I can put in place to see what the PriamProperties is got from SDB are? I think that might be a good idea to print to catalina.out regardless.

from priam.

You can see the request/response by enabling debugging for AWS requests. Add this to your log4j:

log4j.logger.com.amazonaws.request=DEBUG

Also, your cluster name is inferred from your ASG name. I have clarified this in the properties wiki.

from priam.

OK, I bet that is the difference. I was taking the clustername as the name of the ring cluster name, not the ASG. I will try again on Monday and report back.

from priam.

OK, I'm much closer now. One very big worry is nodetool no longer works. I thought well maybe I needed to use Priam "curl http://localhost:8080/Priam/REST/v1/cassadmin/ring" but that just hangs indefinetly. I see

ERROR 15:18:16,005 Couldnt execute the task because of null
java.lang.NullPointerException
at com.netflix.priam.backup.IncrementalBackup.execute(IncrementalBackup.java:36)
at com.netflix.priam.scheduler.Task.execute(Task.java:78)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:546)
ERROR 15:18:16,228 Retry #59 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:17,230 Retry #60 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:18,231 Retry #61 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:19,233 Retry #62 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:20,239 Retry #63 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:21,241 Retry #64 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:22,242 Retry #65 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:23,244 Retry #66 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]
ERROR 15:18:24,246 Retry #67 for: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: localhost; nested exception is:
java.net.ConnectException: Connection refused]

My thought is that Priam is not setting the listen address properly. I still see it as null in the yaml. I do see the correct public IP in InstanceIdentity. I have saved the entire catalina.out in debug mode if you want it.

from priam.

I would look at cassandra system.log to see if there were errors. listen address is fine. You should be able to use nodetool and cassandra-cli.

from priam.

So the overall problem is priam is failing to return the call for seeds. Causing Cassandra to fail to start. Am I doing it wrong by starting on the first node in the ASG and using priam? Should the first node be "Priam-less"?

INFO 15:56:31,180 EC2Snitch using region: us-west-2, zone: 2a.
java.lang.RuntimeException: org.apache.cassandra.config.ConfigurationException: Unable to get data for URL http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_seeds
at com.netflix.priam.utils.SystemUtils.getDataFromUrl(SystemUtils.java:120)
at com.netflix.priam.cassandra.NFSeedProvider.(NFSeedProvider.java:23)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:459)
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:118)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126)
at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
Caused by: org.apache.cassandra.config.ConfigurationException: Unable to get data for URL http://127.0.0.1:8080/Priam/REST/v1/cassconfig/get_seeds
at com.netflix.priam.utils.SystemUtils.getDataFromUrl(SystemUtils.java:105)
... 14 more
ERROR 15:56:32,415 Fatal configuration error
org.apache.cassandra.config.ConfigurationException: The seed provider lists no seeds.
at org.apache.cassandra.config.DatabaseDescriptor.loadYaml(DatabaseDescriptor.java:469)
at org.apache.cassandra.config.DatabaseDescriptor.(DatabaseDescriptor.java:118)
at org.apache.cassandra.service.AbstractCassandraDaemon.setup(AbstractCassandraDaemon.java:126)
at org.apache.cassandra.service.AbstractCassandraDaemon.init(AbstractCassandraDaemon.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.commons.daemon.support.DaemonLoader.load(DaemonLoader.java:160)
The seed provider lists no seeds.
Fatal configuration error; unable to start server. See log for stacktrace.
Service exit with a return value of 1

from priam.

Looks similar to #17

Let me know if this helps too: https://github.com/Netflix/Priam/wiki/FAQ

All nodes should be started with Priam.

from priam.

It does look similar. So Priam needs to be up on multiple zones. I think I get it now, I will try that after this fire is out.

from priam.

You can run in single zone by setting the 'priam.zones.available' property to your zone.

from priam.

OK, I got get_seeds working, but ring still blows up. They did both join the ring this time at least...

INFO 14:36:48,024 node tool ring being called
INFO 14:36:48,024 JMX ring being called
May 2, 2012 2:36:48 PM com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
SEVERE: The RuntimeException could not be mapped to a response, re-throwing to the HTTP container
java.lang.ClassCastException: java.lang.String cannot be cast to org.apache.cassandra.dht.Token
at com.netflix.priam.utils.JMXNodeTool.ring(JMXNodeTool.java:200)
at com.netflix.priam.resources.CassandraAdmin.cassRing(CassandraAdmin.java:105)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:708)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:662)

from priam.

I presume you got your nodetool working. What version of cassandra are you running?

from priam.

Yes it is working. That was latest, 1.1.x. Do I need to drop back to 1.0.x?

from priam.

Yes. Its not yet compatible with 1.1. Some interface changes seem to break.

apache/cassandra@c0af7f6#src/java/org/apache/cassandra/tools/NodeProbe.java

from priam.

I guess i'm just running into too many problems (tokens not matching, etc). Once the docs are flushed out more (permissions seems to be part of the problem), I'll give this a shot again, until then I'll just use my shell script and a move. Thanks for the help.

from priam.