Stalling during handshake phase about ccm HOT 19 CLOSED

Am I correct in understanding that when you say you have "multiple Travis build instances", you mean multiple instances running ccm at the same time on the same machine? It is incredibly likely you could have crosstalk problems in that situation. Are you ensuring that there is no clash of IP's and ports, and more importantly that none of the cluster names are the same?

Also, which version of Cassandra are you running on?

from ccm.

It is possible that if you are underspecced for three instances of Cassandra, that could be the case, but if there is a handshaking issue only 25% of the time, I would not assume that is the problem.

from ccm.

Travis creates a matrix of 4 builds for our project gocql (https://travis-ci.org/gocql/gocql) based on two different versions of Golang each tested against Cassandra 1.2.18 and 2.0.9. Each variant is run in its own machine, so what we have is 4 VMs that get booted at pretty much the same time and each one runs a 3 node ccm cluster. These instances are theoretically separate machines and should not see other, but I have started a support case with Travis to find out in what circumstances we may be seeing cross talk. I don't think cross talk is likely, but I'd like to be able to rule it out so that we can focus on the cause.

from ccm.

Regarding the spec of the machines - because these are a custom VM from Travis, it is hard to say whether this is an issue or not. Apparently they have 3GB of memory, but I don't know much about CPU or disk IO. Hopefully the support question I raised might shed some light on this.

from ccm.

Regarding the potential cross talk - is there a way to boot a ccm cluster with a per-VM secret of some kind to prevent any cross-talk? Or is there a way in the config to specify per-VM network ports to achieve the same level of cluster privateness?

from ccm.

3GB will collapse under load but it should be fine for testing. I want to say that if they are all on separate VM's there should be no issue, but I don't think I can promise that. I will see if I can reproduce your problem. Assuming your tests are available on the gocql github, I'll try running them exactly.

from ccm.

Yes, the code is available on Github as gocql/gocql - if you have the bandwidth to investigate this, that would be greatly appreciated. Let us know if there's any way we can help.

from ccm.

The log you pasted appears to be a log from a passing build, was that correct?

from ccm.

Ah yes, that is not very helpful - let me just dig out a better log.

from ccm.

Additionally you are running 'ccm updateconf' after having already started the nodes. Those config changes you set become pointless, as they will not take effect unless the node is restarted. It would be better to run:

ccm create test -v binary:$version -n 3 -d --vnodes
ccm updateconf 'concurrent_reads: 8' 'concurrent_writes: 32' 'rpc_server_type: sync' 'rpc_min_threads: 2' 'rpc_max_threads: 8' 'write_request_timeout_in_ms: 5000' 'read_request_timeout_in_ms: 5000'
ccm start
ccm status

from ccm.

Sorry about this but I'm having trouble getting the particular failed log from the Travis UI because I kicked the build off again subsequently - so I'm still looking around. But in the meantime I will also kick off a build based on your suggestion.

from ccm.

So I've created a patch based on your suggestion and it should be building soon on Travis. In addition, I'm going to try to find out from Travis how to look through the history for manually re-triggered builds.

from ccm.

So the first run of the patch with your suggestion (apache/cassandra-gocql-driver#231) passed without error on Travis. So I have kicked it off again to see if this is reproducible.

from ccm.

The 2nd time around all four build constellations passed, so I've re-triggered it again.

from ccm.

Now I seem to be getting some different test failures, which probably have little to do with ccm, so what I've done is to take this down in the gocql issue backlog: apache/cassandra-gocql-driver#232

from ccm.

BTW, I've just heard back from Travis support about getting logs for manually re-triggered builds, but the news is not so great:

Right now it is not possible to see the history of retriggered builds, though we have discussed the feature already.

from ccm.

I've kicked this off for the 4th time and everything passed 100%, so I'm inclined to say that your suggestion of re-ordering the ccm commands has improved things significantly and that some of the other failures we are seeing are due to fact the code base hasn't yet been thoroughly regression tested against a cluster.

from ccm.

In the light of the fact that we can't reproduce this issue after having implemented your suggested change, I think it is best to close this issue. Thank you very much for all of help, it is very much appreciated.

from ccm.

If you have any more problems, feel free to keep filing issues. Glad to help.

from ccm.