Git Product home page Git Product logo

cluster-bigartm's People

Contributors

mellain avatar ofrei avatar

Watchers

avatar avatar

cluster-bigartm's Issues

glog doesn't create log file

... or I don't know where to find it.
For now I've added

  FLAGS_minloglevel = 0;
  FLAGS_alsologtostderr = 1;
  FLAGS_logbuflevel = -1;
  FLAGS_stderrthreshold = 0;

before InitGoogleLogging(...), this way I get at logs least on stderr - but still can't see a file.

executor_main throws RedisCluster::NodeSearchException

Redis cluster is up and running, and I'm able to connect to it using redis-cli. However, executor_main does not connect to the cluster, throwing RedisCluster::NodeSearchException.

/usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0   --redis-ip 10.110.31.14 --redis-port 7001  --executor-id 0 --token-begin-index 0 --token-end-index 14051 --batch-begin-index 0 --batch-end-index 40 &
[1] 30807
-bash-4.1$ terminate called after throwing an instance of 'RedisCluster::NodeSearchException'
  what():  node not found in cluster

[1]+  Aborted                 /usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0 --redis-ip 10.110.31.14 --redis-port 7001 --executor-id 0 --token-begin-index 0 --token-end-index 14051 --batch-begin-index 0 --batch-end-index 40
-bash-4.1$ redis-cli -h 10.110.31.14 -p 7001 CLUSTER NODES
44231fbc86e4b2b66b8835441441879769f7bf81 10.110.31.14:7001@17001 myself,master - 0 0 1 connected 8000-8191
63de986ae73144137a7dc3afa8527586a55b875e 10.110.31.14:7002@17002 master - 0 1543480640624 0 connected 8192-16383

master_main crashes with basic_string::_M_construct null not valid

Here is the sequence.
I start two instances of redis-server on the same node (ip 10.110.31.14). Then:

-bash-4.1$ redis-cli -h c31-14 -p 7001 CLUSTER ADDSLOTS {0..8191}
OK
-bash-4.1$ redis-cli -h c31-14 -p 7002 CLUSTER ADDSLOTS {8192..16383}
OK
-bash-4.1$ redis-cli -h c31-14 -p 7002 CLUSTER MEET 10.110.31.14 7001   # must use IP address
OK
-bash-4.1$ redis-cli -h 10.110.31.14 -p 7001 CLUSTER NODES
20dd68459606bd247fce9d044e178108e7e8f480 10.110.31.14:7002@17002 handshake - 0 0 0 disconnected
b736fea98c9670269eaf37a6d110b2a45b056f4f 10.110.31.14:7001@17001 myself,master - 0 0 0 connected 0-8191
-bash-4.1$ more redis_addr2.txt
10.110.31.14 7001
10.110.31.14 7002
-bash-4.1$ python start_executors.py -v /work/users/oleksanf/bigartm/vocab.enron.txt -b /work/users/oleksanf/bigartm/enron -r redis_addr2.txt -n 1 -t 10 -i 10 -c 0 -o /work/users/oleksanf/bigartm/results
Number of instances is 2
Number of tokens: 28102
Number of batches: 80
/usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0   --redis-ip 10.110.31.14 --redis-port 7001  --executor-id 0 --token-begin-index 0 --token-end-index 14051 --batch-begin-index 0 --batch-end-index 40 &
/usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0   --redis-ip 10.110.31.14 --redis-port 7002  --executor-id 1 --token-begin-index 14051 --token-end-index 28102 --batch-begin-index 40 --batch-end-index 80 &
-bash-4.1$ /usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0   --redis-ip 10.110.31.14 --redis-port 7001  --executor-id 0 --token-begin-index 0 --token-end-index 14051 --batch-begin-index 0 --batch-end-index 40 &
[1] 7024
-bash-4.1$ /usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0   --redis-ip 10.110.31.14 --redis-port 7002  --executor-id 1 --token-begin-index 14051 --token-end-index 28102 --batch-begin-index 40 --batch-end-index 80 &
[2] 7025
-bash-4.1$
-bash-4.1$
-bash-4.1$ more executor_ids.txt
0
1

-bash-4.1$ /usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/master_main --num-topics 10 --num-outer-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0   --redis-ip 10.110.31.14 --redis-port 7002  --show-top-tokens 1 --executor-ids-path executor_ids.txt
num-topics:        10
num-outer-iter:    10
executor-ids-path: executor_ids.txt
batches-dir-path:  /work/users/oleksanf/bigartm/enron
vocab-path:        /work/users/oleksanf/bigartm/vocab.enron.txt
redis-ip:          10.110.31.14
redis-port:        7002
show-top-tokens:   1
continue-fitting:  0
terminate called after throwing an instance of 'std::logic_error'
  what():  basic_string::_M_construct null not valid
Aborted
-bash-4.1$ terminate called after throwing an instance of 'std::runtime_error'
  what():  |data-1 Step 1 start, got termination command
terminate called after throwing an instance of 'std::runtime_error'
  what():  |data-0 Step 1 start, got termination command

[1]-  Aborted                 /usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0 --redis-ip 10.110.31.14 --redis-port 7001 --executor-id 0 --token-begin-index 0 --token-end-index 14051 --batch-begin-index 0 --batch-end-index 40
[2]+  Aborted                 /usit/abel/u1/oleksanf/GitHub/MelLain/cluster-bigartm/build/executor_main --num-topics 10 --num-inner-iter 10 --batches-dir-path /work/users/oleksanf/bigartm/enron --vocab-path /work/users/oleksanf/bigartm/vocab.enron.txt --continue-fitting 0 --redis-ip 10.110.31.14 --redis-port 7002 --executor-id 1 --token-begin-index 14051 --token-end-index 28102 --batch-begin-index 40 --batch-end-index 80

I think we need more logging in both executables, executor_main and master_main, to debut these issues

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.