Validator melea-trust , stop for update the server. now don't sync fast, after unsafe_reset neither. about testnets HOT 8 CLOSED

Which version of gaia? Can you try updated to 0.17.3

from testnets.

Related
tendermint/tendermint#1659

from testnets.

I gave access to @melekes to machine with a wal snapshot that causes the crash

from testnets.

my other node (no validator) is not looking good neither, is working but in this way

Moniker
trust melea #3
Listen Address
142.44.240.204:46656

25:C59779BF393E 113163/00/2(Precommit) 000000000000 /FA5E741998DC.../ @ 2018-05-24T02:16:25.252Z} VoteA: Vote{25:C59779BF393E 113307/00/2(Precommit) 629D1BD8B104 /2AC466DC6EF3.../ @ 2018-05-24T02:26:30.982Z}; VoteB: Vote{25:C59779BF393E 113307/00/2(Precommit) 000000000000 /051FA5F6580A.../ @ 2018-05-24T02:26:33.038Z} VoteA: Vote{25:C59779BF393E 113853/00/2(Precommit) 88B8A3FE1D63 /277237E65FA1.../ @ 2018-05-24T03:02:35.931Z}; VoteB: Vote{25:C59779BF393E 113853/00/2(Precommit) 000000000000 /FAC4EE02CF30.../ @ 2018-05-24T03:02:38.619Z} VoteA: Vote{25:C59779BF393E 114233/02/2(Precommit) 40EEE33B0BC1 /904BAD627E06.../ @ 2018-05-24T03:29:21.590Z}; VoteB: Vote{25:C59779BF393E 114233/02/2(Precommit) 000000000000 /E13523745A11.../ @ 2018-05-24T03:29:23.299Z} VoteA: Vote{25:C59779BF393E 116322/00/2(Precommit) C8DC545EDBEB /D3F84E5D8223.../ @ 2018-05-24T05:22:51.962Z}; VoteB: Vote{25:C59779BF393E 116322/00/2(Precommit) 000000000000 /57E5FEC2DF92.../ @ 2018-05-24T05:22:56.991Z} VoteA: Vote{25:C59779BF393E 116476/00/2(Precommit) 8FA6C1F4AAFF /7E53DD5A8EA2.../ @ 2018-05-24T05:31:18.196Z}; VoteB: Vote{25:C59779BF393E 116476/00/2(Precommit) 000000000000 /4EFCBFE70988.../ @ 2018-05-24T05:31:21.743Z} VoteA: Vote{25:C59779BF393E 116736/00/2(Precommit) 5300D6026A29 /C5BE3B791268.../ @ 2018-05-24T05:45:55.631Z}; VoteB: Vote{25:C59779BF393E 116736/00/2(Precommit) 000000000000 /7D99C073CC9D.../ @ 2018-05-24T05:45:58.012Z} VoteA: Vote{25:C59779BF393E 116776/00/2(Precommit) 368DD785D24F /E6245B1344A2.../ @ 2018-05-24T05:48:08.904Z}; VoteB: Vote{25:C59779BF393E 116776/00/2(Precommit) 000000000000 /2F4DE1A850CF.../ @ 2018-05-24T05:48:13.272Z} VoteA: Vote{25:C59779BF393E 117361/00/2(Precommit) 374593375555 /0E4A902F4692.../ @ 2018-05-24T06:20:59.903Z}; VoteB: Vote{25:C59779BF393E 117361/00/2(Precommit) 000000000000 /806B353ED624.../ @ 2018-05-24T06:21:02.708Z} VoteA: Vote{25:C59779BF393E 118700/00/2(Precommit) 4643CC8DFD1C /D595DC49F5EB.../ @ 2018-05-24T07:35:19.906Z}; VoteB: Vote{25:C59779BF393E 118700/00/2(Precommit) 000000000000 /6AD29940430E.../ @ 2018-05-24T07:35:21.924Z} VoteA: Vote{25:C59779BF393E 119373/00/2(Precommit) 91B5649B2433 /19341778D5BA.../ @ 2018-05-24T08:13:23.023Z}; VoteB: Vote{25:C59779BF393E 119373/00/2(Precommit) 000000000000 /3396394371A0.../ @ 2018-05-24T08:13:25.501Z} VoteA: Vote{25:C59779BF393E 119392/02/2(Precommit) BBB1238042AD /1B72D903EF93.../ @ 2018-05-24T08:14:32.539Z}; VoteB: Vote{25:C59779BF393E 119392/02/2(Precommit) 000000000000 /6D6D084F81BB.../ @ 2018-05-24T08:14:33.846Z} VoteA: Vote{25:C59779BF393E 119904/00/2(Precommit) BBA988422918 /288E665DA68E.../ @ 2018-05-24T08:45:46.712Z}; VoteB: Vote{25:C59779BF393E 119904/00/2(Precommit) 000000000000 /2DA8F07A34C8.../ @ 2018-05-24T08:45:48.782Z} VoteA: Vote{25:C59779BF393E 120421/00/2(Precommit) 000000000000 /4D8C9C5498F6.../ @ 2018-05-24T09:17:32.711Z}; VoteB: Vote{25:C59779BF393E 120421/00/2(Precommit) D6D391CE741F /BA64E6C5D65B.../ @ 2018-05-24T09:17:30.939Z} VoteA: Vote{25:C59779BF393E 120813/00/2(Precommit) 8002BE5623B8 /D122366F275B.../ @ 2018-05-24T09:42:08.182Z}; VoteB: Vote{25:C59779BF393E 120813/00/2(Precommit) 000000000000 /A31EBC99A2B5.../ @ 2018-05-24T09:42:09.731Z} VoteA: Vote{25:C59779BF393E 121427/00/2(Precommit) 7F063203E9AD /0B5DC7F987F1.../ @ 2018-05-24T10:19:00.969Z}; VoteB: Vote{25:C59779BF393E 121427/00/2(Precommit) 000000000000 /1A308830C47B.../ @ 2018-05-24T10:19:03.524Z} VoteA: Vote{25:C59779BF393E 121452/00/2(Precommit) BDE6511076DB /6EA75CE3992C.../ @ 2018-05-24T10:20:37.242Z}; VoteB: Vote{25:C59779BF393E 121452/00/2(Precommit) 000000000000 /F953DB12DC25.../ @ 2018-05-24T10:20:39.669Z} VoteA: Vote{25:C59779BF393E 121588/02/2(Precommit) 34CB7AF82B3B /6713C4465582.../ @ 2018-05-24T10:29:22.377Z}; VoteB: Vote{25:C59779BF393E 121588/02/2(Precommit) 000000000000 /5E9F534684CF.../ @ 2018-05-24T10:29:24.354Z} VoteA: Vote{25:C59779BF393E 121591/00/1(Prevote) 090B571DC3B1 /5A42F7517E7D.../ @ 2018-05-24T10:29:33.765Z}; VoteB: Vote{25:C59779BF393E 121591/00/1(Prevote) 0470E0046633 /9B5380384F48.../ @ 2018-05-24T10:29:36.611Z} VoteA: Vote{25:C59779BF393E 121604/00/1(Prevote) 6A9F93C54AC0 /6719A53E240B.../ @ 2018-05-24T10:30:32.547Z}; VoteB: Vote{25:C59779BF393E 121604/00/1(Prevote) 24F5512DB2FC /3B4953E378E0.../ @ 2018-05-24T10:30:35.009Z} VoteA: Vote{26:C59779BF393E 121749/00/2(Precommit) 33811934D0E2 /1335F6E788C9.../ @ 2018-05-24T10:40:44.863Z}; VoteB: Vote{26:C59779BF393E 121749/00/2(Precommit) 000000000000 /AB04BEDF02DE.../ @ 2018-05-24T10:40:47.504Z} VoteA: Vote{26:C59779BF393E 121756/00/2(Precommit) E8868FD238BF /DE8B123EE554.../ @ 2018-05-24T10:41:23.020Z}; VoteB: Vote{26:C59779BF393E 121756/00/2(Precommit) 000000000000 /589D560A5A78.../ @ 2018-05-24T10:41:25.460Z} VoteA: Vote{26:C59779BF393E 121760/00/2(Precommit) 442560BFCEC2 /E45409087157.../ @ 2018-05-24T10:41:35.213Z}; VoteB: Vote{26:C59779BF393E 121760/00/2(Precommit) 000000000000 /E056D0ADC82A.../ @ 2018-05-24T10:41:37.672Z} VoteA: Vote{26:C59779BF393E 121809/00/2(Precommit) 80FA975E1C01 /6060840961C6.../ @ 2018-05-24T10:45:04.098Z}; VoteB: Vote{26:C59779BF393E 121809/00/2(Precommit) 000000000000 /3AFA1F70C202.../ @ 2018-05-24T10:45:06.283Z} VoteA: Vote{26:C59779BF393E 121942/00/2(Precommit) 4144B2B33E9A /B36A974BE3D8.../ @ 2018-05-24T10:54:40.410Z}; VoteB: Vote{26:C59779BF393E 121942/00/2(Precommit) 000000000000 /E0D5FD14DAE0.../ @ 2018-05-24T10:54:42.937Z} VoteA: Vote{26:C59779BF393E 121947/00/2(Precommit) 92BC6075EA2C /97D012F9886F.../ @ 2018-05-24T10:54:59.773Z}; VoteB: Vote{26:C59779BF393E 121947/00/2(Precommit) 000000000000 /559ED244A976.../ @ 2018-05-24T10:55:01.897Z} VoteA: Vote{26:C59779BF393E 121949/01/2(Precommit) A2ECA881D8DE /B4C4C08E5746.../ @ 2018-05-24T10:55:17.940Z}; VoteB: Vote{26:C59779BF393E 121949/01/2(Precommit) 000000000000 /448792558055.../ @ 2018-05-24T10:55:18.471Z} VoteA: Vote{26:C59779BF393E 121964/00/2(Precommit) CF208B54BF17 /15AE77AF00B6.../ @ 2018-05-24T10:56:20.163Z}; VoteB: Vote{26:C59779BF393E 121964/00/2(Precommit) 000000000000 /7828924C5CEA.../ @ 2018-05-24T10:56:21.884Z} VoteA: Vote{26:C59779BF393E 121982/00/2(Precommit) 15B071B9BAEA /ADDEF3B43550.../ @ 2018-05-24T10:57:28.326Z}; VoteB: Vote{26:C59779BF393E 121982/00/2(Precommit) 000000000000 /3BD4B1828F40.../ @ 2018-05-24T10:57:30.181Z} VoteA: Vote{26:C59779BF393E 122051/00/2(Precommit) 2FB7568532B0 /781E4BBA9D63.../ @ 2018-05-24T11:02:12.711Z}; VoteB: Vote{26:C59779BF393E 122051/00/2(Precommit) 000000000000 /1B8B92A40FBB.../ @ 2018-05-24T11:02:13.978Z} VoteA: Vote{26:C59779BF393E 122060/00/1(Prevote) 000000000000 /6553263AE4BF.../ @ 2018-05-24T11:02:53.945Z}; VoteB: Vote{26:C59779BF393E 122060/00/1(Prevote) DC78067BAD2F /D2940737AD8A.../ @ 2018-05-24T11:02:57.398Z} VoteA: Vote{26:C59779BF393E 122204/00/2(Precommit) DFBD7BD70711 /ECF456E285B4.../ @ 2018-05-24T11:13:01.678Z}; VoteB: Vote{26:C59779BF393E 122204/00/2(Precommit) 000000000000 /5B0AB4F12585.../ @ 2018-05-24T11:13:04.189Z} VoteA: Vote{26:C59779BF393E 122285/00/2(Precommit) B1B5D091E4C6 /848847AAC13C.../ @ 2018-05-24T11:18:51.006Z}; VoteB: Vote{26:C59779BF393E 122285/00/2(Precommit) 000000000000 /D5BE61C2AED3.../ @ 2018-05-24T11:18:52.845Z} VoteA: Vote{26:C59779BF393E 122314/00/2(Precommit) 80888A636E23 /AF063F3A2C8C.../ @ 2018-05-24T11:20:53.605Z}; VoteB: Vote{26:C59779BF393E 122314/00/2(Precommit) 000000000000 /04C7CD54D547.../ @ 2018-05-24T11:20:55.367Z} VoteA: Vote{26:C59779BF393E 122396/00/2(Precommit) B058CB1C347A /6722AC329A7F.../ @ 2018-05-24T11:26:24.779Z}; VoteB: Vote{26:C59779BF393E 122396/00/2(Precommit) 000000000000 /BE92E182EDE5.../ @ 2018-05-24T11:26:26.820Z} VoteA: Vote{26:C59779BF393E 122429/00/2(Precommit) A9160E70A5BF /DD0E778147C5.../ @ 2018-05-24T11:28:38.884Z}; VoteB: Vote{26:C59779BF393E 122429/00/2(Precommit) 000000000000 /56C64DEF6C44.../ @ 2018-05-24T11:28:40.559Z} VoteA: Vote{26:C59779BF393E 122446/00/1(Prevote) 000000000000 /415696E514A7.../ @ 2018-05-24T11:29:47.534Z}; VoteB: Vote{26:C59779BF393E 122446/00/1(Prevote) 37ED39320B87 /A1691811E8B8.../ @ 2018-05-24T11:29:50.529Z} VoteA: Vote{26:C59779BF393E 122466/00/2(Precommit) 430901654F25 /02E9F8DFF701.../ @ 2018-05-24T11:31:18.830Z}; VoteB: Vote{26:C59779BF393E 122466/00/2(Precommit) 000000000000 /BBAB6B1D1DDD.../ @ 2018-05-24T11:31:21.531Z} VoteA: Vote{26:C59779BF393E 122617/01/2(Precommit) 20317A01F3AD /FFF96B4F3237.../ @ 2018-05-24T11:41:33.743Z}; VoteB: Vote{26:C59779BF393E 122617/01/2(Precommit) 000000000000 /B5A4AA48A80D.../ @ 2018-05-24T11:41:34.448Z} VoteA: Vote{26:C59779BF393E 122662/00/2(Precommit) 28BFE025104D /0E273065CEEE.../ @ 2018-05-24T11:44:27.900Z}; VoteB: Vote{26:C59779BF393E 122662/00/2(Precommit) 000000000000 /8C15D962AA4E.../ @ 2018-05-24T11:44:29.710Z} VoteA: Vote{26:C59779BF393E 122685/00/2(Precommit) DC558AB5A70B /254DA16DC7E4.../ @ 2018-05-24T11:46:04.690Z}; VoteB: Vote{26:C59779BF393E 122685/00/2(Precommit) 000000000000 /1F2848881CDC.../ @ 2018-05-24T11:46:06.461Z} VoteA: Vote{26:C59779BF393E 122785/01/2(Precommit) 1BEB3BDE0989 /F55C72B586C9.../ @ 2018-05-24T11:52:54.507Z}; VoteB: Vote{26:C59779BF393E 122785/01/2(Precommit) 000000000000 /7BA8C6506777.../ @ 2018-05-24T11:52:55.458Z} VoteA: Vote{26:C59779BF393E 122787/00/2(Precommit) 1F6C9FE3B552 /D2FCACE36A52.../ @ 2018-05-24T11:52:59.970Z}; VoteB: Vote{26:C59779BF393E 122787/00/2(Precommit) 000000000000 /812CB2AF1023.../ @ 2018-05-24T11:53:02.591Z} VoteA: Vote{26:C59779BF393E 122791/00/2(Precommit) 000000000000 /A49A8A78E205.../ @ 2018-05-24T11:53:16.575Z}; VoteB: Vote{26:C59779BF393E 122791/00/2(Precommit) 2EEF5F273DFC /3BABBF9E4A06.../ @ 2018-05-24T11:53:15.834Z} VoteA: Vote{26:C59779BF393E 122811/00/2(Precommit) 9E3940E6A47D /DD21233DD98E.../ @ 2018-05-24T11:54:44.076Z}; VoteB: Vote{26:C59779BF393E 122811/00/2(Precommit) 000000000000 /145344919959.../ @ 2018-05-24T11:54:45.214Z} VoteA: Vote{26:C59779BF393E 122832/00/2(Precommit) D34206E6A740 /77BF1D287099.../ @ 2018-05-24T11:55:56.634Z}; VoteB: Vote{26:C59779BF393E 122832/00/2(Precommit) 000000000000 /84E58AB21000.../ @ 2018-05-24T11:55:59.608Z} VoteA: Vote{26:C59779BF393E 122934/00/2(Precommit) 80956D8DC4F8 /72B5A0983ABF.../ @ 2018-05-24T12:02:16.256Z}; VoteB: Vote{26:C59779BF393E 122934/00/2(Precommit) 000000000000 /E8AD0B762A58.../ @ 2018-05-24T12:02:18.886Z} VoteA: Vote{26:C59779BF393E 123092/02/2(Precommit) 3A6F80FEB062 /F389473C6515.../ @ 2018-05-24T12:11:47.054Z}; VoteB: Vote{26:C59779BF393E 123092/02/2(Precommit) 000000000000 /D1C5A01006AF.../ @ 2018-05-24T12:11:48.669Z} VoteA: Vote{26:C59779BF393E 123468/00/2(Precommit) 250CF08614CA /38BB43181FAE.../ @ 2018-05-24T12:33:21.321Z}; VoteB: Vote{26:C59779BF393E 123468/00/2(Precommit) 000000000000 /29FC1BC5AA51.../ @ 2018-05-24T12:33:22.598Z} VoteA: Vote{26:C59779BF393E 123510/00/2(Precommit) CCFD21ADAAC1 /FDE9491633CB.../ @ 2018-05-24T12:35:45.091Z}; VoteB: Vote{26:C59779BF393E 123510/00/2(Precommit) 000000000000 /C1FA4E264623.../ @ 2018-05-24T12:35:47.057Z} VoteA: Vote{26:C59779BF393E 123965/00/2(Precommit) B838EC78DCDE /77E4B264CA69.../ @ 2018-05-24T13:02:05.056Z}; VoteB: Vote{26:C59779BF393E 123965/00/2(Precommit) 000000000000 /86B7A38AC180.../ @ 2018-05-24T13:02:06.651Z}]" err="Evidence from height 119904 is too old. Min height is 120282"
E[05-30|23:26:32.895] Stopping peer for error module=p2p peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}" err="Evidence from height 119904 is too old. Min height is 120282"
I[05-30|23:26:32.932] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"
I[05-30|23:26:32.932] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"
I[05-30|23:26:33.083] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"
I[05-30|23:26:34.694] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}"

59779BF393E 122787/00/2(Precommit) 000000000000 /812CB2AF1023.../ @ 2018-05-24T11:53:02.591Z} VoteA: Vote{26:C59779BF393E 122791/00/2(Precommit) 000000000000 /A49A8A78E205.../ @ 2018-05-24T11:53:16.575Z}; VoteB: Vote{26:C59779BF393E 122791/00/2(Precommit) 2EEF5F273DFC /3BABBF9E4A06.../ @ 2018-05-24T11:53:15.834Z} VoteA: Vote{26:C59779BF393E 122811/00/2(Precommit) 9E3940E6A47D /DD21233DD98E.../ @ 2018-05-24T11:54:44.076Z}; VoteB: Vote{26:C59779BF393E 122811/00/2(Precommit) 000000000000 /145344919959.../ @ 2018-05-24T11:54:45.214Z} VoteA: Vote{26:C59779BF393E 122832/00/2(Precommit) D34206E6A740 /77BF1D287099.../ @ 2018-05-24T11:55:56.634Z}; VoteB: Vote{26:C59779BF393E 122832/00/2(Precommit) 000000000000 /84E58AB21000.../ @ 2018-05-24T11:55:59.608Z} VoteA: Vote{26:C59779BF393E 122934/00/2(Precommit) 80956D8DC4F8 /72B5A0983ABF.../ @ 2018-05-24T12:02:16.256Z}; VoteB: Vote{26:C59779BF393E 122934/00/2(Precommit) 000000000000 /E8AD0B762A58.../ @ 2018-05-24T12:02:18.886Z} VoteA: Vote{26:C59779BF393E 123092/02/2(Precommit) 3A6F80FEB062 /F389473C6515.../ @ 2018-05-24T12:11:47.054Z}; VoteB: Vote{26:C59779BF393E 123092/02/2(Precommit) 000000000000 /D1C5A01006AF.../ @ 2018-05-24T12:11:48.669Z} VoteA: Vote{26:C59779BF393E 123468/00/2(Precommit) 250CF08614CA /38BB43181FAE.../ @ 2018-05-24T12:33:21.321Z}; VoteB: Vote{26:C59779BF393E 123468/00/2(Precommit) 000000000000 /29FC1BC5AA51.../ @ 2018-05-24T12:33:22.598Z} VoteA: Vote{26:C59779BF393E 123510/00/2(Precommit) CCFD21ADAAC1 /FDE9491633CB.../ @ 2018-05-24T12:35:45.091Z}; VoteB: Vote{26:C59779BF393E 123510/00/2(Precommit) 000000000000 /C1FA4E264623.../ @ 2018-05-24T12:35:47.057Z} VoteA: Vote{26:C59779BF393E 123965/00/2(Precommit) B838EC78DCDE /77E4B264CA69.../ @ 2018-05-24T13:02:05.056Z}; VoteB: Vote{26:C59779BF393E 123965/00/2(Precommit) 000000000000 /86B7A38AC180.../ @ 2018-05-24T13:02:06.651Z}]" err="Evidence from height 119904 is too old. Min height is 120282"
E[05-30|23:27:25.329] Stopping peer for error module=p2p peer="Peer{MConn{172.105.224.9:46656} 0e8faaecc67d974001dd3b68b439cc397135b229 out}" err="Evidence from height 119904 is too old. Min height is 120282"
I[05-30|23:27:26.843] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{172.105.224.9:46656} 0e8faaecc67d974001dd3b68b439cc397135b229 out}"
I[05-30|23:27:30.336] Ensure peers module=p2p numOutPeers=6 numInPeers=0 numDialing=0 numToDial=4
I[05-30|23:27:30.336] Will dial address module=p2p [email protected]:46656
I[05-30|23:27:30.337] Will dial address module=p2p [email protected]:46656
I[05-30|23:27:30.337] Will dial address module=p2p [email protected]:46656
I[05-30|23:27:30.337] Will dial address module=p2p [email protected]:46656
I[05-30|23:27:30.338] We need more addresses. Sending pexRequest to random peer module=p2p peer="Peer{MConn{54.38.243.139:46656} 4d4acd082f5ad7b1c19cf394b92779846bc9547c out}"
I[05-30|23:27:30.338] Dialing peer module=p2p [email protected]:46656
I[05-30|23:27:30.338] Dialing peer module=p2p [email protected]:46656
E[05-30|23:27:30.343] Dialing failed module=p2p [email protected]:46656 err="Failed to authenticate peer. Dialed [email protected]:46656, but got peer with ID 44631a3e7518e972ad4bb6375fa098796a2e3dbb" attempts=0
I[05-30|23:27:30.344] Remove address from book module=p2p book=/home/puppy/.gaiad/config/addrbook.json [email protected]:46656 ID=5420c43ee76f3c8ebbbd53dc5e28a9072857bde2
I[05-30|23:27:30.344] Ignoring inbound connection: error while adding peer module=p2p address=142.44.240.204:37708 err="Error{Error during handshake (cause: read tcp 142.44.240.204:46656->142.44.240.204:37708: read: connection reset by peer)}"
I[05-30|23:27:30.618] Successful handshake with peer module=p2p peer=138.197.187.76:46656 peerNodeInfo="NodeInfo{id: 6e2b7984aa49cfe8efa63f90ec0c878fe58f9028, moniker: ismail-test, network: gaia-5001 [listen 138.197.187.76:46656], version: 0.19.6 ([amino_version=0.9.9 p2p_version=0.5.0 consensus_version=v1/0.2.2 rpc_version=0.7.0/3 tx_index=on rpc_addr=tcp://0.0.0.0:46657])}"
I[05-30|23:27:30.619] Starting Peer module=p2p peer=138.197.187.76:46656 impl="Peer{MConn{138.197.187.76:46656} 6e2b7984aa49cfe8efa63f90ec0c878fe58f9028 out}"
I[05-30|23:27:30.619] Starting MConnection module=p2p peer=138.197.187.76:46656 impl=MConn{138.197.187.76:46656}
I[05-30|23:27:30.619] Added peer module=p2p peer="Peer{MConn{138.197.187.76:46656} 6e2b7984aa49cfe8efa63f90ec0c878fe58f9028 out}"
E[05-30|23:27:30.744] Connection failed @ recvRoutine (reading byte) module=p2p peer=45.77.53.208:46656 conn=MConn{45.77.53.208:46656} err=EOF
I[05-30|23:27:30.745] Stopping MConnection module=p2p peer=45.77.53.208:46656 impl=MConn{45.77.53.208:46656}
E[05-30|23:27:30.745] Stopping peer for error module=p2p peer="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}" err=EOF
I[05-30|23:27:30.745] Stopping Peer module=p2p peer=45.77.53.208:46656 impl="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}"
I[05-30|23:27:30.788] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}"
I[05-30|23:27:30.788] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}"
E[05-30|23:27:30.799] Connection failed @ recvRoutine (reading byte) module=p2p peer=195.189.195.51:46656 conn=MConn{195.189.195.51:46656} err=EOF
I[05-30|23:27:30.799] Stopping MConnection module=p2p peer=195.189.195.51:46656 impl=MConn{195.189.195.51:46656}
E[05-30|23:27:30.799] Stopping peer for error module=p2p peer="Peer{MConn{195.189.195.51:46656} d3dc165af1ac42a33dad0b37e28f5207b431b013 out}" err=EOF
I[05-30|23:27:30.799] Stopping Peer module=p2p peer=195.189.195.51:46656 impl="Peer{MConn{195.189.195.51:46656} d3dc165af1ac42a33dad0b37e28f5207b431b013 out}"
E[05-30|23:27:30.823] Connection failed @ recvRoutine (reading byte) module=p2p peer=52.78.110.50:46656 conn=MConn{52.78.110.50:46656} err=EOF
I[05-30|23:27:30.823] Stopping MConnection module=p2p peer=52.78.110.50:46656 impl=MConn{52.78.110.50:46656}
E[05-30|23:27:30.823] Stopping peer for error module=p2p peer="Peer{MConn{52.78.110.50:46656} 7093fde2a099b1d3640721373236989c825c29ad out}" err=EOF
I[05-30|23:27:30.824] Stopping Peer module=p2p peer=52.78.110.50:46656 impl="Peer{MConn{52.78.110.50:46656} 7093fde2a099b1d3640721373236989c825c29ad out}"
I[05-30|23:27:30.878] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{195.189.195.51:46656} d3dc165af1ac42a33dad0b37e28f5207b431b013 out}"
I[05-30|23:27:30.879] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{195.189.195.51:46656} d3dc165af1ac42a33dad0b37e28f5207b431b013 out}"
I[05-30|23:27:30.879] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{52.78.110.50:46656} 7093fde2a099b1d3640721373236989c825c29ad out}"
I[05-30|23:27:30.880] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{52.78.110.50:46656} 7093fde2a099b1d3640721373236989c825c29ad out}"
E[05-30|23:27:30.907] Connection failed @ recvRoutine (reading byte) module=p2p peer=167.99.69.153:46656 conn=MConn{167.99.69.153:46656} err=EOF
I[05-30|23:27:30.907] Stopping MConnection module=p2p peer=167.99.69.153:46656 impl=MConn{167.99.69.153:46656}
E[05-30|23:27:30.908] Stopping peer for error module=p2p peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}" err=EOF
I[05-30|23:27:30.908] Stopping Peer module=p2p peer=167.99.69.153:46656 impl="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"
I[05-30|23:27:30.933] Stopping gossipVotesRoutine for peer module=consensus peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"
I[05-30|23:27:30.933] Stopping gossipDataRoutine for peer module=consensus peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"
I[05-30|23:27:30.943] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{52.78.110.50:46656} 7093fde2a099b1d3640721373236989c825c29ad out}"
I[05-30|23:27:31.079] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{167.99.69.153:46656} b908cf8ae2008625180c692d69fe4708a8ebecc7 out}"

need wal file? the node is running now

now gaiacli status not working fast, very slow
and the chain stop latest_block_height":220282, long time ago

from testnets.

is update, since last week or so
~/go/src/github.com/cosmos/cosmos-sdk$ gaiad version
0.17.3-518fe87

~/go/src/github.com/cosmos/cosmos-sdk$ gaiacli version
0.17.3-518fe87

~/go/src/github.com/cosmos/cosmos-sdk$ gaiacli status
{"node_info":{"id":"4d4acd082f5ad7b1c19cf394b92779846bc9547c","listen_addr":"54.38.243.139:46656","network":"gaia-5001","version":"0.19.6-dev","channels":"4020212223303800","moniker":"trust melea V1","other":["amino_version=0.9.9","p2p_version=0.5.0","consensus_version=v1/0.2.2","rpc_version=0.7.0/3","tx_index=on","rpc_addr=tcp://0.0.0.0:46657"]},"sync_info":{"latest_block_hash":"38021A7E96A33D652A22367CDBEB158BEE3AC949","latest_app_hash":"83831FFA777943E9E5A7C559FD7C8AB6A59940CE","latest_block_height":79998,

The chain is here stop, long time ago

Current Block
Block Height
220,282

from testnets.

My two nodes stopped. Before exiting, the following errors appeared a few time.

err="Evidence from height 119904 is too old. Min height is 120282" E[05-31|04:36:23.336] Stopping peer for error module=p2p peer="Peer{MConn{159.89.121.174:46656} b0dd378c3fbc4c156cd6d302a799f0d2e4227201 out}" err="Evidence from height 119904 is too old. Min height is 120282"

This is the same as Melea has posted.

And then after adding a peer, the node "captured terminated, exiting..."

I[05-31|04:36:41.834] Stopping queryMaj23Routine for peer module=consensus peer="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}" I[05-31|04:36:51.006] Successful handshake with peer module=p2p peer=45.77.53.208:46656 peerNodeInfo="NodeInfo{id: aef085c4bfed0c1ffc6705f2e1e3bf85e5164600, moniker: ion.dokia-0, network: gaia-5001 [listen 45.77.53.208:46656], version: 0.19.6-dev ([amino_version=0.9.9 p2p_version=0.5.0 consensus_version=v1/0.2.2 rpc_version=0.7.0/3 tx_index=on rpc_addr=tcp://0.0.0.0:46657])}" I[05-31|04:36:51.006] Starting Peer module=p2p peer=45.77.53.208:46656 impl="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}" I[05-31|04:36:51.006] Starting MConnection module=p2p peer=45.77.53.208:46656 impl=MConn{45.77.53.208:46656} I[05-31|04:36:51.006] Added peer module=p2p peer="Peer{MConn{45.77.53.208:46656} aef085c4bfed0c1ffc6705f2e1e3bf85e5164600 out}" captured terminated, exiting...

If I start the gaiad again, it stops.

I[05-31|05:30:13.173] Time to switch to consensus reactor! module=blockchain height=220283 I[05-31|05:30:13.173] Stopping BlockPool module=blockchain impl=BlockPool I[05-31|05:30:13.173] SwitchToConsensus module=consensus I[05-31|05:30:13.181] Ignoring updateToState() module=consensus newHeight=220283 oldHeight=220283 I[05-31|05:30:13.181] Starting ConsensusState module=consensus impl=ConsensusState I[05-31|05:30:13.183] Starting baseWAL module=consensus wal=/home/kwun/.gaiad/data/cs.wal/wal impl=baseWAL I[05-31|05:30:13.183] Starting TimeoutTicker module=consensus impl=TimeoutTicker panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xacc957]

from testnets.

Node crash stack trace

`I[05-31|07:40:11.948] Starting ABCI with Tendermint module=main
I[05-31|07:40:17.330] Starting multiAppConn module=proxy impl=multiAppConn
I[05-31|07:40:17.330] Starting localClient module=abci-client connection=query impl=localClient
I[05-31|07:40:17.330] Starting localClient module=abci-client connection=mempool impl=localClient
I[05-31|07:40:17.331] Starting localClient module=abci-client connection=consensus impl=localClient
I[05-31|07:40:17.331] ABCI Handshake module=consensus appHeight=108489 appHash=7D9A4A52064AC7E16382C191CD3623AE96E01E8E
I[05-31|07:40:17.331] ABCI Replay Blocks module=consensus appHeight=108489 storeHeight=108489 stateHeight=108489
I[05-31|07:40:17.331] Completed ABCI Handshake - Tendermint and App are synced module=consensus appHeight=108489 appHash=7D9A4A52064AC7E16382C191CD3623AE96E01E8E
I[05-31|07:40:17.332] This node is a validator module=consensus addr=217763B77127EBA9C94F76F4BBD7457C059B5B69 pubKey=PubKeyEd25519{6A1DC9B66040EE06F14A28A63EC2EA42557CE60A4D39404932F9F11B1F1E565A}
I[05-31|07:40:17.397] Starting Node module=node impl=Node
I[05-31|07:40:17.397] Starting EventBus module=events impl=EventBus
I[05-31|07:40:17.397] Local listener module=p2p ip=:: port=46656
I[05-31|07:40:17.397] Getting UPNP external address module=p2p
I[05-31|07:40:20.397] Could not perform UPNP discover module=p2p err="read udp4 0.0.0.0:33044: i/o timeout"
I[05-31|07:40:20.398] Starting DefaultListener module=p2p impl=Listener(@172.105.224.9:46656)
I[05-31|07:40:20.398] P2P Node ID module=node ID=0e8faaecc67d974001dd3b68b439cc397135b229 file=/home/forbole/.gaiad/config/node_key.json
I[05-31|07:40:20.398] Add our address to book module=p2p book=/home/forbole/.gaiad/config/addrbook.json [email protected]:46656
I[05-31|07:40:20.399] Starting RPC HTTP server on tcp://0.0.0.0:46657 module=rpc-server
I[05-31|07:40:20.399] Starting P2P Switch module=p2p impl="P2P Switch"
I[05-31|07:40:20.399] Starting MempoolReactor module=mempool impl=MempoolReactor
I[05-31|07:40:20.399] Starting BlockchainReactor module=blockchain impl=BlockchainReactor
I[05-31|07:40:20.399] Starting ConsensusReactor module=consensus impl=ConsensusReactor
I[05-31|07:40:20.399] ConsensusReactor module=consensus fastSync=false
I[05-31|07:40:20.399] Starting ConsensusState module=consensus impl=ConsensusState
I[05-31|07:40:20.400] Starting baseWAL module=consensus wal=/home/forbole/.gaiad/data/cs.wal/wal impl=baseWAL
I[05-31|07:40:20.400] Starting TimeoutTicker module=consensus impl=TimeoutTicker
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0xaccdf7]

goroutine 1 [running]:
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*baseWAL).SearchForEndHeight(0xc42d985810, 0x1a7ca, 0xc42b588790, 0x0, 0xba6120, 0xeb4930, 0xba6120)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/wal.go:179 +0x2a7
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).catchupReplay(0xc42e4aa000, 0x1a7ca, 0x0, 0x0)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/replay.go:110 +0xbb
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusState).OnStart(0xc42e4aa000, 0xc421505e80, 0x17)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/state.go:264 +0x18f
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start(0xc42e4aa000, 0xcfa400, 0x11)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x3bd
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus.(*ConsensusReactor).OnStart(0xc42e4c0600, 0xc421505c40, 0x19)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/consensus/reactor.go:68 +0x132
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start(0xc42e4c0600, 0x0, 0x0)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x3bd
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p.(*Switch).OnStart(0xc42e4d6000, 0xc421505840, 0x13)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/p2p/switch.go:181 +0xc6
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start(0xc42e4d6000, 0xc424daa430, 0x1)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x3bd
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/node.(*Node).OnStart(0xc4237c6700, 0xc42e4c6f80, 0xd)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tendermint/node/node.go:418 +0x576
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common.(*BaseService).Start(0xc4237c6700, 0xec1de0, 0xc42bfb1680)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/common/service.go:130 +0x3bd
github.com/cosmos/cosmos-sdk/server.startInProcess(0xc4200b6ec0, 0xc4200b6f00, 0x1d, 0x0)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/server/start.go:90 +0x2bd
github.com/cosmos/cosmos-sdk/server.StartCmd.func1(0xc4201a4280, 0xc4201bdc40, 0x0, 0x1, 0x0, 0x0)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/server/start.go:34 +0xaa
github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).execute(0xc4201a4280, 0xc4201bdc20, 0x1, 0x1, 0xc4201a4280, 0xc4201bdc20)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4200eb680, 0xc1c1a0, 0xc420919e01, 0xc4200b70a0)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4200eb680, 0xc4200b70a0, 0xc420919ed8)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/spf13/cobra/command.go:800 +0x2b
github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/cli.Executor.Execute(0xc4200eb680, 0xe4faf0, 0x2, 0xc42092a560)
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/vendor/github.com/tendermint/tmlibs/cli/setup.go:89 +0x4e
main.main()
/home/forbole/go/src/github.com/cosmos/cosmos-sdk/cmd/gaia/cmd/gaiad/main.go:32 +0x209
`

from testnets.

i just update to gaiad version 0.17.3-a5a78eb, now sync the nodes
fast_sync true. working now.

from testnets.