Two nodes which have same configuration have different transaction records about cosmos HOT 12 OPEN

The node record which has specific transaction:
{"height":"5266739","txhash":"31938F53EB5FC2E95396EAF98211B7EF166C0F3B7C938CE7A4DC7418D82CFCA2","data":"0A060A0473656E64","raw_log":"[{\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"send\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"module\",\"value\":\"bank\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"amount\",\"value\":\"1063237205uatom\"}]}]}]","logs":[{"events":[{"type":"message","attributes":[{"key":"action","value":"send"},{"key":"sender","value":"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd"},{"key":"module","value":"bank"}]},{"type":"transfer","attributes":[{"key":"recipient","value":"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw"},{"key":"sender","value":"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd"},{"key":"amount","value":"1063237205uatom"}]}]}],"gas_wanted":"85000","gas_used":"62304","tx":{"type":"cosmos-sdk/StdTx","value":{"msg":[{"type":"cosmos-sdk/MsgSend","value":{"from_address":"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd","to_address":"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw","amount":[{"denom":"uatom","amount":"1063237205"}]}}],"fee":{"amount":[{"denom":"uatom","amount":"2500"}],"gas":"85000"},"signatures":[{"pub_key":{"type":"tendermint/PubKeySecp256k1","value":"Auoah0sTBZ6XM8NekMsqWGWka2TtkNJVVHwHylI5J/xi"},"signature":"+VZl088AFicSJiYj9mU307+8ncE5ah3ZYOmG/zULITUePJ9vhnWnTnOaQSf82yZI+aY9Bss8WxLJNAqnLzSSzg=="}],"memo":"","timeout_height":"0"}},"timestamp":"2021-02-23T23:59:57Z"}

from cosmos.

The node record which didn't has specific transaction
{"error":"tx (31938F53EB5FC2E95396EAF98211B7EF166C0F3B7C938CE7A4DC7418D82CFCA2) not found"}

from cosmos.

This may be because one of the nodes has block pruning while the other does not. Can you double-check this in your app.toml and your application state pruning settings? Did the second node use state sync to catch up? More information context is needed to understand how to help

Can you try searching for the transaction via the tx_search endpoint?

from cosmos.

Hi @raphaellin. Are you absolutely 100% certain both nodes had identical configurations from genesis? Did you perhaps tweak the settings of one node only to a later point in time revert? This is the only logical thing that makes sense.

In addition, can you provide your application pruning settings (found in app.toml) and the Tendermint block pruning settings (found in config.toml) please?

from cosmos.

from cosmos.

This may be because one of the nodes has block pruning while the other does not. Can you double-check this in your app.toml and your application state pruning settings? Did the second node use state sync to catch up? More information context is needed to understand how to help

Can you try searching for the transaction via the tx_search endpoint?

@marbar3778

Got same problem when using 'tx_search' endpoint.
The result on issue node
/gaia curl "http://localhost:26657/tx_search?query=\"tx.height=5266739\"" { "jsonrpc": "2.0", "id": -1, "result": { "txs": [], "total_count": "0" }

The result on right node
/gaia # curl "http://localhost:26657/tx_search?query=\"tx.height=5266739\"" { "jsonrpc": "2.0", "id": -1, "result": { "txs": [ { "hash": "31938F53EB5FC2E95396EAF98211B7EF166C0F3B7C938CE7A4DC7418D82CFCA2", "height": "5266739", "index": 0, "tx_result": { "code": 0, "data": "CgYKBHNlbmQ=", "log": "[{\"events\":[{\"type\":\"message\",\"attributes\":[{\"key\":\"action\",\"value\":\"send\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"module\",\"value\":\"bank\"}]},{\"type\":\"transfer\",\"attributes\":[{\"key\":\"recipient\",\"value\":\"cosmos105hssfshxkdrj4ycpdd5yn00wmscjyfc6vk6zw\"},{\"key\":\"sender\",\"value\":\"cosmos1t5u0jfg3ljsjrh2m9e47d4ny2hea7eehxrzdgd\"},{\"key\":\"amount\",\"value\":\"1063237205uatom\"}]}]}]", "info": "", "gas_wanted": "85000", "gas_used": "62304", "events": [ { "type": "transfer", "attributes": [ { "key": "cmVjaXBpZW50", "value": "Y29zbW9zMTd4cGZ2YWttMmFtZzk2MnlsczZmODR6M2tlbGw4YzVsc2VycXRh", "index": true }, { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true }, { "key": "YW1vdW50", "value": "MjUwMHVhdG9t", "index": true } ] }, { "type": "message", "attributes": [ { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true } ] }, { "type": "message", "attributes": [ { "key": "YWN0aW9u", "value": "c2VuZA==", "index": true } ] }, { "type": "transfer", "attributes": [ { "key": "cmVjaXBpZW50", "value": "Y29zbW9zMTA1aHNzZnNoeGtkcmo0eWNwZGQ1eW4wMHdtc2NqeWZjNnZrNnp3", "index": true }, { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true }, { "key": "YW1vdW50", "value": "MTA2MzIzNzIwNXVhdG9t", "index": true } ] }, { "type": "message", "attributes": [ { "key": "c2VuZGVy", "value": "Y29zbW9zMXQ1dTBqZmczbGpzanJoMm05ZTQ3ZDRueTJoZWE3ZWVoeHJ6ZGdk", "index": true } ] }, { "type": "message", "attributes": [ { "key": "bW9kdWxl", "value": "YmFuaw==", "index": true } ] } ], "codespace": "" }, "tx": "CpYBCpMBChwvY29zbW9zLmJhbmsudjFiZXRhMS5Nc2dTZW5kEnMKLWNvc21vczF0NXUwamZnM2xqc2pyaDJtOWU0N2Q0bnkyaGVhN2VlaHhyemRnZBItY29zbW9zMTA1aHNzZnNoeGtkcmo0eWNwZGQ1eW4wMHdtc2NqeWZjNnZrNnp3GhMKBXVhdG9tEgoxMDYzMjM3MjA1EmUKTgpGCh8vY29zbW9zLmNyeXB0by5zZWNwMjU2azEuUHViS2V5EiMKIQLqGodLEwWelzPDXpDLKlhlpGtk7ZDSVVR8B8pSOSf8YhIECgIIfxITCg0KBXVhdG9tEgQyNTAwEIiYBRpA+VZl088AFicSJiYj9mU307+8ncE5ah3ZYOmG/zULITUePJ9vhnWnTnOaQSf82yZI+aY9Bss8WxLJNAqnLzSSzg==" } ], "total_count": "1" }

from cosmos.

Hi @raphaellin. Are you absolutely 100% certain both nodes had identical configurations from genesis? Did you perhaps tweak the settings of one node only to a later point in time revert? This is the only logical thing that makes sense.

In addition, can you provide your application pruning settings (found in app.toml) and the Tendermint block pruning settings (found in config.toml) please?

Hi @alexanderbez

Yes, the configuration of two nodes are absolutely the same.
and we used default value of pruning settings, can you provide a email so that I can send the configuration file to you.

from cosmos.

sure! [email protected] :)

from cosmos.

@raphaellintoken I took a look at the configurations. They are indeed mainly identical.

However, you're using the default pruning settings. This means you're only going to be able to query the last 100 heights and every 100th after that.

This doesn't explain why one node returns a result and another doesn't, but I bet that the two nodes are just slightly out of sync and so one node prunes blocks at a slightly different rate than the other.

I recommend not using the default setting, but something custom. Perhaps consider keeping enough blocks within the unbonding period.

e.g. if your unbonding period is 3 weeks, then:

--pruning=custom --pruning-keep-recent=362880 --pruning-keep-every=0 --pruning-interval=100

from cosmos.

Thanks @alexanderbez , we will try.
But I think it's better to find the root cause of this issue. other users may also meet this issue. Do you want to have more investigation？ we can supply more data what you need.

from cosmos.

Unfortunately not, as this is the only thing I can think of. Also, we have not seen any other reports like this.

Out of curiosity, the heights you are querying, how far back are they from the latest block? Less than 100?

from cosmos.

When we found this issue, it is more than 100 from the latest block.

from cosmos.