codechain-io / cron-jobs Goto Github PK

View Code? Open in Web Editor NEW

4.0 6.0 8.0 2.51 MB

Cron jobs using CodeChain network

License: ISC License

TypeScript 95.81% Shell 1.12% JavaScript 3.06%

cron script monitoring

cron-jobs's Introduction

cron-jobs

Cron jobs using CodeChain network

cron-jobs's People

Contributors

Stargazers

Watchers

Forkers

hoongee foriequal0 sgkim126 majecty kseo joojis msntcs

cron-jobs's Issues

[indexer-monitor] Reduce block sync test period

if the block sync stopped more than 5 minutes, indexer-monitor sends an error message.
The original intention was 1 hour.

Use environment instead of config

Currently, there are two ways to set up the configuration, because each tool is written by the various developers.
One is using config and the other is using environments.
Some developers, including me, prefer config to environments because the environments vanish when the session closes.
And others prefer environments because it corrupts the git history and it's still hard to track because it depends on environments to choose the configuration file.

But now, we can keep the environment because we use tmuxinator, so I think it's better to unify styles by using environment variables.

Make the helicopter set the max probability for rich account not to monopolize rewards

Send validators' CCC if it exceeds the threshold, or the CCC is accumulated more than a day

[fee-monitor] Fee calculation error in a term's last block

The fee monitor said that there is a fee calculation error in the CodeChain Main network.

I checked the stakeholder's balance at the block, there was no error at the block.
There is a transaction with 100 CCC fee and each stakeholder received 10 CCCs.

I copied the error message:

{
   "server": "mainnet",
   "moment": {
       "tag": "block",
       "value": 5343247
   },
   "errors": [
       {
           "type": "stakeholder",
           "account": "cccqyhk4388mymckk40ctf0skgkl8nf89f6sgpzwzke",
           "expected": "10010145050",
           "actual": "10010145030"
       },
       {
           "type": "stakeholder",
           "account": "cccq9g0ayvrvmzcs2qf8yygat0dg9073g5xccd3sydw",
           "expected": "10010144950",
           "actual": "10010144930"
       },
       {
           "type": "stakeholder",
           "account": "cccq9hpz4m5sx0ea42l2hfm8nja0xzzsnx0vv0p7mzw",
           "expected": "10010144850",
           "actual": "10010144830"
       },
       {
           "type": "stakeholder",
           "account": "cccqxgwashdy0fp5cwlvp4ntkzmy3xvjedfjgfzrxf7",
           "expected": "10010144850",
           "actual": "10010144830"
       },
       {
           "type": "stakeholder",
           "account": "cccqxgupxd5j0squ7t8w3etxjf0fn2z4axz0spycqum",
           "expected": "10010145250",
           "actual": "10010145230"
       },
       {
           "type": "stakeholder",
           "account": "cccqx7ycl2awx6awqyjf9ns4532xj7eg63w6v6hlnu0",
           "expected": "10010144850",
           "actual": "10010144830"
       },
       {
           "type": "stakeholder",
           "account": "cccq8r9as2drgp4xwfzz7t0xc9k977y8mw7hus3zq8e",
           "expected": "10010144950",
           "actual": "10010144930"
       },
       {
           "type": "stakeholder",
           "account": "cccq826j0p6fvpk02p0n5th7ukqu5clrnyldcxgrgtz",
           "expected": "10010144850",
           "actual": "10010144830"
       },
       {
           "type": "stakeholder",
           "account": "cccq8dhpcfqt94mx8dntxxd6420782vtd4tpvpgcgjh",
           "expected": "10010145250",
           "actual": "10010145230"
       },
       {
           "type": "stakeholder",
           "account": "cccq8u8se8k00vcqk4pl707qe9v60hs3jj27g4rqkm9",
           "expected": "9417937350",
           "actual": "9417937330"
       }
   ]
}

Reduce unused UTXOs in the timelock checker

Use additional transaction signer in the timelock tester

TimeLock test creates a transaction that goes a little far in the future and creates a transaction that can pass right away. However the two transactions are signed by the same account and using the same sequence, the second transaction may not be processed as intended.

Fee monitor couldn't handle transactions value 0.

I found this error in the fee monitor in Corgi network.

Here is the log.

BlockNumber: 34926
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 1 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 2 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 3 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 4 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 5 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 6 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 7 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 8 sec(s)
TxTypes: custom
Retry Error: U64 must be a positive integer but found 0x. wait for 9 sec(s)
TxTypes: custom
Too many retries: 10
{ error: Error: U64 must be a positive integer but found 0x
    at new U64 (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:16:19)
    at Function.ensure (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:104:47)
    at Object.decodeU64 (/home/ubuntu/fee-monitor/fee-monitor/build/Stake.js:16:26)
    at DynamicChecker.handleTransactions (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:199:67)
    at DynamicChecker.checkBlockDynamic (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:70:20)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7) }
(node:22669) UnhandledPromiseRejectionWarning: Error: U64 must be a positive integer but found 0x
    at new U64 (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:16:19)
    at Function.ensure (/home/ubuntu/fee-monitor/fee-monitor/node_modules/codechain-primitives/lib/value/U64.js:104:47)
    at Object.decodeU64 (/home/ubuntu/fee-monitor/fee-monitor/build/Stake.js:16:26)
    at DynamicChecker.handleTransactions (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:199:67)
    at DynamicChecker.checkBlockDynamic (/home/ubuntu/fee-monitor/fee-monitor/build/DynamicChecker.js:70:20)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
(node:22669) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 21)
(node:22669) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
fee-monitor has been stalled for 30 seconds:{
    "blockNumber": 34926,

Make the auto-self-nominate send an alarm when the account is jailed or banned

Reduce the number of alarms when two or more nodes do not continue to generate precommit

Send a slack notification immediately

Hi @HoOngEe,

cron-jobs/monitor/SlackNotify.ts

Lines 28 to 30 in eb1cb0f

 this.sendDebounced = _.debounce(() => { 

 this.send(); 

 }, 1500);

The current implementation waits 1.5 seconds before sending a notification.
Would you explain the reason?

[monitor] Send a message when the node starts not signing blocks consecutively or when it has recovered

Currently, it sends a notification forever when a node dies.

[fee-monitor] Stalls sometimes

I found that fee-monitor has been stalled without throwing error or logging output.
We should add a watchdog to fee-monitor.

Monitor indexer status

Describe how to run move-ccc in README.md

Fee monitor does not recover when RPC requests fails temporarily

I'm suspecting the line below:

cron-jobs/fee-monitor/src/index.ts

Line 200 in eda1e3e

blockNumber = await getNextBlockNumber(blockNumber);

Fee distribution warning messages need more details

When fee-monitor finds mismatches between fee-monitor's fee distribution and CodeChain's fee distribution, it sends warning messages through slack. But information in those messages are not enough to analyze the problem directly.

Send a notification to slack and email when a tool starts

To check the tool's notification setting is working, a tool needs to send a notification when it starts.

Add metadata to logs and notifications

We need more information in logs and notifications

timestamp
last observed block height
machine info (ip, aws/gcp machine name, etc)

[fee-monitor] Fee distribution error found on corgi network.

[warn][wc][fee-monitor] finds a problem
{
   "server": "corgi",
   "moment": {
       "tag": "term",
       "value": 469870
   },
   "errors": [
       {
           "type": "validator",
           "account": "wccq9hv8e73frxr2c8txp5snu7gd4pth3z92ganlnz8",
           "expected": "9999990104170",
           "actual": "9999990104155"
       },
       {
           "type": "validator",
           "account": "wccq9gw2hdwfh6et35calxp90lft4qfwmdxpsd2dq5y",
           "expected": "9999990124969",
           "actual": "9999990124939"
       },
       {
           "type": "validator",
           "account": "wccqxmehv99je74c9gfmk9fy4m7qp6rwfcgdq02x7hm",
           "expected": "9999990054265",
           "actual": "9999990054204"
       }
   ]
}

The cause of the problem is an error when getting previous term's validators.
actual:

sortedValidators [ [ 'wccqxmehv99je74c9gfmk9fy4m7qp6rwfcgdq02x7hm' ],                                       
  [ 'wccq9gw2hdwfh6et35calxp90lft4qfwmdxpsd2dq5y' ],                                                        
  [ 'wccq9hv8e73frxr2c8txp5snu7gd4pth3z92ganlnz8' ] ]

expected:

sortedValidators [ [ 'wccqxmehv99je74c9gfmk9fy4m7qp6rwfcgdq02x7hm',
 'wccq9l5ly2ppsddta4gynmy05acxzxp9h2mqvqyzufh'],                                       
  [ 'wccq9gw2hdwfh6et35calxp90lft4qfwmdxpsd2dq5y' ],                                                        
  [ 'wccq9hv8e73frxr2c8txp5snu7gd4pth3z92ganlnz8' ] ]

Need to change term change judgement

Following the codechain issue#1710, the fee-monitor's logic should be changed correspondingly.

Make monitor support dynamic validators

cron-jobs/monitor/index.ts

Line 161 in 6ed9a56

const validatorCount = 30;

Currently, it's hardcoded as 30.

Add diagnostic logs to monitor

Fee monitor "Internal error" should be more specific

Current some error messages only send "Internal error" but it should be more specific.

Add specific reason why the account is banned in auto-self-nominate

When one tries to run auto-self-nominate script with banned account, currently it send the message : "Your account ${account} is permanently banned".
But It would be better to show the reason why the account is banned.

Remove hard-coded from emails

Like [email protected]

Report the balance of validators

Make a script that sends a report when the balance of accounts are out of range.
And I think it's good if it sends a report daily.

Script to send CCC when a hot wallet has more than the threshold

We need a script to send CCC from a hot wallet to a cold wallet.

[auto-self-nominate] Don't constantly retry self-nominate and send noti while in the jail / banned

Fix helicopter

I stopped the helicopter because it generates lots of pending transactions.
Fix and rerun it.

Make fee-monitor to use U64 type when calculating rewards

Currently, fee-monitor is using javascript number type when calculating rewards for validators. But there is a potential hazard of overflow when multiplying a number and dividing later to integrate round-off errors into one.

[fee-monitor] dynamicChecker should be atomic

dynamicChecker class uses internal states. If a connection error occurs in the middle of fee distribution checking routine, retry logic re-call the function checkBlock with the parameter dynamicChecker which has partially updated state. That may leads to an block error and a term error.

Document how to update

Each tool's README.md file should have a section about how to update.

Reorganize monitor notifications

Currently, too much notifications are sending to both slack and e-mail. I think we should divide it into slack only notifications and e-mail only notifications.

Make cron jobs send a daily report

Auto self nominate tries to self nominate eventhough it always fails

Add a REAMD.md file to auto-self-nominate.

Add a script to check the block reward

Script to check whether the block reward is distributed as specified.

Auto self nomination crashed while printing information after first nomination

Here is the log from the auto-self-nominate.

sdk wc http://127.0.0.1:8080
Best block number: 2523960
Current Term Id: 0
Deposit Supplement: 10,000,000
Self Nomination tx sent: f9b72fbd75d98ea9e423456bd86b24b0861e96b5c3f1d17d12a115d83176179e
Candidate info
  TypeError: Cannot read property 'deposit' of undefined
      at send (/home/ubuntu/cron-jobs/auto-self-nominate/build/index.js:150:46)
      at processTicksAndRejections (internal/process/task_queues.js:85:5)

Hard to sustain state given by self-nomination transactions

To calculate nomination deposit returns, fee-monitor had to watch the self-nomination transaction and save the deposit quantity to fee-monitor's own state. But to calculate it correctly, minimum duration of running fee-monitor is about "11 days". Because the path from the state "Jailed" to "Eligible" takes 240 terms to transit.

If update fee-monitor and restart it in the middle, there may be some warnings about balance expectation fails.

Allow `monitor` and `fee-monitor` to run without notification settings

monitor and fee-monitor script currently require following environment variables to be set:

SENDGRID_API_KEY
SENDGRID_TO
SLACK_WEBHOOK_URL

There are some cases that the user wants to run the script, but doesn't want to receive notifications. (eg. in local test network)
I think printing warning messages is enough for this case.

[val-val] Unhandled Promise Rejection because of min validator

(node:7171) UnhandledPromiseRejectionWarning: Error: The number of validators is 2 at 463078. min: 4
    at Object.check [as default] (/home/beagle/val-val/val-val/build/election.js:57:15)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:7171) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
(node:7171) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
Error: The number of validators is 2 at 463078. min: 4
Error: The number of validators is 2 at 463078. min: 4
    at Object.check [as default] (/home/beagle/val-val/val-val/build/election.js:57:15)
    at process._tickCallback (internal/process/next_tick.js:68:7)
(node:7171) PromiseRejectionHandledWarning: Promise rejection was handled asynchronously (rejection id: 1)

	this.sendDebounced = _.debounce(() => {
	this.send();
	}, 1500);