notional-labs / cosmosia Goto Github PK

osmosis-labs/osmosis#695

tendermint/tendermint#8424

if DNS point to a single IP address, bandwidth is limited by the physical etherenet port.
Ech server has single 1Gb port, ~100MB/s downloading snapshot.
If there are 2 clients downloading snapshot, each got 50MB/s. If there are 10 clients downloading snapshot, each got 10MB/s...So it does not scale.

The purpose of this task is to increase bandwidth by using DNSRR point to IP of multiple servers in the cluster. If there are 10 servers with 1Gb port, total bandwidth now is 10Gb ~ 1GB/s.

As servers could be up/down/add/remove at runtime, need a script to monitor and update DNS records automatically.

pebbledb has been running for 8 hours, its stable now.

[root@e6047691c84a data]# ps -aux |grep sif
root      211878  5.5  0.7 7771656 1035072 pts/1 Sl+  10:28  27:35 /root/go/bin/sifnoded start --db_backend=pebbledb --p2p.persistent_peers=23923a2ae66653984e3e35d2ce8c1e1e0e1d4473@tasks.sss_sifchain:26656
root      217915  0.0  0.0   4092  2292 pts/4    S+   18:41   0:00 grep sif
[root@e6047691c84a data]# ps -p 211878 -o etime
    ELAPSED
   08:13:59

Good to do some benchmark to compare to goleveldb/rocksdb.

https://github.com/baabeetaa/cosmprund/blob/add_apps/cmd/pruner.go#L118

1. Osmosis (osmosis) ✅
2. Starname (starname) ✅
3. Regen (regen) ✅
4. Akash (akash) ✅
5. Gaia (cosmoshub) ✅
6. Sentinel (sentinel) ✅
7. E-Money (emoney) ✅
8. Ixo (ixo) ✅
9. Juno (juno) ✅
10. Sifchain (sifchain) ✅
11. Likecoin (likecoin) ✅
12. Ki (kichain) ✅
13. Cyber (cyber) <<== need test
14. Cheqd (cheqd) <<== need test
15. Stargaze (stargaze) <<== need test
16. Band (bandchain) <<== need test
17. Chihuahua (chihuahua) <<== need test
18. Kava (kava) ✅
19. BitCanna (bitcanna) <<== need test
20. Konstellation (konstellation) <<== need test
21. Omniflix (omniflixhub) <<== need test
22. Terra (terra) ✅
23. Vidulum (vidulum) <<== need test
24. Provenance (provenance) <<== need test
25. Dig (dig) <<== need test
26. Gravity-Bridge (gravitybridge) ✅
27. Comdex (comdex) <<== need test
28. Cerberus (cerberus) <<== need test
29. BitSong (bitsong) <<== need test
30. AssetMantle (assetmantle) <<== need test
31. FetchAI (fetchhub) <<== need test
32. Evmos (evmos) ✅
33. Persistence (persistent) <<== need test
34. Crypto.org (cryptoorgchain) <<== need test
35. IRISnet (irisnet) <<== need test
36. Axelar (axelar) <<== need test
37. Pylons Testnet (pylons) <== ignore
38. Umee (umee) <<== need test
39. Sei-Chain Testnet (sei) <== ignore

The purpose is to create a pruned snapshot based on time ( eg interval weekly) or size ( a threshold eg., snapshot size Larger than 200 GB )

There are 3 ways to do:

1. Download pruned snapshot from external providers
2. Using cosmos-pruner (#44)
3. Statesync (#37)

umee

each chain has multiple rpc instances and a snapshot instance.
Atm, these instances dont peer to each other, All sync from internal network.

The purpose of this task is to make them auto discover and peer to each others.

https://github.com/cosmos/chain-registry/tree/master/irisnet

Osmosis (osmosis): OK
Starname (starname): OK
Regen (regen): OK
Akash (akash): OK
Gaia (cosmoshub): OK
Sentinel (sentinel): <<< Error
E-Money (emoney): OK
Ixo (ixo): OK
Juno (juno): OK
Sifchain (sifchain): OK
Likecoin (likecoin): OK
Ki (kichain): OK
Cyber (cyber): OK
Cheqd (cheqd): OK
Stargaze (stargaze): OK
Band (bandchain): OK
Chihuahua (chihuahua): OK
Kava (kava) <<< Error
BitCanna (bitcanna): OK
Konstellation (konstellation): OK
Omniflix (omniflixhub): OK
Terra (terra): OK
Vidulum (vidulum): OK
Provenance (provenance): OK
Dig (dig): OK
Gravity-Bridge (gravitybridge): OK
Comdex (comdex): <<< Error
Cerberus (cerberus): OK
BitSong (bitsong): OK
AssetMantle (assetmantle): <<< Error
FetchAI (fetchhub): <<< Error
Evmos (evmos): OK

alternative to re-statesync and re-download pruned snapshot.
Statesync doesnt work with all chains (https://github.com/notional-labs/cosmosia/blob/main/docs/statesync.md).

This task is to try it. If it works, then use it for chains which dont support statesync.

1. Osmosis (osmosis) ❌
2. Starname (starname) ❌
3. Regen (regen) ✅
4. Akash (akash) ✅
5. Gaia (cosmoshub) ✅
6. Sentinel (sentinel) ✅
7. E-Money (emoney) ✅
8. Ixo (ixo) ✅
9. Juno (juno) ❌
10. Sifchain (sifchain) ✅
11. Likecoin (likecoin) ✅
12. Ki (kichain) ✅
13. Cyber (cyber) ❌
14. Cheqd (cheqd) ❌
15. Stargaze (stargaze) ❌
16. Band (bandchain) ✅ Note: it works if copy the files folder to
17. Chihuahua (chihuahua) ✅
18. Kava (kava) ✅
19. BitCanna (bitcanna) ✅
20. Konstellation (konstellation) ✅
21. Omniflix (omniflixhub) ✅
22. Terra (terra) ❌
23. Vidulum (vidulum) ✅
24. Provenance (provenance) ✅
25. Dig (dig) ✅
26. Gravity-Bridge (gravitybridge) ✅
27. Comdex (comdex) ✅
28. Cerberus (cerberus) ✅
29. BitSong (bitsong) ✅
30. AssetMantle (assetmantle) ✅
31. FetchAI (fetchhub) ❌
32. Evmos (evmos) ✅
33. Persistence (persistent) ✅
34. Crypto.org (cryptoorgchain) ✅
35. IRISnet (irisnet) ✅

RPC instances now peer to each other already ( #34 ) but snapshot node does not.

This task is to make the snapshot node peer to rpc instances for faster sync.

Make it easier to find bad instance and fix.

need it works with 3 versions of rocksdb: v6.15.5 v6.24.2 v7.1.2

To make sure all RPC instance of a service running the same snapshot version (the same data and settings for all instances).

Snapshot file can be synced from external providers, statesync.
Rpc service syncs from internal snapshot service only.

However, snapshot can be pruned by (cosmos-pruner, re-statesync or re-download). Everytime prunning snapshot, need to update data all rpc instances. The issue is this update process.

The easy way is just re-start the RPC service. But there is long down-time. Snapshot data can be a few GB to hundreds of GB, so the update time could be very long (few mins to hours).

Need a solution to update RPC data without downtime ( or very small - seconds to few minutes).

The idea of snapshot versioning is that a new RPC service (with new snapshot data) is created. When everything ready, update the load-balancer to switch to the new RPC service. Then remove the old RPC service.

#70

keep the snapshot-interval high ~ 1 day so wont hurt performance.

We have decided not to do anything with AssetMantle. Pls remove it out of the system

v2.5.1 works fine
v2.5.2 stops working

a quick fix is just running v2.5.1

We already have list of chains in the chain_registry.ini.

This task is to generate the https://github.com/notional-labs/cosmosia/blob/main/proxy_public/snapshot.index.html dynamically when starting the proxy.

Help to reduce a step when adding a new chain https://github.com/notional-labs/cosmosia/blob/main/docs/new_chain.md

http://127.0.0.1:26657/genesis
http://127.0.0.1:26657/genesis_chunked?chunk=0

remove docs, snapshot, scripts...

• will do sifchain as it has highest traffic
• log the requests to file
• replay log file to test load

Some chains have memory leak issue. Need a cronjob to restart the chain periodically.

pruned bandchain of quicksync is huge ~300GB
https://quicksync.io/networks/band.html

https://github.com/cosmos/chain-registry/tree/master/injective

Genesis and snapshot files are huge. Eg., cosmoshub-4-pruned is 68 GB (https://quicksync.io/networks/cosmos.html).

The proxy with cache helps:

• avoid downloading the same files over and over
• speed up creating a service.

https://github.com/sei-protocol/sei-chain/releases/tag/1.0.6beta
chain-id: atlantic-1
https://raw.githubusercontent.com/sei-protocol/testnet/main/sei-incentivized-testnet/genesis.json
https://raw.githubusercontent.com/sei-protocol/testnet/main/sei-incentivized-testnet/addrbook.json
peer: [email protected]:2420
https://rpc-sei-ia.notional.ventures/

simple tool like https://github.com/terra-money/level-to-rocks/
which converts goleveldb databases to pebbledb databases.

External status page is already available at https://status.notional.ventures/status/cosmosia

However uptime-kuma doesnt have api atm.

So we'll build a simple tool to use.

Desc:

• there are multiple rpc services
• each rpc service has multiple containers
• the number of containers for each service is dynamic (up/down at runtime)
  => need to monitor health status for each container

Design:

• Use docker swarm api to get info of services and containers
• Bash script to check health
• Write status to text/json file
• serve status text/json file with nginx
• html page for better view

https://github.com/notional-labs/cosmosia/blob/main/docs/rocksdb.md

1. Osmosis (osmosis) ❌ (didnt test as statesync doesnt work)
2. Starname (starname) ❌ (didnt test as statesync doesnt work)
3. Regen (regen) ✅ (v6.15.5)
4. Akash (akash) ✅ (v6.24.2)
5. Gaia (cosmoshub) ✅ (v6.24.2)
6. Sentinel (sentinel) ✅ (v6.15.5)
7. E-Money (emoney) ✅ (v6.15.5)
8. Ixo (ixo) ✅ (v6.15.5)
9. Juno (juno) ❌ (didnt test as statesync doesnt work)
10. Sifchain (sifchain) ✅ (v6.15.5)
11. Likecoin (likecoin) ✅ (v6.15.5)
12. Ki (kichain) ✅ (v6.15.5)
13. Cyber (cyber) ❌ (didnt test as statesync doesnt work)
14. Cheqd (cheqd) ❌ (didnt test as statesync doesnt work)
15. Stargaze (stargaze) ❌ (didnt test as statesync doesnt work)
16. Band (bandchain) ✅ (v6.15.5)
17. Chihuahua (chihuahua) ✅ (v6.15.5)
18. Kava (kava) ✅ (v7.1.2)
19. BitCanna (bitcanna) ✅ (v6.24.2)
20. Konstellation (konstellation) ✅ (v6.15.5)
21. Omniflix (omniflixhub) ✅ (v6.15.5)
22. Terra (terra) ❌ (didnt test as statesync doesnt work)
23. Vidulum (vidulum) ✅ (v6.15.5)
24. Provenance (provenance) ✅ (v6.24.2)
25. Dig (dig) ✅ (v6.24.2)
26. Gravity-Bridge (gravitybridge) ✅ (v6.15.5)
27. Comdex (comdex) ✅ (v6.15.5)
28. Cerberus (cerberus) ✅ (v6.15.5)
29. BitSong (bitsong) ✅ (v6.15.5)
30. AssetMantle (assetmantle) ✅ (v6.24.2)
31. FetchAI (fetchhub) ❌ (didnt test as statesync doesnt work)
32. Evmos (evmos) ✅ (v6.24.2)
33. Persistence (persistent) ⚠️ (v6.15.5) application.db still in leveldb
34. Crypto.org (cryptoorgchain) ✅ (v6.15.5)
35. IRISnet (irisnet) ✅ (v6.15.5)
36. Axelar (axelar) ⚠️ (v6.24.2) application.db still in leveldb

https://github.com/Pylons-tech/pylons

1 Osmosis (osmosis) ✅
2 Starname (starname) ✅
3 Regen (regen) ✅
4 Akash (akash) ✅
5 Gaia (cosmoshub) ✅
6 Sentinel (sentinel) ✅
7 E-Money (emoney) ✅
8 Ixo (ixo) ✅
9 Juno (juno) ✅
10 Sifchain (sifchain) ✅
11 Likecoin (likecoin) ✅
12 Ki (kichain) ✅
13 Cyber (cyber) ✅
14 Cheqd (cheqd) ✅
15 Stargaze (stargaze) ✅
16 Band (bandchain) ✅
17 Chihuahua (chihuahua) ✅
18 Kava(kava) ✅
19 BitCanna (bitcanna) ✅
20 Konstellation (konstellation) ✅
21 Omniflix (omniflixhub) ✅
22 Terra (terra) ✅
23 Vidulum (vidulum) ✅
24 Provenance (provenance) ✅
25 Dig (dig) ✅
26 Gravity-Bridge (gravitybridge) ✅
27 Comdex (comdex) ✅
28 Cerberus (cerberus) ✅
29 BitSong (bitsong) ✅
30 AssetMantle (assetmantle) <== wait for #80
31 FetchAI (fetchhub) ✅
32 Evmos (evmos) ✅
33 Persistence (persistent) ✅
34 Crypto.org (cryptoorgchain) ✅
35 IRISnet (irisnet) ✅
36 Axelar (axelar) ✅
37 Pylons Testnet(pylons) ✅
38 Umee (umee) ✅
39 Sei-Chain Testnet (sei) ✅
40 Evmos Testnet Archive(evmos-testnet-archive) ✅

https://github.com/cosmos/chain-registry/blob/master/axelar/chain.json

got 502 Bad Gateway

its just the proxy-to-loadbalancer issue, there is no issue when testing on load-balancer directly.

<body>
<h1>An error occurred.</h1>
<p>Sorry, the page you are looking for is currently unavailable.<br/>
Please try again later.</p>
<p>If you are the system administrator of this resource then you should check
the error log for details.</p>
<p><em>Faithfully yours, nginx.</em></p>
</body>

https://github.com/CosmosContracts/incident-response/blob/main/28-July-22/upgrade.md

• https://github.com/cosmos/chain-registry/tree/master/persistence
• https://github.com/cosmos/chain-registry/tree/master/cryptoorgchain