Git Product home page Git Product logo

discv4-dns-lists's Introduction

This repository is a downstream fork of ethereum/discv4-dns-list that includes DNS discovery information for ETC networks. The content of this repository and the associated live DNS ENR records are updated by a scheduled Github Action configured here, and with output visible under the Actions tab.

Source for the devp2p tool mentioned below which support ETC network filters can be found at etclabscore/core-geth, which is also referenced in this repository as a submodule at ./core-geth.

Rather than ethdisco.net, the records held under this repository are published to the blockd.info DNS name.

discv4-dns-lists

This repository contains EIP-1459 node lists built by the go-ethereum devp2p tool. These lists are published to the ethdisco.net blockd.info DNS name.

The nodes in the lists are found by crawling the Ethereum node discovery DHT. The entire output of the crawl is available in the all.json file. We create lists for specific blockchain networks by filtering all.json according to the "eth" ENR entry value provided by each node.

If you want your node in the list, simply run your client and make sure it is reachable through discovery. The crawler will pick it up and sort it into the right list automatically.

discv4-dns-lists's People

Contributors

fjl avatar meowsbits avatar whilei avatar ziogaschr avatar

Stargazers

avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar

Forkers

mrblocktw yuriy0803 pinkdiamond1

discv4-dns-lists's Issues

CI failing more than usual

#15
#16
#17

These (and several prior, I believe) fail because the git push is rejected for being behind the remote.

My best guess is this is happening because of a large git store and not enough time allowed between CI jobs (currently 3 hours).

Possible solutions:

  • Larger time interval between job starts (eg. 3 -> 4 hours).
  • Shallower checkout depth.
  • Squash trunk branch history to reduce overall git store size.

use Github Actions environment files

Actions were failing because of https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/, eg. https://github.com/etclabscore/discv4-dns-lists/actions/runs/388796987.

This commit 23686fe is a workaround using the error message instructions:

The add-path command is disabled. Please upgrade to using Environment Files or opt into unsecure command execution by setting the ACTIONS_ALLOW_UNSECURE_COMMANDS environment variable to true. For more information see: https://github.blog/changelog/2020-10-01-github-actions-deprecating-set-env-and-add-path-commands/.

... but is a temporary fix.

FYI: statistics/visualization script

Here's a tiny script(s) that crunches the numbers on node record counts over time. Except for spark it's pretty out-of-the-box portable.

> for f in $(find . -type f -name '*stats.sh'); do echo;echo '>>>>>>'$f;echo; cat $f; done

>>>>>>./stats.sh

#!/usr/bin/env bash

# Does basic statistics and visualization for node record counts.
# Statistics are aggregated at the DAILY granularity, traversing as far back as the second argument value.
# 
# Use: ./script.sh <nodes.json> <days>
# 
# Where 'days' is the number of days back from now.

> count.txt

for i in $(seq 1 $2); do
  commit=$(git rev-list --max-count 1 --before="$(date --date '-'"$i"' days' --iso-8601=minutes)" etccore)
  n=0
  if [[ ! -z $commit ]]; then
    git checkout $commit -- $1
    n=$(cat $1 | grep record | wc -l)
  else
    echo skipping "$i (set to 0)"
  fi
  echo "$n" >> count.txt
done

sed '1!G;h;$!d' count.txt | spark

git checkout etccore -- $1

echo latest count:$(cat $1 | grep record | wc -l)

# https://stackoverflow.com/a/9790056
perl -e 'use List::Util qw(max min sum); @a=();while(<>){$sqsum+=$_*$_; push(@a,$_)}; $n=@a;$s=sum(@a);$a=$s/@a;$m=max(@a);$mm=min(@a);$std=sqrt($sqsum/$n-($s/$n)*($s/$n));$mid=int @a/2;@srtd=sort @a;if(@a%2){$med=$srtd[$mid];}else{$med=($srtd[$mid-1]+$srtd[$mid])/2;};print "records:$n\nsum:$s\navg:$a\nstd:$std\nmed:$med\nmax:$m\min:$mm";' < count.txt | grep -v sum

rm count.txt

>>>>>>./allstats.sh

#!/usr/bin/env bash

# Use: ./allstats.sh 100 > STATS.md

for f in $(find . -type f -name '*nodes.json' | sort); do
  echo "# $f"
  echo -e \`\`\`
  ./stats.sh "$f" $1
  echo -e \`\`\`
  echo 
done

./allstats.sh 100 > STATS.md && cat STATS.md

./all.classic.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▃▃▃▃▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▄▄▅▇█
latest count:352
records:100
avg:245.72
std:26.7451229198896
med:246
max:372
min:200

./all.kotti.blockd.info/nodes.json

▁▁▁▁▁▃▃▃▃▃▅▅▅▅▅▅▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▂▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▆█▆▆
latest count:9
records:100
avg:8.86
std:1.32680066324977
med:7
max:12
min:7

./all.mordor.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▂▂▂▂▂▂▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▃▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▄▄▄▄▄▄▄▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▆▆▆▇█
latest count:16
records:100
avg:9.08
std:2.71175220106853
med:4
max:16
min:4

./les.classic.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▅▇▇▅▇█▄▁▁▁▁▁▁▂▂▂▂▂▂▂▂▂▂▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁
latest count:2
records:100
avg:1.66
std:3.03387540943922
med:1
max:15
min:0

./les.kotti.blockd.info/nodes.json

▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅▅
latest count:1
records:100
avg:1
std:0
med:1
max:1
min:1

./les.mordor.blockd.info/nodes.json

▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁████████████████████████████████████████████████████████████████████████████████████
latest count:1
records:100
avg:0.84
std:0.366606055596467
med:1
max:1
min:0

move list generation from server daemon to CI

These lists are currently being updated by a server running a daemon that no one can see.

It would be good to move this process to a public space so that the provenance of the lists are transparent and the process reproducible. And I can worry less about monitoring or managing the server.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.