Goal: discover and record version message for as many live nodes as we can

Just run it once.

Ignoring: duration of connection, who told us which addresses (topology),

Should we attempt to prevent duplicates?

• if yes, this would be main thread's job. perhaps workers could just send back (VersionMessage(), AddrMessage()). Main thread can persist VersionMessage and feed any new addresses into the address queue. But it would need to filter out duplicates using sql at this point ...
• I tend to say "no". This is an optimization. Maybe do this if there is time left over ...

Should I even use a queue? How to test a minimal program???

Main thread

• Start workers
• Seed addresses queue
• loop
  • just needs to read from result queue and save them into sqlite

Should worker thread add a timestamp?

Should I make a separate table to store every IP observed? Basically a backup for the queue?

Done

• Eliminate results queues

Next

• create_table
• save_observation

Others

• get_node_count()
• get_protocol_distribution()
  • ipv4, ipv6, tor

addresses table: id, host, port

observations table: all attributes of version message, timestamp, perhaps the list of addresses shared?

• an observations table would still allow us the flexibility to crawl multiple times and see what changes ...

what to do when the connection fails? or we can't connect for some reason? just throw out the address or add it to a blacklist? The simplest thing is to only keep track of successes ...

Logistics

• Make a separate server for just me during the presentation. This way we can have massive congestion and the talk still goes well.
• Deploy more than 1 server
• Use a HUGE box
• Tune the resource limits?
• Move the open file limit to like 100000.

Should I copy the whole git repository to /etc/skel? This way they wouldn't have to do any binder setup ...

Install "tlj" on Digital Ocean

Open terminal

sudo -E pip install --upgrade pip
sudo -E pip install PySocks requests pytest

Start Tor and check that it is running

sudo netstat -plnt | grep 9050

sudo tljh-config set auth.type firstuseauthenticator.FirstUseAuthenticator 
sudo tljh-config reload

tljh-config set auth.FirstUseAuthenticator.create_users true
tljh-config reload

This are the base files everyone gets. I should probably just keep the repo in here ... Then I can git pull right before the talk starts and everyone should get up-to-date code ...

• How to reset user passwords

Python is installed here: /opt/tljh/user/bin/python3