@todo

• turn little tests and experiments which involve multiple "nodes" into proper tests inside a lab
• use dat and substrate testing frameworks systematically

1. I would recommend a lab/ folder with multiple named lab/<scenarioNameFolder>/s inside.
2. each one (e.g. lab/supporter-encoder-seeder/) would have one js file per "node", e.g.:

• supporter.js
• encoder.js
• seeder.js
• and one server called server.js which would later be - of course - replaced by the substrate node

To give one example for the "brotli encoding scenario" you were already working on,
it might be possible to add the server.js using something like the example code below:

server.js

const http = require('http')

const PORT = 8989
const server = http.createServer(handler)
server.listen(PORT, () => console.log('listening on  http://localhost:' + PORT)

const originalFeed, encodedFeed
function handler (request, response) {
  const [ url, params ] = request.url.split('?')
  // request.url === '/feed1?dat=3j239fj23jtw093jtw90tjjtw9'
  if (url === '/feed1') {
    originalFeed = params.split('=')[1]
    response.end()
  } else if (url === '/feed2') {
    if (!originalFeed) throw new Error('did not receive feed address from SUPPORTER yet')
    encodedFeed = params.split('=')[1]
    response.end(originalFeed)
  } else {
    if (!encodedFeed) throw new Error('did not receive feed address from ENCODER yet')
    response.end(encodedFeed)    
  }
}

seeder/encoder/supporter.js

// use the code you already have for each of those "nodes", but add colors
// each different node should use a different color
// so that when they run in the same terminal and log their output
// a developer can differentiate by the color which node logged the message
var colors = require('colors/safe');

// to have each of the three processes log in different color, maybe prefix with
console.log(__dirname.split('/').pop())

console.log(colors.green('hello')); // outputs green text
console.log(colors.red.underline('i like cake and pies')) // outputs red underlined text
console.log(colors.inverse('inverse the color')); // inverses the color
console.log(colors.rainbow('OMG Rainbows!')); // rainbow
console.log(colors.trap('Run the trap')); // Drops the bass

package.json for npm start

{
  "scripts": {
    "start": "node server.js & npm run feed1 & npm run feed2 & npm run feed3",
    "feed1": "node ./supporter.js",
    "feed2": "node ./encoder.js",
    "feed3": "node ./seeder.js"
  }
}

...i hope that makes the idea of all of this more clear :-)

@todo

• implement proof of performance

• https://hackmd.io/J8zIszi-Q-qdxD_VKQ-ykA?both
• https://www.loom.com/share/6926762f2b12457a943c1d922fe197aa

@todo

• make cheatsheet for how to run and use substrate with polkadot.js
• datdot-service#10 Update mauve-sim to account for final workflow
• bring JS up to date with current version of substrate #4
• datdot-service#4 use isInBlock instead of isFinalized
• datdot-service#5 merge datdot-service fork from joshua
• datdot-service#6 implement attestation flow and add a test scenario
• datdot-service#8 implement encoding-hosting-challenge flow and add a test scenario
• datdot-service#7 standard for discovering related hypercores
• datdot-service#13 proof-of-performance

later

• datdot-service#2 make lab environment to reliably test hyper-* scenarios
• implement and test encoding module using brotli
• datdot-service#3 measure the health of a dat address

@todo

• make a command line tool to receive a "dat address" as an argument and return "the health" either as a one time snapshot or allow to subscribe to continuously receive updates.
  • "dat address" (can be for hypercores, hyperdrives, hypertries, etc...)
  • the meaning of "the health" depends on what you think we can get:
    • e.g. how many hypercores are involved (e.g. hyperdrives have 2)?
    • e.g. how many peers participate in the swarm?
    • e.g. what is the throughput (download) possible?
    • e.g. how often is each chunk duplicated?
    • e.g. what is the fluctuation of peers?
    • e.g. what is the latency for particular chunks?
    • e.g. etc...
• additionally it would be cool to support also kappa-* stuff if that's possible :-)
  • if i'm not mistaken kappa still uses hypercores underneath it all

Of course we wan't it modular, so we can re-use individually the logic to figure out from a "dat address" whether we are dealing with a hypercore, a hyperdrive, a hypertrie or something else.
We want to be able to listen to the module to keep on receiving data about the health and we might want to store the history, so when we ask the command line tool for the same address, it already has some historical data.

Any feedback or suggestion for how to approach this or do it maybe even differently are welcome :-)

@todo

• implement attestation
• add scenario in the lab/ folder

explanation
So the chain will announce to some random datdot nodes, that they should join specific hyperswarms to download/request chunks from specifically the peer which is supposed to host to see if they provide them and then report back to the chain. we want to use that opportunity to also measure throughput/latency/etc... of hosters

what was previously named "isFinalized" is now named "isInBlock" - (isFinalized now actually triggers on finality, not block inclusion), we probably want to respond to events when they are isInBlock, not isFinalized.

super-long-term, a more robust mechanism would wait for finality and retry actions if finality is not achieved before dropping the extrinsic callback.

• current service logic https://github.com/ninabreznik/service.js
• Account module (creating custom keypair for each - hoster and encoder which fits with the Noise protocol) https://github.com/playproject-io/datdot-service/blob/mauve-mvp/src/account/index.js
• usage of the Account module https://github.com/playproject-io/datdot-service/blob/mauve-mvp/lab/scenarios/service-mvp.js

@todo

• merge code from joshuas fork back into datdot-service
  • https://github.com/joshua-mir/datdot-service

• Change p2plex to enable listening on channel and filtering incoming by key
• Change hosterAddFeed to download instead of request encoding
• Add encoder public key to hosterAddFeed, use to generated shared channel
• Add event to hoster for notifying chain when it's got an encoded chunk
• Have encoder generate shared channel to send data through
• Update mauve-mvp with new flow, have chain choose encoder and hoster and notify them

@todo

• figure out which teams are building data structures based on hypercores
  • => community/ecosystem
• reach out to community/ecosystem about a standard for discovering related hypercores
  • datdot#36 2020.02.27-17:00pm UTC-0 | dat comm comm call
• write proposal for standard, implement a demo
  • DDEP-0001:hypercore-header-for-manifestfeed
• get feedback from community/ecosystem
  • ask for feedback (individually)
  • ask for feedback on next comm comm call
  • datdot#37 2020.04.29-2020.07.30 | dat consortium call
• sum up all the feedback and update proposal of the standard
  • DDEP-0001:hypercore-header-for-manifestfeed
• notify all teams/people in the community/ecosystem about standard update
  • bounce back & fourth until consensus is reached
  • DDEP-0001:hypercore-header-for-manifestfeed
• support the standard in datdot

CURRENT

• merkle verify data in storage proof (see feed._storage.getNode)
• encode [data, unique_encoder_pos, sign encoded and serve const [data, position] = encoded

BUGS

• gracefully closing hyperbeams (if needed)
• noise handshake error (error only if we try to destroy streams)
• hoster can not be assigned to host same index for same feed

LATER

• feed metadata (sig) should be published to chain with Hosting setup report
• nodes (rootHashes) should be fetched by hoster (not by encoder?)
• replace level db with hyperbee or hypertree?
• proof of performance - new logic
• import account

DONE

• db.feed = { feedkey, swarmkey }
• getProviders
• refactoring Encoder, Hoster, Attestor accounts (splitting up vault and service logic)
• key can not be found in the DB
• extension message with signed event ID
• verify encoder signature
• sign event ID
• account creation - refactoring old Mauve's logic

@todo

• Figure out how to get proof from hypercore
• Add new field to encoder result message
• Have hoster verify against merkle proof
• Have hoster insert merkle proof into hypercore without needing to download the block

@todo

• improve hosting setup for encoder
• improve hosting setup for hoster
• improve hosting setup for attestor
• improve hosting setup for chain
  • add accounting logic to hosting setup
• improve proof of storage for hoster
• improve proof of storage for attester
• improve proof of storage for chain
  • add payment logic to proof of storage

DAY 1

• unregistering
  • removing plans from hosting
  • removing roles
  • removing users (makes no sense)
  • removing feeds from hosting (not avail, then cancel plan and make new one)
  • removing also all open schedulers for this role/plan
• end plan triggers => time, budget, traffic, price => how do we end this (in JS or chain?)
• regions in the form: geohashes

DAY 2

• duplicate plans (what to do if same sponsor published exactly same plan?)

DAY 3

• Scheduler: instead of one time challenge, chain should trigger regular challenges

DAY 4

• remove feed and plan from unhosted

DAY 5

• if CONTRACT doesn't run through (replace additional contract with...)

  • triggered in hostingStarts fn => check if 3 hosters
  • notify failed hoster that they're out (to avoid zombie hosters), repeat the contract with same hosters that are successfully hosting + additional new one + select 1 encoder per 1 additional hoster (role of other encoders can be fulfilled by activeHosters, which send encoded to the attestor to compare with new encoded data for new hoster)

• if provider unregisters, we don't assign them new jobs & we make challenges figure out they dropped for their active jobs

• needed STATUSES

  • hoster => which active contracts they have
  • certain contract => how many hosters active

DAY NEXT

• tryContract only when plan.from (not immediately)
  • when newPlan is published schedule it in x blocks (based on plan.from)
  • don't immediately add plan to DB.unhostedPlans => add it only once it's ready to be hosted
• challenges should happen only when data is supposed to be hosted (Plan start and Plan until time)

DAY NEXT

• add hosterID to performanceChallenge
• rewrite perfomance challenge logic to fetch directly from the hoster

@todo

• datdot#24 encoding compatible hypercores
• check current encoding-hosting scenario code in datdot-service repo
• integrating existing new code into it
• refactor code and modules to use dat-SDK
• adding missing parts to it (what are they?)
• add manual handling of different data structures

later

• datdot-service#7 standard for discovering related hypercores