As a user I would like to specify the output folder for transactions in some kind of configuration or cmd line flag.

the following transactions

      {
        txId: 'e00118d7a414034c6e74a37d79621c6c3df62f70efa5164ef6bbdb44b2ee6f55',
        height: 683929
      },

      {
        txId: '7db9642fe57bf735ac8a93236b90b538fb81ae0f58fc4920a8a075452a8824d9',
        height: 685004
      },
      {
        txId: 'b6f60f9258a942c72b2ec6f3e3177e24830cbe92fc9dda2d0d3eb406d1582d13',
        height: 685004
      },

      {
        txId: '90cd5e990a8d4d13481a67470e416d9daecd688729ea5fde7f78d151af827bc2',
        height: 685007
      },

 

Query Code

    const addressList=['1Pwmd4RCoTbYP6tLWVoDcys1GW5chsve8C']
    
    const queryBitbus: any = {
      q: {
        find: {
          $or: [{ 'out.e.a': { $in: addressList } }, { 'in.e.a': { $in: addressList } }],
          'blk.i': { $gt: 683000 },
        },
        sort: { 'blk.i': 1 },
        project: { tx: 1, blk: 1, i: 1 },
      },
    };

the results of query don't include lost txs.

test timestamp: 2021/05/17

A lot of .env related errors are not handled precisely and the app ends up stopping with errors such as:

BITBUS all finished at block undefined
/usr/local/lib/node_modules/bitbus/block.js:46
      fileStream.write("]")
                 ^

TypeError: Cannot read property 'write' of undefined
    at Stream.<anonymous> (/usr/local/lib/node_modules/bitbus/block.js:46:18)
    at Stream.emit (events.js:203:15)
    at Stream.stream.destroy (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:84:12)
    at _end (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:67:14)
    at Stream.stream.end (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:74:5)
    at Stream.onend (internal/streams/legacy.js:42:10)
    at Stream.emit (events.js:203:15)
    at drain (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:34:23)
    at Stream.stream.queue.stream.push (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:45:5)
    at Stream.end (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:15:35)

The error report needs to be much better so the developer knows what exactly is going wrong.

Feature Request: Possible way to specify path on host file system for Bitbus to save and write bus directory.

Possible use case: Running shared volume mount with separate services, (such as on kubernetes).

Current Behavior: bitbus start command looks to find .json file in path with valid "bitbus" key. It creates bus folder in current file path from which command was ran. In bitbusbox impl, this is on the /mnt path from which we also copy the run.sh file. Problem here is that if we are wanting to write the bus to a shared volume, we need to mount it here, overwriting our run and json file.

Tmp Workaround: To workaround this issue, without the requested feature, it is also possible to change the entry point to a different, clean fs location on which to mount, and instead simply export the locations on the run.sh and <your.json> to your PATH, so that they are found on start. This way should work but is a little hacked.

Possible Implementations: could be 1) using command line flags; --buspath <yourpath> or 2) env variable ; export BUSPATH=<yourpath>

I can't seen to find enough documentation about writing filters.

I want to run this filter:

{
"bitbus": 1,
"name": "md-feed-1",
"from": 590470,
"q": {
"find": { "out.h1": "6d02" }
}
}

But when I run bitbus start (after running bitbus new) I get this output:

bitbus start
BITBUS all finished at block undefined
/usr/local/lib/node_modules/bitbus/block.js:46
fileStream.write("]")
^

TypeError: Cannot read property 'write' of undefined
at Stream. (/usr/local/lib/node_modules/bitbus/block.js:46:18)
at Stream.emit (events.js:205:15)
at Stream.stream.destroy (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:84:12)
at _end (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:67:14)
at Stream.stream.end (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:74:5)
at Stream.onend (internal/streams/legacy.js:44:10)
at Stream.emit (events.js:205:15)
at drain (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:34:23)
at Stream.stream.queue.stream.push (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:45:5)
at Stream.end (/usr/local/lib/node_modules/bitbus/node_modules/through/index.js:15:35)

Thanks very much

In mongo, project can be used to either include specific fields or alternatively exclude specific fields. From the docs:

If you specify the exclusion of a field or fields, all other fields are returned in the output documents.

Recreate issue

Attempt the following query using bitbus:

{
  "bitbus": 1,
  "name": "files",
  "from": 590000,
  "q": {
    "find": { "out.s1": "19HxigV4QyBv3tHpQVcUEQyq1pzZVdoAut" },
    "project": {
      "out.b2": 0,
      "out.h2": 0,
      "out.s2": 0,
      "out.lb2": 0,
      "out.lh2": 0,
      "out.ls2": 0
    }
  }
}

Expected result

Bitbus should operate normally, outputting json data files with transactions excluding the out.*2 fields.

Actual result

Bitbus creates the output directory but crashes before writing any files. The following error is logged:

/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/block.js:32
    fileStream.write("]")
               ^

TypeError: Cannot read property 'write' of undefined
    at Stream.<anonymous> (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/block.js:32:16)
    at Stream.emit (events.js:203:15)
    at Stream.stream.destroy (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/node_modules/through/index.js:84:12)
    at _end (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/node_modules/through/index.js:67:14)
    at Stream.stream.end (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/node_modules/through/index.js:74:5)
    at Stream.onend (internal/streams/legacy.js:42:10)
    at Stream.emit (events.js:203:15)
    at drain (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/node_modules/through/index.js:34:23)
    at Stream.stream.queue.stream.push (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/node_modules/through/index.js:45:5)
    at Stream.end (/Users/whodis/.nvm/versions/node/v10.16.0/lib/node_modules/bitbus/node_modules/through/index.js:15:35)

Other notes

The above query works fine using a planaria API endpoint.
https://neongenesis.bitdb.network/query/1HcBPzWoKDL2FhCMbocQmLuFTYsiD73u1j/ewogICJ2IjogMywKICAicSI6IHsKICAgICJmaW5kIjogewogICAgICAib3V0LnMxIjogIjE5SHhpZ1Y0UXlCdjN0SHBRVmNVRVF5cTFwelpWZG9BdXQiCiAgICB9LAogICAgInByb2plY3QiOiB7CiAgICAgICJvdXQuYjIiOiAwLAogICAgICAib3V0LmgyIjogMCwKICAgICAgIm91dC5zMiI6IDAsCiAgICAgICJvdXQubGIyIjogMCwKICAgICAgIm91dC5saDIiOiAwLAogICAgICAib3V0LmxzMiI6IDAKICAgIH0sCiAgICAic29ydCI6IHsKICAgICAgImJsay5pIjogLTEsCiAgICAgICJpIjogLTEKICAgIH0sCiAgICAibGltaXQiOiAyMAogIH0KfQ==

When the project config is changed to include only the above fields, Bitbus works normally. The problems seems specifically related to excluding fields using Bitbus.

It's ideal to keep the entire bitbus, but in some cases developers may not want to keep the bus state, or may only want to keep the most recent blocks.

Think about adding a feature that allows:

1. Pruning older blocks as a setting, so it only keeps the N most recent blocks
2. Make N configurable.

Will need to take into account the re-org API which will need to be implemented, because if the bus is completely gone it will be difficult to detect re-orgs locally.

an option to display or hide bitbus logs

The sync requires dealing with bitcoin transactions and keys even though it's not being used yet.

Think about whether this can be just extracted out to simplify the module.