tigerbeetle / tigerbeetle-node Goto Github PK

TigerBeetle Node.js client

License: Apache License 2.0

TypeScript 39.36% Zig 54.20% Shell 6.44%

tigerbeetle-node's Introduction

tigerbeetle

TigerBeetle is a financial transactions database designed for mission critical safety and performance to power the next 30 years of OLTP.

Quickstart

First, download a prebuilt copy of TigerBeetle.

# macOS
curl -Lo tigerbeetle.zip https://mac.tigerbeetle.com && unzip tigerbeetle.zip && ./tigerbeetle version

# Linux
curl -Lo tigerbeetle.zip https://linux.tigerbeetle.com && unzip tigerbeetle.zip && ./tigerbeetle version

# Windows
powershell -command "curl.exe -Lo tigerbeetle.zip https://windows.tigerbeetle.com; Expand-Archive tigerbeetle.zip .; .\tigerbeetle version"

Want to build from source locally?

git clone https://github.com/tigerbeetle/tigerbeetle && cd tigerbeetle
./scripts/install_zig.sh # or .bat if you're on Windows.
zig/zig build
./tigerbeetle version

Running TigerBeetle

Then create the TigerBeetle data file.

./tigerbeetle format --cluster=0 --replica=0 --replica-count=1 0_0.tigerbeetle

info(io): creating "0_0.tigerbeetle"...
info(io): allocating 660.140625MiB...

And start the replica.

./tigerbeetle start --addresses=3000 0_0.tigerbeetle

info(io): opening "0_0.tigerbeetle"...
info(main): 0: cluster=0: listening on 127.0.0.1:3000

Using the CLI Client

Now that you've got a cluster running, let's connect to it and do some accounting!

First let's create two accounts. (Don't worry about the details, you can read about them later.)

./tigerbeetle repl --cluster=0 --addresses=3000

TigerBeetle Client
  Hit enter after a semicolon to run a command.

Examples:
  create_accounts id=1 code=10 ledger=700 flags=linked|history,
                  id=2 code=10 ledger=700;
  create_transfers id=1 debit_account_id=1 credit_account_id=2 amount=10 ledger=700 code=10;
  lookup_accounts id=1;
  lookup_accounts id=1, id=2;
  get_account_transfers account_id=1 flags=debits|credits;
  get_account_balances account_id=1 flags=debits|credits;

create_accounts id=1 code=10 ledger=700,
                id=2 code=10 ledger=700;

Now create a transfer of 10 (of some amount/currency) between the two accounts.

create_transfers id=1 debit_account_id=1 credit_account_id=2 amount=10 ledger=700 code=10;

Now, the amount of 10 has been credited to account 2 and debited from account 1. Let's query TigerBeetle for these two accounts to verify!

lookup_accounts id=1, id=2;

{
  "id": "1",
  "user_data": "0",
  "ledger": "700",
  "code": "10",
  "flags": "",
  "debits_pending": "0",
  "debits_posted": "10",
  "credits_pending": "0",
  "credits_posted": "0"
}
{
  "id": "2",
  "user_data": "0",
  "ledger": "700",
  "code": "10",
  "flags": "",
  "debits_pending": "0",
  "debits_posted": "0",
  "credits_pending": "0",
  "credits_posted": "10"
}

And indeed you can see that account 1 has debits_posted as 10 and account 2 has credits_posted as 10. The 10 amount is fully accounted for!

For further reading:

Next Steps

Watch an introduction to TigerBeetle on The Primeagen for our design decisions regarding performance, safety, and financial accounting debit/credit primitives:

Read more about the history of TigerBeetle, the problem of balance tracking at scale, and the solution of a purpose-built financial transactions database.

Check out our DESIGN doc to see an overview of TigerBeetle's data structures, take a look at our roadmap, and join one of our communities to stay in the loop about fixes and features!

Documentation

Check out docs.tigerbeetle.com.

Here are a few key pages you might be interested in:

Deployment
- Hardware
Usage
- Integration
Reference

Clients

Community

Contributing

Read docs/HACKING.md.

Roadmap

See #259.

License

Licensed under the Apache License, Version 2.0 (the "License"); you may not use these files except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

tigerbeetle-node's People

Contributors

Stargazers

Watchers

Forkers

wilsonianb jkulubya hoangpq sublimator utanapishtim lewisdaly bermi abbauer

tigerbeetle-node's Issues

Understanding `Transfers.pending_id`

In the readme, the "Creating a transfer" has the following example:

const transfer = {
    id: 1n, // u128
    // Double-entry accounting:
    debit_account_id: 1n,  // u128
    credit_account_id: 2n, // u128
    // Opaque third-party identifier to link this transfer to an external entity:
    user_data: 0n, // u128  
    reserved: 0n, // u128
    // Timeout applicable for a pending/2-phase transfer:
    timeout: 0n, // u64, in nano-seconds.
    // Collection of accounts usually grouped by the currency: 
    // You can't transfer money between accounts with different ledgers:
    ledger: 720,  // u32, ledger for transfer (e.g. currency).
    // Chart of accounts code describing the reason for the transfer:
    code: 1,  // u16, (e.g. deposit, settlement)
    flags: 0, // u16
    amount: 10n, // u64
    timestamp: 0n, //u64, Reserved: This will be set by the server.
}
const errors = await client.createTransfers([transfer])

This fails typescript validation, since the Transfer type definition in in index.ts has:
https://github.com/coilhq/tigerbeetle-node/blob/00296ed3b032d69aba0a1c1ff9b783e2814f5b86/src/index.ts#L85

And when I try ignoring the type errors I get the following error from the client:

Fatal Error: Error: pending_id must be a BigInt
    at /home/lew/developer/buoy/buoy-money/projects/demo-tb/node_modules/tigerbeetle-node/src/index.ts:299:17
    at new Promise (<anonymous>)
    at Object.createTransfers (/home/lew/developer/buoy/buoy-money/projects/demo-tb/node_modules/tigerbeetle-node/src/index.ts:289:12)
    at async main (/home/lew/developer/buoy/buoy-money/projects/demo-tb/scripts/_003_create_transfers.ts:40:18)

So I'm guessing that pending_id is required in client.createTransfers() -- so the types are correct, but the readme is wrong.

postinstall failing

Both running yarn in the repo and installing 0.3.0 as a dependency fails with yarn 2:

➤ YN0007: │ tigerbeetle-node@workspace:. must be built because it never did before or the last one failed
➤ YN0009: │ tigerbeetle-node@workspace:. couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-6dcc4af8/build.log)
➤ YN0009: │ tigerbeetle-node@workspace:. couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-6dcc4af8/build.log)

./src/tigerbeetle/src/state_machine.zig:269:34: error: no member named 'entry' in struct 'std.hash_map.GetOrPutResult'
            const exists = insert.entry.value;
                                 ^
./src/tigerbeetle/src/state_machine.zig:129:45: note: while checking if 'tigerbeetle.src.state_machine.StateMachine.commit' is async
            .create_accounts => self.execute(.create_accounts, input, output),
                                            ^
./src/tigerbeetle/src/vr/replica.zig:1474:72: note: while checking if 'tigerbeetle.src.vr.replica.Replica.commit_op' is async
        const reply_body_size = @intCast(u32, self.state_machine.commit(
                                                                       ^
./src/tigerbeetle/src/vr/replica.zig:1446:31: note: while checking if 'tigerbeetle.src.vr.replica.Replica.commit_ops_through' is async
                self.commit_op(prepare);
                              ^
./src/tigerbeetle/src/vr/replica.zig:1984:78: note: while checking if 'tigerbeetle.src.vr.replica.Replica.repair' is async
        if (self.commit_min < self.commit_max) return self.commit_ops_through(self.commit_max);
                                                                             ^
./src/tigerbeetle/src/vr/replica.zig:82:5: note: while checking this field
    appending_frame: @Frame(write_to_journal) = undefined,
    ^
./src/tigerbeetle/src/state_machine.zig:326:34: error: no member named 'entry' in struct 'std.hash_map.GetOrPutResult'
            const exists = insert.entry.value;
                                 ^
./src/tigerbeetle/src/state_machine.zig:414:19: error: no member named 'entry' in struct 'std.hash_map.GetOrPutResult'
            insert.entry.value = c;
                  ^
./src/tigerbeetle/src/state_machine.zig:447:26: error: no member named 'value' in struct 'std.hash_map.Entry'
            return &entry.value;
                         ^
./src/tigerbeetle/src/state_machine.zig:288:22: error: no member named 'removeAssertDiscard' in struct 'std.hash_map.HashMap(u128,tigerbeetle.src.tigerbeetle.Account,std.hash_map.AutoContext(u128),80)'
        self.accounts.removeAssertDiscard(a.id);
                     ^
./src/tigerbeetle/src/message_bus.zig:1080:60: error: no member named 'removeAssertDiscard' in struct 'std.hash_map.HashMapUnmanaged(u128,*tigerbeetle.src.message_bus.Connection,std.hash_map.AutoContext(u128),80)'
                            .replica => bus.process.clients.removeAssertDiscard(self.peer.client),
                                                           ^
./src/tigerbeetle/src/state_machine.zig:367:23: error: no member named 'removeAssertDiscard' in struct 'std.hash_map.HashMap(u128,tigerbeetle.src.tigerbeetle.Transfer,std.hash_map.AutoContext(u128),80)'
        self.transfers.removeAssertDiscard(t.id);
                      ^
./src/tigerbeetle/src/state_machine.zig:456:26: error: no member named 'value' in struct 'std.hash_map.Entry'
            return &entry.value;
                         ^
./src/tigerbeetle/src/state_machine.zig:465:26: error: no member named 'value' in struct 'std.hash_map.Entry'
            return &entry.value;
                         ^
./src/tigerbeetle/src/state_machine.zig:436:21: error: no member named 'removeAssertDiscard' in struct 'std.hash_map.HashMap(u128,tigerbeetle.src.tigerbeetle.Commit,std.hash_map.AutoContext(u128),80)'
        self.commits.removeAssertDiscard(c.id);
                    ^

Unable to install with yarn2

$ yarn add tigerbeetle-node
➤ YN0000: ┌ Resolution step
➤ YN0000: └ Completed
➤ YN0000: ┌ Fetch step
➤ YN0013: │ tigerbeetle-node@npm:0.2.4 can't be found in the cache and will be fetched from the remote registry
➤ YN0000: └ Completed
➤ YN0000: ┌ Link step
➤ YN0007: │ tigerbeetle-node@npm:0.2.4 must be built because it never did before or the last one failed
➤ YN0009: │ tigerbeetle-node@npm:0.2.4 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-ebfe6f00/build.log)
➤ YN0009: │ tigerbeetle-node@npm:0.2.4 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-ebfe6f00/build.log)
➤ YN0009: │ tigerbeetle-node@npm:0.2.4 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-baa20dcb/build.log)
➤ YN0009: │ tigerbeetle-node@npm:0.2.4 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-baa20dcb/build.log)
➤ YN0000: └ Completed in 0s 644ms
➤ YN0000: Failed with errors in 0s 700ms

cat /tmp/xfs-ebfe6f00/build.log
# This file contains the result of Yarn building a package (tigerbeetle-node@npm:0.2.4)
# Script name: preinstall

Internal Error: tigerbeetle-node@workspace:.: This package doesn't seem to be present in your lockfile; try to make an install to update your resolutions
    at J.getCandidates (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:348376)
    at n.getCandidates (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:337366)
    at /home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:359022
    at Module.w (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:418493)
    at C (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:358985)

cat /tmp/xfs-baa20dcb/build.log
# This file contains the result of Yarn building a package (tigerbeetle-node@npm:0.2.4)
# Script name: postinstall

Internal Error: tigerbeetle-node@workspace:.: This package doesn't seem to be present in your lockfile; try to make an install to update your resolutions
    at J.getCandidates (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:348376)
    at n.getCandidates (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:337366)
    at /home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:359022
    at Module.w (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:418493)
    at C (/home/brandon/test/test-yarn/.yarn/releases/yarn-berry.cjs:2:358985)

It sounds like yarn hides output from at least postinstall, so I think the log files are missing details.
https://blog.typicode.com/husky-git-hooks-autoinstall/
yarnpkg/yarn#5476

I think the current postinstall script could be removed since yarn build is run in prepack and the README says to run yarn and yarn build for local development.

I'm not sure what if anything should change with preinstall.

invalid opcode

I'm encountering an occasional crash with an application using tigerbeetle-node.

syslog shows

kernel: [  110.144682] traps: node[1827] trap invalid opcode ip:7f5448bc12fd sp:7ffd26aafb60 error:0 in client.node[7f5448bc0000+98000]

I'm guessing the client.node is tigerbeetle-node?

`npm install tigerbeetle-node` fails on Alpine Linux Docker Container

I noticed that installing this library fails out of the box on Alpine linux.

Steps to Reproduce:

$ docker run -it  node:16.14-alpine  sh
# then, from inside the container
$ mkdir -p /home/tb-node && cd /home/tb-node
$ npm install tigerbeetle-node

npm notice
npm notice New minor version of npm available! 8.5.0 -> 8.19.2
npm notice Changelog: https://github.com/npm/cli/releases/tag/v8.19.2
npm notice Run npm install -g [email protected] to update!
npm notice
npm ERR! code 127
npm ERR! path /home/tb-node/node_modules/tigerbeetle-node
npm ERR! command failed
npm ERR! command sh -c npm run install_zig && npm run download_node_headers && npm run build_lib
npm ERR! > [email protected] install_zig
npm ERR! > ./src/tigerbeetle/scripts/install_zig.sh
npm ERR! sh: ./src/tigerbeetle/scripts/install_zig.sh: not found

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-09-30T01_19_33_482Z-debug-0.log

Workaround

We can fix this simply by making sure bash, wget, tar and xz are installed on the image before running npm install

$ docker run -it node:16.14-alpine sh
# then, from inside the container
$ mkdir -p /home/tb-node && cd /home/tb-node
# alpine has wget and tar out of the box, but it seems the wrong ones!
$ apk add bash wget tar xz
$ npm install tigerbeetle-node

Tests call `await client.destroy()`, but `destroy()` doesn't return a promise

This is a minor nitpick, but I noticed it and was confused, so I figured I'd open an issue for it.

According to the Client interface, destroy returns void:
https://github.com/tigerbeetledb/tigerbeetle-node/blob/de2272194719a33a013ab5f0a09ce4bead91283c/src/index.ts#L200

But in the tests we have: https://github.com/tigerbeetledb/tigerbeetle-node/blob/de2272194719a33a013ab5f0a09ce4bead91283c/src/test.ts#L343

I'm guessing that the line in the tests are a simple typo.

install_zig.sh missing from package

Installing the package fails in preinstall because scripts/install_zig.sh isn't included in the files list in package.json

build error with latest zig

$ yarn install:zig
Installing Zig latest build...
Downloading https://ziglang.org/builds/zig-linux-x86_64-0.9.0-dev.256+0134cb021.tar.xz...
zig-linux-x86_64-0.9.0-dev.256+0134cb021.tar.xz                      100%[====================================================================================================================================================================>]  39.77M  25.8MB/s    in 1.5s    
Extracting zig-linux-x86_64-0.9.0-dev.256+0134cb021.tar.xz...
Installing zig-linux-x86_64-0.9.0-dev.256+0134cb021 to 'zig' in current working directory...
Congratulations, you have successfully installed Zig 0.9.0-dev.256+0134cb021 to /home/brandon/tigerbeetle-node/zig/zig. Enjoy!

$ yarn build
./src/translate.zig:12:78: error: incompatible types: 'c_uint' and '(enum literal)'
    if (c.napi_create_function(env, null, 0, function, null, &napi_function) != .napi_ok) {
                                                                             ^
./src/node.zig:29:32: note: called from here
    translate.register_function(env, exports, "init", init) catch return null;
                               ^
./src/node.zig:28:88: note: called from here
export fn napi_register_module_v1(env: c.napi_env, exports: c.napi_value) c.napi_value {
                                                                                       ^
./src/translate.zig:12:31: note: type 'c_uint' here
    if (c.napi_create_function(env, null, 0, function, null, &napi_function) != .napi_ok) {
                              ^
./src/translate.zig:12:81: note: type '(enum literal)' here
    if (c.napi_create_function(env, null, 0, function, null, &napi_function) != .napi_ok) {
                                                                                ^
./src/node.zig:29:61: note: referenced here
    translate.register_function(env, exports, "init", init) catch return null;
                                                            ^
./src/node.zig:295:65: error: incompatible types: 'c_uint' and '(enum literal)'
    if (c.napi_get_cb_info(env, info, &argc, &argv, null, null) != .napi_ok) {
                                                                ^
./src/node.zig:295:27: note: type 'c_uint' here
    if (c.napi_get_cb_info(env, info, &argc, &argv, null, null) != .napi_ok) {
                          ^
./src/node.zig:295:68: note: type '(enum literal)' here
    if (c.napi_get_cb_info(env, info, &argc, &argv, null, null) != .napi_ok) {

Client logging is verbose and not configurable

I've noticed that the client's logging is rather verbose. For example this is logged in the space of under a second:

info(message_bus): connected to replica 0
info(message_bus): peer performed an orderly shutdown: tigerbeetle.src.message_bus.union:375:19{ .replica = 0 }
info(message_bus): connected to replica 0
info(message_bus): peer performed an orderly shutdown: tigerbeetle.src.message_bus.union:375:19{ .replica = 0 }
info(message_bus): connected to replica 0
info(message_bus): peer performed an orderly shutdown: tigerbeetle.src.message_bus.union:375:19{ .replica = 0 }
info(message_bus): connected to replica 0
info(message_bus): connected to replica 0
info(message_bus): peer performed an orderly shutdown: tigerbeetle.src.message_bus.union:375:19{ .replica = 0 }
info(message_bus): connected to replica 0
info(message_bus): peer performed an orderly shutdown: tigerbeetle.src.message_bus.union:375:19{ .replica = 0 }
info(message_bus): connected to replica 0

Looking up these log messages, they appear to come from src/message_bus.zig in Tigerbeetle itself:
https://github.com/tigerbeetledb/tigerbeetle/blob/130692a16fc39bfeb5fd23739eeb449a2fb541c0/src/message_bus.zig

Is there any way to control the log verbosity from the node client? Or if not, any pointers on how I might go about implementing such a feature myself?

postinstall failing

https://github.com/interledger/rafiki/runs/4168593074?check_suite_focus=true

tigerbeetle-node@npm:0.4.1 STDOUT Installing Zig 0.8.0 release build...
tigerbeetle-node@npm:0.4.1 STDOUT Downloading https://ziglang.org/builds/zig-linux-x86_64-0.9.0-dev.1599+008b0ec5e.tar.xz
tigerbeetle-node@npm:0.4.1 STDOUT https://ziglang.org/download/0.8.0/zig-linux-x86_64-0.8.0.tar.xz...
tigerbeetle-node@npm:0.4.1 STDOUT Extracting zig-linux-x86_64-0.8.0.tar.xz...
tigerbeetle-node@npm:0.4.1 STDOUT Installing zig-linux-x86_64-0.8.0 to 'zig' in current working directory...
tigerbeetle-node@npm:0.4.1 STDERR  40200K .......... .......... .......... .......... .....     100%  136M=0.4smv: cannot stat 'zig-linux-x86_64-0.8.0': No such file or directory
tigerbeetle-node@npm:0.4.1 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-92ba0a68/build.log)

It looks like it's trying to get 0.8.0 from a 0.9.0 url?
Not sure if updating the tigerbeetle submodule to include this will solve it
tigerbeetle/tigerbeetle@3f572cb

Tigerbeetle probably needs to run latest version of Zig (0.10.0)

Hello, I am using MacOS Ventura 13 and Zig version lower than 0.10.0 'panics' as discussed in thread ziglang/zig#10478.

Looks like tigerbeetle-node should run this newer version of Zig in order to have working packages on Ventura 13.

The issue is happening when I try to run pnpm i.
The stack trace:
Scope: all 7 workspace projects
Lockfile is up to date, resolution step is skipped
Packages: +1646
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Progress: resolved 1646, reused 1630, downloaded 0, added 0, done
node_modules/.pnpm/[email protected]/node_modules/tigerbeetle-node: Running postinstall script, failed in 9s
.../node_modules/tigerbeetle-node postinstall$ npm run install_zig && npm run download_node_headers && npm run build_lib
│ > [email protected] install_zig
│ > ./src/tigerbeetle/scripts/install_zig.sh
│ Installing Zig 0.9.1 release build...
│ Downloading https://ziglang.org/download/0.9.1/zig-macos-aarch64-0.9.1.tar.xz...
│ Extracting zig-macos-aarch64-0.9.1.tar.xz...
│ Installing zig-macos-aarch64-0.9.1 to 'zig' in current working directory...
│ Congratulations, you have successfully installed Zig 0.9.1 to /Users/timeanagy/Documents/rafiki/node_modules/.pnpm/[email protected]/node_modules/tigerbeetl
│ > [email protected] download_node_headers
│ > ./scripts/download_node_headers.sh
│ Downloading https://nodejs.org/download/release/v16.18.0/node-v16.18.0-headers.tar.gz...
│ Extracting ./build/node-v16.18.0-headers.tar.gz...
│ > [email protected] build_lib
│ > mkdir -p dist && zig/zig build-lib -mcpu=baseline -OReleaseSafe -dynamic -lc -isystem build/node-$(node --version)/include/node src/node.zig -fallow-shlib-undef
│ thread 403088 panic: Darwin is handled separately via std.zig.system.darwin module
│ Unable to dump stack trace: debug info stripped
│ sh: line 1: 55468 Abort trap: 6 zig/zig build-lib -mcpu=baseline -OReleaseSafe -dynamic -lc -isystem build/node-$(node --version)/include/node src/node.
└─ Failed in 9s

panic from lookup of non-existent account(s)

When I call lookupAccounts and none of my specified ids exist, I get the following:

debug: 746649394563965214: on_message: Header{ .checksum = 26737541277414919673681114361701047635, .checksum_body = 98081761841810822994326224503485043631, .nonce = 65988244422842662187530359719024616725, .clien = 90, .commit = 90, .offset = 0, .size = 128, .epoch = 0, .request = 1, .replica = 0, .command = Command.reply, .operation = Operation.lookup_accounts }
thread 379696 panic: index out of bounds
/opt/tigerbeetle-node/src/tigerbeetle/src/client.zig:419:98: 0x7f0a9456a9e2 in tigerbeetle.src.client.Batch(tigerbeetle.src.state_machine.Operation.lookup_accounts,1000).deliver (node)
/opt/tigerbeetle-node/src/tigerbeetle/src/client.zig:292:30: 0x7f0a94567c2c in tigerbeetle.src.client.BatchManager.deliver (node)
/opt/tigerbeetle-node/src/tigerbeetle/src/client.zig:102:35: 0x7f0a9456654f in tigerbeetle.src.client.Client.on_message (node)
...

Relatedly, when I call lookupAccounts and some of the specified ids exist, the returned AccountLookupResult[] only includes the existing Accounts. Is AccountLookupError only expected to be returned if none of the specified accounts exist?