ironcorelabs / recrypt-node-binding Goto Github PK

View Code? Open in Web Editor NEW

21.0 8.0 6.0 672 KB

Bindings from recrypt-rs to work with NodeJS

Home Page: https://docs.ironcorelabs.com

License: GNU Affero General Public License v3.0

JavaScript 58.76% Rust 40.14% Nix 1.10%

cryptography proxy-re-encryption transform-encryption nodejs

recrypt-node-binding's Introduction

Recrypt Node Binding

Bindings to be able to use Recrypt Rust from NodeJS code. Improves the performance of Recrypt operations by using native code.

This library uses the Neon Bindings toolchain to compile the Recrypt Rust library into a binary NodeJS file that can be used from Node applications. The Neon Bindings provide a way to write a shim which converts data in/out of the Recrypt Rust code. The resulting binary Node file can then be included into a NodeJS module just like any other NPM dependency.

Supported Platforms

	Node 16	Node 18	Node 20	Node 21
Linux x64 - glibc	✓	✓	✓	✓
Linux x64 - musl-libc	✓	✓	✓	✓
Linux arm64 - glibc	✓	✓	✓	✓
Linux arm64 - musl-libc	✓	✓	✓	✓
OSX x64	✓	✓	✓	✓
OSX arm64	✓	✓	✓	✓
Windows x64	✓	✓	✓	✓

Install

npm install @ironcorelabs/recrypt-node-binding

The binary that is generated via the Neon Bindings toolchain is platform specific. We use the node-pre-gyp tool to pre-compile binaries for the most popular platforms. When you npm install this library it will automatically pull down the proper binary for your platform from the binaries uploaded to the releases page.

This means that you'll need to make sure that the machine that runs npm install is the machine where the code will run. This library will not work if you run npm install on an OSX machine and move the node_modules directory over to a Linux machine, for example.

If the machine you run npm install on is not one of the supported architectures you will get an install failure. If there's an architecture that you'd like supported that isn't yet available, open a new issue and we'll look into adding support for it. You can also build the bindings yourself to generate a binary file for whichever architecture you need. Refer to the local development section for details.

Types

This library contains a TypeScript definitions file which shows the available classes and methods.

Examples

The following examples show how to use this library from a NodeJS application

Basic Encrypt/Decrypt Example

const assert = require("assert");
const Recrypt = require("@ironcorelabs/recrypt-node-binding");

//Create a new Recrypt API instance
const Api256 = new Recrypt.Api256();

//Generate both a user key pair and a signing key pair
const keys = Api256.generateKeyPair();
const signingKeys = Api256.generateEd25519KeyPair();

//Generate a plaintext to encrypt
const plaintext = Api256.generatePlaintext();

//Encrypt the data to the public key and then attempt to decrypt with the private key
const encryptedValue = Api256.encrypt(plaintext, keys.publicKey, signingKeys.privateKey);
const decryptedValue = Api256.decrypt(encryptedValue, keys.privateKey);

assert.equal(decryptedValue, plaintext);

Single-hop Transform Encryption Example

const assert = require("assert");
const Recrypt = require("@ironcorelabs/recrypt-node-binding");

//Create a new Recrypt API instance
const Api256 = new Recrypt.Api256();

//Generate both a user key pair and a signing key pair
const userKeys = Api256.generateKeyPair();
const signingKeys = Api256.generateEd25519KeyPair();

//Generate a plaintext to encrypt
const plaintext = Api256.generatePlaintext();

//Encrypt the data to the user public key
const encryptedValue = Api256.encrypt(plaintext, userKeys.publicKey, signingKeys.privateKey);

//Generate a second public/private key pair as the target of the transform. This will allow the encrypted data to be
//transformed to this second key pair and allow it to be decrypted.
const deviceKeys = Api256.generateKeyPair();

//Generate a transform key from the user private key to the device public key
const userToDeviceTransformKey = Api256.generateTransformKey(userKeys.privateKey, deviceKeys.publicKey, signingKeys.privateKey);

//Transform the encrypted data (without decrypting it!) so that it can be decrypted with the second key pair
const transformedEncryptedValue = Api256.transform(encryptedValue, userToDeviceTransformKey, signingKeys.privateKey);

//Decrypt the data using the second private key
const decryptedValue = Api256.decrypt(transformedEncryptedValue, deviceKeys.privateKey);

assert.equal(decryptedValue, plaintext);

Local Development

In order to build the binary Node file for Recrypt, you'll need the dependencies specified on the Neon Bindings site. Follow their getting started directions and install Rust and the Node Build Tools. The Neon CLI is already installed as a dependency of this project so you don't have to install that as a global dependency.

Once all of those dependencies are installed, the following can be run.

npm run compile

yarn compile

This will produce an index.node file within the bin-package directory. This file can be used directly, or via index.js:

const recrypt = require('index.js');

Benchmarks

From this repos root, run npm i or yarn.
Run npm/yarn run compile to compile the Rust source into a bin-package/index.node module.
Run npm/yarn run benchmark.

Unit Tests

From this repos root, run npm i or yarn.
Run npm/yarn run compile to compile the Rust source into a bin-package/index.node module.
Run npm/yarn run test.

recrypt-node-binding's People

Contributors

Stargazers

Watchers

Forkers

arsen3d nodehaven lazyyi ckartik tianyu0901

recrypt-node-binding's Issues

Expose hash_256 method

Expose the new hash_256 method that was introduced here.

Consume API tweaks, Schnorr signing, and expose ed25519 sign/verify

Now that recrypt-rs has Schnorr signing implemented, we need to consume those changes in the Node binding and expose a method to be able to do it. Also consume the security changes PR as well.

Top level build/publish script

As a (Travis) build process
I want to have a script I can execute to build, package, publish binaries, and publish to npm

Probably will need to be worked on in conjunction with #9

Needs further research on how the Travis multi-arch stuff works. Does it kick off a job for each arch? How do we tie them back together? I started thinking about this when trying to write the "top level build/publish script" ticket. We need to publish a binary for each arch/node combo, and then publish to npm.

See https://github.com/IronCoreLabs/recrypt-node-binding/blob/c7f06ba9097bce2ee064409f19ee0bfae131ea8e/publish.js

Import into NextJs

HI, i'm trying to import this node-binding into my NextJs project to run, but it keeps tlling me this error: ./node_modules/@ironcorelabs/recrypt-node-binding/bin-package/index.node
Module parse failed: Unexpected character '�' (1:0)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
(Source code omitted for this binary file)

Please show me the proper way to import into framework like NextJs, thanks

Upload binary package to github releases

As a (Travis) build process
I want a way to upload a binary tar.gz to github releases
so that it can be downloaded at NPM install time.

AC:

the resultant upload can be successfully downloaded by ./node_modules/.bin/node-pre-gyp install

A good starting point might be
node-pre-gyp-github. During inital assessment this package did not upload the full archive and only created an 100 byte (correctly named) file in github.

Also note that file can only be downloaded once the release is not marked DRAFT.

Introduce Typescript build process, generate types

Now that there is a manually written index.js as the entrypoint, we're duplicating a lot of work between index.js and index.d.ts that is error prone. Write an index.ts that replaces both in the repo, and a build process that produces both of the original files for NPM packaging.

As a node-binding consumer I can create distributable libraries that work on various platforms without forcing developers to compile the binding at install time.

Problem

When we compile the Neon node-binding source, the Node addon binary that it generates is not portable between architectures. This prevents us from being able to compile the source and publish it to NPM as we don't know the platform the user is installing from. Having users compile the addon at install time isn't very feasible since the chance that they have the various build dependencies (Rust, Neon, etc) is very minimal. Instead, we need to generate compiled binaries for various platforms and have those be retrieved at install time depending on the environment we're present in.

This is a known issue within Neon, but not something that appears to be actively developed or discussed even.

node-gyp

A rough parallel to what we're doing within Neon is node-gyp which is a tool for building Node addons from C++ code for Node. Where Neon helps us build Rust code into a native Node module, node-gyp does the same thing from C++ to native Node modules. As node-gyp is a more stable build tool, there's a larger ecosystem of tooling around dealing with this issue.

node-pre-gyp

What appears to be the most popular solution for handling the pre-compiled binary problem is the node-pre-gyp toolkit. This project aims to make it easy for distributing Node addons via pre-compiling binaries for various platforms. The quick and dirty workflow is

Configure node-pre-gyp for the location of your binaries as well as where they are hosted
Auto build and upload your binaries to an AWS bucket
Add a postinstall script to your package.json so that when users run NPM install on it, once it's been installed it discovers the machines platform and requests the appropriate binary from the AWS bucket, extracts it and configures all paths.
Optionally falls back to compiling the code via node-gyp if a binary cannot be found for the users machine.

This rough workflow is about the same as we'd ideally do with our node-binding. We likely won't have support for falling back to compiling locally given the lifting required there, but everything else should work roughly the same.

Resources

Other various resources/thoughts I've found so far after looking into this

By default node-pre-gyp defaults to uploading/downloading to AWS buckets. There are tools such as node-pre-gyp-github that upload to GitHub releases. It seems like using GitHub releases for our binaries might be a good idea. Alternatively we would probably use a GCP bucket.
Building the binaries for various platforms should be able to easily be done via Travis and there are a number of examples about how this is done for node-pre-gyp projects. Travis can build for both linux/osx as well as various NodeJS versions. There isn't Windows support or 32-bit support, so we'd have to decide how much we care about those platforms.
This is a very detailed article which goes through the lengthy process that is involved to setup node-pre-gyp.
I'm pretty skeptical that we'd be able to use node-pre-gyp with our stuff since we're in Rust. Maybe there's a way to use it for only the uploading/post-install downloading, but we might end up doing the same amount of work if we just wrote a tool ourselves. More investigation would be needed before we can decide on a route there.
I've proven that the Neon module that is generated for these bindings works on both Node 8 and Node 10. However, there are no guarantees that future versions will work. So compiled versions would need to be build per Node version. This means that the cross-product of binaries would be [version of binding] x [version of node] x [platform (linux,osx,win)] x [supported architecture (x32/x64)]. Yeesh.
A good project to look at for using node-pre-gyp is the Node Sqlite3 project.

Open Questions

Even if we end up getting this all working above, there's still an issue that the whole process only works if consumers of the binding build their code on the same platform where it runs. If we have the above setup correctly but a developer is building on their local Mac before pushing the results to some location where it'll be run in Linux, the whole system breaks down. We might end up needing a way for developers to specify their platform via some environment variable so that I could build on one architecture but have the results be built for a different architecture.
It should also be noted that Node 8/10 support WASM modules and I've proven that they work as expected. The WASM modules are also platform agnostic so they'd work to publish them to NPM. However the performance isn't great compared to these bindings so it depends on how much performance we need to wring out of this library.

Consume new 0.5.x release of recrypt-rs

Contains updates to new rand 0.6 and other dep updates. Consuming this will also fix the next release of the bindings to not be pulled from recrypt-rs master. (https://github.com/IronCoreLabs/recrypt-node-binding/blob/master/native/Cargo.toml#L13)

Travis multi-arch, multi-Node.js config

As a release manager
I want to be able to have Travis build and release recrypt-node-binding for multiple architecture and multiple Node.js versions when a new release is created.

When a release tag is created in git the build/publish process should kick off
Releases should be created for Linux x86_64, MacOSX for Node 8 and Node 10
An NPM package should be published

Consume 1.0.0 release of recrypt and bump version of binding to 1.0.0

Publish version to match recrypt to designate "production ready" usage.

Support Mac M1

https://github.com/IronCoreLabs/recrypt-node-binding/runs/6751924442?check_suite_focus=true seems to indicate that we need to support arm64 on Mac:

dlopen(/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node, 0x0001): tried: '/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/index.node' (no such file), '/usr/lib/index.node' (no such file)

Build a MUSL linked binding

In support of the common alpine-linux deployment environment, build and upload to releases a musl linked version of these bindings.

Creating this should "just work" per neon if the build is run in a musl environment, causing recrypt-rs and its gridiron dependency to be built linked against musl.

Compile for windows support

Add a compile/publish step to build a binary for windows. We might need to move this repo over to GitHub Actions as part of this in order to add windows builds.

Implement better error message text

Similar to what we did for the wasm-binding code, we should implement changes to the node-bindings to map errors to extract out the internal error message from recrypt-rs and expose that to the caller. Currently the caller just gets a fairly unhelpful "neon error" style message.

Migrate CI: Travis -> GitHub

Node 21 support

Hey, I'd love to start using this and the associated wasm libraries but we have recently upgraded to Node 21, which you don't support yet.

Are there any plans to support Node 21? If so, when?

Thanks in advance.

Expose methods for augment_plus and augment_minus from Recrypt to support private key rotation

Support versions 9 and 11 of NodeJS

Apparently some people run non-LTS versions of NodeJS. We should support those versions so we don't get ugly error messages for them and now make us iterate our version support above version 8 in the consuming projects package.json files.

Fix vulnerabilities

GitHub is reporting three vulnerabilities:

lodash critical CVE-2019-10744
mixin-deep high sev CVE-2019-10746
set-value high sev CVE-2019-10747

[SPIKE] Investigate new neon build process

neon is coming up with a new build/distribution process, the experimental example project is here. Check it out and see if it will work to replace node-pre-gyp for us.

Update depdendencies

Known high severity security vulnerability detected in handlebars < 4.0.14 defined in yarn.lock.

yarn.lock update suggested: handlebars ~> 4.0.14.

Known high severity security vulnerability detected in js-yaml < 3.13.1 defined in yarn.lock.

yarn.lock update suggested: js-yaml ~> 3.13.1.

We should also update Recrypt to the latest and consume all breaking changes there as well. Also add Node12 support which will require us to rely on a fork of Neon.

`index-v0.5.1-node-v67-linux-x64.tar.gz` tar ball missing from releases

$ yarn add @ironcorelabs/recrypt-node-binding
yarn add v1.17.3
warning package.json: No license field
warning No license field
[1/4] Resolving packages...
[2/4] Fetching packages...
info [email protected]: The platform "linux" is incompatible with this module.
info "[email protected]" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
error /home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding: Command failed.
Exit code: 1
Command: node-pre-gyp install
Arguments:
Directory: /home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding
Output:
node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | linux | x64
node-pre-gyp WARN Using request for node-pre-gyp https download
node-pre-gyp info check checked for "/home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding/bin-package/index.node" (not found)
node-pre-gyp http GET https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
node-pre-gyp http 404 https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
node-pre-gyp ERR! install error
node-pre-gyp ERR! stack Error: 404 status code downloading tarball https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
node-pre-gyp ERR! stack     at Request.<anonymous> (/home/foobaruser/project/pondjs/node_modules/node-pre-gyp/lib/install.js:149:27)
node-pre-gyp ERR! stack     at Request.emit (events.js:198:15)
node-pre-gyp ERR! stack     at Request.onRequestResponse (/home/foobaruser/project/pondjs/node_modules/request/request.js:1066:10)
node-pre-gyp ERR! stack     at ClientRequest.emit (events.js:193:13)
node-pre-gyp ERR! stack     at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:560:23)
node-pre-gyp ERR! stack     at HTTPParser.parserOnHeadersComplete (_http_common.js:113:17)
node-pre-gyp ERR! stack     at TLSSocket.socketOnData (_http_client.js:447:22)
node-pre-gyp ERR! stack     at TLSSocket.emit (events.js:193:13)
node-pre-gyp ERR! stack     at addChunk (_stream_readable.js:295:12)
node-pre-gyp ERR! stack     at readableAddChunk (_stream_readable.js:276:11)
node-pre-gyp ERR! System Linux 5.0.0-23-generic
node-pre-gyp ERR! command "/usr/bin/node" "/home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding/node_modules/.bin/node-pre-gyp" "install"
node-pre-gyp ERR! cwd /home/foobaruser/project/pondjs/node_modules/@ironcorelabs/recrypt-node-binding
node-pre-gyp ERR! node -v v11.15.0
node-pre-gyp ERR! node-pre-gyp -v v0.13.0
node-pre-gyp ERR! not ok
404 status code downloading tarball https://github.com/IronCoreLabs/recrypt-node-binding/releases/download/0.5.1/index-v0.5.1-node-v67-linux-x64.tar.gz
info Visit https://yarnpkg.com/en/docs/cli/add for documentation about this command.

[SPIKE] Only build for one node version

Apparently neon binaries are supposed to be portable across Node versions. Investigate how that would affect our build and release process, we may be able to cut down on the number of artifacts and builds significantly.

CI test failure

This CI run failed. It only failed for our Mac builds.

dlopen(/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node, 0x0001): tried: '/Users/ironcore/actions-runner/_work/recrypt-node-binding/recrypt-node-binding/bin-package/index.node' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/local/lib/index.node' (no such file), '/usr/lib/index.node' (no such file)

[SPIKE] Investigate possibility of using node-pre-gyp for our use case

Figure out if we can use node-pre-gyp for the postinstall NPM step of pulling down the expected architecture binary from the expected location. We don't want node-pre-gyp to do anything with node-gyp, just the configuration/simplification of determining an architecture and pulling down the proper binary.

Create binary package

As a (Travis) build process
I want a yarn target to call
so that a .tar.gz file is produced for the target architecture.

node-pre-gyp is used to produce this tar.gz so we can rely on it for naming and downloading at install time
This should be a target like yarn run package unless there's a reason for it to be separate.
The resultant tar.gz should only contain the index.node file to minimize size

Can be based off of the current node-pre-gyp