Large Language Model on Node.js
Picture generated by stable diffusion.This is a nodejs client library for llama (or llama based) LLM built on top of llama-rs and llama.cpp. It uses napi-rs for channel messages between node.js and llama thread.
This project is in an early stage, the API for nodejs may change in the future, use it with caution.
From v0.0.21, both llama-rs and llama.cpp backends are supported!
Currently supported platforms:
- darwin-x64
- darwin-arm64
- linux-x64-gnu (glibc >= 2.31)
- linux-x64-musl
- win32-x64-msvc
Node.js version: >= 16
- Install main package
npm install llama-node
- Install llama-rs backend
npm install @llama-node/core
- Install llama.cpp backend
npm install @llama-node/llama-cpp
We provide prebuild binaries for linux-x64, win32-x64, apple-x64, apple-silicon. For other platforms, before you install the npm package, please install rust environment for self built.
Due to complexity of cross compilation, it is hard for pre-building a binary that fits all platform needs with best performance.
If you face low performance issue, I would strongly suggest you do a manual compilation. Otherwise you have to wait for a better pre-compiled native binding. I am trying to investigate the way to produce a matrix of multi-platform supports.
WIP