Git Product home page Git Product logo

hubingliang / llama-node Goto Github PK

View Code? Open in Web Editor NEW

This project forked from atome-fe/llama-node

0.0 0.0 0.0 30.53 MB

Believe in AI democratization. llama for nodejs backed by llama-rs and llama.cpp, work locally on your laptop CPU. support llama/alpaca/gpt4all/vicuna model.

Home Page: https://llama-node.vercel.app/

License: MIT License

JavaScript 21.85% Python 1.69% C 0.02% Rust 54.62% TypeScript 21.11% CSS 0.72%

llama-node's Introduction

llama-node

Large Language Model on Node.js

GitHub Workflow Status NPM npm npm type definitions twitter

Node.js Library for LLaMA/RWKV

LLaMA generated by Stable diffusion

Picture generated by stable diffusion.

Introduction

This is a nodejs client library for llama (or llama based) LLM built on top of llama-rs and llama.cpp. It uses napi-rs for channel messages between node.js and llama thread.

This project is in an early stage, the API for nodejs may change in the future, use it with caution.

From v0.0.21, both llama-rs and llama.cpp backends are supported!

Currently supported platforms:

  • darwin-x64
  • darwin-arm64
  • linux-x64-gnu (glibc >= 2.31)
  • linux-x64-musl
  • win32-x64-msvc

Node.js version: >= 16


Install

  • Install main package
npm install llama-node
  • Install llama-rs backend
npm install @llama-node/core
  • Install llama.cpp backend
npm install @llama-node/llama-cpp

Performance related

We provide prebuild binaries for linux-x64, win32-x64, apple-x64, apple-silicon. For other platforms, before you install the npm package, please install rust environment for self built.

Due to complexity of cross compilation, it is hard for pre-building a binary that fits all platform needs with best performance.

If you face low performance issue, I would strongly suggest you do a manual compilation. Otherwise you have to wait for a better pre-compiled native binding. I am trying to investigate the way to produce a matrix of multi-platform supports.

Manual compilation

WIP


llama-node's People

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.