rapid-llama is a super HIGH-performance inference engine for LLMs like LLaMA (3x of llama.cpp) written in C++ which can run a 8-bit quantized LLaMA2-7B model on a cpu with 56 cores in speed of ~30 tokens / s. It outperforms all current open-source inference engines, especially when compared to the renowned llama.cpp, with 2~3 times better inference speed on a CPU.

Advantages

• Fast
  • Extremely fast on CPU. Faster than any other engines on Github including llama.cpp (3 times faster than llama.cpp).
• Simple
  • Totally 6k+ lines of C++ codes with well-orgnized code structures and no dependencies except NUMA (if needed for multi-cpus).

⚠️ Only CPU is supported currently. Support for GPU is coming soon.

Only Linux is supported currently. Support of other platforms including Windows, Mac, GPU is coming soon.

• gcc version 10.x or newer versions
• libnuma-dev

libraries like mpi, openblas, mkl, etc are NOT needed currently.

Method 1. Using the provided build script:

bash ./build.sh

Method 2. Using Make:

make -j 4

To run the inference engine, execute the following command:

Only gguf and llama2.c format of models are currently supported. Independent formort is coming soon.

./main -c ./models/cnllama-7b/ggml-model-f32.gguf -f gguf -j 56 -q int8 -n 200 -i 'That was a long long story happened in the ancient China.'

The command-line options are as follows:

• -c: Path to the model file
• -f: Model file format (e.g., gguf)
• -j: Number of threads to use (e.g., 56)
• -q: Quantization mode (e.g., int8)
• -n: Number of tokens to generate (e.g., 200)
• -i: Input text (e.g., 'That was a long long story happened in the ancient China.')
• -h: show usage information

rapid-llama achieves a generation speed of approximately 25-30 tokens/s for an 8-bit quantized 7B model running on the following CPU configuration:

Architecture:            x86_64
Model name:              Intel(R) Xeon(R) Platinum 8350C CPU @ 2.60GHz
CPU(s):                  112
Thread(s) per core:      2
Core(s) per socket:      28
Socket(s):               2

Latancy of first token will be optimized laterly.

Why is it so fast?

• Ultimate memory efficiency
  • Zero memory allocations and frees during inferencing.
  • Maximization of memory locality.
• Well-designed thread scheduling algorithm
• Optimized operators
  • Fuse all operators that can be fused together
  • Optmize calculation of several operators
• Proper Quantizations

rapid-llama is licensed under the Apache 2 License.

We would like to express our gratitude to all contributors and users of RapidLLaMA. Your support and feedback have been invaluable in making this project a success. If you encounter any issues or have any suggestions, please feel free to open an issue on the GitHub repository.

Email: 📩[email protected]

Contact me if you any questions.