This is a lab for efficientml.ai course.

Running large language models (LLMs) on the edge is of great importance. By embedding LLMs directly into real-world systems such as in-car entertainment systems or spaceship control interfaces, users can access instant responses and services without relying on a stable internet connection. Moreover, this approach alleviates the inconvenience of queuing delays often associated with cloud services. As such, running LLMs on the edge not only enhances user experience but also addresses privacy concerns, as sensitive data remains localized and reduces the risk of potential breaches.

However, despite their impressive capabilities, LLMs have traditionally been quite resource-intensive. They require considerable computational power and memory resources, which makes it challenging to run these models on edge devices with limited capabilities.

In this lab, you will learn the following:

• How to deploy an LLaMA2-7B-chat with TinyChatEngine on your computer.
• Implement different optimization techniques (loop unrolling, multithreading, and SIMD programming) for the linear kernel.
• Observe the end-to-end latency improvement achieved by each technique.

This tutorial is based on TinyChatEngine, a powerful neural network library specifically designed for the efficient deployment of quantized large language models (LLMs) on edge devices.

Please check this document and follow the instructions which will walk you through the tutorial: https://docs.google.com/document/d/13IaTfPKjp0KiSBEhPdX9IxgXMIAZfiFjor37OWQJhMM/edit?usp=sharing

• Report: Please write a report (form) that includes your code and the performance improvement for each starter code.
• Code: Use git diff to generate a patch for your implementation. We will use this patch to test the correctness of your code. Please name your patch as {studentID}-{ISA}.patch where {ISA} should be one of x86 and ARM, depending on your computer.

TinyChatEngine.

TinyEngine.

Smoothquant.

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

llama.cpp

transformers