Apple Silicone Neural Engine: Core ML model package format support about autogpt HOT 6 OPEN

I'll be interested to look into this at some point

from autogpt.

Do you have a quick start on this? Haven’t looked into core ML at all

from autogpt.

Hi, yes. @ntindle

This is about running LLMs locally on Apple Silicone. Core ML is a framework that can redistribute workload across CPU, GPU & Nural Engine (ANE). ANE is available on all modern Apple Devices: iPhones & Macs (A14 or newer and M1 or newer). Ideally, we want to run LLMs on ANE only as it has optimizations for running ML tasks compared to GPU. Apple claims "deploying your Transformer models on Apple devices with an A14 or newer and M1 or newer chip to achieve up to 10 times faster and 14 times lower peak memory consumption compared to baseline implementations".

1. To utilize Core ML first, you need to convert a model from TensorFlow, PyTorch to Core ML model package format using coremltools (or simply utilize existing models in Core ML package format ).
2. Second, you must now use that converted package with an implementation designed for Apple Devices. Here is the Apple XCode reference PyTorch implementation.

https://machinelearning.apple.com/research/neural-engine-transformers

from autogpt.

We don't package any models with our code. Is it possible to use tools like Llamafile to do this?

from autogpt.

I mean, it was a general overview; you don't have to package models; you just need to be able to use CoreML packaged models.

Work in progress on CoreML implementation for [whisper.cpp]. They see x3 performance improvements for some models. (ggerganov/whisper.cpp#548) you might be interested in.

You might also be interested in another implementation Swift Transformers. Example of CoreML application
https://github.com/huggingface/swift-chat

from autogpt.

https://appleinsider.com/articles/24/05/07/secret-apple-project-acdc-to-pioneer-ai-chips-for-data-centers

Under the internal name "Project ACDC," Apple is developing Apple Silicon designed specifically for server farms dedicated to AI processing. The company aims to optimize AI applications within its data centers for future versions of its platforms.

from autogpt.