Comments (2)
I could be wrong, but I think this project is mostly a wrapper around the ggml projects like llama.cpp. I think if those papers were implemented over there, it'd be much more likely to make it into apps like this.
from llmfarm.
Latest Research Papers from Apple. I would say a lot of people are not looking into these. I am not trying to spam nor be annoying. I would love to see AI be more available for people with less computational resources, such as myself.
I would love to help with LLMFarm and implement these.
Additional Technologies that may be implemented into LLMFarm that may take a GGUF model, convert & create an optimized model utilizing the below technologies.
ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models
"Large Language Models (LLMs) with billions of parameters have drastically transformed AI applications. However, their demanding computation during inference has raised significant challenges for deployment on resource-constrained devices. Despite recent trends favoring alternative activation functions such as GELU or SiLU, known for increased computation, this study strongly advocates for reinstating ReLU activation in LLMs. We demonstrate that using the ReLU activation function has a negligible impact on convergence and performance while significantly reducing computation and weight transfer. This reduction is particularly valuable during the memory-bound inference step, where efficiency is paramount. Exploring sparsity patterns in ReLU-based LLMs, we unveil the reutilization of activated neurons for generating new tokens and leveraging these insights we propose practical strategies to substantially reduce LLM inference computation up to three times, using ReLU activations with minimal performance trade-offs."
PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model
"Autoregressive models for text sometimes generate repetitive and low-quality output because errors accumulate during the steps of generation. This issue is often attributed to exposure bias - the difference between how a model is trained and how it is used during inference. Denoising diffusion models provide an alternative approach in which a model can revisit and revise its output. However, they can be computationally expensive, and prior efforts on text have led to models that produce less fluent output compared to autoregressive models, especially for longer text and paragraphs. In this paper, we propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation, to generate fluent text while exercising global control over paragraphs. The model achieves this by combining an autoregressive "decoding" module with a "planning" module that uses latent diffusion to generate semantic paragraph embeddings in a coarse-to-fine manner. The proposed method is evaluated on various conditional generation tasks, and results on semantic generation, text completion, and summarization show its effectiveness in generating high-quality long-form text in an efficient manner."
HyperDiffusion: Generating Implicit Neural Fields with Weight-Space Diffusion
"Implicit neural fields, typically encoded by a multilayer perceptron (MLP) that maps from coordinates (e.g., xyz) to signals (e.g., signed distances), have shown remarkable promise as a high-fidelity and compact representation. However, the lack of a regular and explicit grid structure also makes it challenging to apply generative modeling directly on implicit neural fields in order to synthesize new data. To this end, we propose HyperDiffusion, a novel approach for unconditional generative modeling of implicit neural fields. HyperDiffusion operates directly on MLP weights and generates new neural implicit fields encoded by synthesized MLP parameters. Specifically, a collection of MLPs is first optimized to faithfully represent individual data samples. Subsequently, a diffusion process is trained in this MLP weight space to model the underlying distribution of neural implicit fields. HyperDiffusion enables diffusion modeling over a implicit, compact, and yet high-fidelity representation of complex signals across 3D shapes and 4D mesh animations within one single unified framework."
from llmfarm.
Related Issues (20)
- Dolphin models problem (modelLoadError) HOT 5
- Small bugs/unexpected behavior and feedback with suggestions HOT 6
- Can the model get information from the outside? HOT 3
- Crash when running with RWKV 5 (Raven) HOT 2
- Sampling stuck in greedy HOT 6
- iOS 15 support HOT 2
- Additional Support for Vision Based Models Like Llava? HOT 2
- Apple Model Available HOT 2
- Few feature requests.. HOT 3
- Adding instruction for TheBloke/TinyLlama-1.1B-Chat-v0.3-GGUF HOT 1
- App crashes right away HOT 1
- It's so slow HOT 6
- Bugs and suggestions HOT 2
- Issue with LLAVA 1.5 HOT 4
- The development build will either crash or produce incorrect output content. HOT 11
- Add support for image generation models like stable diffusion HOT 1
- The version downloaded via git crashes on the physical device iPhone 14 Plus HOT 1
- Google Gemma 1.1 Support HOT 1
- Add support for Llama 3 HOT 6
- Support for Phi-3 models HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from llmfarm.