quentin-anthony Goto Github PK

5800_holography

6341-public

CSE 6341 Au19 publicly available code

apex

A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

aws-neuron-reference-for-megatron-lm

bert-pytorch

BERT for Distributed PyTorch + AMP Training

climdeeplearn

Repo for climate deep learning codes

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

deepspeedexamples

Example models using DeepSpeed

dl-sr

Tensorflow/keras implementation for image transformation from low-resolution (LR) image to super-resolved one, including single wide-field (WF) image super-resolution prediction and SIM reconstruction.

easybuild-easyconfigs

A collection of easyconfig files that describe which software to build using which build options with EasyBuild.

examples

A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.

flash-attention

Fast and memory-efficient exact attention

grace

GRACE - GRAdient ComprEssion for distributed deep learning

hierarchicalkv

HierarchicalKV is a part of NVIDIA Merlin and provides hierarchical key-value storage to meet RecSys requirements. The key capability of Merlin-KV is to store key-value feature-embeddings on high-bandwidth memory (HBM) of GPUs and in host memory. It also can be used as a generic key-value storage.

horovod

Distributed training framework for TensorFlow, Keras, PyTorch, and MXNet.

incubator-mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Scala, Go, Javascript and more

ior

IOR and mdtest

magma

MAGMA - a GPT-style multimodal model that can understand any combination of images and language. NOTE: The freely available model from this repo is only a demo. For the latest multimodal and multilingual models from Aleph Alpha check out our website https://app.aleph-alpha.com

megatron-deepspeed

Ongoing research training transformer language models at scale, including: BERT & GPT-2

megatron-deepspeed-ms

Ongoing research training transformer language models at scale, including: BERT & GPT-2

mingpt

minGPT in JAX

ml-engineering

Machine Learning Engineering Open Book

models

Models and examples built with TensorFlow

nanogpt

The simplest, fastest repository for training/finetuning medium-sized GPTs.

nemo

NeMo: a toolkit for conversational AI

ok-topk

Ok-Topk is a scheme for distributed training with sparse gradients. Ok-Topk integrates a novel sparse allreduce algorithm (less than 6k communication volume which is asymptotically optimal) with the decentralized parallel Stochastic Gradient Descent (SGD) optimizer, and its convergence is proved theoretically and empirically.

pythia

pytorch

Tensors and Dynamic neural networks in Python with strong GPU acceleration

quentin-anthony.github.io

A beautiful, simple, clean, and responsive Jekyll theme for academics

rwkv-lm

RWKV is an RNN with transformer-level LLM performance. It can be directly trained like a GPT (parallelizable). So it's combining the best of RNN and transformer - great performance, fast inference, saves VRAM, fast training, "infinite" ctx_len, and free sentence embedding.