Comments (5)
Here's how I'm currently solving this (adapted from usage in README) :
from evo import Evo
import torch
device = 'cuda:0'
evo_model = Evo('evo-1-131k-base')
model, tokenizer = evo_model.model, evo_model.tokenizer
model.to(device)
model.eval()
# monkey patch the unembed function with identity
# this removes the final projection back from the embedding space into tokens
# so the "logits" of the model is now the final layer embedding
# see source for unembed - https://huggingface.co/togethercomputer/evo-1-131k-base/blob/main/model.py#L339
from torch import nn
class CustomEmbedding(nn.Module):
def unembed(self, u):
return u
model.unembed = CustomEmbedding()
# end custom code
sequence = 'ACGT'
input_ids = torch.tensor(
tokenizer.tokenize(sequence),
dtype=torch.int,
).to(device).unsqueeze(0)
embed, _ = model(input_ids) # (batch, length, embed dim)
print('Embed: ', embed)
print('Shape (batch, length, embed dim): ', embed.shape)
# you can now use embedding for downstream classification tasks
# you probably want to aggregate over position dimension
# e.g. mean value = embed.mean(dim=1) or final token embedding = embed[:, -1, :]
Note that this is for the model object returned by evo-model, which is an instance of StripedHyena. If you are using Huggingface directly, this is wrapped with StripedHyenaModelForCausalLM, so you need to do model.backbone.unembed = CustomEmbedding()
from evo.
Thanks @davidkell !
from evo.
I had a similar experience. I was able to get inference working for 2k sequences on A100 80GB (e.g. available on Paperspace), although around 2.5-3k I would get OOM. I haven't looked in depth on what is driving the memory requirement
from evo.
Quoting from this issue #24:
Prompting with longer sequences requires sharding for the model, which is currently not supported
So I think if you want to generate embeddings for longer sequences, you will need to manually shard on GPUs or setup CPU offloading or something like that
from evo.
@davidkell I tried your code on a A100 40GB using the evo-8k model, embedding the 4-letter sequence in the example costs over 400MB GPU RAM, the model itself needs 13GB. The embedding dimension is 4096. I don't understand why it cost so much memory. 4x4096 BF16 should only take 32KB, right? I tried to embed a 2kb sequence but always ran out of cuda memory. Anyone has a similar problem?
from evo.
Related Issues (20)
- Default pretrained_model_path is personal directory
- Is it posisble for us to access the embeddings layer? HOT 2
- TORCH_HOME not used when downloading models HOT 2
- gene essentiality HOT 1
- T4 GPUs support? Any recommendation HOT 1
- Scoring DNA with lineages HOT 1
- hello_evo.ipynb code error
- ModuleNotFoundError: No module named 'transformers_modules.togethercomputer.evo-1-131k-base.9562f3fdc38f09b92594864c5e98264f1bfbca33.tokenizer' HOT 8
- 131k base seems to output non-DNA characters HOT 4
- AttributeError: 'InferenceParams' object has no attribute 'fused_ft_kernel' HOT 1
- Limiting attention radius and extracting embeddings HOT 1
- How can I use multiple graphics cards to fine tune this model
- Memory usage? HOT 1
- issue on model.to("cuda") with device_map="auto" HOT 1
- PEFT Prompt Tuning: forward() got an unexpected keyword argument 'inputs_embeds'
- Scoring Question HOT 1
- Require for GWAS(genome wide association studies) HOT 1
- more prompt scheme documentation? HOT 1
- Estimate Required RAM for inference
- Pretrain
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from evo.