facebookresearch / fairseq2 Goto Github PK

View Code? Open in Web Editor NEW

574.0 14.0 54.0 9.96 MB

FAIR Sequence Modeling Toolkit 2

Home Page: https://facebookresearch.github.io/fairseq2/

License: MIT License

TeX 0.84% Shell 0.40% Python 73.12% CMake 2.31% C++ 23.10% C 0.04% Cuda 0.19%

artificial-intelligence deep-learning machine-learning python pytorch

fairseq2's People

Contributors

Stargazers

Watchers

fairseq2's Issues

Introduce MultilingualTextDataset

As part of the NLLB pretraining recipe, introduce a new MultilingualTextDataset interface, optionally with a basic implementation. Also consider having dataset type specific registries, so we can construct then by load_<type>_dataset(), e.g. load_multilingual_text_dataset().

Integration of TFGridNet from ESPNet

To Integrate a new speech enhancement model, TF-GridNet, into Fairseq2

`freqs` in the updated Rotary encoder is of type `torch.complex64` which is not supported in DistributedDataParallel

Describe the bug:
The updated RotaryEncoder implementation introduced in this PR changes the dtype of freqs to torch.complex64. However, complex tensors are not supported by pytorch's DistributedDataParallel. Specifically it will error out during _sync_module_states in DDP's __init__.

Describe how to reproduce:
Here's a simple script to reproduce the error. Make sure that you're in a GPU-enabled environment.

from fairseq2.models.llama import load_llama_model
import os
import torch
import torch.nn as nn
import torch.distributed as dist


rank = int(os.environ["RANK"])
world_size = int(os.environ["WORLD_SIZE"])
dist.init_process_group(
    backend="nccl",
    init_method="env://",
    rank=rank,
    world_size=world_size
)
torch.cuda.set_device(rank)

model = load_llama_model("llama_7b", device=torch.device("cuda"))
ddp_model = nn.parallel.DistributedDataParallel(model, device_ids=[rank])

And run with torchrun --nproc_per_node 1 test.py will produce the following error

Traceback (most recent call last):
  File "/mnt/fsx-home/chaoweihuang/fairseq2-public-fork/test.py", line 19, in <module>
    ddp_model = nn.parallel.DistributedDataParallel(model, device_ids=[rank])
  File "/fsx-ust/chaoweihuang/miniconda3/envs/seamless-fairseq2/lib/python3.10/site-packages/torch/nn/parallel/distributed.py", line 676, in __init__
    _sync_module_states(
  File "/fsx-ust/chaoweihuang/miniconda3/envs/seamless-fairseq2/lib/python3.10/site-packages/torch/distributed/utils.py", line 142, in _sync_module_states
    _sync_params_and_buffers(
  File "/fsx-ust/chaoweihuang/miniconda3/envs/seamless-fairseq2/lib/python3.10/site-packages/torch/distributed/utils.py", line 160, in _sync_params_and_buffers
    dist._broadcast_coalesced(
RuntimeError: Input tensor data type is not supported for NCCL process group: ComplexFloat

A workaround for this is that we can set the _ddp_params_and_buffers_to_ignore attribute in the root module that's going to be wrapped with DDP since we don't really need to sync freqs across ranks:

model._ddp_params_and_buffers_to_ignore = [
    f"decoder.layers.{i}.self_attn.pos_encoder.freqs"
    for i in range(len(model.decoder.layers))
]

Revise DownloadManager API to handle concurrent calls

When multiple processes in a distributed job attempt to download the same asset, we do not handle it in a race-free way if the file system is shared. Think of a way to avoid it in FileDownloadManager

Add pip built distributions / Support for Windows OS

Is your feature request related to a problem? Please describe:
Not able to install fairseq2 / fairseq2n using pip as no windows built distributions available

Describe the solution you would like:
I was not able to build from source myself on windows , if anyone can help todo so or update documentations to include instruction how to build from source on windows systems or add more supports for build distribution using pip

Additional Context:
windows 11 , amd64, py 3.10, cu118

w2v-bert weights

Hi fairseq2 team!

README mentions that w2v-bert is avaliable as a pre-trained model, but I cannot find the link to download the weights anywhere - only the classes. I can see the links for S2T under assets/cards/*.yaml, but not for w2v-bert. Are you planning to release the weights?

Also, SeamlessM4T mentions w2v-bert 2.0 - are there any plans to open-source it as well?

Thank you for your excellent work, much appreciated!

Best,
Artem

I want to perform inference with cpu but I have encountered an error.

from fairseq2n.bindings.data.string import CString
Traceback (most recent call last):
File "", line 1, in
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory

torch 2.1.1+cpu pypi_0 pypi
torchaudio 2.1.1+cpu pypi_0 pypi
torcheval 0.0.7 pypi_0 pypi
torchvision 0.16.1+cpu pypi_0 pypi

Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-88-generic x86_64)

Error when using file:// URI in asset card yaml

Hi - I'm evaluating seamless-m4t and am trying to load a model checkpoint locally. When I specify a file:// URI I get the following error:

fairseq2.assets.card.AssetCardError: The value of the field 'checkpoint' of the asset card 'seamlessM4T_large' must be a valid URI, but is 'file:///foo/bar/seamless-m4t-large/multitask_unity_large.pt' instead.

I can work around this by starting up a python http.server and serve the file locally, but it would be nice to have proper file:// URI support.

Originally posted by @jgwinnup in facebookresearch/seamless_communication#50 (comment)

`WaveformToFbankConverter` does not work as expected for `channels > 1`

Describe the bug:
While the AudioDecoder can read a stereo wav file, the output of WaveformToFbankConverter is always num_frames x num_mel_bins.

Describe how to reproduce:

audio_wav_path, device, dtype = ...
audio_decoder = AudioDecoder(dtype=torch.float32, device=device)
fbank_converter = WaveformToFbankConverter(
    num_mel_bins=80,
    waveform_scale=2**15,
    channel_last=True,
    standardize=True,
    device=device,
    dtype=dtype,
)
with Path(audio_wav_path).open("rb") as fb:
    block = MemoryBlock(fb.read())

decoded_audio = audio_decoder(block) # for stereo waveform shape is (num_samples x num_channels=2)
fbank_features = fbank_converter(decoded_audio)['fbank']  
# shape is always (num_frames x num_mel_bins) irrespective of num_channels

Describe the expected behavior:
WaveformToFbankConverter should handle stereo/multi-channel audio and return (num_channels x num_frames x num_mel_bins)

Environment:
Installed with following command:
pip install fairseq2 --pre --extra-index-url https://fair.pkg.atmeta.com/fairseq2/whl/nightly/pt2.2.0/cu118

fairseq2n is not compatible with fmt in PyTorch nightlies

ome/ezyang/local/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc:7:
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h: In instantiation of ‘constexpr fmt::v10::detail::value<Context> fmt::v10::detail::make_arg(T&) [with bool PACKED = true; Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; T = const fairseq2n::immutable_string; typename std::enable_if<PACKED, int>::type <anonymous> = 0]’:
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1808:51:   required from ‘constexpr fmt::v10::format_arg_store<Context, Args>::format_arg_store(T& ...) [with T = {const fairseq2n::immutable_string}; Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; Args = {fairseq2n::immutable_string}]’
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1826:18:   required from ‘constexpr fmt::v10::format_arg_store<Context, typename std::remove_cv<typename std::remove_reference<T>::type>::type ...> fmt::v10::make_format_args(T& ...) [with Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; T = {const fairseq2n::immutable_string}]’
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:2788:44:   required from ‘std::string fmt::v10::format(fmt::v10::format_string<T ...>, T&& ...) [with T = {const fairseq2n::immutable_string&}; std::string = std::__cxx11::basic_string<char>; fmt::v10::format_string<T ...> = fmt::v10::basic_format_string<char, const fairseq2n::immutable_string&>]’
/home/ezyang/local/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc:68:35:   required from here
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1576:63: error: ‘fmt::v10::detail::type_is_unformattable_for<const fairseq2n::immutable_string, char> _’ has incomplete type
 1576 |     type_is_unformattable_for<T, typename Context::char_type> _;
      |                                                               ^
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1580:7: error: static assertion failed: Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
 1580 |       formattable,
      |       ^~~~~~~~~~~
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1580:7: note: ‘formattable’ evaluates to false
                 from /home/ezyang/local/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc:7:
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h: In instantiation of ‘constexpr fmt::v10::detail::value<Context> fmt::v10::detail::make_arg(T&) [with bool PACKED = true; Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; T = const fairseq2n::immutable_string; typename std::enable_if<PACKED, int>::type <anonymous> = 0]’:
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1808:51:   required from ‘constexpr fmt::v10::format_arg_store<Context, Args>::format_arg_store(T& ...) [with T = {const fairseq2n::immutable_string}; Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; Args = {fairseq2n::immutable_string}]’
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1826:18:   required from ‘constexpr fmt::v10::format_arg_store<Context, typename std::remove_cv<typename std::remove_reference<T>::type>::type ...> fmt::v10::make_format_args(T& ...) [with Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; T = {const fairseq2n::immutable_string}]’
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:2788:44:   required from ‘std::string fmt::v10::format(fmt::v10::format_string<T ...>, T&& ...) [with T = {const fairseq2n::immutable_string&}; std::string = std::__cxx11::basic_string<char>; fmt::v10::format_string<T ...> = fmt::v10::basic_format_string<char, const fairseq2n::immutable_string&>]’
/home/ezyang/local/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc:68:35:   required from here
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1576:63: error: ‘fmt::v10::detail::type_is_unformattable_for<const fairseq2n::immutable_string, char> _’ has incomplete type
 1576 |     type_is_unformattable_for<T, typename Context::char_type> _;
      |                                                               ^
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1580:7: error: static assertion failed: Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
 1580 |       formattable,
      |       ^~~~~~~~~~~
/home/ezyang/local/pytorch-venv/torch/include/fmt/core.h:1580:7: note: ‘formattable’ evaluates to false

workaround

diff --git a/fairseq2n/python/src/fairseq2n/bindings/data/string.cc b/fairseq2n/python/src/fairseq2n/bindings/data/string.cc
index 8e5ab07..67cfa74 100644
--- a/fairseq2n/python/src/fairseq2n/bindings/data/string.cc
+++ b/fairseq2n/python/src/fairseq2n/bindings/data/string.cc
@@ -65,7 +65,7 @@ def_string(py::module_ &data_module)
             "__repr__",
             [](const immutable_string &self)
             {
-                return fmt::format("CString('{}')", self);
+                return fmt::format("CString('{}')", self.to_string());
             })
 
         .def(

Benchmark different SDPA implementations

As of today, we support TorchSDPA (which can be backed by different implementations) and NaiveSDPA. Benchmark their performance and, in particular, how they behave when we have padding in the input which is typical in machine translation tasks.

Finish NLLB training recipe

Publish our first public training recipe where we demonstrate how users can train an NLLB model using fairseq2 components.

ImportError:libfairseq2n.so.0: undefined symbol

from fairseq2.data import Collater

File "/root/miniconda3/envs/xj/lib/python3.10/site-packages/fairseq2/data/init.py", line 7, in
from fairseq2.data.cstring import CString as CString
File "/root/miniconda3/envs/xj/lib/python3.10/site-packages/fairseq2/data/cstring.py", line 58, in
from fairseq2n.bindings.data.string import CString as CString
ImportError: /root/miniconda3/envs/xj/lib/python3.10/site-packages/fairseq2n/lib/libfairseq2n.so.0: undefined symbol: _ZN3c106detail14torchCheckFailEPKcS2_jRKSs

Restore text file offset in constant time

As of today, the reload_position() implementation of TextDataSource is inefficient. It iterates through the entire file until it reaches the restored line number. This works fairly well for relatively small file sizes, but for files larger than a couple GBs it starts to become significantly slower. Revise the implementation of record_position and reload_position to make it a constant time operation if possible. Might be trickier than it sounds due to buffered reading we typically perform.

why fairseq2 and has fairseq been deprecated?

license issue for w2v-bert

Hi fairseq2 team,
Thank you for open-sourcing w2v-bert, I wonder:

What will be the license to the w2v-bert weights? it is released with seamless-m4t, which is cc-by-nc, but fairseq2 is MIT licensed. I spotted in previous issues that this repo does not plan to release standalone w2v-bert, so If I extract the weights from m4t, which license should I use?

sample_rate defined as float in data/audio.py

In src/fairseq2/data/audio.py AudioDecoderOutput, WaveformToFbankInput and WaveformToFbankOutput, the sample_rate is defined as float and not as an integer.

I think this might be an error, but git history shows that it used to be an integer. In every other library I know (espnet, fairseq, soundfile, librosa, torchaudio, etc.), the sample rate is assumed to be an integer, as it should be since it is the number of frames per second, which cannot be non integer.

Here is an example of a problem that could occur (and that I have personally experienced):

import torch
import soundfile as sf
from seamless_communication.models.inference import Translator

waveform, sample_rate = sf.read("any_audio_file.wav")
waveform = torch.from_numpy(waveform)
translator = Translator("seamlessM4T_medium", vocoder_name_or_card="vocoder_36langs", device=torch.device("cuda:0"), dtype=torch.float32)
translated_text, *_ = translator.predict(waveform, "s2st", "fra")
# ValueError: The input sample rate must be of type `float`, but is of type `int` instead.

Does this mean you're deprecating fairseq?

controllable `WaveformToFbankConverter` multithreading

Describe the bug:
WaveformToFbankConverter is running in multithread parallel.
This method (as possibly some others) uses parallel_for statement for the execution.
Currently, there's no obvious way to control the number of threads it uses.
That could lead to some performance drawback (like threads/cpu oversubscription).
Moreover, it turns out that even when used inside DataPipeline.map(...),
it does not respect the number of required parallel calls.

Describe how to reproduce:

from fairseq2.data.data_pipeline import read_sequence
from fairseq2.data.audio import WaveformToFbankConverter

_convert_to_fbank = WaveformToFbankConverter(
                num_mel_bins=80,
                waveform_scale=2**15,
                channel_last=True,
                standardize=True,
                device=torch.device("cpu"),
                dtype=torch.float16)
def convert_to_fbank(wav):
    return _convert_to_fbank({"waveform": torch.unsqueeze(wav, 1),
                              "sample_rate": 16_000})['fbank'].shape

xx = [torch.rand(10**5) for i in range(100)]
data_pipeline = read_sequence(xx).map(convert_to_fbank, num_parallel_calls=1).and_return()
list(iter(data_pipeline))  # this will use typically half of available cpus

Describe the expected behavior:

a context manager to control number of threads the method uses (with fairseq2_nb_threads(2): ...)
make WaveformToFbankConverter respect num_parallel_calls in data pipelining

Environment:
fairseq2==0.1.1+cu118
fairseq2n==0.1.1+cu118

pip install fairseq2 requires pytorch2 via fairseq2n

I am really struggling to install faiseq2 via a dependency manager as the installation triggers fairseq2n which requires pytorch2. Have tried specifying --extra-index-url https://fair.pkg.atmeta.com/fairseq2/whl/pt1.13.1/cu116 but that still doesn't circumvent pytorch2 requirement

Audio Data Loader Example?

Hi, I'd like to use FairSeq2 for my audio models. Is there an example of an audio data loader in the works? That would be super helpful.

What I need in the dataloader:

I have lots of audio that I'd like to load. This is currently in mp3, but I've been cutting chunks out as tuples of (audio: Wav, labels: Json).
This data is hosted on R2 and I'd like the loader to stream it via an endpoint_url.
I also would like for this to save to local cache as it's a big dataset.
Distribute amongst cpu workers.

I can accommodate other data saving formats as needed, e.g. a flat zip, but the most important thing is that it's fast. I'm right now I/O bottlenecked.

Thanks!

Adapters in fairseq2

Since LORA is implemented in fairseq2, it would be good to have general adapter modules available as part of PEFT.
Can someone please add them to the repository?

Calling `SequenceToTextGeneratorBase._do_generate` with empty data throws IndexError

Describe the bug:
When the base class SequenceToTextGeneratorBase processes empty data in its method _do_generate(self, source_seqs: Tensor, source_seq_lens: Optional[Tensor]), at line 80, the code generates an IndexError:

fairseq2/src/fairseq2/generation/text.py

Lines 67 to 84 in 0fc5de7

 @torch.inference_mode() 

 def _do_generate( 

 self, source_seqs: Tensor, source_seq_lens: Optional[Tensor] 

 ) -> "SequenceToTextOutput": 

 """A subclass should call this function for the actual text generation.""" 

 encoder_output, encoder_padding_mask = self.model.encode( 

 source_seqs, source_seq_lens 

 ) 

 gen_output = self.generator( 

 encoder_output, encoder_padding_mask, source_seq_len=source_seqs.size(1) 

 ) 

 sentences = [self.token_decoder(b[0].seq)[0] for b in gen_output.results] 

 return SequenceToTextOutput( 

 sentences, gen_output, encoder_output, encoder_padding_mask 

 )

Which throws:

[/usr/local/lib/python3.10/dist-packages/fairseq2/generation/text.py](https://localhost:8080/#) in <listcomp>(.0)
     78 
     79         # TODO: use parallel_invoke
---> 80         sentences = [self.token_decoder(b[0].seq)[0] for b in gen_output.results]
     81 
     82         return SequenceToTextOutput(

IndexError: list index out of range

Describe how to reproduce:

import torch
import torchaudio
from seamless_communication.models.inference import Translator

translator = Translator(
    model_name_or_card="seamlessM4T_medium",
    vocoder_name_or_card="vocoder_36langs",
    device=torch.device('cuda:0')
)

# Generate a silent audio clip.
sample_rate = 16000
duration = 10
channels = 1
empty_waveform = torch.zeros((channels, duration*sample_rate))
torchaudio.save('empty_audio.wav', src=empty_waveform, sample_rate=sample_rate, format='wav')

# Perform speech-to-text translation.
text, _, _ = translator.predict(
    input="empty_audio.wav",
    task_str="S2TT",
    tgt_lang="ben",
    sample_rate=sample_rate
)

Full stack trace


---------------------------------------------------------------------------
IndexError Traceback (most recent call last)
[](https://localhost:8080/#) in ()
15 torchaudio.save('empty_audio.wav', src=empty_waveform, sample_rate=sample_rate, format='wav')
16
---> 17 text, _, _ = translator.predict(
18 input="empty_audio.wav",
19 task_str="S2TT",
8 frames

/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs)

113 def decorate_context(*args, **kwargs):

114 with ctx_factory():

--> 115 return func(*args, **kwargs)

116

117 return decorate_context
/usr/local/lib/python3.10/dist-packages/seamless_communication/models/inference/translator.py in predict(self, input, task_str, tgt_lang, src_lang, spkr, ngram_filtering, sample_rate, text_max_len_a, text_max_len_b, unit_max_len_a, unit_max_len_b)

223 src = self.collate(self.token_encoder(text))

224

--> 225 result = self.get_prediction(

226 self.model,

227 self.text_tokenizer,
/usr/local/lib/python3.10/dist-packages/seamless_communication/models/inference/translator.py in get_prediction(cls, model, text_tokenizer, unit_tokenizer, src, input_modality, output_modality, tgt_lang, ngram_filtering, text_max_len_a, text_max_len_b, unit_max_len_a, unit_max_len_b)

139 unit_opts=unit_opts,

140 )

--> 141 return generator(

142 src["seqs"],

143 src["seq_lens"],
/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs)

113 def decorate_context(*args, **kwargs):

114 with ctx_factory():

--> 115 return func(*args, **kwargs)

116

117 return decorate_context
/usr/local/lib/python3.10/dist-packages/seamless_communication/models/unity/generator.py in call(self, source_seqs, source_seq_lens, input_modality, output_modality, ngram_filtering)

171

172 if input_modality == "speech":

--> 173 text_output = self.s2t_generator.generate_ex(source_seqs, source_seq_lens)

174 elif input_modality == "text" and self.t2t_generator is not None:

175 text_output = self.t2t_generator.generate_ex(source_seqs, source_seq_lens)
/usr/local/lib/python3.10/dist-packages/fairseq2/generation/text.py in generate_ex(self, source_seqs, source_seq_lens)

153 :math:N is the batch size.

154 """

--> 155 return self._do_generate(source_seqs, source_seq_lens)

156

157
/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py in decorate_context(*args, **kwargs)

113 def decorate_context(*args, **kwargs):

114 with ctx_factory():

--> 115 return func(*args, **kwargs)

116

117 return decorate_context
/usr/local/lib/python3.10/dist-packages/fairseq2/generation/text.py in _do_generate(self, source_seqs, source_seq_lens)

78

79 # TODO: use parallel_invoke

---> 80 sentences = [self.token_decoder(b[0].seq)[0] for b in gen_output.results]

81

82 return SequenceToTextOutput(
/usr/local/lib/python3.10/dist-packages/fairseq2/generation/text.py in (.0)

78

79 # TODO: use parallel_invoke

---> 80 sentences = [self.token_decoder(b[0].seq)[0] for b in gen_output.results]

81

82 return SequenceToTextOutput(
IndexError: list index out of range

Describe the expected behavior:
There shouldn't be any IndexError exceptions thrown by SequenceToTextGeneratorBase._do_generate. Instead, it should just return normally with an empty result.

Environment:
Dependencies:

PyTorch: pip install -q torch==2.0.1+cu118 torchaudio==2.0.2+cu118
FairSeq2: pip install -q fairseq2==0.1.1 fairseq2n==0.1.1
Seamless M4T: pip install -q git+https://github.com/facebookresearch/seamless_communication.git@17adc4035904a1015fc39312884ab0347cb969f5#egg=seamless_communication

Additional Context:
I'm not familiar with the library so I'm afraid to say that there is most definitely a much simpler code example than what I've given above.

Cannot build: undefined reference to `libiconv_close'

I don't see any target_link_libraries to iconv so I'm not sure why this is expected to work in the first place

About dataset part and trainer part of fairseq2

Until now, I don't seem to have found any details about the specific example of dataset organizing and training from scratch , Would fairseq2 intend to provide a separate trainer library and dataset library like lightning and accelerate ,datasets from hugging face.? I think that fairseq2 mainly focuses on model details like transformers library from hugging face now.

I'd like to decouple training and modeling so that it's more conducive to updating the training part quickly, one of the main pain points of self-supervised large models at the moment is that training is time-consuming and slow。
While, I hope that fairseq2 has a good recipe organized for example, maybe like speech open source projects (i.e. kaldi, espnet,wenet and so on), to solve fairseq example is unfriendly and some details missing for new users.

Can I add support to other backends?

First of all, thanks for this awesome project. But, I'm wondering whether it's welcome to contribute other backends support except CUDA and CPU.

Support more backends, allowing fairseq2 to be used in more scenarios, what do you think?

build fairseq2n fails with "undefined reference to `_gfortran_etime@GFORTRAN_8'"

Describe the bug:
I am trying to install fairseq2 from source using the instruction but getting a fail when building fairseq2n.

Describe how to reproduce:
Env: scratch docker image for python:3.11

> uname -a
Linux 582069085efd 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64 GNU/Linux

apt install -y libsndfile1 libsndfile-dev
git clone --recurse-submodules https://github.com/facebookresearch/fairseq2.git
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu
cd fairseq2
pip install -r fairseq2n/python/requirements-build.txt
cd fairseq2n

> cmake -GNinja -B build

-- Module support is disabled.
-- Version: 9.1.0
-- Build type: RelWithDebInfo
-- CXX_STANDARD: 14
-- Required features: cxx_variadic_templates
CMake Deprecation Warning at third-party/kaldi-native-fbank/CMakeLists.txt:24 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- BUILD_SHARED_LIBS: FALSE
-- KALDI_NATIVE_FBANK_BUILD_TESTS: OFF
-- KALDI_NATIVE_FBANK_BUILD_PYTHON: OFF
-- KALDI_NATIVE_FBANK_ENABLE_CHECK: OFF
-- KALDI_NATIVE_FBANK_ENABLE_CHECK: OFF
-- CMAKE_CXX_FLAGS: 
-- CMAKE_INSTALL_PREFIX: /usr/local
-- Disable building Python
CMake Deprecation Warning at third-party/sentencepiece/CMakeLists.txt:15 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- VERSION: 0.1.97
-- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
-- pybind11 v2.11.1 
-- 
-- ******************** Summary ********************
--   CMake Version                      : 3.27.7
--   CMake Command                      : /usr/local/lib/python3.11/site-packages/cmake/data/bin/cmake
--   System                             : Linux
--   C++ Compiler                       : /usr/bin/c++
--   C++ Compiler Version               : 12.2.0
--   Python Version                     : 3.11.6
--   Python Interpreter                 : /usr/local/bin/python3.11
--   Torch Version                      : 2.1.1
--   Torch Library                      : /usr/local/lib/python3.11/site-packages/torch/lib/libtorch.so
--   Build Type                         : RelWithDebInfo
-- 
--   PROJECT_VERSION                    : 0.2.0
--   FAIRSEQ2N_BUILD_FOR_NATIVE         : OFF
--   FAIRSEQ2N_INSTALL_STANDALONE       : ON
--   FAIRSEQ2N_PERFORM_LTO              : OFF
--   FAIRSEQ2N_RUN_CLANG_TIDY           : OFF
--   FAIRSEQ2N_SANITIZERS               : -
--   FAIRSEQ2N_TREAT_WARNINGS_AS_ERRORS : OFF
--   FAIRSEQ2N_USE_LIBTORCH             : OFF
--   FAIRSEQ2N_USE_CUDA                 : OFF
--   FAIRSEQ2N_BUILD_PYTHON_BINDINGS    : ON
--   FAIRSEQ2N_PYTHON_DEVEL             : ON
-- 
--   System Dependencies
--     libsndfile                       : 1.2.0
-- 
-- Configuring done (3.3s)
-- Generating done (0.3s)
-- Build files have been written to: /opt/app/build/fairseq2/fairseq2n/build

> cmake --build build

[2/2] Linking CXX executable tests/run-tests
FAILED: tests/run-tests 
: && /usr/bin/c++ -O2 -g -DNDEBUG -Wl,--as-needed -Wl,--build-id=sha1 -Wl,-z,noexecstack -Wl,-z,now -Wl,-z,relro -Wl,-z,defs tests/CMakeFiles/tests.dir/test_float.cc.o tests/CMakeFiles/tests.dir/test_memory.cc.o tests/CMakeFiles/tests.dir/test_span.cc.o tests/CMakeFiles/tests.dir/data/test_immutable_string.cc.o tests/CMakeFiles/tests.dir/data/test_tape.cc.o tests/CMakeFiles/tests.dir/data/detail/test_lru_cache.cc.o tests/CMakeFiles/tests.dir/utils/test_cast.cc.o -o tests/run-tests  -Wl,-rpath,/opt/app/build/fairseq2/fairseq2n/build/src/fairseq2n:/usr/local/lib/python3.11/site-packages/torch/lib  lib/libgtest_main.a  src/fairseq2n/libfairseq2n.so.0.2.0  lib/libgtest.a  /usr/local/lib/python3.11/site-packages/torch/lib/libtorch.so  /usr/local/lib/python3.11/site-packages/torch/lib/libtorch_cpu.so  /usr/local/lib/python3.11/site-packages/torch/lib/libc10.so && :
/usr/bin/ld: warning: libgfortran-b6d57c85.so.5.0.0, needed by /usr/local/lib/python3.11/site-packages/torch/lib/../../torch.libs/libopenblasp-r0-56e95da7.3.24.so, not found (try using -rpath or -rpath-link)
/usr/bin/ld: /usr/local/lib/python3.11/site-packages/torch/lib/../../torch.libs/libopenblasp-r0-56e95da7.3.24.so: undefined reference to `_gfortran_concat_string@GFORTRAN_8'
/usr/bin/ld: /usr/local/lib/python3.11/site-packages/torch/lib/../../torch.libs/libopenblasp-r0-56e95da7.3.24.so: undefined reference to `_gfortran_etime@GFORTRAN_8'
collect2: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.

compile fairseq2n error

Describe the bug:
A clear and concise description of what the bug is.
I follow your install readme and step by step to install.

install pytorch
conda install pytorch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 pytorch-cuda=11.7 -c pytorch -c nvidia -c https://mirrors.bfsu.edu.cn/anaconda/cloud/pytorch/linux-64/ -y
install libsndfile
conda install -c conda-forge libsndfile
install comiler environment in conda
conda install -c conda-forge compilers
compile fairseq2n
cmake -GNinja -DFAIRSEQ2N_INSTALL_STANDALONE=ON -DFAIRSEQ2N_USE_CUDA=ON -B build
However , the error is as follows:

-- Found SndFile: /home/maduo/miniconda3/envs/fsq2/lib/libsndfile.so (found suitable version "1.0.31", minimum required is "1.0.25")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Checking for oneTBB under the Python environment...
-- Found TBB: /home/maduo/miniconda3/envs/fsq2/lib/libtbb.so (found suitable version "2021.8.0", minimum required is "2021.8")
-- Found Torch: /home/maduo/miniconda3/envs/fsq2/lib/python3.11/site-packages/torch/lib/libtorch.so (found suitable version "2.0.1", minimum required is "1.12")
-- Found Python3: /home/maduo/miniconda3/envs/fsq2/bin/python3.11 (found version "3.11.5") found components: Interpreter Development.Module
-- Module support is disabled.
-- Version: 9.1.0
-- Build type: RelWithDebInfo
-- CXX_STANDARD: 11
-- Performing Test has_std_11_flag
-- Performing Test has_std_11_flag - Success
-- Performing Test has_std_0x_flag
-- Performing Test has_std_0x_flag - Success
-- Required features: cxx_variadic_templates
CMake Deprecation Warning at third-party/kaldi-native-fbank/CMakeLists.txt:24 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- CMAKE_BUILD_TYPE: RelWithDebInfo
-- BUILD_SHARED_LIBS: FALSE
-- KALDI_NATIVE_FBANK_BUILD_TESTS: OFF
-- KALDI_NATIVE_FBANK_BUILD_PYTHON: OFF
-- KALDI_NATIVE_FBANK_ENABLE_CHECK: OFF
-- KALDI_NATIVE_FBANK_ENABLE_CHECK: OFF
-- CMAKE_CXX_FLAGS:
-- CMAKE_INSTALL_PREFIX: /usr/local
-- Looking for C++ include cxxabi.h
-- Looking for C++ include cxxabi.h - found
-- Looking for C++ include execinfo.h
-- Looking for C++ include execinfo.h - found
-- Disable building Python
CMake Deprecation Warning at third-party/sentencepiece/CMakeLists.txt:15 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- VERSION: 0.1.97
-- Not Found TCMalloc: TCMALLOC_LIB-NOTFOUND
CMake Deprecation Warning at third-party/pybind11/CMakeLists.txt:8 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- pybind11 v2.10.0
-- Performing Test HAS_FLTO
-- Performing Test HAS_FLTO - Success
-- Found Python: /home/maduo/miniconda3/envs/fsq2/bin/python3.11 (found version "3.11.5") found components: Interpreter
CMake Error at /home/maduo/miniconda3/envs/fsq2/lib/python3.11/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeDetermineCompilerId.cmake:753 (message):
  Compiling the CUDA compiler identification source file
  "CMakeCUDACompilerId.cu" failed.

  Compiler: /usr/local/cuda-11.7/bin/nvcc

  Build flags:

  Id flags: --keep;--keep-dir;tmp -v



  The output was:

  1

  #$ _NVVM_BRANCH_=nvvm

  #$ _SPACE_=

  #$ _CUDART_=cudart

  #$ _HERE_=/usr/local/cuda-11.7/bin

  #$ _THERE_=/usr/local/cuda-11.7/bin

  #$ _TARGET_SIZE_=

  #$ _TARGET_DIR_=

  #$ _TARGET_DIR_=targets/x86_64-linux

  #$ TOP=/usr/local/cuda-11.7/bin/..

  #$ NVVMIR_LIBRARY_DIR=/usr/local/cuda-11.7/bin/../nvvm/libdevice

  #$
  LD_LIBRARY_PATH=/usr/local/cuda-11.7/bin/../lib:/usr/local/cuda-11.7/lib64::/workspace2/maduo/kaldi/src/base/:/workspace2/maduo/kaldi/src/fstext/:/workspace2/maduo/kaldi/tools/python:/workspace2/maduo/kaldi/src/bin:/workspace2/maduo/kaldi/src/chainbin:/workspace2/maduo/kaldi/src/featbin:/workspace2/maduo/kaldi/src/fgmmbin:/workspace2/maduo/kaldi/src/fstbin:/workspace2/maduo/kaldi/src/gmmbin:/workspace2/maduo/kaldi/src/ivectorbin:/workspace2/maduo/kaldi/src/kwsbin:/workspace2/maduo/kaldi/src/latbin:/workspace2/maduo/kaldi/src/lmbin:/workspace2/maduo/kaldi/src/nnet2bin:/workspace2/maduo/kaldi/src/nnet3bin:/workspace2/maduo/kaldi/src/nnetbin:/workspace2/maduo/kaldi/src/online2bin:/workspace2/maduo/kaldi/src/onlinebin:/workspace2/maduo/kaldi/src/rnnlmbin:/workspace2/maduo/kaldi/src/sgmm2bin:/workspace2/maduo/kaldi/src/sgmmbin:/workspace2/maduo/kaldi/src/tfrnnlmbin:/workspace2/maduo/kaldi/src/cudadecoderbin:/workspace2/maduo/kaldi/src/cudafeatbin:/workspace2/maduo/kaldi/tool/openfst/bin:/home/maduo/cmake-3.26.0/bin:/home/maduo/miniconda3/condabin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin


  #$
  PATH=/usr/local/cuda-11.7/bin/../nvvm/bin:/usr/local/cuda-11.7/bin:/home/maduo/miniconda3/envs/fsq2/bin:/workspace2/maduo/kaldi/tools/python:/workspace2/maduo/kaldi/src/bin:/workspace2/maduo/kaldi/src/chainbin:/workspace2/maduo/kaldi/src/featbin:/workspace2/maduo/kaldi/src/fgmmbin:/workspace2/maduo/kaldi/src/fstbin:/workspace2/maduo/kaldi/src/gmmbin:/workspace2/maduo/kaldi/src/ivectorbin:/workspace2/maduo/kaldi/src/kwsbin:/workspace2/maduo/kaldi/src/latbin:/workspace2/maduo/kaldi/src/lmbin:/workspace2/maduo/kaldi/src/nnet2bin:/workspace2/maduo/kaldi/src/nnet3bin:/workspace2/maduo/kaldi/src/nnetbin:/workspace2/maduo/kaldi/src/online2bin:/workspace2/maduo/kaldi/src/onlinebin:/workspace2/maduo/kaldi/src/rnnlmbin:/workspace2/maduo/kaldi/src/sgmm2bin:/workspace2/maduo/kaldi/src/sgmmbin:/workspace2/maduo/kaldi/src/tfrnnlmbin:/workspace2/maduo/kaldi/src/cudadecoderbin:/workspace2/maduo/kaldi/src/cudafeatbin:/workspace2/maduo/kaldi/tool/openfst/bin:/home/maduo/cmake-3.26.0/bin:/home/maduo/miniconda3/condabin:/usr/local/cuda-11.7/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin


  #$ INCLUDES="-I/usr/local/cuda-11.7/bin/../targets/x86_64-linux/include"

  #$ LIBRARIES=
  "-L/usr/local/cuda-11.7/bin/../targets/x86_64-linux/lib/stubs"
  "-L/usr/local/cuda-11.7/bin/../targets/x86_64-linux/lib"

  #$ CUDAFE_FLAGS=

  #$ PTXAS_FLAGS=

  #$ rm tmp/a_dlink.reg.c

  #$ gcc -D__CUDA_ARCH__=520 -D__CUDA_ARCH_LIST__=520 -E -x c++
  -DCUDA_DOUBLE_MATH_FUNCTIONS -D__CUDACC__ -D__NVCC__
  "-I/usr/local/cuda-11.7/bin/../targets/x86_64-linux/include"
  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=7
  -D__CUDACC_VER_BUILD__=64 -D__CUDA_API_VER_MAJOR__=11
  -D__CUDA_API_VER_MINOR__=7 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
  "cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
  "tmp/CMakeCUDACompilerId.cpp1.ii"

  #$ cicc --c++17 --gnu_version=110200 --display_error_number
  --orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
  "/workspace2/maduo/fairseq2/fairseq2n/build/CMakeFiles/3.27.7/CompilerIdCUDA/CMakeCUDACompilerId.cu"
  --allow_managed -arch compute_52 -m64 --no-version-ident -ftz=0 -prec_div=1
  -prec_sqrt=1 -fmad=1 --include_file_name "CMakeCUDACompilerId.fatbin.c"
  -tused --gen_module_id_file --module_id_file_name
  "tmp/CMakeCUDACompilerId.module_id" --gen_c_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.c" --stub_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.stub.c" --gen_device_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.gpu" "tmp/CMakeCUDACompilerId.cpp1.ii" -o
  "tmp/CMakeCUDACompilerId.ptx"

  #$ ptxas -arch=sm_52 -m64 "tmp/CMakeCUDACompilerId.ptx" -o
  "tmp/CMakeCUDACompilerId.sm_52.cubin"

  #$ fatbinary --create="tmp/CMakeCUDACompilerId.fatbin" -64
  --cicc-cmdline="-ftz=0 -prec_div=1 -prec_sqrt=1 -fmad=1 "
  "--image3=kind=elf,sm=52,file=tmp/CMakeCUDACompilerId.sm_52.cubin"
  "--image3=kind=ptx,sm=52,file=tmp/CMakeCUDACompilerId.ptx"
  --embedded-fatbin="tmp/CMakeCUDACompilerId.fatbin.c"

  #$ gcc -D__CUDA_ARCH_LIST__=520 -E -x c++ -D__CUDACC__ -D__NVCC__
  "-I/usr/local/cuda-11.7/bin/../targets/x86_64-linux/include"
  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=7
  -D__CUDACC_VER_BUILD__=64 -D__CUDA_API_VER_MAJOR__=11
  -D__CUDA_API_VER_MINOR__=7 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -include
  "cuda_runtime.h" -m64 "CMakeCUDACompilerId.cu" -o
  "tmp/CMakeCUDACompilerId.cpp4.ii"

  #$ cudafe++ --c++17 --gnu_version=110200 --display_error_number
  --orig_src_file_name "CMakeCUDACompilerId.cu" --orig_src_path_name
  "/workspace2/maduo/fairseq2/fairseq2n/build/CMakeFiles/3.27.7/CompilerIdCUDA/CMakeCUDACompilerId.cu"
  --allow_managed --m64 --parse_templates --gen_c_file_name
  "tmp/CMakeCUDACompilerId.cudafe1.cpp" --stub_file_name
  "CMakeCUDACompilerId.cudafe1.stub.c" --module_id_file_name
  "tmp/CMakeCUDACompilerId.module_id" "tmp/CMakeCUDACompilerId.cpp4.ii"

  #$ gcc -D__CUDA_ARCH__=520 -D__CUDA_ARCH_LIST__=520 -c -x c++
  -DCUDA_DOUBLE_MATH_FUNCTIONS
  "-I/usr/local/cuda-11.7/bin/../targets/x86_64-linux/include" -m64
  "tmp/CMakeCUDACompilerId.cudafe1.cpp" -o "tmp/CMakeCUDACompilerId.o"

  #$ nvlink -m64 --arch=sm_52 --register-link-binaries="tmp/a_dlink.reg.c"
  "-L/usr/local/cuda-11.7/bin/../targets/x86_64-linux/lib/stubs"
  "-L/usr/local/cuda-11.7/bin/../targets/x86_64-linux/lib" -cpu-arch=X86_64
  "tmp/CMakeCUDACompilerId.o" -lcudadevrt -o "tmp/a_dlink.sm_52.cubin"
  --host-ccbin "gcc"

  #$ fatbinary --create="tmp/a_dlink.fatbin" -64 --cicc-cmdline="-ftz=0
  -prec_div=1 -prec_sqrt=1 -fmad=1 " -link
  "--image3=kind=elf,sm=52,file=tmp/a_dlink.sm_52.cubin"
  --embedded-fatbin="tmp/a_dlink.fatbin.c"

  #$ gcc -D__CUDA_ARCH_LIST__=520 -c -x c++
  -DFATBINFILE="\"tmp/a_dlink.fatbin.c\""
  -DREGISTERLINKBINARYFILE="\"tmp/a_dlink.reg.c\"" -I.
  -D__NV_EXTRA_INITIALIZATION= -D__NV_EXTRA_FINALIZATION=
  -D__CUDA_INCLUDE_COMPILER_INTERNAL_HEADERS__
  "-I/usr/local/cuda-11.7/bin/../targets/x86_64-linux/include"
  -D__CUDACC_VER_MAJOR__=11 -D__CUDACC_VER_MINOR__=7
  -D__CUDACC_VER_BUILD__=64 -D__CUDA_API_VER_MAJOR__=11
  -D__CUDA_API_VER_MINOR__=7 -D__NVCC_DIAG_PRAGMA_SUPPORT__=1 -m64
  "/usr/local/cuda-11.7/bin/crt/link.stub" -o "tmp/a_dlink.o"

  #$ g++ -D__CUDA_ARCH_LIST__=520 -m64 -Wl,--start-group "tmp/a_dlink.o"
  "tmp/CMakeCUDACompilerId.o"
  "-L/usr/local/cuda-11.7/bin/../targets/x86_64-linux/lib/stubs"
  "-L/usr/local/cuda-11.7/bin/../targets/x86_64-linux/lib" -lcudadevrt
  -lcudart_static -lrt -lpthread -ldl -Wl,--end-group -o "a.out"


  /home/maduo/miniconda3/envs/fsq2/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld:
  /home/maduo/miniconda3/envs/fsq2/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib64/libstdc++.so:
  undefined reference to `memcpy@GLIBC_2.14'


  /home/maduo/miniconda3/envs/fsq2/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld:
  /home/maduo/miniconda3/envs/fsq2/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib64/libstdc++.so:
  undefined reference to `aligned_alloc@GLIBC_2.16'


  /home/maduo/miniconda3/envs/fsq2/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/bin/ld:
  /home/maduo/miniconda3/envs/fsq2/bin/../lib/gcc/x86_64-conda-linux-gnu/11.2.0/../../../../x86_64-conda-linux-gnu/lib/../lib64/libstdc++.so:
  undefined reference to `clock_gettime@GLIBC_2.17'

  collect2: error: ld returned 1 exit status

  # --error 0x1 --





Call Stack (most recent call first):
  /home/maduo/miniconda3/envs/fsq2/lib/python3.11/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeDetermineCompilerId.cmake:8 (CMAKE_DETERMINE_COMPILER_ID_BUILD)
  /home/maduo/miniconda3/envs/fsq2/lib/python3.11/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeDetermineCompilerId.cmake:53 (__determine_compiler_id_test)
  /home/maduo/miniconda3/envs/fsq2/lib/python3.11/site-packages/cmake/data/share/cmake-3.27/Modules/CMakeDetermineCUDACompiler.cmake:307 (CMAKE_DETERMINE_COMPILER_ID)
  CMakeLists.txt:220 (enable_language)


-- Configuring incomplete, errors occurred!

Environment:
pytorch 2.0.1, cuda 11.7
python3.11,
operating system: 20.04.2 LTS (GNU/Linux 5.8.0-43-generic x86_64)

Support feature extraction in w2v2

Is your feature request related to a problem? Please describe:
After pretraining Wav2Vec2, users should be able to fine-tune by using it as a feature extractor. Right now there's no method for doing that (I think?).

Describe the solution you would like:

I think there should be a new method extract_features for the model that calls self.run_frontend and then self.encoder.
Then forward will call this new method and have an additional new argument return_features defaulting to False. If this argument is True, forward will return the output of the transformer layers (and optionally convolutional feature extractor) in addition to the Wav2Vec2 output object that is the output of self.quantize_and_contrast.

Describe the alternatives you have considered:
Don't modify forward. Disadvantage: harder to do feature extraction and fine-tuning in a DDP context since DDP wraps model.forward but not other methods (I think?).

Additional Context:
Add any other context about the feature request here.

fairseq2n compiling error with kaldi_fbank.cc (77:35 invalid initialization of reference of type ‘const std::vector<float>&’)

Describe the bug:
A clear and concise description of what the bug is.
I want to compile and install fairseq2 from source code. I know fairseq2 must use kaldi-native-fbank (https://github.com/cbalioglu/kaldi-native-fbank) to process audio, and i followed the link https://facebookresearch.github.io/fairseq2/stable/installation/from_source .

Describe how to reproduce:
Steps to reproduce the behavior. Ideally attach a minimal code sample to reproduce the described issue.
After using cmake -GNinja -DFAIRSEQ2N_USE_CUDA=ON -B build, I found a build directory successfully generated.
However, when i used "cmake --build build" command, the error occurs as:
[1/7] Building CXX object src/fairseq2n/CMakeFiles/fairseq2n.dir/data/audio/detail/kaldi_fbank.cc.o
FAILED: src/fairseq2n/CMakeFiles/fairseq2n.dir/data/audio/detail/kaldi_fbank.cc.o
/usr/bin/c++ -DFAIRSEQ2N_USE_CUDA -DFAIRSEQ2N_USE_TBB -D_FORTIFY_SOURCE=2 -D_GLIBCXX_USE_CXX11_ABI=0 -Dfairseq2n_EXPORTS -I/home/huangwei/fairseq2/fairseq2n/src -I/home/huangwei/fairseq2/fairseq2n/build/src -isystem /home/huangwei/fairseq2/fairseq2n/third-party/fmt/include -isystem /home/huangwei/fairseq2/fairseq2n/third-party/kaldi-native-fbank -isystem /home/huangwei/fairseq2/fairseq2n/third-party/zip/src -isystem /home/huangwei/fairseq2/fairseq2n/third-party/natsort -isystem /home/huangwei/fairseq2/fairseq2n/third-party -isystem /home/huangwei/fairseq2/fairseq2n/third-party/sentencepiece/third_party/protobuf-lite -isystem /home/huangwei/.local/lib/python3.8/site-packages/torch/include -isystem /home/huangwei/.local/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /usr/local/cuda-11.7/include -O2 -g -DNDEBUG -std=c++17 -fPIC -fvisibility=hidden -Wall -Wcast-align -Wconversion -Wdouble-promotion -Wextra -Wfloat-equal -Wformat=2 -Winit-self -Wlogical-op -Wno-unknown-pragmas -Wpointer-arith -Wshadow=compatible-local -Wsign-conversion -Wswitch -Wunused -Wnon-virtual-dtor -Woverloaded-virtual -Wuseless-cast -fasynchronous-unwind-tables -fstack-protector-strong -pthread -MD -MT src/fairseq2n/CMakeFiles/fairseq2n.dir/data/audio/detail/kaldi_fbank.cc.o -MF src/fairseq2n/CMakeFiles/fairseq2n.dir/data/audio/detail/kaldi_fbank.cc.o.d -o src/fairseq2n/CMakeFiles/fairseq2n.dir/data/audio/detail/kaldi_fbank.cc.o -c /home/huangwei/fairseq2/fairseq2n/src/fairseq2n/data/audio/detail/kaldi_fbank.cc
/home/huangwei/fairseq2/fairseq2n/src/fairseq2n/data/audio/detail/kaldi_fbank.cc: In lambda function:
/home/huangwei/fairseq2/fairseq2n/src/fairseq2n/data/audio/detail/kaldi_fbank.cc:77:35: error: invalid initialization of reference of type ‘const std::vector&’ from expression of type ‘fairseq2n::span::pointer’ {aka ‘const float*’}
77 | waveform_data.data(),
| ~~~~~~~~~~~~~~~~~~^~
In file included from /home/huangwei/fairseq2/fairseq2n/third-party/kaldi-native-fbank/kaldi-native-fbank/csrc/feature-fbank.h:28,
from /home/huangwei/fairseq2/fairseq2n/src/fairseq2n/data/audio/detail/kaldi_fbank.h:10,
from /home/huangwei/fairseq2/fairseq2n/src/fairseq2n/data/audio/detail/kaldi_fbank.cc:7:
/home/huangwei/fairseq2/fairseq2n/third-party/kaldi-native-fbank/kaldi-native-fbank/csrc/feature-window.h:140:69: note: in passing argument 2 of ‘void knf::ExtractWindow(int64_t, const std::vector&, int32_t, const knf::FrameExtractionOptions&, const knf::FeatureWindowFunction&, std::vector, float)’
140 | void ExtractWindow(int64_t sample_offset, const std::vector &wave,
| ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~
ninja: build stopped: subcommand failed.

Describe the expected behavior:
A clear and concise description of what you expected to happen.
I want to know how to compile fairseq2n without such an error.

Environment:
At the very least, specify the versions of fairseq2, PyTorch, Python, and CUDA along with your operating system and, if relevant, GPU model.
My environment is ubuntu 20.04, with gcc==11.4
Additional Context:
Add any other context about the bug here.

pip install fairseq2 - doesn't work

pip install fairseq2
Collecting fairseq2
  Obtaining dependency information for fairseq2 from https://files.pythonhosted.org/packages/cd/27/46c14e28e8cb0aa602660ce64d4547a37f460d382e4fcf94f2a53d47e5b0/fairseq2-0.1.0-py3-none-any.whl.metadata
  Using cached fairseq2-0.1.0-py3-none-any.whl.metadata (1.2 kB)
INFO: pip is looking at multiple versions of fairseq2 to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement fairseq2n==0.1.0 (from fairseq2) (from versions: none)
ERROR: No matching distribution found for fairseq2n==0.1.0

Use fairseq2 without a gpu?

Hi folks! Is there a way to run fairseq2 on a machine without GPU? I'm getting the following error trying to use it on my cpu only machine:

>>> from fairseq2.data import Collater
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/czhang/Documents/work/venv/lib/python3.8/site-packages/fairseq2/data/__init__.py", line 7, in <module>
    from fairseq2.data.cstring import CString as CString
  File "/home/czhang/Documents/work/venv/lib/python3.8/site-packages/fairseq2/data/cstring.py", line 58, in <module>
    from fairseq2n.bindings.data.string import CString as CString
ImportError: libcudart.so.11.0: cannot open shared object file: No such file or directory

SentencePieceDecoder to have a helper function which converts token indices to a list of string tokens and not just complete sentences

Is your feature request related to a problem? Please describe:
Currently we have to loop through each batch, and each token in the batch, and call the SPM model's index_to_token method which is very efficient.

The SentencePieceDecoder converts token indices to complete sentences directly. (https://github.com/facebookresearch/fairseq2/blob/main/src/fairseq2/data/text/sentencepiece.py#L85-L91)

       @finaloverride
        def __call__(self, token_indices: Tensor) -> List[StringLike]:
            ...

I need a method which takes in token indices, and just converts it to a list of string tokens. Currently we are having to call index_to_token on each token index.

Describe the solution you would like:
Instead of joining all the string tokens into a single sentence, it would be nice to have an option to just return the list of string tokens as well, without the concatenation into a single string sentence.

       @finaloverride
        def get_tokens(self, token_indices: Tensor) -> List[List[StringLike]]:
            ...

Describe the alternatives you have considered:
A clear and concise description of any alternative solutions or features you have considered.

Additional Context:
We want to deprecate this method:

https://github.com/fairinternal/seamless_communication/blob/main/src/seamless_communication/models/unity/nar_decoder_frontend.py#L134-L145

with the get_tokens method built in the SentencePieceDecoder.

ROCM HIP / Vulkan support for GPU acceleration

Is your feature request related to a problem? Please describe:
Right now, it is not possible to use this library on AMD hardware because it is ABI-linked to either the CPU or CUDA version of Torch. It doesn't appear to be possible to build it with just replacing the torch version with rocm-enabled torch unlike what usually works for other ML projects.

Describe the solution you would like:
Be able to have rocm enabled build of this library available or at least implement HIP/Vulkan support so that it can be built from source.

Describe the alternatives you have considered:
I have none at this moment besides running on the CPU.

Additional Context:
ROCM 5.6.x / Radeon 7900 XTX working with most other tools relying on Torch.

Should `Wav2Vec2Loss` obey tensor contracts?

Wrappers like huggingface accelerate and lightning fabric wrap loss.backward to support DDP and other distributed training, where they do things like gradient accumulation (e.g. https://github.com/huggingface/accelerate/blob/main/src/accelerate/accelerator.py#L1956) and changes to float precision that expect the loss to be a tensor, not a dataclass. I think the underlying issue is that we usually expect loss to be a tensor and obey those contracts. I wonder if it would be better as a tensordict (https://github.com/pytorch/tensordict) or similar?

Support for MoE architecture

Do you have a plan to support the previous mos branch, like fairseq-moe on this library?

pytorch version of WaveformToFbankConverter

I have a question and would greatly appreciate your help! For the audio feature processor WaveformToFbankConverter, is there a PyTorch implementation available for understanding? Thank you very much!

OSError: libsndfile is not found!. Use your system package manager to install it (e.g. `apt install libsndfile1`

Hi, I have a bug when try to run https://github.com/facebookresearch/seamless_communication repo:
$ m4t_predict

Traceback (most recent call last):
  File "/home/myuser/.local/bin/m4t_predict", line 5, in <module>
    from m4t_scripts.predict.predict import main
  File "/home/myuser/.local/lib/python3.8/site-packages/m4t_scripts/predict/predict.py", line 10, in <module>
    from seamless_communication.models.inference import Translator
  File "/home/myuser/.local/lib/python3.8/site-packages/seamless_communication/models/inference/__init__.py", line 6, in <module>
    from seamless_communication.models.inference.translator import Translator as Translator
  File "/home/myuser/.local/lib/python3.8/site-packages/seamless_communication/models/inference/translator.py", line 12, in <module>
    from fairseq2.data import Collater
  File "/home/myuser/.local/lib/python3.8/site-packages/fairseq2/data/__init__.py", line 7, in <module>
    from fairseq2.data.cstring import CString as CString
  File "/home/myuser/.local/lib/python3.8/site-packages/fairseq2/data/cstring.py", line 58, in <module>
    from fairseq2n.bindings.data.string import CString as CString
  File "/home/myuser/.local/lib/python3.8/site-packages/fairseq2n/__init__.py", line 91, in <module>
    _load_sndfile()
  File "/home/myuser/.local/lib/python3.8/site-packages/fairseq2n/__init__.py", line 81, in _load_sndfile
    raise OSError(
OSError: libsndfile is not found!. Use your system package manager to install it (e.g. `apt install libsndfile1`).

SeamlessM4T installed by instruction in README.md by pip install .

Installation verification of libsndfile1

$ dpkg -L libsndfile1
/.
/usr
/usr/lib
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/libsndfile.so.1.0.28
/usr/share
/usr/share/doc
/usr/share/doc/libsndfile1
/usr/share/doc/libsndfile1/changelog.Debian.gz
/usr/share/doc/libsndfile1/copyright
/usr/lib/x86_64-linux-gnu/libsndfile.so.1

System info:
Linux DD8SXD3 5.15.0-76-generic #83~20.04.1-Ubuntu SMP Wed Jun 21 20:23:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
memory 31GiB System memory
processor Intel(R) Core(TM) i7-10850H CPU @ 2.70GHz
bridge 10th Gen Core Processor Host Bridge/DRAM Registers
display UHD Graphics

seamless_communication require fairseq2n == 0.2.0

I try to install seamless_communcation on a man intel. I've followed all the instructions to install fairseg2 but when I try to install seamless_communcation I receive the Error:
ERROR: Could not find a version that satisfies the requirement fairseq2n==0.2.0 (from fairseq2) (from versions: none)
ERROR: No matching distribution found for fairseq2n==0.2.0

doing pip show fairseq2n the result is:
Name: fairseq2n
Version: 0.3.0.dev0
Summary: FAIR Sequence Modeling Toolkit (Native)
Home-page: https://github.com/facebookresearch/fairseq2
Author: Fundamental AI Research (FAIR) at Meta
Author-email:
License: MIT
Location: /Users/me/python/AI/SeamlessM4T/venv/lib/python3.9/site-packages
Requires: tbb, torch
Required-by: fairseq2

How can I install version 0.2.0?
thank you very much

Missing documentation on how to train a model

Is there any documentation or examples that I can refer to train a transformer model from scratch using fairseq2? The examples folder in the repository seems empty.

Tests in `test_sample.py` failing on `main` branch

I installed fairseq2 in edit mode following the instructions here. I did a quick pytest to test the installation to see following these errors:

=================================================================================================================== short test summary info ===================================================================================================================
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected_with_different_lenghts - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected_with_default_weights - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected_with_low_prob - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected_with_single_data_pipeline - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected_with_empty_data_pipelines - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_works_as_expected_with_manual_seed - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_raises_invalid_argument_if_negative_weights - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_raises_invalid_argument_if_weights_wrong_size - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_raises_invalid_argument_if_weights_sum_to_zero - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_raises_error_if_no_data_pipeline - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_op_raises_error_if_one_of_the_pipelines_is_broken - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
FAILED tests/unit/data/data_pipeline/test_sample.py::TestSampleOp::test_record_reload_position_works_as_expected - AttributeError: type object 'fairseq2n.bindings.data.data_pipeline.DataPipeline' has no attribute 'sample'
========================================================================================================== 13 failed, 377 passed, 1 skipped in 1.46s ==================================================================================================

I took a look at the latest PR (#20), and the CI seems to be skip them:

Are these failures expected? And, is there a way a ignore them locally?

Allow setting the number of threads in TBB's task_arena

We should expose a set_num_threads() function that the user can call before using other APIs of fairseq2 to adjust the number of threads we are allowed to use from the process thread pool.

`drop_remainder` in `DataPipeline.shard`

Is your feature request related to a problem? Please describe:
I'm using fairseq2 data pipeline for distributed inference. The current behavior of DataPipeline.shard is to discard remaining samples since it only returns the sample after iterating through world_size samples.

Describe the solution you would like:
An option drop_remainder which is similar to the one in DataPipeline.bucket.

Describe the alternatives you have considered:
I was thinking about maybe implement a custom sharding pipeline function in python but couldn't find a way to do that.

Fairseq2 AdamW GPU memory consumption

Attempts to switch M4T model training/finetuning recipes to fairseq2 AdamW work only with a significant reduction of batch size

Perf drop with OMP_NUM_THREAD/MKL_NUM_THREADS

Thanks to @mavlyutovr's investigation we found out that the throughput of our TBB-based data pipeline API can drop significantly if OMP_NUM_THREADS and/or MKL_NUM_THREADS are not bound to a low thread count. We should investigate if there is anyway for us to mitigate this issue on our end. If not, we should document this behavior and advise users to be cautious about it.

not compatible with torch version 2.1.0

Hi, due to my cuda version, I need to use torch 2.1.0 version. In the document the supported torch version is given as 2.0.1, but is there a way to use it with torch version 2.1.0?

Cannot build with GCC 11.4.1: error: ‘fmt::v10::detail::type_is_unformattable_for<const fairseq2n::immutable_string, char> _’ has incomplete type

Describe the bug:

FAILED: python/src/fairseq2n/bindings/CMakeFiles/py_bindings.dir/data/string.cc.o 
/usr/bin/c++ -D_FORTIFY_SOURCE=2 -D_GLIBCXX_USE_CXX11_ABI=1 -Dpy_bindings_EXPORTS -I/home/ezyang/local/d/fairseq2/fairseq2n/python/src -I/home/ezyang/local/d/fairseq2/fairseq2n/src -I/home/ezyang/local/d/fairseq2/fairseq2n/build/src -isystem /home/ezyang/local/d/pytorch-env/include/python3.10 -isystem /home/ezyang/local/d/fairseq2/fairseq2n/third-party/pybind11/include -isystem /home/ezyang/local/d/pytorch/torch/include -isystem /home/ezyang/local/d/pytorch/torch/include/torch/csrc/api/include -isystem /home/ezyang/local/d/fairseq2/fairseq2n/third-party/fmt/include -isystem /home/ezyang/local/d/fairseq2/fairseq2n/third-party/zip/src -O2 -g -DNDEBUG -std=c++17 -fPIC -fvisibility=hidden -Wall -Wcast-align -Wconversion -Wdouble-promotion -Wextra -Wfloat-equal -Wformat=2 -Winit-self -Wlogical-op -Wno-unknown-pragmas -Wpointer-arith -Wshadow=compatible-local -Wsign-conversion -Wswitch -Wunused -Wnon-virtual-dtor -Woverloaded-virtual -Wuseless-cast -fasynchronous-unwind-tables -fstack-protector-strong -MD -MT python/src/fairseq2n/bindings/CMakeFiles/py_bindings.dir/data/string.cc.o -MF python/src/fairseq2n/bindings/CMakeFiles/py_bindings.dir/data/string.cc.o.d -o python/src/fairseq2n/bindings/CMakeFiles/py_bindings.dir/data/string.cc.o -c /home/ezyang/local/d/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc
In file included from /home/ezyang/local/d/fairseq2/fairseq2n/src/fairseq2n/fmt.h:23,
                 from /home/ezyang/local/d/fairseq2/fairseq2n/src/fairseq2n/data/data.h:24,
                 from /home/ezyang/local/d/fairseq2/fairseq2n/python/src/fairseq2n/bindings/type_casters/data.h:11,
                 from /home/ezyang/local/d/fairseq2/fairseq2n/python/src/fairseq2n/bindings/module.h:14,
                 from /home/ezyang/local/d/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc:7:
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h: In instantiation of ‘constexpr fmt::v10::detail::value<Context> fmt::v10::detail::make_arg(T&) [with bool PACKED = true; Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; T = const fairseq2n::immutable_string; typename std::enable_if<PACKED, int>::type <anonymous> = 0]’:
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h:1808:51:   required from ‘constexpr fmt::v10::format_arg_store<Context, Args>::format_arg_store(T& ...) [with T = {const fairseq2n::immutable_string}; Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; Args = {fairseq2n::immutable_string}]’
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h:1826:18:   required from ‘constexpr fmt::v10::format_arg_store<Context, typename std::remove_cv<typename std::remove_reference<T>::type>::type ...> fmt::v10::make_format_args(T& ...) [with Context = fmt::v10::basic_format_context<fmt::v10::appender, char>; T = {const fairseq2n::immutable_string}]’
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h:2788:44:   required from ‘std::string fmt::v10::format(fmt::v10::format_string<T ...>, T&& ...) [with T = {const fairseq2n::immutable_string&}; std::string = std::__cxx11::basic_string<char>; fmt::v10::format_string<T ...> = fmt::v10::basic_format_string<char, const fairseq2n::immutable_string&>]’
/home/ezyang/local/d/fairseq2/fairseq2n/python/src/fairseq2n/bindings/data/string.cc:68:35:   required from here
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h:1576:63: error: ‘fmt::v10::detail::type_is_unformattable_for<const fairseq2n::immutable_string, char> _’ has incomplete type
 1576 |     type_is_unformattable_for<T, typename Context::char_type> _;
      |                                                               ^
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h:1580:7: error: static assertion failed: Cannot format an argument. To make type T formattable provide a formatter<T> specialization: https://fmt.dev/latest/api.html#udt
 1580 |       formattable,
      |       ^~~~~~~~~~~
/home/ezyang/local/d/pytorch/torch/include/fmt/core.h:1580:7: note: ‘formattable’ evaluates to false
[132/142] Building CXX object python/src/fairseq2n/bindings/CMakeFiles/py_bindings.dir/data/data_pipeline.cc.o

Describe how to reproduce:
On a conda environment on a Meta devgpu, run CONDA_BUILD_SYSROOT=1 cmake -GNinja -DFAIRSEQ2N_INSTALL_STANDALONE=ON -DFAIRSEQ2N_USE_CUDA=ON -B build and then cmake --build build

Describe the expected behavior:
It works

Environment:
fairseq2 bf976b5
c++ (GCC) 11.4.1 20230605 (Red Hat 11.4.1-2)

Why only one previous word is used as input when predict the current word

I have a question and would greatly appreciate your help! In the UnitY paper, the translation process is conducted in an autoregressive manner, where each prediction of the next word utilizes all previously predicted words. This is illustrated here:

However, in the actual implementation, why is only the previous word used as input each time? This is demonstrated here: code

# fairseq2/generation/beam_search.py
 def _step(self) -> bool:
        # Generate the next step output.
        model_output = self._decode(self.seqs[:, self.step_nr - 1 : self.step_nr])  # here 

        self.state_bag.increment_step_nr()

        logits = model_output.logits

not compatible with torch version 2.1.0

Hi, due to my cuda version, I need to use torch 2.1.0 version. In the document the supported torch version is given as 2.0.1, but is there a way to use it with torch version 2.1.0?

Installing Fairseq2 fails on M1 Pro

@cbalioglu created v0.1.1, as described in issue #1 . I have been trying to install it on a Macbook (M1 Pro) in a fresh environment, but I still get an error. Since that issue is closed, I opened a new one. I have been trying to install it for days without any success. I need to use Seamless models from Meta, which require Fairseq2.

Any suggestions?

$ pip install fairseq2==0.1.1
Collecting fairseq2==0.1.1
  Using cached fairseq2-0.1.1-py3-none-any.whl.metadata (1.2 kB)
INFO: pip is looking at multiple versions of fairseq2 to determine which version is compatible with other requirements. This could take a while.
ERROR: Could not find a version that satisfies the requirement fairseq2n==0.1.1 (from fairseq2) (from versions: none)
ERROR: No matching distribution found for fairseq2n==0.1.1

And:

$ pip install fairseq2
Collecting fairseq2
  Using cached fairseq2-0.2.0-py3-none-any.whl.metadata (1.2 kB)
INFO: pip is looking at multiple versions of fairseq2 to determine which version is compatible with other requirements. This could take a while.
  Using cached fairseq2-0.1.1-py3-none-any.whl.metadata (1.2 kB)
  Using cached fairseq2-0.1.0-py3-none-any.whl.metadata (1.2 kB)
ERROR: Cannot install fairseq2==0.1.0, fairseq2==0.1.1 and fairseq2==0.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    fairseq2 0.2.0 depends on fairseq2n==0.2.0
    fairseq2 0.1.1 depends on fairseq2n==0.1.1
    fairseq2 0.1.0 depends on fairseq2n==0.1.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

	@torch.inference_mode()
	def _do_generate(
	self, source_seqs: Tensor, source_seq_lens: Optional[Tensor]
	) -> "SequenceToTextOutput":
	"""A subclass should call this function for the actual text generation."""
	encoder_output, encoder_padding_mask = self.model.encode(
	source_seqs, source_seq_lens
	)

	gen_output = self.generator(
	encoder_output, encoder_padding_mask, source_seq_len=source_seqs.size(1)
	)

	sentences = [self.token_decoder(b[0].seq)[0] for b in gen_output.results]

	return SequenceToTextOutput(
	sentences, gen_output, encoder_output, encoder_padding_mask
	)

facebookresearch / fairseq2 Goto Github PK

fairseq2's People

Contributors

Stargazers

Watchers

Forkers

fairseq2's Issues

Recommend Projects

Recommend Topics

Recommend Org