raivnlab / mrl Goto Github PK

Hi @adityakusupati ,
Great work! After the reading the paper, it wasn't quite clear how you pretrained BERT with MRL.

Could please share the code/scripts for BERT MRL?

Thanks!

Thanks for the implementation. Can this method be used in the training of self-supervised vision transformer models like DINO, iBOT, ... etc. Note, these methods involve multiple global and local crops. Along with regularization mechanisms like centering and sharpening.

Hey there,

This paper and work is awesome. I think it would be great to publish it within a Lightning Studio: https://lightning.ai/.

They are fully persistent cloud IDE that anyone can duplicate. When they do, they get dependencies, data, model checkpoints, API endpoints, servers, etc...

So everything available and ready to go in a single click.

Hi Team! Firstly, thank you for the nice paper and approach. Secondly, while browsing through the paper of MRL, it is mentioned that the BERT-Base model was trained for MLM task on English Wikipedia and BookCorpus in Appendix C and values are provided in Table 7. However, I was not able to locate the script that trains the BERT-based model in the repository to reproduce the results. It will be really helpful if you can locate the script. Thanks in advance.

First, thank you for sharing this project with us!

Could you please add an explicit LICENSE file to the repo so that it's clear
under what terms the content is provided, and under what terms user
contributions are licensed?

Per GitHub docs on licensing:

[...] without a license, the default copyright laws apply, meaning that you
retain all rights to your source code and no one may reproduce, distribute,
or create derivative works from your work. If you're creating an open source
project, we strongly encourage you to include an open source license. The
Open Source Guide provides additional guidance on choosing the correct
license for your project.

Thanks!

After installing the packages present in the requirements file, FFCV module is missing to run the formatting error, when I try to install using the mentioned command in ffcv repo, it gives the following error:

Collecting ffcv
Using cached ffcv-1.0.2.tar.gz (2.6 MB)
Preparing metadata (setup.py) ... error
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
Traceback (most recent call last):
File "", line 2, in
File "", line 34, in
File "/tmp/pip-install-ijqwteab/ffcv_59e99b0ddf5845ea8a74d649f62207d9/setup.py", line 96, in
extension_kwargs = pkgconfig('libturbojpeg', extension_kwargs)
File "/tmp/pip-install-ijqwteab/ffcv_59e99b0ddf5845ea8a74d649f62207d9/setup.py", line 74, in pkgconfig
raise RuntimeError(f"Could not find required package: {package}.")
RuntimeError: Could not find required package: libturbojpeg.
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.

I tried to install ffcv using the below mentioned command
conda create -y -n ffcv python=3.9 cupy pkg-config libjpeg-turbo opencv pytorch torchvision cudatoolkit=11.3 numba -c pytorch -c conda-forge
conda activate ffcv
pip install ffcv