xplip / pixel Goto Github PK

It seems that you are using 2D sincos embedding for PIXEL here. This essentially implies that when the patches are fed to the transformer, they are actually reordered as a square image, even though they are initially rendered horizontally. Is my understanding correct here?

Hi,

I succesfully configured the repository and everything works fine, except for dependency parsing. When trying to either run the sanity check or train a new model, I get this error:
Traceback (most recent call last): File "/home/lys/pixel_mainlp/pixel/scripts/training/run_ud.py", line 466, in <module> main() File "/home/lys/pixel_mainlp/pixel/scripts/training/run_ud.py", line 423, in main outputs = trainer.predict(test_dataset=eval_dataset, metric_key_prefix="eval") File "/home/lys/anaconda3/envs/pixel-env/lib/python3.9/site-packages/transformers/trainer.py", line 2331, in predict output = eval_loop( File "/home/lys/pixel_mainlp/pixel/src/pixel/trainer.py", line 179, in evaluation_loop mask = arc_labels.ne(model.config.pad_token_id) File "/home/lys/anaconda3/envs/pixel-env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'DataParallel' object has no attribute 'config'

I am using PyTorch 2.0.1 with CUDA 11.7. I managed to install it with the cuda 11.3 version suggested on the repository, and it works, but it installs the CPU PyTorch version instead of the GPU one. I do not know how to fix both things at the same time. Thanks.

Dear Author,

I tried to pre-train the model on book-corpus data only, whose training config is exactly the same as yours.
But I found that the training loss is around 0.76 after 3 epochs.
It seems that the model could not converge.
And the reconstruction images are bad.
Could you tell me what's the problem?

Thanks,
Chawdoe

Thanks for your wonderful work!
I am very interested in your work and try to extend the ideas to other tasks.
I wonder when the rendering instructions will be available?

When using the modified datasets library, I am not able to train nor run the sanity checks on squad and glue as it is not able to download these datasets. This only get solved when updating the library to the latest datasets version. Any solutions for this? Thanks a lot!

  File "scripts/training/run_pretraining.py", line 465, in preprocess_images
    examples["pixel_values"] = [transforms(image) for image in examples[image_column_name]]  # bytes, path
  File "scripts/training/run_pretraining.py", line 465, in <listcomp>
    examples["pixel_values"] = [transforms(image) for image in examples[image_column_name]]  # bytes, path
  File "/usr/local/lib/python3.8/dist-packages/torchvision/transforms/transforms.py", line 61, in __call__
    img = t(img)
  File "/usr/local/lib/python3.8/dist-packages/torchvision/transforms/transforms.py", line 437, in __call__
    return self.lambd(img)
  File "/cpfs/shared/research/public-data/cv/driving/pixel/pixel-main/src/pixel/pixel_utils/transforms.py", line 211, in <lambda>
    transforms = [Lambda(lambda img: img.convert("RGB") if img.mode != "RGB" else img)]
AttributeError: 'dict' object has no attribute 'mode'

I used the pre-rendered data which i manually downloaded from the website.
And I found that the data is a dict which contains keys {'path', 'bytes'}, which is not RGB image.
How could I do data transform?

I am using the following code to extract sentence embedding.

pixel didn't perform well on this project.

Hi @xplip , thanks for sharing your code. I'm currently running the pre-training scripts, and have met the issues with finding validation dataset.
I got errors like "Repository Not Found for url: https://huggingface.co/api/datasets/plip/wiki_dev', which probably suggests that the "validation_dataset_name": "plip/wiki_dev" which was specified in their config file specified an evaluation dataset that is not on the huggingface.

The detailed error msg is as below:

Traceback (most recent call last):
File "/exports/eddie/scratch/s2522559/pixel_project/pixel/modify_running_script.py", line 145, in
main()
File "/exports/eddie/scratch/s2522559/pixel_project/pixel/modify_running_script.py", line 142, in main
trainer()
File "/exports/eddie/scratch/s2522559/pixel_project/pixel/modify_running_script.py", line 106, in call
trainer.main(self.config_dict)
File "/exports/eddie/scratch/s2522559/pixel_project/pixel/scripts/training/run_pretraining.py", line 325, in main
validation_dataset = load_dataset(
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/datasets/load.py", line 1676, in load_dataset
builder_instance = load_dataset_builder(
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/datasets/load.py", line 1502, in load_dataset_builder
dataset_module = dataset_module_factory(
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/datasets/load.py", line 1254, in dataset_module_factory
raise e1 from None
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/datasets/load.py", line 1225, in dataset_module_factory
raise e
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/datasets/load.py", line 1205, in dataset_module_factory
dataset_info = hf_api.dataset_info(
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/huggingface_hub/hf_api.py", line 1761, in dataset_info
hf_raise_for_status(r)
File "/exports/eddie/scratch/s2522559/conda/envs/pixel/lib/python3.9/site-packages/huggingface_hub/utils/_errors.py", line 293, in hf_raise_for_status
raise RepositoryNotFoundError(message, response) from e
huggingface_hub.utils._errors.RepositoryNotFoundError: 401 Client Error. (Request ID: Root=1-654baa8e-2186f9fe22cc891357596294;926d6ac1-fd23-457c-b19d-8707baa23362)

Repository Not Found for url: https://huggingface.co/api/datasets/plip/wiki_dev.
Please make sure you specified the correct repo_id and repo_type.
If you are trying to access a private or gated repo, make sure you are authenticated.
Invalid username or password.

Do you have any idea how to get access to the development set?

Hi,

I succesfully configured the repository and everything works fine, except for dependency parsing. When trying to either run the sanity check or train a new model, I get this error:
Traceback (most recent call last): File "/home/lys/pixel_mainlp/pixel/scripts/training/run_ud.py", line 466, in <module> main() File "/home/lys/pixel_mainlp/pixel/scripts/training/run_ud.py", line 423, in main outputs = trainer.predict(test_dataset=eval_dataset, metric_key_prefix="eval") File "/home/lys/anaconda3/envs/pixel-env/lib/python3.9/site-packages/transformers/trainer.py", line 2331, in predict output = eval_loop( File "/home/lys/pixel_mainlp/pixel/src/pixel/trainer.py", line 179, in evaluation_loop mask = arc_labels.ne(model.config.pad_token_id) File "/home/lys/anaconda3/envs/pixel-env/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1614, in __getattr__ raise AttributeError("'{}' object has no attribute '{}'".format( AttributeError: 'DataParallel' object has no attribute 'config'
I am using PyTorch 2.0.1 with CUDA 11.7. I managed to install it with the cuda 11.3 version suggested on the repository, and it works, but it installs the CPU PyTorch version instead of the GPU one. I do not know how to fix both things at the same time. Thanks.

After following the Setup, I found that I need to downgrade protobuf to 3.20 and intall wrapt, opt_einsum, gast, astunparse, termcolor, keras-preprocessing, and python-flatbuffers to load the package and model correctly.

In your work, it seems like the decoder is only used to pre-train the model, and then discarded for downstream tasks, and only the encoder is being used.

Is there a way for the decoder to predict text, and get back to the original text unicode representation? Is an OCR model (that can be trained in conjunction perhaps) required?

For instance, PIXELForSequenceClassification is used for finetuning GLUE. Its underlining ViT implementation is ViTModel instead of PIXELModel. It seems that PIXELModel follows MAE and uses fixed positional embeddings, while ViTModel follows the original ViT paper and uses learnable positional embeddings.

Trying to install this repository:

pip install git+https://github.com/xplip/pixel

Collecting git+https://github.com/xplip/pixel
Cloning https://github.com/xplip/pixel to ./tmp/pip-req-build-ijxd55pn
Running command git clone --quiet https://github.com/xplip/pixel /home/nlp/amit/tmp/pip-req-build-ijxd55pn
Resolved https://github.com/xplip/pixel to commit efb38f5
Running command git submodule update --init --recursive -q
Preparing metadata (setup.py) ... done

Then

python -c "import pixel"

Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'pixel'

I guess it does not install the actual python "package" called "pixel".

If you would add PyPi distribution as well, it would not need to also install the git submodules, which actually takes long as datasets is big.

Here's my suggestion:
#3

However, the name pixel is taken, so you would need to come up with a different pypi package name (this does not change the fact that the installed package will be called pixel)

I work with a 2D script (Sutton SignWriting), which specifies characters, and their 2D position. Linearly looks like this

but can be rendered like this

I am wondering how I could extend your work to not only work from left-to-right, but up-to-down, and create 2D patches instead of the current 1D

Hi, when I run "python scripts/training/run_pos.py
--model_name_or_path="Team-PIXEL/pixel-base-finetuned-pos-ud-vietnamese-vtb"
--data_dir="data/ud-treebanks-v2.10/UD_Vietnamese-VTB"
--remove_unused_columns=False
--output_dir="sanity_check"
--do_eval
--max_seq_length=256
--overwrite_cache", there is a following error:
Traceback (most recent call last):
File "scripts/training/run_pos.py", line 27, in <module>
from pixel import (
File ".../pixel/src/pixel/init.py", line 1, in <module>
from .data import *
File ".../pixel/src/pixel/data/init.py", line 1, in <module>
from .datasets import *
File ".../pixel/src/pixel/data/datasets/init.py", line 1, in <module>
from .ner_dataset import NERDataset
File ".../pixel/src/pixel/data/datasets/ner_dataset.py", line 29, in <module>
from ...utils import Modality, Split, get_attention_mask
File ".../pixel/src/pixel/utils/init.py", line 7, in <module>
from .optimization import *
File "<fstring>", line 1
(num_warmup_steps = )
^
SyntaxError: invalid syntax