ikergarcia1996 / easy-translate Goto Github PK

Hello,
I have a great many headlines and articles, most not in English, stored in a database, which I can store in python lists.
I would like to loop through the lists, and translate each variable, without writing the contents to a file first.
This is part of a website, which displays a great many headlines, to start, for the users to choose from.
Writing each headline to a file, translate, and read the file back to a list will take too much time.
Thank you, in advance, for your help.
Baruch

In some (english-centric or low ressource) cases other models could yield better results, e.g. mBART or SMaLL100:
https://huggingface.co/facebook/mbart-large-50-many-to-one-mmt
https://huggingface.co/alirezamsh/small100 has another tokenizer
Could we include them?

Hello,
Tested with Debian 11/12, cuda 11.7/11.8, different models, different precisions,with and without accel, etc. Other projects based on torch and transformers work well on the same machine.

I have these errors when running the script:
`python3 translate.py --sentences_path sample_text/en.txt --output_path sample_text/en2es.translation.m2m100_1.2B.txt --source_lang en --target_lang es --model_name models/pytorch_model.bin
Loading model from models/pytorch_model.bin
Traceback (most recent call last):
File "Easy-Translate/.env/lib/python3.11/site-packages/transformers/configuration_utils.py", line 702, in _get_config_dict
config_dict = cls._dict_from_json_file(resolved_config_file)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Easy-Translate/.env/lib/python3.11/site-packages/transformers/configuration_utils.py", line 793, in _dict_from_json_file
text = reader.read()
^^^^^^^^^^^^^
File "", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 64: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "Easy-Translate/translate.py", line 443, in
main(
File "Easy-Translate/translate.py", line 115, in main
model, tokenizer = load_model_for_inference(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "Easy-Translate/model.py", line 75, in load_model_for_inference
config = AutoConfig.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Easy-Translate/.env/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 983, in from_pretrained
config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Easy-Translate/.env/lib/python3.11/site-packages/transformers/configuration_utils.py", line 617, in get_config_dict
config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Easy-Translate/.env/lib/python3.11/site-packages/transformers/configuration_utils.py", line 705, in _get_config_dict
raise EnvironmentError(

OSError: It looks like the config file at 'models/pytorch_model.bin' is not a valid JSON file.`

Can the model be loaded only once instead of waiting for the load to complete each time?

I was previously using this from sample and its no longer works now after updates.
python3 translate.py \ --sentences_path sample_text/en.txt \ --output_path sample_text/en2es.translation.small100.txt \ --source_lang en \ --target_lang es \ --model_name alirezamsh/small100

This is the log:
2023-12-06T12:38:42.665144096Z Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.
2023-12-06T12:38:43.268330478Z The tokenizer class you load from this checkpoint is not the same type as the class this function is called from. It may result in unexpected tokenization.
2023-12-06T12:38:43.268375418Z The tokenizer class you load from this checkpoint is 'M2M100Tokenizer'.
2023-12-06T12:38:43.268383058Z The class this function is called from is 'SMALL100Tokenizer'.
2023-12-06T12:38:43.269837017Z Loading model from alirezamsh/small100
2023-12-06T12:38:43.269851767Z Loading custom small100 tokenizer for utils.tokenization_small100
2023-12-06T12:38:43.270022978Z Traceback (most recent call last):
2023-12-06T12:38:43.270132589Z File "//Easy-Translate/translate.py", line 538, in
2023-12-06T12:38:43.270510501Z main(
2023-12-06T12:38:43.270625462Z File "//Easy-Translate/translate.py", line 134, in main
2023-12-06T12:38:43.270820843Z model, tokenizer = load_model_for_inference(
2023-12-06T12:38:43.270937283Z File "/Easy-Translate/model.py", line 90, in load_model_for_inference
2023-12-06T12:38:43.271180565Z tokenizer: PreTrainedTokenizerBase = AutoTokenizer.from_pretrained(
2023-12-06T12:38:43.271340276Z File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2045, in from_pretrained
2023-12-06T12:38:43.271888289Z return cls._from_pretrained(
2023-12-06T12:38:43.271961479Z File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py", line 2256, in _from_pretrained
2023-12-06T12:38:43.272524103Z tokenizer = cls(*init_inputs, **init_kwargs)
2023-12-06T12:38:43.272535813Z File "/Easy-Translate/utils/tokenization_small100.py", line 153, in init
2023-12-06T12:38:43.272685754Z super().init(
2023-12-06T12:38:43.272714444Z File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 366, in init
2023-12-06T12:38:43.272931705Z self._add_tokens(self.all_special_tokens_extended, special_tokens=True)
2023-12-06T12:38:43.272935405Z File "/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils.py", line 462, in _add_tokens
2023-12-06T12:38:43.273221467Z current_vocab = self.get_vocab().copy()
2023-12-06T12:38:43.273318857Z File "/Easy-Translate/utils/tokenization_small100.py", line 289, in get_vocab
2023-12-06T12:38:43.273603199Z vocab = {self.convert_ids_to_tokens(i): i for i in range(self.vocab_size)}
2023-12-06T12:38:43.273645449Z File "/Easy-Translate/utils/tokenization_small100.py", line 192, in vocab_size
2023-12-06T12:38:43.273817970Z return len(self.encoder) + len(self.lang_token_to_id) + self.num_madeup_words
2023-12-06T12:38:43.274002501Z AttributeError: 'SMALL100Tokenizer' object has no attribute 'encoder'. Did you mean: 'encode'?

This model own space has same error:
https://huggingface.co/spaces/alirezamsh/small100

I use this command

python3 translate.py \
--sentences_path input.srt \
--output_path result.srt \
--source_lang eng_Latn \
--target_lang ind_Latn \
--model_name facebook/nllb-200-distilled-600M \
--precision fp16

with input.srt

1
00:00:07,312 --> 00:00:09,993
Hello.

2
00:00:09,994 --> 00:00:11,227
Where are you right now?

3
00:00:11,228 --> 00:00:13,360
Right now I am on my way
to South Dakota.

4
00:00:13,361 --> 00:00:16,093
Gonna do a little camping,
do a little fishing.

5
00:00:16,094 --> 00:00:17,426
Good for you, Colter.

but the result.srt has problems:

• wrong order
• empty line replace with (dalam bahasa Inggris)
• appended unknown

1
00:00:07,312 --> 00:00:09,993
Hei, apa yang kau lakukan?
(dalam bahasa Inggris) <-- this should be empty line
2 (satu) <-- the '(satu)' should not be exist
00:00:09,994 --> 00:00:11,227
Di mana kau sekarang?
(dalam bahasa Inggris) ....
3 Pemberantasan Korupsi <-- this also should not be exist
00:00:11,228 --> 00:00:13,360
Saat ini aku sedang dalam perjalanan
ke Dakota Selatan.
(dalam bahasa Inggris) ...
4
00:00:13,361 --> 00:00:16,093
Akan pergi berkemah sedikit,
lakukan sedikit memancing.
(dalam bahasa Inggris) ...
5
00:00:16,094 --> 00:00:17,426
Bagus untukmu, Colter.
(dalam bahasa Inggris) ...

I would love to use this as a gui on windows

Any plans to add a simple API even on a server running from Python?

BaGRoS