itranslit's Introduction

iTRANSLIT

iTRANSLIT is a deep learning based transliteration package for indic language

Installation

pip install itranslit

Dependency

pytorch 1.7.0 or 1.7.0+

NB: No GPU need. It's CPU based.

Supported Language and Language Code

Language Name	Langauage Code
Bangla	bn
Gujarati	gu
Hindi	hi
Punjabi	pa
Sindhi	sd
Urdu	ur
Malayalam	ml
Tamil	ta

API

from itranslit import Translit

translit = Translit('bn')
word = "aami"
output = translit.predict(word, topk=10)
print(output)

Datasets and Training Details

We used Google Dakshina Dataset
Thanks to AI4Bharat for providing training notebook with details explanation
We trained Google Dakshina lexicons train datasets for 10 epochs with batch size 128, 1e-3, embedding dim = 300, hidden dim = 512, lstm, used attention
We evaluated our trained model with Google Dakshina lexicon test data using AI4Bharat evaluation script
You can find evaluation summary here

itranslit's People

Contributors

Stargazers

Watchers

itranslit's Issues

Doesn't seem to work with pytorch 2.0.1

This command results in this error:
translit = Translit('ml')

ℹ ml model downloading inside itranlit-models..
✔ ml model download successfull. model path itranlit-models/ml.pth
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/me/testtransliterategithubthing/venv/lib/python3.11/site-packages/itranslit/transliterate.py", line 582, in __init__
    self.model = load_pretrained(self.model, self.model_path)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/testtransliterategithubthing/venv/lib/python3.11/site-packages/itranslit/transliterate.py", line 525, in load_pretrained
    pretrain_dict = torch.load(weight_path, map_location=torch.device('cpu'))
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/testtransliterategithubthing/venv/lib/python3.11/site-packages/torch/serialization.py", line 815, in load
    return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/me/testtransliterategithubthing/venv/lib/python3.11/site-packages/torch/serialization.py", line 1033, in _legacy_load
    magic_number = pickle_module.load(f, **pickle_load_args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
_pickle.UnpicklingError: invalid load key, '<'.```

Recommend Projects