notai-tech / anuvaad Goto Github PK
View Code? Open in Web Editor NEWState of the art open-source translation for Indic languages.
License: GNU General Public License v3.0
State of the art open-source translation for Indic languages.
License: GNU General Public License v3.0
Are these models bidirectional? I can easily translate English to Hindi for example, but I don't see how I can translate Hindi back to English.
For example
from anuvaad import Anuvaad
anv = Anuvaad("english-hindi")
anu.anuvaad('नमस्ते।', target_lang="EN", source_lang="HI")
# returns 'नमस्ते।'
Is it not possible with these models to go the other way?
anu = anuvaad("english-malayalam")
Traceback (most recent call last):
File "", line 1, in
TypeError: 'module' object is not callable
anuvaad
<module 'anuvaad.anuvaad' from '/opt/conda/envs/translate/lib/python3.6/site-packages/anuvaad/anuvaad.py'>
anu = anuvaad("english-telugu")
Traceback (most recent call last):
File "", line 1, in
TypeError: 'module' object is not callable
Hi, great work! Can you share the checkpoints and training scripts for further fine tuning on custom dataset.
Thanks.
Hey,
I have installed the anuvaad package using pip and it is taking too long for translating the sentences. Is there any solution for this, like how can we optimize the inference time?
from anuvaad import Anuvaad
anu = Anuvaad("english-kannada")
input_text = 'Welcome back home'
output = anu.anuvaad(input_text)
Hi,
When translating long sentences I am getting part of the translation and other part is missing. It is happening in mostly long sentences.
Some examples given below.
In the early days of artificial intelligence , the field rapidly tackled and solved problems that are intellectually difficult for human beings but relatively straight - forward for computers
Many of the early successes of AI took place in relatively sterile and formal environments and did not require computers to have much knowledge about the world .
Chess can be completely described by a very brief list of completely formal rules , easily provided ahead of time by the programmer .
पूरी तरह से औपचारिक नियमों की एक बहुत संक्षिप्त सूची से शतरंज को पूरी तरह से वर्णित किया जा सकता है, जिसे प्रोग्रामर द्वारा आसानी से समय से आगे
I wanted to know , is there any length restrictions in the current model ?
Thanks,
Sriram
When I run anu = Anuvaad("english-telugu"), I am running into the following error. I have tried to set the flag "from_tf=True" in all the mentioned places, but it did not resolve nor change the error.
Traceback (most recent call last):
File "/home/harsha/.local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 950, in from_pretrained
state_dict = torch.load(resolved_archive_file, map_location="cpu")
File "/home/harsha/.local/lib/python3.6/site-packages/torch/serialization.py", line 587, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/home/harsha/.local/lib/python3.6/site-packages/torch/serialization.py", line 242, in init
super(_open_zipfile_reader, self).init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading zip archive: failed finding central directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "", line 1, in
File "/home/harsha/.local/lib/python3.6/site-packages/anuvaad/anuvaad.py", line 96, in init
lang_path, return_dict=True, from_tf=True
File "/home/harsha/.local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 953, in from_pretrained
f"Unable to load weights from pytorch checkpoint file for '{pretrained_model_name_or_path}' "
OSError: Unable to load weights from pytorch checkpoint file for '/home/harsha/.Anuvaad_en-te' at '/home/harsha/.Anuvaad_en-te/pytorch_model.bin'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.
Do suggest on how to resolve this. Thanking you !
I've created an MR to add your repository to indic.page. knadh/indic.page#12 is the pull request. Please do let me know if there's any concern
Hi,
I read your point of view here
But still in-depth visual inspection and evaluation is not possible .. as with language there are many possibilities & variation.
Hence benchmarking should be done regardless.
Google, Facebook. HuggingFace do benchmarking on Tatoeba .. its a small sentence dataset and it won't give very drastically different translations. I suggest you guys should post these BLEU Scores with your models.
It will add more trust in users who are going to use it and invest time in.
Thanks.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.