This is a tensorflow implementation of the byte-net model from DeepMind's paper Neural Machine Translation in Linear Time.
From the abstract
The ByteNet decoder attains state-of-the-art performance on character-level language modeling and outperforms the previous best results obtained with recurrent neural networks. The ByteNet also achieves a performance on raw character-level machine translation that approaches that of the best neural translation models that run in quadratic time. The implicit structure learnt by the ByteNet mirrors the expected alignments between the sequences.
Image Source - Neural Machine Translation in Linear Time paper
The model applies dilated 1d convolutions on the sequential data, layer by layer to obain the source encoding. The decoder then applies masked 1d convolutions on the target sequence (conditioned by the encoder output) to obtain the next character in the target sequence.The character generation model is just the byteNet decoder, while the machine translation model is the combined encoder and decoder.
- The character generation model is defined in
ByteNet/generator.pyand the translation model is defined inByteNet/translator.py.ByteNet/ops.pycontains the bytenet residual block, dilated conv1d and layer normalization. - The model can be configured by editing model_config.py.
- Number of residual channels 512 (Configurable in model_config.py).
- Python 2.7.6
- Tensorflow 1.2.0
- The character generation model has been trained on Shakespeare text. I have included the text file in the repository
Data/generator_training_data/shakespeare.txt. - The machine translation model has been trained for german to english translation. You may download the news commentary dataset from here http://www.statmt.org/wmt16/translation-task.html
Create the following directories Data/tb_summaries/translator_model, Data/tb_summaries/generator_model, Data/Models/generation_model, Data/Models/translation_model.
-
Text Generation
- Configure the model by editing
model_config.py. - Save the text files to train on, in
Data/generator_training_data. A sample shakespeare.txt is included in the repo. - Train the model by :
python train_generator.py --text_dir="Data/generator_training_data" python train_generator.py --helpfor more options.
- Configure the model by editing
-
Machine Translation
- Configure the model by editing
model_config.py. - Save the source and target sentences in separate files in
Data/MachineTranslation. You may download the new commentary training corpus using this link. - The model is trained on buckets of sentence pairs of length in mutpiples of a configurable parameter
bucket_quant. The sentences are padded with a special character beyond the actual length. - Train translation model using:
python train_translator.py --source_file=<source sentences file> --target_file=<target sentences file> --bucket_quant=50python train_translator.py--help for more options.
- Configure the model by editing
- Generate new samples using :
python generate.py --seed="SOME_TEXT_TO_START_WITH" --sample_size=<SIZE OF GENERATED SEQUENCE>
- You can test sample translations from the dataset using
python translate.py.- This will pick random source sentences from the dataset and translate them.
ANTONIO:
What say you to this part of this to thee?
KING PHILIP:
What say these faith, madam?
First Citizen:
The king of England, the will of the state,
That thou dost speak to me, and the thing that shall
In this the son of this devil to the storm,
That thou dost speak to thee to the world,
That thou dost see the bear that was the foot,
- Evaluating the translation Model
- Implement beam search - Contributors welcomed. Currently the model samples from the probability distribution from the top k most probable predictions.
- Neural Machine Translation in Linear Time paper
- Tensorflow Wavenet code
- Sugar Tensor Source Code For implementing some ops.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent