killua7362 / jaxformers Goto Github PK

Transformers

Transformers implementation from scratch using jax!
Report Bug · Request Feature

Transformers are a groundbreaking class of deep learning models that have revolutionized natural language processing and various other domains of artificial intelligence. They are characterized by their attention mechanism, which allows them to capture complex relationships and dependencies in data, making them exceptionally powerful for sequence-to-sequence tasks.

• Python
• Jax

Install all the libraries

pip install jax 

Change the dataset path in dataset.py file and run train.py file

ANGELO:
And cowards it be strawn to my bed,
And thrust the gates of my threats,
Because he that ale away, and hang'd
An one with him.

DUKE VINCENTIO:
I thank your eyes against it.

DUKE VINCENTIO:
Then will answer him to save the malm:
And what have you tyrannous shall do this?

DUKE VINCENTIO:
If you have done evils of all disposition
To end his power, the day of thrust for a common men
That I leave, to fight with over-liking
Hasting in a roseman.

• Implementing Modules class for converting classes to python function for jax.jit
• Implementing character based transformer architecture
• Training on tinyshakesphere dataset

See the open issues for a full list of proposed features (and known issues).

• https://www.youtube.com/watch?v=kCc8FmEb1nY
• https://arxiv.org/abs/1706.03762