emreaksan / deepwriting Goto Github PK

View Code? Open in Web Editor NEW

101.0 12.0 29.0 1.09 MB

Code for `DeepWriting: Making Digital Ink Editable via Deep Generative Modeling` paper

License: MIT License

Python 100.00%

handwriting-synthesis handwriting-generation recurrent-neural-networks variational-autoencoder tensorflow

deepwriting's Introduction

DeepWriting

Code for DeepWriting: Making Digital Ink Editable via Deep Generative Modeling paper.

Implementation of conditional variational RNNs (C-VRNN) fow handwriting synthesis, generation, beautification and editing tasks.

Dataset

(November 2021) We released json files includeing timestamp information and stroke-based segmentations.

We collected data from 94 authors by using IAMOnDB corpus. After discarding noisy samples of IAMOnDB, we compiled a dataset of 294 authors, fully segmented. You can download our preprocessed data. For the raw dataset, license and other information, please visit our project page.

The dataset is in compressed .npz format. Strokes, labels and statistics can be accessed by using <key, value> pairs. We split handwriting samples by end-of-characters such that each sample consists of ~300 strokes. Samples are then moved to the origin and represented by using the offset values between consecutive strokes.

For example, you can visualize validation samples with indices 1, 5 and 20 by running

python visualize_hw.py -D ./data/deepwriting_validation.npz -O ./data_images -S 1 5 20

If you use our data, we kindly ask you to cite our work, and also fill IAMOnDB's registration form and follow their citation requirements since our dataset extends IAMOnDB.

Training Model

Training details and hyper-parameters are defined in config.py.
Download training dataset and copy into data folder. Otherwise, don't forget to update training_data and validation_data entries in config.py.
Set PYTHONPATH to include source.

export PYTHONPATH=$PYTHONPATH:./source

Run training

python tf_train_hw.py -S <path_to_save_experiment>

If you want to continue training

python tf_train_hw.py -S <path_to_save_experiment> -M <model_folder_name>

Evaluating Model

You can either train a model or download our pretrained model.
You can run

python tf_evaluate_hw.py -S <path_to_save_experiment> -M <model_folder_name> -QL

Evaluation options are defined in tf_evaluate_hw.py globally.

Dependencies

numpy
Tensorflow 1.2+ (not sure if earlier versions work.)
matplotlib
OpenCV (pip install opencv-python is enough)
svgwrite
scikit
sklearn

Missing

We are not planning to release demo interface.

Citation

If you use this code or dataset in your research, please cite us as follows:

@inproceedings{Aksan:2018:DeepWriting,
	author = {Aksan, Emre and Pece, Fabrizio and Hilliges, Otmar},
	title = {{DeepWriting: Making Digital Ink Editable via Deep Generative Modeling}},
	booktitle = {SIGCHI Conference on Human Factors in Computing Systems},
	series = {CHI '18},
	year = {2018},
	location = {Montr{\'}eal, Canada},
	publisher = {ACM},
	address = {New York, NY, USA},
}

deepwriting's People

Contributors

Stargazers

Watchers

deepwriting's Issues

Lack some .py files?

Hi, author,
I want to learn you paper and codes, and try to run your codes, but when I run the python tf_train_hw.py -S <path_to_save_experiment> ,python tf_evaluate_hw.py -S <path_to_save_experiment> -M <model_folder_name> -QL,
I find it lacks of some .py files
Will you updata them , thank you !

char_label's value isn't equal to the alphabet's id

@emreaksan hello, I found the release dataset char_label's id isn't correspond to the alphabet's id. especially, the first id. Others's id - 33 = alphabet's id.

Seems you reorgnized the code but didn't add source. to files

I don't know why nobody pointed it out, should be a easy fix

Help

python tf_evaluate_hw.py -S <path_to_model_folder> -M tf-1514981744-deepwriting_synthesis_model -QL
This generates a set of Images and Plots.
Also I do not see any tensorflow log files, to visualize in tensorflow.
How can I try this with my own handwriting ?

Please help me.

Data preprocessing scripts

Inference file for single and batch inferencing for conditional handwriting generation

Hi
I want to use the pretrained model for conditional handwriting generation. I tried to understand the evaluation file in an effort to make an inference file where I can give some text and can obtain the final svg file of handwritten text in a choosen style but the evaulation file has too many things specific to your experimental setup.
So, can you make a simple inference file or give some pointers on how I can make one?

Simplified model code for training

scikit don't want to install

scikit has changed its name, how can I use it?

Timestamp information or sampling rate

Thank you for this excellent work, as well as the valuable dataset. I've been trying to find any information about the timestamps for each stroke point, or a sampling rate. I think the original IAM-OnDB had timestamps, but even a sampling rate (if the stroke points are picked after regular intervals) for this dataset would be sufficient.

I'd appreciate any help you can provide in this regard. Thanks!

I received the error:

ModuleNotFoundError: No module named 'dataset'

I tried:

pip install dataset

But then get the error:

Traceback (most recent call last):
  File "C:\ci\code\deepwriting\visualize_hw.py", line 164, in <module>
    from dataset_hw import HandWritingDatasetConditional as Dataset
  File "C:\ci\code\deepwriting\dataset_hw.py", line 7, in <module>
    from dataset import BaseDataset
ImportError: cannot import name 'BaseDataset' from 'dataset' (......\Python\Python310\site-packages\dataset\__init__.py)

I then cd'ed into source and changed the command I was running from:

python visualize_hw.py -D ./data/deepwriting_validation.npz -O ./data_images -S 1 5 20

to:

python utils_visualization.py -D ./data/deepwriting_validation.npz -O ./data_images -S 1 5 20

the command ran, but I'm not seeing that the data_images folder is created.

Dataset and training codes

Missing character images in .json file

Hi,

I could not find the character images as mentioned in the .json file as follows:
"char_image_path": "../incomingData/12345/b03-205/b03-205z-01_Summons_0_S_0.png"

I am just wondering, do you have these character images, and are you able to share them?

Thanks!

Data link is dead

The link to the preprocessed data in the readme (https://ait.ethz.ch/projects/2018/deepwriting/downloads/deepwriting_dataset.tar.gz) now returns HTTP 404. Is a new link available?

Data scripts

to read/load JSON formatted data.
to preprocess strokes and create data dictionary that our input pipeline expects.