dcorney / text-generation Goto Github PK

Simple page where user enters parameters (or selects from examples) such as:

• protagonists' names
• locations / objects etc.
• type/length of text required

And then they click 'generate' and are shown a brand new page of text

E.g. W2V models, processed text documents, grammars etc.

Before importing texts from Project Gutenburg (for example), it'd be nice to review them and optionally reject unwanted texts. Also maybe mark the start/end point of the main text to be imported.

Should speed up NER/tokenization

There are several related classes that share a 'synonym' function: thesaurus.py (based on Wordnet); word_vectors.py (w2v) and potentially more (e.g. based on Roget's thesaurus).
Should create an abstract class 'Synonymizer' which is implemented by thesaurus.py (which should be renamed to reference Wordnet) and by word_vectors and by Roget's etc.

The Redis config file specifies the directory to store the database in. Switching between different Redis stores would be nice, so we could (e.g.) have a novel-markov-chain and a news-markov-chain and a shakespearean-markov-chain etc.

Make sure code can deploy/run (easily) to an EC2 instance, so it can get on with downloading and processing texts. E.g.

• write Redis to S3 for persistence
• or just take text, run through tokenizer and NER step (which is slow), writing processed texts to S3 files. Then separate process to add processed texts quickly into a Redis Markov Chain

Write a screensaver (e.g. for MacOS) that generates an endless scrolling stream of text. Possibly use Processing?

Currently, have code to pull text from Project Gutenburg. Should diversify to other sources, including:

• Reuters-21578
• Brown Corpus
• Signal-1M
• more modern literature sources (LiveJournal?)