isc-hemc / burrobot Goto Github PK

To the json message retrived from GraphAPI add a field that contains the conversation id, for example:

{
"message": "Hello",
"conversation_id": x_0000000,
"from": {},
"to": {}
}

README File

Add a brief explanation to the project repository README.md file.

Scraper Implementation

Create the scraper module implementation.

Update NLP module README.md according to the updates.

In docker-compose file add "restart: always" in all services.

En caso de que la bandera de DB se encuentre en verdadero y no se pueda establecer una conexión con la base de datos, cambiarla a falso para prevenir errores.

Bibliography section is empty. Add references.

The NLP module must support stemming, add the implementation in text_processing.py file.

By the moment the mongo helper class in the scraper module doesn't validate if the document trying to insert already exists, implement this functionality, if the document's message id already exists (not _id, this _id it's created by mongo) only update the message not re-insert it.

Add pre-commit configuration to scraper module.

Configure a logger for topics modeling, this logger must sent it's output a tmp file named topic.log and each element must be a LDA output that will be plotted.

Implement a logs configuration similar to nlp/logs/config.yaml.

Create mongodb configuration.

• Mongo Dockerfile.
• Docker compose to build and run automatically the db.
• Custom mongod.conf file.
• Script mongo-init.js to create the burrobot user and it's related database.
• Variables.env file for mongo root user and password.

Update the README.md file adding a description of the mongo database and related environment variables to establish a connection to the db.

Create the initial configuration for the NLP project.

The NLP project must be able to take the information retrivered from scraper (mongodb or file system storage) and clean that data, make exploratory analysis, get the most related topics, use analizers like gensim, etc., by the moment an initial configuration is enough.

• Pipenv file
• Development dependencies.
• VSCode settings, launch and tasks files.
• README.md file
• Pre-commit file.

Check Scraper project for reference.

Add logs in the main processes of the module, also in the functions in which the execution can failure.

Introduction module

The introduction must have the following information.

1.- Presentación.
2.- Motivación.
3.- Planteamiento del problema.
4.- Objetivos.
5.- Objetivo general.
6.- Objetivo especifico.
7.- Estado del arte.
8.- Descripción del documento.

Solve the TODO comments.

Solve the TODO comments.

Implement a module capable of displaying plots with different sets of data for understanding and future analysis.

Add .gitignore file to the repository.

Ignore all Python, Tex, etc., files.

Don't forget to add environment files, for example: .env, .local.env, .dev.env, .stg.env, .prod.env, etc.

MySQL docker configuration:

• Dockerfile configuration.
• .conf file if it's needed.
• Docker compose configuration.
• Create burrobot user and set its password.
• Create burrobot-mysqldb database.
• Create the DB schema, when running docker-compose the schema it's created automatically.

If the registry already exists only update it.

Implement a module that obtain related topics from the corpus:

• Get Related Topics.
• LDA.

Documentation Initial Configuration

1. Create documentation folder.
2. Search for a latex template.
3. Add Cover Page.

Cover page information

1. IPN and ESCOM logos
2. TT Name and ID
3. TT student names
4. TT directors names

Use the following example

Implement the LDA multicore algorithm provided by gensim in topics_analysis.py file.

Stemming support, add a column that stores the stemmed value.

Store the datasets in MySQL database:

Add the text_processing.py unittests.

Add logs in the main processes of the module, also in the functions in which the execution can failure.

Module that connects to MySQL and performs actions over the database.

In the questionnaire's graphs, data must match with the graphs axis and results.

Update the root README.md file according to the updates in every module.

Plot related topics or most important topics from a file.

Add the topics_analysis.py unittests.

Implement a module capable of clean raw texts:

• Tokenize.
• Lemmatize.
• Remove Stopwords.
• Remove Symbols.
• Build Vocabulary
• Unittests

Implement a method or class that can be able to connect with mongo and get/post/update/delete data from it.

Update the README.md file adding a description of how to run scraper script with a connection to mongo.

Modify the column created_time in Message table of the SQL db to varchar(50).

Scraper Initial Configuration

Prepare the Python's scraper initial configuration for downloading messages from Facebook and retrieve linguistic information.

1.- Create a folder for the scraper program.
2.- Upload pipenv file with dependencies.
3.- Pipenv file must include development dependencies.
4.- Add README.md for a proper configuration and an explanation of how to execute the program.