viisix / dancecats Goto Github PK

Add tests for DatabaseConnector class.

Target: 85% coverage

User.version should be integer and can't be string value.
Please investigate this bug

An auto scheduled job should be resumed to retry running few times (configurable) if it fails. After few retries but cannot succeed, it should alert to that job's recipient emails.

Support job scheduling:

• Once
• Hourly
• Daily
• Weekly
• Monthly

Using sphinx to create document for DanceCat.

Add unit test for Models:

• Job
• QueryJob
• JobFeature

For more convenience when view jobs, add paging and sorting jobs function when view job list.

Add unit test for Connection Model

We are missing delete schedule function. Need to add ASAP.

Should cache to avoid run from the beginning while the data is growing and obese with time

Add basic python documentation like MANIFEST, HISTORY...

A simple migrator script that support SQL-Alchemy for easy database schema migrating.

Currently blocking #67
Default support connector: MySQL
Convert to connector plugins: PostgreSQL + SQLServer
Add option config path of plugins directory.

DanceCat should raise error if user doesn't set CONFIG_FILE CONFIG_FILE=/etc/dancecat/config.cfg

Add document about Jobs and schedules management.

RC4 is very old, so we have to upgrade old encrypt method to AES.
Also, add a console command to upgrade existing password on the the database.

We have the function to add full database schema and the migration tool can only database from somewhere in the past of development. But if we use that we can't use the helpful migration tool by alembic. We have to roll back and fix this.

Add test for FrequencyTaskChecker module.

Target: 70%

As a user, I want to use variable on job, so that I can have more freely control on what will run on that job.

Add support for PostgreSQL query job

I read that Numba and PyPy can help improve python performance. Since I also read that PyPy can't be used for numpy and others like scikitlearn, I want to try out the other option: Numba.

AllowedEmail

• Version column's type should be Integer
• Add filter to verify email for Email column.

If the result file is too large, do not send it via email, instead send a link via email to user.

• Have a configurable number in MB that if a result file is larger than this number, do not send the whole file via email.
• Discuss: Download link can be used by everyone, or user have to login to download it. An option for job creator/editor maybe needed.

Related issue: #90

As now every users in DanceCat can edit and run jobs. But we may want to limit that. In this scope, find a solutions, discuss how can we use roles to grant permission on users, and how we grant it.

Using Flask-Misaka to use markdown on job's annotation.

Currently DanceCat is only tested on Google Chrome. After I tried on Safari 9, I found out that because of ES6 format Safari do not open web socket, the realtime job's result is not work. Need to work around more with webpack to fix this.

Add document about publishing DanceCat using WSGI and Nginx.

Fix relative import issue.

Add support for 2 factors authentication and a method to enable it.
https://github.com/pyotp/pyotp

Write tests for Helper's functions.

Add new console command:

• Add new email to allowed_email table so that he/she can start using DanceCat

Add document for user management include

• Add allowed user
• Sign Up and Login

Add document about Connection management in DanceCat.

Function: aes_decrypt
Location: https://github.com/scattm/DanceCat/blob/dev/DanceCat/Helpers.py#L166
For example:

• Input data to aes_encrypt: 8ajYk2aaaaaaab
• Expected output from aes_decrypt: 8ajYk2aaaaaaab
• Actually output: 8ajYk2aaaaaaa

Need to investigate and fix.

Add more test to have better coverage, current target is 45 percent

Currently whenever a job is running it will run the query and then send mail to users afterwards. We must change this behavior by separate them:

• Create new RQ queue for send email.
• After running query, enqueue a job for send email
• New statuses for Job: "Success, mails sent" and "Success, mails failed"

Currently we support view job result on Web as JSON. It would be great if there are also a table view for it.

Add test for FrequencyTaskChecker module.

Target: 70%

This is a critical issue since at some point we can not set the next run value of schedule objects. Need to trace down the root cause.

Scope of work:

• Find root cause and fix
• Add test for Schedule model

Add test for DB models of User and related. Include:

• User
• AllowedUser
  Target: 85% coverage

Basic SQL syntax autocomplete.

• Add one more attribute to Job Model: Timeout
• Add timeout field to Query Job Form
• Pass timeout parameter when execute query job

We need one command so that user can use it with console to check full database schema.

Add a function so that user can use it to reset their password.

Add test for workers:

• Query worker
• Send mail worker

Target: 90% coverage

Currently we if a connection is deleted, that connection's jobs is still available. Need to fix this.

This issue may be caused by email server configuration:
1- Cannot reach recipient email at the time configured for that scheduled job (really late arrival)
2- Cannot send to recipient email

As the result of User Model unit testing in #61. We need to add more features and improve the User Model. Those items list below just for note, we will add more issues for them later.

Feature:

• Disable user
• Delete user
• Change email
• Change password #72
• User roles (Execute/View/Edit) #71

Improve:

• Process of duplicate
• More user states
• Password expiration

Scheduling is base on next_run timestamp on schedule table. If the application is closed on scheduled time, scheduled jobs won't be run and updated to next run time. So I have to create a new method to manual update those outdate schedules.