bd-r / bdclean Goto Github PK

Describe the bug
Verify if bdchecks has unreasonably high package size

To Reproduce
Download repo and verify, build executable and verify

Is your feature request related to a problem? Please describe.
Right now bdclean is highly customizable, but only through source code.

Describe the solution you'd like
We have to bring the same level of customizability outside source code, so that anybody can easily change the bdclean structure.

Describe alternatives you've considered
Need to follow bdchecks style - Having an XML or JSON file that can populate questions and checks and a .config file that can manipulate environment settings and shiny app configs.

A function to change names of the columns to Darwin Core standard.

Artifacts & Documentation --> Data & Cleaning Reports

Is your feature request related to a problem? Please describe.
We need to provide proper visualizations to understand data in bdclean itself.

Describe the solution you'd like
Implement bdviz and bdtools features.

In the Cleaning Report table its better to display the clean records and the records deleted as well.

Right now only the number of deleted records are shown.

Issue pointed out by Vijay

Console Error:

For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=10:10

Running any check from bdchecks crashes bdclean, console error massage:

For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=12:12

Need to move the data frame of questions to be asked to users to a .csv file for easy managament

Is your feature request related to a problem? Please describe.
Good to have kurator integration/support or something similar.

Describe the bug
Messages in the message panel is broken to several lines. Print a message in single line without breaking.

To Reproduce
Steps to reproduce the behavior:

1. Initiate bdclean worflow '...'
2. Allow messages to be populated '....'
3. Click Notifications in the top right corner.'
4. See error

Is your feature request related to a problem? Please describe.
The process now requires a lot of manual inputs. If w could develop a system to automatically continue the entire part of the process it will be easily reproducible.

Describe the solution you'd like
We have a limited console functionality to clean data. It doesn't have as much features as shiny app but it can be improved further.

Describe the bug
Remove test csv to bdtests

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• OS: [e.g. iOS]
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Smartphone (please complete the following information):

• Device: [e.g. iPhone6]
• OS: [e.g. iOS8.1]
• Browser [e.g. stock browser, safari]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

Need to do it sooner than later since GSoC applicants would start visiting the repository once projects are on ideas page.

👋 @thiloshon!

I see the GitHub version of bdclean does not import {finch} any more, good news as we plan to archive {finch} on CRAN since it's no longer maintained. Do you intend to soon submit {bdclean} to CRAN?

• Storing mechanism for questions (Data frame/csv)
  -Editing / adding new questions and linking data check functions to questions
  -Handling conditional / sub questions (tree structure)
• Darwinayzer and kurator-web
• Input data

Describe the bug
The code coverage budge is not working and CI badge is linked with @thiloshon 's fork. It needs to be brought to bdr.

Is downloading the generateReport.R on the fly is good idea or should we just add it as part of the package?

In the newest version (thiloshon/bdclean; 0.1.13) : questionnaire is not available:

Console Error:

For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=13:13

On running get_config questions are not displayed

 myconfig <- get_config()
What is the lowest taxonomic level you require in your data?  

: 

Selection: 0
Error in get_config() : The entered choice is wrong.

bdclean folder (Google Drive)

Here are some great resources when developing an R package:

• Mastering Software Development in R
• CRAN- Writing R Extensions
• How to develop good R packages (Maëlle Salmon)

Shiny tips & tricks:

• advanced-shiny

Is your feature request related to a problem? Please describe.
We need domain specific Questionnaire and cleaning workflow like marine, Desert and such.

We have two data files quest and responses which we use in function get_config. We need to document those items in roxygen

Data

Questionnaire

Flag Data

Console

Happens almost every time 😭

Generating all necessary citations. This is part of the reports generation module and need to be properly implemented in the data checks design.

Is your feature request related to a problem? Please describe.
Right now only one online DB can be queried at a given time.

Describe the solution you'd like
We need to be able to gather datasets from different sources and possibly local data as well for a single cleaning run and de duplicate.

• Storing resultant records
• Storing records for cleaning / editing
• Managing versions of data after each operation
• File types
• More than 5M records, do we need seperate mechanism to run checks?

After answering the questionnaire >> Flagging >> returning to the questionnaire and changing the answers >> some input from the first of the questionnaire session still exist.
A simple example:

Fresh bdclean session>> answering the questionnaire:

Performing flagging:

Results:

Now, changing the questionnaire to:

Results aren't changing as they should:

For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=11:11

Describe the bug
The documentation lacks developer details for users to contribute. Add more details in the guide.

Shall we make it compulsory for a pull request to be reviewed by at least one other, to be able to merge it with master?

Right now I am able to send a PR and then merge it myself which is not good practice. Review will ensure the PR is in the accepted quality and free from any future conflicts.

https://help.github.com/articles/enabling-required-reviews-for-pull-requests/ this explains how to setup the feature.

Combining our DwC fields grouping mechanism and a really good tabular data R viewer (if exists???) can be really helpful when reviewing DwC data in R. If we can't find a decent viewer, I wonder if maybe we can implement the fields grouping mechanism in OpenRefine...

https://github.com/bd-R/bdclean/tree/1183e302ef6be80095a5b9455d56c37b39c96408/inst/txts
@thiloshon: Am I right?

Data checks characterization (GoogleDocs)

Data checks DB (GoogleSheets)

Is your feature request related to a problem? Please describe.
Being able to save and laod the current environment and code of the shiny environment.

https://github.com/MangoTheCat/goodpractice

No that we have our questions are in .rda file we will have to have some helper function to modify the questions?

I created few slides to better understand how bdclean will be structured and how it will interact with other packages in the bdverse ecosystem.

Access the slide here and any suggestion is welcomed.

Using bdclean checks within bdchecks:

1. https://github.com/bd-R/bdchecks/blob/dev/R/dc_taxo_level.R
2. https://github.com/bd-R/bdchecks/blob/dev/R/dc_spatial_resolution.R
3. https://github.com/bd-R/bdchecks/blob/dev/R/dc_earliest_date.R
4. https://github.com/bd-R/bdchecks/blob/dev/R/dc_temporal_resolution.R

Instead of bdclean own quality checks: https://github.com/bd-R/bdclean/blob/dev/R/quality_checks.R

I think we just need to modify the questionnaire file (right @thiloshon ?):
https://github.com/bd-R/bdclean/blob/076fa9bf39c1f799e16665121a1260188a8d66d9/R/questionnaire.R
By:

• Replacing all the bdclean data quality functions with the correlative bdchecks functions
• To modify the current names of bdchecks functions to their new name (I can do it @tom-gu )
• 🙏 Pray 🙏

Brunches:

bdchecks>>dev | bdclean>>dev