bd-r / bdclean Goto Github PK
View Code? Open in Web Editor NEWUser-friendly data cleaning pipeline for the inexperienced R user
Home Page: https://bd-r.github.io/bdclean-guide/
License: GNU General Public License v3.0
User-friendly data cleaning pipeline for the inexperienced R user
Home Page: https://bd-r.github.io/bdclean-guide/
License: GNU General Public License v3.0
Describe the bug
Verify if bdchecks has unreasonably high package size
To Reproduce
Download repo and verify, build executable and verify
Is your feature request related to a problem? Please describe.
Right now bdclean is highly customizable, but only through source code.
Describe the solution you'd like
We have to bring the same level of customizability outside source code, so that anybody can easily change the bdclean structure.
Describe alternatives you've considered
Need to follow bdchecks style - Having an XML or JSON file that can populate questions and checks and a .config file that can manipulate environment settings and shiny app configs.
A function to change names of the columns to Darwin Core standard.
Artifacts & Documentation --> Data & Cleaning Reports
Is your feature request related to a problem? Please describe.
We need to provide proper visualizations to understand data in bdclean itself.
Describe the solution you'd like
Implement bdviz and bdtools features.
In the Cleaning Report table its better to display the clean records and the records deleted as well.
Right now only the number of deleted records are shown.
Issue pointed out by Vijay
For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=10:10
Running any check from bdchecks crashes bdclean, console error massage:
For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=12:12
Need to move the data frame of questions to be asked to users to a .csv file for easy managament
Is your feature request related to a problem? Please describe.
Good to have kurator integration/support or something similar.
Describe the bug
Messages in the message panel is broken to several lines. Print a message in single line without breaking.
To Reproduce
Steps to reproduce the behavior:
Is your feature request related to a problem? Please describe.
The process now requires a lot of manual inputs. If w could develop a system to automatically continue the entire part of the process it will be easily reproducible.
Describe the solution you'd like
We have a limited console functionality to clean data. It doesn't have as much features as shiny app but it can be improved further.
Describe the bug
Remove test csv to bdtests
To Reproduce
Steps to reproduce the behavior:
Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Desktop (please complete the following information):
Smartphone (please complete the following information):
Additional context
Add any other context about the problem here.
Need to do it sooner than later since GSoC applicants would start visiting the repository once projects are on ideas page.
π @thiloshon!
I see the GitHub version of bdclean does not import {finch} any more, good news as we plan to archive {finch} on CRAN since it's no longer maintained. Do you intend to soon submit {bdclean} to CRAN?
Describe the bug
The code coverage budge is not working and CI badge is linked with @thiloshon 's fork. It needs to be brought to bdr.
Is downloading the generateReport.R
on the fly is good idea or should we just add it as part of the package?
In the newest version (thiloshon/bdclean; 0.1.13) : questionnaire is not available:
For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=13:13
On running get_config questions are not displayed
myconfig <- get_config()
What is the lowest taxonomic level you require in your data?
:
Selection: 0
Error in get_config() : The entered choice is wrong.
bdclean folder (Google Drive)
Here are some great resources when developing an R package:
Shiny tips & tricks:
Is your feature request related to a problem? Please describe.
We need domain specific Questionnaire and cleaning workflow like marine, Desert and such.
We have two data files quest and responses which we use in function get_config
. We need to document those items in roxygen
Generating all necessary citations. This is part of the reports generation module and need to be properly implemented in the data checks design.
Is your feature request related to a problem? Please describe.
Right now only one online DB can be queried at a given time.
Describe the solution you'd like
We need to be able to gather datasets from different sources and possibly local data as well for a single cleaning run and de duplicate.
After answering the questionnaire >> Flagging >> returning to the questionnaire and changing the answers >> some input from the first of the questionnaire session still exist.
A simple example:
Fresh bdclean session>> answering the questionnaire:
Now, changing the questionnaire to:
Results aren't changing as they should:
For more details see QA working file:
https://docs.google.com/spreadsheets/d/1VFhHy6uo7qVBiHhcuLqUJ3RXKpaG-4C3RD0jqvB0xtU/edit#gid=21537633&range=11:11
Describe the bug
The documentation lacks developer details for users to contribute. Add more details in the guide.
Shall we make it compulsory for a pull request to be reviewed by at least one other, to be able to merge it with master?
Right now I am able to send a PR and then merge it myself which is not good practice. Review will ensure the PR is in the accepted quality and free from any future conflicts.
https://help.github.com/articles/enabling-required-reviews-for-pull-requests/ this explains how to setup the feature.
Combining our DwC fields grouping mechanism and a really good tabular data R viewer (if exists???) can be really helpful when reviewing DwC data in R. If we can't find a decent viewer, I wonder if maybe we can implement the fields grouping mechanism in OpenRefine...
Is your feature request related to a problem? Please describe.
Being able to save and laod the current environment and code of the shiny environment.
No that we have our questions are in .rda file we will have to have some helper function to modify the questions?
I created few slides to better understand how bdclean will be structured and how it will interact with other packages in the bdverse ecosystem.
Access the slide here and any suggestion is welcomed.
Using bdclean checks within bdchecks:
Instead of bdclean own quality checks: https://github.com/bd-R/bdclean/blob/dev/R/quality_checks.R
I think we just need to modify the questionnaire file (right @thiloshon ?):
https://github.com/bd-R/bdclean/blob/076fa9bf39c1f799e16665121a1260188a8d66d9/R/questionnaire.R
By:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. πππ
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google β€οΈ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.