zooniverse / data-digging Goto Github PK

This repo contains multiple examples of individually-customized, project-specific scripts that all perform a similar transformation:
INPUT = the Panoptes classifications export CSV (with JSON-encoded annotations info)
OUTPUT = a flat CSV file containing extracted marking/classification data in a flat, non-hierarchical format

Using the information provided in the workflow export CSV, one should be able to write a generalized reduction script that could replace many project-specific scripts.

Adapt repo for better use and visibility of its content.

• Reorganize folders and content to sort project-specific examples, general data reduction scripts, and other utility scripts.
• Create explicitly a list of links to external repos and code, replacing current stub text files.

Hy,
In the readme,.md you specify the basic_classification_stats.py file but unless I'm mistaken, I don't find this script in the list of script examples
https://github.com/zooniverse/Data-digging/tree/master/example_scripts

Thanks in advance and sorry if the question is silly ;)

This will need to happen eventually.

Let's say i we modify the workflow but in essence the tasks are the same across a number of major workflow version ids (e.g. seabird watch). Could we instead of providing an explicit version number take a minimum? E.g. worfklow_version >= 3

I am trying to install this Python Environment using the yml file. However I run into the following error :

NoPackagesFoundError: Package missing in current Win64 channels: 
                  - freetype 2.5.5 1

I am using the following code to install the environment using Anaconda 2 (after putting the .yml file in the current directory for Python.)

conda env create -f basic_project_stats.yml

I was not sure what the last line in the yml file which is 'prefix' is doing, but I changed as following to the directory where the python environments reside in my computer .

 prefix: /Users/Public/Anaconda2/envs/python279_volcrowe

I also tried installing a win 64 channel as suggested in a stackoverflow post using the following code( https://stackoverflow.com/questions/38739694/install-python-package-package-missing-in-current-win-64-channels?rq=1

conda config --add channels bioninja

I am able to add the bioninja thing but after it when I run the following command:
conda env create -f environment.yml

It gives me the same Win64 channel error as earlier.

Can anyone please help me solve this issue ?

Required Cleanup

In April 2020, the Zooniverse sent out an email to project owners informing them that they should visit https://github.com/zooniverse/Data-digging/blob/master/example_scripts/check_for_duplicate_marks.py to get a duplicate-checking script.

Simultaneously in April 2020, we had a big repo cleanup (#51) which fundamentally changed where the files are located and organised.

As a result, we have temporarily reinstated (#53) the /example_scripts folder with a copy of the check_for_duplicate_marks.py script for the sake of URL integrity.

❗ The /example_scripts folder should be deleted after 31 May 2020 as the "old script URL" would have had plenty of time to serve its purpose by then.

cc @mrniaboc @lcjohnso

It would be useful for projects to be able to identify single-answer prolific classifiers (which are typically bots).

To do: add this to basic_project_stats.py.

• Read in a pre-processed workflow-id+version classification file
• Read in workflow, workflow_contents exports and extract needed info
• 1 flat csv for question-task annotations, multiple for drawing-task annotations
• use WWK scripts to do the same for survey tasks
• allow for different separators than ,, e.g. \t