I want to make a cleaner README with the GitHub Issues parts and contributions part very forward-facing.

See title.

I found 14 tabular data files that don't have a corresponding data dictionaries. See attached for the list. Requires further investigation and testing.
data_files_without_dictionaries.txt

It is currently written in MATLAB, so it needs refactored into Python to satisfy the style in this repository.

One of the ideas is to have a GUI that follows this layout: https://docs.newrelic.com/attribute-dictionary/
In the linked example, there are three columns on the main table: "Attribute name", "Definition", and "Data types". We could have "Name" (the question), "Description" (Description of the question), and "Dependencies" (If there is any dependency between this question and any other on the questionnaire.

We may need to inject a "false" dataset_description.json and some "blank" data for the validator to run, but we should still be able to validate the phenotypic data.

See issue #12 for more details.

A total of 120 dictionaries don't have a corresponding data file in the ABCD Release 4.0 tabular data package (see attached file). Further investigation required as to why. Maybe someone else can download the data package and run the data_convert.py script before we look into it further?

data_dictionaries_without_data_files.txt

Inputs

The HBN README covers how to download the HBN Data Dictionaries/ folder into this repository's HBN/ subfolder.

Task

The real GitHub Issue here is to create a dictionary.py script using a copy of the ABCD/dictionary.py as an example starter. ABCD should be used as the starter because it is the only other study right now that works with multiple data dictionaries sitting inside one folder.

Implementation

1. I suggest using the pandas read_excel() function: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_excel.html
2. It looks like the first row of each Excel file is always an unnecessary title (for BIDS data dictionary JSONs anyway). But this assumption needs to be confirmed before trying to read in all the data dictionaries programmatically.
3. It looks like every Excel file always only has the content on Sheet 1, but that should be checked as well.
4. The Question looks like the BIDS Description, the Variable Name looks like the ShortName, the Variable Type and Values look unnecessary, but could be included if desired, and the Value Labels will have to be parsed into BIDS Levels usually.
5. If you're unsure about what should go where, refer to the BIDS Tabular Files reference here: https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#tabular-files

Concept

A browsable, filterable, searchable GUI for selecting fields of interest from any data dictionary (or possibly all data dictionaries).

Inputs

The GUI would take as input this repository. It would read from all the available data dictionaries within <STUDY>/phenotype/ subfolders, as output by dictionary.py scripts.

Drafts

Please insert draft concepts/ideas into this Google Slides deck here:

https://docs.google.com/presentation/d/1OWgNupUz2FyKCEzoAjjDtgunaiWHmjT1ArG1asgndig/edit?usp=sharing

Implementation

I recommend using PySimpleGUI as I know it works well and is simple and stable and built in Python. Concepts can of course be drawn and pictured, or drafted, or even just a messy draft on a napkin.

Outputs

As output, you should receive a small data dictionary subset of the selected fields of interest. You should also be able to "auto-save" progress as you click so if the window closes you don't lose a bunch of reading and selection work.

The dream

In the far-flung future, this would also integrate with BIDS TSVs to provide a sneak preview to the user what the distribution of data points are. Like if it were a Yes/No question, then how many Yes and how many No are in the dataset?

Some questionnaires (like Diagnosis_KSADS from the HBN dataset) do not have Value Range column but this information could be inferred using the levels, which are provided.

The rule should just be: "Whatever the study's unique subject ID is for each subject, then THAT turned into an alphanumeric-only becomes the alphanumeric after sub- in the participant_id column."

For instance, datasets in this list could be chosen to increase the amount of supported datasets.

https://docs.google.com/spreadsheets/d/1t1VtW_ScMunEcrloxIv8197ezHQXCZQq_dxsSrH7SQI/edit?usp=sharing

See title

See issue #12 for more details.

Make it more obvious and more central for who is contributing or has contributed to this repository.

Consider whether or not to support not only BIDS, but also Psych-DS (which is modeled heavily after BIDS) for all Python scripts and output files.

See issue #12 for more details.

Something more than just the presence or absence of files. Look for them by exact name or regular expression pattern and check them for expected inputs (a header with commas for CSVs, etc.).

See issue #12 for more details.

Perhaps call them tabular_data.py or data_convert.py or something similar for each study independently.

Each page should just be about how to reproduce the data and dictionaries with version 2 and onward.

See issue #12 for more details. This could be a little more difficult than the rest because of the way UKB stores their tabular data for retrieval.

See issue title.

For all of the following:

1. Questions
2. Feedback
3. Features
4. Ideas
5. Bugs in scripts
6. Corrections in dictionaries
7. Corrections to READMEs
8. Style suggestions or common conventions for scripts, dictionaries, or READMEs
9. Anything else that isn't a code contribution or pull request

something like:

python bids-phenotype.py $STUDY $INPUT_DIR $OUTPUT_DIR --option

Where --option could be one of --dict, --data, or --all.

See title.