Example from the 7T-trt dataset from bids-example

$ tree -d sub-0[12]
sub-01
├── ses-1
│   ├── anat
│   ├── fmap
│   └── func
└── ses-2
    ├── fmap
    └── func
sub-02
├── ses-1
│   ├── anat
│   ├── fmap
│   └── func
└── ses-2
    ├── fmap
    └── func

Actual behavior

BIDS = bids.layout(dataset_path);
bids.query(BIDS,  'modalities',  'sub',  '01',  'ses',  '2')

ans =

  1×3 cell array

    'anat'    'func'    'fmap'

Expected behavior

BIDS = bids.layout(dataset_path);
bids.query(BIDS,  'modalities',  'sub',  '01',  'ses',  '2')

ans =

  1×2 cell array

   'func'    'fmap'

Notes

This also applies to spm_bids

Right now most of the metadata is automatically read (bval, bvec, events, channels), but other metadata is not read (headshape, photo, etc.). I think this mainly distinguishes between the metadata being in tsv files (which are read) versus another format (which are not read).

I think it would be nice to make this configurable, so that the user would know whether he can always expect a MATLAB struct or a string pointing to a filename.

I am tempted to say we should try to put in place some sort of input validation with more meaningful error messages to make the behavior between matlab and octace more consistent.

This is the kind of output on the parts that fails

With matlab

>> BIDS = bids.layout(fullfile(pwd, 'tests', 'bids-examples', '7t_trt'));

>> bids.query(BIDS, 'subjects')
ans =
  1×22 cell array
  Columns 1 through 12
    '01'    '02'    '03'    '04'    '05'    '06'    '07'    '08'    '09'    '10'    '11'    '12'
  Columns 13 through 22
    '13'    '14'    '15'    '16'    '17'    '18'    '19'    '20'    '21'    '22'

>> bids.query(BIDS, 'sessions', 'sub', '01')
ans =
  1×2 cell array
    '1'    '2'

% the following should at leat throw a warning 
%   that subject label should be a string
>> bids.query(BIDS, 'modalities', 'sub', 1)
ans =
  0×0 empty cell array

>> bids.query(BIDS, 'modalities', 'sub', '01')
ans =
  1×3 cell array
    'anat'    'fmap'    'func'

% the following should at leat throw a warning
%    that session '01' does not exist with a list of the valid sessions
>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '01')
ans =
  0×0 empty cell array

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '1')
ans =
  1×3 cell array
    'anat'    'fmap'    'func'

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '2')
ans =
  1×2 cell array
    'fmap'    'func'

With octave

>> BIDS = bids.layout(fullfile(pwd, 'tests', 'bids-examples', '7t_trt'));

>> bids.query(BIDS, 'subjects')
ans =
{
  [1,1] = 01
  [1,2] = 02
  [1,3] = 03
  [1,4] = 04
  [1,5] = 05
  [1,6] = 06
  [1,7] = 07
  [1,8] = 08
  [1,9] = 09
  [1,10] = 10
  [1,11] = 11
  [1,12] = 12
  [1,13] = 13
  [1,14] = 14
  [1,15] = 15
  [1,16] = 16
  [1,17] = 17
  [1,18] = 18
  [1,19] = 19
  [1,20] = 20
  [1,21] = 21
  [1,22] = 22
}

>> bids.query(BIDS, 'sessions', 'sub', '01')
ans =
{
  [1,1] = 1
  [1,2] = 2
}

Asking for modalities on sessions that do not exist return the same thing. (??)

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '02')
ans =
{
  [1,1] = anat
  [1,2] = fmap
  [1,3] = func
}

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '12')
ans =
{
  [1,1] = anat
  [1,2] = fmap
  [1,3] = func
}

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '023')
ans =
{
  [1,1] = anat
  [1,2] = fmap
  [1,3] = func
}

Both of the following return empty cell arrays in matlab.
We can have an error on the second and third one but they should be more informative.

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '')
ans =
{
  [1,1] = anat
  [1,2] = fmap
  [1,3] = func
}

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', [])
error: Invalid call to lookup.  Correct usage is:

 -- IDX = lookup (TABLE, Y)
 -- IDX = lookup (TABLE, Y, OPT)
error: called from
    print_usage at line 91 column 5
    ismember at line 116 column 10
    query at line 82 column 15

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', 2)
error: Invalid call to lookup.  Correct usage is:

 -- IDX = lookup (TABLE, Y)
 -- IDX = lookup (TABLE, Y, OPT)
error: called from
    print_usage at line 91 column 5
    ismember at line 116 column 10
    query at line 82 column 15

This is the part of the test that goes "BOINK" as session 2 only has fmap and func.

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '1')
ans =
{
  [1,1] = anat
  [1,2] = fmap
  [1,3] = func
}

>> bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '2')
ans =
{
  [1,1] = anat
  [1,2] = fmap
  [1,3] = func
}

Originally posted by @Remi-Gau in #108 (comment)

I have taken the liberty to "protect" the master branch so that things MUST go through a pull request (though admin can override this).

I am tempted to create a dev branch and to have all PRs that are not bug fixes send there: since we do not have official realeases this could be way to have our "stable" and "latest" release.

Just a thought.

Installing Matlab libraries for use with other libraries is a big pain. There's no standard mechanism to do so, so it's "some assembly required" for end users. So having dependencies sucks. BIDS-MATLAB users on Matlab pre-R2016b need to figure out how to install JSONio or SPM12 in one of their environments.

What do you think about bundling a version of JSONio or SPM12 with BIDS-MATLAB itself, and have BIDS-MATLAB automatically pull it in if it detects that it is needed? That way BIDS-MATLAB users wouldn't have to worry about it, and you could guarantee that the version of the JSON library that BIDS-MATLAB is using is one that it is tested against.

You could make a minor fork of one of them and stick it under a +bids/+internal/+opp package to prevent it from conflicting with any user-installed JSONio or SPM12. ("opp" standing for "other parties' programs" here.)

Wouldn't work for Octave (without getting really fancy), because oct-files need to be built for the specific Octave installation they're running against. (Or maybe Octave can use MEX files now? I can't remember the binary-compatibility situation.)

Also we only support bold as contrast label for now and cbv and phase are also in the specs.

This seems more of an edge case at the moment and might require more coding that could be rendered moot when we have a schema to help us. So I would suggest that we open an issue for this point (to remind ourselves and let others know we know) but do not move on to fix it right away.

Originally posted by @Remi-Gau in #99 (comment)

@jasmainak gave me an overview of pybids yesterday. I recognize some design choices of that in bids-matlab as well, which I think is good. I would like to lay out in a bit more detail how the technical representation of the layout is (or should be) and how the user interface/experience should be. I will first post this here for discussion, and hope that it can then be added to the README.md and the online help in MATLAB/Octave.

Certain types of MRI fieldmap come witha magnitude and phase images.

At the moment, it seems that bids.query only returns the phase images when asking for

bids.query(BIDS, data', 'modality', 'fmap')

It would be more convenient to return all the files (phase AND magnitude) associated to a given run of a fieldmap.

I made an extra change in spm_save.m a couple of weeks ago to save an issue with the NaN to n/a conversion - I'll try to add a new test case to highlight the issue.

--- spm_save.m	(revision 7946)
+++ spm_save.m	(revision 7947)
@@ -35,17 +35,19 @@
                     var{i} = var{i}(:);
                     if ~iscell(var{i})
                         var{i} = cellstr(num2str(var{i},16));
+                        var{i} = strtrim(var{i});
                         var{i}(cellfun(@(x) strcmp(x,'NaN'),var{i})) = {'n/a'};
                     end
                 end
                 var = [fn'; var{:}];
-            elseif iscell(var)
+            elseif iscell(var) || isnumeric(var) || islogical(var)
+                if isnumeric(var) || islogical(var)
+                    var = num2cell(var);
+                end
                 var = cellfun(@(x) num2str(x,16), var, 'UniformOutput',false);
-            elseif isnumeric(var) || islogical(var)
-                var = num2cell(var);
-                var = cellfun(@(x) num2str(x,16), var, 'UniformOutput',false);
+                var = strtrim(var);
+                var(cellfun(@(x) strcmp(x,'NaN'),var)) = {'n/a'};
             end
-            try, var = strtrim(var); end
             
             fid = fopen(f,'Wt');
             if fid == -1

Originally posted by @gllmflndn in #40 (comment)

right now it implements participants.tsv, but not scans.tsv.

The scans.tsv not only contains a row for each dataset (run), but can also contain run-specific details. E.g. in http://www.fieldtriptoolbox.org/workshop/madrid2019/bids_sedation/ it contains sedation levels for each of the 4 runs.

I want the data structures to be formally described (in json/yaml) and not in the MATLAB code, that representation should be shared with pybids (MATLAB has support for JSON - see here

This would rely on using the BIDS schema that exists as a set of yml files for now and should therefore be converted to something that matlab can handle like JSON.

Questions

• where to do the conversion and hosting of the JSON schema?

See other comments in the "wish-list" issue #60

#60 (comment)
#60 (comment)
#60 (comment)
#60 (comment)
#60 (comment)

I think there is going to be a growing needs for bids-matlab to be able to parse deviratives if only to make it easier for people to use the output of fMRIprep for example.

So unless there is clear disagreement I suggest opening a draft PR to start working on this.

Another issue about infra aspects:

How about switching to using MOxUnit and MoCov to run our tests and get some idea of our code coverage?

https://github.com/MOxUnit/MOxUnit
https://github.com/marketplace/actions/moxunit-action

As far as I can see right now, each session from the subjects will appear as a separate 'subject' after reading in the layout of the BIDS dataset. See this line in layout.m.

For example, a 134 subjects study, where each subject underwent scanning (or data acquisition) over 2 sessions, subjects will be as a 1 x 268 sub-structure array and the sessions field is nor populated:

oBIDS = 

             dir: 'path_to_BIDS_dataset'
     description: [1x1 struct]
        sessions: {}
           scans: [0x0 struct]
            sess: [0x0 struct]
    participants: [1x1 struct]
        subjects: [1x268 struct]

while I would expect to have subjects sub-structure array of size 134 x 2 and the sessions field as a 134 x 1 cell array (or something similar):

BIDS = 

             dir: 'path_to_BIDS_dataset'
     description: [1x1 struct]
        sessions: {1x134 cell}
           scans: [0x0 struct]
            sess: [0x0 struct]
    participants: [1x1 struct]
        subjects: [134x2 struct]

I have slightly update the layout.m function to do this but there remain a few open questions on how exactly describe multi-session dataset:

• what about the case where subjects may have a different number of sessions?
  • should the subjects field be a Nsubj x Nsess structure array, with some of these empty when the session is missing?
    ➡️ this is straightforward and in line with how modalities are left empty if not present.
  • the subjects field could also be Nsubj x 1 cell array, each containing a 1 x Nsess_subj array of subjects structure?
    ➡️ more complicated to handle IMO.
• what about listing the sessions in the field sessions?
  • just a cell array of the unique sessions in the data set, i.e. a 1 x Nsess cell array?
    ➡️ minimalist option, which may well be sufficient.
  • Nsubj x 1 cell array with each cells containing the 1 x Nsess_subj cell array for that specific subject?
    ➡️ more comprehensive but more complicated to handle (that's what I did in my update for no specific reason actually)
  • a Nsubj x Nsess cell array with some cells left empty when the session is missing?
    ➡️ this feels really redundant.

where Nsess is the total number of different sessions and Nsess_subj the number of session for a specific subject.

This should be resolved when we implement the schema but I am opening an issue for the record.

• add ce entity
• remove fa, inv, recording entities

If I have a data collection with BDF data (one file per recording), and if I would convert the BDFs to BrainVision (triplets), I would expect the same BIDS layout to be returned and the same MATLAB analysis script to run. The script should not have to care about file formats, but simply pass them into e.g. ft_preprocessing.

Keeping annotations up to date

Well I think that something like this should be part of our contributing guideline and realease protocols to make sure that people know about them when they submit code and for us to make sure we check those function signatures are still valid before realease.

I can only agree - and perhaps just saying "when adding top-level functions or modifying their inputs, make sure that they have valid entries in functionSignatures.json (insert link to mathworks docs)" would be enough. I guess it is just conceptualization for now, or should I be adding it as part of this PR?

Originally posted by @mslw in #42 (comment)

GitHub Actions allows for easy testing on Windows, MacOS, and Linux, which would improve the testing pipeline to also cover "special" bugs that only appear on some OSses.

Only capture the raw data, channel and event file. Misses the JSON files.
Also, there are only 2 subjects (with 3 sessions each). Yet the program think there are 6 subjects

Thanks for making this resource available !

It would be great to be able to cite it directly. I'm not sure if you're yet ready to version this and create a DOI, but I wonder if generating an RRID would be of interest. Happy to help, but I wanted to check in with you, first !

As discussed in the last meeting (see #59) we will be keeping track of what "people" have already created to deal BIDS that could be added to this repository

• FieldTrip (i.e. the toolbox at large) includes low-level code to read all file formats that BIDS deals with
• bids-matlab-tools mentioned in this post
  • https://github.com/sccn/bids-matlab-tools/blob/master/rename_brainvision_files.m
  • https://github.com/sccn/bids-matlab-tools/blob/master/bids_spreadsheet2participants.m
• CPP-BIDS: to help with creating certain files (tsv and json) for raw BIDS

When I load and do a report on example ieeg_filtered_speech, I get an empty report. Is this expected?

>> b = bids.layout('bids-examples/ieeg_filtered_speech');
>> bids.report(b)
>> 

Same for example ieeg_motorMiller2007.

>> b = bids.layout('bids-examples/ieeg_motorMiller2007');
>> bids.report(b)
>> 

Found while running bids-matlab-ut. Failed test output here.

I am tempted to have some suggestions on code style in the repo.

A very easy way to do this would be to use the miss_hit package that can format code automatically to a certain extend and flag things that it cannot fix (like using a certain type of case for the functions, classes). Pretty much like a black or pep8 for matlab.

It can also help enforcing a certain level of code quality (flags functions that are too complex, with too many nested blocks...)

I am using it on several repos and added as part of the CI to ensure consistency.

It is quite tweakable so we could decide on what features to implement.

how do people feel about this?

@gllmflndn @robertoostenveld @apjanke @tanguyduval

Hi all,

Thank you for this very handy repo. This issue is to make you aware of my contributions since I'm using BIDS with MATLAB for nearly a year now:

1. I created this BIDS viewer for MATLAB based on the excellent imtool3D viewer:
   https://github.com/tanguyduval/imtool3D_td
   There is a dependency with the excellent dicm2nii toolbox for loading files
   You can call it using imtool3D_BIDS
   I am using this bids-matlab repo in order to parse the BIDS folder. But I had to make small changes to parse my BIDS folder since it was too sensitive to errors in .tsv files.

2. I recently collaborated with xiangruili to include an interactive BIDS converter in dicm2nii.

It would be great to have your feedbacks on the viewer and converter. And I would be very pleased to add these tools in bids-standard.

Best,
Tanguy

Are you interested in rearranging the repo so that it could be installed using Octave's pkg tool? That's Octave's standard package manager. It provides conveniences like install/uninstall commands, a load command, and the ability to specify dependencies.

This would be a simple rearrangement:

• stick the +bids source code directory in a new top-level inst/ directory
• add a DESCRIPTION file in Octave pkg format
• copy or rename LICENSE to COPYING, to follow GNU's conventions
• optionally, add a CITATION file to describe how people should cite your package for academic publications

I could throw together a PR to do this, if you're interested.

• add tests to check warning VS error handling of the tolerant option of layout
• add test to make sure query can also filter files that do NOT have a given entity (ex: when querying for bold but want to ignore the files that do not have the dir entitiy)
• add test for the `dir``entity (see #99)

Originally posted by @Remi-Gau in #11 (comment)

I will install the all contributor bot on this repo: https://github.com/all-contributors/all-contributors

Makes it easy to keep track of contributions.

Let me know in the coming 48 hours if you feel deeply against this.

Playing around with a dataset that has different acq for the bold data and I realized that bids.query does not cover that parameter (nor rec).

It would be useful to have the possibility to just do:
fullbrain_bold_ls = bids.query(BIDS, 'data', 'sub', '01', 'type', 'bold', 'acq', 'fullbrain').

acq and rec are already listed in the structure returned by layout should it should be fairly straightforward. @gllmflndn you know those functions better so do you want to have a go or should I have a crack at it?

Mentioned in PR #8.

Adding unit testing is definitely required - with continuous integration through Octave.

I am trying to get some stuff running on a pure octave + SPM12 environment so while I wanted to test this matlab-bids there to see if it behaved well with octave.

Tested on the empty dataset ds001 from the BIDS-example repo.

Running the following that relies on the SPM machinery (i.e spm_BIDS.m) works fine

docker run -it --rm \
-v /c/Users/Remi/Documents/BIDS/ds001/:/data: \
-v /c/Users/Remi/Documents/BIDS/code/:/code \
spmcentral/spm:octave-latest eval "spm_BIDS('/data/')"

But trying to do the same without using SPM by running this fails.

docker run -it --rm \
-v /c/Users/Remi/Documents/BIDS/ds001/:/data: \
-v /c/Users/Remi/Documents/BIDS/code/:/code \
spmcentral/spm:octave-latest script '/code/test_bids_matlab.m'

With bids-matlab in the /code folder and with test_bids_matlab.m being:

addpath(genpath(fullfile('/code', 'bids-matlab')))
data_dir = '/data/';
BIDS_dir = fullfile(data_dir);
layout  = bids.layout(data_dir)

This is the error message I get.

Execution failed: /code/test_bids_matlab.m
'file_utils' undefined near line 24 column 16
Error in layout (line 24)
Error in test_bids_matlab (line 4)
Error in spm (line 1218)
Error in run (line 71)
Error in spm (line 1218)
Error in spm_standalone (line 115)
Error in /usr/local/bin/spm12 (line 38)

Looks like older version of octave did not deal well with subfolders to define classes (that's what "+" folders are, correct?).
http://octave.1599824.n4.nabble.com/Are-package-folder-supported-td4630147.html

Is that still the case with octave 4.4 (the one that this container uses)?

Am I missing some matlab / octave subtle difference here?

We created some functions to help write and create several common BIDS files (tsv, json) in this repo.

As most of bids-matlab is for querying already existing BIDS datasets rather than "creating" then I did not thing of suggesting to add those functions in here.

But maybe some of them could have their place here.

https://github.com/cpp-lln-lab/CPP_BIDS
https://github.com/cpp-lln-lab/CPP_BIDS/blob/master/notebooks/basic_usage.ipynb

See here: https://app.codecov.io/gh/bids-standard/bids-matlab/branch/master

I mean I know we are not hitting every line in the codebase but that's just passive aggressive now... 😉

The attached file is from a valid dataset published on OpenNeuro (ds003195)
The import function fails with the error

Error using bids.util.tsvread>dsv_read (line 178)
Invalid DSV file '/data/matlab/dataquality/ds003195/participants.tsv': Varying number of delimiters per line.

Attaching the file

participants.tsv.zip

Hi, BIDS folks,

Could you document which versions of Matlab are supported by bids-matlab?

And to what degree is Octave supported? Is bids-matlab strictly limiting itself to features & functions available in Octave as well as Matlab? And if so, which versions of Octave are supported?

Example from the 7T-trt dataset from bids-example

$ tree -l --nolinks sub-01
sub-01
├── ses-1
│   ├── anat
│   │   ├── sub-01_ses-1_inv-1_mp2rage.nii
│   │   ├── sub-01_ses-1_inv-2_mp2rage.nii
│   │   ├── sub-01_ses-1_T1map.nii
│   │   └── sub-01_ses-1_T1w.nii
│   ├── fmap
│   │   ├── sub-01_ses-1_run-1_magnitude1.nii
│   │   ├── sub-01_ses-1_run-1_magnitude2.nii
│   │   ├── sub-01_ses-1_run-1_phasediff.json
│   │   ├── sub-01_ses-1_run-1_phasediff.nii
│   │   ├── sub-01_ses-1_run-2_magnitude1.nii
│   │   ├── sub-01_ses-1_run-2_magnitude2.nii
│   │   ├── sub-01_ses-1_run-2_phasediff.json
│   │   └── sub-01_ses-1_run-2_phasediff.nii
│   ├── func
│   │   ├── mean_sub-01_ses-1_task-rest_acq-fullbrain_run-1_bold.nii
│   │   ├── sub-01_ses-1_task-rest_acq-fullbrain_run-1_bold.nii
│   │   ├── sub-01_ses-1_task-rest_acq-fullbrain_run-1_physio.tsv.gz
│   │   ├── sub-01_ses-1_task-rest_acq-fullbrain_run-2_bold.nii
│   │   ├── sub-01_ses-1_task-rest_acq-fullbrain_run-2_physio.tsv.gz
│   │   ├── sub-01_ses-1_task-rest_acq-prefrontal_bold.nii
│   │   └── sub-01_ses-1_task-rest_acq-prefrontal_physio.tsv.gz
│   └── sub-01_ses-1_scans.tsv
└── ses-2
    ├── fmap
    │   ├── sub-01_ses-2_run-1_magnitude1.nii
    │   ├── sub-01_ses-2_run-1_magnitude2.nii
    │   ├── sub-01_ses-2_run-1_phasediff.json
    │   ├── sub-01_ses-2_run-1_phasediff.nii
    │   ├── sub-01_ses-2_run-2_magnitude1.nii
    │   ├── sub-01_ses-2_run-2_magnitude2.nii
    │   ├── sub-01_ses-2_run-2_phasediff.json
    │   └── sub-01_ses-2_run-2_phasediff.nii
    ├── func
    │   ├── sub-01_ses-2_task-rest_acq-fullbrain_run-1_bold.nii
    │   ├── sub-01_ses-2_task-rest_acq-fullbrain_run-1_physio.tsv.gz
    │   ├── sub-01_ses-2_task-rest_acq-fullbrain_run-2_bold.nii
    │   ├── sub-01_ses-2_task-rest_acq-fullbrain_run-2_physio.tsv.gz
    │   ├── sub-01_ses-2_task-rest_acq-prefrontal_bold.nii
    │   └── sub-01_ses-2_task-rest_acq-prefrontal_physio.tsv.gz
    └── sub-01_ses-2_scans.tsv

Actual behavior

BIDS = bids.layout(dataset_path);
fmapFiles = bids.query(BIDS, 'data', 'modality', 'fmap', 'subject', '01')
fmapFiles =

  0×0 empty cell array

Actual behavior

BIDS = bids.layout(dataset_path);
fmapFiles = bids.query(BIDS, 'data', 'modality', 'fmap', 'subject', '01')
fmapFiles =

  4×1 cell array

    '/home/remi/openneuro/ds001168/raw/sub-01/ses-1/fmap/sub-01_ses-1_run-1_phasediff.nii'
    '/home/remi/openneuro/ds001168/raw/sub-01/ses-1/fmap/sub-01_ses-1_run-2_phasediff.nii'
    '/home/remi/openneuro/ds001168/raw/sub-01/ses-2/fmap/sub-01_ses-2_run-1_phasediff.nii'
    '/home/remi/openneuro/ds001168/raw/sub-01/ses-2/fmap/sub-01_ses-2_run-2_phasediff.nii'

A few basic functions that I would find useful, from the perspective of creating pipelines that generate/process derivatives in a BIDS-compliant manner:

• A function for changing a BIDS filename, based on changes to specific key/value pairs (i.e., changing, adding, and removing key/value pairs to a name, and maintaining the proper order of fields in the name).
• A function for changing one or several fields in a .json metadata file (e.g., to generate a new metadata file after applying some transformation / preproc step to a dataset)
• A non-private version of bids-matlab's get_metadata function. I frequently find myself wanting a simple function to check a specific metadata field of a single file, without having to run both bids.layout and bids.query. I understand the rationale for keeping get_metadata private, but from my perspective it would be very useful to have a public version of something like this command.

Originally posted by @bmdeen in #60 (comment)

See also comments:

#60 (comment)

There are a few circumstances (see here under) where I would like to initialize the derivatives with a copy, complete or partial, of the original full BIDS dataset. I have been working on a prepare_derivatives.m for that specific purpose with some features, such as:

• copy the data of all or a sub-list of subjects;
• if there are multiple sessions, copy the data from all or only some sessions;
• copy all or a sub-list of modalities, and for func data possibly all or some tasks;
• unzip the .nii.gz files and 4D-to-3D unpacking the func and dwi images, if requested;
• decide where that derivatives/tool subfolder will be created, i.e. not necessarily next to the "raw" data in the BIDS folder.

Why would I want to do that? Well because, it is convenient to work on a subset of data and/or in a "sandpit", for example

• to develop a processing pipeline with 5 subjects instead of 100 or simply to keep the subjects that are kept for the specific analysis, i.e. leave out some "outliers"
• analysis can focus on a single session for some question and therefore the processing will be different from that combing multiple sessions
• analysis can also focus on a single modality (or task for func data), therefore no need to bring along everything
• some operations in SPM's standard special processing pipelines, e.g. realignment and coregistration, do change the header of the images, so they ought to be in a derivatives subfolder before any processing. Moreover SPM and some other tools, would prefer dealing with a series of 3D images rather than a single 4D volume, e.g. ditching the 1st few functional images into a "dummy" subfolder.
• if the dataset is saved on an external storage place (like the centralized "mass-storage" system at ULiège), one cannot directly work on this server and therefore data need to be copied locally before any processing is applied, i.e. I might as well call this my derivatives\some_tool_or_step data and start working on these. Afterwards, it is straightforward to copy that derivatives folder back on the external storage place.

Originally posted by @ChristophePhillips in #60 (comment)

• possibility to copy part of the raw only by filtering by subject, session, modality, task
• possibility to unzip and split 4D into 3D (FYI: Rémi is not in favor of this. 😉 )
• this derivative folder can be anywhere relative to the raw data but make the default path follow one of the recommended ways to store derivatives with respec to the raw (see BIDS specs here
• initialize a dataset_description.json in the root of that derivatiive folder

See comments:

• #60 (comment)
• #60 (comment)
• #60 (comment)
• #60 (comment)

In other repos:

• https://github.com/Remi-Gau/CPP_BIDS_SPM_pipeline/blob/bf850e944b0f9e2ee8381c11b2a3ef21960da54c/src/workflows/bidsCopyRawFolder.m
• https://github.com/CPernet/spmup/blob/master/bids/spmup_BIDS_unpack.m

In the tests/test_get_metadata.m file, there are some commented-out tests.

% define the expected output from bids query metadata
func.RepetitionTime = 7;
func_sub_01.RepetitionTime = 10;
anat.FlipAngle = 5;
anat_sub_01.FlipAngle = 10;
anat_sub_01.Manufacturer = 'Siemens';

[...]

%% test func metadata base directory
metadata = bids.query(BIDS, 'metadata', 'type', 'bold');
%assert(metadata.RepetitionTime == func.RepetitionTime);


%% test func metadata subject 01
metadata = bids.query(BIDS, 'metadata', 'sub', '01', 'type', 'bold');
%assert(metadata.RepetitionTime == func_sub_01.RepetitionTime);


%% test anat metadata base directory
metadata = bids.query(BIDS, 'metadata', 'type', 'T1w');
%assert(metadata.FlipAngle == anat.FlipAngle);


%% test anat metadata subject 01
metadata = bids.query(BIDS, 'metadata', 'sub', '01', 'type', 'T1w');
assert(metadata.FlipAngle == anat_sub_01.FlipAngle);
assert(strcmp(metadata.Manufacturer, anat_sub_01.Manufacturer));

When I comment these asserts back in, the tests for the "base directory" cases fail. It looks like even when you're not passing the 'sub','01' filter to bids.query(), it still picks up the subject 01 metadata instead of the metadata from the task-auditory_bold.json file in the root of this BIDS directory. (E.g. RepetitionTime is 10 in both of the 'type','bold' queries, instead of 7 and 10, respectively.) Is this expected behavior?

I think we need to establish some "ground rules" how to make this repo functions because I am growing less comfortable playing the benevolent dictator and also we should talk make some decisions about the overall architecture before we move further (for example see #39 and #19).

Also this might good occasion to let a couple of people (who have also written some matlab/octave code to deal with BIDS) know about this repo, so they can think about bringing some of it here.

@gllmflndn @robertoostenveld @apjanke @tanguyduval

I can set up a doodle to find a date in the coming weeks if you send me a way to contact you here : remi dot gau at gmail dot com.

Let me know your time zone too.

Meeting outcome

TODO

• open mattermost channel and update README
• upload meeting video on G drive
• update meeting notes
• share outcomes on bids mailing list, mattermost town square, neurostars, twitter
• open issue to collect "needs" / "requests" for bids-matlab
• call to the community for already existing code for matlab / octave to deal with BIDS
• doodle for next meeting
• set up "governance" proposal for the repo

I was looking into how to make some aspects of the documentation writing easier.

I have started playing with sphinx to automatically generate part of a Matlab based project and serving it with read the docs. There is an extension for sphinx that can deal with matlab and read automatically from the "help" sections of functions, classes...

I know that cosmomvpa uses it.

I suspect that reStructured text has a higher "tech" requirements than markdown so might be scarier to new contributors but I feel that this could make the developping / maintaining of the doc easier in the long term.

OK there is one assert that now fails for some unknown reason.

  pth_bids_example = get_test_data_dir();
  
  BIDS = bids.layout(fullfile(pth_bids_example, '7t_trt'));

  %   test
  mods = {'anat', 'fmap', 'func'};
  
  assert(isequal(bids.query(BIDS, 'modalities'), mods));
  assert(isequal(bids.query(BIDS, 'modalities', 'sub', '01'), mods));
  assert(isequal(bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '1'), mods));
  
  % assert(isequal(bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '2'), mods(2:3)));
  % this now fails on octave 4.2.2 but not on Matlab
  %
  % On octave this gives:
  % 
  % bids.query(BIDS, 'modalities', 'sub', '01', 'ses', '2')
  %
  % ans =
  % {
  %   [1,1] = anat
  %   [1,2] = fmap
  %   [1,3] = func
  % }

Originally posted by @Remi-Gau in #108 (comment)

I have started using BIDS model from the BIDS model BEP in the pipeline of our lab because not everyone can use fitlins. :-p

Things are getting to a point where I am thinking to create a function / set of function / class to deal with those (a bit like the analysis module does in pybids).

I think those functionalities would fit better in the bids-matlab repo so I am tempted to work on that here.

My questions are :

• how do people feel about working on things that are not "vanilla / official" BIDS yet like the BIDS model?

  • pros:
    • those BEP offer some standard which is better than no standard
    • this BEP might not change much
    • if pybids does it, so can we (I like the idea of both packages being more in synch)
  • con :
    • we might have to redo some things and potentially break things downstream if people start using bids-matlab once the BEP become official (I think we need to start having versioned releases).

• if we agree to support this, how should it be implemented?

  • suggestion : mimicking the pybids implementation maybe? (that I have not looked at yet)

If I try to query the metadata for the eeg_face13 example, it errors out:

>> b = bids.layout('bids-examples/eeg_face13');
>> md = bids.query(b, 'metadata');
Error using fieldnames
Invalid input argument of type 'double'. Input must be a structure or a Java or COM object.

Error in get_metadata>update_metadata (line 76)
fn = fieldnames(s2);

Error in get_metadata (line 58)
                meta = update_metadata(meta,bids.util.jsondecode(metafile{i}));

Error in bids.query (line 100)
                                result{end+1} = get_metadata(f);
 
>> 

Looks like it's because bids.util.jsondecode() is returning an empty [] in this case, instead of a struct?

K>> dbstack
  In get_metadata>update_metadata (line 76)
> In get_metadata (line 58)
  In bids.query (line 100)
K>> metafile{i}
ans =
    '/Users/janke/local/repos/bids-matlab-ut/upstream/bids-examples/eeg_face13/task-faceFO_events.json'
K>> bids.util.jsondecode(metafile{i})
ans =
     []
K>> 

And that looks like it's happening because this metadata file in the example contains just an empty JSON array, instead of a JSON object.

$ cat /Users/janke/local/repos/bids-matlab-ut/upstream/bids-examples/eeg_face13/task-faceFO_events.json                        master
[]

Found while running bids-matlab-ut; failed test output here.

If stimulus presentation was the same for all participants, event.tsv files can be set in the root folder (inheritance principle).

But it seems that bids.query fails to find it.

Here is a minimal working example of the bug using the bids-example repo.

This also affects spm_BIDS

BIDS = bids.layout(fullfile(pwd,  'ds114'));

% query for any event file for one task comes back empty
event_file = bids.query(BIDS, 'data', ...
                        'task', 'overtverbgeneration', ...
                        'type', 'events')

event_file =

  0×0 empty cell array

% yet the file is present in the root of this dataset
dir(fullfile(pwd,  'ds114',  'task-overtverbgeneration_events.tsv'))

task-overtverbgeneration_events.tsv  

List what you would need from BIDS-matlab

As discussed in the last meeting (see #59) we will be keeping track of what people would like bids-matlab to do in this issue.

I will keep this top message updated with new ideas from the discussion below.

• a code base that keeps up with new BIDS extensions
• multimodal, not only MRI

Moved to their own issue

• using a schema (see #76)

I want the data structures to be formally described (in json/yaml) and not in the MATLAB code, that representation should be shared with pybids (MATLAB has support for JSON - see here

• document current features (see #77)

documentation is missing: what can the code actually do and not do

• help with reading BIDS model (see #46)

• add basic "write" functions (see #47)

To help with bids compliant filename generation and the creation of json files and data dictionary

• read and parse derivatives (see #50)

Like fMRIprep output

• Initialize derivative data set with a copy of the raw data (see #78)

• Change handling of multi-session dataset (see #63)

• better handling of the metadata (see #79)

reading and representing the metadata and the structure, I know how to deal with the data itself

Hi everyone

I'd like to cite or refer to bids-matlab in a manuscript, and would prefer to refer to a specific release. Would it be possible to create a release for that purpose? I can of course fork the repo and create a release myself, but it would be better to refer to the main code base.

I have created 3 google docs (mostly "pandoc-ed" from the maarkdown in the bids-specs repo) so we can work on:

• code of conduct
• contributing guidelines
• decision making

I have set them in "suggesting" mode and I will work on those to have a draft of some decision-making process before our next meeting. Obvisouly feel free to comment and make suggestions too.

See here.
https://github.com/bids-standard/bids-matlab/runs/1734150473

documentation is missing: what can the code actually do and not do

Add a "features" section in the read explaining what bids-matlab can do.

Also tempted to have a link to the "enhancement" issue label for the people who want to know what is in the pipeline.