Specify absolute paths in experiment_config.yaml about iohub HOT 4 OPEN

• Check that dir exists. If it does not, show dog and ask if it could be created or cancel exp.
• I'd hdf5 file exists, check it is for current exp, else give dialog asking whether to replace or exit exp.

from iohub.

Here is a proposed new format for specifying the experiments virtual folder structure. This would be in the experiment_config.yaml:

directories:
    - experiment:
        root:
        type: [r,]
        file_types:
            - .yaml
    - results:
        root: D:/results/iohub_experiments/<code>
        type: [w,]
        file_types:
            - .hdf5                
        directories:
            - session:
                root: <session_defaults.code>
                type: [w,]
                file_types: 
                    - .xls
                    - .edf    
    - resources:
        root: resources
        directories:
            - condition_files:
                root: condition_files
                type: [r,]
                file_types: 
                    - .xls                
            - images:
                root: images
                type: [r,]
                file_types: 
                    - .png
                    - .jpeg
            - audio:
                root: audio
                type: [r,]
                file_types: 
                    - .wav
                    - .mp4

Where:

• Each element of the list under 'directories' specifies a virtual directory with the name of the element, so in this example the three 'top level' virtual directories are 'experiment', results', and 'resources'.
• A virtual directory can itself specify 'directories', that would become child folders of that virtual dir.
• The 'root' attribute defines the physical system directory path for the virtual folder:
  • If root is empty, the directory script directory is used.
  • If root is an absolute path, it gets used as such. For example the 'results' v. folder gives an absolute path.
  • If root is a not an absolute path or empty, then it is treated and add the parent path + the current root gives the virtual folders system directory. For example, 'images' would have an expected system path of <experiment_script_path>/resources/images.
  • If a path string contains a <....>, this is replaced by the runtime value of the associated experiment_config parameter. So for example, if the experiment_config.yaml defined the code: parameter to be 'my_exp_1', the results virtual folder would have a path of [experiment_script_path]/results/my_exp_1
• The type attribute tells the experiment if the folder can contain files that it may want to 'read' from or 'write' to.
• The file_types attribute specifies the file extensions expected to be in the given virtual folder.
• When the experiment looks for possible locations to find an image called 'picture.png' for example, then it would look in any virtual folder path path is type 'r' (read) and has '.png' in the list of possible file types.
• Within your experiment, you can specify a virtual folder by using io.vfolders.[folder_name].[folder_name] .... For example, using the above v. folder definition, to get to the images v. folder you could go images_vfolder = io.vfolders.resources.images. You could then say images_vfolder.getFiles() to get all the files in that vfolder, or images_vfolder.getSystemPath() to get the physical path of the v. folder.
• Within your experiment you can also get all the vfolders that a given file type can be in: io.file_vfolders.png would return [images,], since only 1 vfolder is defined as a possible .png file location. If 2 vfolders had been defined to contain png's, the result would be [images1,images2 for example].
• Within your experiment you can also get all the system paths that a given file type can be in: io.file_dirs.png would return ['<experiment_script_path>/resources/images',] for example.
• Finally, all the files for a given type can be retrieved as the absolute path to every file that exists of that type by:
  io.file_paths.png. If the image folder had 3 images in it, called 'p1.png', 'p2.png','p3.png', then the result of that reference would be: ['<experiment_script_path>/resources/images/p1.png','<experiment_script_path>/resources/images/p2.png','<experiment_script_path>/resources/images/p3.png']

Let me know if you see any issues with this, ways to make it simpler, or if you do not think it will meet your requirements. Once I know I will plan implementation.

This will also address the bug about 'session folders being empty' and the 'hdf5 file is split to 2 locations' bug.

The string representation of the above directories config is the following based on the current prototype code:

Processing File:  D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\experiment_Config.yaml

Virtual Folder:
    Name: vfolders
    System Path: D:\ioHub\examples\ioHubAccessDelayTest
    File Type Filter: []
    Type: ['r', 'w']

    Virtual Folder:
        Name: experiment
        System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\
        File Type Filter: ['.yaml']
        Type: ['r']

    Virtual Folder:
        Name: results
        System Path: D:/results/iohub_experiments/<code>
        File Type Filter: ['.hdf5']
        Type: ['w']

        Virtual Folder:
            Name: session
            System Path: D:/results/iohub_experiments/<code>\<session_defaults.code>
            File Type Filter: ['.xls', '.edf']
            Type: ['w']

    Virtual Folder:
        Name: resources
        System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources
        File Type Filter: []
        Type: ['r', 'w']

        Virtual Folder:
            Name: condition_files
            System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\condition_files
            File Type Filter: ['.xls']
            Type: ['r']

        Virtual Folder:
            Name: images
            System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\images
            File Type Filter: ['.png', '.jpeg']
            Type: ['r']

        Virtual Folder:
            Name: audio
            System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\audio
            File Type Filter: ['.wav', '.mp4']
            Type: ['r']

from iohub.

It definitely satisfies the original request. I like the idea of passing runtime vars in to separate data based on session ids or experiment versions. Though specifying folder privileges and file_types is really neat I'm imagining those would stick with the defaults for most uses (or mine at least). If you're looking to simplify, that would be my suggestion.

……………………
Pierce Edmiston
[email protected]
sapir.psych.wisc.edu

On Apr 16, 2013, at 2:43 PM, Sol Simpson wrote:

Here is a proposed new format for specifying the experiments virtual folder structure. This would be in the experiment_config.yaml:

directories:
- experiment:
root:
type: [r,]
file_types:
- .yaml
- results:
root: D:/results/iohub_experiments/
type: [w,]
file_types:
- .hdf5
directories:
- session:
root: <session_defaults.code>
type: [w,]
file_types:
- .xls
- .edf
- resources:
root: resources
directories:
- condition_files:
root: condition_files
type: [r,]
file_types:
- .xls
- images:
root: images
type: [r,]
file_types:
- .png
- .jpeg
- audio:
root: audio
type: [r,]
file_types:
- .wav
- .mp4
Where:

#. Each element of the list under 'directories' specifies a virtual directory with the name of the element, so in this example the three 'top level' virtual directories are 'experiment', results', and 'resources'.
#. A virtual directory can itself specify 'directories', that would become child folders of that virtual dir.
#. The 'root' attribute defines the physical system directory path for the virtual folder:
#. If root is empty, the directory script directory is used.
#. If root is an absolute path, it gets used as such. For example the 'results' v. folder gives an absolute path.
#. If root is a not an absolute path or empty, then it is treated and add the parent path + the current root gives the virtual folders system directory. For example, 'images' would have an expected system path of /resources/images.
#. If a path string contains a <....>, this is replaced by the runtime value of the associated experiment_config parameter. So for example, if the experiment_config.yaml defined the code: parameter to be 'my_exp_1', the results virtual folder would have a path of [experiment_script_path]/results/my_exp_1
#. The type attribute tells the experiment if the folder can contain files that it may want to 'read' from or 'write' to.
#. The file_types attribute specifies the file extensions expected to be in the given virtual folder.
#. When the experiment looks for possible locations to find an image called 'picture.png' for example, then it would look in any virtual folder path path is type 'r' (read) and has '.png' in the list of possible file types.

Within your experiment, you can specify a virtual folder by using io.vfolders.[folder_name].[folder_name] .... For example, using the above v. folder definition, to get to the images v. folder you could go images_vfolder = io.vfolders.resources.images. You could then say images_vfolder.getFiles() to get all the files in that vfolder, or images_vfolder.getSystemPath() to get the physical path of the v. folder.

Within your experiment you can also get all the vfolders that a given file type can be in: io.file_vfolders.png would return [images,], since only 1 vfolder is defined as a possible .png file location. If 2 vfolders had been defined to contain png's, the result would be [images1,images2 for example].

Within your experiment you can also get all the system paths that a given file type can be in: io.file_dirs.png would return ['/resources/images',] for example.

Finally, all the files for a given type can be retrieved as the absolute path to every file that exists of that type by:

io.file_paths.png. If the image folder had 3 images in it, called 'p1.png', 'p2.png','p3.png', then the result of that reference would be: ['/resources/images/p1.png','/resources/images/p2.png','/resources/images/p3.png']

Let me know if you see any issues with this, ways to make it simpler, or if you do not think it will meet your requirements. Once I know I will plan implementation.

This will also address the bug about 'session folders being empty' and the 'hdf5 file is split to 2 locations' bug.

The string representation of the above directories config is the following based on the current prototype code:

Processing File: D:\Dropbox\DEV\OpenPsycho\DEV\OpenPsycho2.7.3.1-32bit\my-code\ioHub\examples\ioHubAccessDelayTest\experiment_Config.yaml

Virtual Folder:
Name: vfolders
System Path: D:\ioHub\examples\ioHubAccessDelayTest
File Type Filter: []
Type: ['r', 'w']

Virtual Folder:
    Name: experiment
    System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\
    File Type Filter: ['.yaml']
    Type: ['r']

Virtual Folder:
    Name: results
    System Path: D:/results/iohub_experiments/<code>
    File Type Filter: ['.hdf5']
    Type: ['w']

    Virtual Folder:
        Name: session
        System Path: D:/results/iohub_experiments/<code>\<session_defaults.code>
        File Type Filter: ['.xls', '.edf']
        Type: ['w']

Virtual Folder:
    Name: resources
    System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources
    File Type Filter: []
    Type: ['r', 'w']

    Virtual Folder:
        Name: condition_files
        System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\condition_files
        File Type Filter: ['.xls']
        Type: ['r']

    Virtual Folder:
        Name: images
        System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\images
        File Type Filter: ['.png', '.jpeg']
        Type: ['r']

    Virtual Folder:
        Name: audio
        System Path: D:\Dropbox\DEV\ioHub\examples\ioHubAccessDelayTest\resources\audio
        File Type Filter: ['.wav', '.mp4']
        Type: ['r']

—
Reply to this email directly or view it on GitHub.

from iohub.

due to time; moving this feature request to 0.8

from iohub.