Unable to parse zipped mcd and txt about imctools HOT 24 CLOSED

from imctools.

Hi there,
Sorry for the inconvenience!
I just updated the example notebook in https://github.com/BodenmillerGroup/ImcSegmentationPipeline/tree/development/scripts to not silently ignore errors but print the error message instead. Could you run this updated notebook and report the error that gets produced?

Thanks!

from imctools.

Thanks for the prompt responses @smaffiol @votti ,

I am using the ImcSegmentationPipeline-development,

I am running on a windows 10 system,

I setup imctools using the steps provided in the jupyter notebook under the cell the IMC preprocessing pipeline for multiplexed image analysis

The error occurs converting the zipped IMC acquisitions to input format,

I suspect something is wrong with the affiliated .txt files but I tried exporting a fresh batch from MCDViewer and checked hat they were similarly formatted to the example datasets

Here is the error,
ERROR:root:Error in ../../IMPx4/Human\1197344-13-2.zip
Traceback (most recent call last):
File "", line 11, in
File "C:\Users\dzamler\AppData\Local\Continuum\anaconda2\envs\imctools\lib\site-packages\imctools\scripts\convertfolder2imcfolder.py", line 57, in convert_folder2imcfolder
imc_fol.write_imc_folder(zipfolder=dozip)
File "C:\Users\dzamler\AppData\Local\Continuum\anaconda2\envs\imctools\lib\site-packages\imctools\io\imcfolderwriter.py", line 68, in write_imc_folder
self.mcd.save_slideimage(sid, out_folder)
File "C:\Users\dzamler\AppData\Local\Continuum\anaconda2\envs\imctools\lib\site-packages\imctools\io\mcdparserbase.py", line 284, in save_slideimage
with open(os.path.join(out_folder, fn_out), 'wb') as f:
OSError: [Errno 22] Invalid argument: '/Users/dzamler/Documents/OMEtiffsLogs\ometiff\1197344-13\1197344-13_s1_slide""'

Please don't hesitate to let me know if you need anything else,

Cheers,

from imctools.

Cool! That helps a lot.
@.txt files: in the end they are just used as a backup in vase if the mcd is corrupted - something that would happen often back in the days. They are optional nowadays - I will try to reflect thia better in the description.

It seems like an issue with how a file path is constructed in a non Windows conform way - I will need to investigate a bit more and et you know if I need more infos.

from imctools.

Were any files generated in the folder: /Users/dzamler/Documents/OMEtiffsLogs\ometiff\1197344-13 at all?

from imctools.

@votti I thought that might be a problem originally as well, but it works with the example dataset, unless zipping the files on windows caused some sort of difference

here is the contents of the folder
Directory of C:\Users\dzamler\Documents\OMEtiffsLogs\ometiff\1197344-13

12/20/2018 11:01 AM

from imctools.

from imctools.

Ah I think I see what the problem is:
Somehow the file ending of the encoded slide image filename is parsed wrongly to be "" .
To better understand why this is: could you copy-paste what the entry ImageFile in the 1197344-13_Slide_meta.csv file is? (in the C:\Users\dzamler\Documents\OMEtiffsLogs\ometiff\1197344-13 folder)

from imctools.

Some of the ome.tiff files channels look okay but some appear to be inverted as far as black and white.

ImageFile =

""

from imctools.

@Imagefile: I think that is the reasons for the SlideImage failing - the instead of having a real empty filename the Mcd contains a "" as the filename - this can be easily fixed.

Some of the ome.tiff files channels look okay but some appear to be inverted as far as black and white.

If true that would be quite worrisome - what kind of image viewer do you use to check if they are valid?
I really would recommend using ImageJ/Fiji to look at the images.

If somehow possible at this point it would be likely easiest you could give me access to an example zipped .mcd file, such that I could check if there is something unusual there.

from imctools.

should I just change the name in the meta .csv? I have no problem providing access what would be the best way?

from imctools.

Changing the name in the 'meta.csv' would not change anything. It really needs a quick fix in the code.
But as said, in the all the relevant parts of the data are already parsed at the time of the error, so this does anyway only affect some metadata images that are likely not so relevant for analysis.

For sharing, please share the zipped .mcd data as well as a zipped version of your ometiff output folder by uploading them to something like Dropbox/Google Drive and send me the link per email. I can also provide you a link to a secure platform from our University to upload if you prefer that.
My email is [email protected]
Cheers!

from imctools.

from imctools.

from imctools.

Hi!
It is important that all the channels that are in the pannel.csv and selected for a analysis stack (e.g. either full or ilastik) were actually present in your acquisition. According to the error, some of your acquisitions do not contain measurements for Ru100 or Nd148, which causes these errors. Thus you need to adapt your pannel.csv to fit the actual acquisitions that you did. Does that make sense?

I will note down that these error messages should be improved.

from imctools.

from imctools.

Hey there!
I reallized that currently there is no real expaination of the pannel.csv - I thus updated the example scripts with comments (in the section where the pannel.csv is first used):
https://github.com/BodenmillerGroup/ImcSegmentationPipeline/blob/development/scripts/imc_preprocessing.ipynb

I hope that helps!

In the end the pannel.csv should really reflect the channels/antibodies that were used for the acquisition - thus I would rather remove channels you did actually not measure.

from imctools.

from imctools.

I assume everything worked?

from imctools.

from imctools.

Hi @votti

I'm having a similar problem using our own data with the panel.csv file. Even after adjusting the panel file for our own antibody panel we encounter the below error. I'm not sure why we would be getting the 'Ru102' and 'Ru100' key errors occuring after these were deleted from your base panel. I did verify that we included all the appropriate channels associated with the MCD file.

Thanks

from imctools.

Hi @veenstje,

could you check please imctools version that is installed?

Best regards,
Anton

from imctools.

from imctools.

Thank you both Anton and Vito for your quick responses. After further investigation I found that I was adjusting the wrong panel.csv file (the one in the cpout folder instead of the correct file in the config folder). I am now able to complete the pipleline. Thank you!

Have a wonderful day

Jesse

from imctools.