e-koch / filfinder Goto Github PK
View Code? Open in Web Editor NEWDetection of filamentary structure in molecular clouds
Home Page: http://fil-finder.readthedocs.io/en/latest
License: MIT License
Detection of filamentary structure in molecular clouds
Home Page: http://fil-finder.readthedocs.io/en/latest
License: MIT License
Giving branch_thresh
to fil_finder_2D
works, but giving it to fil_finder_2D.analyze_skeletons
does not.
Thanks to Alexander Howard for reporting the issue.
Pruning only uses a single-pass through, and only branches that don't affect the connectivity of a skeleton are eligible to be pruned. This should be done iteratively until there are no branches shorter than the given threshold.
Thanks to Alexander Howard for reporting the issue.
This is an easy way to make the skeleton more robust by removing small spurs. I have an implementation in BaSiCs already: https://github.com/e-koch/BaSiCs/blob/master/basics/iterative_watershed.py#L117
If a mask is already given to fil_finder_2D
, it will only be used if use_existing_mask
is enabled in fil_finder_2D.create_mask
, which means the user can't use fil_finder_2D.run
function.
I think networkx has dropped the graphviz layout option. The call in analyze_skeletons
should be removed to it.
Hi Eric,
What a nice tool ! Exactly what I've been looking for!
I tried FilFinder on one of my favorite filaments (not really to "find", but to get properties automatically). The results are very good from visual inspection. However the returned total intensity is 1.2e-5 while the median brightness is 12. This is obviously wrong, unless you have different units for the two values? I don't know how to attach files here, but the table is here:
Amplitude Error,Branches,Orientation,Lengths,Total Intensity,Width Error,Curvature,FWHM Error,Width,Background Error,Amplitude,Background,Median Brightness,FWHM,Fit Type 0.02470738489337417,3,-0.5256068557493918,1.3387213469544617,1.2238113258610874e-05,0.0007197117146968454,0.6962065802395621,0.0017615915057050487,0.037069003036394065,0.0953009146975467,11.962578119690637,3.3136256476938657,11.932802903734958,0.0839807440745629,g
BTW, in the width fitting, all pixels within the mask are used to build the radial profile, and that means the fitting is performed on the averaged profile - is that right?
Is it possible to tell the fitter to use one of the models (g, l, c, n)?
Thanks,
Ke
When running the example script in the example folder, I got the following error:
AttributeError Traceback (most recent call last)
/usr/local/lib/python3.9/site-packages/FilFinder/examples/example_filament_profile.py in
45 # p.ion()
46 dists, profiles, extents, fit_table =
---> 47 filament_profile(labels == 3, noiseimg, hdr, max_dist=0.14*u.pc,
48 distance=250.*u.pc, bright_unit="", noise=None,
49 verbose=True)
/usr/local/lib/python3.9/site-packages/fil_finder-1.7.2-py3.9.egg/fil_finder/width_profiles/profile_line_width.py in filament_profile(skeleton, image, pixscale, max_dist, distance, num_avg, verbose, bright_unit, noise, fit_profiles)
95 '''
96
---> 97 deg_per_pix = pixscale.to(u.deg) / u.pixel
98
99 if distance is not None:
AttributeError: 'Header' object has no attribute 'to'
With some help from a friend, we figured that the problem is that the function filament_profile expected the pixscale as a variable with a unit, but instead had received the header object. So we changed "hdr" in line 47 to "abs(hdr['cdelt1'])" and it works. This is just a note that perhaps the example script should have this replacement.
Not sure where this is coming from, since I haven't changed any of that code. Maybe a change in numpy?
Traceback (most recent call last):
File "/home/travis/build/e-koch/FilFinder/fil_finder/tests/test_whole.py", line 63, in test_without_rht_branches
test2.find_widths()
File "/home/travis/build/e-koch/FilFinder/fil_finder/filfind_class.py", line 926, in find_widths
self.array_offsets[n], self.imgscale)
File "/home/travis/build/e-koch/FilFinder/fil_finder/width.py", line 454, in radial_profile
range(1, int(nbins) + 1)])
File "/home/travis/miniconda/envs/test/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3084, in median
overwrite_input=overwrite_input)
File "/home/travis/miniconda/envs/test/lib/python2.7/site-packages/numpy/lib/function_base.py", line 2997, in _ureduce
r = func(a, **kwargs)
File "/home/travis/miniconda/envs/test/lib/python2.7/site-packages/numpy/lib/function_base.py", line 3138, in _median
n = np.isnan(part[..., -1])
IndexError: index -1 is out of bounds for axis 0 with size 0
RuntimeError Traceback (most recent call last)
in
----> 1 fil.create_mask(verbose=True, border_masking=False)
~/.local/lib/python3.8/site-packages/fil_finder/filfinder2D.py in create_mask(self, glob_thresh, adapt_thresh, smooth_size, size_thresh, verbose, test_mode, regrid, border_masking, border_kwargs, fill_hole_size, use_existing_mask, save_png)
402
403 med_filter_size = int(round(self.smooth_size.value * ratio))
--> 404 smooth_img = nd.median_filter(masking_img,
405 size=med_filter_size)
406
~/.local/lib/python3.8/site-packages/scipy/ndimage/_filters.py in median_filter(input, size, footprint, output, mode, cval, origin)
1394 >>> plt.show()
1395 """
-> 1396 return _rank_filter(input, 0, size, footprint, output, mode, cval,
1397 origin, 'median')
1398
~/.local/lib/python3.8/site-packages/scipy/ndimage/_filters.py in _rank_filter(input, rank, size, footprint, output, mode, cval, origin, operation)
1260 print(len(fshape))
1261 if len(fshape) != input.ndim:
-> 1262 raise RuntimeError('filter footprint array has incorrect shape.')
1263 for origin, lenf in zip(origins, fshape):
1264 if (lenf // 2 + origin < 0) or (lenf // 2 + origin >= lenf):
RuntimeError: filter footprint array has incorrect shape.
Hello, thank you for this great package
Is there a way to identify the pixels of branches that contain an endpoint?
best
valerian
Gracefully accept pixel and physical units for each parameter.
I am simply trying to reproduce the results of the example script in the FilFinder Tutorial and it gets stuck at:
fil.create_mask(verbose=True, border_masking=False, size_thresh=400 * u.pix**2, glob_thresh=0.0267)
The error is:
astropy.units.core.UnitConversionError: 'Unit("K")' is not a scaled version of 'Unit(dimensionless)'
raise TypeError('only dimensionless scalar quantities can be '
TypeError: only dimensionless scalar quantities can be converted to Python scalars
Algorithm can fail intermittently when looking for "cornerpoints"
194 if len(match)==1:
195 pairs.append([i,j])
--> 196 cornerpts.remove(i);cornerpts.remove(j)
197 if len(cornerpts)>0:
198 for l in cornerpts:
ValueError: list.remove(x): x not in list
I think this may be caused by duplicates of pixels in the cornerpts list.
There are 2 kinds of problems with looped filaments introduced in filfinder2D:
How to avoid losing information contained with loops when, e.g., extracting longest paths?
OS: Linux, Ubuntu 18.04
Environment: Pycharm 2020.3.3
PlantCV Version 3.11.0]
Specifying a threshold in brightness units might be more convenient than percentiles.
It's not straightforward to return a single filament back into the original image shape. A convenience function of the following would be good to add (fil
is a FilFinder2D
instance):
from fil_finder.pixel_ident import recombine_skeletons
# For ith filament in fil.filaments
i = 0
mask_i = recombine_skeletons([fil.filaments[i].skeleton()], [fil.array_offsets[i]], fil.image.shape, 0)
AttributeError Traceback (most recent call last)
in
1 fil.preprocess_image(flatten_percent=95)
2 fil = FilFinder2D(hdu, distance=260 * u.pc)
----> 3 fil.create_mask(verbose=True, border_masking=False, size_thresh=400 * u.pix**2)
~/.local/lib/python3.8/site-packages/fil_finder/filfinder2D.py in create_mask(self, glob_thresh, adapt_thresh, smooth_size, size_thresh, verbose, test_mode, regrid, border_masking, border_kwargs, fill_hole_size, use_existing_mask, save_png)
361
362 # Make a copy of the flattened image
--> 363 flat_copy = self.flat_img.copy()
364
365 # Make the nan mask
AttributeError: 'FilFinder2D' object has no attribute 'flat_img'
Some properties in FilFinder2D are not described in the docs (i.e. FilFinder2D.skeleton_longpath
).
if you pass the Plummer1D model from Radfil to find_widths, there is the need, close to line 978 in filament.py, to declare the variable found_width, and set idx (for the Plummer1D it will be idx=2.
With this fixed in my PC's version of the code, I'm getting an error when issuing a width_fits call after find_widths; could it be because I'm getting some nan values in the widths?:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/astropy/table/operations.py", line 1309, in _vstack
col = col_cls.info.new_like(cols, n_rows, metadata_conflicts, out_name)
File "/usr/local/lib/python3.8/dist-packages/astropy/table/column.py", line 326, in new_like
attrs = self.merge_cols_attributes(cols, metadata_conflicts, name,
File "/usr/local/lib/python3.8/dist-packages/astropy/utils/data_info.py", line 713, in merge_cols_attributes
out['dtype'] = metadata.common_dtype(cols)
File "/usr/local/lib/python3.8/dist-packages/astropy/utils/metadata.py", line 64, in common_dtype
raise tme
astropy.utils.metadata.MergeConflictError: Arrays have incompatible types ['object', 'str256']
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "filfinder2d.py", line 73, in
tmp_table = fil.width_fits(xunit=u.pc)
File "/home/manferna/.local/lib/python3.8/site-packages/fil_finder/filfinder2D.py", line 1037, in width_fits
tab = tab_vstack([tab, add_tab])
File "/usr/local/lib/python3.8/dist-packages/astropy/table/operations.py", line 643, in vstack
out = _vstack(tables, join_type, col_name_map, metadata_conflicts)
File "/usr/local/lib/python3.8/dist-packages/astropy/table/operations.py", line 1313, in _vstack
raise TableMergeError("The '{}' columns have incompatible types: {}"
astropy.table.np_utils.TableMergeError: The 'model_type' columns have incompatible types: ['object', 'str256']
Running the following code with the latest version of FilFinder (1.6) produces the following error. A copy of the mask and image files can be downloaded from here. Any idea what's going on? I think it might have something to do with the lack of padding, but it looks like the pad argument went away in 1.6, right? Thanks!!
from fil_finder import FilFinder2D
image=fits.getdata("x115y25_NH2_60deg.fits")
mask=fits.getdata("x115y25_filmask_60deg.fits")
fils = FilFinder2D(image,beamwidth = 1*u.pix, mask = mask)
fils.medskel(verbose=False)
fils.analyze_skeletons(verbose=False,skel_thresh=15*u.pix)
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-17-f16944ffed01> in <module>()
4 fils = FilFinder2D(image,beamwidth = 1*u.pix, mask = mask)
5 fils.medskel(verbose=False)
----> 6 fils.analyze_skeletons(verbose=False,skel_thresh=15*u.pix)
/anaconda3/lib/python3.6/site-packages/fil_finder/filfinder2D.py in analyze_skeletons(self, prune_criteria, relintens_thresh, nbeam_lengths, branch_nbeam_lengths, skel_thresh, branch_thresh, max_prune_iter, verbose, save_png, save_name)
639 relintens_thresh=relintens_thresh,
640 branch_thresh=self.branch_thresh,
--> 641 max_prune_iter=max_prune_iter)
642
643 self.number_of_filaments = num
/anaconda3/lib/python3.6/site-packages/fil_finder/filament.py in skeleton_analysis(self, image, verbose, save_png, save_name, prune_criteria, relintens_thresh, max_prune_iter, branch_thresh)
218 # The mask and sliced image better have the same shape!
219 if input_image.shape != skel_mask.shape:
--> 220 raise AssertionError("Sliced image shape does not equal the mask "
221 "shape. This should never happen! If you see"
222 " this issue, please report it as a bug!")
AssertionError: Sliced image shape does not equal the mask shape. This should never happen! If you see this issue, please report it as a bug!
Small thing here - My linter is complaining about the code here and here, because we should really be using ==
for string comparison, not is
, right?
if bintype == "log":
vs if bintype is "log"
I looked through the issues and didn't see anyone else bringing this up. I'm happy to submit a PR but I wanted to bring it up here, before I just went ahead and did that.
Please let me know if a quick PR for this would be welcomed!
Just want to point out that some of the alternative FITS header standard keywords, like "CD1_1"
and "CD2_2"
are not taken by FilFinder
, which I think should be fixed.
Add docs page on scaling a test image with the same number of filaments to larger sizes.
Currently does both together. This isn't helpful when the image does not need to be padded.
TypeError Traceback (most recent call last)
in ()
----> 1 fils.create_mask(verbose=True)
D:\Anaconda3\envs\py2\lib\site-packages\fil_finder\filfind_class.pyc in create_mask(self, glob_thresh, adapt_thresh, smooth_size, size_thresh, verbose, test_mode, regrid, border_masking, zero_border, fill_hole_size, use_existing_mask, save_png)
464
465 smooth_img = nd.median_filter(masking_img,
--> 466 size=round(self.smooth_size*ratio))
467
468 # Set the border to zeros for the adaptive thresholding. Avoid border
D:\Anaconda3\envs\py2\lib\site-packages\scipy\ndimage\filters.pyc in median_filter(input, size, footprint, output, mode, cval, origin)
1280 """
1281 return _rank_filter(input, 0, size, footprint, output, mode, cval,
-> 1282 origin, 'median')
1283
1284
D:\Anaconda3\envs\py2\lib\site-packages\scipy\ndimage\filters.pyc in _rank_filter(input, rank, size, footprint, output, mode, cval, origin, operation)
1163 raise RuntimeError("no footprint or filter size provided")
1164 sizes = _ni_support._normalize_sequence(size, input.ndim)
-> 1165 footprint = numpy.ones(sizes, dtype=bool)
1166 else:
1167 footprint = numpy.asarray(footprint, dtype=bool)
D:\Anaconda3\envs\py2\lib\site-packages\numpy\core\numeric.pyc in ones(shape, dtype, order)
190
191 """
--> 192 a = empty(shape, dtype, order)
193 multiarray.copyto(a, 1, casting='unsafe')
194 return a
TypeError: 'float' object cannot be interpreted as an index
I'm trying to find the width information on filaments, other methods were ok, but no matter what, find_widths() gives error.
# there is a skeleton length reading package
from fil_finder import FilFinder2D
import astropy.units as u
fil = FilFinder2D(processed, distance=250 * u.pc, mask=skeleton)
fil.medskel(verbose=False)
fil.analyze_skeletons(branch_thresh=0* u.pix, skel_thresh=0 * u.pix, prune_criteria='length')
f = fil.filaments[0]
plt.imshow(f.skeleton(pad_size=10))
fil.exec_rht()
fil.find_widths(max_dist=0.2 * u.pc)
AttributeError Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_34180/1915991156.py in <module>
5 #f.plot_radial_profile(xunit=u.pc)
6 fil.exec_rht()
----> 7 fil.find_widths(max_dist=0.2 * u.pc)
C:\ProgramData\Anaconda3\envs\cv\lib\site-packages\fil_finder\filfinder2D.py in find_widths(self, max_dist, pad_to_distance, fit_model, fitter, try_nonparam, use_longest_path, add_width_to_length, deconvolve_width, fwhm_function, chisq_max, verbose, save_png, save_name, xunit, **kwargs)
973 print("Filament: %s / %s" % (n + 1, self.number_of_filaments))
974
--> 975 fil.width_analysis(self.image, all_skeleton_array=self.skeleton,
976 max_dist=max_dist,
977 pad_to_distance=pad_to_distance,
C:\ProgramData\Anaconda3\envs\cv\lib\site-packages\fil_finder\filament.py in width_analysis(self, image, all_skeleton_array, max_dist, pad_to_distance, fit_model, fitter, try_nonparam, use_longest_path, add_width_to_length, deconvolve_width, beamwidth, fwhm_function, chisq_max, **kwargs)
762
763 # Convert quantities to pixel units.
--> 764 max_dist = self._converter.to_pixel(max_dist).value
765 pad_to_distance = self._converter.to_pixel(pad_to_distance).value
766
C:\ProgramData\Anaconda3\envs\cv\lib\site-packages\fil_finder\base_conversions.py in to_pixel(self, value)
206 return value.to(u.pix, equivalencies=self.angular_equiv)
207 elif value.unit.is_equivalent(u.pc):
--> 208 return value.to(u.pix, equivalencies=self.physical_equiv)
209 else:
210 raise u.UnitConversionError("value has units of {}. It must have "
C:\ProgramData\Anaconda3\envs\cv\lib\site-packages\fil_finder\base_conversions.py in physical_equiv(self)
186 def physical_equiv(self):
187 if not hasattr(self, "_distance"):
--> 188 raise AttributeError("No distance has not been given.")
189
190 return [(u.pix, self.distance.unit,
AttributeError: No distance has not been given.
When separating each filament into an array, the edges are padded by 10 pixels for the distance transform to be performed. It's possible for the padded arrays to exceed the size of the original array.
The corners used to locate where an individual filament array is located in the original image will be wrong if the coordinates of the skeleton pixels, plus the pad size, is less than zero: https://github.com/e-koch/FilFinder/blob/master/fil_finder/pixel_ident.py#L82. The derived parameters will then all be incorrect. The default pad size is skeleton_pad_size=1
, so any filaments that have this issue will be offset by 1 pixel if the default settings are used. Typically the skeletons never reach the edge with border_masking=True
, which should be used for most observed data.
A temporary fix is to set pad_size=1
in fil_finder_2D
, which will add a 1 pixel pad on the image used to create the mask and skeletons. A larger pad size is not typically needed, but if it is, make sure skeleton_pad_size == pad_size
in fil_finder_2D
.
Thanks to Remy Indebetouw for reporting the issue.
The docs don't clearly explain that the mask, skeleton, longest path skeleton, and model image are saved as extensions in the same FITS file.
May need to save branch properties by themselves in another format.
Hi, I am working on a skeleton of 3D mesh. I saved the trimesh.path3D to trimesh.path2D as a dxf
file. Then I used online convertor to convert dxf
to png
. Next, I used the fil_finder to find the largest path. However, when I plot the final result, it is all black without any path or original skeleton. Could someone help me solve this issue? I attached the source image file to this issue.
import numpy as np
import cv2
import matplotlib.pyplot as plt
from fil_finder import FilFinder2D
import astropy.units as u
skeleton = cv2.imread("/Users/mikamixiao/Desktop/test.png", 0) #in numpy array format
fil = FilFinder2D(skeleton, distance=250 * u.pc, mask=skeleton)
fil.preprocess_image(flatten_percent=85)
fil.create_mask(border_masking=True, verbose=False,
use_existing_mask=True)
fil.medskel(verbose=False)
plt.imshow(fil.skeleton, cmap='gray', origin='lower')
fil.analyze_skeletons(branch_thresh=40 * u.pix, prune_criteria='length', skel_thresh=10*u.pix)
# Show the longest path
plt.imshow(fil.skeleton, cmap='gray', origin='lower')
plt.contour(fil.skeleton_longpath, colors='r')
plt.axis('off')
plt.show()
Here is the result I get from above code:
By the way, is there anyway to do the branch pruning on 3D path instead of a 2D png image?
Add option to find_widths
When running FilFinder (version 1.6) I got the following AttributeError message:
File "/usr/local/lib/python3.7/dist-packages/fil_finder/filfinder2D.py", line 641, in analyze_skeletons
max_prune_iter=max_prune_iter)
File "/usr/local/lib/python3.7/dist-packages/fil_finder/filament.py", line 286, in skeleton_analysis
if prev_G.node == G[0].node:
AttributeError: 'Graph' object has no attribute 'node'
Changing line 286 in filament.py from
if prev_G.node == G[0].node:
to
if prev_G.nodes == G[0].nodes:
solved the problem for me.
See also https://networkx.github.io/documentation/stable/release/release_2.4.html#deprecations
Need to bring in astropy_helpers
so this is less of a pain.
I don't think values associated with the key "number" was typically generated for branch_properties
. This is an issue when it comes to executing the following line:
FilFinder/fil_finder/length.py
Line 600 in 81b6012
Note: the "number" values are not a part of the branch_properties
returned by init_lengths()
:
FilFinder/fil_finder/length.py
Line 127 in 81b6012
Hello and thank you for this very useful piece of software.
I would just like to point out that the version available in PyPI is outdated (v1.2.2 from 2015) when it appears that the latest one on this repository is v1.7 . This may be confusing since both the README and the docs indicates pip as way to install FilFinder.
Best,
AssertionError: Sliced image shape does not equal the mask shape. This should never happen! If you see this issue, please report it as a bug!
fil = FilFinder2D(thresh, mask=thresh, distance= 100*u.pc)
fil.create_mask(border_masking=True, verbose=False, use_existing_mask=True)
fil.preprocess_image(flatten_percent=100)
fil.medskel(verbose=False)
fil.analyze_skeletons()
This error appears when pip installing fil_finder into a new environment. Here is the full error log.
WARNING: Discarding https://files.pythonhosted.org/packages/db/b4/a0739e22bd23b4e8405efca8b414537144c5e0fc405243f897a581e9b0f9/fil_finder-1.3.tar.gz#sha256=6458d1f2d84b723dd96dc07843f8dd66e839ff0f6931312cc6f2b66b497dc1af (from https://pypi.org/simple/fil-finder/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement fil-finder (from versions: 1.3, 1.4, 1.5, 1.6, 1.7)
ERROR: No matching distribution found for fil-finder
Hello. The Astropy Project has decided to retire pytest-openfiles, a plugin to detect open files in pytest runs. We see that you are using it in this repo. While you may continue to use it, please be advised that it will no longer be maintained going forward. See https://github.com/astropy/pytest-openfiles#important-retirement-roadmap for more information. We apologize for any inconvenienced cause.
If this issue was opened in error and is irrelevant, feel free to close.
Thank you for your patience.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.