Git Product home page Git Product logo

agglomeration's People

Contributors

jingpengw avatar jonathanzung avatar nicholasturner1 avatar ranlu avatar tartavull avatar

Stargazers

avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

agglomeration's Issues

stackoverflow error

I am running pipeline using Agglomerator and encountered an stackoverflow error.
Here is an reference of this error:
http://stackoverflow.com/questions/214741/what-is-a-stackoverflowerror

It is the largest image stack I have ever tried for the pipeline. The image size is 2574x2574x164, about 1GB. The affinity map is 9GB. is there any solution to modify the code?

Here is the error message:

mean affinity agglomeration...
Merged to 865 regions
ERROR: LoadError: StackOverflowError:
 in push! at abstractarray.jl:1391 (repeats 16869 times)
 in build_mst at /usr/local/share/julia/site/v0.4/Agglomerator/src/MST.jl:30
 in forward at /usr/local/share/julia/site/v0.4/Agglomerator/src/Process.jl:40
 in aff2segm at /opt/spipe/src/core/aff2segm.jl:69
 in handletask at /opt/spipe/src/main.jl:60
 in main at /opt/spipe/src/main.jl:17
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in process_options at ./client.jl:280
 in _start at ./client.jl:378

problem for parallel usage

I would like to try to pluralize the computations, but the Process was only imported in the master process.

julia -p 2
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "?help" for help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.5.0 (2016-09-19 18:14 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ release
|__/                   |  x86_64-pc-linux-gnu

julia> @everywhere using Agglomerat
Agglomeration Agglomerators
julia> @everywhere using Agglomerat
Agglomeration Agglomerators
julia> @everywhere using Agglomeration
INFO: Recompiling stale cache file /usr/people/jingpeng/.julia/lib/v0.5/Agglomeration.ji for module Agglomeration.
WARNING: replacing module Agglomeration.
WARNING: replacing module Agglomeration.

julia> @ev
@eval       @evalpoly    @everywhere
julia> @ev
@eval       @evalpoly    @everywhere
julia> @everywhere using Process
INFO: Recompiling stale cache file /usr/people/jingpeng/.julia/lib/v0.5/RegionGraphs.ji for module RegionGraphs.
INFO: Recompiling stale cache file /usr/people/jingpeng/.julia/lib/v0.5/Agglomerators.ji for module Agglomerators.

WARNING: deprecated syntax "call(agg::RandomForestAgglomerator, ...)".
Use "(agg::RandomForestAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.AccumulatingAgglomerator, ...)".
Use "(agg::Agglomerators.AccumulatingAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.AccumulatingAgglomerator, ...)".
Use "(agg::Agglomerators.AccumulatingAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.ConstrainedAgglomerator, ...)".
Use "(agg::Agglomerators.ConstrainedAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.ConstrainedAgglomerator, ...)".
Use "(agg::Agglomerators.ConstrainedAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.LinearAgglomerator, ...)".
Use "(agg::Agglomerators.LinearAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.LinearAgglomerator, ...)".
Use "(agg::Agglomerators.LinearAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.MaxAffinityAgglomerator, ...)".
Use "(agg::Agglomerators.MaxAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.MaxAffinityAgglomerator, ...)".
Use "(agg::Agglomerators.MaxAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.MeanAffinityAgglomerator, ...)".
Use "(agg::Agglomerators.MeanAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.MeanAffinityAgglomerator, ...)".
Use "(agg::Agglomerators.MeanAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.RandomForestAgglomerator, ...)".
Use "(agg::Agglomerators.RandomForestAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.RandomForestAgglomerator, ...)".
Use "(agg::Agglomerators.RandomForestAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.SmoothedMeanAffinityAgglomerator, ...)".
Use "(agg::Agglomerators.SmoothedMeanAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::Agglomerators.SmoothedMeanAffinityAgglomerator, ...)".
Use "(agg::Agglomerators.SmoothedMeanAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::TeacherAgglomerator, ...)".
Use "(agg::TeacherAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::GatedAgglomerator, ...)".
Use "(agg::GatedAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::AccumulatingGatedAgglomerator, ...)".
Use "(agg::AccumulatingGatedAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::LinearAgglomerator, ...)".
Use "(agg::LinearAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::AccumulatingAgglomerator, ...)".
Use "(agg::AccumulatingAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::ConstrainedAgglomerator, ...)".
Use "(agg::ConstrainedAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::MeanAffinityAgglomerator, ...)".
Use "(agg::MeanAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::MaxAffinityAgglomerator, ...)".
Use "(agg::MaxAffinityAgglomerator)(...)" instead.

WARNING: deprecated syntax "call(agg::SmoothedMeanAffinityAgglomerator, ...)".
Use "(agg::SmoothedMeanAffinityAgglomerator)(...)" instead.
INFO: Recompiling stale cache file /usr/people/jingpeng/.julia/lib/v0.5/SegmentationMetrics.ji for module SegmentationMetrics.
INFO: Recompiling stale cache file /usr/people/jingpeng/.julia/lib/v0.5/MSTs.ji for module MSTs.
WARNING: Base.CPU_CORES is deprecated.
  likely near /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/MSTs.jl:6
in blosc_filter at /usr/people/jingpeng/.julia/v0.5/HDF5/src/blosc_filter.jl
WARNING: Base.CPU_CORES is deprecated.
  likely near /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/MSTs.jl:6
in blosc_filter at /usr/people/jingpeng/.julia/v0.5/HDF5/src/blosc_filter.jl
WARNING: Base.CPU_CORES is deprecated.
  likely near /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Process.jl:6
in blosc_filter at /usr/people/jingpeng/.julia/v0.5/HDF5/src/blosc_filter.jl
WARNING: Base.CPU_CORES is deprecated.
  likely near /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Process.jl:6
in blosc_filter at /usr/people/jingpeng/.julia/v0.5/HDF5/src/blosc_filter.jl
WARNING: replacing module Process
WARNING: replacing module Process
ERROR: On worker 2:
LoadError: can only precompile from node 1
 in compilecache at ./loading.jl:560
 in require at ./loading.jl:393
 in include_string at ./loading.jl:441
 in include_from_node1 at ./loading.jl:491
 in eval at ./boot.jl:234
 in require at ./loading.jl:415
 in eval at ./boot.jl:234
 in #5 at ./multi.jl:1957
 in #625 at ./multi.jl:1421
 in run_work_thunk at ./multi.jl:1001
 in macro expansion at ./multi.jl:1421 [inlined]
 in #624 at ./event.jl:68
while loading /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Process.jl, in expression starting on line 4
 in #remotecall_fetch#606(::Array{Any,1}, ::Function, ::Function, ::Base.Worker) at ./multi.jl:1070
 in remotecall_fetch(::Function, ::Base.Worker) at ./multi.jl:1062
 in #remotecall_fetch#609(::Array{Any,1}, ::Function, ::Function, ::Int64) at ./multi.jl:1080
 in remotecall_fetch(::Function, ::Int64) at ./multi.jl:1080
 in (::##6#8)() at ./multi.jl:1959

...and 2 other exceptions.

 in sync_end() at ./task.jl:311
 in macro expansion; at ./multi.jl:1968 [inlined]
 in anonymous at ./<missing>:?

compatible with julia-0.5 ?

@jonathanzung is Agglomeration compitable with julia-0.5 ?
I got a function definition error with julia-0.5

using Agglomeration

julia> using Process
INFO: Recompiling stale cache file /usr/people/jingpeng/.julia/lib/v0.5/Iterators.ji for module Iterators.
WARNING: symbol is deprecated, use Symbol instead.
 in depwarn(::String, ::Symbol) at ./deprecated.jl:64
 in symbol(::String, ::Vararg{String,N}) at ./deprecated.jl:30
 in @memoize(::Expr, ::Vararg{Expr,N}) at /usr/people/jingpeng/.julia/v0.5/Memoize/src/Memoize.jl:19
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in require(::Symbol) at ./loading.jl:357
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in require(::Symbol) at ./loading.jl:357
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in require(::Symbol) at ./loading.jl:357
 in eval(::Module, ::Any) at ./boot.jl:234
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46
while loading /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Features.jl, in expression starting on line 47
ERROR: LoadError: LoadError: LoadError: cannot define function volume; it already has a value
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in require(::Symbol) at ./loading.jl:357
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in require(::Symbol) at ./loading.jl:357
 in include_from_node1(::String) at ./loading.jl:426
 in eval(::Module, ::Any) at ./boot.jl:234
 in require(::Symbol) at ./loading.jl:357
 in eval(::Module, ::Any) at ./boot.jl:234
 in macro expansion at ./REPL.jl:92 [inlined]
 in (::Base.REPL.##1#2{Base.REPL.REPLBackend})() at ./event.jl:46
while loading /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Features.jl, in expression starting on line 50
while loading /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Agglomerators.jl, in expression starting on line 8
while loading /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Process.jl, in expression starting on line 4

Support for restricted agglomeration

Add an extra option to agglomeration which takes in a function f(x,y,z)->{True, False} which indicates whether or not a location is in bounds. Agglomeration should only merge along edges which are at least partially in bounds.

reading error of `affinities.jls`

I tried to run the test and get this error. It seems that the affinities.jls file is not complete.

I am using the updated julia-0.5 branch.

ERROR: LoadError: EOFError: read end of file
 in unsafe_read(::IOStream, ::Ptr{UInt8}, ::UInt64) at ./iostream.jl:167
 in read!(::IOStream, ::Array{Float32,4}) at ./io.jl:248
 in read(::IOStream, ::Type{Float32}, ::Tuple{Int64,Int64,Int64,Int64}) at ./io.jl:239
 in deserialize_array(::SerializationState{IOStream}) at ./serialize.jl:700
 in handle_deserialize(::SerializationState{IOStream}, ::Int32) at ./serialize.jl:569
 in load(::String) at /Users/jpwu/.julia/v0.5/Agglomeration/src/Save.jl:30
 in include_from_node1(::String) at ./loading.jl:488
 in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
 in process_options(::Base.JLOptions) at ./client.jl:262
 in _start() at ./client.jl:318
 in _start() at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
while loading /Users/jpwu/.julia/v0.5/Agglomeration/test/runtests.jl, in expression starting on line 9

tried to run it directly in command line.

deserialize(open("deps/datasets/SNEMI3D/ds_test/affinities.jls","r"))
ERROR: EOFError: read end of file
 in unsafe_read(::IOStream, ::Ptr{UInt8}, ::UInt64) at ./iostream.jl:167
 in read!(::IOStream, ::Array{Float32,4}) at ./io.jl:248
 in read(::IOStream, ::Type{Float32}, ::Tuple{Int64,Int64,Int64,Int64}) at ./io.jl:239
 in deserialize_array(::SerializationState{IOStream}) at ./serialize.jl:700
 in handle_deserialize(::SerializationState{IOStream}, ::Int32) at ./serialize.jl:569
 in deserialize(::IOStream) at ./serialize.jl:538

installation documentation?

@jonathanzung is there any installation documentation?
I have some error to setup the aggromerator.

using Process
INFO: Precompiling module Datasets...
ERROR: LoadError: InitError: PyError (:PyImport_ImportModule) <type 'exceptions.ImportError'>
ImportError('No module named error',)

 [inlined code] from /usr/people/jingpeng/.julia/v0.4/PyCall/src/exception.jl:81
 in pyimport at /usr/people/jingpeng/.julia/v0.4/PyCall/src/PyCall.jl:79
 in __init__ at /usr/people/jingpeng/.julia/v0.4/Agglomerator/src/SegmentationMetrics.jl:10
 in _require_from_serialized at loading.jl:84
 in _require_from_serialized at ./loading.jl:109
 in require at ./loading.jl:219
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:284
 in _start at ./client.jl:411
during initialization of module SegmentationMetrics
while loading /usr/people/jingpeng/.julia/v0.4/Agglomerator/src/InputOutput/Datasets.jl, in expression starting on line 5
ERROR: LoadError: Failed to precompile Datasets to /usr/people/jingpeng/.julia/lib/v0.4/Datasets.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:383
 in require at ./loading.jl:250
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:304
 in require at ./loading.jl:243
while loading /usr/people/jingpeng/.julia/v0.4/Agglomerator/src/Process.jl, in expression starting on line 2

an error in production run

@ranlu I am using your branch in production run of zebra fish. I got an wired error:

 From worker 2:  --------start agglomeration-----------
ERROR: LoadError: BoundsError: attempt to access 5-element Array{Float32,1} at index [0]
 in push! at /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/RegionGraphs.jl:272 [inlined]
 in compute_regiongraph(::Array{UInt32,3}, ::Array{Float32,4}, ::Set{UInt32}) at /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/compute_regiongraph.jl:31
 in #forward#1(::Agglomerators.MeanAffinityAgglomerator, ::Float64, ::Void, ::Function, ::Array{Float32,4}, ::Array{UInt32,3}, ::Set{UInt32}) at /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Process.jl:10
 in forward(::Array{Float32,4}, ::Array{UInt32,3}, ::Set{UInt32}) at /usr/people/jingpeng/.julia/v0.5/Agglomeration/src/Process.jl:10
 in ef_agglomeration!(::ChunkFlow.DictChannel, ::DataStructures.OrderedDict{Symbol,Any}, ::DataStructures.OrderedDict{Symbol,Any}, ::DataStructures.OrderedDict{Symbol,Any}) at /usr/people/jingpeng/.julia/v0.5/ChunkFlow/src/edges/agglomeration.jl:36

some workers just stopped at the agglomeration stage and shows nothing. could you take a look? this might also be a problem of upper stage data problem, such as affinity map error.

use http for watershed

currently, ssh protocol do not work in public machines. We should use http, manually git clone using HTTP fixed it.

originally, I got the following errors in aws:

julia  build.jl 
Cloning into 'deps/watershed'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Clone of '[email protected]:seung-lab/watershed.git' into submodule path 'deps/watershed' failed
ERROR: LoadError: failed process: Process(`git submodule update --init`, ProcessExited(1)) [1]
 in run at ./process.jl:531
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in process_options at ./client.jl:280
 in _start at ./client.jl:378

BTW, you can also use the Watershed repo.
just Pkg.add("Watershed") will finish installing.
the EMIRT repo has julia function to replace seg-error

VI Error memory usage

Rewrite the vi error computation so that it takes time linear in the number of non-zero entries in the incidence matrix.

Output the MST from subgraph specified by selected supervoxels

The agglomerator based on mean affinity does not behave consistently for overlapping chunks at the border, when the supervoxels are truncated, the mean affinity estimated can be very different from the adjacent chunk which contains the whole supervoxel. The best solution after discussing with Jonathan is to use the larger affinity and segmentation map to construct the dendrograms, then select the sub-dendrogram contains only the segments in the target segmentation map, and output the MST from the subgraph.

error: memorize not found

I am trying to install Agglomeration in a docker image. Ubuntu 14.04, julia 0.4.5

julia> using Agglomeration
INFO: Precompiling module Agglomeration...

julia> using Pro

Process Profile
julia> using Pro

Process Profile
julia> using Process
INFO: Precompiling module RegionGraphs...
INFO: Recompiling stale cache file /usr/local/share/julia/site/lib/v0.4/DataStructures.ji for module DataStructures.
INFO: Precompiling module Agglomerators...
ERROR: LoadError: ArgumentError: Memoize not found in path
 in compilecache at loading.jl:393
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /usr/local/share/julia/site/v0.4/Agglomeration/src/Features.jl, in expression starting on line 7
ERROR: LoadError: Failed to precompile Features to /usr/local/share/julia/site/lib/v0.4/Features.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:240
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 [inlined code] from none:2
 in anonymous at no file:0
 in process_options at ./client.jl:257
 in _start at ./client.jl:378
while loading /usr/local/share/julia/site/v0.4/Agglomeration/src/Agglomerators.jl, in expression starting on line 8
ERROR: LoadError: Failed to precompile Agglomerators to /usr/local/share/julia/site/lib/v0.4/Agglomerators.ji
 in error at ./error.jl:21
 in compilecache at loading.jl:400
 in require at ./loading.jl:266
 in include at ./boot.jl:261
 in include_from_node1 at ./loading.jl:320
 in require at ./loading.jl:259
while loading /usr/local/share/julia/site/v0.4/Agglomeration/src/Process.jl, in expression starting on line 4

Current Installation Hurdles

As written, I ran into a few issues installing

  • The first attempt complained that 'Memoize' wasn't installed. I'm not sure if that's supposed to be handled in this repo's code somewhere, but I've added it as a dependency to the README just in case.
  • Next attempt's git clone call failed. This doesn't seem to play nicely with private repos as is
  • Somehow, the SNEMI3D files were decompressed properly on one of my attempts, such that trying the installation again resulted in the gzip calls 'not finding' the compressed versions. Re-compressing the files before trying once more fixed the issue. (Ideally this would be checked within build.jl - and I should be able to fix this soon)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.