Comments (9)
More information, please. The call command in your macro would be nice and the log output, especially the caffe_unet call. There are old tricks?
from unet-segmentation.
Sorry for the lack of specificity. The tricks I was referring to was either using the // for windows paths in macros, or just putting the model.h5 in the fiji directory and the caffemodel.h5 in ~/
It seems the issue is that the caffe_unet call isn't respecting my selected tile size but is in fact picking its default (1908x1764) regardless of what tile size, or Memory (MB):=5000 I specify.
A sample call is:
call('de.unifreiburg.unet.SegmentationJob.processHyperStack', 'modelFilename=2d_cell_net_v0-BZ-ishRGB-18FEB.modeldef.h5,weightsFilename=2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5,Tile shape (px):=996x996,gpuId=all available,useRemoteHost=true,hostname=10.0.0.66,port=22,username=administrator,RSAKeyfile=id_rsa,processFolder=cellnet/,average=none,keepOriginal=true,outputScores=false,outputSoftmaxScores=false');
in the log this is:
Establishing SSH connection for '[email protected]:22'
t = 1: scale = 0.003921569, offset = -0.0
Caffe blobs saved to 'C:\Users\bolek\AppData\Local\Temp\unet-0b9ff258-99be-4d53-873c-3908b20ec26a7177983324116306865.h5'
$ sftp "C:\Users\bolek\AppData\Local\Temp\unet-0b9ff258-99be-4d53-873c-3908b20ec26a7177983324116306865.h5" "[email protected]:22:/home/administrator/cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.h5"
$ sftp "C:\Users\bolek\Desktop\UNET\caffemodels\2d_cell_net_v0-BZ-ishRGB-18FEB.modeldef.h5" "[email protected]:22:/home/administrator/cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.modeldef.h5"
caffe_unet tiled_predict -infileH5 cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.h5 -outfileH5 cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.h5 -model cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.modeldef.h5 -weights cellnet/caffemodels/2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5 -iterations 0 -tile_size 1764x1908 -gpu all
I0219 14:07:15.136282 21135 caffe_unet.cpp:320] Use GPU with device ID 0
I0219 14:07:15.356004 21135 tiled_predict.cpp:342] tile_size: (1764,1908)
I0219 14:07:15.356022 21135 tiled_predict.cpp:352] average_mirror: 0
I0219 14:07:15.356025 21135 tiled_predict.cpp:353] average_rotate: 0
and the error:
I0219 14:07:17.470978 21135 tiled_predict.cpp:655] ----> tile (0,0) / (5,5)
F0219 14:07:18.178520 21135 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f7bca48a0cd google::LogMessage::Fail()
@ 0x7f7bca48bf33 google::LogMessage::SendToLog()
@ 0x7f7bca489c28 google::LogMessage::Flush()
@ 0x7f7bca48c999 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f7bcabd58da caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f7bcaa0f2c2 caffe::Blob<>::mutable_gpu_data()
@ 0x7f7bcac32be4 caffe::CuDNNDeconvolutionLayer<>::Forward_gpu()
@ 0x7f7bcab9c112 caffe::Net<>::ForwardFromTo()
@ 0x7f7bcab9c217 caffe::Net<>::Forward()
@ 0x7f7bcac0db3c caffe::TiledPredict<>()
@ 0x555e50576f13 tiled_predict()
@ 0x555e5057554a main
@ 0x7f7bc8a9ab97 __libc_start_main
@ 0x555e50575f7a _start
and exception:
(Fiji Is Just) ImageJ 2.0.0-rc-43/1.52i; Java 1.8.0_25 [64-bit]; Windows 8 6.2; 5478MB of 392867MB (1%)
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at ij.macro.Functions.call(Functions.java:4379)
at ij.macro.Functions.getStringFunction(Functions.java:274)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1366)
at ij.macro.Interpreter.getString(Interpreter.java:1344)
at ij.macro.Interpreter.doStatement(Interpreter.java:304)
at ij.macro.Interpreter.doStatements(Interpreter.java:236)
at ij.macro.Interpreter.run(Interpreter.java:119)
at ij.macro.Interpreter.run(Interpreter.java:90)
at ij.macro.Interpreter.run(Interpreter.java:101)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.IJ.runMacro(IJ.java:147)
at ij.IJ.runMacro(IJ.java:136)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:936)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:933)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:899)
at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:933)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:116)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:163)
at org.scijava.script.ScriptModule.run(ScriptModule.java:174)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:167)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:126)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:65)
at org.scijava.thread.DefaultThreadService$2.call(DefaultThreadService.java:191)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Macro canceled
at ij.Macro.abort(Macro.java:76)
at ij.IJ.error(IJ.java:672)
at de.unifreiburg.unet.SegmentationJob.run(SegmentationJob.java:742)
at de.unifreiburg.unet.SegmentationJob.processHyperStack(SegmentationJob.java:625)
... 31 more
when running from the GUI the log output is:
Establishing SSH connection for '[email protected]:22'
$ sftp "C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b07991061431350630732.tmp" "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b07991061431350630732.tmp"
[email protected] $ rm "/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b07991061431350630732.tmp"
[email protected]$ caffe_unet
Setting caffe_unet binary path to caffe_unet
Searching for caffe
[email protected]$ caffe
$ sftp "C:\Users\bolek\Desktop\UNET\caffemodels\2d_cell_net_v0-BZ-ishRGB-18FEB.modeldef.h5" "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5"
[email protected]$ caffe_unet check_model_and_weights_h5 -model "cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5" -weights "cellnet/caffemodels/2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5" -n_channels 3 -gpu all
t = 1: scale = 0.003921569, offset = -0.0
Caffe blobs saved to 'C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5'
$ sftp "C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5" "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5"
caffe_unet tiled_predict -infileH5 cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5 -outfileH5 cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5 -model cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5 -weights cellnet/caffemodels/2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5 -iterations 0 -tile_size 996x996 -gpu all
$ sftp "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5" "C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5"
[email protected] $ rm "/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5"
[email protected] $ rm "/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5"
Removing C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5
U-Net job finished
from unet-segmentation.
Thanks, that helps! I will try to reproduce and fix the problem.
from unet-segmentation.
Thank you!
Just to clarify, I say "default" tile size as this is what appears in the GUI segmentation window as well. I'm not sure if it's relevant, but I spoke with my student who says she encounters a similar thing with an even wonkier tile shape (something like 1000x12px). I don't have many more details as she hasn't done any systematic testing yet.
from unet-segmentation.
It looks like the "intelligent" tile shape update from the other options, overwrites the user-selected tile-shape. That's of course not as it is supposed to be... sorry for the inconvenience. At the moment I cannot really give a good solution, but am working on the fix.
from unet-segmentation.
I thought it was just shaming me for having only gotten an EVGA RTX 2080TI with a pitiful 11GB VRAM rather than something with the beef the plugin is asking for. I tried downgrading to an older plugin version but there seems to be incompatibility, perhaps I'll downgrade to the publication version of binary/plugin in the mean time.
Thanks for working on fixing this bug, progress often results in a few broken eggs!
from unet-segmentation.
If the plugin required hardware beyond the most recent consumer card, it would be useless for almost everyone :). No it's just a silly bug and I'm currently refactoring the ModelDefinition code to fix this and make it more maintainable in the future.
from unet-segmentation.
The latest update should hopefully fix the bug. In my tests tiling setup both interactively and in Macros was respected as expected.
from unet-segmentation.
Thanks Thorsten, works like a charm!
from unet-segmentation.
Related Issues (20)
- Exception in thread "Thread-5" java.lang.IllegalArgumentException HOT 1
- Where to obtain test data?
- Tiling makes tens of thousands of images HOT 3
- Use the weights available at https://lmb.informatik.uni-freiburg.de/resources/opensource/unet/ for Tensorflow 2 (Keras) HOT 6
- Invalid Private Key, RSA HOT 4
- CUDA 11 support HOT 3
- How to change binary path if I am using EC2? HOT 2
- output very weird HOT 5
- CUDA 10.1 support HOT 3
- U-net installation HOT 6
- Finetuning does not 'train' ,aborts HOT 6
- Segmentation does not have good separation between multiple instances of cells
- Writing a macro script to run the Finetune model
- caffe_unet not found HOT 1
- Model/weight check failed
- client loop: send disconnect: broken pipe; SFTP Failure 4 HOT 1
- caffe or caffe_unet issues HOT 7
- MacOs installation issue HOT 1
- Exporting the environment
- Installation Roadblock HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unet-segmentation.