Macro execution of segmentation results in GPU memory error about unet-segmentation HOT 9 CLOSED

More information, please. The call command in your macro would be nice and the log output, especially the caffe_unet call. There are old tricks?

from unet-segmentation.

Sorry for the lack of specificity. The tricks I was referring to was either using the // for windows paths in macros, or just putting the model.h5 in the fiji directory and the caffemodel.h5 in ~/

It seems the issue is that the caffe_unet call isn't respecting my selected tile size but is in fact picking its default (1908x1764) regardless of what tile size, or Memory (MB):=5000 I specify.

A sample call is:
call('de.unifreiburg.unet.SegmentationJob.processHyperStack', 'modelFilename=2d_cell_net_v0-BZ-ishRGB-18FEB.modeldef.h5,weightsFilename=2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5,Tile shape (px):=996x996,gpuId=all available,useRemoteHost=true,hostname=10.0.0.66,port=22,username=administrator,RSAKeyfile=id_rsa,processFolder=cellnet/,average=none,keepOriginal=true,outputScores=false,outputSoftmaxScores=false');

in the log this is:
Establishing SSH connection for '[email protected]:22'
t = 1: scale = 0.003921569, offset = -0.0
Caffe blobs saved to 'C:\Users\bolek\AppData\Local\Temp\unet-0b9ff258-99be-4d53-873c-3908b20ec26a7177983324116306865.h5'
$ sftp "C:\Users\bolek\AppData\Local\Temp\unet-0b9ff258-99be-4d53-873c-3908b20ec26a7177983324116306865.h5" "[email protected]:22:/home/administrator/cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.h5"
$ sftp "C:\Users\bolek\Desktop\UNET\caffemodels\2d_cell_net_v0-BZ-ishRGB-18FEB.modeldef.h5" "[email protected]:22:/home/administrator/cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.modeldef.h5"
caffe_unet tiled_predict -infileH5 cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.h5 -outfileH5 cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.h5 -model cellnet/unet-0b9ff258-99be-4d53-873c-3908b20ec26a.modeldef.h5 -weights cellnet/caffemodels/2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5 -iterations 0 -tile_size 1764x1908 -gpu all
I0219 14:07:15.136282 21135 caffe_unet.cpp:320] Use GPU with device ID 0
I0219 14:07:15.356004 21135 tiled_predict.cpp:342] tile_size: (1764,1908)
I0219 14:07:15.356022 21135 tiled_predict.cpp:352] average_mirror: 0
I0219 14:07:15.356025 21135 tiled_predict.cpp:353] average_rotate: 0

and the error:
I0219 14:07:17.470978 21135 tiled_predict.cpp:655] ----> tile (0,0) / (5,5)
F0219 14:07:18.178520 21135 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory
*** Check failure stack trace: ***
@ 0x7f7bca48a0cd google::LogMessage::Fail()
@ 0x7f7bca48bf33 google::LogMessage::SendToLog()
@ 0x7f7bca489c28 google::LogMessage::Flush()
@ 0x7f7bca48c999 google::LogMessageFatal::~LogMessageFatal()
@ 0x7f7bcabd58da caffe::SyncedMemory::mutable_gpu_data()
@ 0x7f7bcaa0f2c2 caffe::Blob<>::mutable_gpu_data()
@ 0x7f7bcac32be4 caffe::CuDNNDeconvolutionLayer<>::Forward_gpu()
@ 0x7f7bcab9c112 caffe::Net<>::ForwardFromTo()
@ 0x7f7bcab9c217 caffe::Net<>::Forward()
@ 0x7f7bcac0db3c caffe::TiledPredict<>()
@ 0x555e50576f13 tiled_predict()
@ 0x555e5057554a main
@ 0x7f7bc8a9ab97 __libc_start_main
@ 0x555e50575f7a _start

and exception:

(Fiji Is Just) ImageJ 2.0.0-rc-43/1.52i; Java 1.8.0_25 [64-bit]; Windows 8 6.2; 5478MB of 392867MB (1%)

java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at ij.macro.Functions.call(Functions.java:4379)
at ij.macro.Functions.getStringFunction(Functions.java:274)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1366)
at ij.macro.Interpreter.getString(Interpreter.java:1344)
at ij.macro.Interpreter.doStatement(Interpreter.java:304)
at ij.macro.Interpreter.doStatements(Interpreter.java:236)
at ij.macro.Interpreter.run(Interpreter.java:119)
at ij.macro.Interpreter.run(Interpreter.java:90)
at ij.macro.Interpreter.run(Interpreter.java:101)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.IJ.runMacro(IJ.java:147)
at ij.IJ.runMacro(IJ.java:136)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:936)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:933)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:899)
at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:933)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:116)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:163)
at org.scijava.script.ScriptModule.run(ScriptModule.java:174)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:167)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:126)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:65)
at org.scijava.thread.DefaultThreadService$2.call(DefaultThreadService.java:191)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Macro canceled
at ij.Macro.abort(Macro.java:76)
at ij.IJ.error(IJ.java:672)
at de.unifreiburg.unet.SegmentationJob.run(SegmentationJob.java:742)
at de.unifreiburg.unet.SegmentationJob.processHyperStack(SegmentationJob.java:625)
... 31 more

when running from the GUI the log output is:
Establishing SSH connection for '[email protected]:22'
$ sftp "C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b07991061431350630732.tmp" "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b07991061431350630732.tmp"
[email protected] $ rm "/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b07991061431350630732.tmp"
[email protected]$ caffe_unet
Setting caffe_unet binary path to caffe_unet
Searching for caffe
[email protected]$ caffe
$ sftp "C:\Users\bolek\Desktop\UNET\caffemodels\2d_cell_net_v0-BZ-ishRGB-18FEB.modeldef.h5" "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5"
[email protected]$ caffe_unet check_model_and_weights_h5 -model "cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5" -weights "cellnet/caffemodels/2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5" -n_channels 3 -gpu all
t = 1: scale = 0.003921569, offset = -0.0
Caffe blobs saved to 'C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5'
$ sftp "C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5" "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5"
caffe_unet tiled_predict -infileH5 cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5 -outfileH5 cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5 -model cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5 -weights cellnet/caffemodels/2d_cell_net_v0-BZ-ishRGB-18FEB.caffemodel.h5 -iterations 0 -tile_size 996x996 -gpu all
$ sftp "[email protected]:22:/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5" "C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5"
[email protected] $ rm "/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.modeldef.h5"
[email protected] $ rm "/home/administrator/cellnet/unet-2a225514-4f46-453c-9dfe-f0fd6bf322b0.h5"
Removing C:\Users\bolek\AppData\Local\Temp\unet-2a225514-4f46-453c-9dfe-f0fd6bf322b01168548472581875004.h5
U-Net job finished

from unet-segmentation.

Thanks, that helps! I will try to reproduce and fix the problem.

from unet-segmentation.

Thank you!
Just to clarify, I say "default" tile size as this is what appears in the GUI segmentation window as well. I'm not sure if it's relevant, but I spoke with my student who says she encounters a similar thing with an even wonkier tile shape (something like 1000x12px). I don't have many more details as she hasn't done any systematic testing yet.

from unet-segmentation.

It looks like the "intelligent" tile shape update from the other options, overwrites the user-selected tile-shape. That's of course not as it is supposed to be... sorry for the inconvenience. At the moment I cannot really give a good solution, but am working on the fix.

from unet-segmentation.

I thought it was just shaming me for having only gotten an EVGA RTX 2080TI with a pitiful 11GB VRAM rather than something with the beef the plugin is asking for. I tried downgrading to an older plugin version but there seems to be incompatibility, perhaps I'll downgrade to the publication version of binary/plugin in the mean time.

Thanks for working on fixing this bug, progress often results in a few broken eggs!

from unet-segmentation.

If the plugin required hardware beyond the most recent consumer card, it would be useless for almost everyone :). No it's just a silly bug and I'm currently refactoring the ModelDefinition code to fix this and make it more maintainable in the future.

from unet-segmentation.

The latest update should hopefully fix the bug. In my tests tiling setup both interactively and in Macros was respected as expected.

from unet-segmentation.

Thanks Thorsten, works like a charm!

from unet-segmentation.