Comments (3)
I didn't post the actual error dialog for the previous message it is:
While the above message is the more frequent one, on the same 6-8 image cycle I sometimes instead get the message below. It's less frequent but seems to happen in lieu of the above message (interchangeably with the above message). As previously, restarting fiji and rerunning with same settings fixes it - rerunning it without restarting Fiji results in the same error.
The specs:
Ubuntu 18.04.1 LTS
i9-9900k 3.6Ghz
32GB RAM
2x EVGA GeForce RTX 2080 Ti 11264MB (not linked)
(one training is running on one GPU via network, macro is running locally on IJ on other GPU)
TileSize setting is: 1396x1396 which results in 6.5GB being used when running - I had it higher before but brought it down thinking that might be the problem. Happens at same frequency since It did with larger tile size / more memory being used.
Any ideas or advice? Thanks!
(Fiji Is Just) ImageJ 2.0.0-rc-69/1.52n; Java 1.8.0_172 [64-bit]; Linux 4.15.0-47-generic; 15761MB of 30000MB (52%)
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedMethodAccessor102.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at ij.macro.Functions.call(Functions.java:4363)
at ij.macro.Functions.getStringFunction(Functions.java:274)
at ij.macro.Interpreter.getStringTerm(Interpreter.java:1407)
at ij.macro.Interpreter.getString(Interpreter.java:1385)
at ij.macro.Interpreter.doStatement(Interpreter.java:324)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.runUserFunction(Interpreter.java:365)
at ij.macro.Interpreter.doStatement(Interpreter.java:276)
at ij.macro.Interpreter.doIf(Interpreter.java:1044)
at ij.macro.Interpreter.doStatement(Interpreter.java:291)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.doStatement(Interpreter.java:315)
at ij.macro.Interpreter.doFor(Interpreter.java:588)
at ij.macro.Interpreter.doStatement(Interpreter.java:297)
at ij.macro.Interpreter.doBlock(Interpreter.java:666)
at ij.macro.Interpreter.runUserFunction(Interpreter.java:365)
at ij.macro.Interpreter.doStatement(Interpreter.java:276)
at ij.macro.Interpreter.doStatements(Interpreter.java:256)
at ij.macro.Interpreter.run(Interpreter.java:152)
at ij.macro.Interpreter.run(Interpreter.java:91)
at ij.macro.Interpreter.run(Interpreter.java:102)
at ij.plugin.Macro_Runner.runMacro(Macro_Runner.java:161)
at ij.IJ.runMacro(IJ.java:148)
at ij.IJ.runMacro(IJ.java:137)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1107)
at net.imagej.legacy.IJ1Helper$3.call(IJ1Helper.java:1103)
at net.imagej.legacy.IJ1Helper.runMacroFriendly(IJ1Helper.java:1054)
at net.imagej.legacy.IJ1Helper.runMacro(IJ1Helper.java:1103)
at net.imagej.legacy.plugin.IJ1MacroEngine.eval(IJ1MacroEngine.java:147)
at org.scijava.script.ScriptModule.run(ScriptModule.java:160)
at org.scijava.module.ModuleRunner.run(ModuleRunner.java:168)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:127)
at org.scijava.module.ModuleRunner.call(ModuleRunner.java:66)
at org.scijava.thread.DefaultThreadService$3.call(DefaultThreadService.java:238)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Macro canceled
at ij.Macro.abort(Macro.java:76)
at ij.IJ.error(IJ.java:691)
at de.unifreiburg.unet.SegmentationJob.run(SegmentationJob.java:737)
at de.unifreiburg.unet.SegmentationJob.processHyperStack(SegmentationJob.java:620)
... 41 more
from unet-segmentation.
So the problematic process runs locally? I am not aware of a memory leak between calls to caffe_unet. Since each call spawns a completely independent process, and the next should only be able to start when the previous one finished, the GPU should be completely free between two segmentations. The only thing I could imagine is that the different calls in the macro try to execute in parallel. This should show up in nvidia-smi top and ps as multiple caffe_unet processes. I am currently on vacation, but will look into it when I'm back in two weeks.
from unet-segmentation.
Hope my previous post didn't put a dent in your vacation :-)
After some testing it seems to be isolated to running the script locally on Fiji on ubuntu as I didn't encounter the problem running it on Windows and remotely tasking caffe on the Ubuntu box. Perhaps it's an issue with my Fiji ubuntu install. I'll follow up if I figure out the underlying cause.
from unet-segmentation.
Related Issues (20)
- Exception in thread "Thread-5" java.lang.IllegalArgumentException HOT 1
- Where to obtain test data?
- Tiling makes tens of thousands of images HOT 3
- Use the weights available at https://lmb.informatik.uni-freiburg.de/resources/opensource/unet/ for Tensorflow 2 (Keras) HOT 6
- Invalid Private Key, RSA HOT 4
- CUDA 11 support HOT 3
- How to change binary path if I am using EC2? HOT 2
- output very weird HOT 5
- CUDA 10.1 support HOT 3
- U-net installation HOT 6
- Finetuning does not 'train' ,aborts HOT 6
- Segmentation does not have good separation between multiple instances of cells
- Writing a macro script to run the Finetune model
- caffe_unet not found HOT 1
- Model/weight check failed
- client loop: send disconnect: broken pipe; SFTP Failure 4 HOT 1
- caffe or caffe_unet issues HOT 7
- MacOs installation issue HOT 1
- Exporting the environment
- Installation Roadblock HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from unet-segmentation.