Comments (3)
Hi @sharpwood,
Thanks for opening the issue!
As you have noticed, this exception only happens when using the CNTK backend. Sadly, the fact that this impossible-to-catch exception has been raised means that something has corrupted the memory of the process, including internal data structures used by the CLR. The CLR has detected this corruption, and therefore decided to stop executing code because it becomes unsafe to continue doing so. As such, it is very likely that whatever is causing this memory corruption is unrelated to the line of code currently highlighted in your screenshot, but instead have only been detected while executing that line.
Since Keras# does not use unsafe operations and therefore never touches memory directly, the only possibility is that the memory corruption is being caused by CNTK itself. The most likely explanation is that Keras# is calling some of the CNTK APIs in an unexpected manner, CNTK is not performing enough argument checks to detect those incorrect calls, and proceeds to execute those incorrectly defined operations, resulting in memory corruption.
The CNTK project should be releasing a new version tomorrow. We can try again with the new version to see if at least the error message improves, but for the time being, I would suggest experimenting with the TensorFlow backend instead.
Regards,
Cesar
from keras-sharp.
use cntk 2.3
from keras-sharp.
Hi @sharpwood,
Thanks for the update. I've just updated to CNTK 2.3 but the issue is still the same. In fact, if you comment the lines in file CNTKFunction.cs:
keras-sharp/Backends/CNTK.CPU/CNTKFunction.cs
Lines 173 to 176 in 8134e8e
and instead replace them by
// this.trainer.TrainMinibatch(input_dict, isSweepEndInarguments: false, computeDevice: DeviceDescriptor.CPUDevice);
updated.Add(c.constant(this.trainer.PreviousMinibatchLossAverage()));
updated.Add(c.constant(this.trainer.PreviousMinibatchEvaluationAverage()));
therefore disabling the mini-batch training but still letting Keras# do all the rest besides that, you will see that the issue disappears.
Keras# will still be preparing the mini-batches, coping memory from C# to CNTK's NDArrayView/Value, build the execution graph, just as before. The exception is that this time we will not let CNTK update the model, and as you see we will not run into memory issues during the training part (the model will never learn anything though, since no weight update is being made).
Regards,
Cesar
from keras-sharp.
Related Issues (20)
- When run SampleApp goes to break mode! HOT 5
- Implement the Dropout layer
- CNTK GPU use? HOT 2
- Add support for saving and loading Keras models HOT 6
- Implement the GRU and LSTM layer
- Error occurred while restoring NuGet packages: The operation failed as details for project SampleApp (GPU) could not be loaded. HOT 3
- Support for .net core 2.0?
- Unity support for keras-sharp?
- New Sample Request
- Is it currently possible to create non-sequential models?
- Run on .Net 4.5
- loading models HOT 2
- Setup procedure (nuget HOT 1
- Fails silently on tensorflowsharp 1.7 HOT 1
- Failure to using Conv2D with Tensorflow (CPU) HOT 1
- Fail to run tests
- Still active? HOT 9
- How do I open this solution?
- Dataset no longer available
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from keras-sharp.