Comments (1)
The problem is that the most efficient way to do this is to use forward-mode inside of the loss function and then reverse over that. Using ForwardDiff in here kind of works, but it requires a few workaround and hard to generalize. But Zygote got a forward mode: FluxML/Zygote.jl#503 . It needs a few more things to be fully compatible with standard NNs people want to use for physics-informed neural networks though: FluxML/Zygote.jl#654 . But we are well aware of this, and @Keno has been working on a major improvement to the AD system which make this a lot better, and @DhairyaLGandhi is aware of this use case.
That said, the reason why it's not a huge issue is that the computational complexity of numerical and forward mode is the same, with forward mode just decreasing the number of primal calculations and allowing a bit more SIMD/CSE in some cases. You never see forward mode more than 4x better than the fastest numerical differentiation schemes, usually more around 2x. What's essential for performance is the reverse mode of the loss function, which is already there, since that has a massive complexity change. This is why I haven't made a big push to get "something for now and better for later", and instead am just waiting for the big nested AD changes coming later this year since the actual difference to a user will be rather slim (much slimmer than you might suspect), so it's not worth making a fuss about until special compiler tools allow for faster high order derivatives (which something like PyTorch doesn't do, so that would be something to write home about).
Pinging @KirillZubov since I know he was curious about this detail as well.
But indeed, this is an issue so thanks for opening it so we can formally track it. We'll start updating the public on this more often since there's a lot going on here.
from neuralpde.jl.
Related Issues (20)
- Parameter Estimation in NNODE
- Parameter Estimation formulation in PINNs HOT 5
- Documentation for BayesianPINN and ahmc_bayesian_pinn_pde
- Encountering Precompilation Error Messages When Adding NeuralPDE Package in Julia HOT 3
- Distance functions as a description of complex geometries HOT 1
- Bump CUDA Compat HOT 1
- How can I solve complex-valued ordinary differential equations (ODEs) using neural networks, given limitations with complex data types in libraries like Lux? HOT 7
- Can't get the @named pde_system to work with a ode system HOT 4
- Add support for equations with complex numbers HOT 20
- Neural Adapter Documentation Example Not Working HOT 6
- DGM Tutorial fails HOT 2
- KeyError when solving Parameter Identification problem using NeuralPDE.jl HOT 9
- Linear parabolic example throws an error on new julia install HOT 2
- NeuralPDE slows when using old weights and biases on a new but related ode HOT 3
- NeuralPDE with LuxAMDGPU, Optimize.solve throws error
- GPU Compatibility Issue: Compilation Error with Complex-Valued Data in LuxCUDA Broadcasting Kernel HOT 8
- Tutorial: imposing constraint on PINN solution of Fokker Planck fails HOT 5
- The trial solutions to additional_loss function are different depending on parameters
- API PINO PDE HOT 5
- How to solve SDEs with NeuralPDE?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from neuralpde.jl.