mstksg / backprop Goto Github PK

View Code? Open in Web Editor NEW

177.0 17.0 22.0 11.31 MB

Heterogeneous automatic differentiation ("backpropagation") in Haskell

Home Page: https://backprop.jle.im

License: BSD 3-Clause "New" or "Revised" License

Haskell 100.00%

backprop graph backpropagation deep-learning neural-network gradient-descent automatic-differentiation

backprop's People

Contributors

Stargazers

Watchers

backprop's Issues

Op between different types

Big fan of your work, I'm trying to understand/port backprop to Scala in https://github.com/fabianmurariu/ad4s

I'm curious if you've come across the issue of operations in between types such as Matrix / Double where the type of Op [Matrix,Double] is (Matrix, Matrix => [Matrix,Double]) however because you're returning product when you multiply Matrix with Double you get Matrix back.

For example (pardon my Haskell, I can barely read it)

-- | 'Op' for division
(/.) :: Matrix a Fractional b Matrix c => Op '[a, b] c
(/.) = op2 $ \x y -> (x / y, \g -> (g/y, -g*x/(y*y)))

The types don't alight because g/y and -gx/(yy) are all [Matrix, Matrix] but you expect [Matrix, Fractional]

What's the strategy for functions with no gradient?

autograd has a few functions with no gradient
https://github.com/HIPS/autograd/blob/e3b525302529d7490769d5c0bcfc7457e24e3b3e/autograd/numpy/numpy_vjps.py#L14

noGrad Op seems to blowup when used so how does one interact with functions that do not have a gradient, floor and ceil come to mind but that is a large list over at autograd?

If you don't remember I'm the guy trying to copy your work into Scala https://github.com/fabianmurariu/ad4s

backprop as functor

Hi,

Did you read Backprop as Functor: A compositional perspective on supervised learning ??

Will you implement the concepts described in this paper in Haskell with backprop?

Hessian computation

It would be nice if there were combinators for computing the Hessian matrix (second partial derivatives).

Why not have an undecidable instance Num a => Backprop a ?

which would be satisfied with the *Num functions?

defining an explicit sum of squared error loss function

Hey @mstksg -- I was trying to write the above loss function, explictily, and made it to the following:

> {-# LANGUAGE ScopedTypeVariables #-}
> {-# LANGUAGE PatternSynonyms #-}
> import qualified Numeric.Backprop as BP
> import qualified Generics.SOP     as SOP
> import Prelude
> import Numeric.LinearAlgebra.Static (L, R, (<.>), konst)
> import GHC.TypeLits (KnownNat)
> import Numeric.Backprop (BPOp, Op, gTuple, (#<~), pattern (:<), (~$))
>
> instance SOP.Generic (R o)
>
> loss :: forall s n .  (KnownNat n) => R n -> BPOp s '[ R n ] Double
> loss target = BP.withInps $ \(outVar :< _) -> do
>   err :< _ <- gTuple #<~ (BP.constVar target - outVar)
>   dot ~$ (err :< BP.only err)

using the definition of dot from the MNIST tutorial:

> dot :: forall n . KnownNat n => Op '[ R n, R n ] Double
> dot = BP.op2' $ \x y ->
>     let
>       back :: Maybe Double -> (R n, R n)
>       back = \case Nothing -> (y, x)
>                    Just g  -> (konst g * y, x * konst g)
>     in
>       (x <.> y, back)

But I'm running into a type error where it seems like the type of R n is getting lost:

QNetworkBackprop.lhs:16:13: error:
    • Couldn't match type ‘hmatrix-0.18.0.0:Internal.Static.Dim
                             n (Data.Vector.Storable.Vector ℝ)’
                     with ‘R n0’
      Expected type: Prod (BVar s '[R n]) '[R n0, R n0]
        Actual type: Prod (BVar s '[R n]) '[a2, a2]
    • In the second argument of ‘(~$)’, namely ‘(err :< err :< Ø)’
      In a stmt of a 'do' block: dot ~$ (err :< err :< Ø)
      In the expression:
        do { err :< _ <- gTuple #<~ (constVar target - outVar);
             dot ~$ (err :< err :< Ø) }
    • Relevant bindings include
        err :: BVar s '[R n] a2 (bound at QNetworkBackprop.lhs:15:5)
        outVar :: BVar s '[R n] a1 (bound at QNetworkBackprop.lhs:14:33)
        target :: R n (bound at QNetworkBackprop.lhs:14:8)
        loss :: R n -> BPOp s '[R n] Double
          (bound at QNetworkBackprop.lhs:14:3)
Failed, modules loaded: none.

Would you be able to point me in the right direction? You should be able to copy/paste this issue's markdown into a literate haskell file to see the error. Thanks!

errors when running "./Build.hs exe"

I am trying to build examples by running "./Build.hs exe". However, build fails.
How to solve it? I am using Ubuntu 16.04

Thank a lot.

# stack (for install)
backprop-0.2.6.1: configure (lib)
Configuring backprop-0.2.6.1...
backprop-0.2.6.1: build (lib)
Preprocessing library for backprop-0.2.6.1..
Building library for backprop-0.2.6.1..
ignoring (possibly broken) abi-depends field for packages
backprop-0.2.6.1: copy/register
Installing library in /home/vincent/worksapce/backprop/.stack-work/install/x86_64-linux/nightly-2018-08-06/8.4.3/lib/x86_64-linux-ghc-8.4.3/backprop-0.2.6.1-1KhuolUzAbT8z2lKXwLAVK
Registering library for backprop-0.2.6.1..
# stack (for samples-exe/extensible-neural)
[1 of 1] Compiling Main             ( samples/extensible-neural.lhs, samples/extensible-neural.o )

samples/extensible-neural.lhs:51:3: error:
    Could not find module ‘Data.List.Split’
    Use -v to see a list of the files searched for.
   |
51 | > import           Data.List.Split
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:60:3: error:
    Could not find module ‘Lens.Micro.TH’
    Perhaps you meant
      Lens.Micro (from microlens-0.4.9.1)
      Lens.Micro.Type (from microlens-0.4.9.1)
    Use -v to see a list of the files searched for.
   |
60 | > import           Lens.Micro.TH
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:63:3: error:
    Could not find module ‘Numeric.LinearAlgebra.Static’
    Use -v to see a list of the files searched for.
   |
63 | > import           Numeric.LinearAlgebra.Static
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:69:3: error:
    Could not find module ‘Numeric.LinearAlgebra’
    Use -v to see a list of the files searched for.
   |
69 | > import qualified Numeric.LinearAlgebra           as HM
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:70:3: error:
    Could not find module ‘System.Random.MWC’
    Perhaps you meant System.Random (from random-1.1)
    Use -v to see a list of the files searched for.
   |
70 | > import qualified System.Random.MWC               as MWC
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:71:3: error:
    Could not find module ‘System.Random.MWC.Distributions’
    Use -v to see a list of the files searched for.
   |
71 | > import qualified System.Random.MWC.Distributions as MWC
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Error when running Shake build system:
* exe
* samples-exe/extensible-neural
user error (Development.Shake.cmd, system command failed
Command: stack ghc --stack-yaml stack.yaml --package mnist-idx --package singletons --package one-liner-instances -- samples/extensible-neural.lhs -o samples-exe/extensible-neural -hidir .build -Wall -O2
Exit code: 1
Stderr:
samples/extensible-neural.lhs:51:3: error:
    Could not find module ‘Data.List.Split’
    Use -v to see a list of the files searched for.
   |
51 | > import           Data.List.Split
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:60:3: error:
    Could not find module ‘Lens.Micro.TH’
    Perhaps you meant
      Lens.Micro (from microlens-0.4.9.1)
      Lens.Micro.Type (from microlens-0.4.9.1)
    Use -v to see a list of the files searched for.
   |
60 | > import           Lens.Micro.TH
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:63:3: error:
    Could not find module ‘Numeric.LinearAlgebra.Static’
    Use -v to see a list of the files searched for.
   |
63 | > import           Numeric.LinearAlgebra.Static
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:69:3: error:
    Could not find module ‘Numeric.LinearAlgebra’
    Use -v to see a list of the files searched for.
   |
69 | > import qualified Numeric.LinearAlgebra           as HM
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:70:3: error:
    Could not find module ‘System.Random.MWC’
    Perhaps you meant System.Random (from random-1.1)
    Use -v to see a list of the files searched for.
   |
70 | > import qualified System.Random.MWC               as MWC
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

samples/extensible-neural.lhs:71:3: error:
    Could not find module ‘System.Random.MWC.Distributions’
    Use -v to see a list of the files searched for.
   |
71 | > import qualified System.Random.MWC.Distributions as MWC
   |   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)

Does generic-lens help backprop?

Just a simple thought. Is using generic-lens beneficial to backprop?
I don't have an answer.

Handling of case splits

While trying to use backprop with ReLU activation functions, I ran into wrong results.
Consider this (admittedly strange) implementation of the identity:

f :: (Reifies s W) => BVar s Double -> BVar s Double
f x = if x < 0 then x else x
f' = BP.gradBP f

It returns a derivative of 0 both for positive and negative inputs:

main :: IO ()
main = do
  putStrLn ("Derivative at -5 = " ++ show (f' (-5)))
  putStrLn ("Derivative at 5 = " ++ show (f' (5)))

Executing the code yields

Derivative at -5 = 0.0
Derivative at 5 = 0.0

Other examples (my original ReLU) produce wrong results, too:

f :: (Reifies s W) => BVar s Double -> BVar s Double
f x = if x < 0 then 0 else x
f' = BP.gradBP f

Any hints are highly appreciated. Are there any specifics we have to keep in mind when
using case splits in the functions to be differentiated?

Backpropagating through "stateful" operations.

I am trying to use this library with a third party C library, but I need to carry some additional context to perform the computations.

My issue boils down to how do I integrate library calls like:

-- Haskell wrapper function signature 
zero :: (Int, Int) -> StateT IO Context Matrix
one :: (Int, Int) -> StateT IO Context Matrix
add :: Matrix -> Matrix -> StateT IO Context Matrix
matMul :: Matrix -> Matrix -> StateT IO Context Matrix

-- FFI signature 
zero :: Ptr Context -> CInt -> CInt -> IO (Ptr Matrix)
one :: Ptr Context -> CInt -> CInt -> IO (Ptr Matrix)
add :: Ptr Context -> Ptr Matrix -> Ptr Matrix -> IO (Ptr Matrix)
matmul :: Ptr Context -> Ptr Matrix -> Ptr Matrix -> IO (Ptr Matrix)

There are no standalone functions to zero, one, or add a Matrix without carrying the Context.

[wiki] add note about wiki being written for backprop >=0.2.0

I still seem to have some issues pushing to the wiki. Maybe write access is for contributors only?

Can it be used with accelerate?

This looks fantastic!

One (possibly) trivial question I have is: can this be used with accelerate? My Haskell got a little bit rusty - I can't tell for sure just from looking at the types...

Thanks!

Jacobians and Op

When dealing with vector-valued functions of vector inputs, e.g. f :: V m -> V n, I'd expect the gradient to be a (n * m) matrix, essentially of a different type than the input of f, but the type of op1 doesn't seem to support this.

What's the idiomatic backprop way of representing this situation ?

[wiki] Fix return type of sumElements

Above Multiple-argument functions the function signatures of sumElements should be:

sumElements :: Reifies s W => BVar s (R n) -> BVar s Double

I tried to submit a PR to the wiki, but I don't think it's possible! I know that page is currently a work in progress, but this one might have slipped through.

backprop now fails to compile with GHC 9.0 (due to vinyl updates?)

I think merging a couple of PRs and releasing would be enough to fix this. Thank you!

mstksg / backprop Goto Github PK

backprop's People

Contributors

Stargazers

Watchers

Forkers

backprop's Issues

Recommend Projects

Recommend Topics

Recommend Org