denizyuret / knet.jl Goto Github PK
View Code? Open in Web Editor NEWKoç University deep learning framework.
Home Page: https://denizyuret.github.io/Knet.jl/latest
License: Other
Koç University deep learning framework.
Home Page: https://denizyuret.github.io/Knet.jl/latest
License: Other
using Knet
function initweights(atype, hidden, embed, winit)
param = Array(Any, 2*length(hidden))
input = embed
for k = 1:length(hidden)
param[2k-1] = winit*randn(input+hidden[k], 4*hidden[k])
param[2k] = zeros(1, 4*hidden[k])
param[2k][1:hidden[k]] = 1 # forget gate bias
input = hidden[k]
end
return map(p->convert(atype,p), param)
end
# state[2k-1]: hidden, state[2k]: cell
function initstate(atype, hidden, batchsize)
state = Array(Any, 2*length(hidden))
for k = 1:length(hidden)
state[2k-1] = zeros(batchsize,hidden[k])
state[2k] = zeros(batchsize,hidden[k])
end
return map(s->convert(atype,s), state)
end
function lstm(weight, bias, hidden, cell, input)
gates = hcat(input,hidden) * weight .+ bias
hsize = size(hidden,2)
forget = sigm(gates[:,1:hsize])
ingate = sigm(gates[:,1+hsize:2hsize])
outgate = sigm(gates[:,1+2hsize:3hsize])
change = tanh(gates[:,1+3hsize:end])
cell = cell .* forget + ingate .* change
hidden = outgate .* tanh(cell)
return (hidden,cell)
end
function sforw(parameters, states, input)
x = input
for i=1:2:length(states)
(states[i], states[i+1]) = lstm(parameters[i], parameters[i+1], states[i], states[i+1], x)
x = states[i]
end
return x
end
function loss(parameters, states, input)
total = 0
count = 0
for t=1:8
ypred = sforw(parameters[:lstm], states, input)
total += sum(ypred)
count +=1
end
return -total/count
end
function test()
atype = Array{Float32}
embed = 256
hidden = [64 128]
winit = 0.01
batchsize = 2
input = randn(Float32, 2,256) # assume embedding already worked
parameters = initweights(atype, hidden, embed, winit)
states = initstate(atype, hidden, batchsize)
paramdict = Dict()
# hint: if you try working with feedforward model it works
paramdict[:lstm] = parameters
lossval = loss(paramdict, states, input)
println("loss value is $lossval")
lossgradient = grad(loss)
gradient = lossgradient(paramdict, states, input)
end
test()
Need more flexibility with both input and output of both f and g=grad(f). Currently cannot use struct as f's first arg. One idea is to replace argnum with a params method that takes the same args as f and returns an iterator for model parameters, which now can be kept anywhere, in structs, multiple arguments etc. I am not sure how grad can box these, the iterator is not a direct argument of f, its elements will need to get boxed in a way that effects whoever refers to them. g needs to return an iterator of gradients, again not sure how, update! can then use the two iterators in parallel. We also need a way to specify which output of f the gradient should be with respect to, supporting multiple outputs for RNN hidden states etc. May look at helper functions in Python autograd for ideas.
This error also happens with other files in the examples directory. Using julia 0.5.
INFO: Knet using GPU 0
INFO: Loading MNIST...
ERROR: LoadError: UndefVarError: dir not defined
in loaddata() at /Github/Knet.jl/examples/mnist.jl:137
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /Github/Knet.jl/examples/mnist.jl, in expression starting on line 152
Add enough primitives to knetarray to pass the autograd tests converted to knetarrays.
Add tests for cuda kernels in libknet8.
Add code coverage service.
Running lenet.jl on Power machine:
in macro expansion at /localhome/paulito/.julia/v0.5/Knet/src/gpu.jl:10 [inlined]
in (::Knet.##call#117#119)(::Int64, ::Int64, ::Int64, ::Int64, ::Int64, ::Type{T}, ::
Knet.KnetArray{Float32,4}) at /localhome/paulito/.julia/v0.5/Knet/src/cuda44.jl:110
in Knet.PD(::Knet.KnetArray{Float32,4}) at /localhome/paulito/.julia/v0.5/Knet/src/cu
da44.jl:107
in macro expansion at /localhome/paulito/.julia/v0.5/Knet/src/gpu.jl:7 [inlined]
in #pool#92(::Ptr{Void}, ::Float32, ::Float32, ::Array{Any,1}, ::Function, ::Knet.Kne
tArray{Float32,4}) at /localhome/paulito/.julia/v0.5/Knet/src/cuda44.jl:40
in pool(::Knet.KnetArray{Float32,4}) at /localhome/paulito/.julia/v0.5/Knet/src/cuda4
4.jl:39
in predict(::Array{Any,1}, ::Knet.KnetArray{Float32,4}) at /localhome/paulito/.julia/
v0.5/Knet/examples/lenet.jl:83
in #accuracy#8(::Int64, ::Function, ::Array{Any,1}, ::Array{Any,1}) at /localhome/pau
lito/.julia/v0.5/Knet/examples/lenet.jl:122
in main(::String) at /localhome/paulito/.julia/v0.5/Knet/examples/lenet.jl:53
in eval(::Module, ::Any) at ./boot.jl:234
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68WARNING: cudnn.cudnnP
oolingForward error 3
in macro expansion at /localhome/paulito/.julia/v0.5/Knet/src/gpu.jl:10 [inlined]
in #conv4#66(::Ptr{Void}, ::Float32, ::Float32, ::Int64, ::Ptr{Void}, ::Int64, ::Arra
y{Any,1}, ::Function, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}) at /lo
calhome/paulito/.julia/v0.5/Knet/src/cuda44.jl:7
in conv4(::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}) at /localhome/paul
ito/.julia/v0.5/Knet/src/cuda44.jl:6
in predict(::Array{Any,1}, ::Knet.KnetArray{Float32,4}) at /localhome/paulito/.julia/
v0.5/Knet/examples/lenet.jl:83
in #accuracy#8(::Int64, ::Function, ::Array{Any,1}, ::Array{Any,1}) at /localhome/pau
lito/.julia/v0.5/Knet/examples/lenet.jl:122
in main(::String) at /localhome/paulito/.julia/v0.5/Knet/examples/lenet.jl:53
in eval(::Module, ::Any) at ./boot.jl:234
in eval_user_input(::Any, ::Base.REPL.REPLBackend) at ./REPL.jl:64
in macro expansion at ./REPL.jl:95 [inlined]
in (::Base.REPL.##3#4{Base.REPL.REPLBackend})() at ./event.jl:68WARNING: cudnn.cudnnS
etPoolingNdDescriptor error 3
From knet-users.
Using julia0.5 and the Knet obtained from the julia package system (i.e. Pkg.add("Knet")),
the mnist example dies with this error:
INFO: Loading MNIST...
ERROR: LoadError: UndefVarError: dir not defined
in loaddata() at /Users/zilla/zilla/Devl/Julia/Knet/Mnist/mnist.jl:137
in include_from_node1(::String) at ./loading.jl:488
in include_from_node1(::String) at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
in _start() at /Applications/Julia-0.5.app/Contents/Resources/julia/lib/julia/sys.dylib:?
while loading /Users/zilla/zilla/Devl/Julia/Knet/Mnist/mnist.jl, in expression starting on line 152
--------
julia> @doc Knet.dir
No documentation found.
Binding Knet.dir does not exist.
A version cloned from this repo (Knet.jl) works however.
When trying to install Knet I get the error message "CUDA not installed", however, success('nvcc --version')
returns true
. After capturing the exception, the actual error thrown is could not spawn make
. Unfortunatelly, I'm on windows. Any plan to support it?
If I run julia -p 12
and then using Knet
my RAM usage blows up +6GB. If I run just julia
and then using Knet
it consumes ~500MB. So the amount of RAM consumed scales linearly with the number of worker processes. Is this a normal behaviour? I saw AutoGrad
consumed just ~ 50 MB for 12 processes.
Error during compile(:model02). Does not happen if "input" is renamed "xinput". Name conflict?
@knet function model02(x; ftype=Float32, input=784, hidden=32, output=10,
wvec=fill!(CudaArray(ftype,input*hidden+hidden*output),0))
h = wf(x; f=:relu, winit=CudaArray(wvec.ptr, (hidden,input), wvec.dev))
return wf(h; f=:soft, winit=CudaArray(wvec.ptr+hidden*input*sizeof(ftype), (output,hidden), wvec.dev))
end
I'm trying to implement a policy gradient method using Knet. To do this, I need to multiply the loss gradients from by a scalar before doing backprop on the rest of the net. Is there a recommended way to do that through the Knet API? If there is a way to run a part of the computational graph that would do the job as well.
The following error is captured when the package is tested on a windows machine.
julia> Pkg.test("Knet")
INFO: Testing Knet
WARNING: Cannot find libknet
WARNING: Cannot find libcuda
WARNING: Cannot find libcudart
WARNING: Cannot find libcublas
WARNING: Cannot find libcudnn
WARNING: Using the cpu
ERROR: LoadError: LoadError: HDF5 not properly installed. Please run Pkg.build("
HDF5")
in error at error.jl:21
while loading C:\Users\erenay\.julia\v0.4\HDF5\src\plain.jl, in expression start
ing on line 32
while loading C:\Users\erenay\.julia\v0.4\HDF5\src\HDF5.jl, in expression starti
ng on line 3
ERROR: LoadError: Failed to precompile HDF5 to C:\Users\erenay\.julia\lib\v0.4\H
DF5.ji
in error at error.jl:21
while loading C:\Users\erenay\.julia\v0.4\JLD\src\JLD.jl, in expression starting
on line 8
ERROR: LoadError: LoadError: LoadError: Failed to precompile JLD to C:\Users\ere
nay\.julia\lib\v0.4\JLD.ji
in error at error.jl:21
while loading C:\Users\erenay\.julia\v0.4\Knet\src\net.jl, in expression startin
g on line 170
while loading C:\Users\erenay\.julia\v0.4\Knet\src\Knet.jl, in expression starti
ng on line 36
while loading C:\Users\erenay\.julia\v0.4\Knet\test\runtests.jl, in expression s
tarting on line 1
================================[ ERROR: Knet ]=================================
failed process: Process(`'C:\Users\erenay\AppData\Local\Julia-0.4.3\bin\julia' -
-check-bounds=yes --code-coverage=none --color=yes 'C:\Users\erenay\.julia\v0.4\
Knet\test\runtests.jl'`, ProcessExited(1)) [1]
================================================================================
ERROR: Knet had test errors
in error at error.jl:21
julia>
The machine does not have a GPU and it runs Windows 7 Professional Service Pack 1
a = randn(256,10000); x = convert(Knet.KnetArray, a); x[[1,2,3]]
creates the following error:
ERROR: MethodError: no method matching getindex(::Knet.KnetArray{Float64,2}, ::Array{Int64,1})
The following is from Yonatan. A workaround is to use "o..." in the SM definition.
LoadError: unrecognized keyword argument "nrepeat"
@knet function dropwbf(x; hidden=100, dropout=0.5)
h = wbf(x, out=hidden, f=:relu)
return drop(h, pdrop=dropout) ## Prob of dropping
end
@knet function SM(x; dropout=0.5, outdim=20, nlayers=2)
r = repeat(x; frepeat=:dropwbf, nrepeat=nlayers, dropout=0.5, hidden=100)
return wbf(r; out=outdim, f=:soft)
end
Hello All,
I would like to ask if anyone experienced with KNet can explain me, what I am doing wrong and why a simple code like this is incredibly slow
using StatsBase
using Knet
function predict(w,x)
for i=1:2:length(w)-2
x = max(0, w[i]*x .+ w[i+1])
end
return w[end-1]*x .+ w[end]
end
x=randn(Float32,64,100000)
y=sample(1:2,100000)
loss(w,x,y) = mean(max(0,1-y.*predict(w,x)))
function train(k)
w=[randn(k,size(x,1)),zeros(k),randn(k,k),zeros(k),randn(1,k),0.0]
lossgradient=grad(loss)
prms=Sgd(;lr=0.001)
for i in 1:100
idxs=sample(1:size(x,2),1000);
xx=x[:,idxs];
yy=y[idxs]
dw = lossgradient(w,xx,yy)
update!(w,dw,prms)
println(i," ",loss(w,xx,yy))
end
end
train(25)
I got the following error
ERROR: LoadError: MethodError: no method matching sum_outgrads(::Array{Knet.KnetArray{Float32,2},1}, ::Array{Any,1})
in backward_pass(::AutoGrad.Rec{Array{Array{T,1},1}}, ::AutoGrad.Rec{Float32}, ::Array{AutoGrad.Node,1}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:214
...
when I apply mean over an array
function predict_multi(w, xs)
a = []
for x ∈ xs
x = predict(w, x)
push!(a, x)
end
mean(a)
end
The forward pass is OK. I temporarily fix it by adding this line to AutoGrad/src/core.jl, line 453
sum_outgrads(a::AbstractArray,b::AbstractArray)=[sum_outgrads(x,y) for (x,y) in zip(a,b)]
Is my fix correct? If it is correct, could you insert this line to AutoGrad/src/core.jl, line 453?
Otherwise, do I have to define a new @primitive
? Will something like the following work?:
mymean(a) = sum(a) / length(a)
@primitive mymean(x),dy (dy/length(x)) .* ones(size(x))
Hello,
this may not be an actual issue, however I am a quite puzzled because I read between the lines that Knet is not running with the GPU support on Win 10 (it has been tested with GPU support only on linux/osx), however I don't find any explicit claim about this. Am I right?
Is Knet.jl running on Win 10 with GPU support? If yes, where can I find some tips to follow?
Thanks,
So I followed this tutorial for installation of Knet. The tests are OK, but first example does not work:
julia> Pkg.add("Knet")
INFO: Installing AutoGrad v0.0.4
INFO: Installing Knet v0.8.1
INFO: Building Knet
make: 'libknet8.so' is up to date.
INFO: Package database updated
julia> Pkg.build("Knet")
INFO: Building Knet
make: 'libknet8.so' is up to date.
julia> Pkg.test("Knet")
INFO: Testing Knet
INFO: No GPU found, Knet using the CPU
(:epoch,0,:loss,461.77524366804886)
(:epoch,1,:loss,3.3320559275652433)
(:epoch,2,:loss,0.1199614974011731)
(:epoch,3,:loss,0.005568748814864047)
(:epoch,4,:loss,0.0012351227916297506)
(:epoch,5,:loss,0.0010684746173929085)
(:epoch,6,:loss,0.00109180688895586)
(:epoch,7,:loss,0.001015281814462818)
(:epoch,8,:loss,0.0010581459658679027)
(:epoch,9,:loss,0.0010342780204682569)
(:epoch,10,:loss,0.0010601555948864918)
2.051864 seconds (1.83 M allocations: 508.324 MB, 1.79% gc time)
INFO: Knet tests passed
julia> using Knet
julia> @knet function linreg(x)
w = par(dims=(1,13), init=Gaussian(0, 0.1))
b = par(dims=(1,1), init=Constant(0))
return w * x .+ b
end
ERROR: UndefVarError: @knet not defined
using Knet
a=[1;2;3]
a*a'
ERROR: MethodError:
gemm!has no method matching gemm!(::Char, ::Char, ::Int64, ::Array{Int64,2}, ::Array{Int64,2}, ::Int64, ::Array{Int64,2}) in * at linalg/matmul.jl:88 in A_mul_Bc at operators.jl:164
Without using Knet it works
How would you implement, say, a stack RNN in Knet ? (or Mem NN)
I am trying to run examples/charlm.jl and get the following error "ERROR: LoadError: UndefVarError: axpy! not defined":
➜ examples git:(master) julia charlm.jl --data 100.txt --epochs 10 --winit 0.3 --save shakespeare.jld
WARNING: cudart.cudaMemGetInfo error 2
in gpumem() at /home/phuoc/.julia/v0.5/Knet/src/gpu.jl:10
in gpufree at /home/phuoc/.julia/v0.5/Knet/src/gpu.jl:56 [inlined]
in gpu(::Bool) at /home/phuoc/.julia/v0.5/Knet/src/gpu.jl:41
in __init__() at /home/phuoc/.julia/v0.5/Knet/src/Knet.jl:26
in _include_from_serialized(::String) at ./loading.jl:150
in _require_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:187
in _require_search_from_serialized(::Int64, ::Symbol, ::String, ::Bool) at ./loading.jl:217
in require(::Symbol) at ./loading.jl:371
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318INFO: Knet using GPU 0
charlm.jl (c) Emre Yolcu, Deniz Yuret, 2016. Character level language model based on http://karpathy.github.io/2015/05/21/rnn-effectiveness.
opts=(:lr,4.0)(:atype,"KnetArray{Float32}")(:winit,0.3)(:savefile,"shakespeare.jld")(:loadfile,nothing)(:generate,0)(:bestfile,nothing)(:gclip,3.0)(:hidden,[256])(:epochs,10)(:decay,0.9)(:gcheck,0)(:seqlength,100)(:seed,-1)(:embed,256)(:batchsize,128)(:datafiles,Any["100.txt"])(:fast,false)
INFO: Chars read: Tuple{String,Int64}[("100.txt",5589917)]
INFO: 92 unique chars.
(:epoch,0,:loss,5.9404273783175165)
ERROR: LoadError: UndefVarError: axpy! not defined
in #train1#15(::Int64, ::Float64, ::Float64, ::Function, ::Array{Any,1}, ::Array{Any,1}, ::Array{BitArray{2},1}) at /home/phuoc/git/Knet.jl/examples/charlm.jl:165
in (::CharLM.#kw##train1)(::Array{Any,1}, ::CharLM.#train1, ::Array{Any,1}, ::Array{Any,1}, ::Array{BitArray{2},1}) at ./<missing>:0
in macro expansion at ./util.jl:184 [inlined]
in train!(::Array{Any,1}, ::Array{Any,1}, ::Dict{Char,Int64}, ::Dict{Symbol,Any}) at /home/phuoc/git/Knet.jl/examples/charlm.jl:127
in main(::Array{String,1}) at /home/phuoc/git/Knet.jl/examples/charlm.jl:99
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/phuoc/git/Knet.jl/examples/charlm.jl, in expression starting on line 309
After typing using Knet
in the Julia REPL and checking memory usage by nvidia-smi
command, I see that Knet causes small memory usages in all the GPUs available in the environment.
Here is the commands I used:
julia> using Knet
INFO: Knet using GPU 2
$ nvidia-smi
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 59598 C julia 64MiB |
| 1 59598 C julia 64MiB |
| 2 59598 C julia 107MiB |
| 3 59598 C julia 64MiB |
| 4 59598 C julia 64MiB |
| 5 59598 C julia 64MiB |
| 6 59598 C julia 64MiB |
| 7 59598 C julia 64MiB |
+-----------------------------------------------------------------------------+
I was thinking of the update! interface. The requirement of doing individual updates to each numeric weight array is counter-intuitive even if we better documented it. People will naturally want a single update! for the whole model:
(weights, params) = update!(weights, params, grads)
The problem is we do not know the structure of the weights, params, or grads (Array, Tuple, Dict, other Type etc.) What if we require the minimum common interface for all three to be iterators i.e. that they only support for ... in ...
? Then, update! can use zip to iterate over them:
for (w,p,g) in zip(weights, params, grads)
...
end
We probably need to think about what, if anything, to return. If we just guarantee that the returned values are also iterators, would that be sufficient?
Is there any support for sparse matrix in GPU? I tried the following but I guess KnetArray uses full representation:
julia> KnetArray(speye(1000000))
.+ERROR: Out of gpu memory
in Knet.KnetPtr(::Int64) at /home/phuoc/.julia/v0.5/Knet/src/kptr.jl:89
in Type at /home/phuoc/.julia/v0.5/Knet/src/karray.jl:36 [inlined]
in convert(::Type{Knet.KnetArray{Float64,2}}, ::SparseMatrixCSC{Float64,Int64}) at /home/phuoc/.julia/v0.5/Knet/src/karray.jl:54
in Knet.KnetArray{T,N}(::SparseMatrixCSC{Float64,Int64}) at ./sysimg.jl:53
@knet function pooltest4(x)
y = pool(x; window=3, stride=2)
return y
end
gives infersize error with
x = reshape([1.0:6.0...], (6,1,1,1))
KnetArray supports broadcasting for scalar-array, vector-array (vector can have multiple dims but only a single dim > 1), and array-array if they have the same dimensions. General array-array broadcasting for multidimensional arrays is complicated and needs to be implemented. The existing implementations are in cuda12gen.jl.
I am getting the following error "ERROR: LoadError: ccall: could not find function add_32_12 in library". The detail error log is as follows:
➜ examples git:(038c46a) julia lenet.jl
INFO: Knet using GPU 1
INFO: Loading MNIST...
lenet.jl (c) Deniz Yuret, 2016. The LeNet model on the MNIST handwritten digit recognition problem from http://yann.lecun.com/exdb/mnist.
opts=(:seed,-1)(:batchsize,100)(:epochs,3)(:lr,0.1)(:gcheck,0)(:fast,false)
ERROR: LoadError: ccall: could not find function add_32_12 in library
in .+(::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}) at /home/phuoc/.julia/v0.5/Knet/src/cuda12.jl:69
in predict(::Array{Any,1}, ::Knet.KnetArray{Float32,4}) at /home/phuoc/.julia/v0.5/Knet/examples/lenet.jl:83
in #accuracy#8(::Int64, ::Function, ::Array{Any,1}, ::Array{Any,1}) at /home/phuoc/.julia/v0.5/Knet/examples/lenet.jl:122
in main(::Array{String,1}) at /home/phuoc/.julia/v0.5/Knet/examples/lenet.jl:53
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/phuoc/.julia/v0.5/Knet/examples/lenet.jl, in expression starting on line 153
julia> using Knet
WARNING: Base.SparseMatrix is deprecated.
likely near /home/rluser/.julia/v0.5/CUSPARSE/src/CUSPARSE.jl:2
WARNING: Method definition (::Type{Knet._CudaArray})(CUDArt.CudaArray{#T<:Any, #N<:Any}) in module Knet at /home/rluser/.julia/v0.5/Knet/src/util/cudart.jl:127 overwritten at /home/rluser/.julia/v0.5/Knet/src/util/cudart.jl:128.
WARNING: Base.writemime is deprecated.
likely near /home/rluser/.julia/v0.5/Knet/src/util/cudart.jl:133
WARNING: Base.writemime is deprecated.
likely near /home/rluser/.julia/v0.5/Knet/src/util/cudart.jl:133
WARNING: Base.LambdaStaticData is deprecated, use LambdaInfo instead.
likely near /home/rluser/.julia/v0.5/Knet/src/util/deepcopy.jl:21
ERROR: LoadError: LoadError: UndefVarError: TopNode not defined
in include_from_node1(::String) at ./loading.jl:426 (repeats 2 times)
in eval(::Module, ::Any) at ./boot.jl:234
in require(::Symbol) at ./loading.jl:357
while loading /home/rluser/.julia/v0.5/Knet/src/util/deepcopy.jl, in expression starting on line 21
while loading /home/rluser/.julia/v0.5/Knet/src/Knet.jl, in expression starting on line 17
I tried to run optimizers.jl and got a lot of the following warnings:
% julia optimizers.jl
in #conv4x#70(::Ptr{Void}, ::Float32, ::Float32, ::Int64, ::Ptr{Void}, ::Int64, ::Array{Any,1}, ::Function, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}) at /home/phuoc/.julia/v0.5/Knet/src/cuda44.jl:50
in #conv4x#85(::Array{Any,1}, ::Function, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::Knet.KnetArray{Float32,4}) at ./<missing>:0
in #conv4#74(::Array{Any,1}, ::Function, ::Type{AutoGrad.Grad{2}}, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}) at ./<missing>:0
in conv4(::Type{AutoGrad.Grad{2}}, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}) at ./<missing>:0
in backward_pass(::AutoGrad.Rec{Array{Any,1}}, ::AutoGrad.Rec{Float32}, ::Array{AutoGrad.Node,1}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:212
in (::AutoGrad.##gradfun#1#3{Optimizers.#loss,Int64})(::Array{Any,1}, ::Function, ::Array{Any,1}, ::Vararg{Any,N}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:47
in (::AutoGrad.#gradfun#2)(::Array{Any,1}, ::Vararg{Any,N}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:47
in #train#4(::Float64, ::Int64, ::Int64, ::Function, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}) at /home/phuoc/.julia/v0.5/Knet/examples/optimizers.jl:68
in (::Optimizers.#kw##train)(::Array{Any,1}, ::Optimizers.#train, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}) at ./<missing>:0
in macro expansion at /home/phuoc/.julia/v0.5/Knet/examples/optimizers.jl:57 [inlined]
in macro expansion at ./util.jl:184 [inlined]
in main(::Array{String,1}) at /home/phuoc/.julia/v0.5/Knet/examples/optimizers.jl:56
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318WARNING: cudnn.cudnnConvolutionBackwardData error 3
in #conv4x#70(::Ptr{Void}, ::Float32, ::Float32, ::Int64, ::Ptr{Void}, ::Int64, ::Array{Any,1}, ::Function, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}) at /home/phuoc/.julia/v0.5/Knet/src/cuda44.jl:50
in #conv4x#85(::Array{Any,1}, ::Function, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::Knet.KnetArray{Float32,4}) at ./<missing>:0
in #conv4#74(::Array{Any,1}, ::Function, ::Type{AutoGrad.Grad{2}}, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}) at ./<missing>:0
in conv4(::Type{AutoGrad.Grad{2}}, ::Knet.KnetArray{Float32,4}, ::Knet.KnetArray{Float32,4}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}, ::AutoGrad.Rec{Knet.KnetArray{Float32,4}}) at ./<missing>:0
in backward_pass(::AutoGrad.Rec{Array{Any,1}}, ::AutoGrad.Rec{Float32}, ::Array{AutoGrad.Node,1}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:212
in (::AutoGrad.##gradfun#1#3{Optimizers.#loss,Int64})(::Array{Any,1}, ::Function, ::Array{Any,1}, ::Vararg{Any,N}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:47
in (::AutoGrad.#gradfun#2)(::Array{Any,1}, ::Vararg{Any,N}) at /home/phuoc/.julia/v0.5/AutoGrad/src/core.jl:47
in #train#4(::Float64, ::Int64, ::Int64, ::Function, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}) at /home/phuoc/.julia/v0.5/Knet/examples/optimizers.jl:68
in (::Optimizers.#kw##train)(::Array{Any,1}, ::Optimizers.#train, ::Array{Any,1}, ::Array{Any,1}, ::Array{Any,1}) at ./<missing>:0
in macro expansion at /home/phuoc/.julia/v0.5/Knet/examples/optimizers.jl:57 [inlined]
in macro expansion at ./util.jl:184 [inlined]
in main(::Array{String,1}) at /home/phuoc/.julia/v0.5/Knet/examples/optimizers.jl:56
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318. 77.394763 seconds (26.57 M allocations: 1.085 GB, 1.71% gc time)
...
It finally ran but the result seems wrong:
(:epoch,0,:trn,(0.091516666f0,2.3076072f0),:tst,(0.086f0,2.3085766f0))
(:epoch,1,:trn,(0.11236667f0,2.3013859f0),:tst,(0.1135f0,2.3013237f0))
(:epoch,2,:trn,(0.11236667f0,2.301366f0),:tst,(0.1135f0,2.3012974f0))
(:epoch,3,:trn,(0.11236667f0,2.301365f0),:tst,(0.1135f0,2.301296f0))
(:epoch,4,:trn,(0.11236667f0,2.301363f0),:tst,(0.1135f0,2.301292f0))
(:epoch,5,:trn,(0.11236667f0,2.301363f0),:tst,(0.1135f0,2.3012917f0))
(:epoch,6,:trn,(0.11236667f0,2.301362f0),:tst,(0.1135f0,2.3012917f0))
(:epoch,7,:trn,(0.11236667f0,2.3013616f0),:tst,(0.1135f0,2.301291f0))
(:epoch,8,:trn,(0.11236667f0,2.301362f0),:tst,(0.1135f0,2.301291f0))
(:epoch,9,:trn,(0.11236667f0,2.3013618f0),:tst,(0.1135f0,2.3012905f0))
(:epoch,10,:trn,(0.11236667f0,2.3013618f0),:tst,(0.1135f0,2.301291f0))
The other examples ran correctly without warning.
Just wondering, but is there a way to use conv
and pool
without a GPU?
I'm running a windows machine and even though I have a nvidia card installed, I failed to install CUDA. If any of you have tips on how to get this working that would be appreciated.
Thanks!
We should add a summary table in the "Function reference" section of the README, and more detailed docstrings in update.jl.
Hi all,
I am doing research in multiple instance learning and I require two key features:
The paper where I describe the models I am interested is available here:
Discriminative Models for Multi-instance Problems with Tree Structure.
Thanks for help.
Tomas
I have updated Knet with Pkg.clone("https://github.com/denizyuret/Knet.jl")
and got various errors running several examples:
0 ~/.j/v/K/examples git:(master) julia housing.jl
INFO: Knet using GPU 0
housing.jl (c) Deniz Yuret, 2016. Linear regression model for the Housing dataset from the UCI Machine Learning
Repository.
opts=(:seed,-1)(:epochs,20)(:lr,0.1)(:atype,"KnetArray{Float32}")(:test,0.0)(:gcheck,0)(:fast,false)
size(data) = (14,506)
ERROR: LoadError: ccall: could not find function add_32_01 in library
in .+(::Float32, ::Knet.KnetArray{Float32,2}) at /home/phuoc/.julia/v0.5/Knet/src/broadcast.jl:53
in .+(::Knet.KnetArray{Float32,2}, ::Float32) at /home/phuoc/.julia/v0.5/Knet/src/broadcast.jl:154
in (::Housing.#report#6)(::Int64) at /home/phuoc/.julia/v0.5/Knet/examples/housing.jl:91
in main(::Array{String,1}) at /home/phuoc/.julia/v0.5/Knet/examples/housing.jl:95
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/phuoc/.julia/v0.5/Knet/examples/housing.jl, in expression starting on line 110
0 ~/.j/v/K/examples git:(master) julia mnist.jl
INFO: Cloning cache of GZip from https://github.com/JuliaIO/GZip.jl.git
INFO: Installing GZip v0.2.20
INFO: Package database updated
INFO: Knet using GPU 0
mnist.jl (c) Deniz Yuret, 2016. Multi-layer perceptron model on the MNIST handwritten digit recognition problem from http://yann.lecun.com/exdb/mnist.
opts=(:seed,-1)(:batchsize,100)(:hidden,Int64[])(:epochs,10)(:lr,0.5)(:atype,"KnetArray{Float32}")(:gcheck,0)(:winit,0.1)(:fast,false)
INFO: Loading MNIST...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 9680k 100 9680k 0 0 1404k 0 0:00:06 0:00:06 --:--:-- 2243k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1610k 100 1610k 0 0 437k 0 0:00:03 0:00:03 --:--:-- 437k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 28881 100 28881 0 0 41835 0 --:--:-- --:--:-- --:--:-- 41795
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4542 100 4542 0 0 9831 0 --:--:-- --:--:-- --:--:-- 9852
ERROR: LoadError: ccall: could not find function add_32_12 in library
in .+(::Knet.KnetArray{Float32,2}, ::Knet.KnetArray{Float32,2}) at /home/phuoc/.julia/v0.5/Knet/src/broadcast.jl:64
in predict(::Array{Any,1}, ::Knet.KnetArray{Float32,2}) at /home/phuoc/.julia/v0.5/Knet/examples/mnist.jl:32
in accuracy(::Array{Any,1}, ::Array{Any,1}, ::MNIST.#predict) at /home/phuoc/.julia/v0.5/Knet/examples/mnist.jl:64
in (::MNIST.#report#9)(::Int64) at /home/phuoc/.julia/v0.5/Knet/examples/mnist.jl:142
in main(::Array{String,1}) at /home/phuoc/.julia/v0.5/Knet/examples/mnist.jl:146
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/phuoc/.julia/v0.5/Knet/examples/mnist.jl, in expression starting on line 161
0 ~/.j/v/K/examples git:(master) julia lenet
ERROR: could not open file /home/phuoc/.julia/v0.5/Knet/examples/lenet
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
0 ~/.j/v/K/examples git:(master) julia lenet.jl
INFO: Knet using GPU 0
lenet.jl (c) Deniz Yuret, 2016. The LeNet model on the MNIST handwritten digit recognition problem from http://yann.lecun.com/exdb/mnist.
opts=(:seed,-1)(:batchsize,100)(:epochs,3)(:lr,0.1)(:gcheck,0)(:fast,false)
ERROR: LoadError: UndefVarError: MNIST not defined
in main(::Array{String,1}) at /home/phuoc/.julia/v0.5/Knet/examples/lenet.jl:123
in include_from_node1(::String) at ./loading.jl:488
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /home/phuoc/.julia/v0.5/Knet/examples/lenet.jl, in expression starting on line 148
In Mocha.jl, I can build arbitrary graph using Concat Layer
and Split Layer
like this issue. How can I implement this in Knet?
We should define another version of grad(f) so that it returns loss, (and possibly hidden state for rnns) as well as the gradient.
This is the error when libcudart is not found. It is not very informative.
_
_ _ _(_)_ | A fresh approach to technical computing
(_) | (_) (_) | Documentation: http://docs.julialang.org
_ _ _| |_ __ _ | Type "?help" for help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 0.4.3 (2016-01-12 21:37 UTC)
_/ |\__'_|_|_|\__'_| | Official http://julialang.org/ release
|__/ | x86_64-apple-darwin13.4.0
julia> using Knet
WARNING: Cannot find libknet
WARNING: Cannot find libcuda
WARNING: Cannot find libcudart
WARNING: Cannot find libcublas
WARNING: Cannot find libcudnn
ERROR: LoadError: LoadError: error compiling anonymous: could not load library "libcudart"
dlopen(libcudart.dylib, 1): image not found
in include at /opt/homebrew-cask/Caskroom/julia/0.4.3/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
in include_from_node1 at /opt/homebrew-cask/Caskroom/julia/0.4.3/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
in include at /opt/homebrew-cask/Caskroom/julia/0.4.3/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
in include_from_node1 at /opt/homebrew-cask/Caskroom/julia/0.4.3/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
in require at /opt/homebrew-cask/Caskroom/julia/0.4.3/Julia-0.4.3.app/Contents/Resources/julia/lib/julia/sys.dylib
while loading /Users/emre/.julia/v0.4/Knet/src/util/gpu.jl, in expression starting on line 12
while loading /Users/emre/.julia/v0.4/Knet/src/Knet.jl, in expression starting on line 9
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 184
On-line CPU(s) list: 0-151,160-191
Thread(s) per core: 8
Core(s) per socket: 5
Socket(s): 4
NUMA node(s): 4
Model: 8247-42L
CPU max MHz: 3923.0000
Machine:
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 184
On-line CPU(s) list: 0-151,160-191
Thread(s) per core: 8
Core(s) per socket: 5
Socket(s): 4
NUMA node(s): 4
I got this error after this command:
julia charlm.jl --data 100.txt --epochs 10 --winit 0.3 --save shakespeare.jld
ERROR: LoadError: ccall: could not find function add_32_12 in library
in .+(::Knet.KnetArray{Float32,2}, ::Knet.KnetArray{Float32,2}) at /localhome/paulito/.julia/v0.6/Knet/src/cuda12.jl:69
in lstm(::Knet.KnetArray{Float32,2}, ::Knet.KnetArray{Float32,2}, ::Knet.KnetArray{Float32,2}, ::Knet.KnetArray{Float32,2}, ::Knet.KnetArray{Float32,2}) at /localhome/paulito/.julia/v0.6/Knet/examples/charlm.jl:137
in predict(::Array{Any,1}, ::Array{Any,1}, ::Knet.KnetArray{Float32,2}) at /localhome/paulito/.julia/v0.6/Knet/examples/charlm.jl:156
in loss(::Array{Any,1}, ::Array{Any,1}, ::Array{BitArray{2},1}, ::UnitRange{Int64}) at /localhome/paulito/.julia/v0.6/Knet/examples/charlm.jl:169
in map(::CharLM.##15#19{Array{Any,1},Array{Any,1}}, ::Array{Any,1}) at ./essentials.jl:124
in macro expansion at ./util.jl:184 [inlined]
in train!(::Array{Any,1}, ::Array{Any,1}, ::Dict{Char,Int64}, ::Dict{Symbol,Any}) at /localhome/paulito/.julia/v0.6/Knet/examples/charlm.jl:222
in main(::Array{String,1}) at /localhome/paulito/.julia/v0.6/Knet/examples/charlm.jl:95
in include_from_node1(::String) at ./loading.jl:481
in process_options(::Base.JLOptions) at ./client.jl:262
in _start() at ./client.jl:318
while loading /localhome/paulito/.julia/v0.6/Knet/examples/charlm.jl, in expression starting on line 301
KnetArray currently supports scalar and vector reductions only. e.g. for a 4-D array A, sum(A) works, sum(A, (1,3,4)) works, but sum(A, 3) or sum(A, (1,3)) does not yet work. The second argument in sum specifies the dimensions over which summation takes place, the remaining dimensions give the size of the output. We need to figure out how to implement general reductions efficiently. The existing implementations are in cuda20.jl (for scalar reductions) and cuda21.jl (for vector reductions).
Take a look at the examples in Knet/test/conv.jl test script in the latest master.
GPU Memory allocation (converting Array to KnetArray) is really slow, 0.06s for each batch of 64x1000 Float32. 100 batches would take 6 seconds when I convert each batch to KnetArray in a training loop. I saw my GPU utilization was just about 5% through out training. 100 epochs training would take 10 minutes.
If I convert the whole training set to KnetArray prior to training. 100 epoches training takes just 1 minute, my GPU ultilization jumped to 90%. But for a large dataset, there is not enough memory.
My questions are:
I defined module "abc.jl" as follows:
__precompile__()
module abc
export f
using Knet
function f()
x = 1
end
end
Knet produced the following warning:
julia> using abc
INFO: Recompiling stale cache file /home/phuoc/.julia/lib/v0.5/abc.ji for module abc.
WARNING: eval from module Knet to abc:
Expr(:||, Expr(:call, :==, Expr(:ccall, Expr(:tuple, "nvmlInit", "libnvidia-ml")::Any, :UInt32, Expr(:tuple)::Any)::Any, 0)::Any, Expr(:call, :error)::Any)::Any
** incremental compilation may be broken for this module **
WARNING: eval from module Knet to abc:
Expr(:||, Expr(:call, :==, Expr(:ccall, Expr(:tuple, "nvmlDeviceGetCount", "libnvidia-ml")::Any, :UInt32, Expr(:tuple, Expr(:curly, :Ptr, :Cuint)::Any)::Any, Array{UInt32, 1}[0x00000000])::Any, 0)::Any, Expr(:call, :error)::Any)::Any
** incremental compilation may be broken for this module **
INFO: Knet using GPU 1
INFO: Knet using GPU 1
I tried implementing a fairly simple muli-layer perceptron in Knet.jl
using Knet
using MNIST
using ProgressMeter
function onehot(labels; min=0, max=9)
labels = round(Int, labels)
res = zeros(Float64, max-min+1, length(labels))
for (i, label) in enumerate(labels)
res[label-min+1, i] = 1.0
end
return res
end
function train!(model, dataX, labels, loss, batchsize=32, steps=1000)
@assert size(dataX,2) == size(labels,2) "$(size(dataX)) != $(size(labels))"
@assert batchsize <= size(labels, 2)
step = 0
bs = 1
prog = Progress(steps, 0.5)
while step < steps
be = bs + batchsize - 1
step += 1
input = dataX[:, bs:be]
output = labels[:, bs:be]
forw(model, input)
back(model, output, loss)
Knet.update!(model)
ProgressMeter.update!(prog, step)
end
end
function test(model, dataX, labels, loss)
ypred = forw(model, dataX)
lossval = loss(ypred, labels)
return lossval
end
@knet function mlp_softmax(x, input_dim=784, output_dim=10, hidden_dim=64)
W1 = par(init=Gaussian(0,0.001), dims=(hidden_dim,input_dim))
b1 = par(init=Constant(0), dims=(hidden_dim,1))
W2 = par(init=Gaussian(0, 0.001), dims=(output_dim, hidden_dim))
b2 = par(init=Constant(0), dims=(output_dim,1))
y = W2*tanh(W1 * x + b1) + b2
return soft(y)
end
function main()
trainX, trainY = traindata()
testX, testY = testdata()
trainX /= maximum(trainX)
testX /= maximum(testX)
model = compile(:mlp_softmax)
setp(model; ls=0.001, adagrad=true)
@show test(model, trainX, onehot(trainY), softloss)
@show test(model, testX, onehot(testY), softloss)
train!(model, trainX, onehot(trainY), softloss)
@show test(model, trainX, onehot(trainY), softloss)
@show test(model, testX, onehot(testY), softloss)
end
main()
One would expect the test loss value to decrease after training a few steps. But instead it diverges to a larger value:
test(model,trainX,onehot(trainY),softloss) = 2.300456469738548
test(model,testX,onehot(testY),softloss) = 2.30038973959615
Progress: 100% Time: 0:00:06
test(model,trainX,onehot(trainY),softloss) = 5.154692176004155
test(model,testX,onehot(testY),softloss) = 5.0927675134887505
Trying different optimization parameters did not seem to help.
If a gpu is full, Knet produces a memory error while checking the memory of that gpu. If there is an available gpu, it starts without any problem.
WARNING: cudart.cudaMemGetInfo error 2
INFO: Knet using GPU 0
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.