sebgao / ctensor Goto Github PK

I guess because you transpose the tensor in forward but why not transpose again in backward?I can not understand what would happen dilation=2.It seems strange when the params of as_stride of numpy is in such a situation. istrides = list(istrides+istrides[-2:])
istrides[2] *= stride[0]
istrides[3] *= stride[1]
istrides[4] *= dilation[0]
istrides[5] *= dilation[1]
return np.lib.stride_tricks.as_strided(input,
(isize[0], isize[1], H,
W, ksize[0], ksize[1]),
istrides,
writeable=writeable,
)`

Thank you for the awesome code!

I am little confused about the batch_conv2d_weight_backward_f function

ctensor/operator.py: batch_conv2d_weight_backward_f

def batch_conv2d_weight_backward_f(kernel, input, stride=(1, 1)):
    '''kernel is result tensor grad, input is original tensor'''
    B, C, H, W = kernel.shape
    x = im2bchwkl(input, kernel.shape[-2:], dilation=stride)
    return np.tensordot(x, kernel, [(0, 4, 5), (0, 2, 3)]).transpose(0, 3, 1, 2)*(1.0/(B*H*W))

1. Why is the argument dilation of im2bchwkl set to stride ?

2. Is *(1.0/(B*H*W)) necessary in the gradient of weights ?

According to the above equation, it seems np.tensordot(x, kernel, [(0, 4, 5), (0, 2, 3)]).transpose(0, 3, 1, 2) can get the final results.

Although the implementation of strided convolution is complete, pooling methods such as maxpool and avgpool are still problems.

I am trying following codes. The correct answer should be 6, but it gives 7. I think it is due to the gradient flow does not propgrate propably. Note here I use the variable xa two times. And that is where is the problem is? If I rename one of them as xb, the result would be correct. Andyway, ctensor is a nice library for studying the autodifferentitation.

import numpy as np
from ctensor import Tensor
from ctensor.functional import conv2d, relu
x  = Tensor(np.array([1]),requires_grad=True)
a  = Tensor(np.array([1]),requires_grad=False)
xa  =   x * a
x2  =   x * x
f_1  =  x * xa
f_2  =  x * xa
f_3  =  x2 * a
f = f_1 +f_3+f_2;
print(f)
f.backward(False)
print(x.grad)