musyoku / chainer-sru Goto Github PK

Hi， I have used your code for other speech recognition task. and I found that the speed is faster but it have bad result compared to result which use GRU. can you give me some advice? Thanks.

hi @musyoku

your implementation is awesome! I'm very excited about your results!

I noticed there are two main differences between your version and mine:

• you used conv1d() instead of batched matrix multiplication (gemm)
• you used a (batch, dimension, length) shape for U, while i used (length, batch, dimension) instead.

did you compare the speed between yours and my pytorch version? I would love to hear your feedback and see if there's a chance to improve the pytorch version as well.

thanks a lot!

Hi Musyoku,

I'm implementing SRU in tensorflow with a accelrated implement, I have finished the forward part referening taolei87's code, but I have trouble implementing the gradient part, in taolei87's code, it calculated the grad_u from cuda code, but tensorflow require a grad wrt x, and grad wrt weights.

I found your code have some code about convert from grad_u to grad_x, and grad_w,

backward_gpu:
..
col = conv_nd.im2col_nd_gpu(grad_u, (1,), (1,), (0,), cover_all=False)
grad_x = xp.tensordot(col, W.T[..., None], ((1, 2), (1, 2))).astype(dtype, copy=False).transpose((0, 2, 1)) + grad_highway_x
grad_w = xp.tensordot(grad_u, self.col, ((0, 2), (0, 3))).astype(dtype, copy=False).reshape((feature_dimension * 3, feature_dimension))

But I have some trouble to understand how this code convert grad_u to grad_x, and grad_w,
Could you explain some about it, or maybe there is some well-known paper about this method ?

Thanks in advance.