Hi,
torch.fft() and torch.irfft() are no more functions, those are modules. And there appears to be a lof of modification in the parameters. I am currently trying to combine the two types of features with compact bilinear pooling, do you know how to port this code to pytorch 1.11?

How do you save and load the model, I'm using torch.save, which cause the following error:

File "x/anaconda3/lib/python3.6/site-packages/tor                                                                                                                               ch/serialization.py", line 135, in save
   return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickl                                                                                                                               e_protocol))
 File "x/anaconda3/lib/python3.6/site-packages/tor                                                                                                                               ch/serialization.py", line 117, in _with_file_like
   return body(f)
 File "xanaconda3/lib/python3.6/site-packages/tor                                                                                                                               ch/serialization.py", line 135, in <lambda>
   return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickl                                                                                                                               e_protocol))
 File "x/anaconda3/lib/python3.6/site-packages/tor                                                                                                                               ch/serialization.py", line 198, in _save
   pickler.dump(obj)
TypeError: can't pickle Rfft objects

Hi,

Thanks for this awesome implementation :)
I used for this code.

Unfortunately, it is not working with pytorch 1.0.
Do you plan to port it?

Thanks

RT

Hi @gdlg, thanks for this nice work. I'm confused about the backward procedure of complex multiplication. So I hope you can help me to figure it out.

In forward,

Z = XY = (Rx + i * Ix)(Ry + i * Iy) = (RxRy - IxIy) + i * (IxRy + RxIy) = Rz + i * Iz

In backward, according the chain rule, it will has

grad_(L/X) = grad_(L/Z) * grad(Z/X)
           = grad_Z * Y
           = (R_gz + i * I_gz)(Ry + i * Iy)
           = (R_gzRy - I_gzIy) + i * (I_gzRy + R_gzIy)

So, why is this line implemented by using the value = 1 for real part and value = -1 for image part?

Is there something wrong in my thoughts? Thanks.

When I run the test module, it indicates that the module of pytorch_fft of fft in autograd does not have attribute of Rfft. What version of pytorch_fft should I install to fit this code?

Hi,
I am back again. When running the test.py, I got the following error
File "test.py", line 69, in test_gradients self.assertTrue(torch.autograd.gradcheck(cbp, (x,y), eps=1)) AssertionError: False is not true

What does this mean?

Source code：
x = self.features(x) #[4,512,28,28]
batch_size = x.size(0)
x = (torch.bmm(x, torch.transpose(x, 1, 2)) / 28 ** 2).view(batch_size, -1)
x = torch.nn.functional.normalize(torch.sign(x) * torch.sqrt(torch.abs(x) + 1e-10))
x = self.classifiers(x)
return x
my code:
x = self.features(x) #[4,512,28,28]
x = x.view(x.shape[0], x.shape[1], -1) #[4,512,784]
x = x.permute(0, 2, 1) #[4,784,512]
x = self.mcb(x,x) #[4,784,512]
batch_size = x.size(0)
x = x.sum(1) #对于二维来说，dim=0，对列求和；dim=1对行求和；在这里是三维所以是对列求和
x = torch.nn.functional.normalize(torch.sign(x) * torch.sqrt(torch.abs(x) + 1e-10))
x = self.classifiers(x)
return x

The training does not converge after modification. Why? Is it a problem with my code?

Hi, I have run the same code, but got this error in this line.

For input:

from compact_bilinear_pooling import CountSketch, CompactBilinearPooling

input_size = 2048
output_size = 16000
mcb = CompactBilinearPooling(input_size, input_size, output_size).cuda()
x = torch.rand(4,input_size).cuda()
y = torch.rand(4,input_size).cuda()

z = mcb(x,y)

Output error:

Traceback (most recent call last):
  File "/compact_bilinear_pooling.py", line 273, in <module>
    z = mcb(x,y)
  File "/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "compact_bilinear_pooling.py", line 263, in forward
    return CompactBilinearPoolingFn.apply(self.sketch1.h, self.sketch1.s, self.sketch2.h, self.sketch2.s, self.output_size, x, y, self.force_cpu_scatter_add)
  File "/python3.9/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/compact_bilinear_pooling.py", line 154, in forward
    im_fx = fx.select(-1, 1)
IndexError: select(): index 1 out of range for tensor of size [2, 1] at dimension 1

Nice work. However, I do not understand how I can apply it with a mini batch of tensors

I modify

class CompactBilinearPooling(nn.Module):   
     def forward(self, x, y):    
            return CompactBilinearPoolingFn.apply(self.sketch1.h, self.sketch1.s, self.sketch2.h, self.sketch2.s, self.output_size, x, y)

to

def forward(self, x):    
    x = x.permute(0, 2, 3, 1) #NCHW to NHWC   
    y = Variable(x.data.clone())    
    out = (CompactBilinearPoolingFn.apply(self.sketch1.h, self.sketch1.s, self.sketch2.h, self.sketch2.s, self.output_size, x, y)).permute(0,3,1,2) #to NCHW    
    out = nn.functional.adaptive_avg_pool2d(out, 1) # N,C,1,1   
    #add an element-wise signed square root layer and an instance-wise l2 normalization    
    out = (torch.sqrt(nn.functional.relu(out)) - torch.sqrt(nn.functional.relu(-out)))/torch.norm(out,2,1,True)   
    return out 

This makes the compact pooling layer can be plugged to PyTorch CNNs more easily:

model.avgpool = CompactBilinearPooling(input_C, input_C, bilinear['dim'])
model.fc = nn.Linear(int(model.fc.in_features/input_C*bilinear['dim']), num_classes)

However, when I run this using multiple GPUs, I got the following error:

Traceback (most recent call last):
File "train3_bilinear_pooling.py", line 400, in
run()
File "train3_bilinear_pooling.py", line 219, in run
train(train_loader, model, criterion, optimizer, epoch)
File "train3_bilinear_pooling.py", line 326, in train
return _each_epoch('train', train_loader, model, criterion, optimizer, epoch)
File "train3_bilinear_pooling.py", line 270, in _each_epoch
output = model(input_var)
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 319, in call
result = self.forward(*input, **kwargs)
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 67, in forward
replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 72, in replicate
return replicate(module, device_ids)
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/nn/parallel/replicate.py", line 19, in replicate
buffer_copies = comm.broadcast_coalesced(buffers, devices)
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/cuda/comm.py", line 55, in broadcast_coalesced
for chunk in _take_tensors(tensors, buffer_size):
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/_utils.py", line 232, in _take_tensors
if tensor.is_sparse:
File "/home/member/fuwang/opt/anaconda/lib/python3.6/site-packages/torch/autograd/variable.py", line 68, in getattr
return object.getattribute(self, name)
AttributeError: 'Variable' object has no attribute 'is_sparse'

Do you have any ideas?

What if I want to merge two feature maps? The shape of the feature map is (batch_size, channel, w, h) ,,,, Thank you ！

I am running this code for compact bilinear pooling. I have 18000 data points but for some values I am getting zero output vector. My inputs are of shape 1024, 1440 and output shape is 1024. Inputs are not very small and rages from 0 to 1e-2. Please help