quva-lab / e2cnn Goto Github PK
View Code? Open in Web Editor NEWE(2)-Equivariant CNNs Library for Pytorch
Home Page: https://quva-lab.github.io/e2cnn/
License: Other
E(2)-Equivariant CNNs Library for Pytorch
Home Page: https://quva-lab.github.io/e2cnn/
License: Other
Thanks for your work. I met a problem: I use R2conv in my network, I found when training R2conv module does not have attribute filter, but when testing, R2conv module has attribute 'filter', could you help me with this problem?Why does this happen
Hello,
Thank you for your nice work. I'm a heavy user of this library :)
Recently I'm working on O(2) groups and I have some questions about irreps.
I'm following your approach here to use every possible irreps in the fibergroup.
irreps = []
for n, irr in gc.fibergroup.irreps.items():
if n != gc.trivial_repr.name:
irreps += [irr] * int(irr.size // irr.sum_of_squares_constituents)
irreps = list(irreps)
What I can see is that the number of the irreps is independent to the maximum_frequency assigned when building a gspace.
My questions are
In addition, I have a minor bug to report related to the code snippet above.
In short: PyTorch DDP does not support them.
Here are the reasons and my solution.
Simply fixed by changing 'field_type.py' as following, then install from the source (pip install .)
# self._unique_representations = set(self.representations)
_unique_representations = list(dict.fromkeys(self.representations).keys())
self._unique_representations = _unique_representations
I'm not raising a pull request because there might be any unexpected side effects.
Best,
Ahyun Seo
Is this package accelerated by cuda?
When I transform a tensor that's on a GPU (using FieldType.transform
or GeometricTensor.transform
), I get TypeError: can't convert cuda:0 device type tensor to numpy.
The issue is that in this line, it would have to be input.detach().cpu().numpy()
instead of input.detach().numpy()
.
Only changing that line is probably bad because it would return a tensor on CPU when a GPU tensor is fed in. You could store the device of the tensor, move the tensor to CPU, and at the end return a new tensor on the original device. Of course, moving the tensor implicitly like that isn't ideal as well because it might be a performance hit without the user being aware of it. Alternatively, maybe just document this and raise a custom Exception ("tensors need to be moved to cpu before transforming")? Not sure which solution is better.
Hi Gabriele, thanks again for the great library!
I apologize in advance if the answer to this can be found in the docs/paper/thesis, I couldn't find it there.
My question is rather soft: what is a "reasonable" SO(2)
-equivariant architecture? E.g. what would be the "proper" way to generalize the Wide ResNet
s in examples/e2wrn.py
to be (approximately) equivariant wrt SO(2)
rather than CN
?
My attempt so far is:
N=-1
with max_frequency=10
(leaving default value)e2cnn.nn.InnerBatchNorm
with e2cnn.nn.NormBatchNorm
gspace.trivial_repr
for the output of the last layer, I have used InnerBatchNorm
for the corresponding BatchNorm
e2cnn.nn.ReLU
in this case)e2cnn.nn.FieldType(gspace, [gspace.irrep(1)] * a + [gspace.irrep(2)] * b + [gspace.irrep(3)] * c)
for some positive integers a
, b
, and c
.This gist contains my attempt to make a SO(2)
-invariant ResNet-18
. An equivariance test, however, shows that I am rather off. In particular, the output of
model.cuda()
model.eval()
img_size = 221
repeats = 500
rot_diffs = []
rand_diffs = []
for _ in range(repeats):
x = torch.randn([1, 3, img_size, img_size]).cuda()
xrot = torch.rot90(x, k=1, dims=[2, 3])
xrand = torch.randn([1, 3, img_size, img_size]).cuda()
with torch.no_grad():
z, lat = model(x, with_latent=True)
zrot, latrot = model(xrot, with_latent=True)
zrand, latrand = model(xrand, with_latent=True)
rot_diffs.append(torch.norm(z - zrot))
rand_diffs.append(torch.norm(z - zrand))
print(f'l2 diff between logits of image and its 90-degree rotation: {torch.mean(torch.stack(rot_diffs)).item():.3f}')
print(f'l2 diff between logits of two arbitrary images: {torch.mean(torch.stack(rand_diffs)).item():.3f}')
is
l2 diff between logits of image and its 90-degree rotation: 0.617
l2 diff between logits of two arbitrary images: 0.739
What is going wrong here?
In case this is helpful, this is the model "topology":
MODEL TOPOLOGY:
0 -
1 - conv1
2 - conv1._basisexpansion
3 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_1')
4 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_2')
5 - conv1._basisexpansion.block_expansion_('irrep_0', 'irrep_3')
6 - relu1
7 - bn1
8 - maxpool
9 - layer1
10 - layer1.0
11 - layer1.0.conv1
12 - layer1.0.conv1._basisexpansion
13 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_1')
14 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_2')
15 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_1', 'irrep_3')
16 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_1')
17 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_2')
18 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_2', 'irrep_3')
19 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_1')
20 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_2')
21 - layer1.0.conv1._basisexpansion.block_expansion_('irrep_3', 'irrep_3')
22 - layer1.0.bn1
23 - layer1.0.conv2
24 - layer1.0.conv2._basisexpansion
25 - layer1.0.bn2
26 - layer1.1
27 - layer1.1.conv1
28 - layer1.1.conv1._basisexpansion
29 - layer1.1.bn1
30 - layer1.1.conv2
31 - layer1.1.conv2._basisexpansion
32 - layer1.1.bn2
33 - layer2
34 - layer2.0
35 - layer2.0.conv1
36 - layer2.0.conv1._basisexpansion
37 - layer2.0.bn1
38 - layer2.0.conv2
39 - layer2.0.conv2._basisexpansion
40 - layer2.0.bn2
41 - layer2.0.shortcut
42 - layer2.0.shortcut.0
43 - layer2.0.shortcut.0._basisexpansion
44 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_1', 'irrep_1')
45 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_2', 'irrep_2')
46 - layer2.0.shortcut.0._basisexpansion.block_expansion_('irrep_3', 'irrep_3')
...
(layers 2.1 to 4.0 only change multiplicities in field types, no 'block_expansion_'s here)
...
92 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_1', 'irrep_0')
93 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_2', 'irrep_0')
94 - layer4.1.conv2._basisexpansion.block_expansion_('irrep_3', 'irrep_0')
95 - layer4.1.relu2
96 - layer4.1.bn2
97 - layer4.1.bn2.batch_norm_[1]
98 - linear
99 - avgpool
Thanks in advance!
Hi, @Gabri95 , thank you for your nice work! And I have got inspiration from your work!
But a question (may be very simple) confused me:
How to generate rotation equivariant features on a specific angle (e.g., 45, 33.3), but not all the 360 degree features.
In other words, I just want the features on a particular orientation.
I guess the FieldType defined in SO(2)
can resolve this problem, but I dont know how to use related functions.
Hi Gabriele, thank you very much for the great library!
Is equivariance guaranteed for images of arbitrary sizes/resolutions? It seems like equivariance is satisfied only for certain sizes. As an example, the C4
-invariant 10-4 Wide ResNet
defined and tested in examples/e2wrn.py
seems to only be invariant to 90
-degree rotations for images with widths of the form 4k+3
.
To exemplify this, the output of this snippet
m = Wide_ResNet(10, 4, 0.3, initial_stride=1, N=4, f=False, r=0, num_classes=10)
m.eval()
for img_size in range(30, 50, 1):
x = torch.randn([1, 3, 30, img_size])
xrot = torch.rot90(x, k=1, dims=[2, 3])
with torch.no_grad():
z = m(x)
zrot = m(xrot)
print(f'Image size is 30x{img_size}. Equivariant:' +\
('YES' if torch.allclose(z, zrot, atol=1e-6) else 'NO'))
is
Image size is 30x30. Equivariant: NO
Image size is 30x31. Equivariant: NO
Image size is 30x32. Equivariant: NO
Image size is 30x33. Equivariant: YES
Image size is 30x34. Equivariant: NO
Image size is 30x35. Equivariant: NO
Image size is 30x36. Equivariant: NO
Image size is 30x37. Equivariant: YES
Image size is 30x38. Equivariant: NO
Image size is 30x39. Equivariant: NO
Image size is 30x40. Equivariant: NO
Image size is 30x41. Equivariant: YES
Image size is 30x42. Equivariant: NO
Image size is 30x43. Equivariant: NO
Image size is 30x44. Equivariant: NO
Image size is 30x45. Equivariant: YES
Image size is 30x46. Equivariant: NO
Image size is 30x47. Equivariant: NO
Image size is 30x48. Equivariant: NO
Image size is 30x49. Equivariant: YES.
This seems like an artifact caused by dilation/pooling but I am not sure whether there's any other factor causing this behaviour. Is this the case?
Lastly, would we observe a similar behaviour if we are working with field types composed of irreps of SO(2)
? I don't have an intuition on how dilation is going to interact with guaranteeing equivariance there.
Thanks in advance!
Is there a way I can use this library with python 3.6 as well?
Hi,
When I'm testing the equivariance for a sample example in C8 space following the "introduction.ipynb", I've observed it to fail. So, I wanted to check if there is a correct way of evaluating the c8 space equivariance? Further, It would help me if you could point out how could one go about defining an architecture such that equivariance could be observed across all angles of C8 space?
My code:
import torch
from e2cnn import gspaces
from e2cnn import nn
def conv3x3(in_type: nn.FieldType, out_type: nn.FieldType, stride=1, padding=1,
dilation=1, bias=False):
"""3x3 convolution with padding"""
return nn.R2Conv(in_type, out_type, 3,
stride=stride,
padding=padding,
dilation=dilation,
bias=bias,
sigma=None,
frequencies_cutoff=lambda r: 3*r,
)
def conv1x1(in_type: nn.FieldType, out_type: nn.FieldType, stride=1, padding=0,
dilation=1, bias=False):
"""1x1 convolution with padding"""
return nn.R2Conv(in_type, out_type, 1,
stride=stride,
padding=padding,
dilation=dilation,
bias=bias,
sigma=None,
frequencies_cutoff=lambda r: 3*r,
)
r2_act = gspaces.Rot2dOnR2(N=8)
feat_type_in = nn.FieldType(r2_act, [r2_act.trivial_repr])
feat_type_out = nn.FieldType(r2_act, 3*[r2_act.regular_repr])
conv = conv3x3(feat_type_in, feat_type_out)
x = torch.randn(4, 1, 32, 32)
x = nn.GeometricTensor(x, feat_type_in)
assert isinstance(x.tensor, torch.Tensor)
y = conv(x)
assert y.type == feat_type_out
print(y.tensor.shape)
for g in r2_act.testing_elements:
# transform the input with the current group element according to the input type
x_transformed = x.transform(g)
# feed the transformed input in the convolutional layer
y_from_x_transformed = conv(x_transformed)
# the result should be equivalent to rotating the output produced in the
# previous block according to the output type
y_transformed_from_x = y.transform(g)
assert torch.allclose(y_from_x_transformed.tensor, y_transformed_from_x.tensor, atol=1e-05), g
print("success")
result:
Traceback (most recent call last):
File "e2_test.py", line 62, in <module>
assert torch.allclose(y_from_x_transformed.tensor, y_transformed_from_x.tensor, atol=1e-05), g
AssertionError: 1
Thank you for this great implementation of equivariant CNNs. I noticed that in your paper and in this implementation, while broad groups such as O(2) are mentioned, only transformations of translation/rotation/reflection are explicitly discussed.
In light of new work on scale-equivariant CNNs, such as this one, would scale-equivariance also be covered under O(2), since the origin is preserved under scaling? If so, how could scale-equivariance be incorporated in tandem with the transformations covered in this library?
I would appreciate any tips or thoughts you might have on this subject.
Hi there. Many thanks for this amazing library and the whole equivariant framework, it's a very inspiring work!
I think the title of the issue pretty much sums it all up. I am trying to pass 2 positional arguments as input to an R2Conv module, both the input data and also the weight parameters.
My issue is that I get an error saying that 3 positional arguments were given when 2 were expected.
The standard conv2D module of PyTorch allows this, so I was wondering if it something wrong in my implementation or in general this behavior cannot be used for an R2Conv module.
Many thanks in advance,
Hi @Gabri95 ,
I noticed the current implementation only supports bias for trivial representation as commented:
e2cnn/e2cnn/nn/modules/r2_conv/r2convolution.py
Lines 184 to 189 in 2604f67
From a naive intuition, I feel that bias acted on the norm of an irrep seems also equivariant. I didn't find relevant description in the paper. So just a short question for any relevant reference to deny this intuition. Thanks!
Thank you for this wonderful implementation of equivariant models. I'm currently using them to investigate segmentation problems. I have a question, which I'm really cannot find a clear answer to. What is the difference between the parameters weights
and filter
in R2Conv
?
I'm confused as to how to connect the terms: if a regular square kernel is sampled on a PolarBasis, we obtain rotated versions of the filter. These are in turn used to in R2Conv
. Does that mean, they are learnt directly in the back propagation phase?
Hello, I am looking for something equivalent to torch.cat, but takes GeometricTensors as inputs and outputs another GeometricTensor. I am developing a rotation-equivariant version of U-net and I need something equivalent to torch.cat for the skip connections. I haven't been able to find something inside the E2CNN library yet that seems to do this.
Thanks again for a really nice framework, I credited it in a recent publication.
As a minor convenience, it would be nice to have an equivariant equivalent to torch.nn.ModuleList.
The main use cases:
It would not subclass EquivariantModule, since it can't meaningfully conform to most of that API, but it would probably get used mostly in EquivariantModule subclass implementations.
I'm currently trying out the following pattern in my code:
# Inside a bigger class that is itself implementing .export()... with self.up_path also a ModuleList...
def export(self):
self.eval()
up_path_exported = torch.nn.ModuleList()
for module in self.up_path:
up_path_exported.append(module.export())
If the EquivariantModules were already living in a e2cnn.nn.ModuleList, it would just be:
def export(self):
self.eval()
up_path_exported = self.up_path.export()
So a couple lines of boilerplate would be saved, but the discovery aspect is probably worth more... "just build everything using e2cnn 1:1 equivalents, train, then .export()"
Thank you for a very well-crafted and important package. I have been using this package to test a rotation-equivariant UNet type network. I already see VERY good results; the standard UNet has 92 % accuracy on our dataset, whereas the most naive rotation-equivariant implementation achieves 99 %(!)
I would deploy this model right now if it was faster to instantiate: It currently takes 16 seconds, not including loading parameters and moving the model to GPU (which takes less than a second).
I can make some changes to our existing software to make this work and the accuracy improvements would certainly be worth it; however I am hoping for a way to make instantiation a bit faster.
Thanks,
Jacob
The doc states usage as "e2cnn.nn.Sequential(...)", but it should be "e2cnn.nn.SequentialModule(...)"
I would like to build rotation equivariant networks in the UNet family, but it seems downsampling and upsampling are missing necessary building blocks.
For continuous signals it seems pretty obvious that downsampling and upsampling operators can be defined that are equivariant. i.e. if an image is downscaled by 2, then rotations happen by the same angle, translations happen in the same direction but half as far. Similar story for upscaling but translations are doubled. Both operators are lossless in an idealized infinite continuous 2d plane setting.
For discrete signals it should be possible too up to numerics, although you have to be careful to use downsampling and upsampling kernels that are (approximately) rotation symmetric, aka "radial".
Pytorch already has down and upsampling operators; maybe it's just a matter of wrapping them correctly in your framework.
I really like your visualizations of rotation equivariance (i.e. vectorfield.gif and conventional_cnn.gif
), and would like to generate similar ones for my network. Is the code for generating those something that could be easily added, or should I go ahead write my own code for that?
I'm working on optimizing my model inference, trying conversion to torchscript as a first step. When I call torch.jit.script() on my model, I hit:
name = '_weights_ranges', item = {('irrep_0,0', 'regular'): (0, 288)}
def infer_type(name, item):
# The forward function from Module is special; never use this annotations; we
# need to infer type directly using JIT. I originally wanted to write
# this test as isinstance(class_annotations[name], Callable) but
# isinstance on typing things doesn't seem to work: isinstance(list, Callable)
# is also true!
if name in class_annotations and class_annotations[name] != torch.nn.Module.__annotations__["forward"]:
attr_type = torch.jit.annotations.ann_to_type(class_annotations[name], _jit_internal.fake_range())
elif isinstance(item, torch.jit.Attribute):
attr_type = torch.jit.annotations.ann_to_type(item.type, _jit_internal.fake_range())
else:
> attr_type = torch._C._jit_try_infer_type(item)
E RuntimeError: Cannot create dict for key type '(str, str)', only int, float, Tensor and string keys are supported
This pytorch code resides here:
https://github.com/pytorch/pytorch/blob/22902b9242853a4ce319e7c5c4a1c94bc00ccb7a/torch/jit/_recursive.py#L126
Torch can't trace through a Dict[Tuple[str,str],_], which is used here:
My goal is to get the model to run as fast as possible on NVIDIA hardware, probably using tensorrt. Is there another known-good conversion path?
The above error was with torch 1.6.0, e2cnn v0.1.
This looks like a really nice library and I want to test it out, so I am converting an existing neural network to a rotational invariant one.
In one of my layers, I have a convolution with kernel_size=2. This breaks in the R2Conv while kernel_size=3 or more works fine.
Is there a reason why a kernel_size=2 convolution cannot work in the rotational setting or is this a bug?
Minimal example:
r2_act = gspaces.Rot2dOnR2(N=8)
in_type = nn.FieldType(r2_act, 3*[r2_act.trivial_repr])
out_type = nn.FieldType(r2_act, 6*[r2_act.regular_repr])
conv = nn.R2Conv(in_type, out_type, kernel_size=2, initialize=False, padding=0).cuda()
x = torch.ones(size=[1,3,190,190]).cuda()
x = nn.GeometricTensor(x, in_type)
conv(x)
Error:
list index out of range in basisexpansion_blocks.py in line 327
Hello,
Thank you for your nice work!
I'm about to use the e2cnn based network for my project.
I need to use the method grid_sample in PyTorch and its underlying implementation is just some interpolation.
https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html?highlight=grid_sample#torch.nn.functional.grid_sample
I noticed the upsample in this repo simply computes the value and reassigns the output type.
https://github.com/QUVA-Lab/e2cnn/blob/master/e2cnn/nn/modules/r2upsampling.py#L86
I think I can do exactly like this with grid_sample.
Please let me know if there will be any issues with this.
Ahyun
Hi!
I found your package quite helpful! However, when I want to save the model or model.state_dict with torch.save, it failed with the error: AttributeError: Can't pickle local object 'DihedralGroup.irrep..'. I wonder if you have any solution for this? Thank you a lot!
When running model.ipynb, I get the error shown below when training, in box [9]:
When commenting out the RandomRotation in the train_transform function, there are no errors and I able to run the model without errors.
Any idea what could be wrong?
And thanks a lot for this really nice implementation and tutorial!
TypeError Traceback (most recent call last)
in
2 for epoch in range(51):
3 model.train()
----> 4 for i, (x, t) in enumerate(train_loader):
5 if i > 20:
6 break
~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in next(self)
343
344 def next(self):
--> 345 data = self._next_data()
346 self._num_yielded += 1
347 if self._dataset_kind == _DatasetKind.Iterable and \
~/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py in _next_data(self)
383 def _next_data(self):
384 index = self._next_index() # may raise StopIteration
--> 385 data = self._dataset_fetcher.fetch(index) # may raise StopIteration
386 if self._pin_memory:
387 data = _utils.pin_memory.pin_memory(data)
~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]
~/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py in (.0)
42 def fetch(self, possibly_batched_index):
43 if self.auto_collation:
---> 44 data = [self.dataset[idx] for idx in possibly_batched_index]
45 else:
46 data = self.dataset[possibly_batched_index]
in getitem(self, index)
21 image = Image.fromarray(image)
22 if self.transform is not None:
---> 23 image = self.transform(image)
24 return image, label
25
~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in call(self, img)
68 def call(self, img):
69 for t in self.transforms:
---> 70 img = t(img)
71 return img
72
~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/transforms.py in call(self, img)
1001 angle = self.get_params(self.degrees)
1002
-> 1003 return F.rotate(img, angle, self.resample, self.expand, self.center, self.fill)
1004
1005 def repr(self):
~/anaconda3/lib/python3.7/site-packages/torchvision/transforms/functional.py in rotate(img, angle, resample, expand, center, fill)
727 fill = tuple([fill] * 3)
728
--> 729 return img.rotate(angle, resample, expand, center, fillcolor=fill)
730
731
~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in rotate(self, angle, resample, expand, center, translate, fillcolor)
1913
1914 return self.transform((w, h), AFFINE, matrix, resample,
-> 1915 fillcolor=fillcolor)
1916
1917 def save(self, fp, format=None, **params):
~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in transform(self, size, method, data, resample, fill, fillcolor)
2203 raise ValueError("missing method data")
2204
-> 2205 im = new(self.mode, size, fillcolor)
2206 if method == MESH:
2207 # list of quads
~/anaconda3/lib/python3.7/site-packages/PIL/Image.py in new(mode, size, color)
2373 color = ImageColor.getcolor(color, mode)
2374
-> 2375 return Image()._new(core.fill(mode, size, color))
2376
2377
TypeError: must be real number, not tuple
Hi developers,
I have set up a python environment with python 3.6.9.
When I install the package,
pip install .
I encounter the following error
ERROR: Command errored out with exit status 1: command: /home/kaikai/anaconda3/envs/python3.6.9/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-nx2wsig7/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-nx2wsig7/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-iki7taiq cwd: /tmp/pip-req-build-nx2wsig7/ Complete output (2 lines): running egg_info error: Invalid distribution name or version syntax: e2cnn-py3.6--0.1 ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
It turns out that parentheses are invalid characters in the "name" of setuptools.
I have verified if I change the line in e2cnn/__about__.py
from
__title__ = "e2cnn (py3.6)"
to
__title__ = "e2cnn-py3.6"
,
the installation will work.
Would you help take a look at this?
Hello,
Thank you for sharing this amazing work!
While playing it a little bit, I wonder how many flops the network consumes.
I found that the existing library (e.g., https://github.com/1adrianb/pytorch-estimate-flops) does not work for this as they don't support equivariant modules.
Is there any simple method for this one? If the counting for kernel construction parts would be hard, I just want to know how to translate the model into torch model as in test phase to count using the existing library.
Is it possible to do ZeroPad2D on the geometric tensor? Specifically, I want my input image and output image to have the same dimension and I am using kernel size = 4. With the normal CNN, I can achieve this with pad = torch.nn.ZeroPad2d((2, 1, 2, 1)). How can I do this type of padding with equivariant CNNs?
While auditing e2cnn too see if the functions I need actually support .export(), I noticed that:
PointwiseMaxPoolAntialiased:
inherits the .export() function from PointwiseMaxPool, whereas it should be exporting ops for the antialiased version.
hi~i want to ask if i can build rotated-equivariant swin transformer/Transformer anchitectural in E2CNN library?
thank you!!!
Dear Author:
I change the wide_resnet by deleting the pooling and linear operation after out=out.tensor:
def forward(self, x):
# wrap the input tensor in a GeometricTensor
x = enn.GeometricTensor(x, self.in_type)
out = self.conv1(x)
out = self.layer1(out)
out = self.layer2(self.restrict1(out))
out = self.layer3(self.restrict2(out))
out = self.bn(out)
out = self.relu(out)
# extract the tensor from the GeometricTensor to use the common Pytorch operations
out = out.tensor
# b, c, w, h = out.shape
# out = F.avg_pool2d(out, (w, h))
# out = out.view(out.size(0), -1)
# out = self.linear(out)
return out
But find that the equivalence does not exist anymore:
print()
print('TESTING INVARIANCE: ')
print('REFLECTIONS along the VERTICAL axis: ' + (str(torch.norm(y-y_fv))))
print('REFLECTIONS along the HORIZONTAL axis: ' + (str(torch.norm(y-y_fh))))
print('90 degrees ROTATIONS: ' + (str(torch.norm(y-y90))))
print('REFLECTIONS along the 45 degrees axis: ' + (str(torch.norm(y-y90_fh))))
The output is:
REFLECTIONS along the VERTICAL axis: tensor(1.9236, grad_fn=<CopyBackwards>)
REFLECTIONS along the HORIZONTAL axis: tensor(2.0343, grad_fn=<CopyBackwards>)
90 degrees ROTATIONS: tensor(2.0621, grad_fn=<CopyBackwards>)
REFLECTIONS along the 45 degrees axis: tensor(2.0622, grad_fn=<CopyBackwards>)
May I have a ask about the reason?
When running python setup.py install
on the legacy_3.6
branch, I'm getting:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/tmp/pip-install-pv1d_p_r/e2cnn/setup.py", line 19, in <module>
long_description = f.read()
File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 6561: ordinal not in range(128)
If I hotfix this by setting long_description = ""
, the installation works.
(Python 3.6.9)
Fix:
cell 7:
instead of
x_transformed.to(device)
this:
x_transformed = x_transformed.to(device)
cell 9:
instead of:
loss = loss_function(y, t)
this:
loss = loss_function(y, t.to(device=device))
instead of:
correct += (prediction == t).sum().item()
this:
correct += (prediction == t.to(device=device)).sum().item()
I'm porting over this function to e2cnn:
def center_crop(self, layer:torch.Tensor, target_size:List[int]):
_, _, layer_height, layer_width = layer.size()
diff_y = (layer_height - target_size[0]) // 2
diff_x = (layer_width - target_size[1]) // 2
return layer[
:, :, diff_y : (diff_y + target_size[0]), diff_x : (diff_x + target_size[1])
]
When I pass a GeometricTensor into it, I hit "TypeError: 'GeometricTensor' object is not subscriptable" where the tensor is sliced.
That's probably too much info. I only need to slice along the spatial dimensions... I will experiment with casting back to torch.Tensor, but please let me know if you have a better workaround or update up your sleave. Thanks!
Error while import with Pytorch version 1.9.0+cu111 (maybe earlier)
----> 6 from e2cnn import nn as e2nn
7
8 import utils
/venv/lib/python3.8/site-packages/e2cnn/nn/__init__.py in <module>
3 from .geometric_tensor import GeometricTensor, tensor_directsum
4
----> 5 from .modules import *
6
7
/venv/lib/python3.8/site-packages/e2cnn/nn/modules/__init__.py in <module>
48
49 from .sequential_module import SequentialModule
---> 50 from .module_list import ModuleList
51 from .identity_module import IdentityModule
52
/venv/lib/python3.8/site-packages/e2cnn/nn/modules/module_list.py in <module>
2 from .equivariant_module import EquivariantModule
3
----> 4 from torch._six import container_abcs
5
6 import torch
ImportError: cannot import name 'container_abcs' from 'torch._six' (/home/s51972mb/Documents/EquivariantSelfAttention/venv/lib/python3.8/site-packages/torch/_six.py)
Potential fix:
NVIDIA/apex#1049 (comment)
Thank you for this great library for equivariant steerable CNNs. However, I am not able to build an equivariant model for my project through the examples given in this repository. Can you include examples for ResNet & VGG like architectures or can someone tell me how to induce equivariance in these steerable networks. For example, I tried building equivariant model based on WideResNet example given in the repository but I am struggling to induce equivariance and getting only invariance through that.
There is a situation when images are stacked twice. The shape of the structured image tensor is (B, S, C, H, W) where B represents batch size and S the number of stacked images.
Could R2conv directly act or broadcast on such a 5-D tensor?
Just a small problem I just encountered, currently a convolutional models state_dict in train mode is different to its state_dict in eval mode. Which results in a "Missing Keys" error when loading a state_dict saved during training into a model in eval mode. It is not critical though as loading with "strict=False" and then running model.eval() restores the filter and the expanded_bias.
Hi all,
Thank you very much for your amazing framework!
I would like to ask a few things. Since I am trying to build an auto-encoder for image segmentation based on your framework, I was wondering about how and if I should use your mask module.
I have seen your example and read the documentation for the mask module. My question is, should I use it often? I have many blocks of sequential layers with different operations like convolutions, pooling, upsampling etc.
Do you think it's necessary to add a lot of them to maintain rotation equivariance?
Thank you.
I believe that the package needs to be updated for new versions of PyTorch. The code works with PyTorch 1.1
, but it does not work for PyTorch 1.8
.
This issue can be reproduced via the use of another package:
Another issue is the support of machine without CUDA. To reproduce this issue, execute the following two steps on a CPU-only machine:
Currently the e2cnn.R2Upsampling class only has four parameters: in_type, scale_factor, mode and align_corners. However, I think it would be helpful to add another "size" parameter, to be passed to the torch.nn.functional.interpolate method used inside the R2Upsampling class. Sometimes we need to up-sample the input to a specific dimension, especially when dealing with even and odd dimension issues
I happened to observe that when I change the dilation value, the number of parameters change. This is not the case with the standard torch.nn.Conv2D module. Is there any specific reason it happens in e2cnn.nn.R2Conv.
If this behaviour is expected, can you please direct me to the right resource.
Environment:
Python=3.7.9
torch=1.7.1
e2cnn=0.1.5
Code to reproduce issue
import torch
import torch.nn as nn
import torch.nn.functional as F
import numpy as np
import math
import e2cnn
import e2cnn.nn as enn
from e2cnn.nn import init
from e2cnn import gspaces
class Model(nn.Module):
def __init__(self):
super(Model, self).__init__()
N = 8
self.gspace = gspaces.Rot2dOnR2(N)
self.in_type = enn.FieldType(self.gspace, [self.gspace.trivial_repr] * 3)
self.out_type = enn.FieldType(self.gspace, [self.gspace.regular_repr] * 16)
self.layer = enn.R2Conv(self.in_type, self.out_type, 3,
stride=1,
padding=1,
dilation=1,
bias=True,
)
self.invariant = enn.GroupPooling(self.out_type)
def forward(self, x):
x = enn.GeometricTensor(x, self.in_type)
out = self.layer(x)
out = self.invariant(out)
out = out.tensor
return out
class ModelDilated(nn.Module):
def __init__(self):
super(ModelDilated, self).__init__()
N = 8
self.gspace = gspaces.Rot2dOnR2(N)
self.in_type = enn.FieldType(self.gspace, [self.gspace.trivial_repr] * 3)
self.out_type = enn.FieldType(self.gspace, [self.gspace.regular_repr] * 16)
self.layer = enn.R2Conv(self.in_type, self.out_type, 3,
stride=1,
padding=2,
dilation=2,
bias=True,
)
self.invariant = enn.GroupPooling(self.out_type)
def forward(self, x):
x = enn.GeometricTensor(x, self.in_type)
out = self.layer(x)
out = self.invariant(out)
out = out.tensor
return out
if __name__=="__main__":
m = Model()
md = ModelDilated()
ip = torch.randn(1,3,100,100)
op1 = m(ip)
op2 = md(ip)
totalParams = sum(p.numel() for p in m.parameters())
totalParams2 = sum(p.numel() for p in md.parameters())
print(totalParams, totalParams2)
print(op1.shape, op2.shape)
Output
304 400
torch.Size([1, 16, 100, 100]) torch.Size([1, 16, 100, 100])
Hi admins, Thanks for the nice paper and handy code.
e2cnn/e2cnn/kernels/steerable_basis.py
Line 174 in 2604f67
The weight tensor return by SteerableBasis.sample
has the shape of out_channels x in_channels x dim x len(angles). dim
is the summation of basis dimensions of all input/output representation pairs. I feel that this shape is redundant since each basis is only used by its corresponding input/output pair.
e2cnn/e2cnn/kernels/steerable_basis.py
Lines 180 to 192 in 2604f67
The sampling code also implies that the current weight tensor is very sparse by filling a 3D tensor with 2 for loops. I feel that theoretically out_channels x in_channels x 4 x len(angles) should be enough to save all sampled basis as a non-trivial irrep pair usually corresponds to 4 bases.
Could you please let me know if this interpretation is correct? Thanks!
Hi,
First off, thanks a lot for this nice package!
I am trying to split a regular field into all of its field components. According to GeometricTensor.split
this can be done via x.split(None)
, which however errors out:
import torch
import e2cnn
r2_act = e2cnn.gspaces.Rot2dOnR2(N=12)
feat_type_in = e2cnn.nn.FieldType(r2_act, [r2_act.trivial_repr])
feat_type_out = e2cnn.nn.FieldType(r2_act, [r2_act.regular_repr])
conv = e2cnn.nn.R2Conv(feat_type_in, feat_type_out, kernel_size=3, padding=1)
x = e2cnn.nn.GeometricTensor(torch.randn((1,1,32,32)), feat_type_in)
x = conv(x)
print(x.tensor.shape)
x.split(None)
>>> AssertionError: Error! "breaks" must be an increasing list of positive indexes
Am I doing something wrong here?
I'm experiencing some surprisingly large models when using equivariance. I am taking advantage of the (very convenient!) TrivialOnR2 fallback for comparisons been my non-equivariant and my equivariant model. This means all of the code is the same aside from the layer (and tensor) types.
Sizes of saved model files:
TrivialOnR2(): 18M
FlipRot(4): 733M
That's a 40X difference in size.
I get gpu OOM killed when I try to load it onto my local machine's 2GB gpu.
I was under the impression that there was no extra cost for equivariance at inference time. Even with a naive guess that you pay linearly for the equivariance, I would expect FlipRot(4) to be about 8X bigger.
I unzipped the model file (the size was unchanged, so I guess it wasn't actually compressed).
These are the largest files in the "data" subdirectory of the archive:
144.0 MiB [##########] 93996286070688
135.2 MiB [######### ] 93996315383952
103.6 MiB [####### ] 93996297647440
36.0 MiB [## ] 93996309406176
36.0 MiB [## ] 93996301356576
31.6 MiB [## ] 93996287309120
24.8 MiB [# ] 93996405222784
23.0 MiB [# ] 93996315305440
23.0 MiB [# ] 93996283837952
15.5 MiB [# ] 93996319486144
15.5 MiB [# ] 93996405222048
15.5 MiB [# ] 93996309417568
Running my model through (a slightly hacked) pytorch-model-summary:
Layer (type) Input Shape Param # Tr. Param #
Layer (type) Input Shape Param # Tr. Param #
There actually ~are 8X more trainable parameters than with TrivialOnR2, so I was probably doing an unfair comparison; My TrivialOnR2 should get 8X more channels.
So the parameters take up 51 MB, leaving 682 MB of unexplained size. Do you have a guess what mistake I might have made to be causing that bloat? Thanks!
I'm working on creating an equivariant network for classification task. Based on the example available here, it seems that only Equivariant
module can be exported to pytorch. As shown below, an assertion error occurs as torch.nn.Linear
is not an Equivariant module.
spc = e2cnn.gspaces.Rot2dOnR2(8)
in_ = e2cnn.nn.FieldType(s, [s.trivial_repr])
out_ = e2cnn.nn.FieldType(s, [s.regular_repr]*16)
net = SequentialModule(
R2Conv(in_, out_, 3, bias=False),
ReLU(out_, inplace=True),
PointwiseMaxPool(out_, kernel_size=2, stride=2),
GroupPooling(out_),
torch.nn.Linear(in_channels, out_channels)
)
Is there another way in which, the entire net can be exported to pytorch after training?
When creating a FieldDropout Layer, I get the following error: TypeError: cannot assign 'torch.LongTensor' object to parameter 'indices_8' (torch.nn.Parameter or None required). Initializing PointwiseDropout with the same parameters works.
Is this a known issue?
Hi!
First, I really appreciate your project, and it is very helpful in my project. However, I was a bit confused when I tried to compare vanilla resnet18 with e2resnet18. I found e2resnet18 consumed 4 times more GPU memory and 10 times training time. I wonder if I had done anything wrong, or it is just designed like that. Thank you very much!
Best,
Harold
Hi, thanks again for a very nice package and your support of it.
When I am running my own architecture, there seems to be some kind of problem with its equivariance properties after training.
I am using my own dataset contaning biomedical images, and the classifier needs to classify cancer being present or not. So therefore there are two classes. When I run the test function on an image on the test set, I get these results:
angle | 0 1 2 3 4 5 6 7 8 9
0 : [-0.0123 -0.2154]
90 : [-0.0118 -0.2155]
180 : [-0.0126 -0.2156]
270 : [-0.012 -0.2157]
#############################################################################
So the network has a high degree of equivariance.
Then I train my model, run the same test function again and I get these results:
#############################################################################
angle | 0 1 2 3 4 5 6 7 8 9
0 : [-1.6111 1.8046]
90 : [-1.0136 1.043 ]
180 : [-1.3092 1.5487]
270 : [-1.5571 1.8682]
#############################################################################
Meaning very different outputs even though I use the C4 group. Any insight into what could be happening to 'break' the equivariance properties of the network?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.