Comments (14)
VainF
commented on June 8, 2024
Hi, @ooodragon94 , SwishImplementation is not a standard module in pytorch. I'm trying to skip those customized ops and treat them as element-wise OP.
from torch-pruning.
ooodragon94
commented on June 8, 2024
OMG thanks
if I may,
may I ask when it would be done
from torch-pruning.
VainF
commented on June 8, 2024
BTW, I found that EfficientNet contains group conv which is not supported yet.
from torch-pruning.
ooodragon94
commented on June 8, 2024
oh... then no future plan on supporting it?
from torch-pruning.
VainF
commented on June 8, 2024
I'm working on this.
maybe tomorrow?
from torch-pruning.
ooodragon94
commented on June 8, 2024
oh my god that is fast...
thanks!
from torch-pruning.
VainF
commented on June 8, 2024
Hi @ooodragon94 , please try the latest version
from efficientnet_pytorch import EfficientNet
import torch_pruning as tp
import torch
import torch.nn as nn
import random
model = EfficientNet.from_pretrained("efficientnet-b0", advprop=True)
model.eval()
example_inputs = torch.randn(1,3,224,224)
prunable_module_type = ( nn.Conv2d, nn.BatchNorm2d )
prunable_modules = [ m for m in model.modules() if isinstance(m, prunable_module_type) ]
ori_size = tp.utils.count_params( model )
DG = tp.DependencyGraph().build_dependency( model, example_inputs=example_inputs )
for layer_to_prune in prunable_modules:
# select a layer
if isinstance( layer_to_prune, nn.Conv2d ):
prune_fn = tp.prune_conv
elif isinstance(layer_to_prune, nn.BatchNorm2d):
prune_fn = tp.prune_batchnorm
ch = tp.utils.count_prunable_channels( layer_to_prune )
rand_idx = random.sample( list(range(ch)), min(10, ch//2))
plan = DG.get_pruning_plan( layer_to_prune, prune_fn, rand_idx)
plan.exec()
print(model)
with torch.no_grad():
out = model( example_inputs )
print( " Params: %s => %s"%( ori_size, tp.utils.count_params(model) ) )
print( " Output: ", out.shape )
print("------------------------------------------------------\n")from torch-pruning.
VainF
commented on June 8, 2024
My output:
Loaded pretrained weights for efficientnet-b0
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953410>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953320>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953230>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953140>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191953050>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954e60>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954d70>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954c80>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954b90>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954aa0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919549b0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919548c0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919547d0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919546e0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919545f0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954500>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954410>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954320>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954230>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954140>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191954050>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958e60>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958d70>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958c80>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958b90>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958aa0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919589b0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919588c0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919587d0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919586e0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91919585f0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958500>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958410>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958320>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958230>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958140>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9191958050>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812e60>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812d70>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812c80>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812b90>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812aa0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928129b0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928128c0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928127d0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928126e0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f91928125f0>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812500>. It will be treated as element-wise op
[Warning] Unrecognized operation: <torch.autograd.function.SwishImplementationBackward object at 0x7f9192812410>. It will be treated as element-wise op
EfficientNet(
(_conv_stem): Conv2dStaticSamePadding(
3, 2, kernel_size=(3, 3), stride=(2, 2), bias=False
(static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
)
(_bn0): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_blocks): ModuleList(
(0): MBConvBlock(
(_depthwise_conv): Conv2dStaticSamePadding(
2, 2, kernel_size=(3, 3), stride=[1, 1], groups=2, bias=False
(static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
)
(_bn1): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
2, 4, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
4, 2, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
2, 4, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(4, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(1): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
4, 46, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(46, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
46, 46, kernel_size=(3, 3), stride=[2, 2], groups=46, bias=False
(static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
)
(_bn1): BatchNorm2d(46, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
46, 2, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
2, 46, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
46, 2, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(2): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
2, 94, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
94, 94, kernel_size=(3, 3), stride=(1, 1), groups=94, bias=False
(static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
)
(_bn1): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
94, 3, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
3, 94, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
94, 2, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(2, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(3): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
2, 94, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
94, 94, kernel_size=(5, 5), stride=[2, 2], groups=94, bias=False
(static_padding): ZeroPad2d(padding=(1, 2, 1, 2), value=0.0)
)
(_bn1): BatchNorm2d(94, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
94, 3, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
3, 94, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
94, 5, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(5, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(4): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
5, 190, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
190, 190, kernel_size=(5, 5), stride=(1, 1), groups=190, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
190, 5, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
5, 190, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
190, 5, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(5, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(5): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
5, 190, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
190, 190, kernel_size=(3, 3), stride=[2, 2], groups=190, bias=False
(static_padding): ZeroPad2d(padding=(0, 1, 0, 1), value=0.0)
)
(_bn1): BatchNorm2d(190, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
190, 5, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
5, 190, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
190, 20, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(20, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(6): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
20, 430, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
430, 430, kernel_size=(3, 3), stride=(1, 1), groups=430, bias=False
(static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
)
(_bn1): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
430, 10, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
10, 430, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
430, 20, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(20, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(7): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
20, 430, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
430, 430, kernel_size=(3, 3), stride=(1, 1), groups=430, bias=False
(static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
)
(_bn1): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
430, 10, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
10, 430, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
430, 20, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(20, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(8): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
20, 430, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
430, 430, kernel_size=(5, 5), stride=[1, 1], groups=430, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(430, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
430, 10, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
10, 430, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
430, 52, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(52, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(9): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
52, 622, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
622, 622, kernel_size=(5, 5), stride=(1, 1), groups=622, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
622, 18, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
18, 622, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
622, 52, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(52, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(10): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
52, 622, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
622, 622, kernel_size=(5, 5), stride=(1, 1), groups=622, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
622, 18, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
18, 622, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
622, 52, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(52, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(11): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
52, 622, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
622, 622, kernel_size=(5, 5), stride=[2, 2], groups=622, bias=False
(static_padding): ZeroPad2d(padding=(1, 2, 1, 2), value=0.0)
)
(_bn1): BatchNorm2d(622, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
622, 18, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
18, 622, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
622, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(12): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
1102, 1102, kernel_size=(5, 5), stride=(1, 1), groups=1102, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
1102, 38, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
38, 1102, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
1102, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(13): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
1102, 1102, kernel_size=(5, 5), stride=(1, 1), groups=1102, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
1102, 38, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
38, 1102, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
1102, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(14): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
1102, 1102, kernel_size=(5, 5), stride=(1, 1), groups=1102, bias=False
(static_padding): ZeroPad2d(padding=(2, 2, 2, 2), value=0.0)
)
(_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
1102, 38, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
38, 1102, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
1102, 112, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(112, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
(15): MBConvBlock(
(_expand_conv): Conv2dStaticSamePadding(
112, 1102, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn0): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_depthwise_conv): Conv2dStaticSamePadding(
1102, 1102, kernel_size=(3, 3), stride=[1, 1], groups=1102, bias=False
(static_padding): ZeroPad2d(padding=(1, 1, 1, 1), value=0.0)
)
(_bn1): BatchNorm2d(1102, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_se_reduce): Conv2dStaticSamePadding(
1102, 38, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_se_expand): Conv2dStaticSamePadding(
38, 1102, kernel_size=(1, 1), stride=(1, 1)
(static_padding): Identity()
)
(_project_conv): Conv2dStaticSamePadding(
1102, 300, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn2): BatchNorm2d(300, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_swish): MemoryEfficientSwish()
)
)
(_conv_head): Conv2dStaticSamePadding(
300, 1260, kernel_size=(1, 1), stride=(1, 1), bias=False
(static_padding): Identity()
)
(_bn1): BatchNorm2d(1260, eps=0.001, momentum=0.010000000000000009, affine=True, track_running_stats=True)
(_avg_pooling): AdaptiveAvgPool2d(output_size=1)
(_dropout): Dropout(p=0.2, inplace=False)
(_fc): Linear(in_features=1260, out_features=1000, bias=True)
(_swish): MemoryEfficientSwish()
)
Params: 5288548 => 3784884
Output: torch.Size([1, 1000])
------------------------------------------------------
from torch-pruning.
ooodragon94
commented on June 8, 2024
it's working perfectly!
one question if I may...
can I specify percentage in which I want to prune?
hm.. I tried validating on the pruned model (which is loaded as pretrained) and its giving me 0.1% accuracy
is this normal before finetuning?
Oh I see.. so it is pruning random indexes and giving back a newly made modules
hm.. is there option on pruning a pretrained model?
from torch-pruning.
VainF
commented on June 8, 2024
Finetuning is necessary after pruning. The exaple code is only for testing.
from torch-pruning.
ooodragon94
commented on June 8, 2024
ok great!
thanks a lot for your help!
from torch-pruning.
ooodragon94
commented on June 8, 2024
hello sorry to interrupt again
I'm trying to prune another efficientnet b0 model from here
from models import *
import torch
import torch.nn as nn
import random
import torch_pruning as tp
model = EfficientNetB0()
model.eval()
example_inputs = torch.randn(1,3,224,224)
prunable_module_type = ( nn.Conv2d, nn.BatchNorm2d )
prunable_modules = [ m for m in model.modules() if isinstance(m, prunable_module_type) ]
DG = tp.DependencyGraph().build_dependency(model, example_inputs=example_inputs)
for layer_to_prune in prunable_modules:
# select a layer
if isinstance( layer_to_prune, nn.Conv2d ):
prune_fn = tp.prune_conv
elif isinstance(layer_to_prune, nn.BatchNorm2d):
prune_fn = tp.prune_batchnorm
ch = tp.utils.count_prunable_channels( layer_to_prune )
rand_idx = random.sample(list(range(ch)), min(10, ch//2))
plan = DG.get_pruning_plan( layer_to_prune, prune_fn, rand_idx)
plan.exec()
and such error shows up
File "/home/user/anaconda3/lib/python3.7/site-packages/torch_pruning/dependency.py", line 328, in get_pruning_plan
root_node = self.module_to_node[module]
KeyError: Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)
I'm looking at self.module_to_node dictionary at it seems like the model above is not added for some reason and I can't see why it fails.
May I get some help on this?
thank you
from torch-pruning.
KIM7AZEN
commented on June 8, 2024
hello sorry to interrupt again
I'm trying to prune another efficientnet b0 model from herefrom models import * import torch import torch.nn as nn import random import torch_pruning as tp model = EfficientNetB0() model.eval() example_inputs = torch.randn(1,3,224,224) prunable_module_type = ( nn.Conv2d, nn.BatchNorm2d ) prunable_modules = [ m for m in model.modules() if isinstance(m, prunable_module_type) ] DG = tp.DependencyGraph().build_dependency(model, example_inputs=example_inputs) for layer_to_prune in prunable_modules: # select a layer if isinstance( layer_to_prune, nn.Conv2d ): prune_fn = tp.prune_conv elif isinstance(layer_to_prune, nn.BatchNorm2d): prune_fn = tp.prune_batchnorm ch = tp.utils.count_prunable_channels( layer_to_prune ) rand_idx = random.sample(list(range(ch)), min(10, ch//2)) plan = DG.get_pruning_plan( layer_to_prune, prune_fn, rand_idx) plan.exec()and such error shows up
File "/home/user/anaconda3/lib/python3.7/site-packages/torch_pruning/dependency.py", line 328, in get_pruning_plan root_node = self.module_to_node[module] KeyError: Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False)I'm looking at self.module_to_node dictionary at it seems like the model above is not added for some reason and I can't see why it fails.
May I get some help on this?thank you
Has the problem been solved? I have the same issue. get error while pruning a bottleneck layer
from torch-pruning.
ooodragon94
commented on June 8, 2024
@KIM7AZEN no I think it worked quite well
maybe check your code's version??
from torch-pruning.
Related Issues (20)
- Question: Why is Eigendamage and FPGM not docummented nor exposed in importance.py HOT 1
- Prune Swin Transfomer:Could not find image processor class in the image processor config or the model config.
- Pruned Yolov8 model not loading? HOT 3
- detecting unwrapped parameters HOT 1
- AttributeError: Can't get attribute '__main__' on <module 'builtins' (built-in)>
- Error when pruning a group with GLU Layer HOT 4
- Improve Iterative Pruning: Verify Pruning Status Before Training HOT 1
- The QRcode of WeChat group has expired, could anyone please share a new one? HOT 1
- Yolov8 upgrade ultralytics version HOT 2
- Cannot calculate importance score for a single group HOT 1
- What is 'N' in "Iห = N ยท I / P{TopN(I )}" HOT 1
- Yolov8 pruning with Taylor criteria
- Add hooks to a module (make sure the hooks are removed), cannot deepcopy OBB model HOT 1
- How to make tracer account for arbitrary restrictions such as making a particular dim divisible by 4? HOT 1
- Hi i have already sparsity the gamma of layerNorm in Segformer. Can I use the methode to prune the model ?
- Clarification around MetaPruner
- How can I get the mask of the pruned model when I apply the soft pruning?
- loss stop reducing after pruning.
- Can the process of structural pruning be rolled back?
- The parameters and computation decreased, but FPS was worse than the baseline model HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from torch-pruning.