Comments (3)
Hi,
From what I understand, you're reimplementing the meProp method using your own code. As I do not know your implementation, I can't answer your question in detail. But here are some points that I think could be useful for you:
- Implement simple unified top-k. The original top-k is not suitable for GPU-style parallel computation.
- Refrain from using sparse matrix operations. Again, it is not suitable for GPUs to accelerate sparse matrix operations. Instead, extract the columns or rows to build a new matrix and then use normal matrix operations (the provided code uses this implementation).
- Speedup on GPUs is more meaningful for heavy models that can fully assume the GPU's computational power. The reason is explained in Section 4.8 in the paper. In essence, models of relatively small sizes (e.g. 64 and 512) have almost the same speeds on GPUs.
Best regards
from meprop.
Dear @jklj077,
Can you please just provide a proper simple unified top-k implementation, which we can plug & play into our codes (as in the following example). Thank you very much for your time and consideration.
Note: The following one works but doesn't result in any speed improvement in the training of the CNN.
def topk(x, k = 0.5):
original_size = None
if x.dim() > 2:
original_size = x.size()
x = x.view(x.size(0), -1)
ax = torch.abs(x.data)
topk, _ = ax.topk(int(x.size(-1)*k))
topk = topk[:, -1]
y = x.clone()
y[ax < topk.repeat(x.size(-1), 1).transpose(0, 1)] = 0
if original_size: y = y.view(original_size)
return y
....
self.conv = nn.Conv1d(in_channels, out_channels, **kwargs)
def forward(self, x):
return topk(self.conv(x)) if self.training else self.conv(x)
from meprop.
Dear @jklj077,
Can you please just provide a proper simple unified top-k implementation, which we can plug & play into our codes (as in the following example). Thank you very much for your time and consideration.
Note: The following one works but doesn't result in any speed improvement in the training of the CNN.
def topk(x, k = 0.5): original_size = None if x.dim() > 2: original_size = x.size() x = x.view(x.size(0), -1) ax = torch.abs(x.data) topk, _ = ax.topk(int(x.size(-1)*k)) topk = topk[:, -1] y = x.clone() y[ax < topk.repeat(x.size(-1), 1).transpose(0, 1)] = 0 if original_size: y = y.view(original_size) return y
.... self.conv = nn.Conv1d(in_channels, out_channels, **kwargs) def forward(self, x): return topk(self.conv(x)) if self.training else self.conv(x)
@akaniklaus We're sorry for your inconvenience and we really wish we could help. For linear layers, we actually found a simple way (i.e., simple unified top-k) in the Python side of PyTorch (pre v0.3) that can optimize the speed on GPUs. That is what this repo is doing. For CNNs, there is simply no API we can tweak in the Python side to show a similar effect because all the operations are coded in the C++ side. Plug & play and showing real speedups needs digging really deeper into the PyTorch implementation and writing specific optimization for every different scenario, which is beyond our means.
PS: Actually, the snippet you provided is doing top-k in the forward propagation. You may need to write a backward method for the module class and doing top-k there, if meProp is what you're trying to implement. But it is irrelevant to speed, so I think it doesn't matter now.
Best regards
from meprop.
Related Issues (3)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from meprop.