Comments (1)
Hi! Sorry for the late response. You can find the attention weights as tensor "attention_weights" here:
DeepRC/deeprc/architectures.py
Line 417 in 108d08d
You would have to pass those tensors into the evaluation function somehow.
A quick-and-dirty hack would be to just set the attention weights as attribute. E.g. like this as modification of the part between lines 412- 434 :
# Compute representation per bag (N times shape (d_v,))
mb_emb_seqs_after_attention = []
mb_attention_weights_list = []
start_i = 0
for n_seqs in n_sequences_per_bag:
# Get sequence embedding h() for single bag (shape: (n_sequences_per_bag, d_v))
attention_weights = mb_attention_weights[start_i:start_i+n_seqs]
mb_attention_weights_list.append(attention_weights.detach())
# Get attention weights for single bag (shape: (n_sequences_per_bag, 1))
emb_seqs = mb_emb_seqs[start_i:start_i+n_seqs]
# Calculate attention activations (softmax over n_sequences_per_bag) (shape: (n_sequences_per_bag, 1))
attention_weights = torch.softmax(attention_weights, dim=0)
# Apply attention weights to sequence features (shape: (n_sequences_per_bag, d_v))
emb_seqs_after_attention = emb_seqs * attention_weights
# Compute weighted sum over sequence features after attention (format: (d_v,))
mb_emb_seqs_after_attention.append(emb_seqs_after_attention.sum(dim=0))
start_i += n_seqs
# Stack representations of bags (shape (N, d_v))
emb_seqs_after_attention = torch.stack(mb_emb_seqs_after_attention, dim=0)
# Calculate predictions (shape (N, n_outputs))
predictions = self.output_nn(emb_seqs_after_attention)
self.mb_attention_weights_list = mb_attention_weights_list
return predictions
Later you could access the attribute model.mb_attention_weights_list
in the evaluation function.
I hope that helps. Let me know if you need further help.
Best wishes, Michael
from deeprc.
Related Issues (9)
- conda env create -f condal_install.yml --name deeprdc_env HOT 1
- It seems that the link to cmv_with_implanted_signal, lstm_generated or simulated datasets has expired.
- docs on how to apply DeepRC to BCR HOT 2
- update torch version HOT 1
- Where is the self-attention operation HOT 1
- Datasets issue (?) HOT 3
- RuntimeError: "unfolded2d_copy" not implemented for 'Half' HOT 1
- example_single_task_cnn.py HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deeprc.