Comments (6)
Hi Jannik,
I think here fc_o
(in the 5th line of the code you pasted) is the W_O
in the paper, what do you think?
from set_transformer.
Dear Jingweiz,
thanks for your reply!
I would have identified fc_o
with the rFF(H)
of the MAB and not with W_O
.
from set_transformer.
Hi, thanks for your interest!
- Multiplying W_0 after the concat and 2) multiplying W to the query to get Q and then split-attend-concat, in essence, makes a small difference (one of them is a restricted version of another). For the paper, I followed the description in the original transformer paper, and for the code, I chose the current form following the code available for original transfomer (also, it gives a cleaner code). But they don't make a big empirical difference.
from set_transformer.
Hey Juho Lee!
Thanks for your reply.
It makes sense that this does not give a big empirical difference. I just wanted to check if I missed something.
And LayerNorm(X + Multihead(X, Y, Y ; ω))
in the paper, should probably be something like LayerNorm(W_q X + Multihead(X, Y, Y ; ω))
, correct?
from set_transformer.
Dear Jingweiz,
thanks for your reply!
I would have identifiedfc_o
with therFF(H)
of the MAB and not withW_O
.
Oh right exactly, I got messed up, thanks!
from set_transformer.
I have a follow up question linked to this topic.
In the paper a row-wise FF block is used for the pooling, and unlike the rFF in the MAB, the rFF in the pooling doesn't have an activation function. Should the PMA rFF have an activation function or not?
from set_transformer.
Related Issues (19)
- License HOT 1
- A little puzzle about the implementation details. HOT 4
- PMA implementation missing rFF? HOT 2
- Question about model's input HOT 13
- Inputs of the SetTransformer HOT 1
- Question on dim_split in MAB
- Updating Multihead attention to include batch size>1?
- Why is LayerNorm default to False? HOT 2
- Question about ISAB
- Question about the normalization in the attention weight calculation
- Is there code for Amortized Clustering on CIFAR-100 available?
- about shape classification experiment HOT 1
- question about the network architecture for set transformer HOT 3
- Question about Deep Sets Implementation HOT 2
- about Set Anomaly Detection experiment code HOT 2
- Question about Counting Unique Characters code HOT 3
- 4-D equivalent? HOT 3
- LayerNorm HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from set_transformer.