Comments (1)
Hi,
You are right, if the embeddings are not tied, we would have to use the output embedding table in which case the language model will not provide gradients to the input being generated. So at any update, a token gets signals either from the past tokens (via the usual forward pass) or from the future (via the gradients). If we use the output embeddings, the tokens won't receive gradients from the future tokens.
While it will affect the performance a little bit, the approach can still generate good samples with control because the token will still get gradients on the entire sequence from the control functions, which can now be defined as functions of output embeddings. I did some small-scale experiments with this setting and didn't observe a significant drop in performance (for toxicity avoidance). In general, I have observed that the gradients from the future tokens are not very meaningful especially if the sequence length is long.
Hope this answers your question.
Thanks,
from mucoco.
Related Issues (6)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mucoco.