Comments (2)
ChenMnZ
commented on August 21, 2024
2
- In our observation, most of the activation outliers exist after LN, so the initialization of scale right after LN is important. As for the QK, the distributions are relatively uniform, and we find that
torch.ones()initialization is enough. We have tried your proposal and find that the SmoothQuant initialization can not bring benefit to QK. - Yes, it is feasible. In our method, we apply both qkv-scales and qkt-scales. qkv-scales change the weight in the input_channel dimension, and qkt-scales change the weight in the output channel dimension. Therefore, the collaboration of these two types scales can significantly enlarge the solution space.
from omniquant.
superdocker
commented on August 21, 2024
Thanks a lot!
from omniquant.
Related Issues (20)
- KeyError: 'Cache only has 0 layers, attempted to access layer with index 0' HOT 2
- When reproducing evaluation results for Llama-2-13b w4a4, I got nan HOT 4
- W4A4 in llama2-7b HOT 4
- AttributeError: 'FalconAttention' object has no attribute 'maybe_rotary' HOT 1
- TypeError: FalconRotaryEmbedding.forward() missing 1 required positional argument: position_ids
- Why is the compressed file one file instead of the pre trained weights, where there are many files for training the mode HOT 1
- Is activation get quantized on-the-fly? HOT 5
- Llama-3-8B HOT 5
- OPT-30B
- Questions regarding Infusing Omniquant into MLC HOT 3
- Some questions about the results of weight only quantification in the paper
- Which bug do you fix for auto_gptq HOT 1
- How to accelerate the inference speed with real_quant HOT 3
- Questions about quantization
- Questions about quantization
- The llama-1-65b model seems unstable in this code HOT 2
- [Model Request] MiniCPM
- question about let
- [New Feature] Seek MLA Supported by Smooth
- Quantify tinyllama-1.1B-Chat-v1.0, a CUDA assertion error occurred
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from omniquant.