Comments (2)
Issue 2 makes me sad. AFAICT the symbol is not being simplified out because it's part of guard expressions:
Eq(s5*s6*s7, 60)
from the view decomp. Solving fors7
givesEq(s7, 60/(s5*s6))
, and then the logic attempts to replaces7
with60/(s5*s6)
, but the latter doesn't haveis_integer=True
, so this branch is skipped:
pytorch/torch/fx/experimental/symbolic_shapes.py
Lines 4947 to 4951 in 2e5366f
Eq(s7, (15//s6))
doesn't allow for symbol replacement either; it throws here:
pytorch/torch/fx/experimental/symbolic_shapes.py
Lines 4913 to 4914 in 2e5366f
TL;DR: can't do symbol replacement for any of the guards so the ephemeral symbol created during view replay can't be simplified out
from pytorch.
Here we know that s5, s6 and s7 divide 60 evenly. Thus, the replacement should be allowed. I would probably try a fix along those lines.
However, I've generally found it is better to express things in terms of s5s6s7, instead of trying to replace s7 with 60/(s5*s6), so it would be good to understand why the replacement is preferred as well.
from pytorch.
Related Issues (20)
- DISABLED test_flash_attention_vs_math_ref_grads_batch_size_1_seq_len_q_1024_seq_len_k_1024_head_dim_16_is_causal_False_dropout_p_0_48_bfloat16_scale0_cuda_bfloat16 (__main__.TestSDPACudaOnlyCUDA) HOT 1
- DISABLED test_flash_attention_vs_math_ref_grads_batch_size_1_seq_len_q_1024_seq_len_k_1024_head_dim_64_is_causal_False_dropout_p_0_48_float16_scale_l1_cuda_float16 (__main__.TestSDPACudaOnlyCUDA) HOT 1
- DISABLED test_flash_attention_vs_math_ref_grads_batch_size_1_seq_len_q_1024_seq_len_k_1024_head_dim_16_is_causal_False_dropout_p_0_0_bfloat16_scale_l1_cuda_bfloat16 (__main__.TestSDPACudaOnlyCUDA) HOT 4
- DISABLED test_flash_attention_vs_math_ref_grads_batch_size_1_seq_len_q_1024_seq_len_k_1024_head_dim_64_is_causal_True_dropout_p_0_48_bfloat16_scale0_cuda_bfloat16 (__main__.TestSDPACudaOnlyCUDA) HOT 1
- DISABLED test_flash_attention_vs_math_ref_grads_batch_size_1_seq_len_q_1024_seq_len_k_1024_head_dim_16_is_causal_True_dropout_p_0_48_bfloat16_scale0_cuda_bfloat16 (__main__.TestSDPACudaOnlyCUDA) HOT 1
- The empty 1D or more D tensor for `mode()` with the deepest `dim` doesn't work HOT 3
- Masked Attention has no effect in ``TransformerEncoderLayer``
- DISABLED test_sdp_math_gradcheck_contiguous_inputs_True_cuda (__main__.TestSDPACUDA) HOT 1
- torch.nonzero with known count to avoid host-device synchronization HOT 2
- FlashAttention IMA HOT 1
- An API to re-enable dynamo-disabled functions HOT 6
- [torch.compile][HF] torch.compile issue tracker for torch.compile on forward method of Meta Llama model HOT 3
- ONNX: wrong operator for ceil_mode Pooling in case of skip the last window HOT 1
- The empty 0D or more D tensor with exceeding `dim`(`dim=100` or `dim=-100`) for `cummin()` works
- The empty 0D or more D tensor with exceeding `dim`(`dim=100` or `dim=-100`) for `cummax()` works HOT 1
- Failing to export ONNX when `GroupNorm` input shape = 2 HOT 1
- The shape of weights in lstm seems to be wrong. HOT 1
- `input` argument of `nan_to_num()` works with `complex` type but `nan`, `posinf` and `neginf` argument don't work with `complex` type
- torch cuda-121 fails to load .so library HOT 2
- MPS and cpu method results are drastically different HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch.