Comments (4)
quetric
commented on September 14, 2024
Hi @Mellich
Could you clarify this: "It occurs using the UDP and TCP stack."
I can see a mechanism whereby backpressure from the RX pipeline causes the UDP stack to drop packets. This shouldn't happen for the TCP stack though.
Please add ILAs to the input and output of the UDP stack to give us visibility into what's going on, and share the waveforms.
from accl.
Mellich
commented on September 14, 2024
Could you clarify this: "It occurs using the UDP and TCP stack."
I can see a mechanism whereby backpressure from the RX pipeline causes the UDP stack to drop packets. This shouldn't happen for the TCP stack though.
I executed the same experiment with another bitstream containing the TCP stack. Also there, the execution gets stuck after several iterations when large message sizes are used.
The issue occurred more frequently for message sizes of 512KB and above (-s 131072). I mainly tested powers of two so far, but it seems there is no strict line where this issue occurs. It rather seems to get more likely the larger the message.
But I did not observe many differences between the UDP and TCP bitstreams, for now, so I would assume this is an issue independent of the network stack... or two different issues leading to similar behavior.
from accl.
TristanLaan
commented on September 14, 2024
I have been able to replicate this bug on our infrastructure with the following settings: bin/test -d -u -f -x ${XCLBIN_UDP} -b 1024 -s 131072 --rsfec -n 2000
I have changed @Mellich his code a little bit in my branch so that it automatically prints some statistics if the send/recv gets stuck. I've attached the statistics that I got from our infrastructure. Most notably, the CMAC numbers seem to match, but the Network Layer numbers don't. Is this maybe a problem with VNx instead of ACCL? (This of course doesn't explain the TCP problems)
Rank 0:
CMAC stats:
Cycle count: 831533921
Tx:
bad_fcs: 0
bytes: 305785848
good_bytes: 305785848
good_packets: 195108
packets: 195108
packets_1024_1518B: 0
packets_128_255B: 0
packets_1519_1522B: 0
packets_1523_1548B: 0
packets_1549_2047B: 193006
packets_2048_4095B: 0
packets_256_511B: 0
packets_4096_8191B: 0
packets_512_1023B: 566
packets_64B: 1536
packets_65_127B: 0
packets_8192_9215B: 0
packets_large: 0
packets_small: 0
pause: 0
user_pause: 0
Rx:
bad_fcs: 0
bytes: 306265447
good_bytes: 306265447
good_packets: 194461
packets: 194461
packets_1024_1518B: 0
packets_128_255B: 0
packets_1519_1522B: 0
packets_1523_1548B: 0
packets_1549_2047B: 193347
packets_2048_4095B: 0
packets_256_511B: 13
packets_4096_8191B: 0
packets_512_1023B: 567
packets_64B: 534
packets_65_127B: 0
packets_8192_9215B: 0
packets_bad_fcs: 0
packets_fragmented: 0
packets_jabber: 0
packets_large: 0
packets_oversize: 0
packets_small: 0
packets_toolong: 0
packets_undersize: 0
pause: 0
stomped_fcs: 0
user_pause: 0
Network Layer stats:
udp out bytes = 302203248
ethhi_out_bytes = 304913256
eth_out_bytes = 305005416
udp in bytes = 301939892
udp app out bytes = 296783232
udp app in bytes = 296524608
Rank 1:
CMAC stats:
Cycle count: 3733636461
Tx:
bad_fcs: 0
bytes: 306325932
good_bytes: 306325932
good_packets: 195450
packets: 195450
packets_1024_1518B: 0
packets_128_255B: 0
packets_1519_1522B: 0
packets_1523_1548B: 0
packets_1549_2047B: 193347
packets_2048_4095B: 0
packets_256_511B: 0
packets_4096_8191B: 0
packets_512_1023B: 567
packets_64B: 1536
packets_65_127B: 0
packets_8192_9215B: 0
packets_large: 0
packets_small: 0
pause: 0
user_pause: 0
Rx:
bad_fcs: 0
bytes: 305724888
good_bytes: 305724888
good_packets: 194115
packets: 194115
packets_1024_1518B: 0
packets_128_255B: 0
packets_1519_1522B: 0
packets_1523_1548B: 0
packets_1549_2047B: 193006
packets_2048_4095B: 0
packets_256_511B: 12
packets_4096_8191B: 0
packets_512_1023B: 566
packets_64B: 531
packets_65_127B: 0
packets_8192_9215B: 0
packets_bad_fcs: 0
packets_fragmented: 0
packets_jabber: 0
packets_large: 0
packets_oversize: 0
packets_small: 0
packets_toolong: 0
packets_undersize: 0
pause: 0
stomped_fcs: 0
user_pause: 0
Network Layer stats:
udp out bytes = 302737176
ethhi_out_bytes = 305451972
eth_out_bytes = 305544132
udp in bytes = 302203248
udp app out bytes = 297307584
udp app in bytes = 296783232
from accl.
quetric
commented on September 14, 2024
Traced this problem back to ACCL applying backpressure into the POE which causes packet loss with UDP and with TCP if RX bypass is enabled.
The current work-around is to use TCP and disable RX bypass. I made this configuration default for now.
Keeping the issue open while I debug the cause for the backpressure
from accl.
Related Issues (20)
- Deadlocks with "complex" communication patterns HOT 8
- Copy fails with Coyote implementation
- "lvalue required as left operand of assignment" when using Coyote buffer subscript operator
- Unit Tests hang and fail on dev branch HOT 2
- ETH messages received by rank 1 although not destination of message HOT 1
- A soft reset routine for the CCLO should be introduced and applied during initialization
- Address not mapped: 0x78 HOT 1
- Data transfer not working above 8 MB HOT 1
- Build system misconfiguration for refdesign with Coyote + RDMA + U250 HOT 2
- Enable multiple kernel builds with different configurations HOT 1
- Update firmware in Xclbin HOT 2
- Error in Building hardware for Alveou280 HOT 5
- Sim support for host buffers HOT 1
- Update to latest Vitis HOT 1
- Variable clock frequency
- Emulator breaks with compiler optimizations (-O3)
- Reduce/Allreduce issues on cyt_rdma HOT 2
- Gather wrong order HOT 9
- Allreduce hangs after AlltoAll HOT 5
- Broadcast hangs on cyt_rdma HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from accl.