Error when compiling YOLO-NAS model about deepsparse HOT 11 CLOSED

@Y-T-G Here is a quick colab notebook I made using T4 GPU to one-shot sparsify/quantize the model. https://colab.research.google.com/drive/1DLB-tE1ide-55b9gzq6kQyrrW0lvT7xj?usp=sharing

It uses our Sparsify tool (in alpha right now, so leave feedback!) to optimize the ONNX with some (dummy) calibration data: https://github.com/neuralmagic/sparsify

Here is the low-sparsity ONNX: https://drive.google.com/file/d/1qMZCtikHtS4Edy0EBP9R7qX5i9eLyOvz/view?usp=sharing
Here is the high-sparsity ONNX: https://drive.google.com/file/d/1XkVYhX4SJfM0mLRuH-RIx6F0xQ9-vAYy/view?usp=drive_link

I used dummy data so the model likely isn't accurate, but you can substitute real input data to maintain.

If you want to talk more on this, happy to jump on a call or join our slack to ask question: https://join.slack.com/t/discuss-neuralmagic/shared_invite/zt-q1a1cnvo-YBoICSIw3L1dmQpjBeDurQ

Base model benchmark:

Sparsified model benchmark:

from deepsparse.

Hey @Y-T-G, thanks for the bug report. We can reproduce this issue and have a fix for it. I'll ping you once it's available in deepsparse-nightly

from deepsparse.

Hello @Y-T-G
The latest nightly has been mounted. Please now try pip install deepsparse-nightly - THANK YOU! 🥇
Jeannie / Neural Magic

from deepsparse.

Hi @Y-T-G here is a Colab notebook showing how to export the ONNX and run it on deepsparse-nightly: https://colab.research.google.com/drive/16r8fLUgAEqPWbDlQmgrmmuq8WvFXxLnQ?usp=sharing

from deepsparse.

A fix will be available the next time a nightly release goes out, and I'll close the issue then!

from deepsparse.

@tlrmchlsmth Cool. Thanks for the fix.

from deepsparse.

When can I expect the nightly to be available?

from deepsparse.

@jeanniefinks @mgoin Thanks. I will try it out.

from deepsparse.

I was able to test it on C++ and it works. Thanks.

from deepsparse.

Thanks for sharing @Y-T-G , very cool project! Let me know if you'd be interested in sparsifying the model for more performance

from deepsparse.

@mgoin Sure. That would be great. I was wondering how to improve the FPS further.

from deepsparse.