could you give some examples about ragged input config for tensorrt backend about server HOT 4 OPEN

Hi @wanghuihhh, would you be able to try enabling ragged batching for the input on the config?

from server.

Hi @wanghuihhh, would you be able to try enabling ragged batching for the input on the config?

I have understand this blog. But I still don't success because the shape of model's input is 3D, but expect of ragged input is 1D. I don't know how to modify my model to adapt the feature.
In your introduction, you did not show how to flatten the input with feature dimensions to one dimension. Can you provide a configuration example of a standard BERT model? I believe many people need it.

from server.

Hi @matthewkotila @nv-hwoo, wondering if you have worked with a BERT model in the past? If so, would you be able to share the model config?

from server.

@kthui: Hi @matthewkotila @nv-hwoo, wondering if you have worked with a BERT model in the past? If so, would you be able to share the model config?

I am not familiar with any BERT models right now :/

from server.