How to process data in a minibatch setting about pytorch-widedeep HOT 2 CLOSED

Hey @Aceticia thank you!

ok, let's see:

1. If I create a dataframe that contains the same columns as real data and put in dummy examples that contain all possible combinations of categorical features it should work fine.

Not sure I follow...could you perhaps insert some pseudo code or an illustration of what you want to do? Let's see...if you real data has two columns, you would create a dataframe with the two columns, and then each column will have...what? combination of the categorical features IN THE TWO COLUMNS?

2. For the second step of standardizing continuous features, I'm thinking I can just write some dummy values before calling fit and overwriting the StandardScaler object with one that was already fit to all of my data.

to be honest, you could even contribute to the library and add the option of passing a custom "Standarizer". If not I can do it in the next release 🙂. I am still a bit confused with this: "I can just write some dummy values before calling fit [...]"

In any case, in case it helps, we now have custom dataloaders: https://github.com/jrzaurin/pytorch-widedeep/blob/master/pytorch_widedeep/dataloaders.py

You could add any that you like in that module. Overall, let me see if I understand. You want to Preprocess data on the fly? i.e. like running the TabPreprocessor per minibatch?

Let me know and see how can I help

Also, apart from commenting here (so is visible for more users), please, join the slack channel if you wanted :) : https://join.slack.com/t/pytorch-widedeep/shared_invite/zt-soss7stf-iXpVuLeKZz8lGTnxxtHtTw

from pytorch-widedeep.

Closing this due to lack of response/activity

from pytorch-widedeep.