Featurewiz key error during fit and transform about featurewiz HOT 7 CLOSED

Please note that you need to send in dataframes as input, not numpy arrays. You may have sent in arrays when you split the data intro train and test. Please double check. This code snippet works for me.

from featurewiz import FeatureWiz
features = FeatureWiz(corr_limit=0.70, feature_engg='', category_encoders='', dask_xgboost_flag=False, nrows=None, verbose=2)
X_train_selected = features.fit_transform(ord_train_t, y_train)
X_test_selected = features.transform(ord_test_t) # error is encountered here
features.features  ### provides the list of selected features ###

Please post your sample ord_train_t and ord_test_t data sets here for me to test out. Thanks for trying out featurewiz.
Thanks
AutoViML

from featurewiz.

@AutoViML - My ord_train_t is a dataframe. my y_train is also a dataframe, my ord_test_t is also a dataframe. please help me. The issue happens only when I try to transform the ord_test_t. But yes, ord_test_t contains only the input columns.

Please find below the sample data. I can't share original data due to confidentiality reasons. But I can say that most of my columns are ordinally encoded (except 4 columns).

ord_train_t

ord_test_t

my y_train is dataframe with discrete values 0 and 1. Hope this helps

from featurewiz.

@AutoViML - Hope this sample data useful for you to test. Both ord_train_t and ord_test_t doesn't contain the output column (label column)

from featurewiz.

Hi @SSMK-wq 👍
I have attached your dataset and my notebook to test it. There was no error at all. Can you please download this and use it in your laptop.
All_Main.zip
please download it and test it
Thanks
AutoViML

from featurewiz.

@AutoViML - I am sorry. For some reason that I am not sure why, I still get the same key error when I pass in ordinally encoded dataframes as input. Meaning,my fit function works well. It's only my transform on test data fails with key error. But my test data also has the same columns as train data. Anyway, I also understand for you it works fine. Am not sure, how can this be reproduced to solve this issue. Thanks for your help anyway. I will go back to manual feature selection. Moreover, when I run the fit function, I don't see the graphs for Xgboost feature that you have in your sample code. I don't know why it is behaving unusually in my instance. Meaning, there is no Xgboost graphs, no SULOV graphs as well. Might be there is no correlation.

Anyway to be specific, the below is where I get the key error

X_test_selected = features.transform(ord_test_t) #ord_test_t is test data (ordinally encoded) with no target column

from featurewiz.

@AutoViML Oh, I think I found what is the issue now. It was subtle. I think the package fills the spaces in column names with underscores. Meaning if my original column name is product group, the package converts them to product_group (with underscore in between). This results in key error (and prevents from transformation). Guess, during fit it is automatically handled. Apologies for overlooking this

from featurewiz.

Hi @SSMK-wq 👍
I think you caught a nice bug! thanks for bringing it to my attention. I will be uploading a new version 0.1.01

Please upgrade to it via:

pip install featurewiz --upgrade

You should be able to see that fit and transform work correctly now even if the column names are changed.

Thanks
AutoViML

from featurewiz.