gianlucamalato / collinearity Goto Github PK

Hi @gianlucamalato , nice and clean little gem you have there. Wish I found it earlier, as I did something very similar but not so clean as your solution.
Anyway would like to suggest an enhancement for my use case. I'm using mainly pandas dataframes with some 3000-4000 columns.
As these result from joining several pivot statements the number of columns can vary, also the sequence can change. Also in a new dataset, new columns may appear and previously existing ones may be gone.
Currently if I use .values on my dataframe to fit the X, collinearity will plow through. However when I try to transform a new dataframe, there's no guarantee the mask is still going to be correct or even fit.
So I would propose to detect if X is a pd.DataFrame and if so, keep next to the mask, a list of column names that were selected.
That way, if you next try to transform a new dataframe, the column names can be used to correctly reduce the features. Also columns that are not available in the new data, can be added and filled with a default value. Doing so would allow me finally to have all the steps to create my model in a single pipeline