atif-hassan / frufs Goto Github PK

hi

Thanks for this library and the clear description on how to use it. I wish to use it to compare to ur manual analysis of feature importances on the dataset that we have. Our data set is a bit bigger than the datasets i have seen in the readme. It has around 180000 samples and we have 99 features on which we wish to run analysis using FRUFS.
Furthermore I used XGBoost regressors for both models that one has to specify in the calls to make the FRUFS model.
I seem to always run into a crash where in the interpreter complains that it ran out of memory

I initialize my model as follows
import xgboost as xgb

# Initialize the FRUFS object
model_frufs = FRUFS(model_r=xgb.XGBRegressor(random_state=27, n_jobs=24), model_c=xgb.XGBRegressor(random_state=27, class_weight="balanced"), k=13, n_jobs=24, verbose=0, random_state=27)
# The fit_transform function is a wrapper for the fit and transform functions, individually.
# The fit function ranks the features and the transform function prunes the dataset to selected set of features
df_train_pruned = model_frufs.fit_transform(XGB_init.dataset_handler.x_train)
df_test_pruned = model_frufs.transform(XGB_init.dataset_handler.x_test)
# Get a plot of the feature importance scores
model_frufs.feature_importance()

I get the following error

TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

The exit codes of the workers are {SIGSEGV(-11)}

I am wondering if you have any suggestion on what i might be doing wrong?