SparkXGBClassifier does not validate params about xgboost HOT 6 CLOSED

Can you see a warning in the worker log?

from xgboost.

Hi @NicoG94, when you call fit, the xgboost will show the warning of unused parameters,

2024-03-13 19:03:55,097 INFO XGBoost-PySpark: _fit Running xgboost-2.0.3 on 1 workers with
	booster params: {'objective': 'binary:logistic', 'device': 'cpu', 'this_does_not_exist': 'really_not', 'nthread': 1}
	train_call_kwargs_params: {'verbose_eval': True, 'num_boost_round': 100}
	dmatrix_kwargs: {'nthread': 1, 'missing': nan}
[19:03:59] task 0 got new rank 0
/home/bobwang/anaconda3/envs/xgboost/lib/python3.10/site-packages/xgboost/core.py:160: UserWarning: [19:03:59] WARNING: /workspace/src/learner.cc:742: 
Parameters: { "this_does_not_exist" } are not used.

  warnings.warn(smsg, UserWarning)
2024-03-13 19:04:00,234 INFO XGBoost-PySpark: _fit Finished xgboost training!

from xgboost.

for me it does not show that unfortunately.

Call:

model = SparkXGBClassifier(
    num_workers=1,
    label_col=target_col,
    random_state=42,
    this_does_not_exist="really_not",
    n_estimators=10, 
)

trained_model = model.fit(train_data)

Output:

2024-03-13 11:28:57,093 INFO XGBoost-PySpark: _fit Running xgboost-2.0.3 on 1 workers with
	booster params: {'objective': 'binary:logistic', 'device': 'cpu', 'random_state': 42, 'this_does_not_exist': 'really_not', 'nthread': 1}
	train_call_kwargs_params: {'verbose_eval': True, 'num_boost_round': 10}
	dmatrix_kwargs: {'nthread': 1, 'missing': nan}
2024-03-13 11:29:00,789 INFO XGBoost-PySpark: _fit Finished xgboost training!

from xgboost.

in the worker logs it does show up. But it is really hard to find if you work with Databricks. I checked where it is logged. It is here. Any reason you are using print instead of logging here? Maybe if you would use logging, my console would be able to show it

from xgboost.

We have changed it into using the warning module in the latest but haven't released it yet. You can find the call to the warning module in your referenced link (right above the line you referred to).

from xgboost.

ah okay, thanks for the quick responses. Keep up the great work! :)

from xgboost.