Create GridSearch + RandomForest example with Joblib about dask-examples HOT 3 CLOSED

Hi, @mrocklin
In the example, the GridSearch function is used. That means several models will be built, and then their scores will be compared.
I want to know whether either all workers do one single task together, or each worker implements different tasks (models with different hyperparameters).

Recently, I am training a large model:

from sklearn.datasets import make_regression
X, y = make_regression(n_samples = 6300000, n_features = 55, random_state=1) 
### size is 6,300,000, huge!

from sklearn.ensemble import RandomForestRegressor
model = RandomForestRegressor(n_estimators=1000, oob_score=True, random_state=1, n_jobs=36)
model.fit(X, y)

I tried it on one node (180GB memory, 36 cores) of a cluster (I have 16 these nodes in this cluster), but it failed. However, if I use the node with 360GB and 36 cores, it can run smoothly (This is a different type and I just have one).
So. memory is the limitation. Furthermore, training the model on one node with 36 cores is slow, even though the training works on the 360GB node. Of course, I hope that I can use all the cores.
Is this possible to be distributed?

Thank you!

from dask-examples.

0.20 is out

from dask-examples.

This original request is done in https://examples.dask.org/machine-learning/scale-scikit-learn.html

@MichaelChaoLi-cpu if you have usage questions, I think the Dask discourse is the best place for them these days. Make sure to include a minimal example if you can.

from dask-examples.