[BUG] Unable to run LightGBMClassifier on large dataset about synapseml HOT 4 OPEN

Hey @dpdrmj 👋!
Thank you so much for reporting the issue/feature request 🚨.
Someone from SynapseML Team will be looking to triage this issue soon.
We appreciate your patience.

from synapseml.

Hello everyone! Can someone please help here? Does anyone know what could've caused this?

from synapseml.

Hello everyone! Can someone please help here? Does anyone know what could've caused this?

val (trainingData, validationData) =
  if (get(validationIndicatorCol).isDefined && dataset.columns.contains(getValidationIndicatorCol))
    (df.filter(x => !x.getBoolean(x.fieldIndex(getValidationIndicatorCol))),
      Some(sc.broadcast(preprocessData(df.filter(x =>
        x.getBoolean(x.fieldIndex(getValidationIndicatorCol)))).collect())))

if the validationData is large, the "collect" use many memory. you need to set driver.memory and executor.memory very large.

from synapseml.

Hello everyone! Can someone please help here? Does anyone know what could've caused this?

val (trainingData, validationData) =
  if (get(validationIndicatorCol).isDefined && dataset.columns.contains(getValidationIndicatorCol))
    (df.filter(x => !x.getBoolean(x.fieldIndex(getValidationIndicatorCol))),
      Some(sc.broadcast(preprocessData(df.filter(x =>
        x.getBoolean(x.fieldIndex(getValidationIndicatorCol)))).collect())))

if the validationData is large, the "collect" use many memory. you need to set driver.memory and executor.memory very large.

set driver.memory and executor.memory very large can fix it, but it is slow, consume many resource, hope hynapseML team find a new way to rewrite the code, hope to replace the "collect".

from synapseml.