Dataframe Encoders for TransmogrifAI types about transmogrifai HOT 4 OPEN

spark dataframe is dataset of Row, Encoder or Decode for row already exists ,you just need not to define new Encoder or Decoder for Row.

from transmogrifai.

@liuzhenhai93 yeah, Dataframe encoding is currently working. We would like to have a support for the following:

implicit val enc: Encoder[(Real, Text)] = ???
val reals: Dataset[(Real, Text)] = spark.createDataset(Seq(1.0.toReal -> "one".toText))

from transmogrifai.

@tovbinm you can try like this in scala
import spark.implicits._
case class Wrap[T](unwrap: T)
Then whenever you want to use custom type use them inside Wrap like this:
val dataFrame = spark.createDataset(Seq(Wrap(2.0,"hello")))

from transmogrifai.

I don’t believe this would work (I will check it). Ideally I would like to avoid allocating another wrapper class, since we already do so (FeatureType is a wrapper around Option, Seq, Map etc).

from transmogrifai.