Histogram analysis for categorical data with numerical values about deequ HOT 8 CLOSED

Hi Paul,

Happy to see that you are using Deequ! Actually, histograms on numeric columns should work, I wrote a test for this here #96 .

Can you provide us with more details about the error that you are seeing?

Best,
Sebastian

from deequ.

@sscdotopen I was using the column profiler and the histogram analysis was only working with String and boolean columns. On further investigation I found this code in ColumnProfiler.scala
`
/* Identifies all columns, which:
*

• (1) have type string or boolean
• (2) have less than lowCardinalityHistogramThreshold approximate distinct values
  */
  private[this] def findTargetColumnsForHistograms(
  schema: StructType,
  genericStatistics: GenericColumnStatistics,
  lowCardinalityHistogramThreshold: Long)
  : Seq[String] = {

val originalStringOrBooleanColumns = schema
  .flatMap { field =>
    if (field.dataType == StringType || field.dataType == BooleanType) {
      Some(field.name)
    } else {
      None
    }
  }
  .toSet

genericStatistics.approximateNumDistincts
  .filter { case (column, _) => originalStringOrBooleanColumns.contains(column) }
  .filter { case (column, _) =>
    genericStatistics.typeOf(column) == String || genericStatistics.typeOf(column) == Boolean
  }
  .filter { case (_, count) => count <= lowCardinalityHistogramThreshold }
  .map { case (column, _) => column }
  .toSeq

}
`

from deequ.

I could be wrong, but it looks like the column profiler is filtering out the numeric columns before using the histogram class you wrote the test for. Thanks for the quick response btw. Deequ is pretty sweet so far

from deequ.

Ah, good find. You are right that we do not consider low-cardinality numeric columns as categorical here. Would you like to write a PR to fix this?

from deequ.

Ya, I'll write up a pr later today.

from deequ.

Cool, looking forward to that!

from deequ.

pr for fix #97

from deequ.

#97 merged. Closing issue

from deequ.