potential regression on scalding joinWithTiny on 4.5-wip about cascading HOT 7 CLOSED

here is the scalding test:

class TinyJoinAndMergeJob(args: Args) extends Job(args) {
  import TinyJoinAndMergeJob._

  val people = peopleInput.read.mapTo(0 -> 'id) { v: Int => v }

  val messages = messageInput.read
    .mapTo(0 -> 'id) { v: Int => v }
    .joinWithTiny('id -> 'id, people)

  (messages ++ people).groupBy('id)(_.size('count)).write(output)
}

here is the input for the tests:

object TinyJoinAndMergeJob {
  val peopleInput = TypedTsv[Int]("input1")
  val peopleData = List(1, 2, 3, 4)

  val messageInput = TypedTsv[Int]("input2")
  val messageData = List(1, 2, 3)

  val output = TypedTsv[(Int, Int)]("output")
  val outputData = List((1, 2), (2, 2), (3, 2), (4, 1))
}

previous behavior:

(1,2,3) joins against (1,2,3,4) creating (1,2,3). 
(1,2,3) + (1,2,3,4) -> (1,1,2,2,3,3,4).
(1,1,2,2,3,3,4) count by key -> ((1,2), (2,2) (3,2), (4,1))

current behaviour: every key is being over-counted by one.

from cascading.

I think this is also possibly related to cascading.avro.AvroScheme third-party package is leaking an old cascading 2.X version onto classpath. That project seems to be completely abandoned right now.

from cascading.

So there is this: https://github.com/cwensel/cascading-avro

from cascading.

thanks for the fork and link. We're still trying to setup some discussion with the scalding project to figure out what a potential upgrade path might look like. Since it's not my project it's not clear to me what is a "must have" to support. I will try to get back once I have more info on what the plan is.

from cascading.

It makes sense to push the avro fork out to maven central, but I don't have time to patch it since it's using maven as the build. If you guys need it, feel free to push PRs to it so we can get it out. I also don't have a build server for it, so it will need a github workflow action(s) as well.

Note, the hardest part of pushing to maven central will be getting the private keys on the build server. We can collaborate on that bit.

from cascading.

@daniel-sudz has this been resolved?

from cascading.

@cwensel I'm going to close this I think it's unlikely that scalding will adopt cascading 4.X in the near future so I don't really have time to look into it.

from cascading.