After several discussions and a lot of thought, I am proposing and working on the following major change:
Example
Suppose I have a ddf of the iris data:
I want a new division of this data, split by slices of petal length. I want a new variable, plCut
that represents the slicing. If I were to add this variable to a regular data frame, I would do something like this:
iris$plCut <- cut(iris$Petal.Length, seq(0, 7, length = 3))
To add the variable to a ddf, the current approach is to use preTransFn
in divide()
, so that prior to the divide operations being applied to each subset, the transformation function is applied:
Current approach
byPL <- divide(irisDdf, by = "plCut", preTransFn = function(x) {
x$plCut <- cut(x$Petal.Length, seq(0, 7, length = 3))
x
})
Proposed approach
My proposed approach is to provide a function addTransform()
, which allows you to add a transformation function to a distributed data object, creating a new object that is then passed to the divide()
or any other method.
irisDdfPL <- addTransform(irisDdf, function(x) {
x$plCut <- cut(x$Petal.Length, seq(0, 7, length = 3))
x
})
byPL <- divide(irisDdfPL, by = "plCut")
This seems like a much more natural approach. When you call divide()
, you know the object you are passing contains the plCut
variable instead of having to think about supplying a preTransFn
in the same step.
From a design point of view, calling addTransform()
does not result in any computation - the computation is deferred until any of the dr
-prefixed functions are called (drJoin()
, drLapply()
, etc.), as well as divide()
and recombine()
. irisDdfPL
still points to the same data as irisDdf
, but behaves as if it were already transformed. For example, irisDdfPL[[1]]
will return the first key-value pair including the new column.
This approach is even more natural when thinking about how recombine()
is currently implemented. Currently, you specify both the apply
and combine
in recombine()
, which is not completely intuitive. With the proposed approach, the apply would be adding a transform to any ddo/ddf object, and the recombine would just be applying a combine method to the transformed object.
Implications for current design
This change would make the preTransFn
argument in all of the dr
-prefixed functions as well as divide()
and recombine()
redundant and unnecessary. It would be cleanest to completely remove preTransFn
, but that would be a breaking change. We could accomodate both approaches to remain compatible, but with the user base being relatively small and changes to code negligible, I'd rather go with the clean approach, with the option for those who are using this in a project-critical way can stick with version 0.72 until they are ready to update their code.
Any thoughts on whether to remove preTransFn
? Also, is the name addTransform
clear enough?