Proposal: Don't validate co-partitioning if there's no joins about goka HOT 7 CLOSED

We don't have such use cases, but I agree with you. Still you'll need to decide how many partitions (goroutines) the processor should have. The check for the partitions is done here.

from goka.

I think the easiest way how to implement this would be if you look for the input topic with most partitions in prepareTopics() and use that as the number of partitions in the processor. If you do otherwise, you'll also need to adapt the main processor loop so that it applies % operation on the partition of the incoming messages.

from goka.

@db7 I was thinking about changing copartitioned method to be like this:

 func (gg *GroupGraph) copartitioned() Edges {
+       if len(gg.joinCheck) == 0 {
+               return nil
+       }
+
        return append(gg.inputStreams, gg.inputTables...)
 }

So if there's no join topics, this method would return nil, and ensureCopartitioned would have nothing to work with.

But from what you're saying seems like that wouldn't be enough, right?

from goka.

This piece of code is important:

// check co-partitioned (external) topics have the same number of partitions
npar, err = ensureCopartitioned(tm, gg.copartitioned().Topics())
if err != nil {
	return 0, err
}

It returns the number of partitions that the processor will have. Here you should do something like
if "no group table and no joins", then pick npar form the topic with most partitions, otherwise call ensureCopartitioned.

from goka.

I started implementing that, but facing multiple edge cases and many tests started to fail. Will dig into that later. For now, working this limitation around by creating multiple processors in our systems.

from goka.

Ok. If you like, you can tell me the branch with your changes and I can try later to understand what is going on.

from goka.

I'll close it as it didn't have any activity for 2 years now

from goka.