"Transform" topic name before uploading to S3 about secor HOT 5 CLOSED

Secor supports this use case via subclassing MessageParser. You can configure a custom parser by setting the secor.message.parser.class param.

from secor.

Ah, you're right, I completely overlooked the parse(Message message) method, I'll give it a go and see if everything works as expected.

from secor.

For our use case it's actually not possible to transform the topic name in the parse method as the topic name is used in various places throughout Secor, for example for writing offsets to ZK, and slashes are not allowed in topic names. It might still work, but it feels more of a "fortunate" hack than a feature.

from secor.

Actually I think I know what you mean. Sorry about the confusion. I'm still not extremely excited about this kind of feature since it seems very specialized and I would prefer to keep secor code simple. As mentioned before, secor supports custom partition names for messages posted to a topic so you can control the subdirectory structure.

from secor.

Understandable. Perhaps another way to look at it would be to make it possible to exclude the topic name from the path. Would be configurable of course, but as you mentioned there's quite a lot of configuration parameters already :).

I'm gonna look at possible solutions during the coming week, and if I stumble upon a good solution I'll make a pull request and we can take it from there.

from secor.