Data loss during rebalance about secor HOT 7 CLOSED

Or maybe move mFileRegistry.deleteWriters(topicPartition); inside this if statement?

from secor.

@yyejun I don't understand the failure scenario. Every thread has its own directory where it keeps log files which means that writers are not shared between threads.

from secor.

It's not the file.
It's mFileRegistry.deleteWriters(topicPartition) will cause the file be deleted when new message comes in.
This should moved to after the if statement, otherwise, when the if statement return false. Nothing happens, file is not uploaded, but deleted from FileRegistry, everything saved so far will be lost in next new message.

from secor.

Basically mFileRegistry.deleteWriters is not a recoverable operation, it need to be inside the lock/check block. Because the 2nd check can return false in edge case.

from secor.

Not sure if my explanation clear enough.

This happens when https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L209 return True,
But https://github.com/pinterest/secor/blob/master/src/main/java/com/pinterest/secor/uploader/Uploader.java#L112 return False.

So Writer deleted, but file was not uploaded. Next message will call createOrGet writer, which will cause the saved file being deleted.

from secor.

I think I start to get what you are saying. Is this the failure scenario you have in mind?

1. consumer thread A saved locally data from offset 100 to 120. The committed offset is 99,
2. there was a rebalancing event and the ownership of the partition transferred to consumer B who saved locally data from offset 100 to 150,
3. there was another rebalancing that returned the ownership to A,
4. A saves locally data from 100 to 160,
5. A attempts to upload the data. It executes the logic in L209,
6. B uploads the data and commits the offset that now becomes 150,
7. A executes the logic in L112. Offsets 151-160 get lost.

from secor.

That's correct. It happened on a slow machine after 3 rebalance events which caused two thread has exact same file access time and then try to upload at same time.

from secor.