Data loss since karma decay feature about lita-karma HOT 14 CLOSED

Well data loss is clearly not good. I would like to take a stab at fixing it. I suspect that the karma-cleanup branch is at least partially responsible, but there may be a bug in the karma code itself that branch exposed.

I don’t suppose you have a backup of the Redis instance before it all went to hell?

from lita-karma.

Further, especially given the testing we already did, I strongly suspect the problem lies in upgrading existing data. I’ll focus attention on that.

from lita-karma.

I'm afraid I don't have example data to share. I'll also look at this myself soon.

from lita-karma.

I’m going to take a look at this now. Incidentally, when I was working on karma decay, I wrote a script to copy Lita data from a free Redis2Go instance locally: https://gist.github.com/erikogan/a3fd06ae9634aa1256ff

I just added the ability to write to/load from a file, and in reviewing that change I noticed that karma actions appear to clump around certain times, rather than being spread normally.

I don’t yet know if that’s a likely cause of what you saw, but it’s definitely true that there is a bug in the default decay_distributor.

from lita-karma.

Ok, with a decay life of 60 days, an item upgraded with a few hundred karma has actions that clump around ~10 moments (with some outliers). I suspect that with a shorter lifetime that clustering is probably far worse. It certainly might explain why some items disappeared (after dropping to zero quickly), but I cannot say definitively.

I have no idea how that happened. It could very well be a bug in how I implemented the function I chose, or it could be that I chose the wrong function. I spent far too long tweaking it to get that function just right.

In the end, it is overly complex and adds questionable value. I think the right answer is to make the default distributor a linear burn-down. If someone wants something more complex than that, they are free to implement it themselves.

Sorry about that. The remaining question: How do you want to handle installations that have actions added with the broken distributor? I am inclined to delete all actions and re-calculate, but I thought I should ask first.

from lita-karma.

Thanks very much for looking into this. I would be very surprised if anyone but the two of us have been running off master, so I'm going to move the current master to a branch and then remove the decay features from master to make it safe again. Then we can spend as much time as we want perfecting decay until we're confident it can ship.

from lita-karma.

There is definitely an unintentional integer arithmetic (truncation) error in the original implementation, but even fixing that the curve does not behave as expected. I guess I just tweaked it so much I got it mixed up in my head.

(I still think the default should be linear, I was just hoping I could fix the slow + acceleration curve for an example in the README.)

from lita-karma.

I think I just had an epiphany that may explain the clumping. It has to do with how the upgrade adds actions for a user based on who modified them in the past. The more different modifiers, the more clumping.

If I am correct it needs to be fixed for us to get a truly linear distribution with the new default distributor, so I’ll take a stab at proving it in a spec, so that it’s reproducible.

from lita-karma.

Thanks for your fixes! Forget what I said earlier, I'll just let it bake on master and try it out in our room for a while.

from lita-karma.

If you’re going to run with it as is, you should probably blow away the (badly) upgraded actions from previous installs and reinstate them via the new upgrade mechanism.

Either:

1. Simply remove the entire actions sorted-set and regenerate from the data you have, or
2. redis.zremrangebyscore(:actions, '-inf', Time.new(<args for upgrade date>).to_f) To remove the actions added by the upgrade process.

Then reds.del('support:decay_with_negatives') will cause the upgrade to fill in the missing actions on next start.

from lita-karma.

Also, you could download the Redis DB backup (or use the script in the gist above) to get a copy of the data as it stands right now, so that it’s easy to reproduce if we experience further problems (grabbing the backup will reduce the chance that we will need it. Just sayin’ ;)

from lita-karma.

I'm afraid in more testing more data loss and strange behavior occurred, so I've decided to remove the decay and modification count features from master. I also removed all upgrade/data migration code, so you'll have to manually change the "modified:#{term}" Redis keys from sorted sets back to a regular set manually, or likely just delete them. I really appreciate that you spent your time working on this stuff, but I think the risk vs. reward balance is not really right. Perhaps we'll revisit some of these ideas in the future!

from lita-karma.

@jimmycuadra Did you remove the faulty actions? I have not seen any data loss nor odd behavior since we started using it.

from lita-karma.

Yeah, I did. There were a lot of strange things happening, including a bunch of previously negative values suddenly being positive.

from lita-karma.