Performance impact about scientist HOT 5 CLOSED

Being able to run candidates in a non-blocking, out-of-band way would be a really really cool feature that would increase the usefulness of scientist A LOT.

I'm currently working on an experiment that makes a very basic change to how data is fetched in various places in our app. It really needs to be evaluated at the controller action level, but we cannot at least double those request response times. So I'm looking at ways we can support such experiments.

from scientist.

So! After having a look around and testing a most naive way of doing this, most of the raised-exception-expecting tests fail (besides some processes hanging and whatnot) – this made evident that in order to be able to use scientist also on tests without having to rewrite any of them, it should be possible to escalate exceptions.

Naturally, this requirement makes it rather complex to run candidates in isolation and in parallel, given that in any “test mode” it’d be needed to pass around (from candidates to the control process) the results or exceptions, and wait for everything to be finished before either returning the appropriate results, or escalating any exceptions.

So yeah, I can see the why now! 😄

from scientist.

@Ostera I took a crack at something like this a while ago, except not using fork (fork is probably not the best choice here). https://github.com/Headlinerfm/scientist/tree/concurrent-ruby

Tests pass, but this one

it "shuffles behaviors before running" do
    last = nil
    runs = []

    @ex.use { last = "control" }
    @ex.try { last = "candidate" }

    10000.times do
      @ex.run
      runs << last
    end

    assert runs.uniq.size > 1
  end

runs very slowly, ~2s. Without this test, the concurrent-ruby full test suite runs at the same rate as master. I don't have in-depth knowledge of concurrent-ruby but my guess is it's slow here because of mutex locking and thread generation/waiting on free threads -- apparently futures use a global (global to concurrent-ruby?) thread pool, but I'm not sure how large that is, likely much less than 10000. In a real-world scenario, that time is likely to be rendered negligible by whatever you're doing in the scientist experiment/control blocks, especially if that is IO.

My implementation here is half-baked, it was just a proof of concept. I may revisit it. Feel free to copy/hack on it and improve it if you're still interested.

from scientist.

Scientist itself avoids running things at the same time because there are so many ways to do it. 😁 It's limiting for sure, since it means Scientist is only really useful when it's OK for an operation to sometimes be (at least) twice as slow.

I'm closing this since it's been dormant for a while, but it'd be cool to figure out some way to plug in other ways to run all the candidates.

from scientist.

Understood 👌

from scientist.