matt52 / bayesian-testing Goto Github PK

Thank you for the excellent package! I was wondering if this package could provide credible intervals for each variant when reporting A/B testing results.

First, this package is great! I wanted to know if the probability estimates rely on a minimum sample size or how one might go about determining minimum sample size for a Binary test, for example.

Just a quick observation: if you want a flat Beta prior as the default, then based on numpy's parameterization of the Beta function I think Beta(1, 1) is the right choice.

Beta(1/2, 1/2) blows up at p=0 and p=1, and has a minimum at p=1/2. (See here.) If that's the behavior you want, please ignore this issue. But if you want a flat default then consider changing to Beta(1,1).

Cheers!

Hi，first i like your code! Thanks a lot!
I want to know the principles about gamma_posteriors/dirichlet_posteriors/normal_posteriors/lognormal_posteriors
I found something about gamma_posteriors in VWO_SmartStats_technical_whitepaper, but it looks different. (I cant understand the
1 / (b_priors_gamma[i] + totals[i]) in gamma_posteriors)
Is there any recommended material or website that i can learn about the different posteriors and the estimate_probabilities function?
Thanks a lot!

Test example: there 3 users, only one user purchased something (twice). Nobody else purchased anything.

What should be the value of 'totals'?

• the number of ALL participants of the test. totals = 3.
• the number of participants who generated at least something. totals = 1
• the number of purchases made. totals=2

From the first look Normal test will not work. Will Delta Normal test work with negative and zero values?

Hi,

I tested your library and cross-checked against this online calculator:
Here is the result from your library:

[{'variant': 'True True True False False False False',
  'totals': 1172,
  'positives': 461,
  'positive_rate': 0.39334,
  'prob_being_best': 0.7422,
  'expected_loss': 0.0582635},
 {'variant': 'False True True False False False False',
  'totals': 222,
  'positives': 27,
  'positive_rate': 0.12162,
  'prob_being_best': 0.0,
  'expected_loss': 0.3280173},
 {'variant': 'False False True False False False False',
  'totals': 1363,
  'positives': 63,
  'positive_rate': 0.04622,
  'prob_being_best': 0.0,
  'expected_loss': 0.4051768},
 {'variant': 'False False False False False False False',
  'totals': 1052,
  'positives': 0,
  'positive_rate': 0.0,
  'prob_being_best': 0.0,
  'expected_loss': 0.4512031},
 {'variant': 'True False True False False False False',
  'totals': 1,
  'positives': 0,
  'positive_rate': 0.0,
  'prob_being_best': 0.2578,
  'expected_loss': 0.1997566}]

So the best variant has 74% probability to be the winner. On the online calculator it is 63.48% instead (last variant is 36.52% instead of 25.78%).

I used the BinaryDataTest() without any priors.

I did not dig deeper on what might be right here, but wanted to drop this as feedback.