permutations-stats's Introduction

permutations-stats

Python-only permutation-based statistical tests, accelerated with numba.

Status

Statistical tests

Brunner Munzel [1], Mann Whitney Wilcoxon [2, 3], Wilcoxon signed rank test [3], and Friedman [4] tests are implemented.

Exact tests (all combinations) and approximate method (simulation) are available.

Functions for bootstrapping-based confidence intervals calculations for mean, median and standard deviation are also implemented (not documented yet).

Why this package ?

This work aims to provide fast permutation-based statistical tests in Python. Some tests are not available publicly in an exact mode (computing all possible permutations) or with simulations. If certain assumptions cannot be made about the data (such as normality) or if the sample is not large enough, the existing implementations should not be used.
For example, the Brunner Munzel [1] test is implemented in scipy but not with an exact calculation. The statistic can be used with the public API but it can take some time if ran several thousands of times (e.g. the p-value is also calculated for each iteration).

This packages reimplements the looping of permutations and statistical tests with numba. A few seconds are required to compile on the fly for the first function call. Then, acceleration is critical as shown in this output from tests comparing the Brunner Munzel statistic calculation with scipy:

tests/stat_tests.py
200 tests with 5 and 4 data points - Permutations-stats: 2.375s, Scipy: 0.076s, diff: -2.299s
.
200 tests with 3 and 3 data points - Permutations-stats: 0.002s, Scipy: 0.077s, diff: 0.075s
.
200 tests with 10 and 12 data points - Permutations-stats: 0.004s, Scipy: 0.079s, diff: 0.075s
.
10000 tests with 18 and 10 data points - Permutations-stats: 0.220s, Scipy: 3.806s, diff: 3.586s
.
20000 tests with 18 and 15 data points - Permutations-stats: 0.477s, Scipy: 7.645s, diff: 7.168s
.
30000 tests with 18 and 19 data points - Permutations-stats: 0.769s, Scipy: 11.468s, diff: 10.699s

Dependencies

numpy
numba

And for development testing only

scipy ≥1.5
pytest

Usage

Basic usage:

import numpy as np
from permutations_stats.permutations import permutation_test

# Sample data
x = np.arange(9)
y = (np.arange(9) -0.2) * 1.1

permutation_test(x, y, test="brunner_munzel")
# PermutationsResults(statistic=0.2776044311308564, pvalue=0.7475935828877005, permutations=24310, test='brunner_munzel', alternative='TWO_SIDED', method='exact')

More examples on usage.ipynb and a detailed demonstration on doc/demo.ipynb.

Perspective

If sample sizes and/or the number of iterations are small, acceleration is not expected with numba, and using numpy alone should be the fastest option.
Thresholds for numba use will be better determined to decide the function to call (without user intervention).

License

GNU General Public License v3.0 only.

Cite

If you find this software useful for your academic work, please cite as below:

Florian Charlier. (2022). permutations-stats (v0.2). Zenodo. 
https://doi.org/10.5281/zenodo.7213305

Acknowledgements

We would like to thank Marianne Paesmans, Lieveke Ameye, and Luigi Moretti at Institut Jules Bordet for their support during the development of this package.

References

[1] Brunner, E. and Munzel, U. (2000), The Nonparametric Behrens‐Fisher Problem: Asymptotic Theory and a Small‐Sample Approximation. Biom. J., 42: 17-25. doi:10.1002/(SICI)1521-4036(200001)42:1<17::AID-BIMJ17>3.0.CO;2-U

[2] Mann, H. B. and D. R. Whitney (1947). "On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other." Ann. Math. Statist. 18(1): 50-60.

[3] Wilcoxon, F. (1945). "Individual Comparisons by Ranking Methods." Biometrics Bulletin 1(6): 80-83.

[4] Friedman, M. (1937). "The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance." Journal of the American Statistical Association 32(200): 675-701.

permutations-stats's People

Contributors

Stargazers

Watchers

permutations-stats's Issues

Document specific test usage

Document bootstrap module usage

some basics on doing permutations

Hi, I have just started to use your permutation test package. I am not a programmer, so it took a bit of time to figure out how to download and install on my system, where it is not standardly present. I am using the Digital Research Alliance of Canada system, used to be called Compute Canada. I have a couple of questions:

First, I was able to install the wheel version, but for some reason not the tar.gz version, my system had Path troubles it seems. Anyway, I can import the program in Python and run the test. Question 1, where is the package actually? I do not see any obvious new directories or files in my home directory, I wouldn't think it could be installed more generally. Does the installation update some path variables in my home account so python knows where to look for permutations_stats.permutations ?

More importantly, is there some general documentation on setting some parameters in the system, like the number of permutations, whether to permute both X and Y (in my case I am doing a Brunner Munzel test with cases and controls, so just two data variables)? I was able to run the test with default parameters when X and Y both had around a dozen values, but when I tried to use my real data, which has over 100 values for one of the two, the system "killed" the program so presumably I exceeded memory allocation. I can request more memory, but need to have some rough idea of how much memory is required for a certain large number of data points. Is it possible to permute only one of the two data variables? Is it meaningful to do that?

Some of this info can be found on the web for other permutation test packages (like for R), but your implementation may be different so it's best to ask at the source.

You don't seem to have a literature citation posted here, is there one now that I can cite when I write this up? Otherwise do I just mention it as software used in my Methods section with a link to here?

FYI, this is for a human genetics project. We sequenced a bunch of cases and controls to look for mutations causing a rare endocrine phenotype, focused on very rare genomic variants. As a result we are doing a kind of gene burden test, and want to compare the burden of variants in the cases vs controls. For particular reasons, a Fisher or chi-square 2x2 test is not appropriate here, rather we are using Mann-Whitney or Brunner-Munzel, essentially to compare the average per sample mutation burden by ranking the samples. Because the number of cases is small (12), I wanted to run a permutation version just in case the regular tests (BM or MW) are too forgiving. BM is probably better generally as we don't know whether the variances are the same in the cases and controls, although they likely are.

Thanks and cheers from Montreal!
Mark Samuels
Associate Professor in Medicine
University of Montreal

Recommend Projects