raphaelvallat / pingouin Goto Github PK

Hi Raphael,

I'm wondering if there is any possibility to compute BF10 from your implementations of ANOVA alongside p-values?

Wish you a nice day,
Clément.

Hi,

Noticed some issues while trying rmcorr.

• If the column you pass is non-numeric, it just results in an assertion error with no error message. The r library rmcorr() returns message saying: 'Measure 1' and 'Measure 2' must be numeric. Maybe return some message like this?

• In ancova, bw = ss_slopes.sum() / ss.sum() (and ss_c = ss_slopes.sum() * bw) returns NaN if any of the values in ss_slopes is NaN. Basically, sum() on a numpy array == NaN if it has any NaNs in the array. As a result, rmcorr or ancova result turns out to be NaN. I think this should be changed to np.nansum(ss_slopes) instead. This will ignore NaNs but still sum the remaining values.

Thank you!

Currently pingouin.normality() silently discards the W statistic returned by scipy.stats.shapiro(). I would love to have an option to retrieve these stats, as I need to present them together with the p values when drafting a scientific publication. I wonder if the function could not be adjusted to return a DataFrame with numerous information like most of the other pingouin functions?

Hi Raphael,

I was thinking a great addition to the toolbox would be some python based tools for power analysis such as something similar to R's 'pwr'. https://cran.r-project.org/web/packages/pwr/pwr.pdf. Thanks again for your great work on this tool!

Best,
Alex

A great addition in Python 3 are type hints, which can make development less error-prone if an IDE is used that supports this feature.

Here's an example from one of my projects that's using type hints (and keyword-only arguments, for that matter):

https://github.com/hoechenberger/questplus/blob/9d5a6b9fc7bafd000dd82c1737be274ea31ee3de/questplus/qp.py#L10-L20

And another one:
https://github.com/hoechenberger/questplus/blob/9d5a6b9fc7bafd000dd82c1737be274ea31ee3de/questplus/psychometric_function.py#L7-L13

I think type hints could be a welcome addition to pingouin as well.

Good morning.

I'm trying to do a two-way ANOVA. My dataset is looking like this:

gender success d t v 
M      TRUE    1 1 2
M      FALSE   1 2 7
F      TRUE    1 2 1
M      TRUE    3 5 4
F      FALSE   2 1 6

Gender and Success are my between variables.

So I'm doing:
pg.anova(dv=v, between=["gender", "success"], data=output, detailed=False)

And I got this error:

Traceback (most recent call last):

  File "<ipython-input-93-35b88e82087c>", line 8, in <module>
    print(pg.anova(dv=v, between=["gender", "success"], data=output, detailed=False), "\n")

  File "C:\Users\poire\Anaconda3\lib\site-packages\pingouin\parametric.py", line 801, in anova
    export_filename=export_filename)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pingouin\parametric.py", line 939, in anova2
    ss_resid = np.sum(data.groupby([fac1, fac2]).apply(lambda x:

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\groupby\groupby.py", line 701, in apply
    return self._python_apply_general(f)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\groupby\groupby.py", line 707, in _python_apply_general
    self.axis)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\groupby\ops.py", line 190, in apply
    res = f(group)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pingouin\parametric.py", line 940, in <lambda>
    (x - x.mean())**2))[0]

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 2030, in f
    level=level)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1930, in _combine_series_frame
    return self._combine_match_columns(other, func, level=level)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\frame.py", line 5116, in _combine_match_columns
    return ops.dispatch_to_series(left, right, func, axis="columns")

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1157, in dispatch_to_series
    new_data = expressions.evaluate(column_op, str_rep, left, right)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\computation\expressions.py", line 208, in evaluate
    return _evaluate(op, op_str, a, b, **eval_kwargs)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\computation\expressions.py", line 123, in _evaluate_numexpr
    result = _evaluate_standard(op, op_str, a, b)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\computation\expressions.py", line 68, in _evaluate_standard
    return op(a, b)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1144, in column_op
    for i in range(len(a.columns))}

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1144, in <dictcomp>
    for i in range(len(a.columns))}

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1583, in wrapper
    result = safe_na_op(lvalues, rvalues)

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1533, in safe_na_op
    lambda x: op(x, rvalues))

  File "pandas/_libs/algos.pyx", line 690, in pandas._libs.algos.arrmap

  File "C:\Users\poire\Anaconda3\lib\site-packages\pandas\core\ops.py", line 1533, in <lambda>
    lambda x: op(x, rvalues))

TypeError: ufunc 'subtract' did not contain a loop with signature matching types dtype('<U32') dtype('<U32') dtype('<U32')

If I remove either gender or success, the code is working well.
Is there any fix?

Thanks,
Clément.

Hi,
I like the simplicity of the library, especially running a Friedman test on a panda dataframe with one line is nice.

However due to the lack of posthoc test I have to use other library like Scikit-posthoc.
It would be nice to be able to use pingouin for that as well !

Best,

Antoine

Make it 2x2. If b == 0 and c == 0, raise ValueError.

Currently, if the passed iterable contains one or more NaN values, qqplot() will return a basically empty plot. There should be the option to automatically remove NaN values before plotting (might even be the default).

I try to run the example

import numpy as np
import pingouin as pg

np.random.seed(123)
mean, cov, n = [4, 5], [(1, .6), (.6, 1)], 30
x, y = np.random.multivariate_normal(mean, cov, n).T

# T-test
pg.ttest(x, y)

The erroe is module 'numpy' has no attribute 'format_float_positional'

Python 3.6.7
statsmodels-0.10.1
pingouin-0.2.8
numpy-1.16.4

Should we incorporate them on the chi2_exact? Or create a new function? Or just leave those metrics out?

Source to study: http://www.real-statistics.com/chi-square-and-f-distributions/effect-size-chi-square/

• New function: pingouin.power_chi2
• Add Cramer's V and statistical power to pingouin.chi2

https://en.wikipedia.org/wiki/Cochran%27s_Q_test

correlation.shepherd() removes outliers before calculating Spearman's rho. Ideally this function would return a list of removed outliers, just like correlation.skipped() does.

Edit I haven't looked into the literature, but I would assume that the degrees of freedom of Shepherd's pi are N – N_outliers – 1, right? It could be helpful if the correlation functions were also to return the dof, then :)

1. Add more effect size options to pingouin.anova and pingouin.rm_anova
2. Add a measure of effect size for pingouin.welch_anova (but which one -- is partial eta-squared suitable?)

See new version of normality()

Validate contingency tests against implementations from R, SPSS and JASP

Both Wikipedia and Real Statistics mention that the Welch's test is useful when we have samples of different sizes, but pingouin's documentation does not. Wouldn't it be interesting to contemplate that piece of information?

There's a line (245) in the wilcoxon function of nonparametric.py:

pval *= .5 if tail == 'one-sided' else pval

I think this is intended to work as

pval = pval*.5 if tail == 'one-sided' else pval

but is actually doing

pval *= (.5 if tail == 'one-sided' else pval)

Easy fix if I'm right! :)

Also, far less important but while I'm here:
pingouin docs for wilcoxon says "A continuity correction is applied by default..." but this is not the case, right?? Scipy wilcoxon's default is False, and pingouin also explicitly passes that.

See Gitter chat

See gitter chat.

pg.chi2_cmh also requires a stratify parameter.

Sources to study:

• https://en.wikipedia.org/wiki/Cochran%E2%80%93Mantel%E2%80%93Haenszel_statistics
• http://www.biostathandbook.com/cmh.html
• http://rcompanion.org/rcompanion/b_10.html
• https://www.statisticshowto.datasciencecentral.com/cochran-mantel-haenszel-statistic-test/

According to Jammalamadaka & Sengupta (2001, pg. 177) the circular means are not well defined if the marginal distribution of one of the angles is uniform. This leads to wrong estimates of circular correlations in these cases. There is an alternative formulation of the circular correlation coefficient that deals with this problem (see equation 8.2.4 in Jammalamadaka & Sengupta, 2001). This implementation can also lead to slightly different values for the coefficient in other cases, though. This is the implementation:

def circcorr(x, y, tail='two-sided'):
    from scipy.stats import norm
    
    x = np.asarray(x)
    y = np.asarray(y)

    # Check size
    if x.size != y.size:
        raise ValueError('x and y must have the same length.')

    n = x.size
    
    x_sin = np.sin(x - stats.circmean(x))
    y_sin = np.sin(y - stats.circmean(y))

    
    num = np.abs(np.sum(np.exp((x - y)*1j))) - np.abs(np.sum(np.exp((x + y)*1j)))
    denom = 2 * np.sqrt(np.sum(x_sin**2) * np.sum(y_sin**2))
    
    r = num / denom
    
    # Compute T- and p-values
    tval = np.sqrt((n * (x_sin**2).mean() * (y_sin**2).mean())
                   / np.mean(x_sin**2 * y_sin**2)) * r

    # Approximately distributed as a standard normal
    pval = 2 * norm.sf(abs(tval))
    pval = pval / 2 if tail == 'one-sided' else pval
    
    return r, pval

I'm not sure if this should be added to pingouin, whether as an extra function or an alternative to circ_corrcc with a kwarg.

References:

• Jammalamadaka, S. R., & Sengupta, A. (2001). Topics in circular statistics (Vol. 5). world scientific.
   

importing pingouin returns the following:

ModuleNotFoundError Traceback (most recent call last)
in ()
1 import pandas as pd
2 import numpy as np
----> 3 import pingouin

~/.local/lib/python3.6/site-packages/pingouin/init.py in ()
1 # Import pingouin objects
----> 2 from .utils import *
3 from .bayesian import *
4 from .effsize import *
5 from .multicomp import *

~/.local/lib/python3.6/site-packages/pingouin/utils.py in ()
3 import numpy as np
4 from six import string_types
----> 5 from pingouin.external.tabulate import tabulate
6 import pandas as pd
7

ModuleNotFoundError: No module named 'pingouin.external'

I typically end up having MultiIndex columns in my DataFrames during analysis, which currently does not seem to be (well) supported by pingouin. Here's an example when trying to use pairwise_corr() on MultiIndex columns:

In[31]: import pandas as pd
   ...: 
   ...: 
   ...: results_a = list(range(5))
   ...: results_b = list(range(5))
   ...: 
   ...: columns = pd.MultiIndex.from_tuples([('Result', 'a'),
   ...:                                      ('Result', 'b')])
   ...: data = pd.DataFrame(dict(a=results_a, b=results_b))
   ...: data.columns = columns
   ...: 
   ...: print(data)
   ...: 
  Result   
       a  b
0      0  0
1      1  1
2      2  2
3      3  3
4      4  4

In[32]: import pingouin
   ...: pingouin.pairwise_corr(data, columns=[('Result', 'a'),
   ...:                                       ('Result', 'b')])
   ...: 
Traceback (most recent call last):
  File "/Users/hoechenberger/miniconda3/envs/QUOTE/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-32-f548ba39714e>", line 3, in <module>
    ('Result', 'b')])
  File "/Users/hoechenberger/miniconda3/envs/QUOTE/lib/python3.7/site-packages/pingouin/pairwise.py", line 818, in pairwise_corr
    assert comb[0] in keys
AssertionError

In this specific case, this behavior is caused because np.intersect1d(keys, columns) returns array(['Result', 'a', 'b'], dtype='<U6'), which then yields the combinations [('Result', 'a'), ('Result', 'b'), ('a', 'b')]; of course, these don't make much sense.

Would be nice if pingouin could handle MultiIndex columns directly.

Thanks for creating this amazing project! Only found out about it by accident :)

At any rate, I just created a conda recipe and proposed it for inclusion in the conda-forge channel, so users would be able to install pingouin and all dependencies simply by doing conda install -c conda-forge pingouin

I wanted to ask you if you had any interest in

• getting included as a recipe maintainer in the conda-forge recipe, and
• adding a conda badge as well as
• amending the installation instructions

once the PR to conda-forge is accepted and the packages become available? (Meaning, I could do a PR for all of that.)

Using statsmodels:

• https://www.statsmodels.org/stable/anova.html
• https://www.statsmodels.org/stable/generated/statsmodels.stats.anova.anova_lm.html

Need to wait for the next statsmodels release to fix that:
See statsmodels/statsmodels#5747 and statsmodels/statsmodels#5759

Functions that fail are:

• ancovan
• anova2 with unbalanced design

Hello,
one small problem: rm_anova collapses multiple observation for each subject and condition to mean (which is correct behavior) but if you have 2 within subject factors this function runs rm_anova2 which does not. Due to this fact when you run rm_anova with two within-subject variables it returns wrong value. If you do agree I can submit a pull request - as far as i can see the fix is to collapse values to a mean and THEN run rm_anova2.

Hey,

Thanks for the great package.

The documentation of Pingouin states that it is strongly recommended that python >= 3.6 is used, which seems to imply that the package is usable with other versions. This is not the case.
Importing Pingouin on python < 3.6 fails, because this line uses f-strings, which were only introduced in python 3.6.

AFAIK, the file linked above is the only place where f-strings are used.
I'm not sure whether you want to create a work-around for this, but it threw me for a loop when I saw the error. I'll leave it up to you to decide whether removing these f-strings is a good choice, they are the way forward after all.

On a related note, installing pingouin using python 2.7 does not work, because the matching matplotlib version is not available. So it seems to me that python 2.7 support can be dropped from the readme as well.

error: Setup script exited with
Matplotlib 3.0+ does not support Python 2.x, 3.0, 3.1, 3.2, 3.3, or 3.4.
Beginning with Matplotlib 3.0, Python 3.5 and above is required.

Currently, pingouin imports pandas and monkey-patches the DataFrame to add new methods. The "proper" way to achieve this, however, is to use the pandas extension API, most probably via pandas_flavor (pyjanitor, for example, does this too)

One simply needs to import pandas_flavor and add a decorator:

# pingouin/pairwise.py

import pandas as pd
import pandas_flavor as pf

@pf.register_dataframe_method
def pairwise_ttests(dv=None, between=None, within=None, subject=None,
                    data=None, parametric=True, alpha=.05, tail='two-sided',
                    padjust='none', effsize='hedges', return_desc=False,
                    export_filename=None):
...

Usage from the user's side:

import pandas as pd
import pingouin as pg

data = pd.DataFrame(...)
data.pairwise_ttests(...)

Hello,

Might be a bit specific, but I was wondering if that was an idea. I'm currently using it as a cheap way to posthoc test everything and am not super interested in interaction effects as for some cases it would make no sense. However, it seems that doing these tests is

1a. memory hungry, because it probably checks for all interactions
1b. could maybe reuse / make better use of temporary variables, it currently blows up my 16 go of ram the instant it goes into the interaction effects for a dataset with around 300 000 lines (and 6 columns, for each group I'm checking).

At the moment the normality function does not account properly for NaNs in the data.

normality([np.nan, 1, 2])
(False, nan)

normality([np.nan, 1, 2, 10])
(True, 1.0)

Though a p value of 1 is suspicious anyway, it would make sense to provide a warning message or an argument to remove the NaNs.

normality([np.nan, 1, 2, 10], remove_na=True)
(True, 0.194)

See https://rdrr.io/cran/rmcorr/man/plot.rmc.html

This is a test that I use a lot to understand the relationship between categorical data.

SciPy has it's own implementation, but we have to feed it with a contingency table. The contingency table is easy to build (using pandas.crosstab, for instance), but I think it would be an improvement for the user experience to simply call pg.pearson_chi2(x, y) or something like that.

SciPy's implementation returns the χ², the p-value, the degrees of freedom and a contingency table with the expected values (for which there would be no correlation between the columns). But I think it would be possible to infer more information from the test, just like you did with pg.ttest?

Best regards,
Arthur Paulino

Tests currently planned to be supported:

• Fisher's exact test
• Barnard's test

Thanks for the awesome package! I noticed a small concern with the logistic regression implementation. Since pingouin relies on sklearn's LogisticRegression implementation, it's worth noting that it implements L2 regularization by default with a hyperparameter value C=1.

This results in coefficients being penalized by default, and I suspect (but not 100% sure) the standard-error and p-values not being totally correct/interpretable. I say this because estimating standard errors for regularized/penalized models is not trivial and the computation often involves incorporating the penalty term somehow, but even then may not all that informative. See for example this stackoverflow discussion and links.

Here's an example jupyter notebook demonstrating the results different from the same analysis using glm in R (using the rpy2 package).

One possibility is setting the C=1e9 by default but letting it be overridden if desired. This effectively removes the effect of regularization as sklearn doesn't have a flag to disable it completely but this seems to be the accepted community solution.

Opened a pull request if you go with that solution!

The docs mention that one should ideally pass parameters as keyword arguments, not by position. I'm wondering why this isn't simply enforced? Python 3 supports keyword-only arguments. It's as simple as changing a function signature from

def f(foo=1, bar=2)

to

def f(*, foo=1, bar=2)

Now, positional arguments won't be accepted anymore.

I sometimes plot shift functions as / via empirical cumulative distribution functions, to avoid the troubles that come with kernel-density estimates. Would be great to see this feature in Pingouin as well. I typically use statsmodels.distributions.empirical_distribution.ECDF to estimate the CDFs.

Example (based on my own implementation) using a KDE:

Same example using CDFs:

(Yes, could use some visual tuning, I know :))

Good morning,

I had to work on non-normal and heteroscedastic variables but had to compute interactions between those variables. Unfortunately, no methods in your package allow to do so.
I found ART beeing quite straight forward and easy to use. I think it could be very valuable to include ART inside your library.
The actual solution is to run ARTool through rpy2 which is neither performant nor easy to do.

Here is the original paper: http://faculty.washington.edu/wobbrock/pubs/chi-11.06.pdf

And here are some R documentation:

• https://cran.r-project.org/web/packages/ARTool/index.html
• https://cran.r-project.org/web/packages/ARTool/vignettes/art-contrasts.html

Thanks, and have a nice day,
Clément.

See https://garstats.wordpress.com/2016/07/12/shift-function/ and https://garstats.wordpress.com/2016/07/29/matlab-shift-function/

@LegrandNico

See https://github.com/cran/BayesFactor/blob/0a1fe0bedf62549a466c9ec5db8b8b5a0217f0c6/R/correlationBF.R

This affects several Pingouin functions (T-test, effect sizes...).

• T-test: The confidence intervals of the difference in means should be Inf on one of the two sides, as in the R t.test function.

Or should we automatically determine the 'greater' and 'less' using the sign of the T-value?

https://en.wikipedia.org/wiki/McNemar%27s_test

Create pingouin.power_rm_anova function