scipy.stats._stats_py:combine_pvalues

Notes

If this function is applied to tests with a discrete statistics such as any rank test or contingency-table test, it will yield systematically wrong results, e.g. Fisher's method will systematically overestimate the p-value ^[1]. This problem becomes less severe for large sample sizes when the discrete distributions become approximately continuous.

The differences between the methods can be best illustrated by their statistics and what aspects of a combination of p-values they emphasise when considering significance ^[2]. For example, methods emphasising large p-values are more sensitive to strong false and true negatives; conversely methods focussing on small p-values are sensitive to positives.

• The statistics of Fisher's method (also known as Fisher's combined probability test) ^[3] is $-2{\sum }_i\log (p_i)$, which is equivalent (as a test statistics) to the product of individual p-values: ${\prod }_i{p}_i$. Under the null hypothesis, this statistics follows a ${\chi }^2$ distribution. This method emphasises small p-values.

• Pearson's method uses $-2{\sum }_i\log (1-p_i)$, which is equivalent to ${\prod }_i\frac{1}{1-p_i}$ ^[2]. It thus emphasises large p-values.

• Mudholkar and George compromise between Fisher's and Pearson's method by averaging their statistics ^[4]. Their method emphasises extreme p-values, both close to 1 and 0.

• Stouffer's method ^[5] uses Z-scores and the statistic: ${\sum }_i{\Phi }^{-1}(p_i)$, where $\Phi$ is the CDF of the standard normal distribution. The advantage of this method is that it is straightforward to introduce weights, which can make Stouffer's method more powerful than Fisher's method when the p-values are from studies of different size ^{[6] [7]}.

• Tippett's method uses the smallest p-value as a statistic. (Mind that this minimum is not the combined p-value.)

Fisher's method may be extended to combine p-values from dependent tests ^[8]. Extensions such as Brown's method and Kost's method are not currently implemented.

Beginning in SciPy 1.9, np.matrix inputs (not recommended for new code) are converted to np.ndarray before the calculation is performed. In this case, the output will be a scalar or np.ndarray of appropriate shape rather than a 2D np.matrix. Similarly, while masked elements of masked arrays are ignored, the output will be a scalar or np.ndarray rather than a masked array with mask=False.

Array API Standard Support

combine_pvalues has experimental support for Python Array API Standard compatible backends in addition to NumPy. Please consider testing these features by setting an environment variable SCIPY_ARRAY_API=1 and providing CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following combinations of backend and device (or other capability) are supported.

====================  ====================  ====================
Library               CPU                   GPU
====================  ====================  ====================
NumPy                 ✅                     n/a                 
CuPy                  n/a                   ✅                   
PyTorch               ✅                     ⛔                   
JAX                   ⚠️ no JIT             ⚠️ no JIT           
Dask                  ⚠️ computes graph     n/a                 
====================  ====================  ====================

See dev-arrayapi for more information.