{ "__type": "IngestedDoc", "__tag": 4010, "_content": { "Notes": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The three types of permutation tests supported by this function are described below." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Unpaired statistics" } ] }, { "__type": "Text", "__tag": 4046, "value": " (" }, { "__type": "InlineCode", "__tag": 4051, "value": "permutation_type='independent'" }, { "__type": "Text", "__tag": 4046, "value": "):" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The null hypothesis associated with this permutation type is that all observations are sampled from the same underlying distribution and that they have been assigned to one of the samples at random." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Suppose " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " contains two samples; e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "a, b = data" }, { "__type": "Text", "__tag": 4046, "value": ". When " }, { "__type": "InlineCode", "__tag": 4051, "value": "1 < n_resamples < binom(n, k)" }, { "__type": "Text", "__tag": 4046, "value": ", where" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "k" }, { "__type": "Text", "__tag": 4046, "value": " is the number of observations in " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": "," } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " is the total number of observations in " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": ", and" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "binom(n, k)" }, { "__type": "Text", "__tag": 4046, "value": " is the binomial coefficient (" }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " choose " }, { "__type": "InlineCode", "__tag": 4051, "value": "k" }, { "__type": "Text", "__tag": 4046, "value": ")," } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "the data are pooled (concatenated), randomly assigned to either the first or second sample, and the statistic is calculated. This process is performed repeatedly, " }, { "__type": "InlineRole", "__tag": 4003, "value": "permutation", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When " }, { "__type": "InlineCode", "__tag": 4051, "value": "n_resamples >= binom(n, k)" }, { "__type": "Text", "__tag": 4046, "value": ", an exact test is performed: the data are " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "partitioned" } ] }, { "__type": "Text", "__tag": 4046, "value": " between the samples in each distinct way exactly once, and the exact null distribution is formed. Note that for a given partitioning of the data between the samples, only one ordering/permutation of the data " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "within" } ] }, { "__type": "Text", "__tag": 4046, "value": " each sample is considered. For statistics that do not depend on the order of the data within samples, this dramatically reduces computational cost without affecting the shape of the null distribution (because the frequency/count of each value is affected by the same factor)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For " }, { "__type": "InlineCode", "__tag": 4051, "value": "a = [a1, a2, a3, a4]" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "b = [b1, b2, b3]" }, { "__type": "Text", "__tag": 4046, "value": ", an example of this permutation type is " }, { "__type": "InlineCode", "__tag": 4051, "value": "x = [b3, a1, a2, b2]" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "y = [a4, b1, a3]" }, { "__type": "Text", "__tag": 4046, "value": ". Because only one ordering/permutation of the data " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "within" } ] }, { "__type": "Text", "__tag": 4046, "value": " each sample is considered in an exact test, a resampling like " }, { "__type": "InlineCode", "__tag": 4051, "value": "x = [b3, a1, b2, a2]" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "y = [a4, a3, b1]" }, { "__type": "Text", "__tag": 4046, "value": " would " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "not" } ] }, { "__type": "Text", "__tag": 4046, "value": " be considered distinct from the example above." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "permutation_type='independent'" }, { "__type": "Text", "__tag": 4046, "value": " does not support one-sample statistics, but it can be applied to statistics with more than two samples. In this case, if " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " is an array of the number of observations within each sample, the number of distinct partitions is " } ] }, { "__type": "Code", "__tag": 4050, "value": "np.prod([binom(sum(n[i:]), sum(n[i+1:])) for i in range(len(n)-1)])", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Paired statistics, permute pairings" } ] }, { "__type": "Text", "__tag": 4046, "value": " (" }, { "__type": "InlineCode", "__tag": 4051, "value": "permutation_type='pairings'" }, { "__type": "Text", "__tag": 4046, "value": "):" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The null hypothesis associated with this permutation type is that observations within each sample are drawn from the same underlying distribution and that pairings with elements of other samples are assigned at random." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Suppose " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " contains only one sample; e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "a, = data" }, { "__type": "Text", "__tag": 4046, "value": ", and we wish to consider all possible pairings of elements of " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": " with elements of a second sample, " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": ". Let " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " be the number of observations in " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": ", which must also equal the number of observations in " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When " }, { "__type": "InlineCode", "__tag": 4051, "value": "1 < n_resamples < factorial(n)" }, { "__type": "Text", "__tag": 4046, "value": ", the elements of " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": " are randomly permuted. The user-supplied statistic accepts one data argument, say " }, { "__type": "InlineCode", "__tag": 4051, "value": "a_perm" }, { "__type": "Text", "__tag": 4046, "value": ", and calculates the statistic considering " }, { "__type": "InlineCode", "__tag": 4051, "value": "a_perm" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": ". This process is performed repeatedly, " }, { "__type": "InlineRole", "__tag": 4003, "value": "permutation", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When " }, { "__type": "InlineCode", "__tag": 4051, "value": "n_resamples >= factorial(n)" }, { "__type": "Text", "__tag": 4046, "value": ", an exact test is performed: " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": " is permuted in each distinct way exactly once. Therefore, the " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": " is computed for each unique pairing of samples between " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": " exactly once." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For " }, { "__type": "InlineCode", "__tag": 4051, "value": "a = [a1, a2, a3]" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "b = [b1, b2, b3]" }, { "__type": "Text", "__tag": 4046, "value": ", an example of this permutation type is " }, { "__type": "InlineCode", "__tag": 4051, "value": "a_perm = [a3, a1, a2]" }, { "__type": "Text", "__tag": 4046, "value": " while " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": " is left in its original order." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "permutation_type='pairings'" }, { "__type": "Text", "__tag": 4046, "value": " supports " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " containing any number of samples, each of which must contain the same number of observations. All samples provided in " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " are permuted " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "independently" } ] }, { "__type": "Text", "__tag": 4046, "value": ". Therefore, if " }, { "__type": "InlineCode", "__tag": 4051, "value": "m" }, { "__type": "Text", "__tag": 4046, "value": " is the number of samples and " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " is the number of observations within each sample, then the number of permutations in an exact test is " } ] }, { "__type": "Code", "__tag": 4050, "value": "factorial(n)**m", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that if a two-sample statistic, for example, does not inherently depend on the order in which observations are provided - only on the " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "pairings" } ] }, { "__type": "Text", "__tag": 4046, "value": " of observations - then only one of the two samples should be provided in " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": ". This dramatically reduces computational cost without affecting the shape of the null distribution (because the frequency/count of each value is affected by the same factor)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Paired statistics, permute samples" } ] }, { "__type": "Text", "__tag": 4046, "value": " (" }, { "__type": "InlineCode", "__tag": 4051, "value": "permutation_type='samples'" }, { "__type": "Text", "__tag": 4046, "value": "):" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The null hypothesis associated with this permutation type is that observations within each pair are drawn from the same underlying distribution and that the sample to which they are assigned is random." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Suppose " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " contains two samples; e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "a, b = data" }, { "__type": "Text", "__tag": 4046, "value": ". Let " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " be the number of observations in " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": ", which must also equal the number of observations in " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When " }, { "__type": "InlineCode", "__tag": 4051, "value": "1 < n_resamples < 2**n" }, { "__type": "Text", "__tag": 4046, "value": ", the elements of " }, { "__type": "InlineCode", "__tag": 4051, "value": "a" }, { "__type": "Text", "__tag": 4046, "value": " are " }, { "__type": "InlineCode", "__tag": 4051, "value": "b" }, { "__type": "Text", "__tag": 4046, "value": " are randomly swapped between samples (maintaining their pairings) and the statistic is calculated. This process is performed repeatedly, " }, { "__type": "InlineRole", "__tag": 4003, "value": "permutation", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " times, generating a distribution of the statistic under the null hypothesis. The statistic of the original data is compared to this distribution to determine the p-value." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When " }, { "__type": "InlineCode", "__tag": 4051, "value": "n_resamples >= 2**n" }, { "__type": "Text", "__tag": 4046, "value": ", an exact test is performed: the observations are assigned to the two samples in each distinct way (while maintaining pairings) exactly once." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For " }, { "__type": "InlineCode", "__tag": 4051, "value": "a = [a1, a2, a3]" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "b = [b1, b2, b3]" }, { "__type": "Text", "__tag": 4046, "value": ", an example of this permutation type is " }, { "__type": "InlineCode", "__tag": 4051, "value": "x = [b1, a2, b3]" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "y = [a1, b2, a3]" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "permutation_type='samples'" }, { "__type": "Text", "__tag": 4046, "value": " supports " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " containing any number of samples, each of which must contain the same number of observations. If " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " contains more than one sample, paired observations within " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " are exchanged between samples " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "independently" } ] }, { "__type": "Text", "__tag": 4046, "value": ". Therefore, if " }, { "__type": "InlineCode", "__tag": 4051, "value": "m" }, { "__type": "Text", "__tag": 4046, "value": " is the number of samples and " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " is the number of observations within each sample, then the number of permutations in an exact test is " } ] }, { "__type": "Code", "__tag": 4050, "value": "factorial(m)**n", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Several paired-sample statistical tests, such as the Wilcoxon signed rank test and paired-sample t-test, can be performed considering only the " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "difference" } ] }, { "__type": "Text", "__tag": 4046, "value": " between two paired elements. Accordingly, if " }, { "__type": "InlineCode", "__tag": 4051, "value": "data" }, { "__type": "Text", "__tag": 4046, "value": " contains only one sample, then the null distribution is formed by independently changing the " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "sign" } ] }, { "__type": "Text", "__tag": 4046, "value": " of each observation." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "warning", "base_type": "warning", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "warning " } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The p-value is calculated by counting the elements of the null distribution that are as extreme or more extreme than the observed value of the statistic. Due to the use of finite precision arithmetic, some statistic functions return numerically distinct values when the theoretical values would be exactly equal. In some cases, this could lead to a large error in the calculated p-value. " }, { "__type": "CrossRef", "__tag": 4002, "value": "permutation_test", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.stats._resampling:permutation_test" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " guards against this by considering elements in the null distribution that are \"close\" (within a relative tolerance of 100 times the floating point epsilon of inexact dtypes) to the observed value of the test statistic as equal to the observed value of the test statistic. However, the user is advised to inspect the null distribution to assess whether this method of comparison is appropriate, and if not, calculate the p-value manually. See example below." } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Array API Standard Support" } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "CrossRef", "__tag": 4002, "value": "permutation_test", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.stats._resampling:permutation_test" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " has experimental support for Python Array API Standard compatible backends in addition to NumPy. Please consider testing these features by setting an environment variable " }, { "__type": "InlineCode", "__tag": 4051, "value": "SCIPY_ARRAY_API=1" }, { "__type": "Text", "__tag": 4046, "value": " and providing CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following combinations of backend and device (or other capability) are supported." } ] }, { "__type": "Code", "__tag": 4050, "value": "==================== ==================== ====================\nLibrary CPU GPU\n==================== ==================== ====================\nNumPy ✅ n/a \nCuPy n/a ✅ \nPyTorch ✅ ✅ \nJAX ✅ ✅ \nDask ⛔ n/a \n==================== ==================== ====================", "execution_status": null }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "See " }, { "__type": "InlineRole", "__tag": 4003, "value": "dev-arrayapi", "domain": null, "role": "ref", "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " for more information." } ] } ] } ], "title": [], "level": 0, "target": null }, "Warns": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Raises": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Yields": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Methods": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Returns": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Parameters", "__tag": 4026, "children": [ { "__type": "DocParam", "__tag": 4016, "name": "res", "annotation": "PermutationTestResult", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "An object with attributes:" } ] }, { "__type": "DefList", "__tag": 4033, "children": [ { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "statistic" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "statistic" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "pvalue" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "pvalue" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "null_distribution" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "null_distribution" } ] } ] } ] } ] } ] } ], "title": [], "level": 0, "target": null }, "Summary": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Performs a permutation test of a given statistic on provided data." } ] } ], "title": [], "level": 0, "target": null }, "Receives": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Warnings": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Attributes": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Parameters": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Parameters", "__tag": 4026, "children": [ { "__type": "DocParam", "__tag": 4016, "name": "data", "annotation": "iterable of array-like", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Contains the samples, each of which is an array of observations. Dimensions of sample arrays must be compatible for broadcasting except along " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "statistic", "annotation": "callable", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Statistic for which the p-value of the hypothesis test is to be calculated. " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": " must be a callable that accepts samples as separate arguments (e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "statistic(*data)" }, { "__type": "Text", "__tag": 4046, "value": ") and returns the resulting statistic. If " }, { "__type": "ParamRef", "__tag": 4071, "name": "vectorized" }, { "__type": "Text", "__tag": 4046, "value": " is set " }, { "__type": "InlineCode", "__tag": 4051, "value": "True" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": " must also accept a keyword argument " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " and be vectorized to compute the statistic along the provided " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " of the sample arrays." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "permutation_type", "annotation": "{'independent', 'samples', 'pairings'}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The type of permutations to be performed, in accordance with the null hypothesis. The first two permutation types are for paired sample statistics, in which all samples contain the same number of observations and observations with corresponding indices along " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " are considered to be paired; the third is for independent sample statistics." } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "'samples'" }, { "__type": "Text", "__tag": 4046, "value": "observations are assigned to different samples but remain paired with the same observations from other samples. This permutation type is appropriate for paired sample hypothesis tests such as the Wilcoxon signed-rank test and the paired t-test." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "'pairings'" }, { "__type": "Text", "__tag": 4046, "value": "observations are paired with different observations, but they remain within the same sample. This permutation type is appropriate for association/correlation tests with statistics such as Spearman's " }, { "__type": "InlineMath", "__tag": 4057, "value": "\\rho" }, { "__type": "Text", "__tag": 4046, "value": ", Kendall's " }, { "__type": "InlineMath", "__tag": 4057, "value": "\\tau" }, { "__type": "Text", "__tag": 4046, "value": ", and Pearson's " }, { "__type": "InlineMath", "__tag": 4057, "value": "r" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "'independent'" }, { "__type": "Text", "__tag": 4046, "value": " (default)observations are assigned to different samples. Samples may contain different numbers of observations. This permutation type is appropriate for independent sample hypothesis tests such as the Mann-Whitney " }, { "__type": "InlineMath", "__tag": 4057, "value": "U" }, { "__type": "Text", "__tag": 4046, "value": " test and the independent sample t-test." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Please see the Notes section below for more detailed descriptions of the permutation types." } ] } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "vectorized", "annotation": "bool, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If " }, { "__type": "ParamRef", "__tag": 4071, "name": "vectorized" }, { "__type": "Text", "__tag": 4046, "value": " is set " }, { "__type": "InlineCode", "__tag": 4051, "value": "False" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": " will not be passed keyword argument " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " and is expected to calculate the statistic only for 1D samples. If " }, { "__type": "InlineCode", "__tag": 4051, "value": "True" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": " will be passed keyword argument " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " and is expected to calculate the statistic along " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " when passed an ND sample array. If " }, { "__type": "InlineCode", "__tag": 4051, "value": "None" }, { "__type": "Text", "__tag": 4046, "value": " (default), " }, { "__type": "ParamRef", "__tag": 4071, "name": "vectorized" }, { "__type": "Text", "__tag": 4046, "value": " will be set " }, { "__type": "InlineCode", "__tag": 4051, "value": "True" }, { "__type": "Text", "__tag": 4046, "value": " if " }, { "__type": "InlineCode", "__tag": 4051, "value": "axis" }, { "__type": "Text", "__tag": 4046, "value": " is a parameter of " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": ". Use of a vectorized statistic typically reduces computation time." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "n_resamples", "annotation": "int or np.inf, default: 9999", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Number of random permutations (resamples) used to approximate the null distribution. If greater than or equal to the number of distinct permutations, the exact null distribution will be computed. Note that the number of distinct permutations grows very rapidly with the sizes of samples, so exact tests are feasible only for very small data sets." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "batch", "annotation": "int, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The number of permutations to process in each call to " }, { "__type": "ParamRef", "__tag": 4071, "name": "statistic" }, { "__type": "Text", "__tag": 4046, "value": ". Memory usage is O( " }, { "__type": "ParamRef", "__tag": 4071, "name": "batch" }, { "__type": "Text", "__tag": 4046, "value": " * " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " ), where " }, { "__type": "InlineCode", "__tag": 4051, "value": "n" }, { "__type": "Text", "__tag": 4046, "value": " is the total size of all samples, regardless of the value of " }, { "__type": "ParamRef", "__tag": 4071, "name": "vectorized" }, { "__type": "Text", "__tag": 4046, "value": ". Default is " }, { "__type": "InlineCode", "__tag": 4051, "value": "None" }, { "__type": "Text", "__tag": 4046, "value": ", in which case " }, { "__type": "InlineCode", "__tag": 4051, "value": "batch" }, { "__type": "Text", "__tag": 4046, "value": " is the number of permutations." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "alternative", "annotation": "{'two-sided', 'less', 'greater'}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The alternative hypothesis for which the p-value is calculated. For each alternative, the p-value is defined for exact tests as follows." } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "'greater'" }, { "__type": "Text", "__tag": 4046, "value": "the percentage of the null distribution that is greater than or equal to the observed value of the test statistic." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "'less'" }, { "__type": "Text", "__tag": 4046, "value": "the percentage of the null distribution that is less than or equal to the observed value of the test statistic." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "'two-sided'" }, { "__type": "Text", "__tag": 4046, "value": " (default)twice the smaller of the p-values above." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that p-values for randomized tests are calculated according to the conservative (over-estimated) approximation suggested in " }, { "__type": "FootnoteReference", "__tag": 4066, "label": "2" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "FootnoteReference", "__tag": 4066, "label": "3" }, { "__type": "Text", "__tag": 4046, "value": " rather than the unbiased estimator suggested in " }, { "__type": "FootnoteReference", "__tag": 4066, "label": "4" }, { "__type": "Text", "__tag": 4046, "value": ". That is, when calculating the proportion of the randomized null distribution that is as extreme as the observed value of the test statistic, the values in the numerator and denominator are both increased by one. An interpretation of this adjustment is that the observed value of the test statistic is always included as an element of the randomized null distribution. The convention used for two-sided p-values is not universal; the observed test statistic and null distribution are returned in case a different definition is preferred." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "axis", "annotation": "int, default: 0", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The axis of the (broadcasted) samples over which to calculate the statistic. If samples have a different number of dimensions, singleton dimensions are prepended to samples with fewer dimensions before " }, { "__type": "ParamRef", "__tag": 4071, "name": "axis" }, { "__type": "Text", "__tag": 4046, "value": " is considered." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "rng", "annotation": "{None, int, `numpy.random.Generator`}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If " }, { "__type": "ParamRef", "__tag": 4071, "name": "rng" }, { "__type": "Text", "__tag": 4046, "value": " is passed by keyword, types other than " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.random.Generator", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.random._generator:Generator" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " are passed to " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.random.default_rng", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.random:default_rng" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " to instantiate a " }, { "__type": "InlineCode", "__tag": 4051, "value": "Generator" }, { "__type": "Text", "__tag": 4046, "value": ". If " }, { "__type": "ParamRef", "__tag": 4071, "name": "rng" }, { "__type": "Text", "__tag": 4046, "value": " is already a " }, { "__type": "InlineCode", "__tag": 4051, "value": "Generator" }, { "__type": "Text", "__tag": 4046, "value": " instance, then the provided instance is used. Specify " }, { "__type": "ParamRef", "__tag": 4071, "name": "rng" }, { "__type": "Text", "__tag": 4046, "value": " for repeatable function behavior." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If this argument is passed by position or " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " is passed by keyword, legacy behavior for the argument " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " applies:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " is None (or " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.random", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.random" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": "), the " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.random.RandomState", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.random.mtrand:RandomState" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " singleton is used." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " is an int, a new " }, { "__type": "InlineCode", "__tag": 4051, "value": "RandomState" }, { "__type": "Text", "__tag": 4046, "value": " instance is used, seeded with " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " is already a " }, { "__type": "InlineCode", "__tag": 4051, "value": "Generator" }, { "__type": "Text", "__tag": 4046, "value": " or " }, { "__type": "InlineCode", "__tag": 4051, "value": "RandomState" }, { "__type": "Text", "__tag": 4046, "value": " instance then that instance is used." } ] } ] } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionchanged", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionchanged 1.15.0" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "As part of the " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "SPEC-007" } ], "url": "https://scientific-python.org/specs/spec-0007/", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " transition from use of " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.random.RandomState", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.random.mtrand:RandomState" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " to " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.random.Generator", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.random._generator:Generator" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", this keyword was changed from " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " to " }, { "__type": "ParamRef", "__tag": 4071, "name": "rng" }, { "__type": "Text", "__tag": 4046, "value": ". For an interim period, both keywords will continue to work, although only one may be specified at a time. After the interim period, function calls using the " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " keyword will emit warnings. The behavior of both " }, { "__type": "ParamRef", "__tag": 4071, "name": "random_state" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "ParamRef", "__tag": 4071, "name": "rng" }, { "__type": "Text", "__tag": 4046, "value": " are outlined above, but only the " }, { "__type": "ParamRef", "__tag": 4071, "name": "rng" }, { "__type": "Text", "__tag": 4046, "value": " keyword should be used in new code." } ] } ] } ] } ] } ], "title": [], "level": 0, "target": null }, "Extended Summary": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For independent sample statistics, the null hypothesis is that the data are randomly sampled from the same distribution. For paired sample statistics, two null hypothesis can be tested: that the data are paired at random or that the data are assigned to samples at random." } ] } ], "title": [], "level": 0, "target": null }, "Other Parameters": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null } }, "_ordered_sections": [ "Summary", "Extended Summary", "Parameters", "Attributes", "Methods", "Returns", "Yields", "Receives", "Other Parameters", "Raises", "Warns", "Warnings", "Notes" ], "item_file": "/scipy/stats/_resampling.py", "item_line": 1670, "item_type": "function", "aliases": [ "scipy.stats.permutation_test" ], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Text", "__tag": 4046, "value": "Suppose we wish to test whether two samples are drawn from the same\ndistribution. Assume that the underlying distributions are unknown to us,\nand that before observing the data, we hypothesized that the mean of the\nfirst sample would be less than that of the second sample. We decide that\nwe will use the difference between the sample means as a test statistic,\nand we will consider a p-value of 0.05 to be statistically significant.\n\nFor efficiency, we write the function defining the test statistic in a\nvectorized fashion: the samples ``x`` and ``y`` can be ND arrays, and the\nstatistic will be calculated for each axis-slice along `axis`.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "import numpy as np\ndef statistic(x, y, axis):\n return np.mean(x, axis=axis) - np.mean(y, axis=axis)\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nAfter collecting our data, we calculate the observed value of the test\nstatistic.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.stats import norm\nrng = np.random.default_rng()\nx = norm.rvs(size=5, random_state=rng)\ny = norm.rvs(size=6, loc = 3, random_state=rng)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "statistic(x, y, 0)\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nIndeed, the test statistic is negative, suggesting that the true mean of\nthe distribution underlying ``x`` is less than that of the distribution\nunderlying ``y``. To determine the probability of this occurring by chance\nif the two samples were drawn from the same distribution, we perform\na permutation test.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.stats import permutation_test\nres = permutation_test((x, y), statistic, vectorized=True,\n n_resamples=np.inf, alternative='less')\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "print(res.statistic)\n", "execution_status": "failure" }, { "__type": "Code", "__tag": 4050, "value": "print(res.pvalue)\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nThe probability of obtaining a test statistic less than or equal to the\nobserved value under the null hypothesis is 0.4329%. This is less than our\nchosen threshold of 5%, so we consider this to be significant evidence\nagainst the null hypothesis in favor of the alternative.\n\nBecause the size of the samples above was small, `permutation_test` could\nperform an exact test. For larger samples, we resort to a randomized\npermutation test.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "x = norm.rvs(size=100, random_state=rng)\ny = norm.rvs(size=120, loc=0.2, random_state=rng)\nres = permutation_test((x, y), statistic, n_resamples=9999,\n vectorized=True, alternative='less',\n rng=rng)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "print(res.statistic)\nprint(res.pvalue)\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nThe approximate probability of obtaining a test statistic less than or\nequal to the observed value under the null hypothesis is 0.0225%. This is\nagain less than our chosen threshold of 5%, so again we have significant\nevidence to reject the null hypothesis in favor of the alternative.\n\nFor large samples and number of permutations, the result is comparable to\nthat of the corresponding asymptotic test, the independent sample t-test.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.stats import ttest_ind\nres_asymptotic = ttest_ind(x, y, alternative='less')\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "print(res_asymptotic.pvalue)\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nThe permutation distribution of the test statistic is provided for\nfurther investigation.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "import matplotlib.pyplot as plt\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "plt.hist(res.null_distribution, bins=50)\nplt.title(\"Permutation distribution of test statistic\")\nplt.xlabel(\"Value of Statistic\")\nplt.ylabel(\"Frequency\")\n", "execution_status": "failure" }, { "__type": "Code", "__tag": 4050, "value": "plt.show()\n", "execution_status": "success" }, { "__type": "Figure", "__tag": 4024, "value": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "1.17.1", "kind": "assets", "path": "fig-0aa886eb867324f3.png" } }, { "__type": "Text", "__tag": 4046, "value": "\nInspection of the null distribution is essential if the statistic suffers\nfrom inaccuracy due to limited machine precision. Consider the following\ncase:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.stats import pearsonr\nx = [1, 2, 4, 3]\ny = [2, 4, 6, 8]\ndef statistic(x, y, axis=-1):\n return pearsonr(x, y, axis=axis).statistic\nres = permutation_test((x, y), statistic, vectorized=True,\n permutation_type='pairings',\n alternative='greater')\nr, pvalue, null = res.statistic, res.pvalue, res.null_distribution\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nIn this case, some elements of the null distribution differ from the\nobserved value of the correlation coefficient ``r`` due to numerical noise.\nWe manually inspect the elements of the null distribution that are nearly\nthe same as the observed value of the test statistic.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "r\n", "execution_status": "failure" }, { "__type": "Code", "__tag": 4050, "value": "unique = np.unique(null)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "unique\nunique[np.isclose(r, unique)].tolist()\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nIf `permutation_test` were to perform the comparison naively, the\nelements of the null distribution with value ``0.7999999999999998`` would\nnot be considered as extreme or more extreme as the observed value of the\nstatistic, so the calculated p-value would be too small.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "incorrect_pvalue = np.count_nonzero(null >= r) / len(null)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "incorrect_pvalue\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nInstead, `permutation_test` treats elements of the null distribution that\nare within ``max(1e-14, abs(r)*1e-14)`` of the observed value of the\nstatistic ``r`` to be equal to ``r``.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "correct_pvalue = np.count_nonzero(null >= r - 1e-14) / len(null)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "correct_pvalue\nres.pvalue == correct_pvalue\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nThis method of comparison is expected to be accurate in most practical\nsituations, but the user is advised to assess this by inspecting the\nelements of the null distribution that are close to the observed value\nof the statistic. Also, consider the use of statistics that can be\ncalculated using exact arithmetic (e.g. integer statistics)." } ], "title": [], "level": 0, "target": null }, "see_also": [], "signature": { "__type": "SignatureNode", "__tag": 4029, "kind": "function", "parameters": [ { "__type": "SigParam", "__tag": 4030, "name": "data", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": { "__type": "Empty", "__tag": 4031 } }, { "__type": "SigParam", "__tag": 4030, "name": "statistic", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": { "__type": "Empty", "__tag": 4031 } }, { "__type": "SigParam", "__tag": 4030, "name": "permutation_type", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "independent" }, { "__type": "SigParam", "__tag": 4030, "name": "vectorized", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "n_resamples", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "9999" }, { "__type": "SigParam", "__tag": 4030, "name": "batch", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "alternative", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "two-sided" }, { "__type": "SigParam", "__tag": 4030, "name": "axis", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "0" }, { "__type": "SigParam", "__tag": 4030, "name": "rng", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "random_state", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "None" } ], "return_annotation": { "__type": "Empty", "__tag": 4031 }, "target_name": "permutation_test" }, "references": [ ".. [1] R. A. Fisher. The Design of Experiments, 6th Ed (1951).", ".. [2] B. Phipson and G. K. Smyth. \"Permutation P-values Should Never Be", " Zero: Calculating Exact P-values When Permutations Are Randomly Drawn.\"", " Statistical Applications in Genetics and Molecular Biology 9.1 (2010).", ".. [3] M. D. Ernst. \"Permutation Methods: A Basis for Exact Inference\".", " Statistical Science (2004).", ".. [4] B. Efron and R. J. Tibshirani. An Introduction to the Bootstrap", " (1993)." ], "qa": "scipy.stats._resampling:permutation_test", "arbitrary": [], "local_refs": [ "alternative", "axis", "batch", "data", "n_resamples", "permutation_type", "res", "rng", "statistic", "vectorized" ] }