{ "__type": "IngestedDoc", "__tag": 4010, "_content": { "Notes": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Informally, the Shannon entropy quantifies the expected uncertainty inherent in the possible outcomes of a discrete random variable. For example, if messages consisting of sequences of symbols from a set are to be encoded and transmitted over a noiseless channel, then the Shannon entropy " }, { "__type": "InlineCode", "__tag": 4051, "value": "H(pk)" }, { "__type": "Text", "__tag": 4046, "value": " gives a tight lower bound for the average number of units of information needed per symbol if the symbols occur with frequencies governed by the discrete distribution " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": " " }, { "__type": "FootnoteReference", "__tag": 4066, "label": "1" }, { "__type": "Text", "__tag": 4046, "value": ". The choice of base determines the choice of units; e.g., " }, { "__type": "InlineCode", "__tag": 4051, "value": "e" }, { "__type": "Text", "__tag": 4046, "value": " for nats, " }, { "__type": "InlineCode", "__tag": 4051, "value": "2" }, { "__type": "Text", "__tag": 4046, "value": " for bits, etc." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The relative entropy, " }, { "__type": "InlineCode", "__tag": 4051, "value": "D(pk|qk)" }, { "__type": "Text", "__tag": 4046, "value": ", quantifies the increase in the average number of units of information needed per symbol if the encoding is optimized for the probability distribution " }, { "__type": "ParamRef", "__tag": 4071, "name": "qk" }, { "__type": "Text", "__tag": 4046, "value": " instead of the true distribution " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": ". Informally, the relative entropy quantifies the expected excess in surprise experienced if one believes the true distribution is " }, { "__type": "ParamRef", "__tag": 4071, "name": "qk" }, { "__type": "Text", "__tag": 4046, "value": " when it is actually " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A related quantity, the cross entropy " }, { "__type": "InlineCode", "__tag": 4051, "value": "CE(pk, qk)" }, { "__type": "Text", "__tag": 4046, "value": ", satisfies the equation " }, { "__type": "InlineCode", "__tag": 4051, "value": "CE(pk, qk) = H(pk) + D(pk|qk)" }, { "__type": "Text", "__tag": 4046, "value": " and can also be calculated with the formula " }, { "__type": "InlineCode", "__tag": 4051, "value": "CE = -sum(pk * log(qk))" }, { "__type": "Text", "__tag": 4046, "value": ". It gives the average number of units of information needed per symbol if an encoding is optimized for the probability distribution " }, { "__type": "ParamRef", "__tag": 4071, "name": "qk" }, { "__type": "Text", "__tag": 4046, "value": " when the true distribution is " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": ". It is not computed directly by " }, { "__type": "InlineRole", "__tag": 4003, "value": "entropy", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": ", but it can be computed using two calls to the function (see Examples)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "See " }, { "__type": "FootnoteReference", "__tag": 4066, "label": "2" }, { "__type": "Text", "__tag": 4046, "value": " for more information." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Beginning in SciPy 1.9, " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.matrix" }, { "__type": "Text", "__tag": 4046, "value": " inputs (not recommended for new code) are converted to " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.ndarray" }, { "__type": "Text", "__tag": 4046, "value": " before the calculation is performed. In this case, the output will be a scalar or " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.ndarray" }, { "__type": "Text", "__tag": 4046, "value": " of appropriate shape rather than a 2D " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.matrix" }, { "__type": "Text", "__tag": 4046, "value": ". Similarly, while masked elements of masked arrays are ignored, the output will be a scalar or " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.ndarray" }, { "__type": "Text", "__tag": 4046, "value": " rather than a masked array with " }, { "__type": "InlineCode", "__tag": 4051, "value": "mask=False" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Array API Standard Support" } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineRole", "__tag": 4003, "value": "entropy", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " has experimental support for Python Array API Standard compatible backends in addition to NumPy. Please consider testing these features by setting an environment variable " }, { "__type": "InlineCode", "__tag": 4051, "value": "SCIPY_ARRAY_API=1" }, { "__type": "Text", "__tag": 4046, "value": " and providing CuPy, PyTorch, JAX, or Dask arrays as array arguments. The following combinations of backend and device (or other capability) are supported." } ] }, { "__type": "Code", "__tag": 4050, "value": "==================== ==================== ====================\nLibrary CPU GPU\n==================== ==================== ====================\nNumPy ✅ n/a \nCuPy n/a ✅ \nPyTorch ✅ ✅ \nJAX ✅ ✅ \nDask ✅ n/a \n==================== ==================== ====================", "execution_status": null }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "See " }, { "__type": "InlineRole", "__tag": 4003, "value": "dev-arrayapi", "domain": null, "role": "ref", "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " for more information." } ] } ] } ], "title": [], "level": 0, "target": null }, "Warns": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Raises": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Yields": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Methods": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Returns": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Parameters", "__tag": 4026, "children": [ { "__type": "DocParam", "__tag": 4016, "name": "S", "annotation": "{float, array_like}", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The calculated entropy." } ] } ] } ] } ], "title": [], "level": 0, "target": null }, "Summary": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Calculate the Shannon entropy/relative entropy of given distribution(s)." } ] } ], "title": [], "level": 0, "target": null }, "Receives": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Warnings": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Attributes": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Parameters": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Parameters", "__tag": 4026, "children": [ { "__type": "DocParam", "__tag": 4016, "name": "pk", "annotation": "array_like", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Defines the (discrete) distribution. Along each axis-slice of " }, { "__type": "InlineCode", "__tag": 4051, "value": "pk" }, { "__type": "Text", "__tag": 4046, "value": ", element " }, { "__type": "InlineCode", "__tag": 4051, "value": "i" }, { "__type": "Text", "__tag": 4046, "value": " is the (possibly unnormalized) probability of event " }, { "__type": "InlineCode", "__tag": 4051, "value": "i" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "qk", "annotation": "array_like, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Sequence against which the relative entropy is computed. Should be in the same format as " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "base", "annotation": "float, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The logarithmic base to use, defaults to " }, { "__type": "InlineCode", "__tag": 4051, "value": "e" }, { "__type": "Text", "__tag": 4046, "value": " (natural logarithm)." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "axis", "annotation": "int or None, default: 0", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If an int, the axis of the input along which to compute the statistic. The statistic of each axis-slice (e.g. row) of the input will appear in a corresponding element of the output. If " }, { "__type": "InlineCode", "__tag": 4051, "value": "None" }, { "__type": "Text", "__tag": 4046, "value": ", the input will be raveled before computing the statistic." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "nan_policy", "annotation": "{'propagate', 'omit', 'raise'}", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Defines how to handle input NaNs." } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "propagate" }, { "__type": "Text", "__tag": 4046, "value": ": if a NaN is present in the axis slice (e.g. row) along which the statistic is computed, the corresponding entry of the output will be NaN." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "omit" }, { "__type": "Text", "__tag": 4046, "value": ": NaNs will be omitted when performing the calculation. If insufficient data remains in the axis slice along which the statistic is computed, the corresponding entry of the output will be NaN." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "raise" }, { "__type": "Text", "__tag": 4046, "value": ": if a NaN is present, a " }, { "__type": "InlineCode", "__tag": 4051, "value": "ValueError" }, { "__type": "Text", "__tag": 4046, "value": " will be raised." } ] } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "keepdims", "annotation": "bool, default: False", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If this is set to True, the axes which are reduced are left in the result as dimensions with size one. With this option, the result will broadcast correctly against the input array." } ] } ] } ] } ], "title": [], "level": 0, "target": null }, "Extended Summary": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If only probabilities " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": " are given, the Shannon entropy is calculated as " }, { "__type": "InlineCode", "__tag": 4051, "value": "H = -sum(pk * log(pk))" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If " }, { "__type": "ParamRef", "__tag": 4071, "name": "qk" }, { "__type": "Text", "__tag": 4046, "value": " is not None, then compute the relative entropy " }, { "__type": "InlineCode", "__tag": 4051, "value": "D = sum(pk * log(pk / qk))" }, { "__type": "Text", "__tag": 4046, "value": ". This quantity is also known as the Kullback-Leibler divergence." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This routine will normalize " }, { "__type": "ParamRef", "__tag": 4071, "name": "pk" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "ParamRef", "__tag": 4071, "name": "qk" }, { "__type": "Text", "__tag": 4046, "value": " if they don't sum to 1." } ] } ], "title": [], "level": 0, "target": null }, "Other Parameters": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null } }, "_ordered_sections": [ "Summary", "Extended Summary", "Parameters", "Attributes", "Methods", "Returns", "Yields", "Receives", "Other Parameters", "Raises", "Warns", "Warnings", "Notes" ], "item_file": "/scipy/stats/_entropy.py", "item_line": 17, "item_type": "function", "aliases": [ "scipy.stats.entropy" ], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Text", "__tag": 4046, "value": "The outcome of a fair coin is the most uncertain:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "import numpy as np\nfrom scipy.stats import entropy\nbase = 2 # work in units of bits\npk = np.array([1/2, 1/2]) # fair coin\nH = entropy(pk, base=base)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "H\nH == -np.sum(pk * np.log(pk)) / np.log(base)\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nThe outcome of a biased coin is less uncertain:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "qk = np.array([9/10, 1/10]) # biased coin\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "entropy(qk, base=base)\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nThe relative entropy between the fair coin and biased coin is calculated\nas:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "D = entropy(pk, qk, base=base)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "D\nnp.isclose(D, np.sum(pk * np.log(pk/qk)) / np.log(base), rtol=4e-16, atol=0)\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nThe cross entropy can be calculated as the sum of the entropy and\nrelative entropy`:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "CE = entropy(pk, base=base) + entropy(pk, qk, base=base)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "CE\nCE == -np.sum(pk * np.log(qk)) / np.log(base)\n", "execution_status": "failure" } ], "title": [], "level": 0, "target": null }, "see_also": [], "signature": { "__type": "SignatureNode", "__tag": 4029, "kind": "function", "parameters": [ { "__type": "SigParam", "__tag": 4030, "name": "pk", "annotation": "ArrayLike", "kind": "POSITIONAL_OR_KEYWORD", "default": { "__type": "Empty", "__tag": 4031 } }, { "__type": "SigParam", "__tag": 4030, "name": "qk", "annotation": "ArrayLike | None", "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "base", "annotation": "float | None", "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "axis", "annotation": "int", "kind": "POSITIONAL_OR_KEYWORD", "default": "0" }, { "__type": "SigParam", "__tag": 4030, "name": "nan_policy", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "propagate" }, { "__type": "SigParam", "__tag": 4030, "name": "keepdims", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "KEYWORD_ONLY", "default": "False" } ], "return_annotation": "numpy.number | numpy.ndarray", "target_name": "entropy" }, "references": [ ".. [1] Shannon, C.E. (1948), A Mathematical Theory of Communication.", " Bell System Technical Journal, 27: 379-423.", " https://doi.org/10.1002/j.1538-7305.1948.tb01338.x", ".. [2] Thomas M. Cover and Joy A. Thomas. 2006. Elements of Information", " Theory (Wiley Series in Telecommunications and Signal Processing).", " Wiley-Interscience, USA." ], "qa": "scipy.stats._entropy:entropy", "arbitrary": [], "local_refs": [ "S", "axis", "base", "keepdims", "nan_policy", "pk", "qk" ] }