{ "__type": "IngestedDoc", "__tag": 4010, "_content": {}, "_ordered_sections": [], "item_file": null, "item_line": null, "item_type": null, "aliases": [], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "see_also": [], "signature": null, "references": null, "qa": "user:basics.strings", "arbitrary": [ { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "While NumPy is primarily a numerical library, it is often convenient to work with NumPy arrays of strings or bytes. The two most common use cases are:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Working with data loaded or memory-mapped from a data file, where one or more of the fields in the data is a string or bytestring, and the maximum length of the field is known ahead of time. This often is used for a name or label field." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Using NumPy indexing and broadcasting with arrays of Python strings of unknown length, which may or may not have data defined for every value." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For the first use case, NumPy provides the fixed-width " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.void", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:void" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.str_", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:str_" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.bytes_", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:bytes_" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " data types. For the second use case, numpy provides " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.dtypes.StringDType", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.dtypes:StringDType" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ". Below we describe how to work with both fixed-width and variable-width string arrays, how to convert between the two representations, and provide some advice for most efficiently working with string data in NumPy." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Working with Arrays of Strings And Bytes" } ], "level": 0, "target": "basics.strings" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Before NumPy 2.0, the fixed-width " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.str_", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:str_" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.bytes_", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:bytes_" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", and " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.void", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:void" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " data types were the only types available for working with strings and bytestrings in NumPy. For this reason, they are used as the default dtype for strings and bytestrings, respectively:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> np.array([\"hello\", \"world\"])\narray(['hello', 'world'], dtype='>> np.array([b\"hello\", b\"world\"])\narray([b'hello', b'world'], dtype='|S5')", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Since this is a one-byte encoding, the byteorder is " }, { "__type": "InlineRole", "__tag": 4003, "value": "'|'", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " (not applicable), and the data type detected is a maximum 5 character bytestring." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can also use " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.void", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:void" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " to represent bytestrings:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> np.array([b\"hello\", b\"world\"]).astype(np.void)\narray([b'\\x68\\x65\\x6C\\x6C\\x6F', b'\\x77\\x6F\\x72\\x6C\\x64'], dtype='|V5')", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This is most useful when working with byte streams that are not well represented as bytestrings, and instead are better thought of as collections of 8-bit integers." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Fixed-width data types" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Admonition", "__tag": 4056, "kind": "versionadded", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionadded 2.0" } ] } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "note", "base_type": "note", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "note " } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "CrossRef", "__tag": 4002, "value": "numpy.dtypes.StringDType", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.dtypes:StringDType" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " is a new addition to NumPy, implemented using the new support in NumPy for flexible user-defined data types and is not as extensively tested in production workflows as the older NumPy data types." } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Often, real-world string data does not have a predictable length. In these cases it is awkward to use fixed-width strings, since storing all the data without truncation requires knowing the length of the longest string one would like to store in the array before the array is created." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "To support situations like this, NumPy provides " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.dtypes.StringDType", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy.dtypes:StringDType" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", which stores variable-width string data in a UTF-8 encoding in a NumPy array:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> from numpy.dtypes import StringDType\n>>> data = [\"this is a longer string\", \"short string\"]\n>>> arr = np.array(data, dtype=StringDType())\n>>> arr\narray(['this is a longer string', 'short string'], dtype=StringDType())", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that unlike fixed-width strings, " }, { "__type": "InlineCode", "__tag": 4051, "value": "StringDType" }, { "__type": "Text", "__tag": 4046, "value": " is not parameterized by the maximum length of an array element, arbitrarily long or short strings can live in the same array without needing to reserve storage for padding bytes in the short strings." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Also note that unlike fixed-width strings and most other NumPy data types, " }, { "__type": "InlineCode", "__tag": 4051, "value": "StringDType" }, { "__type": "Text", "__tag": 4046, "value": " does not store the string data in the \"main\" " }, { "__type": "InlineCode", "__tag": 4051, "value": "ndarray" }, { "__type": "Text", "__tag": 4046, "value": " data buffer. Instead, the array buffer is used to store metadata about where the string data are stored in memory. This difference means that code expecting the array buffer to contain string data will not function correctly, and will need to be updated to support " }, { "__type": "InlineCode", "__tag": 4051, "value": "StringDType" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Variable-width strings" } ], "level": 1, "target": "stringdtype" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Often string datasets are not complete, and a special label is needed to indicate that a value is missing. By default " }, { "__type": "InlineCode", "__tag": 4051, "value": "StringDType" }, { "__type": "Text", "__tag": 4046, "value": " does not have any special support for missing values, besides the fact that empty strings are used to populate empty arrays:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> np.empty(3, dtype=StringDType())\narray(['', '', ''], dtype=StringDType())", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Optionally, you can create an instance of " }, { "__type": "InlineCode", "__tag": 4051, "value": "StringDType" }, { "__type": "Text", "__tag": 4046, "value": " with support for missing values by passing " }, { "__type": "InlineCode", "__tag": 4051, "value": "na_object" }, { "__type": "Text", "__tag": 4046, "value": " as a keyword argument for the initializer:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> dt = StringDType(na_object=None)\n>>> arr = np.array([\"this array has\", None, \"as an entry\"], dtype=dt)\n>>> arr\narray(['this array has', None, 'as an entry'],\n dtype=StringDType(na_object=None))\n>>> arr[1] is None\nTrue", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The " }, { "__type": "InlineCode", "__tag": 4051, "value": "na_object" }, { "__type": "Text", "__tag": 4046, "value": " can be any arbitrary python object. Common choices are " }, { "__type": "InlineRole", "__tag": 4003, "value": "numpy.nan", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "InlineCode", "__tag": 4051, "value": "float('nan')" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "InlineCode", "__tag": 4051, "value": "None" }, { "__type": "Text", "__tag": 4046, "value": ", an object specifically intended to represent missing data like " }, { "__type": "InlineCode", "__tag": 4051, "value": "pandas.NA" }, { "__type": "Text", "__tag": 4046, "value": ", or a (hopefully) unique string like " }, { "__type": "InlineCode", "__tag": 4051, "value": "\"__placeholder__\"" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "NumPy has special handling for NaN-like sentinels and string sentinels." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Missing data support" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A NaN-like sentinel returns itself as the result of arithmetic operations. This includes the python " }, { "__type": "InlineCode", "__tag": 4051, "value": "nan" }, { "__type": "Text", "__tag": 4046, "value": " float and the Pandas missing data sentinel " }, { "__type": "InlineCode", "__tag": 4051, "value": "pd.NA" }, { "__type": "Text", "__tag": 4046, "value": ". NaN-like sentinels inherit these behaviors in string operations. This means that, for example, the result of addition with any other string is the sentinel:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> dt = StringDType(na_object=np.nan)\n>>> arr = np.array([\"hello\", np.nan, \"world\"], dtype=dt)\n>>> arr + arr\narray(['hellohello', nan, 'worldworld'], dtype=StringDType(na_object=nan))", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Following the behavior of " }, { "__type": "InlineCode", "__tag": 4051, "value": "nan" }, { "__type": "Text", "__tag": 4046, "value": " in float arrays, NaN-like sentinels sort to the end of the array:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> np.sort(arr)\narray(['hello', 'world', nan], dtype=StringDType(na_object=nan))", "execution_status": null } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "NaN-like Missing Data Sentinels" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A string missing data value is an instance of " }, { "__type": "InlineCode", "__tag": 4051, "value": "str" }, { "__type": "Text", "__tag": 4046, "value": " or subtype of " }, { "__type": "InlineCode", "__tag": 4051, "value": "str" }, { "__type": "Text", "__tag": 4046, "value": ". If such an array is passed to a string operation or a cast, \"missing\" entries are treated as if they have a value given by the string sentinel. Comparison operations similarly use the sentinel value directly for missing entries." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "String Missing Data Sentinels" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Other objects, such as " }, { "__type": "InlineCode", "__tag": 4051, "value": "None" }, { "__type": "Text", "__tag": 4046, "value": " are also supported as missing data sentinels. If any missing data are present in an array using such a sentinel, then string operations will raise an error:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> dt = StringDType(na_object=None)\n>>> arr = np.array([\"this array has\", None, \"as an entry\"])\n>>> np.sort(arr)\nTraceback (most recent call last):\n...\nTypeError: '<' not supported between instances of 'NoneType' and 'str'", "execution_status": null } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Other Sentinels" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "By default, non-string data are coerced to strings:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> np.array([1, object(), 3.4], dtype=StringDType())\narray(['1', '', '3.4'], dtype=StringDType())", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If this behavior is not desired, an instance of the DType can be created that disables string coercion by setting " }, { "__type": "InlineCode", "__tag": 4051, "value": "coerce=False" }, { "__type": "Text", "__tag": 4046, "value": " in the initializer:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Code", "__tag": 4050, "value": ">>> np.array([1, object(), 3.4], dtype=StringDType(coerce=False))\nTraceback (most recent call last):\n...\nValueError: StringDType only allows string data when string coercion is disabled.", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This allows strict data validation in the same pass over the data NumPy uses to create the array. Setting " }, { "__type": "InlineCode", "__tag": 4051, "value": "coerce=True" }, { "__type": "Text", "__tag": 4046, "value": " recovers the default behavior allowing coercion to strings." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Coercing Non-strings" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "StringDType" }, { "__type": "Text", "__tag": 4046, "value": " supports round-trip casts between " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.str_", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:str_" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.bytes_", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:bytes_" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", and " }, { "__type": "CrossRef", "__tag": 4002, "value": "numpy.void", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "numpy", "version": "*", "kind": "api", "path": "numpy:void" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ". Casting to a fixed-width string is most useful when strings need to be memory-mapped in an ndarray or when a fixed-width string is needed for reading and writing to a columnar data format with a known maximum string length." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "In all cases, casting to a fixed-width string requires specifying the maximum allowed string length " } ] }, { "__type": "Code", "__tag": 4050, "value": ">>> arr = np.array([\"hello\", \"world\"], dtype=StringDType())\n>>> arr.astype(np.str_) # doctest: +IGNORE_EXCEPTION_DETAIL\nTraceback (most recent call last):\n...\nTypeError: Casting from StringDType to a fixed-width dtype with an\nunspecified size is not currently supported, specify an explicit\nsize for the output dtype instead.\n\nThe above exception was the direct cause of the following\nexception:\n\nTypeError: cannot cast dtype StringDType() to .\n>>> arr.astype(\"U5\")\narray(['hello', 'world'], dtype='>> arr = np.array([\"hello\", \"world\"], dtype=StringDType())\n>>> arr.astype(\"V5\")\narray([b'\\x68\\x65\\x6C\\x6C\\x6F', b'\\x77\\x6F\\x72\\x6C\\x64'], dtype='|V5')", "execution_status": null } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Care must be taken to ensure that the output array has enough space for the UTF-8 bytes in the string, since the size of a UTF-8 bytestream in bytes is not necessarily the same as the number of characters in the string." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Casting To and From Fixed-Width Strings" } ], "level": 2, "target": null } ], "local_refs": [] }