{ "__type": "IngestedDoc", "__tag": 4010, "_content": {}, "_ordered_sections": [], "item_file": null, "item_line": null, "item_type": null, "aliases": [], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "see_also": [], "signature": null, "references": null, "qa": "dev:development_advanced_debugging", "arbitrary": [ { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If you reached here, you want to dive into, or use, more advanced tooling. This is usually not necessary for first-time contributors and most day-to-day development. These are used more rarely, for example close to a new NumPy release, or when a large or particular complex change was made." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Some of these tools are used in NumPy's continuous integration tests. If you see a test failure that only happens under a debugging tool, these instructions should hopefully enable you to reproduce the test failure locally." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Since not all of these tools are used on a regular basis and only available on some systems, please expect differences, issues, or quirks; we will be happy to help if you get stuck and appreciate any improvements or suggestions to these workflows." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Advanced debugging tools" } ], "level": 0, "target": "advanced_debugging" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Most development will not require more than a typical debugging toolchain as shown in " }, { "__type": "CrossRef", "__tag": 4002, "value": "Debugging", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "docs", "path": "dev:development_environment" }, "kind": "exists" }, { "__type": "Text", "__tag": 4046, "value": ". But for example memory leaks can be particularly subtle or difficult to narrow down." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "We do not expect any of these tools to be run by most contributors. However, you can ensure that we can track down such issues more easily:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Tests should cover all code paths, including error paths." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Try to write short and simple tests. If you have a very complicated test consider creating an additional simpler test as well. This can be helpful, because often it is only easy to find which test triggers an issue and not which line of the test." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Never use " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.empty" }, { "__type": "Text", "__tag": 4046, "value": " if data is read/used. " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "Valgrind" } ], "url": "https://valgrind.org/", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " will notice this and report an error. When you do not care about values, you can generate random values instead." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This will help us catch any oversights before your change is released and means you do not have to worry about making reference counting errors, which can be intimidating." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Finding C errors with additional tooling" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Debug builds of Python are easily available for example via the system package manager on Linux systems, but are also available on other platforms, possibly in a less convenient format. If you cannot easily install a debug build of Python from a system package manager, you can build one yourself using " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "pyenv" } ], "url": "https://github.com/pyenv/pyenv", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ". For example, to install and globally activate a debug build of Python 3.13.3, one would do " } ] }, { "__type": "Code", "__tag": 4050, "value": "pyenv install -g 3.13.3\npyenv global 3.13.3", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyenv install" }, { "__type": "Text", "__tag": 4046, "value": " builds Python from source, so you must ensure that Python's dependencies are installed before building, see the pyenv documentation for platform-specific installation instructions. You can use " }, { "__type": "InlineCode", "__tag": 4051, "value": "pip" }, { "__type": "Text", "__tag": 4046, "value": " to install Python dependencies you may need for your debugging session. If there is no debug wheel available on " }, { "__type": "InlineRole", "__tag": 4003, "value": "pypi,", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " you will need to build the dependencies from source and ensure that your dependencies are also compiled as debug builds." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Often debug builds of Python name the Python executable " }, { "__type": "InlineCode", "__tag": 4051, "value": "pythond" }, { "__type": "Text", "__tag": 4046, "value": " instead of " }, { "__type": "InlineCode", "__tag": 4051, "value": "python" }, { "__type": "Text", "__tag": 4046, "value": ". To check if you have a debug build of Python installed, you can run e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "pythond -m sysconfig" }, { "__type": "Text", "__tag": 4046, "value": " to get the build configuration for the Python executable. A debug build will be built with debug compiler options in " }, { "__type": "InlineCode", "__tag": 4051, "value": "CFLAGS" }, { "__type": "Text", "__tag": 4046, "value": " (e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "-g -Og" }, { "__type": "Text", "__tag": 4046, "value": ")." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Running the Numpy tests or an interactive terminal is usually as easy as " } ] }, { "__type": "Code", "__tag": 4050, "value": "python3.8d runtests.py\n# or\npython3.8d runtests.py --ipython", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "and were already mentioned in " }, { "__type": "CrossRef", "__tag": 4002, "value": "Debugging", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "docs", "path": "dev:development_environment" }, "kind": "exists" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A Python debug build will help:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Find bugs which may otherwise cause random behaviour. One example is when an object is still used after it has been deleted." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Python debug builds allows to check correct reference counting. This works using the additional commands " } ] }, { "__type": "Code", "__tag": 4050, "value": "sys.gettotalrefcount()\nsys.getallocatedblocks()", "execution_status": null } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Python debug builds allow easier debugging with gdb and other C debuggers." } ] } ] } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Python debug build" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Running the test suite only with a debug python build will not find many errors on its own. An additional advantage of a debug build of Python is that it allows detecting memory leaks." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A tool to make this easier is " }, { "__type": "CrossRef", "__tag": 4002, "value": "pytest-leaks", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "docs", "path": "dev:development_advanced_debugging" }, "kind": "exists" }, { "__type": "Text", "__tag": 4046, "value": ", which can be installed using " }, { "__type": "InlineCode", "__tag": 4051, "value": "pip" }, { "__type": "Text", "__tag": 4046, "value": ". Unfortunately, " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest" }, { "__type": "Text", "__tag": 4046, "value": " itself may leak memory, but good results can usually (currently) be achieved by removing " } ] }, { "__type": "Code", "__tag": 4050, "value": "@pytest.fixture(autouse=True)\ndef add_np(doctest_namespace):\n doctest_namespace['np'] = numpy\n\n@pytest.fixture(autouse=True)\ndef env_setup(monkeypatch):\n monkeypatch.setenv('PYTHONHASHSEED', '0')", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "from " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy/conftest.py" }, { "__type": "Text", "__tag": 4046, "value": " (This may change with new " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest-leaks" }, { "__type": "Text", "__tag": 4046, "value": " versions or " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest" }, { "__type": "Text", "__tag": 4046, "value": " updates)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This allows to run the test suite, or part of it, conveniently " } ] }, { "__type": "Code", "__tag": 4050, "value": "python3.8d runtests.py -t numpy/_core/tests/test_multiarray.py -- -R2:3 -s", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "where " }, { "__type": "InlineCode", "__tag": 4051, "value": "-R2:3" }, { "__type": "Text", "__tag": 4046, "value": " is the " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest-leaks" }, { "__type": "Text", "__tag": 4046, "value": " command (see its documentation), the " }, { "__type": "InlineCode", "__tag": 4051, "value": "-s" }, { "__type": "Text", "__tag": 4046, "value": " causes output to print and may be necessary (in some versions captured output was detected as a leak)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that some tests are known (or even designed) to leak references, we try to mark them, but expect some false positives." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Use together with " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "Valgrind" } ], "url": "https://valgrind.org/", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " is a powerful tool to find certain memory access problems and should be run on complicated C code. Basic use of " }, { "__type": "InlineCode", "__tag": 4051, "value": "valgrind" }, { "__type": "Text", "__tag": 4046, "value": " usually requires no more than " } ] }, { "__type": "Code", "__tag": 4050, "value": "PYTHONMALLOC=malloc valgrind python runtests.py", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "where " }, { "__type": "InlineCode", "__tag": 4051, "value": "PYTHONMALLOC=malloc" }, { "__type": "Text", "__tag": 4046, "value": " is necessary to avoid false positives from python itself. Depending on the system and valgrind version, you may see more false positives. " }, { "__type": "InlineCode", "__tag": 4051, "value": "valgrind" }, { "__type": "Text", "__tag": 4046, "value": " supports \"suppressions\" to ignore some of these, and Python does have a suppression file (and even a compile time option) which may help if you find it necessary." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Valgrind helps:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Find use of uninitialized variables/memory." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Detect memory access violations (reading or writing outside of allocated memory)." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Find " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "many" } ] }, { "__type": "Text", "__tag": 4046, "value": " memory leaks. Note that for " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "most" } ] }, { "__type": "Text", "__tag": 4046, "value": " leaks the python debug build approach (and " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest-leaks" }, { "__type": "Text", "__tag": 4046, "value": ") is much more sensitive. The reason is that " }, { "__type": "InlineCode", "__tag": 4051, "value": "valgrind" }, { "__type": "Text", "__tag": 4046, "value": " can only detect if memory is definitely lost. If " } ] }, { "__type": "Code", "__tag": 4050, "value": "dtype = np.dtype(np.int64)\narr.astype(dtype=dtype)", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Has incorrect reference counting for " }, { "__type": "InlineCode", "__tag": 4051, "value": "dtype" }, { "__type": "Text", "__tag": 4046, "value": ", this is a bug, but valgrind cannot see it because " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.dtype(np.int64)" }, { "__type": "Text", "__tag": 4046, "value": " always returns the same object. However, not all dtypes are singletons, so this might leak memory for different input. In rare cases NumPy uses " }, { "__type": "InlineCode", "__tag": 4051, "value": "malloc" }, { "__type": "Text", "__tag": 4046, "value": " and not the Python memory allocators which are invisible to the Python debug build. " }, { "__type": "InlineCode", "__tag": 4051, "value": "malloc" }, { "__type": "Text", "__tag": 4046, "value": " should normally be avoided, but there are some exceptions (e.g. the " }, { "__type": "InlineCode", "__tag": 4051, "value": "PyArray_Dims" }, { "__type": "Text", "__tag": 4046, "value": " structure is public API and cannot use the Python allocators.)" } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Even though using valgrind for memory leak detection is slow and less sensitive it can be convenient: you can run most programs with valgrind without modification." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Things to be aware of:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Valgrind does not support the numpy " }, { "__type": "InlineCode", "__tag": 4051, "value": "longdouble" }, { "__type": "Text", "__tag": 4046, "value": ", this means that tests will fail or be flagged errors that are completely fine." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Expect some errors before and after running your NumPy code." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Caches can mean that errors (specifically memory leaks) may not be detected or are only detect at a later, unrelated time." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A big advantage of valgrind is that it has no requirements aside from valgrind itself (although you probably want to use debug builds for better tracebacks)." } ] } ], "title": [ { "__type": "InlineCode", "__tag": 4051, "value": "valgrind" } ], "level": 2, "target": "pytest-leaks" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can run the test suite with valgrind which may be sufficient when you are only interested in a few tests " } ] }, { "__type": "Code", "__tag": 4050, "value": "PYTHONMALLOC=malloc valgrind python runtests.py \\\n -t numpy/_core/tests/test_multiarray.py -- --continue-on-collection-errors", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note the " }, { "__type": "InlineCode", "__tag": 4051, "value": "--continue-on-collection-errors" }, { "__type": "Text", "__tag": 4046, "value": ", which is currently necessary due to missing " }, { "__type": "InlineCode", "__tag": 4051, "value": "longdouble" }, { "__type": "Text", "__tag": 4046, "value": " support causing failures (this will usually not be necessary if you do not run the full test suite)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If you wish to detect memory leaks you will also require " }, { "__type": "InlineCode", "__tag": 4051, "value": "--show-leak-kinds=definite" }, { "__type": "Text", "__tag": 4046, "value": " and possibly more valgrind options. Just as for " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest-leaks" }, { "__type": "Text", "__tag": 4046, "value": " certain tests are known to leak cause errors in valgrind and may or may not be marked as such." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "We have developed " }, { "__type": "CrossRef", "__tag": 4002, "value": "pytest-valgrind", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "docs", "path": "dev:development_advanced_debugging" }, "kind": "exists" }, { "__type": "Text", "__tag": 4046, "value": " which:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Reports errors for each test individually" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Narrows down memory leaks to individual tests (by default valgrind only checks for memory leaks after a program stops, which is very cumbersome)." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Please refer to its " }, { "__type": "InlineCode", "__tag": 4051, "value": "README" }, { "__type": "Text", "__tag": 4046, "value": " for more information (it includes an example command for NumPy)." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Use together with " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Whenever NumPy crashes or when working on changes to NumPy's low-level C or C++ code, it's often convenient to run Python under a C debugger to get more information. A debugger can aid in understanding an interpreter crash (e.g. due to a segmentation fault) by providing a C call stack at the site of the crash. The call stack often provides valuable context to understand the nature of a crash. C debuggers are also very useful during development, allowing interactive debugging in the C implementation of NumPy." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The NumPy developers often use both " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " to debug Numpy. As a rule of thumb, " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " is often easier to use on Linux while " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " is easier to use on a Mac environment. They have disjoint user interfaces, so you will need to learn how to use whichever one you land on. The " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " to " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "command map" } ], "url": "https://lldb.llvm.org/use/map.html", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " is a convenient reference for how to accomplish common recipes in both debuggers." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "C debuggers" } ], "level": 2, "target": "pytest-valgrind" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" }, { "__type": "Text", "__tag": 4046, "value": " " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "development workflow tool" } ], "url": "https://github.com/scientific-python/spin", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ". has built-in support for working with both " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " via the " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin gdb" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin lldb" }, { "__type": "Text", "__tag": 4046, "value": " commands." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "note", "base_type": "note", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "note " } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Building with " }, { "__type": "InlineCode", "__tag": 4051, "value": "-Dbuildtype=debug" }, { "__type": "Text", "__tag": 4046, "value": " has a couple of important effects to be aware of:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Assertions are enabled" } ] }, { "__type": "Text", "__tag": 4046, "value": ": This build type does not define the " }, { "__type": "InlineCode", "__tag": 4051, "value": "NDEBUG" }, { "__type": "Text", "__tag": 4046, "value": " macro, which means that any C-level assertions in the code will be active. This is very useful for debugging, as it can help pinpoint where an unexpected condition occurs." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Strong", "__tag": 4048, "children": [ { "__type": "Text", "__tag": 4046, "value": "Compiler flags may need overriding" } ] }, { "__type": "Text", "__tag": 4046, "value": ": Some compiler toolchains, particularly those from " }, { "__type": "InlineCode", "__tag": 4051, "value": "conda-forge" }, { "__type": "Text", "__tag": 4046, "value": ", may set optimization flags like " }, { "__type": "InlineCode", "__tag": 4051, "value": "-O2" }, { "__type": "Text", "__tag": 4046, "value": " by default. These can override the " }, { "__type": "InlineCode", "__tag": 4051, "value": "debug" }, { "__type": "Text", "__tag": 4046, "value": " build type. To ensure a true debug build in such environments, you may need to manually unset or override this flag." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For more details on both points, see the " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "meson-python guide on debug builds" } ], "url": "https://mesonbuild.com/meson-python/how-to-guides/debug-builds.html", "title": "" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For both debuggers, it's advisable to build NumPy in either the " }, { "__type": "InlineCode", "__tag": 4051, "value": "debug" }, { "__type": "Text", "__tag": 4046, "value": " or " }, { "__type": "InlineCode", "__tag": 4051, "value": "debugoptimized" }, { "__type": "Text", "__tag": 4046, "value": " meson build profile. To use " }, { "__type": "InlineCode", "__tag": 4051, "value": "debug" }, { "__type": "Text", "__tag": 4046, "value": " you can pass the option via " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin build" }, { "__type": "Text", "__tag": 4046, "value": ":" } ] }, { "__type": "Code", "__tag": 4050, "value": "spin build -- -Dbuildtype=debug", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "to use " }, { "__type": "InlineCode", "__tag": 4051, "value": "debugoptimized" }, { "__type": "Text", "__tag": 4046, "value": " you're pass " }, { "__type": "InlineCode", "__tag": 4051, "value": "-Dbuildtype=debugoptimized" }, { "__type": "Text", "__tag": 4046, "value": " instead." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can pass additional arguments to " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "meson setup" } ], "url": "https://mesonbuild.com/Builtin-options.html", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " besides " }, { "__type": "InlineCode", "__tag": 4051, "value": "buildtype" }, { "__type": "Text", "__tag": 4046, "value": " using the same positional argument syntax for " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin build" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Building With Debug Symbols" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Let's say you have a test script named " }, { "__type": "InlineRole", "__tag": 4003, "value": "test.py", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " that lives in a " }, { "__type": "InlineCode", "__tag": 4051, "value": "test" }, { "__type": "Text", "__tag": 4046, "value": " folder in the same directory as the NumPy source checkout. You could execute the test script using the " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" }, { "__type": "Text", "__tag": 4046, "value": " build of NumPy with the following incantation:" } ] }, { "__type": "Code", "__tag": 4050, "value": "spin gdb ../test/test.py", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This will launch into gdb. If all you care about is a call stack for a crash, type \"r\" and hit enter. Your test script will run and if a crash happens, you type \"bt\" to get a traceback. For " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": ", the instructions are similar, just replace " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin gdb" }, { "__type": "Text", "__tag": 4046, "value": " with " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin lldb" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can also set breakpoints and use other more advanced techniques. See the documentation for your debugger for more details." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "One common issue with breakpoints in NumPy is that some code paths get hit repeatedly during the import of the " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy" }, { "__type": "Text", "__tag": 4046, "value": " module. This can make it tricky or tedious to find the first \"real\" call after the NumPy import has completed and the " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy" }, { "__type": "Text", "__tag": 4046, "value": " module is fully initialized." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "One workaround is to use a script like this:" } ] }, { "__type": "Code", "__tag": 4050, "value": "import os\nimport signal\n\nimport numpy as np\n\nPID = os.getpid()\n\ndef do_nothing(*args):\n pass\n\nsignal.signal(signal.SIGUSR1, do_nothing)\n\nos.kill(PID, signal.SIGUSR1)\n\n# the code to run under a debugger follows", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This example installs a signal handler for the " }, { "__type": "InlineCode", "__tag": 4051, "value": "SIGUSR1" }, { "__type": "Text", "__tag": 4046, "value": " signal that does nothing and then calls " }, { "__type": "InlineCode", "__tag": 4051, "value": "os.kill" }, { "__type": "Text", "__tag": 4046, "value": " on the Python process with the " }, { "__type": "InlineCode", "__tag": 4051, "value": "SIGUSR1" }, { "__type": "Text", "__tag": 4046, "value": " signal. This causes the signal handler to fire and critically also causes both " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " to halt execution inside of the " }, { "__type": "InlineCode", "__tag": 4051, "value": "kill" }, { "__type": "Text", "__tag": 4046, "value": " syscall." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If you run " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " you should see output something like this:" } ] }, { "__type": "Code", "__tag": 4050, "value": "Process 67365 stopped\n * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGUSR1\n frame #0: 0x000000019c4b9da4 libsystem_kernel.dylib`__kill + 8\n libsystem_kernel.dylib`__kill:\n -> 0x19c4b9da4 <+8>: b.lo 0x19c4b9dc4 ; <+40>\n 0x19c4b9da8 <+12>: pacibsp\n 0x19c4b9dac <+16>: stp x29, x30, [sp, #-0x10]!\n 0x19c4b9db0 <+20>: mov x29, sp\n Target 0: (python3.13) stopped.\n (lldb) bt\n * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGUSR1\n * frame #0: 0x000000019c4b9da4 libsystem_kernel.dylib`__kill + 8\n frame #1: 0x000000010087f5c4 libpython3.13.dylib`os_kill + 104\n frame #2: 0x000000010071374c libpython3.13.dylib`cfunction_vectorcall_FASTCALL + 276\n frame #3: 0x00000001006c1e3c libpython3.13.dylib`PyObject_Vectorcall + 88\n frame #4: 0x00000001007edd1c libpython3.13.dylib`_PyEval_EvalFrameDefault + 23608\n frame #5: 0x00000001007e7e6c libpython3.13.dylib`PyEval_EvalCode + 252\n frame #6: 0x0000000100852944 libpython3.13.dylib`run_eval_code_obj + 180\n frame #7: 0x0000000100852610 libpython3.13.dylib`run_mod + 220\n frame #8: 0x000000010084fa4c libpython3.13.dylib`_PyRun_SimpleFileObject + 868\n frame #9: 0x000000010084f400 libpython3.13.dylib`_PyRun_AnyFileObject + 160\n frame #10: 0x0000000100874ab8 libpython3.13.dylib`pymain_run_file + 336\n frame #11: 0x0000000100874324 libpython3.13.dylib`Py_RunMain + 1516\n frame #12: 0x000000010087459c libpython3.13.dylib`pymain_main + 324\n frame #13: 0x000000010087463c libpython3.13.dylib`Py_BytesMain + 40\n frame #14: 0x000000019c152b98 dyld`start + 6076\n(lldb)", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "As you can see, the C stack trace is inside of the " }, { "__type": "InlineCode", "__tag": 4051, "value": "kill" }, { "__type": "Text", "__tag": 4046, "value": " syscall and an " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " prompt is active, allowing interactively setting breakpoints. Since the " }, { "__type": "InlineCode", "__tag": 4051, "value": "os.kill" }, { "__type": "Text", "__tag": 4046, "value": " call happens after the " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy" }, { "__type": "Text", "__tag": 4046, "value": " module is already fully initialized, this means any breakpoints set inside of " }, { "__type": "InlineCode", "__tag": 4051, "value": "kill" }, { "__type": "Text", "__tag": 4046, "value": " will happen " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "after" } ] }, { "__type": "Text", "__tag": 4046, "value": " " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy" }, { "__type": "Text", "__tag": 4046, "value": " is finished initializing." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Running a Test Script" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can also run " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest" }, { "__type": "Text", "__tag": 4046, "value": " tests under a debugger. This requires using the debugger in a slightly more manual fashion, since " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" }, { "__type": "Text", "__tag": 4046, "value": " does not yet automate this process. First, run " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin build" }, { "__type": "Text", "__tag": 4046, "value": " to ensure there is a fully built copy of NumPy managed by " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" }, { "__type": "Text", "__tag": 4046, "value": ". Then, to run the tests under " }, { "__type": "InlineCode", "__tag": 4051, "value": "lldb" }, { "__type": "Text", "__tag": 4046, "value": " you would do something like this:" } ] }, { "__type": "Code", "__tag": 4050, "value": "spin lldb $(which python) $(which pytest) build-install/usr/lib/python3.13/site-packages/numpy/_core/tests/test_multiarray.py", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This will execute the tests in " }, { "__type": "InlineCode", "__tag": 4051, "value": "test_multiarray.py" }, { "__type": "Text", "__tag": 4046, "value": " under lldb after typing 'r' and hitting enter. Note that this command comes from a session using Python 3.13 on a Mac. If you are using a different Python version or operating system, the directory layout inside " }, { "__type": "InlineCode", "__tag": 4051, "value": "build-install" }, { "__type": "Text", "__tag": 4046, "value": " may be slightly different." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can set breakpoints as described above. The issue about breakpoints commonly being hit during NumPy import also applies - consider refactoring your test workflow into a test script so you can adopt the workaround using " }, { "__type": "InlineCode", "__tag": 4051, "value": "os.kill" }, { "__type": "Text", "__tag": 4046, "value": " described above." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note the use of " }, { "__type": "InlineCode", "__tag": 4051, "value": "$(which python)" }, { "__type": "Text", "__tag": 4046, "value": " to ensure the debugger receives a path to a Python executable. If you are using " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyenv" }, { "__type": "Text", "__tag": 4046, "value": ", you may need to replace " }, { "__type": "InlineCode", "__tag": 4051, "value": "which python" }, { "__type": "Text", "__tag": 4046, "value": " with " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyenv which python" }, { "__type": "Text", "__tag": 4046, "value": ", since " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyenv" }, { "__type": "Text", "__tag": 4046, "value": " relies on shim scripts that " }, { "__type": "InlineCode", "__tag": 4051, "value": "which" }, { "__type": "Text", "__tag": 4046, "value": " doesn't know about." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Use together with " }, { "__type": "InlineCode", "__tag": 4051, "value": "pytest" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "compiler sanitizer" } ], "url": "https://hpc-wiki.info/hpc/Compiler_Sanitizers", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " suites shipped by both GCC and LLVM offer a means to detect many common programming errors at runtime. The sanitizers work by instrumenting the application code at build time so additional runtime checks fire. Typically, sanitizers are run during the course of regular testing and if a sanitizer check fails, this leads to a test failure or crash, along with a report about the nature of the failure." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "While it is possible to use sanitizers with a \"regular\" build of CPython - it is best if you can set up a Python environment based on a from-source Python build with sanitizer instrumentation, and then use the instrumented Python to build NumPy and run the tests. If the entire Python stack is instrumented using the same sanitizer runtime, it becomes possible to identify issues that happen across the Python stack. This enables detecting memory leaks in NumPy due to misuse of memory allocated in CPython, for example." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Compiler Sanitizers" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "See the " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "section in the Python developer's guide" } ], "url": "https://devguide.python.org/getting-started/setup-building/", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " on this topic for more information about building Python from source. To enable address sanitizer, you will need to pass " }, { "__type": "InlineCode", "__tag": 4051, "value": "--with-address-sanitizer" }, { "__type": "Text", "__tag": 4046, "value": " to the " }, { "__type": "InlineCode", "__tag": 4051, "value": "configure" }, { "__type": "Text", "__tag": 4046, "value": " script invocation when you build Python." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can also use " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "pyenv" } ], "url": "https://github.com/pyenv/pyenv", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " to automate the process of building Python and quickly activate or deactivate a Python installation using a command-line interface similar to virtual environments. With " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyenv" }, { "__type": "Text", "__tag": 4046, "value": " you could install an ASAN-instrumented build of Python 3.13 like this:" } ] }, { "__type": "Code", "__tag": 4050, "value": "CONFIGURE_OPTS=\"--with-address-sanitizer\" pyenv install 3.13", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If you are interested in thread sanitizer, the " }, { "__type": "InlineCode", "__tag": 4051, "value": "cpython_sanity" }, { "__type": "Text", "__tag": 4046, "value": " " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "docker images" } ], "url": "https://github.com/nascheme/cpython_sanity", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " might also be a quicker choice that bypasses building Python from source, although it may be annoying to do debugging work inside of a docker image." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Build Python with Sanitizer Instrumentation" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "However you build Python, once you have an instrumented Python build, you can install NumPy's development and test dependencies and build NumPy with address sanitizer instrumentation. For example, to build NumPy with the " }, { "__type": "InlineCode", "__tag": 4051, "value": "debug" }, { "__type": "Text", "__tag": 4046, "value": " profile and address sanitizer, you would pass additional build options to " }, { "__type": "InlineCode", "__tag": 4051, "value": "meson" }, { "__type": "Text", "__tag": 4046, "value": " like this:" } ] }, { "__type": "Code", "__tag": 4050, "value": "spin build -- -Dbuildtype=debug -Db_sanitize=address", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Once the build is finished, you can use other " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" }, { "__type": "Text", "__tag": 4046, "value": " command like " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin test" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin gdb" }, { "__type": "Text", "__tag": 4046, "value": " as with any other Python build." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Use together with " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" } ], "level": 3, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Some NumPy tests intentionally lead to " }, { "__type": "InlineCode", "__tag": 4051, "value": "malloc" }, { "__type": "Text", "__tag": 4046, "value": " returning " }, { "__type": "InlineCode", "__tag": 4051, "value": "NULL" }, { "__type": "Text", "__tag": 4046, "value": ". In its default configuration, some of the compiler sanitizers flag this as an error. You can disable that check by passing " }, { "__type": "InlineCode", "__tag": 4051, "value": "allocator_may_return_null=1" }, { "__type": "Text", "__tag": 4046, "value": " to the sanitizer as an option. For example, with address sanitizer:" } ] }, { "__type": "Code", "__tag": 4050, "value": "ASAN_OPTIONS=allocator_may_return_null=1 spin test", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You may see memory leaks coming from the Python interpreter, particularly on MacOS. If the memory leak reports are not useful, you can disable leak detection by passing " }, { "__type": "InlineCode", "__tag": 4051, "value": "detect_leaks=0" }, { "__type": "Text", "__tag": 4046, "value": " in " }, { "__type": "InlineCode", "__tag": 4051, "value": "ASAN_OPTIONS" }, { "__type": "Text", "__tag": 4046, "value": ". You can pass more than one option using a colon-delimited list, like this:" } ] }, { "__type": "Code", "__tag": 4050, "value": "ASAN_OPTIONS=allocator_may_return_null=1:halt_on_error=1:detect_leaks=1 spin test", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The " }, { "__type": "InlineCode", "__tag": 4051, "value": "halt_on_error" }, { "__type": "Text", "__tag": 4046, "value": " option can be particularly useful -- it hard-crashes the Python executable whenever it detects an error, along with a report about the error that includes a stack trace." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can also take a look at the " }, { "__type": "InlineCode", "__tag": 4051, "value": "compiler_sanitizers.yml" }, { "__type": "Text", "__tag": 4046, "value": " GitHub actions workflow configuration. It describes several different CI jobs that are run as part of the NumPy tests using Thread, Address, and Undefined Behavior sanitizer." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Special considerations" } ], "level": 3, "target": null } ], "local_refs": [] }