{ "__type": "IngestedDoc", "__tag": 4010, "_content": {}, "_ordered_sections": [], "item_file": null, "item_line": null, "item_type": null, "aliases": [], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "see_also": [], "signature": null, "references": null, "qa": "dev:contributor:debugging_linalg_issues", "arbitrary": [ { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Linear algebra related bug reports are among the most challenging issues to diagnose and address. This is not only because linear algebra can be challenging mathematically/algorithmically (that is true for many parts of SciPy), but because BLAS/LAPACK libraries are a complex build-time as well as runtime dependency - and we support a significant number of BLAS/LAPACK libraries." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This document aims to provide guidance about how to go about debugging linear algebra issues." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If there is a real bug, it can be in one of three places:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "In the BLAS library being used," } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "In SciPy's bindings for BLAS or LAPACK (generated by " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy.f2py" }, { "__type": "Text", "__tag": 4046, "value": " and/or Cython)," } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "In SciPy's algorithmic code." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A key first step is to determine whether the bug is in SciPy or in the BLAS library. To do so, the most efficient way to disambiguate the two is to set up your environment in such a way that you can achieve runtime switching between different BLAS libraries (something we don't support out of the box, and isn't possible with SciPy's wheels from PyPI)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Upstream BLAS library authors strongly prefer to get clean reproducers (just like we do), which means: no Python involved. So this guide will also cover how to create reproducers in C or Fortran." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Debugging linear algebra related issues" } ], "level": 0, "target": "debugging-linalg-issues" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "SciPy has one function, " }, { "__type": "CrossRef", "__tag": 4002, "value": "show_config", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "*", "kind": "api", "path": "scipy.__config__:show" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ", to introspect the build configuration of an installed package. This allows querying for details of BLAS and LAPACK. E.g. " } ] }, { "__type": "Code", "__tag": 4050, "value": ">>> blas_dep = scipy.show_config(mode='dicts')['Build Dependencies']['blas']\n>>> for key in blas_dep:\n... print(f\"{key}: {blas_dep[key]}\")\n...\nname: openblas\nfound: True\nversion: 0.3.23\ndetection method: pkgconfig\ninclude directory: /home/user/miniforge/envs/scipy-dev/include\nlib directory: /home/user/miniforge/envs/scipy-dev/lib\nopenblas configuration: USE_64BITINT=0 DYNAMIC_ARCH=1 DYNAMIC_OLDER= NO_CBLAS= NO_LAPACK=0 NO_LAPACKE= NO_AFFINITY=1 USE_OPENMP=0 PRESCOTT MAX_THREADS=128\npc file directory: /home/user/miniforge/envs/scipy-dev/lib/pkgconfig", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This method will be correct for SciPy wheels and for local dev builds. It " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "may" } ] }, { "__type": "Text", "__tag": 4046, "value": " be correct for other installs, however keep in mind that distros like conda-forge and Debian may be building against stub libraries (typically " }, { "__type": "InlineCode", "__tag": 4051, "value": "blas.so" }, { "__type": "Text", "__tag": 4046, "value": "/" }, { "__type": "InlineCode", "__tag": 4051, "value": "lapack.so" }, { "__type": "Text", "__tag": 4046, "value": ") and then installing another library for the user - in such cases, plain " }, { "__type": "InlineCode", "__tag": 4051, "value": "blas" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "lapack" }, { "__type": "Text", "__tag": 4046, "value": " will be reported even if for example OpenBLAS or MKL is installed in the environment. For such installs, " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "threadpoolctl" } ], "url": "https://github.com/joblib/threadpoolctl", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " will usually be able to report what the actual BLAS library in use is (except it doesn't report on plain Netlib BLAS, see " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "threadpoolctl#159" } ], "url": "https://github.com/joblib/threadpoolctl/issues/159", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ") " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ python -m threadpoolctl -i scipy.linalg\n[\n {\n \"user_api\": \"blas\",\n \"internal_api\": \"openblas\",\n \"prefix\": \"libopenblas\",\n \"filepath\": \"/home/user/miniforge/envs/dev/lib/libopenblasp-r0.3.21.so\",\n \"version\": \"0.3.21\",\n \"threading_layer\": \"pthreads\",\n \"architecture\": \"SkylakeX\",\n \"num_threads\": 24\n }\n]", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Other ways of introspecting that can be helpful in local dev environments include:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": true, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Checking dependencies of shared libraries:" } ] } ] } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": true, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Checking whether the linked library contains a given symbol. E.g., conda-forge installs a " }, { "__type": "InlineCode", "__tag": 4051, "value": "libblas.so" }, { "__type": "Text", "__tag": 4046, "value": " that may be any supported library:" } ] } ] } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Finding the BLAS library being used" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "We'll cover several ways of switching between different BLAS libraries, because the easiest method may depend on your OS/distro and on whether you want a released version of SciPy or a development build." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Setting up your environment for switching BLAS libraries" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Perhaps the easiest way is to use the runtime switching abilities provided by distros. For example, to create a conda environment with the latest SciPy release installed and then switching between OpenBLAS, Netlib BLAS/LAPACK, and MKL is as simple as " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ mamba create -n blas-switch scipy threadpoolctl\n$ mamba activate blas-switch\n$ python -m threadpoolctl -i scipy.linalg\n...\n \"user_api\": \"blas\",\n \"internal_api\": \"openblas\",\n\n$ mamba install \"libblas=*=*netlib\"\n...\n libblas 3.9.0-21_linux64_openblas --> 3.9.0-5_h92ddd45_netlib\n libcblas 3.9.0-21_linux64_openblas --> 3.9.0-5_h92ddd45_netlib\n liblapack 3.9.0-21_linux64_openblas --> 3.9.0-5_h92ddd45_netlib\n\n$ mamba install \"libblas=*=*mkl\"\n...\n libblas 3.9.0-5_h92ddd45_netlib --> 3.9.0-21_linux64_mkl\n libcblas 3.9.0-5_h92ddd45_netlib --> 3.9.0-21_linux64_mkl\n liblapack 3.9.0-5_h92ddd45_netlib --> 3.9.0-21_linux64_mkl\n\n$ python -m threadpoolctl -i scipy.linalg\n...\n \"user_api\": \"blas\",\n \"internal_api\": \"mkl\",", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This can be done for development builds as well, when building via " }, { "__type": "InlineCode", "__tag": 4051, "value": "spin" }, { "__type": "Text", "__tag": 4046, "value": " in the exact same way as in " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "SciPy's conda-forge build recipe" } ], "url": "https://github.com/conda-forge/scipy-feedstock/blob/main/recipe/build.sh", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " (outputs omitted for brevity, they're similar to the ones above) " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ mamba env create -f environment.yml\n$ mamba activate scipy-dev\n$ mamba install \"libblas=*=*netlib\" # necessary, we need to build against blas/lapack\n$ spin build -S-Dblas=blas -S-Dlapack=lapack -S-Duse-g77-abi=true\n$ spin test -s linalg # run tests to verify\n$ mamba install \"libblas=*=*mkl\"\n$ spin test -s linalg\n$ mamba install \"libblas=*=*openblas\"", "execution_status": null } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Conda-forge" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A number of Linux distros use the " }, { "__type": "InlineCode", "__tag": 4051, "value": "update-alternatives" }, { "__type": "Text", "__tag": 4046, "value": " mechanism to allow switching between different BLAS libraries via the system package manager. Note that this is a generic mechanism to manage \"multiple implementations of the same library or tool\" situations, rather than something specific to BLAS/LAPACK. It's similar to the conda-forge method above, in that it works for distro-provided " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy" }, { "__type": "Text", "__tag": 4046, "value": " packages as well as for development builds against the reference " }, { "__type": "InlineCode", "__tag": 4051, "value": "libblas" }, { "__type": "Text", "__tag": 4046, "value": "/" }, { "__type": "InlineCode", "__tag": 4051, "value": "liblapack" }, { "__type": "Text", "__tag": 4046, "value": " interfaces." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The interface looks like " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ update-alternatives --config libblas.so.3\n$ update-alternatives --config liblapack.so.3", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "which will open a menu in the terminal with all available libraries to choose from. Because the interface and available options are likely to vary across distros, we link here to " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "the Debian documentation for BLAS/LAPACK switching" } ], "url": "https://wiki.debian.org/DebianScience/LinearAlgebraLibraries", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " and avoid documenting in more detail how this works on other distros." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that Fedora is an exception; it is the only distro that ships FlexiBLAS (see the next section for more on that) and allows installing multiple BLAS libraries in parallel so true runtime switching without having to invoke the system package manager becomes possible. See " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "the Fedora docs on system-level and user-level selection" } ], "url": "https://docs.fedoraproject.org/en-US/packaging-guidelines/BLAS_LAPACK/#_backend_selection", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " for more details." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Linux distro package managers" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "FlexiBLAS" } ], "url": "https://github.com/mpimd-csc/flexiblas", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " provides runtime switching support (among other things) for all installed BLAS libraries that it can detect. There are a few limitations at the time of writing (March 2024), primarily: no support for Windows, no support for macOS Accelerate (the updated version, with " }, { "__type": "InlineCode", "__tag": 4051, "value": "NEWLAPACK" }, { "__type": "Text", "__tag": 4046, "value": " symbols). If those limitations don't matter for you, FlexiBLAS can be a quite useful tool for efficient debugging, including for versions of OpenBLAS and other BLAS libraries that you have to build from source." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Once you have everything set up, the development experience is " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ spin build -S-Dblas=flexiblas -S-Dlapack=flexiblas\n$ FLEXIBLAS=NETLIB spin test -s linalg\n$ FLEXIBLAS=OpenBLAS spin test -s linalg\n# Or export the environment variable to make the selection stick:\n$ export FLEXIBLAS=OpenBLAS", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You can also provide a path to a built BLAS library (e.g., " }, { "__type": "InlineCode", "__tag": 4051, "value": "FLEXIBLAS=\"libbhlas_atlas.so\"" }, { "__type": "Text", "__tag": 4046, "value": ") - see the " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "usage docs in its README" } ], "url": "https://github.com/mpimd-csc/flexiblas#selecting-the-backend-at-runtime", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " for more details." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Unless you're on Fedora, you will likely have to build FlexiBLAS from source, which is a bit of work. The good news is that this should work no matter if you're on Linux or macOS, and use Python via virtualenvs, conda environments, or in some other way. We'll go through how to build OpenBLAS and FlexiBLAS from source, to allow debugging whether something in the latest OpenBLAS version is different from Netlib BLAS/LAPACK (or MKL) or not." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The below should work in any environment where you can build SciPy itself; the only additional tool we need is CMake (install with, for example, " }, { "__type": "InlineCode", "__tag": 4051, "value": "pip install cmake" }, { "__type": "Text", "__tag": 4046, "value": ")." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Clone each repository " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ cd .. # starting from the root of the local `scipy` repo\n$ mkdir flexiblas-setup && cd flexiblas-setup\n$ git clone https://github.com/OpenMathLib/OpenBLAS.git openblas\n$ git clone https://github.com/mpimd-csc/flexiblas.git\n$ mkdir built-libs # our local prefix to install everything to", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Build OpenBLAS " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ cd openblas\n$ mkdir build && cd build\n$ cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_INSTALL_PREFIX=$PWD/../../built-libs\n$ cmake --build . -j\n$ cmake --install . --prefix $PWD/../../built-libs\n$ cd ../..", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Build FlexiBLAS " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ cd flexiblas\n$ mkdir build && cd build\n$ # Note: this will also pick up the libraries in your system/env libdir\n$ cmake .. -DEXTRA=\"OpenBLAS\" -DLAPACK_API_VERSION=3.9.0 \\\n -DOpenBLAS_LIBRARY=$PWD/../../built-libs/lib/libopenblas.so \\\n -DCMAKE_INSTALL_PREFIX=$PWD/../../built-libs\n$ cmake --build . -j\n$ cmake --install . --prefix $PWD/../../built-libs\n$ cd ../..", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "We're now ready to build SciPy against FlexiBLAS " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ export PKG_CONFIG_PATH=$PWD/flexiblas-setup/built-libs/lib/pkgconfig/\n$ cd scipy\n$ spin build -S-Dblas=flexiblas -S-Dlapack=flexiblas\n...\nRun-time dependency flexiblas found: YES 3.4.2", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Now we can run the tests. Note that the " }, { "__type": "InlineCode", "__tag": 4051, "value": "NETLIB" }, { "__type": "Text", "__tag": 4046, "value": " option is built without having to specify it; it's the default in FlexiBLAS and sources are included in its repository " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ FLEXIBLAS=OpenBLAS spin test -s linalg\n$ FLEXIBLAS=NETLIB spin test -s linalg\n$ spin test -s linalg # uses the default (NETLIB)", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This backend switching can also be done inside a Python interpreter with " }, { "__type": "InlineCode", "__tag": 4051, "value": "threadpoolctl" }, { "__type": "Text", "__tag": 4046, "value": " (see " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "its README" } ], "url": "https://github.com/joblib/threadpoolctl#switching-the-flexiblas-backend", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " for details)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Other libraries available on the system can be inspected with " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ ./flexiblas-setup/built-libs/bin/flexiblas list", "execution_status": null }, { "__type": "Admonition", "__tag": 4056, "kind": "note", "base_type": "note", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "note " } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Using local builds of reference BLAS/LAPACK or BLIS is more difficult, because FlexiBLAS requires a single shared library which contains all needed symbols. It " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "may be feasible" } ], "url": "https://github.com/mpimd-csc/flexiblas#setup-with-precompiled-reference-blas-and-lapack", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " to use a separate " }, { "__type": "InlineCode", "__tag": 4051, "value": "libblas" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "liblapack" }, { "__type": "Text", "__tag": 4046, "value": " as the \"system library\", but this has proven to be more fragile and difficult to build (so this is YMMV). In case you do want to try:" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Build reference BLAS and LAPACK:" } ] }, { "__type": "Blockquote", "__tag": 4059, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "$ git clone https://github.com/Reference-LAPACK/lapack.git $ cd lapack $ mkdir build && cd build $ cmake -DCBLAS=ON -DBUILD_SHARED_LIBS=OFF .. $ cmake --build . -j $ cmake --install . --prefix $PWD/../../built-libs" } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Then add the following two lines to the " }, { "__type": "InlineCode", "__tag": 4051, "value": "cmake .." }, { "__type": "Text", "__tag": 4046, "value": " configure command for FlexiBLAS " } ] }, { "__type": "Code", "__tag": 4050, "value": "-DSYS_BLAS_LIBRARY=$PWD/../../built-libs/lib/libblas.a \\\n-DSYS_LAPACK_LIBRARY=$PWD/../../built-libs/lib/liblapack.a \\", "execution_status": null } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "FlexiBLAS" } ], "level": 2, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Our experience tells us that a large majority of bugs are inside SciPy rather than in OpenBLAS or another BLAS library. If the testing with different BLAS libraries tells us though that the problem is specific to a single BLAS library (maybe even a single version of that library with a regression), the next step is to produce a reproducer in C or Fortran; doing so is necessary for reporting the bug upstream, and makes it much easier for the BLAS library developers to address the problem." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "To get from a Python reproducer which uses a " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy" }, { "__type": "Text", "__tag": 4046, "value": " function with NumPy arrays as input to a C/Fortran reproducer, it is necessary to find the code path taken in SciPy and determine which exact BLAS or LAPACK function is called, and with what inputs (note: the answer may be contained in the " }, { "__type": "InlineCode", "__tag": 4051, "value": ".pyf.src" }, { "__type": "Text", "__tag": 4046, "value": " f2py signature files; looking into the generated " }, { "__type": "InlineCode", "__tag": 4051, "value": "_flapackmodule.c" }, { "__type": "Text", "__tag": 4046, "value": " in the build directory may be useful too). This can then be reproduced in C/Fortran by defining some integer/float variables and arrays (typically small arrays with hardcoded numbers are enough)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Argument lists of BLAS and LAPACK functions can be looked up in for example " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "the Netlib LAPACK docs" } ], "url": "https://www.netlib.org/lapack/explore-html/", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " or the " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "Reference-LAPACK/lapack repository" } ], "url": "https://github.com/Reference-LAPACK/lapack", "title": "" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Below a reproducer is shown for an issue in reference LAPACK, which was reported as a SciPy issue in " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "scipy#11577" } ], "url": "https://github.com/scipy/scipy/issues/11577", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ". We'll name the file " }, { "__type": "InlineCode", "__tag": 4051, "value": "ggev_repro_gh_11577.c|f90" }, { "__type": "Text", "__tag": 4046, "value": ":" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Now we need to compile this reproducer locally and run it. If we're invoking a compiler directly, we need to add the needed compile and link flags by hand. The include path will depend on your local install, and the link flags will depend on which library you're testing. For example, to test against a local build of OpenBLAS:" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For reference BLAS/LAPACK, the " }, { "__type": "InlineCode", "__tag": 4051, "value": "-lopenblas" }, { "__type": "Text", "__tag": 4046, "value": " should be replaced with " }, { "__type": "InlineCode", "__tag": 4051, "value": "-lblas -llapack" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that the explicit paths are only needed for libraries in non-standard locations (like the ones we built in this guide). For building against a package manager-installed library for which the shared library and headers are on the normal compiler search path (e.g., in " }, { "__type": "InlineCode", "__tag": 4051, "value": "/usr/lib" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "/usr/include" }, { "__type": "Text", "__tag": 4046, "value": ", or inside a conda env when using compilers from the same env), they can be left out:" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Alternatively (and probably a more robust way), use a small " }, { "__type": "InlineCode", "__tag": 4051, "value": "meson.build" }, { "__type": "Text", "__tag": 4046, "value": " file to automate this and avoid the manual paths:" } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "warning", "base_type": "warning", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "warning " } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When you have multiple versions/builds of the same BLAS library on your machine, it's easy to accidentally pick up the wrong one during the build (remember: " }, { "__type": "InlineCode", "__tag": 4051, "value": "-lopenblas" }, { "__type": "Text", "__tag": 4046, "value": " only says \"link against " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "some" } ] }, { "__type": "Text", "__tag": 4046, "value": " " }, { "__type": "InlineCode", "__tag": 4051, "value": "libopenblas.so" }, { "__type": "Text", "__tag": 4046, "value": "). If you're not sure, use " }, { "__type": "InlineCode", "__tag": 4051, "value": "ldd" }, { "__type": "Text", "__tag": 4046, "value": " on the test executable you built to inspect which shared library it's linked against." } ] } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Creating reproducers in C or Fortran" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When debugging linalg issues, it is sometimes useful to step through both Python and C code. You can use " }, { "__type": "InlineCode", "__tag": 4051, "value": "pdb" }, { "__type": "Text", "__tag": 4046, "value": " for the former, and " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " for the latter." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "First, prepare a small python reproducer, with a breakpoint. For example " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ cat chol.py\nimport numpy as np\nfrom scipy import linalg\nn = 40\nrng = np.random.default_rng(1234)\nx = rng.uniform(size=n)\na = x[:, None] @ x[None, :] + np.identity(n)\n\nbreakpoint() # note a breakpoint\nlinalg.cholesky(a)", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Then, you will need to run it under the " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " and add a C-level breakpoint at the LAPACK function. This way, your execution will stop twice: first on the Python breakpoint and then on the C breakpoint, and you will be able to step through and inspect values of both Python and C variables." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "To find out the LAPACK name, read the python source of the SciPy function and use " }, { "__type": "InlineCode", "__tag": 4051, "value": "nm" }, { "__type": "Text", "__tag": 4046, "value": " on the " }, { "__type": "InlineCode", "__tag": 4051, "value": ".so" }, { "__type": "Text", "__tag": 4046, "value": " library to find out the exact name. For the Cholesky factorization above, the LAPACK function is " }, { "__type": "InlineCode", "__tag": 4051, "value": "?potrf" }, { "__type": "Text", "__tag": 4046, "value": ", and the C name on Ubuntu linux is " }, { "__type": "InlineCode", "__tag": 4051, "value": "dpotrf_" }, { "__type": "Text", "__tag": 4046, "value": " (it may be spelled with or without the trailing underscore, in uppper case or lower case, depending on the system)." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Here is an example " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" }, { "__type": "Text", "__tag": 4046, "value": " session " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ gdb --args python chol.py\n...\n(gdb) b dpotrf_ # this adds a C breakpoint (type \"y\" below)\nFunction \"dpotrf_\" not defined.\nMake breakpoint pending on future shared library load? (y or [n]) y\nBreakpoint 1 (dpotrf_) pending.\n(gdb) run # run the python script\n...\n> /home/br/temp/chol/chol.py(12)()\n-> linalg.cholesky(a) # execution stopped at the python breakpoint\n(Pdb) p a.shape # ... inspect the python state here\n(40, 40)\n(Pdb) c # continue execution until the C breakpoint\n\nThread 1 \"python\" hit Breakpoint 1, 0x00007ffff4c48820 in dpotrf_ ()\n from /home/br/miniforge/envs/scipy-dev/lib/python3.10/site-packages/numpy/core/../../../../libcblas.so.3\n(gdb) s # step through the C function\nSingle stepping until exit from function dpotrf_,\nwhich has no line number information.\nf2py_rout__flapack_dpotrf (capi_self=, capi_args=,\n capi_keywds=, f2py_func=0x7ffff4c48820 )\n at scipy/linalg/_flapackmodule.c:63281\n....\n(gdb) p lda # inspect values of C variables\n$1 = 40\n\n# print out the C backtrace\n(gdb) bt\n#0 0x00007ffff3056b1e in f2py_rout.flapack_dpotrf ()\n from /path/to/site-packages/scipy/linalg/_flapack.cpython-311-x86_64-linux-gnu.so\n#1 0x0000555555734323 in _PyObject_MakeTpCall (tstate=0x555555ad0558 <_PyRuntime+166328>,\n callable=, args=, nargs=1,\n keywords=('lower', 'overwrite_a', 'clean'))\n at /usr/local/src/conda/python-3.11.8/Objects/call.c:214\n...", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Depending on your system, you may need to build SciPy with debug build type, and unset CFLAGS/CXXFLAGS environment variables. See the " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "NumPy debugging guide" } ], "url": "https://numpy.org/devdocs/dev/development_environment.html#debugging", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " for more details." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Debugging linalg issues with " }, { "__type": "InlineCode", "__tag": 4051, "value": "gdb" } ], "level": 1, "target": null } ], "local_refs": [] }