{ "__type": "IngestedDoc", "__tag": 4010, "_content": {}, "_ordered_sections": [], "item_file": null, "item_line": null, "item_type": null, "aliases": [], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "see_also": [], "signature": null, "references": null, "qa": "building:redistributable_binaries", "arbitrary": [ { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "The intended audience for this section is anyone who wants to build SciPy and\ndeploy it anywhere else than their own machine - from distro packagers to users\nwho want to build wheels to deploy to their production environment" } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "When " }, { "__type": "InlineCode", "__tag": 4051, "value": "python -m build" }, { "__type": "Text", "__tag": 4046, "value": " or " }, { "__type": "InlineCode", "__tag": 4051, "value": "pip wheel" }, { "__type": "Text", "__tag": 4046, "value": " is used to build a SciPy wheel, that wheel will rely on external shared libraries (at least for BLAS/LAPACK and a Fortran compiler runtime library, perhaps other libraries). Such wheels therefore will only run on the system on which they are built. See " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "the pypackaging-native content under \"Building and installing or uploading artifacts\"" } ], "url": "https://pypackaging-native.github.io/meta-topics/build_steps_conceptual/#building-and-installing-or-uploading-artifacts", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " for more context on that." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A wheel like that is therefore an intermediate stage to producing a binary that can be distributed. That final binary may be a wheel - in that case, run " }, { "__type": "InlineCode", "__tag": 4051, "value": "auditwheel" }, { "__type": "Text", "__tag": 4046, "value": " (Linux), " }, { "__type": "InlineCode", "__tag": 4051, "value": "delocate" }, { "__type": "Text", "__tag": 4046, "value": " (macOS), " }, { "__type": "InlineCode", "__tag": 4051, "value": "delvewheel" }, { "__type": "Text", "__tag": 4046, "value": " (Windows) or " }, { "__type": "InlineCode", "__tag": 4051, "value": "repairwheel" }, { "__type": "Text", "__tag": 4046, "value": " (platform-independent) to vendor the required shared libraries into the wheel." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The final binary may also be in another packaging format (e.g., a " }, { "__type": "InlineCode", "__tag": 4051, "value": ".rpm" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "InlineCode", "__tag": 4051, "value": ".deb" }, { "__type": "Text", "__tag": 4046, "value": " or " }, { "__type": "InlineCode", "__tag": 4051, "value": ".conda" }, { "__type": "Text", "__tag": 4046, "value": " package). In that case, there are packaging ecosystem-specific tools to first install the wheel into a staging area, then making the extension modules in that install location relocatable (e.g., by rewriting RPATHs), and then repackaging it into the final package format." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Building redistributable binaries" } ], "level": 0, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The Python build and runtime dependencies that are needed to build SciPy can be found in the " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyproject.toml" }, { "__type": "Text", "__tag": 4046, "value": " metadata. Note that for released versions of SciPy, dependencies will likely have upper bounds. Each upper bound has comments above it; packagers are free to remove or loosen those upper bound in most cases (except for " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy" }, { "__type": "Text", "__tag": 4046, "value": "). E.g. " } ] }, { "__type": "Code", "__tag": 4050, "value": "# The upper bound on pybind11 is preemptive only\n\"pybind11>=2.12.0,<2.13.0\",\n\n# ...\n# 3. The <2.3 upper bound is for matching the numpy deprecation policy,\n# it should not be loosened.\n\"numpy>=2.0.0rc1,<2.3\",", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Non-Python build requirements are:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "C, C++ and Fortran compilers" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "BLAS and LAPACK libraries" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "ninja" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "pkg-config" } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Minimum versions of common compilers are enforced in the top-level " }, { "__type": "InlineCode", "__tag": 4051, "value": "meson.build" }, { "__type": "Text", "__tag": 4046, "value": " file. The minimum LAPACK version is currently 3.7.1. More detailed information on these build dependencies can be found in " }, { "__type": "CrossRef", "__tag": 4002, "value": "toolchain-roadmap", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "docs", "path": "dev:toolchain" }, "kind": "exists" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Build and runtime dependencies" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "SciPy contains a " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "lot" } ] }, { "__type": "Text", "__tag": 4046, "value": " of code that has been vendored from libraries written in C, C++ and Fortran. This is done for a mix of historical and robustness reasons. Distro packagers sometimes have reasons to want to " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "unvendor" } ] }, { "__type": "Text", "__tag": 4046, "value": " this code, ensuring that they only have a single version of a particular library in use. We offer a build option, " }, { "__type": "InlineCode", "__tag": 4051, "value": "-Duse-system-libraries" }, { "__type": "Text", "__tag": 4046, "value": ", to allow them to do so in a much easier fashion than can be done by manually patching SciPy." } ] }, { "__type": "Code", "__tag": 4050, "value": "$ python -m build -wnx -Duse-system-libraries=boost.math", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This build option causes a dependency lookup " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "for the exact same version" } ] }, { "__type": "Text", "__tag": 4046, "value": " as is bundled in SciPy only. A different system version will not be accepted, the build will then fall back to the vendored sources or error out. This is because different versions can easily cause failures at build time or at runtime, and SciPy's CI does not test with any other versions. This may change in the future, however as of now, using a different version requires patching out the relevant version check and is strongly discouraged." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Advice for packagers:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": true, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You should think about this build option as \"if I'm already unbundling a library for some compelling reason, this option makes that quite a bit easier\"." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "You should not default to unbundling everything, this will cause diamond dependencies and more build complexity for little gain. Most of these libraries are not maintained as well as SciPy and/or known to not offer a stable API; effectively you'd be adding an extra " }, { "__type": "InlineCode", "__tag": 4051, "value": "==" }, { "__type": "Text", "__tag": 4046, "value": " dependency for each library you unbundle. Only do that for a good reason. E.g., it allows you to drop patches, you consider both SciPy and the vendored library in question security-critical, or it's a large gain in binary size." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The build option takes the following values:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "none" }, { "__type": "Text", "__tag": 4046, "value": ": Use the source code shipped as part of SciPy, do not look for external dependencies (default)." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "all" }, { "__type": "Text", "__tag": 4046, "value": ": Use external libraries for all components controlled by " }, { "__type": "InlineCode", "__tag": 4051, "value": "use-system-libraries" }, { "__type": "Text", "__tag": 4046, "value": ", and error out if they are not found. Detecting is done by Meson, typically first through the " }, { "__type": "InlineCode", "__tag": 4051, "value": "pkg-config" }, { "__type": "Text", "__tag": 4046, "value": " and then the " }, { "__type": "InlineCode", "__tag": 4051, "value": "cmake" }, { "__type": "Text", "__tag": 4046, "value": " method of the " }, { "__type": "InlineCode", "__tag": 4051, "value": "dependency()" }, { "__type": "Text", "__tag": 4046, "value": " function." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "auto" }, { "__type": "Text", "__tag": 4046, "value": ": Try detecting external libraries, and if they are not found then fall back onto vendored sources." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A comma-separated list of dependency names (e.g., " }, { "__type": "InlineCode", "__tag": 4051, "value": "boost.math,qhull" }, { "__type": "Text", "__tag": 4046, "value": "). If given, uses the " }, { "__type": "InlineCode", "__tag": 4051, "value": "all" }, { "__type": "Text", "__tag": 4046, "value": " behavior for the named dependencies (and " }, { "__type": "InlineCode", "__tag": 4051, "value": "none" }, { "__type": "Text", "__tag": 4046, "value": " for the rest). Supported options are:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "boost.math" }, { "__type": "Text", "__tag": 4046, "value": " (since 1.16.0)" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "qhull" }, { "__type": "Text", "__tag": 4046, "value": " (since 1.16.0)" } ] } ] } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "It's also valid to combine the generic and named-library options above, for example " }, { "__type": "InlineCode", "__tag": 4051, "value": "boost.math,auto" }, { "__type": "Text", "__tag": 4046, "value": " will apply the " }, { "__type": "InlineCode", "__tag": 4051, "value": "auto" }, { "__type": "Text", "__tag": 4046, "value": " behavior to all dependencies other than " }, { "__type": "InlineCode", "__tag": 4051, "value": "boost.math" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Using system dependencies instead of vendored sources" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "By default, an installed version of " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy" }, { "__type": "Text", "__tag": 4046, "value": " includes the full test suite. That test suite, including data files and compiled extension modules that are test-only, takes up about 4.5 MB in a wheel (for x86-64, as of v1.14.0), and more than that on disk. In cases where binary size matters, packagers may want to remove the test suite. As of SciPy 1.14.0, there is a convenient way of doing this, making use of " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "Meson's install tags" } ], "url": "https://mesonbuild.com/Installing.html#installation-tags", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " functionality. It is a one-liner " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ python -m build -wnx -Cinstall-args=--tags=runtime,python-runtime,devel", "execution_status": null }, { "__type": "Admonition", "__tag": 4056, "kind": "note", "base_type": "note", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "note " } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Note that in the above command " }, { "__type": "InlineCode", "__tag": 4051, "value": "-wnx" }, { "__type": "Text", "__tag": 4046, "value": " means " }, { "__type": "InlineCode", "__tag": 4051, "value": "--wheel --no-isolation --skip-dependency-check" }, { "__type": "Text", "__tag": 4046, "value": ". It assumes that the packager has already set up the build environment, which is usually the case for distro packaging. The install tags feature works equally well with isolated builds (e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "pip install scipy --no-binary -Cinstall-args=--tags=runtime,python-runtime,devel" }, { "__type": "Text", "__tag": 4046, "value": ")." } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If you want to produce a separate package for the tests themselves, say under the name " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy-tests" }, { "__type": "Text", "__tag": 4046, "value": ", then edit " }, { "__type": "InlineCode", "__tag": 4051, "value": "pyproject.toml" }, { "__type": "Text", "__tag": 4046, "value": " to change the project name:" } ] }, { "__type": "Code", "__tag": 4050, "value": "[project]\nname = \"scipy-tests\"", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "And then build with " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ python -m build -wnx -Cinstall-args=--tags=tests", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The above would build the whole package twice; in order to rebuild in a cached fashion, use the " }, { "__type": "InlineCode", "__tag": 4051, "value": "-Cbuild-dir=build" }, { "__type": "Text", "__tag": 4046, "value": " build option " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ $ # apply patch to change the project name in pyproject.toml\n$ python -m build -wnx -Cbuild-dir=build -Cinstall-args=--tags=tests", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The end result will look something like " } ] }, { "__type": "Code", "__tag": 4050, "value": "$ ls -lh dist/*.whl\n... 20M ... dist/scipy-1.14.0-cp311-cp311-linux_x86_64.whl\n... 4,5M ... dist/scipy_tests-1.14.0-cp311-cp311-linux_x86_64.whl", "execution_status": null } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Stripping the test suite from a wheel or installed package" } ], "level": 1, "target": null } ], "local_refs": [] }