{ "__type": "IngestedDoc", "__tag": 4010, "_content": {}, "_ordered_sections": [], "item_file": null, "item_line": null, "item_type": null, "aliases": [], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "see_also": [], "signature": null, "references": null, "qa": "dev:roadmap", "arbitrary": [ { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "This roadmap page contains only the most important ideas and needs for SciPy going forward. For a more detailed roadmap, including per-subpackage status, many more ideas, API stability and more, see " }, { "__type": "CrossRef", "__tag": 4002, "value": "scipy-roadmap-detailed", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "docs", "path": "dev:roadmap-detailed" }, "kind": "exists" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "SciPy Roadmap" } ], "level": 0, "target": "scipy-roadmap" }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "NumPy has split its API from its execution engine with " }, { "__type": "InlineCode", "__tag": 4051, "value": "__array_function__" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "__array_ufunc__" }, { "__type": "Text", "__tag": 4046, "value": ". This will enable parts of SciPy to accept distributed arrays (e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "dask.array.Array" }, { "__type": "Text", "__tag": 4046, "value": ") and GPU arrays (e.g. " }, { "__type": "InlineCode", "__tag": 4051, "value": "cupy.ndarray" }, { "__type": "Text", "__tag": 4046, "value": ") that implement the " }, { "__type": "InlineCode", "__tag": 4051, "value": "ndarray" }, { "__type": "Text", "__tag": 4046, "value": " interface. At the moment it is not yet clear which algorithms will work out of the box, and if there are significant performance gains when they do. We want to create a map of which parts of the SciPy API work, and improve support over time." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "In addition to making use of NumPy protocols like " }, { "__type": "InlineCode", "__tag": 4051, "value": "__array_function__" }, { "__type": "Text", "__tag": 4046, "value": ", we can make use of these protocols in SciPy as well. That will make it possible to (re)implement SciPy functions like, e.g., those in " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy.signal" }, { "__type": "Text", "__tag": 4046, "value": " for Dask or GPU arrays (see " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "NEP 18 - use outside of NumPy" } ], "url": "http://www.numpy.org/neps/nep-0018-array-function-protocol.html#use-outside-of-numpy", "title": "" }, { "__type": "Text", "__tag": 4046, "value": "). NumPy's features in this areas are still evolving, see e.g. " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "NEP 37 - A dispatch protocol for NumPy-like modules" } ], "url": "https://numpy.org/neps/nep-0037-array-module.html", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ", and SciPy is an important \"client\" for those features." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Support for distributed arrays and GPU arrays" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Speed improvements, lower memory usage and the ability to parallelize algorithms are beneficial to most science domains and use cases. We have established an API design pattern for multiprocessing - using the " }, { "__type": "InlineCode", "__tag": 4051, "value": "workers" }, { "__type": "Text", "__tag": 4046, "value": " keyword - that can be adopted in many more functions." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Making it easier for users to use Numba's " }, { "__type": "InlineCode", "__tag": 4051, "value": "@njit" }, { "__type": "Text", "__tag": 4046, "value": " in their code that relies on SciPy functionality would unlock a lot of performance gain. That needs a strategy though, e.g. based on " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "Transonic" } ], "url": "https://transonic.readthedocs.io/", "title": "" }, { "__type": "Text", "__tag": 4046, "value": " or custom code that supports multiple just-in-time and ahead-of-time compilers (as is explored in " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "gh-23447" } ], "url": "https://github.com/scipy/scipy/pull/23447", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ")." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Finally, many individual functions can be optimized for performance. " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy.optimize" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "InlineCode", "__tag": 4051, "value": "scipy.interpolate" }, { "__type": "Text", "__tag": 4046, "value": " functions are particularly often requested in this respect." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Performance improvements" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "SciPy now has continuous integration for ARM64 (or " }, { "__type": "InlineCode", "__tag": 4051, "value": "aarch64" }, { "__type": "Text", "__tag": 4046, "value": ") and POWER8/9 (or " }, { "__type": "InlineCode", "__tag": 4051, "value": "ppc64le" }, { "__type": "Text", "__tag": 4046, "value": "), and binaries are available via " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "Miniforge" } ], "url": "https://github.com/conda-forge/miniforge", "title": "" }, { "__type": "Text", "__tag": 4046, "value": ". Wheels on PyPI for these platforms are now also possible (with the " }, { "__type": "InlineCode", "__tag": 4051, "value": "manylinux2014" }, { "__type": "Text", "__tag": 4046, "value": " standard), and requests for those are becoming more frequent." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Additionally, having IBM Z (or " }, { "__type": "InlineCode", "__tag": 4051, "value": "s390x" }, { "__type": "Text", "__tag": 4046, "value": ") in CI is now possible with TravisCI but not yet done - and " }, { "__type": "InlineCode", "__tag": 4051, "value": "manylinux2014" }, { "__type": "Text", "__tag": 4046, "value": " wheels for that platform are also possible then. Finally, resolving open AIX build issues would help users." } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Support for more hardware platforms" } ], "level": 1, "target": null }, { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "SciPy sparse matrices are being replaced by sparse arrays. The sparse matrix formats are mostly feature-complete, however their main issue is that they act like " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy.matrix" }, { "__type": "Text", "__tag": 4046, "value": " (which will be deprecated in NumPy at some point). What we want is sparse " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "arrays" } ] }, { "__type": "Text", "__tag": 4046, "value": " that act like " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy.ndarray" }, { "__type": "Text", "__tag": 4046, "value": " (See discussion at " }, { "__type": "Link", "__tag": 4049, "children": [ { "__type": "Text", "__tag": 4046, "value": "gh-18915" } ], "url": "https://github.com/scipy/scipy/issues/18915", "title": "" }, { "__type": "Text", "__tag": 4046, "value": "). Sparse arrays support all features of sparse matrices as of 1.15. In addition to 2D arrays, 1D sparse arrays are supported in DOK, COO, CSR formats. Further functionality e.g. nD array support and broadcasting for some operations is being developed. The future plan is:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "DefList", "__tag": 4033, "children": [ { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Extend sparse array API to nD arrays:" } ] }, "dd": [ { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "COO, CSR and DOK formats. COO format already partially in place." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The nD formats use 2D CSR code to do nD things like indexing/min-max/arithmetic." } ] } ] } ] } ] } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Sparse array binary operations will support broadcasting in some settings. Broadcasting is tricky for sparse arrays because it leans heavily on the strided memory model of dense arrays, and so does not always fit sparse data formats. Our optimistic goal is to support broadcasting for all operations where that makes sense for sparse data structures. We start with binary operations like " }, { "__type": "InlineRole", "__tag": 4003, "value": "A + B", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Help other libraries convert to sparse arrays from sparse matrices. Create transition guide and helpful scripts to flag code that needs changing." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Deprecate and then remove \"sparse matrix\" in favor of \"sparse array\"." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Work with NumPy on deprecation/removal of " }, { "__type": "InlineCode", "__tag": 4051, "value": "numpy.matrix" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] } ] } ], "title": [ { "__type": "Text", "__tag": 4046, "value": "Implement sparse arrays in addition to sparse matrices" } ], "level": 1, "target": null } ], "local_refs": [] }