{ "__type": "IngestedDoc", "__tag": 4010, "_content": { "Notes": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Method 'lm' (Levenberg-Marquardt) calls a wrapper over a least-squares algorithm implemented in MINPACK (lmder). It runs the Levenberg-Marquardt algorithm formulated as a trust-region type algorithm. The implementation is based on paper " }, { "__type": "CitationReference", "__tag": 4063, "label": "JJMore" }, { "__type": "Text", "__tag": 4046, "value": ", it is very robust and efficient with a lot of smart tricks. It should be your first choice for unconstrained problems. Note that it doesn't support bounds. Also, it doesn't work when m < n." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Method 'trf' (Trust Region Reflective) is motivated by the process of solving a system of equations, which constitute the first-order optimality condition for a bound-constrained minimization problem as formulated in " }, { "__type": "CitationReference", "__tag": 4063, "label": "STIR" }, { "__type": "Text", "__tag": 4046, "value": ". The algorithm iteratively solves trust-region subproblems augmented by a special diagonal quadratic term and with trust-region shape determined by the distance from the bounds and the direction of the gradient. This enhancements help to avoid making steps directly into bounds and efficiently explore the whole space of variables. To further improve convergence, the algorithm considers search directions reflected from the bounds. To obey theoretical requirements, the algorithm keeps iterates strictly feasible. With dense Jacobians trust-region subproblems are solved by an exact method very similar to the one described in " }, { "__type": "CitationReference", "__tag": 4063, "label": "JJMore" }, { "__type": "Text", "__tag": 4046, "value": " (and implemented in MINPACK). The difference from the MINPACK implementation is that a singular value decomposition of a Jacobian matrix is done once per iteration, instead of a QR decomposition and series of Givens rotation eliminations. For large sparse Jacobians a 2-D subspace approach of solving trust-region subproblems is used " }, { "__type": "CitationReference", "__tag": 4063, "label": "STIR" }, { "__type": "Text", "__tag": 4046, "value": ", " }, { "__type": "CitationReference", "__tag": 4063, "label": "Byrd" }, { "__type": "Text", "__tag": 4046, "value": ". The subspace is spanned by a scaled gradient and an approximate Gauss-Newton solution delivered by " }, { "__type": "CrossRef", "__tag": 4002, "value": "scipy.sparse.linalg.lsmr", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "*", "kind": "api", "path": "scipy.sparse.linalg._isolve.lsmr:lsmr" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ". When no constraints are imposed the algorithm is very similar to MINPACK and has generally comparable performance. The algorithm works quite robust in unbounded and bounded problems, thus it is chosen as a default algorithm." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Method 'dogbox' operates in a trust-region framework, but considers rectangular trust regions as opposed to conventional ellipsoids " }, { "__type": "CitationReference", "__tag": 4063, "label": "Voglis" }, { "__type": "Text", "__tag": 4046, "value": ". The intersection of a current trust region and initial bounds is again rectangular, so on each iteration a quadratic minimization problem subject to bound constraints is solved approximately by Powell's dogleg method " }, { "__type": "CitationReference", "__tag": 4063, "label": "NumOpt" }, { "__type": "Text", "__tag": 4046, "value": ". The required Gauss-Newton step can be computed exactly for dense Jacobians or approximately by " }, { "__type": "CrossRef", "__tag": 4002, "value": "scipy.sparse.linalg.lsmr", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "*", "kind": "api", "path": "scipy.sparse.linalg._isolve.lsmr:lsmr" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " for large sparse Jacobians. The algorithm is likely to exhibit slow convergence when the rank of Jacobian is less than the number of variables. The algorithm often outperforms 'trf' in bounded problems with a small number of variables." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Robust loss functions are implemented as described in " }, { "__type": "CitationReference", "__tag": 4063, "label": "BA" }, { "__type": "Text", "__tag": 4046, "value": ". The idea is to modify a residual vector and a Jacobian matrix on each iteration such that computed gradient and Gauss-Newton Hessian approximation match the true gradient and Hessian approximation of the cost function. Then the algorithm proceeds in a normal way, i.e., robust loss functions are implemented as a simple wrapper over standard least-squares algorithms." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionadded", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionadded 0.17.0" } ] } ] } ], "title": [], "level": 0, "target": null }, "Warns": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Raises": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Yields": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Methods": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Returns": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Parameters", "__tag": 4026, "children": [ { "__type": "DocParam", "__tag": 4016, "name": "result", "annotation": "OptimizeResult", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "CrossRef", "__tag": 4002, "value": "OptimizeResult", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._optimize:OptimizeResult" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " with the following fields defined:" } ] }, { "__type": "DefList", "__tag": 4033, "children": [ { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "x" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "x" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "cost" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "cost" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "fun" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "fun" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "jac" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "jac" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "grad" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "grad" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "optimality" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "optimality" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "active_mask" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "active_mask" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "nfev" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "nfev" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "njev" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "njev" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "status" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "status" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "message" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "message" } ] } ] }, { "__type": "DefListItem", "__tag": 4037, "dt": { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "success" } ] }, "dd": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "success" } ] } ] } ] } ] } ] } ], "title": [], "level": 0, "target": null }, "Summary": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Solve a nonlinear least-squares problem with bounds on the variables." } ] } ], "title": [], "level": 0, "target": null }, "Receives": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Warnings": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Attributes": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null }, "Parameters": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Parameters", "__tag": 4026, "children": [ { "__type": "DocParam", "__tag": 4016, "name": "fun", "annotation": "callable", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Function which computes the vector of residuals, with the signature " }, { "__type": "InlineCode", "__tag": 4051, "value": "fun(x, *args, **kwargs)" }, { "__type": "Text", "__tag": 4046, "value": ", i.e., the minimization proceeds with respect to its first argument. The argument " }, { "__type": "InlineCode", "__tag": 4051, "value": "x" }, { "__type": "Text", "__tag": 4046, "value": " passed to this function is an ndarray of shape (n,) (never a scalar, even for n=1). It must allocate and return a 1-D array_like of shape (m,) or a scalar. If the argument " }, { "__type": "InlineCode", "__tag": 4051, "value": "x" }, { "__type": "Text", "__tag": 4046, "value": " is complex or the function " }, { "__type": "InlineCode", "__tag": 4051, "value": "fun" }, { "__type": "Text", "__tag": 4046, "value": " returns complex residuals, it must be wrapped in a real function of real arguments, as shown at the end of the Examples section." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "x0", "annotation": "array_like with shape (n,) or float", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Initial guess on independent variables. If float, it will be treated as a 1-D array with one element. When " }, { "__type": "ParamRef", "__tag": 4071, "name": "method" }, { "__type": "Text", "__tag": 4046, "value": " is 'trf', the initial guess might be slightly adjusted to lie sufficiently within the given " }, { "__type": "ParamRef", "__tag": 4071, "name": "bounds" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "jac", "annotation": "{'2-point', '3-point', 'cs', callable}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Method of computing the Jacobian matrix (an m-by-n matrix, where element (i, j) is the partial derivative of f[i] with respect to x[j]). The keywords select a finite difference scheme for numerical estimation. The scheme '3-point' is more accurate, but requires twice as many operations as '2-point' (default). The scheme 'cs' uses complex steps, and while potentially the most accurate, it is applicable only when " }, { "__type": "ParamRef", "__tag": 4071, "name": "fun" }, { "__type": "Text", "__tag": 4046, "value": " correctly handles complex inputs and can be analytically continued to the complex plane. If callable, it is used as " }, { "__type": "InlineCode", "__tag": 4051, "value": "jac(x, *args, **kwargs)" }, { "__type": "Text", "__tag": 4046, "value": " and should return a good approximation (or the exact value) for the Jacobian as an array_like (np.atleast_2d is applied), a sparse array (csr_array preferred for performance) or a " }, { "__type": "CrossRef", "__tag": 4002, "value": "scipy.sparse.linalg.LinearOperator", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "*", "kind": "api", "path": "scipy.sparse.linalg._interface:LinearOperator" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionchanged", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionchanged 1.16.0" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "An ability to use the '3-point', 'cs' keywords with the 'lm' method. Previously 'lm' was limited to '2-point' and callable." } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "bounds", "annotation": "2-tuple of array_like or `Bounds`, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "There are two ways to specify bounds:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": true, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Instance of " }, { "__type": "CrossRef", "__tag": 4002, "value": "Bounds", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._constraints:Bounds" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " class" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Lower and upper bounds on independent variables. Defaults to no bounds. Each array must match the size of " }, { "__type": "ParamRef", "__tag": 4071, "name": "x0" }, { "__type": "Text", "__tag": 4046, "value": " or be a scalar, in the latter case a bound will be the same for all variables. Use " }, { "__type": "InlineCode", "__tag": 4051, "value": "np.inf" }, { "__type": "Text", "__tag": 4046, "value": " with an appropriate sign to disable bounds on all or some variables." } ] } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "method", "annotation": "{'trf', 'dogbox', 'lm'}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Algorithm to perform minimization." } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'trf'Trust Region Reflective algorithm, particularly suitable for large sparse problems with bounds. Generally robust method." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'dogbox'dogleg algorithm with rectangular trust regions, typical use case is small problems with bounds. Not recommended for problems with rank-deficient Jacobian." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'lm'Levenberg-Marquardt algorithm as implemented in MINPACK. Doesn't handle bounds and sparse Jacobians. Usually the most efficient method for small unconstrained problems." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Default is 'trf'. See Notes for more information." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "ftol", "annotation": "float or None, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Tolerance for termination by the change of the cost function. Default is 1e-8. The optimization process is stopped when " }, { "__type": "InlineCode", "__tag": 4051, "value": "dF < ftol * F" }, { "__type": "Text", "__tag": 4046, "value": ", and there was an adequate agreement between a local quadratic model and the true model in the last step." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If None and 'method' is not 'lm', the termination by this condition is disabled. If 'method' is 'lm', this tolerance must be higher than machine epsilon." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "xtol", "annotation": "float or None, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Tolerance for termination by the change of the independent variables. Default is 1e-8. The exact condition depends on the " }, { "__type": "ParamRef", "__tag": 4071, "name": "method" }, { "__type": "Text", "__tag": 4046, "value": " used:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'trf' and 'dogbox'" }, { "__type": "InlineCode", "__tag": 4051, "value": "norm(dx) < xtol * (xtol + norm(x))" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'lm'" }, { "__type": "InlineCode", "__tag": 4051, "value": "Delta < xtol * norm(xs)" }, { "__type": "Text", "__tag": 4046, "value": ", where " }, { "__type": "InlineCode", "__tag": 4051, "value": "Delta" }, { "__type": "Text", "__tag": 4046, "value": " is a trust-region radius and " }, { "__type": "InlineCode", "__tag": 4051, "value": "xs" }, { "__type": "Text", "__tag": 4046, "value": " is the value of " }, { "__type": "InlineCode", "__tag": 4051, "value": "x" }, { "__type": "Text", "__tag": 4046, "value": " scaled according to " }, { "__type": "ParamRef", "__tag": 4071, "name": "x_scale" }, { "__type": "Text", "__tag": 4046, "value": " parameter (see below)." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If None and 'method' is not 'lm', the termination by this condition is disabled. If 'method' is 'lm', this tolerance must be higher than machine epsilon." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "gtol", "annotation": "float or None, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Tolerance for termination by the norm of the gradient. Default is 1e-8. The exact condition depends on a " }, { "__type": "ParamRef", "__tag": 4071, "name": "method" }, { "__type": "Text", "__tag": 4046, "value": " used:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'trf'" }, { "__type": "InlineCode", "__tag": 4051, "value": "norm(g_scaled, ord=np.inf) < gtol" }, { "__type": "Text", "__tag": 4046, "value": ", where " }, { "__type": "InlineCode", "__tag": 4051, "value": "g_scaled" }, { "__type": "Text", "__tag": 4046, "value": " is the value of the gradient scaled to account for the presence of the bounds " }, { "__type": "CitationReference", "__tag": 4063, "label": "STIR" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'dogbox'" }, { "__type": "InlineCode", "__tag": 4051, "value": "norm(g_free, ord=np.inf) < gtol" }, { "__type": "Text", "__tag": 4046, "value": ", where " }, { "__type": "InlineCode", "__tag": 4051, "value": "g_free" }, { "__type": "Text", "__tag": 4046, "value": " is the gradient with respect to the variables which are not in the optimal state on the boundary." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'lm'the maximum absolute value of the cosine of angles between columns of the Jacobian and the residual vector is less than " }, { "__type": "ParamRef", "__tag": 4071, "name": "gtol" }, { "__type": "Text", "__tag": 4046, "value": ", or the residual vector is zero." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If None and 'method' is not 'lm', the termination by this condition is disabled. If 'method' is 'lm', this tolerance must be higher than machine epsilon." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "x_scale", "annotation": "{None, array_like, 'jac'}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Characteristic scale of each variable. Setting " }, { "__type": "ParamRef", "__tag": 4071, "name": "x_scale" }, { "__type": "Text", "__tag": 4046, "value": " is equivalent to reformulating the problem in scaled variables " }, { "__type": "InlineCode", "__tag": 4051, "value": "xs = x / x_scale" }, { "__type": "Text", "__tag": 4046, "value": ". An alternative view is that the size of a trust region along jth dimension is proportional to " }, { "__type": "InlineCode", "__tag": 4051, "value": "x_scale[j]" }, { "__type": "Text", "__tag": 4046, "value": ". Improved convergence may be achieved by setting " }, { "__type": "ParamRef", "__tag": 4071, "name": "x_scale" }, { "__type": "Text", "__tag": 4046, "value": " such that a step of a given size along any of the scaled variables has a similar effect on the cost function. If set to 'jac', the scale is iteratively updated using the inverse norms of the columns of the Jacobian matrix (as described in " }, { "__type": "CitationReference", "__tag": 4063, "label": "JJMore" }, { "__type": "Text", "__tag": 4046, "value": "). The default scaling for each method (i.e. if " }, { "__type": "InlineCode", "__tag": 4051, "value": "x_scale is None" }, { "__type": "Text", "__tag": 4046, "value": ") is as follows:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'trf'" }, { "__type": "InlineCode", "__tag": 4051, "value": "x_scale == 1" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'dogbox'" }, { "__type": "InlineCode", "__tag": 4051, "value": "x_scale == 1" } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For 'lm'" }, { "__type": "InlineCode", "__tag": 4051, "value": "x_scale == 'jac'" } ] } ] } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionchanged", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionchanged 1.16.0" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The default keyword value is changed from 1 to None to indicate that a default approach to scaling is used. For the 'lm' method the default scaling is changed from 1 to 'jac'. This has been found to give better performance, and is the same scaling as performed by " }, { "__type": "InlineCode", "__tag": 4051, "value": "leastsq" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "loss", "annotation": "str or callable, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Determines the loss function. The following keyword values are allowed:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'linear' (default)" }, { "__type": "InlineCode", "__tag": 4051, "value": "rho(z) = z" }, { "__type": "Text", "__tag": 4046, "value": ". Gives a standard least-squares problem." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'soft_l1'" }, { "__type": "InlineCode", "__tag": 4051, "value": "rho(z) = 2 * ((1 + z)**0.5 - 1)" }, { "__type": "Text", "__tag": 4046, "value": ". The smooth approximation of l1 (absolute value) loss. Usually a good choice for robust least squares." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'huber'" }, { "__type": "InlineCode", "__tag": 4051, "value": "rho(z) = z if z <= 1 else 2*z**0.5 - 1" }, { "__type": "Text", "__tag": 4046, "value": ". Works similarly to 'soft_l1'." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'cauchy'" }, { "__type": "InlineCode", "__tag": 4051, "value": "rho(z) = ln(1 + z)" }, { "__type": "Text", "__tag": 4046, "value": ". Severely weakens outliers influence, but may cause difficulties in optimization process." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'arctan'" }, { "__type": "InlineCode", "__tag": 4051, "value": "rho(z) = arctan(z)" }, { "__type": "Text", "__tag": 4046, "value": ". Limits a maximum loss on a single residual, has properties similar to 'cauchy'." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If callable, it must take a 1-D ndarray " }, { "__type": "InlineCode", "__tag": 4051, "value": "z=f**2" }, { "__type": "Text", "__tag": 4046, "value": " and return an array_like with shape (3, m) where row 0 contains function values, row 1 contains first derivatives and row 2 contains second derivatives. Method 'lm' supports only 'linear' loss." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "f_scale", "annotation": "float, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Value of soft margin between inlier and outlier residuals, default is 1.0. The loss function is evaluated as follows " }, { "__type": "InlineCode", "__tag": 4051, "value": "rho_(f**2) = C**2 * rho(f**2 / C**2)" }, { "__type": "Text", "__tag": 4046, "value": ", where " }, { "__type": "InlineCode", "__tag": 4051, "value": "C" }, { "__type": "Text", "__tag": 4046, "value": " is " }, { "__type": "ParamRef", "__tag": 4071, "name": "f_scale" }, { "__type": "Text", "__tag": 4046, "value": ", and " }, { "__type": "InlineCode", "__tag": 4051, "value": "rho" }, { "__type": "Text", "__tag": 4046, "value": " is determined by " }, { "__type": "ParamRef", "__tag": 4071, "name": "loss" }, { "__type": "Text", "__tag": 4046, "value": " parameter. This parameter has no effect with " }, { "__type": "InlineCode", "__tag": 4051, "value": "loss='linear'" }, { "__type": "Text", "__tag": 4046, "value": ", but for other " }, { "__type": "ParamRef", "__tag": 4071, "name": "loss" }, { "__type": "Text", "__tag": 4046, "value": " values it is of crucial importance." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "max_nfev", "annotation": "None or int, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "For all methods this parameter controls the maximum number of function evaluations used by each method, separate to those used in numerical approximation of the jacobian. If None (default), the value is chosen automatically as 100 * n." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionchanged", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionchanged 1.16.0" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The default for the 'lm' method is changed to 100 * n, for both a callable and a numerically estimated jacobian. Previously the default when using an estimated jacobian was 100 * n * (n + 1), because the method included evaluations used in the estimation." } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "diff_step", "annotation": "None or array_like, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Determines the relative step size for the finite difference approximation of the Jacobian. The actual step is computed as " }, { "__type": "InlineCode", "__tag": 4051, "value": "x * diff_step" }, { "__type": "Text", "__tag": 4046, "value": ". If None (default), then " }, { "__type": "ParamRef", "__tag": 4071, "name": "diff_step" }, { "__type": "Text", "__tag": 4046, "value": " is taken to be a conventional \"optimal\" power of machine epsilon for the finite difference scheme used " }, { "__type": "CitationReference", "__tag": 4063, "label": "NR" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "tr_solver", "annotation": "{None, 'exact', 'lsmr'}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Method for solving trust-region subproblems, relevant only for 'trf' and 'dogbox' methods." } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'exact' is suitable for not very large problems with dense Jacobian matrices. The computational complexity per iteration is comparable to a singular value decomposition of the Jacobian matrix." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "'lsmr' is suitable for problems with sparse and large Jacobian matrices. It uses the iterative procedure " }, { "__type": "CrossRef", "__tag": 4002, "value": "scipy.sparse.linalg.lsmr", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "*", "kind": "api", "path": "scipy.sparse.linalg._isolve.lsmr:lsmr" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " for finding a solution of a linear least-squares problem and only requires matrix-vector product evaluations." } ] } ] } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If None (default), the solver is chosen based on the type of Jacobian returned on the first iteration." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "tr_options", "annotation": "dict, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Keyword options passed to trust-region solver." } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "tr_solver='exact'" }, { "__type": "Text", "__tag": 4046, "value": ": " }, { "__type": "ParamRef", "__tag": 4071, "name": "tr_options" }, { "__type": "Text", "__tag": 4046, "value": " are ignored." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineCode", "__tag": 4051, "value": "tr_solver='lsmr'" }, { "__type": "Text", "__tag": 4046, "value": ": options for " }, { "__type": "CrossRef", "__tag": 4002, "value": "scipy.sparse.linalg.lsmr", "reference": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "*", "kind": "api", "path": "scipy.sparse.linalg._isolve.lsmr:lsmr" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": ". Additionally, " }, { "__type": "InlineCode", "__tag": 4051, "value": "method='trf'" }, { "__type": "Text", "__tag": 4046, "value": " supports 'regularize' option (bool, default is True), which adds a regularization term to the normal equation, which improves convergence if the Jacobian is rank-deficient " }, { "__type": "CitationReference", "__tag": 4063, "label": "Byrd" }, { "__type": "Text", "__tag": 4046, "value": " (eq. 3.4)." } ] } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "jac_sparsity", "annotation": "{None, array_like, sparse array}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Defines the sparsity structure of the Jacobian matrix for finite difference estimation, its shape must be (m, n). If the Jacobian has only few non-zero elements in " }, { "__type": "Emphasis", "__tag": 4047, "children": [ { "__type": "Text", "__tag": 4046, "value": "each" } ] }, { "__type": "Text", "__tag": 4046, "value": " row, providing the sparsity structure will greatly speed up the computations " }, { "__type": "CitationReference", "__tag": 4063, "label": "Curtis" }, { "__type": "Text", "__tag": 4046, "value": ". A zero entry means that a corresponding element in the Jacobian is identically zero. If provided, forces the use of 'lsmr' trust-region solver. If None (default), then dense differencing will be used. Has no effect for 'lm' method." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "verbose", "annotation": "{0, 1, 2}, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Level of algorithm's verbosity:" } ] }, { "__type": "BulletList", "__tag": 4053, "ordered": false, "start": 1, "children": [ { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "0 (default)work silently." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "1display a termination report." } ] } ] }, { "__type": "ListItem", "__tag": 4054, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "2display progress during iterations (not supported by 'lm' method)." } ] } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "args, kwargs", "annotation": "tuple and dict, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Additional arguments passed to " }, { "__type": "ParamRef", "__tag": 4071, "name": "fun" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "ParamRef", "__tag": 4071, "name": "jac" }, { "__type": "Text", "__tag": 4046, "value": ". Both empty by default. The calling signature is " }, { "__type": "InlineCode", "__tag": 4051, "value": "fun(x, *args, **kwargs)" }, { "__type": "Text", "__tag": 4046, "value": " and the same for " }, { "__type": "ParamRef", "__tag": 4071, "name": "jac" }, { "__type": "Text", "__tag": 4046, "value": "." } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "callback", "annotation": "None or callable, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Callback function that is called by the algorithm on each iteration. This can be used to print or plot the optimization results at each step, and to stop the optimization algorithm based on some user-defined condition. Only implemented for the " }, { "__type": "CrossRef", "__tag": 4002, "value": "trf", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._lsq.trf" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " and " }, { "__type": "CrossRef", "__tag": 4002, "value": "dogbox", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._lsq.dogbox" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " methods." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The signature is " }, { "__type": "InlineCode", "__tag": 4051, "value": "callback(intermediate_result: OptimizeResult)" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "InlineRole", "__tag": 4003, "value": "intermediate_result is a 'scipy.optimize.OptimizeResult", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " which contains the intermediate results of the optimization at the current iteration." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The callback also supports a signature like: " }, { "__type": "InlineCode", "__tag": 4051, "value": "callback(x)" } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Introspection is used to determine which of the signatures is invoked." } ] }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "If the " }, { "__type": "ParamRef", "__tag": 4071, "name": "callback" }, { "__type": "Text", "__tag": 4046, "value": " function raises " }, { "__type": "InlineRole", "__tag": 4003, "value": "StopIteration", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " the optimization algorithm will stop and return with status code -2." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionadded", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionadded 1.16.0" } ] } ] } ] }, { "__type": "DocParam", "__tag": 4016, "name": "workers", "annotation": "map-like callable, optional", "desc": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A map-like callable, such as " }, { "__type": "InlineRole", "__tag": 4003, "value": "multiprocessing.Pool.map", "domain": null, "role": null, "inventory": null }, { "__type": "Text", "__tag": 4046, "value": " for evaluating any numerical differentiation in parallel. This evaluation is carried out as " }, { "__type": "InlineCode", "__tag": 4051, "value": "workers(fun, iterable)" }, { "__type": "Text", "__tag": 4046, "value": "." } ] }, { "__type": "Admonition", "__tag": 4056, "kind": "versionadded", "base_type": "neutral", "children": [ { "__type": "AdmonitionTitle", "__tag": 4055, "children": [ { "__type": "Text", "__tag": 4046, "value": "versionadded 1.16.0" } ] } ] } ] } ] } ], "title": [], "level": 0, "target": null }, "Extended Summary": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Given the residuals f(x) (an m-D real function of n real variables) and the loss function rho(s) (a scalar function), " }, { "__type": "CrossRef", "__tag": 4002, "value": "least_squares", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._lsq.least_squares:least_squares" }, "kind": "module" }, { "__type": "Text", "__tag": 4046, "value": " finds a local minimum of the cost function F(x) " } ] }, { "__type": "Code", "__tag": 4050, "value": "minimize F(x) = 0.5 * sum(rho(f_i(x)**2), i = 0, ..., m - 1)\nsubject to lb <= x <= ub", "execution_status": null }, { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "The purpose of the loss function rho(s) is to reduce the influence of outliers on the solution." } ] } ], "title": [], "level": 0, "target": null }, "Other Parameters": { "__type": "Section", "__tag": 4015, "children": [], "title": [], "level": 0, "target": null } }, "_ordered_sections": [ "Summary", "Extended Summary", "Parameters", "Attributes", "Methods", "Returns", "Yields", "Receives", "Other Parameters", "Raises", "Warns", "Warnings", "Notes" ], "item_file": "/scipy/optimize/_lsq/least_squares.py", "item_line": 266, "item_type": "function", "aliases": [ "scipy.optimize.least_squares" ], "example_section_data": { "__type": "Section", "__tag": 4015, "children": [ { "__type": "Text", "__tag": 4046, "value": "In this example we find a minimum of the Rosenbrock function without bounds\non independent variables.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "import numpy as np\ndef fun_rosenbrock(x):\n return np.array([10 * (x[1] - x[0]**2), (1 - x[0])])\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nNotice that we only provide the vector of the residuals. The algorithm\nconstructs the cost function as a sum of squares of the residuals, which\ngives the Rosenbrock function. The exact minimum is at ``x = [1.0, 1.0]``.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.optimize import least_squares\nx0_rosenbrock = np.array([2, 2])\nres_1 = least_squares(fun_rosenbrock, x0_rosenbrock)\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "res_1.x\nres_1.cost\nres_1.optimality\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nWe now constrain the variables, in such a way that the previous solution\nbecomes infeasible. Specifically, we require that ``x[1] >= 1.5``, and\n``x[0]`` left unconstrained. To this end, we specify the `bounds` parameter\nto `least_squares` in the form ``bounds=([-np.inf, 1.5], np.inf)``.\n\nWe also provide the analytic Jacobian:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "def jac_rosenbrock(x):\n return np.array([\n [-20 * x[0], 10],\n [-1, 0]])\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nPutting this all together, we see that the new solution lies on the bound:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "res_2 = least_squares(fun_rosenbrock, x0_rosenbrock, jac_rosenbrock,\n bounds=([-np.inf, 1.5], np.inf))\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "res_2.x\nres_2.cost\nres_2.optimality\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nNow we solve a system of equations (i.e., the cost function should be zero\nat a minimum) for a Broyden tridiagonal vector-valued function of 100000\nvariables:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "def fun_broyden(x):\n f = (3 - x) * x + 1\n f[1:] -= x[:-1]\n f[:-1] -= 2 * x[1:]\n return f\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nThe corresponding Jacobian matrix is sparse. We tell the algorithm to\nestimate it by finite differences and provide the sparsity structure of\nJacobian to significantly speed up this process.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.sparse import lil_array\ndef sparsity_broyden(n):\n sparsity = lil_array((n, n), dtype=int)\n i = np.arange(n)\n sparsity[i, i] = 1\n i = np.arange(1, n)\n sparsity[i, i - 1] = 1\n i = np.arange(n - 1)\n sparsity[i, i + 1] = 1\n return sparsity\nn = 100000\nx0_broyden = -np.ones(n)\nres_3 = least_squares(fun_broyden, x0_broyden,\n jac_sparsity=sparsity_broyden(n))\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "res_3.cost\nres_3.optimality\n", "execution_status": "failure" }, { "__type": "Text", "__tag": 4046, "value": "\nLet's also solve a curve fitting problem using robust loss function to\ntake care of outliers in the data. Define the model function as\n``y = a + b * exp(c * t)``, where t is a predictor variable, y is an\nobservation and a, b, c are parameters to estimate.\n\nFirst, define the function which generates the data with noise and\noutliers, define the model parameters, and generate data:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from numpy.random import default_rng\nrng = default_rng()\ndef gen_data(t, a, b, c, noise=0., n_outliers=0, seed=None):\n rng = default_rng(seed)\n\n y = a + b * np.exp(t * c)\n\n error = noise * rng.standard_normal(t.size)\n outliers = rng.integers(0, t.size, n_outliers)\n error[outliers] *= 10\n\n return y + error\na = 0.5\nb = 2.0\nc = -1\nt_min = 0\nt_max = 10\nn_points = 15\nt_train = np.linspace(t_min, t_max, n_points)\ny_train = gen_data(t_train, a, b, c, noise=0.1, n_outliers=3)\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nDefine function for computing residuals and initial estimate of\nparameters.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "def fun(x, t, y):\n return x[0] + x[1] * np.exp(x[2] * t) - y\nx0 = np.array([1.0, 1.0, 0.0])\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nCompute a standard least-squares solution:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "res_lsq = least_squares(fun, x0, args=(t_train, y_train))\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nNow compute two solutions with two different robust loss functions. The\nparameter `f_scale` is set to 0.1, meaning that inlier residuals should\nnot significantly exceed 0.1 (the noise level used).\n\n" }, { "__type": "Code", "__tag": 4050, "value": "res_soft_l1 = least_squares(fun, x0, loss='soft_l1', f_scale=0.1,\n args=(t_train, y_train))\nres_log = least_squares(fun, x0, loss='cauchy', f_scale=0.1,\n args=(t_train, y_train))\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nAnd, finally, plot all the curves. We see that by selecting an appropriate\n`loss` we can get estimates close to optimal even in the presence of\nstrong outliers. But keep in mind that generally it is recommended to try\n'soft_l1' or 'huber' losses first (if at all necessary) as the other two\noptions may cause difficulties in optimization process.\n\n" }, { "__type": "Code", "__tag": 4050, "value": "t_test = np.linspace(t_min, t_max, n_points * 10)\ny_true = gen_data(t_test, a, b, c)\ny_lsq = gen_data(t_test, *res_lsq.x)\ny_soft_l1 = gen_data(t_test, *res_soft_l1.x)\ny_log = gen_data(t_test, *res_log.x)\nimport matplotlib.pyplot as plt\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "plt.plot(t_train, y_train, 'o')\nplt.plot(t_test, y_true, 'k', linewidth=2, label='true')\nplt.plot(t_test, y_lsq, label='linear loss')\nplt.plot(t_test, y_soft_l1, label='soft_l1 loss')\nplt.plot(t_test, y_log, label='cauchy loss')\nplt.xlabel(\"t\")\nplt.ylabel(\"y\")\nplt.legend()\n", "execution_status": "failure" }, { "__type": "Code", "__tag": 4050, "value": "plt.show()\n", "execution_status": "success" }, { "__type": "Figure", "__tag": 4024, "value": { "__type": "RefInfo", "__tag": 4000, "module": "scipy", "version": "1.17.1", "kind": "assets", "path": "fig-16dd8f77cf2456d2.png" } }, { "__type": "Text", "__tag": 4046, "value": "\nIn the next example, we show how complex-valued residual functions of\ncomplex variables can be optimized with ``least_squares()``. Consider the\nfollowing function:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "def f(z):\n return z - (0.5 + 0.5j)\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nWe wrap it into a function of real variables that returns real residuals\nby simply handling the real and imaginary parts as independent variables:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "def f_wrap(x):\n fx = f(x[0] + 1j*x[1])\n return np.array([fx.real, fx.imag])\n", "execution_status": "success" }, { "__type": "Text", "__tag": 4046, "value": "\nThus, instead of the original m-D complex function of n complex\nvariables we optimize a 2m-D real function of 2n real variables:\n\n" }, { "__type": "Code", "__tag": 4050, "value": "from scipy.optimize import least_squares\nres_wrapped = least_squares(f_wrap, (0.1, 0.1), bounds=([0, 0], [1, 1]))\nz = res_wrapped.x[0] + res_wrapped.x[1]*1j\n", "execution_status": "success" }, { "__type": "Code", "__tag": 4050, "value": "z\n", "execution_status": "failure" } ], "title": [], "level": 0, "target": null }, "see_also": [ { "__type": "SeeAlsoItem", "__tag": 4028, "name": { "__type": "CrossRef", "__tag": 4002, "value": "curve_fit", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._minpack_py:curve_fit" }, "kind": "module" }, "descriptions": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "Least-squares minimization applied to a curve-fitting problem." } ] } ], "type": null }, { "__type": "SeeAlsoItem", "__tag": 4028, "name": { "__type": "CrossRef", "__tag": 4002, "value": "leastsq", "reference": { "__type": "LocalRef", "__tag": 4022, "kind": "module", "path": "scipy.optimize._minpack_py:leastsq" }, "kind": "module" }, "descriptions": [ { "__type": "Paragraph", "__tag": 4045, "children": [ { "__type": "Text", "__tag": 4046, "value": "A legacy wrapper for the MINPACK implementation of the Levenberg-Marquadt algorithm." } ] } ], "type": null } ], "signature": { "__type": "SignatureNode", "__tag": 4029, "kind": "function", "parameters": [ { "__type": "SigParam", "__tag": 4030, "name": "fun", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": { "__type": "Empty", "__tag": 4031 } }, { "__type": "SigParam", "__tag": 4030, "name": "x0", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": { "__type": "Empty", "__tag": 4031 } }, { "__type": "SigParam", "__tag": 4030, "name": "jac", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "2-point" }, { "__type": "SigParam", "__tag": 4030, "name": "bounds", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "(-inf, inf)" }, { "__type": "SigParam", "__tag": 4030, "name": "method", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "trf" }, { "__type": "SigParam", "__tag": 4030, "name": "ftol", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "1e-08" }, { "__type": "SigParam", "__tag": 4030, "name": "xtol", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "1e-08" }, { "__type": "SigParam", "__tag": 4030, "name": "gtol", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "1e-08" }, { "__type": "SigParam", "__tag": 4030, "name": "x_scale", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "loss", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "linear" }, { "__type": "SigParam", "__tag": 4030, "name": "f_scale", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "1.0" }, { "__type": "SigParam", "__tag": 4030, "name": "diff_step", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "tr_solver", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "tr_options", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "jac_sparsity", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "max_nfev", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "verbose", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "0" }, { "__type": "SigParam", "__tag": 4030, "name": "args", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "()" }, { "__type": "SigParam", "__tag": 4030, "name": "kwargs", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "callback", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" }, { "__type": "SigParam", "__tag": 4030, "name": "workers", "annotation": { "__type": "Empty", "__tag": 4031 }, "kind": "POSITIONAL_OR_KEYWORD", "default": "None" } ], "return_annotation": { "__type": "Empty", "__tag": 4031 }, "target_name": "least_squares" }, "references": [ ".. [STIR] M. A. Branch, T. F. Coleman, and Y. Li, \"A Subspace, Interior,", " and Conjugate Gradient Method for Large-Scale Bound-Constrained", " Minimization Problems,\" SIAM Journal on Scientific Computing,", " Vol. 21, Number 1, pp 1-23, 1999.", ".. [NR] William H. Press et. al., \"Numerical Recipes. The Art of Scientific", " Computing. 3rd edition\", Sec. 5.7.", ".. [Byrd] R. H. Byrd, R. B. Schnabel and G. A. Shultz, \"Approximate", " solution of the trust region problem by minimization over", " two-dimensional subspaces\", Math. Programming, 40, pp. 247-263,", " 1988.", ".. [Curtis] A. Curtis, M. J. D. Powell, and J. Reid, \"On the estimation of", " sparse Jacobian matrices\", Journal of the Institute of", " Mathematics and its Applications, 13, pp. 117-120, 1974.", ".. [JJMore] J. J. More, \"The Levenberg-Marquardt Algorithm: Implementation", " and Theory,\" Numerical Analysis, ed. G. A. Watson, Lecture", " Notes in Mathematics 630, Springer Verlag, pp. 105-116, 1977.", ".. [Voglis] C. Voglis and I. E. Lagaris, \"A Rectangular Trust Region", " Dogleg Approach for Unconstrained and Bound Constrained", " Nonlinear Optimization\", WSEAS International Conference on", " Applied Mathematics, Corfu, Greece, 2004.", ".. [NumOpt] J. Nocedal and S. J. Wright, \"Numerical optimization,", " 2nd edition\", Chapter 4.", ".. [BA] B. Triggs et. al., \"Bundle Adjustment - A Modern Synthesis\",", " Proceedings of the International Workshop on Vision Algorithms:", " Theory and Practice, pp. 298-372, 1999." ], "qa": "scipy.optimize._lsq.least_squares:least_squares", "arbitrary": [], "local_refs": [ "args", "bounds", "callback", "diff_step", "f_scale", "ftol", "fun", "gtol", "jac", "jac_sparsity", "kwargs", "loss", "max_nfev", "method", "result", "tr_options", "tr_solver", "verbose", "workers", "x0", "x_scale", "xtol" ] }