Nonlinear Data Fitting APIs - 5.2 English - 68552

AOCL API Guide (68552)
Document ID
68552
Release Date
2025-12-29
Version
5.2 English
class aoclda.nonlinear_model.nlls(n_coef, n_res, weights=None, lower_bounds=None, upper_bounds=None, order='c', model='hybrid', method='galahad', glob_strategy='tr', reg_power='quadratic', verbose=0, check_data=false)#

Nonlinear data fitting.

This class defines a model and solves the problem

\[\underset{\text{subject to} x \in R^{n_{coef}}}{\text{minimize}} F(x) = \frac{1}{2} \sum_{i=0}^{n_{res-1}} r_i(x)^2_W + \frac{\sigma}{p} ||x||_2^p\]

where

  • \(x\) is the \(n_{coef}\)-vector of coefficients (parameters) to be trained,

  • \(r_i(x)\) is the \(n_{res}\)-vector of model residuals evaluated at \(x\),

  • \(\sigma > 0\), and \(p=2,3\) are the regularization hyperparameters, and

  • \(W\) is a vector of the diagonal elements of the weight norm matrix.

Parameters:

  • n_coef (int) – Number of coefficients in the model. Must be positive.

  • n_res (int) – Number of residuals in the model. Must be positive.

  • weights (array-like, optional) – Vector containing the values of the diagonal weight matrix W. It is expected that the values are all non-negative and normalized. Default = None.

  • lower_bounds (array-like, optional) – Vector of lower bounds of the coefficient vector. Default = None.

  • upper_bounds (array-like, optional) – Vector of upper bounds of the coefficient vector. Default = None.

  • order (str, optional) –

    defines the storage scheme for the matrices. Default = 'c'.

    • 'c' (row-major),

    • 'fortran' (column-major)

  • prec (str, optional) –

    defines the data precision to use. Default = 'double'.

    • 'double' uses floats in float64 format,

    • 'single' uses floats in float32 format.

  • model (str, optional) –

    Choose the nonlinear model to use. Default = 'hybrid'.

    • 'gauss-newton' Gauss-Newton method,

    • 'quasi-newton' Quasi-Newton method,

    • 'hybrid' uses a strategy that enables switching between the Gauss-Newton and Quasi-Newton methods,

    • 'tensor-newton' higher order method that uses Hessians. This option requires call-backs for second order derivative Hessians.

  • glob_strategy (str, optional) –

    Globalization strategy. Only relevant when model = 'hybrid', 'gauss-newton', or 'quasi-newton'. Default = 'tr'.

    • 'tr' uses trust-region methods,

    • 'reg' uses regularization techniques.

  • method (str, optional) –

    Optimization solver for the subproblem. Default = 'galahad'.

    • Valid option values for when glob_strategy = 'tr':

      • 'galahad' GALAHAD’s DTRS,

      • 'more-sorensen' variant of More-Sorensen’s method,

      • 'powell-dogleg' Powell’s dog-leg method.

      • 'aint' Generalized eigen value method.

    • Valid option values for when glob_strategy = 'reg':

      • 'galahad' GALAHAD’s DRQS.

      • 'linear solver' solve step system using a linear solver,

    Refer to [Gould et al., 2017] for details on these subproblem solvers.

  • reg_power (str, optional) –

    Regularization power (p). Default = 'quadratic'. This option is only relevant if the regularization term (parameter reg_term in fit()) is positive.

    • 'quadratic' uses second order regularization,

    • 'cubic' uses third order regularization.

  • check_derivatives (str, optional) –

    Verify the user-supplied Jacobian derivatives using finite-differences method.

    • 'no' will not perform any verifications

    • 'yes' will trigger the verification on each element of the Jacobian matrix. The verification requires one call to the residual function for each column of the Jacobian. The verification process will print one line for each Jacobian entry that is deemed to be incorrect in a similar format to

      Jac[2,6] =   1.28402 ~ 1.582  [ 2.097E-01], ( 0.250E-06) XT

      informing with an X that the entry is likely to be wrong. It also informs with a T that the entry at Jac[2,6] matches the correct value in the transposed Jacobian, indicating that the storage scheme should be reviewed (storing Fortran format instead of C format, or vice-versa). Storage scheme is defined with the order argument and defaults to c. If the verbose level is high then one line for each entry of the whole Jacobian is printed regardless of its correctness.

      The finite-difference and derivative checker machinery is controlled via two arguments of the fit function:

      fd_step which defines the overall step size for the finite-difference approximation, and

      fd_ttol the derivative test tolerance which sets the threshold for deciding if a user-supplied derivative is close to the finite-difference approximation.

      If false positives are reported by the derivative checker or if the optimization process fails, then trialing different values for this parameter may help.

  • verbose (int, optional) – Set verbosity level (0 to 3) of the solver. Default = 0.

  • check_data (bool, optional) – Whether to check the data for NaNs. Default = False.

fit(x, fun, jac=None, hes=None, hep=None, data=None, ftol=1e-08, abs_ftol=1e-08, gtol=1e-08, abs_gtol=1e-05, xtol=2.22e-16, reg_term=0, maxit=100, fd_step=1e-07, fd_ttol=0.0001)#

Fit data to a nonlinear model using regularized least-squares.

Parameters:

  • x (array-like) – initial guess to start optimizing from

  • fun (method) –

    function that calculates the n_res residuals. This function must return the residual vector of the model evaluated at the point x. This function has the interface

    def fun(x, residuals, data) -> int:

    Note

    • The function should only modify in-place the residual vector.

    • Must return 0 on success and nonzero otherwise.

    Example:

    def fun(x, r, data) -> int:
    r[:] = data.residuals(x)
    return 0;

  • jac (method) –

    function that calculates the residual Jacobian matrix: This function must return the Jacobian matrix of size n_res by n_coef of the model evaluated at the point x. This function has the interface

    def jac(x, jacobian, data) -> int:

    Note

    • The function should only modify in-place the Jacobian matrix.

    • Must return 0 on success and nonzero otherwise.

    Example:

    def jac(x, j, data) -> int:
    j[:] = data.Jacobian(x)
    return 0;

  • hes (method, optional) –

    function that calculates the n_coef by n_coef symmetric residual Hessian matrix: \(H = \sum_i r_i H_i\). This function has the interface

    def hes(x, r, h, data) -> int:

    Default = None.

    Note

    • The function should only modify in-place the Hessian matrix.

    • Must return 0 on success and nonzero otherwise.

    Example:

    def hes(x, r, h, data) -> int:
    n = data['n_coef'];
    Hi = data['hessians']
    h[:] = Hi.sum(n, x, r)
    return 0;

  • hep (method, optional) –

    function that calculates the matrix-vector product of the symmetric residual Hessian matrices with a vector y: hp[:] = [ H1(x)*y, H2(x)*y, …, Hn(x)*y]. This resulting matrix is of size number of coefficients by number of residuals. This function has the interface

    def hep(x, y, hp, data) -> int:

    Default = None.

    Note

    • The function should only modify in-place the hp Hessian-vector product matrix.

    • Must return 0 on success and nonzero otherwise.

    Example:

    def hep(x, y, hp, data) -> int:
    n = data['n_coef'];
    m = data['n_res'];
    Hi = data['hessians']
    hp[:] = Hi.prod(n, m, x, y)
    return 0

  • data (optional) – The solver does not read or write to this object. Default = None.

  • ftol (float, optional) – Defines the relative tolerance for the residual norm to declare convergence. Default = 1.0e-8.

  • abs_ftol (float, optional) – Defines the absolute tolerance for the residual norm to declare convergence. Default = 1.0e-8.

  • gtol (float, optional) – Defines the relative tolerance of the gradient norm to declare convergence. Default = 1.0e-8.

  • abs_gtol (float, optional) – Defines the absolute tolerance of the gradient norm to declare convergence. Default = 1.0e-5.

  • xtol (float, optional) – Defines the tolerance of the step length of two consecutive iterates to declare convergence. Default = 2.22e-16.

  • reg_term (float, optional) – Defines the regularization penalty term (sigma). Default = 0.

  • maxit (int, optional) – Defines the iteration limit. Default = 100.

  • fd_step (float, optional) – Defines the overall step size to use in

  • process (the finite-difference approximation. If the optimization) –

  • fails (stagnates or) –

  • this (then trialing different values for) –

  • help. (parameter may) –

  • fd_ttol (float, optional) – Derivative test tolerance, sets the

  • acceptably (threshold for deciding if a user-supplied derivative is) –

  • approximation. (close to the finite-difference) –

Returns:

Returns the fitted model, instance itself.

Return type:

self (object)

property metrics#

dictionary with the metrics:

  • obj (float): Objective value at iterate x.

  • norm_g (float): Norm of the residual gradient at iterate x.

  • scl_norm_g (float): Norm of the scaled residual gradient at x.

Type:

(dict) [obj, nrmg, sclg]

property n_eval#

dictionary with the number of function calls for:

  • f (int): Residual function fun (includes fd_f count as well).

  • j (int): Gradient function jac.

  • h (int): Hessian function hes.

  • hp (int): Hessian-vector product function hep.

  • fd_f (int): Residual function fun due to finite-differences or derivative checker.

Type:

(dict) [nevalf, nevalg, nevalh, nevalhp, nevalfd]

property n_iter#

number of iterations the solver made.

Type:

int

typedef da_int da_resfun_t_s(da_int n_coef, da_int n_res, void *data, const float *x, float *res)#
typedef da_int da_resfun_t_d(da_int n_coef, da_int n_res, void *data, const double *x, double *res)#

Nonlinear data fitting call-back. Residual function signature.

This function evaluates the nonlinear model at the point x and returns in res (of length n_res) the model’s error residual:

\[r_i(x) = \theta(t_i,x) - y_i,\]

where \(\theta\) is the model and the pair \((t_i, y_i), i = 1, \ldots, n_{res}\) are the model parameters and observations.

Param n_coef:

[in] number of coefficients in the model.

Param n_res:

[in] number of residuals declared.

Param data:

[inout] user data pointer; the solver does not touch this pointer and passes it on to the call-back.

Param x:

[in] the vector of coefficients (at the current iteration) of size n_coef.

Param res:

[out] residual vector of size n_res for the model evaluated at x.

Return:

flag indicating whether evaluation of the model was successful: zero to indicate success; nonzero to indicate failure, in which case the solver will terminate with da_status_optimization_usrstop.

typedef da_int da_resgrd_t_s(da_int n_coef, da_int n_res, void *data, float const *x, float *jac)#
typedef da_int da_resgrd_t_d(da_int n_coef, da_int n_res, void *data, double const *x, double *jac)#

Nonlinear data fitting call-back. Residual Jacobian function signature.

This function evaluates the nonlinear model’s residual first derivatives (gradients) at the point x and returns in jac (of size n_res by n_coef) the model’s Jacobian matrix:

\[\nabla r(x) = \left [ \nabla r_1(x), \nabla r_2(x), \ldots, \nabla r_{n_{res}}(x)\right ]^{\text{T}},\]
with

\[\nabla r_i(x) = \left [ \frac{\partial r_i(x)}{\partial x_1}, \frac{\partial r_i(x)}{\partial x_2}, \ldots, \frac{\partial r_i(x)}{\partial x_{n_{coef}}} \right].\]
Param n_coef:

[in] number of coefficients in the model.

Param n_res:

[in] number of residuals declared.

Param data:

[inout] user data pointer; the solver does not touch this pointer and passes it on to the call-back.

Param x:

[in] the vector of coefficients (at the current iteration) of size n_coef.

Param jac:

[out] Jacobian matrix (n_res by n_coef), first derivatives of the residual function. evaluated at x. This matrix expects to be stored in the format defined by the optional parameter storage_scheme and defaults to row-major; this can be changed to column-major (Fortran format).

Return:

flag indicating whether evaluation of the model was successful: zero to indicate success; nonzero to indicate failure, in which case the solver will terminate with da_status_optimization_usrstop.

typedef da_int da_reshes_t_s(da_int n_coef, da_int n_res, void *data, float const *x, float const *wr, float *hes)#
typedef da_int da_reshes_t_d(da_int n_coef, da_int n_res, void *data, double const *x, double const *wr, double *hes)#

Nonlinear data fitting call-back. Residual Hessians function signature.

This function evaluates the nonlinear model’s residual second derivatives (Hessians) at the point x and returns in hes (of size n_coef by n_coef) the matrix:

\[H(x) = \sum_{i=1}^{n_{res}} w_{r_i}\nabla^2 r_i(x)\]
with

\[\begin{split}\nabla^2 r_i(x) = \left [ \begin{array}{ccc} \frac{\partial^2 r_i(x)}{\partial x_1 x_1} &\ldots & \frac{\partial^2 r_i(x)}{\partial x_1 x_{n_{coef}}}\\ \vdots & \ddots&\vdots\\ \frac{\partial^2 r_i(x)}{\partial x_{n_{coef}} x_1} &\ldots & \frac{\partial^2 r_i(x)}{\partial x_{n_{coef}} x_{n_{coef}}} \end{array} \right],\end{split}\]

where \(w_{r_i}\) is the \(i\)-th weighted residual.

Param n_coef:

[in] number of coefficients in the model.

Param n_res:

[in] number of residuals declared.

Param data:

[inout] user data pointer; the solver does not touch this pointer and passes it on to the call-back.

Param x:

[in] the vector of coefficients (at the current iteration) of size n_coef.

Param wr:

[in] a scaled (weighted) version of the residual vector evaluated at x, of size n_res.

Param hes:

[out] Hessian matrix (size of n_coef by n_coef) containing second derivatives of the residual function evaluated at x. This symmetric matrix is expected to be stored in the format defined by the optional parameter storage_scheme and defaults to row-major; this can be changed to column-major (Fortran format).

Return:

flag indicating whether evaluation of the model was successful: zero to indicate success; nonzero to indicate failure, in which case the solver will terminate with da_status_optimization_usrstop.

typedef da_int da_reshp_t_s(da_int n_coef, da_int n_res, const float *x, const float *y, float *hp, void *data)#
typedef da_int da_reshp_t_d(da_int n_coef, da_int n_res, const double *x, const double *y, double *hp, void *data)#

Nonlinear data fitting call-back. Residual Hessians-vector product function signature.

This function evaluates the nonlinear model’s residual second derivatives (Hessians) at the point x, performs a matrix-vector product with y and returns in hp (of size n_coef by n_res) the matrix:

\[H_P(x) = \left [ \nabla^2 r_1(x)\,y,\; \nabla^2 r_2(x)\,y,\;\ldots,\;\nabla^2 r_{n_{res}}(x)\,y\right].\]
Param n_coef:

[in] number of coefficients in the model.

Param n_res:

[in] number of residuals declared.

Param data:

[inout] user data pointer; the solver does not touch this pointer and passes it on to the call-back.

Param x:

[in] the vector of coefficients (at the current iteration) of size n_coef.

Param y:

[in] an arbitrary vector of size n_coef.

Param hp:

[out] Hessians matrix-vector product with y. This dense matrix of size n_coef by n_res is expected to be stored in the format defined by the optional parameter storage_scheme and defaults to row-major; this can be changed to column-major (Fortran format).

Return:

flag indicating whether evaluation of the model was successful: zero to indicate success; nonzero to indicate failure, in which case the solver will terminate with da_status_optimization_usrstop.

da_status da_nlls_define_residuals_s(da_handle handle, da_int n_coef, da_int n_res, da_resfun_t_s *resfun, da_resgrd_t_s *resgrd, da_reshes_t_s *reshes, da_reshp_t_s *reshp)#
da_status da_nlls_define_residuals_d(da_handle handle, da_int n_coef, da_int n_res, da_resfun_t_d *resfun, da_resgrd_t_d *resgrd, da_reshes_t_d *reshes, da_reshp_t_d *reshp)#

Nonlinear data fitting function call-backs registration.

This function registers in the nonlinear data fitting handle the residual function call-backs that define the nonlinear model to train. resfun (residual function) and resgrd (residual Jacobian matrix) are mandatory while reshes (residual Hessians) and reshp (residual Hessians matrix-vector products) are only required if the solver for the model chosen requires higher order derivatives. If these are not provided and the solver requires them, an error will be returned.

For details on the function call-backs, see nonlinear data fitting call-back declarations.

Parameters:

  • handle – [inout] a da_handle object, initialized with type da_handle_nlls.

  • n_coef – [in] number of coefficients of the model.

  • n_res – [in] number of residuals.

  • resfun – [in] function callback to provide residual vector for the model evaluated at x.

  • resgrd – [in] function callback to provide the Jacobian matrix (first derivatives) of the residual function evaluated at x. If not available, set to NULL. See information on estimating derivatives in chapter introduction.

  • reshes – [in] function callback to evaluate residual Hessian matrices (second derivatives of the residual function) evaluated at x. Optionally, can be passed as NULL.

  • reshp – [in] function callback to evaluate the residual Hessians evaluated at x and perform for each Hessian matrix-vector product and store the resulting products into a dense matrix. Optionally, can be passed as NULL.

Returns:

da_status. The function returns:

da_status da_nlls_define_weights_s(da_handle handle, da_int n_res, float *weights)#
da_status da_nlls_define_weights_d(da_handle handle, da_int n_res, double *weights)#

Set residual weights on nonlinear models.

This function sets the square-root of the diagonal elements of the weighting matrix \( W\). This diagonal matrix, defines the norm to be used in the least-squares part of the optimization problem

\[\underset{\text{subject to} x \in R^{n_{coef}}}{\text{minimize}} F(x) = \frac{1}{2} \sum_{i=0}^{n_{res-1}} r_i(x)^2_W + \frac{\sigma}{p} ||x||_2^p.\]

By default, the norm of the residuals is taken to be \(\ell_2\) and so \(W = I\).

Changing the weighting matrix provides a means to rescale the importance of certain residuals where it is known that some residuals are more relevant than others.

Note

The handle does not make a copy of the weights vector. Instead it stores a pointer to its location. It is important that this stays valid on all subsequent calls to da_nlls_fit().

Parameters:

  • handle – [inout] a da_handle object, initialized with type da_handle_nlls.

  • n_res – [in] number of residuals of the model. Optionally, if set to zero then all previously defined weights are removed.

  • weights – [in] vector \( w\) of length n_res that defines the square-root of the entries of the diagonal weighting matrix: \( w_i = \sqrt{W_{ii}}\). The vector is not checked for correctness and it is assumed that all entries are valid.

Returns:

da_status. The function returns:

da_status da_nlls_define_bounds_s(da_handle handle, da_int n_coef, float *lower, float *upper)#
da_status da_nlls_define_bounds_d(da_handle handle, da_int n_coef, double *lower, double *upper)#

Set bound constraints on nonlinear models.

This function sets the bound constraints on the coefficients of a nonlinear model. Specifically, defines the lower and upper bounds of the coefficient vector

\[\ell_x[i] \le x[i] \le u_x[i], i = 1,\ldots,n_{coef}.\]

Note

The handle does not make a copy of the bound constraint vectors, it stores a pointer to their location. It is important that these stay valid on all subsequent calls to da_nlls_fit().

Parameters:

  • handle – [inout] a da_handle object, initialized with type da_handle_nlls.

  • n_coef – [in] number of coefficients of the model. Optionally, if set to zero then all previously defined bounds are removed.

  • lower – [in] vector \(\ell_x\) of length n_coef that defines the lower bound constraints on the coefficients. If the problem does not have lower bounds, then lower can be NULL. Any value less than -1e20 is considered as -infinity and the coefficient is considered to not have a lower bound.

  • upper – [in] vector \( u_x\) of length n_coef that defines the upper bound constraints on the coefficients. If the problem does not have upper bounds, then upper can be NULL. Any value greater than 1e20 is considered as infinity and the coefficient is considered to not have an upper bound.

Returns:

da_status. The function returns:

da_status da_nlls_fit_s(da_handle handle, da_int n_coef, float *coef, void *udata)#
da_status da_nlls_fit_d(da_handle handle, da_int n_coef, double *coef, void *udata)#

Fit a nonlinear model.

This function trains a nonlinear model. Specifically, it optimizes the coefficients of a nonlinear model defined in a handle. For further information on how to initialize a handle and on the steps to define and optimize a nonlinear model, see Nonlinear Data Fitting.

Parameters:

  • handle – [inout] a da_handle object, initialized with type da_handle_nlls.

  • n_coef – [in] number of coefficients of the model.

  • coef – [inout] vector of coefficients of size n_coef. On entry, it is the initial guess from where to start the optimization process. On exit, it contains the optimized coefficients.

  • udata – [inout] a generic pointer for the caller to pass any data objects to the residual callbacks. This pointer is passed to the callbacks untouched.

Returns:

da_status. Some of the following da_status flags have been marked as a “warning”. In these cases the returned coefficient vector coef contains a valid iterate, which is potentially a rough estimate of the solution. The function may return:

enum da_optim_info_t_#

Indices of the information vector containing metrics from optimization solvers.

The information vector can be retrieved after a successful return from the fit function da_nlls_fit_? by querying the handle, using da_handle_get_result_? and passing the query da_result_::da_rinfo.

Values:

enumerator info_objective#

objective value

enumerator info_grad_norm#

norm of the objective gradient

enumerator info_iter#

number of iterations

enumerator info_time#

current time

enumerator info_nevalf#

number of objective function callback evaluations

enumerator info_nevalg#

number of gradient callback evaluations

enumerator info_nevalh#

number of Hessian callback evaluations

enumerator info_nevalhp#

number of Hessian-vector callback evaluations

enumerator info_scl_grad_norm#

scaled gradient norm of objective

enumerator info_nevalfd#

number of objective function callback evaluations used for approximating the derivatives or due to derivative checker

enumerator info_number#

for internal use