Logistic regression - 5.2 English

Logistic regression - 5.2 English - 68552

AOCL API Guide (68552)

Document ID

68552

Release Date

2025-12-29

Version

5.2 English

Logistic regression is a type of supervised classification model aimed at assigning labels. In AOCL-DA, the labels are expected to be provided in a categorical response variable, \(y\), encoded by \(\{0, 1, 2, \ldots, K-1 \}\). The fit is based on maximizing the log-likelihood (loss function) of the probabilities that each observation \(i\) belongs to a given class, in turn defined by,

\[\begin{split}p(y_i=k\,|\,X_i, \beta) = \frac{ \exp(\beta_k X_i) }{ 1 + \sum_{l=0}^{K-2}\exp(\beta_l X_i) }, \mathrm{ for }\ 0 \leq k < K-1,\\ p(y_i=K-1\,|\,X_i, \beta) = \frac{ 1 }{ 1 + \sum_{l=0}^{K-2}\exp(\beta_l X_i) }.\end{split}\]

As an example, if \(K=2\), the loss function simplifies to,

\[C\left( \beta \right) = -L(y, \beta X) = \sum_{i=0}^n \bigg( y_i \log p(X_i, \beta) + (1 - y_i) \log \big( 1 - p(X_i, \beta) \big) \bigg).\]

As in the linear regression model, \(\ell_1\) or \(\ell_2\) regularization can be applied by adding the corresponding penalty term to the cost function.

When the model uses \(\ell_1\) regularization is it also known by the name of lasso, while when using \(\ell_2\) is it also called a ridge model. When both regularization terms are present then it is termed an elastic-net model.