Ordinal regression

Regression analysis
Part of a series on Statistics

Models
Linear regression Simple regression Ordinary least squares Polynomial regression General linear model
Generalized linear model Discrete choice Logistic regression Multinomial logit Mixed logit Probit Multinomial probit Ordered logit Ordered probit Poisson
Multilevel model Fixed effects Random effects Mixed model
Nonlinear regression Nonparametric Semiparametric Robust Quantile Isotonic Principal components Least angle Local Segmented
Errors-in-variables
Estimation
Least squares Ordinary least squares Linear (math) Partial Total Generalized Weighted Non-linear Non-negative Iteratively reweighted Ridge regression
Least absolute deviations Bayesian Bayesian multivariate
Background
Regression model validation Mean and predicted response Errors and residuals Goodness of fit Studentized residual Gauss–Markov theorem
Statistics portal

In statistics, ordinal regression (also called "ordinal classification") is a type of regression analysis used for predicting an ordinal variable, i.e. a variable whose value exists on an arbitrary scale where only the relative ordering between different values is significant. It can be considered an intermediate problem between regression and classification.^[1] Ordinal regression turns up often in the social sciences, for example in the modeling of human levels of preference (on a scale from, say, 1–5 for "very poor" through "excellent"), as well as in information retrieval. In machine learning, ordinal regression may also be called ranking learning.^[2]^{[lower-alpha 1]}

Linear models for ordinal regression

Ordinal regression can be performed using a generalized linear model (GLM) that fits both a coefficient vector and a set of thresholds to a dataset. Suppose one has a set of observations, represented by length- $p$ vectors $x 1$ through $x n$ , with associated responses $y 1$ through $y n$ , where each $y i$ is an ordinal variable on a scale $1, ..., K$ . To this data, one fits a length- $p$ coefficient vector $w$ and a set of thresholds $θ 1, ..., θ K -1$ with the property that $θ 1 < θ 2 < ... < θ K -1$ . This set of thresholds divides the real number line into $K$ disjoint segments, corresponding to the $K$ response levels.

The model can now be formulated as

\Pr(y\leq i|\mathbf {x} )=\sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )

or, the cumulative probability of the response $y$ being at most $i$ is given by a function $σ$ (the inverse link function) applied to a linear function of $x$ . Several choices exist for $σ$ ; the logistic function

\sigma (\theta _{i}-\mathbf {w} \cdot \mathbf {x} )={\frac {1}{1+e^{-(\theta _{i}-\mathbf {w} \cdot \mathbf {x} )}}}

gives the ordered logit model, while using the probit function gives the ordered probit model. A third option is to use an exponential function\

\sigma (\theta _{i}-{\mathbf {w}}\cdot {\mathbf {x}})=\exp(-\exp(\theta _{i}-{\mathbf {w}}\cdot {\mathbf {x}}))

which gives the proportional hazards model.^[3]

Latent variable model

The probit version of the above model can be justified by assuming the existence of a real-valued latent variable (unobserved quantity) $y*$ , determined by^[4]

y^{*}={\mathbf {w}}\cdot {\mathbf {x}}+\varepsilon

where $ε$ is normally distributed with zero mean and unit variance, conditioned on $x$ . The response variable $y$ results from an "incomplete measurement" of $y*$ , where one only determines the interval into which $y*$ falls:

y={\begin{cases}1~~{\text{if}}~~y^{*}\leq \theta _{1},\\2~~{\text{if}}~~\theta _{1}<y^{*}\leq \theta _{2},\\3~~{\text{if}}~~\theta _{2}<y^{*}\leq \theta _{3}\\\vdots \\K~~{\text{if}}~~\theta _{{K-1}}<y^{*}.\end{cases}}

Defining $θ 0 = -\infty$ and $θ K = \infty$ , the above can be summarized as $y = k$ if and only if $θ k -1 < y * \leq θ k$ .

From these assumptions, one can derive the conditional distribution of $y$ as^[4]

{\begin{aligned}P(y=k|\mathbf {x} )&=P(\theta _{k-1}<y^{*}\leq \theta _{k}|\mathbf {x} )\\&=P(\theta _{k-1}<\mathbf {w} \cdot \mathbf {x} +\varepsilon \leq \theta _{k})\\&=\Phi (\theta _{k}-\mathbf {w} \cdot \mathbf {x} )-\Phi (\theta _{k-1}-\mathbf {w} \cdot \mathbf {x} )\end{aligned}}

where $Φ$ is the cumulative distribution function of the standard normal distribution, and takes on the role of the inverse link function $σ$ . The log-likelihood of the model for a single training example $x i$ , $y i$ can now be stated as^[4]

\log {\mathcal {L}}({\mathbf {w}},{\mathbf {\theta }}|{\mathbf {x}}_{i},y_{i})=\sum _{{k=1}}^{K}[y_{i}=k]\log[\Phi (\theta _{k}-{\mathbf {w}}\cdot {\mathbf {x}})-\Phi (\theta _{{k-1}}-{\mathbf {w}}\cdot {\mathbf {x}})]

(using the Iverson bracket $[y i = k]$ .) The log-likelihood of the ordered logit model is analogous, using the logistic function instead of $Φ$ .^[5]

Alternative models

In machine learning, alternatives to the latent-variable models of ordinal regression have been proposed. An early result was PRank, a variant of the perceptron algorithm that found multiple parallel hyperplanes separating the various ranks; its output is a weight vector $w$ and a sorted vector of $K -1$ thresholds $θ$ , as in the ordered logit/probit models. The prediction rule for this model is to output the smallest rank $k$ such that $wx < θ k$ .^[6]

Other methods rely on the principle of large-margin learning that also underlies support vector machines.^[7]^[8]

Another approach is given by Rennie and Srebro, who, realizing that "even just evaluating the likelihood of a predictor is not straight-forward" in the ordered logit and ordered probit models, propose fitting ordinal regression models by adapting common loss functions from classification (such as the hinge loss and log loss) to the ordinal case.^[9]

Notes

↑ Not to be confused with learning to rank.

References

↑ Winship, Christopher; Mare, Robert D. (1984). "Regression Models with Ordinal Variables" (PDF). American Sociological Review. 49: 512–525.
↑ Shashua, Amnon; Levin, Anat (2002). Ranking with large margin principle: Two approaches. NIPS.
↑ McCullagh, Peter (1980). "Regression models for ordinal data". Journal of the Royal Statistical Society. Series B (Methodological). 42 (2): 109–142.
1 2 3 Wooldridge, Jeffrey M. (2010). Econometric Analysis of Cross Section and Panel Data. MIT Press. pp. 655–657. ISBN 9780262232586.
↑ Agresti, Alan (23 October 2010). "Modeling Ordinal Categorical Data" (PDF). Retrieved 23 July 2015.
↑ Crammer, Koby; Singer, Yoram (2001). Pranking with ranking. NIPS.
↑ Chu, Wei; Keerthi, S. Sathiya (2007). "Support vector ordinal regression". Neural Computation. 19: 792–815. CiteSeerX 10.1.1.297.3637. doi:10.1162/neco.2007.19.3.792.
↑ Herbrich, Ralf; Graepel, Thore; Obermayer, Klaus (2000). "Large Margin Rank Boundaries for Ordinal Regression". Advances in Large Margin Classifiers. MIT Press. pp. 115–132.
↑ Rennie, Jason D. M.; Srebro, Nathan (2005). Loss Functions for Preference Levels: Regression with Discrete Ordered Labels (PDF). Proc. IJCAI Multidisciplinary Workshop on Advances in Preference Handling.