Solving large-scale L1-regularized SVMs and cousins: the surprising effectiveness of column and constraint generation

by   Antoine Dedieu, et al.

The linear Support Vector Machine (SVM) is one of the most popular binary classification techniques in machine learning. Motivated by applications in modern high dimensional statistics, we consider penalized SVM problems involving the minimization of a hinge-loss function with a convex sparsity-inducing regularizer such as: the L1-norm on the coefficients, its grouped generalization and the sorted L1-penalty (aka Slope). Each problem can be expressed as a Linear Program (LP) and is computationally challenging when the number of features and/or samples is large -- the current state of algorithms for these problems is rather nascent when compared to the usual L2-regularized linear SVM. To this end, we propose new computational algorithms for these LPs by bringing together techniques from (a) classical column (and constraint) generation methods and (b) first order methods for non-smooth convex optimization - techniques that are rarely used together for solving large scale LPs. These components have their respective strengths; and while they are found to be useful as separate entities, they have not been used together in the context of solving large scale LPs such as the ones studied herein. Our approach complements the strengths of (a) and (b) --- leading to a scheme that seems to outperform commercial solvers as well as specialized implementations for these problems by orders of magnitude. We present numerical results on a series of real and synthetic datasets demonstrating the surprising effectiveness of classic column/constraint generation methods in the context of challenging LP-based machine learning tasks.


page 1

page 2

page 3

page 4


High-Dimensional Penalized Bernstein Support Vector Machines

The support vector machines (SVM) is a powerful classifier used for bina...

Sparse Universum Quadratic Surface Support Vector Machine Models for Binary Classification

In binary classification, kernel-free linear or quadratic support vector...

Quantum Sparse Support Vector Machines

We present a quantum machine learning algorithm for Sparse Support Vecto...

Fast L1-Minimization Algorithms For Robust Face Recognition

L1-minimization refers to finding the minimum L1-norm solution to an und...

Computing Estimators of Dantzig Selector type via Column and Constraint Generation

We consider a class of linear-programming based estimators in reconstruc...

Explainable prediction of Qcodes for NOTAMs using column generation

A NOtice To AirMen (NOTAM) contains important flight route related infor...

Safe squeezing for antisparse coding

Spreading the information over all coefficients of a representation is a...

Please sign up or login with your details

Forgot password? Click here to reset