Welcome!

This page is devoted to the subject of "Sparse Bayesian Modelling". It is maintained by Mike Tipping as part of his personal website, and it is also redirected to by www.relevancevector.com.

"Sparse Bayesian Modelling" describes the application of Bayesian automatic relevance determination (ARD) methodology to predictive models that are linear in their parameters. The motivation behind the approach is that one can infer a flexible, nonlinear, predictive model which is accurate and at the same time makes its predictions using only a small number of relevant basis functions which are automatically selected from a potentially large initial set.

The "relevance vector machine" is a special case of this idea, applied to linear kernel models, and may be of interest due to similarity of form with the popular "support vector machine".

Latest news (March 2009):

I have just posted a first release of the long-promised "V2" SparseBayes software for Matlab. This is a library of routines that implement the generic Sparse Bayesian model with inference based on the accelerated algorithm detailed in the paper "Fast marginal likelihood maximisation for Sparse Bayesian models". Links to both software and paper can be found below.

Note that this new "V2" software is intended to supercede the earlier "V1" version. I recently (February 2009) reimplemented this latter release after the unfortunate disappearance of my previous RVM page at Microsoft Research, and for "reference" purposes, this remains available to download below.

Thanks for your continued interest!

Papers

An introductory paper on Bayesian inference in machine learning, focussing on sparse Bayesian models, is available:

• Tipping, M. E. (2004). Bayesian inference: An introduction to principles and practice in machine learning. In O. Bousquet, U. von Luxburg, and G. Rätsch (Eds.), Advanced Lectures on Machine Learning, pp.  41–62. Springer. [Abstract] [PDF] [gzipped PostScript]

A fairly comprehensive full-length journal paper on sparse Bayesian learning:

• Tipping, M. E. (2001). Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Research  1, 211–244. [Abstract] [Available from JMLR]

There are a couple of minor typos in the above paper.

Two early conference publications on the Relevance Vector Machine:

• Tipping, M. E. (2000). The Relevance Vector Machine. In S. A. Solla, T. K. Leen, and K.-R. Müller (Eds.), Advances in Neural Information Processing Systems 12, pp.  652–658. MIT Press. [Abstract] [gzipped PostScript]

• Bishop, C. M. and M. E. Tipping (2000). Variational relevance vector machines. In C. Boutilier and M. Goldszmidt (Eds.), Proceedings of the 16th Conference on Uncertainty in Artificial Intelligence, pp.  46–53. Morgan Kaufmann. [Abstract] [PDF] [gzipped PostScript]

Note that the "variational" relevance vector machine is pretty much identical to the non-variational version, but is a lot slower to train J

Exploiting the sparse Bayes methodology to realise "sparse kernel PCA":

• Tipping, M. E. (2001). Sparse kernel principal component analysis. In Advances in Neural Information Processing Systems 13. MIT Press. [Abstract] [gzipped PostScript]

Robust sparse Bayesian regression:

• Faul, A. and M. E. Tipping (2001). A variational approach to robust regression. In G. Dorffner, H. Bischof, and K. Hornik (Eds.), Proceedings of ICANN'01, pp.  95–102. Springer. [Abstract] [gzipped PostScript]

Some theoretical analysis of marginal likelihood optimisation and sparsity:

• Faul, A. C. and M. E. Tipping (2002). Analysis of sparse Bayesian learning. In T. G. Dietterich, S. Becker, and Z. Ghahramani (Eds.), Advances in Neural Information Processing Systems 14, pp.  383–389. MIT Press. [Abstract] [gzipped PostScript]

An accelerated learning algorithm:

• Tipping, M. E. and A. C. Faul (2003). Fast marginal likelihood maximisation for sparse Bayesian models. In C. M. Bishop and B. J. Frey (Eds.), Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, Key West, FL, Jan 3-6. [Abstract] [PDF] [gzipped PostScript]

This is the algorithm of choice for implementing a sparse Bayes model. Its an order of magnitude faster than the original, uses less memory and analytically (rather than numerically) "prunes" irrelevant basis functions.

Slides

Copies of the slides from my 2003 lectures at the Tübingen "Machine Learning Summer School" are now available in ".ps.gz" format:

• Introduction to Bayesian Inference [180 KB]
• Bayesian Inference: Marginalisation [147 KB]
• Sparse Bayesian Models and the "Relevance Vector Machine" [1.18 MB]
• Sparse Bayesian Models: Analysis, Optimisation and Applications [2.68 MB]

Software

I have just finalised (March 2009) a first release of the "Version 2" SparseBayes library for Matlab. This is an implementation of the generic Sparse Bayesian model based on the accelerated algorithm detailed in the paper Fast marginal likelihood maximisation for Sparse Bayesian models (see above).

The library is freely available for download at the Vector Anomaly website. Note that the SparseBayes package for Matlab is free software, distributed by Vector Anomaly subject to the GNU Public Licence, version 2. Please see the file "licence.txt" included with the distribution for details.

 

A simpler, baseline, Matlab software implementation is also available here: [gzipped tar file]. This "SparseBayes V1.1" code (which dates back to 2002) implements the "original" sparse Bayesian learning algorithm, and so is relatively slow. It is nevertheless retained here for reference purposes.