Using Deep Belief Nets to Learn Covariance Kernels for Gaussian Processes

Part of Advances in Neural Information Processing Systems 20 (NIPS 2007)

Bibtex »Metadata »Paper »


Geoffrey E. Hinton, Russ R. Salakhutdinov


We show how to use unlabeled data and a deep belief net (DBN) to learn a good covariance kernel for a Gaussian process. We first learn a deep generative model of the unlabeled data using the fast, greedy algorithm introduced by Hinton If the data is high-dimensional and highly-structured, a Gaussian kernel applied to the top layer of features in the DBN works much better than a similar kernel applied to the raw input. Performance at both regression and classification can then be further improved by using backpropagation through the DBN to discriminatively fine-tune the covariance kernel.