Part of Advances in Neural Information Processing Systems 7 (NIPS 1994)
Glenn Marion, David Saad
Using a statistical mechanical formalism we calculate the evidence, generalisation error and consistency measure for a linear percep(cid:173) tron trained and tested on a set of examples generated by a non linear teacher. The teacher is said to be unrealisable because the student can never model it without error. Our model allows us to interpolate between the known case of a linear teacher, and an un(cid:173) realisable, nonlinear teacher. A comparison of the hyperparameters which maximise the evidence with those that optimise the perfor(cid:173) mance measures reveals that, in the non-linear case, the evidence procedure is a misleading guide to optimising performance. Finally, we explore the extent to which the evidence procedure is unreliable and find that, despite being sub-optimal, in some circumstances it might be a useful method for fixing the hyperparameters.