Copyright © 2006 The Institute of Electronics, Information and Communication Engineers
Regular Section -- Papers -- Neural Networks and Bioengineering |
Analytic Optimization of Shrinkage Parameters Based on Regularized Subspace Information Criterion
1 The author is with the Department of Computer Science, Tokyo Institute of Technology, Tokyo, 152-8552 Japan. E-mail: sugi{at}cs.titech.ac.jp, http://sugiyama-www.cs.titech.ac.jp/~sugi/, 2 The author is with the Department of Computational Intelligence and Systems Science, Tokyo Institute of Technology, Yokohama-shi, 226-8503 Japan. E-mail: sakura{at}isl.titech.ac.jp
For obtaining a higher level of generalization capability in supervised learning, model parameters should be optimized, i.e., they should be determined in such a way that the generalization error is minimized. However, since the generalization error is inaccessible in practice, model parameters are usually determined in such a way that an estimate of the generalization error is minimized. A standard procedure for model parameter optimization is to first prepare a finite set of candidates of model parameter values, estimate the generalization error for each candidate, and then choose the best one from the candidates. If the number of candidates is increased in this procedure, the optimization quality may be improved. However, this in turn increases the computational cost. In this paper, we give methods for analytically finding the optimal model parameter value from a set of infinitely many candidates. This maximally enhances the optimization quality while the computational cost is kept reasonable.
Key Words: supervised learning, generalization capability, model selection, shrinkage estimator, regularized subspace information criterion
Manuscript received February 27, 2006. Final manuscript received May 11, 2006.