Copyright © 2007 The Institute of Electronics, Information and Communication Engineers
Regular Section -- Papers -- Algorithm Theory |
Incremental Leaning and Model Selection for Radial Basis Function Network through Sleep
1 The authors are with the Graduate School of Information Science and Technology, Hokkaido University, Sapporo-shi, 060–0814 Japan. E-mail: yamauchi{at}complex.eng.hokudai.ac.jp
The model selection for neural networks is an essential procedure to get not only high levels of generalization but also a compact data model. Especially in terms of getting the compact model, neural networks usually outperform other kinds of machine learning methods. Generally, models are selected by trial and error testing using whole learning samples given in advance. In many cases, however, it is difficult and time consuming to prepare whole learning samples in advance. To overcome these inconveniences, we propose a hybrid on-line learning system for a radial basis function (RBF) network that repeats quick learning of novel instances by rote during on-line periods (awake phases) and repeats pseudo rehearsal for model selection during out-of-service periods (sleep phases). We call this system Incremental Learning with Sleep (ILS). During sleep phases, the system basically stops the learning of novel instances, and during awake phases, the system responds quickly. We also extended the system so as to shorten the periodic sleep periods. Experimental results showed the system selects more compact data models than those selected by other machine learning systems.
Key Words: incremental learning, sleep learning, model selection, RBF
Manuscript received December 21, 2005. Manuscript revised August 22, 2006.
References
[1] J.E. Moody, The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems, pp.847–854, Morgan Kaufmann Publishers, 1992.
[2] G.E. Hinton and D. van Camp, "Keeping neural networks simple by minimizing the description length of the weights," Sixth ACM Conference on Computational Learning Theory, Santa Cruz, pp.5–13, July 1993.
[3] N. Murata, S. Yoshizawa, and S. Amari, "Network information criterion: Determining the number of hidden units for an artificial neural network model," IEEE Trans. Neural Netw., vol.5, no.6, pp.865–872, Nov. 1994.[Medline]
[4] P. van de Laar and T. Heskes, "Pruning using parameter and neuronal metrics," Neural Comput., vol.11, pp.977–993, 1999.
[5] D.F. Specht, "A general regression neural network," IEEE Trans. Neural Netw., vol.2, no.6, pp.568–576, Nov. 1991.[Medline]
[6] D. Tomandl and A. Schober, "A modified generalized regression neural network (MGRNN) with a new efficient training algorithm as a robust "black-box"-tool for data analysis," Neural Networks, vol.14, pp.1023–1034, 2001.[Medline]
[7] R.O. Duda, P.E. Hart, and D.G. Stork, Pattern Classification, Second ed., Wiley-Interscience Publication, 2001.
[8] T. Poggio and F. Girosi, "Networks for approximation and learning," Proc. IEEE International Conference on Neural Networks, vol.78, no.9, pp.1481–1497, Sept. 1990.
[9] L. Yingwei, N. Sundararajan, and P. Saratchandran, "A sequential learning scheme for function approximation using minimal radial basis function neural networks," Neural Comput., vol.9, pp.461–478, 1997.[Abstract]
[10] M. Ishikawa, "Structural learning with forgetting," Neural Netw., vol.9, no.3, pp.509–521, 1996.
[11] N. Kasabov, "Evolving fuzzy neural networks for supervised/unsupervised online knowledge-based learning," IEEE Trans. Syst. Man Cybern., vol.31, no.6, pp.902–918, Dec. 2001.
[12] S. Schaal and C.G. Atkeson, "Constructive incremental learning from only local information," Neural Comput., vol.10, no.8, pp.2047–2084, Nov. 1998.[Abstract]
[13] T. Hoya and J.A. Chambers, "Heuristic pattern correction scheme using adaptively trained generalized regression neural networks," IEEE Trans. Neural Netw., vol.12, no.1, pp.91–100, Jan. 2001.[Medline]
[14] I. Hayashi, H. Maeda, and J.R. Williamson, "A formulation of receptive field type input layer for tam network using gabor function," IEEE International Conference on Fuzzy Systems, #1335, July 2004.
[15] J. Moody and C.J. Darken, "Fast learning in neural networks of locally-tuned processing units," Neural Comput., vol.1, pp.281–294, 1989.
[16] R.M. French, "Pseudo-recurrent connectionist networks: An approach to the "sensitivity stability" dilemma," Connection Science, vol.9, no.4, pp.353–379, 1997.
[17] B. Ans and S. Roussert, "Neural networks with a self-refreshing memory: Knowledge transfer in sequential learning tasks without catastrophic forgetting," Connection Science, vol.12, no.1, pp.1–19, 2000.
[18] K. Yamauchi, S. Itho, and N. Ishii, "Combination of fast and slow learning neural networks for quick adaptation and pruning redundant cells," IEEE SMC'99, 1999 IEEE System, Man and Cybernetics Conference, vol.III, pp.390–395, Oct. 1999.
[19] K. Yamauchi, S. Ito, and N. Ishii, "Wake-sleep learning method for quick adaptation and reduction of redundant cells," ICONIP 2000 7th International Conference on Neural Information Processing, vol.1, pp.559–564, Nov. 2000.
[20] K. Yamauchi, "Sequential learning and model selection with sleep," ICONIP 2001 8th International Conference on Neural Information Processing, ed. L. Zhang and F. Gu, vol.1, pp.205–210, Fudan University Press, Nov. 2001.
[21] K. Yamauchi and N. Kobayashi, "Incremental learning with sleep-learning of noiseless datasets," International Conference on Neural Information Processing ICONIP2002, vol.1, pp.398–403, Nov. 2002.
[22] K. Yamauchi and J. Hayami, "Sleep learning," IEEE World Congress on Computational Intelligence (WCCI2006), pp.6295–6302, July 2006.
[23] J. Ma, J. Theiler, and S. Perkins, "Accurate on-line support vector regression," Neural Comput., vol.15, pp.2683–2703, 2003.
[24] C. Burges, "Geometory and invariance in kernel based methods," in Advances in Kernel Methods — Support Vector Learning, ed. B. Schölkoph, C. Burges, and A. Smola, pp.144–152, MIT Press, 1999.
[25] M. Umano, Y. Hosoya, Y. Uno, K. Seta, and M. Okada, "Incremental learning of fuzzy rules by fuzzy neural network with forgetting facility," 19th Fuzzy System Symposium, pp.665–668, Japan Society for Fuzzy Theory and Intelligent Informatics, 2003.
[26] K. Yamauchi, N. Yamaguchi, and N. Ishii, "Incremental learning methods with retrieving interfered patterns," IEEE Trans. Neural Netw., vol.10, no.6, pp.1351–1365, Nov. 1999.[Medline]
[27] H. Akaike, "A new look at the statistical model identification," IEEE Trans. Autom. Control, vol.AC-19, no.6, pp.716–723, Dec. 1974.
[28] J.L. McClelland, R.C. O'Reilly, and B.L. McNaughton, "Why there are complementary learning systems in the hippocampus and neocortex: Insights from the successes and failures of connectionist models of learning and memory," Psychological Review, vol.102, no.3, pp.419–457, 1995.[Medline]
[29] K.A. Norman, E.L. Newman, and A.J. Perotte, "Methods for reducing interference in the complementary learning systems mode: Ocillating inhibition and autonomous memory rehearsal," Neural Netw., vol.18, no.9, pp.1212–1228, Nov. 2005.[Medline]
[30] A.S. Dave and D. Margoliash, "Song replay during sleep and computational rules for sensorymotor vocal learning," Sciense, vol.290, pp.812–816, Oct. 2000.
[31] K. Yamauchi, "Incremental learning and dimension selection with sleep," International Conference on Neural Information Processing ICONIP2004, LNCS3316, pp.489–494, Nov. 2004.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||