Copyright © 2006 The Institute of Electronics, Information and Communication Engineers
Regular Section -- Papers -- Biocybernetics, Neurocomputing |
A Pruning Algorithm for Training Cooperative Neural Network Ensembles
The authors are with the Department of Human and Artificial Intelligence Systems, University of Fukui, Fukui-shi, 910–8507 Japan. E-mail: murase{at}synapse.his.fukui-u.ac.jp
We present a training algorithm to create a neural network (NN) ensemble that performs classification tasks. It employs a competitive decay of hidden nodes in the component NNs as well as a selective deletion of NNs in ensemble, thus named a pruning algorithm for NN ensembles (PNNE). A node cooperation function of hidden nodes in each NN is introduced in order to support the decaying process. The training is based on the negative correlation learning that ensures diversity among the component NNs in ensemble. The less important networks are deleted by a criterion that indicates over-fitting. The PNNE has been tested extensively on a number of standard benchmark problems in machine learning, including the Australian credit card assessment, breast cancer, circle-in-the-square, diabetes, glass identification, ionosphere, iris identification, and soybean identification problems. The results show that classification performances of NN ensemble produced by the PNNE are better than or competitive to those by the conventional constructive and fixed architecture algorithms. Furthermore, in comparison to the constructive algorithm, NN ensemble produced by the PNNE consists of a smaller number of component NNs, and they are more diverse owing to the uniform training for all component NNs.
Key Words: neural network ensemble, ensemble design, negative correlation learning, pruning, node decay, over-fitting, generalization
Manuscript received May 30, 2005. Manuscript revised September 13, 2005.
References
[1] D.W. Opitz and R. Maclin, "Popular ensemble methods: An empirical study," J. Artificial Intelligence Research, vol.11, pp.169–198, 1999.
[2] L. Breidman, "Bagging predictors," Mach. Learn., vol.24, no.2, pp.123–140, 1996.
[3] N.E. Sharkey and A.J.C. Sharkey, "An analysis of catastrophic interference," Connection Science, vol.7, no.3/4, pp.313–341, 1995.
[4] H. Drucker, C. Cortes, L.D. Jackel, Y. LeCun, and V. Vapnik, "Boosting and other ensemble methods," Neural Comput., vol.6, no.6, pp.1289–1301, 1994.
[5] Y. Jiang, Z.H. Zhou, and Z.Q. Chen, "Rule learning based on neural network ensemble," Proc. Int. Joint Conf. on Neural Network, pp.1416–1420, Honolulu, HI, 2002.
[6] B.E. Rosen, "Ensemble learning using decorrelated neural networks," Connection Science, vol.8, pp.373–383, 1996.
[7] A.J.C. Sharkey, "On combining artificial neural nets," Connection Science, vol.8, pp.299–313, 1996.
[8] D.W. Opitz and J.W. Shavlik, "Actively searching for an effective neural-network ensemble," Connection Science, vol.8, no.3/4, pp.337–353, 1996.
[9] L.I. Kuncheva and L.C. Jain, "Designing classifier fusion systems by genetic algorithms," IEEE Trans. Evol. Comput., vol.4, no.4, pp.327–336, 2000.
[10] P.D. Gader, M.A. Mohamed, and J.M. Keller, "Fusion of handwritten word classifiers," Pattern Recognit. Lett., vol.17, pp.577–584, 1996.
[11] J. Kittler, M. Hatef, R.P.W. Duin, and J. Matas, "On combining classifiers," IEEE Trans. Pattern Anal. Mach. Intell., vol.20, no.3, pp.226–239, 1998.
[12] R.A. Jacobs, "Methods for combining experts' probability assessment," Neural Comput., vol.7, pp.867–888, 1995.[Abstract]
[13] J.A. Benediktsson, J.R. Sveinsson, J.I. Ingimundarson, H. Sigurdsson, and O.K. Ersoy, "Multistage classifiers optimized by neural networks and genetic algorithm," Nonlinear Anal. Theory, Methods Appl., vol.30, no.3, pp.1323–1334, 1997.
[14] L.K. Hansen and P. Salamon, "Neural network ensembles," IEEE Trans. Pattern Anal. Mach. Intell., vol.12, no.10, pp.993–1001, 1990.
[15] K.J. Cherlauer, "Human expert level performance on a scientific image analysis task by a system using combined artificial neural networks," Proc. 13th AAAI Workshop on Integrating Multiple Learned Models for Improving and Scaling Machine Learning Algorithms, pp.15–21, Portland, OR, 1996.
[16] J. Hampshire and A. Waibel, "A novel objective function for improved phoneme recognition using time delay neural networks," IEEE Trans. Neural Netw., vol.1, no.2, pp.216–228, 1990.[Medline]
[17] A. Krogh and J. Veledsby, "Neural network ensemble, cross validation and active learning," in Advances in Neural Information Processing Systems 7, ed. G. Tesauro, D.S. Touretzky, and T.K. Leen, MIT Press, 1995.
[18] A.J.C. Sharkey, N.E. Sharkey, and G.O. Chandroth, "Neural nets and diversity," Proc. 14th Int. Conf. on Computer Safety, Reliability and Security, pp.375–389, Belgirate, Italy, Oct. 1995.
[19] K. Tumer and J. Ghosh, "Analysis of decision boundaries in linearly combined neural networks," Pattern Recognit., vol.29, no.2, pp.341–348, 1996.
[20] Y. Liu and X. Yao, "Ensemble learning via negative correlation," Neural Netw., vol.12, pp.1399–1404, 1999.[Medline]
[21] Y. Liu and X. Yao, "Simultaneous training of negatively correlated neural networks in an ensemble," IEEE Trans. Syst., Man Cybern., B, Cybern., vol.29, pp.716–725, June 1999.[Medline]
[22] J.S. Lee, H. Lee, J.Y. Kim, D. Nam, and C.H. Park, "Self-organizing neural networks by construction and pruning," IEICE Trans. Inf. & Syst., vol.E87-D, no.11, pp.2489–2498, Nov. 2004.
[23] R. Setiono and L.C.K. Hui, "Use of a quasi-Newton method in a feed-forward neural network construction algorithm," IEEE Trans. Neural Netw., vol.6, no.1, pp.273–277, 1995.
[24] R. Reed, "Pruning algorithms—A survey," IEEE Trans. Neural Netw., vol.4, no.5, pp.740–747, 1993.[Medline]
[25] M.M. Islam, X. Yao, and K. Murase, "A constructive algorithm for training cooperative neural network ensembles," IEEE Trans. Neural Netw., vol.14, no.4, pp.820–834, 2003.[Medline]
[26] D.E. Rumelhart, G.E. Hinton, and R.J. Williams, "Learning internal representation by error propagation," in Parallel Distributed Processing, vol.1, ed. D.E. Rumelhart and J.L. McClelland, pp.318–362, MIT Press, Cambridge, MA, 1986.
[27] K. Murase, Y. Matsunaga, and Y. Nakaade, "A back-propagation algorithm which automatically determines the number of association units," Proc. IJCNN, pp.783–788, Singapore, 1991.
[28] Y. Matsunaga, Y. Nakada, O. Yamakawa, and K. Murase, "A back-propagation algorithm with automatic reduction of association units in multilayer neural network," IEICE Trans. Inf. & Syst. (Japanese Edition), vol.J74-D-II, no.8, pp.1118–1121, Aug. 1991.
[29] Y. Matsunaga, K. Murase, O. Yamakawa, and M. Tanifuji, "A modified back propagation algorithm that automatically removes the redundant hidden units by competition," IEICE Trans. Inf. & Syst. (Japanese Edition), vol.J79-D-II, no.3, pp.403–412, March 1996.
[30] M. Hagiwara, "Removal of hidden units and weights for back-propagation," Proc. IJCNN, vol.1, pp.351–354, 1993.
[31] M. Shahjahan, M.M. Islam, and K. Murase, "A competitive node decaying method for artificial neural networks," Proc. KES 2002, ed. E. Damiani, et al., pp.11–15, Creama, Italy, 2002.
[32] M. Shahjahan and K. Murase, "Exploring simultaneous node decay to select artificial neural network structure," Proc. HART 2002, pp.424–427, Fukui, Japan, 2002.
[33] M. Shahjahan and K. Murase, "A dynamic node decaying method for pruning artificial neural networks," IEICE Trans. Inf. & Syst., vol.E86-D, no.4, pp.736–751, April 2003.
[34] X. Yao and Y. Liu, "A new evolutionary system for evolving artificial neural network," IEEE Trans. Neural Netw., vol.8, no.3, pp.694–713, 1997.[Medline]
[35] Z. Zhou, S. Chen, and Z. Chen, "FANNC: A fast adaptive neural network classifier," Knowledge and Information Systems, vol.2, pp.115–129, 2000.
[36] L. Prechelt, "Connection pruning with static and adaptive pruning schedules," Neurocomputing, vol.16, no.1, pp.49–61, July 1997.
[37] H. Inoue and H. Narihisa, "Efficient pruning method for ensemble self-generating neural networks," J. Systemics, Cybernetics and Informatics, vol.1, no.6, pp.72–73, 2004.
[38] A. Lazarevic and Z. Obradovic, "Effective pruning of neural network classifier ensembles," Proc. IJCNN, pp.796–801, Washington DC, July 2001.
[39] H. Liu, A. Mandvikar, and J. Mody, "An empirical study of building compact ensemble," 5th Intl. Conference on Web-Age Information Management (WAIM), Dahan, China, July 2004.
[40] I. Witter and E. Frank, Data Mining—Practical Machine Learning Tools and Techniques with JAVA Implementation, Morgan Kaufman, 2000.
[41] E.A. Partrick and F.P. Fischer, "A generalized k-nearest neighbor rule," Inf. Control, vol.16, no.2, pp.128–152, 1970.
[42] K. Tumer and J. Ghosh, "Error correlation and error reduction in ensemble classifiers," University of Texas, TX, July 1996.
[43] G. Wilensky, "Analysis of neural network issues: Scaling, enhancing nodal processing, comparison with standard classification," DARPA Neural Network Program Review, pp.29–30, 1990.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||