Unsupervised learning of parsimonious mixtures on large spaces with integrated feature and component selection

Michael W. Graham, David J. Miller

Research output: Contribution to journalArticlepeer-review

47 Scopus citations


Estimating the number of components (the order) in a mixture model is often addressed using criteria such as the Bayesian information criterion (BIC) and minimum message length. However, when the feature space is very large, use of these criteria may grossly underestimate the order. Here, it is suggested that this failure is not mainly attributable to the criterion (e.g., BIC), but rather to the lack of "structure" in standard mixtures - these models trade off data fitness and model complexity only by varying the order. The authors of the present paper propose mixtures with a richer set of tradeoffs. The proposed model allows each component its own informative feature subset, with all other features explained by a common model (shared by all components). Parameter sharing greatly reduces complexity at a given order. Since the space of these parsimonious modeling solutions is vast, this space is searched in an efficient manner, integrating the component and feature selection within the generalized expectation-maximization (GEM) learning for the mixture parameters. The quality of the proposed (unsupervised) solutions is evaluated using both classification error and test set data likelihood. On text data, the proposed multinomial version - learned without labeled examples, without knowing the "true" number of topics, and without feature preprocessing - compares quite favorably with both alternative unsupervised methods and with a supervised naive Bayes classifier. A Gaussian version compares favorably with a recent method introducing "feature saliency" in mixtures.

Original languageEnglish (US)
Pages (from-to)1289-1303
Number of pages15
JournalIEEE Transactions on Signal Processing
Issue number4
StatePublished - Apr 2006

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'Unsupervised learning of parsimonious mixtures on large spaces with integrated feature and component selection'. Together they form a unique fingerprint.

Cite this