A deterministic, annealing-based approach for learning and model selection in finite mixture models

Research output: Contribution to journalConference articlepeer-review

3 Scopus citations


We address the longstanding problem of learning and model selection in finite mixtures. A common approach is to generate solutions of varying number of components (via the Expectation-Maximization (EM) algorithm) and then select the best model in the sense of a cost such as the Bayesian Information Criterion (BIC). A recent alternative uses component-wise EM (CEM) and, further, integrates model selection within CEM. Both approaches are susceptible to finding poor solutions, the first due to initialization sensitivity of EM and the second due to the sequential (greedy) nature of CEM. Deterministic annealing for clustering (DA) and mixture modeling (DAEM) provide potential for avoiding local optima. However, these methods do not encompass model selection. We propose a new technique with positive attributes of all these methods: it integrates learning and model selection, performs batch optimization over components, and has the character of DA, with the optimization performed over a sequence of decreasing temperatures. Unlike standard DA, with the partition entropy reduced as the temperature is lowered, our approach reduces entropy of binary random variables that express whether each component is active or inactive. At low temperature, the method achieves explicit model order selection. Experiments demonstrate favorable performance of our method, compared with several alternatives. We also give an interesting stochastic generative model interpretation for our method.

Original languageEnglish (US)
Pages (from-to)V-457-V-460
JournalICASSP, IEEE International Conference on Acoustics, Speech and Signal Processing - Proceedings
StatePublished - 2004
EventProceedings - IEEE International Conference on Acoustics, Speech, and Signal Processing - Montreal, Que, Canada
Duration: May 17 2004May 21 2004

All Science Journal Classification (ASJC) codes

  • Software
  • Signal Processing
  • Electrical and Electronic Engineering


Dive into the research topics of 'A deterministic, annealing-based approach for learning and model selection in finite mixture models'. Together they form a unique fingerprint.

Cite this