General statistical inference by an approximate application of the maximum entropy principle

Lian Yan, David J. Miller

Research output: Contribution to conferencePaperpeer-review

1 Scopus citations

Abstract

We propose a new learning method for building a general statistical inference engine, operating on discrete feature spaces. Such a model allows inference on any feature given values for the other features (or for a feature subset). Bayesian networks (BNs) are versatile tools that possess this inference capability. However, while the BN's explicit representation of conditional independencies is informative, this structure is not so easily learned. Typically, learning methods for BNs use (sub-optimal) greedy search techniques. There is also a difficult issue of overfitting in these models. Alternatively, in 1983 Cheeseman proposed finding the maximum entropy (ME) joint pmf consistent with arbitrary lower order probability constraints. This approach has some potential advantages over BNs. However, the huge complexity required for learning the joint pmf has severely limited the use of this approach until now. Here we propose an approximate ME method which also allows incorporation of arbitrary lower order constraints, but while retaining quite tractable learning complexity. The new method approximates the joint feature pmf (during learning) on a subgrid of the full feature space grid. Experimental results on the UC-Irvine repository reveal significant performance gains over two BN approaches: Chow and Liu's dependence trees and Herskovits and Cooper's Kutato. Several extensions of our approach are indicated.

Original languageEnglish (US)
Pages112-121
Number of pages10
StatePublished - 1999
EventProceedings of the 1999 9th IEEE Workshop on Neural Networks for Signal Processing (NNSP'99) - Madison, WI, USA
Duration: Aug 23 1999Aug 25 1999

Other

OtherProceedings of the 1999 9th IEEE Workshop on Neural Networks for Signal Processing (NNSP'99)
CityMadison, WI, USA
Period8/23/998/25/99

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Software
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'General statistical inference by an approximate application of the maximum entropy principle'. Together they form a unique fingerprint.

Cite this