TY - GEN

T1 - Hierarchical maximum entropy modeling for regression

AU - Zhang, Yanxin

AU - Miller, David Jonathan

AU - Kesidis, George

PY - 2009/12/1

Y1 - 2009/12/1

N2 - Maximum entropy/iterative scaling (ME/IS) models have been well developed for classification on categorical (discrete-field) feature spaces. In this paper, we propose a hierarchical maximum entropy regression (HMEreg) model in building a posterior model for continuous target, which encodes constraints in the hierarchical tree structures from both input features and target output variable. In ME models, the tradeoff between model bias and variance is found in the constraints encoded into the model - complex constraints give the model more representation capacity but may over-fit, whereas simple constraints may produce less over-fitting but may have much more model bias. We developed a greedy order-growing constraint search method to sequentially build constraints with flexible order based on likelihood gain on a validation set. Experiments showed the HMEreg model performed comparably to or better than other regression models, including generalized linear regression, multi-layer perceptron, support vector regression, and regression tree.

AB - Maximum entropy/iterative scaling (ME/IS) models have been well developed for classification on categorical (discrete-field) feature spaces. In this paper, we propose a hierarchical maximum entropy regression (HMEreg) model in building a posterior model for continuous target, which encodes constraints in the hierarchical tree structures from both input features and target output variable. In ME models, the tradeoff between model bias and variance is found in the constraints encoded into the model - complex constraints give the model more representation capacity but may over-fit, whereas simple constraints may produce less over-fitting but may have much more model bias. We developed a greedy order-growing constraint search method to sequentially build constraints with flexible order based on likelihood gain on a validation set. Experiments showed the HMEreg model performed comparably to or better than other regression models, including generalized linear regression, multi-layer perceptron, support vector regression, and regression tree.

UR - http://www.scopus.com/inward/record.url?scp=77950932380&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=77950932380&partnerID=8YFLogxK

U2 - 10.1109/MLSP.2009.5306225

DO - 10.1109/MLSP.2009.5306225

M3 - Conference contribution

AN - SCOPUS:77950932380

SN - 9781424449484

T3 - Machine Learning for Signal Processing XIX - Proceedings of the 2009 IEEE Signal Processing Society Workshop, MLSP 2009

BT - Machine Learning for Signal Processing XIX - Proceedings of the 2009 IEEE Signal Processing Society Workshop, MLSP 2009

T2 - Machine Learning for Signal Processing XIX - 2009 IEEE Signal Processing Society Workshop, MLSP 2009

Y2 - 2 September 2009 through 4 September 2009

ER -