Hierarchical maximum entropy modeling for regression

Maximum entropy/iterative scaling (ME/IS) models have been well developed for classification on categorical (discrete-field) feature spaces. In this paper, we propose a hierarchical maximum entropy regression (HMEreg) model in building a posterior model for continuous target, which encodes constraints in the hierarchical tree structures from both input features and target output variable. In ME models, the tradeoff between model bias and variance is found in the constraints encoded into the model - complex constraints give the model more representation capacity but may over-fit, whereas simple constraints may produce less over-fitting but may have much more model bias. We developed a greedy order-growing constraint search method to sequentially build constraints with flexible order based on likelihood gain on a validation set. Experiments showed the HMEreg model performed comparably to or better than other regression models, including generalized linear regression, multi-layer perceptron, support vector regression, and regression tree.