Improving Data Analytics with Fast and Adaptive Regularization

Zhaojing Luo, Shaofeng Cai, Gang Chen, Jinyang Gao, Wang Chien Lee, Kee Yuan Ngiam, Meihui Zhang

Research output: Contribution to journalArticlepeer-review

4 Scopus citations


Deep Learning and Machine Learning models have recently been shown to be effective in many real world applications. While these models achieve increasingly better predictive performance, their structures have also become much more complex. A common and difficult problem for complex models is overfitting. Regularization is used to penalize the complexity of the model in order to avoid overfitting. However, in most learning frameworks, regularization function is usually set with some hyper-parameters where the best setting is difficult to find. In this paper, we propose an adaptive regularization method, as part of a large end-to-end healthcare data analytics software stack, which effectively addresses the above difficulty. First, we propose a general adaptive regularization method based on Gaussian Mixture (GM) to learn the best regularization function according to the observed parameters. Second, we develop an effective update algorithm which integrates Expectation Maximization (EM) with Stochastic Gradient Descent (SGD). Third, we design a lazy update and sparse update algorithm to reduce the computational cost by 4x and 20x, respectively. The overall regularization framework is fast, adaptive, and easy-to-use. We validate the effectiveness of our regularization method through an extensive experimental study over 14 standard benchmark datasets and three kinds of deep learning/machine learning models. The results illustrate that our proposed adaptive regularization method achieves significant improvement over state-of-the-art regularization methods.

Original languageEnglish (US)
Article number8713578
Pages (from-to)551-568
Number of pages18
JournalIEEE Transactions on Knowledge and Data Engineering
Issue number2
StatePublished - Feb 1 2021

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Computational Theory and Mathematics


Dive into the research topics of 'Improving Data Analytics with Fast and Adaptive Regularization'. Together they form a unique fingerprint.

Cite this