Prior Information Guided Regularized Deep Learning for Cell Nucleus Detection

Mohammad Tofighi, Tiantong Guo, Jairam K.P. Vanamala, Vishal Monga

Research output: Contribution to journalArticlepeer-review

57 Scopus citations


Cell nuclei detection is a challenging research topic because of limitations in cellular image quality and diversity of nuclear morphology, i.e., varying nuclei shapes, sizes, and overlaps between multiple cell nuclei. This has been a topic of enduring interest with promising recent success shown by deep learning methods. These methods train convolutional neural networks (CNNs) with a training set of input images and known, labeled nuclei locations. Many such methods are supplemented by spatial or morphological processing. Using a set of canonical cell nuclei shapes, prepared with the help of a domain expert, we develop a new approach that we call shape priors (SPs) with CNNs (SPs-CNN). We further extend the network to introduce an SP layer and then allowing it to become trainable (i.e., optimizable). We call this network as tunable SP-CNN (TSP-CNN). In summary, we present new network structures that can incorporate "expected behavior" of nucleus shapes via two components: learnable layers that perform the nucleus detection and a fixed processing part that guides the learning with prior information. Analytically, we formulate two new regularization terms that are targeted at: 1) learning the shapes and 2) reducing false positives while simultaneously encouraging detection inside the cell nucleus boundary. Experimental results on two challenging datasets reveal that the proposed SP-CNN and TSP-CNN can outperform the state-of-the-art alternatives.

Original languageEnglish (US)
Pages (from-to)2047-2058
Number of pages12
JournalIEEE transactions on medical imaging
Issue number9
StatePublished - Sep 1 2019

All Science Journal Classification (ASJC) codes

  • Software
  • Radiological and Ultrasound Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Prior Information Guided Regularized Deep Learning for Cell Nucleus Detection'. Together they form a unique fingerprint.

Cite this