Lung cancer tends to be detected at an advanced stage, resulting in a high patient mortality rate. Thus, recent research has focused on early disease detection. Autofluorescence bronchoscopy (AFB) is an effective noninvasive way of detecting early manifestations of lung cancer. Unfortunately, manual inspection of AFB video is extremely tedious and error-prone, while limited effort has been expended toward potentially more robust automatic AFB lesion analysis. We propose a real-time deep-learning architecture dubbed ESFPNet for accurate segmentation and robust detection of bronchial lesions in an AFB video stream. Our approach gives the best segmentation results (mDice = 0.756, mIoU=0.624) on our AFB dataset among recent architectures. Moreover, our model shows promising potential applicability to other domains, as evidenced by its state-of-the-art (SOTA) performance on the CVC-ClinicDB,ETIS-LaribPolypDB datasets, and superior performance on the Kvasir, CVC-ColonDB datasets.