Multimodal feature assessment using multibranch 3D CNN to BILSTM for feature level multi-polarization SAR image data fusion and vehicle identification

Ferris I. Arnous, Ram M. Narayanan

Research output: Chapter in Book/Report/Conference proceedingConference contribution


Deep machine learning computer vision algorithms have been widely explored for the purpose of multisensory data fusion. The ability to combine feature, pixel, and decision level information from multiple sensors in order to enhance accurate assessments and decisions made by the platforms has been a significant point of interest for the remote sensing community. In this paper, we propose a dual branch 3D convolutional neural network (CNN) to bi-long short-term memory network (BILSTM) algorithm that seeks to fuse sparse multiresolution, multi-pose and multimodal VV and HV polarizations of synthetic aperture radar (SAR) vehicle image information to enhance vehicle identification in unfamiliar and uncoherent environments. We cultivated and explored the proposed algorithm using the SDMS CV Data Domes repository of 14,430 augmented images per modality, equally represented over ten vehicle classes under similar and dissimilar vehicle pose augmentations with low to high levels of added testing set noise via zero-mean white Gaussian noise. Our results indicated that the local individual modality 3D convolution fusion of multiple poses and resolutions as well as dual-modality fusion of both polarizations enhanced the developed algorithm’s ability to classify SAR vehicle image information in unfamiliar pose, elevation angle and moderate to low noise environments.

Original languageEnglish (US)
Title of host publicationRadar Sensor Technology XXVII
EditorsAbigail S. Hedden, Gregory J. Mazzaro, Ann Marie Raynal
ISBN (Electronic)9781510661844
StatePublished - 2023
EventRadar Sensor Technology XXVII 2023 - Orlando, United States
Duration: May 1 2023May 3 2023

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


ConferenceRadar Sensor Technology XXVII 2023
Country/TerritoryUnited States

All Science Journal Classification (ASJC) codes

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering

Cite this