SNN-ANN Hybrid Networks for Embedded Multimodal Monocular Depth Estimation

Sadia Anjum Tumpa, Anusha Devulapally, Matthew Brehove, Espoir Kyubwa, Vijaykrishnan Narayanan

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Monocular depth estimation is a crucial task in many embedded vision systems with numerous applications in autonomous driving, robotics and augmented reality. Traditional methods often rely only on frame-based approaches, which struggle in dynamic scenes due to their limitations, while event-based cameras offer complementary high temporal resolution, though they lack spatial resolution and context. We propose a novel embedded multimodal monocular depth estimation framework using a hybrid spiking neural network (SNN) and artificial neural network (ANN) architecture. This framework leverages a custom accelerator, TransPIM for efficient transformer deployment, enabling real-time depth estimation on embedded systems. Our approach leverages the advantages of both frame-based and event-based cameras, where SNN extracts low-level features and generates sparse representations from events, which are then fed into an ANN with frame-based input for estimating depth. The SNN-ANN hybrid architecture allows for efficient processing of both RGB and event data showing competitive performance across different accuracy metrics in depth estimation with standard benchmark MVSEC and DENSE dataset. To make it accessible to embedded system we deploy it on TransPIM enabling 9x speedup and 183× lower energy consumption compared to standard GPUs opening up new possibilities for various embedded system applications.

Original languageEnglish (US)
Title of host publication2024 IEEE Computer Society Annual Symposium on VLSI
Subtitle of host publicationEmerging VLSI Technologies and Architectures, ISVLSI 2024
EditorsHimanshu Thapliyal, Jurgen Becker
PublisherIEEE Computer Society
Pages198-203
Number of pages6
ISBN (Electronic)9798350354119
DOIs
StatePublished - 2024
Event2024 IEEE Computer Society Annual Symposium on VLSI, ISVLSI 2024 - Knoxville, United States
Duration: Jul 1 2024Jul 3 2024

Publication series

NameProceedings of IEEE Computer Society Annual Symposium on VLSI, ISVLSI
ISSN (Print)2159-3469
ISSN (Electronic)2159-3477

Conference

Conference2024 IEEE Computer Society Annual Symposium on VLSI, ISVLSI 2024
Country/TerritoryUnited States
CityKnoxville
Period7/1/247/3/24

All Science Journal Classification (ASJC) codes

  • Hardware and Architecture
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint

Dive into the research topics of 'SNN-ANN Hybrid Networks for Embedded Multimodal Monocular Depth Estimation'. Together they form a unique fingerprint.

Cite this