STAR: Efficient SpatioTemporal Modeling for Action Recognition

Abhijeet Kumar, Samuel Abrams, Abhishek Kumar, Vijaykrishnan Narayanan

Research output: Contribution to journalArticlepeer-review

Abstract

Action recognition in video has gained significant attention over the past several years. While conventional 2D CNNs have found great success in understanding images, they are not as effective in capturing temporal relationships present in video. By contrast, 3D CNNs capture spatiotemporal information well, but they incur a high computational cost, making deployment challenging. In video, key information is typically confined to a small number of frames, though many current approaches require decompressing and processing all frames, which wastes resources. Others work directly on the compressed domain but require multiple input streams to understand the data. In our work, we directly operate on compressed video and extract information solely from intracoded frames (I-frames) avoiding the use of motion vectors and residuals for motion information making this a single-stream network. This reduces processing time and energy consumption, by extension, making this approach more accessible for a wider range of machines and uses. Extensive testing is employed on the UCF101 (Soomro et al. in UCF101: a dataset of 101 human actions classes from videos in the Wild, 2012) and HMDB51 (Kuehne et al., in: Jhuang, Garrote, Poggio, Serre (eds) Proceedings of the international conference on computer vision (ICCV), 2011) datasets to evaluate our framework and show that computational complexity is reduced significantly while achieving competitive accuracy to existing compressed domain efforts, i.e., 92.6% top1 accuracy in UCF-101 and 62.9% in HMDB-51 dataset with 24.3M parameters and 4 GFLOPS and energy savings of over 11× for the two datasets versus CoViAR (Wu et al. in Compressed video action recognition, 2018).

Original languageEnglish (US)
Pages (from-to)705-723
Number of pages19
JournalCircuits, Systems, and Signal Processing
Volume42
Issue number2
DOIs
StatePublished - Feb 2023

All Science Journal Classification (ASJC) codes

  • Signal Processing
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'STAR: Efficient SpatioTemporal Modeling for Action Recognition'. Together they form a unique fingerprint.

Cite this