X-VS: Crossbar-based processing-in-memory architecture for video summarization

Video summarization techniques identify the most interesting frames in a video based on their uniqueness/importance or relevance to a user query. Deep learning based automated video summarization techniques have gained significant importance due to the growing need to analyze the exploding video data from user devices, surveillance cameras, and social media platforms etc. In contrast to the image classification, object detection tasks which predominantly use convolutional neural networks (CNNs), video summarization techniques comprise a pipeline of more diverse networks such as text processing networks, attention and content similarity mechanisms. In this work, we present X-VS, a ReRAM processing-in-memory (PIM) hardware accelerator architecture for video summarization workloads. We augment a baseline ReRAM CNN accelerator with a systolic array-based crossbar architecture to incorporate efficient support for recurrent neural networks, attention and content similarity mechanisms and hash-based word embedding lookup to support the video summarization networks. The proposed architecture achieves an average speedup of '450x, and energy savings of '1600x for two state-of-the-art video summarization networks over CPU and GPU implementations.