GRASP: Dynamic and Priority-Aware Gradient Sparsification for Efficient Online Learning

The rich temporal dynamics of Spiking Neural Networks (SNNs) offer numerous ways to interpret neuronal firing patterns, enabling various approaches for training these networks. With a growing demand for on-device intelligence, online learning methods implemented on neuromorphic hardware have emerged as promising solutions, allowing rapid adaptation to new stimuli under stringent constraints on latency and power consumption. Although SNNs' inherent input and output sparsity leads to energy-efficient implementations, conventional gradient-based training techniques still incur significant computational and memory costs that render them impractical for real-time learning on edge devices. Recent local-synaptic plasticity rules have mitigated the memory challenges associated with gradient-based methods; however, instantaneous resource utilization and energy consumption remain high due to dense gradient computations. Although model compression and redundancy reduction methods effectively address static inference costs, they do little to alleviate the ongoing computational burden introduced by continuous synaptic updates in online learning scenarios. Inspired by sharpness-aware minimization techniques, this work analyzes operational inefficiencies in instantaneous gradient updates by leveraging insights from the sharpness of the loss landscape to determine the optimal number of gradients updated at each timestep. Additionally, we dynamically compute the Hessian-vector product to prioritize gradient updates based on their potential impact on model parameters and loss reduction. We introduce GRASP, a dynamic, priority-aware, and model-agnostic enhancement compatible with existing online learning algorithms. Experimental evaluations demonstrate that GRASP significantly enhances the efficiency and effectiveness of online learning approaches, including DECOLLE, ETLP, and OTTT, across multiple diverse datasets.