Semantics-Preserving Reinforcement Learning Attack Against Graph Neural Networks for Malware Detection

Lan Zhang, Peng Liu, Yoon Ho Choi, Ping Chen

Research output: Contribution to journalArticlepeer-review

5 Scopus citations


As an increasing number of deep-learning-based malware scanners have been proposed, the existing evasion techniques, including code obfuscation and polymorphic malware, are found to be less effective. In this work, we propose a reinforcement learning based semantics-preserving (i.e. functionality-preserving) attack against black-box GNNs (Graph Neural Networks) for malware detection. The key factor of adversarial malware generation via semantic Nops insertion is to select the appropriate semantic Nops and their corresponding basic blocks. The proposed attack uses reinforcement learning to automatically make these 'how to select' decisions. To evaluate the attack, we have trained two kinds of GNNs with three types (e.g., Backdoor, Trojan, and Virus) of Windows malware samples and various benign Windows programs. The evaluation results have shown that the proposed attack can achieve a significantly higher evasion rate than four baseline attacks, namely the binary diversification attack, the semantics-preserving random instruction insertion attack, the semantics-preserving accumulative instruction insertion attack, and the semantics-preserving gradient-based instruction insertion attack.

Original languageEnglish (US)
Pages (from-to)1390-1402
Number of pages13
JournalIEEE Transactions on Dependable and Secure Computing
Issue number2
StatePublished - Mar 1 2023

All Science Journal Classification (ASJC) codes

  • General Computer Science
  • Electrical and Electronic Engineering

Cite this