Impeding behavior-based malware analysis via replacement attacks to malware specifications

Jiang Ming, Zhi Xin, Pengwei Lan, Dinghao Wu, Peng Liu, Bing Mao

Research output: Contribution to journalArticlepeer-review

24 Scopus citations


As the underground market of malware flourishes, there is an exponential increase in the number and diversity of malware. A crucial question in malware analysis research is how to define malware specifications or signatures that faithfully describe similar malicious intent and also clearly stand out from other programs. Although the traditional malware specifications based on syntactic signatures are efficient, they can be easily defeated by various obfuscation techniques. Since the malicious behavior is often stable across similar malware instances, behavior-based specifications which capture real malicious characteristics during run time, have become more prevalent in anti-malware tasks, such as malware detection and malware clustering. This kind of specification is typically extracted from the system call dependence graph that a malware sample invokes. In this paper, we present replacement attacks to camouflage similar behaviors by poisoning behavior-based specifications. The key method of our attacks is to replace a system call dependence graph to its semantically equivalent variants so that the similar malware samples within one family turn out to be different. As a result, malware analysts have to put more efforts into reexamining the similar samples which may have been investigated before. We distil general attacking strategies by mining more than 5200 malware samples’ behavior specifications and implement a compiler-level prototype to automate replacement attacks. Experiments on 960 real malware samples demonstrate the effectiveness of our approach to impede various behavior-based malware analysis tasks, such as similarity comparison and malware clustering. In the end, we also discuss possible countermeasures in order to strengthen existing malware defense.

Original languageEnglish (US)
Pages (from-to)193-207
Number of pages15
JournalJournal of Computer Virology and Hacking Techniques
Issue number3
StatePublished - Aug 1 2017

All Science Journal Classification (ASJC) codes

  • Computer Science (miscellaneous)
  • Software
  • Hardware and Architecture
  • Computational Theory and Mathematics


Dive into the research topics of 'Impeding behavior-based malware analysis via replacement attacks to malware specifications'. Together they form a unique fingerprint.

Cite this