LOOP: Logic-oriented opaque predicate detection in obfuscated binary code

Jiang Ming, Dongpeng Xu, Li Wang, Dinghao Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

67 Scopus citations

Abstract

Opaque predicates have been widely used to insert superuous branches for control flow obfuscation. Opaque predicates can be seamlessly applied together with other obfuscation methods such as junk code to turn reverse engineering attempts into arduous work. Previous efforts in detecting opaque predicates are far from mature. They are either ad hoc, designed for a specific problem, or have a considerably high error rate. This paper introduces LOOP, a Logic Oriented Opaque Predicate detection tool for obfuscated binary code. Being different from previous work, we do not rely on any heuristics; instead we construct general logical formulas, which represent the intrinsic characteristics of opaque predicates, by symbolic execution along a trace. We then solve these formulas with a constraint solver. The result accurately answers whether the predicate under examination is opaque or not. In addition, LOOP is obfuscation resilient and able to detect previously unknown opaque predicates. We have developed a prototype of LOOP and evaluated it with a range of common utilities and obfuscated malicious programs. Our experimental results demonstrate the efficacy and generality of LOOP. By integrating LOOP with code normalization for matching metamorphic malware variants, we show that LOOP is an appealing complement to existing malware defenses.

Original languageEnglish (US)
Title of host publicationCCS 2015 - Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security
PublisherAssociation for Computing Machinery
Pages757-768
Number of pages12
ISBN (Electronic)9781450338325
DOIs
StatePublished - Oct 12 2015
Event22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015 - Denver, United States
Duration: Oct 12 2015Oct 16 2015

Publication series

NameProceedings of the ACM Conference on Computer and Communications Security
Volume2015-October
ISSN (Print)1543-7221

Other

Other22nd ACM SIGSAC Conference on Computer and Communications Security, CCS 2015
Country/TerritoryUnited States
CityDenver
Period10/12/1510/16/15

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'LOOP: Logic-oriented opaque predicate detection in obfuscated binary code'. Together they form a unique fingerprint.

Cite this