Generalized dynamic opaque predicates: A new control flow obfuscation method

Dongpeng Xu, Jiang Ming, Dinghao Wu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

32 Scopus citations

Abstract

Opaque predicate obfuscation, a low-cost and stealthy control flow obfuscation method to introduce superfluous branches, has been demonstrated to be effective to impede reverse engineering efforts and broadly used in various areas of software security. Conventional opaque predicates typically rely on the invariant property of well-known number theoretic theorems, making them easy to be detected by the dynamic testing and formal semantics techniques. To address this limitation, previous work has introduced the idea of dynamic opaque predicates, whose values may vary in different runs. However, the systematical design and evaluation of dynamic opaque predicates are far from mature. In this paper, we generalize the concept and systematically develop a new control flow obfuscation scheme called generalized dynamic opaque predicates. Compared to the previous work, our approach has two distinct advantages: (1) We extend the application scope by automatically transforming more common program structures (e.g., straight-line code, branch, and loop) into dynamic opaque predicates; (2) Our system design does not require that dynamic opaque predicates to be strictly adjacent, which is more resilient to the deobfuscation techniques. We have developed a prototype tool based on LLVM IR and evaluated it by obfuscating the GNU core utilities. Our experimental results show the efficacy and generality of our method. In addition, the comparative evaluation demonstrates that our method is resilient to the latest formal program semantics-based opaque predicate detection method.

Original languageEnglish (US)
Title of host publicationInformation Security - 19th International Conference, ISC 2016, Proceedings
EditorsMatt Bishop, Anderson C.A. Nascimento
PublisherSpringer Verlag
Pages323-342
Number of pages20
ISBN (Print)9783319458700
DOIs
StatePublished - 2016
Event19th Annual International Conference on Information Security, ISC 2016 - Honolulu, United States
Duration: Sep 3 2016Sep 6 2016

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume9866 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Other

Other19th Annual International Conference on Information Security, ISC 2016
Country/TerritoryUnited States
CityHonolulu
Period9/3/169/6/16

All Science Journal Classification (ASJC) codes

  • Theoretical Computer Science
  • General Computer Science

Fingerprint

Dive into the research topics of 'Generalized dynamic opaque predicates: A new control flow obfuscation method'. Together they form a unique fingerprint.

Cite this