TY - GEN
T1 - Generalized dynamic opaque predicates
T2 - 19th Annual International Conference on Information Security, ISC 2016
AU - Xu, Dongpeng
AU - Ming, Jiang
AU - Wu, Dinghao
N1 - Publisher Copyright:
© Springer International Publishing Switzerland 2016.
PY - 2016
Y1 - 2016
N2 - Opaque predicate obfuscation, a low-cost and stealthy control flow obfuscation method to introduce superfluous branches, has been demonstrated to be effective to impede reverse engineering efforts and broadly used in various areas of software security. Conventional opaque predicates typically rely on the invariant property of well-known number theoretic theorems, making them easy to be detected by the dynamic testing and formal semantics techniques. To address this limitation, previous work has introduced the idea of dynamic opaque predicates, whose values may vary in different runs. However, the systematical design and evaluation of dynamic opaque predicates are far from mature. In this paper, we generalize the concept and systematically develop a new control flow obfuscation scheme called generalized dynamic opaque predicates. Compared to the previous work, our approach has two distinct advantages: (1) We extend the application scope by automatically transforming more common program structures (e.g., straight-line code, branch, and loop) into dynamic opaque predicates; (2) Our system design does not require that dynamic opaque predicates to be strictly adjacent, which is more resilient to the deobfuscation techniques. We have developed a prototype tool based on LLVM IR and evaluated it by obfuscating the GNU core utilities. Our experimental results show the efficacy and generality of our method. In addition, the comparative evaluation demonstrates that our method is resilient to the latest formal program semantics-based opaque predicate detection method.
AB - Opaque predicate obfuscation, a low-cost and stealthy control flow obfuscation method to introduce superfluous branches, has been demonstrated to be effective to impede reverse engineering efforts and broadly used in various areas of software security. Conventional opaque predicates typically rely on the invariant property of well-known number theoretic theorems, making them easy to be detected by the dynamic testing and formal semantics techniques. To address this limitation, previous work has introduced the idea of dynamic opaque predicates, whose values may vary in different runs. However, the systematical design and evaluation of dynamic opaque predicates are far from mature. In this paper, we generalize the concept and systematically develop a new control flow obfuscation scheme called generalized dynamic opaque predicates. Compared to the previous work, our approach has two distinct advantages: (1) We extend the application scope by automatically transforming more common program structures (e.g., straight-line code, branch, and loop) into dynamic opaque predicates; (2) Our system design does not require that dynamic opaque predicates to be strictly adjacent, which is more resilient to the deobfuscation techniques. We have developed a prototype tool based on LLVM IR and evaluated it by obfuscating the GNU core utilities. Our experimental results show the efficacy and generality of our method. In addition, the comparative evaluation demonstrates that our method is resilient to the latest formal program semantics-based opaque predicate detection method.
UR - http://www.scopus.com/inward/record.url?scp=84988446575&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84988446575&partnerID=8YFLogxK
U2 - 10.1007/978-3-319-45871-7_20
DO - 10.1007/978-3-319-45871-7_20
M3 - Conference contribution
AN - SCOPUS:84988446575
SN - 9783319458700
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 323
EP - 342
BT - Information Security - 19th International Conference, ISC 2016, Proceedings
A2 - Bishop, Matt
A2 - Nascimento, Anderson C.A.
PB - Springer Verlag
Y2 - 3 September 2016 through 6 September 2016
ER -