Toward detecting compromised mapreduce workers through log analysis

MapReduce is a framework for performing data intensive computations in parallel on commodity computers. When MapReduce is carried out in distributed settings, users maintain very little control over these computations, causing several security and privacy concerns. MapReduce activities may be subverted or compromised by malicious or cheating nodes. In this paper, we focus on the analysis and detection of attacks launched by malicious or mis configured nodes, which may tamper with the ordinary functions of the MapReduce framework. Our goal is to investigate the extent to which integrity and correctness of computation in a MapReduce environments can be verified while introducing no modifications on the original MapReduce operations or introductions of extra operations, neither computational nor cryptographic. We identify a number of data and computation integrity checks against aggregated low-level system traces and Hadoop logs, correlated with one another to obtain insights on the operations being performed by nodes. This information is then matched against system and program invariants to effectively detect malicious activities, from lazy nodes to nodes changing input/output or completing different computations.