Measuring and modeling the label dynamics of online anti-malware engines

Shuofei Zhu, Jianjun Shi, Limin Yang, Boqin Qin, Ziyi Zhang, Linhai Song, Gang Wang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

90 Scopus citations


VirusTotal provides malware labels from a large set of anti-malware engines, and is heavily used by researchers for malware annotation and system evaluation. Since different engines often disagree with each other, researchers have used various methods to aggregate their labels. In this paper, we take a data-driven approach to categorize, reason, and validate common labeling methods used by researchers. We first survey 115 academic papers that use VirusTotal, and identify common methodologies. Then we collect the daily snapshots of VirusTotal labels for more than 14,000 files (including a subset of manually verified ground-truth) from 65 VirusTotal engines over a year. Our analysis validates the benefits of threshold-based label aggregation in stabilizing files' labels, and also points out the impact of poorly-chosen thresholds. We show that hand-picked “trusted” engines do not always perform well, and certain groups of engines are strongly correlated and should not be treated independently. Finally, we empirically show certain engines fail to perform in-depth analysis on submitted files and can easily produce false positives. Based on our findings, we offer suggestions for future usage of VirusTotal for data annotation.

Original languageEnglish (US)
Title of host publicationProceedings of the 29th USENIX Security Symposium
PublisherUSENIX Association
Number of pages18
ISBN (Electronic)9781939133175
StatePublished - 2020
Event29th USENIX Security Symposium - Virtual, Online
Duration: Aug 12 2020Aug 14 2020

Publication series

NameProceedings of the 29th USENIX Security Symposium


Conference29th USENIX Security Symposium
CityVirtual, Online

All Science Journal Classification (ASJC) codes

  • Computer Networks and Communications
  • Information Systems
  • Safety, Risk, Reliability and Quality


Dive into the research topics of 'Measuring and modeling the label dynamics of online anti-malware engines'. Together they form a unique fingerprint.

Cite this