TY - GEN
T1 - Opening the blackbox of virustotal
T2 - 19th ACM Internet Measurement Conference, IMC 2019
AU - Peng, Peng
AU - Yang, Limin
AU - Song, Linhai
AU - Wang, Gang
N1 - Publisher Copyright:
© 2019 Association for Computing Machinery. ACM ISBN 978-1-4503-6948-0/19/10...$15.00
PY - 2019/10/21
Y1 - 2019/10/21
N2 - Online scan engines such as VirusTotal are heavily used by researchers to label malicious URLs and files. Unfortunately, it is not well understood how the labels are generated and how reliable the scanning results are. In this paper, we focus on VirusTotal and its 68 third-party vendors to examine their labeling process on phishing URLs. We perform a series of measurements by setting up our own phishing websites (mimicking PayPal and IRS) and submitting the URLs for scanning. By analyzing the incoming network traffic and the dynamic label changes at VirusTotal, we reveal new insights into how VirusTotal works and the quality of their labels. Among other things, we show that vendors have trouble flagging all phishing sites, and even the best vendors missed 30% of our phishing sites. In addition, the scanning results are not immediately updated to VirusTotal after the scanning, and there are inconsistent results between VirusTotal scan and some vendors' own scanners. Our results reveal the need for developing more rigorous methodologies to assess and make use of the labels obtained from VirusTotal.
AB - Online scan engines such as VirusTotal are heavily used by researchers to label malicious URLs and files. Unfortunately, it is not well understood how the labels are generated and how reliable the scanning results are. In this paper, we focus on VirusTotal and its 68 third-party vendors to examine their labeling process on phishing URLs. We perform a series of measurements by setting up our own phishing websites (mimicking PayPal and IRS) and submitting the URLs for scanning. By analyzing the incoming network traffic and the dynamic label changes at VirusTotal, we reveal new insights into how VirusTotal works and the quality of their labels. Among other things, we show that vendors have trouble flagging all phishing sites, and even the best vendors missed 30% of our phishing sites. In addition, the scanning results are not immediately updated to VirusTotal after the scanning, and there are inconsistent results between VirusTotal scan and some vendors' own scanners. Our results reveal the need for developing more rigorous methodologies to assess and make use of the labels obtained from VirusTotal.
UR - http://www.scopus.com/inward/record.url?scp=85074823687&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85074823687&partnerID=8YFLogxK
U2 - 10.1145/3355369.3355585
DO - 10.1145/3355369.3355585
M3 - Conference contribution
AN - SCOPUS:85074823687
T3 - Proceedings of the ACM SIGCOMM Internet Measurement Conference, IMC
SP - 478
EP - 485
BT - IMC 2019 - Proceedings of the 2019 ACM Internet Measurement Conference
PB - Association for Computing Machinery
Y2 - 21 October 2019 through 23 October 2019
ER -