A cloud-based framework for large-scale log mining through Apache Spark And Elasticsearch

Yun Li, Yongyao Jiang, Juan Gu, Mingyue Lu, Manzhu Yu, Edward M. Armstrong, Thomas Huang, David Moroni, Lewis J. McGibbney, Greguska Frank, Chaowei Yang

Research output: Contribution to journalArticlepeer-review

6 Scopus citations


The volume, variety, and velocity of different data, e.g., simulation data, observation data, and social media data, are growing ever faster, posing grand challenges for data discovery. An increasing trend in data discovery is to mine hidden relationships among users and metadata from the web usage logs to support the data discovery process. Web usage log mining is the process of reconstructing sessions from raw logs and finding interesting patterns or implicit linkages. The mining results play an important role in improving quality of search-related components, e.g., ranking, query suggestion, and recommendation. While researches were done in the data discovery domain, collecting and analyzing logs efficiently remains a challenge because (1) the volume of web usage logs continues to grow as long as users access the data; (2) the dynamic volume of logs requires on-demand computing resources for mining tasks; (3) the mining process is compute-intensive and time-intensive. To speed up the mining process, we propose a cloud-based log-mining framework using Apache Spark and Elasticsearch. In addition, a data partition paradigm, logPartitioner, is designed to solve the data imbalance problem in data parallelism. As a proof of concept, oceanographic data search and access logs are chosen to validate performance of the proposed parallel log-mining framework.

Original languageEnglish (US)
Article number1114
JournalApplied Sciences (Switzerland)
Issue number6
StatePublished - 2019

All Science Journal Classification (ASJC) codes

  • Materials Science(all)
  • Instrumentation
  • Engineering(all)
  • Process Chemistry and Technology
  • Computer Science Applications
  • Fluid Flow and Transfer Processes


Dive into the research topics of 'A cloud-based framework for large-scale log mining through Apache Spark And Elasticsearch'. Together they form a unique fingerprint.

Cite this