Graph-based seed selection for web-scale crawlers

Shuyi Zheng; Pavel Dmitriev; C. Lee Giles

doi:10.1145/1645953.1646277

Graph-based seed selection for web-scale crawlers

Shuyi Zheng, Pavel Dmitriev, C. Lee Giles

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

13 Scopus citations

Abstract

One of the most important steps in web crawling is determining the starting points, or seed selection. This paper identifies and explores the problem of seed selection in web-scale incremental crawlers. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a repository with more "good" and less "bad" pages. We propose a graph-based framework for crawler seed selection, and present several algorithms within this framework. Evaluation on real web data showed significant improvements over heuristic seed selection approaches.

Original language	English (US)
Title of host publication	ACM 18th International Conference on Information and Knowledge Management, CIKM 2009
Pages	1967-1970
Number of pages	4
DOIs	https://doi.org/10.1145/1645953.1646277
State	Published - 2009
Event	ACM 18th International Conference on Information and Knowledge Management, CIKM 2009 - Hong Kong, China Duration: Nov 2 2009 → Nov 6 2009

Publication series

Name	International Conference on Information and Knowledge Management, Proceedings

Other

Other	ACM 18th International Conference on Information and Knowledge Management, CIKM 2009
Country/Territory	China
City	Hong Kong
Period	11/2/09 → 11/6/09

All Science Journal Classification (ASJC) codes

Decision Sciences(all)
Business, Management and Accounting(all)

Access to Document

10.1145/1645953.1646277

Cite this

@inproceedings{d3262bce890048b8877d211dca1037e5,

title = "Graph-based seed selection for web-scale crawlers",

abstract = "One of the most important steps in web crawling is determining the starting points, or seed selection. This paper identifies and explores the problem of seed selection in web-scale incremental crawlers. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a repository with more {"}good{"} and less {"}bad{"} pages. We propose a graph-based framework for crawler seed selection, and present several algorithms within this framework. Evaluation on real web data showed significant improvements over heuristic seed selection approaches.",

author = "Shuyi Zheng and Pavel Dmitriev and Giles, {C. Lee}",

year = "2009",

doi = "10.1145/1645953.1646277",

language = "English (US)",

isbn = "9781605585123",

series = "International Conference on Information and Knowledge Management, Proceedings",

pages = "1967--1970",

booktitle = "ACM 18th International Conference on Information and Knowledge Management, CIKM 2009",

note = "ACM 18th International Conference on Information and Knowledge Management, CIKM 2009 ; Conference date: 02-11-2009 Through 06-11-2009",

}

Zheng, S, Dmitriev, P & Giles, CL 2009, Graph-based seed selection for web-scale crawlers. in ACM 18th International Conference on Information and Knowledge Management, CIKM 2009. International Conference on Information and Knowledge Management, Proceedings, pp. 1967-1970, ACM 18th International Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, 11/2/09. https://doi.org/10.1145/1645953.1646277

Graph-based seed selection for web-scale crawlers. / Zheng, Shuyi; Dmitriev, Pavel; Giles, C. Lee.
ACM 18th International Conference on Information and Knowledge Management, CIKM 2009. 2009. p. 1967-1970 (International Conference on Information and Knowledge Management, Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Graph-based seed selection for web-scale crawlers

AU - Zheng, Shuyi

AU - Dmitriev, Pavel

AU - Giles, C. Lee

PY - 2009

Y1 - 2009

N2 - One of the most important steps in web crawling is determining the starting points, or seed selection. This paper identifies and explores the problem of seed selection in web-scale incremental crawlers. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a repository with more "good" and less "bad" pages. We propose a graph-based framework for crawler seed selection, and present several algorithms within this framework. Evaluation on real web data showed significant improvements over heuristic seed selection approaches.

AB - One of the most important steps in web crawling is determining the starting points, or seed selection. This paper identifies and explores the problem of seed selection in web-scale incremental crawlers. We argue that seed selection is not a trivial but very important problem. Selecting proper seeds can increase the number of pages a crawler will discover, and can result in a repository with more "good" and less "bad" pages. We propose a graph-based framework for crawler seed selection, and present several algorithms within this framework. Evaluation on real web data showed significant improvements over heuristic seed selection approaches.

UR - http://www.scopus.com/inward/record.url?scp=74549193422&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=74549193422&partnerID=8YFLogxK

U2 - 10.1145/1645953.1646277

DO - 10.1145/1645953.1646277

M3 - Conference contribution

AN - SCOPUS:74549193422

SN - 9781605585123

T3 - International Conference on Information and Knowledge Management, Proceedings

SP - 1967

EP - 1970

BT - ACM 18th International Conference on Information and Knowledge Management, CIKM 2009

T2 - ACM 18th International Conference on Information and Knowledge Management, CIKM 2009

Y2 - 2 November 2009 through 6 November 2009

ER -

Graph-based seed selection for web-scale crawlers

Abstract

Publication series

Other

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this