Efficient Crawling Through URL Ordering

In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without.

• Publication year

  1998

• Venue

  Comput. Networks

• Publication date

  1998-04-01

• Fields of study

  Computer Science

• Identifiers
  DOI 10.1016/S0169-7552(98)00108-1
• External record

  Open on Semantic Scholar

• Source metadata

  Semantic Scholar

• No claims are published for this paper.

• No concepts are published for this paper.

• The PageRank Citation Ranking : Bringing Order to the Web

  1999cited by this paper
• The Anatomy of a Large-Scale Hypertextual Web Search Engine

  1998cited by this paper
• The Quest for Correct Information on the Web: Hyper Search Engines

  1997cited by this paper
• Preserving the Internet

  1997cited by this paper
• Silk from a sow's ear: extracting usable structures from the Web

  1996cited by this paper
• A World Wide Web Resource Discovery System

  1995cited by this paper
• Robots in the Web : threat or treat ?

  1995cited by this paper
• Finding What People Want : Experiences with the WebCrawler

  1994cited by this paper
• Automatic text processing

  1988cited by this paper
• Introduction to Modern Information Retrieval

  1983cited by this paper

• Craw4LLM: Efficient Web Crawling for LLM Pretraining

  2025influential citation
• Neural Prioritisation for Web Crawling

  2025cites this paper
• Pre-Crawl Prioritization and Seed Classification for Large-Scale Web Archiving

  2025cites this paper
• Faster Graph Algorithms Through DAG Compression

  2024cites this paper
• Timely, Granular, and Actionable: Designing a Social Listening Platform for Public Health 3.0

  2024cites this paper
• CRATOR: a Dark Web Crawler

  2024cites this paper
• Focused crawler based on concept context graph

  2023cites this paper
• Summarizing Web Archive Corpora via Social Media Storytelling by Automatically Selecting and Visualizing Exemplars

  2023cites this paper
• Predicting Aging-Related Bugs Using Network Analysis on Aging-Related Dependency Networks

  2023cites this paper
• Adaptive Behavior-aware Driven Intelligent Method for Detecting News Webpage Structure

  2023cites this paper
• From Web Catalogs to Google: A Retrospective Study of Web Search Engines Sustainable Development

  2023cites this paper
• Applying Web Crawler Technologies for Compiling Parallel Corpora as one Stage of Natural Language Processing

  2022cites this paper
• Real-Time Automatic Translation Algorithm for Chinese Subtitles in Media Playback Using Knowledge Base

  2022cites this paper
• Crawling Deep Web Data Based on Three-stage Template

  2022cites this paper
• Analysis of Semantic and Non-Semantic crawlers

  2021cites this paper
• Retrieving Black-box Optimal Images from External Databases

  2021cites this paper
• Garbage, Glitter, or Gold: Assigning Multi-Dimensional Quality Scores to Social Media Seeds for Web Archive Collections

  2021cites this paper
• On the feasibility of crawling-based attacks against recommender systems

  2021cites this paper
• Intelligent Machine Translation of Chinese English News in the Context of Communication Observations and a Priori Modeling

  2021cites this paper
• KnowCrawler: AI Classification Cloud-Driven Framework for Web Crawling Using Collective Knowledge

  2021cites this paper
• Physical Fundamentals of Biomaterials Surface Electrical Functionalization

  2020influential citation
• Summary of web crawler technology research

  2020cites this paper
• Bootstrapping Web Archive Collections From Micro-Collections in Social Media

  2020cites this paper
• Adversarial Bandits Policy for Crawling Commercial Web Content

  2020cites this paper
• Crawling the German Health Web: Exploratory Study and Graph Analysis

  2020cites this paper
• Modeling Updates of Scholarly Webpages Using Archived Data

  2020cites this paper
• Understanding the limitations of network online learning

  2020cites this paper
• EvenDB: optimizing key-value storage for spatial locality

  2020cites this paper
• Big Enough to Care Not Enough to Scare! Crawling to Attack Recommender Systems

  2020cites this paper
• Crawling the German Health Web: Exploratory Study and Graph Analysis (Preprint)

  2020cites this paper
• Design and development of an automated web crawler used for building image databases

  2019cites this paper
• Comparative Analysis of Web Crawling Algorithms for Improvement in Web Crawler

  2019cites this paper
• An Approach for Focused Crawler to Harvest Digital Academic Documents in Online Digital Libraries

  2019cites this paper
• Implementation of hybrid P2P networking distributed web crawler using AWS for smart work news big data

  2019cites this paper
• Predictive Crawling for Commercial Web Content

  2019cites this paper
• Towards Efficient Crawling of Georeferenced Documents from Location-based Social Networks

  2019cites this paper
• Using Web Pages Dynamicity to Prioritise Web Crawling

  2019influential citation
• Collecting Comparable Corpora

  2019cites this paper
• Perbandingan Performansi Terhadap Algoritma Breadth First Search (BFS) dan Depth First Search (DFS) pada Web Crawler

  2019cites this paper
• An Algorithm Integrating and Updating Rights Management Information on Public Domain Images

  2019cites this paper
• Automatic Summarization and Keyword Extraction from Web Page or Text File

  2019cites this paper
• Predicting New Web Pages on the World Wide Web Using Topological Features

  2019cites this paper
• Efficiency of Web Crawling for Geotagged Image Retrieval

  2019cites this paper
• An automated methodology for rapid information extraction from large drilling datasets

  2019cites this paper
• Enhance the Security and Improve the Performance of Web Crawlers using Web Crawling Algorithms

  2019cites this paper
• An ontology-driven multimedia focused crawler based on linked open data and deep learning techniques

  2019cites this paper
• Expectation maximisation on unsupervised web mined data using probability latent semantic analysis (PLSA) algorithm

  2019influential citation
• Crawling and Classification Strategies for Generating a Multi-Language Corpus of Sign Language Video

  2019cites this paper
• A Survey about Algorithms Utilized by Focused Web Crawler

  2018cites this paper
• Smart Approach to Crawl Web Interfaces Using a Two Stage Framework of Crawler

  2018cites this paper
• Allocation of Graph Jobs in Geo-Distributed Cloud Networks

  2018cites this paper
• An intelligent web spider for online e-commerce data extraction

  2018influential citation
• Change-Aware Scheduling for Effectively Updating Linked Open Data Caches

  2018cites this paper
• Google Hummingbird : Technique to Humanize A Search

  2018cites this paper
• Target oriented network intelligence collection: effective exploration of social networks

  2018cites this paper
• The Anatomy of A Large-Scale Hyper Textualweb Search Engine

  2018cites this paper
• Applying topic modeling techniques to degraded texts: Spanish historical press during the Transición (1977-1982)

  2018cites this paper
• Intelligent Crawler Engine On Cloud Computing Infrastructure

  2018cites this paper
• Unsupervised Domain Ranking in Large-Scale Web Crawls

  2018cites this paper
• A Structure-Oriented Unsupervised Crawling Strategy for Social Media Sites

  2018cites this paper
• Power-Aware Allocation of Graph Jobs in Geo-Distributed Cloud Networks

  2018cites this paper
• Web Crawling and Processing with Limited Resources for Business Intelligence and Analytics Applications

  2018cites this paper
• Domain Adaptation for Statistical Machine Translation

  2018cites this paper
• Integration and Analysis of Agricultural Market Information Based on Web Mining

  2018cites this paper
• AGEN CRAWLER ALAMAT EMAIL MENGGUNAKAN METODE BREADTH-FIRST CRAWLING

  2018cites this paper
• The investigation of the possibility of automated collection of information in the hidden segment of the Internet

  2018cites this paper
• Information Retrieval and Search Engines

  2018cites this paper
• Stream-based live public opinion monitoring approach with adaptive probabilistic topic model

  2018cites this paper
• Machine Learning for Text

  2018cites this paper
• Closed Walk Sampler: An Efficient Method for Estimating Eigenvalues of Large Graphs

  2018influential citation
• Location-based web crawler for geotagged images retrieval

  2018cites this paper
• RaRE: Social Rank Regulated Large-scale Network Embedding

  2018cites this paper
• Latte: Large-Scale Lateral Movement Detection

  2018cites this paper
• Focused Web Crawling

  2018cites this paper
• Sentiment analysis and resources for informal Arabic text on social media

  2018cites this paper
• Automatically assembling a full census of an academic field

  2018cites this paper
• Hadoop and Hive Inspecting Maintenance of Mobile Application for Groceries Expenditure

  2018cites this paper
• Priority Queue Based Estimation of Importance of Web Pages for Web Crawlers

  2017cites this paper
• ε - WGX: Adaptive Edge Probing for Enhancing Incomplete Networks

  2017cites this paper
• A Novel Approach on Focused Crawling with Anchor Text

  2017cites this paper
• Closed walk sampler: An efficient method for estimating the spectral radius of large graphs

  2017influential citation
• Extracting Textual Information from Google Using Wrapper Class

  2017cites this paper
• A Framework for Event Information Extraction from Chinese News Online

  2017cites this paper
• Knowledge Extraction Through Page Rank Using Web-Mining Techniques for E-Business: A Review

  2017cites this paper
• Framework for Scalable File System Metadata Crawling and Differencing

  2017cites this paper
• A survey of Web crawlers for information retrieval

  2017cites this paper
• GENERATING QUERIES TO CRAWL HIDDEN WEB USING KEYWORD SAMPLING AND RANDOM FOREST CLASSIFIER

  2017cites this paper
• School of Computing 2011-06-09 Measuring Design Metrics In Websites

  2017cites this paper
• Towards Optimal Strategy for Adaptive Probing in Incomplete Networks

  2017cites this paper
• Emerging opportunities in Domain Specific Search

  2016cites this paper
• Detecting off-topic pages within TimeMaps in Web archives

  2016cites this paper
• Focussed crawling of environmental Web resources based on the combination of multimedia evidence

  2016cites this paper
• An Investigation of the Use of Innovative Biology-Based Computational Intelligence in Ubiquitous Robotics Systems: Data Mining Perspective

  2016cites this paper
• Targeted cyber-attacks: Unveiling target reconnaissance strategy via Social Networks

  2016cites this paper
• Modified Breadth-first order-based link categorization for finding financial statement documents

  2016cites this paper
• Parallelizing the extraction of fresh information from online social networks

  2016cites this paper
• An Experimental evaluation of Adaptive Real Time Web Crawler

  2016influential citation
• W-tree

  2016cites this paper
• A Scalable P2P RIA Crawling System with Fault Tolerance

  2016cites this paper
• Learning to query: Focused web page harvesting for entity aspects

  2016cites this paper