web crawling
Indexing the internet
Web crawling is the automated process of systematically browsing the internet to index content.
Search engines use web crawlers (also called spiders or bots) to discover and catalog web pages, following links from one page to the next.
Google's web crawling covers billions of pages.
The practice raises questions about server load and data scraping ethics.
- spiderWeb-spinning arachnid
- search engineTool for locating websites
- scrapingRubbing away material
- data miningExtracting insights from datasets
- botAutomated software agent
indexing · page rank · hyperlinks · HTML · parsing · sitemap · robots.txt · harvesting · URLs · depth-first · breadth-first · throttling · fetch · cache · DOM · headless · browser · infinite scroll
- Web thingsspider web, cobweb, funnel web, web page, website
- search technologysearch engines, search results, indexing, queries, ranking
- web archivesWayback Machine, snapshots, crawlers, saved pages, capture
- web researchdata collection, web scraping, online surveys, internet research, web analytics
- networkingrouter, routers, LAN, WAN, switches
- crawl typescrab walk, hands and knees