WebCrawler
Automated site explorer
WebCrawler is a program that systematically browses the internet, following links from page to page to index content for search engines.
Also called a spider or bot, it builds the databases that make web searches possible: Google's webcrawler indexes billions of pages.
The term also refers to an early search engine launched in 1994.
Web-spinning arachnid
- botAutomated software agent
- sitemapWebsite's page directory
- archiveStored historical records
- catalogOrdered list of items
database · directory · repository · index · URL · link · page · domain
Written words for performance
- botAutomated software agent
- scheduledArranged for specific time
- batchQuantity produced together
- taskWork to be done
routine · agent · autonomous · loop · queue · multithreading · asynchronous · pipeline
Tool for removing buildup
- harvestingGathering mature crops
- extractionAct of removing
- parsingGrammatical structure extraction
- dataFacts for analysis
HTML · DOM · selector · selenium · requests · curl · fetch · content
spider
Web-spinning arachnid
catalog · database · directory · repository · index · URL · link · page · domain · depth-first
script
Written words for performance
task · routine · agent · autonomous · loop · queue · multithreading · asynchronous · pipeline · workflow
scraper
Tool for removing buildup
- harvestingGathering mature crops
- extractionAct of removing
- parsingGrammatical structure extraction
data · HTML · DOM · selector · selenium · requests · curl · fetch · content · text
Dominant web search provider
- BaiduChinese search engine giant
- DuckDuckGoPrivacy-focused search engine
- YahooNineties web services pioneer
Yandex · ranking · SEO · PageRank · SERP · keywords · backlink · sitemap · indexation · deindexing
HTTP
Web data transmission standard
header · redirect · robots.txt · API · REST · POST · cookies · session · authentication · timeout