spidering
Automated link following
Spidering is the automated process of systematically browsing the web to index content.
Search engines use spidering programs (also called crawlers or bots) to discover and catalog web pages by following links from site to site: Google's spidering indexed the new pages within hours.
The term evokes a spider traversing its web.
Something that inches along
- botAutomated software agent
- scraperTool for removing buildup
- indexingSystematic organization for retrieval
- robots.txtCrawler directive file
sitemap · hyperlinks · scraping · harvesting · parsing · depth-first · breadth-first · Wayback Machine
Spider building circular webs
- silkFine woven insect fiber
- spinneretSilk-producing appendage
- trapdoorHinged floor panel
- cobwebAbandoned spider silk
ballooning · dragline · egg sac · molt · spinnerets · funnel web
Study of spatial arrangement
- nodesPoints of connection
- meshInterlocking open weave
- routingPath selection process
- distributedDivided and given out
crawler
Something that inches along
- botAutomated software agent
- scraperTool for removing buildup
- indexingSystematic organization for retrieval
robots.txt · sitemap · hyperlinks · scraping · harvesting · parsing · depth-first · breadth-first · Wayback Machine · curl
orb-weaver
Spider building circular webs
cobweb · ballooning · dragline · egg sac · molt · spinnerets · funnel web
topology
Study of spatial arrangement
distributed · spanning · traceroute · adjacency · path finding