web scraping

Harvesting site data

Web scraping is the automated extraction of data from websites, using software to collect information that would be tedious to gather manually.

A program might scrape product prices from retail sites, contact information from directories, or headlines from news sources.

They built a web scraper to monitor competitor pricing.

The practice occupies a legal gray area—sites may prohibit it in their terms of service, and aggressive scraping can overwhelm servers.

extraction

crawler

Something that inches along

botAutomated software agent
parserSyntax-interpreting program
spiderWeb-spinning arachnid

harvesting · Selenium · DOM · data collection · pagination · throttling · sitemap · structured data

legal

robots.txt

Crawler directive file

terms of serviceBinding usage conditions
copyrightExclusive reproduction rights
fair useLimited copying permitted legally

permission · GDPR · CCPA · cease and desist · attribution · commercial use

technical

response

Answer to stimulus

headersTop-of-page identifiers
cookiesSmall sweet baked goods
sessionScheduled time for activity

proxy · Firefox · AJAX · authentication · login · timeout

evasion

CAPTCHA

Automated humanness verification

CloudflareWeb security and performance company
fingerprintingRecording unique digit patterns
detectionAct of finding something

blocking · ban · IP ban · delay · honeypot · stealth · bypass

Categories

OSINTgeolocation, reverse image search, public records, digital footprint, IP tracking
computingcomputer, computers, software, hardware, code
digital technologyinternet, software, streaming, computers, CGI
data collectionsurveys, interviews, questionnaires, forms, polls
digital researchsearch engine, data analysis, database, digital libraries, search
data extractionexport, parsing, crawling, spiders, OCR

Opposites

data collection method

web scrapingcopy and paste

Replicate content elsewhere

data access approach

web scrapingRSS feed

Automated content subscription format

information flow direction

web scrapingweb publishing

Creating internet content

content creation · form filling · blogging · content management

Family

scrape, scraps, scraping, scraped, scrappy, scrapbook, scraper, data mining, data extraction

Familiarity: Top 500,000
Profile: Used worldwide · Rated G