robots.txt
Web crawler permission file
Robots.txt is a text file placed in a website's root directory that instructs web crawlers which pages they may or may not access.
Search engines check this file before indexing a site: The robots.txt blocked Google from crawling the admin pages.
It's a convention rather than enforcement—well-behaved bots respect it, malicious ones ignore it.
Those who move prone
- disallowRefuse to permit
- allowGrant permission for
- sitemapWebsite's page directory
- directivesFormal commands or instructions
spider · exclusion · standard · parsing · rules · syntax · wildcard · host
Systematic organization for retrieval
- search enginesInternet lookup tools
- visibilityRange of clear sight
- rankingOrdered position assignment
- organic trafficUnpaid search-driven visitors
SERP · webmaster · discoverability · optimization · deindexing · canonical · metadata
Arrangement of parts
- root directoryTopmost filesystem location
- serverOne who attends tables
- deploymentOrganized distribution of resources
- validationConfirming truth or accuracy
testing · syntax errors · debugging · access control · hosting · web development · maintenance · admin
Rubbing away material
- blockingImpeding forward progress
- botsAutomated software agents
- protectionGuarding against harm
- spam botsAutomated mass-messaging programs
harvesting · vulnerability · honeypot · bad actors · unauthorized access · exposure · obfuscation
crawlers
Those who move prone
directives · spider · exclusion · standard · parsing · rules · syntax · wildcard · host · fetch
indexing
Systematic organization for retrieval
- search enginesInternet lookup tools
- visibilityRange of clear sight
- rankingOrdered position assignment
organic traffic · SERP · webmaster · discoverability · optimization · deindexing · canonical · metadata
configuration
Arrangement of parts
- root directoryTopmost filesystem location
- serverOne who attends tables
- deploymentOrganized distribution of resources
validation · testing · syntax errors · debugging · access control · hosting · web development · maintenance · admin
scraping
Rubbing away material
- blockingImpeding forward progress
- botsAutomated software agents
- protectionGuarding against harm
spam bots · harvesting · vulnerability · honeypot · bad actors · unauthorized access · exposure · obfuscation
- web toolsbrowser, site builder, HTML editor, URL shortener, browser extension
- SEO ToolsMajestic, meta tags, structured data
- Webmasteringwebsite, hosting, domain name, HTML, CSS
- deindexingdeindexing, delisting, removal, unlisting, purging
- text filestxt, readme, log file, log, markdown
- search technologysearch engines, search results, indexing, queries, ranking