Technical
Robots.txt
A text file that tells web crawlers which pages or files the crawler can or can't request from your site.
Back to Glossary
Definition
A text file that tells web crawlers which pages or files the crawler can or can't request from your site.
Examples
- User-agent: *
- Disallow: /private/
- Allow: /public/
Related terms
- Crawler
A program that systematically browses the World Wide Web in order to create an index of data.
- Sitemap
A file that provides information about the pages, videos, and other files on your site, and the relationships between them.
- Indexing
The process by which search engines add web pages to their database of searchable content.
More Technical terms
6 additional terms in this category
Schema types
- Person — Individual people
- Organization — Companies and groups
- Product — Items for sale
- Event — Concerts and conferences
- Article — Blog posts and news
- Recipe — Cooking instructions