Robots.txt
A text file that tells web crawlers which pages or files the crawler can or can't request from your site.
Examples
- User-agent: *
- Disallow: /private/
- Allow: /public/
Related terms
- Crawler
- A program that systematically browses the World Wide Web in order to create an index of data.
- Sitemap
- A file that provides information about the pages, videos, and other files on your site, and the relationships between them.
- Indexing
- The process by which search engines add web pages to their database of searchable content.