Crawler
A program that systematically browses the World Wide Web in order to create an index of data.
Examples
- Googlebot
- Bingbot
- Web crawler
Related terms
- Indexing
- The process by which search engines add web pages to their database of searchable content.