Technical

Crawler

A program that systematically browses the World Wide Web in order to create an index of data.

Back to Glossary

Technical Terms

Definition

A program that systematically browses the World Wide Web in order to create an index of data.

Examples

Googlebot
Bingbot
Web crawler

Related terms

Indexing
The process by which search engines add web pages to their database of searchable content.

More Technical terms

6 additional terms in this category

Structured Data Linked Data Microdata RDFa Semantic Web RDF

Schema types

Person — Individual people
Organization — Companies and groups
Product — Items for sale
Event — Concerts and conferences
Article — Blog posts and news
Recipe — Cooking instructions

View all schema types