Search Engine Optimization
268 views
Robots.txt
Quick Definition
Text file instructing crawlers which pages to crawl
Full Definition
A text file instructing web crawlers which pages or sections of a site to crawl or skip.
Examples
crawl budget management, blocking pages, search engine directives
Related Terms
technical-seo
xml-sitemap
crawlability
More Search Engine Optimization Terms
Keyword Research
Discovering and analyzing search terms for content strategy
Largest Contentful Paint
Core Web Vitals metric for largest content element render time
XML Sitemap
XML file listing URLs for search engine crawling
Crawl Budget
Number of pages search engines crawl on a site
Featured Snippet
Summary answer at top of search results from a webpage
Title Tag
HTML element specifying page title for search and browsers