Search Engine Optimization
134 views
Robots.txt
Quick Definition
Text file instructing crawlers which pages to crawl
Full Definition
A text file instructing web crawlers which pages or sections of a site to crawl or skip.
Examples
crawl budget management, blocking pages, search engine directives
Related Terms
technical-seo
xml-sitemap
crawlability
More Search Engine Optimization Terms
NAP Consistency
Ensuring consistent business info across all listings
Mobile-First Indexing
Google indexing primarily using mobile version of content
Off-Page SEO
Activities outside the website to improve rankings
Page Speed
How fast a web page loads and becomes interactive
Crawl Budget
Number of pages search engines crawl on a site
Backlinks
Links from other websites signaling authority