robots.txt tells web crawlers
amitmund
June 01, 2026
The rebots.txt
# Allow Google
User-agent: Googlebot
Allow: /
# Allow Bing
User-agent: Bingbot
Allow: /
# Block everyone else
User-agent: *
Disallow: /