Robots.txt

Robots.txt Rules

Crawler Rules
All
  • Allow: /
  • Disallow: /dl/
  • Disallow: /report/
  • Disallow: /print/
  • Disallow: /raw/
  • Disallow: /embed/
  • Disallow: /embed_iframe/
  • Disallow: /embed_js/
  • Disallow: /message/
  • Disallow: /clone/
  • Disallow: /user/search-self
  • Disallow: /api/
  • Disallow: /rating/
  • Disallow: /user/profile
ia_archiver
  • Allow: /
  • Disallow: /

Sitemaps

Priority Location
Nothing to show.

Raw Text

User-agent: *
Disallow: /dl/
Disallow: /report/
Disallow: /print/
Disallow: /raw/
Disallow: /embed/
Disallow: /embed_iframe/
Disallow: /embed_js/
Disallow: /message/
Disallow: /clone/
Disallow: /user/search-self
Disallow: /api/
Disallow: /rating/
Disallow: /user/profile

User-agent: ia_archiver
Disallow: /