Robots.txt

Robots.txt Rules

Crawler Rules
All
  • Allow: /
  • Disallow: /*/actions_panel
  • Disallow: /users/auth/twitter
  • Disallow: /users/auth/github
  • Disallow: /report-abuse?url=
  • Disallow: /connect/@
  • Disallow: /search?q=
  • Disallow: /search/?q=
  • Disallow: /search/feed_content?
  • Disallow: /listings*?q=
  • Disallow: /mod/
  • Disallow: /mod?
  • Disallow: /admin/
  • Disallow: /reactions?
  • Disallow: /async_info/base_data
  • Disallow: /ahoy/
  • Disallow: /auth_pass/

Sitemaps

Priority Location
1https://dev.to/sitemap-index.xml

Raw Text

User-agent: *
Disallow: /*/actions_panel*
Disallow: /users/auth/twitter*
Disallow: /users/auth/github*
Disallow: /report-abuse?url=*
Disallow: /connect/@*
Disallow: /search?q=*
Disallow: /search/?q=*
Disallow: /search/feed_content?*
Disallow: /listings*?q=*
Disallow: /mod/*
Disallow: /mod?*
Disallow: /admin/*
Disallow: /reactions?*
Disallow: /async_info/base_data
Disallow: /ahoy/*
Disallow: /auth_pass/*

Sitemap: https://dev.to/sitemap-index.xml