Robots.txt

Robots.txt Rules

Crawler Rules
All
  • Allow: /
  • Disallow: /admin/
  • Disallow: /cpresources/
  • Disallow: /vendor/
  • Disallow: /.env
  • Disallow: /docs/3.x/fr/
  • Disallow: /index.php/actions/
  • Disallow: /cdn-cgi/
  • Disallow: /*/page/*/json
Google News
  • Allow: /
  • Disallow: /shareables/

Sitemaps

Priority Location
1https://sitemaps.thenextweb.com/sitemap_index.xml

Raw Text

User-agent: *
Disallow: /admin/*
Disallow: /cpresources/
Disallow: /vendor/
Disallow: /.env
Disallow: /docs/3.x/fr/
Disallow: /index.php/actions/
Disallow: /cdn-cgi/
Disallow: /*/page/*/json

User-agent: googlebot-news
Disallow: /shareables/

Sitemap: https://sitemaps.thenextweb.com/sitemap_index.xml