Robots.txt

Robots.txt Rules

Crawler Rules
All
  • Allow: /
  • Allow: /news/directory
  • Allow: /booksrightsholders
  • Allow: /patents/about
  • Allow: /safebrowsing/diagnostic
  • Allow: /safebrowsing/report_error/
  • Allow: /safebrowsing/report_phish/
  • Allow: /profiles
  • Allow: /s2/profiles
  • Allow: /s2/photos
  • Allow: /s2/static
  • Allow: /accounts/o8/id
  • Allow: /toolkit/*.html
  • Disallow: /search
  • Disallow: /groups
  • Disallow: /images
  • Disallow: /catalogs
  • Disallow: /catalogues
  • Disallow: /news
  • Disallow: /nwshp
  • Disallow: /setnewsprefs?
  • Disallow: /index.html?
  • Disallow: /?
  • Disallow: /addurl/image?
  • Disallow: /pagead/
  • Disallow: /relpage/
  • Disallow: /relcontent
  • Disallow: /imgres
  • Disallow: /imglanding
  • Disallow: /keyword/
  • Disallow: /u/
  • Disallow: /univ/
  • Disallow: /cobrand
  • Disallow: /custom
  • Disallow: /advanced_group_search
  • Disallow: /googlesite
  • Disallow: /preferences
  • Disallow: /setprefs
  • Disallow: /swr
  • Disallow: /url
  • Disallow: /default
  • Disallow: /m?
  • Disallow: /m/?
  • Disallow: /m/blogs?
  • Disallow: /m/ig
  • Disallow: /m/images?
  • Disallow: /m/local?
  • Disallow: /m/movies?
  • Disallow: /m/news?
  • Disallow: /m/news/i?
  • Disallow: /m/place?
  • Disallow: /m/setnewsprefs?
  • Disallow: /m/search?
  • Disallow: /m/swmloptin?
  • Disallow: /m/trends
  • Disallow: /wml?
  • Disallow: /wml/?
  • Disallow: /wml/search?
  • Disallow: /xhtml?
  • Disallow: /xhtml/?
  • Disallow: /xhtml/search?
  • Disallow: /xml?
  • Disallow: /imode?
  • Disallow: /imode/?
  • Disallow: /imode/search?
  • Disallow: /jsky?
  • Disallow: /jsky/?
  • Disallow: /jsky/search?
  • Disallow: /pda?
  • Disallow: /pda/?
  • Disallow: /pda/search?
  • Disallow: /sprint_xhtml
  • Disallow: /sprint_wml
  • Disallow: /pqa
  • Disallow: /palm
  • Disallow: /gwt/
  • Disallow: /purchases
  • Disallow: /hws
  • Disallow: /bsd?
  • Disallow: /linux?
  • Disallow: /mac?
  • Disallow: /microsoft?
  • Disallow: /unclesam?
  • Disallow: /answers/search?q=
  • Disallow: /local?
  • Disallow: /local_url
  • Disallow: /froogle?
  • Disallow: /products?
  • Disallow: /products/
  • Disallow: /froogle_
  • Disallow: /product_
  • Disallow: /products_
  • Disallow: /print
  • Disallow: /books
  • Disallow: /bkshp?q=
  • Disallow: /patents?
  • Disallow: /patents/
  • Disallow: /scholar
  • Disallow: /complete
  • Disallow: /sponsoredlinks
  • Disallow: /videosearch?
  • Disallow: /videopreview?
  • Disallow: /videoprograminfo?
  • Disallow: /maps?
  • Disallow: /mapstt?
  • Disallow: /mapslt?
  • Disallow: /maps/stk/
  • Disallow: /maps/br?
  • Disallow: /mapabcpoi?
  • Disallow: /maphp?
  • Disallow: /places/
  • Disallow: /maps/place
  • Disallow: /help/maps/streetview/partners/welcome/
  • Disallow: /lochp?
  • Disallow: /center
  • Disallow: /ie?
  • Disallow: /sms/demo?
  • Disallow: /katrina?
  • Disallow: /blogsearch?
  • Disallow: /blogsearch/
  • Disallow: /blogsearch_feeds
  • Disallow: /advanced_blog_search
  • Disallow: /reader/
  • Disallow: /uds/
  • Disallow: /chart?
  • Disallow: /transit?
  • Disallow: /mbd?
  • Disallow: /extern_js/
  • Disallow: /calendar/feeds/
  • Disallow: /calendar/ical/
  • Disallow: /cl2/feeds/
  • Disallow: /cl2/ical/
  • Disallow: /coop/directory
  • Disallow: /coop/manage
  • Disallow: /trends?
  • Disallow: /trends/music?
  • Disallow: /notebook/search?
  • Disallow: /musica
  • Disallow: /musicad
  • Disallow: /musicas
  • Disallow: /musicl
  • Disallow: /musics
  • Disallow: /musicsearch
  • Disallow: /musicsp
  • Disallow: /musiclp
  • Disallow: /browsersync
  • Disallow: /call
  • Disallow: /archivesearch?
  • Disallow: /archivesearch/url
  • Disallow: /archivesearch/advanced_search
  • Disallow: /base/search?
  • Disallow: /base/reportbadoffer
  • Disallow: /base/s2
  • Disallow: /urchin_test/
  • Disallow: /movies?
  • Disallow: /codesearch?
  • Disallow: /codesearch/feeds/search?
  • Disallow: /wapsearch?
  • Disallow: /safebrowsing
  • Disallow: /reviews/search?
  • Disallow: /orkut/albums
  • Disallow: /jsapi
  • Disallow: /views?
  • Disallow: /c/
  • Disallow: /cbk
  • Disallow: /recharge/dashboard/car
  • Disallow: /recharge/dashboard/static/
  • Disallow: /translate_a/
  • Disallow: /translate_c
  • Disallow: /translate_f
  • Disallow: /translate_static/
  • Disallow: /translate_suggestion
  • Disallow: /profiles/me
  • Disallow: /s2/profiles/me
  • Disallow: /s2
  • Disallow: /transconsole/portal/
  • Disallow: /gcc/
  • Disallow: /aclk
  • Disallow: /cse?
  • Disallow: /cse/panel
  • Disallow: /cse/manage
  • Disallow: /tbproxy/
  • Disallow: /comparisonads/
  • Disallow: /imesync/
  • Disallow: /shenghuo/search?
  • Disallow: /support/forum/search?
  • Disallow: /reviews/polls/
  • Disallow: /hosted/images/
  • Disallow: /hosted/life/
  • Disallow: /ppob/?
  • Disallow: /ppob?
  • Disallow: /ig/add?
  • Disallow: /adwordsresellers
  • Disallow: /accounts/o8
  • Disallow: /topicsearch?q=
  • Disallow: /xfx7/
  • Disallow: /squared/api
  • Disallow: /squared/search
  • Disallow: /squared/table
  • Disallow: /toolkit/
  • Disallow: /qnasearch?
  • Disallow: /errors/
  • Disallow: /app/updates
  • Disallow: /sidewiki/entry/
  • Disallow: /quality_form?
  • Disallow: /labs/popgadget/search

Sitemaps

Priority Location
4http://www.google.com/hostednews/sitemap_index.xml
3http://www.google.com/ventures/sitemap_ventures.xml
2http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
1http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml

Raw Text

User-agent: *
Disallow: /search
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Disallow: /catalogues
Disallow: /news
Allow: /news/directory
Disallow: /nwshp
Disallow: /setnewsprefs?
Disallow: /index.html?
Disallow: /?
Disallow: /addurl/image?
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /imgres
Disallow: /imglanding
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /default
Disallow: /m?
Disallow: /m/?
Disallow: /m/blogs?
Disallow: /m/ig
Disallow: /m/images?
Disallow: /m/local?
Disallow: /m/movies?
Disallow: /m/news?
Disallow: /m/news/i?
Disallow: /m/place?
Disallow: /m/setnewsprefs?
Disallow: /m/search?
Disallow: /m/swmloptin?
Disallow: /m/trends
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /froogle?
Disallow: /products?
Disallow: /products/
Disallow: /froogle_
Disallow: /product_
Disallow: /products_
Disallow: /print
Disallow: /books
Disallow: /bkshp?q=
Allow: /booksrightsholders
Disallow: /patents?
Disallow: /patents/
Allow: /patents/about
Disallow: /scholar
Disallow: /complete
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /places/
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /reader/
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /notebook/search?
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/search?
Disallow: /base/reportbadoffer
Disallow: /base/s2
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Disallow: /jsapi
Disallow: /views?
Disallow: /c/
Disallow: /cbk
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /translate_a/
Disallow: /translate_c
Disallow: /translate_f
Disallow: /translate_static/
Disallow: /translate_suggestion
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/photos
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /comparisonads/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /hosted/life/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /ig/add?
Disallow: /adwordsresellers
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /toolkit/
Allow: /toolkit/*.html
Disallow: /qnasearch?
Disallow: /errors/
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml
Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml