Should I block duplicate pages using robots.txt?

Channel:
United States Google Search Central
Subscribers:
751,000
Published on ● Video Link: https://www.youtube.com/watch?v=CJMFYpYQZ0c


Duration: 2:13
12,142 views
29

Halfdeck from Davis, CA asks: "If Google crawls 1,000 pages/day, Googlebot crawling many dupe content pages may slow down indexing of a large site. In that scenario, do you recommend blocking dupes using robots.txt or is using META ROBOTS NOINDEX,NOFOLLOW a better alternative?"

Short answer: No, don't block them using robots.txt. Learn more about duplicate content here: http://www.google.com/support/webmasters/bin/answer.py?answer=66359



Other Videos By Google Search Central


2010-03-29Are there any negatives to generating a short list of 'related posts?'
2010-03-25Would you accept a job at Bing if the opportunity arose?
2010-03-24Can I make sure Google always uses my meta description tags for snippets?
2010-03-23How can I get Google to index more of my Sitemap URLs?
2010-03-22Should we be worried about a penalty if we quickly removed hacked content?
2010-03-18What do Googlers usually talk about?
2010-03-17Will including generic product descriptions reduce the significance of my unique content?
2010-03-16At what level is nofollow honored?
2010-03-15Can we serve Googlebot a different page with no ads?
2010-03-11Does Google think the world will end in 2012?
2010-03-10Should I block duplicate pages using robots.txt?
2010-03-09Does Google treat "brandname" and "brandname®" differently?
2010-03-08Does the PageRank of Twitter profiles matter?
2010-03-04What are some effective techniques for building links?
2010-03-03Are you on the Google PR team?
2010-03-02Does conference attendance affect rankings?
2010-03-01Should we create a mobile version of our site?
2010-02-25How will Google search work with pages built with GWT?
2010-02-24If I use multiple versions of a phrase, would it be seen as keyword spam?
2010-02-23How can I add more than 1,000,000 URLs to a Sitemap?
2010-02-22Can having dofollow comments on my blog affect its reputation?


Tags:
google
seo
robots.txt
duplicate content