Can I use robots.txt to optimize Googlebot's crawl?

Channel:
United States Google Search Central
Subscribers:
751,000
Published on ● Video Link: https://www.youtube.com/watch?v=I2giR-WKUfY


Duration: 2:47
29,229 views
97

Can I use robots.txt to optimize Googlebot's crawl? For example, can I disallow all but one section of a site (for a week) to ensure it is crawled, and then revert to a 'normal' robots.txt?


Blind Five Year Old, SF, CA



Other Videos By Google Search Central


2010-09-13Are there any SEO certification programs you are aware of?
2010-09-07Is it OK to sell links as long as we use the nofollow attribute?
2010-09-01Why are links used in ranking when they all have the nofollow attribute?
2010-08-31How should I configure If-Modified-Since on database-driven pages?
2010-08-30What's a good way to kick off a new RSS feed?
2010-08-26How do PageRank updates work?
2010-08-25Is it better to have keywords in the URL path or filename?
2010-08-23How are sitelinks generated?
2010-08-18Which Chrome extensions are popular at Google?
2010-08-17Can a site's downtime affect its ranking?
2010-08-16Can I use robots.txt to optimize Googlebot's crawl?
2010-08-11Are there any plans to implement a "negative keyword" meta tag?
2010-08-10Does PageRank take into account cross-browser compatibility?
2010-08-09Why doesn't search query data in Webmaster Tools appear to match what I see when searching?
2010-08-04Can a spammy website affect the ranking of other sites hosted on the same server?
2010-08-03Do tag clouds help or hinder SEO?
2010-08-02Are Chrome's 'usage statistics' used in evaluating site speed?
2010-07-28Will adding my Twitter feed to my website increase my PageRank?
2010-07-27Can you make the realtime results more conspicuous?
2010-07-26Should I remove widgets that increase my site's load time?
2010-07-21Where do you see Google in 5 years?


Tags:
google
seo
robots.txt
crawling