Why might Googlebot get errors when trying to access my robots.txt file?

Channel:
United States Google Search Central
Subscribers:
751,000
Published on ● Video Link: https://www.youtube.com/watch?v=XMyz7-D96vQ


Duration: 1:33
8,386 views
46

I'm getting errors from Google Webmaster Tools about the Googlebot crawler being unable to fetch my robots.txt 50% of the time (but I can fetch it with 100% success rate from various other hosts). (On a plain old nginx server and an mit.edu host.)
Yang, Palo Alto, CA

Fetch as Google:
http://support.google.com/webmasters/bin/answer.py?answer=158587

Have a question? Ask it in our Webmaster Help Forum: http://groups.google.com/a/googleproductforums.com/forum/#!forum/webmasters

Want your question to be answered on a video like this? Follow us on Twitter and look for an announcement when we take new questions: http://twitter.com/googlewmc

More videos: http://www.youtube.com/GoogleWebmasterHelp
Webmaster Central Blog: http://googlewebmastercentral.blogspot.com/
Webmaster Central: http://www.google.com/webmasters/



Other Videos By Google Search Central


2012-12-12What should be included in a proper reconsideration request?
2012-12-12Introduction to Data Highlighter
2012-12-11What are the advantages of 301 redirects over rel="canonical"?
2012-12-10How can I make the pages on my site unique?
2012-12-05HTML meta tag verification in Webmaster Tools
2012-12-05Will Google get a webspam team outside of the US?
2012-12-04Why might the estimated number of results change when going from page 1 to page 2?
2012-12-03How many messages did Google send about unnatural links?
2012-11-28Should I incorporate synonyms for important terms into my site?
2012-11-27Why do paid links violate Google's guidelines while other ads don't?
2012-11-26Why might Googlebot get errors when trying to access my robots.txt file?
2012-11-21What has having your own blog taught you about SEO?
2012-11-20Do you think that "Search Engine Optimization" should be renamed?
2012-11-19How long does a reconsideration request take to process?
2012-11-14How will Google interpret links to URLs ending with a campaign tag?
2012-11-13How does Google consider site-wide backlinks?
2012-11-12Does Google take action on spammy guest blogging activities?
2012-11-07Should I keep a domain parked without content before I launch the website?
2012-11-06Why does Google shut down products?
2012-11-05Can I fetch an https URL as Googlebot in Webmaster Tools?
2012-10-31Should I structure my site using subdomains or subdirectories?


Tags:
google
seo
webmaster tools
robots.txt
googlebot
fetch as google