Just a note of example robots.txt file that I’m using in Sitecore:
User-agent: *
Disallow: /sitecore
Disallow: /Sitecore
Disallow: /sitecore_files/
Disallow: /sitecore modules/
Disallow: /App_Browsers/
Disallow: /App_config/
Disallow: /App_Data/
Disallow: /temp/
Disallow: /upload/
Disallow: /xsl/
Sitemap: http://www.example.com/sitemap.xml
Also, don’t forget to set up your Sitecore.Analytics.ExcludeRobots.config