This is a complete guide to SEO structure and optimization by Google. Page 21 deals specifically with robots.txt.This rule tells crawlers if they can access parts of your site. If you do not want them to access that particular section you can limit their access by using these rules. This can save users from finding non-helpful information.
Page 24 explains how to accurate modify your site to be searched and indexed properly by Google for mobile. The document contains links for further instructions.