Create A Robots.txt File

There are a few ways to create a robots.txt file for the webbots that will want to crawl your site.
Searchbots look into your site to know what you have in there so that the search engine can comeback and find it to answer a search.

Open Notepad and type this exactly if you want no robots to look at your site. Robots can ignore your request, bad robots usually do.

User-agent: *
Disallow: /

Save the robots.txt file to your server along with your webpages.
Save it as robots.txt

 Or if you want a robot to see everything then type this.

User-agent: *
Disallow:

To exclude some sections of the site you can type something like this:

In the case shown above you would add the section to be excluded.

 To allow robots to look around your site but to direct them to your xml sitemap specifically type this:

User-agent: *
Allow: /

Sitemap: http://yourwebsite.com/sitemap.xml

Obviously you would replace yourwebsite.com with the actual address of your site.

This is helpful to the sitemap crawlers