A robot is a program that automatically traverses the Web by retrieving documents from servers. Normal Web browsers are not robots, because they are operated by a human, and don't automatically retrieve documents . Web robots are sometimes referred to as Web Wanderers, Web Crawlers, or Spiders. Search engines amongst other things use these robots to fill up their search results.

As robots hit your site you are able to tell the various robots which sections or areas of your site you want indexed or not. That is you can tell the search engine spiders/crawlers not to index your personal folders as an example, or in fact you could tell it to not index your entire site.

The actual robot.txt file is a simple text file that needs to be placed in your sites root folder. Meaning you place the file next to your index or homepage file.

Simple answer really is you don't need a robot.txt file, unless you want to stop certain sections of your site being crawled. Some robots will ignore this file anyway and some will even use it as an indication of which directories to have a snoop in .

Best practise though is to add a robot.txt file to your site the majority of honest robots are looking for this file and infact you will see that certain search engines such as google have specific interfaces built in to their webmaster tools. That is they have pages that allow you to edit your robot .txt file direct from there interfaces.

So why not add one in. It is quite easy to create.

I have stored a page oh how to create a robot.txt file here