WordPress has a virtual robots.txt file; best not to muck around with the filesystem. There are a bunch of plugins to allow you to edit the robots.txt.
I choose to use WP Robots.txt, it is simple and the least intrusive that I found, its control panel is added to Settings -> Reading dashboard panel. This is explained in the support forum.
I took advice from stackflow and spiderwebpress to add the following line
sitemap: http://wiki.davelevy.info/sitemap.xml
The spiderweb article has some more advice about disallowing the filesystem since the content isn’t there.
One Comment