Posted by Dave on

WordPress and robots.txt

WordPress has a virtual robots.txt file; best not to muck around with the filesystem. There are a bunch of plugins to allow you to edit the robots.txt.

I choose to use WP Robots.txt, it is simple and the least intrusive that I found, its control panel is added to Settings -> Reading dashboard panel. This is explained in the support forum.

I took advice from stackflow and spiderwebpress to add the following line

sitemap: http://wiki.davelevy.info/sitemap.xml

The spiderweb article has some more advice about disallowing the filesystem since the content isn’t there.


Comment ( 1 )

  1. Google Tools and Analytics | wiki.davelevy.info
    […] Having created sitemaps, I want these to be advertised in the robots.txt file. See WordPress and robots.txt. […]

This site uses Akismet to reduce spam. Learn how your comment data is processed.