{"id":2065,"date":"2014-03-04T00:11:18","date_gmt":"2014-03-04T00:11:18","guid":{"rendered":"http:\/\/wiki.davelevy.info\/?p=2065"},"modified":"2018-07-13T11:49:02","modified_gmt":"2018-07-13T11:49:02","slug":"wordpress-and-robots-txt","status":"publish","type":"post","link":"https:\/\/davelevy.info\/wiki\/wordpress-and-robots-txt\/","title":{"rendered":"WordPress and robots.txt"},"content":{"rendered":"<p>WordPress has a virtual robots.txt file; best not to muck around with the filesystem. There are a bunch of plugins to allow you to edit the robots.txt.<!--more--><\/p>\n<p>I choose to use <a href=\"http:\/\/wordpress.org\/plugins\/wp-robots-txt\/screenshots\/\">WP Robots.txt<\/a>, it is simple and the least intrusive that I found, its control panel is added to Settings -&gt; Reading dashboard panel. This is explained in the support forum.<\/p>\n<p>I took advice from stackflow and <a href=\"https:\/\/web.archive.org\/web\/20150307043619\/http:\/\/www.spiderwebpress.com:80\/wordpress-tutorials\/using-a-robots-txt-file-with-wordpress\/\">spiderwebpress<\/a> to add the following line<\/p>\n<p><code>sitemap: http:\/\/wiki.davelevy.info\/sitemap.xml<\/code><\/p>\n<p>The spiderweb article has some more advice about disallowing the filesystem since the content isn&#8217;t there.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>WordPress has a virtual robots.txt file; best not to muck around with the filesystem. There are a bunch of plugins to allow you to edit the robots.txt.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","_share_on_mastodon":"0"},"categories":[3,584],"tags":[637,144,143,911,28],"class_list":["post-2065","post","type-post","status-publish","format-standard","hentry","category-technology","category-wordpress-technology","tag-robots-txt","tag-search","tag-seo","tag-technology","tag-wordpress"],"share_on_mastodon":{"url":"","error":""},"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/posts\/2065","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/comments?post=2065"}],"version-history":[{"count":5,"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/posts\/2065\/revisions"}],"predecessor-version":[{"id":3797,"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/posts\/2065\/revisions\/3797"}],"wp:attachment":[{"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/media?parent=2065"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/categories?post=2065"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/davelevy.info\/wiki\/wp-json\/wp\/v2\/tags?post=2065"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}