Revert "FEATURE: add Noindex to robots.txt for disallowed routes"

Revert “FEATURE: add Noindex to robots.txt for disallowed routes”

This reverts commit d84256a876a9fa4fc7bcb4b8ac8c5865f8c10701.

This is not supported by Google and causes robots.txt to be flagged as invalid

Removing Noindex

diff --git a/app/views/robots_txt/index.erb b/app/views/robots_txt/index.erb
index d86f72e..71ca94b 100644
--- a/app/views/robots_txt/index.erb
+++ b/app/views/robots_txt/index.erb
@@ -10,7 +10,6 @@ Crawl-delay: <%= agent[:delay] %>
 <%- end -%>
 <% agent[:disallow].each do |path| %>
 Disallow: <%= path %>
-Noindex: <%= path %>
 <% end %>
 
 
diff --git a/spec/requests/robots_txt_controller_spec.rb b/spec/requests/robots_txt_controller_spec.rb
index 413e806..edd79c4 100644
--- a/spec/requests/robots_txt_controller_spec.rb
+++ b/spec/requests/robots_txt_controller_spec.rb
@@ -52,7 +52,6 @@ RSpec.describe RobotsTxtController do
         Discourse.stubs(:base_uri).returns('/forum')
         get '/robots.txt'
         expect(response.body).to include("\nDisallow: /forum/admin")
-        expect(response.body).to include("\nNoindex: /forum/admin")
       end
     end

GitHub sha: 5feb3429

1 Like

This commit has been mentioned on Discourse Meta. There might be relevant details there: